Supervised local runs

startSupervisedRun from @obversa/runner starts a graph in a separate worker process. The calling process acts as its watchdog: it waits for the worker, stops discoverable child processes, and restarts work within declared time and restart limits.

Run the offline example

From a built Obversa checkout, run:

pnpm --filter @obversa/runner exec tsx ../../examples/packages/supervised-run.ts

The example creates a disposable Git repository and runs two data nodes without an engine or network access. It checks their files and saved results, prints a JSON report, and removes its temporary directory after teardown succeeds. examples/packages/supervised-host.mjs supplies the worker’s node bindings.

Start, inspect, and stop

import {
  readSupervisedRunStatus,
  startSupervisedRun,
  type SupervisedRunOptions,
} from '@obversa/runner';

async function run(options: SupervisedRunOptions) {
  const handle = await startSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}

async function inspect(options: SupervisedRunOptions) {
  return await readSupervisedRunStatus({
    storage: options.storage,
    runId: options.definition.runId,
  });
}

Call handle.stop() while work is running to request a stop and await cleanup. A clean stop without a recorded worker result returns kind: 'fail' and code: 'STOPPED'. Calling it after completion returns the settled result. handle.done resolves to a graph result: complete, pause, or fail. Infrastructure errors can reject the promise. After verified process cleanup and workspace lease release, the watchdog returns the current worker’s durably recorded result even if a stop request or timeout arrives before worker exit. Without a recorded result, stop and timeout return STOPPED and TIMEOUT. Cleanup or lease-release failures take precedence over the recorded result. Completion output is stored once as an artifact, subject to the storage policy’s artifact size and quota limits. Worker and watchdog completion events carry the same artifact reference, not the output bytes. The watchdog verifies the stored bytes against that reference before returning done.output. An unreadable or corrupt output artifact returns kind: 'fail' with code: 'TERMINAL_ARTIFACT'. Successful and incomplete engine calls also store their response parts as runner-engine-parts artifacts. Supervision events carry partsArtifact references instead of those parts. Each artifact must fit storage.policy.maxArtifactBytes; distinct stored parts consume the shared storage.policy.maxTotalArtifactBytesPerRun quota alongside the run’s other artifacts. A response can fit the node’s output limit and still exceed either storage limit. Engine parts use contentMode: 'state'. The store rejects parts containing a configured knownSecrets value instead of redacting them. Size or quota refusal raises StorageError with code: 'STORAGE_LIMIT_EXCEEDED'; a known-secret match raises code: 'KNOWN_SECRET'. Inside the worker, a refused parts write becomes EngineIncompleteResultError, even when the engine returned successfully. Rejected parts are not durable, and status reports unknown usage when the engine’s final supervision event could not be written. The caller does not receive those storage codes directly from handle.done. For an otherwise valid engine result within the node’s limits, the runtime records graph:node-failed with code: 'EFFECT_FAILED'. With the built-in DAG form, a failed required node makes handle.done resolve to kind: 'fail' with code: 'DAG_NODE_FAILED', and status reports phase: 'failed'. Other graph forms decide their own response to a failed node. The status reader can run in another process with the same storage settings. It reads recorded progress and inspects the worker’s process identity. It does not take ownership of the run or stop another process’s watchdog. A stored run without a host binding rejects with SupervisedRunError and code: 'HOST_MODULE'.

Supply a host module

The runner loads host modules and supervises processes. The runtime receives bound graph, node, and engine instances; it does not load host modules. Set runRoot to the working directory and module to a relative file specifier, such as ./host.mjs. The file must remain inside runRoot after resolving symlinks; a path outside it is refused with code: 'HOST_MODULE'. Every filesystem error while resolving the selected entry path is rejected with code: 'HOST_MODULE' and the original error as its cause. This includes a missing file, an unreadable directory, a non-directory path component, or a symlink loop. This error boundary does not cover resolving runRoot itself. The module must export:

export async function bindRun({ definition, scratchDirectory }) {
  // Compile definition.graphDefinition.value with its graph form.
  // Return { graph, nodes, engines }, plus memory if the form requires it.
}

The runner stores the host module’s path and byte digest. Each worker checks the digest before import and again after import. Changed module bytes produce a HOST_MODULE_CHANGED failure. Each worker imports the module once and calls bindRun with the stored definition and scratch directory. The digest covers the selected module file only. Transitive imports are not pinned. An edit after a worker has imported the module does not replace that worker’s loaded bindings. The stored graph definition, resolved plan, and resolved inputs supply the execution contract. The worker loads them from storage. Its bindings must match that contract; it does not reconstruct the run from mutable command-line inputs. By default, the values copied from the watchdog environment into the worker are the present values of PATH, HOME, TMPDIR, TMP, TEMP, SystemRoot, USERPROFILE, and PATHEXT. Arbitrary parent variables, including API keys and NODE_OPTIONS, are not inherited. The owned-command layer separately sets OBVERSA_ATTEMPT_ID, OBVERSA_RUN_ID, OBVERSA_HEADLESS, and OBVERSA_RUN_OWNER for runner identity and process cleanup. To supply engine credentials, list the required environment variable names in environmentVariables on startSupervisedRun or resumeSupervisedRun:

const handle = await startSupervisedRun({
  ...options,
  environmentVariables: ['ANTHROPIC_API_KEY'],
});

The runner copies each listed variable only when it is present in the watchdog environment. It snapshots the list at each start or resume call. Supply the list again when resuming; credentials are not stored in run inputs or host records. resolvedInputs is durable storage. Do not put secrets in it. The workspace provider must capture the same root as runRoot. Capture and storage preflight failures start no worker and write no run. If storage fails during publication, startup returns RUN_STORAGE and starts no worker. Artifacts or a start event that were published before the error remain available. The runner never deletes that evidence or promises a transaction across both stores.

Recover after a worker crash

The watchdog waits for worker exit and cleanup before starting a replacement. The replacement loads the stored run and folds its graph events to decide what remains.

Crash boundary	Recovery
Before a dispatch is recorded	Decide and dispatch the work.
After dispatch, before node code starts	Start the recorded occurrence.
After node code starts, before its result is recorded	Pause for reconciliation unless the binding explicitly declares `retrySafe: true`.
After the result is recorded	Continue without repeating the completed occurrence.

Set retrySafe: true only when repeating the node is acceptable after an unknown outcome. A saved start without a saved result does not prove whether an outward effect happened. For an unsafe repeat, the executor records a pause with a reconcile-attempt request. Unattended action decisions must return allow before node effects run. A host decision to wait records a graph pause with its reason and request. A decision to deny records a failed node. The worker does not open a prompt.

Resume a paused run

resumeSupervisedRun reopens one recorded graph pause. Pass the same storage settings and workspace provider, the run ID, and the exact position from its graph:node-paused event. Supply the runner directory, worker root, restart policy, and cleanup grace as for a start:

import {
  resumeSupervisedRun,
  type ResumeSupervisedRunOptions,
} from '@obversa/runner';

async function resume(options: ResumeSupervisedRunOptions) {
  const handle = await resumeSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}

The definition, host module, and run limits come from the stored run; resume does not accept replacements. It writes no second run-start event and does not repeat completed occurrences. Status shows a resumed worker as active rather than reporting the earlier pause as its terminal result. A resume is bound to one pause event. If a replacement worker finds a different pause event at the same position, handle.done resolves to kind: 'pause' with code: 'RESUME_EVENT_MISMATCH'. Its reason preserves the exact message: Resume expected pause event "<expected>" but found "<found>". Both placeholders contain the recorded event IDs. After worker cleanup, the watchdog saves the workspace anchor and records runner:paused. A later explicit resumeSupervisedRun call can reopen that pause. The replacement does not resume the newer pause or approve reconciliation automatically. The elapsed budget freezes at the settled runner pause and resumes at the next stored worker launch. Resume preflight, including workspace checks, does not spend that budget. Earlier execution and restart backoff remain spent across repeated pauses and worker replacements. Logical dispatches and recorded worker restarts also keep their counts. An exhausted elapsed budget returns TIMEOUT without starting a worker. The host’s action policy runs again. Record approval in the system that policy reads before calling resume. A repeated wait pauses again with its request; calling resume is not an approval flag. For a reconcile-attempt pause, the caller must decide whether offering the uncertain work again is safe. An explicit resume may repeat an effect whose completion was not recorded; completed occurrences are not repeated. A worker replacement does not automatically approve a new reconciliation pause. Before releasing ownership after a pause, the watchdog cleans up the worker and saves a workspace snapshot as a runner-pause-anchor artifact. Resume acquires the process lock and workspace lease, then checks that saved snapshot before starting a worker. It never adopts edits made while paused. The snapshot shares the run’s artifact size, quota, and sensitive-content rules. If pause-time capture, the artifact write, or its runner:pause-anchor event append fails, handle.done returns kind: 'fail' with code: 'WORKSPACE_ANCHOR_WRITE'. A stored artifact remains available when the event append fails, but no resumable runner pause is written. This failure is terminal; the runner has no anchor-persistence retry path. Initial workspace capture failures remain WORKSPACE_CAPTURE. A workspace-drift pause saves the earlier anchor, without capturing the changed workspace. Workspace evidence failures resolve handle.done to kind: 'pause':

WORKSPACE_ANCHOR_MISSING: the pause has no saved snapshot, including pauses recorded without this artifact.
WORKSPACE_ANCHOR_INVALID: the saved reference or snapshot cannot be verified. Invalid evidence is refused before acquiring a workspace lease.
WORKSPACE_DRIFT: the workspace differs from the saved snapshot.

An occupied process lock rejects startup with PROCESS_LOCKED; an unavailable workspace lease rejects with WORKSPACE_LEASE. A run without a safely released pause rejects with RUN_NOT_PAUSED. A position that is not paused rejects with RESUME_POSITION. Resume does not take over a dead watchdog’s retained lock.

Bound the run

Elapsed time: limits.timeoutMs covers worker execution and restart backoff across pauses and resumes. Counting starts at the stored run timestamp, freezes at each settled runner pause, and resumes at the next stored worker launch. Teardown can continue beyond that work deadline.
Logical dispatches: limits.maxDispatches counts recorded graph dispatches across workers. Resuming one recorded occurrence does not create another dispatch.
Restarts: restart.maxRestarts caps replacement workers. Backoff grows from initialBackoffMs up to maxBackoffMs.
Cleanup grace: teardownGraceMs allows processes to stop before forceful termination.

Each worker launch has a fixed 1 MB (1,000,000 bytes) limit across its combined stdout and stderr. When excess output triggers this limit, the watchdog stops the worker without restarting it. After successful process cleanup, handle.done resolves to kind: 'fail' with code: 'OUTPUT_LIMIT', and status reports phase: 'failed'. If cleanup reports surviving processes, handle.done instead resolves to kind: 'fail' with code: 'TEARDOWN_INCOMPLETE', and the watchdog retains its workspace lease and process lock. This limit applies to worker console output. Engine response parts and graph completion output use the separate artifact size and quota limits described above. SupervisedRunOptions exposes no setting for the worker output cap or a worker memory ceiling. The watchdog passes Number.MAX_SAFE_INTEGER as its worker memory limit; it does not enforce a practical memory budget for the worker. Token budgets apply per worker process and do not enforce a shared token cap across worker restarts.

Read progress

Status includes the run phase, worker liveness, inspected processes, restart count, backoff, elapsed execution time, remaining timeout, and pause reasons. Run-level elapsed time and remaining timeout stay fixed during settled pauses and resume preflight. They use the same execution budget as the watchdog. cleanupVerified is null before a terminal result, true when cleanup was verified within the reported capability, and false when cleanup could not be verified. leaseRetained reports a lease held after terminal failure. A dead worker or an empty process list does not prove cleanup succeeded; inspect these fields before treating the workspace as released. active contains all recorded active node occurrences while the worker is alive. Each entry has its node ID, unchanged position, plan phase, start time, elapsed time, remaining node timeout, and usage state. Each active entry uses its node’s total. The top-level usage list retains node totals after completion. The runner exports SupervisedRunUsage with three states:

Reported: Every engine call has a measured receipt.
Partial: At least one call has a measured receipt and at least one does not. inputTokens and outputTokens are measured lower bounds; unknownCalls counts calls without receipts, including still-open calls.
Unknown: No call has a measured receipt, or measured totals exceed the safe integer range.

Missing usage is not zero. A later successful retry adds its measured tokens without erasing the gap left by a crashed call. Measured receipts from failed calls also contribute to the totals. Graph phases group nodes for display. They do not define execution parents. Position strings identify occurrences; the status reader does not split them into a parent chain or choose a supposed deepest node. Checkpoints cache display state only. Missing, stale, or malformed checkpoints can be rebuilt from graph events. The executor always makes its decisions from the events, never from a status checkpoint.

Process cleanup and workspace ownership

The watchdog uses the public @obversa/engine/command API. Command and cleanup requests accept an optional ownerId, a SHA-256 digest used to track the outer command’s processes. The command passes it to child processes as OBVERSA_RUN_OWNER. Nested commands keep a valid inherited marker instead of replacing it with their own ownerId. During Linux cleanup, only the outer command uses that marker to discover processes to stop. commandCleanupCapability() returns the CommandCleanupCapability type: 'inherited-owner' on Linux or 'observed-processes' elsewhere. inspectOwnerMarkedProcesses(ownerId) returns matching process identities on Linux and an empty list elsewhere. An empty list on another platform does not prove that the owner’s processes have stopped. cleanupCapability states the cleanup scope:

inherited-owner on Linux: cleanup also searches for processes carrying the worker’s inherited owner marker. Commands that remove or forge markers are outside this cleanup scope.
observed-processes on other platforms: cleanup follows observed process identities and the platform’s process-tree mechanisms.

Linux owner cleanup has an implementation and tests, but its Linux-specific tests have not run on Linux for this release, so Linux cleanup is best-effort. On macOS, a helper that moves into a new session before the watchdog samples it is not swept. The watchdog holds the workspace lease token while the worker runs and while cleanup executes. After a crash that permits restart, it captures and records a fresh workspace snapshot while still holding the lease. It then releases the lease for backoff. A replacement must acquire a lease and verify that snapshot before starting. Changes during the released interval produce a pause with code: 'WORKSPACE_DRIFT'. This restart snapshot includes workspace changes left by the stopped worker; it does not replace the stored graph plan or inputs. A terminal result releases the lease after successful cleanup. Incomplete cleanup or failed lease release retains the process lock instead of starting another worker. The process lock is scoped to the storage directory, namespace, and run ID. A second watchdog for that stored run receives PROCESS_LOCKED, even if it uses another scratch directory. Worker-crash recovery requires a live watchdog. If the watchdog dies, its lock and any held lease remain. A new watchdog does not force takeover or infer that the workspace is safe from a missing PID. This API has no automatic recovery for watchdog death.

Start

Concepts

Workflows

Reviewing

Recording runs

Driving runs

Writing workflow shapes

Memory

Workspace

Core packages

Plugins

Supervised local runs

Run the offline example

Start, inspect, and stop

Supply a host module

Recover after a worker crash

Resume a paused run

Bound the run

Read progress

Process cleanup and workspace ownership

​Run the offline example

​Start, inspect, and stop

​Supply a host module

​Recover after a worker crash

​Resume a paused run

​Bound the run

​Read progress

​Process cleanup and workspace ownership

Run the offline example

Start, inspect, and stop

Supply a host module

Recover after a worker crash

Resume a paused run

Bound the run

Read progress

Process cleanup and workspace ownership