Supervised local runs

startSupervisedRun from @obversa/runner starts a graph in a separate worker process. The calling process acts as its watchdog: it waits for the worker, stops discoverable child processes, and restarts work within declared time and restart limits.

Run the offline example

From a built Obversa checkout, run:

pnpm --filter @obversa/runner exec tsx ../../examples/packages/supervised-run.ts

The example creates a disposable Git repository and runs two data nodes without an engine or network access. It checks their files and saved results, prints a JSON report, and removes its temporary directory after teardown succeeds. examples/packages/supervised-host.mjs supplies the worker’s node bindings.

Start, inspect, and stop

import {
  readSupervisedRunStatus,
  startSupervisedRun,
  type SupervisedRunOptions,
} from '@obversa/runner';

async function run(options: SupervisedRunOptions) {
  const handle = await startSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}

async function inspect(options: SupervisedRunOptions) {
  return await readSupervisedRunStatus({
    storage: options.storage,
    runId: options.definition.runId,
  });
}

Call handle.stop() while work is running to request a stop and await cleanup. A clean stop returns kind: 'fail' and code: 'STOPPED'. Calling it after completion returns the settled result. handle.done resolves to a graph result: complete, pause, or fail. Infrastructure errors can reject the promise. Completion output is stored once as an artifact, subject to the storage policy’s artifact size and quota limits. Worker and watchdog completion events carry the same artifact reference, not the output bytes. The watchdog verifies the stored bytes against that reference before returning done.output. An unreadable or corrupt output artifact returns kind: 'fail' with code: 'TERMINAL_ARTIFACT'. Successful and incomplete engine calls also store their response parts as runner-engine-parts artifacts. Supervision events carry partsArtifact references instead of those parts. Each artifact must fit storage.policy.maxArtifactBytes; distinct stored parts consume the shared storage.policy.maxTotalArtifactBytesPerRun quota alongside the run’s other artifacts. A response can fit the node’s output limit and still exceed either storage limit. Engine parts use contentMode: 'state'. The store rejects parts containing a configured knownSecrets value instead of redacting them. Size or quota refusal raises StorageError with code: 'STORAGE_LIMIT_EXCEEDED'; a known-secret match raises code: 'KNOWN_SECRET'. Inside the worker, a refused parts write becomes EngineIncompleteResultError, even when the engine returned successfully. Rejected parts are not durable, and status reports unknown usage when the engine’s final supervision event could not be written. The caller does not receive those storage codes directly from handle.done. For an otherwise valid engine result within the node’s limits, the runtime records graph:node-failed with code: 'EFFECT_FAILED'. With the built-in DAG form, a failed required node makes handle.done resolve to kind: 'fail' with code: 'DAG_NODE_FAILED', and status reports phase: 'failed'. Other graph forms decide their own response to a failed node. The status reader can run in another process with the same storage settings. It reads recorded progress and inspects the worker’s process identity. It does not take ownership of the run or stop another process’s watchdog. A stored run without a host binding rejects with SupervisedRunError and code: 'HOST_MODULE'.

Supply a host module

The runner loads host modules and supervises processes. The runtime receives bound graph, node, and engine instances; it does not load host modules. Set runRoot to the working directory and module to a relative file specifier, such as ./host.mjs. The file must remain inside runRoot after resolving symlinks; a path outside it is refused with code: 'HOST_MODULE'. The module must export:

export async function bindRun({ definition, scratchDirectory }) {
  // Compile definition.graphDefinition.value with its graph form.
  // Return { graph, nodes, engines }, plus memory if the form requires it.
}

The runner stores the host module’s path and byte digest. Each worker checks the digest before import and again after import. Changed module bytes produce a HOST_MODULE_CHANGED failure. Each worker imports the module once and calls bindRun with the stored definition and scratch directory. The digest covers the selected module file only. Transitive imports are not pinned. An edit after a worker has imported the module does not replace that worker’s loaded bindings. The stored graph definition, resolved plan, and resolved inputs supply the execution contract. The worker loads them from storage. Its bindings must match that contract; it does not reconstruct the run from mutable command-line inputs. The workspace provider must capture the same root as runRoot. Capture and storage preflight failures start no worker and write no run. If storage fails during publication, startup returns RUN_STORAGE and starts no worker. Artifacts or a start event that were published before the error remain available. The runner never deletes that evidence or promises a transaction across both stores.

Recover after a worker crash

The watchdog waits for worker exit and cleanup before starting a replacement. The replacement loads the stored run and folds its graph events to decide what remains.

Crash boundary	Recovery
Before a dispatch is recorded	Decide and dispatch the work.
After dispatch, before node code starts	Start the recorded occurrence.
After node code starts, before its result is recorded	Pause for reconciliation unless the binding explicitly declares `retrySafe: true`.
After the result is recorded	Continue without repeating the completed occurrence.

Set retrySafe: true only when repeating the node is acceptable after an unknown outcome. A saved start without a saved result does not prove whether an outward effect happened. For an unsafe repeat, the executor records a pause with a reconcile-attempt request. Unattended action decisions must return allow before node effects run. A host decision to wait records a graph pause with its reason and request. A decision to deny records a failed node. The worker does not open a prompt.

Resume a paused run

resumeSupervisedRun reopens one recorded graph pause. Pass the same storage settings and workspace provider, the run ID, and the exact position from its graph:node-paused event. Supply the runner directory, worker root, restart policy, and cleanup grace as for a start:

import {
  resumeSupervisedRun,
  type ResumeSupervisedRunOptions,
} from '@obversa/runner';

async function resume(options: ResumeSupervisedRunOptions) {
  const handle = await resumeSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}

The definition, host module, and run limits come from the stored run; resume does not accept replacements. It writes no second run-start event and does not repeat completed occurrences. Status shows a resumed worker as active rather than reporting the earlier pause as its terminal result. The original wall-clock deadline includes time spent paused. Logical dispatches and recorded worker restarts also keep their counts. A pause does not restore spent budgets; resuming after the deadline returns TIMEOUT without a worker. The host’s action policy runs again. Record approval in the system that policy reads before calling resume. A repeated wait pauses again with its request; calling resume is not an approval flag. For a reconcile-attempt pause, the caller must decide whether offering the uncertain work again is safe. A worker replacement does not automatically approve a new reconciliation pause. Before releasing ownership after a pause, the watchdog cleans up the worker and saves a workspace snapshot as a runner-pause-anchor artifact. Resume acquires the process lock and workspace lease, then checks that saved snapshot before starting a worker. It never adopts edits made while paused. The snapshot shares the run’s artifact size, quota, and sensitive-content rules; a failed write cannot produce a resumable pause. Workspace evidence failures resolve handle.done to kind: 'pause':

WORKSPACE_ANCHOR_MISSING: the pause has no saved snapshot, including pauses recorded without this artifact.
WORKSPACE_ANCHOR_INVALID: the saved reference or snapshot cannot be verified. Invalid evidence is refused before acquiring a workspace lease.
WORKSPACE_DRIFT: the workspace differs from the saved snapshot.

An occupied process lock rejects startup with PROCESS_LOCKED; an unavailable workspace lease rejects with WORKSPACE_LEASE. A run without a safely released pause rejects with RUN_NOT_PAUSED. A position that is not paused rejects with RESUME_POSITION. Resume does not take over a dead watchdog’s retained lock.

Bound the run

Elapsed time: limits.timeoutMs covers worker execution and restart backoff. Teardown can continue beyond that work deadline.
Logical dispatches: limits.maxDispatches counts recorded graph dispatches across workers. Resuming one recorded occurrence does not create another dispatch.
Restarts: restart.maxRestarts caps replacement workers. Backoff grows from initialBackoffMs up to maxBackoffMs.
Cleanup grace: teardownGraceMs allows processes to stop before forceful termination.

Each worker launch has a fixed 1 MB (1,000,000 bytes) limit across its combined stdout and stderr. When excess output triggers this limit, the watchdog stops the worker without restarting it. After successful process cleanup, handle.done resolves to kind: 'fail' with code: 'OUTPUT_LIMIT', and status reports phase: 'failed'. If cleanup reports surviving processes, handle.done instead resolves to kind: 'fail' with code: 'TEARDOWN_INCOMPLETE', and the watchdog retains its workspace lease and process lock. This limit applies to worker console output. Engine response parts and graph completion output use the separate artifact size and quota limits described above. SupervisedRunOptions exposes no setting for the worker output cap or a worker memory ceiling. The watchdog passes Number.MAX_SAFE_INTEGER as its worker memory limit; it does not enforce a practical memory budget for the worker. Token budgets apply per worker process and do not enforce a shared token cap across worker restarts.

Read progress

Status includes the run phase, worker liveness, inspected processes, restart count, backoff, elapsed time, remaining timeout, and pause reasons. cleanupVerified is null before a terminal result, true when cleanup was verified within the reported capability, and false when cleanup could not be verified. leaseRetained reports a lease held after terminal failure. A dead worker or an empty process list does not prove cleanup succeeded; inspect these fields before treating the workspace as released. active contains all recorded active node occurrences while the worker is alive. Each entry has its node ID, unchanged position, plan phase, start time, elapsed time, remaining node timeout, and usage state. Unknown usage is not zero. The top-level usage list keeps recorded totals by node after completion. An unreported or unfinished engine call makes that node’s total unknown. Graph phases group nodes for display. They do not define execution parents. Position strings identify occurrences; the status reader does not split them into a parent chain or choose a supposed deepest node. Checkpoints cache display state only. Missing, stale, or malformed checkpoints can be rebuilt from graph events. The executor always makes its decisions from the events, never from a status checkpoint.

Process cleanup and workspace ownership

The watchdog uses the public @obversa/engine/command API. Command and cleanup requests accept an optional ownerId, a SHA-256 digest used to track the outer command’s processes. The command passes it to child processes as OBVERSA_RUN_OWNER. Nested commands keep a valid inherited marker instead of replacing it with their own ownerId. During Linux cleanup, only the outer command uses that marker to discover processes to stop. commandCleanupCapability() returns the CommandCleanupCapability type: 'inherited-owner' on Linux or 'observed-processes' elsewhere. inspectOwnerMarkedProcesses(ownerId) returns matching process identities on Linux and an empty list elsewhere. An empty list on another platform does not prove that the owner’s processes have stopped. cleanupCapability states the cleanup scope:

inherited-owner on Linux: cleanup also searches for processes carrying the worker’s inherited owner marker. Commands that remove or forge markers are outside this cleanup scope.
observed-processes on other platforms: cleanup follows observed process identities and the platform’s process-tree mechanisms.

Linux owner cleanup has an implementation and tests, but its Linux-specific tests have not run on Linux for this release, so Linux cleanup is best-effort. On macOS, a helper that moves into a new session before the watchdog samples it is not swept. The watchdog holds the workspace lease token while the worker runs and while cleanup executes. After a crash that permits restart, it captures and records a fresh workspace snapshot while still holding the lease. It then releases the lease for backoff. A replacement must acquire a lease and verify that snapshot before starting. Changes during the released interval produce a pause with code: 'WORKSPACE_DRIFT'. This restart snapshot includes workspace changes left by the stopped worker; it does not replace the stored graph plan or inputs. A terminal result releases the lease after successful cleanup. Incomplete cleanup or failed lease release retains the process lock instead of starting another worker. The process lock is scoped to the storage directory, namespace, and run ID. A second watchdog for that stored run receives PROCESS_LOCKED, even if it uses another scratch directory. Worker-crash recovery requires a live watchdog. If the watchdog dies, its lock and any held lease remain. A new watchdog does not force takeover or infer that the workspace is safe from a missing PID. This API has no automatic recovery for watchdog death.

Start

Concepts

Memory

Workspace

Graphs

Proof

Storage

Runtime

Processes

Hosts

Packages

Supervised local runs

Run the offline example

Start, inspect, and stop

Supply a host module

Recover after a worker crash

Resume a paused run

Bound the run

Read progress

Process cleanup and workspace ownership

​Run the offline example

​Start, inspect, and stop

​Supply a host module

​Recover after a worker crash

​Resume a paused run

​Bound the run

​Read progress

​Process cleanup and workspace ownership

Run the offline example

Start, inspect, and stop

Supply a host module

Recover after a worker crash

Resume a paused run

Bound the run

Read progress

Process cleanup and workspace ownership