> ## Documentation Index
> Fetch the complete documentation index at: https://docs.obversa.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Supervised local runs

> Run a graph in a worker process, inspect its progress, and recover safely after a worker crash.

`startSupervisedRun` from `@obversa/runner` starts a graph in a separate worker process. The calling
process acts as its watchdog: it waits for the worker, stops discoverable child
processes, and restarts work within declared time and restart limits.

## Run the offline example

From a built Obversa checkout, run:

```bash theme={null}
pnpm --filter @obversa/runner exec tsx ../../examples/packages/supervised-run.ts
```

The example creates a disposable Git repository and runs two data nodes without
an engine or network access. It checks their files and saved results, prints a
JSON report, and removes its temporary directory after teardown succeeds.
`examples/packages/supervised-host.mjs` supplies the worker's node bindings.

## Start, inspect, and stop

```ts theme={null}
import {
  readSupervisedRunStatus,
  startSupervisedRun,
  type SupervisedRunOptions,
} from '@obversa/runner';

async function run(options: SupervisedRunOptions) {
  const handle = await startSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}

async function inspect(options: SupervisedRunOptions) {
  return await readSupervisedRunStatus({
    storage: options.storage,
    runId: options.definition.runId,
  });
}
```

Call `handle.stop()` while work is running to request a stop and await cleanup.
A clean stop returns `kind: 'fail'` and `code: 'STOPPED'`. Calling it after completion
returns the settled result. `handle.done` resolves to a graph result:
`complete`, `pause`, or `fail`. Infrastructure errors can reject the promise.

Completion output is stored once as an artifact, subject to the storage policy's
artifact size and quota limits. Worker and watchdog completion events carry the
same artifact reference, not the output bytes. The watchdog verifies the stored
bytes against that reference before returning `done.output`. An unreadable or
corrupt output artifact returns `kind: 'fail'` with `code: 'TERMINAL_ARTIFACT'`.

Successful and incomplete engine calls also store their response parts as
`runner-engine-parts` artifacts. Supervision events carry `partsArtifact`
references instead of those parts. Each artifact must fit
`storage.policy.maxArtifactBytes`; distinct stored parts consume the shared
`storage.policy.maxTotalArtifactBytesPerRun` quota alongside the run's other
artifacts. A response can fit the node's output limit and still exceed either
storage limit.

Engine parts use `contentMode: 'state'`. The store rejects parts containing a
configured `knownSecrets` value instead of redacting them. Size or quota refusal
raises `StorageError` with `code: 'STORAGE_LIMIT_EXCEEDED'`; a known-secret match
raises `code: 'KNOWN_SECRET'`. Inside the worker, a refused parts write becomes
`EngineIncompleteResultError`, even when the engine returned successfully.
Rejected parts are not durable, and status reports unknown usage when the
engine's final supervision event could not be written.

The caller does not receive those storage codes directly from `handle.done`.
For an otherwise valid engine result within the node's limits, the runtime
records `graph:node-failed` with `code: 'EFFECT_FAILED'`. With the built-in DAG
form, a failed required node makes `handle.done` resolve to `kind: 'fail'` with
`code: 'DAG_NODE_FAILED'`, and status reports `phase: 'failed'`. Other graph
forms decide their own response to a failed node.

The status reader can run in another process with the same storage settings.
It reads recorded progress and inspects the worker's process identity. It does
not take ownership of the run or stop another process's watchdog.
A stored run without a host binding rejects with `SupervisedRunError` and
`code: 'HOST_MODULE'`.

## Supply a host module

The runner loads host modules and supervises processes. The runtime receives
bound graph, node, and engine instances; it does not load host modules.

Set `runRoot` to the working directory and `module` to a relative file specifier,
such as `./host.mjs`. The file must remain inside `runRoot` after resolving
symlinks; a path outside it is refused with `code: 'HOST_MODULE'`.
The module must export:

```js theme={null}
export async function bindRun({ definition, scratchDirectory }) {
  // Compile definition.graphDefinition.value with its graph form.
  // Return { graph, nodes, engines }, plus memory if the form requires it.
}
```

The runner stores the host module's path and byte digest. Each worker checks the
digest before import and again after import. Changed module bytes produce a
`HOST_MODULE_CHANGED` failure. Each worker imports the module once and calls
`bindRun` with the stored definition and scratch directory.

The digest covers the selected module file only. Transitive imports are not
pinned. An edit after a worker has imported the module does not replace that
worker's loaded bindings.

The stored graph definition, resolved plan, and resolved inputs supply the
execution contract. The worker loads them from storage. Its bindings must match
that contract; it does not reconstruct the run from mutable command-line inputs.

The workspace provider must capture the same root as `runRoot`. Capture and
storage preflight failures start no worker and write no run. If storage fails
during publication, startup returns `RUN_STORAGE` and starts no worker. Artifacts
or a start event that were published before the error remain available. The
runner never deletes that evidence or promises a transaction across both stores.

## Recover after a worker crash

The watchdog waits for worker exit and cleanup before starting a replacement.
The replacement loads the stored run and folds its graph events to decide what
remains.

| Crash boundary                                        | Recovery                                                                           |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Before a dispatch is recorded                         | Decide and dispatch the work.                                                      |
| After dispatch, before node code starts               | Start the recorded occurrence.                                                     |
| After node code starts, before its result is recorded | Pause for reconciliation unless the binding explicitly declares `retrySafe: true`. |
| After the result is recorded                          | Continue without repeating the completed occurrence.                               |

Set `retrySafe: true` only when repeating the node is acceptable after an unknown
outcome. A saved start without a saved result does not prove whether an outward
effect happened. For an unsafe repeat, the executor records a pause with a
`reconcile-attempt` request.

Unattended action decisions must return `allow` before node effects run. A host
decision to `wait` records a graph pause with its reason and request. A decision
to `deny` records a failed node. The worker does not open a prompt.

## Resume a paused run

`resumeSupervisedRun` reopens one recorded graph pause. Pass the same storage
settings and workspace provider, the run ID, and the exact `position` from its
`graph:node-paused` event. Supply the runner directory, worker root, restart
policy, and cleanup grace as for a start:

```ts theme={null}
import {
  resumeSupervisedRun,
  type ResumeSupervisedRunOptions,
} from '@obversa/runner';

async function resume(options: ResumeSupervisedRunOptions) {
  const handle = await resumeSupervisedRun(options);
  try {
    console.log(await handle.status());
    return await handle.done;
  } finally {
    await handle.stop();
  }
}
```

The definition, host module, and run limits come from the stored run; resume
does not accept replacements. It writes no second run-start event and does not
repeat completed occurrences. Status shows a resumed worker as active rather
than reporting the earlier pause as its terminal result.

The original wall-clock deadline includes time spent paused. Logical dispatches
and recorded worker restarts also keep their counts. A pause does not restore
spent budgets; resuming after the deadline returns `TIMEOUT` without a worker.

The host's action policy runs again. Record approval in the system that policy
reads before calling resume. A repeated `wait` pauses again with its request;
calling resume is not an approval flag. For a `reconcile-attempt` pause, the
caller must decide whether offering the uncertain work again is safe. A worker
replacement does not automatically approve a new reconciliation pause.

Before releasing ownership after a pause, the watchdog cleans up the worker and
saves a workspace snapshot as a `runner-pause-anchor` artifact. Resume acquires
the process lock and workspace lease, then checks that saved snapshot before
starting a worker. It never adopts edits made while paused. The snapshot shares
the run's artifact size, quota, and sensitive-content rules; a failed write
cannot produce a resumable pause.

Workspace evidence failures resolve `handle.done` to `kind: 'pause'`:

* **`WORKSPACE_ANCHOR_MISSING`:** the pause has no saved snapshot, including
  pauses recorded without this artifact.
* **`WORKSPACE_ANCHOR_INVALID`:** the saved reference or snapshot cannot be
  verified. Invalid evidence is refused before acquiring a workspace lease.
* **`WORKSPACE_DRIFT`:** the workspace differs from the saved snapshot.

An occupied process lock rejects startup with `PROCESS_LOCKED`; an unavailable
workspace lease rejects with `WORKSPACE_LEASE`. A run without a safely released
pause rejects with `RUN_NOT_PAUSED`. A position that is not paused rejects with
`RESUME_POSITION`. Resume does not take over a dead watchdog's retained lock.

## Bound the run

* **Elapsed time:** `limits.timeoutMs` covers worker execution and restart
  backoff. Teardown can continue beyond that work deadline.
* **Logical dispatches:** `limits.maxDispatches` counts recorded graph
  dispatches across workers. Resuming one recorded occurrence does not create
  another dispatch.
* **Restarts:** `restart.maxRestarts` caps replacement workers. Backoff grows
  from `initialBackoffMs` up to `maxBackoffMs`.
* **Cleanup grace:** `teardownGraceMs` allows processes to stop before forceful
  termination.

Each worker launch has a fixed 1 MB (1,000,000 bytes) limit across its combined
stdout and stderr. When excess output triggers this limit, the watchdog stops
the worker without restarting it. After successful process cleanup,
`handle.done` resolves to `kind: 'fail'` with `code: 'OUTPUT_LIMIT'`, and status
reports `phase: 'failed'`. If cleanup reports surviving processes, `handle.done`
instead resolves to `kind: 'fail'` with `code: 'TEARDOWN_INCOMPLETE'`, and the
watchdog retains its workspace lease and process lock.

This limit applies to worker console output. Engine response parts and graph
completion output use the separate artifact size and quota limits described
above. `SupervisedRunOptions` exposes no setting for the worker output cap or
a worker memory ceiling. The watchdog passes `Number.MAX_SAFE_INTEGER` as its
worker memory limit; it does not enforce a practical memory budget for the
worker.

Token budgets apply per worker process and do not enforce a shared token cap
across worker restarts.

## Read progress

Status includes the run phase, worker liveness, inspected processes, restart
count, backoff, elapsed time, remaining timeout, and pause reasons.

`cleanupVerified` is `null` before a terminal result, `true` when cleanup was
verified within the reported capability, and `false` when cleanup could not be
verified. `leaseRetained` reports a lease held after terminal failure. A dead
worker or an empty process list does not prove cleanup succeeded; inspect these
fields before treating the workspace as released.

`active` contains all recorded active node occurrences while the worker is
alive. Each entry has its node ID, unchanged position, plan phase, start time,
elapsed time, remaining node timeout, and usage state. Unknown usage is not zero.
The top-level `usage` list keeps recorded totals by node after completion.
An unreported or unfinished engine call makes that node's total unknown.

Graph phases group nodes for display. They do not define execution parents.
Position strings identify occurrences; the status reader does not split them
into a parent chain or choose a supposed deepest node.

Checkpoints cache display state only. Missing, stale, or malformed checkpoints
can be rebuilt from graph events. The executor always makes its decisions from
the events, never from a status checkpoint.

## Process cleanup and workspace ownership

The watchdog uses the public `@obversa/engine/command` API. Command and cleanup
requests accept an optional `ownerId`, a SHA-256 digest used to track the outer
command's processes. The command passes it to child processes as
`OBVERSA_RUN_OWNER`. Nested commands keep a valid inherited marker instead of
replacing it with their own
`ownerId`. During Linux cleanup, only the outer command uses that marker to
discover processes to stop.

`commandCleanupCapability()` returns the `CommandCleanupCapability` type:
`'inherited-owner'` on Linux or `'observed-processes'` elsewhere.
`inspectOwnerMarkedProcesses(ownerId)` returns matching process identities on
Linux and an empty list elsewhere. An empty list on another platform does not
prove that the owner's processes have stopped.

`cleanupCapability` states the cleanup scope:

* **`inherited-owner` on Linux:** cleanup also searches for processes carrying
  the worker's inherited owner marker. Commands that remove or forge markers
  are outside this cleanup scope.
* **`observed-processes` on other platforms:** cleanup follows observed process
  identities and the platform's process-tree mechanisms.

Linux owner cleanup has an implementation and tests, but its Linux-specific
tests have not run on Linux for this release, so Linux cleanup is best-effort.

On macOS, a helper that moves into a new session before the watchdog samples it is not swept.

The watchdog holds the workspace lease token while the worker runs and while
cleanup executes. After a crash that permits restart, it captures and records a
fresh workspace snapshot while still holding the lease. It then releases the
lease for backoff. A replacement must acquire a lease and verify that snapshot
before starting. Changes during the released interval produce a pause with
`code: 'WORKSPACE_DRIFT'`.

This restart snapshot includes workspace changes left by the stopped worker;
it does not replace the stored graph plan or inputs. A terminal result releases
the lease after successful cleanup. Incomplete cleanup or failed lease release
retains the process lock instead of starting another worker.

The process lock is scoped to the storage directory, namespace, and run ID.
A second watchdog for that stored run receives `PROCESS_LOCKED`, even if it
uses another scratch directory.

Worker-crash recovery requires a live watchdog. If the watchdog dies, its lock
and any held lease remain. A new watchdog does not force takeover or infer that
the workspace is safe from a missing PID. This API has no automatic recovery
for watchdog death.