Distillation

Current State

• Active task (Sep 15, 2026, 20:07 onward): Read-only adversarial correctness audit of the CURRENT files in /home/byk/Code/opencode, focused only on V2 interrupted-session recovery after the latest fixes.
• Audit restrictions: Do not edit files, run tests/builds, commit, touch services, use lore/recall, or retrieve prior context. This is the user-designated single evidence-or-blocker retry. Never return empty. Do not give a merge verdict.
• Required output: Substantive findings ordered by severity, each with exact current file:line evidence and a concrete failure sequence. If no defect exists, explicitly say PASS and provide evidence-backed residual risks/testing gaps. If blocked, quote the exact tool/error and affected scope.
• Recheck order:
  1. Durable "continue" after Step.Ended and Compaction.Ended.
  2. Replay-safe recovery-outbox settlement.
  3. Exact claim cancellation when a user declines after step-finish.
  4. Conflicting retry must not arm wake.
  5. Poison-row quarantine and bounded progress.
  6. Interactions with newest-assistant supersession, exact recovery-input priority, ordinary durable wake intent including resume:false, lease expiry/takeover, supported pre-feature-to-final migration, fresh schema, and replay.
  7. Whether regression tests reach the real failure windows.
• Scope decision: Compatibility with discarded unpublished intermediate migrations is explicitly out of scope. Supported pre-feature-to-final migration remains in scope.
• Current audit progress: Source and tests were inspected through approximately 20:11. The audit had not yet issued its final PASS/findings response.
• Current worktree/branch: Repository is /home/byk/Code/opencode; branch was reported as byk/cumulative. One status reported HEAD dc661f30cb2055ad6248fdc59b14998d834497be (dc661f30cb, feat(core): recover interrupted sessions), while another reported revision was 631f67a9f330e2e0b1c064db358e67133a053655; this discrepancy was not resolved.
• Latest known changed files: Recovery/session source, migration/schema artifacts, and tests remain dirty. Untracked files include:
  • packages/core/src/session/execution/claim.ts
  • packages/core/src/session/recovery-id.ts
  • perf.data
  • perf.data.old
• Latest migration chain currently inspected:
  1. packages/core/src/database/migration/20260914170650_session-recovery.ts
  2. packages/core/src/database/migration/20260914214636_session-execution.ts
  3. packages/core/src/database/migration/20260914222526_session-execution-phase.ts
  4. packages/core/src/database/migration/20260915193000_session-recovery-fencing.ts
• Read-only audit evidence inspected most recently:
  • packages/core/src/session/recovery.ts
  • packages/core/src/session/input.ts
  • packages/core/src/session/projector.ts
  • packages/core/src/session/execution/local.ts
  • packages/core/src/session/execution/claim.ts
  • packages/core/src/session/runner/llm.ts
  • packages/core/src/session/runner/publish-llm-event.ts
  • packages/core/src/session/sql.ts
  • all four recovery-related migrations
  • packages/core/test/session-runner.test.ts
  • packages/core/test/session-projector.test.ts
  • packages/core/test/session-prompt.test.ts
  • packages/core/test/database-migration.test.ts
• Current blocker status: No source-inspection blocker. Repository skill/file lookup earlier returned exactly No files found, so inspection proceeded directly from repository files and instructions.

Key Decisions

• Recovery ownership model
  • Execution ownership is durable and fenced by session_id, execution id, owner_id, and a live lease.
  • An expired execution claim never renews itself.
  • Ordinary local acquisition must never replace either a live or expired existing claim; takeover belongs to recovery.
  • Recovery must never recover a live lease and must compare-and-swap on execution/session/old owner/expiry/quarantine state.
  • Execution IDs provide fencing immediately after takeover.
• Durable continuation design
  • A new execution phase "continue" represents a committed Step/compaction that still requires another provider turn.
  • Step.Ended commits "continue" when a settled tool or other condition requires continuation; otherwise it commits "safe".
  • Compaction.Ended similarly commits "continue" before the next provider turn.
  • Recovery events expose phases "ready" | "unknown" | "continue"; execution-only "safe" is cleared without recovery.
  • This replaced the unsafe design where continuation intent existed only in process-local return state after committing "safe".
• Recovery outbox design
  • Recovery continuation IDs are deterministic: msg_recovery_continue_${executionID}.
  • Recovery rows are keyed by execution_id, have unique continuation_message_id, store phase, wake_pending, and nullable recovery_error.
  • SessionEvent.Prompted projection settles the exact recovery wake by matching both session_id and continuation_message_id, making projection replay safe.
  • Exact recovery input has priority over ordinary pending inputs. Poisoned rows are excluded after quarantine.
• Supersession invariant
  • A newer assistant turn supersedes stale incomplete rows; recovery must never resume an older assistant projection.
  • Recovery checks newest assistant before publishing, and the projector checks again when applying Step.Recovered.
• Cancellation/decline semantics
  • Declining permission or dismissing a question after step-finish halts continuation rather than converting the decision into model-facing tool output.
  • The publisher emits SessionEvent.Step.Interrupted; interruption must clear the exact active claim so later recovery cannot resume it.
  • Permission correction is different: feedback is returned to the model and continuation proceeds.
• Prompt retry semantics
  • SessionV2.prompt(...) durably admits one session_input row before advisory SessionExecution.wake(sessionID).
  • resume:false is admit-only and must leave wake_pending=false.
  • An exact retry may turn previously admit-only work into durable execution intent.
  • Conflicting reuse of a message ID must fail before mutating wake state.
  • Reusing a Session ID adopts the existing Session. Historical projected prompts lazily synthesize inbox records on exact retry.
• Migration strategy
  • The initially rewritten already-journaled migration was rejected as unsafe for existing databases.
  • The final approach restored the three prior migrations and added 20260915193000_session-recovery-fencing.ts to transform the earlier table forms into the final schema.
  • Discarding rows from unpublished intermediate recovery/execution tables is accepted as out of scope; preserving supported pre-feature user/session/input data remains required.
  • Fresh databases use generated full schema and mark all TypeScript migrations completed.
• Coordinator ordering
  • Cancellation must settle and cleanup before a successor drain starts.
  • Wakes arriving during stopping are retained as one coalesced successor.
  • Same-Session explicit resumes join active work; different Sessions may run concurrently.
• Repository architecture decisions
  • Runtime dependency direction: Schema → Core and Protocol; Core and Protocol → Server. Client runtime may depend on Schema and Protocol, never Core or Server. sdk-next composes Client, Core, and Server.
  • SessionExecution remains process-global and Session-ID-based. Local implementation owns the process-local coordinator and resolves placement only when a drain begins.
  • SessionRunner, model resolution, tools, permissions, and filesystem remain Location-scoped.
  • Exactly one explicit llm.stream(request) per provider turn; reload projected history before continuation. Do not bridge through legacy SessionPrompt.loop(...).
  • EventV2 replay ownership remains separate from clustered Session execution ownership.
• Coding/import rules
  • No aliased imports and no star imports.
  • No export namespace; use flat exports and self-reexports.
  • Multi-sibling directories such as src/session/ must not add barrel index.ts files.
  • Prefer Effect.gen, Effect.fn, Effect.fnUntraced, Effect.callback, Effect.void, DateTime.nowAsDate, Effect services, Effect.cached, and scoped background fibers.
  • Effect v4 beta has no Effect.fork or Effect.forkDaemon; use Effect.forkIn(scope).
  • Verify Effect APIs against current effect-smol source, not v2/v3 memory.
  • Avoid try/catch, any, unnecessary destructuring/reassignment/else, premature helpers, and obvious comments.
  • Drizzle fields use snake_case directly.
• Testing/tooling rules
  • Never run package tests from repository root; run from package directories such as packages/core.
  • Typecheck with bun typecheck/package tsgo --noEmit, never direct tsc.
  • Avoid mocks and globalThis.* unless unavoidable; tests should exercise actual implementation and real failure windows.
  • Default branch is dev; use dev or origin/dev for diffs.
  • Branch names: at most three hyphen-separated words, no slash/type prefix.
  • Commits/PR titles: type(scope): summary, with type among feat, fix, docs, chore, refactor, test.
• Audit stance: No merge verdict because audits are pre-freeze reviews.

Technical Changes

Current recovery/session implementation

• packages/core/src/session/execution/claim.ts:25-66
  • Claim predicate requires matching session_id, execution id, owner_id, and expires_at > now.
  • update() renews to now + SESSION_EXECUTION_LEASE_MS.
  • clear() deletes only a still-live matching claim.
  • Both die with SessionExecutionClaim.Lost if no row matches.
  • transaction() starts an immediate DB transaction, renews first, then performs the supplied Effect.
  • Comment records wall-clock/clock-skew requirements and the invariant that expired claims never renew themselves.
• packages/core/src/session/execution/local.ts:23-99
  • Creates process-local random ownerID and per-drain random executionID.
  • Inserts a "ready" row with expires_at = now + 30_000 using onConflictDoNothing().
  • Does not start work when another claim exists.
  • Selects the oldest recovery row for the Session with wake_pending=true and recovery_error IS NULL, ordered by time_created, then execution_id.
  • Passes the exact continuation ID to input.run(...).
  • Races the run with claim renewal every lease-duration/3.
  • On success clears claim; on interruption preserves it except explicit interruption cleanup; on non-interrupt failure clears unless phase is "unknown" or "continue".
  • interrupt(sessionID) clears the exact active claim before coordinator interruption and suppresses only SessionExecutionClaim.Lost.
• packages/core/src/session/sql.ts:141-225
  • SessionInputTable: primary-key id, session_id, JSON prompt, delivery, admitted/promoted sequence, wake_pending DEFAULT false, creation time, and pending-wake index.
  • SessionRecoveryTable: execution_id primary key; nullable assistant_message_id; unique continuation_message_id; phase "ready" | "unknown" | "continue"; wake_pending DEFAULT true; nullable recovery_error.
  • SessionExecutionTable: primary-key id; unique session_id; owner; nullable assistant ID; phase "ready" | "safe" | "unknown" | "continue" with SQL check; creation/expiry; nullable recovery error.
  • SESSION_EXECUTION_LEASE_MS = 30_000.
• packages/core/src/session/recovery-id.ts
  • message(executionID, assistantMessageID?) returns the supplied assistant ID or msg_recovery_${executionID}.
  • continuation(executionID) returns msg_recovery_continue_${executionID}.
• packages/core/src/session/recovery.ts
  • BATCH_SIZE = 100.
  • Scans expired, unquarantined execution claims in stable creation/ID order.
  • Takes over with guarded update matching execution ID, Session ID, old owner, expiry, and null quarantine state.
  • Runtime phase schema now allows "ready", "safe", "unknown", and "continue".
  • Invalid phases are quarantined with exact text Invalid execution phase: ${String(claim.phase)}.
  • "safe" claims and claims superseded by a newer assistant are cleared without recovery.
  • Publishes SessionEvent.Step.Recovered; commit conditionally deletes the unchanged claim.
  • Duplicate publication is tolerated only when the matching recovery row already exists.
  • Pending recovery rows are selected only with wake_pending=true and recovery_error IS NULL.
  • Validates deterministic continuation identity, continuation input ownership/shape, and projection identity.
  • Quarantine messages include:
    • Invalid recovery continuation identity
    • Invalid recovery continuation input
    • Invalid recovery projection identity
  • Invalid rows are quarantined rather than repeatedly consuming bounded scan capacity.
  • Ordinary wake scanning is separate and selects unpromoted session_input rows with wake_pending=true.
  • Per-row failures are isolated and logged.
  • Background recovery repeats every 10 seconds in a scoped fiber; top-level failures log Failed to recover Sessions (recovery.ts:286-298).
• packages/core/src/session/input.ts:43-122
  • admit() checks existing-row equivalence before requesting wake.
  • New prompt admission uses the wake request as the durable event’s commit callback.
  • requestWake() updates only matching unpromoted input.
  • Already-promoted equivalent rows are accepted; non-equivalent rows die with SessionInput.LifecycleConflict.
• packages/core/src/session/input.ts:159-229
  • projectPrompted() atomically assigns promoted_seq, clears input wake_pending, supports exact replay, and calls settleRecoveryWake().
  • settleRecoveryWake() clears only the exact recovery row matching both session_id and continuation_message_id.
• packages/core/src/session/input.ts:277-380
  • Steers are promoted by ascending admitted_seq.
  • Queued input promotion chooses the oldest unpromoted queue item.
  • promoteExact() requires exact message ID, Session, unpromoted state, and queue delivery; otherwise dies with Pending recovery input not found: ${messageID}.
• packages/core/src/session/projector.ts:357-448
  • Prompted projects through SessionInput.projectPrompted() before common event projection.
  • PromptAdmitted projects through SessionInput.projectAdmitted().
  • Step.Recovered:
    • derives deterministic continuation ID;
    • is idempotent when the same execution/session recovery row already exists;
    • verifies supplied assistant ID is the newest assistant and is actually an assistant;
    • projects the recovery event;
    • admits exact queued prompt text "continue";
    • inserts a recovery row with phase, pending wake, and continuation ID.
  • Exact errors include:
    • Recovery target is not the latest assistant: ...
    • Recovery target is not an assistant: ...
  • Step.Started, Step.Ended, Step.Failed, Step.Interrupted, and Compaction.Ended route through common projection.
  • SessionEvent.Retried remains commented out around line 460.
  • A duplicate continuation-message lifecycle conflict found during tests was fixed here at 19:54.
• packages/core/src/session/runner/llm.ts
  • Execution state recognizes phase "continue".
  • Startup resets non-continuation work to { phase: "ready", assistant_message_id: null }.
  • Recovery promotes only the exact continuation via SessionInput.promoteExact(...).
  • Uncertain execution is set to { phase: "unknown", assistant_message_id: null }.
  • terminalCommit can set phase "safe" (llm.ts:285).
  • Step.Ended commits phase: needsContinuation ? "continue" : "safe" (llm.ts:426).
  • Recovery suppresses steer promotion until exact recovery input is handled.
  • Decline handling calls publisher.interruptAssistant("Tool execution declined").
  • Some comments/checklist text remained stale at 19:07, still describing durable continuation as future work despite implementation being present.
• packages/core/src/session/runner/publish-llm-event.ts
  • Durable events invoke commit hooks atomically; live-only events commit before publish.
  • step-finish flushes fragments, makes assistant inactive, rejects duplicate settlement, and stores finish/tokens (:441-452 in one inspection, :421-468 in latest layout).
  • failAssistant() publishes Step.Failed through terminalCommit.
  • interruptAssistant() publishes Step.Interrupted through interruptedCommit (:220-256).
  • failUnsettledTools() marks matching unsettled tools failed before publishing.
  • Provider tool event validation rejects malformed ordering, changed names, and duplicates with exact errors such as:
    • Tool input name changed for ${event.id}: ${tool.name} -> ${event.name}
    • Tool call name changed for ${event.id}: ${tool.name} -> ${event.name}
    • Duplicate tool call: ${event.id}
    • Duplicate tool result: ${event.id}
    • Duplicate step finish
• packages/core/src/session/message-updater.ts:251-297
  • "continue" recovery performs no assistant error mutation.
  • "unknown" uses Provider Step outcome unknown after server restart.
  • Other recoverable phases use Session execution interrupted by server restart.
  • Non-continue recovery of an existing assistant marks it errored and converts pending/running tools to unknown-outcome errors.
• packages/schema/src/session-event.ts:207-217
  • Step.Recovered event type is "session.next.step.recovered".
  • Fields include execution ID, optional assistant ID, and phase "ready" | "unknown" | "continue"; "safe" is intentionally execution-table-only.

Migration/schema changes

• Earlier unsafe consolidation was reversed.
• Restored:
  • packages/core/src/database/migration/20260914170650_session-recovery.ts
    • Creates original recovery table with composite key (session_id, assistant_message_id).
  • packages/core/src/database/migration/20260914214636_session-execution.ts
    • Creates initial execution table, adds execution_id, and indexes.
  • packages/core/src/database/migration/20260914222526_session-execution-phase.ts
    • Adds required execution phase.
• Added packages/core/src/database/migration/20260915193000_session-recovery-fencing.ts:4-53
  • Conditionally adds session_input.wake_pending integer DEFAULT false NOT NULL.
  • Drops/recreates intermediate session_execution and session_recovery forms.
  • Final execution schema adds expires_at, recovery_error, PK, session FK, and phase check for ready/safe/unknown/continue.
  • Final recovery schema uses execution_id as PK; nullable assistant; required phase; wake_pending DEFAULT true; nullable recovery_error.
  • Recreates:
    • session_execution_session_idx
    • session_execution_owner_idx
    • session_input_wake_pending_promoted_time_created_id_idx
    • session_recovery_continuation_message_idx
• packages/core/src/database/migration.gen.ts:36-47
  • Registers the four recovery migrations in chronological order.
  • Total generated migration list is now 40 after adding the fencing migration; before addition it contained 39, ending in 20260914170650_session-recovery.
• Generated schema artifacts updated:
  • packages/core/schema.json
  • packages/core/src/database/schema.gen.ts
  • Schema remains version "7" / SQLite.
• packages/core/script/migration.ts --check succeeded at 20:03:
  • Incremental comparison: No schema changes, nothing to migrate.
  • Full schema generated successfully at /tmp/opencode-core-migration-check-kktghk/full/20260915200156_schema/migration.sql.

Tests added/fixed and latest validation

• packages/core/test/database-migration.test.ts
  • Added populated predecessor-to-final migration coverage.
  • Preserves rows ses_existing, msg_existing, and msg_input.
  • Verifies:
    • execution ID required;
    • input wake_pending defaults false;
    • recovery execution ID required and primary-keyed;
    • assistant ID nullable;
    • phase required;
    • recovery error nullable;
    • invalid execution phase "invalid" fails.
• packages/core/test/session-projector.test.ts
  • Coverage includes:
    1. Recovery without forged assistant identity.
    2. Never recover live lease.
    3. Never recover execution superseded by newer assistant.
    4. Redrive durable recovery wake after admission.
    5. Quarantine malformed row without blocking valid wake.
    6. Quarantine BATCH_SIZE + 1 poison rows before later valid work.
    7. Quarantine continuation belonging to another Session.
    8. Redrive ordinary durable input intent after lost advisory wake.
    9. Quarantine recovery wake behind a newer assistant.
    10. Quarantine malformed claim without blocking valid claim.
    11. Fence paused owner after real expiry/reconciliation/takeover.
  • Poison-batch test creates SessionRecovery.BATCH_SIZE + 1 poison rows and verifies bounded progress on a later scan.
• packages/core/test/session-prompt.test.ts:293-404
  • Verifies durable wake before advisory wake.
  • Verifies exact retry can arm previously admit-only work.
  • Verifies conflicting retry with resume:true leaves original wake_pending:false, makes no advisory wake, and remains unwoken after SessionRecovery.recover().
• packages/core/test/session-runner.test.ts:1511-1560
  • Crashes after Compaction.Ended commits phase "continue", expires claim, runs recovery, and verifies the recovery model receives "continue" while preserving compacted context.
• packages/core/test/session-runner.test.ts:1942-2000
  • Crashes after settled-tool Step.Ended commits "continue", expires claim, recovers, verifies tool ran exactly once and recovered request contains ["Recover settled tool continuation", "continue"].
• packages/core/test/session-runner.test.ts:3235-3288
  • Permission decline after step-finish produces interruption-only failure, exactly one provider request, Step.Interrupted, empty execution table, no later recovery row, and no extra provider request.
• packages/core/test/session-runner.test.ts:3340-3394
  • Dismissed question similarly interrupts continuation after one provider request.
• packages/core/test/session-execution-local.test.ts
  • Covers claim cleanup, child-scope cancellation preserving claims, explicit interrupt cleanup, no second live owner, no overwrite of expired lease, no uncertain reentry, heartbeat renewal from current clock, no revival of expired claim, and cancellation-before-successor ordering.
• Test run at 19:51 initially failed exactly:
  • SessionProjector > atomically reconciles one abandoned execution into one continuation
  • SessionInput.LifecycleConflict
  • ID msg_recovery_continue_execution
  • at packages/core/src/session/input.ts:141
  • path through SessionInput.projectAdmitted in packages/core/src/session/projector.ts:414
• Root cause: recovery projection attempted to admit a continuation whose message ID already existed in SessionMessageTable.
• After fixing projector handling, rerun at 19:56 succeeded:
  • 186 pass
  • 0 fail
  • 535 expect() calls
  • 7 files
  • 15.32 seconds
• Type checking:
  • tsgo --noEmit succeeded at 19:57.
  • A second tsgo --noEmit succeeded at 19:59.
• These executions preceded the active 20:07 read-only audit; no tests/builds may be run during the current audit.

Historical first-audit findings and disposition

At 16:52, the earlier dirty worktree had seven substantive findings:

1. Critical: Rewriting already-journaled migration ID left existing DBs without final columns.
   • Superseded by restoring intermediate migrations and adding the new fencing migration.
   • User later explicitly excluded unpublished intermediate compatibility, while retaining supported pre-feature migration scope.
2. High: Crash after successful tool Step.Ended could lose required continuation.
   • Addressed by durable "continue" phase and real crash-window test.
3. High: Crash after committed compaction could lose post-compaction continuation.
   • Addressed by durable "continue" phase and real crash-window test.
4. High: Replay rebuilt settled recovery outbox as pending forever.
   • Addressed by exact settleRecoveryWake() during Prompted projection.
5. High: User-declined tool left "unknown" claim that recovery resumed.
   • Addressed by Step.Interrupted/exact claim cancellation and post-recovery test.
6. Medium: Conflicting retry armed original admit-only prompt.
   • Addressed by equivalence check before wake mutation and dedicated regression.
7. Medium: Cross-Session poison rows could permanently block bounded scans.
   • Addressed by row validation, quarantine, filtering, and BATCH_SIZE + 1 progress test.

Checks known to hold

• Lease renewal uses current Effect clock and never renews expired claims.
• Ordinary acquisition replaces neither live nor expired claims.
• Recovery takeover uses guarded compare-and-swap.
• Provider dispatch, streamed publication, local tools, promotion, and Context Epoch writes are fenced by execution claims.
• Newest-assistant supersession is checked in recovery and projection.
• Invalid phases and poisoned recovery rows are isolated/quarantined.
• Effect v4 Effect.repeat, schedules, and scoped background-fiber usage matched inspected effect-smol APIs.

Session Timeline

• Sep 15, 2026, 16:32: User requested first read-only adversarial audit of dirty V2 recovery worktree; prohibited edits/tests/builds/services/commit and requested severity-ordered exact evidence.
• 16:33–16:51: Inspected dirty diff, claim/local execution, recovery, projector, runner, coordinator, compaction, Context Epoch, migration/schema, prompt/replay tests, and Effect APIs.
• 16:52: First audit reported seven defects: one Critical migration issue, four High continuation/replay/decline issues, and two Medium retry/poison-row issues. No files were changed or tests run during that audit.
• 19:07: User requested a second read-only recheck after fixes, explicitly excluding discarded unpublished intermediate migration compatibility.
• 19:07–19:18: Inspected revised recovery phases/outbox validation, prompt wake behavior, decline path, schema, migration tests, and runner recovery tests.
• 19:20–19:30: Further migration/event/claim inspection. Confirmed package tooling and lease invariant.
• 19:31: Implementation work resumed outside the read-only audit:
  • Added 20260915193000_session-recovery-fencing.ts.
  • Restored/updated the two deleted intermediate migrations.
  • Updated migration registry and original recovery migration.
• 19:39–19:50: Updated projector, runner integration, fencing migration, and migration/projector test coverage.
• 19:51: First package test run: 185 pass, 1 fail, 533 expectations; duplicate continuation message caused SessionInput.LifecycleConflict.
• 19:54: Fixed recovery projector lifecycle handling.
• 19:56: Tests passed: 186/186, 535 expectations, 15.32 seconds.
• 19:57 and 19:59: Two successful tsgo --noEmit runs.
• 20:03: Migration consistency check passed with no pending incremental schema change and successful full-schema generation.
• 20:07: User began final read-only evidence-or-blocker audit of current files, forbidding lore/recall, edits, tests, builds, services, and commits.
• 20:07–20:09: Inspected recovery loop, admission/wake mutation, replay settlement, exact promotion, Step.Recovered projection, local claim lifecycle, lease fencing, and publisher interruption behavior.
• 20:10: Inspected final four-migration chain and final recovery/execution schema.
• 20:11: Inspected real crash-window tests for committed compaction and settled-tool continuation, plus permission-decline regression. Final audit response remained pending.