Distillation

Current State

• Repository/task: /home/byk/Code/opencode, baseline commit dc661f30cb feat(core): recover interrupted sessions, on the existing dirty worktree. No commit has been made. The current task, requested Sep 15, 2026 at 17:33, is to close remaining parent-audit durability/liveness gaps in the revised V2 interrupted-session recovery ownership protocol by writing code and deterministic tests.
• Current investigation status (17:34–17:36):
  • packages/core/src/session/sql.ts currently models SessionExecutionTable.phase as "ready" | "safe" | "unknown" with session_execution_phase_check; SessionExecutionTable.recovery_error is nullable.
  • SessionRecoveryTable.assistant_message_id is non-null; primary key is (session_id, assistant_message_id); execution_id and continuation_message_id are unique; wake_pending defaults to true.
  • packages/core/src/session/recovery.ts scans at most BATCH_SIZE = 100 expired claims with null recovery_error, ordered by time_created, then id. It clears "safe" or superseded claims; otherwise it publishes SessionEvent.Step.Recovered.
  • Recovery-outbox redrive currently checks only continuation_message_id === continuation(execution_id). Invalid rows are logged as "Invalid recovery continuation identity" but not quarantined, so poison rows can repeatedly occupy bounded scans.
  • packages/core/src/session/projector.ts currently does not settle SessionRecoveryTable.wake_pending in the Prompted projector; the live recovery promotion callback clears it, but EventV2 commit callbacks are local and do not replay.
  • packages/core/src/session/recovery-id.ts derives recovery IDs as the assistant ID when available or msg_recovery_${executionID}, and continuation IDs as msg_recovery_continue_${executionID}.
  • packages/schema/src/session-event.ts currently allows optional assistantMessageID on Step.Recovered, with phase restricted to "ready" | "unknown".
  • packages/core/src/session/runner/publish-llm-event.ts records provider step-finish separately from later tool settlement.
• Requested implementation now in progress:
  1. Make next-Step continuation durable atomically with successful Step.Ended and Compaction.Ended, likely using an explicit execution phase such as "continue".
  2. Distinguish known-settled continuation from interrupted/unknown provider work during recovery.
  3. Never mark a successfully ended assistant or compaction as errored; post-compaction continuation with no assistant must not invent an error message.
  4. Preserve “never replay a physical provider/tool call” and “never resume an older assistant projection.”
  5. Settle recovery outbox state deterministically during Prompted projection/replay, while making live commit logic idempotent.
  6. Settle user-declined local tools atomically with Step.Interrupted or equivalent and exact-claim deletion, including declines after provider step-finish; such declines must never recover as "unknown" or synthesize continue.
  7. Fix SessionInput.admit ordering so exact-retry equivalence is checked before arming wake_pending; a conflicting default-resume retry against an original resume:false input must conflict without setting durable wake intent.
  8. Fully validate persisted session_recovery rows at the recovery boundary and quarantine invalid rows, including cross-Session continuation poison.
  9. Add poison-row tests with more than 100 invalid rows preceding a valid row, proving quarantine allows valid work on a later bounded scan.
  10. Keep the existing squashed unpublished migration approach; compatibility with discarded intermediate recovery migrations is explicitly not required.
• Blockers: None reported. The required repo-setup skill was unavailable earlier, so repository/package instructions were read directly. Effect v4 source is available under /home/byk/.local/share/opencode/repos/github.com/Effect-TS/effect-smol.
• Files likely open/under active inspection:
  packages/core/src/session/sql.ts, packages/core/src/session/recovery.ts, packages/core/src/session/input.ts, packages/core/src/session/projector.ts, packages/core/src/session/recovery-id.ts, packages/core/src/session/runner/llm.ts, packages/core/src/session/runner/publish-llm-event.ts, packages/core/src/session/compaction.ts, packages/schema/src/session-event.ts, and related Core tests/migration artifacts.
• Unrelated files that must remain untouched: .lore.md, AGENTS.md, perf.data, and perf.data.old. The latter two are untracked.
• Validation required for the active task: focused tests from packages/core with --printConsoleTrace, affected Core/Schema bun typecheck, bun script/migration.ts --check from packages/core, and git diff --check. Do not run package tests from repository root. No full Core suite unless needed.

Key Decisions

• Repository workflow and style
  • Package tests must never run from repository root. Root package.json intentionally fails with "test": "echo 'do not run tests from root' && exit 1.
  • Always run bun typecheck from the relevant package directory; never invoke tsc directly.
  • Use apply_patch for edits.
  • No commits, builds, SEA builds, service starts/restarts, or live-service operations.
  • Default branch is dev; use dev or origin/dev for diffs because local main may not exist.
  • Branch names: at most three hyphen-separated words, no slashes/type prefixes. Commit/PR titles, if later requested, use type(scope): summary.
  • Never alias imports and never use star imports. Prefer a module’s named namespace export.
  • Avoid any, non-null assertions, unchecked casts, stale Effect APIs, unnecessary try/catch, unnecessary destructuring/else; prefer Bun APIs, inference, functional arrays, const, and early returns.
• Dependency architecture
  • Runtime direction: Schema → Core and Protocol; Core and Protocol → Server.
  • Client runtime may depend on Schema and Protocol, never Core or Server.
  • sdk-next composes Client, Core, and Server.
  • Core must remain independent of Server.
  • EventV2 replay ownership remains separate from clustered Session execution ownership.
• Effect conventions
  • Use current Effect v4/effect-smol APIs after checking source and nearby code.
  • Prefer Effect.gen(function* () {}); named/reusable workflows use Effect.fn("Name") or Effect.fnUntraced(...).
  • Bind services to named variables before method calls.
  • Prefer Schema, branded ID schemas, Schema.TaggedErrorClass, Schema.UnknownFromJsonString, and Schema.decodeUnknownOption.
  • HTTP handlers stay thin; business rules belong in services; layer/test dependencies remain explicit.
  • Service/platform tests use testEffect(...); real filesystem/git/server/socket/process/time behavior uses it.live(...) with scoped fixtures/finalizers.
• Session execution/recovery model
  • SessionExecution remains process-global and Session-ID based. It owns a process-local SessionRunCoordinator, looks up placement only when a drain starts through SessionStore and LocationServiceMap, and no layer takes a Session ID.
  • SessionRunner, model resolution, tools, permissions, and filesystem are Location-scoped.
  • Ordinary acquisition never overwrites an existing claim, including expired claims. Expired claim reconciliation belongs exclusively to recovery.
  • Expired "ready"/"unknown" claims reconcile before replacement. "safe" claims may be discarded only by recovery ownership.
  • Leases are fenced by Session ID + execution ID + owner ID and require expires_at > now; an expired claim never renews itself.
  • Lease time uses node wall clocks. SESSION_EXECUTION_LEASE_MS = 30_000 must exceed maximum deployment clock skew plus one renewal interval. Execution IDs fence a stale node immediately after takeover.
  • Every renew, phase update, event commit, promotion/context mutation, and immediate provider/tool/compaction dispatch check must match the exact claim, require a live lease, and affect exactly one row.
  • External provider/tool calls retain an unavoidable instruction-level race after the immediate synchronous ownership check; no speculative distributed-call infrastructure should be added.
  • Explicit cancellation must atomically settle Step.Interrupted and delete the exact claim; owner-scoped cleanup must never delete a successor claim. Successor wake starts only after cancellation cleanup.
• Prompt admission/delivery
  • SessionV2.prompt(...) durably admits one session_input before advisory wake unless resume:false.
  • Exact retries reconcile only when Session, prompt, and delivery match. Conflicting reuse fails.
  • Durable ordinary wake intent is stored in session_input.wake_pending. resume:false never sets it; true retries may set/retain it only after equivalence succeeds; false requests never clear prior true intent.
  • Promotion atomically clears session_input.wake_pending.
  • Recovery must never scan all pending inputs; ordinary redrive queries only wake_pending=true AND promoted_seq IS NULL, grouped deterministically by Session.
  • Recovery wakes are registered before ordinary wakes so coalescing schedules ordinary work as a successor. Recovery drains promote only their exact synthetic continuation.
  • Delivery semantics remain: steers promote at safe provider-turn boundaries; queued input promotes only when the Session would otherwise idle.
• Provider/tool/continuation semantics
  • Exactly one explicit llm.stream(request) per provider turn.
  • Reload projected history before durable continuation.
  • Never call SessionPrompt; never replay prior provider/tool calls.
  • Tool calls are durably recorded before side effects.
  • A stale owner must never dispatch a provider, publish streamed output, start a local tool, fail interrupted tools, promote input, mutate context, or commit compaction.
  • New active requirement: after successful tool-bearing Step.Ended or Compaction.Ended, the need for a fresh Step must be durable in the same commit. Recovery must treat this as settled continuation, not interrupted work.
• Recovery event/outbox identity
  • Startup recovery admits a fresh canonical queued continue prompt through durable recovery state and wakes exactly that input.
  • Canonical prompt remains Prompt.make({ text: "continue" }); existing user-visible interruption text must remain unchanged.
  • Recovery IDs are deterministic from execution ID.
  • Recovery relationships must validate deterministic continuation ID, same-Session ownership, still-pending continuation, canonical queue/continue payload, and any assistant/recovery target’s Session.
  • Invalid recovery rows must be quarantined or permanently settled so bounded scans make progress.
  • Prompted replay must clear matching recovery wake_pending because EventV2 local commit hooks are not replayed.
• Migration decision
  • The recovery work is unpublished 0-to-1. Keep one squashed migration at packages/core/src/database/migration/20260914170650_session-recovery.ts.
  • Deleted intermediate migrations must remain deleted; no compatibility code for databases that ran those discarded migrations.
  • Supported migration path is the actual populated pre-feature schema → final schema.
• Schema conventions
  • @opencode-ai/schema owns browser-safe wire/storage contracts; runtime behavior stays in owning packages.
  • Current contracts are unversioned; retained legacy is V1; permanent replacement architecture should not retain V2.
  • Optional properties normally use package optional(...); public contracts are readonly; avoid Schema.Any.
  • Contract changes require focused tests for omission, identifiers, facade identity, and event-manifest boundaries.
  • Public Protocol/Server HttpApi changes require bun run generate from packages/client; generated sources must not be edited directly.

Technical Changes

• Completed interrupted-session recovery redesign, Sep 15, 2026 14:42–16:12
  • Added exact, lease-fenced execution claim support:
    • New packages/core/src/session/execution/claim.ts.
    • Claims match session_id, execution id, owner_id, and expires_at > now.
    • SessionExecutionClaim.update() renews to current time plus SESSION_EXECUTION_LEASE_MS; stale/expired claims die with SessionExecutionClaim.Lost.
    • SessionExecutionClaim.clear() deletes only the exact live claim.
    • SessionExecutionClaim.transaction() performs renewal/verification plus guarded work in an SQLite { behavior: "immediate" } transaction.
  • Reworked packages/core/src/session/execution/local.ts:
    • Ordinary acquisition uses onConflictDoNothing() and never takes over expired claims.
    • Heartbeats reevaluate current time on every renewal.
    • Success clears exact claim; unsuccessful/uncertain exits retain claim.
    • Interrupt cleanup uses the exact active claim.
  • Reworked packages/core/src/session/run-coordinator.ts:
    • Same-Session runs serialize, different Sessions run concurrently.
    • Wakes coalesce.
    • Interruption marks stopping, waits for cleanup, then starts any pending successor.
    • Fixed "runs a wake registered during interruption cleanup" race and successor ordering.
  • Reworked runner fencing across:
    • packages/core/src/session/runner/index.ts
    • packages/core/src/session/runner/llm.ts
    • packages/core/src/session/runner/publish-llm-event.ts
    • packages/core/src/session/message-updater.ts
    • packages/core/src/session/input.ts
    • packages/core/src/session/context-epoch.ts
    • packages/core/src/session/compaction.ts
  • Runner input now carries executionID, ownerID, and optional recoveryInputID.
  • Added immediate ownership checks around provider dispatch, streamed publication, tool startup, interrupted-tool failure, promotion/context mutation, and compaction.
  • Automatic/overflow compaction transitions to safe only in the durable Compaction.Ended commit.
  • Added deterministic recovery IDs in new packages/core/src/session/recovery-id.ts.
  • Recovery uses bounded deterministic scans (BATCH_SIZE = 100), per-row isolation, takeover CAS, malformed-phase quarantine, latest-assistant supersession checks, and exact continuation wake state.
  • Recovery phase validation currently accepts "ready", "safe", and "unknown".
  • Added durable session_recovery.wake_pending and later durable ordinary session_input.wake_pending.
  • Ordinary redrive currently selects one candidate per Session where wake_pending=true and promoted_seq IS NULL, ordered by minimum time_created, minimum input ID, then Session ID, limited to 100.
• Durable ordinary prompt wake fix, Sep 15, 2026 15:47–16:12
  • Modified:
    • packages/core/src/session.ts
    • packages/core/src/session/input.ts
    • packages/core/src/session/projector.ts
    • packages/core/src/session/recovery.ts
    • packages/core/src/session/sql.ts
    • migration/generated artifacts and related tests.
  • Session.prompt passes requestExecution: input.resume !== false into SessionInput.admit and advisory-wakes only after durable intent exists.
  • New admissions atomically set wake_pending=true using EventV2’s local commit transaction.
  • Exact retries can promote a prior false intent to true; false intent never clears true.
  • Prompt promotion atomically sets promoted_seq and clears wake_pending.
  • Added session_input_wake_pending_promoted_time_created_id_idx on (wake_pending, promoted_seq, time_created, id).
  • A temporary generated migration 20260915155751_session-input-wake.ts was deleted after folding the change into the squashed recovery migration.
• Context Epoch transaction-gap fix, Sep 15, 2026 16:15–16:29
  • User identified a concrete check-then-write bug: SessionContextEpoch.advance renewed the claim, then mutated SessionContextEpochTable outside the claim transaction.
  • Fixed packages/core/src/session/context-epoch.ts so claim renewal/verification and epoch snapshot update occur in one immediate SessionExecutionClaim.transaction(...).
  • SessionContextEpoch.initialize() and .prepare() now accept and propagate a SessionExecutionClaim.Claim.
  • Updated packages/core/test/session-projector.test.ts with:
    1. A deterministic ownership-takeover regression through real SessionContextEpoch.prepare().
    2. A rollback regression proving failed epoch update rolls back claim renewal.
    3. A mutation check: restoring split statements caused expected failure, with expiry changing from 30_000 to 31_000.
  • Adjacent audit found no equivalent gap: Context Epoch insert/replace already used claim transactions; event commits are within EventV2 immediate transactions; recovery claim changes are fenced statements or transactional commits.
• Migration/schema consolidation
  • Squashed recovery schema into packages/core/src/database/migration/20260914170650_session-recovery.ts.
  • Deleted:
    • packages/core/src/database/migration/20260914214636_session-execution.ts
    • packages/core/src/database/migration/20260914222526_session-execution-phase.ts
    • packages/core/src/database/migration/20260914235657_session-execution-lease.ts
  • Updated:
    • packages/core/schema.json
    • packages/core/src/database/migration.gen.ts
    • packages/core/src/database/schema.gen.ts
    • packages/core/src/session/sql.ts
  • Corrected session_execution.id generated metadata to explicit NOT NULL named primary key session_execution_pk.
  • Added session_execution_phase_check enforcing allowed phases.
  • Added nullable session_execution.recovery_error.
  • Added session_recovery.wake_pending integer DEFAULT true NOT NULL.
  • Added session_input.wake_pending integer DEFAULT false NOT NULL.
  • Populated-predecessor migration tests verify existing ses_existing, msg_existing, and msg_input survive; invalid execution phase insertion fails.
• Current known bugs/gaps to fix
  1. Step.Ended writes phase "safe" even when process-local needsContinuation is true after local tool results. Crash before next Step loses required continuation.
  2. Compaction.Ended commits "safe" before process-local rebuilt Step starts. Crash in that window loses continuation.
  3. Recovery currently treats "safe" as discardable, so it cannot distinguish settled-with-required-continuation from no-continuation.
  4. Prompted replay does not clear matching SessionRecoveryTable.wake_pending; live-only commit callbacks do not replay.
  5. User-declined local tools can leave execution "unknown" and be incorrectly recovered with synthetic continue.
  6. Existing-row retry in SessionInput.admit can arm wake_pending before verifying equivalence; conflicting default-resume retry may mutate original admit-only input.
  7. Recovery-outbox validation is incomplete and invalid rows are not quarantined, enabling poison rows to repeatedly block bounded scans.
• Relevant current data model
  • SessionInputTable: id, session_id, JSON prompt, delivery, admitted_seq, nullable promoted_seq, wake_pending default false, time_created.
  • SessionRecoveryTable: session_id, execution_id, non-null assistant_message_id, continuation_message_id, wake_pending default true, time_created; current PK (session_id, assistant_message_id).
  • SessionExecutionTable: exact execution id, unique session_id, owner_id, nullable assistant_message_id, phase, time_created, expires_at, nullable recovery_error.
  • SESSION_EXECUTION_LEASE_MS = 30_000.
• Tests already added/updated in the redesign
  • packages/core/test/database-migration.test.ts
  • packages/core/test/session-execution-local.test.ts
  • packages/core/test/session-projector.test.ts
  • packages/core/test/session-prompt.test.ts
  • packages/core/test/session-runner-recorded.test.ts
  • packages/core/test/session-runner-tool-events.test.ts
  • packages/core/test/session-runner.test.ts
  • Existing coordinator coverage is in packages/core/test/session-run-coordinator.test.ts.
  • Existing compaction-specific coverage is in packages/core/test/session-compaction.test.ts.
• Required new tests for current task
  • Crash after committed tool-result Step.Ended: exactly one recovery continuation, settled assistant/tool result unchanged.
  • Crash after committed Compaction.Ended: exactly one continuation, summary unchanged, no invented assistant error.
  • Replay Step.Recovered followed by continuation Prompted, rebuild projection, verify outbox settled and ordinary work later runs.
  • User-declined local tool, including after provider step-finish: exact claim settled/deleted; after expiry/recovery, no continuation.
  • Conflicting retry: original resume:false, retry defaults to resume; conflict occurs and startup recovery performs no wake.
  • More than 100 invalid recovery rows before a valid row: invalid rows quarantined and valid work progresses on a later scan.
  • Cross-Session continuation-poison regression.
• Validation history
  • 15:32: full Core suite passed 1124 tests, 0 failures, 3088 expect() calls across 145 files in 61.93s.
  • 15:39: 8 focused files passed 166, 0 failed, 459 expect() calls in 13.96s; typecheck passed.
  • 16:04: 4-file wake-intent suite passed 160, 0 failed, 453 expect() calls in 15.01s.
  • 16:07: expanded 9-file suite passed 195, 0 failed, 525 expect() calls in 15.91s.
  • 16:11: full Core suite passed 1129, 0 failed, 3100 expect() calls across 145 files in 61.53s.
  • 16:12: migration check, Core typecheck, Schema typecheck, and git diff --check passed.
  • 16:28 mutation check: original split Context Epoch implementation failed as intended: 21 pass, 1 fail, 67 expect(); expected 30_000, received 31_000.
  • 16:29 final Context Epoch suite: 117 pass, 0 fail, 354 expect() calls across test/session-projector.test.ts and test/session-runner.test.ts in 11.89s; Core typecheck and git diff --check passed.
• Latest reported broad worktree state, 16:05
  • 29 changed files, exactly 1913 insertions(+), 517 deletions(-).
  • Modified production/schema files included .lore.md, AGENTS.md, generated DB artifacts, Session execution/recovery/runner files, and packages/schema/src/session-event.ts.
  • Deleted two migrations were shown then; the third lease migration is also reported deleted in final summaries.
  • Untracked: packages/core/src/session/execution/claim.ts, packages/core/src/session/recovery-id.ts, perf.data, perf.data.old.
  • Later Context Epoch task intentionally changed only packages/core/src/session/context-epoch.ts and packages/core/test/session-projector.test.ts relative to the preceding worktree.

Session Timeline

• Sep 15, 2026 14:42: User requested complete implementation of the revised interrupted-session ownership/recovery protocol against dc661f30cb plus dirty lease/fencing work. Nine DO-NOT-MERGE findings were enumerated: fixed heartbeat expiry, expired-runner revival, acquisition bypassing recovery, cancellation/successor deletion race, compaction safe too early, recovery wake loss/admit-only leakage, populated migration failure, malformed-phase scan poisoning, and cross-Session recovery identity.
• 14:43–14:50: Repository rules, Session architecture, Effect conventions, execution/recovery code, SQL schema, migrations, coordinator, projectors, input lifecycle, context epoch, compaction, EventV2 transaction semantics, and existing tests were inspected. Initial worktree had 20 modified tracked files, 3 untracked files, and exactly 467 insertions/268 deletions.
• 14:54–15:09: Implemented claim helper, local execution changes, runner claim threading/fencing, recovery IDs/state, projector/schema changes, squashed migration changes, and compatibility fixes. Initial typecheck errors across six files were resolved. A 2-file test run passed 21 tests.
• 15:09–15:24: Expanded runner/coordinator/migration failures exposed retry, wake, migration, tool registration, scope-close, and interruption-successor races. Iterative fixes produced a relevant suite result of 166 passed, 0 failed, 459 assertions.
• 15:25–15:40: Corrected explicit non-null execution primary-key metadata and phase check; verified migration consistency; ran full Core (1124/1124) and focused (166/166) suites; Core/Schema typechecks and diff check passed. First recovery redesign was reported complete with no commit.
• 15:47: User identified ordinary prompt wake liveness gap and requested durable session_input.wake_pending, exact-retry escalation, atomic clear on promotion, and deterministic bounded ordinary redrive without waking resume:false.
• 15:47–16:08: Implemented durable ordinary wake intent, updated migration/schema artifacts, added prompt/projector/runner/migration tests, and fixed test-fixture issues. Focused suites reached 195/195; migration check and typecheck passed.
• 16:11–16:12: Full Core suite passed 1129/1129, 3100 assertions, 145 files, 61.53s. Migration check, Core/Schema typechecks, and diff check passed. No commit/service operation.
• 16:15: User identified Context Epoch advance check-then-write gap and requested immediate transaction plus deterministic real-path regression and adjacent audit.
• 16:18–16:29: Wrapped claim renewal and Context Epoch update in one immediate transaction. Mutation testing proved the regression: split code leaked expiry from 30_000 to 31_000; fixed code passed. Final focused suite passed 117/117; typecheck/diff check passed. Audit found no adjacent equivalent durable gap.
• 17:33: User requested the next parent-audit implementation: durable continuation after settled tool Steps/compaction, replay-safe recovery outbox settlement, user-decline settlement, retry-equivalence ordering, and robust recovery-row quarantine/validation. Migration-ID compatibility finding was explicitly rejected as not applicable to unpublished 0-to-1 work.
• 17:34–17:36: Current schemas and recovery/projector/runner paths were inspected. Concrete gaps were confirmed: no durable settled-continuation phase, no replay settlement of recovery wake_pending, incomplete recovery-row validation/quarantine, and separate provider step-finish/tool-result handling. No implementation or validation result for this newest task has yet been recorded.