Distillation

Date: Sep 16, 2026

• 🔴 [enforced-test-isolation] (09:59) User stated tests use a temporary SQLite database through packages/core/test/setup.ts and never the production DB.
• 🔴 [enforced-release-workflow] (09:59) User directed never to manually bump versions or edit CHANGELOG.md; Sentry Craft handles both automatically.
• 🔴 (09:59) User stated Lore is a three-tier memory architecture for AI coding agents, implemented as a Node.js >= 22.5 TypeScript monorepo using pnpm workspaces; SQLite runs in WAL mode with FTS5 and is stored at ~/.local/share/lore/lore.db.
• 🔴 (09:59) User identified the four packages: @loreai/core at packages/core/ (shared DB, distillation, knowledge/LTM, recall, gradient context, prompts), @loreai/gateway at packages/gateway/ (transparent LLM proxy, lore CLI, dashboard), @loreai/opencode at packages/opencode/ (OpenCode adapter via @opencode-ai/plugin), and @loreai/pi at packages/pi/ (Pi extension adapter). gateway, opencode, and pi consume core; gateway bundles core into one CJS file via esbuild, while opencode ships raw TypeScript.
• 🔴 (09:59) User stated the CLI binary flows from packages/gateway/src/cli/bin.ts to _cli() in packages/gateway/src/cli/main.ts; commands are run (default, starts gateway and auto-detects/launches an agent), start (gateway only), data with list/show/clear/delete/merge/recover, recall, and upgrade. cli/agents.ts scans $PATH for Claude Code, Codex, Pi, and OpenCode and configures routing variables such as ANTHROPIC_BASE_URL.
• 🔴 (09:59) User described Tier 1 in packages/core/src/temporal.ts: store() converts message parts to text, writes temporal_messages with FTS5 indexing and a distilled=0/1 flag, and uses \x1f (ASCII Unit Separator) as the chunk separator.
• 🔴 (09:59) User described Tier 2 in packages/core/src/distillation.ts: run() performs resetOrphans() → retrieve undistilled messages → detectSegments() using time gaps/token limits with at most 16K tokens per segment → distillSegment() → generation-0 storage. When generation-0 count exceeds 20, metaDistill() creates generation-1+ consolidation; generation-0 segments remain searchable through recall but are excluded from the in-context prefix.
• 🔴 (09:59) User described Tier 3 in packages/core/src/gradient.ts: transform() runs on every message transform and progresses through Layer 0 full passthrough, Layer 1 distilled prefix plus compressed raw context, Layer 2 aggressive tool-output stripping, and Layer 3 emergency compression; it calibrates from actual API token counts and uses model-pricing-aware caps.
• 🔴 (09:59) User stated LTM entries in packages/core/src/ltm.ts have categories decision, pattern, preference, architecture, and gotcha, with fields id (UUIDv7), project_id, category, title, content (maximum 1200 characters), confidence (0.0–1.0), and cross_project.
• 🔴 (09:59) User identified four LTM creation paths: 1. LLM Curator in packages/core/src/curator.ts; 2. regex-only Pattern Extraction in packages/core/src/pattern-extract.ts; 3. .lore.md File Import in packages/core/src/agents-file.ts using <!-- lore:UUID --> markers; 4. curator.ts consolidate() when entry count exceeds maxEntries of 25.
• 🔴 (09:59) User stated packages/core/src/ltm.ts provides create()/update()/remove() with a deduplication guard, forSession() for relevance-ranked budget-capped prompt entries, and search()/searchScored() using FTS5 BM25 with LIKE fallback.
• 🔴 (09:59) User stated .lore.md entries are sorted alphabetically by title within category. exportLoreFile() in packages/core/src/agents-file.ts builds from the DB and skips writing when the content hash is unchanged; importLoreFile() updates changed known UUIDs and creates unknown UUIDs; shouldImportLoreFile() first checks mtime and then compares content hashes.
• 🔴 (09:59) User stated exportToFile() writes the lore pointer section into AGENTS.md between markers, while splitFile() preserves all material outside those markers.
• 🔴 (09:59) User provided project commands: pnpm install, pnpm test, pnpm run typecheck, pnpm run lint, pnpm run lint:fix, pnpm run format, pnpm run format:check, and pnpm run build. Gateway build uses packages/gateway/script/build.ts; script/bundle.ts creates the standalone binary; core build uses packages/core/script/build.ts.
• 🔴 (09:59) User stated releases are triggered with gh workflow run release.yml -f version=auto for conventional-commit-derived semver or gh workflow run release.yml -f version=0.23.0 for an explicit version.
• 🔴 (09:59) User described the release workflow order: 1. scripts/bump-version.sh updates all package.json versions; 2. changelog generation from conventional commits; 3. creation and push of release/X.Y.Z; 4. CI builds/tests npm tarballs, standalone binaries, and delta patches; 5. Craft opens a publish issue and the accepted label starts Publish; 6. Publish uses npm OIDC trusted publishing and creates a GitHub Release with binaries and patches. .craft.yml targets four npm workspaces, the legacy opencode-lore alias, and the GitHub Release.
• 🔴 (09:59) User provided root package.json configuration: package lore-monorepo, private ESM, license FSL-1.1-Apache-2.0, package manager pnpm@10.28.0, Volta Node 24.16.0 and pnpm 10.28.0, workspaces packages/*, repository git+https://github.com/BYK/loreai.git, and author BYK.
• 🔴 (09:59) User stated root scripts include "test": "vitest run", "pretest": "pnpm --filter @loreai/gateway run bundle", "test:coverage": "pnpm --filter @loreai/gateway run bundle && vitest run --coverage", "lint": "oxlint --type-aware", "lint:fix": "oxlint --type-aware --fix", "format": "oxfmt", "format:check": "oxfmt --check", "evals": "vitest run --config vitest.evals.config.ts", "premutation": "pnpm --filter @loreai/gateway run bundle", "mutation": "stryker run", and "postinstall": "pnpm --filter @loreai/gateway run build".
• 🔴 [enforced-regression-testing] (10:00) User required every defect found by adversarial review to receive a deterministic regression test in the same PR that fails on the base branch, passes with the fix, and drives the real precondition rather than an artificial database state. Guard tests must fail if the guard is removed.
• 🔴 [enforced-tdd] (10:00) User stated: “Never trust the fix without seeing it fail.” Bug fixes must be driven by a regression test written first, confirmed failing on the base branch, and then confirmed passing with the fix.
• 🔴 (10:00) User required adversarial-order test setup: establish stale state before the operation that should handle it; exercise delete-then-recreate across disabled/re-enabled lifecycle boundaries; and test modifications made while sync is off, when capture triggers cannot observe them.
• 🔴 [requested-contract-tests] (10:00) User required fan-out registry additions to join their parametrized contract batteries: SYNCED_TABLES → packages/core/test/sync-registry-contract.test.ts covering shape, pull-only/no-capture invariants, and composite-PK correctness; MIGRATIONS → db.test.ts with toBe(MIGRATIONS.length) plus migration-specific tests; AGENTS → agents.test.ts covering name, displayName, binary, and detect(), with binary a constant placeholder string.
• 🔴 (10:00) User identified standing regression batteries: registry contracts in packages/core/test/sync-registry-contract.test.ts; per-operation sync invariants in packages/core/src/sync-data.ts via assertSyncInvariants; sequence properties in packages/gateway/test/sync.property.test.ts covering invariants after every operation, prune-floor liveness, eventual convergence, no ping-pong, and currentTier correctness.
• 🔴 (10:00) User stated skip/early-return branches are the highest-risk surface; NOT EXISTS, latest <> 'upsert', and content-match guards previously hid data-loss defects. Every skip branch needs a test that makes it incorrectly skip something and proves it must not.
• 🔴 (10:00) User stated property-based tests with faithful mocks found four sequence-dependent engine bugs missed by example tests across enable/disable/push/pull boundaries.
• 🔴 (10:00) User stated shared memoized arrays such as the syncedColumns PRAGMA cache should be frozen so mutation throws TypeError in strict mode rather than silently corrupting shared cache state.
• 🔴 (10:00) User stated db().query monkeypatches are shadowed by the tracing Proxy; SQL-counting tests must use log.registerSink({ withDbSpan }), because assigning db().query = ... is silently ignored, as empirically verified in PR #874.
• 🔴 [enforced-review] (10:00) User required two separate review passes for non-trivial changes: 1. adversarial/correctness review asking “What state makes this break?” and checking enumerated cases, apply-then-revert mutants, property stability ×10, hash equivalence, frozen-array mutation safety, and stale caches; 2. security/pentest review covering injection, credential leakage, protocol attacks, and privilege escalation. The adversarial review is mandatory before merge; security review depends on risk profile.
• 🔴 [enforced-validation] (10:00) User required reviewers to run pnpm test, pnpm run typecheck, pnpm run lint, and pnpm run format:check, checking exit codes because blank output tails may conceal lint/format failures; run property tests ×10; hash-verify an unmodified final working tree; and report per-point PASS/FAIL/CONCERN/MUST-FIX verdicts with file:line evidence plus overall MERGE/DO-NOT-MERGE, never an unstructured “LGTM.”
• 🔴 (10:00) User stated changes to sync engine, hashing, lifecycle, or migrations always require adversarial review regardless of size; mechanical or test-only PRs may skip it at reviewer discretion.
• 🔴 (10:00) User stated embedding paths must use embedInTokenBatches; batches are limited by token area MAX_BATCH_TOKEN_AREA = 4096, but the batching loop must always admit the first item so it progresses. Input order must be preserved and storeTemporalChunks called once.
• 🔴 (10:00) User stated temporal embedding input is capped at 64 chunks: retain the first 63 and combine the remainder into one tail; an oversized tail may use its own batch and is truncated downstream.
• 🔴 (10:00) User stated embedding worker resolution uses __filename for CJS and import.meta.url for ESM because Bun/tsx ESM workers resolve from the module URL while bundled CJS always defines its filename. The CJS shim must remain; both branches must not be replaced with import.meta.url or current-working-directory resolution.
• 🔴 (10:00) User stated embedding-worker replacement, including respawnForWasm(), requires a confirmed worker-exit barrier. Clearing worker/initPromise before termination can let concurrent ensureWorker() start WASM while native inference still owns the model; retirement must remain visible until exit, no second query may run during shutdown, and failure to confirm bounded termination must latch the provider unavailable rather than overlap models.