Distillation

Date: Sep 17, 2026

• 🔴 (09:34) User stated packages/core/test/embedding-pool.test.ts exercises cross-worker dispatch via _setTestWorkerFactory, including least-busy routing, lazy/memory-gated growth, the broken-provider latch, and pool shutdown; the per-worker OOM/self-heal lifecycle belongs exclusively in packages/core/test/embedding-oom-recovery.test.ts, and embedding-pool.test.ts never touches it.
• 🔴 (09:34) User stated embedding-pool initialization watchdog failures must be normalized at the readiness boundary to LocalProviderUnavailableError, retaining EmbeddingWorkerWatchdogError with stage: "init" as the cause.
• 🔴 (09:34) User stated container-aware pool admission must remain at one worker when host os.freemem() reports 64 GiB but the cgroup cap is 256 MiB; this prevents a second native-ONNX worker from exceeding memory.max and being SIGKILLed, as occurred in Aditya’s Railway container.
• 🔴 (09:34) User stated embedding-pool admission must reserve memory headroom while a newly admitted worker is still initializing, so the same headroom cannot be spent repeatedly.
• 🔴 (09:34) User stated a fatal worker initialization error degrades the module-global provider to unavailable/FTS-only, while a transiently failed slot is retired after a sibling recovers and must not be replaced before the transient retry cooldown.
• 🔴 (09:34) User stated the pool’s cold bootstrap must be serialized until one real result proves model readiness; only after bootstrap may genuine concurrent demand grow the pool.
• 🔴 (09:34) User stated the settle<T>() helper in packages/core/test/embedding-pool.test.ts attaches fulfillment and rejection handlers immediately so expected rejections are never flagged as "unhandled".
• 🔴 (09:34) User stated persisted embedding-cap behavior in packages/core/test/embedding-cap-persistence.test.ts: _readPersistedEmbedCap() returns null when absent, corrupt, or missing required fields; _persistEmbedCap() round-trips cap and positive freeMemBytes, overwrites via ON CONFLICT, preserves positive knownBadCap, and omits/ignores knownBadCap when 0, undefined, non-positive, or non-numeric. Persistence key is lore:embedding_cap.
• 🔴 (09:36) User stated packages/core/test/embedding-oom-recovery.test.ts exclusively exercises the stateful LocalProvider OOM lifecycle: worker "exit" with EMBED_OOM_EXIT_CODE → handleOomBackoff → resubmitPending, including the token-floor latch and synchronous respawn failure.
• 🔴 (09:36) User stated OOM recovery lowers the cap by ×0.7 using backoffEmbedCap(EMBED_TOKEN_CEILING), persists the lowered cap, spawns a fresh worker, and resubmits the same in-flight request ID; an OOM at MIN_EMBED_TOKENS does not respawn, rejects with LocalProviderUnavailableError, and latches the provider into FTS-only mode.
• 🔴 (09:36) User stated repeated OOMs must descend monotonically through the full backoff ladder to MIN_EMBED_TOKENS before latching, and a synchronous failure on the second worker spawn must reject pending requests rather than leave them unsettled.
• 🔴 (09:36) User stated worker creation must snapshot the host’s live stderr-silence state into workerData.stderrSilenced on every spawn because worker threads have separate globalThis; otherwise the worker’s [lore] OOM line can corrupt the host TUI.
• 🔴 (09:36) User stated settle<T>() in packages/core/test/embedding-oom-recovery.test.ts attaches handlers immediately so a rejection occurring while worker events are driven is never flagged as "unhandled".
• 🟡 (09:36) Search found healthy in packages/core/src/embedding/pool.ts at lines 84, 308, and 551: the slot field is declared, initialized to false, and later set to true.
• 🟡 (09:36) Search found embedding-cap persistence calls in packages/core/src/embedding/local.ts: persistEmbedCap at line 139, _persistEmbedCap at line 177, delegation at line 182, and production persistence at lines 791 and 857.
• 🔴 (09:37) User stated token-area batching must always admit the first item so the batching loop progresses, while preserving input order.
• 🔴 (09:37) User stated temporal embedding uses embedInTokenBatches with MAX_BATCH_TOKEN_AREA = 4096; temporal chunks are capped at 64 by retaining chunks 0..62 and joining index 63+ into one final tail, and storeTemporalChunks is called once.
• 🔴 (09:37) User stated structural "\n\x1f" boundaries and a 64-chunk temporal cap were chosen over splitting on tool-looking text, dropping overflow, or allowing unlimited vec0 rows because tool output may imitate headers and unbounded fan-out expands storage and KNN widening; units remain dependency-free, prose/reasoning is preserved verbatim, reduced tool envelopes are retained, and vectors are stored atomically.
• 🔴 (09:37) User stated embedding-worker stdout/stderr streams must be silently drained with resume() and no payload-observing "data" listeners, rather than logging sanitized or rate-limited lines, because native dependencies can emit model paths, provider diagnostics, IDs, and process state while every core log reaches persistent storage and LogSink; only fixed categorical lifecycle messages are retained.
• 🔴 (09:37) User stated lazy, container-aware embedding-pool growth was chosen over eagerly loading multiple ONNX workers because each model costs about 680 MB and host freemem() overstates capacity in constrained containers; worker 0 remains available, extra workers require concurrent demand plus clampFreeToContainerLimit(...) headroom, memDivisor shares the budget, and upward re-probes must never reach a prior OOM ceiling.
• 🔴 (09:37) User stated cgroup-aware availableParallelism() limits are used only when below cpus().length; always setting ONNX Runtime threads to the logical CPU count was rejected because native ORT is cgroup-blind and may create host-core-sized pools in a 1-vCPU container, wasting RSS and causing context-switch thrash. Unconstrained hosts return undefined, preserving ORT’s physical-core default instead of raising it to the hyper-thread count.
• 🔴 (09:37) User stated temporal backfill always uses "document" input type, so it never self-gates against the recall-load predicate.
• 🔴 (09:37) User stated recall load counts only single-text "query" embeds; write-time and backfill "document" embeds are excluded, admission reads the live counter rather than a snapshot, and provider disappearance, OOM, timeout, or rejection releases ownership in finally.
• 🔴 (09:37) User stated unresolved "document" embeds must not be used for embedding-pool concurrency or memory-cap fixtures because production document work is intentionally serialized and one unresolved request can leak admission into later tests; fixtures requiring simultaneous provider calls use single-text "query" embeds, while dedicated scheduler tests cover document serialization.
• 🔴 (09:37) User stated resolveReadMode() must trust physical storage layout over vec.storage_mode; committed vec0 and the connection latch override forged "blob" metadata, transaction-local provisional state remains rollback-safe, and blob fallback is allowed only after proving base embedding columns exist.
• 🔴 (09:37) User stated embedding startup must always run provider-independent checkConfigChange() before the provider-availability return and before maybeCutoverToVec0(); otherwise unavailable embeddings can skip physical vec0 repair, while reversing cutover order can seed vec0 from stale blobs.
• 🟡 (09:38) Focused test run under Vitest v4.1.8 passed 2 test files and all 13 tests in 2.89 seconds.
• 🟡 (09:38) Working copy tmzmtlss / commit 6a5b70c9 on fix/embedding-runtime-admission had description fix(core): correct embedding runtime selection and admission, parent 2cbf4006 on main, no revision conflict, and a conflicting bookmark named fix/responses-private-references.
• 🟡 (09:38) Working-copy changes comprised 20 files: added packages/core/src/cgroup-memory.ts and packages/core/test/cgroup-memory.test.ts; modified packages/core/src/embedding/local.ts, packages/core/src/embedding/pool.ts, packages/core/src/embedding-cap.ts, packages/core/src/embedding-worker-types.ts, packages/core/src/embedding-worker.ts, packages/core/src/embedding.ts, packages/core/src/ort-native.ts, packages/core/test/embedding-cap.test.ts, packages/core/test/embedding-pool-memory.test.ts, packages/core/test/embedding-pool.test.ts, packages/core/test/embedding-wasm-fallback.test.ts, packages/core/test/embedding-worker-stdio.test.ts, packages/core/test/embedding-worker-types.test.ts, packages/core/test/ort-native.test.ts, packages/gateway/script/bundle.ts, packages/gateway/script/ort-platform-package.ts, packages/gateway/test/bundle-exports.test.ts, and packages/gateway/test/ort-platform-package.test.ts.
• 🟡 (09:38) Scoped patch SHA-256 was ab8a298011cb88a89aa2ac41f70ad718afc296605e867007d1c298f679f43b5f; a later scoped patch SHA-256 was a781cff720fd7624addc687a1830db1cbebde9153c10bd961dc08ecbe374b9f7.
• 🟡 (09:40) Focused Vitest run produced 2 failed and 122 passed tests across 3 files: packages/core/test/embedding-cap.test.ts:102 expected resolveMemoryHeadroom(0, undefined, 7 * GB) to equal 7516192768 but received undefined; packages/core/test/embedding-pool-memory.test.ts:176 expected MIN_EMBED_TOKENS (256) but received 4948 when cgroup accounting failed without a hard limit.
• 🟡 (09:43) Assistant gave a scoped DO-NOT-MERGE verdict for tmzmtlss at 6a5b70c9ee77, parent 2cbf40060c67, and stated no files or VCS state were mutated.
• 🟡 (09:43) Assistant identified high-severity issue 1: unknown memory headroom fails open to host memory because constrainedMemoryLimit() uses 0 for both unconstrained and unknown states in packages/core/src/embedding/local.ts:86-97, clampFreeToContainerLimit() returns hostFree without a finite hard limit/headroom in packages/core/src/embedding-cap.ts:165-173, and packages/core/src/embedding/pool.ts:282-287 blocks unknown headroom only when constrained > 0. Evidence: packages/core/test/embedding-pool-memory.test.ts:164-176 expected 256 but got 4948. Recommended fix: return a tagged known, known-unconstrained, or unknown sample; only explicit unconstrained state may use host memory, while unknown becomes zero headroom; add a pool-growth regression for unknown soft-limit accounting.
• 🟡 (09:43) Assistant identified high-severity issue 2: pool admission can double-spend memory during OOM recovery. Slots begin unhealthy and become healthy after one success at packages/core/src/embedding/pool.ts:290-315 and :550-575, but never become unhealthy again; growth reserves only for !slot.healthy at :262-269; an OOM replaces the worker internally at packages/core/src/embedding/local.ts:720-727 and :876-919 without notifying the pool, so temporarily increased sampled headroom can admit another pool slot while the replacement model reloads. Recommended fix: track current worker-generation residency separately from historical health, reserve a full model budget from exit until the replacement completes a real embed, and add an OOM-replacement-plus-concurrent-growth regression.
• 🟡 (09:43) Assistant identified high-severity issue 3: the known-bad cap is neither hard nor monotonic. reprobeEmbedCap() applies Math.max(cap, …) after calculating the known-bad ceiling at packages/core/src/embedding-cap.ts:354-365, making reprobeEmbedCap(3000, 64 * GB, 2400) === 3000; handleOomBackoff() uses Math.max(this.lastOomCap, capBefore) at packages/core/src/embedding/local.ts:845-857, so failures at 4000 then 2800 remember 4000 and may re-probe to the failed 2800; each provider snapshots knownBadCap once at local.ts:468-475 and unconditionally overwrites shared persistence at :139-155 and :782-791, allowing stale siblings to erase tighter learned limits. Recommended fix: keep the lowest positive failed cap in shared pool/process state, atomically merge it into persistence, apply it as a final clamp even when lowering the current cap, and test repeated descending OOMs plus two stale sibling providers.
• 🟡 (09:43) Assistant identified high-severity issue 4: one admission decision combines inconsistent memory samples. containerFreeBytes() samples the limit and then availableMemoryHeadroom() samples it again at packages/core/src/embedding/local.ts:99-136; the pool repeats this at packages/core/src/embedding/pool.ts:282-287. A first sample of 0, second finite sample, and failed scoped-headroom read can yield (hostFree, 0, undefined) and fail open; constructor sampling at local.ts:472-475 can also anchor a low initial cap to a high re-probe baseline. Recommended fix: use one immutable memory snapshot per admission decision and one initial sample for both cap and baseline; add a sequence-driven limit-change regression.
• 🟡 (09:43) Assistant identified medium-severity issue 5: non-finite values do not consistently fail closed. clampFreeToContainerLimit() at packages/core/src/embedding-cap.ts:160-173 does not validate hostFree, so positive infinity can admit workers to the pool ceiling; backoffEmbedCap() at embedding-cap.ts:256-259 can return Infinity or NaN; persisted values at packages/core/src/embedding/local.ts:53-64 only receive typeof number checks, and JSON 1e309 parses as infinity. Recommended fix: normalize non-finite or negative memory/cap inputs to a safe floor or zero before arithmetic and add table tests for NaN, both infinities, negative values, and production pool growth.
• 🟡 (09:43) Assistant identified medium-severity issue 6: persisted free memory and startup free memory use inconsistent units. Startup divides current memory by memDivisor before reconciliation at packages/core/src/embedding/local.ts:160-170, while persistence at :782-791 and :845-857 and the in-process baseline at :475 use undivided memory; with pool ceiling 2, unchanged total memory appears halved after restart and doubled total memory appears unchanged. Recommended fix: persist the divisor with the learned cap, or compare total memory with total memory while deriving per-worker model cap; test restarts across pool-ceiling changes.
• 🟡 (09:43) Assistant validation reported 2 failed and 122 passed tests in the focused candidate run (embedding-cap.test.ts:102, embedding-pool-memory.test.ts:176), while OOM recovery and persistence suites passed all 13 tests but lacked coverage for lower repeated failures and stale sibling writers.
• 🟡 (09:43) Assistant excluded cgroup mount-resolution internals from the audit as requested and manually read AGENTS.md, quality/REVIEW.md, and package.json because the required repo-setup skill was unavailable.