Distillation

Date: Sep 17, 2026

• 🔴 (09:50) User stated embedding-cap recovery must never re-probe up to or past a persisted known-bad cap on a memory-rich reboot.
• 🔴 (09:50) User stated promise rejections that occur while driving worker events must never be flagged as "unhandled".
• 🔴 (09:50) User stated embedding-provider failure paths—including provider disappearance, OOM, and timeout—must never leak a permanent "busy" state.
• 🔴 (09:50) User stated the one-way process-shutdown latch must never reopen provider admission, including during config resets or late callbacks.
• 🟡 (09:50) packages/core/test/embedding-oom-recovery.test.ts uses FakeWorker extends EventEmitter and _setTestWorkerFactory to deterministically exercise the live LocalProvider OOM lifecycle: worker "exit" with EMBED_OOM_EXIT_CODE → handleOomBackoff → resubmitPending, floor latching, and synchronous respawn failure.
• 🟡 (09:50) FakeWorker.postMessage() snapshots each EmbedMsg as { type, id, maxTokens } because production mutates p.payload.maxTokens in place during resubmission; retaining object references would retroactively alter earlier captured messages.
• 🟡 (09:50) settle<T>() in packages/core/test/embedding-oom-recovery.test.ts immediately attaches fulfillment and rejection handlers, returning { ok: true, value } or { ok: false, err }, so early worker-driven rejections are not reported as unhandled.
• 🟡 (09:50) OOM recovery test setup removes VOYAGE_API_KEY and OPENAI_API_KEY, sets _setConstrainedMemoryForTest(0) and _setContainerFreeForTest(16 * 1024 * 1024 * 1024), resets the local-provider probe, and saves/clears the provider. Teardown restores environment variables and provider state and clears worker and memory overrides.
• 🟡 (09:50) OOM recovery test 1 persists _persistEmbedCap(8192, 0), verifies the first request starts at EMBED_TOKEN_CEILING, emits EMBED_OOM_EXIT_CODE, and verifies a fresh worker receives the same request ID at backoffEmbedCap(EMBED_TOKEN_CEILING), strictly below the first cap. The lowered cap is persisted through _readPersistedEmbedCap(), and the original promise resolves when the replacement worker returns one vector.
• 🟡 (09:50) OOM recovery test 2 persists MIN_EMBED_TOKENS, triggers OOM at that floor, and verifies there is no respawn, the pending request rejects with LocalProviderUnavailableError, and isAvailable() becomes false for FTS-only degradation.
• 🟡 (09:50) OOM recovery test 3 makes the second _setTestWorkerFactory invocation throw Error("synthetic spawn failure"); it verifies exactly 2 spawn calls and that the otherwise orphaned pending request rejects with LocalProviderUnavailableError.
• 🟡 (09:50) OOM recovery test 4 repeatedly emits EMBED_OOM_EXIT_CODE for at most 30 iterations, verifies caps are monotonically non-increasing, reaches MIN_EMBED_TOKENS, then latches without another respawn; the pending request rejects with LocalProviderUnavailableError and isAvailable() is false.
• 🟡 (09:50) OOM recovery test 5 calls silenceStderr(true) and verifies every worker spawn snapshots the host’s live stderr-silence state into workerData.stderrSilenced; this prevents a worker thread’s [lore] OOM output from corrupting the host TUI.
• 🟡 (09:50) packages/core/src/embedding/runtime.ts uses providerAdmissionClosed as a one-way process-shutdown latch and OwnedRetirements<object> as providerGenerations, retaining ownership of every local-provider generation until worker termination is known.
• 🟡 (09:50) resetProvider(timeoutMs?) tracks EmbeddingPool.shutdown(timeoutMs) through trackProviderGenerationShutdown() and sets cachedProvider to null when providerAdmissionClosed is already true, otherwise to undefined; this allows normal config reset to create a fresh generation without permitting late reset callbacks to reopen admission after shutdown.
• 🟡 (09:50) shutdownProvider(timeoutMs?) first sets providerAdmissionClosed = true, tracks shutdown of the current EmbeddingPool, sets cachedProvider = null, then awaits settleProviderGenerations(); failure to confirm termination uses message "embedding worker termination was not confirmed".
• 🟡 (09:50) _shutdownAndDisable is retained as a deprecated compatibility alias for shutdownProvider; _resetProviderShutdownTrackingForTest() clears settled generation failures.
• 🟡 (09:50) _saveAndClearProvider() preserves both cachedProvider and providerAdmissionClosed, temporarily resets admission to open and clears the cache; _restoreProvider() restores both fields.
• 🟡 (09:50) packages/core/src/embedding.ts publicly re-exports local-memory test seams _setAvailableMemoryForTest, _setConstrainedMemoryForTest, and _setContainerFreeForTest; pool seams _setEmbedPoolSizeForTest and _setPoolFreememForTest; and runtime lifecycle APIs including _resetProviderShutdownTrackingForTest, _saveAndClearProvider, _restoreProvider, _shutdownAndDisable, resetProvider, and shutdownProvider.
• 🟡 (09:51) Local embedding pool size is configured by search.embeddings.embedPoolSize or LORE_EMBED_POOL_SIZE; documented bounds are minimum 1 and maximum 8, default sizing is memory-gated to 1–2 workers, and each local worker loads its own approximately 137MB model.