Distillation

Date: Sep 16, 2026

• 🔴 (23:44) User stated cgroup memory headroom is never admitted from a stale cgroup directory or a cached discovery miss.
• 🔴 (23:44) User stated sibling cgroup usage is never subtracted from the wrong scope; if the correct usage file is not readable, memory headroom must remain unknown rather than use another scope.
• 🔴 (23:44) User stated a host logger or sink must never interrupt embedding-worker stream consumption and let the stream stop draining.
• 🔴 (23:44) User stated embedding-pool growth never multiplies memory pressure on a constrained host.
• 🔴 (23:44) User stated container-aware memory clamping can only lower the free-memory figure, never raise it, so it never increases memory use.
• 🟡 (23:44) Added packages/core/src/cgroup-memory.ts to parse Linux cgroup membership and mount information, resolve cgroup v1/v2 memory files while retaining host-side mount roots, decode octal path escapes, and read container memory headroom.
• 🟡 (23:44) packages/core/src/cgroup-memory.ts treats a cgroup v2 limit value of max as Number.POSITIVE_INFINITY.
• 🟡 (23:44) Cgroup discovery is performed freshly so stale cgroup directories and cached discovery misses cannot be used for memory admission.
• 🟡 (23:44) Embedding worker admission now separates MODELED_WORKER_MEM_BUDGET_BYTES—the model plus a 2048-token attention estimate—from the resident high-water reserve used to prevent unsafe pool growth.
• 🟡 (23:44) PER_WORKER_MEM_BUDGET_BYTES now reflects EMBED_WORKER_RESIDENT_RESERVE_BYTES; tests expect it to equal exactly 1536 * MB.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:577-657, runOperation() owns the worker slot until native inference settles; caller AbortSignals are intentionally omitted from slot.provider.embed(...), because callers own only their waiters.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:581-615, each operation has a watchdog that starts in "init" using embedInitWatchdogMs, switches to "execution" using embedExecutionWatchdogMs, logs embedding worker watchdog expired: stage=${stage} timeout_ms=${timeoutMs}, rejects with EmbeddingWorkerWatchdogError(stage), uses Math.max(1, timeoutMs), and calls watchdogTimer.unref?.().
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:616-632, successful operations call recordSlotSuccess(slot), clone vectors only when retainResult === true and no waiters remain, clear operation.texts, settle all waiters, and either prune retained completed operations or remove the operation from this.operations.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:633-655, failed operations are removed, release text/vector payloads, settle every waiter with the owned error, and retire the slot for either LocalProviderUnavailableError or EmbeddingWorkerWatchdogError; the watchdog is cleared, slot.inflight is decremented, and dispatch resumes in finally.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:659-708, dispatch drops queued operations with no waiters and does not replace the last active slot or grow the pool while retiredWorkers still owns live native model/heap memory.
• 🟡 (23:44) dispatch() handles EmbeddingWorkerRetryCooldownError by scheduling retry at error.retryAt; other slot-selection failures remove the operation and settle all waiters with the owned error or LocalProviderUnavailableError.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:710-754, embed() rejects pre-aborted signals with EmbeddingRequestAbortedError, rejects a closing pool with LocalProviderUnavailableError("embedding pool is unavailable"), deduplicates requests by embeddingOperationKey(ownedTexts, inputType), prioritizes recall embeddings as "high", measures queued bytes with Buffer.byteLength(text), and rejects over-capacity work with EmbeddingQueueCapacityError.
• 🟡 (23:44) In packages/core/src/embedding/pool.ts:760-802, shutdown() is idempotent via shutdownPromise, marks the pool closing, clears retry dispatch, rejects all waiters with LocalProviderUnavailableError("embedding pool shut down"), releases queued texts/vectors and token-batch checkpoint vectors, shuts down all providers with Promise.allSettled, waits for retired workers, and throws AggregateError("embedding worker termination was not confirmed") if any shutdown cannot be confirmed.
• 🟡 (23:45) Added coverage in packages/core/test/cgroup-memory.test.ts for resolveLinuxCgroupMemoryFiles() and readLinuxCgroupMemoryHeadroomBytes(), including v1/v2 path resolution, malformed metadata, headroom results of exactly 100, 50, and 400, and fail-closed undefined outcomes.
• 🟡 (23:45) packages/core/test/embedding-cap.test.ts covers clampFreeToContainerLimit() and resolveMemoryHeadroom(): invalid or unknown limits (0, Number.NaN, negative) do not constrain host free memory unless current container availability is known; a constrained cgroup with unknown or invalid current usage fails closed to 0.
• 🟡 (23:45) resolveMemoryHeadroom() tests require the minimum valid headroom source: (12 * GB, 23 * MB, 7 * GB) → 23 * MB, (12 * GB, 7 * GB, 23 * MB) → 23 * MB, (12 * GB, undefined, 2 * GB) → 2 * GB, and (1 * GB, undefined, 2 * GB) → 1 * GB; wholly unknown inputs return undefined.
• 🟡 (23:45) desiredEmbedPoolSize() tests establish exact boundaries: 3 * GB - 1 → 1 worker, 3 * GB → 2 workers, 64 * GB → DEFAULT_MAX_EMBED_POOL, (64 * GB, 4) → 4, (64 * GB, 3.9) → 3, (1024 * GB, 100) → EMBED_POOL_ABS_MAX, (3 * PER_WORKER_MEM_BUDGET_BYTES, 4) → 3, and invalid/negative/zero free memory → 1.
• 🟡 (23:45) packages/core/test/embedding-pool-memory.test.ts contains the OOM-regression suite verifying that each pool worker’s token cap is sized from free / ceiling rather than full free memory, and that the token floor is used when constrained primary-worker headroom is unknown.