Distillation

Date: Sep 16, 2026

• 🔴 (12:07) User stated native ONNX OOM can SIGKILL the entire gateway, so the ×0.7 backoff never fires; the only defense is never over-allocating in the first place.
• 🔴 (12:07) User stated the embedding-pool OOM regression reported by Onur occurred because the pool grew to N workers and each independently sized its token cap to consume roughly half of free memory, causing aggregate host RAM exhaustion.
• 🔴 (12:07) User’s required embedding-pool memory guards are: 1. each worker sizes from free / ceiling; 2. the cap is re-clamped to current free memory on every request, not only at construction.
• 🔴 (12:07) packages/core/test/embedding-pool-memory.test.ts contains 4 end-to-end OOM regressions: 1. with a pool ceiling of 2 and 6 * GB free, each worker’s maxTokens must equal memoryModelEmbedCap((6 * GB) / 2) and be below the undivided cap; 2. with a constrained 12 * GB limit but Number.NaN available memory, the primary worker must use MIN_EMBED_TOKENS; 3. with ceiling 1, the same worker must re-clamp from memoryModelEmbedCap(6 * GB) to memoryModelEmbedCap(2 * GB) after free memory drops—the pool never grew because its ceiling was 1; 4. after EMBED_OOM_EXIT_CODE, a respawned worker must resubmit the same request ID at memoryModelEmbedCap(2 * GB), below backoffEmbedCap(first.maxTokens).
• 🔴 (12:07) In packages/core/src/cgroup-memory.ts, cgroup membership and mount discovery are re-read from /proc/self/cgroup and /proc/self/mountinfo on every call so a worker is never admitted from a stale cgroup directory or a cached discovery miss.
• 🔴 (12:07) resolveLinuxCgroupMemoryFiles() prefers an explicit cgroup v1 memory-controller membership over a unified cgroup v2 membership on hybrid hosts; it selects the longest matching direct mount root and accepts a namespaced interpretation only when exactly one candidate exists.
• 🔴 (12:07) decodeProcPath() decodes octal procfs escapes, rejects paths that are not absolute, contain \0, or include a .. segment, and normalizes accepted paths with posix.normalize().
• 🔴 (12:07) readLinuxCgroupMemoryHeadroomBytes() traverses from the process cgroup directory to the cgroup mount point and takes the minimum finite limit - usage headroom across boundaries. Every descendant boundary must be readable so sibling usage is never subtracted from the wrong scope; only a cgroup v2 root mount with root === "/" may omit its own memory files.
• 🔴 (12:08) User’s numeric cap policy requires non-finite input to resolve to the token floor, never NaN or Infinity; clampEmbedCap(Number.NaN) must equal MIN_EMBED_TOKENS.
• 🔴 (12:08) packages/core/test/embedding-cap.test.ts specifies resolveMemoryHeadroom() behavior: choose the tighter Linux or Node headroom; honor a Linux soft-only boundary when the hard limit is 0; use libuv headroom on constrained non-Linux Node; return undefined for constrained Bun without trustworthy usage; and never invent headroom on an unconstrained host.
• 🔴 (12:08) clampFreeToContainerLimit() contract in packages/core/test/embedding-cap.test.ts: unconstrained (<= 0) or invalid/negative limits leave host free memory unchanged; constrained headroom clamps host free memory; a finite constrained limit with missing or Number.NaN usage fails closed to 0; discovered headroom is honored even with an unlimited hard limit; and the result may never exceed the original host-free figure.
• 🔴 (12:08) Worker admission pricing uses PER_WORKER_MEM_BUDGET_BYTES === EMBED_WORKER_RESIDENT_RESERVE_BYTES === 1536 * MB; desiredEmbedPoolSize(3 * GB - 1) must be 1, while desiredEmbedPoolSize(3 * GB) must be 2.
• 🔴 (12:09) packages/core/test/embedding-pool.test.ts includes a burst-growth regression where _setEmbedPoolSizeForTest(4), constrained memory is 12 * GB, and available headroom is 1600 * 1024 * 1024; three concurrent requests may create only 2 workers because an admitted but not-yet-initialized worker’s budget must be reserved and not spent again.
• 🔴 (12:09) Additional pool admission regressions require remaining at one worker when: 1. free memory is 0 despite ceiling 2; 2. host free reports 64 * GB but the cgroup limit is 256 * 1024 * 1024; 3. host free reports 64 * GB, constrained memory is 12 * GB, and current cgroup headroom is only 23 * 1024 * 1024.
• 🔴 (12:09) Production-path pool tests require: 64 * GB free memory permits the default ceiling of 2 workers; 0 headroom keeps the pool at one worker; and a pool constructed under 0 headroom must later grow to 2 workers when headroom rises to 64 * GB.
• 🟡 (12:10) Assistant concluded the composed primary sizing path already fails closed: finite constraints without trustworthy usage yield zero free bytes, while conflicting Linux/libuv headroom readings resolve to their minimum.
• 🟡 (12:10) Assistant concluded the burst-growth path now charges every unproven worker slot before admitting another; primary sizing and burst reservation both have end-to-end regressions and were retained for later mutation proof.
• 🟡 (12:10) Work-plan status changed: completed—cgroup discovery/live-move regressions, primary sizing under unknown or conflicting constrained headroom, and reservation of unreflected memory during embedding-pool burst growth; in progress—native resolver and malformed-native fallback findings; pending—critical-guard failing-first/mutation proof, complete validation and exact-candidate freeze, independent correctness/security reviews, and committing the Jujutsu change with runtime guidance.