Distillation

Date: Sep 16, 2026

• 🔴 (23:43) User stated the adaptive embedding token cap never exceeds the Nomic v1.5 maximum sequence length.
• 🔴 (23:43) User stated each embedding OOM lowers the token cap by a factor of 0.7 and respawns the worker with a fresh heap.
• 🔴 (23:43) User stated MIN_EMBED_TOKENS = 256.
• 🔴 (23:43) User stated that when free memory never improves, the learned embedding limit is never re-probed.
• 🔴 (23:43) User stated a memory-rich reboot must never re-probe upward to a token cap beyond what the WASM heap can support.
• 🔴 (23:43) User stated embedding-pool sizing must never fall below 1; the primary worker always runs.
• 🔴 (23:43) User stated that in a memory-capped container, relying on host-level os.freemem() can lead to an uncatchable process termination, so the ×0.7 OOM backoff never fires; the WASM path must instead self-limit before that point.
• 🟡 (23:43) The embedding memory model subtracts a measured approximately 665 MB baseline footprint before sizing transient allocation, with additional margin and OOM backoff for residual error.
• 🟡 (23:43) Pool sizing accepts configured from search.embeddings.embedPoolSize or LORE_EMBED_POOL_SIZE, estimates how many workers fit into freeBytes using Math.floor(free / PER_WORKER_MEM_BUDGET_BYTES), and preserves a minimum of one primary worker.
• 🟡 (23:43) resolveMemoryHeadroom selects trustworthy current headroom across runtimes, accounting for Linux/container limits and runtime-reported availability rather than trusting host os.freemem() inside a constrained container.
• 🟡 (23:43) capFreememAtLearn is monotonic and acts as a no-op on an unconstrained host.
• 🟡 (23:43) The WASM token-cap calculation accounts for the ONNX WASM runtime linear-memory hard cap, WASM heap fragmentation, ORT non-attention intermediates, and quadratic attention memory through Math.sqrt(budget / EMBED_ATTENTION_BYTES_PER_TOKEN_SQ) followed by clampEmbedCap(...).
• 🟡 (23:43) The token-cap rationale notes that effective context plateaus around approximately 2048 tokens, while a cap around approximately 4962 trades some theoretical sequence length for memory safety.
• 🟡 (23:43) Persisted-cap handling uses PersistedEmbedCap, EMBED_CAP_TRUST_BAND, and an explicit ceiling; stale learned caps from before the ceiling existed, such as an old value of 7000, are constrained by the newer bound.
• 🟡 (23:43) Upward cap re-probing never steps downward, and its path cannot return a cap above the ceiling even if supplied an already above-ceiling cap.
• 🟡 (23:43) Temporal re-chunk backfill throttling uses BACKFILL_FULLSPEED_CORES, computes scaled = cores / BACKFILL_FULLSPEED_CORES, and exposes backfillThrottleSleepMs(...).