Distillation

Date: Sep 16, 2026

• 🔴 (15:08) User stated the model-cache cleanup path in packages/core/src/embedding-worker.ts is only safe in npm mode and is never called for vendored binaries.
• 🔴 (15:08) User stated the embedding runtime switched to native because native ONNX Runtime scales with CPU cores; the corresponding WASM tuning is therefore a no-op on the native path.
• 🟡 (15:08) packages/gateway/script/ort-platform-package.ts generates esbuild-style per-platform packages containing the native onnxruntime-node addon and shared libraries, avoiding onnxruntime-node’s GitHub-release postinstall download and remaining compatible with npm 12, offline, air-gapped, and proxied installs.
• 🟡 (15:08) ORT_NPM_PLATFORM_DEFINITIONS contains exactly 5 targets in order: 1. linux-x64 (linux/x64), 2. linux-arm64 (linux/arm64), 3. darwin-arm64 (darwin/arm64), 4. win32-x64 (win32/x64), 5. win32-arm64 (win32/arm64). darwin-x64 is absent because onnxruntime-node 1.27.0 dropped its Intel macOS prebuilt, so Intel Macs fall back to WASM.
• 🟡 (15:08) buildOrtPlatformPackages() writes each package beneath outDir/onnxruntime-<target>/, copies native files flat into the package root, and generates package.json with name, gateway-locked version, loreOrtNodeVersion, one-element os and cpu arrays, preferUnplugged: true, sorted files, license: "FSL-1.1-Apache-2.0", and publishConfig: { access: "public" }. It intentionally omits an exports field so require.resolve("<pkg>/onnxruntime_binding.node") can deep-resolve the addon.
• 🟡 (15:08) The platform-package CLI is tsx script/ort-platform-package.ts --out <dir> [--version x.y.z]; --out is required and exits with status 1 when absent, while --version defaults to the gateway package version.
• 🟡 (15:09) packages/core/src/embedding/local.ts persists adaptive embedding-cap state under kv_meta key lore:embedding_cap with fields cap, freeMemBytes, and optional positive knownBadCap; malformed or inaccessible persisted data is ignored.
• 🟡 (15:09) packages/core/src/embedding/local.ts sets EMBED_REPROBE_INTERVAL_MS = 5 * 60_000. containerFreeBytes() clamps host freemem() through clampFreeToContainerLimit() using constrainedMemoryLimit() and availableMemoryHeadroom().
• 🟡 (15:09) availableMemoryHeadroom() uses Linux cgroup headroom from readLinuxCgroupMemoryHeadroomBytes(). For Bun it avoids trusting process.availableMemory() because Bun currently aliases host freemem() inside constrained cgroups; Node uses process.availableMemory() when available and falls back safely if it throws.
• 🔴 (15:09) User stated the embedding pool’s primary worker is always present; constrained-host protection for that worker comes from its own OOM backoff rather than pool sizing, preserving the prior single-worker behavior.
• 🟡 (15:09) packages/core/src/embedding/pool.ts lazily grows beyond the primary worker only after at least one slot has completed a real embed. This prevents concurrent cold workers sharing the HuggingFace cache from reading or purging one another’s partial downloads.
• 🟡 (15:09) Pool growth requires a healthy slot, all eligible workers being busy, total active plus retired workers below this.ceiling, and live free memory of at least (unhealthy slot count + 1) * PER_WORKER_MEM_BUDGET_BYTES.
• 🟡 (15:09) Each LocalProvider receives the pool ceiling as its memory divisor, sizing each worker’s token cap from free / ceiling; because the ceiling is memory-gated by desiredEmbedPoolSize, provisioned workers collectively remain within one EMBED_MEM_FRACTION share of free memory instead of independently claiming memory and summing to an OOM.
• 🟡 (15:09) During transient initialization-failure debt, packages/core/src/embedding/pool.ts routes new work only to healthy slots. After cooldown it admits exactly one recovery probe, while exhausted retry debt prevents a replacement storm; if a healthy sibling remains, preserveHealthyServiceAfterExhaustion() clears the global FTS-only latch without resetting the exhausted retry budget.
• 🟡 (15:09) Assistant identified a resolver-integrity investigation: native loading is intentionally fail-closed and depends jointly on package identity, exact version locks, libc/target checks, canonical paths, and binary magic. The assistant was checking those contracts against generated bundle artifacts, with two suspected regression risks: stale artifacts silently forcing production back to WASM, or an overly broad fallback loading the wrong addon.