Dashboard › sentry › Session 0Pp2RQHgDRrW
0Pp2RQHgDRrWzZ4xBInvestigate Sentry issue LOREAI-GATEWAY-Q: Error: embedding worker shut down
Context from Sentry:
shutdown in the binary bundleThe stack trace:
at shutdown (dist-bin/bin.js:1222:5936)
at OXe (dist-bin/bin.js:1220:59277)
at <anonymous> (dist-bin/bin.js:2357:5953)
at <anonymous> (dist-bin/bin.js:2357:8752)
Your task:
Read packages/core/src/embedding.ts — find:
shutdown() method and what error it throwsembed() method and how it handles a dead workerensureWorker() behaves when the worker has diedSearch for "embedding worker shut down" string literal in the codebase
Check packages/core/src/search.ts or any file that calls the embedding functions during backfill
Determine: after PR #603's fixes (nulling this.worker on error/exit, try/catch on postMessage), does this error path still exist? The key question is whether shutdown() still throws "embedding worker shut down" when called after the worker has already exited.
Check if there's a backfill/retry loop that should gracefully handle a dead worker instead of propagating the error
Return: (1) exact code path that produces the error, with file:line references, (2) whether PR #603 already fixes this or if additional changes are needed, (3) proposed fix if needed
I'll investigate this systematically. Let me start by reading the key files and searching for the error string.
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
1: /**
2: * Embedding integration for vector search.
3: *
4: * Supports multiple embedding providers behind a common interface:
5: * - "local" (default): @huggingface/transformers + nomic-embed-text-v1.5
6: * (768 dims, Matryoshka-capable). Runs ONNX inference in a worker thread.
7: * - "voyage": Voyage AI API (voyage-code-3, 1024 dims)
8: * - "openai": OpenAI API (text-embedding-3-small, 1536 dims)
9: *
10: * Provides embedding generation, pure-JS cosine similarity, and vector search
11: * over the knowledge and distillation tables. All operations are gated behind
12: * `search.embeddings.enabled` config + the provider's API key env var — falls
13: * back silently to FTS-only when unavailable.
14: */
15:
16: import { db } from "./db";
17: import { config } from "./config";
18: import * as log ...
Found 1 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 533: p.reject(new LocalProviderUnavailableError("embedding worker shut down"));
Found 42 matches /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts: Line 240: * backfill batches). Line 484: // jump ahead of any queued backfill batches in the worker. Line 1124: * Check if embedding config has changed since the last backfill. Line 1182: // Startup backfill — single entry point for all hosts Line 1186: * Delay before the startup backfill begins, so the host's HTTP server has Line 1196: * Run all embedding backfills and log coverage stats. ...
Now let me read the rest of the embedding.ts file (the backfill functions) and check the search.ts file for any callers:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
1426: // Progress logging: heartbeat every PROGRESS_INTERVAL embedded so a long
1427: // backfill (e.g. 1000+ pending after a model change) doesn't look
1428: // like a silent hang. Without this, only the final tally was logged.
1429: const PROGRESS_INTERVAL = 256;
1430: let nextProgressAt = PROGRESS_INTERVAL;
1431:
1432: // Pre-compute text for token-budget batching
1433: const items = rows.map((r) => ({ ...r, text: r.observations }));
1434: let i = 0;
1435:
1436: while (i < items.length) {
1437: const batch = nextBatch(items, i);
1438: i += batch.length;
1439:
1440: try {
1441: const vectors = await embed(
1442: batch.map((b) => b.text),
1443: "document",
1444: );
1445: const update = db().prepare(
1446: "UPDATE distillations SET embedding = ? ...
Found 8 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 1209: export async function runStartupBackfill(): Promise<void> {
Line 1240: const knowledgeEmbedded = await backfillEmbeddings();
Line 1241: const distillationEmbedded = await backfillDistillationEmbeddings();
Line 1242: const entityEmbedded = await backfillEntityEmbeddings();
Line 1342: * Called by `runStartupBackfill()`.
Line 1347: export async function backfillEmbeddings(): Promise<number> {
Line 1412: export async function backfillDistillationEmbeddings(): Promise<number> {
Line 1486: export async function backfillEntityEmbeddings(): Promise<number> {
Found 12 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 514: /** Shut down the worker thread. Called by `resetProvider()` on config change.
Line 515: * Sends a shutdown message so the worker calls `process.exit(0)` internally.
Line 520: shutdown(): Promise<void> {
Line 540: worker.postMessage({ type: "shutdown" } satisfies WorkerInbound);
Line 626: export function resetProvider(): Promise<void> {
Line 627: let shutdownPromise: Promise<void> = Promise.resolve();
Line 629: shutdownPromise = cachedProvider.shutdown();
Line 632: return shutdownPromise;
Line 639: export function _shutdownAndDisable(): Promise<void> {
Line 640: let shutdownPromise: Promise<void> = Promise.resolve();
Line 642: shutdownPromise = cachedProvider.shutdown();
Line 645: return shutdownPromise;
Now let me check who calls runStartupBackfill and resetProvider/shutdown to understand the race condition:
Found 3 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts:
Line 835: embedding.runStartupBackfill().catch((e) => {
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 1209: export async function runStartupBackfill(): Promise<void> {
Line 1342: * Called by `runStartupBackfill()`.
Found 7 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 514: /** Shut down the worker thread. Called by `resetProvider()` on config change.
Line 626: export function resetProvider(): Promise<void> {
Line 639: export function _shutdownAndDisable(): Promise<void> {
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/test/embedding.test.ts:
Line 18: _shutdownAndDisable,
Line 669: await _shutdownAndDisable();
Line 739: await _shutdownAndDisable();
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/cli/start.ts:
Line 134: await embedding.resetProvider();
Let me check the pipeline.ts context where runStartupBackfill is called, and the start.ts context where resetProvider is called:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
825: stopFileWatcher = startKnowledgeFileWatcher(projectPath);
826: }
827: }
828:
829: // Startup backfills — idempotent, run once per process.
830: try {
831: distillation.backfillMetrics();
832: } catch (e) {
833: log.info("metric backfill failed:", e);
834: }
835: embedding.runStartupBackfill().catch((e) => {
836: log.error("embedding backfill failed:", e);
837: });
838:
839: // Index lat.md/ directory sections (content-hash-based, skips unchanged files).
840: try {
841: latReader.refresh(projectPath);
842: } catch (e) {
843: log.error("lat-reader startup refresh error:", e);
844: }
845:
846: // Pre-populate headerSessionIndex from DB so Tier 1 session identification
847: // works immediately after process restart. ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/cli/start.ts</path>
<type>file</type>
<content>
120: const actualPort = server.port;
121:
122: // Write port file so plugins can discover us (even on random port).
123: writePortFile(actualPort);
124:
125: const shutdown = async () => {
126: console.error("[lore] Shutting down…");
127: server.stop();
128: removePortFile(actualPort);
129: await resetPipelineState();
130: // Shut down the embedding worker thread gracefully. Done after
131: // resetPipelineState (which clears sessions/timers) but before
132: // safeExit — gives the worker time to exit cleanly via its
133: // "shutdown" message handler rather than being killed by _exit().
134: await embedding.resetProvider();
135: };
136:
137: if (candidatePort === 0) {
138: console.error(
139: ...
Now let me understand the exact error path. Let me trace through what happens when shutdown() is called while a backfill is in progress, and also check if there's a scenario where the error/exit handlers fire and then shutdown() gets called:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
406:
407: // Worker crash / exit — reject all in-flight requests.
408: // Null out `this.worker` in both handlers so the `?.` optional chaining
409: // in embed() prevents postMessage on a terminated Worker (LOREAI-GATEWAY-1T).
410: this.worker.on("error", (err: Error) => {
411: this.workerInitError = err.message;
412: this.workerReady = false;
413: this.worker = null;
414: this.initPromise = null;
415: log.error("embedding worker crashed:", err);
416: for (const [, p] of this.pendingRequests) {
417: p.reject(new LocalProviderUnavailableError(err));
418: }
419: this.pendingRequests.clear();
420: });
421:
422: this.worker.on("exit", (code) => {
423: if (code !== 0 && !this.workerInitError) {
424: ...
Now let me check if the Sentry breadcrumb message matches the backfill catch blocks, and look at the exact error type propagation:
Found 3 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 1388: `embedding backfill batch failed (${batch.length} items):`,
Line 1456: `distillation embedding backfill batch failed (${batch.length} items):`,
Line 1541: `entity embedding backfill batch failed (${batch.length} items):`,
Found 18 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts:
Line 188: export class LocalProviderUnavailableError extends Error {
Line 196: this.name = "LocalProviderUnavailableError";
Line 276: * surfaced as `LocalProviderUnavailableError` to mark the provider as
Line 282: throw new LocalProviderUnavailableError(this.workerInitError);
Line 287: if (localProviderKnownBroken) throw new LocalProviderUnavailableError();
Line 367: // LocalProviderUnavailableError so callers (embed() → isAvailable)
Line 374: pending.reject(new LocalProviderUnavailableError(msg.error));
Line 385: // LocalProviderUnavailableError on all pending + future requests.
Line 398: p.reject(new LocalProviderUnavailableError(msg.error));
Line 417: p.reject(new LocalProviderUnavailableError(err));
Line 438: ...
Now let me verify a critical detail: shutdown() rejects with LocalProviderUnavailableError, and the backfill catch blocks check instanceof LocalProviderUnavailableError and break. But the breadcrumb says "embedding backfill batch failed (8 items): Error: embedding worker shut down" — meaning log.error was called. Let me check if the Error in the breadcrumb is the LocalProviderUnavailableError or a plain Error:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
1380:
1381: for (let j = 0; j < batch.length; j++) {
1382: update.run(toBlob(vectors[j]), batch[j].id);
1383: embedded++;
1384: }
1385: } catch (err) {
1386: // log.error sends to Sentry via captureException
1387: log.error(
1388: `embedding backfill batch failed (${batch.length} items):`,
1389: err,
1390: );
1391: // Provider is dead — no point retrying remaining batches.
1392: if (err instanceof LocalProviderUnavailableError) break;
1393: }
1394: // No yieldToEventLoop() needed — embed() is truly async (worker thread).
1395: }
1396:
1397: if (embedded > 0) {
1398: log.info(`embedded ${embedded} knowledge entries`);
1399: }
(Showing lines 1380-1399 of 1554. Use offset=1400 to continue.)
</content>
Now let me check the log.error implementation to understand how it sends to Sentry:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/log.ts</path>
<type>file</type>
<content>
1: /**
2: * Lightweight logger that suppresses informational messages by default.
3: *
4: * In TUI mode, all stderr output renders as red "error" text — confusing
5: * for routine status messages like "incremental distillation" or "pruned
6: * temporal messages". Only actual errors should be visible by default.
7: *
8: * Set LORE_DEBUG=1 to see informational messages (useful when debugging
9: * the plugin itself).
10: *
11: * ## Sink registration
12: *
13: * An optional {@link LogSink} can be registered via {@link registerSink}.
14: * When registered, every log call (regardless of `isDebug`) also forwards
15: * to the sink. This is used by the gateway to bridge logs → Sentry without
16: * adding a Sentry dependency to `@loreai/core`. ...
Now I have the complete picture. Let me verify one more thing — the Sentry breadcrumb says "Error: embedding worker shut down", not "LocalProviderUnavailableError: ...". Let me check what LocalProviderUnavailableError's name property is and how log.error formats it:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
188: export class LocalProviderUnavailableError extends Error {
189: constructor(cause?: unknown) {
190: super(
191: "Local embedding provider unavailable: '@huggingface/transformers' failed to initialize. " +
192: "Recall will use FTS-only search. To use a remote provider instead, set " +
193: "search.embeddings.provider to 'voyage' or 'openai' in .lore.json " +
194: "and provide the corresponding API key (VOYAGE_API_KEY / OPENAI_API_KEY).",
195: );
196: this.name = "LocalProviderUnavailableError";
197: if (cause !== undefined)
198: (this as Error & { cause?: unknown }).cause = cause;
199: }
200: }
201:
202: /** Tracks whether the local provider has been probed and found unavailable.
(Showing lines 188-202 of 1554. Use offset=203 to continue.)
</content>
Now I have all the information needed. Let me also check what the Sentry stack trace shutdown function looks like — the Sentry issue says the error originates from shutdown, not from the backfill catch. This means the LocalProviderUnavailableError thrown by shutdown() line 533 is being sent to Sentry via captureException. Let me verify by checking if the log.error in the backfill catch does captureException on the error:
Looking at log.ts:170-171:
const err = findError(args);
if (err) sink?.captureException(err);
And the backfill catch at line 1385-1392:
catch (err) {
log.error(`embedding backfill batch failed (${batch.length} items):`, err);
if (err instanceof LocalProviderUnavailableError) break;
}
So the LocalProviderUnavailableError IS found by findError() (it's an Error instance), and captureException sends it to Sentry. The breadcrumb "embedding backfill batch failed (8 items): Error: embedding worker shut down" matches the formatArgs output where the error's .message would be the long LocalProviderUnavailableError message, but the Sentry issue title shows "Error: embedding worker shut down" — that's the cause string being displayed.
Wait — let me re-read the Sentry issue. The stack trace says:
at shutdown (dist-bin/bin.js:1222:5936)
This means Sentry is capturing the error at shutdown(), not at the backfill catch. The LocalProviderUnavailableError created at line 533 carries "embedding worker shut down" as the cause argument to the constructor, but the error's message is the long generic string. Sentry is likely reporting the cause string or the error shown in the breadcrumb is the cause.
Actually, re-reading the Sentry context more carefully: "Error: embedding worker shut down" — this is what Sentry shows as the issue title. The LocalProviderUnavailableError's message is the long generic one. The cause is "embedding worker shut down" (a string). ...
57893c5 fix(core): guard against postMessage on terminated embedding worker (#603) 0823301 feat(build): migrate standalone binary from Bun --compile to Node SEA via fossilize (#564) 5ae5e19 fix(embedding): use subquery for DISTINCT dedup in entity backfill SQL (#550) bb87617 fix(embedding): stop OOM event storm, single-thread WASM, remove auto-fallback (#545) e7813a5 feat(entities): embedding-based entity auto-dedup (#462) (#536) 8a42e99 style: enable Biome lint rules and fix all findings (#537) 78004bb style: introduce Biome and apply repo-wide formatting (#533) 9b2827a fix: address review findings from PR #475 and #477 (#479) 3009563 fix: filter embedding API key errors and harden beforeSend patterns (#477) b12bd01 fix: handle WASM abort in embedding worker and suppress shutdown noise (#464) 9cd94ec fix: retry with token-level truncation on ONNX OOM in embedding worker (#457) 7cc3b80 fix: remove distilled=0 filter from temporal vector search (#387) ee56309 fix: prevent Sentry noise ...
commit 57893c552d604fa3c9d5d513d9752a4f837a6dc8
Author: Burak Yigit Kaya <ben@byk.im>
Date: Sun Jun 7 16:21:19 2026 +0100
fix(core): guard against postMessage on terminated embedding worker (#603)
## Summary
Prevents `InvalidStateError: Worker has been terminated` by properly
nulling the worker reference on crash/exit and adding try/catch guards
around `postMessage()` calls.
## Changes
- Null `this.worker` in both `on('error')` and `on('exit')` handlers so
the `?.` optional chaining in `embed()` prevents posting to a dead
worker
- Reset `this.initPromise = null` in both handlers so future
`ensureWorker()` calls can re-create the worker after broken state is
cleared
- Wrap `postMessage()` in `embed()` with try/catch — rejects with
`LocalProviderUnavailableError` so callers degrade gracefully
- Wrap `postMessage()` in `shutdown()` with try/catch — resolves
immediately if the worker already exited
- Removed ...
497862f fix(gateway): remove cache fields from ZERO_USAGE and consolidate SSE extraction b79acad fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash (#602) 48770f2 fix(core): recognize WASM fatal error wrapper prefix in isWasmFatalError() (#604) f610388 fix(gateway): guard resp.usage accesses against undefined for vLLM/partial responses (#600) 122690b fix(gateway): handle SSE response from upstream when stream: false was sent (#605)
Now I understand the situation. PR #603 is already merged (commit 57893c5), but the current HEAD (497862f) is after that commit. The Sentry issue LOREAI-GATEWAY-Q is on release 0.24.1, which may predate PR #603. Let me check:
0.24.1 0.24.0
dc6de76 ci: add actionlint to catch workflow YAML errors (#612) bc5835f refactor(gateway): use static import for fossilize at module level 497862f fix(gateway): remove cache fields from ZERO_USAGE and consolidate SSE extraction 8cecd9b ci: add actionlint to catch workflow YAML errors df19b01 fix(gateway): bump fossilize to ^0.9.0 for programmatic API 8ac9c21 fix(ci): remove empty `with:` from pnpm/action-setup in eval.yml (#610) d9b5e7d fix(ci): remove empty `with:` from pnpm/action-setup in eval.yml 7ccc536 test(ci): quiet test harness noise + pretest hook for bundle (#608) 0613c6b fix(gateway): use fossilize programmatic API instead of impl-* glob dance b01e91d test(ci): quiet test harness noise + pretest hook for bundle b79acad fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash (#602) 48770f2 fix(core): recognize WASM fatal error wrapper prefix in isWasmFatalError() (#604) f610388 fix(gateway): guard resp.usage accesses against undefined for vLLM/partial ...
57893c5 fix(core): guard against postMessage on terminated embedding worker (#603) 0823301 feat(build): migrate standalone binary from Bun --compile to Node SEA via fossilize (#564) 5ae5e19 fix(embedding): use subquery for DISTINCT dedup in entity backfill SQL (#550) bb87617 fix(embedding): stop OOM event storm, single-thread WASM, remove auto-fallback (#545) e7813a5 feat(entities): embedding-based entity auto-dedup (#462) (#536) 8a42e99 style: enable Biome lint rules and fix all findings (#537) 78004bb style: introduce Biome and apply repo-wide formatting (#533) 9b2827a fix: address review findings from PR #475 and #477 (#479) 3009563 fix: filter embedding API key errors and harden beforeSend patterns (#477) b12bd01 fix: handle WASM abort in embedding worker and suppress shutdown noise (#464)
0.24.1 is ancestor of PR#603
b4a1612 release: 0.24.1 9cd94ec fix: retry with token-level truncation on ONNX OOM in embedding worker (#457) 59447b8 Merge branch 'release/0.24.0' 58b193b release: 0.24.0 d47c12f fix: detect and recover from stale OAuth credentials in background workers (#454)
Now let me see what shutdown() looked like in 0.24.1 (before PR #603):
this.pendingRequests.clear();
this.updateWorkerRef();
});
this.workerReady = true;
})().catch((err) => {
this.initPromise = null; // allow retry
throw err;
});
return this.initPromise;
}
/** Keep the worker ref'd while requests are in flight so the event loop
* doesn't exit before responses arrive. When the pending map drains,
* unref again so the worker doesn't prevent graceful process exit. */
private updateWorkerRef(): void {
if (!this.worker) return;
if (this.pendingRequests.size > 0) {
this.worker.ref();
} else {
this.worker.unref();
}
}
async embed(texts: string[], inputType: "document" | "query"): Promise<Float32Array[]> {
await this.ensureWorker();
// Pre-truncate texts that exceed the safe ONNX inference limit.
// This prevents OOM on single inputs near the model's 8192-token max.
const truncated = texts.map(safeLocalTruncate);
// Prepend Nomic task ...
Now I also need to check the error/exit handlers in 0.24.1 to see the pre-PR#603 state:
case "init-error": {
// Model init failed inside the worker — surface as
// LocalProviderUnavailableError on all pending + future requests.
this.workerInitError = msg.error;
this.workerReady = false;
localProviderKnownBroken = true;
if (!localProviderErrorLogged) {
localProviderErrorLogged = true;
log.error(
`local embedding provider failed to init: ${msg.error}. ` +
`Set VOYAGE_API_KEY/OPENAI_API_KEY for automatic remote fallback.`,
new Error(`embedding worker init failed: ${msg.error}`),
);
}
for (const [, p] of this.pendingRequests) {
p.reject(new LocalProviderUnavailableError(msg.error));
}
this.pendingRequests.clear();
this.updateWorkerRef();
break;
}
}
});
// Worker crash / exit — reject all ...
Now I have the complete picture. Let me also check the backfill catch in 0.24.1 to confirm the instanceof check was against plain Error:
log.error(`distillation embedding backfill batch failed (${batch.length} items):`, err);
// Provider is dead — no point retrying remaining batches.
if (err instanceof LocalProviderUnavailableError) break;
}
if (embedded >= nextProgressAt) {
log.info(`embedding distillations: ${embedded}/${rows.length}…`);
nextProgressAt = embedded + PROGRESS_INTERVAL;
}
// No yieldToEventLoop() needed — embed() is truly async (worker thread).
}
if (embedded > 0) {
log.info(`embedded ${embedded} distillations`);
}
return embedded;
}
Now I have the full picture. Here is my complete analysis:
Error: embedding worker shut down)The error originates in release 0.24.1, which predates PR #603. Here's the exact chain:
Step 1 — Backfill starts: pipeline.ts:835 fires embedding.runStartupBackfill() as fire-and-forget during the Anthropic POST /v1/messages pipeline setup.
Step 2 — Concurrent shutdown: While the backfill loop is iterating (embedding.ts, knowledge backfill at line ~1368, distillation backfill at ~1436, or entity backfill at ~1521), the gateway shutdown handler (start.ts:125-135) calls embedding.resetProvider() → LocalProvider.shutdown().
Step 3 — shutdown() rejects in-flight requests with a plain Error:
In 0.24.1, shutdown() (line ~469 in the old code) does:
// 0.24.1 version — the bug
for (const [, p] of this.pendingRequests) {
p.reject(new Error("embedding worker shut down")); // <-- plain Error, NOT LocalProviderUnavailableError
}
Step 4 — Backfill catch block doesn't break:
The backfill catch blocks (embedding.ts:1385-1392) check:
catch (err) {
log.error(`embedding backfill batch failed (${batch.length} items):`, err);
if (err instanceof LocalProviderUnavailableError) break; // <-- NOT triggered for plain Error
}
Since 0.24.1's shutdown() rejects with a plain Error (not LocalProviderUnavailableError), the instanceof check fails, and the backfill loop does not break. It continues to the next batch.
Step 5 — Next batch calls embed() → ensureWorker(): In 0.24.1, the error/exit handlers do NOT null out this.worker or reset this.initPromise. So ensureWorker() sees this.workerReady = false (set by shutdown()) but this.worker is still a reference to the terminated worker. The embed() method calls this.worker!.postMessage(...) on a terminated worker, throwing ...