Dashboard › sentry › Session 1Yjc8a0YxYNu
1Yjc8a0YxYNutCblIExplore the embedding worker code in the loreai monorepo at /home/byk/Code/opencode-lore. I need a thorough understanding of:
packages/core/src/embedding-worker.ts - Read the full file. I need:
./embedding-worker-types)packages/core/src/embedding-worker-types.ts - Read the full file. I need to know what is exported and whether there are any imports of its own.
packages/core/src/embedding.ts - Focus on:
new Worker(...) calls around lines 331, 365)packages/core/test/embedding.test.ts - Focus on:
withLocalModel wrapper (lines 483-516 mentioned in the issue)loggedModelSkip flagwithLocalModelpackages/core/test/setup.ts - Read the test setup/preload file to understand vitest configuration.
packages/core/package.json - Note the test scripts and any related config (vitest config, etc.)
packages/core/vitest.config.ts or any vitest config file - Read to understand the test setup.
packages/core/tsconfig.json - Read to understand TypeScript module resolution (moduleResolution setting).
packages/gateway/script/build.ts - Read to understand how the worker is bundled for production (esbuild config related to the embedding-worker).
packages/gateway/src/embedding-worker.cjs if it exists - Note the bundled output (don't read fully, just confirm it exists). ...
The user wants me to explore the embedding worker code in the loreai monorepo thoroughly. Let me start by reading all the relevant files in parallel to get a comprehensive understanding.
Let me make parallel calls for:
I'll also need to check if any vitest config exists since it might be a different name.
I'll explore the embedding worker code thoroughly. Let me read all the relevant files in parallel.
<path>/home/byk/Code/opencode-lore/packages/core/src/embedding-worker.ts</path> <type>file</type> <content> 1: /** 2: * Embedding worker thread — runs ONNX inference via @huggingface/transformers 3: * off the main thread. 4: * 5: * This file is the entry point for a `node:worker_threads` Worker spawned by 6: * `LocalProvider` in `embedding.ts`. It owns the transformers.js pipeline 7: * and processes embed requests sequentially from a priority queue. Moving 8: * inference here keeps the main thread's event loop free — HTTP requests, 9: * SSE streams, and session APIs are no longer blocked during embedding. 10: * 11: * Communication uses `parentPort` message passing with structured clone. 12: * Float32Array vectors are sent back directly (Bun preserves identity). 13: * 14: * The worker applies Nomic's recommended post-processing: 15: * 1. Mean pooling (via pipeline option) 16: * 2. Layer normalization 17: * 3. ...
<path>/home/byk/Code/opencode-lore/packages/core/src/embedding-worker-types.ts</path>
<type>file</type>
<content>
1: /**
2: * Shared message types for the embedding worker thread.
3: *
4: * The embedding worker (`embedding-worker.ts`) runs ONNX inference via
5: * `@huggingface/transformers` in a separate `node:worker_threads` Worker
6: * so the main thread's event loop stays free during inference. This file
7: * defines the message protocol between the main thread (`LocalProvider`
8: * in `embedding.ts`) and the worker.
9: *
10: * Imported by both sides — keep this file free of runtime dependencies.
11: */
12:
13: // ---------------------------------------------------------------------------
14: // Main thread → Worker
15: // ---------------------------------------------------------------------------
16:
17: /** Request an embedding batch. */
18: export interface EmbedRequest {
19: type: "embed";
20: /** Monotonic request ID for correlating responses. ...
<path>/home/byk/Code/opencode-lore/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
1: /**
2: * Embedding integration for vector search.
3: *
4: * Supports multiple embedding providers behind a common interface:
5: * - "local" (default): @huggingface/transformers + nomic-embed-text-v1.5
6: * (768 dims, Matryoshka-capable). Runs ONNX inference in a worker thread.
7: * - "voyage": Voyage AI API (voyage-code-3, 1024 dims)
8: * - "openai": OpenAI API (text-embedding-3-small, 1536 dims)
9: *
10: * Provides embedding generation, pure-JS cosine similarity, and vector search
11: * over the knowledge and distillation tables. All operations are gated behind
12: * `search.embeddings.enabled` config + the provider's API key env var — falls
13: * back silently to FTS-only when unavailable.
14: */
15:
16: import { db } from "./db";
17: import { config } from "./config";
18: import * as log from "./log";
19: import { vendorModelInfo } from ...
<path>/home/byk/Code/opencode-lore/packages/core/test/embedding.test.ts</path>
<type>file</type>
<content>
1: import {
2: afterAll,
3: afterEach,
4: describe,
5: test,
6: expect,
7: beforeEach,
8: } from "vitest";
9: import { db, ensureProject } from "../src/db";
10: import {
11: cosineSimilarity,
12: toBlob,
13: fromBlob,
14: isAvailable,
15: vectorSearch,
16: vectorSearchEntities,
17: checkConfigChange,
18: _shutdownAndDisable,
19: _saveAndClearProvider,
20: _restoreProvider,
21: embed,
22: LocalProviderUnavailableError,
23: pickRemoteFallback,
24: _resetLocalProviderProbe,
25: _markLocalProviderUnavailable,
26: } from "../src/embedding";
27:
28: describe("cosineSimilarity", () => {
29: test("identical vectors return 1.0", () => {
30: const a = new Float32Array([1, 2, 3]);
31: expect(cosineSimilarity(a, a)).toBeCloseTo(1.0, 5);
32: });
33:
34: test("opposite vectors return -1.0", () => {
35: const a = new Float32Array([1, ...
<path>/home/byk/Code/opencode-lore/packages/core/test/setup.ts</path>
<type>file</type>
<content>
1: import { mkdtempSync, rmSync } from "node:fs";
2: import { join } from "node:path";
3: import { tmpdir } from "node:os";
4: import { afterAll } from "vitest";
5: import { close } from "../src/db";
6:
7: // Create an isolated temporary database for the entire test run.
8: // This prevents test fixtures from leaking into the live lore DB
9: // at ~/.local/share/lore/lore.db.
10: const tmp = mkdtempSync(join(tmpdir(), "lore-test-"));
11: process.env.LORE_DB_PATH = join(tmp, "test.db");
12:
13: // ---------------------------------------------------------------------------
14: // Block live network to models.dev during tests.
15: //
16: // `fetchModelData()` (gateway/src/worker-model.ts) hits
17: // https://models.dev/api.json to pull pricing/limits, and the gateway
18: // pre-warms it on startup (pipeline.ts). ...
<path>/home/byk/Code/opencode-lore/packages/core/package.json</path>
<type>file</type>
<content>
1: {
2: "name": "@loreai/core",
3: "version": "0.24.1",
4: "type": "module",
5: "license": "FSL-1.1-Apache-2.0",
6: "description": "Shared memory engine for Lore — three-tier storage, distillation, gradient context management",
7: "main": "./dist/node/index.js",
8: "types": "./dist/node/index.d.ts",
9: "exports": {
10: ".": {
11: "types": "./dist/types/index.d.ts",
12: "bun": "./dist/bun/index.js",
13: "default": "./dist/node/index.js"
14: }
15: },
16: "imports": {
17: "#db/driver": {
18: "bun": "./src/db/driver.bun.ts",
19: "default": "./src/db/driver.node.ts"
20: }
21: },
22: "scripts": {
23: "typecheck": "tsc --noEmit",
24: "build": "tsx script/build.ts"
25: },
26: "dependencies": {
27: "@huggingface/hub": "2.11.0",
28: "@huggingface/transformers": "^3.7.1",
29: "micromark": "^4.0.0",
30: ...
No files found
<path>/home/byk/Code/opencode-lore/packages/core/tsconfig.json</path>
<type>file</type>
<content>
1: {
2: "extends": "../../tsconfig.base.json",
3: "compilerOptions": {
4: "noEmit": true
5: },
6: "include": ["src", "test", "script"],
7: "exclude": ["src/db/driver.bun.ts"]
8: }
(End of file - total 8 lines)
</content>
<path>/home/byk/Code/opencode-lore/packages/gateway/script/build.ts</path> <type>file</type> <content> 1: /** 2: * Build @loreai/gateway. 3: * 4: * Two build modes: 5: * 6: * 1. `bun run script/build.ts` (default) 7: * Produces dist/index.js — publishable ESM bundle for npm. 8: * @loreai/core is external (workspace dep, installed alongside). 9: * 10: * 2. `bun run script/build.ts --binary` 11: * Delegates to `script/build-binary-sea.ts` which produces a 12: * standalone Node SEA binary via fossilize. The legacy Bun 13: * `--compile` pipeline was removed in #551 in favor of Node SEA 14: * because Bun's WASM engine has unfixed bugs that cause ONNX 15: * embedding OOM on all platforms (oven-sh/bun#18145, #25677, #31158). ...
No files found
I have most of the data I need. Let me search for more details:
Let me do these in parallel.
Let me find the vitest config, bundled output, and bundle script:
/home/byk/Code/opencode-lore/vitest.config.ts
No files found
/home/byk/Code/opencode-lore/packages/gateway/script/build.ts /home/byk/Code/opencode-lore/packages/gateway/script/build-binary-sea.ts /home/byk/Code/opencode-lore/packages/gateway/script/smoke-test.ts /home/byk/Code/opencode-lore/packages/gateway/script/bundle.ts /home/byk/Code/opencode-lore/packages/gateway/script/record-session.ts /home/byk/Code/opencode-lore/packages/gateway/script/native-loader.cjs /home/byk/Code/opencode-lore/packages/gateway/script/vendor-embeddings.ts /home/byk/Code/opencode-lore/packages/gateway/script/vendor-paths.ts /home/byk/Code/opencode-lore/packages/gateway/script/debug-id.ts
/home/byk/Code/opencode-lore/packages/website/tsconfig.json /home/byk/Code/opencode-lore/tsconfig.base.json /home/byk/Code/opencode-lore/packages/opencode/tsconfig.json /home/byk/Code/opencode-lore/packages/gateway/tsconfig.json /home/byk/Code/opencode-lore/packages/core/tsconfig.json /home/byk/Code/opencode-lore/packages/core/tsconfig.build.json /home/byk/Code/opencode-lore/packages/pi/tsconfig.build.json /home/byk/Code/opencode-lore/packages/pi/tsconfig.json /home/byk/Code/opencode-lore/tsconfig.json
<path>/home/byk/Code/opencode-lore/packages/core/src/embedding.ts</path>
<type>file</type>
<content>
1433: `distillation embedding backfill batch failed (${batch.length} items):`,
1434: err,
1435: );
1436: // Provider is dead — no point retrying remaining batches.
1437: if (err instanceof LocalProviderUnavailableError) break;
1438: }
1439:
1440: if (embedded >= nextProgressAt) {
1441: log.info(`embedding distillations: ${embedded}/${rows.length}…`);
1442: nextProgressAt = embedded + PROGRESS_INTERVAL;
1443: }
1444: // No yieldToEventLoop() needed — embed() is truly async (worker thread).
1445: }
1446:
1447: if (embedded > 0) {
1448: log.info(`embedded ${embedded} distillations`);
1449: }
1450: return embedded;
1451: }
1452:
1453: // ---------------------------------------------------------------------------
1454: // Backfill — entities
1455: // ...
Now let me read the vitest config and the bundle.ts and build-binary-sea.ts scripts, plus the base tsconfig:
Let me read the vitest config, the bundle script, build-binary-sea, and the base tsconfig:
<path>/home/byk/Code/opencode-lore/vitest.config.ts</path>
<type>file</type>
<content>
1: import { defineConfig } from "vitest/config";
2: import path from "node:path";
3:
4: export default defineConfig({
5: test: {
6: // Run all packages' tests
7: include: [
8: "packages/core/test/**/*.test.ts",
9: "packages/gateway/test/**/*.test.ts",
10: "packages/opencode/test/**/*.test.ts",
11: "packages/pi/test/**/*.test.ts",
12: ],
13: // Preload test setup for DB isolation
14: setupFiles: ["./packages/core/test/setup.ts"],
15: // Environment
16: environment: "node",
17: pool: "threads",
18: // Timeouts — generous for gateway startup and LLM operations
19: testTimeout: 300_000, // 5 min per test
20: hookTimeout: 300_000,
21: // Environment variables for test isolation
22: env: {
23: NODE_ENV: "test",
24: SENTRY_ENABLED: "0",
25: LORE_DEBUG: "0",
26: },
27: // Coverage is optional and run ...
<path>/home/byk/Code/opencode-lore/tsconfig.base.json</path>
<type>file</type>
<content>
1: {
2: "compilerOptions": {
3: "target": "ESNext",
4: "module": "ESNext",
5: "moduleResolution": "bundler",
6: "strict": true,
7: "esModuleInterop": true,
8: "skipLibCheck": true,
9: "declaration": true,
10: "types": []
11: }
12: }
(End of file - total 12 lines)
</content>
<path>/home/byk/Code/opencode-lore/tsconfig.json</path>
<type>file</type>
<content>
1: {
2: "extends": "./tsconfig.base.json",
3: "compilerOptions": {
4: "noEmit": true
5: },
6: "include": [
7: "packages/*/src",
8: "packages/*/test",
9: "packages/*/script",
10: "packages/*/scripts",
11: "packages/*/eval"
12: ],
13: "exclude": ["node_modules", "**/dist"]
14: }
(End of file - total 14 lines)
</content>
<path>/home/byk/Code/opencode-lore/packages/gateway/script/bundle.ts</path> <type>file</type> <content> 1: /** 2: * Bundle @loreai/gateway into a self-contained CJS package for npm/npx. 3: * 4: * Produces: 5: * dist/index.cjs — single CJS bundle (gateway + core + all JS deps) 6: * dist/bin.cjs — thin CLI wrapper with Node.js version check 7: * 8: * Everything is bundled except: 9: * - node:* built-ins (resolved at runtime) 10: * 11: * Source code is pure Node.js — no `Bun.*` polyfill layer is needed. 12: * (xxHash64 lives in src/xxhash.ts as a standalone module since there is 13: * no Node.js equivalent of `Bun.hash.xxHash64`.) 14: * 15: * Debug IDs are injected into the JS + sourcemap after bundling for Sentry 16: * source map resolution. When SENTRY_AUTH_TOKEN is set, sourcemaps are 17: * uploaded to Sentry and then deleted (they shouldn't ship to users). ...
<path>/home/byk/Code/opencode-lore/packages/gateway/script/build-binary-sea.ts</path> <type>file</type> <content> 1: /** 2: * Build @loreai/gateway standalone binary via Node SEA + fossilize. 3: * 4: * This replaces the legacy `bun build --compile` pipeline. The new 5: * pipeline uses: 6: * 7: * 1. esbuild → single CJS bundle (target: Node 22) 8: * 2. esbuild → worker CJS bundle 9: * 3. fossilize → Node SEA per target, with WASM files + model 10: * files + worker CJS embedded as SEA assets 11: * 12: * At runtime, the binary uses the WASM backend of 13: * `@huggingface/transformers` (i.e. `onnxruntime-web`'s Node entry). 14: * This is the path of least resistance: WASM runs correctly under 15: * Node's V8 engine (the bugs that forced this migration were 16: * specific to Bun's WASM engine — see `oven-sh/bun#18145`, `#25677`, 17: * `#31158`). 18: * 19: * Targets: 4 currently supported (Apple Silicon-only macOS, plus 20: * Linux x64/arm64 and Windows x64). ...
Now let me look at a few more things:
dist/embedding-worker.cjs file actually existingembedding-vendor.ts file referenced in the importsI need to:
Let me also check what bun test config was migrated from (per the lore knowledge).
Let me check the sea-entry.ts, the embedding-vendor, and verify the bundled output exists:
/home/byk/Code/opencode-lore/packages/gateway/src/cli/sea-entry.ts
No files found
/home/byk/Code/opencode-lore/packages/core/src/embedding-vendor.ts /home/byk/Code/opencode-lore/packages/core/src/embedding-vendor.d.ts /home/byk/Code/opencode-lore/packages/core/src/embedding-vendor.d.ts.map
total 11212 drwxr-xr-x 1 byk byk 182 Jun 6 20:01 . drwxr-xr-x 1 byk byk 150 Jun 7 14:44 .. -rwxrwxrwx 1 byk byk 802 Jun 5 14:01 bin.cjs -rw-r--r-- 1 byk byk 501577 Jun 5 14:01 embedding-worker.cjs -rw-r--r-- 1 byk byk 501023 Jun 5 14:01 embedding-worker.js -rw-r--r-- 1 byk byk 33 Jun 6 20:01 index.bun.js -rw-r--r-- 1 byk byk 1894752 Jun 5 14:01 index.cjs -rw-r--r-- 1 byk byk 8562894 Jun 5 14:01 index.cjs.map -rw-r--r-- 1 byk byk 3016 Jun 5 14:01 index.d.cts
Found 4 matches /home/byk/Code/opencode-lore/package.json: Line 20: "test": "vitest run", Line 28: "evals": "vitest run --config vitest.evals.config.ts", Line 37: "vitest": "^4.1.7", Line 38: "vitest-evals": "^0.10.0"
Now let me read sea-entry.ts and embedding-vendor.ts to fully understand the worker source mechanism:
Let me read the sea-entry.ts and the embedding-vendor.ts to understand the full picture:
<path>/home/byk/Code/opencode-lore/packages/gateway/src/cli/sea-entry.ts</path>
<type>file</type>
<content>
1: /**
2: * Fossilize binary entry.
3: *
4: * This is the single entry point that fossilize bundles into the SEA.
5: *
6: * Architecture:
7: * - The native loader shim (auto-injected by esbuild's `inject:`)
8: * runs FIRST and extracts the WASM runtime files to a per-pid
9: * tmp dir.
10: * - This file (sea-entry.ts) reads the embedding worker source
11: * from a SEA asset and exposes it via
12: * `globalThis.__LORE_WORKER_SOURCE__` so `embedding.ts` can
13: * pass it to `new Worker(source, { eval: true, filename, workerData })`.
14: * - If vendoring is enabled, we materialize the model files from
15: * SEA assets to `~/.lore/embeddings-vendored/`.
16: * - We hand off to the main CLI in `bin.ts`. ...
<path>/home/byk/Code/opencode-lore/packages/core/src/embedding-vendor.ts</path> <type>file</type> <content> 1: /** 2: * Vendored model registration for the standalone Lore binary. 3: * 4: * The fossilize-built `lore` binary bundles `@huggingface/transformers` 5: * (which includes onnxruntime-web) into the executable. The model 6: * weights and tokenizer files (nomic-embed-text-v1.5, INT8 quantized) 7: * are embedded as SEA assets and extracted to a cache dir on first run. 8: * 9: * The binary's sea-entry.ts sets `globalThis.__LORE_VENDOR_MODEL__` to 10: * the extraction path. This module exposes that registration to the 11: * `LocalProvider` so it can configure the transformers.js `env` to 12: * load from the local path instead of downloading from HuggingFace Hub. 13: * 14: * In npm-mode usage from `@loreai/opencode` / `@loreai/pi` the global 15: * is unset and `vendorModelInfo()` returns `null`, so transformers.js 16: * downloads the model from HF Hub on first use and ...
<path>/home/byk/Code/opencode-lore/package.json</path>
<type>file</type>
<content>
1: {
2: "name": "lore-monorepo",
3: "private": true,
4: "type": "module",
5: "license": "FSL-1.1-Apache-2.0",
6: "description": "Monorepo root for Lore — three-tier memory architecture",
7: "main": "./packages/opencode/src/index.ts",
8: "exports": {
9: ".": "./packages/opencode/src/index.ts"
10: },
11: "packageManager": "pnpm@10.28.0",
12: "pnpm": {
13: "onlyBuiltDependencies": [
14: "esbuild",
15: "sharp"
16: ]
17: },
18: "scripts": {
19: "typecheck": "pnpm -r run typecheck",
20: "test": "vitest run",
21: "build": "pnpm -r run build",
22: "site:dev": "pnpm --filter '@loreai/website' dev",
23: "site:build": "pnpm --filter '@loreai/website' build",
24: "site:preview": "pnpm --filter '@loreai/website' preview",
25: "lint": "biome check .",
26: "lint:fix": "biome check --write .",
27: "format": "biome format --write .",
28: ...
total 24 drwxr-xr-x 1 byk byk 208 Jun 7 14:44 . drwxr-xr-x 1 byk byk 68 Jun 7 14:19 .. -rw-r--r-- 1 byk byk 67 May 14 19:40 bunfig.toml drwxr-xr-x 1 byk byk 24 Jun 7 14:01 dist drwxr-xr-x 1 byk byk 276 Jun 6 12:55 eval -rw-r--r-- 1 byk byk 1060 Apr 21 17:51 LICENSE drwxr-xr-x 1 byk byk 148 Jun 7 13:37 node_modules -rw-r--r-- 1 byk byk 1353 Jun 7 14:44 package.json -rw-r--r-- 1 byk byk 2469 May 19 09:10 README.md drwxr-xr-x 1 byk byk 16 May 13 15:01 script drwxr-xr-x 1 byk byk 2588 Jun 7 14:19 src drwxr-xr-x 1 byk byk 1132 Jun 7 14:01 test -rw-r--r-- 1 byk byk 289 Jun 7 13:01 tsconfig.build.json -rw-r--r-- 1 byk byk 170 Jun 7 13:01 tsconfig.json
Let me look at the bunfig.toml to see if there's any related test config, and the dist directory of core:
Let me check the remaining items: bunfig.toml, core/dist contents, and any vendor-build artifacts:
<path>/home/byk/Code/opencode-lore/packages/core/bunfig.toml</path> <type>file</type> <content> 1: [test] 2: preload = ["./test/setup.ts"] 3: 4: [test.env] 5: NODE_ENV = "test" (End of file - total 5 lines) </content>
<path>/home/byk/Code/opencode-lore/packages/core/tsconfig.build.json</path>
<type>file</type>
<content>
1: {
2: "extends": "../../tsconfig.base.json",
3: "compilerOptions": {
4: "outDir": "dist/types",
5: "rootDir": "src",
6: "emitDeclarationOnly": true,
7: "declaration": true,
8: "declarationMap": true,
9: "noEmit": false
10: },
11: "include": ["src"],
12: "exclude": ["src/db/driver.bun.ts"]
13: }
(End of file - total 13 lines)
</content>
total 0 drwxr-xr-x 1 byk byk 24 Jun 7 14:01 . drwxr-xr-x 1 byk byk 208 Jun 7 14:44 .. drwxr-xr-x 1 byk byk 2340 Jun 7 14:01 bun drwxr-xr-x 1 byk byk 2340 Jun 7 14:01 node drwxr-xr-x 1 byk byk 2216 Jun 7 14:01 types
total 0 drwxrwxr-x 1 byk byk 50 May 13 12:48 . drwxrwxr-x 1 byk byk 64 May 10 18:29 .. drwxrwxr-x 1 byk byk 178 May 10 01:08 bge-small-en-v1.5 drwxr-xr-x 1 byk byk 42 May 13 12:48 nomic-ai
Found 1 matches
/home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.cjs:
Line 24: `:case"\r":return!1;default:return new RegExp("^\\p{Cc}|\\p{Cf}|\\p{Co}|\\p{Cs}$","u").test(c)}}_clean_text(c){let L=[];for(let U of c){let $=U.charCodeAt(0);$===0||$===65533||this._is_control(U)||(/^\s$/.test(U)?L.push(" "):L.push(U))}return L.join("")}normalize(c){return this.config.clean_text&&(c=this._clean_text(c)),this.config.handle_chinese_chars&&(c=this._tokenize_chinese_chars(c)),this.config.lowercase?(c=c.toLowerCase(),this.config.strip_accents!==!1&&(c=this.stripAccents(c))):this.config.strip_accents&&(c=this.stripAccents(c)),c}}class ge extends n.Callable{static fromConfig(c){if(c===null)return null;switch(c.type){case"BertPreTokenizer":return new m(c);case"Sequence":return new $t(c);case"Whitespace":return new Hs(c);case"WhitespaceSplit":return new Js(c);case"Metaspace":return new Pt(c);case"ByteLevel":return new q(c);case"Split":return new de(c);case"Punctuation":return ...
Interesting! The dist/embedding-worker.cjs exists, but let me check what's in it more carefully — and also look at the source's referenced file path. The grep returned line 24 with a :case"\r":return!1; which is part of a minified file. The dist file exists.
Let me now also check the import.meta.url behavior in vitest. I want to look at the core/src/embedding-worker.ts import line carefully. The user mentioned the "extensionless import (around line 31)". Let me look at it again:
import {
isOomError,
isWasmFatalError,
type WorkerInbound,
type WorkerOutbound,
type WorkerInitData,
type EmbedRequest,
} from "./embedding-worker-types";
That's an extensionless import! It's line 31. In TypeScript with moduleResolution: "bundler", this is allowed. But when this is bundled with esbuild for the CJS output, the bundler needs to resolve it. Let me check what extension the bundled output uses for this import.
Let me also check the actual embedding-worker.cjs and ...
Let me look closer at the actual issue — the extensionless import in embedding-worker.ts and how the bundled output handles it:
...output truncated...
Full output saved to: /home/byk/.local/share/opencode/tool-output/tool_ea3e009fe001ikTM7Djl1OVRIs
`],["t"," "],["r","\r"],["b","\b"],["f","\f"],["v","\v"],["'","'"],['"','"'],["\\","\\"]]);function w(i,j={}){return i.endsWith(`
`)&&(i=i.slice(0,-1)),j.lstrip_blocks&&(i=i.replace(/^[ \t]*({[#%-])/gm,"$1")),j.trim_blocks&&(i=i.replace(/([#%-]})\n/g,"$1")),i.replace(/{%\s*(end)?generation\s*%}/gs,"")}function d(i,j={}){let v=[],O=w(i,j),z=0,Z=0,ue=Oe=>{let st="";for(;Oe(O[z]);){if(O[z]==="\\"){if(++z,z>=O.length)throw new SyntaxError("Unexpected end of input");let We=O[z++],it=p.get(We);if(it===void 0)throw new SyntaxError(`Unexpected escaped character: ${We}`);st+=it;continue}if(st+=O[z++],z>=O.length)throw new SyntaxError("Unexpected end of input")}return st},Ge=()=>{let Oe=v.at(-1);Oe&&Oe.type===n.Text&&(Oe.value=Oe.value.trimEnd(),Oe.value===""&&v.pop())},Je=()=>{for(;z<O.length&&g(O[z]);)++z};e:for(;z<O.length;){let Oe=v.at(-1)?.type;if(Oe===void ...
...output truncated...
Full output saved to: /home/byk/.local/share/opencode/tool-output/tool_ea3e00a20001xqP6r5OZkfY6Td
`],["t"," "],["r","\r"],["b","\b"],["f","\f"],["v","\v"],["'","'"],['"','"'],["\\","\\"]]);function w(i,j={}){return i.endsWith(`
`)&&(i=i.slice(0,-1)),j.lstrip_blocks&&(i=i.replace(/^[ \t]*({[#%-])/gm,"$1")),j.trim_blocks&&(i=i.replace(/([#%-]})\n/g,"$1")),i.replace(/{%\s*(end)?generation\s*%}/gs,"")}function d(i,j={}){let v=[],O=w(i,j),z=0,Z=0,ue=Oe=>{let st="";for(;Oe(O[z]);){if(O[z]==="\\"){if(++z,z>=O.length)throw new SyntaxError("Unexpected end of input");let We=O[z++],it=p.get(We);if(it===void 0)throw new SyntaxError(`Unexpected escaped character: ${We}`);st+=it;continue}if(st+=O[z++],z>=O.length)throw new SyntaxError("Unexpected end of input")}return st},Ge=()=>{let Oe=v.at(-1);Oe&&Oe.type===n.Text&&(Oe.value=Oe.value.trimEnd(),Oe.value===""&&v.pop())},Je=()=>{for(;z<O.length&&g(O[z]);)++z};e:for(;z<O.length;){let Oe=v.at(-1)?.type;if(Oe===void ...
24 /home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.cjs
24 /home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.js
3548 /home/byk/Code/opencode-lore/packages/gateway/dist/index.cjs
3596 total
No files found
No files found
No files found
The bundled output is interesting! Both embedding-worker.cjs and embedding-worker.js exist (24 lines each, ~500KB). The bundle is minified, so the grep couldn't find the embedding-worker-types import by name. Both bundled files are minified into a single line of code essentially.
Let me check the first few lines to understand how the import is resolved:
Let me check the structure of the bundled worker files (just first 200 chars to see the IIFE wrapper):
"use strict";var am=Object.create;var Wa=Object.defineProperty;var nm=Object.getOwnPropertyDescriptor;var im=Object.getOwnPropertyNames;var lm=Object.getPrototypeOf,cm=Object.prototype.hasOwnProperty;var Bt=(C,a)=>()=>(C&&(a=C(C=0)),a);var wu=(C,a)=>{for(var e in a)Wa(C,e,{get:a[e],enumerable:!0})},_m=(C,a,e,n)=>{if(a&&typeof a=="object"||typeof a=="function")for(let _ of im(a))!cm.call(C,_)&&_!==
---LENGTH---
48907
---END LINE 1---
var rm=Object.defineProperty;var Bt=(L,a)=>()=>(L&&(a=L(L=0)),a);var fu=(L,a)=>{for(var e in a)rm(L,e,{get:a[e],enumerable:!0})};var Nr,Ks,Pu,om,wu,za=Bt(()=>{Nr=new Map,Ks=[],Pu=(L,a,e)=>{if(a&&typeof a.init=="function"&&typeof a.createInferenceSessionHandler=="function"){let n=Nr.get(L);if(n===void 0)Nr.set(L,{backend:a,priority:e});else{if(n.priority>e)return;if(n.priority===e&&n.backend!==a)th
---LENGTH---
48423
(no output)
0
The bundled CJS file does NOT contain isOomError or isWasmFatalError from the source! The minified output doesn't have those identifiers visible. This makes sense — they were renamed by the minifier. The bundle successfully merged all the code.
But wait, let me look more carefully. The ESM bundle also doesn't show them. Let me search differently - look for what the source code (embedding-worker-types.ts) actually compiles to in the bundle:
Actually, looking at the grep result: 0 occurrences. The minifier probably renamed them. The 501577 byte CJS file is the worker, and the 24 line count is suspicious. Let me check the structure more.
Now I have enough information to compile the comprehensive report. Let me also check the core test setup more carefully and check for any related issue files. Let me also check what's in core/dist/:
Let me check the bun and node dist directories, and look for any test helpers:
/home/byk/Code/opencode-lore/packages/core/dist/bun/: total 8252 drwxr-xr-x 1 byk byk 2340 Jun 7 14:01 . drwxr-xr-x 1 byk byk 24 Jun 7 14:01 .. -rw-r--r-- 1 byk byk 4578 Jun 7 14:01 agents-file.d.ts -rw-r--r-- 1 byk byk 1289 Jun 7 14:01 agents-file.d.ts.map -rw-r--r-- 1 byk byk 4456 Jun 7 14:01 config.d.ts -rw-r--r-- 1 byk byk 454 Jun 7 14:01 config.d.ts.map -rw-r--r-- 1 byk byk 4198 Jun 7 14:01 curator.d.ts -rw-r--r-- 1 byk byk 2586 Jun 7 14:01 curator.d.ts.map -rw-r--r-- 1 byk byk 747 Jun 7 14:01 data-dir.d.ts -rw-r--r-- 1 byk byk 224 Jun 7 14:01 data-dir.d.ts.map -rw-r--r-- 1 byk byk 8789 Jun 7 14:01 data.d.ts -rw-r--r-- 1 byk byk 4350 Jun 7 14:01 data.d.ts.map drwxr-xr-x 1 byk byk 72 Jun 7 14:01 db -rw-r--r-- 1 byk byk 10251 Jun 7 14:01 db.d.ts -rw-r--r-- 1 byk byk 4746 Jun 7 14:01 db.d.ts.map -rw-r--r-- 1 byk byk 10050 Jun 7 14:01 distillation.d.ts -rw-r--r-- 1 byk byk 2981 Jun 7 14:01 ...
agents-file.test.ts config.test.ts context-health.test.ts data-dir.test.ts db.test.ts dedup.test.ts distillation.test.ts embedding.test.ts embedding-vendor.test.ts entities.test.ts entity-dedup.test.ts fixtures git.test.ts gradient-reasoning.test.ts gradient.test.ts hosted.test.ts import instruction-detect.test.ts integrity.test.ts knowledge-transfers.test.ts lat-reader.test.ts ltm.test.ts markdown.test.ts pattern-extract.test.ts prompt.test.ts refs.test.ts search.test.ts session-limiter.test.ts setup.ts temporal.test.ts tool-trace.test.ts worker-model.test.ts worker.test.ts workspace.test.ts
agents-file.d.ts agents-file.d.ts.map agents-file.ts config.d.ts config.d.ts.map config.ts curator.d.ts curator.d.ts.map curator.ts data-dir.d.ts data-dir.d.ts.map data-dir.ts data.d.ts data.d.ts.map data.ts db db.d.ts db.d.ts.map db.ts distillation.d.ts distillation.d.ts.map distillation.ts embedding.d.ts embedding.d.ts.map embedding.ts embedding-vendor.d.ts embedding-vendor.d.ts.map embedding-vendor.ts embedding-worker.ts embedding-worker-types.d.ts
Found 42 matches
/home/byk/Code/opencode-lore/packages/core/dist/bun/embedding-worker.d.ts.map:
Line 1: {"version":3,"file":"embedding-worker.d.ts","sourceRoot":"","sources":["../../src/embedding-worker.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG"}
/home/byk/Code/opencode-lore/packages/core/dist/bun/embedding-worker.d.ts:
Line 20: * @see embedding-worker-types.ts for the message protocol.
Line 23: //# sourceMappingURL=embedding-worker.d.ts.map
/home/byk/Code/opencode-lore/packages/core/dist/bun/embedding-worker-types.d.ts:
Line 4: * The embedding worker (`embedding-worker.ts`) runs ONNX inference via
Line 91: //# sourceMappingURL=embedding-worker-types.d.ts.map
/home/byk/Code/opencode-lore/packages/core/dist/bun/embedding-worker-types.d.ts.map:
Line 1: ...
Now I have a very clear picture. The key finding is:
In packages/core/dist/node/index.js:28755:
workerUrl = new URL(
`./embedding-worker${selfUrl.endsWith(".ts") ? ".ts" : ".js"}`,
selfUrl,
);
This matches the source: embedding.ts:330-333.
And at line 28761:
workerUrl = new URL(
"./embedding-worker.cjs",
pathToFileURL(__filename),
);
This is the CJS fallback at embedding.ts:336-339.
The dist/node/embedding-worker.js and dist/bun/embedding-worker.js files exist as Bun's built version (39010 lines) - they have the original source code preserved with // src/embedding-worker.ts and // src/embedding-worker-types.ts comments.
The gateway's dist/embedding-worker.cjs (24 lines, ~500KB) is the bundled esbuild output for the npm package.
Now I have all the information needed. Let me also do a final check on the index.cjs to see how the worker reference works in the main bundle:
Let me also verify the gateway dist/index.cjs to see the bundled worker resolution:
embedding-worker.cjs embedding-worker.js
(no output)
-rw-r--r-- 1 byk byk 501577 Jun 5 14:01 /home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.cjs -rw-r--r-- 1 byk byk 501023 Jun 5 14:01 /home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.js
1780668118 /home/byk/Code/opencode-lore/packages/gateway/dist/embedding-worker.cjs 1780668118 /home/byk/Code/opencode-lore/packages/gateway/dist/index.cjs 1780609177 /home/byk/Code/opencode-lore/packages/core/src/embedding-worker.ts 1780609177 /home/byk/Code/opencode-lore/packages/core/src/embedding-worker-types.ts
Now I have all the information I need to provide a comprehensive report. Let me compile everything.
The dist files were modified later than the source files, which is normal. The source files were last modified on June 4 (Unix timestamp 1780609177), and the dist files on June 5 (1780668118).
Now let me also check if there's any "vite-node" or runtime transformation in vitest that could cause the extensionless import to break at runtime:
Now I have all the data I need. Let me deliver the comprehensive structured summary.
/home/byk/Code/opencode-lore/packages/core/src/embedding-worker.tsAbsolute path: /home/byk/Code/opencode-lore/packages/core/src/embedding-worker.ts
Total lines: 408
// packages/core/src/embedding-worker.ts:24-31
import {
isOomError,
isWasmFatalError,
type WorkerInbound,
type WorkerOutbound,
type WorkerInitData,
type EmbedRequest,
} from "./embedding-worker-types";
Key point: This is the extensionless import the user asked about. The path ./embedding-worker-types has no .ts or .js extension. It works in the source TypeScript because tsconfig.base.json:5 sets moduleResolution: "bundler", but this is fragile at runtime when vitest is resolving the path under CJS.
| Section | Lines | Purpose |
|---|---|---|
| Imports | 23-31 | node:worker_threads + types from ./embedding-worker-types |
| workerData capture | 33-45 | Asserts parentPort non-null; reads WorkerInitData |
| OOM retry constants | 47-59 | OOM_RETRY_START_TOKENS=4096, OOM_MAX_RETRIES=3 |
| Model lifecycle types | 65-105 | FeatureExtractionPipeline + lazy state |
| ensurePipeline() | 107-183 | Lazy pipeline("feature-extraction", modelId, {dtype:"q8", device:"cpu"}); configures WASM numThreads: 1, proxy: false to dodge Bun bugs (oven-sh/bun#25677, #31158, #18145) |
| Priority queue | 185-228 | High-priority requests jump ahead of normal-priority; FIFO within tier |
| runInference() | 230-303 | Mean pooling → layer_norm → Matryoshka slice → L2 normalize; extracts per-text vectors |
| processEmbed() | 305-389 | OOM retry loop with token-level truncation; exits on fatal WASM errors |
| Message handler | 391-408 | port.on("message") handles embed / shutdown |
Notable detail (line 254-255): The ...