Dashboard › sentry › Session 05rjs8fSV9k7
05rjs8fSV9k7m1phcYou are reviewing PR #600 on BYK/loreai. This PR guards resp.usage accesses against undefined for vLLM/partial responses.
Read these files and review the changes critically:
packages/gateway/src/translate/types.ts — look for the ZERO_USAGE constant and the usage type changepackages/gateway/src/pipeline.ts — search for ZERO_USAGE and resp.usage to review all guard sitespackages/gateway/src/translate/openai.ts — search for ZERO_USAGE and usage guardpackages/gateway/src/translate/openai-responses.ts — search for ZERO_USAGE and usage guardspackages/gateway/src/translate/anthropic.ts — search for ZERO_USAGE and usage guardpackages/gateway/src/stream/openai.ts — search for ZERO_USAGEpackages/gateway/src/stream/openai-responses.ts — search for ZERO_USAGEpackages/gateway/src/stream/anthropic.ts — search for ZERO_USAGEpackages/gateway/test/compaction.test.ts — check test changespackages/gateway/test/openai-responses-stream.test.ts — check test changespackages/gateway/test/recall.test.ts — check test changesFor each file, look for:
resp.usage accesses properly guarded?resp.usage accesses that were missed?Also check the PR description at https://github.com/BYK/loreai/pull/600
Return a structured review with: (1) critical issues, (2) medium issues, (3) minor/style issues, (4) verdict (approve/request changes)
I'll start by reading all the relevant files and the PR description in parallel.
fix(gateway): guard resp.usage accesses against undefined for vLLM/partial responses by BYK · Pull Request #600 · BYK/loreai · GitHub
[Skip to content](#start-of-content)
## Navigation Menu
Toggle navigation
[](/)
[Sign in](/login?return_to=https%3A%2F%2Fgithub.com%2FBYK%2Floreai%2Fpull%2F600)
Appearance settings
- Platform
- AI CODE CREATION
- [
GitHub CopilotWrite better code with AI
](https://github.com/features/copilot)
- [
GitHub Copilot appDirect agents from issue to merge
](https://github.com/features/ai/github-app)
- [
MCP RegistryNewIntegrate external tools
](https://github.com/mcp)
- DEVELOPER WORKFLOWS
- [
ActionsAutomate any ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/translate/types.ts</path> <type>file</type> <content> 1: /** 2: * Internal representation types for the Lore gateway. 3: * 4: * The gateway accepts both Anthropic (`/v1/messages`) and OpenAI 5: * (`/v1/chat/completions`) protocol requests, normalizes them into these 6: * types for Lore pipeline processing, then translates back to the original 7: * protocol for the upstream response. 8: * 9: * Design: types are intentionally minimal — only fields that Lore's context 10: * management (gradient, LTM, distillation) actually reads/writes. Protocol- 11: * specific fields the gateway doesn't process live in `metadata`. ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path> <type>file</type> <content> 1: /** 2: * Core request processing pipeline for the Lore gateway. 3: * 4: * Orchestrates the full flow for every request: 5: * session identification → LTM injection → gradient transform → 6: * upstream forwarding → response accumulation → calibration → 7: * temporal storage → background work scheduling. 8: * 9: * Three request classes are handled: 10: * 1. Compaction requests → intercepted, never forwarded upstream. 11: * 2. Meta requests (title gen, summaries, etc.) → forwarded transparently, no Lore processing. 12: * 3. Normal conversation turns → full pipeline. ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/translate/openai.ts</path>
<type>file</type>
<content>
1: /**
2: * OpenAI ↔ Gateway translation layer.
3: *
4: * Converts between OpenAI's `/v1/chat/completions` API format and the gateway's
5: * internal `GatewayRequest`/`GatewayResponse` types.
6: */
7: import type {
8: GatewayContentBlock,
9: GatewayMessage,
10: GatewayRequest,
11: GatewayResponse,
12: GatewayTool,
13: } from "./types";
14: import { blocksToText, forwardClientHeaders } from "./types";
15: import { extractAuth } from "../auth";
16:
17: // ---------------------------------------------------------------------------
18: // OpenAI → GatewayRequest
19: // ---------------------------------------------------------------------------
20:
21: export function parseOpenAIRequest(
22: body: unknown,
23: headers: Record<string, string>,
24: ): GatewayRequest {
25: const raw = (body ?? ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/translate/openai-responses.ts</path>
<type>file</type>
<content>
1: /**
2: * OpenAI Responses API ↔ Gateway translation layer.
3: *
4: * Converts between OpenAI's `/v1/responses` API format and the gateway's
5: * internal `GatewayRequest`/`GatewayResponse` types.
6: *
7: * The Responses API uses a different message format than Chat Completions:
8: * - Input is an array of "input items" (message, function_call, function_call_output, etc.)
9: * - Output is an array of "output items" with similar structure
10: * - System prompt is in the `instructions` field
11: * - Tools use `parameters` directly (not wrapped in `function`)
12: */
13: import { log } from "@loreai/core";
14: import type {
15: GatewayContentBlock,
16: GatewayMessage,
17: GatewayRequest,
18: GatewayResponse,
19: GatewayTool,
20: } from "./types";
21: import { blocksToText, ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/translate/anthropic.ts</path>
<type>file</type>
<content>
1: /**
2: * Anthropic ↔ Gateway translation layer.
3: *
4: * Converts between Anthropic's `/v1/messages` API format and the gateway's
5: * internal `GatewayRequest`/`GatewayResponse` types. The parser is lenient —
6: * unknown fields pass through in `metadata` rather than causing errors.
7: */
8: import type {
9: GatewayContentBlock,
10: GatewayMessage,
11: GatewayRequest,
12: GatewayResponse,
13: GatewayTool,
14: } from "./types";
15: import { forwardClientHeaders } from "./types";
16: import { extractAuth, authHeaders } from "../auth";
17:
18: // ---------------------------------------------------------------------------
19: // Anthropic API version — used in all outgoing requests
20: // ---------------------------------------------------------------------------
21:
22: const ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/stream/openai.ts</path> <type>file</type> <content> 1: /** 2: * Anthropic SSE → OpenAI Chat Completions SSE streaming translator. 3: * 4: * Reads Anthropic-format SSE events from an upstream Response and emits 5: * OpenAI Chat Completions streaming chunks incrementally, so the client 6: * receives tokens as they arrive rather than waiting for the full response. 7: * 8: * Anthropic lifecycle: 9: * message_start → content_block_start → content_block_delta (repeated) 10: * → content_block_stop → message_delta → message_stop 11: * 12: * OpenAI Chat Completions streaming lifecycle: 13: * chunk with delta.role → chunk with delta.content (repeated) 14: * → chunk with finish_reason → data: [DONE] 15: * 16: * Uses `parseSSEStream` from the Anthropic stream module to parse upstream 17: * events, and `createStreamAccumulator` to build the internal ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/stream/openai-responses.ts</path>
<type>file</type>
<content>
1: /**
2: * OpenAI Responses API SSE stream accumulator.
3: *
4: * Parses upstream Responses API streaming events and accumulates the full
5: * response into a `GatewayResponse`. The Responses API uses a different
6: * SSE event lifecycle than Anthropic:
7: *
8: * response.created → response.in_progress →
9: * response.output_item.added → response.output_text.delta (repeated) →
10: * response.output_item.done → response.function_call_arguments.delta →
11: * response.function_call_arguments.done →
12: * response.completed
13: *
14: * Reuses `parseSSEStream` from the Anthropic stream module since the
15: * underlying SSE wire format is the same.
16: */
17: import type {
18: GatewayContentBlock,
19: GatewayResponse,
20: GatewayUsage,
21: } from "../translate/types";
22: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/stream/anthropic.ts</path>
<type>file</type>
<content>
1: /**
2: * Anthropic SSE stream handling.
3: *
4: * Parses upstream Anthropic streaming responses (named SSE events), accumulates
5: * the full response into a `GatewayResponse`, and provides helpers for
6: * generating synthetic SSE event sequences (e.g. for compaction interception).
7: *
8: * Anthropic uses named SSE events with a lifecycle:
9: * message_start -> content_block_start/delta/stop (repeated) -> message_delta -> message_stop
10: *
11: * All functions are pure (no side effects) except `parseSSEStream` which is
12: * an async generator consuming a byte stream.
13: */
14: import type {
15: GatewayContentBlock,
16: GatewayResponse,
17: GatewayUsage,
18: } from "../translate/types";
19: import { scaleUsageForClient } from "../compaction";
20:
21: // ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/test/compaction.test.ts</path>
<type>file</type>
<content>
1: import { describe, test, expect } from "vitest";
2: import {
3: isCompactionRequest,
4: detectCompactionRequest,
5: isStructuralCompaction,
6: extractPreviousSummary,
7: isMetaRequest,
8: LORE_AGENT_HEADER,
9: buildCompactionResponse,
10: COMPACTION_SYSTEM_PATTERNS,
11: COMPACTION_USER_PATTERNS,
12: } from "../src/compaction";
13: import type { GatewayRequest } from "../src/translate/types";
14:
15: // ---------------------------------------------------------------------------
16: // Helpers
17: // ---------------------------------------------------------------------------
18:
19: /** Minimal valid GatewayRequest with sensible defaults. */
20: function makeRequest(
21: overrides: Partial<GatewayRequest> & {
22: system?: string;
23: messages?: GatewayRequest["messages"];
24: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/test/openai-responses-stream.test.ts</path>
<type>file</type>
<content>
1: /**
2: * Tests for the OpenAI Responses API SSE stream accumulator.
3: *
4: * Covers:
5: * - Text output accumulation from delta events
6: * - Function call accumulation from arguments delta events
7: * - Usage extraction from response.completed
8: * - Stop reason mapping from status
9: * - Mixed text + function_call output
10: */
11: import { describe, test, expect } from "vitest";
12: import { accumulateResponsesSSEStream } from "../src/stream/openai-responses";
13:
14: // ---------------------------------------------------------------------------
15: // Helpers
16: // ---------------------------------------------------------------------------
17:
18: /** Build a fake SSE Response from event/data pairs. ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/test/recall.test.ts</path>
<type>file</type>
<content>
1: /**
2: * Unit tests for gateway recall interception helpers.
3: *
4: * Tests the pure functions in recall.ts:
5: * - Tool definition
6: * - Detection helpers (findRecallToolUse, hasRecallToolUse, hasOtherToolUse)
7: * - Follow-up request builder
8: * - Pending recall injection
9: * - Response stripping
10: */
11: import { describe, test, expect } from "vitest";
12: import {
13: RECALL_GATEWAY_TOOL,
14: RECALL_TOOL_NAME,
15: MAX_RECALL_DEPTH,
16: findRecallToolUse,
17: hasRecallToolUse,
18: hasOtherToolUse,
19: clientHasRecallTool,
20: buildRecallFollowUpRequest,
21: runRecallFollowUpStreaming,
22: runRecallFollowUpJSON,
23: type RecallFollowUpCtx,
24: buildRecallMarker,
25: parseRecallMarker,
26: isRecallMarker,
27: scopeToLabel,
28: labelToScope,
29: ...
Now I need to check the actual diff to see what was changed. Let me look at the git diff for this PR, and also search for any remaining unguarded resp.usage accesses.
33ad12e fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash 6ba34e5 fix(site): convert brand-mark PNG to SVG, add light variant, mobile site-nav 8291bbf fix(core,gateway): restore bun export conditions for OpenCode plugin (#597) 7fe80db feat(website): Migrate website to Astro (#559) 75f1ce4 fix(gateway): decouple worker wire protocol from provider identity (#596)
No files found
Found 65 matches /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts: Line 2506: input_tokens: resp.usage.inputTokens, Line 2507: output_tokens: resp.usage.outputTokens, Line 2508: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2509: cache_creation_input_tokens: resp.usage.cacheCreationInputTokens, Line 2560: inputTokens: resp.usage.inputTokens, Line 2561: outputTokens: resp.usage.outputTokens, Line 2597: (resp.usage.inputTokens ?? 0) + Line 2598: (resp.usage.cacheReadInputTokens ?? 0) + Line 2599: (resp.usage.cacheCreationInputTokens ?? 0); Line 2603: setSentryCacheContext(resp.usage); Line 2605: input_tokens: resp.usage.inputTokens, Line 2606: output_tokens: resp.usage.outputTokens, Line 2607: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2608: ...
Found 78 matches /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts: Line 2506: input_tokens: resp.usage.inputTokens, Line 2507: output_tokens: resp.usage.outputTokens, Line 2508: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2509: cache_creation_input_tokens: resp.usage.cacheCreationInputTokens, Line 2560: inputTokens: resp.usage.inputTokens, Line 2561: outputTokens: resp.usage.outputTokens, Line 2597: (resp.usage.inputTokens ?? 0) + Line 2598: (resp.usage.cacheReadInputTokens ?? 0) + Line 2599: (resp.usage.cacheCreationInputTokens ?? 0); Line 2605: input_tokens: resp.usage.inputTokens, Line 2606: output_tokens: resp.usage.outputTokens, Line 2607: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2608: cache_creation_input_tokens: resp.usage.cacheCreationInputTokens, Line ...
This is very revealing. The PR claims to have been merged, but I see no evidence of the changes in the codebase. There is no ZERO_USAGE constant anywhere, GatewayResponse.usage is still typed as non-optional (usage: GatewayUsage on line 228 of types.ts), and all resp.usage accesses are still unguarded.
Let me verify this is the right worktree and check the commit history more carefully.
7cc73c6 test(ci): quiet test harness noise + pretest hook for bundle b79acad fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash (#602) 48770f2 fix(core): recognize WASM fatal error wrapper prefix in isWasmFatalError() (#604) f610388 fix(gateway): guard resp.usage accesses against undefined for vLLM/partial responses (#600) 122690b fix(gateway): handle SSE response from upstream when stream: false was sent (#605) 57893c5 fix(core): guard against postMessage on terminated embedding worker (#603) 33ad12e fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash 688f863 fix(gateway): handle SSE response from upstream when stream: false was sent 79639fd fix(core): recognize WASM fatal error wrapper prefix in isWasmFatalError() 740b530 fix(core): guard against postMessage on terminated embedding worker 932b1b9 fix(site): grain z-index, blog footer, CWD-relative fs read c084a94 fix(site): grain z-index, blog footer, CWD-relative fs read af58aa1 ...
33ad12e fix(gateway): polyfill getSystemErrorMap for Bun to prevent Sentry SDK crash chore/biome-lint-rules chore/biome-safe-fixes chore/biome-tooling chore/responses-stateless-cleanup ci/docs-preview-and-skip codex/social-icons-size docs/rsi-marketing-copy feat/astro-site-migration feat/ci-workflow-cleanup feat/cross-project-transfer-metrics feat/entity-auto-dedup feat/tool-call-trace fix-pi-baseurl fix/516-per-agent-auto-import fix/516-review-followup fix/anthropic-compat-ttl * fix/bun-system-error-map-polyfill fix/codex-compact-404 fix/decouple-worker-protocol-from-provider fix/dismiss-dedup-suggestions fix/embedding-oom-storm fix/fossilize-programmatic-api-correct fix/fossilize-programmatic-api-final fix/fossilize-variadic-platforms fix/lossless-content-passthrough fix/oauth-worker-headers fix/perf-test-flake fix/preserve-5m-cache-ttl fix/provider-id-routing fix/provider-routing-followups ...
I see — the current worktree is on fix/bun-system-error-map-polyfill, not on the PR branch. The PR branch fix/usage-undefined-guard exists locally. Let me check out that branch to see the actual changes.
packages/gateway/src/pipeline.ts | 88 +++++++++++++--------- packages/gateway/src/stream/anthropic.ts | 19 ++--- packages/gateway/src/stream/openai-responses.ts | 11 +-- packages/gateway/src/stream/openai.ts | 11 +-- packages/gateway/src/translate/anthropic.ts | 15 ++-- packages/gateway/src/translate/openai-responses.ts | 24 +++--- packages/gateway/src/translate/openai.ts | 13 ++-- packages/gateway/src/translate/types.ts | 18 ++++- packages/gateway/test/compaction.test.ts | 10 +-- .../gateway/test/openai-responses-stream.test.ts | 14 ++-- packages/gateway/test/recall.test.ts | 4 +- 11 files changed, 132 insertions(+), 95 deletions(-)
diff --git a/packages/gateway/src/pipeline.ts b/packages/gateway/src/pipeline.ts
index 1e75b6b..518d00e 100644
--- a/packages/gateway/src/pipeline.ts
+++ b/packages/gateway/src/pipeline.ts
@@ -70,7 +70,11 @@ import type {
SessionState,
UpstreamSnapshot,
} from "./translate/types";
-import { blocksToText, forwardClientHeaders } from "./translate/types";
+import {
+ blocksToText,
+ forwardClientHeaders,
+ ZERO_USAGE,
+} from "./translate/types";
import type { GatewayConfig } from "./config";
import {
getProjectPath,
@@ -2500,13 +2504,16 @@ function nonStreamHttpResponse(
clientStream?: boolean,
extraHeaders?: Record<string, string>,
): Response {
+ // Guard: resp.usage can be undefined at runtime for vLLM / partial responses.
+ const usage = resp.usage ?? ZERO_USAGE;
+
// Scale usage so the client's token total stays below auto-compact threshold.
// postResponse() has already consumed the real values for calibration/bustRate. ...
Now let me check a few more things to be thorough — the stream accumulators that produce GatewayResponse objects, and the buildSSEMessageStart in context.
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
2500: clientStream?: boolean,
2501: extraHeaders?: Record<string, string>,
2502: ): Response {
2503: // Scale usage so the client's token total stays below auto-compact threshold.
2504: // postResponse() has already consumed the real values for calibration/bustRate.
2505: const scaledUsage = scaleUsageForClient({
2506: input_tokens: resp.usage.inputTokens,
2507: output_tokens: resp.usage.outputTokens,
2508: cache_read_input_tokens: resp.usage.cacheReadInputTokens,
2509: cache_creation_input_tokens: resp.usage.cacheCreationInputTokens,
2510: });
2511: const scaledResp: GatewayResponse = {
2512: ...resp,
2513: usage: {
2514: inputTokens: scaledUsage.input_tokens,
2515: outputTokens: scaledUsage.output_tokens,
2516: cacheReadInputTokens: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
4560: // Run the recall-interception loop over an already-accumulated
4561: // (internal Anthropic-format) GatewayResponse and return the client HTTP
4562: // response. Shared by the non-streaming path AND the OpenAI/openai-responses
4563: // streaming paths — those accumulate the upstream SSE into the same internal
4564: // Anthropic-format response, so the recall loop is protocol-agnostic here.
4565: // Without this, a `recall` tool_use injected by the gateway would leak to the
4566: // client (e.g. "Model tried to call unavailable tool 'recall'").
4567: const finalizeWithRecall = async (
4568: resp: GatewayResponse,
4569: ): Promise<Response> => {
4570: // --- Recall interception (non-streaming) ---
4571: // Loop allows the model to call recall multiple times (e.g. ...
Found 65 matches /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts: Line 2506: input_tokens: resp.usage.inputTokens, Line 2507: output_tokens: resp.usage.outputTokens, Line 2508: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2509: cache_creation_input_tokens: resp.usage.cacheCreationInputTokens, Line 2560: inputTokens: resp.usage.inputTokens, Line 2561: outputTokens: resp.usage.outputTokens, Line 2597: (resp.usage.inputTokens ?? 0) + Line 2598: (resp.usage.cacheReadInputTokens ?? 0) + Line 2599: (resp.usage.cacheCreationInputTokens ?? 0); Line 2603: setSentryCacheContext(resp.usage); Line 2605: input_tokens: resp.usage.inputTokens, Line 2606: output_tokens: resp.usage.outputTokens, Line 2607: cache_read_input_tokens: resp.usage.cacheReadInputTokens, Line 2608: ...
Now I have the complete picture. Let me check the accumulateResponsesSSEStream return statement and the accumulateNonStreamResponse to understand if accumulators always produce usage, and also check if there are more callers that construct GatewayResponse objects.
Found 31 matches
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts:
Line 245: function injectContextWarning(resp: GatewayResponse): GatewayResponse {
Line 1800: onComplete: (response: GatewayResponse) => void,
Line 2223: ): GatewayResponse {
Line 2285: ): GatewayResponse {
Line 2498: resp: GatewayResponse,
Line 2511: const scaledResp: GatewayResponse = {
Line 2552: function streamHttpResponse(resp: GatewayResponse): Response {
Line 2584: resp: GatewayResponse,
Line 4568: resp: GatewayResponse,
Line 5339: const resp: GatewayResponse = {
/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/translate/openai.ts:
Line 297: resp: GatewayResponse,
Line 306: function buildOpenAINonStreamResponse(resp: GatewayResponse): Response {
Line 384: function buildOpenAIStreamResponse(resp: ...
Found 10 matches /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts: Line 2202: async function accumulateNonStreamResponse( Line 2376: async function accumulateNonStreamOpenAIStream( Line 3642: const resp = await accumulateNonStreamResponse( Line 3650: const resp = await accumulateNonStreamResponse( Line 4630: // continuation is consumed (accumulateNonStreamResponse). Line 4640: parseJSON: accumulateNonStreamResponse, Line 4759: const resp = await accumulateNonStreamOpenAIStream(upstreamResponse); Line 4789: const resp = await accumulateNonStreamResponse( /home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/recall.ts: Line 404: * `parseJSON` wraps `accumulateNonStreamResponse`. Line 431: * the upstream response: `false` → JSON via `accumulateNonStreamResponse()`,
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
2202: async function accumulateNonStreamResponse(
2203: upstreamResponse: Response,
2204: protocol: "anthropic" | "openai" | "openai-responses" = "anthropic",
2205: ): Promise<GatewayResponse> {
2206: const json = (await upstreamResponse.json()) as Record<string, unknown>;
2207:
2208: switch (protocol) {
2209: case "openai":
2210: return accumulateOpenAINonStreamJSON(json);
2211: case "openai-responses":
2212: return accumulateResponsesNonStreamJSON(json);
2213: default:
2214: return accumulateAnthropicNonStreamJSON(json);
2215: }
2216: }
2217:
2218: // Anthropic non-stream JSON → GatewayResponse: use shared parseAnthropicResponseJSON
2219: const accumulateAnthropicNonStreamJSON = parseAnthropicResponseJSON;
2220:
2221: function accumulateOpenAINonStreamJSON(
2222: json: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
2376: async function accumulateNonStreamOpenAIStream(
2377: upstreamResponse: Response,
2378: ): Promise<GatewayResponse> {
2379: let id = "";
2380: let model = "";
2381: let stopReason = "end_turn";
2382: let textContent = "";
2383: const toolCalls = new Map<
2384: number,
2385: { id: string; name: string; args: string }
2386: >();
2387: let inputTokens = 0;
2388: let outputTokens = 0;
2389: let cachedTokens: number | undefined;
2390:
2391: if (!upstreamResponse.body) {
2392: throw new Error("Upstream response has no body");
2393: }
2394: const reader = upstreamResponse.body.getReader();
2395:
2396: for await (const { data } of parseSSEStream(reader)) {
2397: if (data === "[DONE]") break;
2398:
2399: let parsed: Record<string, unknown>;
2400: try {
2401: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
5330:
5331: /** Build a synthetic slash-command response in the client's wire format. */
5332: function slashResponse(
5333: req: GatewayRequest,
5334: text: string,
5335: msgId: string,
5336: ): Response {
5337: // Build a GatewayResponse and use the protocol-aware response builders
5338: // so slash commands work correctly for all client protocols.
5339: const resp: GatewayResponse = {
5340: id: msgId,
5341: model: req.model,
5342: content: [{ type: "text", text }],
5343: stopReason: "end_turn",
5344: usage: {
5345: inputTokens: 0,
5346: outputTokens: 0,
5347: cacheReadInputTokens: 0,
5348: cacheCreationInputTokens: 0,
5349: },
5350: };
5351:
5352: if (req.stream) {
5353: // Build Anthropic SSE, then translate to client's format if needed
5354: ...
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/recall.ts</path>
<type>file</type>
<content>
640: * Build a GatewayResponse with recall tool_use blocks replaced by marker text.
641: *
642: * Used for both recall-only and mixed-tools cases to produce a response
643: * where the client sees human-readable markers instead of tool call mechanics.
644: */
645: export function replaceRecallWithMarker(
646: resp: GatewayResponse,
647: ): GatewayResponse {
648: return {
649: ...resp,
650: content: resp.content.map((b) => {
651: if (b.type === "tool_use" && b.name === RECALL_TOOL_NAME) {
652: const input = b.input as Record<string, unknown>;
653: const query = typeof input.query === "string" ? input.query : "";
654: const scope = (input.scope as string) ?? "all";
655: const id =
656: typeof input.id === "string" && input.id ? input.id : undefined;
657: ...
Now I have a complete picture. Let me also quickly check how accumulateOpenAINonStreamJSON returns its usage to see if there's a code path that could produce undefined usage:
<path>/home/byk/.local/share/opencode/worktree/51d1015fd1f616acf9a5ffaad9331ec6b304586b/clever-circuit/packages/gateway/src/pipeline.ts</path>
<type>file</type>
<content>
2252: input,
2253: });
2254: }
2255: }
2256: }
2257:
2258: // Map OpenAI finish_reason to gateway stop reason
2259: const finishReason = firstChoice?.finish_reason as string | undefined;
2260: let stopReason = "end_turn";
2261: if (finishReason === "stop") stopReason = "end_turn";
2262: else if (finishReason === "length") stopReason = "max_tokens";
2263: else if (finishReason === "tool_calls") stopReason = "tool_use";
2264:
2265: const usage = json.usage as Record<string, unknown> | undefined;
2266: const promptTokensDetails = usage?.prompt_tokens_details as
2267: | Record<string, number>
2268: | undefined;
2269:
2270: return {
2271: id: String(json.id ?? ""),
2272: model: String(json.model ?? ...
Now I have a complete picture of the PR. Here is my structured review:
fix(gateway): guard resp.usage accesses against undefined for vLLM/partial responsesThe PR makes GatewayResponse.usage optional (usage?: GatewayUsage), introduces a ZERO_USAGE constant as a safe fallback, and guards all resp.usage access sites with ?? ZERO_USAGE (or inline ?? { inputTokens: 0, outputTokens: 0 }). Test assertions are updated from resp.usage.field to resp.usage?.field.
CRITICAL: ZERO_USAGE includes cacheReadInputTokens: 0 and cacheCreationInputTokens: 0, which changes the semantics of null checks downstream
The ZERO_USAGE constant sets cacheReadInputTokens: 0 and cacheCreationInputTokens: 0. These fields are normally undefined when no cache data is present (they're declared as optional on GatewayUsage). Throughout the codebase, there are many != null / != undefined guards like:
if (resp.usage.cacheReadInputTokens != null) {
usage.cache_read_input_tokens = resp.usage.cacheReadInputTokens;
}
When ZERO_USAGE is used as the fallback, these != null checks now pass (because 0 !== null), causing cache_read_input_tokens: 0 to be emitted in the wire response. This changes client-visible API behavior: clients that check for the presence of these optional fields to determine whether the provider supports prompt caching will now incorrectly see caching fields where they didn't exist before.
Fix: Remove cacheReadInputTokens and cacheCreationInputTokens from ZERO_USAGE, or set them to undefined:
export const ZERO_USAGE: GatewayUsage = {
inputTokens: 0,
outputTokens: 0,
};
This preserves the semantic distinction: "no usage data at all" ≠ "caching was used but had zero tokens."
**2a. ...