Dashboard › sentry › Session 0uHzjTP5RFaB

Session 0uHzjTP5RFaB

Full ID: 0uHzjTP5RFaBCeUXc
Messages: 13
Distillations: 0
Time range: 2026-06-07 19:49:22 — 2026-06-07 19:51:21

Conversation (13 messages)

Research the feasibility of running llama.cpp inference inside a Node.js/Bun process. Look for:

  1. Search the web for "node-llama-cpp" npm package - what is its current state, API, size?
  2. Search for "@anthropic-ai/tokenizers" or similar GGUF inference packages for Node
  3. Search for "llama.node" or any native addon that wraps llama.cpp for Node.js
  4. Check if there's a WASM build of llama.cpp that could run in Node (like we do with ONNX for embeddings)
  5. Look at https://www.npmjs.com/package/node-llama-cpp for current version and API

For each option found, note:

  • Package name and npm weekly downloads
  • Binary/addon size
  • API surface (does it support OpenAI-compatible chat completions?)
  • GGUF model loading support
  • GPU acceleration support (CUDA, Metal, Vulkan)
  • Whether it works with Node.js SEA (single executable application)

Return a structured comparison of all viable options.

gateway · 2026-06-07 19:49:22

I'll research all the options systematically. Let me fetch information from multiple sources in parallel.

claude-opus-4-6 · 2026-06-07 19:49:22 · ~36 tokens

NPM is blocking direct fetches. Let me try the npm registry API and the project documentation sites instead.

claude-opus-4-6 · 2026-06-07 19:49:37 · ~37 tokens
result 

...791774 bytes truncated...

The tool call succeeded but the output was truncated. Full output saved to: /home/byk/.local/share/opencode/tool-output/tool_ea3a22cc0001ZvmGW2Sn1yHSxl
Use the Task tool to have explore agent process this file with Grep and Read (with offset/limit). Do NOT read the full file yourself - delegate to save context.
result 
  Getting Started | node-llama-cpp                             

[Skip to content](#VPContent)

[![](/icon.svg)node-llama-cpp](/)

SearchK

Main Navigation [Guide](/guide/)[CLI](/cli/)[API Reference](/api/functions/getLlama)[Blog](/blog/)

3.18.1

[Changelog](https://github.com/withcatai/node-llama-cpp/releases)

[Roadmap](https://github.com/orgs/withcatai/projects/1)

[npm](https://www.npmjs.com/package/node-llama-cpp)

[GitHub Discussions](https://github.com/withcatai/node-llama-cpp/discussions)

[Awesome List](/guide/awesome)

[Contribute](/guide/contributing)

[Sponsor](https://github.com/sponsors/giladgd)

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Appearance

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Menu

Return to top

Sidebar Navigation

## Guide

[

Getting Started

](/guide/)

[

Chat Session

](/guide/chat-session)

[

Chat ...
gateway · 2026-06-07 19:49:48

Good, I got the documentation. Now let me fetch more details — download counts, binary size info, and alternative packages.

claude-opus-4-6 · 2026-06-07 19:49:48 · ~42 tokens
result 
{"downloads":184257,"start":"2026-05-27","end":"2026-06-02","package":"node-llama-cpp"}
result 
{"downloads":1105031,"start":"2026-05-27","end":"2026-06-02","package":"@huggingface/transformers"}
result 
{"name":"node-llama-cpp","version":"3.18.1","description":"Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level","main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","bin":{"node-llama-cpp":"dist/cli/cli.js","nlc":"dist/cli/cli.js"},"exports":{".":{"import":"./dist/index.js","node":"./dist/index.js","types":"./dist/index.d.ts","default":"./dist/index.js"},"./commands":{"import":"./dist/commands.js","node":"./dist/commands.js","types":"./dist/commands.d.ts","default":"./dist/commands.js"},"./commands.js":{"import":"./dist/commands.js","node":"./dist/commands.js","types":"./dist/commands.d.ts","default":"./dist/commands.js"}},"engines":{"node":">=20.0.0"},"scripts":{"prepare":"node --experimental-require-module -e \"process.env.CI !== 'true' && console.log(require('husky').default())\"","postinstall":"node ./dist/cli/cli.js postinstall","postversion":"vite-node ...
result 
  Building From Source | node-llama-cpp                              

[Skip to content](#VPContent)

[![](/icon.svg)node-llama-cpp](/)

SearchK

Main Navigation [Guide](/guide/)[CLI](/cli/)[API Reference](/api/functions/getLlama)[Blog](/blog/)

3.18.1

[Changelog](https://github.com/withcatai/node-llama-cpp/releases)

[Roadmap](https://github.com/orgs/withcatai/projects/1)

[npm](https://www.npmjs.com/package/node-llama-cpp)

[GitHub Discussions](https://github.com/withcatai/node-llama-cpp/discussions)

[Awesome List](/guide/awesome)

[Contribute](/guide/contributing)

[Sponsor](https://github.com/sponsors/giladgd)

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Appearance

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Menu

Return to top

Sidebar Navigation

## Guide

[

Getting Started

](/guide/)

[

Chat Session

](/guide/chat-session)

[

Chat ...
result 
  Using in Electron | node-llama-cpp                             

[Skip to content](#VPContent)

[![](/icon.svg)node-llama-cpp](/)

SearchK

Main Navigation [Guide](/guide/)[CLI](/cli/)[API Reference](/api/functions/getLlama)[Blog](/blog/)

3.18.1

[Changelog](https://github.com/withcatai/node-llama-cpp/releases)

[Roadmap](https://github.com/orgs/withcatai/projects/1)

[npm](https://www.npmjs.com/package/node-llama-cpp)

[GitHub Discussions](https://github.com/withcatai/node-llama-cpp/discussions)

[Awesome List](/guide/awesome)

[Contribute](/guide/contributing)

[Sponsor](https://github.com/sponsors/giladgd)

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Appearance

[](https://www.npmjs.com/package/node-llama-cpp)[](https://github.com/withcatai/node-llama-cpp)

Menu

Return to top

Sidebar Navigation

## Guide

[

Getting Started

](/guide/)

[

Chat Session

](/guide/chat-session)

[

Chat ...
gateway · 2026-06-07 19:49:56

Now let me get the binary sizes and check for alternative packages.

claude-opus-4-6 · 2026-06-07 19:49:56 · ~23 tokens
result 
dist.unpackedSize = 21331472
dist.fileCount = 23
version = '3.18.1'
gateway · 2026-06-07 19:50:04
result 
dist.unpackedSize = 5457019
dist.fileCount = 12
version = '3.18.1'
---
dist.unpackedSize = 159456082
dist.fileCount = 24
version = '3.18.1'
---
dist.unpackedSize = 78845354
dist.fileCount = 24
version = '3.18.1'
---
dist.unpackedSize = 35868775
dist.fileCount = 24
version = '3.18.1'
result 
{"name":"@anthropic-ai/tokenizer","version":"0.0.4","author":{"name":"Anthropic","email":"support@anthropic.com"},"license":"Apache-2.0","_id":"@anthropic-ai/tokenizer@0.0.4","maintainers":[{"name":"bkrausz","email":"briankrausz@gmail.com"},{"name":"ejlangev-ant","email":"ethan.langevin@anthropic.com"},{"name":"nikhil-anthropic","email":"nikhil@anthropic.com"},{"name":"jenan-anthropic","email":"jenan@anthropic.com"},{"name":"benjmann","email":"ben@anthropic.com"},{"name":"jspahrsummers","email":"justin@jspahrsummers.com"}],"dist":{"shasum":"d1f5dab07bbf9289414dad1f7c57b812b27bb857","tarball":"https://registry.npmjs.org/@anthropic-ai/tokenizer/-/tokenizer-0.0.4.tgz","fileCount":34,"integrity":"sha512-EHRKbxlxlc8W4KCBEseByJ7YwyYCmgu9OyN59H9+IYIGPoKv8tXyQXinkeGDI+cI8Tiuz9wk2jZb/kK7AyvL7g==","signatures":[{"sig":"MEYCIQCg6kpyWSS9fGZu97tzABPzKeeEZ40LJQ/kzPCXBs9f4gIhANiwVLPkQwFRqc5UTdreujuAf/g1IUXAYx6XheOvGiq2","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":140...
result 
{"downloads":201268,"start":"2026-05-27","end":"2026-06-02","package":"@anthropic-ai/tokenizer"}
gateway · 2026-06-07 19:50:13

Let me search for WASM alternatives and other packages.

claude-opus-4-6 · 2026-06-07 19:50:13 · ~19 tokens
result 
{"name":"@wllama/wllama","version":"3.4.1","description":"WebAssembly binding for llama.cpp - Enabling on-browser LLM inference","main":"index.js","type":"module","directories":{"example":"examples"},"scripts":{"serve":"node ./scripts/http_server.js","serve:mt":"MULTITHREAD=1 node ./scripts/http_server.js","clean":"rm -rf ./esm && rm -rf ./docs && rm -rf ./wasm","build:worker":"./scripts/build_worker.sh","build:glue":"node ./cpp/generate_glue_prototype.js","build:wasm":"./scripts/build_wasm.sh && npm run build:glue","build:tsup":"tsup src/index.ts --format cjs,esm --clean","build:minified":"terser esm/index.js -o esm/index.min.js --compress --mangle --source-map","build:typedef":"tsc --emitDeclarationOnly --declaration -p tsconfig.build.json","build":"npm run clean && npm run build:worker && npm run build:tsup && npm run build:minified && npm run build:typedef","postbuild":"./scripts/post_build.sh && npm run docs","docs":"typedoc --tsconfig tsconfig.build.json ...
result 
@huggingface/gguf                        a GGUF parser that works on remotely hosted files
@react-native-ai/llama                   llama.rn provider for Vercel AI SDK
node-llama-cpp                           Run AI models locally on your machine with node.js bindings for llama.cpp. Enfor
llama-cpp-capacitor                      A native Capacitor plugin that embeds llama.cpp directly into mobile apps, enabl
@node-llama-cpp/linux-x64                Prebuilt binary for node-llama-cpp for Linux x64
@node-llama-cpp/linux-x64-cuda-ext       Extension of @node-llama-cpp/linux-x64-cuda - prebuilt binary for node-llama-cpp
@node-llama-cpp/linux-x64-vulkan         Prebuilt binary for node-llama-cpp for Linux x64 with Vulkan support
@node-llama-cpp/linux-x64-cuda           Prebuilt binary for node-llama-cpp for Linux x64 with CUDA support
@node-llama-cpp/linux-armv7l             Prebuilt binary for node-llama-cpp for Linux armv7l
@node-llama-cpp/linux-arm64              Prebuilt binary for ...
gateway · 2026-06-07 19:50:19
result 
{"downloads":4154,"start":"2026-05-27","end":"2026-06-02","package":"@wllama/wllama"}
result 
                       GitHub - ngxson/wllama: WebAssembly binding for llama.cpp - Enabling on-browser LLM inference · GitHub                                             

[Skip to content](#start-of-content)   

## Navigation Menu

Toggle navigation

[](/)

[Sign in](/login?return_to=https%3A%2F%2Fgithub.com%2Fngxson%2Fwllama)

Appearance settings

  

-   Platform
    
    -   AI CODE CREATION
        
        -   [
            
            GitHub CopilotWrite better code with AI
            
            ](https://github.com/features/copilot)
        -   [
            
            GitHub Copilot appDirect agents from issue to merge
            
            ](https://github.com/features/ai/github-app)
        -   [
            
            MCP RegistryNewIntegrate external tools
            
            ](https://github.com/mcp)
        
    -   DEVELOPER WORKFLOWS
        
        -   [
            
            ActionsAutomate any workflow
            
            ...
result 
32332515
result 
{"name":"@huggingface/gguf","version":"0.4.2","description":"a GGUF parser that works on remotely hosted files","keywords":["gguf","hub","huggingface"],"license":"MIT","author":{"name":"Hugging Face"},"repository":{"type":"git","url":"git+https://github.com/huggingface/huggingface.js.git"},"bin":{"gguf-view":"dist/cli.js"},"source":"index.ts","main":"./dist/index.js","module":"./dist/index.mjs","browser":{"./src/cli.ts":false,"./src/utils/FileBlob.ts":false,"./dist/index.js":"./dist/browser/index.js","./dist/index.mjs":"./dist/browser/index.mjs"},"types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","require":"./dist/index.js","import":"./dist/index.mjs"}},"publishConfig":{"access":"public"},"dependencies":{"@huggingface/tasks":"^0.20.12"},"devDependencies":{"@types/node":"^20.12.8"},"engines":{"node":">=20"},"scripts":{"lint":"eslint --quiet --fix --ext .cjs,.ts .","lint:check":"eslint --ext .cjs,.ts .","format":"oxfmt .","format:check":"oxfmt --check ...
result 
{"downloads":4044,"start":"2026-05-27","end":"2026-06-02","package":"@huggingface/gguf"}
gateway · 2026-06-07 19:51:21

Now I have all the data needed. Here's the structured comparison:

llama.cpp Inference in Node.js/Bun — Feasibility Report

Option 1: node-llama-cpp (Native Addon — Clear Winner)

| Attribute | Details | |---|---| | Package | node-llama-cpp v3.18.1 | | Weekly downloads | 184,257 | | License | MIT | | Node requirement | >= 20.0.0, ESM-only ("type": "module") | | JS package size | 32.3 MB (unpacked, includes templates, docs, CLI, git bundle of llama.cpp source) | | Native binary sizes | See below |

Prebuilt binary sizes (per-platform optional deps):

| Platform package | Unpacked size | |---|---| | @node-llama-cpp/mac-arm64-metal | 5.5 MB | | @node-llama-cpp/linux-x64 | 21.3 MB | | @node-llama-cpp/linux-x64-vulkan | 78.8 MB | | @node-llama-cpp/linux-x64-cuda | 159.5 MB | | @node-llama-cpp/win-x64 | 35.9 MB |

API surface:

  • High-level: getLlama() → loadModel() → createContext() → LlamaChatSession.prompt()
  • Chat wrappers with auto-detection for Llama 3, DeepSeek R1, Mistral, etc.
  • JSON schema grammar enforcement at generation level
  • Function calling (built-in for Llama 3 Instruct, Functionary; generic fallback)
  • Embeddings, reranking, text completion, batching
  • Speculative decoding (token prediction)
  • LoRA adapter support
  • Low-level token-by-token evaluation API
  • No built-in OpenAI-compatible HTTP server — but the API is rich enough to build one trivially

GPU acceleration:

  • Metal: auto-enabled on Apple Silicon Macs
  • CUDA: auto-detected, separate @node-llama-cpp/linux-x64-cuda binary (~160 MB)
  • Vulkan: auto-detected, separate binary (~79 MB)
  • Accelerate framework always enabled on macOS

GGUF support: Full — loads any GGUF model file directly. Includes CLI for downloading models from HuggingFace.

Node.js SEA compatibility: Problematic. The package:

  1. ...
claude-opus-4-6 · 2026-06-07 19:51:21 · ~2.6K tokens