feat(llama-cpp): in-process local GGUF text inference provider (#109444)

* feat(llama-cpp): add in-process text inference

* test(llama-cpp): narrow setup provider fixture

* fix(llama-cpp): trim public surface and refresh docs map

* fix(llama-cpp): import Context type in inference test

docs/docs_map.md

@@ -5831,9 +5831,11 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
 
 - Route: /plugins/llama-cpp
 - Headings:
-  - H2: Configuration
-  - H2: Native Runtime
-  - H2: Runtime diagnostics
+  - H2: Local text inference
+  - H3: Use another GGUF model
+  - H2: Memory embedding configuration
+  - H2: Native runtime
+  - H2: Memory runtime diagnostics
   - H2: Troubleshooting
 
 ## plugins/logbook.md

docs/plugins/llama-cpp.md

@@ -1,6 +1,7 @@
 ---
-summary: "Install the official llama.cpp provider for local GGUF memory embeddings"
+summary: "Run local GGUF text inference and memory embeddings in OpenClaw with llama.cpp"
 read_when:
+  - You want local text inference without an API key or model server
   - You want memory search embeddings from a local GGUF model
   - You are configuring memorySearch.provider = "local"
   - You need the OpenClaw plugin that owns the node-llama-cpp runtime
@@ -8,11 +9,11 @@ title: "llama.cpp Provider"
 sidebarTitle: "llama.cpp Provider"
 ---
 
-`llama-cpp` is the official external provider plugin for local GGUF
-embeddings. It registers embedding provider id `local` and owns the
-`node-llama-cpp` runtime dependency used by `memorySearch.provider: "local"`.
+`llama-cpp` is the official external provider plugin for in-process local GGUF
+text inference and embeddings. It registers text provider `llama-cpp`,
+embedding provider `local`, and owns the `node-llama-cpp` native runtime.
 
-Install it before using local memory embeddings:
+Install it before using either local inference or local memory embeddings:
 
 ```bash
 openclaw plugins install @openclaw/llama-cpp-provider
@@ -22,7 +23,77 @@ The main `openclaw` npm package does not include `node-llama-cpp`. Keeping the
 native dependency in this plugin prevents normal OpenClaw npm updates from
 deleting a manually installed runtime inside the OpenClaw package directory.
 
-## Configuration
+## Local text inference
+
+Choose **Local model (llama.cpp)** during interactive onboarding. OpenClaw asks
+before downloading the default model:
+
+`hf:bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf`
+
+The Qwen3 4B Instruct 2507 Q4_K_M file is about 2.5 GB. Budget roughly 3 GB of
+RAM for model weights, plus context and OpenClaw runtime overhead. The default
+context is automatically sized with an 8,192-token cap so it remains practical
+on 8 GB machines. Configure a larger context only when the machine has enough
+memory.
+
+The onboarding discovery check is read-only. It offers llama.cpp automatically
+only when the default or configured GGUF file is already in the model cache; it
+never downloads during discovery. Ollama and LM Studio remain separate local
+service choices and keep their own discovery flows. Manually choosing llama.cpp
+is the path that prompts for the default model download.
+
+The provider uses the GGUF model's embedded chat template and native
+node-llama-cpp function calling. Text streams token by token. Tool calls return
+to OpenClaw for execution rather than running inside node-llama-cpp.
+
+### Use another GGUF model
+
+Add a model to `models.providers.llama-cpp`. Put a local path or full `hf:` file
+URI in `params.modelPath`:
+
+```json5
+{
+  models: {
+    mode: "merge",
+    providers: {
+      "llama-cpp": {
+        baseUrl: "local://llama-cpp",
+        api: "openai-completions",
+        params: {
+          modelCacheDir: "~/.node-llama-cpp/models",
+        },
+        models: [
+          {
+            id: "my-local-model",
+            name: "My local GGUF",
+            reasoning: false,
+            input: ["text"],
+            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+            contextWindow: 8192,
+            maxTokens: 2048,
+            params: {
+              modelPath: "~/Models/my-model.Q4_K_M.gguf",
+              contextSize: 8192,
+            },
+            compat: { supportsTools: true },
+          },
+        ],
+      },
+    },
+  },
+  agents: {
+    defaults: {
+      model: { primary: "llama-cpp/my-local-model" },
+    },
+  },
+}
+```
+
+Inference never downloads a missing model implicitly. For a custom `hf:` URI,
+download the GGUF into `modelCacheDir` first. Discovery uses node-llama-cpp's
+own read-only cache resolver, including repository, branch, and split-file naming.
+
+## Memory embedding configuration
 
 Set `memorySearch.provider` to `local`:
 
@@ -52,7 +123,7 @@ to node-llama-cpp's automatic GPU-layer placement. This lets node-llama-cpp fit
 the model and embedding context together while retaining its memory-safety
 checks. With `"auto"`, node-llama-cpp keeps its normal automatic placement.
 
-## Native Runtime
+## Native runtime
 
 Use Node 24 for the smoothest native install path. Source checkouts using
 pnpm may need to approve and rebuild the native dependency:
@@ -62,7 +133,7 @@ pnpm approve-builds
 pnpm rebuild node-llama-cpp
 ```
 
-## Runtime diagnostics
+## Memory runtime diagnostics
 
 Run `openclaw memory status --deep` after the provider has loaded to inspect
 the selected backend and build, device names, GPU offloaded layers, requested
@@ -83,6 +154,7 @@ with:
 2. Use Node 24 for native installs/updates.
 3. From a pnpm source checkout: `pnpm approve-builds`, then `pnpm rebuild node-llama-cpp`.
 
-For lower-friction local embeddings without the native build step, set
+For local inference without an in-process native dependency, use the Ollama or
+LM Studio provider instead. For lower-friction local embeddings, set
 `memorySearch.provider` to a remote embedding provider such as `lmstudio`,
 `ollama`, `openai`, or `voyage` instead.

docs/plugins/plugin-inventory.md

@@ -263,7 +263,7 @@ Each entry lists the package, distribution route, and description.
 
 - **[line](/plugins/reference/line)** (`@openclaw/line`) - npm; ClawHub. OpenClaw LINE channel plugin for LINE Bot API chats.
 
-- **[llama-cpp](/plugins/reference/llama-cpp)** (`@openclaw/llama-cpp-provider`) - npm; ClawHub. Local GGUF embeddings through node-llama-cpp.
+- **[llama-cpp](/plugins/reference/llama-cpp)** (`@openclaw/llama-cpp-provider`) - npm; ClawHub. Local GGUF text inference and embeddings through node-llama-cpp.
 
 - **[lobster](/plugins/reference/lobster)** (`@openclaw/lobster`) - npm; ClawHub. Lobster workflow tool plugin for typed pipelines and resumable approvals.
 

docs/plugins/reference/llama-cpp.md

@@ -1,5 +1,5 @@
 ---
-summary: "Local GGUF embeddings through node-llama-cpp."
+summary: "Local GGUF text inference and embeddings through node-llama-cpp."
 read_when:
   - You are installing, configuring, or auditing the llama-cpp plugin
 title: "Llama Cpp plugin"
@@ -7,7 +7,7 @@ title: "Llama Cpp plugin"
 
 # Llama Cpp plugin
 
-Local GGUF embeddings through node-llama-cpp.
+Local GGUF text inference and embeddings through node-llama-cpp.
 
 ## Distribution
 
@@ -16,7 +16,7 @@ Local GGUF embeddings through node-llama-cpp.
 
 ## Surface
 
-contracts: `embeddingProviders`
+providers: `llama-cpp`; contracts: `embeddingProviders`
 
 ## Related docs
 

extensions/llama-cpp/README.md

@@ -1,8 +1,9 @@
 # @openclaw/llama-cpp-provider
 
-Official llama.cpp embedding provider for OpenClaw.
+Official llama.cpp text-inference and embedding provider for OpenClaw.
 
-This plugin runs local GGUF embedding models through `node-llama-cpp`.
+This plugin runs local GGUF chat and embedding models in-process through
+`node-llama-cpp`.
 
 ## Install
 
@@ -13,7 +14,16 @@ openclaw plugins install @openclaw/llama-cpp-provider
 Restart the Gateway after installing or updating the plugin. Use Node 24 for
 native installs and updates.
 
-## Configure
+## Configure text inference
+
+Choose **Local model (llama.cpp)** during onboarding. After explicit consent,
+OpenClaw downloads the approximately 2.5 GB Qwen3 4B Instruct 2507 Q4_K_M
+default. Discovery never downloads a model.
+
+See the [llama.cpp provider guide](https://docs.openclaw.ai/plugins/llama-cpp)
+for custom GGUF model configuration and hardware guidance.
+
+## Configure embeddings
 
 Set `agents.defaults.memorySearch.provider` to `local`. By default, the plugin
 downloads and uses the EmbeddingGemma GGUF model. Configure

extensions/llama-cpp/index.test.ts

@@ -1,6 +1,7 @@
 import os from "node:os";
 import path from "node:path";
 import { expectDefined } from "@openclaw/normalization-core";
+import { createTestPluginApi } from "openclaw/plugin-sdk/plugin-test-api";
 import {
   createPluginRegistryFixture,
   registerVirtualTestPlugin,
@@ -47,6 +48,31 @@ afterEach(() => {
 });
 
 describe("llama.cpp provider plugin", () => {
+  it("registers the local text-inference provider", () => {
+    const registerProvider = vi.fn();
+
+    llamaCppPlugin.register(
+      createTestPluginApi({
+        id: "llama-cpp",
+        name: "llama.cpp Provider",
+        source: "test",
+        config: {},
+        pluginConfig: {},
+        runtime: {} as never,
+        registerProvider,
+      }),
+    );
+
+    expect(registerProvider).toHaveBeenCalledWith(
+      expect.objectContaining({
+        id: "llama-cpp",
+        label: "Local model (llama.cpp)",
+        createStreamFn: expect.any(Function),
+        auth: [expect.objectContaining({ id: "local" })],
+      }),
+    );
+  });
+
   it("registers the local embedding provider through the generic SDK contract", () => {
     const { config, registry } = createPluginRegistryFixture();
 

extensions/llama-cpp/index.ts

@@ -1,11 +1,74 @@
-import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
+import { definePluginEntry, type OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry";
+import {
+  LLAMA_CPP_PROVIDER_ID,
+  LLAMA_CPP_PROVIDER_LABEL,
+  buildLlamaCppProviderConfig,
+  resolveLlamaCppSyntheticApiKey,
+} from "./src/defaults.js";
 import { llamaCppEmbeddingProviderAdapter } from "./src/embedding-provider.js";
+import { createLlamaCppStreamFn } from "./src/inference-provider.js";
+import { detectLlamaCppSetup, prepareLlamaCppSetup, runLlamaCppSetup } from "./src/setup.js";
 
 export default definePluginEntry({
   id: "llama-cpp",
   name: "llama.cpp Provider",
-  description: "Local GGUF embeddings through node-llama-cpp",
-  register(api) {
+  description: "Local GGUF text inference and embeddings through node-llama-cpp",
+  register(api: OpenClawPluginApi) {
     api.registerEmbeddingProvider(llamaCppEmbeddingProviderAdapter);
+    api.registerProvider({
+      id: LLAMA_CPP_PROVIDER_ID,
+      label: LLAMA_CPP_PROVIDER_LABEL,
+      docsPath: "/plugins/llama-cpp",
+      auth: [
+        {
+          id: "local",
+          label: LLAMA_CPP_PROVIDER_LABEL,
+          hint: "In-process local GGUF model (about 2.5 GB download)",
+          kind: "custom",
+          appGuidedSetup: {
+            detect: detectLlamaCppSetup,
+            prepare: prepareLlamaCppSetup,
+          },
+          run: runLlamaCppSetup,
+        },
+      ],
+      catalog: {
+        order: "late",
+        run: async (ctx) => ({
+          provider: buildLlamaCppProviderConfig(
+            ctx.config.models?.providers?.[LLAMA_CPP_PROVIDER_ID],
+          ),
+        }),
+      },
+      staticCatalog: {
+        order: "late",
+        run: async () => ({ provider: buildLlamaCppProviderConfig() }),
+      },
+      createStreamFn: ({ config, provider }) =>
+        createLlamaCppStreamFn({
+          providerConfig: config?.models?.providers?.[provider],
+        }),
+      resolveSyntheticAuth: () => ({
+        apiKey: resolveLlamaCppSyntheticApiKey(),
+        source: "local llama.cpp runtime",
+        mode: "api-key" as const,
+      }),
+      wizard: {
+        setup: {
+          choiceId: LLAMA_CPP_PROVIDER_ID,
+          choiceLabel: LLAMA_CPP_PROVIDER_LABEL,
+          choiceHint: "In-process local model (about 2.5 GB download)",
+          groupId: LLAMA_CPP_PROVIDER_ID,
+          groupLabel: "Local llama.cpp",
+          groupHint: "No API key required",
+          methodId: "local",
+        },
+        modelPicker: {
+          label: "llama.cpp (local GGUF)",
+          hint: "Run a GGUF model in the OpenClaw process",
+          methodId: "local",
+        },
+      },
+    });
   },
 });

extensions/llama-cpp/openclaw.plugin.json

@@ -1,11 +1,41 @@
 {
   "id": "llama-cpp",
   "name": "llama.cpp Provider",
-  "description": "Local GGUF embeddings through node-llama-cpp.",
+  "description": "Local GGUF text inference and embeddings through node-llama-cpp.",
   "activation": {
     "onStartup": false
   },
   "enabledByDefault": true,
+  "providers": ["llama-cpp"],
+  "providerRequest": {
+    "providers": {
+      "llama-cpp": {
+        "family": "llama-cpp"
+      }
+    }
+  },
+  "modelPricing": {
+    "providers": {
+      "llama-cpp": {
+        "external": false
+      }
+    }
+  },
+  "syntheticAuthRefs": ["llama-cpp"],
+  "nonSecretAuthMarkers": ["llama-cpp-local"],
+  "providerAuthChoices": [
+    {
+      "provider": "llama-cpp",
+      "method": "local",
+      "choiceId": "llama-cpp",
+      "appGuidedDiscovery": true,
+      "choiceLabel": "Local model (llama.cpp)",
+      "choiceHint": "Downloads an approximately 2.5 GB local model",
+      "groupId": "llama-cpp",
+      "groupLabel": "Local llama.cpp",
+      "groupHint": "No API key required"
+    }
+  ],
   "contracts": {
     "embeddingProviders": ["local"]
   },

extensions/llama-cpp/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@openclaw/llama-cpp-provider",
   "version": "2026.7.2",
-  "description": "OpenClaw llama.cpp embedding provider plugin",
+  "description": "OpenClaw llama.cpp text inference and embedding provider plugin",
   "repository": {
     "type": "git",
     "url": "https://github.com/openclaw/openclaw"

extensions/llama-cpp/src/defaults.ts

@@ -0,0 +1,108 @@
+import os from "node:os";
+import path from "node:path";
+import type {
+  ModelDefinitionConfig,
+  ModelProviderConfig,
+} from "openclaw/plugin-sdk/provider-model-shared";
+
+export const LLAMA_CPP_PROVIDER_ID = "llama-cpp";
+export const LLAMA_CPP_PROVIDER_LABEL = "Local model (llama.cpp)";
+const LLAMA_CPP_LOCAL_AUTH_MARKER = "llama-cpp-local";
+const LLAMA_CPP_LOCAL_BASE_URL = "local://llama-cpp";
+
+export function resolveLlamaCppSyntheticApiKey(): string {
+  return LLAMA_CPP_LOCAL_AUTH_MARKER;
+}
+
+export const DEFAULT_LLAMA_CPP_MODEL_ID = "qwen3-4b-instruct-2507-q4_k_m";
+export const DEFAULT_LLAMA_CPP_MODEL_REF = `${LLAMA_CPP_PROVIDER_ID}/${DEFAULT_LLAMA_CPP_MODEL_ID}`;
+// Verified 2026-07-16: 2,497,280,736 bytes (about 2.5 GB) from the public
+// bartowski mirror. Qwen does not publish an official Instruct-2507 GGUF repo.
+export const DEFAULT_LLAMA_CPP_MODEL_URI =
+  "hf:bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF/Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf";
+export const DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE =
+  "hf_bartowski_Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf";
+export const DEFAULT_LLAMA_CPP_MODEL_SIZE_BYTES = 2_497_280_736;
+export const DEFAULT_LLAMA_CPP_CONTEXT_SIZE = 8192;
+
+export function resolveLlamaCppModelCacheDir(provider?: ModelProviderConfig): string {
+  const configured = provider?.params?.modelCacheDir;
+  return typeof configured === "string" && configured.trim()
+    ? resolveHomePath(configured.trim())
+    : path.join(os.homedir(), ".node-llama-cpp", "models");
+}
+
+function resolveHomePath(value: string): string {
+  if (value === "~") {
+    return os.homedir();
+  }
+  if (value.startsWith("~/")) {
+    return path.join(os.homedir(), value.slice(2));
+  }
+  return value;
+}
+
+export function resolveLlamaCppModelSource(model: {
+  id: string;
+  params?: Record<string, unknown>;
+}): string {
+  const configured = model.params?.modelPath;
+  if (typeof configured === "string" && configured.trim()) {
+    return resolveHomePath(configured.trim());
+  }
+  return model.id === DEFAULT_LLAMA_CPP_MODEL_ID
+    ? DEFAULT_LLAMA_CPP_MODEL_URI
+    : resolveHomePath(model.id);
+}
+
+export function resolveCachedLlamaCppModelPath(params: {
+  model: Pick<ModelDefinitionConfig, "id" | "params">;
+  provider?: ModelProviderConfig;
+}): string | null {
+  const source = resolveLlamaCppModelSource(params.model);
+  const cacheDir = resolveLlamaCppModelCacheDir(params.provider);
+  if (source === DEFAULT_LLAMA_CPP_MODEL_URI) {
+    return path.join(cacheDir, DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE);
+  }
+  if (/^hf:/i.test(source)) {
+    return null;
+  }
+  if (/^https?:\/\//i.test(source)) {
+    return null;
+  }
+  const localPath = resolveHomePath(source);
+  return path.isAbsolute(localPath) ? localPath : path.resolve(cacheDir, localPath);
+}
+
+function buildDefaultLlamaCppModel(): ModelDefinitionConfig {
+  return {
+    id: DEFAULT_LLAMA_CPP_MODEL_ID,
+    name: "Qwen3 4B Instruct 2507 (Q4_K_M)",
+    api: "openai-completions",
+    reasoning: false,
+    input: ["text"],
+    cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+    contextWindow: DEFAULT_LLAMA_CPP_CONTEXT_SIZE,
+    contextTokens: DEFAULT_LLAMA_CPP_CONTEXT_SIZE,
+    maxTokens: 2048,
+    params: {
+      modelPath: DEFAULT_LLAMA_CPP_MODEL_URI,
+      contextSize: "auto",
+    },
+    compat: { supportsTools: true, supportsUsageInStreaming: true },
+  };
+}
+
+export function buildLlamaCppProviderConfig(existing?: ModelProviderConfig): ModelProviderConfig {
+  const defaultModel = buildDefaultLlamaCppModel();
+  const configuredModels = existing?.models ?? [];
+  const models = configuredModels.some((model) => model.id === defaultModel.id)
+    ? configuredModels
+    : [...configuredModels, defaultModel];
+  return {
+    ...existing,
+    baseUrl: existing?.baseUrl ?? LLAMA_CPP_LOCAL_BASE_URL,
+    api: existing?.api ?? "openai-completions",
+    models,
+  };
+}

extensions/llama-cpp/src/embedding-provider.ts

@@ -1,7 +1,5 @@
-import { createRequire } from "node:module";
 import os from "node:os";
 import path from "node:path";
-import { pathToFileURL } from "node:url";
 import type {
   EmbeddingInput,
   EmbeddingProvider,
@@ -16,6 +14,7 @@ import {
   type MemoryEmbeddingProviderCreateOptions,
   type MemoryEmbeddingProviderCreateResult,
 } from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
+import { formatLlamaCppSetupError, resolveNodeLlamaCppImportUrl } from "./node-llama.runtime.js";
 
 type LlamaCppLocalOptions = {
   modelPath?: string;
@@ -118,48 +117,6 @@ function toMemoryEmbeddingInput(input: EmbeddingInput): MemoryEmbeddingInput {
   return typeof input === "string" ? { text: input } : input;
 }
 
-function isNodeLlamaCppMissing(err: unknown): boolean {
-  if (!(err instanceof Error)) {
-    return false;
-  }
-  const code = (err as Error & { code?: unknown }).code;
-  return code === "ERR_MODULE_NOT_FOUND" && err.message.includes("node-llama-cpp");
-}
-
-function formatErrorMessage(err: unknown): string {
-  if (err instanceof Error) {
-    return err.message;
-  }
-  return String(err);
-}
-
-function formatLlamaCppSetupError(err: unknown): string {
-  const detail = formatErrorMessage(err);
-  const missing = isNodeLlamaCppMissing(err);
-  return [
-    "Local llama.cpp embeddings unavailable.",
-    missing
-      ? "Reason: node-llama-cpp is missing or failed to install."
-      : detail
-        ? `Reason: ${detail}`
-        : undefined,
-    missing && detail ? `Detail: ${detail}` : null,
-    "To enable local GGUF embeddings:",
-    "1) Install the official provider plugin: openclaw plugins install @openclaw/llama-cpp-provider",
-    "2) Use Node 24 for native installs/updates.",
-    "3) If you use pnpm from source: pnpm approve-builds, then pnpm rebuild node-llama-cpp.",
-    'Or set agents.defaults.memorySearch.provider to a remote embedding provider such as "openai", "ollama", "lmstudio", or "voyage".',
-  ]
-    .filter(Boolean)
-    .join("\n");
-}
-
-const requireFromPlugin = createRequire(import.meta.url);
-
-function resolveNodeLlamaCppImportUrl(): string {
-  return pathToFileURL(requireFromPlugin.resolve("node-llama-cpp")).href;
-}
-
 function copyLocalRuntimeFacts(source: object, target: object): void {
   const getRuntimeFacts = Reflect.get(source, LOCAL_EMBEDDING_RUNTIME_FACTS);
   if (typeof getRuntimeFacts === "function") {

extensions/llama-cpp/src/inference-provider.test.ts

@@ -0,0 +1,397 @@
+import os from "node:os";
+import path from "node:path";
+import type { AssistantMessageEvent, Context, Model } from "openclaw/plugin-sdk/llm";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+const mocks = vi.hoisted(() => {
+  const generateResponse = vi.fn();
+  const resolveModelFile = vi.fn(
+    async (source: string) => `/models/${source.replaceAll("/", "_")}`,
+  );
+  const contextDispose = vi.fn(async () => {});
+  const modelDispose = vi.fn(async () => {});
+  const llamaDispose = vi.fn(async () => {});
+  const diff = vi.fn(() => ({ usedInputTokens: 7, usedOutputTokens: 2 }));
+  const getState = vi.fn(() => ({ usedInputTokens: 0, usedOutputTokens: 0 }));
+  const sequence = { tokenMeter: { getState, diff } };
+  const context = {
+    getSequence: vi.fn(() => sequence),
+    dispose: contextDispose,
+  };
+  const model = {
+    createContext: vi.fn(async () => context),
+    dispose: modelDispose,
+  };
+  const llama = {
+    loadModel: vi.fn(async () => model),
+    dispose: llamaDispose,
+  };
+  return {
+    generateResponse,
+    resolveModelFile,
+    contextDispose,
+    modelDispose,
+    llamaDispose,
+    getState,
+    diff,
+    sequence,
+    context,
+    model,
+    llama,
+    getLlama: vi.fn(async () => llama),
+  };
+});
+
+vi.mock("node-llama-cpp", () => ({
+  getLlama: mocks.getLlama,
+  resolveModelFile: mocks.resolveModelFile,
+  createModelDownloader: vi.fn(),
+  LlamaChat: class {
+    generateResponse = mocks.generateResponse;
+    dispose = vi.fn();
+  },
+}));
+
+import { createLlamaCppStreamFn } from "./inference-provider.js";
+
+const {
+  clearLlamaCppInferenceCacheForTests,
+  mapContextToLlamaChatHistory,
+  mapToolsToLlamaFunctions,
+} = (globalThis as Record<PropertyKey, unknown>)[
+  Symbol.for("openclaw.llamaCppInferenceTestApi")
+] as {
+  clearLlamaCppInferenceCacheForTests: () => Promise<void>;
+  mapContextToLlamaChatHistory: (context: Context) => unknown[];
+  mapToolsToLlamaFunctions: (context: Context) => Record<string, unknown> | undefined;
+};
+
+const model: Model = {
+  id: "test.gguf",
+  name: "test",
+  api: "openai-completions",
+  provider: "llama-cpp",
+  baseUrl: "local://llama-cpp",
+  reasoning: false,
+  input: ["text"],
+  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+  contextWindow: 8192,
+  contextTokens: 8192,
+  maxTokens: 2048,
+  params: { modelPath: "test.gguf" },
+};
+
+async function collectEvents(
+  stream: AsyncIterable<AssistantMessageEvent>,
+): Promise<AssistantMessageEvent[]> {
+  const events: AssistantMessageEvent[] = [];
+  for await (const event of stream) {
+    events.push(event);
+  }
+  return events;
+}
+
+beforeEach(async () => {
+  await clearLlamaCppInferenceCacheForTests();
+  vi.clearAllMocks();
+  mocks.generateResponse.mockResolvedValue({
+    response: "",
+    functionCalls: undefined,
+    metadata: { stopReason: "eogToken" },
+  });
+});
+
+afterEach(async () => {
+  await clearLlamaCppInferenceCacheForTests();
+});
+
+describe("llama.cpp inference provider", () => {
+  it("maps OpenClaw history and tool results into the model chat template history", () => {
+    const context = {
+      systemPrompt: "Be concise.",
+      messages: [
+        { role: "user" as const, content: "weather?", timestamp: 1 },
+        {
+          role: "assistant" as const,
+          api: "openai-completions",
+          provider: "test",
+          model: "test",
+          stopReason: "toolUse" as const,
+          usage: {
+            input: 1,
+            output: 1,
+            cacheRead: 0,
+            cacheWrite: 0,
+            totalTokens: 2,
+            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
+          },
+          timestamp: 2,
+          content: [
+            { type: "text" as const, text: "Checking." },
+            {
+              type: "toolCall" as const,
+              id: "call-1",
+              name: "weather",
+              arguments: { city: "Berlin" },
+            },
+          ],
+        },
+        {
+          role: "toolResult" as const,
+          toolCallId: "call-1",
+          toolName: "weather",
+          content: [{ type: "text" as const, text: "Sunny" }],
+          isError: false,
+          timestamp: 3,
+        },
+        { role: "user" as const, content: "thanks", timestamp: 4 },
+      ],
+    };
+
+    expect(mapContextToLlamaChatHistory(context)).toEqual([
+      { type: "system", text: "Be concise." },
+      { type: "user", text: "weather?" },
+      {
+        type: "model",
+        response: [
+          "Checking.",
+          {
+            type: "functionCall",
+            name: "weather",
+            params: { city: "Berlin" },
+            result: "Sunny",
+          },
+        ],
+      },
+      { type: "user", text: "thanks" },
+    ]);
+  });
+
+  it("maps JSON-schema tools to native node-llama-cpp function definitions", () => {
+    expect(
+      mapToolsToLlamaFunctions({
+        messages: [],
+        tools: [
+          {
+            name: "weather",
+            description: "Get weather",
+            parameters: {
+              type: "object",
+              properties: { city: { type: "string" } },
+              required: ["city"],
+            },
+          },
+        ],
+      }),
+    ).toEqual({
+      weather: {
+        description: "Get weather",
+        params: {
+          type: "object",
+          properties: { city: { type: "string" } },
+          required: ["city"],
+        },
+      },
+    });
+  });
+
+  it("streams text deltas and reports native token-meter usage", async () => {
+    mocks.generateResponse.mockImplementationOnce(async (_history, options) => {
+      options.onTextChunk("Hel");
+      options.onTextChunk("lo");
+      return {
+        response: "Hello",
+        functionCalls: undefined,
+        metadata: { stopReason: "eogToken" },
+      };
+    });
+    const stream = await createLlamaCppStreamFn({})(
+      model,
+      { messages: [{ role: "user", content: "Hi", timestamp: 1 }] },
+      { stop: ["END"] },
+    );
+
+    const events = await collectEvents(stream);
+
+    expect(events.map((event) => event.type)).toEqual([
+      "start",
+      "text_start",
+      "text_delta",
+      "text_delta",
+      "text_end",
+      "done",
+    ]);
+    expect(events.at(-1)).toMatchObject({
+      type: "done",
+      reason: "stop",
+      message: {
+        content: [{ type: "text", text: "Hello" }],
+        usage: { input: 7, output: 2, totalTokens: 9 },
+      },
+    });
+    expect(mocks.generateResponse.mock.calls[0]?.[1]).toMatchObject({
+      maxTokens: 2048,
+      customStopTriggers: ["END"],
+    });
+  });
+
+  it("emits native function calls in the final assistant message", async () => {
+    mocks.generateResponse.mockResolvedValueOnce({
+      response: "",
+      functionCalls: [{ functionName: "weather", params: { city: "Paris" }, raw: [] }],
+      metadata: { stopReason: "functionCalls" },
+    });
+    const stream = await createLlamaCppStreamFn({})(model, {
+      messages: [{ role: "user", content: "Weather?", timestamp: 1 }],
+      tools: [
+        {
+          name: "weather",
+          description: "Get weather",
+          parameters: { type: "object", properties: { city: { type: "string" } } },
+        },
+      ],
+    });
+
+    const events = await collectEvents(stream);
+
+    expect(events.map((event) => event.type)).toEqual(["done"]);
+    expect(events.at(-1)).toMatchObject({
+      type: "done",
+      reason: "toolUse",
+      message: {
+        content: [
+          {
+            type: "toolCall",
+            id: expect.stringMatching(/^llama_cpp_call_/),
+            name: "weather",
+            arguments: { city: "Paris" },
+          },
+        ],
+      },
+    });
+  });
+
+  it("disposes the previous model and context when the model changes", async () => {
+    const streamFn = createLlamaCppStreamFn({});
+    await collectEvents(
+      await streamFn(model, { messages: [{ role: "user", content: "one", timestamp: 1 }] }),
+    );
+    await collectEvents(
+      await streamFn(
+        { ...model, id: "other.gguf", params: { modelPath: "other.gguf" } },
+        { messages: [{ role: "user", content: "two", timestamp: 2 }] },
+      ),
+    );
+
+    expect(mocks.contextDispose).toHaveBeenCalledTimes(1);
+    expect(mocks.modelDispose).toHaveBeenCalledTimes(1);
+    expect(mocks.llama.loadModel).toHaveBeenCalledTimes(2);
+  });
+
+  it("reuses one context sequence across serialized requests for the same model", async () => {
+    const streamFn = createLlamaCppStreamFn({});
+    await collectEvents(
+      await streamFn(model, { messages: [{ role: "user", content: "one", timestamp: 1 }] }),
+    );
+    await collectEvents(
+      await streamFn(model, { messages: [{ role: "user", content: "two", timestamp: 2 }] }),
+    );
+
+    expect(mocks.context.getSequence).toHaveBeenCalledTimes(1);
+    expect(mocks.llama.loadModel).toHaveBeenCalledTimes(1);
+  });
+
+  it("expands home-relative local model paths before resolving the file", async () => {
+    const stream = await createLlamaCppStreamFn({})(
+      { ...model, params: { modelPath: "~/Models/test.gguf" } },
+      { messages: [{ role: "user", content: "Hi", timestamp: 1 }] },
+    );
+
+    await collectEvents(stream);
+
+    expect(mocks.resolveModelFile).toHaveBeenCalledWith(
+      path.join(os.homedir(), "Models", "test.gguf"),
+      expect.objectContaining({ download: false }),
+    );
+  });
+
+  it("preserves streamed text in a terminal error message", async () => {
+    mocks.generateResponse.mockImplementationOnce(async (_history, options) => {
+      options.onTextChunk("Partial");
+      throw new Error("generation failed");
+    });
+    const stream = await createLlamaCppStreamFn({})(model, {
+      messages: [{ role: "user", content: "Hi", timestamp: 1 }],
+    });
+
+    await expect(stream.result()).resolves.toMatchObject({
+      stopReason: "error",
+      content: [{ type: "text", text: "Partial" }],
+      errorMessage: expect.stringContaining("generation failed"),
+    });
+  });
+
+  it("returns an aborted stream error when the signal is cancelled", async () => {
+    const controller = new AbortController();
+    controller.abort();
+    const stream = await createLlamaCppStreamFn({})(
+      model,
+      { messages: [{ role: "user", content: "stop", timestamp: 1 }] },
+      { signal: controller.signal },
+    );
+
+    await expect(stream.result()).resolves.toMatchObject({
+      stopReason: "aborted",
+      errorMessage: "Request was aborted",
+    });
+    expect(mocks.generateResponse).not.toHaveBeenCalled();
+  });
+
+  it("maps a native abort result to an aborted stream error", async () => {
+    mocks.generateResponse.mockResolvedValueOnce({
+      response: "",
+      functionCalls: undefined,
+      metadata: { stopReason: "abort" },
+    });
+    const stream = await createLlamaCppStreamFn({})(model, {
+      messages: [{ role: "user", content: "stop", timestamp: 1 }],
+    });
+
+    await expect(stream.result()).resolves.toMatchObject({
+      stopReason: "aborted",
+      errorMessage: "Request was aborted",
+    });
+  });
+
+  it("ends an aborted queued request without loading or switching its model", async () => {
+    let resolveFirst: ((value: unknown) => void) | undefined;
+    mocks.generateResponse.mockImplementationOnce(
+      async () =>
+        await new Promise((resolve) => {
+          resolveFirst = resolve;
+        }),
+    );
+    const streamFn = createLlamaCppStreamFn({});
+    const firstStream = await streamFn(model, {
+      messages: [{ role: "user", content: "first", timestamp: 1 }],
+    });
+    await vi.waitFor(() => expect(mocks.generateResponse).toHaveBeenCalledTimes(1));
+    const controller = new AbortController();
+    const queuedStream = await streamFn(
+      { ...model, id: "other.gguf", params: { modelPath: "other.gguf" } },
+      { messages: [{ role: "user", content: "second", timestamp: 2 }] },
+      { signal: controller.signal },
+    );
+
+    controller.abort();
+    await expect(queuedStream.result()).resolves.toMatchObject({ stopReason: "aborted" });
+    expect(mocks.llama.loadModel).toHaveBeenCalledTimes(1);
+
+    resolveFirst?.({
+      response: "",
+      functionCalls: undefined,
+      metadata: { stopReason: "eogToken" },
+    });
+    await firstStream.result();
+  });
+});

extensions/llama-cpp/src/inference-provider.ts

@@ -0,0 +1,418 @@
+import { randomUUID } from "node:crypto";
+import type {
+  ChatHistoryItem,
+  ChatModelFunctions,
+  Llama,
+  LlamaContext,
+  LlamaContextSequence,
+  LlamaModel,
+} from "node-llama-cpp";
+import type { StreamFn } from "openclaw/plugin-sdk/agent-core";
+import type {
+  AssistantMessage,
+  Context,
+  StopReason,
+  ToolCall,
+  Usage,
+} from "openclaw/plugin-sdk/llm";
+import { createAssistantMessageEventStream } from "openclaw/plugin-sdk/llm";
+import type { ModelProviderConfig } from "openclaw/plugin-sdk/provider-model-shared";
+import {
+  DEFAULT_LLAMA_CPP_CONTEXT_SIZE,
+  resolveLlamaCppModelCacheDir,
+  resolveLlamaCppModelSource,
+} from "./defaults.js";
+import {
+  formatLlamaCppSetupError,
+  importNodeLlamaCpp,
+  type NodeLlamaCppModule,
+} from "./node-llama.runtime.js";
+
+type LoadedModel = {
+  key: string;
+  model: LlamaModel;
+  context: LlamaContext;
+  sequence: LlamaContextSequence;
+};
+
+// Process-owned, single-slot cache. A model/context pair lives until another
+// model replaces it or the process exits, bounding resident model memory.
+let loadedModel: LoadedModel | undefined;
+let llamaInstance: Llama | undefined;
+let operationQueue: Promise<void> = Promise.resolve();
+
+function zeroCostUsage(input = 0, output = 0): Usage {
+  return {
+    input,
+    output,
+    cacheRead: 0,
+    cacheWrite: 0,
+    totalTokens: input + output,
+    cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
+  };
+}
+
+function buildMessage(params: {
+  model: Parameters<StreamFn>[0];
+  content: AssistantMessage["content"];
+  stopReason: StopReason;
+  usage?: Usage;
+  errorMessage?: string;
+}): AssistantMessage {
+  return {
+    role: "assistant",
+    content: params.content,
+    api: params.model.api,
+    provider: params.model.provider,
+    model: params.model.id,
+    stopReason: params.stopReason,
+    usage: params.usage ?? zeroCostUsage(),
+    timestamp: Date.now(),
+    ...(params.errorMessage ? { errorMessage: params.errorMessage } : {}),
+  };
+}
+
+function extractText(content: unknown): string {
+  if (typeof content === "string") {
+    return content;
+  }
+  if (!Array.isArray(content)) {
+    return "";
+  }
+  return content
+    .filter(
+      (part): part is { type: "text"; text: string } =>
+        Boolean(part) && typeof part === "object" && part.type === "text",
+    )
+    .map((part) => part.text)
+    .join("");
+}
+
+function normalizeArguments(value: unknown): Record<string, unknown> {
+  return value && typeof value === "object" && !Array.isArray(value)
+    ? (value as Record<string, unknown>)
+    : {};
+}
+
+function mapContextToLlamaChatHistory(context: Context): ChatHistoryItem[] {
+  const history: ChatHistoryItem[] = [];
+  if (context.systemPrompt?.trim()) {
+    history.push({ type: "system", text: context.systemPrompt });
+  }
+  const toolResults = new Map(
+    context.messages
+      .filter((message) => message.role === "toolResult")
+      .map((message) => [message.toolCallId, extractText(message.content)]),
+  );
+  const consumedToolResults = new Set<string>();
+
+  for (const message of context.messages) {
+    if (message.role === "user") {
+      history.push({ type: "user", text: extractText(message.content) });
+      continue;
+    }
+    if (message.role === "assistant") {
+      const response: Extract<ChatHistoryItem, { type: "model" }>["response"] = [];
+      for (const part of message.content) {
+        if (part.type === "text") {
+          if (part.text) {
+            response.push(part.text);
+          }
+          continue;
+        }
+        if (part.type === "thinking") {
+          if (part.thinking) {
+            response.push({
+              type: "segment",
+              segmentType: "thought",
+              text: part.thinking,
+              ended: true,
+            });
+          }
+          continue;
+        }
+        const result = toolResults.get(part.id);
+        if (result !== undefined) {
+          consumedToolResults.add(part.id);
+        }
+        response.push({
+          type: "functionCall",
+          name: part.name,
+          params: part.arguments,
+          result: result ?? "",
+        });
+      }
+      history.push({ type: "model", response });
+      continue;
+    }
+    if (!consumedToolResults.has(message.toolCallId)) {
+      history.push({
+        type: "user",
+        text: `Tool result (${message.toolName}): ${extractText(message.content)}`,
+      });
+    }
+  }
+  return history;
+}
+
+function mapToolsToLlamaFunctions(context: Context): ChatModelFunctions | undefined {
+  if (!context.tools?.length) {
+    return undefined;
+  }
+  return Object.fromEntries(
+    context.tools.map((tool) => [
+      tool.name,
+      {
+        description: tool.description,
+        params: tool.parameters as ChatModelFunctions[string]["params"],
+      },
+    ]),
+  );
+}
+
+function readContextSizeValue(value: unknown): number | "auto" | undefined {
+  if (value === "auto") {
+    return value;
+  }
+  return typeof value === "number" && Number.isFinite(value) && value > 0
+    ? Math.floor(value)
+    : undefined;
+}
+
+function resolveContextSize(
+  model: Parameters<StreamFn>[0],
+  providerConfig?: ModelProviderConfig,
+): number | { max: number } {
+  const configured =
+    readContextSizeValue(model.params?.contextSize) ??
+    readContextSizeValue(providerConfig?.params?.contextSize);
+  if (typeof configured === "number") {
+    return configured;
+  }
+  const modelCap =
+    typeof model.contextTokens === "number" && model.contextTokens > 0
+      ? Math.floor(model.contextTokens)
+      : DEFAULT_LLAMA_CPP_CONTEXT_SIZE;
+  return { max: modelCap };
+}
+
+async function disposeLoadedModel(): Promise<void> {
+  if (!loadedModel) {
+    return;
+  }
+  const previous = loadedModel;
+  loadedModel = undefined;
+  await previous.context.dispose();
+  await previous.model.dispose();
+}
+
+async function getLoadedModel(params: {
+  runtime: NodeLlamaCppModule;
+  model: Parameters<StreamFn>[0];
+  providerConfig?: ModelProviderConfig;
+  signal?: AbortSignal;
+}): Promise<LoadedModel> {
+  const source = resolveLlamaCppModelSource(params.model);
+  const modelPath = await params.runtime.resolveModelFile(source, {
+    directory: resolveLlamaCppModelCacheDir(params.providerConfig),
+    download: false,
+  });
+  const contextSize = resolveContextSize(params.model, params.providerConfig);
+  const key = `${modelPath}\0${JSON.stringify(contextSize)}`;
+  if (loadedModel?.key === key) {
+    return loadedModel;
+  }
+  await disposeLoadedModel();
+  const llama = llamaInstance ?? (await params.runtime.getLlama());
+  llamaInstance = llama;
+  const fitContextSize = typeof contextSize === "number" ? contextSize : contextSize.max;
+  const model = await llama.loadModel({
+    modelPath,
+    loadSignal: params.signal,
+    gpuLayers: { fitContext: { contextSize: fitContextSize } },
+  });
+  let context: LlamaContext | undefined;
+  try {
+    context = await model.createContext({ contextSize, createSignal: params.signal });
+    // Serialized requests reuse this one sequence. Disposing/reallocating it per
+    // turn races node-llama-cpp's asynchronous sequence-id reclamation.
+    const sequence = context.getSequence();
+    loadedModel = { key, model, context, sequence };
+    return loadedModel;
+  } catch (error) {
+    await context?.dispose();
+    await model.dispose();
+    throw error;
+  }
+}
+
+async function serialize(operation: () => Promise<void>): Promise<void> {
+  const current = operationQueue.then(operation, operation);
+  operationQueue = current.catch(() => undefined);
+  await current;
+}
+
+async function clearLlamaCppInferenceCacheForTests(): Promise<void> {
+  await serialize(async () => {
+    await disposeLoadedModel();
+    if (llamaInstance) {
+      await llamaInstance.dispose();
+      llamaInstance = undefined;
+    }
+  });
+}
+
+export function createLlamaCppStreamFn(params: { providerConfig?: ModelProviderConfig }): StreamFn {
+  return (model, context, options) => {
+    const stream = createAssistantMessageEventStream();
+    let streamedText = "";
+    let generationAborted = false;
+    let started = false;
+    let ended = false;
+    const signal = options?.signal;
+    const abortWhileQueued = () => {
+      if (started || ended) {
+        return;
+      }
+      ended = true;
+      stream.push({
+        type: "error",
+        reason: "aborted",
+        error: buildMessage({
+          model,
+          content: [],
+          stopReason: "aborted",
+          errorMessage: "Request was aborted",
+        }),
+      });
+      stream.end();
+    };
+    signal?.addEventListener("abort", abortWhileQueued, { once: true });
+    if (signal?.aborted) {
+      abortWhileQueued();
+    }
+    const run = async () => {
+      if (ended) {
+        return;
+      }
+      started = true;
+      signal?.removeEventListener("abort", abortWhileQueued);
+      try {
+        const runtime = await importNodeLlamaCpp();
+        const loaded = await getLoadedModel({
+          runtime,
+          model,
+          providerConfig: params.providerConfig,
+          signal: options?.signal,
+        });
+        const sequence = loaded.sequence;
+        const chat = new runtime.LlamaChat({
+          contextSequence: sequence,
+          chatWrapper: "auto",
+          autoDisposeSequence: false,
+        });
+        const before = sequence.tokenMeter.getState();
+        let textStarted = false;
+        const partial = () =>
+          buildMessage({
+            model,
+            content: streamedText ? [{ type: "text", text: streamedText }] : [],
+            stopReason: "stop",
+          });
+        const appendTextDelta = (delta: string) => {
+          if (!delta) {
+            return;
+          }
+          if (!textStarted) {
+            textStarted = true;
+            stream.push({ type: "start", partial: partial() });
+            stream.push({ type: "text_start", contentIndex: 0, partial: partial() });
+          }
+          streamedText += delta;
+          stream.push({ type: "text_delta", contentIndex: 0, delta });
+        };
+        try {
+          const result = await chat.generateResponse(mapContextToLlamaChatHistory(context), {
+            functions: mapToolsToLlamaFunctions(context),
+            documentFunctionParams: true,
+            signal: options?.signal,
+            maxTokens: options?.maxTokens ?? model.maxTokens,
+            temperature: options?.temperature,
+            customStopTriggers: options?.stop,
+            onTextChunk: appendTextDelta,
+          });
+          if (result.metadata.stopReason === "abort" || signal?.aborted) {
+            generationAborted = true;
+            throw signal?.reason ?? new Error("Request was aborted");
+          }
+          const usageDelta = sequence.tokenMeter.diff(before);
+          if (!streamedText && result.response) {
+            appendTextDelta(result.response);
+          }
+          const content: AssistantMessage["content"] = streamedText
+            ? [{ type: "text", text: streamedText }]
+            : [];
+          if (textStarted) {
+            stream.push({
+              type: "text_end",
+              contentIndex: 0,
+              content: streamedText,
+              partial: partial(),
+            });
+          }
+          const toolCalls: ToolCall[] = (result.functionCalls ?? []).map((call) => ({
+            type: "toolCall",
+            id: `llama_cpp_call_${randomUUID()}`,
+            name: call.functionName,
+            arguments: normalizeArguments(call.params),
+          }));
+          content.push(...toolCalls);
+          const reason: Extract<StopReason, "stop" | "length" | "toolUse"> =
+            toolCalls.length > 0
+              ? "toolUse"
+              : result.metadata.stopReason === "maxTokens"
+                ? "length"
+                : "stop";
+          const message = buildMessage({
+            model,
+            content,
+            stopReason: reason,
+            usage: zeroCostUsage(usageDelta.usedInputTokens, usageDelta.usedOutputTokens),
+          });
+          stream.push({ type: "done", reason, message });
+        } finally {
+          chat.dispose();
+        }
+      } catch (error) {
+        const aborted = generationAborted || options?.signal?.aborted === true;
+        const reason = aborted ? "aborted" : "error";
+        const errorMessage = aborted ? "Request was aborted" : formatLlamaCppSetupError(error);
+        stream.push({
+          type: "error",
+          reason,
+          error: buildMessage({
+            model,
+            content: streamedText ? [{ type: "text", text: streamedText }] : [],
+            stopReason: reason,
+            errorMessage,
+          }),
+        });
+      } finally {
+        ended = true;
+        stream.end();
+      }
+    };
+    if (!ended) {
+      queueMicrotask(() => void serialize(run));
+    }
+    return stream;
+  };
+}
+
+if (process.env.VITEST || process.env.NODE_ENV === "test") {
+  (globalThis as Record<PropertyKey, unknown>)[Symbol.for("openclaw.llamaCppInferenceTestApi")] = {
+    mapContextToLlamaChatHistory,
+    mapToolsToLlamaFunctions,
+    clearLlamaCppInferenceCacheForTests,
+  };
+}

extensions/llama-cpp/src/node-llama.runtime.ts

@@ -0,0 +1,46 @@
+import { createRequire } from "node:module";
+import { pathToFileURL } from "node:url";
+
+export type NodeLlamaCppModule = typeof import("node-llama-cpp");
+
+function isNodeLlamaCppMissing(error: unknown): boolean {
+  if (!(error instanceof Error)) {
+    return false;
+  }
+  const code = (error as Error & { code?: unknown }).code;
+  return code === "ERR_MODULE_NOT_FOUND" && error.message.includes("node-llama-cpp");
+}
+
+function formatErrorMessage(error: unknown): string {
+  return error instanceof Error ? error.message : String(error);
+}
+
+export function formatLlamaCppSetupError(error: unknown): string {
+  const detail = formatErrorMessage(error);
+  const missing = isNodeLlamaCppMissing(error);
+  return [
+    "Local llama.cpp is unavailable.",
+    missing
+      ? "Reason: node-llama-cpp is missing or failed to install."
+      : detail
+        ? `Reason: ${detail}`
+        : undefined,
+    missing && detail ? `Detail: ${detail}` : null,
+    "To enable local GGUF models:",
+    "1) Install the official provider plugin: openclaw plugins install @openclaw/llama-cpp-provider",
+    "2) Use Node 24 for native installs/updates.",
+    "3) If you use pnpm from source: pnpm approve-builds, then pnpm rebuild node-llama-cpp.",
+  ]
+    .filter(Boolean)
+    .join("\n");
+}
+
+const requireFromPlugin = createRequire(import.meta.url);
+
+export function resolveNodeLlamaCppImportUrl(): string {
+  return pathToFileURL(requireFromPlugin.resolve("node-llama-cpp")).href;
+}
+
+export async function importNodeLlamaCpp(): Promise<NodeLlamaCppModule> {
+  return await import("node-llama-cpp");
+}

extensions/llama-cpp/src/setup.test.ts

@@ -0,0 +1,198 @@
+import fs from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import type {
+  ProviderAppGuidedSetupContext,
+  ProviderAuthContext,
+} from "openclaw/plugin-sdk/plugin-entry";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import {
+  DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE,
+  DEFAULT_LLAMA_CPP_MODEL_REF,
+  LLAMA_CPP_PROVIDER_ID,
+} from "./defaults.js";
+
+const nodeLlamaMocks = vi.hoisted(() => ({
+  download: vi.fn(async () => "/models/default.gguf"),
+  createModelDownloader: vi.fn(),
+  resolveModelFile: vi.fn(),
+}));
+
+vi.mock("node-llama-cpp", () => ({
+  createModelDownloader: nodeLlamaMocks.createModelDownloader,
+  getLlama: vi.fn(),
+  resolveModelFile: nodeLlamaMocks.resolveModelFile,
+  LlamaChat: vi.fn(),
+}));
+
+import { detectLlamaCppSetup, prepareLlamaCppSetup, runLlamaCppSetup } from "./setup.js";
+
+let tempRoot: string;
+let cacheDir: string;
+
+beforeEach(async () => {
+  tempRoot = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "llama-cpp-setup-")));
+  cacheDir = path.join(tempRoot, "models");
+  await fs.mkdir(cacheDir);
+  nodeLlamaMocks.download.mockReset().mockResolvedValue("/models/default.gguf");
+  nodeLlamaMocks.createModelDownloader.mockReset().mockResolvedValue({
+    download: nodeLlamaMocks.download,
+  });
+  nodeLlamaMocks.resolveModelFile.mockReset().mockImplementation(async (_source, options) => {
+    const candidate = path.join(options.directory, DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE);
+    await fs.access(candidate);
+    return candidate;
+  });
+});
+
+afterEach(async () => {
+  await fs.rm(tempRoot, { recursive: true, force: true });
+});
+
+function configWithCache(): ProviderAppGuidedSetupContext["config"] {
+  return {
+    models: {
+      providers: {
+        [LLAMA_CPP_PROVIDER_ID]: {
+          baseUrl: "local://llama-cpp",
+          api: "openai-completions" as const,
+          params: { modelCacheDir: cacheDir },
+          models: [],
+        },
+      },
+    },
+  };
+}
+
+function createAuthContext(confirm: boolean): ProviderAuthContext {
+  return {
+    config: configWithCache(),
+    prompter: {
+      confirm: vi.fn(async () => confirm),
+      note: vi.fn(async () => {}),
+      progress: vi.fn(() => ({ update: vi.fn(), stop: vi.fn() })),
+    },
+    runtime: {},
+  } as unknown as ProviderAuthContext;
+}
+
+describe("llama.cpp setup", () => {
+  it("returns null when the configured model is not cached", async () => {
+    await expect(detectLlamaCppSetup({ config: configWithCache(), env: {} })).resolves.toBeNull();
+  });
+
+  it("detects the cached default model without downloading", async () => {
+    await fs.writeFile(path.join(cacheDir, DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE), "fixture");
+
+    await expect(detectLlamaCppSetup({ config: configWithCache(), env: {} })).resolves.toEqual({
+      modelRef: DEFAULT_LLAMA_CPP_MODEL_REF,
+      detail: "qwen3-4b-instruct-2507-q4_k_m (downloaded)",
+    });
+    expect(nodeLlamaMocks.createModelDownloader).not.toHaveBeenCalled();
+    expect(nodeLlamaMocks.resolveModelFile).toHaveBeenCalledWith(
+      expect.stringMatching(/^hf:/),
+      expect.objectContaining({ directory: cacheDir, download: false, cli: false }),
+    );
+  });
+
+  it("uses node-llama-cpp cache resolution for a configured HF branch", async () => {
+    const cachedPath = path.join(cacheDir, "hf_org_repo_release_model.gguf");
+    await fs.writeFile(cachedPath, "fixture");
+    nodeLlamaMocks.resolveModelFile.mockResolvedValueOnce(cachedPath);
+    const config = configWithCache();
+    const provider = config.models?.providers?.[LLAMA_CPP_PROVIDER_ID];
+    if (!provider) {
+      throw new Error("expected llama.cpp provider config");
+    }
+    provider.models.push({
+      id: "custom",
+      name: "Custom",
+      reasoning: false,
+      input: ["text"],
+      cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+      contextWindow: 8192,
+      maxTokens: 2048,
+      params: { modelPath: "hf:org/repo/model.gguf#release" },
+    });
+
+    await expect(detectLlamaCppSetup({ config, env: {} })).resolves.toEqual({
+      modelRef: "llama-cpp/custom",
+      detail: "custom (downloaded)",
+    });
+    expect(nodeLlamaMocks.resolveModelFile).toHaveBeenCalledWith(
+      "hf:org/repo/model.gguf#release",
+      expect.objectContaining({ download: false, cli: false }),
+    );
+  });
+
+  it("prepares config only for a currently cached detected model", async () => {
+    await expect(
+      prepareLlamaCppSetup({
+        config: configWithCache(),
+        env: {},
+        modelRef: DEFAULT_LLAMA_CPP_MODEL_REF,
+      }),
+    ).resolves.toBeNull();
+
+    await fs.writeFile(path.join(cacheDir, DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE), "fixture");
+    await expect(
+      prepareLlamaCppSetup({
+        config: configWithCache(),
+        env: {},
+        modelRef: DEFAULT_LLAMA_CPP_MODEL_REF,
+      }),
+    ).resolves.toMatchObject({
+      profiles: [],
+      defaultModel: DEFAULT_LLAMA_CPP_MODEL_REF,
+      configPatch: {
+        models: {
+          mode: "merge",
+          providers: {
+            [LLAMA_CPP_PROVIDER_ID]: {
+              baseUrl: "local://llama-cpp",
+              models: [expect.objectContaining({ id: "qwen3-4b-instruct-2507-q4_k_m" })],
+            },
+          },
+        },
+      },
+    });
+  });
+
+  it("exits without config or download when consent is declined", async () => {
+    const ctx = createAuthContext(false);
+
+    await expect(runLlamaCppSetup(ctx)).resolves.toEqual({ profiles: [] });
+
+    expect(ctx.prompter.confirm).toHaveBeenCalledWith(
+      expect.objectContaining({ message: expect.stringContaining("about 2.5 GB") }),
+    );
+    expect(nodeLlamaMocks.createModelDownloader).not.toHaveBeenCalled();
+  });
+
+  it("downloads after consent and returns the provider patch", async () => {
+    const ctx = createAuthContext(true);
+
+    await expect(runLlamaCppSetup(ctx)).resolves.toMatchObject({
+      profiles: [],
+      defaultModel: DEFAULT_LLAMA_CPP_MODEL_REF,
+      configPatch: {
+        models: {
+          providers: {
+            [LLAMA_CPP_PROVIDER_ID]: expect.objectContaining({
+              baseUrl: "local://llama-cpp",
+            }),
+          },
+        },
+      },
+    });
+
+    expect(nodeLlamaMocks.createModelDownloader).toHaveBeenCalledWith(
+      expect.objectContaining({
+        dirPath: cacheDir,
+        fileName: DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE,
+        showCliProgress: false,
+      }),
+    );
+    expect(nodeLlamaMocks.download).toHaveBeenCalledTimes(1);
+  });
+});

extensions/llama-cpp/src/setup.ts

@@ -0,0 +1,156 @@
+import fs from "node:fs/promises";
+import type {
+  ProviderAppGuidedSetupContext,
+  ProviderAuthContext,
+  ProviderAuthResult,
+} from "openclaw/plugin-sdk/plugin-entry";
+import type {
+  ModelDefinitionConfig,
+  ModelProviderConfig,
+} from "openclaw/plugin-sdk/provider-model-shared";
+import {
+  DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE,
+  DEFAULT_LLAMA_CPP_MODEL_ID,
+  DEFAULT_LLAMA_CPP_MODEL_REF,
+  DEFAULT_LLAMA_CPP_MODEL_SIZE_BYTES,
+  DEFAULT_LLAMA_CPP_MODEL_URI,
+  LLAMA_CPP_PROVIDER_ID,
+  buildLlamaCppProviderConfig,
+  resolveCachedLlamaCppModelPath,
+  resolveLlamaCppModelCacheDir,
+  resolveLlamaCppModelSource,
+} from "./defaults.js";
+import {
+  formatLlamaCppSetupError,
+  importNodeLlamaCpp,
+  type NodeLlamaCppModule,
+} from "./node-llama.runtime.js";
+
+function readPrimaryModel(config: ProviderAppGuidedSetupContext["config"]): string | undefined {
+  const model = config.agents?.defaults?.model;
+  return typeof model === "string" ? model : model?.primary;
+}
+
+function configuredCandidates(
+  config: ProviderAppGuidedSetupContext["config"],
+): Array<{ model: ModelDefinitionConfig; provider: ModelProviderConfig }> {
+  const existing = config.models?.providers?.[LLAMA_CPP_PROVIDER_ID];
+  const provider = buildLlamaCppProviderConfig(existing);
+  const primary = readPrimaryModel(config);
+  const primaryId = primary?.startsWith(`${LLAMA_CPP_PROVIDER_ID}/`)
+    ? primary.slice(LLAMA_CPP_PROVIDER_ID.length + 1)
+    : undefined;
+  return provider.models
+    .map((model) => ({ model, provider }))
+    .toSorted((a, b) => Number(b.model.id === primaryId) - Number(a.model.id === primaryId));
+}
+
+async function isFile(filePath: string): Promise<boolean> {
+  try {
+    return (await fs.stat(filePath)).isFile();
+  } catch {
+    return false;
+  }
+}
+
+export async function detectLlamaCppSetup(ctx: ProviderAppGuidedSetupContext) {
+  let runtime: NodeLlamaCppModule;
+  try {
+    runtime = await importNodeLlamaCpp();
+  } catch {
+    return null;
+  }
+  for (const candidate of configuredCandidates(ctx.config)) {
+    try {
+      const cachedPath = await runtime.resolveModelFile(
+        resolveLlamaCppModelSource(candidate.model),
+        {
+          directory: resolveLlamaCppModelCacheDir(candidate.provider),
+          download: false,
+          cli: false,
+        },
+      );
+      if (!(await isFile(cachedPath))) {
+        continue;
+      }
+      return {
+        modelRef: `${LLAMA_CPP_PROVIDER_ID}/${candidate.model.id}`,
+        detail: `${candidate.model.id} (downloaded)`,
+      };
+    } catch {
+      // Discovery is read-only: a missing model or native module is not a setup error.
+    }
+  }
+  return null;
+}
+
+function buildSetupResult(
+  config: ProviderAppGuidedSetupContext["config"],
+  defaultModel = DEFAULT_LLAMA_CPP_MODEL_REF,
+): ProviderAuthResult {
+  return {
+    profiles: [],
+    defaultModel,
+    configPatch: {
+      models: {
+        mode: config.models?.mode ?? "merge",
+        providers: {
+          [LLAMA_CPP_PROVIDER_ID]: buildLlamaCppProviderConfig(
+            config.models?.providers?.[LLAMA_CPP_PROVIDER_ID],
+          ),
+        },
+      },
+    },
+  };
+}
+
+export async function prepareLlamaCppSetup(
+  ctx: ProviderAppGuidedSetupContext & { modelRef: string },
+): Promise<ProviderAuthResult | null> {
+  const detected = await detectLlamaCppSetup(ctx);
+  return detected?.modelRef === ctx.modelRef ? buildSetupResult(ctx.config, ctx.modelRef) : null;
+}
+
+export async function runLlamaCppSetup(ctx: ProviderAuthContext): Promise<ProviderAuthResult> {
+  const existing = ctx.config.models?.providers?.[LLAMA_CPP_PROVIDER_ID];
+  const cacheDir = resolveLlamaCppModelCacheDir(existing);
+  const cachedPath = resolveCachedLlamaCppModelPath({
+    model: {
+      id: DEFAULT_LLAMA_CPP_MODEL_ID,
+      params: { modelPath: DEFAULT_LLAMA_CPP_MODEL_URI },
+    },
+    provider: existing,
+  });
+  if (!cachedPath || !(await isFile(cachedPath))) {
+    const consent = await ctx.prompter.confirm({
+      message:
+        "Download Qwen3 4B Instruct 2507 Q4_K_M (about 2.5 GB) for local llama.cpp inference?",
+      initialValue: false,
+    });
+    if (!consent) {
+      await ctx.prompter.note("Local model download skipped.", "Setup skipped");
+      return { profiles: [] };
+    }
+    const progress = ctx.prompter.progress("Preparing Qwen3 4B model download…");
+    try {
+      const runtime = await importNodeLlamaCpp();
+      const downloader = await runtime.createModelDownloader({
+        modelUri: DEFAULT_LLAMA_CPP_MODEL_URI,
+        dirPath: cacheDir,
+        fileName: DEFAULT_LLAMA_CPP_MODEL_CACHE_FILE,
+        showCliProgress: false,
+        onProgress: ({ downloadedSize, totalSize }) => {
+          const expectedSize = totalSize || DEFAULT_LLAMA_CPP_MODEL_SIZE_BYTES;
+          const percent = Math.min(100, Math.floor((downloadedSize / expectedSize) * 100));
+          progress.update(`Downloading Qwen3 4B model… ${percent}%`);
+        },
+      });
+      await downloader.download({ signal: ctx.signal });
+      progress.stop("Qwen3 4B model downloaded");
+    } catch (error) {
+      progress.stop("Model download failed");
+      throw new Error(formatLlamaCppSetupError(error), { cause: error });
+    }
+  }
+  return buildSetupResult(ctx.config);
+}