openclaw/docs/plugins/llama-cpp.md

summary, read_when, title, sidebarTitle

summary	read_when	title	sidebarTitle
Run GGUF chat and memory embeddings with managed or external llama.cpp servers	You want OpenClaw to install and manage a local llama.cpp server You want OpenClaw to connect to an existing llama-server You want memory search embeddings from a local GGUF model You are configuring memory.search.provider = "local"	llama.cpp Provider	llama.cpp Provider

The llama-cpp plugin supports two llama.cpp server ownership modes. The llama-cpp provider installs and manages a loopback-only llama-server for chat and embeddings. The llama-server provider connects to an existing server that you manage outside OpenClaw.

Install the official plugin before using either mode or local memory embeddings:

openclaw plugins install @openclaw/llama-cpp-provider

Choose a server

Interactive onboarding shows two choices in the Local llama.cpp group:

Choice	Model reference	Process owner
Managed local server	llama-cpp/<model>	OpenClaw
Existing llama-server	llama-server/<model>	User

Both providers can be configured at the same time. The external provider supports passive model discovery, router status, chat-template capabilities, and optional authentication. See llama-server.

Managed server setup

Choose Managed local server during interactive onboarding or configuration. OpenClaw then:

1. Selects the verified llama-server build for the Gateway platform.
2. Verifies the archive SHA-256 and the extracted server version.
3. Downloads and verifies the default chat and embedding models after consent.
4. Writes a durable OpenAI-compatible provider with a loopback baseUrl and localService process definition.
5. Live-tests the candidate before saving it.

The default chat model remains:

hf:unsloth/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf

Gemma 4 E4B IT Q4_K_M is about 5.0 GB. OpenClaw offers that download only on machines with at least 16 GiB of RAM. The default context cap is 65,536 tokens, which the full agent system prompt requires. The bundled EmbeddingGemma model is about 0.3 GB.

Discovery is read-only. It reports a prepared choice only when the managed binary, server preset, and selected model already exist; it never installs or downloads during discovery.

How requests run

The provider uses OpenClaw's normal OpenAI-compatible chat, image, streaming, and tool transport. llama-server applies the GGUF chat template; OpenClaw executes tool calls and returns their results to the model. The existing llamacpp-gbnf tool-schema compatibility profile remains enabled.

One managed router owns separate presets for chat and embeddings. This lets memory.search.provider: "local" use a dedicated embedding GGUF through /v1/embeddings without creating a second OpenClaw process supervisor.

Use another GGUF model

Add the model to models.providers.llama-cpp.models, set params.modelPath, make it the selected llama-cpp/<model-id>, then run interactive llama.cpp setup again. modelPath accepts:

• an absolute or ~/ local GGUF path;
• a cache-relative GGUF filename;
• a full hf: file URI, including #branch when needed;
• an HTTPS GGUF URL that publishes a SHA-256 response digest.

Example model entry:

{
  id: "my-local-model",
  name: "My local GGUF",
  reasoning: false,
  input: ["text"],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 65536,
  maxTokens: 2048,
  params: {
    modelPath: "~/Models/my-model.Q4_K_M.gguf",
    contextSize: 65536,
  },
  compat: { supportsTools: true },
}

The default managed cache is ~/.openclaw/models/llama.cpp. Existing modelCacheDir settings still win, and setup recognizes the former ~/.node-llama-cpp/models default cache so upgrades do not redownload a model that is already present.

Local memory embeddings

Set the memory provider to local:

{
  memory: {
    search: {
      provider: "local",
      local: {
        modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf",
      },
    },
  },
}

The plugin preserves the historical local provider/model cache identity, so the transport migration does not require a SQLite schema change or automatic memory reindex. A custom embedding modelPath remains its literal index identity. Run openclaw memory status --index if you intentionally change it.

Diagnostics

Run:

openclaw memory status --deep
openclaw doctor

After the managed embedding server has handled a request, deep status reports facts observed from /health, /models, /props, and /metrics: server build, model id and path, endpoint state, and configured capabilities. Vision is reported only when /props confirms it. Draft and multimodal projector support are not currently configured by this plugin and are never inferred from a model name.

Local-service startup and exit logs include bounded, redacted stderr tails. See Logging and Local model services.

Platform requirements

• macOS arm64 uses the official Metal build. macOS x64 uses the CPU build.
• Linux x64 needs glibc 2.34 or newer; Linux arm64 needs glibc 2.38 or newer. Install the OpenMP runtime (libgomp1 on Debian/Ubuntu or libgomp on Fedora) if the version probe reports libgomp.so.1 missing.
• Windows x64 and arm64 use the CPU build and require the Microsoft Visual C++ 2015-2022 Redistributable.
• Alpine/musl and platforms without a pinned official build fail with an actionable manual-server path rather than silently skipping setup.

OpenClaw intentionally does not auto-select CUDA, ROCm, SYCL, OpenVINO, or Vulkan archives. Those builds add driver and companion-runtime contracts that cannot be verified safely from onboarding alone.

Troubleshooting

Binary missing or wrong version: run interactive llama.cpp setup again. It reinstalls the pinned build and rewrites the absolute localService.command.

Model missing: configure a local GGUF path or rerun setup and approve the verified default download.

Server starts but the model fails to load: inspect openclaw logs --follow and openclaw memory status --deep. The managed service error includes the bounded server stderr tail.

Only keyword memory matches: run openclaw memory status --deep, repair the reported endpoint/model issue, then run openclaw memory index --force only if status reports an index identity mismatch.