--- summary: "Run GGUF chat with managed or existing llama.cpp servers and managed local embeddings" read_when: - You want OpenClaw to install and manage a local llama.cpp server - You want OpenClaw to connect to an existing llama-server - You want memory search embeddings from a local GGUF model - You are configuring memory.search.provider = "local" title: "llama.cpp Provider" sidebarTitle: "llama.cpp Provider" --- The `llama-cpp` plugin provides one `llama-cpp` model provider. OpenClaw can manage a local `llama-server` or connect to one that you operate. Both choices use `llama-cpp/` references and the OpenAI-compatible transport. ```bash openclaw plugins install @openclaw/llama-cpp-provider openclaw onboard ``` ## Choose server ownership | Setup choice | Process owner | Local embeddings | | --------------------- | ----------------------------- | ---------------- | | Managed local server | OpenClaw | Yes | | Existing llama-server | You or an external supervisor | No | `models.providers.llama-cpp.localService` is the ownership discriminator. If it exists, OpenClaw manages the process. Without it, `baseUrl` identifies an existing endpoint. Switching choices rewrites ownership-specific state on the same provider; it never creates another provider namespace. ## Managed local server Choose **Managed local server** when OpenClaw should install, start, and stop the server. After consent, setup verifies a pinned llama.cpp build, downloads verified chat and embedding models, writes the loopback endpoint and `localService` definition, and probes the result before saving it. The default chat model is Gemma 4 E4B IT Q4_K_M (about 5.0 GB) with a 65,536 token context cap. OpenClaw offers it only on machines with at least 16 GiB of RAM. The managed EmbeddingGemma model is about 0.3 GB. Setup discovery remains read-only and never installs or downloads anything. ### Use another managed GGUF Add a model under `models.providers.llama-cpp.models`, select its `llama-cpp/` reference, and run managed setup again: ```json5 { id: "my-local-model", name: "My local GGUF", reasoning: false, input: ["text"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 65536, maxTokens: 2048, params: { modelPath: "~/Models/my-model.Q4_K_M.gguf", contextSize: 65536, }, compat: { supportsTools: true }, } ``` `modelPath` accepts local paths, cache-relative filenames, full `hf:` file URIs, and HTTPS GGUF URLs that publish a SHA-256 response digest. The default cache is `~/.openclaw/models/llama.cpp`; a configured `modelCacheDir` remains authoritative for managed setup. ## Existing llama-server Choose **Existing llama-server** when another terminal, container, service manager, or machine owns the process. Give the model a stable alias: ```bash llama-server \ --model /path/to/model.gguf \ --alias my-model \ --host 127.0.0.1 \ --port 8080 ``` Run `openclaw onboard`, choose **Existing llama-server**, and enter the endpoint. Enable API-key authentication only when the server or proxy requires it. ```bash openclaw models list --provider llama-cpp openclaw models set llama-cpp/my-model ``` OpenClaw reads `/health`, `/models` (falling back to `/v1/models`), and `/props`. Router property probes use `autoload=false`; discovery never loads, wakes, unloads, downloads, or reloads models. Explicit configured model rows remain authoritative over discovered rows with the same ID. ### Authentication and endpoint replacement Existing endpoints support no auth, API keys, SecretRefs, auth profiles, and explicit authorization headers. An explicit `Authorization` header wins over ambient API-key discovery. Endpoint URLs containing a username or password are rejected. ```bash export LLAMA_SERVER_API_KEY="" openclaw onboard ``` When the endpoint changes, setup does not send the old endpoint's environment, profile, configured key, or header credentials to the replacement. Switching from managed mode also removes `localService`, managed model/cache parameters, and the managed request timeout before discovery. For non-interactive setup: ```bash openclaw onboard \ --non-interactive \ --accept-risk \ --auth-choice llama-cpp-existing-server \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id my-model ``` Use `--llama-server-api-key ` when a replacement endpoint requires a new credential. `LLAMA_SERVER_API_KEY` remains available for initial setup and unchanged endpoints. ### Manual configuration Guided setup is recommended because it verifies discovery. The minimal manual shape is: ```json5 { models: { mode: "merge", providers: { "llama-cpp": { baseUrl: "http://127.0.0.1:8080/v1", api: "openai-completions", request: { allowPrivateNetwork: true }, models: [], }, }, }, } ``` Custom provider IDs may also point at llama-server through the generic OpenAI-compatible path. They remain custom providers and should declare the `llamacpp` tool-schema profile explicitly; see [custom provider capability declarations](/gateway/config-tools#custom-provider-capability-declarations). ## Requests and local embeddings Both ownership choices use OpenClaw's normal chat, image, streaming, and tool transport. The llama.cpp compatibility family cleans unsupported tool-schema constraints, maps thinking-off requests to the Qwen chat-template flag, and adapts JSON Schema requests for older llama-server builds. Local memory embeddings require managed mode: ```json5 { memory: { search: { provider: "local", local: { modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf", }, }, }, } ``` The plugin preserves the historical `local` embedding provider and index identity. Run `openclaw memory status --index` after intentionally changing the embedding model. ## Troubleshooting - Managed setup: run `openclaw doctor` and `openclaw memory status --deep`. - Existing server: inspect `/health`, `/models`, and `/props`; HTTP 503 means the model is still loading. - Missing tools: verify both tool capability flags in `/props` and use a tool-capable Jinja chat template. - Managed Linux builds require glibc 2.34 on x64 or 2.38 on arm64. Windows builds require the Microsoft Visual C++ 2015-2022 Redistributable. - Platforms without a verified managed build should use an existing server. OpenClaw does not auto-select CUDA, ROCm, SYCL, OpenVINO, or Vulkan archives. ## Related - [Local model services](/gateway/local-model-services) - [Model providers](/concepts/model-providers) - [LM Studio](/providers/lmstudio)