ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-20 17:41:33 -06:00
Code Issues Packages Projects Releases Wiki Activity
Files
fix/github-hovercard-interactive
openclaw/docs/nodes/media-understanding.md
T
Ayaan Zaidi b8ddddd702 feat(media): let agents inspect unsupported local documents (#122408) 
Let eligible embedded host runs inspect root-approved unsupported documents after final sandbox, filesystem, provider, owner, and tool-policy gates. Generic ACP, sandboxed, URL-only, and restricted-tool paths retain the plain marker.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
2026-08-12 11:42:24 +05:30

434 lines
23 KiB
Markdown
Raw Permalink Blame History

---
summary: "Inbound image/audio/video understanding (optional) with provider + CLI fallbacks"
read_when:
- Designing or refactoring media understanding
- Tuning inbound audio/video/image preprocessing
title: "Media understanding"
sidebarTitle: "Media understanding"
---
OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged.
Vendor plugins register capability metadata (which provider supports which media type, default model, priority). OpenClaw core owns the shared `tools.media` config, fallback order, and reply-pipeline integration.
## How it works
<Steps>
<Step title="Collect attachments">
Collect ordered inbound media facts (`path`, `url`, `contentType`, and `kind`).
</Step>
<Step title="Select per capability">
For each enabled capability (image/audio/video), select attachments per the `attachments` policy (default: first attachment only).
</Step>
<Step title="Choose a model">
Pick the first eligible model entry (size + capability + auth available).
</Step>
<Step title="Fall back on failure">
If a model errors, times out, or the media exceeds `maxBytes`, try the next entry.
</Step>
<Step title="Apply on success">
`Body` becomes an `[Image]`, `[Audio]`, or `[Video]` block. Audio also sets `{{Transcript}}`; command parsing uses caption text when present, otherwise the transcript. Captions are preserved as `User text:` inside the block.
</Step>
</Steps>
## Config
`tools.media` holds one capability-tagged model list plus small per-capability controls:
```json5
{
tools: {
media: {
concurrency: 2, // max concurrent capability runs (default)
models: [
{ provider: "openai", model: "gpt-4o-mini-transcribe", capabilities: ["audio"] },
{ provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "video"] },
],
image: { preferredModel: "google/gemini-3-flash-preview" },
audio: { enabled: true },
video: { enabled: true },
},
},
}
```
Per-capability (`image`/`audio`/`video`) keys:
| Key | Type | Default | Notes |
| ---------------- | --------- | -------------------------------------- | -------------------------------------------------------------------- |
| `enabled` | `boolean` | auto (`false` disables) | Set `false` to turn off auto-detect for this capability |
| `preferredModel` | `string` | first compatible entry | Prefer `provider/model`, model id, `provider:<id>`, or `cli:command` |
| `prompt` | `string` | capability default | Default prompt when an entry does not override it |
| `maxChars` | `number` | `500` image/video, unset audio | Default output limit |
| `maxBytes` | `number` | 10MB image, 20MB audio, 50MB video | Default input limit |
| `timeoutSeconds` | `number` | `60` image/audio, `120` video | Default request timeout |
| `language` | `string` | unset | Audio transcription hint |
| `scope` | object | unset | Gate by channel/chat type/source key |
| `attachments` | object | `{ mode: "first", maxAttachments: 1 }` | Select which matching attachments are processed |
| `echoTranscript` | `boolean` | `false` | Audio only: echo the transcript before agent processing |
| `echoFormat` | `string` | `'📝 "{transcript}"'` | Audio only: format for the echoed transcript |
Prompts, limits, language hints, request overrides, and provider options can be set as capability defaults or overridden on individual `tools.media.models[]` entries. Capability defaults also cover auto-detected providers when no explicit model is configured.
### Model entries
Each `models[]` entry is a **provider** entry (default) or a **CLI** entry:
<Tabs>
<Tab title="Provider entry">
```json5
{
type: "provider", // default if omitted
provider: "openai",
model: "gpt-5.6-sol",
prompt: "Describe the image in <= 500 chars.",
maxChars: 500,
maxBytes: 10485760,
timeoutSeconds: 60,
capabilities: ["image"],
profile: "vision-profile",
preferredProfile: "vision-fallback",
}
```
</Tab>
<Tab title="CLI entry">
```json5
{
type: "cli",
command: "gemini",
args: [
"-m",
"gemini-3-flash",
"--allowed-tools",
"read_file",
"Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
],
maxChars: 500,
maxBytes: 52428800,
timeoutSeconds: 120,
capabilities: ["video", "image"],
}
```
CLI templates can also use `{{AttachmentUrl}}`, `{{AttachmentContentType}}`, `{{AttachmentDir}}`, `{{AttachmentIndex}}`, `{{OutputDir}}` (scratch dir created for this run), and `{{OutputBase}}` (scratch file base path, no extension). The older `{{MediaPath}}`, `{{MediaUrl}}`, `{{MediaType}}`, and `{{MediaDir}}` names remain deprecated compatibility aliases.
</Tab>
</Tabs>
### Provider credentials
Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then `models.providers.<providerId>.apiKey`. `tools.media.models[]` entries do not accept an inline `apiKey` field.
```json5
{
models: {
providers: {
openai: { apiKey: "<OPENAI_API_KEY>" },
moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
},
},
}
```
See [Tools and custom providers](/gateway/config-tools) for profiles, env vars, and custom base URLs.
## Rules and behavior
- Media exceeding `maxBytes` skips that model and tries the next one.
- Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
- If the active primary image model already supports vision natively, OpenClaw skips the `[Image]` summary block and passes the original image into the model directly. MiniMax is an exception: `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` always route image understanding through the plugin-owned `MiniMax-VL-01` media provider, even if legacy MiniMax M2.x chat metadata claims image input (only `MiniMax-M3` and later are treated as natively vision-capable).
- If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded `media://inbound/*` refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
- Explicit `openclaw infer image describe --file <path> --model <provider/model>` (alias: `openclaw capability image describe`) runs that image-capable provider/model directly, including Ollama refs such as `ollama/qwen2.5vl:7b` when a matching image-capable model is configured under `models.providers.ollama.models[]`.
- If `<capability>.enabled` is not `false` but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.
### Auto-detect (default)
When `tools.media.<capability>.enabled` is not `false` and no models are configured, OpenClaw tries these in order and stops at the first working option:
<Steps>
<Step title="Configured image model (image only)">
`agents.defaults.imageModel` primary/fallback refs, unless the active reply model already supports vision natively. Prefer `provider/model` refs; bare refs are qualified from configured image-capable provider model entries only when the match is unique.
</Step>
<Step title="Active reply model">
The active reply model, when its provider supports the capability.
</Step>
<Step title="Provider auth (audio only, before local CLIs)">
Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI &rarr; xAI &rarr; Deepgram &rarr; OpenRouter &rarr; Google/SenseAudio &rarr; Deepinfra/ElevenLabs &rarr; Mistral.
</Step>
<Step title="Local CLIs (audio only)">
Ready local binaries become an ordered fallback list:
- `whisper-cli` first only after an earlier model invocation in the current process observed Metal or CUDA
- CPU-default `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`)
- `whisper-cli` when acceleration is merely build-capable or unobserved
- `parakeet-mlx` on Apple Silicon (MLX-capable, device use unobserved)
- `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically)
Backend capability inspection is cached and does not load a model. Build capability, requested backend flags, and backend observed from a real invocation remain separate. Auto-detected whisper.cpp leaves model-run logs enabled so the upstream selected-backend line can be recorded. Explicit CLI entries keep their configured order, backend flags, and output flags.
</Step>
<Step title="Provider auth (image/video)">
Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.
Bundled provider priority order (ties break alphabetically by provider id):
- Image: Anthropic/OpenAI &rarr; Google &rarr; MiniMax &rarr; Deepinfra &rarr; MiniMax Portal &rarr; Z.AI
- Video: Google &rarr; Qwen &rarr; Moonshot
</Step>
</Steps>
To disable auto-detection for a capability:
```json5
{
tools: {
media: {
audio: {
enabled: false,
},
},
},
}
```
<Note>
Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model entry with a full command path.
</Note>
### Proxy support (audio/video provider calls)
Provider-based **audio** and **video** understanding honors standard outbound proxy environment variables, including `NO_PROXY`/`no_proxy` bypass rules: `HTTPS_PROXY`, `HTTP_PROXY`, `ALL_PROXY`, `https_proxy`, `http_proxy`, `all_proxy`. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.
## Capabilities
Set `capabilities` on a `models[]` entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:
| Provider | Capabilities |
| ------------------------------------------------------------------------ | --------------------- |
| `openai`, `anthropic`, `minimax` | image |
| `minimax-portal` | image |
| `moonshot` | image + video |
| `openrouter` | image + audio |
| `google` (Gemini API) | image + audio + video |
| `qwen` | image + video |
| `deepinfra` | image + audio |
| `mistral` | audio |
| `zai` | image |
| `groq`, `xai`, `deepgram`, `senseaudio` | audio |
| Any `models.providers.<id>.models[]` catalog with an image-capable model | image |
For CLI entries, set `capabilities` explicitly to avoid surprising matches; if omitted, the entry is eligible for every capability list it appears in.
## Provider support matrix
| Capability | Providers | Notes |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | Anthropic, Codex app-server, Deepinfra, Google, MiniMax, MiniMax Portal, Moonshot, OpenAI, OpenAI Codex OAuth, OpenRouter, Qwen, Z.AI, config providers | Vendor plugins register image support; `openai/*` can use API-key or Codex OAuth routing; `codex/*` uses a bounded Codex app-server turn; image-capable config providers auto-register. |
| Audio | Deepgram, Deepinfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, xAI | Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral). |
| Video | Google, Moonshot, Qwen | Provider video understanding via vendor plugins; Qwen video understanding uses the standard DashScope endpoints. |
<Note>
**MiniMax note**: `minimax`, `minimax-cn`, `minimax-portal`, and `minimax-portal-cn` image understanding always comes from the plugin-owned `MiniMax-VL-01` media provider, even if legacy MiniMax M2.x chat metadata claims image input.
</Note>
## Model selection guidance
- Prefer the strongest current-generation model for each media capability when quality and safety matter.
- For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
- Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
- CLI fallbacks (`whisper-cli`, `whisper`, `gemini`) help when provider APIs are unavailable.
- Known file-output modes are authoritative: an empty or missing inferred transcript file produces no transcript instead of falling back to CLI progress output.
- `parakeet-mlx`: use `--output-format txt` (or `all`) with `--output-dir` and the default `{filename}` output template. The upstream `PARAKEET_OUTPUT_FORMAT` and `PARAKEET_OUTPUT_TEMPLATE` environment variables are also honored. OpenClaw reads `<output-dir>/<media-basename>.txt`; the default `srt` format, other formats, and custom output templates continue to use stdout.
## Attachment policy
Per-capability `attachments` controls which attachments are processed:
<ParamField path="mode" type='"first" | "all"' default="first">
Process only the first selected attachment, or all of them.
</ParamField>
<ParamField path="maxAttachments" type="number" default="1">
Cap the number processed.
</ParamField>
<ParamField path="prefer" type='"first" | "last" | "path" | "url"'>
Selection preference among candidate attachments.
</ParamField>
When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
### File-attachment extraction
- Every inbound document attachment ends in a model-visible file block. Attachments routed to image, audio, or video understanding are outside this contract; those stages own their outcomes.
- Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like `<<<EXTERNAL_UNTRUSTED_CONTENT id="...">>>` / `<<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>>` plus a `Source: External` metadata line.
- This path intentionally omits the long `SECURITY NOTICE:` banner to keep the media prompt short; the boundary markers and metadata still apply.
- Unsupported files saved on local disk get self-serve guidance only when the reply runtime proves it can read host-local paths (currently non-sandboxed embedded sessions). The path is fenced as untrusted external metadata; the trusted guidance tells the agent to extract the file with its own tools, and modern Office files get an unzip hint. Generic ACP backends, URL-only attachments, and sandboxed sessions keep the plain `[Unsupported document format: <mime>. PDF and plain-text attachments can be read.]` marker.
- Files rejected by an operator-configured allowlist never include the self-serve path; a policy rejection must not coach the agent around the operator's decision.
- Files rejected by an operator-configured `allowedMimes` list get `[Attachment type not allowed: <mime>]` instead, so the prompt never claims support the active configuration disables.
- Read failures get `[Attachment could not be read]`.
- URL attachments get `[Attachment skipped: URL file sources are disabled]` when URL file sources are disabled.
- A file with no extractable text gets `[No extractable text]`.
- At most five skip markers render per message; further skipped attachments collapse into one reason-neutral `[<n> more attachments skipped]` summary so junk attachments cannot grow the prompt without bound. File and image, audio, or video markers share this five-marker budget.
- If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder `[PDF content rendered to images]` in the file block.
- Image, audio, and video decisions record one closed disposition for every attachment candidate: handled, handed to native vision, not selected after the attachment limit, disabled, missing a model, denied by chat scope, or failed.
- Unhandled media gets a bounded model-visible marker. Images handed to native vision and media turns owned by another harness do not add markers.
## Config examples
<Tabs>
<Tab title="Shared models + overrides">
```json5
{
tools: {
media: {
models: [
{ provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
{
provider: "google",
model: "gemini-3-flash-preview",
capabilities: ["image", "audio", "video"],
},
{
type: "cli",
command: "gemini",
args: [
"-m",
"gemini-3-flash",
"--allowed-tools",
"read_file",
"Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
],
capabilities: ["image", "video"],
},
],
audio: {
attachments: { mode: "all", maxAttachments: 2 },
},
video: {
maxChars: 500,
},
},
},
}
```
</Tab>
<Tab title="Audio + video only">
```json5
{
tools: {
media: {
models: [
{
provider: "openai",
model: "gpt-4o-mini-transcribe",
capabilities: ["audio"],
},
{
type: "cli",
command: "whisper",
args: ["--model", "base", "{{AttachmentPath}}"],
capabilities: ["audio"],
},
{
provider: "google",
model: "gemini-3-flash-preview",
capabilities: ["video"],
},
{
type: "cli",
command: "gemini",
args: [
"-m",
"gemini-3-flash",
"--allowed-tools",
"read_file",
"Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
],
capabilities: ["video"],
},
],
audio: {
enabled: true,
},
video: {
enabled: true,
maxChars: 500,
},
},
},
}
```
</Tab>
<Tab title="Image only">
```json5
{
tools: {
media: {
models: [
{ provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },
{ provider: "anthropic", model: "claude-opus-5", capabilities: ["image"] },
{
type: "cli",
command: "gemini",
args: [
"-m",
"gemini-3-flash",
"--allowed-tools",
"read_file",
"Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
],
capabilities: ["image"],
},
],
image: {
enabled: true,
maxBytes: 10485760,
maxChars: 500,
},
},
},
}
```
</Tab>
<Tab title="Multi-modal single entry">
```json5
{
tools: {
media: {
models: [
{
provider: "google",
model: "gemini-3.1-pro-preview",
capabilities: ["image", "video", "audio"],
},
],
},
},
}
```
</Tab>
</Tabs>
## Status output
When media understanding runs, `/status` includes a per-capability summary line:
```
📎 Media: image ok (openai/gpt-5.6-sol) · audio ok (whisper-cli observed=metal)
```
For preflight inventory, run `openclaw capability audio providers`. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding:
```bash
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info
```
## Notes
- Understanding is best-effort. Errors do not block replies.
- Attachments are still passed to models even when understanding is disabled.
- Use `scope` to limit where understanding runs (for example, only DMs).
## Related
- [Configuration](/gateway/configuration)
- [Image & media support](/nodes/images)
Reference in New Issue View Git Blame Copy Permalink