From 7920af0c9ec44b43d308bd00df141e333901454d Mon Sep 17 00:00:00 2001 From: scotthuang <101131451@qq.com> Date: Sun, 31 May 2026 07:00:19 +0800 Subject: [PATCH] refactor: route browser screenshot vision through shared media understanding * feat(browser): add optional vision understanding to screenshot tool * fix(browser): wrap vision output as external content, enforce maxBytes, forward auth profiles * fix(browser): remove no-op scope/attachments config, drop profile pass-through lacking runtime support * feat(media-understanding): add profile/preferredProfile to DescribeImageFileWithModelParams and forward to describeImage * style(browser): add curly braces to satisfy eslint curly rule * fix(browser): correct tools.browser.enabled help text to match actual behavior * fix(browser): thread agentDir/workspaceDir from plugin tool context into browser vision * refactor(browser): move vision config from tools.browser to browser.models The browser plugin's vision configuration now lives on the top-level `browser` config namespace (browser.models, browser.visionEnabled, browser.visionPrompt, etc.) instead of `tools.browser`. This aligns with the plugin's existing config location and avoids confusion between tool-level and plugin-level settings. - Remove tools.browser from ToolsSchema and ToolsConfig - Add models/vision* fields to BrowserConfig and its zod schema - Update getBrowserVisionConfig to read from cfg.browser - Update schema help, labels, and quality test - Update vision.test.ts to use new config shape * docs(browser): add screenshot vision configuration section Document the new browser.models config for automatic screenshot description via vision models, enabling text-only main models to reason about web page content. * fix(browser): remove deliverable media markers from vision result, drop unused import P1: Vision-success path no longer exposes the raw screenshot as deliverable media (removes MEDIA: line and details.media.mediaUrl). This prevents channel delivery from auto-sending sensitive page content when the intended output is a text description. P2: Remove unused ToolsMediaUnderstandingSchema import that would fail noUnusedLocals typecheck. * fix(browser): add command/args fields to browser models schema The browser vision model schema uses .strict(), so CLI-type entries with command/args were rejected by TypeScript. Add these fields to align with MediaUnderstandingModelSchema. * chore(browser): remove debug console.log statements * fix(browser): harden screenshot vision result against MEDIA: directive injection and restore image sanitization on failure fallback ClawSweeper #84247 review round 2: P1 (security, high): neutralize line-start MEDIA: directives in vision descriptions before wrapping with wrapExternalContent. The agent media extractor scans every browser tool-result text block via splitMediaFromOutput which treats line-start MEDIA: as a trusted local-media delivery directive, and browser is on the trusted-media allowlist. Without neutralization, page or vision-provider output containing 'MEDIA:/tmp/secret.png' could synthesize a channel-deliverable media artifact from untrusted content. wrapExternalContent itself does not strip line-start directives. Introduce neutralizeMediaDirectives in vision.ts that prepends '[neutralized] ' to any line whose trimStart() begins with MEDIA: (case-insensitive), defanging the parser anchor while keeping the original text human-readable. P2 (compatibility): pass resolveRuntimeImageSanitization() to imageResultFromFile in the vision-failure catch fallback. The non-vision screenshot path already forwards this option (d5cc0d53b7e3) so configured agents.defaults.imageMaxDimensionPx takes effect. Without this fix, any provider timeout/error silently bypasses the sanitization guard and returns a raw full-resolution screenshot. Regression coverage: - vision.test.ts: 6 unit cases for neutralizeMediaDirectives (no-op fast path, mid-line MEDIA: untouched, line-start defanged, leading-whitespace defanged, case-insensitive, multiple directives per blob). - browser-tool.test.ts: 2 integration cases that drive the full screenshot tool execute path: - 'neutralizes MEDIA: directives in vision text and does not attach media' asserts no line matches /^\s*MEDIA:/i in returned text, secret path text is preserved verbatim, details.media is absent, and imageResultFromFile is not called on the success path. - 'preserves screenshot image sanitization on vision failure fallback' mocks describeImageFileWithModel to reject and asserts the fallback imageResultFromFile call receives imageSanitization: {maxDimensionPx:1600} plus the 'browser screenshot vision failed' extraText. * fix(browser): apply clawsweeper fallback media fix from PR #84247 * refactor: reuse media image understanding for browser screenshots * refactor: use structured media delivery * test: update music completion media instruction expectation * fix: trim buffered reply directive padding * test: refresh codex prompt snapshots for message media aliases --------- Co-authored-by: scotthuang Co-authored-by: Peter Steinberger --- docs/concepts/streaming.md | 12 +- docs/help/faq.md | 4 +- docs/nodes/camera.md | 12 +- docs/nodes/index.md | 2 +- docs/reference/rich-output-protocol.md | 46 ++--- docs/start/openclaw.md | 24 +-- docs/tools/browser-control.md | 2 +- docs/tools/browser.md | 51 ++++++ docs/tools/tts.md | 4 +- extensions/browser/index.test.ts | 66 +++++++ extensions/browser/plugin-registration.ts | 79 ++++++++- .../browser/src/browser-tool.runtime.ts | 2 + extensions/browser/src/browser-tool.test.ts | 124 +++++++++++++ extensions/browser/src/browser-tool.ts | 99 ++++++++++- extensions/browser/src/browser/vision.test.ts | 166 ++++++++++++++++++ extensions/browser/src/browser/vision.ts | 123 +++++++++++++ .../browser/src/cli/browser-cli-inspect.ts | 8 +- extensions/browser/src/cli/browser-cli.ts | 2 +- extensions/browser/src/sdk-setup-tools.ts | 1 + extensions/canvas/src/cli.test.ts | 6 +- extensions/canvas/src/cli.ts | 4 +- .../src/providers/mock-openai/server.test.ts | 6 +- .../src/providers/mock-openai/server.ts | 44 ++++- .../src/suite-runtime-agent-media.test.ts | 19 +- .../qa-lab/src/suite-runtime-agent-media.ts | 54 +++++- .../qqbot/src/engine/group/history.test.ts | 12 +- .../src/engine/messaging/decode-media-path.ts | 13 +- .../engine/messaging/streaming-media-send.ts | 6 +- .../src/engine/ref/format-ref-entry.test.ts | 4 +- .../src/engine/utils/attachment-tags.test.ts | 16 +- .../qqbot/src/engine/utils/attachment-tags.ts | 10 +- packages/speech-core/src/tts.test.ts | 14 ++ packages/speech-core/src/tts.ts | 9 +- src/agents/command/delivery.ts | 6 +- .../embedded-agent-runner/run/attempt.ts | 2 +- .../embedded-agent-runner/run/payloads.ts | 23 ++- ...-agent-subscribe.handlers.messages.test.ts | 50 +++++- ...edded-agent-subscribe.handlers.messages.ts | 49 ++++-- ...ent-subscribe.handlers.tools.media.test.ts | 54 ++++-- ...embedded-agent-subscribe.handlers.tools.ts | 41 +---- ...sion.subscribeembeddedagentsession.test.ts | 33 +++- ...bedded-agent-subscribe.tools.media.test.ts | 152 +++++----------- src/agents/embedded-agent-subscribe.tools.ts | 63 ++----- src/agents/embedded-agent-subscribe.ts | 18 +- src/agents/embedded-agent-subscribe.types.ts | 2 +- src/agents/system-prompt.ts | 6 +- src/agents/tool-images.log.test.ts | 10 -- src/agents/tool-images.ts | 27 +-- src/agents/tools/image-generate-tool.ts | 2 +- .../tools/media-generate-background-shared.ts | 2 +- src/agents/tools/message-tool.ts | 10 ++ .../tools/music-generate-background.test.ts | 4 +- src/agents/tools/music-generate-tool.ts | 2 +- src/agents/tools/tts-tool.ts | 11 +- src/agents/tools/video-generate-tool.ts | 2 +- src/auto-reply/reply/prompt-prelude.ts | 2 +- src/auto-reply/reply/reply-delivery.test.ts | 23 ++- src/auto-reply/reply/reply-delivery.ts | 5 +- src/auto-reply/reply/reply-directives.ts | 2 + src/auto-reply/reply/reply-utils.test.ts | 84 +++------ src/auto-reply/reply/streaming-directives.ts | 43 ++--- src/cli/nodes-cli/register.camera.ts | 8 +- src/cli/nodes-cli/register.screen.ts | 4 +- src/cli/program.nodes-media.e2e.test.ts | 6 +- src/commands/agent-via-gateway.ts | 2 +- src/gateway/server-methods/chat.ts | 2 +- src/infra/outbound/deliver.test.ts | 10 +- .../message-action-runner.media.test.ts | 39 +++- src/infra/outbound/message-action-runner.ts | 21 ++- src/infra/outbound/payloads.test.ts | 9 +- src/infra/outbound/payloads.ts | 2 +- src/media-understanding/runtime-types.ts | 8 + src/media-understanding/runtime.test.ts | 43 +++++ src/media-understanding/runtime.ts | 20 ++- src/media/load-options.ts | 6 +- src/media/parse.ts | 6 +- src/media/web-media.test.ts | 19 ++ src/media/web-media.ts | 14 +- .../codex-dynamic-tools.discord-group.json | 11 ++ .../codex-dynamic-tools.heartbeat-turn.json | 11 ++ .../codex-dynamic-tools.telegram-direct.json | 11 ++ .../discord-group-codex-message-tool.md | 19 +- .../telegram-direct-codex-message-tool.md | 19 +- .../telegram-heartbeat-codex-tool.md | 19 +- 84 files changed, 1487 insertions(+), 594 deletions(-) create mode 100644 extensions/browser/src/browser/vision.test.ts create mode 100644 extensions/browser/src/browser/vision.ts diff --git a/docs/concepts/streaming.md b/docs/concepts/streaming.md index 189e126b337b..c7dcc5f488c4 100644 --- a/docs/concepts/streaming.md +++ b/docs/concepts/streaming.md @@ -54,16 +54,16 @@ Legend: ### Media delivery with block streaming -`MEDIA:` directives are normal delivery metadata. When block streaming sends a -media block early, OpenClaw remembers that delivery for the turn. If the final -assistant payload repeats the same media URL, the final delivery strips the -duplicate media instead of sending the attachment again. +Streaming media must use structured payload fields such as `mediaUrl` or +`mediaUrls`; streamed text is not parsed as an attachment command. When block +streaming sends media early, OpenClaw remembers that delivery for the turn. If +the final assistant payload repeats the same media URL, the final delivery +strips the duplicate media instead of sending the attachment again. Exact duplicate final payloads are suppressed. If the final payload adds distinct text around media that was already streamed, OpenClaw still sends the new text while keeping the media single-delivery. This prevents duplicate voice -notes or files on channels such as Telegram when an agent emits `MEDIA:` during -streaming and the provider also includes it in the completed reply. +notes or files on channels such as Telegram. ## Chunking algorithm (low/high bounds) diff --git a/docs/help/faq.md b/docs/help/faq.md index 558d797a6c9a..be5d9838ecff 100644 --- a/docs/help/faq.md +++ b/docs/help/faq.md @@ -1737,7 +1737,7 @@ lives on the [Models FAQ](/help/faq-models). - Outbound attachments from the agent must include a `MEDIA:` line (on its own line). See [OpenClaw assistant setup](/start/openclaw) and [Agent send](/tools/agent-send). + Outbound attachments from the agent must use structured media fields such as `media`, `mediaUrl`, `path`, or `filePath`. See [OpenClaw assistant setup](/start/openclaw) and [Agent send](/tools/agent-send). CLI sending: @@ -1750,7 +1750,7 @@ lives on the [Models FAQ](/help/faq-models). - The target channel supports outbound media and isn't blocked by allowlists. - The file is within the provider's size limits (images are resized to max 2048px). - `tools.fs.workspaceOnly=true` keeps local-path sends limited to workspace, temp/media-store, and sandbox-validated files. - - `tools.fs.workspaceOnly=false` lets `MEDIA:` send host-local files the agent can already read, but only for media plus safe document types (images, audio, video, PDF, and Office docs). Plain text and secret-like files are still blocked. + - `tools.fs.workspaceOnly=false` lets structured local media sends use host-local files the agent can already read, but only for media plus safe document types (images, audio, video, PDF, and Office docs). Plain text and secret-like files are still blocked. See [Images](/nodes/images). diff --git a/docs/nodes/camera.md b/docs/nodes/camera.md index 7ac293033cb0..7982ddf2579f 100644 --- a/docs/nodes/camera.md +++ b/docs/nodes/camera.md @@ -59,9 +59,9 @@ All camera access is gated behind **user-controlled settings**. Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`. -### CLI helper (temp files + MEDIA) +### CLI helper -The easiest way to get attachments is via the CLI helper, which writes decoded media to a temp file and prints `MEDIA:`. +The easiest way to get media files is via the CLI helper, which writes decoded media to a temp file and prints the saved path. Examples: @@ -126,12 +126,12 @@ Examples: ```bash openclaw nodes camera list --node # list camera ids -openclaw nodes camera snap --node # prints MEDIA: +openclaw nodes camera snap --node # prints saved path openclaw nodes camera snap --node --max-width 1280 openclaw nodes camera snap --node --delay-ms 2000 openclaw nodes camera snap --node --device-id -openclaw nodes camera clip --node --duration 10s # prints MEDIA: -openclaw nodes camera clip --node --duration-ms 3000 # prints MEDIA: (legacy flag) +openclaw nodes camera clip --node --duration 10s # prints saved path +openclaw nodes camera clip --node --duration-ms 3000 # prints saved path (legacy flag) openclaw nodes camera clip --node --device-id openclaw nodes camera clip --node --no-audio ``` @@ -152,7 +152,7 @@ Notes: For _screen_ video (not camera), use the macOS companion: ```bash -openclaw nodes screen record --node --duration 10s --fps 15 # prints MEDIA: +openclaw nodes screen record --node --duration 10s --fps 15 # prints saved path ``` Notes: diff --git a/docs/nodes/index.md b/docs/nodes/index.md index 047f5437acb8..7d4deac2343c 100644 --- a/docs/nodes/index.md +++ b/docs/nodes/index.md @@ -218,7 +218,7 @@ and approve the new request so the gateway stores the updated command snapshot. If the node is showing the Canvas (WebView), `canvas.snapshot` returns `{ format, base64 }`. -CLI helper (writes to a temp file and prints `MEDIA:`): +CLI helper (writes to a temp file and prints the saved path): ```bash openclaw nodes canvas snapshot --node --format png diff --git a/docs/reference/rich-output-protocol.md b/docs/reference/rich-output-protocol.md index b4a544273999..416c5d4c138f 100644 --- a/docs/reference/rich-output-protocol.md +++ b/docs/reference/rich-output-protocol.md @@ -1,59 +1,51 @@ --- -summary: "Rich output shortcode protocol for embeds, media, audio hints, and replies" +summary: "Rich output protocol for structured media, embeds, audio hints, and replies" read_when: - Changing assistant output rendering in the Control UI - - Debugging `[embed ...]`, `MEDIA:`, reply, or audio presentation directives + - Debugging `[embed ...]`, structured media, reply, or audio presentation directives title: "Rich output protocol" --- Assistant output can carry a small set of delivery/render directives: -- `MEDIA:` for attachment delivery +- structured `mediaUrl` / `mediaUrls` fields for attachment delivery - `[[audio_as_voice]]` for audio presentation hints - `[[reply_to_current]]` / `[[reply_to:]]` for reply metadata - `[embed ...]` for Control UI rich rendering -Remote `MEDIA:` attachments must be public `https:` URLs. Plain `http:`, +Remote media attachments must be public `https:` URLs. Plain `http:`, loopback, link-local, private, and internal hostnames are ignored as attachment directives; server-side media fetchers still enforce their own network guards. -Local `MEDIA:` attachments can use absolute paths, workspace-relative paths, or +Local media attachments can use absolute paths, workspace-relative paths, or home-relative `~/` paths. They still pass through the agent file-read policy and media type checks before delivery. -`MEDIA:` is parsed only as plain text. Wrapping the directive in Markdown -formatting (bold, inline code, fenced code) prevents the parser from -recognizing it, and the attachment is silently dropped from delivery. +Do not emit text commands for attachments from tools, plugins, streaming blocks, +browser output, or message actions. Use structured media fields instead. -Valid: +Valid message-tool payload: -```text -MEDIA:/workspace/image.png +```json +{ "message": "Here is your image.", "mediaUrl": "/workspace/image.png" } ``` -Invalid (parsed as prose, no attachment delivered): - -```text -**MEDIA:/workspace/image.png** -`MEDIA:/workspace/image.png` -Here is your image: MEDIA:/workspace/image.png -``` - -Keep `MEDIA:` on its own line, in plain text, with no surrounding formatting. +Legacy final assistant reply text may still be normalized for compatibility, but +it is not a general plugin/tool protocol. Plain Markdown image syntax stays text by default. Channels that intentionally map Markdown image replies to media attachments opt in at their outbound adapter; Telegram does this so `![alt](url)` can still become a media reply. -These directives are separate. `MEDIA:` and reply/voice tags remain delivery metadata; `[embed ...]` is the web-only rich render path. -Trusted tool-result media uses the same `MEDIA:` / `[[audio_as_voice]]` parser before delivery, so text tool outputs can still mark an audio attachment as a voice note. +These directives are separate. Structured media fields and reply/voice tags are +delivery metadata; `[embed ...]` is the web-only rich render path. -When block streaming is enabled, `MEDIA:` remains single-delivery metadata for a -turn. If the same media URL is sent in a streamed block and repeated in the final -assistant payload, OpenClaw delivers the attachment once and strips the duplicate -from the final payload. +When block streaming is enabled, media must be carried on structured payload +fields. If the same media URL is sent in a streamed block and repeated in the +final assistant payload, OpenClaw delivers the attachment once and strips the +duplicate from the final payload. ## `[embed ...]` @@ -72,7 +64,7 @@ Rules: - Only URL-backed embeds are rendered. Use `ref="..."` or `url="..."`. - Block-form inline HTML embed shortcodes are not rendered. - The web UI strips the shortcode from visible text and renders the embed inline. -- `MEDIA:` is not an embed alias and should not be used for rich embed rendering. +- Structured media is not an embed alias and should not be used for rich embed rendering. ## Stored rendering shape diff --git a/docs/start/openclaw.md b/docs/start/openclaw.md index ca10ac5f191a..71b007b81962 100644 --- a/docs/start/openclaw.md +++ b/docs/start/openclaw.md @@ -196,27 +196,21 @@ Inbound attachments (images/audio/docs) can be surfaced to your command via temp - `{{MediaUrl}}` (pseudo-URL) - `{{Transcript}}` (if audio transcription is enabled) -Outbound attachments from the agent: include `MEDIA:` on its own line (no spaces). The directive must start the line as plain text, outside code fences and without Markdown wrappers such as bold or inline code. Example: +Outbound attachments from the agent use structured media fields on the message tool or reply payload, such as `media`, `mediaUrl`, `mediaUrls`, `path`, or `filePath`. Example message-tool arguments: -``` -Here's the screenshot. -MEDIA:https://example.com/screenshot.png +```json +{ + "message": "Here's the screenshot.", + "mediaUrl": "https://example.com/screenshot.png" +} ``` -OpenClaw extracts these and sends them as media alongside the text. - -These forms are not attachment directives and are sent as normal text: - -```md -**MEDIA:https://example.com/screenshot.png** -`MEDIA:https://example.com/screenshot.png` -Here is the screenshot: MEDIA:https://example.com/screenshot.png -``` +OpenClaw sends structured media alongside the text. Legacy final assistant replies may still be normalized for compatibility, but tool output, browser output, streaming blocks, and message actions do not parse text as attachment commands. Local-path behavior follows the same file-read trust model as the agent: -- If `tools.fs.workspaceOnly` is `true`, outbound `MEDIA:` local paths stay restricted to the OpenClaw temp root, the media cache, agent workspace paths, and sandbox-generated files. -- If `tools.fs.workspaceOnly` is `false`, outbound `MEDIA:` can use host-local files the agent is already allowed to read. +- If `tools.fs.workspaceOnly` is `true`, outbound local media paths stay restricted to the OpenClaw temp root, the media cache, agent workspace paths, and sandbox-generated files. +- If `tools.fs.workspaceOnly` is `false`, outbound local media can use host-local files the agent is already allowed to read. - Local paths can be absolute, workspace-relative, or home-relative with `~/`. - Host-local sends still only allow media and safe document types (images, audio, video, PDF, and Office documents). Plain text and secret-like files are not treated as sendable media. diff --git a/docs/tools/browser-control.md b/docs/tools/browser-control.md index b5e583163fbc..164cb2ddc7ea 100644 --- a/docs/tools/browser-control.md +++ b/docs/tools/browser-control.md @@ -252,7 +252,7 @@ Snapshot flags at a glance: - `--format aria`: accessibility tree with `axN` refs. When Playwright is available, OpenClaw binds refs with backend DOM ids to the live page so follow-up actions can use them; otherwise treat the output as inspection-only. - `--efficient` (or `--mode efficient`): compact role snapshot preset. Set `browser.snapshotDefaults.mode: "efficient"` to make this the default (see [Gateway configuration](/gateway/configuration-reference#browser)). - `--interactive`, `--compact`, `--depth`, `--selector` force a role snapshot with `ref=e12` refs. `--frame "