ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-25 20:05:46 -06:00
Code Issues Packages Projects Releases Wiki Activity

refactor(canvas): make the panel a widget presenter (#126030)

Browse Source 
* refactor(canvas): retire legacy host and commands

* refactor(apple): narrow shared Canvas contracts

* refactor(macos): keep Canvas as widget presenter

* refactor(ios): remove Canvas client

* refactor(android): remove Canvas client

* refactor(linux): remove Canvas client

* fix(ci): isolate native locale artifacts

* fix(linux): regenerate companion lockfile

* fix(canvas): refresh native tool display metadata

* test(canvas): align coverage with presenter surface

* test(canvas): remove obsolete asset root seam

* test(canvas): stabilize retirement CI coverage

* refactor(swift): remove orphaned resource wrapper

* test(ios): remove retired canvas layout assertion

* fix(macos): reserve retired canvas command namespace

* refactor(macos): isolate canvas command policy

* fix(canvas): select only eligible macOS panels

* fix(canvas): keep panel selection plugin-owned
This commit is contained in:
Peter Steinberger
2026-08-19 08:21:07 -07:00
committed by GitHub
parent c90226dc53
commit ae55a4090c
236 changed files with 1994 additions and 15556 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/labeler.yml
-1
View File
@@ -253,7 +253,6 @@
- any-glob-to-any-file:
- "apps/linux/**"
- "docs/platforms/linux.md"
- "extensions/linux-canvas/**"
"app: web-ui":
- changed-files:
- any-glob-to-any-file:
SECURITY.md
+2 -2
View File
@@ -92,7 +92,7 @@ These are frequently reported but are typically closed with no code change:
- Prompt-injection-only chains without a boundary bypass (prompt injection is out of scope).
- Operator-intended local features (for example TUI local `!` shell) presented as remote injection.
- Reports that treat explicit operator-control surfaces (for example `canvas.eval`, browser evaluate/script execution, or direct `node.invoke` execution primitives) as vulnerabilities without demonstrating an auth/policy/sandbox boundary bypass. These capabilities are intentional when enabled and are trusted-operator features, not standalone security bugs.
- Reports that treat explicit operator-control surfaces (for example browser evaluate/script execution or direct `node.invoke` execution primitives) as vulnerabilities without demonstrating an auth/policy/sandbox boundary bypass. These capabilities are intentional when enabled and are trusted-operator features, not standalone security bugs.
- Reports that treat an admin-gated enablement or arming step as requiring `operator.admin` for every subsequent action, when the documented contract delegates use of the enabled capability to `operator.write` and no auth, arming, allowlist, sandbox, or policy bypass is shown. This is an arm-then-use operator guardrail, not privilege escalation.
- Authorized user-triggered local actions presented as privilege escalation. Example: an allowlisted/owner sender running `/export-session /absolute/path.html` to write on the host. In this trust model, authorized user actions are trusted host actions unless you demonstrate an auth/sandbox/boundary bypass.
- Reports that only show a malicious plugin executing privileged actions after a trusted operator installs/enables it.
@@ -169,7 +169,7 @@ Plugins/extensions are part of OpenClaw's trusted computing base for a gateway.
- Reports whose only claim is post-approval executable identity drift on a trusted host via same-path file replacement/rewrite unless a separate untrusted boundary bypass is shown for that host write primitive.
- Reports whose only claim is environment-variable-driven executable behavior change, including path lookup changes, preload hooks, wrapper/interpreter selection, package-manager/runtime hooks, or variables that make an executable invoke another executable, unless a separate OpenClaw boundary bypass lets untrusted input set or mutate that environment.
- Reports where the only demonstrated impact is an already-authorized sender intentionally invoking a local-action command (for example `/export-session` writing to an absolute host path) without bypassing auth, sandbox, or another documented boundary
- Reports whose only claim is use of an explicit trusted-operator control surface (for example `canvas.eval`, browser evaluate/script execution, or direct `node.invoke` execution) without demonstrating an auth, policy, allowlist, approval, or sandbox bypass.
- Reports whose only claim is use of an explicit trusted-operator control surface (for example browser evaluate/script execution or direct `node.invoke` execution) without demonstrating an auth, policy, allowlist, approval, or sandbox bypass.
- Reports where the only claim is that a trusted-installed/enabled plugin can execute with gateway/host privileges (documented trust model behavior).
- Any report whose only claim is that an operator-enabled `dangerous*`/`dangerously*` config option weakens defaults (these are explicit break-glass tradeoffs by design)
- Reports that depend on trusted operator-supplied configuration values to trigger availability impact (for example custom regex patterns). These may still be fixed as defense-in-depth hardening, but are not security-boundary bypasses.
apps/.i18n/native-source.json
Generated
-573
View File
@@ -1433,17 +1433,6 @@
}
]
},
{
"id": "native.android.5b8201964dde6f85",
"source": "Active on this phone",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.ccd226a2c0e85242",
"source": "Active task list is at its limit",
@@ -2821,17 +2810,6 @@
}
]
},
{
"id": "native.android.b62482f58e031dc3",
"source": "Bridge",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.eb2895729924f303",
"source": "Brining",
@@ -3089,43 +3067,6 @@
}
]
},
{
"id": "native.android.866ba9a5bb1a7f78",
"source": "Canvas",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/ShellScreen.kt"
}
]
},
{
"id": "native.android.5af14aa58bca179d",
"source": "Canvas Status",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt"
}
]
},
{
"id": "native.android.44a4fb62733e24c6",
"source": "Canvas output needs an active gateway connection.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.00c4489830ccafcf",
"source": "Capability approval pending",
@@ -3631,17 +3572,6 @@
}
]
},
{
"id": "native.android.182fd214002b29fd",
"source": "Close Canvas",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/ShellScreen.kt"
}
]
},
{
"id": "native.android.9e3c928c58928895",
"source": "Close image preview",
@@ -3926,17 +3856,6 @@
}
]
},
{
"id": "native.android.f8d28df6a1df6381",
"source": "Connect the gateway",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.c295f12117ba7386",
"source": "Connect the gateway to browse workspace files.",
@@ -4209,17 +4128,6 @@
}
]
},
{
"id": "native.android.ec9908b9ab6cfb7b",
"source": "Connect to load your agents",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.8aba779b768f9497",
"source": "Connect to your Gateway",
@@ -4377,17 +4285,6 @@
}
]
},
{
"id": "native.android.02bdbe53eaafab28",
"source": "Connected to $gatewayLabel",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.f81d190c1511e44a",
"source": "Connecting",
@@ -4457,10 +4354,6 @@
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeForegroundService.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt"
@@ -5311,17 +5204,6 @@
}
]
},
{
"id": "native.android.6ba7183d09548c7a",
"source": "Current screen output and interactive app surface.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.b8a8f25579564d21",
"source": "Current session",
@@ -5446,10 +5328,6 @@
"source": "Default agent",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SkillWorkshopSettingsScreen.kt"
@@ -6516,17 +6394,6 @@
}
]
},
{
"id": "native.android.a1baf41e20e30419",
"source": "Failed to request restore. Tap to retry.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.1062364401943984",
"source": "Failed — $it",
@@ -6815,10 +6682,6 @@
"source": "Gateway",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/HealthLogsSettingsScreen.kt"
@@ -7350,17 +7213,6 @@
}
]
},
{
"id": "native.android.0257c2760551e7a0",
"source": "Gateway session in progress",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.1fe19418fbbcd7aa",
"source": "Gateway status, phone node readiness, and recent log stream.",
@@ -7693,17 +7545,6 @@
}
]
},
{
"id": "native.android.ef14e6a42656428f",
"source": "Home canvas",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.46e7f94e1f9784bd",
"source": "Host",
@@ -7793,17 +7634,6 @@
}
]
},
{
"id": "native.android.7258ce613d19f304",
"source": "If the gateway is reachable, reconnect should complete without intervention.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.2410b214b9995569",
"source": "Image",
@@ -8478,17 +8308,6 @@
}
]
},
{
"id": "native.android.da02288f33f35aaa",
"source": "Live page",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.96c78912cf0124e0",
"source": "Live transcript",
@@ -8701,10 +8520,6 @@
"source": "Main",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt"
@@ -9720,17 +9535,6 @@
}
]
},
{
"id": "native.android.4dacf7dcc6d48f88",
"source": "No canvas update yet. Tap to retry.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.d8115083618341f7",
"source": "No channels",
@@ -10286,17 +10090,6 @@
}
]
},
{
"id": "native.android.cc49ac364268f663",
"source": "Node offline. Reconnect and retry.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.f1e946108f81769e",
"source": "Nodes",
@@ -10717,10 +10510,6 @@
"kind": "ui-named-argument",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/HealthLogsSettingsScreen.kt"
@@ -10886,10 +10675,6 @@
"source": "Online",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/HealthLogsSettingsScreen.kt"
@@ -11033,17 +10818,6 @@
}
]
},
{
"id": "native.android.4a0022fb1c008b0d",
"source": "Open Screen",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.296fd1428b8f59bc",
"source": "Open Settings",
@@ -11206,17 +10980,6 @@
}
]
},
{
"id": "native.android.abc9911c021e13cb",
"source": "Open the current Canvas surface to inspect or interact with it.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.88b9aad3b451616f",
"source": "Open thread",
@@ -11541,17 +11304,6 @@
}
]
},
{
"id": "native.android.b73f7afaed5e0a0d",
"source": "OpenClaw is syncing back up",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.d6542e1282f7812e",
"source": "OpenClaw is working",
@@ -11894,17 +11646,6 @@
}
]
},
{
"id": "native.android.44c3c35d0c823ff2",
"source": "Pair this device to your gateway to wake it only for real work, keep a live agent overview handy, and avoid battery-draining background loops.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.4bfab01326a3cfb2",
"source": "Paired",
@@ -13129,14 +12870,6 @@
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/GatewayTalkSetupReadiness.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/ChannelsSettingsScreen.kt"
@@ -13394,10 +13127,6 @@
"source": "Reconnect",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt"
@@ -13448,17 +13177,6 @@
}
]
},
{
"id": "native.android.94a87a408a998f5b",
"source": "Reconnecting",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.377c08edb4a24718",
"source": "Reconnecting…",
@@ -13550,17 +13268,6 @@
}
]
},
{
"id": "native.android.fbe6cca48fc3c648",
"source": "Refresh Screen",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.28322842224d1259",
"source": "Refresh background tasks",
@@ -13599,10 +13306,6 @@
"source": "Refreshing",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/ChannelsSettingsScreen.kt"
@@ -14017,17 +13720,6 @@
}
]
},
{
"id": "native.android.9d5a067d783c42b6",
"source": "Restore canvas now for session=$sessionKey source=$source. If existing A2UI state exists, replay it immediately. If not, create and render a compact mobile-friendly dashboard in Canvas.",
"surface": "android",
"sites": [
{
"kind": "ui-named-argument-concatenated",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.f48f5f2e61a7282d",
"source": "Retry",
@@ -14621,28 +14313,6 @@
}
]
},
{
"id": "native.android.f3090cbb8f561d55",
"source": "Screen surface",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/ShellScreen.kt"
}
]
},
{
"id": "native.android.2034fd6620e681f5",
"source": "Screen surface ready",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.cbf9a102a28066d1",
"source": "Script",
@@ -15006,17 +14676,6 @@
}
]
},
{
"id": "native.android.401c3f75ad88d7c4",
"source": "Selected on this phone",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.bf9df75bbd74480d",
"source": "Selected or full photo access granted.",
@@ -15600,17 +15259,6 @@
}
]
},
{
"id": "native.android.41e12872a460ce83",
"source": "Show screen-sharing debug state.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt"
}
]
},
{
"id": "native.android.59c0981a6bba60dd",
"source": "Showing ${visibleApps.size} of ${apps.size}. Refine search for more.",
@@ -16127,17 +15775,6 @@
}
]
},
{
"id": "native.android.bfed07d0a38974f2",
"source": "Standby",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.0a047512bbbf8a70",
"source": "Start Chat",
@@ -16511,17 +16148,6 @@
}
]
},
{
"id": "native.android.0ede4a4211dbe0ed",
"source": "Surface",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt"
}
]
},
{
"id": "native.android.ca52f5e355c99ff9",
"source": "Surfacing",
@@ -17046,28 +16672,6 @@
}
]
},
{
"id": "native.android.f465630c9d0e3e7d",
"source": "The gateway session is coming back online. Agent shortcuts should settle automatically in a moment.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.ad658d20177609a1",
"source": "The overview refreshes on reconnect and when this screen opens.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.773b10f8ee342bf4",
"source": "The result for $slug is unknown. Reconnect, refresh Skills, then retry; the Gateway safely joins a matching install that is still running.",
@@ -17348,17 +16952,6 @@
}
]
},
{
"id": "native.android.baa142039556e0ee",
"source": "This phone stays dormant until the gateway needs it, then wakes, syncs, and goes back to sleep.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.7aa25cff23229ddd",
"source": "This skill is blocked by the gateway allowlist. Allowlist changes stay on desktop or CLI.",
@@ -18741,10 +18334,6 @@
"source": "Welcome to OpenClaw",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
},
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/ui/OnboardingFlow.kt"
@@ -18762,17 +18351,6 @@
}
]
},
{
"id": "native.android.be9350e46b23487b",
"source": "When connected, the gateway can wake the phone with a silent push instead of holding an always-on session.",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.ac10cd8e6869baf8",
"source": "Where do I get a setup code?",
@@ -18996,17 +18574,6 @@
}
]
},
{
"id": "native.android.33bd96ad84f6b1cd",
"source": "Your agents are ready",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.b4e0d4aa03353c07",
"source": "Your phone is paired with ${recoveryGatewayName(serverName = serverName, attemptedGatewayName = attemptedGatewayName)}. Continue to finish node access.",
@@ -19029,17 +18596,6 @@
}
]
},
{
"id": "native.android.7cc7f3a18d2be1bc",
"source": "Your phone stays quiet until it is needed",
"surface": "android",
"sites": [
{
"kind": "ui-call",
"path": "apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt"
}
]
},
{
"id": "native.android.6698826454daa7b0",
"source": "Your preferred phone changed. Open the app to reload the session before replying.",
@@ -22745,10 +22301,6 @@
"source": "Available",
"surface": "apple",
"sites": [
{
"kind": "conditional-branch",
"path": "apps/ios/Sources/RootTabs.swift"
},
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/ChannelsSettings+View.swift"
@@ -24474,17 +24026,6 @@
}
]
},
{
"id": "native.apple.9621ed74cd72289e",
"source": "Close canvas",
"surface": "apple",
"sites": [
{
"kind": "ui-modifier",
"path": "apps/ios/Sources/RootTabs.swift"
}
]
},
{
"id": "native.apple.90284f8660d14a12",
"source": "Close image preview",
@@ -27047,17 +26588,6 @@
}
]
},
{
"id": "native.apple.640ff6aab8d97fda",
"source": "Debug Screen Status",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/ios/Sources/Design/SettingsProTabSections.swift"
}
]
},
{
"id": "native.apple.62c6ef2db7c5dd05",
"source": "Decide from watch",
@@ -28981,28 +28511,6 @@
}
]
},
{
"id": "native.apple.75dd34360c117103",
"source": "Eval",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.689e209eeeb7cca9",
"source": "Eval JS",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.b143af13bdd1e109",
"source": "Event text",
@@ -30408,10 +29916,6 @@
{
"kind": "ui-localized-call",
"path": "apps/ios/Sources/Design/AgentAutomationDetailScreen.swift"
},
{
"kind": "conditional-branch",
"path": "apps/ios/Sources/RootTabs.swift"
}
]
},
@@ -30441,17 +29945,6 @@
}
]
},
{
"id": "native.apple.264053ca15a19c21",
"source": "Gateway needs attention",
"surface": "apple",
"sites": [
{
"kind": "conditional-branch",
"path": "apps/ios/Sources/RootTabs.swift"
}
]
},
{
"id": "native.apple.ea07a483f49f64fe",
"source": "Gateway not configured",
@@ -37464,17 +36957,6 @@
}
]
},
{
"id": "native.apple.3dfe5d3525846721",
"source": "OpenClaw iOS",
"surface": "apple",
"sites": [
{
"kind": "conditional-branch",
"path": "apps/ios/Sources/RootTabs.swift"
}
]
},
{
"id": "native.apple.40d6e689933c49c2",
"source": "OpenClaw installation failed",
@@ -41723,17 +41205,6 @@
}
]
},
{
"id": "native.apple.477f40964635800c",
"source": "Reveal",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.f2ed0e4a3e0e2b6c",
"source": "Reveal app in Finder",
@@ -42000,17 +41471,6 @@
}
]
},
{
"id": "native.apple.090579ae88131c3f",
"source": "Routed on this phone",
"surface": "apple",
"sites": [
{
"kind": "conditional-branch",
"path": "apps/ios/Sources/RootTabs.swift"
}
]
},
{
"id": "native.apple.4bf71fe3e547a4ed",
"source": "Run",
@@ -44714,17 +44174,6 @@
}
]
},
{
"id": "native.apple.bf0b00e244c5b237",
"source": "Snapshot",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.9bd57456b80dbeeb",
"source": "Sounds",
@@ -51148,17 +50597,6 @@
}
]
},
{
"id": "native.apple.ec16f68de25be2eb",
"source": "eval → \\(canvasEvalResult)",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.3e7adde6c3359c0c",
"source": "every",
@@ -51968,17 +51406,6 @@
}
]
},
{
"id": "native.apple.0c793c9099f80ae3",
"source": "snapshot → \\(canvasSnapshotPath)",
"surface": "apple",
"sites": [
{
"kind": "ui-call",
"path": "apps/macos/Sources/OpenClaw/DebugSettings.swift"
}
]
},
{
"id": "native.apple.0feb96c6b97603af",
"source": "ssh tunnel did not open local port \\(localPort)",
apps/android/README.md
+3 -14
View File
@@ -19,7 +19,6 @@ OpenClaw Android is the officially released Google Play app. It connects to an O
- [x] Authenticated background presence beacons
- [x] Voice tab full functionality
- [x] Foreground on-device Voice Wake with Gateway-synced wake words
- [x] Screen tab full functionality
- [x] Skill Workshop settings can filter proposals, inspect proposal content, and apply/reject/quarantine drafts through Gateway RPCs
- [x] Skills settings can search installed skills, enable or disable them, and install Gateway-verified ClawHub releases
- [x] Per-app language selection for translated resources follows Android system settings and persistence
@@ -28,7 +27,6 @@ OpenClaw Android is the officially released Google Play app. It connects to an O
## Open in Android Studio
- Run `pnpm install` from the repository root so native Canvas resources can be generated.
- Open the folder `apps/android`.
## Wear OS companion
@@ -248,7 +246,6 @@ This app is native Kotlin + Jetpack Compose.
- For Compose UI edits: use Android Studio **Live Edit** on a debug build (works on physical devices; project `minSdk=31` already meets API requirement).
- For many non-structural code/resource changes: use Android Studio **Apply Changes**.
- For structural/native/manifest/Gradle changes: do full reinstall (`pnpm android:run`).
- Canvas web content already supports live reload when loaded from Gateway `__openclaw__/canvas/` (see `docs/platforms/android.md`).
## Connect / Pair
@@ -338,12 +335,9 @@ Pre-req checklist:
1) Gateway is running and reachable from the Android app.
2) Android app is connected to that gateway and `openclaw nodes status` shows it as paired + connected.
3) App stays unlocked and in foreground for the whole run.
4) Open the app **Screen** tab and keep it active during the run (canvas/A2UI commands require the canvas WebView attached there).
5) Grant runtime permissions for capabilities you expect to pass (camera/mic/location/notification listener/location, etc.).
6) No interactive system dialogs should be pending before test start.
7) Canvas host is enabled and reachable from the device for remote Canvas checks (do not run gateway with `OPENCLAW_SKIP_CANVAS_HOST=1`; startup logs should include `canvas host mounted at .../__openclaw__/`).
8) Local operator test client pairing is approved. If first run fails with `pairing required`, preview the latest pending request, approve the printed request ID, then rerun:
9) For A2UI checks, keep the app on **Screen** tab; the node uses its bundled app-owned A2UI page for message application.
4) Grant runtime permissions for capabilities you expect to pass (camera/mic/location/notification listener/location, etc.).
5) No interactive system dialogs should be pending before test start.
6) Local operator test client pairing is approved. If first run fails with `pairing required`, preview the latest pending request, approve the printed request ID, then rerun:
```bash
openclaw devices list
@@ -377,11 +371,6 @@ Common failure quick-fixes:
- `pairing required` before tests start:
- list pending requests (`openclaw devices list`), then approve with the exact ID (`openclaw devices approve <requestId>`) and rerun.
- `A2UI host not reachable` / `A2UI_HOST_UNAVAILABLE`:
- keep the app foregrounded on the **Screen** tab and rerun. A2UI commands use the bundled app-owned A2UI page; the Gateway Canvas host is still needed for remote Canvas checks, but not for A2UI message application.
- `NODE_BACKGROUND_UNAVAILABLE: canvas unavailable`:
- app is not effectively ready for canvas commands; keep app foregrounded and **Screen** tab active.
## Contributions
Maintainer: @obviyus. For issues/questions/contributions, please open an issue or reach out on Discord.
apps/android/app/build.gradle.kts
-69
View File
@@ -1,27 +1,14 @@
import com.android.build.api.variant.impl.VariantOutputImpl
import org.gradle.api.DefaultTask
import org.gradle.api.file.ConfigurableFileCollection
import org.gradle.api.file.DirectoryProperty
import org.gradle.api.tasks.Exec
import org.gradle.api.tasks.InputFiles
import org.gradle.api.tasks.Internal
import org.gradle.api.tasks.OutputDirectory
import org.gradle.api.tasks.PathSensitive
import org.gradle.api.tasks.PathSensitivity
import org.gradle.api.tasks.TaskAction
import org.gradle.process.ExecOperations
import java.time.Instant
import java.time.ZoneOffset
import java.time.format.DateTimeFormatter
import java.util.Properties
import javax.inject.Inject
val dnsjavaInetAddressResolverService = "META-INF/services/java.net.spi.InetAddressResolverProvider"
val openClawAndroidApplicationId = "ai.openclaw.app"
val openClawAndroidVersionFile = rootProject.file("Config/Version.properties")
val thirdPartyLicensesDir = rootProject.file("THIRD_PARTY_LICENSES")
val openClawRepositoryRoot = rootProject.projectDir.resolve("../..").canonicalFile
val canvasA2uiAssetsDir = layout.buildDirectory.dir("generated/canvasA2uiAssets")
val openClawAndroidVersionProperties =
Properties().apply {
if (!openClawAndroidVersionFile.isFile) {
@@ -134,58 +121,6 @@ plugins {
alias(libs.plugins.ksp)
}
abstract class StageCanvasA2uiTask
@Inject
constructor(
private val execOperations: ExecOperations,
) : DefaultTask() {
@get:Internal abstract val repoRoot: DirectoryProperty
@get:InputFiles
@get:PathSensitive(PathSensitivity.RELATIVE)
abstract val sourceFiles: ConfigurableFileCollection
@get:OutputDirectory abstract val outputDirectory: DirectoryProperty
@TaskAction
fun stage() {
val root = repoRoot.get().asFile
execOperations.exec {
workingDir(root)
commandLine(
"node",
"--import",
"tsx",
"scripts/sync-native-a2ui.mts",
"--write",
"--output",
outputDirectory
.get()
.dir("CanvasA2UI")
.asFile.absolutePath,
)
}
}
}
val stageCanvasA2ui =
tasks.register<StageCanvasA2uiTask>("stageCanvasA2ui") {
group = "build"
description = "Stages the plugin-owned Canvas A2UI renderer for native apps."
repoRoot.set(openClawRepositoryRoot)
sourceFiles.from(
openClawRepositoryRoot.resolve("package.json"),
openClawRepositoryRoot.resolve("pnpm-lock.yaml"),
openClawRepositoryRoot.resolve("scripts/bundle-a2ui.mts"),
openClawRepositoryRoot.resolve("scripts/sync-native-a2ui.mts"),
openClawRepositoryRoot.resolve("extensions/canvas/package.json"),
openClawRepositoryRoot.resolve("extensions/canvas/scripts/bundle-a2ui.mjs"),
openClawRepositoryRoot.resolve("extensions/canvas/src/host/a2ui/index.html"),
)
sourceFiles.from(openClawRepositoryRoot.resolve("extensions/canvas/src/host/a2ui-app"))
outputDirectory.set(canvasA2uiAssetsDir)
}
ksp {
arg("room.schemaLocation", "$projectDir/schemas")
}
@@ -342,10 +277,6 @@ android {
androidComponents {
val adbExecutable = sdkComponents.adb
onVariants { variant ->
variant.sources.assets?.addGeneratedSourceDirectory(
stageCanvasA2ui,
StageCanvasA2uiTask::outputDirectory,
)
variant.outputs
.filterIsInstance<VariantOutputImpl>()
.forEach { output ->
apps/android/app/src/androidTest/java/ai/openclaw/app/ui/CanvasHostLifecycleTest.kt
-274
View File
@@ -1,274 +0,0 @@
package ai.openclaw.app.ui
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.ui.chat.ChatWidgetExportDestination
import ai.openclaw.app.ui.chat.exportChatWidgetImage
import ai.openclaw.app.ui.chat.widgetExportFileName
import android.content.ClipboardManager
import android.content.ContentUris
import android.content.pm.ActivityInfo
import android.graphics.BitmapFactory
import android.os.SystemClock
import android.provider.MediaStore
import android.view.View
import androidx.test.core.app.ActivityScenario
import androidx.test.ext.junit.rules.ActivityScenarioRule
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import androidx.test.uiautomator.UiDevice
import kotlinx.coroutines.runBlocking
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotEquals
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import java.io.File
import java.util.UUID
@RunWith(AndroidJUnit4::class)
class CanvasHostLifecycleTest {
@get:Rule
val activityRule = ActivityScenarioRule(CanvasLifecycleTestActivity::class.java)
@Before
fun resetMetrics() {
CanvasLifecycleTestMetrics.reset()
}
@Test
fun hiddenHostRetainsOneWebViewWithoutBlockingShellInput() {
activityRule.scenario.onActivity { activity ->
assertEquals(CanvasController.PresentationState.Unmounted, activity.controller.presentationState.value)
assertNull(activity.host)
}
val presentElapsedMs = activityRule.scenario.readActivity { activity -> activity.presentSlowPage() }
assertTrue(
"present waited for the remote page: ${presentElapsedMs}ms",
presentElapsedMs < canvasLifecycleSlowPageDelayMs / 2,
)
assertTrue("slow page never finished", activityRule.scenario.waitForPageFinished())
val firstWebView =
activityRule.scenario.readActivity { activity ->
val host = checkNotNull(activity.host)
assertEquals(1, host.childCount)
assertEquals(CanvasController.PresentationState.Visible, activity.controller.presentationState.value)
val webView = checkNotNull(activity.currentWebView())
activity.hideCanvas()
assertEquals(CanvasController.PresentationState.Hidden, activity.controller.presentationState.value)
webView
}
assertTrue(
"hidden host remained visible",
activityRule.scenario.waitUntilActivity { activity -> activity.host?.visibility == View.INVISIBLE },
)
val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
assertTrue(device.click(device.displayWidth / 2, device.displayHeight / 2))
device.waitForIdle()
activityRule.scenario.onActivity { activity ->
assertEquals(1, activity.underlayClickCount)
repeat(3) {
activity.presentFastPage()
activity.hideCanvas()
}
assertEquals(1, activity.host?.childCount)
assertTrue(firstWebView === activity.currentWebView())
}
}
@Test
fun stalePageCompletionCannotReshowCanvasAfterHide() {
activityRule.scenario.onActivity { activity -> activity.presentSlowPage() }
assertTrue(
"Canvas host was not attached",
activityRule.scenario.waitUntilActivity { activity -> activity.currentWebView() != null },
)
activityRule.scenario.onActivity { activity -> activity.hideCanvas() }
assertTrue(
"hidden host remained visible",
activityRule.scenario.waitUntilActivity { activity -> activity.host?.visibility == View.INVISIBLE },
)
assertTrue("slow page never finished", activityRule.scenario.waitForPageFinished())
activityRule.scenario.onActivity { activity ->
assertEquals(CanvasController.PresentationState.Hidden, activity.controller.presentationState.value)
assertEquals(View.INVISIBLE, checkNotNull(activity.host).visibility)
assertNotNull(activity.currentWebView())
}
}
@Test
fun rendererTerminationForgetsFailedPageAndNextShowRecreatesIt() {
activityRule.scenario.onActivity { activity -> activity.presentFastPage() }
assertTrue("initial page never finished", activityRule.scenario.waitForPageFinished())
val firstWebView =
activityRule.scenario.readActivity { activity ->
assertNotNull(activity.controller.currentUrl())
checkNotNull(activity.currentWebView())
}
val terminated =
activityRule.scenario.readActivity { activity ->
activity.currentWebView()?.webViewRenderProcess?.terminate() == true
}
assertTrue("WebView renderer did not terminate", terminated)
assertTrue(
"renderer loss did not clear the invalid WebView",
activityRule.scenario.waitUntilActivity { activity -> activity.currentWebView() == null },
)
activityRule.scenario.onActivity { activity ->
assertEquals(CanvasController.PresentationState.Hidden, activity.controller.presentationState.value)
assertNull(activity.controller.currentUrl())
assertEquals(0, activity.host?.childCount)
activity.showCanvas()
}
assertTrue(
"next show did not create a replacement WebView",
activityRule.scenario.waitUntilActivity { activity -> activity.currentWebView() != null },
)
assertTrue("replacement scaffold never finished", activityRule.scenario.waitForPageFinished())
activityRule.scenario.onActivity { activity ->
assertEquals(CanvasController.PresentationState.Visible, activity.controller.presentationState.value)
assertEquals(1, activity.host?.childCount)
assertNotEquals(firstWebView, activity.currentWebView())
}
}
@Test
fun configurationChangesKeepTheSameHostAndWebView() {
activityRule.scenario.onActivity { activity -> activity.presentFastPage() }
assertTrue("initial page never finished", activityRule.scenario.waitForPageFinished())
val firstHost = activityRule.scenario.readActivity { activity -> checkNotNull(activity.host) }
val firstWebView = activityRule.scenario.readActivity { activity -> checkNotNull(activity.currentWebView()) }
activityRule.scenario.onActivity { activity ->
activity.requestedOrientation = ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE
}
UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()).waitForIdle()
activityRule.scenario.onActivity { activity ->
assertTrue(firstHost === activity.host)
assertTrue(firstWebView === activity.currentWebView())
activity.requestedOrientation = ActivityInfo.SCREEN_ORIENTATION_PORTRAIT
}
}
@Test
fun renderedWebViewExportsPngToClipboardAndDownloads() {
activityRule.scenario.onActivity { activity -> activity.presentFastPage() }
assertTrue("widget proof page never finished", activityRule.scenario.waitForPageFinished())
val activity = activityRule.scenario.readActivity { it }
val webView = activityRule.scenario.readActivity { checkNotNull(it.currentWebView()) }
val title = "Widget export proof ${UUID.randomUUID()}"
val fileName = widgetExportFileName(title)
val clipboard = activity.getSystemService(ClipboardManager::class.java)
var downloadsUri: android.net.Uri? = null
try {
runBlocking {
exportChatWidgetImage(activity, webView, title, ChatWidgetExportDestination.Clipboard)
}
val clipboardUri = checkNotNull(clipboard.primaryClip?.getItemAt(0)?.uri)
assertEquals("image/png", activity.contentResolver.getType(clipboardUri))
activity.contentResolver.openInputStream(clipboardUri).use { input ->
val bitmap = checkNotNull(BitmapFactory.decodeStream(input))
assertTrue(bitmap.width > 0)
assertTrue(bitmap.height > 0)
bitmap.recycle()
}
runBlocking {
exportChatWidgetImage(activity, webView, title, ChatWidgetExportDestination.Downloads)
}
downloadsUri =
activity.contentResolver
.query(
MediaStore.Downloads.EXTERNAL_CONTENT_URI,
arrayOf(MediaStore.MediaColumns._ID),
"${MediaStore.MediaColumns.DISPLAY_NAME} = ?",
arrayOf(fileName),
null,
).use { cursor ->
checkNotNull(cursor)
assertTrue(cursor.moveToFirst())
ContentUris.withAppendedId(
MediaStore.Downloads.EXTERNAL_CONTENT_URI,
cursor.getLong(cursor.getColumnIndexOrThrow(MediaStore.MediaColumns._ID)),
)
}
assertEquals("image/png", activity.contentResolver.getType(downloadsUri))
activity.contentResolver.openInputStream(downloadsUri).use { input ->
val bitmap = checkNotNull(BitmapFactory.decodeStream(input))
assertTrue(bitmap.width > 0)
assertTrue(bitmap.height > 0)
bitmap.recycle()
}
} finally {
downloadsUri?.let { activity.contentResolver.delete(it, null, null) }
clipboard.clearPrimaryClip()
File(activity.cacheDir, "exports")
.walkTopDown()
.firstOrNull { it.isFile && it.name == fileName }
?.parentFile
?.deleteRecursively()
}
}
}
@RunWith(AndroidJUnit4::class)
class CanvasHostReleaseTest {
@Before
fun resetMetrics() {
CanvasLifecycleTestMetrics.reset()
}
@Test
fun activityTeardownReleasesTheHostAndWebView() {
ActivityScenario.launch(CanvasLifecycleTestActivity::class.java).use { scenario ->
scenario.onActivity { activity -> activity.presentFastPage() }
}
assertTrue(
"AndroidView onRelease was not called",
waitUntil { CanvasLifecycleTestMetrics.hostReleaseCount.get() == 1 },
)
assertEquals(1, CanvasLifecycleTestMetrics.webViewDestroyCount.get())
}
}
private inline fun <T> ActivityScenario<CanvasLifecycleTestActivity>.readActivity(crossinline block: (CanvasLifecycleTestActivity) -> T): T {
var result: Result<T>? = null
onActivity { activity -> result = runCatching { block(activity) } }
return checkNotNull(result).getOrThrow()
}
private fun ActivityScenario<CanvasLifecycleTestActivity>.waitForPageFinished(): Boolean = waitUntilActivity { activity -> activity.currentWebView()?.progress == 100 }
private inline fun ActivityScenario<CanvasLifecycleTestActivity>.waitUntilActivity(
timeoutMs: Long = 5_000L,
crossinline predicate: (CanvasLifecycleTestActivity) -> Boolean,
): Boolean =
waitUntil(timeoutMs) {
readActivity(predicate)
}
private fun waitUntil(
timeoutMs: Long = 5_000L,
predicate: () -> Boolean,
): Boolean {
val deadline = SystemClock.elapsedRealtime() + timeoutMs
while (SystemClock.elapsedRealtime() < deadline) {
if (predicate()) return true
SystemClock.sleep(20)
}
return predicate()
}
apps/android/app/src/debug/AndroidManifest.xml
-5
View File
@@ -12,11 +12,6 @@
android:name="androidx.activity.ComponentActivity"
android:exported="false"
android:theme="@style/Theme.OpenClawNode" />
<activity
android:name=".ui.CanvasLifecycleTestActivity"
android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|uiMode|density|keyboard|keyboardHidden|navigation"
android:exported="false"
android:theme="@android:style/Theme.Material.Light.NoActionBar" />
<receiver
android:name=".VoiceE2eReceiver"
android:permission="${applicationId}.permission.RUN_VOICE_E2E"
apps/android/app/src/debug/java/ai/openclaw/app/ui/CanvasLifecycleTestActivity.kt
-129
View File
@@ -1,129 +0,0 @@
package ai.openclaw.app.ui
import ai.openclaw.app.node.CanvasController
import android.os.Bundle
import android.os.SystemClock
import android.util.Base64
import android.webkit.WebView
import android.widget.Button
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.viewinterop.AndroidView
import java.util.concurrent.atomic.AtomicInteger
const val canvasLifecycleSlowPageDelayMs = 2_000L
class CanvasLifecycleTestActivity : ComponentActivity() {
val controller = CanvasController()
internal var host: CanvasHostView? = null
internal set
var underlayClickCount: Int = 0
private set
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContent {
CanvasLifecycleTestContent(
activity = this,
onUnderlayClick = { underlayClickCount += 1 },
)
}
}
fun presentSlowPage(): Long = presentHtml(slowPageHtml)
fun presentFastPage(): Long = presentHtml("<html><body>ready</body></html>")
fun hideCanvas() {
controller.hide()
}
fun showCanvas() {
controller.show()
}
fun currentWebView(): WebView? = host?.currentWebView
private fun presentHtml(html: String): Long {
val startedAt = SystemClock.elapsedRealtime()
val encoded = Base64.encodeToString(html.toByteArray(Charsets.UTF_8), Base64.NO_WRAP)
controller.navigate("data:text/html;base64,$encoded")
controller.show()
return SystemClock.elapsedRealtime() - startedAt
}
}
@Composable
private fun CanvasLifecycleTestContent(
activity: CanvasLifecycleTestActivity,
onUnderlayClick: () -> Unit,
) {
val state by activity.controller.presentationState.collectAsState()
Box(modifier = Modifier.fillMaxSize()) {
AndroidView(
factory = { context ->
Button(context).apply {
setOnClickListener { onUnderlayClick() }
}
},
modifier = Modifier.fillMaxSize(),
)
if (state != CanvasController.PresentationState.Unmounted) {
AndroidView(
factory = { context ->
CanvasHostView(
context = context,
controller = activity.controller,
isTrustedPage = { false },
onA2uiMessage = {},
).also { host ->
activity.host = host
host.updateVisible(state == CanvasController.PresentationState.Visible)
}
},
update = { host ->
host.updateVisible(state == CanvasController.PresentationState.Visible)
},
modifier = Modifier.fillMaxSize(),
onRelease = { host ->
val hadWebView = host.currentWebView != null
host.release()
if (activity.host === host) activity.host = null
CanvasLifecycleTestMetrics.hostReleaseCount.incrementAndGet()
if (hadWebView) CanvasLifecycleTestMetrics.webViewDestroyCount.incrementAndGet()
},
)
}
}
}
object CanvasLifecycleTestMetrics {
val hostReleaseCount = AtomicInteger()
val webViewDestroyCount = AtomicInteger()
fun reset() {
hostReleaseCount.set(0)
webViewDestroyCount.set(0)
}
}
private val slowPageHtml =
"""
<html>
<body>
<script>
const deadline = Date.now() + $canvasLifecycleSlowPageDelayMs;
while (Date.now() < deadline) {}
</script>
ready
</body>
</html>
""".trimIndent()
apps/android/app/src/main/java/ai/openclaw/app/AssistantLaunch.kt
-1
View File
@@ -22,7 +22,6 @@ enum class HomeDestination {
Connect,
Chat,
Voice,
Screen,
Settings,
}
apps/android/app/src/main/java/ai/openclaw/app/MainViewModel.kt
-38
View File
@@ -28,7 +28,6 @@ import ai.openclaw.app.gateway.GatewayRegistryEntry
import ai.openclaw.app.gateway.GatewayRegistryEntryKind
import ai.openclaw.app.gateway.GatewayUpdateAvailableSummary
import ai.openclaw.app.node.CameraCaptureManager
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.node.SmsManager
import ai.openclaw.app.systemagent.SystemAgentChatState
import ai.openclaw.app.ui.GatewayConnectPlan
@@ -493,13 +492,6 @@ class MainViewModel private constructor(
.flatMapLatest { runtime -> flowOf(runtime != null) }
.stateIn(viewModelScope, SharingStarted.Eagerly, false)
val canvasCurrentUrl: StateFlow<String?> = runtimeState(initial = null) { it.canvas.currentUrl }
val canvasPresentationState: StateFlow<CanvasController.PresentationState> =
runtimeState(initial = CanvasController.PresentationState.Unmounted) { it.canvas.presentationState }
val canvasA2uiHydrated: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasA2uiHydrated }
val canvasRehydratePending: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasRehydratePending }
val canvasRehydrateErrorText: StateFlow<String?> = runtimeState(initial = null) { it.canvasRehydrateErrorText }
val gateways: StateFlow<List<GatewayEndpoint>> = runtimeState(initial = emptyList()) { it.gateways }
val discoveryStatusText: StateFlow<String> = runtimeState(initial = "Searching…") { it.discoveryStatusText }
val notificationForwardingEnabled: StateFlow<Boolean> = prefs.notificationForwardingEnabled
@@ -617,7 +609,6 @@ class MainViewModel private constructor(
val activeGatewayStableId: StateFlow<String?> = prefs.gatewayRegistry.activeStableId
val connectedGatewayStableIds: StateFlow<List<String>> = prefs.gatewayRegistry.connectedStableIds
val onboardingCompleted: StateFlow<Boolean> = prefs.onboardingCompleted
val canvasDebugStatusEnabled: StateFlow<Boolean> = prefs.canvasDebugStatusEnabled
val installedAppsSharingEnabled: StateFlow<Boolean> = prefs.installedAppsSharingEnabled
val accessibilityControlEnabled: StateFlow<Boolean> = prefs.accessibilityControlEnabled
val speakerEnabled: StateFlow<Boolean> = prefs.speakerEnabled
@@ -695,9 +686,6 @@ class MainViewModel private constructor(
val execApprovalsErrorText: StateFlow<String?> = runtimeState(initial = null) { it.execApprovalsErrorText }
val execApprovalsNotice: StateFlow<GatewayExecApprovalNotice?> = runtimeState(initial = null) { it.execApprovalsNotice }
val canvas: CanvasController
get() = ensureRuntime().canvas
val camera: CameraCaptureManager
get() = ensureRuntime().camera
@@ -935,10 +923,6 @@ class MainViewModel private constructor(
_startOnboardingAtGatewaySetup.value = false
}
fun setCanvasDebugStatusEnabled(value: Boolean) {
prefs.setCanvasDebugStatusEnabled(value)
}
fun grantInstalledAppsDisclosureConsent() {
ensureRuntime().grantInstalledAppsDisclosureConsent()
}
@@ -1384,12 +1368,6 @@ class MainViewModel private constructor(
runtimeRef.value?.declineGatewayTrustPrompt()
}
fun handleCanvasA2UIActionFromWebView(payloadJson: String) {
ensureRuntime().handleCanvasA2UIActionFromWebView(payloadJson)
}
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean = ensureRuntime().isTrustedCanvasActionUrl(rawUrl)
internal suspend fun resolveInlineWidgetResource(
path: String,
failedResource: ChatWidgetResource?,
@@ -1403,22 +1381,6 @@ class MainViewModel private constructor(
playbackRendition: Boolean,
) = ensureRuntime().loadChatMediaArtifact(artifactId, kind, playbackRendition)
fun requestCanvasRehydrate(source: String = "screen_tab") {
ensureRuntime().requestCanvasRehydrate(source = source, force = true)
}
fun showCanvas() {
ensureRuntime().canvas.show()
}
fun hideCanvas() {
runtimeRef.value?.canvas?.hide()
}
fun refreshHomeCanvasOverviewIfConnected() {
ensureRuntime().refreshHomeCanvasOverviewIfConnected()
}
fun refreshModelCatalog() {
ensureRuntime().refreshModelCatalog()
}
apps/android/app/src/main/java/ai/openclaw/app/NodeApp.kt
+2 -2
View File
@@ -120,8 +120,8 @@ class NodeApp : Application() {
override fun onConfigurationChanged(newConfig: Configuration) {
super.onConfigurationChanged(newConfig)
// The process runtime survives Activity recreation, so retained text and
// serialized Home Canvas state need an explicit locale refresh signal.
// The process runtime survives Activity recreation, so retained text needs an
// explicit locale refresh signal.
NativeStringResources.setConfigurationLocales(newConfig)
notifyNativeLocaleChanged()
}
apps/android/app/src/main/java/ai/openclaw/app/NodeRuntime.kt
+1 -404
View File
@@ -62,17 +62,14 @@ import ai.openclaw.app.gateway.normalizeGatewayTlsFingerprintInput
import ai.openclaw.app.gateway.parseChatSendAck
import ai.openclaw.app.gateway.probeGatewayTlsFingerprint
import ai.openclaw.app.i18n.NativeText
import ai.openclaw.app.i18n.nativeLocaleChanges
import ai.openclaw.app.i18n.nativeString
import ai.openclaw.app.i18n.nativeText
import ai.openclaw.app.i18n.resolveOptionalNativeText
import ai.openclaw.app.i18n.verbatimText
import ai.openclaw.app.node.A2UIHandler
import ai.openclaw.app.node.CalendarHandler
import ai.openclaw.app.node.CallLogHandler
import ai.openclaw.app.node.CameraCaptureManager
import ai.openclaw.app.node.CameraHandler
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.node.ConnectionManager
import ai.openclaw.app.node.ContactsHandler
import ai.openclaw.app.node.DEFAULT_SEAM_COLOR_ARGB
@@ -87,7 +84,6 @@ import ai.openclaw.app.node.MotionHandler
import ai.openclaw.app.node.NodePresenceAliveBeacon
import ai.openclaw.app.node.NotificationsHandler
import ai.openclaw.app.node.PhotosHandler
import ai.openclaw.app.node.Quad
import ai.openclaw.app.node.SmsHandler
import ai.openclaw.app.node.SmsManager
import ai.openclaw.app.node.SystemHandler
@@ -97,7 +93,6 @@ import ai.openclaw.app.node.asStringOrNull
import ai.openclaw.app.node.invokeErrorFromThrowable
import ai.openclaw.app.node.parseHexColorArgb
import ai.openclaw.app.node.readAndroidPermissionSnapshot
import ai.openclaw.app.protocol.OpenClawCanvasA2UIAction
import ai.openclaw.app.systemagent.SystemAgentChatController
import ai.openclaw.app.systemagent.SystemAgentChatState
import ai.openclaw.app.systemagent.SystemAgentGatewayAccess
@@ -162,7 +157,6 @@ import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeout
import kotlinx.coroutines.withTimeoutOrNull
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonElement
@@ -903,7 +897,6 @@ class NodeRuntime private constructor(
private val appContext = context.applicationContext
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
private val deviceAuthStore = DeviceAuthStore(prefs)
val canvas = CanvasController()
val camera = CameraCaptureManager(appContext) { prefs.preferredCameraFacing.value }
val location = LocationCaptureManager(appContext)
val sms = SmsManager(appContext)
@@ -1038,12 +1031,6 @@ class NodeRuntime private constructor(
private val mobileUiHandler = MobileUiHandler()
private var lastMobileUiConnected = mobileUiHandler.isConnected.value
private val a2uiHandler: A2UIHandler =
A2UIHandler(
canvas = canvas,
json = json,
)
private val connectionManager: ConnectionManager =
ConnectionManager(
prefs = prefs,
@@ -1078,7 +1065,6 @@ class NodeRuntime private constructor(
private val invokeDispatcher: InvokeDispatcher =
InvokeDispatcher(
canvas = canvas,
cameraHandler = cameraHandler,
locationHandler = locationHandler,
deviceHandler = deviceHandler,
@@ -1099,7 +1085,6 @@ class NodeRuntime private constructor(
calendarHandler = calendarHandler,
motionHandler = motionHandler,
smsHandler = smsHandlerImpl,
a2uiHandler = a2uiHandler,
debugHandler = debugHandler,
callLogHandler = callLogHandler,
mobileUiHandler = mobileUiHandler,
@@ -1114,12 +1099,6 @@ class NodeRuntime private constructor(
photosAvailable = { SensitiveFeatureConfig.photosEnabled },
installedAppsSharingEnabled = { installedAppsSharingEnabled.value },
debugBuild = { BuildConfig.DEBUG },
onCanvasA2uiPush = {
_canvasA2uiHydrated.value = true
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = null
},
onCanvasA2uiReset = { _canvasA2uiHydrated.value = false },
motionActivityAvailable = { motionHandler.isActivityAvailable() },
motionPedometerAvailable = { motionHandler.isPedometerAvailable() },
mobileUiAvailable = {
@@ -1194,13 +1173,6 @@ class NodeRuntime private constructor(
private val _cameraHud = MutableStateFlow<CameraHudState?>(null)
val cameraHud: StateFlow<CameraHudState?> = _cameraHud.asStateFlow()
private val _canvasA2uiHydrated = MutableStateFlow(false)
val canvasA2uiHydrated: StateFlow<Boolean> = _canvasA2uiHydrated.asStateFlow()
private val _canvasRehydratePending = MutableStateFlow(false)
val canvasRehydratePending: StateFlow<Boolean> = _canvasRehydratePending.asStateFlow()
private val _canvasRehydrateErrorText = MutableStateFlow<NativeText?>(null)
val canvasRehydrateErrorText: StateFlow<String?> = _canvasRehydrateErrorText.resolveOptionalNativeText()
private val _serverName = MutableStateFlow<String?>(null)
val serverName: StateFlow<String?> = _serverName.asStateFlow()
@@ -1410,9 +1382,6 @@ class NodeRuntime private constructor(
private val voiceReplySpeechDepth = AtomicInteger(0)
private val voiceCapturePreparationMutex = Mutex()
private var didAutoRequestCanvasRehydrate = false
private val canvasRehydrateSeq = AtomicLong(0)
@Volatile private var nodePresenceAliveLastSuccessAtMs: Long? = null
private var operatorConnected = false
private var operatorStatusText: String = "Offline"
@@ -1458,7 +1427,6 @@ class NodeRuntime private constructor(
subscribeOperatorSessionEvents()
refreshWakeWordsFromGateway()
refreshExecApprovalsFromGateway()
refreshHomeCanvasOverviewIfConnected()
if (voiceReplySpeakerLazy.isInitialized()) {
voiceReplySpeaker.refreshConfig()
}
@@ -1814,17 +1782,12 @@ class NodeRuntime private constructor(
deviceAuthStore = deviceAuthStore,
onConnected = {
recordConnectedGateway()
didAutoRequestCanvasRehydrate = false
_canvasA2uiHydrated.value = false
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = null
updateStatus {
nodeConnectionProblem = null
_nodeConnected.value = true
nodeStatusText = "Connected"
}
notificationOutbox.onConnected()
resetLocalCanvas()
publishNodePresenceAliveBeacon(NodePresenceAliveBeacon.Trigger.Connect)
val endpoint = connectedEndpoint
val auth = activeGatewayAuth
@@ -1836,16 +1799,11 @@ class NodeRuntime private constructor(
},
onDisconnected = { message ->
invalidateNodeCapabilityApprovalState()
didAutoRequestCanvasRehydrate = false
_canvasA2uiHydrated.value = false
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = null
updateStatus {
_nodeConnected.value = false
nodeStatusText = message
nodeConnectionProblem = gatewayProblemAfterDisconnect(nodeConnectionProblem, message)
}
resetLocalCanvas()
},
onConnectFailure = { error, pauseReconnect ->
updateStatus {
@@ -2319,7 +2277,6 @@ class NodeRuntime private constructor(
} else {
chat.applyMainSessionKey(resolvedKey)
}
updateHomeCanvasState()
}
private fun prepareMainSessionKey(agentId: String?): String {
@@ -2328,7 +2285,6 @@ class NodeRuntime private constructor(
talkMode.setMainSessionKey(resolvedKey)
if (_mainSessionKey.value != resolvedKey) {
_mainSessionKey.value = resolvedKey
updateHomeCanvasState()
}
chat.prepareMainSessionKey(resolvedKey)
return resolvedKey
@@ -2340,7 +2296,6 @@ class NodeRuntime private constructor(
_mainSessionKey.value = resolvedKey
chat.prepareAndSelectMainSessionKey(resolvedKey)
chat.onGatewayConnected(mainSessionBinding(resolvedKey))
updateHomeCanvasState()
}
private fun mainSessionBinding(sessionKey: String): MainSessionBinding =
@@ -2367,7 +2322,6 @@ class NodeRuntime private constructor(
_statusText.value = display.statusText
_gatewayConnectionProblem.value = display.problem
}
updateHomeCanvasState()
}
private fun setStandaloneGatewayStatus(statusText: String) {
@@ -2378,7 +2332,6 @@ class NodeRuntime private constructor(
_statusText.value = display.statusText
_gatewayConnectionProblem.value = display.problem
}
updateHomeCanvasState()
}
private fun gatewayConnectionProblem(
@@ -2406,39 +2359,10 @@ class NodeRuntime private constructor(
return if (trimmed.isEmpty()) "main" else trimmed
}
private fun resetLocalCanvas() {
_canvasA2uiHydrated.value = false
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = null
canvas.navigate("")
}
private fun launchGatewayRefresh(refresh: suspend () -> Unit) {
if (mode != NodeRuntimeMode.ScreenshotFixture) scope.launch { refresh() }
}
fun refreshHomeCanvasOverviewIfConnected() {
if (mode == NodeRuntimeMode.ScreenshotFixture) return
if (!operatorConnected) {
updateHomeCanvasState()
return
}
scope.launch {
refreshBrandingFromGateway()
refreshAgentsFromGateway()
refreshModelCatalogFromGateway()
refreshProviderModelsFromGateway()
refreshTalkSetupReadinessFromGateway()
refreshCronFromGateway()
refreshUsageFromGateway()
refreshSkillsFromGateway()
refreshNodesDevicesFromGateway()
refreshChannelsFromGateway()
refreshDreamingFromGateway()
refreshHealthLogsFromGateway()
}
}
fun refreshModelCatalog() = launchGatewayRefresh { refreshModelCatalogFromGateway() }
fun refreshProviderModels() = launchGatewayRefresh { refreshProviderModelsFromGateway() }
@@ -2828,62 +2752,6 @@ class NodeRuntime private constructor(
fun refreshHealthLogs() = launchGatewayRefresh { refreshHealthLogsFromGateway() }
fun requestCanvasRehydrate(
source: String = "manual",
force: Boolean = true,
) {
val gatewayId = connectedEndpoint?.stableId
scope.launch {
if (gatewayId == null || !_nodeConnected.value) {
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = nativeText("Node offline. Reconnect and retry.")
return@launch
}
if (!force && didAutoRequestCanvasRehydrate) return@launch
didAutoRequestCanvasRehydrate = true
val requestId = canvasRehydrateSeq.incrementAndGet()
_canvasRehydratePending.value = true
_canvasRehydrateErrorText.value = null
val sessionKey = resolveMainSessionKey()
val prompt =
"Restore canvas now for session=$sessionKey source=$source. " +
"If existing A2UI state exists, replay it immediately. " +
"If not, create and render a compact mobile-friendly dashboard in Canvas."
val sent =
nodeSession.sendNodeEventForEndpoint(
expectedEndpointStableId = gatewayId,
event = "agent.request",
payloadJson =
buildJsonObject {
put("message", JsonPrimitive(prompt))
put("sessionKey", JsonPrimitive(sessionKey))
put("thinking", JsonPrimitive("low"))
put("deliver", JsonPrimitive(false))
}.toString(),
)
if (!sent) {
if (!force) {
didAutoRequestCanvasRehydrate = false
}
if (canvasRehydrateSeq.get() == requestId) {
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = nativeText("Failed to request restore. Tap to retry.")
}
Log.w("OpenClawCanvas", "canvas rehydrate request failed ($source): transport unavailable")
return@launch
}
scope.launch {
delay(20_000)
if (canvasRehydrateSeq.get() != requestId) return@launch
if (!_canvasRehydratePending.value) return@launch
if (_canvasA2uiHydrated.value) return@launch
_canvasRehydratePending.value = false
_canvasRehydrateErrorText.value = nativeText("No canvas update yet. Tap to retry.")
}
}
}
val instanceId: StateFlow<String> = prefs.instanceId
val displayName: StateFlow<String> = prefs.displayName
val cameraEnabled: StateFlow<Boolean> = prefs.cameraEnabled
@@ -2959,7 +2827,6 @@ class NodeRuntime private constructor(
val pairedGateways: StateFlow<List<GatewayRegistryEntry>> = prefs.gatewayRegistry.entries
val activeGatewayStableId: StateFlow<String?> = prefs.gatewayRegistry.activeStableId
val connectedGatewayStableIds: StateFlow<List<String>> = prefs.gatewayRegistry.connectedStableIds
val canvasDebugStatusEnabled: StateFlow<Boolean> = prefs.canvasDebugStatusEnabled
val installedAppsSharingEnabled: StateFlow<Boolean> = prefs.installedAppsSharingEnabled
val notificationForwardingEnabled: StateFlow<Boolean> = prefs.notificationForwardingEnabled
val notificationForwardingMode: StateFlow<NotificationPackageFilterMode> =
@@ -3164,28 +3031,6 @@ class NodeRuntime private constructor(
}
}
scope.launch {
combine(
canvasDebugStatusEnabled,
statusText,
serverName,
remoteAddress,
) { debugEnabled, status, server, remote ->
Quad(debugEnabled, status, server, remote)
}.distinctUntilChanged()
.collect { (debugEnabled, status, server, remote) ->
canvas.setDebugStatusEnabled(debugEnabled)
if (!debugEnabled) return@collect
canvas.setDebugStatus(status, server ?: remote)
}
}
scope.launch {
nativeLocaleChanges.drop(1).collect {
updateHomeCanvasState()
}
}
scope.launch {
chatModelCatalog.drop(1).distinctUntilChanged().collect {
// Chat metadata arrives after the connection event. Invalidate the Watch snapshot so
@@ -3193,8 +3038,6 @@ class NodeRuntime private constructor(
if (operatorSession.isReady()) wearProxyBridge()?.publishResync()
}
}
updateHomeCanvasState()
}
/** Updates foreground state and triggers reconnect/presence behavior on app visibility changes. */
@@ -3534,10 +3377,6 @@ class NodeRuntime private constructor(
prefs.setManualTls(value)
}
fun setCanvasDebugStatusEnabled(value: Boolean) {
prefs.setCanvasDebugStatusEnabled(value)
}
fun grantInstalledAppsDisclosureConsent() {
if (prefs.installedAppsSharingEnabled.value) return
prefs.grantInstalledAppsDisclosureConsent()
@@ -5014,93 +4853,6 @@ class NodeRuntime private constructor(
_pendingGatewayTrust.value = null
}
fun handleCanvasA2UIActionFromWebView(payloadJson: String) {
val gatewayId = connectedEndpoint?.stableId
scope.launch {
val trimmed = payloadJson.trim()
if (trimmed.isEmpty()) return@launch
val root =
try {
json.parseToJsonElement(trimmed).asObjectOrNull() ?: return@launch
} catch (_: Throwable) {
return@launch
}
val userActionObj = (root["userAction"] as? JsonObject) ?: root
val actionId =
(userActionObj["id"] as? JsonPrimitive)?.content?.trim().orEmpty().ifEmpty {
java.util.UUID
.randomUUID()
.toString()
}
val name = OpenClawCanvasA2UIAction.extractActionName(userActionObj) ?: return@launch
val surfaceId =
(userActionObj["surfaceId"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
.ifEmpty { "main" }
val sourceComponentId =
(userActionObj["sourceComponentId"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
.ifEmpty { "-" }
val contextJson = (userActionObj["context"] as? JsonObject)?.toString()
val sessionKey = resolveMainSessionKey()
val message =
OpenClawCanvasA2UIAction.formatAgentMessage(
actionName = name,
sessionKey = sessionKey,
surfaceId = surfaceId,
sourceComponentId = sourceComponentId,
host = displayName.value,
instanceId = instanceId.value.lowercase(),
contextJson = contextJson,
)
val connected = _nodeConnected.value
var error: String? = null
if (connected && gatewayId != null) {
val sent =
nodeSession.sendNodeEventForEndpoint(
expectedEndpointStableId = gatewayId,
event = "agent.request",
payloadJson =
buildJsonObject {
put("message", JsonPrimitive(message))
put("sessionKey", JsonPrimitive(sessionKey))
put("thinking", JsonPrimitive("low"))
put("deliver", JsonPrimitive(false))
put("key", JsonPrimitive(actionId))
}.toString(),
)
if (!sent) {
error = "send failed"
}
} else {
error = "gateway not connected"
}
try {
canvas.eval(
OpenClawCanvasA2UIAction.jsDispatchA2UIActionStatus(
actionId = actionId,
ok = connected && error == null,
error = error,
),
)
} catch (_: Throwable) {
// ignore
}
}
}
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean = a2uiHandler.isTrustedCanvasActionUrl(rawUrl)
internal suspend fun resolveInlineWidgetResource(
path: String,
failedResource: ChatWidgetResource?,
@@ -5287,7 +5039,7 @@ class NodeRuntime private constructor(
if (normalizedAgentId.isEmpty()) return
stopMessageSpeech()
// Agent selection owns every main-session consumer; switching chat alone would
// leave Talk mode and the home canvas bound to the previous agent.
// leave Talk mode bound to the previous agent.
selectedChatAgentId = normalizedAgentId
selectMainSessionKey(normalizedAgentId)
}
@@ -5784,7 +5536,6 @@ class NodeRuntime private constructor(
val parsed = parseHexColorArgb(raw)
publishGatewayData(gatewayScope) {
_seamColorArgb.value = parsed ?: DEFAULT_SEAM_COLOR_ARGB
updateHomeCanvasState()
}
} catch (_: Throwable) {
// ignore
@@ -5837,7 +5588,6 @@ class NodeRuntime private constructor(
val selectedAgentId = selectedChatAgentId?.takeIf { id -> agents.any { it.id == id } }
selectedChatAgentId = selectedAgentId
syncMainSessionKey(selectedAgentId ?: resolveAgentIdFromMainSessionKey(mainKey) ?: gatewayDefaultAgentId.value)
updateHomeCanvasState()
}
} catch (_: Throwable) {
// ignore
@@ -8353,84 +8103,6 @@ class NodeRuntime private constructor(
?: nativeText("No prompt")
}
private fun updateHomeCanvasState() {
val payload =
try {
json.encodeToString(makeHomeCanvasPayload())
} catch (_: Throwable) {
null
}
canvas.updateHomeCanvasState(payload)
}
private fun makeHomeCanvasPayload(): HomeCanvasPayload {
val state = resolveHomeCanvasGatewayState()
val gatewayName = normalized(_serverName.value)
val gatewayAddress = normalized(_remoteAddress.value)
val gatewayLabel = gatewayName ?: gatewayAddress ?: nativeString("Gateway")
val activeAgentId = resolveActiveAgentId()
val agents = homeCanvasAgents(activeAgentId)
return when (state) {
HomeCanvasGatewayState.Connected ->
HomeCanvasPayload(
gatewayState = "connected",
eyebrow = nativeString("Connected to \$gatewayLabel", gatewayLabel),
title = nativeString("Your agents are ready"),
subtitle =
nativeString("This phone stays dormant until the gateway needs it, then wakes, syncs, and goes back to sleep."),
gatewayLabel = gatewayLabel,
activeAgentName = resolveActiveAgentName(activeAgentId),
activeAgentBadge = agents.firstOrNull { it.isActive }?.badge ?: "OC",
activeAgentCaption = nativeString("Selected on this phone"),
agentCount = agents.size,
agents = agents.take(6),
footer = nativeString("The overview refreshes on reconnect and when this screen opens."),
)
HomeCanvasGatewayState.Connecting ->
HomeCanvasPayload(
gatewayState = "connecting",
eyebrow = nativeString("Reconnecting"),
title = nativeString("OpenClaw is syncing back up"),
subtitle =
nativeString("The gateway session is coming back online. Agent shortcuts should settle automatically in a moment."),
gatewayLabel = gatewayLabel,
activeAgentName = resolveActiveAgentName(activeAgentId),
activeAgentBadge = "OC",
activeAgentCaption = nativeString("Gateway session in progress"),
agentCount = agents.size,
agents = agents.take(4),
footer = nativeString("If the gateway is reachable, reconnect should complete without intervention."),
)
HomeCanvasGatewayState.Error, HomeCanvasGatewayState.Offline ->
HomeCanvasPayload(
gatewayState = if (state == HomeCanvasGatewayState.Error) "error" else "offline",
eyebrow = nativeString("Welcome to OpenClaw"),
title = nativeString("Your phone stays quiet until it is needed"),
subtitle =
nativeString("Pair this device to your gateway to wake it only for real work, keep a live agent overview handy, and avoid battery-draining background loops."),
gatewayLabel = gatewayLabel,
activeAgentName = nativeString("Main"),
activeAgentBadge = "OC",
activeAgentCaption = nativeString("Connect to load your agents"),
agentCount = agents.size,
agents = agents.take(4),
footer = nativeString("When connected, the gateway can wake the phone with a silent push instead of holding an always-on session."),
)
}
}
private fun resolveHomeCanvasGatewayState(): HomeCanvasGatewayState {
val display = gatewayConnectionDisplay.value
val lower = display.statusText.trim().lowercase()
return when {
display.isConnected -> HomeCanvasGatewayState.Connected
lower.contains("connecting") || lower.contains("reconnecting") -> HomeCanvasGatewayState.Connecting
lower.contains("error") || lower.contains("failed") -> HomeCanvasGatewayState.Error
else -> HomeCanvasGatewayState.Offline
}
}
private fun resolveActiveAgentId(): String {
val mainKey = _mainSessionKey.value.trim()
if (mainKey.startsWith("agent:")) {
@@ -8440,50 +8112,6 @@ class NodeRuntime private constructor(
return gatewayDefaultAgentId.value?.trim().orEmpty()
}
private fun resolveActiveAgentName(activeAgentId: String): String {
if (activeAgentId.isNotEmpty()) {
gatewayAgents.value.firstOrNull { it.id == activeAgentId }?.let { agent ->
return normalized(agent.name) ?: agent.id
}
return activeAgentId
}
return gatewayAgents.value.firstOrNull()?.let { normalized(it.name) ?: it.id } ?: nativeString("Main")
}
private fun homeCanvasAgents(activeAgentId: String): List<HomeCanvasAgentCard> {
val defaultAgentId = gatewayDefaultAgentId.value?.trim().orEmpty()
return gatewayAgents.value
.map { agent ->
val isActive = activeAgentId.isNotEmpty() && agent.id == activeAgentId
val isDefault = defaultAgentId.isNotEmpty() && agent.id == defaultAgentId
HomeCanvasAgentCard(
id = agent.id,
name = normalized(agent.name) ?: agent.id,
badge = homeCanvasBadge(agent),
caption =
when {
isActive -> nativeString("Active on this phone")
isDefault -> nativeString("Default agent")
else -> nativeString("Ready")
},
isActive = isActive,
)
}.sortedWith(compareByDescending<HomeCanvasAgentCard> { it.isActive }.thenBy { it.name.lowercase() })
}
private fun homeCanvasBadge(agent: GatewayAgentSummary): String {
val emoji = normalized(agent.emoji)
if (emoji != null) return emoji
val initials =
(normalized(agent.name) ?: agent.id)
.split(' ', '-', '_')
.filter { it.isNotBlank() }
.take(2)
.mapNotNull { token -> token.uppercaseFirstGraphemeOrNull() }
.joinToString("")
return if (initials.isNotEmpty()) initials else "OC"
}
private fun normalized(value: String?): String? {
val trimmed = value?.trim().orEmpty()
return trimmed.ifEmpty { null }
@@ -8707,13 +8335,6 @@ internal fun gatewayControlPageBaseUrl(endpoint: GatewayEndpoint): String {
return "$scheme://${formatGatewayAuthority(endpoint.host, endpoint.port)}"
}
private enum class HomeCanvasGatewayState {
Connected,
Connecting,
Error,
Offline,
}
data class GatewayModelSummary(
val id: String,
val name: String,
@@ -9218,30 +8839,6 @@ fun channelDisplayLabel(channel: String): String =
.ifBlank { "Channel" }
}
@Serializable
private data class HomeCanvasPayload(
val gatewayState: String,
val eyebrow: String,
val title: String,
val subtitle: String,
val gatewayLabel: String,
val activeAgentName: String,
val activeAgentBadge: String,
val activeAgentCaption: String,
val agentCount: Int,
val agents: List<HomeCanvasAgentCard>,
val footer: String,
)
@Serializable
private data class HomeCanvasAgentCard(
val id: String,
val name: String,
val badge: String,
val caption: String,
val isActive: Boolean,
)
private fun gatewayControlPageTlsFingerprint(
prefs: SecurePrefs,
endpoint: GatewayEndpoint,
apps/android/app/src/main/java/ai/openclaw/app/SecurePrefs.kt
-9
View File
@@ -148,10 +148,6 @@ class SecurePrefs(
)
val lastDiscoveredStableId: StateFlow<String> = _lastDiscoveredStableId
private val _canvasDebugStatusEnabled =
MutableStateFlow(plainPrefs.getBoolean("canvas.debugStatusEnabled", false))
val canvasDebugStatusEnabled: StateFlow<Boolean> = _canvasDebugStatusEnabled
private val _installedAppsSharingEnabled =
MutableStateFlow(loadInstalledAppsSharingEnabled())
val installedAppsSharingEnabled: StateFlow<Boolean> = _installedAppsSharingEnabled
@@ -298,11 +294,6 @@ class SecurePrefs(
_onboardingCompleted.value = value
}
fun setCanvasDebugStatusEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("canvas.debugStatusEnabled", value) }
_canvasDebugStatusEnabled.value = value
}
fun grantInstalledAppsDisclosureConsent() {
plainPrefs.edit {
putBoolean(installedAppsSharingEnabledKey, true)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayDiscovery.kt
-4
View File
@@ -286,7 +286,6 @@ class GatewayDiscovery(
val lanHost = txt(resolved, "lanHost")
val tailnetDns = txt(resolved, "tailnetDns")
val gatewayPort = txtInt(resolved, "gatewayPort")
val canvasPort = txtInt(resolved, "canvasPort")
val tlsEnabled = txtBool(resolved, "gatewayTls")
val tlsFingerprint = txt(resolved, "gatewayTlsSha256")
val id = stableId(serviceName, "local.")
@@ -300,7 +299,6 @@ class GatewayDiscovery(
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
canvasPort = canvasPort,
tlsEnabled = tlsEnabled,
tlsFingerprintSha256 = tlsFingerprint,
)
@@ -408,7 +406,6 @@ class GatewayDiscovery(
val lanHost = txtValue(txt, "lanHost")
val tailnetDns = txtValue(txt, "tailnetDns")
val gatewayPort = txtIntValue(txt, "gatewayPort")
val canvasPort = txtIntValue(txt, "canvasPort")
val tlsEnabled = txtBoolValue(txt, "gatewayTls")
val tlsFingerprint = txtValue(txt, "gatewayTlsSha256")
val id = stableId(instanceName, domain)
@@ -421,7 +418,6 @@ class GatewayDiscovery(
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
canvasPort = canvasPort,
tlsEnabled = tlsEnabled,
tlsFingerprintSha256 = tlsFingerprint,
)
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewayEndpoint.kt
-1
View File
@@ -9,7 +9,6 @@ data class GatewayEndpoint(
val lanHost: String? = null,
val tailnetDns: String? = null,
val gatewayPort: Int? = null,
val canvasPort: Int? = null,
val tlsEnabled: Boolean = false,
val tlsFingerprintSha256: String? = null,
val contextPath: String = "",
apps/android/app/src/main/java/ai/openclaw/app/gateway/GatewaySession.kt
+1 -4
View File
@@ -1980,11 +1980,8 @@ class GatewaySession(
?: endpoint.host.trim()
if (fallbackHost.isEmpty()) return trimmed.ifBlank { null }
// For TLS connections, use the connected endpoint's scheme/port instead of raw canvas metadata.
val fallbackScheme = if (isTlsConnection) "https" else scheme
// For TLS, always use the connected endpoint port.
val fallbackPort = if (isTlsConnection) endpoint.port else (endpoint.canvasPort ?: endpoint.port)
return buildCanvasUrl(host = fallbackHost, scheme = fallbackScheme, port = fallbackPort, suffix = suffix)
return buildCanvasUrl(host = fallbackHost, scheme = fallbackScheme, port = endpoint.port, suffix = suffix)
}
private fun buildCanvasUrl(
apps/android/app/src/main/java/ai/openclaw/app/node/A2UIHandler.kt
-149
View File
@@ -1,149 +0,0 @@
package ai.openclaw.app.node
import kotlinx.coroutines.delay
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
/**
* Android bridge for applying gateway A2UI messages to the canvas WebView.
*/
class A2UIHandler(
private val canvas: CanvasController,
private val json: Json,
) {
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean = CanvasActionTrust.isTrustedCanvasActionUrl(rawUrl)
suspend fun ensureA2uiReady(): Boolean {
val alreadyOnA2uiHost = canvas.currentUrl()?.trim() == CanvasActionTrust.localA2uiAssetUrl
if (!canvas.showAndAwaitHost()) return false
if (!alreadyOnA2uiHost) {
canvas.showLocalA2ui()
}
if (alreadyOnA2uiHost && isA2uiReady()) {
return true
}
// The bundled A2UI host bootstraps asynchronously after navigation; poll briefly before failing the command.
repeat(50) {
if (isA2uiReady()) return true
delay(120)
}
return false
}
private suspend fun isA2uiReady(): Boolean =
try {
canvas.eval(a2uiReadyCheckJS) == "true"
} catch (_: Throwable) {
false
}
fun decodeA2uiMessages(
command: String,
paramsJson: String?,
): String {
val raw = paramsJson?.trim().orEmpty()
if (raw.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: paramsJSON required")
val obj =
json.parseToJsonElement(raw) as? JsonObject
?: throw IllegalArgumentException("INVALID_REQUEST: expected object params")
val jsonlField = (obj["jsonl"] as? JsonPrimitive)?.content?.trim().orEmpty()
val hasMessagesArray = obj["messages"] is JsonArray
if (command == "canvas.a2ui.pushJSONL" || (!hasMessagesArray && jsonlField.isNotBlank())) {
val jsonl = jsonlField
if (jsonl.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: jsonl required")
// JSONL keeps large A2UI streams model-friendly while still validating each message.
val messages =
jsonl
.lineSequence()
.map { it.trim() }
.filter { it.isNotBlank() }
.mapIndexed { idx, line ->
val el = json.parseToJsonElement(line)
val msg =
el as? JsonObject
?: throw IllegalArgumentException("A2UI JSONL line ${idx + 1}: expected a JSON object")
validateA2uiV0_8(msg, idx + 1)
msg
}.toList()
return JsonArray(messages).toString()
}
val arr = obj["messages"] as? JsonArray ?: throw IllegalArgumentException("INVALID_REQUEST: messages[] required")
val out =
arr.mapIndexed { idx, el ->
val msg =
el as? JsonObject
?: throw IllegalArgumentException("A2UI messages[$idx]: expected a JSON object")
validateA2uiV0_8(msg, idx + 1)
msg
}
return JsonArray(out).toString()
}
private fun validateA2uiV0_8(
msg: JsonObject,
lineNumber: Int,
) {
if (msg.containsKey("createSurface")) {
// Android scaffold currently implements A2UI v0.8, not the v0.9 createSurface shape.
throw IllegalArgumentException(
"A2UI JSONL line $lineNumber: looks like A2UI v0.9 (`createSurface`). Canvas supports v0.8 messages only.",
)
}
val allowed = setOf("beginRendering", "surfaceUpdate", "dataModelUpdate", "deleteSurface")
val matched = msg.keys.filter { allowed.contains(it) }
if (matched.size != 1) {
val found = msg.keys.sorted().joinToString(", ")
throw IllegalArgumentException(
"A2UI JSONL line $lineNumber: expected exactly one of ${allowed.sorted().joinToString(", ")}; found: $found",
)
}
}
companion object {
const val a2uiReadyCheckJS: String =
"""
(() => {
try {
const host = globalThis.openclawA2UI;
return !!host && typeof host.applyMessages === 'function';
} catch (_) {
return false;
}
})()
"""
const val a2uiResetJS: String =
"""
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return { ok: false, error: "missing openclawA2UI" };
return host.reset();
} catch (e) {
return { ok: false, error: String(e?.message ?? e) };
}
})()
"""
fun a2uiApplyMessagesJS(messagesJson: String): String =
"""
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return { ok: false, error: "missing openclawA2UI" };
const messages = $messagesJson;
return host.applyMessages(messages);
} catch (e) {
return { ok: false, error: String(e?.message ?? e) };
}
})()
""".trimIndent()
}
}
apps/android/app/src/main/java/ai/openclaw/app/node/CanvasActionTrust.kt
-21
View File
@@ -1,21 +0,0 @@
package ai.openclaw.app.node
/**
* Trust helper for WebView-originated canvas/A2UI actions.
*/
object CanvasActionTrust {
/** Local canvas scaffold is the only trusted file URL. */
const val scaffoldAssetUrl: String = "file:///android_asset/CanvasScaffold/scaffold.html"
/** Local bundled A2UI is the only action-capable A2UI host. */
const val localA2uiAssetUrl: String = "file:///android_asset/CanvasA2UI/index.html"
/** Accepts only app-owned bundled pages. Remote WebView content is render-only. */
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean {
val candidate = rawUrl?.trim().orEmpty()
if (candidate.isEmpty()) return false
if (candidate == scaffoldAssetUrl) return true
if (candidate == localA2uiAssetUrl) return true
return false
}
}
apps/android/app/src/main/java/ai/openclaw/app/node/CanvasController.kt
-388
View File
@@ -1,388 +0,0 @@
package ai.openclaw.app.node
import ai.openclaw.app.BuildConfig
import android.graphics.Bitmap
import android.graphics.Canvas
import android.os.Looper
import android.util.Base64
import android.util.Log
import android.webkit.WebView
import androidx.core.graphics.createBitmap
import androidx.core.graphics.scale
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeoutOrNull
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import org.json.JSONObject
import java.io.ByteArrayOutputStream
import kotlin.coroutines.resume
/**
* Owns the Android WebView canvas surface used by canvas and A2UI commands.
*/
class CanvasController {
enum class PresentationState {
Unmounted,
Hidden,
Visible,
}
enum class SnapshotFormat(
val rawValue: String,
) {
Png("png"),
Jpeg("jpeg"),
}
@Volatile private var webView: WebView? = null
@Volatile private var url: String? = null
@Volatile private var debugStatusEnabled: Boolean = false
@Volatile private var debugStatusTitle: String? = null
@Volatile private var debugStatusSubtitle: String? = null
@Volatile private var homeCanvasStateJson: String? = null
private val _currentUrl = MutableStateFlow<String?>(null)
val currentUrl: StateFlow<String?> = _currentUrl.asStateFlow()
private val _presentationState = MutableStateFlow(PresentationState.Unmounted)
val presentationState: StateFlow<PresentationState> = _presentationState.asStateFlow()
private val hostAttachedState = MutableStateFlow(false)
private val scaffoldAssetUrl = CanvasActionTrust.scaffoldAssetUrl
private val localA2uiAssetUrl = CanvasActionTrust.localA2uiAssetUrl
private fun clampJpegQuality(quality: Double?): Int {
val q = (quality ?: 0.82).coerceIn(0.1, 1.0)
return (q * 100.0).toInt().coerceIn(1, 100)
}
private fun Bitmap.scaleForMaxWidth(maxWidth: Int?): Bitmap {
if (maxWidth == null || maxWidth <= 0 || width <= maxWidth) {
return this
}
val scaledHeight = (height.toDouble() * (maxWidth.toDouble() / width.toDouble())).toInt().coerceAtLeast(1)
return scale(maxWidth, scaledHeight)
}
/** Attaches the active WebView and replays state that may have arrived before the view existed. */
fun attach(webView: WebView) {
this.webView = webView
hostAttachedState.value = true
// Replay persisted state because WebView attachment can happen after gateway events arrive.
reload()
applyDebugStatus()
applyHomeCanvasState()
}
/** Releases the shell-owned host when its UI owner permanently leaves composition. */
fun releaseHost() {
webView = null
hostAttachedState.value = false
_presentationState.value = PresentationState.Unmounted
}
/** Invalid renderer processes cannot be reused; retain the host but require a new child. */
fun onRenderProcessGone(webView: WebView) {
if (this.webView !== webView) return
this.webView = null
// Do not replay the page that terminated its renderer into the replacement WebView.
url = null
_currentUrl.value = null
hostAttachedState.value = false
_presentationState.value = PresentationState.Hidden
}
fun show() {
_presentationState.value = PresentationState.Visible
}
fun hide() {
if (_presentationState.value != PresentationState.Unmounted) {
_presentationState.value = PresentationState.Hidden
}
}
/**
* Requests presentation and waits only for the shell host to accept it.
* Remote page loading remains asynchronous and must not delay invoke completion.
*/
suspend fun showAndAwaitHost(): Boolean {
val previousState = _presentationState.value
show()
if (hostAttachedState.value) return true
val attached =
withTimeoutOrNull(hostAttachTimeoutMs) {
hostAttachedState.first { it }
true
} ?: hostAttachedState.value
if (!attached && _presentationState.value == PresentationState.Visible) {
// A failed foreground handoff must not leave a pending overlay for the next Activity.
_presentationState.value = previousState
}
return attached
}
/** Navigates the canvas to a remote URL or back to the bundled scaffold for blank/root input. */
fun navigate(url: String) {
this.url = CanvasNavigationPolicy.normalize(url).ifBlank { null }
_currentUrl.value = this.url
reload()
}
/** Shows the app-owned A2UI renderer that is allowed to dispatch native actions. */
fun showLocalA2ui() {
this.url = localA2uiAssetUrl
_currentUrl.value = localA2uiAssetUrl
reload()
}
fun currentUrl(): String? = url
fun setDebugStatusEnabled(enabled: Boolean) {
debugStatusEnabled = enabled
applyDebugStatus()
}
fun setDebugStatus(
title: String?,
subtitle: String?,
) {
debugStatusTitle = title
debugStatusSubtitle = subtitle
applyDebugStatus()
}
fun onPageFinished() {
applyDebugStatus()
applyHomeCanvasState()
}
fun updateHomeCanvasState(json: String?) {
homeCanvasStateJson = json
applyHomeCanvasState()
}
private inline fun withWebViewOnMain(crossinline block: (WebView) -> Unit) {
val wv = webView ?: return
if (Looper.myLooper() == Looper.getMainLooper()) {
block(wv)
} else {
// WebView APIs must run on the main thread.
wv.post { block(wv) }
}
}
private fun reload() {
val currentUrl = url
withWebViewOnMain { wv ->
if (currentUrl == null) {
if (BuildConfig.DEBUG) {
Log.d("OpenClawCanvas", "load scaffold: $scaffoldAssetUrl")
}
wv.loadUrl(scaffoldAssetUrl)
} else {
if (BuildConfig.DEBUG) {
Log.d("OpenClawCanvas", "load url: $currentUrl")
}
wv.loadUrl(currentUrl)
}
}
}
private fun applyDebugStatus() {
val enabled = debugStatusEnabled
val title = debugStatusTitle
val subtitle = debugStatusSubtitle
withWebViewOnMain { wv ->
val titleJs = title?.let { JSONObject.quote(it) } ?: "null"
val subtitleJs = subtitle?.let { JSONObject.quote(it) } ?: "null"
val js =
"""
(() => {
try {
const api = globalThis.__openclaw;
if (!api) return;
if (typeof api.setDebugStatusEnabled === 'function') {
api.setDebugStatusEnabled(${if (enabled) "true" else "false"});
}
if (!${if (enabled) "true" else "false"}) return;
if (typeof api.setStatus === 'function') {
api.setStatus($titleJs, $subtitleJs);
}
} catch (_) {}
})();
""".trimIndent()
wv.evaluateJavascript(js, null)
}
}
private fun applyHomeCanvasState() {
val payload = homeCanvasStateJson ?: "null"
withWebViewOnMain { wv ->
val js =
"""
(() => {
try {
const api = globalThis.__openclaw;
if (!api || typeof api.renderHome !== 'function') return;
api.renderHome($payload);
} catch (_) {}
})();
""".trimIndent()
wv.evaluateJavascript(js, null)
}
}
/** Evaluates JavaScript against the attached WebView on the main thread. */
suspend fun eval(javaScript: String): String =
withContext(Dispatchers.Main) {
val wv = webView ?: throw IllegalStateException("no webview")
suspendCancellableCoroutine { cont ->
wv.evaluateJavascript(javaScript) { result ->
cont.resume(result ?: "")
}
}
}
/** Captures the WebView as PNG/JPEG base64 with optional width and quality bounds. */
suspend fun snapshotBase64(
format: SnapshotFormat,
quality: Double?,
maxWidth: Int?,
): String =
withContext(Dispatchers.Main) {
val wv = webView ?: throw IllegalStateException("no webview")
val bmp = wv.captureBitmap()
try {
val scaled = bmp.scaleForMaxWidth(maxWidth)
try {
val out = ByteArrayOutputStream()
val (compressFormat, compressQuality) =
when (format) {
SnapshotFormat.Png -> Bitmap.CompressFormat.PNG to 100
SnapshotFormat.Jpeg -> Bitmap.CompressFormat.JPEG to clampJpegQuality(quality)
}
scaled.compress(compressFormat, compressQuality, out)
Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
} finally {
if (scaled !== bmp) scaled.recycle()
}
} finally {
bmp.recycle()
}
}
private suspend fun WebView.captureBitmap(): Bitmap =
suspendCancellableCoroutine { cont ->
val width = width.coerceAtLeast(1)
val height = height.coerceAtLeast(1)
val bitmap = createBitmap(width, height, Bitmap.Config.ARGB_8888)
// WebView isn't supported by PixelCopy.request(...) directly; draw() is the most reliable
// cross-version snapshot for this lightweight "canvas" use-case.
draw(Canvas(bitmap))
cont.resume(bitmap)
}
companion object {
private const val hostAttachTimeoutMs = 5_000L
/**
* Parsed canvas.snapshot options used by invoke dispatch.
*/
data class SnapshotParams(
val format: SnapshotFormat,
val quality: Double?,
val maxWidth: Int?,
)
/** Parses canvas.navigate params and returns blank when the payload is missing or invalid. */
fun parseNavigateUrl(paramsJson: String?): String {
val obj = parseParamsObject(paramsJson) ?: return ""
return obj.string("url").trim()
}
/** Parses non-blank JavaScript from canvas.eval params. */
fun parseEvalJs(paramsJson: String?): String? {
val obj = parseParamsObject(paramsJson) ?: return null
val js = obj.string("javaScript").trim()
return js.takeIf { it.isNotBlank() }
}
fun parseSnapshotMaxWidth(paramsJson: String?): Int? {
val obj = parseParamsObject(paramsJson) ?: return null
if (!obj.containsKey("maxWidth")) return null
val width = obj.int("maxWidth") ?: 0
return width.takeIf { it > 0 }
}
fun parseSnapshotFormat(paramsJson: String?): SnapshotFormat {
val obj = parseParamsObject(paramsJson) ?: return SnapshotFormat.Jpeg
val raw = obj.string("format").trim().lowercase()
return when (raw) {
"png" -> SnapshotFormat.Png
"jpeg", "jpg" -> SnapshotFormat.Jpeg
"" -> SnapshotFormat.Jpeg
else -> SnapshotFormat.Jpeg
}
}
fun parseSnapshotQuality(paramsJson: String?): Double? {
val obj = parseParamsObject(paramsJson) ?: return null
if (!obj.containsKey("quality")) return null
val q = obj.double("quality") ?: Double.NaN
if (!q.isFinite()) return null
// Keep JPEG quality inside encoder-safe bounds; PNG ignores it.
return q.coerceIn(0.1, 1.0)
}
/** Parses canvas.snapshot params using JPEG defaults and encoder-safe bounds. */
fun parseSnapshotParams(paramsJson: String?): SnapshotParams =
SnapshotParams(
format = parseSnapshotFormat(paramsJson),
quality = parseSnapshotQuality(paramsJson),
maxWidth = parseSnapshotMaxWidth(paramsJson),
)
private val json = Json { ignoreUnknownKeys = true }
private fun parseParamsObject(paramsJson: String?): JsonObject? {
val raw = paramsJson?.trim().orEmpty()
if (raw.isEmpty()) return null
return try {
json.parseToJsonElement(raw).asObjectOrNull()
} catch (_: Throwable) {
null
}
}
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
private fun JsonObject.string(key: String): String {
val prim = this[key] as? JsonPrimitive ?: return ""
val raw = prim.content
return raw.takeIf { it != "null" }.orEmpty()
}
private fun JsonObject.int(key: String): Int? {
val prim = this[key] as? JsonPrimitive ?: return null
return prim.content.toIntOrNull()
}
private fun JsonObject.double(key: String): Double? {
val prim = this[key] as? JsonPrimitive ?: return null
return prim.content.toDoubleOrNull()
}
}
}
apps/android/app/src/main/java/ai/openclaw/app/node/CanvasNavigationPolicy.kt
-196
View File
@@ -1,196 +0,0 @@
package ai.openclaw.app.node
import java.net.InetAddress
import java.net.URI
/** Shared policy for gateway-driven loads and WebView main-frame navigations. */
internal object CanvasNavigationPolicy {
fun normalize(rawUrl: String): String {
val trimmed = rawUrl.trim()
if (trimmed.isBlank() || trimmed == "/") return ""
return trimmed.takeUnless(::shouldBlock).orEmpty()
}
fun shouldBlock(rawUrl: String): Boolean {
val trimmed = rawUrl.trim()
if (trimmed.isBlank() || trimmed == "/") return false
val target = parseTarget(trimmed)
val isWebUrl = target.scheme == "http" || target.scheme == "https"
if (isWebUrl && (target.host.isEmpty() || isMalformedWebHost(target.host))) return true
return target.host.isNotEmpty() && isDeviceLocalHost(target.host)
}
/** WebView does not expose POST redirects to shouldOverrideUrlLoading; fail closed before dispatch. */
fun shouldBlockNonGetMainFrame(
method: String,
isForMainFrame: Boolean,
): Boolean = isForMainFrame && !method.equals("GET", ignoreCase = true)
private data class Target(
val scheme: String,
val host: String,
)
private fun parseTarget(rawUrl: String): Target {
val parsed = runCatching { URI(rawUrl) }.getOrNull()
val scheme =
parsed
?.scheme
?.trim()
?.lowercase()
.orEmpty()
.ifBlank { rawScheme(rawUrl) }
val host =
parsed
?.host
?.trim()
.orEmpty()
.ifBlank { authorityHost(parsed?.rawAuthority ?: rawAuthority(rawUrl)) }
return Target(scheme = scheme, host = percentDecodeAscii(host))
}
private fun rawScheme(rawUrl: String): String {
val schemeSeparator = rawUrl.indexOf("://")
val colonSeparator = rawUrl.indexOf(':')
val end =
when {
schemeSeparator > 0 -> schemeSeparator
colonSeparator > 0 -> colonSeparator
else -> return ""
}
val candidate = rawUrl.substring(0, end).trim().lowercase()
return candidate
.takeIf {
it.all { char ->
char in 'a'..'z' || char in '0'..'9' || char == '+' || char == '-' || char == '.'
}
}.orEmpty()
}
private fun rawAuthority(rawUrl: String): String? {
val schemeSeparator = rawUrl.indexOf("://")
if (schemeSeparator < 0) return null
val authorityStart = schemeSeparator + 3
val authorityEnd =
rawUrl
.indexOfAny(charArrayOf('/', '\\', '?', '#'), startIndex = authorityStart)
.takeIf { it >= 0 }
?: rawUrl.length
return rawUrl.substring(authorityStart, authorityEnd)
}
private fun authorityHost(rawAuthority: String?): String {
val authority = rawAuthority?.trim().orEmpty()
if (authority.isEmpty()) return ""
val hostPort = authority.substringAfterLast('@')
if (hostPort.startsWith("[")) {
return hostPort.substringAfter('[').substringBefore(']')
}
return if (hostPort.count { it == ':' } == 1) hostPort.substringBefore(':') else hostPort
}
private fun percentDecodeAscii(value: String): String {
if (!value.contains('%')) return value
val out = StringBuilder(value.length)
var index = 0
while (index < value.length) {
if (value[index] == '%' && index + 2 < value.length) {
val byte = value.substring(index + 1, index + 3).toIntOrNull(16)
if (byte != null) {
out.append(byte.toChar())
index += 3
continue
}
}
out.append(value[index])
index += 1
}
return out.toString()
}
private fun isMalformedWebHost(rawHost: String): Boolean =
// Chromium applies UTS #46 before resolving special-scheme hosts. Reject raw Unicode here so
// compatibility characters cannot become localhost or a loopback IP after this check.
rawHost.any { char ->
char <= ' ' ||
char.code > 0x7f ||
char == '/' ||
char == '\\' ||
char == '?' ||
char == '#' ||
char == '@'
} ||
rawHost.contains('%')
private fun isDeviceLocalHost(rawHost: String): Boolean {
var host =
rawHost
.trim()
.lowercase()
.trim('[', ']')
.trimEnd('.')
host = host.substringBefore('%')
if (host == "localhost" || host.endsWith(".localhost")) return true
parseWebViewIpv4Address(host)?.let { address ->
return address == 0L || ((address ushr 24) and 0xffL) == 127L
}
if (!host.contains(':') || !host.all(::isIpv6LiteralChar)) return false
val address = runCatching { InetAddress.getByName(host).address }.getOrNull() ?: return false
if (address.size == 4) {
return address.all { it == 0.toByte() } || address[0] == 127.toByte()
}
if (address.size != 16) return false
if (address.all { it == 0.toByte() }) return true
if (address.copyOfRange(0, 15).all { it == 0.toByte() } && address[15] == 1.toByte()) return true
val mappedPrefix =
address.copyOfRange(0, 10).all { it == 0.toByte() } &&
address[10] == 0xff.toByte() &&
address[11] == 0xff.toByte()
val compatiblePrefix = address.copyOfRange(0, 12).all { it == 0.toByte() }
return (mappedPrefix || compatiblePrefix) &&
(address[12] == 127.toByte() || address.copyOfRange(12, 16).all { it == 0.toByte() })
}
private fun isIpv6LiteralChar(char: Char): Boolean = char == ':' || char == '.' || char in '0'..'9' || char.lowercaseChar() in 'a'..'f'
/** Matches Chromium/WebView's accepted decimal, octal, hex, and shorthand IPv4 forms. */
private fun parseWebViewIpv4Address(rawHost: String): Long? {
val host = rawHost.trim().lowercase().trimEnd('.')
if (host.isEmpty() || host.contains(':') || host.contains('%')) return null
val parts = host.split('.')
if (parts.size !in 1..4 || parts.any { it.isEmpty() }) return null
val numbers = parts.map { parseWebViewIpv4Number(it) ?: return null }
if (numbers.dropLast(1).any { it > 255L }) return null
val lastMax =
when (numbers.size) {
1 -> 0xffffffffL
2 -> 0x00ffffffL
3 -> 0x0000ffffL
else -> 0xffL
}
val last = numbers.last()
if (last > lastMax) return null
return when (numbers.size) {
1 -> last
2 -> (numbers[0] shl 24) or last
3 -> (numbers[0] shl 24) or (numbers[1] shl 16) or last
else -> (numbers[0] shl 24) or (numbers[1] shl 16) or (numbers[2] shl 8) or last
}
}
private fun parseWebViewIpv4Number(raw: String): Long? {
val normalized = raw.trim().lowercase()
if (normalized.isEmpty()) return null
val (digits, radix) =
when {
normalized.startsWith("0x") -> normalized.drop(2) to 16
normalized.length > 1 && normalized.startsWith("0") -> normalized.drop(1) to 8
else -> normalized to 10
}
if (digits.isEmpty()) return 0L
return digits.toLongOrNull(radix)
}
}
apps/android/app/src/main/java/ai/openclaw/app/node/InvokeCommandRegistry.kt
-35
View File
@@ -3,8 +3,6 @@ package ai.openclaw.app.node
import ai.openclaw.app.protocol.OpenClawCalendarCommand
import ai.openclaw.app.protocol.OpenClawCallLogCommand
import ai.openclaw.app.protocol.OpenClawCameraCommand
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
import ai.openclaw.app.protocol.OpenClawCanvasCommand
import ai.openclaw.app.protocol.OpenClawCapability
import ai.openclaw.app.protocol.OpenClawContactsCommand
import ai.openclaw.app.protocol.OpenClawDeviceCommand
@@ -81,7 +79,6 @@ object InvokeCommandRegistry {
/** Capabilities mirror gateway protocol ids and are filtered by device state. */
val capabilityManifest: List<NodeCapabilitySpec> =
listOf(
NodeCapabilitySpec(name = OpenClawCapability.Canvas.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Device.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Notifications.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.System.rawValue),
@@ -125,38 +122,6 @@ object InvokeCommandRegistry {
/** Complete Android node command catalog before runtime availability filtering. */
val all: List<InvokeCommandSpec> =
listOf(
InvokeCommandSpec(
name = OpenClawCanvasCommand.Present.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Hide.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Navigate.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Eval.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Snapshot.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.Push.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.PushJSONL.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.Reset.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawSystemCommand.Notify.rawValue,
),
apps/android/app/src/main/java/ai/openclaw/app/node/InvokeDispatcher.kt
-111
View File
@@ -4,8 +4,6 @@ import ai.openclaw.app.gateway.GatewaySession
import ai.openclaw.app.protocol.OpenClawCalendarCommand
import ai.openclaw.app.protocol.OpenClawCallLogCommand
import ai.openclaw.app.protocol.OpenClawCameraCommand
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
import ai.openclaw.app.protocol.OpenClawCanvasCommand
import ai.openclaw.app.protocol.OpenClawContactsCommand
import ai.openclaw.app.protocol.OpenClawDeviceCommand
import ai.openclaw.app.protocol.OpenClawLocationCommand
@@ -15,8 +13,6 @@ import ai.openclaw.app.protocol.OpenClawNotificationsCommand
import ai.openclaw.app.protocol.OpenClawSmsCommand
import ai.openclaw.app.protocol.OpenClawSystemCommand
import ai.openclaw.app.protocol.OpenClawTalkCommand
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
/** Runtime state for SMS search, split so permission prompts are not reported as hard unavailability. */
internal enum class SmsSearchAvailabilityReason {
@@ -64,7 +60,6 @@ internal fun smsSearchAvailabilityError(
* Gateway node.invoke command router for Android-owned capabilities.
*/
class InvokeDispatcher(
private val canvas: CanvasController,
private val cameraHandler: CameraHandler,
private val locationHandler: LocationHandler,
private val deviceHandler: DeviceHandler,
@@ -76,7 +71,6 @@ class InvokeDispatcher(
private val calendarHandler: CalendarHandler,
private val motionHandler: MotionHandler,
private val smsHandler: SmsHandler,
private val a2uiHandler: A2UIHandler,
private val debugHandler: DebugHandler,
private val callLogHandler: CallLogHandler,
private val mobileUiHandler: MobileUiHandler,
@@ -91,14 +85,10 @@ class InvokeDispatcher(
private val photosAvailable: () -> Boolean,
private val installedAppsSharingEnabled: () -> Boolean,
private val debugBuild: () -> Boolean,
private val onCanvasA2uiPush: () -> Unit,
private val onCanvasA2uiReset: () -> Unit,
private val motionActivityAvailable: () -> Boolean,
private val motionPedometerAvailable: () -> Boolean,
private val mobileUiAvailable: () -> Boolean,
) {
private val canvasCommandMutex = Mutex()
/** Dispatches one gateway node.invoke command after foreground and availability gates pass. */
suspend fun handleInvoke(
command: String,
@@ -119,11 +109,6 @@ class InvokeDispatcher(
}
availabilityError(spec.availability)?.let { return it }
if (command.startsWith(OpenClawCanvasCommand.NamespacePrefix)) {
// GatewaySession may deliver invokes concurrently. Canvas presentation, navigation, and
// A2UI evaluation share one WebView and must observe command arrival order.
return canvasCommandMutex.withLock { dispatchInvoke(command, paramsJson) }
}
return dispatchInvoke(command, paramsJson)
}
@@ -133,81 +118,6 @@ class InvokeDispatcher(
): GatewaySession.InvokeResult {
// Command strings come from OpenClawProtocolConstants; the registry above owns advertised availability.
return when (command) {
// Canvas commands
OpenClawCanvasCommand.Present.rawValue -> {
val url = CanvasController.parseNavigateUrl(paramsJson)
withCanvasAvailable {
check(canvas.showAndAwaitHost()) { "canvas host unavailable" }
canvas.navigate(url)
GatewaySession.InvokeResult.ok(null)
}
}
OpenClawCanvasCommand.Hide.rawValue -> {
canvas.hide()
GatewaySession.InvokeResult.ok(null)
}
OpenClawCanvasCommand.Navigate.rawValue -> {
val url = CanvasController.parseNavigateUrl(paramsJson)
withCanvasAvailable {
check(canvas.showAndAwaitHost()) { "canvas host unavailable" }
canvas.navigate(url)
GatewaySession.InvokeResult.ok(null)
}
}
OpenClawCanvasCommand.Eval.rawValue -> {
val js =
CanvasController.parseEvalJs(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: javaScript required",
)
withCanvasAvailable {
val result = canvas.eval(js)
GatewaySession.InvokeResult.ok("""{"result":${result.toJsonString()}}""")
}
}
OpenClawCanvasCommand.Snapshot.rawValue -> {
val snapshotParams = CanvasController.parseSnapshotParams(paramsJson)
withCanvasAvailable {
val base64 =
canvas.snapshotBase64(
format = snapshotParams.format,
quality = snapshotParams.quality,
maxWidth = snapshotParams.maxWidth,
)
GatewaySession.InvokeResult.ok("""{"format":"${snapshotParams.format.rawValue}","base64":"$base64"}""")
}
}
// A2UI commands
OpenClawCanvasA2UICommand.Reset.rawValue ->
withReadyA2ui {
withCanvasAvailable {
val res = canvas.eval(A2UIHandler.a2uiResetJS)
onCanvasA2uiReset()
GatewaySession.InvokeResult.ok(res)
}
}
OpenClawCanvasA2UICommand.Push.rawValue, OpenClawCanvasA2UICommand.PushJSONL.rawValue -> {
val messages =
try {
a2uiHandler.decodeA2uiMessages(command, paramsJson)
} catch (err: Throwable) {
return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = err.message ?: "invalid A2UI payload",
)
}
withReadyA2ui {
withCanvasAvailable {
val js = A2UIHandler.a2uiApplyMessagesJS(messages)
val res = canvas.eval(js)
onCanvasA2uiPush()
GatewaySession.InvokeResult.ok(res)
}
}
}
// Camera commands
OpenClawCameraCommand.List.rawValue -> cameraHandler.handleList(paramsJson)
OpenClawCameraCommand.Snap.rawValue -> cameraHandler.handleSnap(paramsJson)
@@ -272,27 +182,6 @@ class InvokeDispatcher(
}
}
private suspend fun withReadyA2ui(block: suspend () -> GatewaySession.InvokeResult): GatewaySession.InvokeResult {
if (!a2uiHandler.ensureA2uiReady()) {
return GatewaySession.InvokeResult.error(
code = "A2UI_HOST_UNAVAILABLE",
message = "A2UI_HOST_UNAVAILABLE: bundled A2UI host not reachable",
)
}
return block()
}
private suspend fun withCanvasAvailable(block: suspend () -> GatewaySession.InvokeResult): GatewaySession.InvokeResult =
try {
block()
} catch (_: Throwable) {
// WebView calls throw when the Activity is backgrounded between the foreground check and execution.
GatewaySession.InvokeResult.error(
code = "NODE_BACKGROUND_UNAVAILABLE",
message = "NODE_BACKGROUND_UNAVAILABLE: canvas unavailable",
)
}
private fun availabilityError(availability: InvokeCommandAvailability): GatewaySession.InvokeResult? =
when (availability) {
InvokeCommandAvailability.Always -> null
apps/android/app/src/main/java/ai/openclaw/app/node/NodeUtils.kt
-19
View File
@@ -11,25 +11,6 @@ import kotlinx.serialization.json.contentOrNull
/** Default canvas seam color used when gateway/user params omit a hex color. */
const val DEFAULT_SEAM_COLOR_ARGB: Long = 0xFF4F7A9A
/** Small tuple used by Android node handlers that need four return values. */
data class Quad<A, B, C, D>(
val first: A,
val second: B,
val third: C,
val fourth: D,
)
/** Escapes a Kotlin string into a JSON string literal without building a JsonElement. */
fun String.toJsonString(): String {
val escaped =
this
.replace("\\", "\\\\")
.replace("\"", "\\\"")
.replace("\n", "\\n")
.replace("\r", "\\r")
return "\"$escaped\""
}
fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
/** Parses invoke params into a JSON object, returning null for absent/malformed input. */
apps/android/app/src/main/java/ai/openclaw/app/protocol/OpenClawCanvasA2UIAction.kt
-76
View File
@@ -1,76 +0,0 @@
package ai.openclaw.app.protocol
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
object OpenClawCanvasA2UIAction {
/** Reads the agent-facing action name from either the modern name field or legacy action field. */
fun extractActionName(userAction: JsonObject): String? {
val name =
(userAction["name"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
if (name.isNotEmpty()) return name
val action =
(userAction["action"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
return action.ifEmpty { null }
}
/** Normalizes prompt tag values so the compact CANVAS_A2UI envelope stays parser-friendly. */
fun sanitizeTagValue(value: String): String {
val trimmed = value.trim().ifEmpty { "-" }
val normalized = trimmed.replace(" ", "_")
val out = StringBuilder(normalized.length)
for (c in normalized) {
val ok =
c.isLetterOrDigit() ||
c == '_' ||
c == '-' ||
c == '.' ||
c == ':'
out.append(if (ok) c else '_')
}
return out.toString()
}
/** Formats the compact text envelope sent to the agent when a canvas UI action fires. */
fun formatAgentMessage(
actionName: String,
sessionKey: String,
surfaceId: String,
sourceComponentId: String,
host: String,
instanceId: String,
contextJson: String?,
): String {
val ctxSuffix = contextJson?.takeIf { it.isNotBlank() }?.let { " ctx=$it" }.orEmpty()
return listOf(
"CANVAS_A2UI",
"action=${sanitizeTagValue(actionName)}",
"session=${sanitizeTagValue(sessionKey)}",
"surface=${sanitizeTagValue(surfaceId)}",
"component=${sanitizeTagValue(sourceComponentId)}",
"host=${sanitizeTagValue(host)}",
"instance=${sanitizeTagValue(instanceId)}$ctxSuffix",
"default=update_canvas",
).joinToString(separator = " ")
}
/** Builds JS that reports an agent action result back to the canvas runtime. */
fun jsDispatchA2UIActionStatus(
actionId: String,
ok: Boolean,
error: String?,
): String {
val err = jsonStringLiteral(error ?: "")
val okLiteral = if (ok) "true" else "false"
val idLiteral = jsonStringLiteral(actionId)
return "window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail: { id: $idLiteral, ok: $okLiteral, error: $err } }));"
}
private fun jsonStringLiteral(raw: String): String = JsonPrimitive(raw).toString().replace("\u2028", "\\u2028").replace("\u2029", "\\u2029")
}
apps/android/app/src/main/java/ai/openclaw/app/protocol/OpenClawProtocolConstants.kt
-28
View File
@@ -21,34 +21,6 @@ enum class OpenClawCapability(
MobileUI("mobileUI"),
}
enum class OpenClawCanvasCommand(
val rawValue: String,
) {
Present("canvas.present"),
Hide("canvas.hide"),
Navigate("canvas.navigate"),
Eval("canvas.eval"),
Snapshot("canvas.snapshot"),
;
companion object {
const val NamespacePrefix: String = "canvas."
}
}
enum class OpenClawCanvasA2UICommand(
val rawValue: String,
) {
Push("canvas.a2ui.push"),
PushJSONL("canvas.a2ui.pushJSONL"),
Reset("canvas.a2ui.reset"),
;
companion object {
const val NamespacePrefix: String = "canvas.a2ui."
}
}
enum class OpenClawCameraCommand(
val rawValue: String,
) {
apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasScreen.kt
-319
View File
@@ -1,319 +0,0 @@
package ai.openclaw.app.ui
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.node.CanvasNavigationPolicy
import android.annotation.SuppressLint
import android.content.Context
import android.net.Uri
import android.util.Log
import android.view.View
import android.view.ViewGroup
import android.webkit.ConsoleMessage
import android.webkit.RenderProcessGoneDetail
import android.webkit.WebChromeClient
import android.webkit.WebResourceError
import android.webkit.WebResourceRequest
import android.webkit.WebResourceResponse
import android.webkit.WebSettings
import android.webkit.WebView
import android.webkit.WebViewClient
import android.widget.FrameLayout
import androidx.compose.runtime.Composable
import androidx.compose.ui.Modifier
import androidx.compose.ui.viewinterop.AndroidView
import androidx.webkit.JavaScriptReplyProxy
import androidx.webkit.WebMessageCompat
import androidx.webkit.WebSettingsCompat
import androidx.webkit.WebViewCompat
import androidx.webkit.WebViewFeature
import java.io.ByteArrayInputStream
import java.util.concurrent.atomic.AtomicReference
/** Hosts the gateway canvas WebView and attaches it to the runtime canvas controller. */
@SuppressLint("SetJavaScriptEnabled")
@Suppress("DEPRECATION")
@Composable
fun CanvasScreen(
viewModel: MainViewModel,
visible: Boolean,
modifier: Modifier = Modifier,
) {
AndroidView(
modifier = modifier,
factory = { context ->
CanvasHostView(
context = context,
controller = viewModel.canvas,
isTrustedPage = viewModel::isTrustedCanvasActionUrl,
onA2uiMessage = viewModel::handleCanvasA2UIActionFromWebView,
).apply {
updateVisible(visible)
}
},
update = { host -> host.updateVisible(visible) },
onRelease = CanvasHostView::release,
)
}
/**
* Retained shell host whose WebView child can be replaced after renderer death.
*
* Compose creates this host directly; XML inflation cannot supply its controller and callbacks.
*/
@SuppressLint("SetJavaScriptEnabled", "ViewConstructor")
@Suppress("DEPRECATION")
internal class CanvasHostView(
context: Context,
private val controller: CanvasController,
private val isTrustedPage: (String?) -> Boolean,
private val onA2uiMessage: (String) -> Unit,
) : FrameLayout(context) {
internal var currentWebView: WebView? = null
private set
private val isDebuggable =
(context.applicationInfo.flags and android.content.pm.ApplicationInfo.FLAG_DEBUGGABLE) != 0
private val currentPageUrlRef = AtomicReference<String?>(null)
init {
visibility = View.INVISIBLE
}
fun updateVisible(visible: Boolean) {
if (visible) {
val webView = currentWebView ?: createWebView()
visibility = View.VISIBLE
webView.visibility = View.VISIBLE
webView.onResume()
return
}
visibility = View.INVISIBLE
currentWebView?.let { webView ->
webView.visibility = View.INVISIBLE
webView.onPause()
}
}
fun release() {
controller.releaseHost()
currentWebView?.let(::destroyWebView)
}
private fun createWebView(): WebView =
WebView(context).also { webView ->
val webSettings = webView.settings
webSettings.setAllowContentAccess(false)
webSettings.setAllowFileAccess(false)
webSettings.setAllowFileAccessFromFileURLs(false)
webSettings.setAllowUniversalAccessFromFileURLs(false)
webSettings.setSafeBrowsingEnabled(true)
webSettings.javaScriptEnabled = true
webSettings.domStorageEnabled = true
webSettings.mixedContentMode = WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE
webSettings.useWideViewPort = false
webSettings.loadWithOverviewMode = false
webSettings.builtInZoomControls = false
webSettings.displayZoomControls = false
webSettings.setSupportZoom(false)
webView.visibility = View.INVISIBLE
// targetSdk 33+ ignores Force Dark APIs, so only opt out through the supported
// algorithmic darkening flag when this WebView implementation exposes it.
if (WebViewFeature.isFeatureSupported(WebViewFeature.ALGORITHMIC_DARKENING)) {
WebSettingsCompat.setAlgorithmicDarkeningAllowed(webSettings, false)
}
if (isDebuggable) {
Log.d("OpenClawWebView", "userAgent: ${webSettings.userAgentString}")
}
webView.isScrollContainer = true
webView.overScrollMode = View.OVER_SCROLL_IF_CONTENT_SCROLLS
webView.isVerticalScrollBarEnabled = true
webView.isHorizontalScrollBarEnabled = true
webView.webViewClient =
object : WebViewClient() {
override fun shouldOverrideUrlLoading(
view: WebView,
request: WebResourceRequest,
): Boolean {
if (!request.isForMainFrame) return false
return blockUnsafeCanvasNavigation(controller, currentPageUrlRef, request.url.toString())
}
override fun shouldInterceptRequest(
view: WebView,
request: WebResourceRequest,
): WebResourceResponse? {
val shouldBlock =
CanvasNavigationPolicy.shouldBlockNonGetMainFrame(
method = request.method,
isForMainFrame = request.isForMainFrame,
)
if (!shouldBlock) return null
// shouldOverrideUrlLoading excludes POST navigations and their redirects. WebView does
// not expose those redirect targets, so non-GET main-frame loads fail closed here.
currentPageUrlRef.set(null)
view.post { controller.navigate("") }
return blockedCanvasResponse()
}
override fun onPageStarted(
view: WebView,
url: String?,
favicon: android.graphics.Bitmap?,
) {
currentPageUrlRef.set(url)
}
override fun onReceivedError(
view: WebView,
request: WebResourceRequest,
error: WebResourceError,
) {
if (!isDebuggable || !request.isForMainFrame) return
Log.e("OpenClawWebView", "onReceivedError: ${error.errorCode} ${error.description} ${request.url}")
}
override fun onReceivedHttpError(
view: WebView,
request: WebResourceRequest,
errorResponse: WebResourceResponse,
) {
if (!isDebuggable || !request.isForMainFrame) return
Log.e(
"OpenClawWebView",
"onReceivedHttpError: ${errorResponse.statusCode} ${errorResponse.reasonPhrase} ${request.url}",
)
}
override fun onPageFinished(
view: WebView,
url: String?,
) {
currentPageUrlRef.set(url)
if (isDebuggable) {
Log.d("OpenClawWebView", "onPageFinished: $url")
}
controller.onPageFinished()
}
override fun onRenderProcessGone(
view: WebView,
detail: RenderProcessGoneDetail,
): Boolean {
if (isDebuggable) {
Log.e(
"OpenClawWebView",
"onRenderProcessGone didCrash=${detail.didCrash()} priorityAtExit=${detail.rendererPriorityAtExit()}",
)
}
if (view === currentWebView) {
controller.onRenderProcessGone(view)
destroyWebView(view)
visibility = View.INVISIBLE
}
return true
}
}
webView.webChromeClient =
object : WebChromeClient() {
override fun onConsoleMessage(consoleMessage: ConsoleMessage?): Boolean {
if (!isDebuggable) return false
val msg = consoleMessage ?: return false
Log.d(
"OpenClawWebView",
"console ${msg.messageLevel()} @ ${msg.sourceId()}:${msg.lineNumber()} ${msg.message()}",
)
return false
}
}
// The listener accepts any WebView origin at registration time; native
// dispatch still requires the live URL to be an app-owned bundled page.
val bridge =
CanvasA2UIActionBridge(
isTrustedPage = { isTrustedPage(currentPageUrlRef.get()) },
onMessage = onA2uiMessage,
)
if (WebViewFeature.isFeatureSupported(WebViewFeature.WEB_MESSAGE_LISTENER)) {
WebViewCompat.addWebMessageListener(
webView,
CanvasA2UIActionBridge.interfaceName,
CanvasA2UIActionBridge.allowedOriginRules,
bridge,
)
} else if (isDebuggable) {
Log.w("OpenClawWebView", "WebMessageListener unsupported; canvas actions disabled")
}
addView(
webView,
ViewGroup.LayoutParams(
ViewGroup.LayoutParams.MATCH_PARENT,
ViewGroup.LayoutParams.MATCH_PARENT,
),
)
currentWebView = webView
controller.attach(webView)
}
private fun destroyWebView(webView: WebView) {
if (currentWebView !== webView) return
if (WebViewFeature.isFeatureSupported(WebViewFeature.WEB_MESSAGE_LISTENER)) {
WebViewCompat.removeWebMessageListener(webView, CanvasA2UIActionBridge.interfaceName)
}
removeView(webView)
webView.stopLoading()
webView.destroy()
currentWebView = null
}
}
private fun blockUnsafeCanvasNavigation(
controller: CanvasController,
currentPageUrlRef: AtomicReference<String?>,
rawUrl: String,
): Boolean {
val url = rawUrl.trim()
if (!CanvasNavigationPolicy.shouldBlock(url)) return false
currentPageUrlRef.set(null)
controller.navigate("")
return true
}
private fun blockedCanvasResponse(): WebResourceResponse =
WebResourceResponse(
"text/plain",
"UTF-8",
403,
"Blocked",
mapOf("Cache-Control" to "no-store"),
ByteArrayInputStream(ByteArray(0)),
)
/** Filters WebView postMessage payloads before they enter the A2UI action handler. */
internal class CanvasA2UIActionBridge(
private val isTrustedPage: () -> Boolean,
private val onMessage: (String) -> Unit,
) : WebViewCompat.WebMessageListener {
override fun onPostMessage(
view: WebView,
message: WebMessageCompat,
sourceOrigin: Uri,
isMainFrame: Boolean,
replyProxy: JavaScriptReplyProxy,
) {
if (!isMainFrame) return
postMessage(message.data)
}
fun postMessage(payload: String?) {
val msg = payload?.trim().orEmpty()
if (msg.isEmpty()) return
if (!isTrustedPage()) return
onMessage(msg)
}
companion object {
const val interfaceName: String = "openclawCanvasA2UIAction"
val allowedOriginRules: Set<String> = setOf("*")
}
}
apps/android/app/src/main/java/ai/openclaw/app/ui/CanvasSettingsScreen.kt
-127
View File
@@ -1,127 +0,0 @@
package ai.openclaw.app.ui
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.i18n.nativeString
import ai.openclaw.app.ui.design.ClawPanel
import ai.openclaw.app.ui.design.ClawPrimaryButton
import ai.openclaw.app.ui.design.ClawSecondaryButton
import ai.openclaw.app.ui.design.ClawTheme
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.automirrored.filled.ScreenShare
import androidx.compose.material3.Icon
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
/** Settings detail surface for live canvas status, refresh, and presentation. */
@Composable
internal fun CanvasSettingsScreen(
viewModel: MainViewModel,
onBack: () -> Unit,
) {
val isConnected by viewModel.isConnected.collectAsState()
val currentUrl by viewModel.canvasCurrentUrl.collectAsState()
val hydrated by viewModel.canvasA2uiHydrated.collectAsState()
val rehydratePending by viewModel.canvasRehydratePending.collectAsState()
val rehydrateErrorText by viewModel.canvasRehydrateErrorText.collectAsState()
val hasLivePage = currentUrl?.isNotBlank() == true
val canvasLabel = if (hasLivePage) nativeString("Live page") else nativeString("Home canvas")
LaunchedEffect(isConnected) {
if (isConnected) {
// Refresh once when the gateway comes online so current Canvas status is
// populated before the user manually asks for a rehydrate.
viewModel.refreshHomeCanvasOverviewIfConnected()
}
}
SettingsDetailFrame(
title = nativeString("Canvas"),
subtitle = nativeString("Current screen output and interactive app surface."),
icon = Icons.AutoMirrored.Filled.ScreenShare,
onBack = onBack,
) {
SettingsMetricPanel(
rows =
listOf(
SettingsMetric(nativeString("Connection"), if (isConnected) nativeString("Online") else nativeString("Offline")),
SettingsMetric(nativeString("Surface"), canvasLabel),
SettingsMetric(nativeString("Bridge"), if (hasLivePage && hydrated) nativeString("Ready") else nativeString("Standby")),
),
)
Row(modifier = Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(8.dp)) {
ClawPrimaryButton(
text = if (rehydratePending) nativeString("Refreshing") else nativeString("Refresh Screen"),
onClick = { viewModel.requestCanvasRehydrate(source = "settings_canvas") },
enabled = isConnected && !rehydratePending,
modifier = Modifier.weight(1f),
)
ClawSecondaryButton(
text = if (isConnected) nativeString("Open Screen") else nativeString("Reconnect"),
onClick = if (isConnected) viewModel::showCanvas else viewModel::refreshGatewayConnection,
modifier = Modifier.weight(1f),
)
}
rehydrateErrorText?.let {
ClawPanel {
Text(text = it, style = ClawTheme.type.body, color = ClawTheme.colors.warning)
}
}
ClawPanel(contentPadding = PaddingValues(horizontal = 16.dp, vertical = 14.dp)) {
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
Text(text = canvasLabel, style = ClawTheme.type.section, color = ClawTheme.colors.text, maxLines = 1, overflow = TextOverflow.Ellipsis)
CanvasStandbyPanel(isConnected = isConnected)
}
}
}
}
@Composable
private fun CanvasStandbyPanel(isConnected: Boolean) {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 28.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.Center,
) {
Surface(
modifier = Modifier.size(54.dp),
shape = RoundedCornerShape(ClawTheme.radii.panel),
color = ClawTheme.colors.surfacePressed,
border = BorderStroke(1.dp, ClawTheme.colors.borderStrong),
contentColor = ClawTheme.colors.text,
) {
Box(contentAlignment = Alignment.Center) {
Icon(imageVector = Icons.AutoMirrored.Filled.ScreenShare, contentDescription = null, modifier = Modifier.size(26.dp))
}
}
Text(
text = if (isConnected) nativeString("Screen surface ready") else nativeString("Connect the gateway"),
style = ClawTheme.type.title,
color = ClawTheme.colors.text,
modifier = Modifier.padding(top = 18.dp),
)
Text(
text = if (isConnected) nativeString("Open the current Canvas surface to inspect or interact with it.") else nativeString("Canvas output needs an active gateway connection."),
style = ClawTheme.type.body,
color = ClawTheme.colors.textMuted,
modifier = Modifier.padding(top = 6.dp),
)
}
}
apps/android/app/src/main/java/ai/openclaw/app/ui/SettingsScreens.kt
-4
View File
@@ -187,7 +187,6 @@ internal enum class SettingsRoute {
NodesDevices,
Channels,
Dreaming,
Canvas,
Terminal,
Desktop,
Notifications,
@@ -223,7 +222,6 @@ internal fun SettingsDetailScreen(
SettingsRoute.NodesDevices -> NodesDevicesSettingsScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Channels -> ChannelsSettingsScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Dreaming -> DreamingSettingsScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Canvas -> CanvasSettingsScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Terminal -> TerminalSettingsScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Desktop -> DesktopScreen(viewModel = viewModel, onBack = onBack)
SettingsRoute.Notifications -> NotificationSettingsScreen(viewModel = viewModel, onBack = onBack)
@@ -1277,7 +1275,6 @@ private fun PhoneCapabilitiesScreen(
val locationMode by viewModel.locationMode.collectAsState()
val locationPreciseEnabled by viewModel.locationPreciseEnabled.collectAsState()
val preventSleep by viewModel.preventSleep.collectAsState()
val canvasDebugStatusEnabled by viewModel.canvasDebugStatusEnabled.collectAsState()
val installedAppsSharingEnabled by viewModel.installedAppsSharingEnabled.collectAsState()
val photosAvailable = remember { SensitiveFeatureConfig.photosEnabled }
val backgroundLocationAvailable = remember { SensitiveFeatureConfig.backgroundLocationEnabled }
@@ -1490,7 +1487,6 @@ private fun PhoneCapabilitiesScreen(
::setInstalledAppsSharing,
),
SettingsToggleRow(nativeString("Keep Awake"), nativeString("Keep the node available during active work."), Icons.Default.Bolt, preventSleep, viewModel::setPreventSleep),
SettingsToggleRow(nativeString("Canvas Status"), nativeString("Show screen-sharing debug state."), Icons.AutoMirrored.Filled.ScreenShare, canvasDebugStatusEnabled, viewModel::setCanvasDebugStatusEnabled),
),
)
if (SensitiveFeatureConfig.accessibilityControlEnabled) {
apps/android/app/src/main/java/ai/openclaw/app/ui/ShellScreen.kt
+3 -58
View File
@@ -24,13 +24,11 @@ import ai.openclaw.app.i18n.nativeString
import ai.openclaw.app.i18n.nativeText
import ai.openclaw.app.i18n.resolveNativeTextResource
import ai.openclaw.app.i18n.verbatimText
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.systemagent.SystemAgentChatAccess
import ai.openclaw.app.ui.design.AgentAvatarSource
import ai.openclaw.app.ui.design.ClawAgentAvatar
import ai.openclaw.app.ui.design.ClawDesignTheme
import ai.openclaw.app.ui.design.ClawEmptyState
import ai.openclaw.app.ui.design.ClawIconButton
import ai.openclaw.app.ui.design.ClawPanel
import ai.openclaw.app.ui.design.ClawPlainIconButton
import ai.openclaw.app.ui.design.ClawPrimaryButton
@@ -52,28 +50,23 @@ import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.WindowInsetsSides
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.ime
import androidx.compose.foundation.layout.only
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.safeDrawing
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.automirrored.filled.ExitToApp
import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight
import androidx.compose.material.icons.automirrored.filled.ScreenShare
import androidx.compose.material.icons.filled.ArrowDropDown
import androidx.compose.material.icons.filled.Bolt
import androidx.compose.material.icons.filled.Close
import androidx.compose.material.icons.filled.Cloud
import androidx.compose.material.icons.filled.Favorite
import androidx.compose.material.icons.filled.GraphicEq
@@ -179,8 +172,6 @@ fun ShellScreen(
val requestedHomeDestination by viewModel.requestedHomeDestination.collectAsState()
val pendingTrust by viewModel.pendingGatewayTrust.collectAsState()
val runtimeInitialized by viewModel.runtimeInitialized.collectAsState()
val canvasPresentationState by viewModel.canvasPresentationState.collectAsState()
val canvasVisible = canvasPresentationState == CanvasController.PresentationState.Visible
val gatewayAgents by viewModel.gatewayAgents.collectAsState()
val gatewayDefaultAgentId by viewModel.gatewayDefaultAgentId.collectAsState()
val chatSessionOwnerAgentId by viewModel.chatSessionOwnerAgentId.collectAsState()
@@ -190,11 +181,6 @@ fun ShellScreen(
LaunchedEffect(requestedHomeDestination) {
val destination = requestedHomeDestination ?: return@LaunchedEffect
if (destination == HomeDestination.Screen) {
viewModel.showCanvas()
viewModel.clearRequestedHomeDestination()
return@LaunchedEffect
}
// HomeDestination is a one-shot command from launch intents and settings
// actions; consume it after translating to local shell state.
nav.selectTab(
@@ -202,7 +188,6 @@ fun ShellScreen(
HomeDestination.Connect -> Tab.Overview
HomeDestination.Chat -> Tab.Chat
HomeDestination.Voice -> Tab.Chat
HomeDestination.Screen -> Tab.Overview
HomeDestination.Settings -> Tab.Settings
},
)
@@ -236,14 +221,14 @@ fun ShellScreen(
commandOpen = false
}
LaunchedEffect(commandOpen, canvasVisible, pendingTrust) {
if (commandOpen || canvasVisible || pendingTrust != null) sidebarDrawerState.close()
LaunchedEffect(commandOpen, pendingTrust) {
if (commandOpen || pendingTrust != null) sidebarDrawerState.close()
}
val density = LocalDensity.current
val keyboardVisible = WindowInsets.ime.getBottom(density) > 0
val compactNavigationVisible =
shellBottomNavVisible(keyboardVisible = keyboardVisible, commandOpen = commandOpen) && !canvasVisible
shellBottomNavVisible(keyboardVisible = keyboardVisible, commandOpen = commandOpen)
val activeSidebarDestination =
when {
@@ -400,14 +385,6 @@ fun ShellScreen(
)
}
if (canvasPresentationState != CanvasController.PresentationState.Unmounted) {
CanvasOverlay(
viewModel = viewModel,
visible = canvasVisible,
onClose = viewModel::hideCanvas,
)
}
pendingTrust?.let { prompt ->
// Gateway certificate trust is modal across the shell so navigation
// cannot hide a changed TLS identity prompt.
@@ -422,36 +399,6 @@ fun ShellScreen(
}
}
@Composable
private fun CanvasOverlay(
viewModel: MainViewModel,
visible: Boolean,
onClose: () -> Unit,
) {
BackHandler(enabled = visible, onBack = onClose)
val overlayColor = if (visible) ClawTheme.colors.canvas else Color.Transparent
Box(modifier = Modifier.fillMaxSize().background(overlayColor)) {
// The shell owns system-bar avoidance; arbitrary Canvas pages cannot know Android insets.
CanvasScreen(
viewModel = viewModel,
visible = visible,
modifier = Modifier.fillMaxSize().windowInsetsPadding(WindowInsets.safeDrawing),
)
if (visible) {
ClawIconButton(
icon = Icons.Default.Close,
contentDescription = nativeString("Close Canvas"),
onClick = onClose,
modifier =
Modifier
.align(Alignment.TopEnd)
.windowInsetsPadding(WindowInsets.safeDrawing.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal))
.padding(top = 12.dp, end = 12.dp),
)
}
}
}
/** Modal trust decision for first-seen or changed gateway TLS fingerprints. */
@Composable
private fun GatewayTrustDialog(
@@ -1810,7 +1757,6 @@ private fun SettingsShellScreen(
null
},
SettingsRow(nativeText("Voice"), if (speakerEnabled) nativeText("Speaker on") else nativeText("Speaker muted"), Icons.Default.Mic, route = SettingsRoute.Voice),
SettingsRow(nativeText("Canvas"), nativeText("Screen surface"), Icons.AutoMirrored.Filled.ScreenShare, status = isConnected, route = SettingsRoute.Canvas),
SettingsRow(nativeText("Notifications"), if (notificationForwardingEnabled) nativeText("Smart delivery") else nativeText("Off"), Icons.Default.Notifications, route = SettingsRoute.Notifications),
SettingsRow(nativeText("Phone Capabilities"), if (cameraEnabled) nativeText("Camera enabled") else nativeText("Locked"), Icons.Default.Lock, status = !cameraEnabled, route = SettingsRoute.PhoneCapabilities),
SettingsRow(
@@ -2065,7 +2011,6 @@ internal fun settingsSectionTitleForRoute(route: SettingsRoute): NativeText =
-> nativeText("Agents & automation")
SettingsRoute.Voice,
SettingsRoute.Canvas,
SettingsRoute.Notifications,
SettingsRoute.PhoneCapabilities,
-> nativeText("Phone context & privacy")
apps/android/app/src/test/java/ai/openclaw/app/gateway/InvokeErrorParserTest.kt
+3 -3
View File
@@ -17,9 +17,9 @@ class InvokeErrorParserTest {
@Test
fun parseInvokeErrorMessage_parsesNumericCodePrefix() {
val parsed = parseInvokeErrorMessage("A2UI_HOST_UNAVAILABLE: bundled A2UI host not reachable")
assertEquals("A2UI_HOST_UNAVAILABLE", parsed.code)
assertEquals("bundled A2UI host not reachable", parsed.message)
val parsed = parseInvokeErrorMessage("API2_UNAVAILABLE: service not reachable")
assertEquals("API2_UNAVAILABLE", parsed.code)
assertEquals("service not reachable", parsed.message)
assertTrue(parsed.hadExplicitCode)
}
apps/android/app/src/test/java/ai/openclaw/app/node/CanvasActionTrustTest.kt
-63
View File
@@ -1,63 +0,0 @@
package ai.openclaw.app.node
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class CanvasActionTrustTest {
@Test
fun acceptsBundledScaffoldAsset() {
assertTrue(CanvasActionTrust.isTrustedCanvasActionUrl(CanvasActionTrust.scaffoldAssetUrl))
}
@Test
fun acceptsBundledA2uiAsset() {
assertTrue(CanvasActionTrust.isTrustedCanvasActionUrl(CanvasActionTrust.localA2uiAssetUrl))
}
@Test
fun rejectsRemoteHttpA2uiPageEvenWhenGatewayAdvertised() {
assertFalse(
CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = "http://canvas.example.com:9443/__openclaw__/cap/token/__openclaw__/a2ui/?platform=android",
),
)
}
@Test
fun rejectsRemoteHttpsA2uiPageEvenWhenGatewayAdvertised() {
assertFalse(
CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = "https://canvas.example.com:9443/__openclaw__/cap/token/__openclaw__/a2ui/?platform=android",
),
)
}
@Test
fun rejectsRemoteCanvasPage() {
assertFalse(
CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = "https://canvas.example.com:9443/__openclaw__/canvas/",
),
)
}
@Test
fun rejectsDescendantPathUnderBundledA2uiRoot() {
assertFalse(
CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = "file:///android_asset/CanvasA2UI/child/index.html",
),
)
}
@Test
fun rejectsQueryOrFragmentChangesToBundledA2uiAsset() {
assertFalse(
CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = "${CanvasActionTrust.localA2uiAssetUrl}?platform=android",
),
)
assertFalse(CanvasActionTrust.isTrustedCanvasActionUrl("${CanvasActionTrust.localA2uiAssetUrl}#step2"))
}
}
apps/android/app/src/test/java/ai/openclaw/app/node/CanvasControllerPresentationTest.kt
-38
View File
@@ -1,38 +0,0 @@
package ai.openclaw.app.node
import kotlinx.coroutines.test.runTest
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Test
class CanvasControllerPresentationTest {
@Test
fun presentationStateKeepsTheHostUnmountedUntilFirstShow() {
val controller = CanvasController()
controller.hide()
assertEquals(CanvasController.PresentationState.Unmounted, controller.presentationState.value)
controller.show()
assertEquals(CanvasController.PresentationState.Visible, controller.presentationState.value)
controller.hide()
assertEquals(CanvasController.PresentationState.Hidden, controller.presentationState.value)
controller.show()
assertEquals(CanvasController.PresentationState.Visible, controller.presentationState.value)
controller.releaseHost()
assertEquals(CanvasController.PresentationState.Unmounted, controller.presentationState.value)
}
@Test
fun failedHostHandoffRestoresThePreviousPresentationState() =
runTest {
val controller = CanvasController()
assertFalse(controller.showAndAwaitHost())
assertEquals(CanvasController.PresentationState.Unmounted, controller.presentationState.value)
}
}
apps/android/app/src/test/java/ai/openclaw/app/node/CanvasControllerSnapshotParamsTest.kt
-43
View File
@@ -1,43 +0,0 @@
package ai.openclaw.app.node
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Test
class CanvasControllerSnapshotParamsTest {
@Test
fun parseSnapshotParamsDefaultsToJpeg() {
val params = CanvasController.parseSnapshotParams(null)
assertEquals(CanvasController.SnapshotFormat.Jpeg, params.format)
assertNull(params.quality)
assertNull(params.maxWidth)
}
@Test
fun parseSnapshotParamsParsesPng() {
val params = CanvasController.parseSnapshotParams("""{"format":"png","maxWidth":900}""")
assertEquals(CanvasController.SnapshotFormat.Png, params.format)
assertEquals(900, params.maxWidth)
}
@Test
fun parseSnapshotParamsParsesJpegAliases() {
assertEquals(
CanvasController.SnapshotFormat.Jpeg,
CanvasController.parseSnapshotParams("""{"format":"jpeg"}""").format,
)
assertEquals(
CanvasController.SnapshotFormat.Jpeg,
CanvasController.parseSnapshotParams("""{"format":"jpg"}""").format,
)
}
@Test
fun parseSnapshotParamsClampsQuality() {
val low = CanvasController.parseSnapshotParams("""{"quality":0.01}""")
assertEquals(0.1, low.quality)
val high = CanvasController.parseSnapshotParams("""{"quality":5}""")
assertEquals(1.0, high.quality)
}
}
apps/android/app/src/test/java/ai/openclaw/app/node/CanvasNavigationPolicyTest.kt
-99
View File
@@ -1,99 +0,0 @@
package ai.openclaw.app.node
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Test
class CanvasNavigationPolicyTest {
@Test
fun blocksDeviceLocalWebUrls() {
listOf(
"http://127.0.0.1:18789",
"https://localhost",
"https://canvas.localhost/path",
"http://0.0.0.0:18789",
"http://0/canvas",
"http://[::]/canvas",
"http://[::1]/canvas",
"http://[::1%25lo]/canvas",
"http://[::ffff:127.0.0.1]/canvas",
"http://2130706433/canvas",
"http://0x7f000001/canvas",
"http://017700000001/canvas",
"http://127.1/canvas",
"http://0x7f.1/canvas",
"http://127.0.0.1/a raw space",
"http://127.0.0.1/#raw space",
"http://127.0.0.1\\@example.com/",
"http://%31%32%37.0.0.1:18789/",
"http://%6c%6f%63%61%6c%68%6f%73%74/",
"http://ｌｏｃａｌｈｏｓｔ:18789/",
"http://１２７.０.０.１:18789/",
"http:\\127.0.0.1:18789/",
"http:127.0.0.1:18789/",
).forEach { url ->
assertEquals(url, true, CanvasNavigationPolicy.shouldBlock(url))
assertEquals(url, "", CanvasNavigationPolicy.normalize(url))
}
}
@Test
fun blocksMalformedWebHosts() {
listOf(
"http:///missing-host",
"https://double%252dencoded.example/",
"http://example.com%00.evil/",
).forEach { url -> assertEquals(url, true, CanvasNavigationPolicy.shouldBlock(url)) }
}
@Test
fun keepsRemoteEmulatorBridgeAndBundledUrls() {
val accepted =
listOf(
"https://example.com/canvas",
"https://xn--mnich-kva.example/canvas",
"http://gateway.local:18789/__openclaw__/canvas/",
"http://10.0.2.2:18789/__openclaw__/canvas/",
CanvasActionTrust.scaffoldAssetUrl,
)
accepted.forEach { url ->
assertEquals(url, false, CanvasNavigationPolicy.shouldBlock(url))
assertEquals(url, url, CanvasNavigationPolicy.normalize(" $url "))
}
}
@Test
fun blankAndRootSelectBundledCanvasWithoutBeingSecurityBlocks() {
listOf("", " / ").forEach { url ->
assertEquals(url, false, CanvasNavigationPolicy.shouldBlock(url))
assertEquals(url, "", CanvasNavigationPolicy.normalize(url))
}
}
@Test
fun controllerUsesSharedPolicyForDirectLoads() {
val controller = CanvasController()
controller.navigate("http://127.0.0.1:18789")
assertNull(controller.currentUrl())
controller.navigate("http://10.0.2.2:18789/__openclaw__/canvas/")
assertEquals("http://10.0.2.2:18789/__openclaw__/canvas/", controller.currentUrl())
}
@Test
fun nonGetMainFrameRequestsFailClosedBeforeRedirects() {
assertEquals(
true,
CanvasNavigationPolicy.shouldBlockNonGetMainFrame("POST", isForMainFrame = true),
)
assertEquals(
false,
CanvasNavigationPolicy.shouldBlockNonGetMainFrame("GET", isForMainFrame = true),
)
assertEquals(
false,
CanvasNavigationPolicy.shouldBlockNonGetMainFrame("POST", isForMainFrame = false),
)
}
}
apps/android/app/src/test/java/ai/openclaw/app/node/InvokeCommandRegistryTest.kt
-1
View File
@@ -24,7 +24,6 @@ import org.junit.Test
class InvokeCommandRegistryTest {
private val coreCapabilities =
setOf(
OpenClawCapability.Canvas.rawValue,
OpenClawCapability.Device.rawValue,
OpenClawCapability.Notifications.rawValue,
OpenClawCapability.System.rawValue,
apps/android/app/src/test/java/ai/openclaw/app/node/InvokeDispatcherTest.kt
-64
View File
@@ -4,7 +4,6 @@ import ai.openclaw.app.gateway.GatewaySession
import ai.openclaw.app.gateway.testDeviceIdentityStore
import ai.openclaw.app.protocol.OpenClawCallLogCommand
import ai.openclaw.app.protocol.OpenClawCameraCommand
import ai.openclaw.app.protocol.OpenClawCanvasCommand
import ai.openclaw.app.protocol.OpenClawDeviceCommand
import ai.openclaw.app.protocol.OpenClawLocationCommand
import ai.openclaw.app.protocol.OpenClawMobileUiCommand
@@ -14,9 +13,7 @@ import ai.openclaw.app.protocol.OpenClawSmsCommand
import ai.openclaw.app.protocol.OpenClawTalkCommand
import android.content.Context
import android.content.pm.PackageManager
import android.webkit.WebView
import kotlinx.coroutines.test.runTest
import kotlinx.serialization.json.Json
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Test
@@ -291,58 +288,6 @@ class InvokeDispatcherTest {
assertEquals(listOf("start", "stop", "cancel"), talk.calls)
}
@Test
fun handleInvoke_presentAndHideDriveTheShellOwnedCanvasState() =
runTest {
val appContext = RuntimeEnvironment.getApplication()
val canvas = CanvasController()
val webView = WebView(appContext)
canvas.attach(webView)
val dispatcher = newDispatcher(canvas = canvas)
val present =
dispatcher.handleInvoke(
OpenClawCanvasCommand.Present.rawValue,
"""{"url":"https://example.com/canvas"}""",
)
assertNull(present.error)
assertEquals("https://example.com/canvas", canvas.currentUrl())
assertEquals(CanvasController.PresentationState.Visible, canvas.presentationState.value)
val hide = dispatcher.handleInvoke(OpenClawCanvasCommand.Hide.rawValue, null)
assertNull(hide.error)
assertEquals(CanvasController.PresentationState.Hidden, canvas.presentationState.value)
canvas.releaseHost()
webView.destroy()
}
@Test
fun handleInvoke_rejectsBackgroundCanvasPresentationBeforeMountingAHost() =
runTest {
val canvas = CanvasController()
val result =
newDispatcher(isForeground = false, canvas = canvas)
.handleInvoke(OpenClawCanvasCommand.Present.rawValue, """{"url":"https://example.com"}""")
assertEquals("NODE_BACKGROUND_UNAVAILABLE", result.error?.code)
assertEquals(CanvasController.PresentationState.Unmounted, canvas.presentationState.value)
}
@Test
fun handleInvoke_doesNotCommitNavigationWhenTheShellHostCannotAttach() =
runTest {
val canvas = CanvasController()
val result =
newDispatcher(canvas = canvas)
.handleInvoke(OpenClawCanvasCommand.Present.rawValue, """{"url":"https://example.com"}""")
assertEquals("NODE_BACKGROUND_UNAVAILABLE", result.error?.code)
assertNull(canvas.currentUrl())
assertEquals(CanvasController.PresentationState.Unmounted, canvas.presentationState.value)
}
private fun newDispatcher(
isForeground: Boolean = true,
cameraEnabled: Boolean = false,
@@ -359,12 +304,10 @@ class InvokeDispatcherTest {
motionPedometerAvailable: Boolean = false,
mobileUiAvailable: Boolean = false,
talkHandler: TalkHandler = InvokeDispatcherFakeTalkHandler(),
canvas: CanvasController = CanvasController(),
): InvokeDispatcher {
val appContext = RuntimeEnvironment.getApplication()
shadowOf(appContext.packageManager).setSystemFeature(PackageManager.FEATURE_TELEPHONY, smsTelephonyAvailable)
return InvokeDispatcher(
canvas = canvas,
cameraHandler = newCameraHandler(appContext),
locationHandler =
LocationHandler.forTesting(
@@ -384,11 +327,6 @@ class InvokeDispatcherTest {
calendarHandler = CalendarHandler.forTesting(appContext, InvokeDispatcherFakeCalendarDataSource()),
motionHandler = MotionHandler.forTesting(appContext, InvokeDispatcherFakeMotionDataSource()),
smsHandler = SmsHandler(SmsManager(appContext)),
a2uiHandler =
A2UIHandler(
canvas = canvas,
json = Json { ignoreUnknownKeys = true },
),
debugHandler = DebugHandler(appContext, testDeviceIdentityStore(appContext)),
callLogHandler = CallLogHandler.forTesting(appContext, InvokeDispatcherFakeCallLogDataSource()),
mobileUiHandler = MobileUiHandler(),
@@ -403,8 +341,6 @@ class InvokeDispatcherTest {
photosAvailable = { photosAvailable },
installedAppsSharingEnabled = { installedAppsSharingEnabled },
debugBuild = { debugBuild },
onCanvasA2uiPush = {},
onCanvasA2uiReset = {},
motionActivityAvailable = { motionActivityAvailable },
motionPedometerAvailable = { motionPedometerAvailable },
mobileUiAvailable = { mobileUiAvailable },
apps/android/app/src/test/java/ai/openclaw/app/protocol/OpenClawCanvasA2UIActionTest.kt
-63
View File
@@ -1,63 +0,0 @@
package ai.openclaw.app.protocol
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.jsonObject
import org.junit.Assert.assertEquals
import org.junit.Test
class OpenClawCanvasA2UIActionTest {
@Test
fun extractActionNameAcceptsNameOrAction() {
val nameObj = Json.parseToJsonElement("{\"name\":\"Hello\"}").jsonObject
assertEquals("Hello", OpenClawCanvasA2UIAction.extractActionName(nameObj))
val actionObj = Json.parseToJsonElement("{\"action\":\"Wave\"}").jsonObject
assertEquals("Wave", OpenClawCanvasA2UIAction.extractActionName(actionObj))
val fallbackObj =
Json.parseToJsonElement("{\"name\":\" \",\"action\":\"Fallback\"}").jsonObject
assertEquals("Fallback", OpenClawCanvasA2UIAction.extractActionName(fallbackObj))
}
@Test
fun formatAgentMessageMatchesSharedSpec() {
val msg =
OpenClawCanvasA2UIAction.formatAgentMessage(
actionName = "Get Weather",
sessionKey = "main",
surfaceId = "main",
sourceComponentId = "btnWeather",
host = "Peter’s iPad",
instanceId = "ipad16,6",
contextJson = "{\"city\":\"Vienna\"}",
)
assertEquals(
"CANVAS_A2UI action=Get_Weather session=main surface=main component=btnWeather host=Peter_s_iPad instance=ipad16_6 ctx={\"city\":\"Vienna\"} default=update_canvas",
msg,
)
}
@Test
fun jsDispatchA2uiStatusIsStable() {
val js = OpenClawCanvasA2UIAction.jsDispatchA2UIActionStatus(actionId = "a1", ok = true, error = null)
assertEquals(
"window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail: { id: \"a1\", ok: true, error: \"\" } }));",
js,
)
}
@Test
fun jsDispatchA2uiStatusQuotesControlCharacters() {
val js =
OpenClawCanvasA2UIAction.jsDispatchA2UIActionStatus(
actionId = "a1\n\u2028\"",
ok = false,
error = "parse failed\n\t\u2029\\",
)
assertEquals(
"window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail: { id: \"a1\\n\\u2028\\\"\", ok: false, error: \"parse failed\\n\\t\\u2029\\\\\" } }));",
js,
)
}
}
apps/android/app/src/test/java/ai/openclaw/app/protocol/OpenClawProtocolConstantsTest.kt
-2
View File
@@ -17,8 +17,6 @@ class OpenClawProtocolConstantsTest {
fun generatedCommandGroupsMatchTheirNamespaces() {
val groups =
listOf(
OpenClawCanvasCommand.NamespacePrefix to OpenClawCanvasCommand.entries.map { it.rawValue },
OpenClawCanvasA2UICommand.NamespacePrefix to OpenClawCanvasA2UICommand.entries.map { it.rawValue },
OpenClawCameraCommand.NamespacePrefix to OpenClawCameraCommand.entries.map { it.rawValue },
OpenClawSmsCommand.NamespacePrefix to OpenClawSmsCommand.entries.map { it.rawValue },
OpenClawTalkCommand.NamespacePrefix to OpenClawTalkCommand.entries.map { it.rawValue },
apps/android/app/src/test/java/ai/openclaw/app/ui/CanvasA2UIActionBridgeTest.kt
-50
View File
@@ -1,50 +0,0 @@
package ai.openclaw.app.ui
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
class CanvasA2UIActionBridgeTest {
@Test
fun forwardsTrimmedPayloadFromTrustedPage() {
val forwarded = mutableListOf<String>()
val bridge =
CanvasA2UIActionBridge(
isTrustedPage = { true },
onMessage = { forwarded += it },
)
bridge.postMessage(" {\"ok\":true} ")
assertEquals(listOf("{\"ok\":true}"), forwarded)
}
@Test
fun rejectsPayloadFromUntrustedPage() {
val forwarded = mutableListOf<String>()
val bridge =
CanvasA2UIActionBridge(
isTrustedPage = { false },
onMessage = { forwarded += it },
)
bridge.postMessage("{\"ok\":true}")
assertTrue(forwarded.isEmpty())
}
@Test
fun rejectsBlankPayloadBeforeForwarding() {
val forwarded = mutableListOf<String>()
val bridge =
CanvasA2UIActionBridge(
isTrustedPage = { true },
onMessage = { forwarded += it },
)
bridge.postMessage(" ")
bridge.postMessage(null)
assertTrue(forwarded.isEmpty())
}
}
apps/ios/README.md
+3 -3
View File
@@ -297,13 +297,13 @@ gateway can only send pushes for iOS devices that paired with that gateway.
- Pairing via QR or setup code flow (`/pair qr` or `/pair`, then `/pair approve` in Telegram).
- Gateway connection via discovery or manual host/port with TLS fingerprint trust prompt.
- One Chat surface for text, realtime voice, dictation, and voice notes through the operator gateway session.
- iOS node commands in foreground: camera snap/clip, canvas present/navigate/eval/snapshot, screen record, location, contacts, calendar, reminders, photos, motion, local notifications.
- iOS node commands in foreground: camera snap/clip, screen record, location, contacts, calendar, reminders, photos, motion, local notifications.
- Authenticated background `node.presence.alive` beacons that update gateway last-seen metadata when the app moves between foreground and background, without treating suspended sockets as connected.
- Share extension deep-link forwarding into the connected gateway session.
## Computer Use Relationship
The iOS app is not a Codex Computer Use backend. Computer Use and `cua-driver mcp` are macOS desktop-control paths; iOS exposes device capabilities as OpenClaw node commands through the gateway. Agents can drive the iPhone canvas, camera, screen, location, voice, and other node capabilities with `node.invoke`, subject to iOS foreground/background limits.
The iOS app is not a Codex Computer Use backend. Computer Use and `cua-driver mcp` are macOS desktop-control paths; iOS exposes device capabilities as OpenClaw node commands through the gateway. Agents can drive the iPhone camera, screen recorder, location, voice, and other node capabilities with `node.invoke`, subject to iOS foreground/background limits.
## Location Automation Use Case (Testing)
@@ -339,7 +339,7 @@ Pass criteria:
## Known Issues / Limitations / Problems
- Foreground-first: iOS can suspend sockets in background; reconnect recovery is still being tuned.
- Background command limits are strict: `canvas.*`, `camera.*`, `screen.*`, and `talk.*` are blocked when backgrounded.
- Background command limits are strict: `camera.*`, `screen.*`, and `talk.*` are blocked when backgrounded.
- Background location requires `Always` location permission.
- Pairing/auth errors intentionally pause reconnect loops until a human fixes auth/pairing state.
- Voice Wake and Talk contend for the same microphone; Talk suppresses wake capture while active.
apps/ios/Sources/Design/SettingsProTab.swift
-1
View File
@@ -38,7 +38,6 @@ struct SettingsProTab: View {
@AppStorage("gateway.manual.port") var manualGatewayPort: Int = 18789
@AppStorage("gateway.manual.tls") var manualGatewayTLS: Bool = true
@AppStorage("gateway.discovery.debugLogs") var discoveryDebugLogsEnabled: Bool = false
@AppStorage("canvas.debugStatusEnabled") var canvasDebugStatusEnabled: Bool = false
@AppStorage("gateway.setupCode") var setupCode: String = ""
@AppStorage("gateway.onboardingComplete") var onboardingComplete: Bool = false
@AppStorage("gateway.hasConnectedOnce") var hasConnectedOnce: Bool = false
apps/ios/Sources/Design/SettingsProTabActions.swift
+2 -4
View File
@@ -1189,10 +1189,8 @@ extension SettingsProTab {
var lines: [String] = []
if let lanHost = gateway.lanHost { lines.append("LAN: \(lanHost)") }
if let tailnet = gateway.tailnetDns { lines.append("Tailnet: \(tailnet)") }
let gw = gateway.gatewayPort.map(String.init)
let canvas = gateway.canvasPort.map(String.init)
if gw != nil || canvas != nil {
lines.append("Ports: gateway \(gw ?? "-") / canvas \(canvas ?? "-")")
if let gatewayPort = gateway.gatewayPort {
lines.append("Port: \(gatewayPort)")
}
return lines.isEmpty ? [gateway.debugID] : lines
}
apps/ios/Sources/Design/SettingsProTabSections.swift
-1
View File
@@ -1515,7 +1515,6 @@ extension SettingsProTab {
self.settingsToggle("Discovery Debug Logs", isOn: self.$discoveryDebugLogsEnabled) { enabled in
self.gatewayController.setDiscoveryDebugLoggingEnabled(enabled)
}
self.settingsToggle("Debug Screen Status", isOn: self.$canvasDebugStatusEnabled)
NavigationLink {
GatewayDiscoveryDebugLogView()
} label: {
apps/ios/Sources/Gateway/GatewayConnectionController+Capabilities.swift
+1 -12
View File
@@ -111,10 +111,7 @@ extension GatewayConnectionController {
}
private func currentCaps() -> [String] {
var caps = [
OpenClawCapability.canvas.rawValue,
OpenClawCapability.screen.rawValue,
]
var caps = [OpenClawCapability.screen.rawValue]
// Default-on: if the key doesn't exist yet, treat it as enabled.
let cameraEnabled =
@@ -151,14 +148,6 @@ extension GatewayConnectionController {
private func currentCommands() -> [String] {
var commands: [String] = [
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
OpenClawCanvasCommand.evalJS.rawValue,
OpenClawCanvasCommand.snapshot.rawValue,
OpenClawCanvasA2UICommand.push.rawValue,
OpenClawCanvasA2UICommand.pushJSONL.rawValue,
OpenClawCanvasA2UICommand.reset.rawValue,
OpenClawScreenCommand.record.rawValue,
OpenClawSystemCommand.notify.rawValue,
OpenClawChatCommand.push.rawValue,
apps/ios/Sources/Gateway/GatewayDiscoveryModel.swift
-3
View File
@@ -24,7 +24,6 @@ final class GatewayDiscoveryModel {
var lanHost: String?
var tailnetDns: String?
var gatewayPort: Int?
var canvasPort: Int?
var tlsEnabled: Bool
var tlsFingerprintSha256: String?
var cliPath: String?
@@ -37,7 +36,6 @@ final class GatewayDiscoveryModel {
lhs.lanHost == rhs.lanHost &&
lhs.tailnetDns == rhs.tailnetDns &&
lhs.gatewayPort == rhs.gatewayPort &&
lhs.canvasPort == rhs.canvasPort &&
lhs.tlsEnabled == rhs.tlsEnabled &&
lhs.tlsFingerprintSha256 == rhs.tlsFingerprintSha256 &&
lhs.cliPath == rhs.cliPath
@@ -100,7 +98,6 @@ final class GatewayDiscoveryModel {
lanHost: Self.txtValue(txt, key: "lanHost"),
tailnetDns: Self.txtValue(txt, key: "tailnetDns"),
gatewayPort: Self.txtIntValue(txt, key: "gatewayPort"),
canvasPort: Self.txtIntValue(txt, key: "canvasPort"),
tlsEnabled: Self.txtBoolValue(txt, key: "gatewayTls"),
tlsFingerprintSha256: Self.txtValue(txt, key: "gatewayTlsSha256"),
cliPath: Self.txtValue(txt, key: "cliPath"))
apps/ios/Sources/Gateway/GatewayQuickSetupSheet.swift
-1
View File
@@ -516,7 +516,6 @@ extension GatewayDiscoveryModel.DiscoveredGateway {
lanHost: "openclaw.local",
tailnetDns: nil,
gatewayPort: 18789,
canvasPort: 18789,
tlsEnabled: true,
tlsFingerprintSha256: "preview",
cliPath: "/opt/homebrew/bin/openclaw")
apps/ios/Sources/Model/NodeAppModel+Canvas.swift
-35
View File
@@ -1,35 +0,0 @@
import Foundation
import OpenClawKit
enum A2UIReadyState {
case ready
case hostUnavailable
}
extension NodeAppModel {
func showA2UIOnConnectIfNeeded() async {
await MainActor.run {
// Keep the bundled home canvas as the default connected view.
// Agents can still explicitly present a remote or local canvas later.
self.screen.showDefaultCanvas()
}
}
func ensureA2UIReadyWithCapabilityRefresh(timeoutMs: Int = 5000) async -> A2UIReadyState {
if self.screen.isShowingLocalA2UI(),
await self.screen.waitForA2UIReady(timeoutMs: timeoutMs)
{
return .ready
}
self.screen.showLocalA2UI()
if await self.screen.waitForA2UIReady(timeoutMs: timeoutMs) {
return .ready
}
return .hostUnavailable
}
func showLocalCanvasOnDisconnect() {
self.screen.showDefaultCanvas()
}
}
apps/ios/Sources/Model/NodeAppModel.swift
+3 -277
View File
@@ -357,7 +357,6 @@ final class NodeAppModel {
}
var isBackgrounded: Bool = false
let screen: ScreenController
private let camera: any CameraServicing
private(set) var preferredCameraFacing: OpenClawCameraFacing
private let screenRecorder: any ScreenRecordingServicing
@@ -415,7 +414,6 @@ final class NodeAppModel {
var selectedAgentId: String?
var gatewayDefaultAgentId: String?
var gatewayAgents: [AgentSummary] = []
var homeCanvasRevision: Int = 0
var lastShareEventText: String = "No share events yet."
var openChatRequestID: Int = 0
var newChatRequestID: Int = 0
@@ -731,7 +729,6 @@ final class NodeAppModel {
self.mainSessionBaseKey = identity.mainSessionKey
self.gatewayDefaultAgentId = identity.defaultAgentID
self.synchronizeTalkSessionKey()
self.homeCanvasRevision &+= 1
}
func loadCachedChatSessions() async -> [OpenClawChatSessionEntry] {
@@ -929,7 +926,6 @@ final class NodeAppModel {
activationState: "notActivated")
init(
screen: ScreenController = ScreenController(),
camera: any CameraServicing = CameraController(),
screenRecorder: any ScreenRecordingServicing = ScreenRecordService(),
locationService: any LocationServicing = LocationService(),
@@ -946,7 +942,6 @@ final class NodeAppModel {
voiceNoteRecorder: OpenClawVoiceNoteRecorder = OpenClawVoiceNoteRecorder(),
audioAdmissionInitiallyAllowed: Bool = true)
{
self.screen = screen
self.camera = camera
self.preferredCameraFacing = Self.cameraFacingPreference(
rawValue: UserDefaults.standard.string(forKey: Self.preferredCameraFacingKey))
@@ -1059,99 +1054,6 @@ final class NodeAppModel {
mode: self.locationMode(),
authorizationStatus: snapshot.authorizationStatus)
}
// Wire up deep links from canvas taps
self.screen.onDeepLink = { [weak self] url in
guard let self else { return }
Task { @MainActor in
await self.handleDeepLink(url: url)
}
}
// Wire up A2UI action clicks (buttons, etc.)
self.screen.onA2UIAction = { [weak self] body in
guard let self else { return }
Task { @MainActor in
await self.handleCanvasA2UIAction(body: body)
}
}
}
func handleCanvasA2UIAction(body: [String: Any]) async {
let userActionAny = body["userAction"] ?? body
let userAction: [String: Any] = {
if let dict = userActionAny as? [String: Any] { return dict }
if let dict = userActionAny as? [AnyHashable: Any] {
return dict.reduce(into: [String: Any]()) { acc, pair in
guard let key = pair.key as? String else { return }
acc[key] = pair.value
}
}
return [:]
}()
guard !userAction.isEmpty else { return }
guard let name = OpenClawCanvasA2UIAction.extractActionName(userAction) else { return }
let actionId: String = {
let id = (userAction["id"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
return id.isEmpty ? UUID().uuidString : id
}()
let surfaceId: String = {
let raw = (userAction["surfaceId"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
return raw.isEmpty ? "main" : raw
}()
let sourceComponentId: String = {
let raw = (userAction[
"sourceComponentId",
] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
return raw.isEmpty ? "-" : raw
}()
let host = NodeDisplayName.resolve(
existing: UserDefaults.standard.string(forKey: "node.displayName"),
deviceName: UIDevice.current.name,
interfaceIdiom: UIDevice.current.userInterfaceIdiom)
let instanceId = (UserDefaults.standard.string(forKey: "node.instanceId") ?? "ios-node").lowercased()
let contextJSON = OpenClawCanvasA2UIAction.compactJSON(userAction["context"])
let sessionKey = mainSessionKey
let messageContext = OpenClawCanvasA2UIAction.AgentMessageContext(
actionName: name,
session: .init(key: sessionKey, surfaceId: surfaceId),
component: .init(id: sourceComponentId, host: host, instanceId: instanceId),
contextJSON: contextJSON)
let message = OpenClawCanvasA2UIAction.formatAgentMessage(messageContext)
let ok: Bool
var errorText: String?
if await !isGatewayConnected() {
ok = false
errorText = "gateway not connected"
} else {
do {
try await sendAgentRequest(link: AgentDeepLink(
message: message,
sessionKey: sessionKey,
thinking: "low",
deliver: false,
to: nil,
channel: nil,
timeoutSeconds: nil,
key: actionId))
ok = true
} catch {
ok = false
errorText = error.localizedDescription
}
}
let js = OpenClawCanvasA2UIAction.jsDispatchA2UIActionStatus(actionId: actionId, ok: ok, error: errorText)
do {
_ = try await self.screen.eval(javaScript: js)
} catch {
// ignore
}
}
func setScenePhase(_ phase: ScenePhase) {
@@ -1348,7 +1250,6 @@ final class NodeAppModel {
LiveActivityManager.shared.endActivity(reason: "background_idle")
self.gatewayServerName = nil
self.gatewayRemoteAddress = nil
self.showLocalCanvasOnDisconnect()
}
}
}
@@ -1653,7 +1554,6 @@ final class NodeAppModel {
private static let apnsDeviceTokenUserDefaultsKey = "push.apns.deviceTokenHex"
private static let deepLinkKeyUserDefaultsKey = "deeplink.agent.key"
private static let canvasUnattendedDeepLinkKey: String = NodeAppModel.generateDeepLinkKey()
private func refreshBrandingFromGateway(shouldApply: () -> Bool = { true }) async {
do {
@@ -1677,7 +1577,6 @@ final class NodeAppModel {
self.mainSessionBaseKey = mainKey
self.gatewaySessionScope = scope
self.synchronizeTalkSessionKey()
self.homeCanvasRevision &+= 1
}
} catch {
if let gatewayError = error as? GatewayResponseError {
@@ -1721,7 +1620,6 @@ final class NodeAppModel {
self.focusedChatSessionKey = nil
}
self.synchronizeTalkSessionKey()
self.homeCanvasRevision &+= 1
}
if let routingIdentity {
await sourceStore.storeSessionRoutingIdentity(routingIdentity)
@@ -1761,7 +1659,6 @@ final class NodeAppModel {
self.focusedChatSessionKey = nil
}
self.synchronizeTalkSessionKey()
self.homeCanvasRevision &+= 1
if let relay = ShareGatewayRelaySettings.loadConfig() {
ShareGatewayRelaySettings.saveConfig(
ShareGatewayRelayConfig(
@@ -2264,7 +2161,7 @@ final class NodeAppModel {
ok: false,
error: OpenClawNodeError(
code: .backgroundUnavailable,
message: "NODE_BACKGROUND_UNAVAILABLE: canvas/camera/screen/talk commands require foreground"))
message: "NODE_BACKGROUND_UNAVAILABLE: camera/screen/talk commands require foreground"))
}
if command.hasPrefix("camera."), !isCameraEnabled() {
@@ -2348,7 +2245,7 @@ final class NodeAppModel {
}
private func isBackgroundRestricted(_ command: String) -> Bool {
command.hasPrefix("canvas.") || command.hasPrefix("camera.") || command.hasPrefix("screen.") ||
command.hasPrefix("camera.") || command.hasPrefix("screen.") ||
command.hasPrefix("talk.")
}
@@ -2410,131 +2307,6 @@ final class NodeAppModel {
return try Self.successfulInvokeResponse(req, payload: payload)
}
private func handleCanvasInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
switch req.command {
case OpenClawCanvasCommand.present.rawValue:
// iOS ignores placement hints; canvas always fills the screen.
let params = (try? Self.decodeParams(OpenClawCanvasPresentParams.self, from: req.paramsJSON)) ??
OpenClawCanvasPresentParams()
let url = params.url?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
if url.isEmpty {
self.screen.presentDefaultCanvas()
} else {
self.screen.present(urlString: url)
}
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.hide.rawValue:
self.screen.hideCanvas()
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.navigate.rawValue:
let params = try Self.decodeParams(OpenClawCanvasNavigateParams.self, from: req.paramsJSON)
let trimmedURL = params.url.trimmingCharacters(in: .whitespacesAndNewlines)
self.screen.present(urlString: trimmedURL)
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.evalJS.rawValue:
let params = try Self.decodeParams(OpenClawCanvasEvalParams.self, from: req.paramsJSON)
let result = try await screen.eval(javaScript: params.javaScript)
let payload = try Self.encodePayload(["result": result])
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload)
case OpenClawCanvasCommand.snapshot.rawValue:
let params = try? Self.decodeParams(OpenClawCanvasSnapshotParams.self, from: req.paramsJSON)
let format = params?.format ?? .jpeg
let maxWidth: CGFloat? = {
if let raw = params?.maxWidth, raw > 0 { return CGFloat(raw) }
// Keep default snapshots comfortably below the gateway client's maxPayload.
// For full-res, clients should explicitly request a larger maxWidth.
return switch format {
case .png: 900
case .jpeg: 1600
}
}()
let base64 = try await screen.snapshotBase64(
maxWidth: maxWidth,
format: format,
quality: params?.quality)
let payload = try Self.encodePayload([
"format": format == .jpeg ? "jpeg" : "png",
"base64": base64,
])
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload)
default:
return Self.unknownInvokeResponse(req)
}
}
private func handleCanvasA2UIInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
let command = req.command
switch command {
case OpenClawCanvasA2UICommand.reset.rawValue:
switch await ensureA2UIReadyWithCapabilityRefresh(timeoutMs: 5000) {
case .ready:
break
case .hostUnavailable:
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: OpenClawNodeError(
code: .unavailable,
message: "A2UI_HOST_UNAVAILABLE: bundled A2UI host not reachable"))
}
let json = try await screen.eval(javaScript: """
(() => {
const host = globalThis.openclawA2UI;
if (!host) return JSON.stringify({ ok: false, error: "missing openclawA2UI" });
return JSON.stringify(host.reset());
})()
""")
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json)
case OpenClawCanvasA2UICommand.push.rawValue, OpenClawCanvasA2UICommand.pushJSONL.rawValue:
let messages: [OpenClawKit.AnyCodable]
if command == OpenClawCanvasA2UICommand.pushJSONL.rawValue {
let params = try Self.decodeParams(OpenClawCanvasA2UIPushJSONLParams.self, from: req.paramsJSON)
messages = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(params.jsonl)
} else {
do {
let params = try Self.decodeParams(OpenClawCanvasA2UIPushParams.self, from: req.paramsJSON)
messages = params.messages
} catch {
// Be forgiving: some clients still send JSONL payloads to `canvas.a2ui.push`.
let params = try Self.decodeParams(OpenClawCanvasA2UIPushJSONLParams.self, from: req.paramsJSON)
messages = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(params.jsonl)
}
}
switch await ensureA2UIReadyWithCapabilityRefresh(timeoutMs: 5000) {
case .ready:
break
case .hostUnavailable:
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: OpenClawNodeError(
code: .unavailable,
message: "A2UI_HOST_UNAVAILABLE: bundled A2UI host not reachable"))
}
let messagesJSON = try OpenClawCanvasA2UIJSONL.encodeMessagesJSONArray(messages)
let js = """
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return JSON.stringify({ ok: false, error: "missing openclawA2UI" });
const messages = \(messagesJSON);
return JSON.stringify(host.applyMessages(messages));
} catch (e) {
return JSON.stringify({ ok: false, error: String(e?.message ?? e) });
}
})()
"""
let resultJSON = try await screen.eval(javaScript: js)
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: resultJSON)
default:
return Self.unknownInvokeResponse(req)
}
}
private func handleCameraInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
switch req.command {
case OpenClawCameraCommand.list.rawValue:
@@ -3283,20 +3055,6 @@ extension NodeAppModel {
register([OpenClawLocationCommand.get.rawValue]) { try await $0.handleLocationInvoke($1) }
register([
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
OpenClawCanvasCommand.evalJS.rawValue,
OpenClawCanvasCommand.snapshot.rawValue,
]) { try await $0.handleCanvasInvoke($1) }
register([
OpenClawCanvasA2UICommand.reset.rawValue,
OpenClawCanvasA2UICommand.push.rawValue,
OpenClawCanvasA2UICommand.pushJSONL.rawValue,
]) { try await $0.handleCanvasA2UIInvoke($1) }
register([
OpenClawCameraCommand.list.rawValue,
OpenClawCameraCommand.snap.rawValue,
@@ -4024,7 +3782,6 @@ extension NodeAppModel {
self.chatSessionRoutingRestoreTask = nil
self.synchronizeTalkSessionKey()
ShareGatewayRelaySettings.clearConfig()
showLocalCanvasOnDisconnect()
}
private func disableGatewayAutoReconnect() {
@@ -4095,7 +3852,6 @@ extension NodeAppModel {
self.focusedChatSessionKey = nil
}
self.synchronizeTalkSessionKey()
self.homeCanvasRevision &+= 1
self.apnsLastRegisteredTokenHex = nil
self.apnsLastRegisteredGatewayStableID = nil
self.chatSessionRoutingRestoreTask = Task { [weak self] in
@@ -4142,7 +3898,6 @@ extension NodeAppModel {
self.gatewayServerName = nil
self.gatewayRemoteAddress = nil
self.gatewayConnected = false
showLocalCanvasOnDisconnect()
if problem.pauseReconnect {
self.gatewayAutoReconnectEnabled = false
}
@@ -4593,7 +4348,6 @@ extension NodeAppModel {
_ = GatewaySettingsStore.markGatewayConnected(
stableID: stableID,
atMs: Int(Date().timeIntervalSince1970 * 1000))
self.screen.errorText = nil
UserDefaults.standard.set(true, forKey: "gateway.autoconnect")
LiveActivityManager.shared.handleReconnect()
guard self.isCurrentGatewayRoute(generation: routeGeneration, stableID: stableID) else { return }
@@ -4613,8 +4367,6 @@ extension NodeAppModel {
self.gatewayRemoteAddress = address
}
guard self.isCurrentGatewayRoute(generation: routeGeneration, stableID: stableID) else { return }
await showA2UIOnConnectIfNeeded()
guard self.isCurrentGatewayRoute(generation: routeGeneration, stableID: stableID) else { return }
let shouldContinue = self.gatewayRouteCheck(
generation: routeGeneration,
stableID: stableID)
@@ -4949,7 +4701,6 @@ extension NodeAppModel {
self.gatewayServerName = nil
self.gatewayRemoteAddress = nil
self.gatewayConnected = false
self.showLocalCanvasOnDisconnect()
}
GatewayDiagnostics.log("gateway disconnected reason: \(reason)")
},
@@ -5083,7 +4834,6 @@ extension NodeAppModel {
self.gatewayServerName = nil
self.gatewayRemoteAddress = nil
self.gatewayConnected = false
self.showLocalCanvasOnDisconnect()
}
}
@@ -5113,7 +4863,6 @@ extension NodeAppModel {
// Retain the last verified routing contract for offline capture; reconnect compares it
// with the live gateway before replay.
self.synchronizeTalkSessionKey()
self.showLocalCanvasOnDisconnect()
}
private func shouldRequestOperatorApprovalScope(
@@ -5380,7 +5129,6 @@ extension NodeAppModel {
self.talkMode.setEnabled(false)
self.talkMode.statusText = "Demo mode only"
self.configureLocalGatewayFixtureSession(agents: AppleReviewDemoMode.agents)
self.homeCanvasRevision &+= 1
}
func enterScreenshotFixtureMode() {
@@ -5394,7 +5142,6 @@ extension NodeAppModel {
self.hasOperatorAdminScope = true
self.configureLocalGatewayFixtureSession(agents: ScreenshotFixtureMode.agents)
self.talkMode.enterScreenshotFixtureMode()
self.homeCanvasRevision &+= 1
}
}
@@ -10213,14 +9960,11 @@ extension NodeAppModel {
"agent deep link messageChars=\(message.count) url=\(originalURL.absoluteString, privacy: .public)")
if message.count > IOSDeepLinkAgentPolicy.maxMessageChars {
self.screen.errorText = "Deep link too large (message exceeds "
+ "\(IOSDeepLinkAgentPolicy.maxMessageChars) characters)."
self.recordShareEvent("Rejected: message too large (\(message.count) chars).")
return
}
guard await self.isGatewayConnected() else {
self.screen.errorText = "Gateway not connected (cannot forward deep link)."
self.recordShareEvent("Failed: gateway not connected.")
self.deepLinkLogger.error("agent deep link rejected: gateway not connected")
return
@@ -10229,7 +9973,6 @@ extension NodeAppModel {
let allowUnattended = self.isUnattendedDeepLinkAllowed(link.key)
if !allowUnattended {
if message.count > IOSDeepLinkAgentPolicy.maxUnkeyedConfirmChars {
self.screen.errorText = "Deep link blocked (message too long without key)."
self.recordShareEvent(
"Rejected: deep link over \(IOSDeepLinkAgentPolicy.maxUnkeyedConfirmChars) chars without key.")
self.deepLinkLogger.error(
@@ -10322,7 +10065,6 @@ extension NodeAppModel {
guard let prompt = pendingAgentDeepLinkPrompt else { return }
self.pendingAgentDeepLinkPrompt = nil
guard await self.isGatewayConnected() else {
self.screen.errorText = "Gateway not connected (cannot forward deep link)."
self.recordShareEvent("Failed: gateway not connected.")
self.deepLinkLogger.error("agent deep link approval failed: gateway not connected")
return
@@ -10333,7 +10075,6 @@ extension NodeAppModel {
func declinePendingAgentDeepLinkPrompt() {
guard self.pendingAgentDeepLinkPrompt != nil else { return }
self.pendingAgentDeepLinkPrompt = nil
self.screen.errorText = "Deep link cancelled."
self.recordShareEvent("Cancelled: deep link confirmation declined.")
self.deepLinkLogger.info("agent deep link cancelled by local user")
}
@@ -10395,12 +10136,10 @@ extension NodeAppModel {
private func submitAgentDeepLink(_ link: AgentDeepLink, messageCharCount: Int) async {
do {
try await self.sendAgentRequest(link: link)
self.screen.errorText = nil
self.recordShareEvent("Sent to gateway (\(messageCharCount) chars).")
self.deepLinkLogger.info("agent deep link forwarded to gateway")
self.openChatRequestID &+= 1
} catch {
self.screen.errorText = "Agent request failed: \(error.localizedDescription)"
self.recordShareEvent("Failed: \(error.localizedDescription)")
self.deepLinkLogger.error("agent deep link send failed: \(error.localizedDescription, privacy: .public)")
}
@@ -10422,7 +10161,7 @@ extension NodeAppModel {
private func isUnattendedDeepLinkAllowed(_ key: String?) -> Bool {
let normalizedKey = key?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
guard !normalizedKey.isEmpty else { return false }
return normalizedKey == Self.canvasUnattendedDeepLinkKey || normalizedKey == Self.expectedDeepLinkKey()
return normalizedKey == Self.expectedDeepLinkKey()
}
static func expectedDeepLinkKey() -> String {
@@ -10471,19 +10210,6 @@ extension NodeAppModel {
self.admitTalkAfterSessionHydration()
}
func _test_applyPendingForegroundNodeActions(
_ actions: [(id: String, command: String, paramsJSON: String?)]) async
{
let mapped = actions.map { action in
PendingForegroundNodeAction(
id: action.id,
command: action.command,
paramsJSON: action.paramsJSON,
enqueuedAtMs: nil)
}
await self.applyPendingForegroundNodeActions(mapped, trigger: "test")
}
func _test_makeOperatorConnectOptions(
clientId: String,
displayName: String?,
apps/ios/Sources/RootTabs.swift
-194
View File
@@ -1,5 +1,4 @@
import OpenClawKit
import OpenClawProtocol
import SwiftUI
import UIKit
@@ -28,7 +27,6 @@ struct RootTabs: View {
@AppStorage("gateway.manual.enabled") private var manualGatewayEnabled: Bool = false
@AppStorage("gateway.manual.host") private var manualGatewayHost: String = ""
@AppStorage("onboarding.quickSetupDismissed") private var quickSetupDismissed: Bool = false
@AppStorage("canvas.debugStatusEnabled") private var canvasDebugStatusEnabled: Bool = false
@State private var selectedSidebarDestination: SidebarDestination = Self.initialSidebarDestination
@State private var selectedSettingsRoute: SettingsRoute? =
Self.initialSettingsRoute ?? Self.initialSidebarDestination.settingsRoute
@@ -563,13 +561,6 @@ struct RootTabs: View {
// flashes without treating a later remount as a new capture.
RootCameraFlashOverlay(nonce: self.appModel.cameraFlashNonce)
}
.overlay {
if self.appModel.screen.isCanvasPresented {
self.canvasPresentationOverlay
.transition(.opacity)
.zIndex(20)
}
}
}
private var activeGatewayProblemToast: GatewayConnectionProblem? {
@@ -626,29 +617,6 @@ struct RootTabs: View {
self.isGatewayToastSwipeDismissed = false
}
private var canvasPresentationOverlay: some View {
ZStack(alignment: .topTrailing) {
Color.black.ignoresSafeArea()
ScreenWebView(controller: self.appModel.screen)
.ignoresSafeArea()
Button {
self.appModel.screen.hideCanvas()
} label: {
Image(systemName: "xmark.circle.fill")
.font(.system(size: 30, weight: .semibold))
.symbolRenderingMode(.hierarchical)
.foregroundStyle(.white)
.shadow(color: .black.opacity(0.32), radius: 8, y: 2)
.frame(width: 48, height: 48)
.contentShape(Rectangle())
}
.buttonStyle(.plain)
.accessibilityLabel("Close canvas")
.safeAreaPadding(.top, 8)
.padding(.trailing, 12)
}
}
private func rootLifecycle(_ content: some View) -> some View {
self.rootRequestLifecycle(
self.rootGatewayLifecycle(
@@ -678,7 +646,6 @@ struct RootTabs: View {
private func rootAppearLifecycle(_ content: some View) -> some View {
content
.onAppear { self.updateIdleTimer() }
.onAppear { self.updateCanvasState() }
.onAppear { self.evaluateOnboardingPresentation(force: false) }
.onAppear { self.maybeAutoOpenSettings() }
.onAppear { self.maybeOpenSettingsForGatewaySetup() }
@@ -688,7 +655,6 @@ struct RootTabs: View {
.onChange(of: self.appModel.talkMode.isEnabled) { _, _ in self.updateIdleTimer() }
.onChange(of: self.scenePhase) { _, newValue in
self.updateIdleTimer()
self.updateHomeCanvasState()
guard newValue == .active else {
self.clearVoiceWakeToast()
return
@@ -696,9 +662,6 @@ struct RootTabs: View {
self.maybeRequestLocalNetworkAccess(reason: "scene_active")
Task {
await self.appModel.refreshGatewayOverviewIfConnected()
await MainActor.run {
self.updateHomeCanvasState()
}
}
}
.onDisappear {
@@ -726,7 +689,6 @@ struct RootTabs: View {
private func rootGatewayLifecycle(_ content: some View) -> some View {
self.rootGatewayProblemLifecycle(content)
.onChange(of: self.canvasDebugStatusEnabled) { _, _ in self.updateCanvasDebugStatus() }
.onChange(of: self.gatewayController.gateways.count) { _, _ in self.maybeShowQuickSetup() }
.onChange(of: self.appModel.gatewayServerName) { _, newValue in
if newValue != nil {
@@ -736,18 +698,6 @@ struct RootTabs: View {
}
self.maybeAutoOpenSettings()
self.maybeShowQuickSetup()
self.updateCanvasState()
}
.onChange(of: self.appModel.gatewayStatusText) { _, _ in self.updateCanvasState() }
.onChange(of: self.appModel.gatewayRemoteAddress) { _, _ in self.updateCanvasState() }
.onChange(of: self.appModel.gatewayDisplayStatusText) { _, _ in self.updateCanvasState() }
.onChange(of: self.appModel.homeCanvasRevision) { _, _ in self.updateHomeCanvasState() }
.onChange(of: self.appModel.gatewayAgents.count) { _, _ in self.updateHomeCanvasState() }
.onChange(of: self.appModel.selectedAgentId) { _, _ in self.updateHomeCanvasState() }
.onChange(of: self.appModel.gatewayDefaultAgentId) { _, _ in self.updateHomeCanvasState() }
.onChange(of: self.appModel.activeAgentName) { _, _ in self.updateHomeCanvasState() }
.onChange(of: self.appModel.connectedGatewayID) { _, _ in
self.updateCanvasState()
}
}
@@ -835,133 +785,12 @@ struct RootTabs: View {
})
}
private var gatewayStatus: GatewayDisplayState {
GatewayStatusBuilder.build(appModel: self.appModel)
}
private func updateIdleTimer() {
UIApplication.shared.isIdleTimerDisabled =
self.scenePhase == .active && (self.preventSleep || self.appModel.talkMode.isEnabled)
}
}
extension RootTabs {
private func updateCanvasState() {
self.updateHomeCanvasState()
self.updateCanvasDebugStatus()
}
private func updateCanvasDebugStatus() {
self.appModel.screen.setDebugStatusEnabled(self.canvasDebugStatusEnabled)
guard self.canvasDebugStatusEnabled else { return }
let title = self.appModel.gatewayDisplayStatusText.trimmingCharacters(in: .whitespacesAndNewlines)
let subtitle = self.appModel.gatewayServerName ?? self.appModel.gatewayRemoteAddress
self.appModel.screen.updateDebugStatus(title: title, subtitle: subtitle)
}
private func updateHomeCanvasState() {
let payload = self.makeHomeCanvasPayload()
guard let data = try? JSONEncoder().encode(payload),
let json = String(data: data, encoding: .utf8)
else {
self.appModel.screen.updateHomeCanvasState(json: nil)
return
}
self.appModel.screen.updateHomeCanvasState(json: json)
}
private func makeHomeCanvasPayload() -> RootTabsHomeCanvasPayload {
let gatewayName = normalized(appModel.gatewayServerName)
let gatewayAddress = normalized(appModel.gatewayRemoteAddress)
let gatewayLabel = gatewayName ?? gatewayAddress ?? "Gateway"
let activeAgentID = self.resolveActiveAgentID()
let agents = self.homeCanvasAgents(activeAgentID: activeAgentID)
switch self.gatewayStatus {
case .connected:
return RootTabsHomeCanvasPayload(
gatewayState: "connected",
eyebrow: "\(gatewayLabel) online",
title: "Command center",
subtitle:
"Use Chat for code work or realtime voice, plus gateway tools for approved device actions.",
gatewayLabel: gatewayLabel,
activeAgentName: self.appModel.activeAgentName,
activeAgentBadge: agents.first(where: { $0.isActive })?.badge ?? "OC",
activeAgentCaption: "Routes chat and voice",
agentCount: agents.count,
agents: Array(agents.prefix(6)),
footer: "OpenClaw only runs phone-side capabilities while the app is connected and permitted.")
case .connecting:
return RootTabsHomeCanvasPayload(
gatewayState: "connecting",
eyebrow: "Gateway handshake",
title: "Reconnecting",
subtitle:
"Restoring the local node session, agent list, voice config, and device capability state.",
gatewayLabel: gatewayLabel,
activeAgentName: self.appModel.activeAgentName,
activeAgentBadge: "OC",
activeAgentCaption: "Session in progress",
agentCount: agents.count,
agents: Array(agents.prefix(4)),
footer: "If the gateway is reachable, the local node should recover without re-pairing.")
case .error, .disconnected:
return RootTabsHomeCanvasPayload(
gatewayState: self.gatewayStatus == .error ? "error" : "offline",
eyebrow: self.gatewayStatus == .error ? "Gateway needs attention" : "OpenClaw iOS",
title: "Pair a gateway",
subtitle:
"Connect this phone as a local node for chat, realtime voice, share intake, and approved device tools.",
gatewayLabel: gatewayLabel,
activeAgentName: "Main",
activeAgentBadge: "OC",
activeAgentCaption: "Connect to load your agents",
agentCount: agents.count,
agents: Array(agents.prefix(4)),
footer:
"Use Settings to scan a pairing QR code or paste a setup code from your OpenClaw gateway.")
}
}
private func resolveActiveAgentID() -> String {
let selected = normalized(appModel.selectedAgentId) ?? ""
if !selected.isEmpty {
return selected
}
return self.resolveDefaultAgentID()
}
private func resolveDefaultAgentID() -> String {
normalized(self.appModel.gatewayDefaultAgentId) ?? ""
}
private func homeCanvasAgents(activeAgentID: String) -> [RootTabsHomeCanvasAgentCard] {
let defaultAgentID = self.resolveDefaultAgentID()
let cards = self.appModel.gatewayAgents.map { agent -> RootTabsHomeCanvasAgentCard in
let isActive = !activeAgentID.isEmpty && agent.id == activeAgentID
let isDefault = !defaultAgentID.isEmpty && agent.id == defaultAgentID
return RootTabsHomeCanvasAgentCard(
id: agent.id,
name: self.homeCanvasName(for: agent),
badge: self.homeCanvasBadge(for: agent),
caption: isActive ? "Routed on this phone" : (isDefault ? "Gateway default" : "Available"),
isActive: isActive)
}
return cards.sorted { lhs, rhs in
if lhs.isActive != rhs.isActive {
return lhs.isActive
}
return lhs.name.localizedCaseInsensitiveCompare(rhs.name) == .orderedAscending
}
}
private func homeCanvasName(for agent: AgentSummary) -> String {
normalized(agent.name) ?? agent.id
}
}
extension RootTabs {
private func selectSidebarDestination(_ destination: SidebarDestination) {
self.sidebarNavigationPath.removeAll()
@@ -1069,29 +898,6 @@ extension RootTabs {
self.isSidebarVisible = isVisible
}
private func homeCanvasBadge(for agent: AgentSummary) -> String {
if let identity = agent.identity,
let emoji = identity["emoji"]?.value as? String,
let normalizedEmoji = normalized(emoji)
{
return normalizedEmoji
}
let words = self.homeCanvasName(for: agent)
.split(whereSeparator: { $0.isWhitespace || $0 == "-" || $0 == "_" })
.prefix(2)
let initials = words.compactMap(\.first).map(String.init).joined()
if !initials.isEmpty {
return initials.uppercased()
}
return "OC"
}
private func normalized(_ value: String?) -> String? {
guard let value else { return nil }
let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines)
return trimmed.isEmpty ? nil : trimmed
}
private func gatewayProblemPrimaryActionTitle(_ problem: GatewayConnectionProblem) -> String? {
GatewayProblemPrimaryAction.title(
for: problem,
apps/ios/Sources/RootTabsCanvasPayload.swift
-21
View File
@@ -1,21 +0,0 @@
struct RootTabsHomeCanvasPayload: Codable {
var gatewayState: String
var eyebrow: String
var title: String
var subtitle: String
var gatewayLabel: String
var activeAgentName: String
var activeAgentBadge: String
var activeAgentCaption: String
var agentCount: Int
var agents: [RootTabsHomeCanvasAgentCard]
var footer: String
}
struct RootTabsHomeCanvasAgentCard: Codable {
var id: String
var name: String
var badge: String
var caption: String
var isActive: Bool
}
apps/ios/Sources/Screen/ScreenController.swift
-320
View File
@@ -1,320 +0,0 @@
import Observation
import OpenClawKit
import UIKit
import WebKit
@MainActor
@Observable
final class ScreenController {
private weak var activeWebView: WKWebView?
var urlString: String = ""
var errorText: String?
var isCanvasPresented: Bool = false
/// Callback invoked when an openclaw:// deep link is tapped in the canvas
var onDeepLink: ((URL) -> Void)?
/// Callback invoked when the user clicks an A2UI action (e.g. button) inside the canvas web UI.
var onA2UIAction: (([String: Any]) -> Void)?
private var debugStatusEnabled: Bool = false
private var debugStatusTitle: String?
private var debugStatusSubtitle: String?
private var homeCanvasStateJSON: String?
init() {
self.reload()
}
func navigate(to urlString: String, trustA2UIActions _: Bool = false) {
let trimmed = urlString.trimmingCharacters(in: .whitespacesAndNewlines)
if trimmed.isEmpty {
self.urlString = ""
self.reload()
return
}
if let url = URL(string: trimmed),
!url.isFileURL,
let host = url.host,
LoopbackHost.isLoopback(host)
{
// Never try to load loopback URLs from a remote gateway.
self.showDefaultCanvas()
return
}
self.urlString = (trimmed == "/" ? "" : trimmed)
self.reload()
}
func reload() {
self.applyScrollBehavior()
guard let webView = self.activeWebView else { return }
let trimmed = self.urlString.trimmingCharacters(in: .whitespacesAndNewlines)
if trimmed.isEmpty {
guard let url = Self.canvasScaffoldURL else { return }
self.errorText = nil
webView.loadFileURL(url, allowingReadAccessTo: url.deletingLastPathComponent())
return
}
guard let url = URL(string: trimmed) else {
self.errorText = "Invalid URL: \(trimmed)"
return
}
self.errorText = nil
if url.isFileURL {
webView.loadFileURL(url, allowingReadAccessTo: url.deletingLastPathComponent())
} else {
webView.load(URLRequest(url: url))
}
}
func showDefaultCanvas() {
self.urlString = ""
self.reload()
}
func presentDefaultCanvas() {
self.isCanvasPresented = true
self.showDefaultCanvas()
}
func present(urlString: String) {
self.isCanvasPresented = true
self.navigate(to: urlString)
}
func hideCanvas() {
self.isCanvasPresented = false
self.showDefaultCanvas()
}
func showLocalA2UI() {
self.isCanvasPresented = true
guard let url = Self.localA2UIURL else {
self.showDefaultCanvas()
return
}
self.urlString = url.absoluteString
self.reload()
}
func isShowingLocalA2UI() -> Bool {
guard let url = URL(string: self.urlString),
url.isFileURL,
let expected = Self.localA2UIURL
else { return false }
return url.standardizedFileURL == expected.standardizedFileURL
}
func setDebugStatusEnabled(_ enabled: Bool) {
self.debugStatusEnabled = enabled
self.applyDebugStatusIfNeeded()
}
func updateDebugStatus(title: String?, subtitle: String?) {
self.debugStatusTitle = title
self.debugStatusSubtitle = subtitle
self.applyDebugStatusIfNeeded()
}
func applyDebugStatusIfNeeded() {
guard let webView = self.activeWebView else { return }
WebViewJavaScriptSupport.applyDebugStatus(
webView: webView,
enabled: self.debugStatusEnabled,
title: self.debugStatusTitle,
subtitle: self.debugStatusSubtitle)
}
func updateHomeCanvasState(json: String?) {
self.homeCanvasStateJSON = json
self.applyHomeCanvasStateIfNeeded()
}
func applyHomeCanvasStateIfNeeded() {
guard let webView = self.activeWebView else { return }
let payload = self.homeCanvasStateJSON ?? "null"
let js = """
(() => {
try {
const api = globalThis.__openclaw;
if (!api || typeof api.renderHome !== 'function') return;
api.renderHome(\(payload));
} catch (_) {}
})()
"""
webView.evaluateJavaScript(js) { _, _ in }
}
func waitForA2UIReady(timeoutMs: Int) async -> Bool {
let clock = ContinuousClock()
let deadline = clock.now.advanced(by: .milliseconds(timeoutMs))
while clock.now < deadline {
do {
let res = try await self.eval(javaScript: """
(() => {
try {
const host = globalThis.openclawA2UI;
return !!host && typeof host.applyMessages === 'function';
} catch (_) { return false; }
})()
""")
let trimmed = res.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
if trimmed == "true" || trimmed == "1" { return true }
} catch {
// ignore; page likely still loading
}
try? await Task.sleep(nanoseconds: 120_000_000)
}
return false
}
func eval(javaScript: String) async throws -> String {
guard let webView = self.activeWebView else {
throw NSError(domain: "Screen", code: 3, userInfo: [
NSLocalizedDescriptionKey: "web view unavailable",
])
}
return try await WebViewJavaScriptSupport.evaluateToString(webView: webView, javaScript: javaScript)
}
func snapshotBase64(
maxWidth: CGFloat? = nil,
format: OpenClawCanvasSnapshotFormat,
quality: Double? = nil) async throws -> String
{
let image = try await self.snapshotImage(maxWidth: maxWidth)
let data: Data?
switch format {
case .png:
data = image.pngData()
case .jpeg:
let q = (quality ?? 0.82).clamped(to: 0.1...1.0)
data = image.jpegData(compressionQuality: q)
}
guard let data else {
throw NSError(domain: "Screen", code: 1, userInfo: [
NSLocalizedDescriptionKey: "snapshot encode failed",
])
}
return data.base64EncodedString()
}
private func snapshotImage(maxWidth: CGFloat?) async throws -> UIImage {
let config = WKSnapshotConfiguration()
if let maxWidth {
config.snapshotWidth = NSNumber(value: Double(maxWidth))
}
guard let webView = self.activeWebView else {
throw NSError(domain: "Screen", code: 3, userInfo: [
NSLocalizedDescriptionKey: "web view unavailable",
])
}
return try await withCheckedThrowingContinuation { cont in
webView.takeSnapshot(with: config) { image, error in
if let error {
cont.resume(throwing: error)
return
}
guard let image else {
cont.resume(throwing: NSError(domain: "Screen", code: 2, userInfo: [
NSLocalizedDescriptionKey: "snapshot failed",
]))
return
}
cont.resume(returning: image)
}
}
}
func attachWebView(_ webView: WKWebView) {
self.activeWebView = webView
self.reload()
self.applyDebugStatusIfNeeded()
self.applyHomeCanvasStateIfNeeded()
}
func detachWebView(_ webView: WKWebView) {
guard self.activeWebView === webView else { return }
self.activeWebView = nil
}
private static func bundledResourceURL(
name: String,
ext: String,
subdirectory: String)
-> URL?
{
let bundle = OpenClawKitResources.bundle
return bundle.url(forResource: name, withExtension: ext, subdirectory: subdirectory)
?? bundle.url(forResource: name, withExtension: ext)
}
private static let canvasScaffoldURL: URL? = ScreenController.bundledResourceURL(
name: "scaffold",
ext: "html",
subdirectory: "CanvasScaffold")
private static let localA2UIURL: URL? = ScreenController.bundledResourceURL(
name: "index",
ext: "html",
subdirectory: "CanvasA2UI")
func isTrustedCanvasUIURL(_ url: URL) -> Bool {
if url.isFileURL {
let std = url.standardizedFileURL
if let expected = Self.canvasScaffoldURL,
std == expected.standardizedFileURL
{
return true
}
if let expected = Self.localA2UIURL,
std == expected.standardizedFileURL
{
return true
}
return false
}
return false
}
nonisolated static func parseA2UIActionBody(_ body: Any) -> [String: Any]? {
if let dict = body as? [String: Any] { return dict.isEmpty ? nil : dict }
if let str = body as? String,
let data = str.data(using: .utf8),
let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any]
{
return json.isEmpty ? nil : json
}
if let dict = body as? [AnyHashable: Any] {
let mapped = dict.reduce(into: [String: Any]()) { acc, pair in
guard let key = pair.key as? String else { return }
acc[key] = pair.value
}
return mapped.isEmpty ? nil : mapped
}
return nil
}
private func applyScrollBehavior() {
guard let webView = self.activeWebView else { return }
let trimmed = self.urlString.trimmingCharacters(in: .whitespacesAndNewlines)
let allowScroll = !trimmed.isEmpty
let scrollView = webView.scrollView
// Default canvas needs raw touch events; external pages should scroll.
scrollView.isScrollEnabled = allowScroll
scrollView.bounces = allowScroll
}
}
extension Double {
fileprivate func clamped(to range: ClosedRange<Double>) -> Double {
if self < range.lowerBound { return range.lowerBound }
if self > range.upperBound { return range.upperBound }
return self
}
}
apps/ios/Sources/Screen/ScreenWebView.swift
-189
View File
@@ -1,189 +0,0 @@
import OpenClawKit
import SwiftUI
import WebKit
struct ScreenWebView: UIViewRepresentable {
var controller: ScreenController
func makeCoordinator() -> ScreenWebViewCoordinator {
ScreenWebViewCoordinator(controller: self.controller)
}
func makeUIView(context: Context) -> UIView {
context.coordinator.makeContainerView()
}
func updateUIView(_: UIView, context: Context) {
context.coordinator.updateController(self.controller)
}
static func dismantleUIView(_: UIView, coordinator: ScreenWebViewCoordinator) {
coordinator.teardown()
}
}
@MainActor
final class ScreenWebViewCoordinator: NSObject {
private weak var controller: ScreenController?
private let navigationDelegate = ScreenNavigationDelegate()
private let a2uiActionHandler = CanvasA2UIActionMessageHandler()
private let userContentController = WKUserContentController()
private(set) var managedWebView: WKWebView?
private weak var containerView: UIView?
init(controller: ScreenController) {
self.controller = controller
super.init()
self.navigationDelegate.controller = controller
self.a2uiActionHandler.controller = controller
}
func makeContainerView() -> UIView {
if let containerView {
return containerView
}
let container = UIView(frame: .zero)
container.backgroundColor = .black
let webView = Self.makeWebView(userContentController: self.userContentController)
webView.navigationDelegate = self.navigationDelegate
self.installA2UIHandlers()
webView.translatesAutoresizingMaskIntoConstraints = false
container.addSubview(webView)
NSLayoutConstraint.activate([
webView.leadingAnchor.constraint(equalTo: container.leadingAnchor),
webView.trailingAnchor.constraint(equalTo: container.trailingAnchor),
webView.topAnchor.constraint(equalTo: container.topAnchor),
webView.bottomAnchor.constraint(equalTo: container.bottomAnchor),
])
self.managedWebView = webView
self.containerView = container
self.controller?.attachWebView(webView)
return container
}
func updateController(_ controller: ScreenController) {
let previousController = self.controller
let controllerChanged = self.controller !== controller
self.controller = controller
self.navigationDelegate.controller = controller
self.a2uiActionHandler.controller = controller
if controllerChanged, let managedWebView {
previousController?.detachWebView(managedWebView)
controller.attachWebView(managedWebView)
}
}
func teardown() {
if let managedWebView {
self.controller?.detachWebView(managedWebView)
managedWebView.navigationDelegate = nil
}
self.removeA2UIHandlers()
self.navigationDelegate.controller = nil
self.a2uiActionHandler.controller = nil
self.managedWebView = nil
self.containerView = nil
}
private static func makeWebView(userContentController: WKUserContentController) -> WKWebView {
let config = WKWebViewConfiguration()
config.websiteDataStore = .nonPersistent()
config.userContentController = userContentController
let webView = WKWebView(frame: .zero, configuration: config)
// Canvas scaffold is a fully self-contained HTML page; avoid relying on transparency underlays.
webView.isOpaque = true
webView.backgroundColor = .black
let scrollView = webView.scrollView
scrollView.backgroundColor = .black
scrollView.contentInsetAdjustmentBehavior = .never
scrollView.contentInset = .zero
scrollView.scrollIndicatorInsets = .zero
scrollView.automaticallyAdjustsScrollIndicatorInsets = false
return webView
}
private func installA2UIHandlers() {
for name in CanvasA2UIActionMessageHandler.handlerNames {
self.userContentController.add(self.a2uiActionHandler, name: name)
}
}
private func removeA2UIHandlers() {
for name in CanvasA2UIActionMessageHandler.handlerNames {
self.userContentController.removeScriptMessageHandler(forName: name)
}
}
}
// MARK: - Navigation Delegate
/// Handles navigation policy to intercept OpenClaw deep links from canvas.
@MainActor
private final class ScreenNavigationDelegate: NSObject, WKNavigationDelegate {
weak var controller: ScreenController?
func webView(
_: WKWebView,
decidePolicyFor navigationAction: WKNavigationAction,
decisionHandler: @escaping @MainActor @Sendable (WKNavigationActionPolicy) -> Void)
{
guard let url = navigationAction.request.url else {
decisionHandler(.allow)
return
}
let scheme = url.scheme?.lowercased()
if scheme == "openclaw" || scheme == "openclaw-debug" {
decisionHandler(.cancel)
self.controller?.onDeepLink?(url)
return
}
decisionHandler(.allow)
}
func webView(
_: WKWebView,
didFailProvisionalNavigation _: WKNavigation?,
withError error: any Error)
{
self.controller?.errorText = error.localizedDescription
}
func webView(_: WKWebView, didFinish _: WKNavigation?) {
self.controller?.errorText = nil
self.controller?.applyDebugStatusIfNeeded()
self.controller?.applyHomeCanvasStateIfNeeded()
}
func webView(_: WKWebView, didFail _: WKNavigation?, withError error: any Error) {
self.controller?.errorText = error.localizedDescription
}
}
private final class CanvasA2UIActionMessageHandler: NSObject, WKScriptMessageHandler {
static let messageName = "openclawCanvasA2UIAction"
static let handlerNames = [messageName]
weak var controller: ScreenController?
func userContentController(_: WKUserContentController, didReceive message: WKScriptMessage) {
guard Self.handlerNames.contains(message.name) else { return }
guard let controller else { return }
guard let url = message.webView?.url else { return }
guard controller.isTrustedCanvasUIURL(url) else { return }
guard let body = ScreenController.parseA2UIActionBody(message.body) else { return }
controller.onA2UIAction?(body)
}
}
apps/ios/Tests/GatewayConnectionControllerTests.swift
+6 -4
View File
@@ -304,13 +304,16 @@ private func waitUntil(
let controller = GatewayConnectionController(appModel: appModel, startDiscovery: false)
let caps = Set(controller._test_currentCaps())
#expect(caps.contains(OpenClawCapability.canvas.rawValue))
#expect(!caps.contains(OpenClawCapability.canvas.rawValue))
#expect(caps.contains(OpenClawCapability.screen.rawValue))
#expect(!caps.contains(OpenClawGatewayClientCapability.inlineWidgets))
#expect(caps.contains(OpenClawCapability.camera.rawValue))
#expect(caps.contains(OpenClawCapability.location.rawValue))
#expect(caps.contains(OpenClawCapability.voiceWake.rawValue))
#expect(caps.contains(OpenClawCapability.talk.rawValue))
let commands = controller._test_currentCommands()
#expect(!commands.contains(where: { $0.hasPrefix("canvas.") }))
}
}
@@ -635,7 +638,7 @@ private func waitUntil(
bootstrapToken: lhs.bootstrapToken,
password: lhs.password,
nodeOptions: Self.makeNodeOptions(
caps: ["canvas", "screen"],
caps: ["camera", "screen"],
commands: ["location.get", "notify"],
permissions: ["screen": true]))
@@ -1557,7 +1560,6 @@ private func waitUntil(
lanHost: nil,
tailnetDns: nil,
gatewayPort: nil,
canvasPort: nil,
tlsEnabled: true,
tlsFingerprintSha256: nil,
cliPath: nil)
@@ -2645,7 +2647,7 @@ private func waitUntil(
bootstrapToken: nil,
password: nil,
nodeOptions: self.makeNodeOptions(
caps: ["screen", "canvas"],
caps: ["screen", "camera"],
commands: ["notify", "location.get"],
permissions: ["screen": true]))
}
apps/ios/Tests/GatewayConnectionSecurityTests.swift
-1
View File
@@ -28,7 +28,6 @@ import Testing
lanHost: lanHost,
tailnetDns: tailnetDns,
gatewayPort: gatewayPort,
canvasPort: nil,
tlsEnabled: tlsEnabled,
tlsFingerprintSha256: fingerprint,
cliPath: nil)
apps/ios/Tests/NodeAppModelInvokeTests.swift
+13 -119
View File
@@ -649,14 +649,6 @@ private func waitForMainActorWork(
return condition()
}
@MainActor
private func mountScreen(_ screen: ScreenController) throws -> ScreenWebViewCoordinator {
let coordinator = ScreenWebViewCoordinator(controller: screen)
_ = coordinator.makeContainerView()
_ = try #require(coordinator.managedWebView)
return coordinator
}
@MainActor
private final class MockWatchMessagingService: @preconcurrency WatchMessagingServicing, @unchecked Sendable {
var currentStatus = WatchMessagingStatus(
@@ -984,8 +976,12 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
@Suite(.serialized) struct NodeAppModelInvokeTests {
@Test @MainActor func `decode params fails without JSON`() {
struct RequiredPayload: Decodable {
var value: String
}
#expect(throws: Error.self) {
_ = try NodeAppModel.decodeParams(OpenClawCanvasNavigateParams.self, from: nil)
_ = try NodeAppModel.decodeParams(RequiredPayload.self, from: nil)
}
}
@@ -6445,7 +6441,7 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
let appModel = NodeAppModel()
appModel.setScenePhase(.background)
let req = BridgeInvokeRequest(id: "bg", command: OpenClawCanvasCommand.present.rawValue)
let req = BridgeInvokeRequest(id: "bg", command: OpenClawScreenCommand.record.rawValue)
let res = await appModel.handleInvoke(req)
#expect(res.ok == false)
#expect(res.error?.code == .backgroundUnavailable)
@@ -6659,94 +6655,6 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
#expect(res.error?.message.contains("screen format must be mp4") == true)
}
@Test @MainActor func `handle invoke canvas commands update screen`() async throws {
let appModel = NodeAppModel()
let coordinator = try mountScreen(appModel.screen)
defer { coordinator.teardown() }
appModel.screen.navigate(to: "http://example.com")
let present = BridgeInvokeRequest(id: "present", command: OpenClawCanvasCommand.present.rawValue)
let presentRes = await appModel.handleInvoke(present)
#expect(presentRes.ok == true)
#expect(appModel.screen.urlString.isEmpty)
// Loopback URLs are rejected (they are not meaningful for a remote gateway).
let navigate = try makeInvokeRequest(
id: "nav",
command: OpenClawCanvasCommand.navigate.rawValue,
params: OpenClawCanvasNavigateParams(url: "http://example.com/"))
let navRes = await appModel.handleInvoke(navigate)
#expect(navRes.ok == true)
#expect(appModel.screen.urlString == "http://example.com/")
let eval = try makeInvokeRequest(
id: "eval",
command: OpenClawCanvasCommand.evalJS.rawValue,
params: OpenClawCanvasEvalParams(javaScript: "1+1"))
var evalRes = await appModel.handleInvoke(eval)
let deadline = ContinuousClock().now.advanced(by: .seconds(3))
while evalRes.ok != true, ContinuousClock().now < deadline {
try? await Task.sleep(nanoseconds: 100_000_000)
evalRes = await appModel.handleInvoke(eval)
}
#expect(evalRes.ok == true)
let payloadData = try #require(evalRes.payloadJSON?.data(using: .utf8))
let payload = try JSONSerialization.jsonObject(with: payloadData) as? [String: Any]
#expect(payload?["result"] as? String == "2")
}
@Test @MainActor func `pending foreground actions replay canvas navigate`() async throws {
let appModel = NodeAppModel()
let navJSON = try String(
decoding: JSONEncoder().encode(OpenClawCanvasNavigateParams(url: "http://example.com/")),
as: UTF8.self)
await appModel._test_applyPendingForegroundNodeActions([
(
id: "pending-nav-1",
command: OpenClawCanvasCommand.navigate.rawValue,
paramsJSON: navJSON),
])
#expect(appModel.screen.urlString == "http://example.com/")
}
@Test @MainActor func `pending foreground actions do not apply while backgrounded`() async throws {
let appModel = NodeAppModel()
appModel.setScenePhase(.background)
let navJSON = try String(
decoding: JSONEncoder().encode(OpenClawCanvasNavigateParams(url: "http://example.com/")),
as: UTF8.self)
await appModel._test_applyPendingForegroundNodeActions([
(
id: "pending-nav-bg",
command: OpenClawCanvasCommand.navigate.rawValue,
paramsJSON: navJSON),
])
#expect(appModel.screen.urlString.isEmpty)
}
@Test @MainActor func `handle invoke A 2 UI commands fail when local host unavailable`() async throws {
let appModel = NodeAppModel()
let reset = BridgeInvokeRequest(id: "reset", command: OpenClawCanvasA2UICommand.reset.rawValue)
let resetRes = await appModel.handleInvoke(reset)
#expect(resetRes.ok == false)
#expect(resetRes.error?.message.contains("A2UI_HOST_UNAVAILABLE") == true)
let jsonl = "{\"beginRendering\":{}}"
let push = try makeInvokeRequest(
id: "push",
command: OpenClawCanvasA2UICommand.pushJSONL.rawValue,
params: OpenClawCanvasA2UIPushJSONLParams(jsonl: jsonl))
let pushRes = await appModel.handleInvoke(push)
#expect(pushRes.ok == false)
#expect(pushRes.error?.message.contains("A2UI_HOST_UNAVAILABLE") == true)
}
@Test @MainActor func `handle invoke unknown command returns invalid request`() async {
let appModel = NodeAppModel()
let req = BridgeInvokeRequest(id: "unknown", command: "nope")
@@ -7494,19 +7402,19 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
#expect(appModel.watchMessageOutbox.queuedCount(kind: .quickReply) == 0)
}
@Test @MainActor func `handle deep link sets error when not connected`() async throws {
@Test @MainActor func `handle deep link records failure when not connected`() async throws {
let appModel = NodeAppModel()
let url = try #require(URL(string: "openclaw://agent?message=hello"))
await appModel.handleDeepLink(url: url)
#expect(appModel.screen.errorText?.contains("Gateway not connected") == true)
#expect(appModel.lastShareEventText.contains("gateway not connected"))
}
@Test @MainActor func `handle deep link rejects oversized message`() async throws {
@Test @MainActor func `handle deep link records oversized message rejection`() async throws {
let appModel = NodeAppModel()
let msg = String(repeating: "a", count: 20001)
let url = try #require(URL(string: "openclaw://agent?message=\(msg)"))
await appModel.handleDeepLink(url: url)
#expect(appModel.screen.errorText?.contains("Deep link too large") == true)
#expect(appModel.lastShareEventText.contains("message too large"))
}
@Test @MainActor func `handle deep link requires confirmation when connected and unkeyed`() async {
@@ -7522,7 +7430,7 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
await appModel.approvePendingAgentDeepLinkPrompt()
#expect(appModel.pendingAgentDeepLinkPrompt == nil)
#expect(appModel.openChatRequestID == 1)
#expect(appModel.screen.errorText == nil)
#expect(appModel.lastShareEventText.contains("Sent to gateway"))
}
@Test @MainActor func `handle deep link coalesces prompt when rate limited`() async throws {
@@ -7563,7 +7471,7 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
await appModel.handleDeepLink(url: url)
#expect(appModel.pendingAgentDeepLinkPrompt == nil)
#expect(appModel.screen.errorText?.contains("blocked") == true)
#expect(appModel.lastShareEventText.contains("Rejected"))
}
@Test @MainActor func `handle deep link bypasses prompt with valid key`() async {
@@ -7576,7 +7484,7 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
await appModel.handleDeepLink(url: url)
#expect(appModel.pendingAgentDeepLinkPrompt == nil)
#expect(appModel.openChatRequestID == 1)
#expect(appModel.screen.errorText == nil)
#expect(appModel.lastShareEventText.contains("Sent to gateway"))
}
@Test @MainActor func `operator scopes use the active gateway token`() throws {
@@ -7653,18 +7561,4 @@ private func overrideNotificationServingPreference(_ enabled: Bool) -> () -> Voi
}
}
@Test @MainActor func `canvas A 2 UI action dispatches status`() async {
let appModel = NodeAppModel()
let body: [String: Any] = [
"userAction": [
"name": "tap",
"id": "action-1",
"surfaceId": "main",
"sourceComponentId": "button-1",
"context": ["value": "ok"],
],
]
await appModel.handleCanvasA2UIAction(body: body)
#expect(appModel.screen.urlString.isEmpty)
}
}
apps/ios/Tests/RootTabsSourceGuardTests.swift
-1
View File
@@ -59,7 +59,6 @@ struct RootTabsSourceGuardTests {
#expect(source.contains("layoutMode: self.isSidebarDrawerLayout ? .drawer : .split"))
#expect(componentSource.contains("OpenClawSidebarHeaderLeadingSlot"))
#expect(componentSource.contains(".frame(width: 44, height: 44)"))
#expect(source.contains(".safeAreaPadding(.top, 8)"))
#expect(source.contains("Self.sidebarShowButtonAccessibilityIdentifier"))
#expect(source.contains("Self.sidebarHideButtonAccessibilityIdentifier"))
#expect(source.contains("accessibilityLabel: .localized(\"Hide Sidebar\")"))
apps/ios/Tests/ScreenControllerTests.swift
-124
View File
@@ -1,124 +0,0 @@
import Testing
import WebKit
@testable import OpenClaw
@MainActor
private func mountScreen(_ screen: ScreenController) throws -> (ScreenWebViewCoordinator, WKWebView) {
let coordinator = ScreenWebViewCoordinator(controller: screen)
_ = coordinator.makeContainerView()
let webView = try #require(coordinator.managedWebView)
return (coordinator, webView)
}
@Suite struct ScreenControllerTests {
@Test @MainActor func canvasModeConfiguresWebViewForTouch() throws {
let screen = ScreenController()
let (coordinator, webView) = try mountScreen(screen)
defer { coordinator.teardown() }
#expect(webView.isOpaque == true)
#expect(webView.backgroundColor == .black)
let scrollView = webView.scrollView
#expect(scrollView.backgroundColor == .black)
#expect(scrollView.contentInsetAdjustmentBehavior == .never)
#expect(scrollView.isScrollEnabled == false)
#expect(scrollView.bounces == false)
}
@Test @MainActor func navigateEnablesScrollForWebPages() throws {
let screen = ScreenController()
let (coordinator, webView) = try mountScreen(screen)
defer { coordinator.teardown() }
screen.navigate(to: "https://example.com")
let scrollView = webView.scrollView
#expect(scrollView.isScrollEnabled == true)
#expect(scrollView.bounces == true)
}
@Test @MainActor func navigateSlashShowsDefaultCanvas() {
let screen = ScreenController()
screen.navigate(to: "/")
#expect(screen.urlString.isEmpty)
}
@Test @MainActor func canvasPresentationTracksExplicitPresentAndHide() {
let screen = ScreenController()
#expect(screen.isCanvasPresented == false)
screen.showDefaultCanvas()
#expect(screen.isCanvasPresented == false)
screen.presentDefaultCanvas()
#expect(screen.isCanvasPresented == true)
#expect(screen.urlString.isEmpty)
screen.hideCanvas()
#expect(screen.isCanvasPresented == false)
#expect(screen.urlString.isEmpty)
}
@Test @MainActor func evalExecutesJavaScript() async throws {
let screen = ScreenController()
let (coordinator, _) = try mountScreen(screen)
defer { coordinator.teardown() }
let deadline = ContinuousClock().now.advanced(by: .seconds(3))
while true {
do {
let result = try await screen.eval(javaScript: "1+1")
#expect(result == "2")
return
} catch {
if ContinuousClock().now >= deadline {
throw error
}
try? await Task.sleep(nanoseconds: 100_000_000)
}
}
}
@Test("remote A2UI URL is not trusted for native actions")
@MainActor func remoteA2UIURLIsNotTrustedForNativeActions() throws {
let screen = ScreenController()
let trusted = "https://node.ts.net:18789/__openclaw__/a2ui/?platform=ios"
screen.navigate(to: trusted, trustA2UIActions: true)
#expect(screen.isShowingLocalA2UI() == false)
let urls = try [
trusted,
"https://node.ts.net:18789/__openclaw__/a2ui/?platform=ios#step2",
"http://192.168.0.10:18789/__openclaw__/a2ui/?platform=ios",
"https://node.ts.net:18789/__openclaw__/a2ui/?platform=android",
"https://node.ts.net:18789/__openclaw__/canvas/",
"https://evil.ts.net:18789/__openclaw__/a2ui/?platform=ios",
].map { try #require(URL(string: $0)) }
for url in urls {
#expect(screen.isTrustedCanvasUIURL(url) == false)
}
}
@Test("local A2UI URL is trusted for native actions")
@MainActor func localA2UIURLIsTrustedForNativeActions() throws {
let screen = ScreenController()
screen.showLocalA2UI()
let url = try #require(URL(string: screen.urlString))
#expect(url.isFileURL)
#expect(screen.isShowingLocalA2UI() == true)
#expect(screen.isTrustedCanvasUIURL(url) == true)
}
@Test func parseA2UIActionBodyAcceptsJSONString() throws {
let body = ScreenController.parseA2UIActionBody("{\"userAction\":{\"name\":\"hello\"}}")
let userAction = try #require(body?["userAction"] as? [String: Any])
#expect(userAction["name"] as? String == "hello")
}
}
apps/ios/project.yml
-16
View File
@@ -106,22 +106,6 @@ targets:
export PATH="$PATH:/opt/homebrew/bin:/usr/local/bin"
"$SRCROOT/../../scripts/check-swift-tools.sh" swiftlint
swiftlint lint --strict --config "$SRCROOT/.swiftlint.yml" --use-script-input-file-lists
postBuildScripts:
- name: Stage Canvas A2UI resources
basedOnDependencyAnalysis: false
script: |
set -euo pipefail
export PATH="$PATH:/opt/homebrew/bin:/usr/local/bin"
repo_root="$(cd "$SRCROOT/../.." && pwd -P)"
resource_product="$BUILT_PRODUCTS_DIR/OpenClawKit_OpenClawKit.bundle"
resource_bundle="$TARGET_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH/OpenClawKit_OpenClawKit.bundle"
test -d "$resource_product"
test -d "$resource_bundle"
node --import tsx "$repo_root/scripts/sync-native-a2ui.mts" \
--write \
--output "$resource_product/CanvasA2UI"
rm -rf "$resource_bundle/CanvasA2UI"
cp -R "$resource_product/CanvasA2UI" "$resource_bundle/CanvasA2UI"
settings:
base:
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
apps/linux/README.md
+3 -12
View File
@@ -33,8 +33,8 @@ workflow instead of relying on GStreamer packages from the user's system.
## Develop and build
The companion frontend is static HTML, CSS, and JavaScript. The shared Canvas A2UI renderer is
generated from the Canvas plugin, so install repository dependencies once before building:
The companion frontend is static HTML, CSS, and JavaScript. Install repository dependencies once
before building:
```bash
pnpm install
@@ -53,18 +53,9 @@ On first run, release builds automatically install the stable CLI channel, while
The companion checks the latest GitHub release shortly after launch and from **Check for Updates** in the tray menu. AppImage installs download and verify the signed update in place, then wait for **Restart to update**. Package-managed installs such as `.deb` stay owned by the system package manager and link to the release download page instead of replacing installed files. The macOS and Windows test builds use a separate opt-in desktop-test update channel; macOS self-updates like the AppImage build, while Windows downloads the update first and runs its installer only after **Restart to update**.
## Canvas bridge
The running app gives the headless `openclaw node run` host a single Canvas WebView. The bundled `linux-canvas` plugin advertises `canvas.*` only while the app socket exists. The app listens at `$XDG_RUNTIME_DIR/openclaw-canvas.sock` (or `/tmp/openclaw-canvas-$UID.sock`) with mode `0600`; a headless Linux node without the app does not advertise Canvas.
The Canvas plugin sources remain the source of truth for the A2UI renderer. Each native build
generates `index.html` and `a2ui.bundle.js` into its isolated build output before compiling. Run
`node --import tsx scripts/sync-native-a2ui.mts --check` from the repository root to verify fresh bundles are
byte-identical and every native build owner is wired.
## Quick Chat widgets
Quick Chat advertises the Gateway `inline-widgets` capability and renders hosted `show_widget` results in isolated child WebViews. The parent Quick Chat WebView is the only one granted Tauri commands; widget WebViews match no capability and therefore have no IPC access. Quick Chat accepts only assistant-message Canvas previews under the capability-scoped `/__openclaw__/canvas/documents/` route, blocks navigation away from the original document, uses nonpersistent WebViews, and keeps stable widget instances while switching among multiple previews. Connections that require a custom Gateway TLS leaf pin remain text-only because the platform WebView cannot bind that pin. Like the other native clients, Quick Chat does not expose the Control UI `sendPrompt` bridge.
Quick Chat advertises the Gateway `inline-widgets` capability and renders hosted `show_widget` results in isolated child WebViews. The parent Quick Chat WebView is the only one granted Tauri commands; widget WebViews match no capability and therefore have no IPC access. Quick Chat accepts only assistant-message widget previews under the capability-scoped `/__openclaw__/canvas/documents/` route, blocks navigation away from the original document, uses nonpersistent WebViews, and keeps stable widget instances while switching among multiple previews. Connections that require a custom Gateway TLS leaf pin remain text-only because the platform WebView cannot bind that pin. Like the other native clients, Quick Chat does not expose the Control UI `sendPrompt` bridge.
## Installer resource
apps/linux/src-tauri/Cargo.lock
Generated
-20
View File
@@ -2018,8 +2018,6 @@ dependencies = [
"moxcms",
"num-traits",
"png 0.18.1",
"zune-core",
"zune-jpeg",
]
[[package]]
@@ -2790,11 +2788,9 @@ name = "openclaw-desktop-linux"
version = "0.1.0"
dependencies = [
"base64 0.23.0",
"cairo-rs",
"ed25519-dalek",
"futures-util",
"getrandom 0.4.3",
"image",
"libc",
"mdns-sd",
"rustls",
@@ -2816,7 +2812,6 @@ dependencies = [
"tokio",
"tokio-tungstenite",
"uuid",
"webkit2gtk",
"zbus",
"zeroize",
]
@@ -6133,21 +6128,6 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
[[package]]
name = "zune-core"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb8a0807f7c01457d0379ba880ba6322660448ddebc890ce29bb64da71fb40f9"
[[package]]
name = "zune-jpeg"
version = "0.5.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296"
dependencies = [
"zune-core",
]
[[package]]
name = "zvariant"
version = "5.13.1"
apps/linux/src-tauri/Cargo.toml
-3
View File
@@ -17,7 +17,6 @@ base64 = "0.23.0"
ed25519-dalek = "3.0.0"
futures-util = "0.3.33"
getrandom = "0.4.3"
image = { version = "0.25.10", default-features = false, features = ["jpeg", "png"] }
mdns-sd = { version = "0.20", default-features = false }
rustls = { version = "0.23.42", default-features = false, features = ["ring", "std", "tls12"] }
serde = { version = "1.0.229", features = ["derive"] }
@@ -48,10 +47,8 @@ uuid = { version = "1.24.0", features = ["v4"] }
zeroize = "1.9.0"
[target.'cfg(target_os = "linux")'.dependencies]
cairo-rs = { version = "0.18.5", features = ["png"] }
libc = "0.2.189"
tauri-plugin-notifications = { git = "https://github.com/steipete/tauri-plugin-notifications.git", rev = "d20b4ff0e0e327e49fee80903478f825eac5a71a" }
webkit2gtk = "2.0.2"
zbus = { version = "5", default-features = false, features = ["tokio"] }
[target.'cfg(target_os = "macos")'.dependencies]
apps/linux/src-tauri/build.rs
-50
View File
@@ -1,15 +1,8 @@
use std::path::PathBuf;
use std::process::Command;
fn main() {
stage_canvas_a2ui();
link_macos_swift_runtime();
// Command metadata generates capability permissions independently of the
// target's invoke handler, so keep the Linux-only command permission known.
const COMMANDS: &[&str] = &[
"bootstrap",
"build_info",
"canvas_a2ui_action",
"check_for_updates",
"connect_discovered_gateway",
"discover_gateways",
@@ -26,49 +19,6 @@ fn main() {
.expect("Tauri build configuration should be valid");
}
fn stage_canvas_a2ui() {
let repo_root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../..");
let output_dir = PathBuf::from(std::env::var_os("OUT_DIR").expect("Cargo must set OUT_DIR"))
.join("canvas-a2ui");
for input in [
"package.json",
"pnpm-lock.yaml",
"scripts/bundle-a2ui.mts",
"scripts/sync-native-a2ui.mts",
"extensions/canvas/package.json",
"extensions/canvas/scripts/bundle-a2ui.mjs",
"extensions/canvas/src/host/a2ui/index.html",
"extensions/canvas/src/host/a2ui-app",
] {
println!("cargo:rerun-if-changed={}", repo_root.join(input).display());
}
let status = Command::new("node")
.args([
"--import",
"tsx",
"scripts/sync-native-a2ui.mts",
"--write",
"--output",
])
.arg(&output_dir)
.current_dir(&repo_root)
.status()
.expect("Canvas A2UI staging requires Node.js; run pnpm install from the repository root");
assert!(
status.success(),
"Canvas A2UI resource staging failed; run pnpm install from the repository root"
);
println!(
"cargo:rustc-env=OPENCLAW_CANVAS_A2UI_INDEX_HTML={}",
output_dir.join("index.html").display()
);
println!(
"cargo:rustc-env=OPENCLAW_CANVAS_A2UI_BUNDLE_JS={}",
output_dir.join("a2ui.bundle.js").display()
);
}
/// tauri-plugin-notifications links a Swift static library into us, but nothing
/// adds an rpath for the Swift runtime it pulls in. Bundled apps get one from
/// the bundler; plain `cargo run` and `cargo test` binaries do not, so they die
apps/linux/src-tauri/permissions/autogenerated/canvas_a2ui_action.toml
-11
View File
@@ -1,11 +0,0 @@
# Automatically generated - DO NOT EDIT!
[[permission]]
identifier = "allow-canvas-a2ui-action"
description = "Enables the canvas_a2ui_action command without any pre-configured scope."
commands.allow = ["canvas_a2ui_action"]
[[permission]]
identifier = "deny-canvas-a2ui-action"
description = "Denies the canvas_a2ui_action command without any pre-configured scope."
commands.deny = ["canvas_a2ui_action"]
apps/linux/src-tauri/src/canvas.rs
-1022
View File
File diff suppressed because it is too large Load Diff
apps/linux/src-tauri/src/main.rs
+1 -43
View File
@@ -1,5 +1,3 @@
#[cfg(target_os = "linux")]
mod canvas;
mod cli;
mod discovery;
mod gateway;
@@ -736,11 +734,9 @@ fn main() {
.plugin(tauri_plugin_process::init())
.plugin(
tauri_plugin_window_state::Builder::default()
.with_denylist(&["canvas", quickchat::QUICKCHAT_LABEL])
.with_denylist(&[quickchat::QUICKCHAT_LABEL])
.build(),
);
#[cfg(target_os = "linux")]
let builder = canvas::register_protocol(builder);
let builder = builder.setup(move |app| {
let window = app
@@ -788,44 +784,9 @@ fn main() {
app.manage(discovery::GatewayDiscovery::default());
app.manage(quickchat_state.clone());
app.manage(updater::UpdaterState::default());
#[cfg(target_os = "linux")]
match canvas::CanvasBridge::start(app.handle().clone()) {
Ok(bridge) => {
app.manage(bridge);
}
Err(error) => eprintln!("Canvas bridge unavailable: {error}"),
}
state.set_tray(tray::build(app, state.clone(), global_shortcuts_supported)?);
Ok(())
});
#[cfg(target_os = "linux")]
let builder = builder.invoke_handler(tauri::generate_handler![
bootstrap,
build_info,
canvas::canvas_a2ui_action,
updater::check_for_updates,
discovery::connect_discovered_gateway,
discovery::discover_gateways,
install_cli,
gateway_action,
quickchat::quickchat_activate,
quickchat::quickchat_agents,
quickchat::quickchat_hide,
quickchat::quickchat_identity,
quickchat::quickchat_ready,
quickchat::quickchat_select_agent,
quickchat::quickchat_send,
quickchat::quickchat_set_expanded,
quickchat::quickchat_set_shortcut,
quickchat::quickchat_shortcut,
quickchat::quickchat_show_dashboard,
quickchat_widgets::quickchat_refresh_widget_surface,
quickchat_widgets::quickchat_sync_widgets,
updater::open_release_page,
updater::relaunch,
updater::updater_ready
]);
#[cfg(not(target_os = "linux"))]
let builder = builder.invoke_handler(tauri::generate_handler![
bootstrap,
build_info,
@@ -887,9 +848,6 @@ fn main() {
if let Some(bridge) = app.try_state::<gateway_sleep_logind::SleepBridge>() {
bridge.shutdown();
}
if let Some(bridge) = app.try_state::<canvas::CanvasBridge>() {
bridge.shutdown();
}
}
#[cfg(not(target_os = "linux"))]
let _ = (app, event);
apps/linux/src-tauri/tauri.conf.json
-7
View File
@@ -60,13 +60,6 @@
"updater:default"
]
},
{
"identifier": "canvas-renderer",
"description": "The bundled Canvas renderer can relay A2UI actions.",
"local": true,
"windows": ["canvas"],
"permissions": ["allow-canvas-a2ui-action"]
},
{
"identifier": "quickchat",
"description": "Quick Chat can select agents, configure its shortcut, send messages, render hosted widgets, and receive window events.",
apps/macos/Sources/OpenClaw/CanvasA2UIActionMessageHandler.swift
-165
View File
@@ -1,165 +0,0 @@
import AppKit
import Foundation
import OpenClawKit
import WebKit
final class CanvasA2UIActionMessageHandler: NSObject, WKScriptMessageHandler {
static let messageName = "openclawCanvasA2UIAction"
static let allMessageNames = [messageName]
private let sessionKey: String
private var expectedRemoteURL: URL?
init(sessionKey: String) {
self.sessionKey = sessionKey
super.init()
}
func setTrustedRemoteURL(_ url: URL?) {
self.expectedRemoteURL = url.flatMap {
CanvasHostedURLResolver.isCapabilityScopedA2UIURL($0) ? $0 : nil
}
}
func updateTrustForMainFrameNavigation(to url: URL) {
guard let expectedRemoteURL = self.expectedRemoteURL else { return }
// Hosted action trust is load-scoped. Once the main frame leaves the
// selected A2UI request, page navigation must never re-arm it.
if !Self.isExactRemoteSourceURL(url, expectedRemoteURL: expectedRemoteURL) {
self.expectedRemoteURL = nil
}
}
func isTrustedSourceURL(_ url: URL) -> Bool {
Self.isTrustedSourceURL(url, expectedRemoteURL: self.expectedRemoteURL)
}
static func isTrustedSourceURL(_ url: URL, expectedRemoteURL: URL?) -> Bool {
if let scheme = url.scheme?.lowercased(), CanvasScheme.allSchemes.contains(scheme) {
return true
}
return self.isExactRemoteSourceURL(url, expectedRemoteURL: expectedRemoteURL)
}
private static func isExactRemoteSourceURL(_ url: URL, expectedRemoteURL: URL?) -> Bool {
guard let expectedRemoteURL,
CanvasHostedURLResolver.isCapabilityScopedA2UIURL(expectedRemoteURL),
let actual = URLComponents(url: url, resolvingAgainstBaseURL: false),
let expected = URLComponents(url: expectedRemoteURL, resolvingAgainstBaseURL: false),
let actualScheme = actual.scheme?.lowercased(),
let expectedScheme = expected.scheme?.lowercased(),
actualScheme == expectedScheme,
actual.host?.lowercased() == expected.host?.lowercased(),
self.effectivePort(actual) == self.effectivePort(expected),
actual.user == nil,
actual.password == nil,
expected.user == nil,
expected.password == nil
else {
return false
}
return actual.percentEncodedPath == expected.percentEncodedPath &&
actual.percentEncodedQuery == expected.percentEncodedQuery
}
func userContentController(_: WKUserContentController, didReceive message: WKScriptMessage) {
guard Self.allMessageNames.contains(message.name) else { return }
// Only the main in-app document or the exact capability-scoped A2UI
// document may dispatch. Other web content remains render-only.
guard message.frameInfo.isMainFrame else { return }
guard let webView = message.webView, let url = message.frameInfo.request.url else { return }
guard self.isTrustedSourceURL(url) else {
return
}
let body: [String: Any] = {
if let dict = message.body as? [String: Any] { return dict }
if let dict = message.body as? [AnyHashable: Any] {
return dict.reduce(into: [String: Any]()) { acc, pair in
guard let key = pair.key as? String else { return }
acc[key] = pair.value
}
}
return [:]
}()
guard !body.isEmpty else { return }
let userActionAny = body["userAction"] ?? body
let userAction: [String: Any] = {
if let dict = userActionAny as? [String: Any] { return dict }
if let dict = userActionAny as? [AnyHashable: Any] {
return dict.reduce(into: [String: Any]()) { acc, pair in
guard let key = pair.key as? String else { return }
acc[key] = pair.value
}
}
return [:]
}()
guard !userAction.isEmpty else { return }
guard let name = OpenClawCanvasA2UIAction.extractActionName(userAction) else { return }
let actionId =
(userAction["id"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines).nonEmpty
?? UUID().uuidString
canvasWindowLogger.info("A2UI action \(name, privacy: .public) session=\(self.sessionKey, privacy: .public)")
let surfaceId = (userAction["surfaceId"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines)
.nonEmpty ?? "main"
let sourceComponentId = (userAction["sourceComponentId"] as? String)?
.trimmingCharacters(in: .whitespacesAndNewlines).nonEmpty ?? "-"
let instanceId = InstanceIdentity.instanceId.lowercased()
let contextJSON = OpenClawCanvasA2UIAction.compactJSON(userAction["context"])
// Token-efficient and unambiguous. The agent should treat this as a UI event and (by default) update Canvas.
let messageContext = OpenClawCanvasA2UIAction.AgentMessageContext(
actionName: name,
session: .init(key: self.sessionKey, surfaceId: surfaceId),
component: .init(id: sourceComponentId, host: InstanceIdentity.displayName, instanceId: instanceId),
contextJSON: contextJSON)
let text = OpenClawCanvasA2UIAction.formatAgentMessage(messageContext)
Task { [weak webView] in
if AppStateStore.shared.connectionMode == .local {
GatewayProcessManager.shared.setActive(true)
}
let result = await GatewayConnection.shared.sendAgent(
GatewayAgentInvocation(
message: text,
sessionKey: self.sessionKey,
thinking: "low",
deliver: false,
to: nil,
channel: .last,
idempotencyKey: actionId))
await MainActor.run {
guard let webView else { return }
let js = OpenClawCanvasA2UIAction.jsDispatchA2UIActionStatus(
actionId: actionId,
ok: result.ok,
error: result.error)
webView.evaluateJavaScript(js) { _, _ in }
}
if !result.ok {
canvasWindowLogger.error(
"""
A2UI action send failed name=\(name, privacy: .public) \
error=\(result.error ?? "unknown", privacy: .public)
""")
}
}
}
private static func effectivePort(_ components: URLComponents) -> Int? {
if let port = components.port { return port }
return switch components.scheme?.lowercased() {
case "http": 80
case "https": 443
default: nil
}
}
// Formatting helpers live in OpenClawKit (`OpenClawCanvasA2UIAction`).
}
apps/macos/Sources/OpenClaw/CanvasHostedURLResolver.swift
+21 -42
View File
@@ -1,18 +1,11 @@
import Foundation
struct CanvasHostedTarget: Equatable {
let url: URL
let allowsA2UIActions: Bool
}
enum CanvasHostedURLResolver {
private static let canvasPath = "/__openclaw__/canvas"
private static let a2uiPath = "/__openclaw__/a2ui"
private static let capabilityMarker = "/__openclaw__/cap/"
static func resolve(surfaceURL rawSurfaceURL: String?, target rawTarget: String) -> CanvasHostedTarget? {
guard let target = self.relativeHostedTarget(rawTarget),
var surface = self.capabilitySurface(rawSurfaceURL)
static func resolve(surfaceURL rawSurfaceURL: String?, target rawTarget: String) -> URL? {
guard let target = relativeHostedTarget(rawTarget),
var surface = capabilitySurface(rawSurfaceURL)
else {
return nil
}
@@ -24,39 +17,27 @@ enum CanvasHostedURLResolver {
surface.percentEncodedPath = surfacePath + target.percentEncodedPath
surface.percentEncodedQuery = target.percentEncodedQuery
surface.fragment = target.fragment
guard let url = surface.url else { return nil }
return CanvasHostedTarget(
url: url,
allowsA2UIActions: self.isA2UIPath(target.percentEncodedPath))
}
static func resolveA2UIURL(surfaceURL: String?) -> String? {
self.resolve(
surfaceURL: surfaceURL,
target: "\(self.a2uiPath)/?platform=macos")?.url.absoluteString
return surface.url
}
static func isHostedTarget(_ rawTarget: String) -> Bool {
self.relativeHostedTarget(rawTarget) != nil
}
static func isCapabilityScopedA2UIURL(_ url: URL) -> Bool {
guard let components = URLComponents(url: url, resolvingAgainstBaseURL: false),
self.isWebURL(components),
let marker = components.percentEncodedPath.range(of: self.capabilityMarker)
static func isAppLocalTarget(_ rawTarget: String) -> Bool {
let target = rawTarget.trimmingCharacters(in: .whitespacesAndNewlines)
guard let components = URLComponents(string: target),
components.scheme?.lowercased() == CanvasScheme.scheme,
components.host?.isEmpty == false,
components.user == nil,
components.password == nil,
components.port == nil,
components.percentEncodedPath.isEmpty ||
isCanonicalHostedPath(components.percentEncodedPath)
else {
return false
}
let suffix = components.percentEncodedPath[marker.upperBound...]
guard let separator = suffix.firstIndex(of: "/"), separator != suffix.startIndex else {
return false
}
let encodedCapability = String(suffix[..<separator])
guard let capability = encodedCapability.removingPercentEncoding, !capability.isEmpty else {
return false
}
let hostedPath = String(suffix[separator...])
return self.isCanonicalHostedPath(hostedPath) && self.isA2UIPath(hostedPath)
return true
}
private static func relativeHostedTarget(_ rawTarget: String) -> URLComponents? {
@@ -67,8 +48,8 @@ enum CanvasHostedURLResolver {
components.host == nil,
components.user == nil,
components.password == nil,
self.isCanonicalHostedPath(components.percentEncodedPath),
self.isCanvasPath(components.percentEncodedPath) || self.isA2UIPath(components.percentEncodedPath)
isCanonicalHostedPath(components.percentEncodedPath),
isCanvasPath(components.percentEncodedPath)
else {
return nil
}
@@ -79,7 +60,7 @@ enum CanvasHostedURLResolver {
let raw = rawSurfaceURL?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
guard !raw.isEmpty,
let components = URLComponents(string: raw),
self.isWebURL(components),
isWebURL(components),
components.user == nil,
components.password == nil,
components.percentEncodedQuery == nil,
@@ -117,7 +98,9 @@ enum CanvasHostedURLResolver {
var segment = String(encodedSegment)
while true {
guard let decoded = segment.removingPercentEncoding else { return false }
if decoded == segment { break }
if decoded == segment {
break
}
segment = decoded
}
if segment == "." || segment == ".." || segment.contains("/") || segment.contains("\\") {
@@ -130,8 +113,4 @@ enum CanvasHostedURLResolver {
private static func isCanvasPath(_ path: String) -> Bool {
path == self.canvasPath || path.hasPrefix("\(self.canvasPath)/")
}
private static func isA2UIPath(_ path: String) -> Bool {
path == self.a2uiPath || path.hasPrefix("\(self.a2uiPath)/")
}
}
apps/macos/Sources/OpenClaw/CanvasManager.swift
+26 -150
View File
@@ -12,12 +12,8 @@ final class CanvasManager {
private var panelController: CanvasWindowController?
private var panelSessionKey: String?
private var lastAutoA2UIUrl: String?
private var gatewayWatchTask: Task<Void, Never>?
private init() {
self.startGatewayObserver()
}
private init() {}
var onPanelVisibilityChanged: ((Bool) -> Void)?
@@ -32,21 +28,18 @@ final class CanvasManager {
func show(
sessionKey: String,
path: String? = nil,
placement: CanvasPlacement? = nil,
trustedA2UIActions: Bool = false) throws -> String
placement: CanvasPlacement? = nil) throws -> String
{
try self.showDetailed(
sessionKey: sessionKey,
target: path,
placement: placement,
trustedA2UIActions: trustedA2UIActions).directory
placement: placement).directory
}
func showDetailed(
sessionKey: String,
target: String? = nil,
placement: CanvasPlacement? = nil,
trustedA2UIActions: Bool = false) throws -> CanvasShowResult
placement: CanvasPlacement? = nil) throws -> CanvasShowResult
{
Self.logger.debug(
"""
@@ -58,7 +51,7 @@ final class CanvasManager {
let normalizedTarget = target?
.trimmingCharacters(in: .whitespacesAndNewlines)
.nonEmpty
let ensured = try self.ensureController(sessionKey: sessionKey)
let ensured = try ensureController(sessionKey: sessionKey)
let controller = ensured.controller
if !ensured.created {
@@ -67,7 +60,7 @@ final class CanvasManager {
// Existing session: only navigate when an explicit target was provided.
if let normalizedTarget {
controller.load(target: normalizedTarget, trustedA2UIActions: trustedA2UIActions)
controller.load(target: normalizedTarget)
self.refreshDebugStatus()
return self.makeShowResult(
directory: controller.directoryPath,
@@ -75,7 +68,6 @@ final class CanvasManager {
effectiveTarget: normalizedTarget)
}
self.maybeAutoNavigateToA2UIAsync(controller: controller)
self.refreshDebugStatus()
return CanvasShowResult(
directory: controller.directoryPath,
@@ -87,15 +79,11 @@ final class CanvasManager {
controller.applyPreferredPlacement(placement)
// New session: default to "/" so the user sees either the welcome page or `index.html`.
// New session: default to the local document root.
let effectiveTarget = normalizedTarget ?? "/"
Self.logger.debug("showDetailed showCanvas hasExplicitTarget=\(normalizedTarget != nil)")
// showCanvas presents the panel (presentAnchoredPanel) and loads the target.
controller.showCanvas(path: effectiveTarget, trustedA2UIActions: trustedA2UIActions)
controller.showCanvas(path: effectiveTarget)
Self.logger.debug("showDetailed showCanvas done")
if normalizedTarget == nil {
self.maybeAutoNavigateToA2UIAsync(controller: controller)
}
self.refreshDebugStatus()
return self.makeShowResult(
@@ -114,105 +102,8 @@ final class CanvasManager {
self.panelController?.hideCanvas()
}
func eval(sessionKey: String, javaScript: String) async throws -> String {
let ensured = try self.ensureController(sessionKey: sessionKey)
if ensured.created {
ensured.controller.load(target: "/")
}
let controller = ensured.controller
return try await controller.eval(javaScript: javaScript)
}
func snapshot(sessionKey: String, outPath: String?) async throws -> String {
let session = sessionKey.trimmingCharacters(in: .whitespacesAndNewlines)
// Snapshot is read-only: it must not create, switch, or reveal panel state.
// WebKit suspends rendering for ordered-out windows, so a hidden capture
// has no usable image; refuse with a clear contract instead.
guard let controller = self.panelController,
self.panelSessionKey == session,
controller.window?.isVisible == true
else {
throw NSError(domain: "Canvas", code: 21, userInfo: [
NSLocalizedDescriptionKey:
"CANVAS_HIDDEN: canvas snapshot needs a visible panel; run canvas.present first",
])
}
return try await controller.snapshot(to: outPath)
}
func prepare(sessionKey: String, target: String, trustedA2UIActions: Bool = false) throws {
let controller = try self.ensureController(sessionKey: sessionKey).controller
controller.load(target: target, trustedA2UIActions: trustedA2UIActions)
}
// MARK: - Gateway A2UI auto-nav
private func startGatewayObserver() {
self.gatewayWatchTask?.cancel()
self.gatewayWatchTask = Task { [weak self] in
guard let self else { return }
let stream = await GatewayConnection.shared.subscribe(bufferingNewest: 1)
for await push in stream {
self.handleGatewayPush(push)
}
}
}
private func handleGatewayPush(_ push: GatewayPush) {
guard case let .snapshot(snapshot) = push else { return }
let raw =
(snapshot.pluginsurfaceurls?["canvas"]?.value as? String)?
.trimmingCharacters(in: CharacterSet.whitespacesAndNewlines) ?? ""
if raw.isEmpty {
Self.logger.debug("canvas plugin surface URL missing in gateway snapshot")
} else {
Self.logger.debug("canvas plugin surface URL present in gateway snapshot")
}
let a2uiUrl = CanvasHostedURLResolver.resolveA2UIURL(surfaceURL: raw)
if a2uiUrl == nil, !raw.isEmpty {
Self.logger.debug("canvas plugin surface URL invalid; cannot resolve A2UI")
}
guard let controller = self.panelController else {
if a2uiUrl != nil {
Self.logger.debug("canvas panel not visible; skipping auto-nav")
}
return
}
self.maybeAutoNavigateToA2UI(controller: controller, a2uiUrl: a2uiUrl)
}
private func maybeAutoNavigateToA2UIAsync(controller: CanvasWindowController) {
Task { [weak self] in
guard let self else { return }
let a2uiUrl = await self.resolveA2UIHostUrl()
await MainActor.run {
guard self.panelController === controller else { return }
self.maybeAutoNavigateToA2UI(controller: controller, a2uiUrl: a2uiUrl)
}
}
}
private func maybeAutoNavigateToA2UI(controller: CanvasWindowController, a2uiUrl: String?) {
guard let a2uiUrl else { return }
let shouldNavigate = controller.shouldAutoNavigateToA2UI(
lastAutoTarget: self.lastAutoA2UIUrl,
candidateTarget: a2uiUrl)
guard shouldNavigate else {
Self.logger.debug("canvas auto-nav skipped; target unchanged")
return
}
Self.logger.debug("canvas auto-nav to capability-scoped A2UI")
controller.load(target: a2uiUrl, trustedA2UIActions: true)
self.lastAutoA2UIUrl = a2uiUrl
}
private func resolveA2UIHostUrl() async -> String? {
let raw = await GatewayConnection.shared.canvasPluginSurfaceUrl()
return CanvasHostedURLResolver.resolveA2UIURL(surfaceURL: raw)
}
func refreshDebugStatus() {
guard let controller = self.panelController else { return }
guard let controller = panelController else { return }
let enabled = AppStateStore.shared.debugPaneEnabled
let mode = AppStateStore.shared.connectionMode
let title: String?
@@ -247,18 +138,14 @@ final class CanvasManager {
return NSRect(x: pt.x, y: pt.y, width: 1, height: 1)
}
// placement interpretation is handled by the window controller.
// MARK: - Helpers
/// Content operations must not reveal Canvas or activate the app. Only explicit presentation or user intent may
/// re-present it, or reconnect-time agent content would reopen a panel the user closed.
/// A session switch keeps the single-panel model: the previous panel closes and the new one stays hidden.
/// A session switch keeps the single-panel model by replacing the previous panel.
private func ensureController(sessionKey: String) throws -> (controller: CanvasWindowController, created: Bool) {
let anchorProvider = self.defaultAnchorProvider ?? Self.mouseAnchorProvider
let session = sessionKey.trimmingCharacters(in: .whitespacesAndNewlines)
if let controller = self.panelController, self.panelSessionKey == session {
if let controller = panelController, panelSessionKey == session {
Self.logger.debug("ensureController reuse existing session=\(session, privacy: .public)")
controller.onVisibilityChanged = { [weak self] visible in
self?.onPanelVisibilityChanged?(visible)
@@ -292,20 +179,10 @@ final class CanvasManager {
let trimmed = target.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return nil }
if let url = URL(string: trimmed), let scheme = url.scheme?.lowercased() {
if scheme == "https" || scheme == "http" || scheme == "file" { return url }
}
// Convenience: existing absolute *file* paths resolve as local files.
// (Avoid treating Canvas routes like "/" as filesystem paths.)
if trimmed.hasPrefix("/") {
var isDir: ObjCBool = false
if FileManager().fileExists(atPath: trimmed, isDirectory: &isDir), !isDir.boolValue {
return URL(fileURLWithPath: trimmed)
}
}
return nil
guard let url = URL(string: trimmed), let scheme = url.scheme?.lowercased() else { return nil }
return scheme == "https" || scheme == "http" || CanvasScheme.allSchemes.contains(scheme)
? url
: nil
}
private func makeShowResult(
@@ -340,15 +217,19 @@ final class CanvasManager {
let withoutQuery = trimmed.split(separator: "?", maxSplits: 1, omittingEmptySubsequences: false).first
.map(String.init) ?? trimmed
var path = withoutQuery
if path.hasPrefix("/") { path.removeFirst() }
if path.hasPrefix("/") {
path.removeFirst()
}
path = path.removingPercentEncoding ?? path
// Root special-case: built-in scaffold page when no index exists.
// Root special-case: resolve an existing index document.
if path.isEmpty {
let a = sessionDir.appendingPathComponent("index.html", isDirectory: false)
let b = sessionDir.appendingPathComponent("index.htm", isDirectory: false)
if fm.fileExists(atPath: a.path) || fm.fileExists(atPath: b.path) { return .ok }
return .welcome
if fm.fileExists(atPath: a.path) || fm.fileExists(atPath: b.path) {
return .ok
}
return .notFound
}
// Direct file or directory.
@@ -375,12 +256,12 @@ final class CanvasManager {
private static func indexExists(in dir: URL) -> Bool {
let fm = FileManager()
let a = dir.appendingPathComponent("index.html", isDirectory: false)
if fm.fileExists(atPath: a.path) { return true }
if fm.fileExists(atPath: a.path) {
return true
}
let b = dir.appendingPathComponent("index.htm", isDirectory: false)
return fm.fileExists(atPath: b.path)
}
// no bundled A2UI shell; scaffold fallback is purely visual
}
#if DEBUG
@@ -389,15 +270,10 @@ extension CanvasManager {
self.panelController?.window?.isVisible
}
var _testHasPanelController: Bool {
self.panelController != nil
}
func _testResetPanel() {
self.panelController?.close()
self.panelController = nil
self.panelSessionKey = nil
self.lastAutoA2UIUrl = nil
}
}
#endif
apps/macos/Sources/OpenClaw/CanvasSchemeHandler.swift
-57
View File
@@ -1,5 +1,4 @@
import Foundation
import OpenClawKit
import OSLog
import WebKit
@@ -65,17 +64,6 @@ final class CanvasSchemeHandler: NSObject, WKURLSchemeHandler {
if path.hasPrefix("/") { path.removeFirst() }
path = path.removingPercentEncoding ?? path
// Special-case: welcome page when root index is missing.
if path.isEmpty {
let indexA = sessionRoot.appendingPathComponent("index.html", isDirectory: false)
let indexB = sessionRoot.appendingPathComponent("index.htm", isDirectory: false)
if !FileManager().fileExists(atPath: indexA.path),
!FileManager().fileExists(atPath: indexB.path)
{
return self.scaffoldPage(sessionRoot: sessionRoot)
}
}
let resolved = self.resolveFileURL(sessionRoot: sessionRoot, requestPath: path)
guard let fileURL = resolved else {
return self.html("Not Found", title: "Canvas: 404")
@@ -191,51 +179,6 @@ final class CanvasSchemeHandler: NSObject, WKURLSchemeHandler {
return CanvasResponse(mime: "text/html", data: Data(html.utf8))
}
private func welcomePage(sessionRoot: URL) -> CanvasResponse {
let escaped = sessionRoot.path
.replacingOccurrences(of: "&", with: "&amp;")
.replacingOccurrences(of: "<", with: "&lt;")
.replacingOccurrences(of: ">", with: "&gt;")
let body = """
<div style="font-weight:600; font-size:14px;">Canvas is ready.</div>
<div class="muted">Create <code>index.html</code> in:</div>
<div style="margin-top:10px;"><code>\(escaped)</code></div>
"""
return self.html(body, title: "Canvas")
}
private func scaffoldPage(sessionRoot: URL) -> CanvasResponse {
// Default Canvas UX: when no index exists, show the built-in scaffold page.
if let data = self.loadBundledResourceData(relativePath: "CanvasScaffold/scaffold.html") {
return CanvasResponse(mime: "text/html", data: data)
}
// Fallback for dev misconfiguration: show the classic welcome page.
return self.welcomePage(sessionRoot: sessionRoot)
}
private func loadBundledResourceData(relativePath: String) -> Data? {
let trimmed = relativePath.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return nil }
if trimmed.contains("..") || trimmed.contains("\\") { return nil }
let parts = trimmed.split(separator: "/")
guard let filename = parts.last else { return nil }
let subdirectory =
parts.count > 1 ? parts.dropLast().joined(separator: "/") : nil
let fileURL = URL(fileURLWithPath: String(filename))
let ext = fileURL.pathExtension
let name = fileURL.deletingPathExtension().lastPathComponent
guard !name.isEmpty, !ext.isEmpty else { return nil }
let bundle = OpenClawKitResources.bundle
let resourceURL =
bundle.url(forResource: name, withExtension: ext, subdirectory: subdirectory)
?? bundle.url(forResource: name, withExtension: ext)
guard let resourceURL else { return nil }
return try? Data(contentsOf: resourceURL)
}
private func textEncodingName(forMimeType mimeType: String) -> String? {
if mimeType.hasPrefix("text/") { return "utf-8" }
switch mimeType {
apps/macos/Sources/OpenClaw/CanvasWindowController+Helpers.swift
-5
View File
@@ -12,11 +12,6 @@ extension CanvasWindowController {
return String(scalars)
}
static func jsStringLiteral(_ value: String) -> String {
let data = try? JSONEncoder().encode(value)
return data.flatMap { String(data: $0, encoding: .utf8) } ?? "\"\""
}
static func storedFrameDefaultsKey(sessionKey: String) -> String {
"openclaw.canvas.frame.\(self.sanitizeSessionKey(sessionKey))"
}
apps/macos/Sources/OpenClaw/CanvasWindowController+Navigation.swift
-15
View File
@@ -56,23 +56,8 @@ extension CanvasWindowController {
decisionHandler(.cancel)
}
@MainActor
func webView(
_: WKWebView,
decidePolicyFor navigationResponse: WKNavigationResponse,
decisionHandler: @escaping @MainActor @Sendable (WKNavigationResponsePolicy) -> Void)
{
// Revoke only once navigation produces a response. Requests canceled
// above leave the original trusted A2UI document active.
if navigationResponse.isForMainFrame, let url = navigationResponse.response.url {
self.updateA2UITrustForMainFrameNavigation(to: url)
}
decisionHandler(.allow)
}
func webView(_ webView: WKWebView, didCommit _: WKNavigation?) {
if let url = webView.url {
self.updateA2UITrustForMainFrameNavigation(to: url)
self.updateFilePollingForCommittedNavigation(to: url)
}
}
apps/macos/Sources/OpenClaw/CanvasWindowController.swift
+17 -178
View File
@@ -11,12 +11,10 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
private let sessionDir: URL
private let schemeHandler: CanvasSchemeHandler
let webView: WKWebView
private var a2uiActionMessageHandler: CanvasA2UIActionMessageHandler?
private let watcher: CanvasFileWatcher
private let container: HoverChromeContainerView
let presentation: CanvasPresentation
var preferredPlacement: CanvasPlacement?
private(set) var currentTarget: String?
private var debugStatusEnabled = false
private var debugStatusTitle: String?
private var debugStatusSubtitle: String?
@@ -51,77 +49,9 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
}
canvasWindowLogger.debug("CanvasWindowController init scheme handler installed")
// Bridge A2UI "a2uiaction" DOM events back into the native agent loop.
//
// This fallback event bridge runs only on the app-owned scheme. The
// script-message handler separately gates hosted A2UI to its exact URL.
canvasWindowLogger.debug("CanvasWindowController init building A2UI bridge script")
let injectedSessionKey = sessionKey.trimmingCharacters(in: .whitespacesAndNewlines).nonEmpty ?? "main"
let allowedSchemesJSON = (
try? String(
data: JSONSerialization.data(withJSONObject: CanvasScheme.allSchemes),
encoding: .utf8)) ?? "[]"
let bridgeScript = """
(() => {
try {
const allowedSchemes = \(allowedSchemesJSON);
const protocol = location.protocol.replace(':', '');
if (!allowedSchemes.includes(protocol)) return;
if (globalThis.__openclawA2UIBridgeInstalled) return;
globalThis.__openclawA2UIBridgeInstalled = true;
const sessionKey = \(Self.jsStringLiteral(injectedSessionKey));
const machineName = \(Self.jsStringLiteral(InstanceIdentity.displayName));
const instanceId = \(Self.jsStringLiteral(InstanceIdentity.instanceId));
globalThis.addEventListener('a2uiaction', (evt) => {
try {
const payload = evt?.detail ?? evt?.payload ?? null;
if (!payload || payload.eventType !== 'a2ui.action') return;
const action = payload.action ?? null;
const name = action?.name ?? '';
if (!name) return;
const context = Array.isArray(action?.context) ? action.context : [];
const userAction = {
name,
surfaceId: payload.surfaceId ?? 'main',
sourceComponentId: payload.sourceComponentId ?? '',
dataContextPath: payload.dataContextPath ?? '',
timestamp: new Date().toISOString(),
...(context.length ? { context } : {}),
};
const handler = globalThis.webkit?.messageHandlers?.openclawCanvasA2UIAction;
// If the bundled A2UI shell is present, let it forward actions so we keep its richer
// context resolution (data model path lookups, surface detection, etc.).
const hasBundledA2UIHost =
!!globalThis.openclawA2UI ||
!!document.querySelector('openclaw-a2ui-host');
if (hasBundledA2UIHost && handler?.postMessage) return;
// Otherwise, forward directly when possible.
if (!hasBundledA2UIHost && handler?.postMessage) {
handler.postMessage({ userAction });
return;
}
// Without the native handler, fail closed instead of exposing an
// unattended deep-link credential to page JavaScript.
} catch {}
}, true);
} catch {}
})();
"""
config.userContentController.addUserScript(
WKUserScript(source: bridgeScript, injectionTime: .atDocumentStart, forMainFrameOnly: true))
canvasWindowLogger.debug("CanvasWindowController init A2UI bridge installed")
canvasWindowLogger.debug("CanvasWindowController init creating WKWebView")
self.webView = WKWebView(frame: .zero, configuration: config)
// Canvas scaffold is a fully self-contained HTML page; avoid relying on transparency underlays.
// Presented documents render against an opaque surface.
self.webView.setValue(true, forKey: "drawsBackground")
let sessionDir = self.sessionDir
@@ -154,12 +84,6 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
canvasWindowLogger.debug("CanvasWindowController init makeWindow done")
super.init(window: window)
let handler = CanvasA2UIActionMessageHandler(sessionKey: sessionKey)
self.a2uiActionMessageHandler = handler
for name in CanvasA2UIActionMessageHandler.allMessageNames {
self.webView.configuration.userContentController.add(handler, name: name)
}
self.webView.navigationDelegate = self
self.webView.uiDelegate = self
self.window?.delegate = self
@@ -174,14 +98,11 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
}
@available(*, unavailable)
required init?(coder: NSCoder) {
required init?(coder _: NSCoder) {
fatalError("init(coder:) is not supported")
}
@MainActor deinit {
for name in CanvasA2UIActionMessageHandler.allMessageNames {
self.webView.configuration.userContentController.removeScriptMessageHandler(forName: name)
}
self.watcher.stop()
}
@@ -189,68 +110,51 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
self.preferredPlacement = placement
}
func showCanvas(path: String? = nil, trustedA2UIActions: Bool = false) {
if case let .panel(anchorProvider) = self.presentation {
self.presentAnchoredPanel(anchorProvider: anchorProvider)
func showCanvas(path: String? = nil) {
if case let .panel(anchorProvider) = presentation {
presentAnchoredPanel(anchorProvider: anchorProvider)
if let path {
self.load(target: path, trustedA2UIActions: trustedA2UIActions)
self.load(target: path)
}
return
}
// The window is built in init, so skip showWindow(_:); it would make the
// window key and steal focus from the user's current window.
self.window?.orderFrontRegardless()
window?.orderFrontRegardless()
if let path {
self.load(target: path, trustedA2UIActions: trustedA2UIActions)
self.load(target: path)
}
self.setCanvasVisible(true)
}
func hideCanvas() {
if case .panel = self.presentation {
self.persistFrameIfPanel()
persistFrameIfPanel()
}
self.window?.orderOut(nil)
window?.orderOut(nil)
self.setCanvasVisible(false)
}
func load(target: String, trustedA2UIActions: Bool = false) {
func load(target: String) {
let trimmed = target.trimmingCharacters(in: .whitespacesAndNewlines)
self.currentTarget = trimmed
self.a2uiActionMessageHandler?.setTrustedRemoteURL(nil)
if let url = URL(string: trimmed), let scheme = url.scheme?.lowercased() {
if CanvasScheme.allSchemes.contains(scheme) {
canvasWindowLogger.debug("canvas load app-local URL")
self.webView.load(URLRequest(url: url))
return
}
if scheme == "https" || scheme == "http" {
if trustedA2UIActions {
self.a2uiActionMessageHandler?.setTrustedRemoteURL(url)
}
canvasWindowLogger.debug(
"canvas load web scheme=\(scheme, privacy: .public) host=\(url.host ?? "-", privacy: .public)")
self.webView.load(URLRequest(url: url))
return
}
if scheme == "file" {
canvasWindowLogger.debug("canvas load file \(url.absoluteString, privacy: .public)")
self.loadFile(url)
return
}
}
// Convenience: absolute file paths resolve as local files when they exist.
// (Avoid treating Canvas routes like "/" as filesystem paths.)
if trimmed.hasPrefix("/") {
var isDir: ObjCBool = false
if FileManager().fileExists(atPath: trimmed, isDirectory: &isDir), !isDir.boolValue {
let url = URL(fileURLWithPath: trimmed)
canvasWindowLogger.debug("canvas load file \(url.absoluteString, privacy: .public)")
self.loadFile(url)
return
}
}
guard let url = CanvasScheme.makeURL(
session: CanvasWindowController.sanitizeSessionKey(self.sessionKey),
session: CanvasWindowController.sanitizeSessionKey(sessionKey),
path: trimmed)
else {
canvasWindowLogger
@@ -283,10 +187,6 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
self.watcher.isPolling
}
func updateA2UITrustForMainFrameNavigation(to url: URL) {
self.a2uiActionMessageHandler?.updateTrustForMainFrameNavigation(to: url)
}
func updateDebugStatus(enabled: Bool, title: String?, subtitle: String?) {
self.debugStatusEnabled = enabled
self.debugStatusTitle = title
@@ -302,68 +202,7 @@ final class CanvasWindowController: NSWindowController, WKNavigationDelegate, WK
subtitle: self.debugStatusSubtitle)
}
private func loadFile(_ url: URL) {
let fileURL = url.isFileURL ? url : URL(fileURLWithPath: url.path)
let accessDir = fileURL.deletingLastPathComponent()
self.webView.loadFileURL(fileURL, allowingReadAccessTo: accessDir)
}
func eval(javaScript: String) async throws -> String {
try await WebViewJavaScriptSupport.evaluateToString(webView: self.webView, javaScript: javaScript)
}
func snapshot(to outPath: String?) async throws -> String {
let image: NSImage = try await withCheckedThrowingContinuation { cont in
self.webView.takeSnapshot(with: nil) { image, error in
if let error {
cont.resume(throwing: error)
return
}
guard let image else {
cont.resume(throwing: NSError(domain: "Canvas", code: 11, userInfo: [
NSLocalizedDescriptionKey: "snapshot returned nil image",
]))
return
}
cont.resume(returning: image)
}
}
guard let tiff = image.tiffRepresentation,
let rep = NSBitmapImageRep(data: tiff),
let png = rep.representation(using: .png, properties: [:])
else {
throw NSError(domain: "Canvas", code: 12, userInfo: [
NSLocalizedDescriptionKey: "failed to encode png",
])
}
let snapshotID = "\(CanvasWindowController.sanitizeSessionKey(self.sessionKey))-\(UUID().uuidString)"
let path: String = if let outPath, !outPath.isEmpty {
outPath
} else {
"/tmp/openclaw-canvas-\(snapshotID).png"
}
try png.write(to: URL(fileURLWithPath: path), options: [.atomic])
return path
}
var directoryPath: String {
self.sessionDir.path
}
func shouldAutoNavigateToA2UI(lastAutoTarget: String?, candidateTarget: String) -> Bool {
let current = (self.currentTarget ?? "").trimmingCharacters(in: .whitespacesAndNewlines)
let candidate = candidateTarget.trimmingCharacters(in: .whitespacesAndNewlines)
if current.isEmpty || current == "/" { return true }
if !candidate.isEmpty, current == candidate { return false }
if let lastAuto = lastAutoTarget?.trimmingCharacters(in: .whitespacesAndNewlines),
!lastAuto.isEmpty,
current == lastAuto
{
return true
}
return false
}
}
apps/macos/Sources/OpenClaw/DebugSettings.swift
-73
View File
@@ -30,9 +30,6 @@ struct DebugSettings: View {
@State private var canvasSessionKey: String = "main"
@State private var canvasStatus: String?
@State private var canvasError: String?
@State private var canvasEvalJS: String = "document.title"
@State private var canvasEvalResult: String?
@State private var canvasSnapshotPath: String?
init(state: AppState = AppStateStore.shared) {
self.state = state
@@ -584,52 +581,12 @@ struct DebugSettings: View {
Spacer(minLength: 0)
}
HStack(spacing: 8) {
TextField("Eval JS", text: self.$canvasEvalJS)
.textFieldStyle(.roundedBorder)
.font(.caption.monospaced())
.frame(maxWidth: 520)
Button("Eval") {
Task { await self.canvasEval() }
}
.buttonStyle(.bordered)
Button("Snapshot") {
Task { await self.canvasSnapshot() }
}
.buttonStyle(.bordered)
Spacer(minLength: 0)
}
if let canvasStatus {
Text(canvasStatus)
.font(.caption2.monospaced())
.foregroundStyle(.secondary)
.textSelection(.enabled)
}
if let canvasEvalResult {
Text("eval → \(canvasEvalResult)")
.font(.caption2.monospaced())
.foregroundStyle(.secondary)
.lineLimit(2)
.truncationMode(.middle)
.textSelection(.enabled)
}
if let canvasSnapshotPath {
HStack(spacing: 8) {
Text("snapshot → \(canvasSnapshotPath)")
.font(.caption2.monospaced())
.foregroundStyle(.secondary)
.lineLimit(1)
.truncationMode(.middle)
.textSelection(.enabled)
Button("Reveal") {
NSWorkspace.shared
.activateFileViewerSelecting([URL(fileURLWithPath: canvasSnapshotPath)])
}
.buttonStyle(.bordered)
Spacer(minLength: 0)
}
}
if let canvasError {
Text(canvasError)
.font(.caption2)
@@ -885,36 +842,6 @@ extension DebugSettings {
self.canvasError = error.localizedDescription
}
}
@MainActor
private func canvasEval() async {
self.canvasError = nil
self.canvasEvalResult = nil
do {
let session = self.canvasSessionKey.trimmingCharacters(in: .whitespacesAndNewlines)
let result = try await CanvasManager.shared.eval(
sessionKey: session.isEmpty ? "main" : session,
javaScript: self.canvasEvalJS)
self.canvasEvalResult = result
} catch {
self.canvasError = error.localizedDescription
}
}
@MainActor
private func canvasSnapshot() async {
self.canvasError = nil
self.canvasSnapshotPath = nil
do {
let session = self.canvasSessionKey.trimmingCharacters(in: .whitespacesAndNewlines)
let path = try await CanvasManager.shared.snapshot(
sessionKey: session.isEmpty ? "main" : session,
outPath: nil)
self.canvasSnapshotPath = path
} catch {
self.canvasError = error.localizedDescription
}
}
}
struct PlainSettingsGroupBoxStyle: GroupBoxStyle {
apps/macos/Sources/OpenClaw/NodeMode/MacNodeCanvasHostedSurfaceResolver.swift
+1 -12
View File
@@ -12,18 +12,7 @@ struct MacNodeCanvasHostedSurfaceResolver: Sendable {
self.refreshSurfaceURL = refreshSurfaceURL
}
func resolveA2UIURL(forceRefresh: Bool = false) async -> String? {
let observedSurface = await currentSurfaceURL()
if !forceRefresh,
let current = CanvasHostedURLResolver.resolveA2UIURL(surfaceURL: observedSurface)
{
return current
}
let refreshedSurface = await refreshSurfaceURL(observedSurface)
return CanvasHostedURLResolver.resolveA2UIURL(surfaceURL: refreshedSurface)
}
func resolveTarget(_ target: String?) async throws -> CanvasHostedTarget? {
func resolveTarget(_ target: String?) async throws -> URL? {
guard let target, CanvasHostedURLResolver.isHostedTarget(target) else { return nil }
let observedSurface = await currentSurfaceURL()
if let refreshedSurface = await refreshSurfaceURL(observedSurface),
apps/macos/Sources/OpenClaw/NodeMode/MacNodeModeCoordinator.swift
-5
View File
@@ -1209,11 +1209,6 @@ extension MacNodeModeCoordinator {
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
OpenClawCanvasCommand.evalJS.rawValue,
OpenClawCanvasCommand.snapshot.rawValue,
OpenClawCanvasA2UICommand.push.rawValue,
OpenClawCanvasA2UICommand.pushJSONL.rawValue,
OpenClawCanvasA2UICommand.reset.rawValue,
]
if computerControlProvider == .peekaboo {
apps/macos/Sources/OpenClaw/NodeMode/MacNodeRuntime.swift
+61 -252
View File
@@ -1,4 +1,3 @@
import AppKit
import Foundation
import OpenClawIPC
import OpenClawKit
@@ -89,7 +88,7 @@ actor MacNodeClaudeSessionCatalogWorker {
}
private func cancel(id: UUID) {
if let index = self.pending.firstIndex(where: { $0.id == id }) {
if let index = pending.firstIndex(where: { $0.id == id }) {
let pending = self.pending.remove(at: index)
pending.continuation.resume(throwing: CancellationError())
return
@@ -198,45 +197,34 @@ actor MacNodeRuntime {
/// One branch per advertised native command keeps command ownership explicit.
func handleInvoke(_ req: BridgeInvokeRequest) async -> BridgeInvokeResponse {
let command = req.command
if self.isCanvasCommand(command), !Self.canvasEnabled() {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: OpenClawNodeError(
code: .unavailable,
message: "CANVAS_DISABLED: enable Canvas in Settings"))
if let rejection = Self.canvasCommandRejection(req) {
return rejection
}
if let cuaResponse = await self.handleCuaInvokeIfSelected(req) {
if let cuaResponse = await handleCuaInvokeIfSelected(req) {
return cuaResponse
}
do {
switch command {
case OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
OpenClawCanvasCommand.evalJS.rawValue,
OpenClawCanvasCommand.snapshot.rawValue:
return try await self.handleCanvasInvoke(req)
case OpenClawCanvasA2UICommand.reset.rawValue,
OpenClawCanvasA2UICommand.push.rawValue,
OpenClawCanvasA2UICommand.pushJSONL.rawValue:
return try await self.handleA2UIInvoke(req)
OpenClawCanvasCommand.navigate.rawValue:
return try await handleCanvasInvoke(req)
case OpenClawCameraCommand.snap.rawValue,
OpenClawCameraCommand.clip.rawValue,
OpenClawCameraCommand.list.rawValue,
OpenClawCameraCommand.ptzStatus.rawValue,
OpenClawCameraCommand.ptzControl.rawValue:
return try await self.handleCameraInvoke(req)
return try await handleCameraInvoke(req)
case OpenClawLocationCommand.get.rawValue:
return try await self.handleLocationInvoke(req)
return try await handleLocationInvoke(req)
case MacNodeScreenCommand.snapshot.rawValue:
return try await self.handleScreenSnapshotInvoke(req)
return try await handleScreenSnapshotInvoke(req)
case MacNodeScreenCommand.record.rawValue:
return try await self.handleScreenRecordInvoke(req)
return try await handleScreenRecordInvoke(req)
case OpenClawComputerCommand.act.rawValue:
return try await self.handleComputerActInvoke(req)
return try await handleComputerActInvoke(req)
case OpenClawSystemCommand.notify.rawValue:
return try await self.handleSystemNotify(req)
return try await handleSystemNotify(req)
case MacNodeCodexThreadCatalogContract.listCommand,
MacNodeCodexThreadCatalogContract.turnsCommand:
return try await self.handleCodexThreadInvoke(req)
@@ -265,13 +253,36 @@ actor MacNodeRuntime {
case .deviceNotFound, .unsupported, .partial: .unavailable
}
return Self.errorResponse(req, code: code, message: error.localizedDescription)
} catch let error as MacNodeCanvasTargetError {
return Self.errorResponse(req, code: .invalidRequest, message: error.localizedDescription)
} catch {
return Self.errorResponse(req, code: .unavailable, message: error.localizedDescription)
}
}
private func isCanvasCommand(_ command: String) -> Bool {
command.hasPrefix("canvas.") || command.hasPrefix("canvas.a2ui.")
private static let canvasCommands: Set<String> = [
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
]
private static func canvasCommandRejection(_ req: BridgeInvokeRequest) -> BridgeInvokeResponse? {
guard req.command.hasPrefix("canvas.") else { return nil }
guard self.canvasCommands.contains(req.command) else {
return self.errorResponse(
req,
code: .invalidRequest,
message: "INVALID_REQUEST: unknown command")
}
guard self.canvasEnabled() else {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: OpenClawNodeError(
code: .unavailable,
message: "CANVAS_DISABLED: enable Canvas in Settings"))
}
return nil
}
private func handleCuaInvokeIfSelected(_ req: BridgeInvokeRequest) async -> BridgeInvokeResponse? {
@@ -301,13 +312,13 @@ actor MacNodeRuntime {
message: "UNAVAILABLE: Codex session catalog is disabled")
}
let payload: String = if req.command == MacNodeCodexThreadCatalogContract.listCommand {
if let request = self.codexThreadListRequest {
if let request = codexThreadListRequest {
try await request(req.paramsJSON)
} else {
try await self.codexThreadCatalogClient.list(paramsJSON: req.paramsJSON)
}
} else {
if let request = self.codexThreadTurnsRequest {
if let request = codexThreadTurnsRequest {
try await request(req.paramsJSON)
} else {
try await self.codexThreadCatalogClient.turns(paramsJSON: req.paramsJSON)
@@ -341,8 +352,7 @@ extension MacNodeRuntime {
OpenClawCanvasPresentParams()
let urlTrimmed = params.url?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let url = urlTrimmed.isEmpty ? nil : urlTrimmed
let hostedTarget = try await self.canvasHostedSurfaceResolver.resolveTarget(url)
let effectiveURL = hostedTarget?.url.absoluteString ?? url
let effectiveURL = try await resolveCanvasTarget(url)
let placement = params.placement.map {
CanvasPlacement(x: $0.x, y: $0.y, width: $0.width, height: $0.height)
}
@@ -351,8 +361,7 @@ extension MacNodeRuntime {
_ = try CanvasManager.shared.showDetailed(
sessionKey: sessionKey,
target: effectiveURL,
placement: placement,
trustedA2UIActions: hostedTarget?.allowsA2UIActions == true)
placement: placement)
}
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.hide.rawValue:
@@ -363,70 +372,38 @@ extension MacNodeRuntime {
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.navigate.rawValue:
let params = try Self.decodeParams(OpenClawCanvasNavigateParams.self, from: req.paramsJSON)
let hostedTarget = try await self.canvasHostedSurfaceResolver.resolveTarget(params.url)
let effectiveURL = hostedTarget?.url.absoluteString ?? params.url
let effectiveURL = try await resolveCanvasTarget(params.url)
let sessionKey = self.mainSessionKey
try await MainActor.run {
_ = try CanvasManager.shared.show(
sessionKey: sessionKey,
path: effectiveURL,
trustedA2UIActions: hostedTarget?.allowsA2UIActions == true)
path: effectiveURL)
}
return BridgeInvokeResponse(id: req.id, ok: true)
case OpenClawCanvasCommand.evalJS.rawValue:
let params = try Self.decodeParams(OpenClawCanvasEvalParams.self, from: req.paramsJSON)
let sessionKey = self.mainSessionKey
let result = try await CanvasManager.shared.eval(
sessionKey: sessionKey,
javaScript: params.javaScript)
let payload = try Self.encodePayload(["result": result] as [String: String])
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload)
case OpenClawCanvasCommand.snapshot.rawValue:
let params = try? Self.decodeParams(OpenClawCanvasSnapshotParams.self, from: req.paramsJSON)
let format = params?.format ?? .jpeg
let maxWidth: Int? = {
if let raw = params?.maxWidth, raw > 0 {
return raw
}
return switch format {
case .png: 900
case .jpeg: 1600
}
}()
let quality = params?.quality ?? 0.9
let sessionKey = self.mainSessionKey
let path = try await CanvasManager.shared.snapshot(sessionKey: sessionKey, outPath: nil)
defer { try? FileManager().removeItem(atPath: path) }
let data = try Data(contentsOf: URL(fileURLWithPath: path))
guard let image = NSImage(data: data) else {
return Self.errorResponse(req, code: .unavailable, message: "canvas snapshot decode failed")
}
let encoded = try Self.encodeCanvasSnapshot(
image: image,
format: format,
maxWidth: maxWidth,
quality: quality)
let payload = try Self.encodePayload([
"format": format == .jpeg ? "jpeg" : "png",
"base64": encoded.base64EncodedString(),
])
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload)
default:
return Self.errorResponse(req, code: .invalidRequest, message: "INVALID_REQUEST: unknown command")
}
}
private func handleA2UIInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
switch req.command {
case OpenClawCanvasA2UICommand.reset.rawValue:
try await self.handleA2UIReset(req)
case OpenClawCanvasA2UICommand.push.rawValue,
OpenClawCanvasA2UICommand.pushJSONL.rawValue:
try await self.handleA2UIPush(req)
default:
Self.errorResponse(req, code: .invalidRequest, message: "INVALID_REQUEST: unknown command")
private func resolveCanvasTarget(_ rawTarget: String?) async throws -> String? {
guard let target = rawTarget?.trimmingCharacters(in: .whitespacesAndNewlines).nonEmpty else {
return nil
}
if CanvasHostedURLResolver.isHostedTarget(target) {
return try await self.canvasHostedSurfaceResolver.resolveTarget(target)?.absoluteString
}
guard CanvasHostedURLResolver.isAppLocalTarget(target) else {
throw MacNodeCanvasTargetError.invalidTarget
}
return target
}
}
private enum MacNodeCanvasTargetError: LocalizedError {
case invalidTarget
var errorDescription: String? {
"INVALID_REQUEST: canvas target must be a hosted widget-document path or app-local Canvas URL"
}
}
@@ -802,119 +779,6 @@ extension MacNodeRuntime {
}
}
// MARK: - A2UI host
extension MacNodeRuntime {
private func handleA2UIReset(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
try await self.ensureA2UIHost()
let sessionKey = self.mainSessionKey
let json = try await CanvasManager.shared.eval(sessionKey: sessionKey, javaScript: """
(() => {
const host = globalThis.openclawA2UI;
if (!host) return JSON.stringify({ ok: false, error: "missing openclawA2UI" });
return JSON.stringify(host.reset());
})()
""")
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json)
}
private func handleA2UIPush(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse {
let command = req.command
let messages: [OpenClawKit.AnyCodable]
if command == OpenClawCanvasA2UICommand.pushJSONL.rawValue {
let params = try Self.decodeParams(OpenClawCanvasA2UIPushJSONLParams.self, from: req.paramsJSON)
messages = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(params.jsonl)
} else {
do {
let params = try Self.decodeParams(OpenClawCanvasA2UIPushParams.self, from: req.paramsJSON)
messages = params.messages
} catch {
let params = try Self.decodeParams(OpenClawCanvasA2UIPushJSONLParams.self, from: req.paramsJSON)
messages = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(params.jsonl)
}
}
try await self.ensureA2UIHost()
let messagesJSON = try OpenClawCanvasA2UIJSONL.encodeMessagesJSONArray(messages)
let js = """
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return JSON.stringify({ ok: false, error: "missing openclawA2UI" });
const messages = \(messagesJSON);
return JSON.stringify(host.applyMessages(messages));
} catch (e) {
return JSON.stringify({ ok: false, error: String(e?.message ?? e) });
}
})()
"""
let sessionKey = self.mainSessionKey
let resultJSON = try await CanvasManager.shared.eval(sessionKey: sessionKey, javaScript: js)
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: resultJSON)
}
private func ensureA2UIHost() async throws {
if await self.isA2UIReady() {
return
}
guard let a2uiUrl = await self.canvasHostedSurfaceResolver.resolveA2UIURL() else {
throw NSError(domain: "Canvas", code: 30, userInfo: [
NSLocalizedDescriptionKey: "A2UI_HOST_NOT_CONFIGURED: gateway did not advertise canvas host",
])
}
let sessionKey = self.mainSessionKey
_ = try await MainActor.run {
try CanvasManager.shared.prepare(
sessionKey: sessionKey,
target: a2uiUrl,
trustedA2UIActions: true)
}
if await self.isA2UIReady(poll: true) {
return
}
if let refreshedUrl = await self.canvasHostedSurfaceResolver.resolveA2UIURL(forceRefresh: true) {
_ = try await MainActor.run {
try CanvasManager.shared.prepare(
sessionKey: sessionKey,
target: refreshedUrl,
trustedA2UIActions: true)
}
if await self.isA2UIReady(poll: true) {
return
}
}
throw NSError(domain: "Canvas", code: 31, userInfo: [
NSLocalizedDescriptionKey: "A2UI_HOST_UNAVAILABLE: A2UI host not reachable",
])
}
private func isA2UIReady(poll: Bool = false) async -> Bool {
let deadline = poll ? Date().addingTimeInterval(6.0) : Date()
while true {
do {
let sessionKey = self.mainSessionKey
let ready = try await CanvasManager.shared.eval(sessionKey: sessionKey, javaScript: """
(() => {
const host = globalThis.openclawA2UI;
return String(Boolean(host));
})()
""")
let trimmed = ready.trimmingCharacters(in: .whitespacesAndNewlines)
if trimmed == "true" {
return true
}
} catch {
// Ignore transient eval failures while the page is loading.
}
guard poll, Date() < deadline else { return false }
try? await Task.sleep(nanoseconds: 120_000_000)
}
}
}
// MARK: - Native system notifications
extension MacNodeRuntime {
@@ -1048,59 +912,4 @@ extension MacNodeRuntime {
ok: false,
error: OpenClawNodeError(code: code, message: message))
}
private static func encodeCanvasSnapshot(
image: NSImage,
format: OpenClawCanvasSnapshotFormat,
maxWidth: Int?,
quality: Double) throws -> Data
{
let source = Self.scaleImage(image, maxWidth: maxWidth) ?? image
guard let tiff = source.tiffRepresentation,
let rep = NSBitmapImageRep(data: tiff)
else {
throw NSError(domain: "Canvas", code: 22, userInfo: [
NSLocalizedDescriptionKey: "snapshot encode failed",
])
}
switch format {
case .png:
guard let data = rep.representation(using: .png, properties: [:]) else {
throw NSError(domain: "Canvas", code: 23, userInfo: [
NSLocalizedDescriptionKey: "png encode failed",
])
}
return data
case .jpeg:
let clamped = min(1.0, max(0.05, quality))
guard let data = rep.representation(
using: .jpeg,
properties: [.compressionFactor: clamped])
else {
throw NSError(domain: "Canvas", code: 24, userInfo: [
NSLocalizedDescriptionKey: "jpeg encode failed",
])
}
return data
}
}
private static func scaleImage(_ image: NSImage, maxWidth: Int?) -> NSImage? {
guard let maxWidth, maxWidth > 0 else { return image }
let size = image.size
guard size.width > 0, size.width > CGFloat(maxWidth) else { return image }
let scale = CGFloat(maxWidth) / size.width
let target = NSSize(width: CGFloat(maxWidth), height: size.height * scale)
let out = NSImage(size: target)
out.lockFocus()
image.draw(
in: NSRect(origin: .zero, size: target),
from: NSRect(origin: .zero, size: size),
operation: .copy,
fraction: 1.0)
out.unlockFocus()
return out
}
}
apps/macos/Tests/OpenClawIPCTests/CanvasManagerVisibilityTests.swift
-64
View File
@@ -1,22 +1,9 @@
import Foundation
import Testing
@testable import OpenClaw
@Suite(.serialized)
@MainActor
struct CanvasManagerVisibilityTests {
@Test func `eval on fresh state creates a hidden surface`() async throws {
let manager = CanvasManager.shared
manager._testResetPanel()
defer { manager._testResetPanel() }
let result = try await manager.eval(sessionKey: "visibility-eval", javaScript: "1 + 1")
#expect(result == "2")
#expect(manager._testHasPanelController)
#expect(manager._testPanelWindowIsVisible == false)
}
@Test func `showDetailed presents the panel`() throws {
let manager = CanvasManager.shared
manager._testResetPanel()
@@ -26,55 +13,4 @@ struct CanvasManagerVisibilityTests {
#expect(manager._testPanelWindowIsVisible == true)
}
@Test func `content operations respect a user hide`() async throws {
let manager = CanvasManager.shared
manager._testResetPanel()
defer { manager._testResetPanel() }
_ = try manager.showDetailed(sessionKey: "visibility-hidden")
manager.hideAll()
try manager.prepare(sessionKey: "visibility-hidden", target: "/")
_ = try await manager.eval(sessionKey: "visibility-hidden", javaScript: "1 + 1")
#expect(manager._testPanelWindowIsVisible == false)
}
@Test func `snapshot while hidden throws without presenting`() async throws {
let manager = CanvasManager.shared
manager._testResetPanel()
defer { manager._testResetPanel() }
let output = FileManager.default.temporaryDirectory
.appendingPathComponent("openclaw-canvas-hidden-\(UUID().uuidString).png")
defer { try? FileManager.default.removeItem(at: output) }
try manager.prepare(sessionKey: "visibility-snapshot", target: "/")
do {
_ = try await manager.snapshot(sessionKey: "visibility-snapshot", outPath: output.path)
Issue.record("hidden snapshot should throw CANVAS_HIDDEN")
} catch {
#expect(error.localizedDescription.contains("CANVAS_HIDDEN"))
}
#expect(manager._testHasPanelController)
#expect(manager._testPanelWindowIsVisible == false)
#expect(FileManager.default.fileExists(atPath: output.path) == false)
}
@Test func `snapshot for another session leaves the visible panel alone`() async throws {
let manager = CanvasManager.shared
manager._testResetPanel()
defer { manager._testResetPanel() }
_ = try manager.showDetailed(sessionKey: "visibility-live")
#expect(manager._testPanelWindowIsVisible == true)
await #expect(throws: (any Error).self) {
try await manager.snapshot(sessionKey: "visibility-other", outPath: nil)
}
// Read-only snapshot must not close or switch the live panel.
#expect(manager._testPanelWindowIsVisible == true)
}
}
apps/macos/Tests/OpenClawIPCTests/CanvasWindowSmokeTests.swift
+14 -74
View File
@@ -7,7 +7,7 @@ import Testing
@Suite(.serialized)
@MainActor
struct CanvasWindowSmokeTests {
@Test func `panel controller shows and hides`() async throws {
@Test func `panel controller shows and hides`() throws {
let root = FileManager().temporaryDirectory
.appendingPathComponent("openclaw-canvas-test-\(UUID().uuidString)")
try FileManager().createDirectory(at: root, withIntermediateDirectories: true)
@@ -39,7 +39,6 @@ struct CanvasWindowSmokeTests {
#expect(controller._testIsFilePollingActive == false)
controller.updateFilePollingForCommittedNavigation(to: localURL)
#expect(controller._testIsFilePollingActive)
_ = try await controller.eval(javaScript: "1 + 1")
controller.windowDidMove(Notification(name: NSWindow.didMoveNotification))
controller.windowDidEndLiveResize(Notification(name: NSWindow.didEndLiveResizeNotification))
controller.hideCanvas()
@@ -65,97 +64,38 @@ struct CanvasWindowSmokeTests {
controller.close()
}
@Test func `A2UI auto navigation is idempotent for current host target`() throws {
let root = FileManager().temporaryDirectory
.appendingPathComponent("openclaw-canvas-test-\(UUID().uuidString)")
try FileManager().createDirectory(at: root, withIntermediateDirectories: true)
defer { try? FileManager().removeItem(at: root) }
let controller = try CanvasWindowController(
sessionKey: "main",
root: root,
presentation: .window)
defer { controller.close() }
let oldTarget = "http://127.0.0.1:18789/__openclaw__/a2ui/?platform=macos"
let currentTarget = "http://127.0.0.1:18790/__openclaw__/a2ui/?platform=macos"
let userTarget = "https://github.com/openclaw/openclaw"
#expect(controller.shouldAutoNavigateToA2UI(lastAutoTarget: nil, candidateTarget: currentTarget) == true)
controller.load(target: "/")
#expect(controller.shouldAutoNavigateToA2UI(lastAutoTarget: nil, candidateTarget: currentTarget) == true)
controller.load(target: currentTarget)
#expect(controller
.shouldAutoNavigateToA2UI(lastAutoTarget: currentTarget, candidateTarget: currentTarget) == false)
controller.load(target: oldTarget)
#expect(controller.shouldAutoNavigateToA2UI(lastAutoTarget: oldTarget, candidateTarget: currentTarget) == true)
controller.load(target: userTarget)
#expect(controller
.shouldAutoNavigateToA2UI(lastAutoTarget: currentTarget, candidateTarget: currentTarget) == false)
}
@Test func `hosted Canvas URL resolver keeps capability scope and only trusts A2UI`() throws {
@Test func `hosted Canvas URL resolver keeps capability scope`() throws {
let surface = "https://gateway.example/root/__openclaw__/cap/token%20value"
let canvas = try #require(CanvasHostedURLResolver.resolve(
surfaceURL: surface,
target: "/__openclaw__/canvas/demo%20page.html?mode=proof#result"))
#expect(canvas.url.absoluteString ==
#expect(canvas.absoluteString ==
"https://gateway.example/root/__openclaw__/cap/token%20value/__openclaw__/canvas/demo%20page.html?mode=proof#result")
#expect(canvas.allowsA2UIActions == false)
let a2ui = try #require(CanvasHostedURLResolver.resolve(
surfaceURL: surface,
target: "/__openclaw__/a2ui/?platform=macos"))
#expect(a2ui.url.absoluteString ==
"https://gateway.example/root/__openclaw__/cap/token%20value/__openclaw__/a2ui/?platform=macos")
#expect(a2ui.allowsA2UIActions)
#expect(CanvasHostedURLResolver.resolve(surfaceURL: surface, target: "/local.html") == nil)
#expect(CanvasHostedURLResolver.resolve(surfaceURL: surface, target: "https://example.com/") == nil)
#expect(CanvasHostedURLResolver.resolve(
surfaceURL: surface,
target: "/__openclaw__/a2ui/../canvas/") == nil)
target: "/__openclaw__/canvas/../other/") == nil)
#expect(CanvasHostedURLResolver.resolve(
surfaceURL: surface,
target: "/__openclaw__/a2ui/%252e%252e/canvas/") == nil)
target: "/__openclaw__/canvas/%252e%252e/other/") == nil)
#expect(CanvasHostedURLResolver.resolve(
surfaceURL: surface,
target: "/__openclaw__/a2ui/%25252525252e%25252525252e/canvas/") == nil)
target: "/__openclaw__/canvas/%25252525252e%25252525252e/other/") == nil)
#expect(CanvasHostedURLResolver.resolve(
surfaceURL: "https://gateway.example/not-capability-scoped",
target: "/__openclaw__/canvas/") == nil)
}
@Test func `A2UI action trust is exact and capability scoped`() throws {
let expected = try #require(URL(string:
"https://gateway.example/__openclaw__/cap/current-token/__openclaw__/a2ui/?platform=macos"))
let sameWithFragment = try #require(URL(string: expected.absoluteString + "#card"))
let staleCapability = try #require(URL(string:
"https://gateway.example/__openclaw__/cap/stale-token/__openclaw__/a2ui/?platform=macos"))
let changedQuery = try #require(URL(string:
"https://gateway.example/__openclaw__/cap/current-token/__openclaw__/a2ui/?platform=other"))
let canvasPage = try #require(URL(string:
"https://gateway.example/__openclaw__/cap/current-token/__openclaw__/canvas/"))
let traversingA2UI = try #require(URL(string:
"https://gateway.example/__openclaw__/cap/current-token/__openclaw__/a2ui/%2e%2e/canvas/"))
let localCanvas = try #require(URL(string: "openclaw-canvas://main/"))
@Test func `Canvas target resolver accepts only hosted paths and app-local URLs`() {
#expect(CanvasHostedURLResolver.isHostedTarget(
"/__openclaw__/canvas/documents/widget/index.html"))
#expect(CanvasHostedURLResolver.isAppLocalTarget("openclaw-canvas://main/widget/index.html"))
#expect(CanvasA2UIActionMessageHandler.isTrustedSourceURL(expected, expectedRemoteURL: expected))
#expect(CanvasA2UIActionMessageHandler.isTrustedSourceURL(sameWithFragment, expectedRemoteURL: expected))
#expect(!CanvasA2UIActionMessageHandler.isTrustedSourceURL(staleCapability, expectedRemoteURL: expected))
#expect(!CanvasA2UIActionMessageHandler.isTrustedSourceURL(changedQuery, expectedRemoteURL: expected))
#expect(!CanvasA2UIActionMessageHandler.isTrustedSourceURL(canvasPage, expectedRemoteURL: expected))
#expect(!CanvasHostedURLResolver.isCapabilityScopedA2UIURL(traversingA2UI))
#expect(CanvasA2UIActionMessageHandler.isTrustedSourceURL(localCanvas, expectedRemoteURL: nil))
let handler = CanvasA2UIActionMessageHandler(sessionKey: "main")
handler.setTrustedRemoteURL(expected)
#expect(handler.isTrustedSourceURL(expected))
handler.updateTrustForMainFrameNavigation(to: canvasPage)
#expect(!handler.isTrustedSourceURL(expected))
#expect(!CanvasHostedURLResolver.isAppLocalTarget("https://example.com/widget.html"))
#expect(!CanvasHostedURLResolver.isAppLocalTarget("file:///tmp/widget.html"))
#expect(!CanvasHostedURLResolver.isAppLocalTarget("openclaw-canvas:///widget.html"))
#expect(!CanvasHostedURLResolver.isAppLocalTarget("openclaw-canvas://main/%252e%252e/secret"))
}
}
apps/macos/Tests/OpenClawIPCTests/LowCoverageHelperTests.swift
+2 -5
View File
@@ -345,6 +345,7 @@ struct LowCoverageHelperTests {
let missing = try #require(CanvasScheme.makeURL(session: "missing", path: "/"))
let missingResponse = handler._testResponse(for: missing)
#expect(missingResponse.mime == "text/html")
#expect(String(data: missingResponse.data, encoding: .utf8)?.contains("Not Found") == true)
#expect(handler._testTextEncodingName(for: "text/html") == "utf-8")
#expect(handler._testTextEncodingName(for: "application/octet-stream") == nil)
@@ -376,15 +377,11 @@ struct LowCoverageHelperTests {
#expect(!body.contains("top-secret"))
}
@Test @MainActor func `canvas window helper functions`() throws {
@Test @MainActor func `canvas window helper functions`() {
let rect = NSRect(x: 10, y: 12, width: 400, height: 420)
let key = CanvasWindowController._testStoredFrameKey(sessionKey: "test")
let loaded = CanvasWindowController._testStoreAndLoadFrame(sessionKey: "test", frame: rect)
UserDefaults.standard.removeObject(forKey: key)
#expect(loaded?.size.width == rect.size.width)
let trusted = try #require(URL(string:
"http://127.0.0.1:18789/__openclaw__/cap/token/__openclaw__/a2ui/?platform=macos"))
#expect(CanvasA2UIActionMessageHandler.isTrustedSourceURL(trusted, expectedRemoteURL: trusted))
}
}
apps/macos/Tests/OpenClawIPCTests/MacNodeHostWorkerTests.swift
+23 -2
View File
@@ -245,8 +245,12 @@ struct MacNodeHostWorkerTests {
#expect(await worker.invokedCommands().isEmpty)
}
@Test(arguments: [OpenClawCanvasCommand.present.rawValue, "canvas.plugin.render"])
func `worker cannot bypass the canvas namespace consent gate`(command: String) async {
@Test(arguments: [
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
])
func `worker cannot bypass the canvas presenter consent gate`(command: String) async {
await TestIsolation.withUserDefaultsValues([canvasEnabledKey: false]) {
let worker = StubMacNodeHostWorker(commands: [command])
let runtime = MacNodeRuntime(nodeHostWorker: worker)
@@ -262,6 +266,23 @@ struct MacNodeHostWorkerTests {
}
}
@Test func `worker cannot claim commands in the retired canvas namespace`() async {
await TestIsolation.withUserDefaultsValues([canvasEnabledKey: true]) {
let command = "canvas.plugin.render"
let worker = StubMacNodeHostWorker(commands: [command])
let runtime = MacNodeRuntime(nodeHostWorker: worker)
let response = await runtime.handleInvoke(BridgeInvokeRequest(
id: "canvas-retired",
command: command))
#expect(!response.ok)
#expect(response.error?.code == .invalidRequest)
#expect(response.error?.message == "INVALID_REQUEST: unknown command")
#expect(await worker.invokedCommands().isEmpty)
}
}
@Test func `capability union preserves native order and adds worker commands once`() {
#expect(MacNodeModeCoordinator.mergingUnique(
["canvas", "screen", "system"],
apps/macos/Tests/OpenClawIPCTests/MacNodeModeCoordinatorTests.swift
+5 -1
View File
@@ -666,7 +666,11 @@ struct MacNodeModeCoordinatorTests {
#expect(!caps.contains(OpenClawCapability.browser.rawValue))
#expect(!commands.contains(OpenClawBrowserCommand.proxy.rawValue))
#expect(commands.contains(OpenClawCanvasCommand.present.rawValue))
#expect(commands.filter { $0.hasPrefix("canvas.") } == [
OpenClawCanvasCommand.present.rawValue,
OpenClawCanvasCommand.hide.rawValue,
OpenClawCanvasCommand.navigate.rawValue,
])
#expect(commands.contains(OpenClawSystemCommand.notify.rawValue))
#expect(!commands.contains(OpenClawFileSystemCommand.listDir.rawValue))
#expect(!commands.contains(OpenClawSystemCommand.run.rawValue))
apps/macos/Tests/OpenClawIPCTests/MacNodeRuntimeTests.swift
+51 -46
View File
@@ -281,7 +281,7 @@ struct MacNodeRuntimeTests {
@Test func `handle invoke rejects unknown command`() async {
let runtime = MacNodeRuntime()
let response = await self.invoke(runtime, "req-1", "unknown.command")
let response = await invoke(runtime, "req-1", "unknown.command")
#expect(response.ok == false)
}
@@ -293,7 +293,7 @@ struct MacNodeRuntimeTests {
#expect(paramsJSON == #"{"limit":7}"#)
return payload
})
let response = await self.invoke(
let response = await invoke(
runtime, "req-codex-threads", MacNodeCodexThreadCatalogContract.listCommand, #"{"limit":7}"#)
#expect(response.ok)
@@ -307,7 +307,7 @@ struct MacNodeRuntimeTests {
Issue.record("disabled Codex catalog request must not execute")
return #"{"sessions":[]}"#
})
let response = await self.invoke(
let response = await invoke(
runtime, "req-codex-disabled", MacNodeCodexThreadCatalogContract.listCommand)
#expect(!response.ok)
@@ -323,7 +323,7 @@ struct MacNodeRuntimeTests {
#expect(paramsJSON == #"{"threadId":"thread-1","limit":50}"#)
return payload
})
let response = await self.invoke(
let response = await invoke(
runtime, "req-codex-items", MacNodeCodexThreadCatalogContract.turnsCommand,
#"{"threadId":"thread-1","limit":50}"#)
@@ -345,9 +345,9 @@ struct MacNodeRuntimeTests {
return readPayload
})
let list = await self.invoke(
let list = await invoke(
runtime, "req-claude-list", MacNodeClaudeSessionCatalogContract.listCommand, #"{"limit":7}"#)
let read = await self.invoke(
let read = await invoke(
runtime, "req-claude-read", MacNodeClaudeSessionCatalogContract.readCommand,
#"{"threadId":"thread-1","limit":20}"#)
@@ -434,7 +434,7 @@ struct MacNodeRuntimeTests {
Issue.record("disabled Claude catalog request must not execute")
return #"{"sessions":[]}"#
})
let response = await self.invoke(
let response = await invoke(
runtime, "req-claude-disabled", MacNodeClaudeSessionCatalogContract.listCommand)
#expect(!response.ok)
@@ -442,23 +442,6 @@ struct MacNodeRuntimeTests {
#expect(response.error?.message == "UNAVAILABLE: Claude session catalog is disabled")
}
@Test func `A2UI host capability refresh uses injected node session refresher`() async {
let probe = CanvasRefreshProbe()
let resolver = MacNodeCanvasHostedSurfaceResolver(
currentSurfaceURL: { "http://127.0.0.1:18789/__openclaw__/cap/current-token" },
refreshSurfaceURL: { _ in await probe.refresh() })
let current = await resolver.resolveA2UIURL()
#expect(current ==
"http://127.0.0.1:18789/__openclaw__/cap/current-token/__openclaw__/a2ui/?platform=macos")
#expect(await probe.calls == 0)
let refreshed = await resolver.resolveA2UIURL(forceRefresh: true)
#expect(refreshed ==
"http://127.0.0.1:18789/__openclaw__/cap/refreshed-token/__openclaw__/a2ui/?platform=macos")
#expect(await probe.calls == 1)
}
@Test func `hosted Canvas commands refresh capability and preserve target components`() async throws {
let probe = CanvasRefreshProbe()
let resolver = MacNodeCanvasHostedSurfaceResolver(
@@ -467,9 +450,8 @@ struct MacNodeRuntimeTests {
let resolved = try await resolver.resolveTarget(
"/__openclaw__/canvas/demo%20page.html?mode=proof#result")
#expect(resolved?.url.absoluteString ==
#expect(resolved?.absoluteString ==
"http://127.0.0.1:18789/__openclaw__/cap/refreshed-token/__openclaw__/canvas/demo%20page.html?mode=proof#result")
#expect(resolved?.allowsA2UIActions == false)
#expect(await probe.calls == 1)
let external = try await resolver.resolveTarget("https://example.com/")
@@ -485,14 +467,37 @@ struct MacNodeRuntimeTests {
let resolved = try await resolver.resolveTarget("/__openclaw__/canvas/demo.html")
#expect(resolved?.url.absoluteString ==
#expect(resolved?.absoluteString ==
"http://127.0.0.1:18789/__openclaw__/cap/new-token/__openclaw__/canvas/demo.html")
}
@Test func `removed Canvas commands are unknown even when Canvas is disabled`() async {
await TestIsolation.withUserDefaultsValues([canvasEnabledKey: false]) {
let response = await self.invoke(MacNodeRuntime(), "req-canvas-eval", "canvas.eval")
#expect(response.error?.code == .invalidRequest)
#expect(response.error?.message == "INVALID_REQUEST: unknown command")
}
}
@Test func `Canvas commands reject arbitrary web targets`() async {
await TestIsolation.withUserDefaultsValues([canvasEnabledKey: true]) {
let response = await self.invoke(
MacNodeRuntime(),
"req-canvas-external",
OpenClawCanvasCommand.navigate.rawValue,
#"{"url":"https://example.com/widget.html"}"#)
#expect(response.error?.code == .invalidRequest)
#expect(response.error?.message ==
"INVALID_REQUEST: canvas target must be a hosted widget-document path or app-local Canvas URL")
}
}
@Test func `handle invoke rejects empty notification`() async throws {
let runtime = MacNodeRuntime()
let params = OpenClawSystemNotifyParams(title: "", body: "")
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-3", OpenClawSystemCommand.notify.rawValue, params: params)
#expect(response.ok == false)
}
@@ -535,7 +540,7 @@ struct MacNodeRuntimeTests {
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let params = MacNodeScreenRecordParams(durationMs: 250)
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-5", MacNodeScreenCommand.record.rawValue, params: params)
#expect(response.ok == true)
let payloadJSON = try #require(response.payloadJSON)
@@ -571,7 +576,7 @@ struct MacNodeRuntimeTests {
maxWidth: 800,
quality: 0.5,
format: .jpeg)
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-screen-snapshot", MacNodeScreenCommand.snapshot.rawValue, params: params)
#expect(response.ok == true)
let payloadJSON = try #require(response.payloadJSON)
@@ -601,7 +606,7 @@ struct MacNodeRuntimeTests {
let services = await MainActor.run { MainActorServicesProbe() }
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-screen-snapshot-invalid", MacNodeScreenCommand.snapshot.rawValue, #"{"screenIndex":"#)
#expect(response.ok == false)
@@ -615,7 +620,7 @@ struct MacNodeRuntimeTests {
let services = await MainActor.run { MainActorServicesProbe() }
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-screen-snapshot-defaults", MacNodeScreenCommand.snapshot.rawValue)
#expect(response.ok == true)
@@ -630,7 +635,7 @@ struct MacNodeRuntimeTests {
computerControlEnabled: { false })
let params = OpenClawComputerActParams(action: .leftClick, x: 5, y: 6, refWidth: 1280)
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-computer-disabled", OpenClawComputerCommand.act.rawValue, params: params)
#expect(response.ok == false)
@@ -647,7 +652,7 @@ struct MacNodeRuntimeTests {
computerControlEnabled: { true })
let params = OpenClawComputerActParams(action: .leftClick, x: 12, y: 34, refWidth: 1280)
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-computer-ok", OpenClawComputerCommand.act.rawValue, params: params)
#expect(response.ok == true)
@@ -714,7 +719,7 @@ struct MacNodeRuntimeTests {
makeMainActorServices: { unavailableServices },
computerControlEnabled: { true },
computerControlProvider: { .cua })
let unavailable = await self.invoke(
let unavailable = await invoke(
unavailableRuntime,
"cua-unavailable",
MacNodeScreenCommand.snapshot.rawValue)
@@ -744,11 +749,11 @@ struct MacNodeRuntimeTests {
let first = Task {
await self.invoke(runtime, "req-computer-single-flight-1", OpenClawComputerCommand.act.rawValue, json)
}
try #require(await self.waitForCount(1, counter: factoryCalls))
try #require(await waitForCount(1, counter: factoryCalls))
let second = Task {
await self.invoke(runtime, "req-computer-single-flight-2", OpenClawComputerCommand.act.rawValue, json)
}
try #require(await self.waitForCount(2, counter: admissionCalls))
try #require(await waitForCount(2, counter: admissionCalls))
// The actor barrier proves the second invoke reached its first suspension.
await runtime.updateMainSessionKey("single-flight-barrier")
@@ -777,7 +782,7 @@ struct MacNodeRuntimeTests {
let invoke = Task {
await self.invoke(runtime, "req-computer-release-during-init", OpenClawComputerCommand.act.rawValue, json)
}
try #require(await self.waitForCount(1, counter: factoryCalls))
try #require(await waitForCount(1, counter: factoryCalls))
await runtime.releaseHeldComputerInput()
factoryGate.open()
@@ -844,7 +849,7 @@ struct MacNodeRuntimeTests {
computerControlEnabled: { true })
let params = OpenClawComputerActParams(action: .leftClick, x: 1, y: 1, refWidth: 1280)
let response = try await self.invoke(
let response = try await invoke(
runtime, "req-computer-ax", OpenClawComputerCommand.act.rawValue, params: params)
#expect(response.ok == false)
@@ -858,7 +863,7 @@ struct MacNodeRuntimeTests {
makeMainActorServices: { services },
computerControlEnabled: { true })
let response = await self.invoke(
let response = await invoke(
runtime, "req-computer-bad", OpenClawComputerCommand.act.rawValue, #"{"action":"#)
#expect(response.ok == false)
@@ -879,7 +884,7 @@ struct MacNodeRuntimeTests {
}
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-screen-snapshot-error", MacNodeScreenCommand.snapshot.rawValue)
#expect(response.ok == false)
@@ -893,7 +898,7 @@ struct MacNodeRuntimeTests {
snapshotError: ScreenSnapshotService.ScreenSnapshotError.invalidScreenIndex(4))
}
let invalidIndexRuntime = MacNodeRuntime(makeMainActorServices: { invalidIndexServices })
let invalidIndexResponse = await self.invoke(
let invalidIndexResponse = await invoke(
invalidIndexRuntime, "req-screen-snapshot-bad-index", MacNodeScreenCommand.snapshot.rawValue)
#expect(invalidIndexResponse.ok == false)
@@ -904,7 +909,7 @@ struct MacNodeRuntimeTests {
MainActorServicesProbe(snapshotError: ScreenSnapshotService.ScreenSnapshotError.noDisplays)
}
let noDisplaysRuntime = MacNodeRuntime(makeMainActorServices: { noDisplaysServices })
let noDisplaysResponse = await self.invoke(
let noDisplaysResponse = await invoke(
noDisplaysRuntime, "req-screen-snapshot-no-displays", MacNodeScreenCommand.snapshot.rawValue)
#expect(noDisplaysResponse.ok == false)
@@ -926,7 +931,7 @@ struct MacNodeRuntimeTests {
}
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-screen-snapshot-too-large", MacNodeScreenCommand.snapshot.rawValue)
#expect(response.ok == false)
@@ -949,7 +954,7 @@ struct MacNodeRuntimeTests {
}
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-screen-snapshot-slash-heavy", MacNodeScreenCommand.snapshot.rawValue,
nodeId: "node-slash-heavy")
@@ -972,7 +977,7 @@ struct MacNodeRuntimeTests {
}
let runtime = MacNodeRuntime(makeMainActorServices: { services })
let response = await self.invoke(
let response = await invoke(
runtime, "req-fit", MacNodeScreenCommand.snapshot.rawValue, nodeId: "node-fit")
#expect(response.ok == true)
apps/shared/OpenClawKit/Sources/OpenClawKit/CanvasA2UIAction.swift
-104
View File
@@ -1,104 +0,0 @@
import Foundation
public enum OpenClawCanvasA2UIAction: Sendable {
public struct AgentMessageContext: Sendable {
public struct Session: Sendable {
public var key: String
public var surfaceId: String
public init(key: String, surfaceId: String) {
self.key = key
self.surfaceId = surfaceId
}
}
public struct Component: Sendable {
public var id: String
public var host: String
public var instanceId: String
public init(id: String, host: String, instanceId: String) {
self.id = id
self.host = host
self.instanceId = instanceId
}
}
public var actionName: String
public var session: Session
public var component: Component
public var contextJSON: String?
public init(actionName: String, session: Session, component: Component, contextJSON: String?) {
self.actionName = actionName
self.session = session
self.component = component
self.contextJSON = contextJSON
}
}
public static func extractActionName(_ userAction: [String: Any]) -> String? {
let keys = ["name", "action"]
for key in keys {
if let raw = userAction[key] as? String {
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
if !trimmed.isEmpty { return trimmed }
}
}
return nil
}
public static func sanitizeTagValue(_ value: String) -> String {
let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines)
let nonEmpty = trimmed.isEmpty ? "-" : trimmed
let normalized = nonEmpty.replacingOccurrences(of: " ", with: "_")
let allowed = CharacterSet(charactersIn: "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-.:")
let scalars = normalized.unicodeScalars.map { allowed.contains($0) ? Character($0) : "_" }
return String(scalars)
}
public static func compactJSON(_ obj: Any?) -> String? {
guard let obj else { return nil }
guard JSONSerialization.isValidJSONObject(obj) else { return nil }
guard let data = try? JSONSerialization.data(withJSONObject: obj, options: []),
let str = String(data: data, encoding: .utf8)
else { return nil }
return str
}
public static func formatAgentMessage(_ context: AgentMessageContext) -> String {
let ctxSuffix = context.contextJSON.flatMap { $0.isEmpty ? nil : " ctx=\($0)" } ?? ""
return [
"CANVAS_A2UI",
"action=\(self.sanitizeTagValue(context.actionName))",
"session=\(self.sanitizeTagValue(context.session.key))",
"surface=\(self.sanitizeTagValue(context.session.surfaceId))",
"component=\(self.sanitizeTagValue(context.component.id))",
"host=\(self.sanitizeTagValue(context.component.host))",
"instance=\(self.sanitizeTagValue(context.component.instanceId))\(ctxSuffix)",
"default=update_canvas",
].joined(separator: " ")
}
public static func jsDispatchA2UIActionStatus(actionId: String, ok: Bool, error: String?) -> String {
let payload: [String: Any] = [
"id": actionId,
"ok": ok,
"error": error ?? "",
]
let json: String = {
if let data = try? JSONSerialization.data(withJSONObject: payload, options: []),
let str = String(data: data, encoding: .utf8)
{
return str
}
return "{\"id\":\"\(actionId)\",\"ok\":\(ok ? "true" : "false"),\"error\":\"\"}"
}()
return """
(() => {
const detail = \(json);
window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail }));
})();
"""
}
}
apps/shared/OpenClawKit/Sources/OpenClawKit/CanvasA2UICommands.swift
-22
View File
@@ -1,22 +0,0 @@
import Foundation
public enum OpenClawCanvasA2UICommand: String, Codable, Sendable {
/// Render A2UI content on the device canvas.
case push = "canvas.a2ui.push"
/// Legacy alias for `push` when sending JSONL.
case pushJSONL = "canvas.a2ui.pushJSONL"
/// Reset the A2UI renderer state.
case reset = "canvas.a2ui.reset"
}
public struct OpenClawCanvasA2UIPushParams: Codable, Sendable, Equatable {
public var messages: [AnyCodable]
}
public struct OpenClawCanvasA2UIPushJSONLParams: Codable, Sendable, Equatable {
public var jsonl: String
public init(jsonl: String) {
self.jsonl = jsonl
}
}
apps/shared/OpenClawKit/Sources/OpenClawKit/CanvasA2UIJSONL.swift
-81
View File
@@ -1,81 +0,0 @@
import Foundation
public enum OpenClawCanvasA2UIJSONL: Sendable {
public struct ParsedItem: Sendable {
public var lineNumber: Int
public var message: AnyCodable
public init(lineNumber: Int, message: AnyCodable) {
self.lineNumber = lineNumber
self.message = message
}
}
public static func parse(_ text: String) throws -> [ParsedItem] {
var out: [ParsedItem] = []
var lineNumber = 0
for rawLine in text.split(omittingEmptySubsequences: false, whereSeparator: \.isNewline) {
lineNumber += 1
let line = String(rawLine).trimmingCharacters(in: .whitespacesAndNewlines)
if line.isEmpty { continue }
let data = Data(line.utf8)
let decoded = try JSONDecoder().decode(AnyCodable.self, from: data)
out.append(ParsedItem(lineNumber: lineNumber, message: decoded))
}
return out
}
public static func validateV0_8(_ items: [ParsedItem]) throws {
let allowed = Set([
"beginRendering",
"surfaceUpdate",
"dataModelUpdate",
"deleteSurface",
])
for item in items {
guard let dict = item.message.value as? [String: AnyCodable] else {
throw NSError(domain: "A2UI", code: 1, userInfo: [
NSLocalizedDescriptionKey: "A2UI JSONL line \(item.lineNumber): expected a JSON object",
])
}
if dict.keys.contains("createSurface") {
throw NSError(domain: "A2UI", code: 2, userInfo: [
NSLocalizedDescriptionKey: """
A2UI JSONL line \(item.lineNumber): looks like A2UI v0.9 (`createSurface`).
Canvas currently supports A2UI v0.8 server→client messages
(`beginRendering`, `surfaceUpdate`, `dataModelUpdate`, `deleteSurface`).
""",
])
}
let matched = dict.keys.filter { allowed.contains($0) }
if matched.count != 1 {
let found = dict.keys.sorted().joined(separator: ", ")
throw NSError(domain: "A2UI", code: 3, userInfo: [
NSLocalizedDescriptionKey: """
A2UI JSONL line \(item.lineNumber): expected exactly one of \(allowed.sorted()
.joined(separator: ", ")); found: \(found)
""",
])
}
}
}
public static func decodeMessagesFromJSONL(_ text: String) throws -> [AnyCodable] {
let items = try self.parse(text)
try self.validateV0_8(items)
return items.map(\.message)
}
public static func encodeMessagesJSONArray(_ messages: [AnyCodable]) throws -> String {
let data = try JSONEncoder().encode(messages)
guard let json = String(data: data, encoding: .utf8) else {
throw NSError(domain: "A2UI", code: 10, userInfo: [
NSLocalizedDescriptionKey: "Failed to encode messages payload as UTF-8",
])
}
return json
}
}
apps/shared/OpenClawKit/Sources/OpenClawKit/CanvasCommandParams.swift
-37
View File
@@ -24,40 +24,3 @@ public struct OpenClawCanvasPresentParams: Codable, Sendable, Equatable {
self.placement = placement
}
}
public struct OpenClawCanvasEvalParams: Codable, Sendable, Equatable {
public var javaScript: String
public init(javaScript: String) {
self.javaScript = javaScript
}
}
public enum OpenClawCanvasSnapshotFormat: String, Codable, Sendable {
case png
case jpeg
public init(from decoder: Decoder) throws {
let c = try decoder.singleValueContainer()
let raw = try c.decode(String.self).trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
switch raw {
case "png":
self = .png
case "jpeg", "jpg":
self = .jpeg
default:
throw DecodingError.dataCorruptedError(in: c, debugDescription: "Invalid snapshot format: \(raw)")
}
}
public func encode(to encoder: Encoder) throws {
var c = encoder.singleValueContainer()
try c.encode(self.rawValue)
}
}
public struct OpenClawCanvasSnapshotParams: Codable, Sendable, Equatable {
public var maxWidth: Int?
public var quality: Double?
public var format: OpenClawCanvasSnapshotFormat?
}
apps/shared/OpenClawKit/Sources/OpenClawKit/CanvasCommands.swift
-2
View File
@@ -4,6 +4,4 @@ public enum OpenClawCanvasCommand: String, Codable, Sendable {
case present = "canvas.present"
case hide = "canvas.hide"
case navigate = "canvas.navigate"
case evalJS = "canvas.eval"
case snapshot = "canvas.snapshot"
}
apps/shared/OpenClawKit/Sources/OpenClawKit/OpenClawKitResources.swift
-83
View File
@@ -1,83 +0,0 @@
import Foundation
public enum OpenClawKitResources {
/// Resource bundle for OpenClawKit.
///
/// Locates the SwiftPM-generated resource bundle, checking multiple locations:
/// 1. Inside Bundle.main (packaged apps)
/// 2. Bundle.module (SwiftPM development/tests)
/// 3. Falls back to Bundle.main if not found (resource lookups will return nil)
///
/// This avoids a fatal crash when Bundle.module can't locate its resources
/// in packaged .app bundles where the resource bundle path differs from
/// SwiftPM's expectations.
public static let bundle: Bundle = locateBundle()
private static let bundleName = "OpenClawKit_OpenClawKit"
private static func locateBundle() -> Bundle {
// 1. Check inside Bundle.main (packaged apps copy resources here)
if let mainResourceURL = Bundle.main.resourceURL {
let bundleURL = mainResourceURL.appendingPathComponent("\(self.bundleName).bundle")
if let bundle = Bundle(url: bundleURL) {
return bundle
}
}
// 2. Check Bundle.main directly for embedded resources
if Bundle.main.url(forResource: "tool-display", withExtension: "json") != nil {
return Bundle.main
}
// 3. Try Bundle.module (works in SwiftPM development/tests)
// Wrap in a function to defer the fatalError until actually called
if let moduleBundle = loadModuleBundleSafely() {
return moduleBundle
}
// 4. Fallback: return Bundle.main (resource lookups will return nil gracefully)
return Bundle.main
}
private static func loadModuleBundleSafely() -> Bundle? {
// Bundle.module is generated by SwiftPM and will fatalError if not found.
// We check likely locations manually to avoid the crash.
let candidates: [URL?] = [
Bundle.main.resourceURL,
Bundle.main.bundleURL,
Bundle(for: BundleLocator.self).resourceURL,
Bundle(for: BundleLocator.self).bundleURL,
]
for candidate in candidates {
guard let baseURL = candidate else { continue }
// SwiftPM often places the resource bundle next to (or near) the test runner bundle,
// not inside it. Walk up a few levels and check common container paths.
var roots: [URL] = []
roots.append(baseURL)
roots.append(baseURL.appendingPathComponent("Resources"))
roots.append(baseURL.appendingPathComponent("Contents/Resources"))
var current = baseURL
for _ in 0..<5 {
current = current.deletingLastPathComponent()
roots.append(current)
roots.append(current.appendingPathComponent("Resources"))
roots.append(current.appendingPathComponent("Contents/Resources"))
}
for root in roots {
let bundleURL = root.appendingPathComponent("\(self.bundleName).bundle")
if let bundle = Bundle(url: bundleURL) {
return bundle
}
}
}
return nil
}
}
/// Helper class for bundle lookup via Bundle(for:)
private final class BundleLocator {}
apps/shared/OpenClawKit/Sources/OpenClawKit/Resources/CanvasScaffold/scaffold.html
-672
View File
@@ -1,672 +0,0 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
<title>OpenClaw</title>
<script>
(() => {
try {
const params = new URLSearchParams(window.location.search);
const platform = (params.get('platform') || '').trim().toLowerCase();
if (platform) {
document.documentElement.dataset.platform = platform;
return;
}
if (/android/i.test(navigator.userAgent || '')) {
document.documentElement.dataset.platform = 'android';
} else {
document.documentElement.dataset.platform = 'ios';
}
} catch (_) {}
})();
</script>
<style>
:root {
color-scheme: dark;
--bg: #08080a;
--panel: rgba(18, 19, 23, 0.76);
--panel-strong: rgba(28, 30, 35, 0.88);
--line: rgba(255, 255, 255, 0.11);
--line-strong: rgba(255, 255, 255, 0.2);
--text: rgba(250, 250, 250, 0.96);
--muted: rgba(224, 224, 226, 0.72);
--soft: rgba(224, 224, 226, 0.5);
--accent: #c6312a;
--accent-strong: #ef3e52;
--accent-warm: #f59e0b;
--state: #7d8ca3;
--safe-top: env(safe-area-inset-top, 0px);
--safe-bottom: env(safe-area-inset-bottom, 0px);
}
html, body {
height: 100%;
margin: 0;
}
body {
font-family: -apple-system, BlinkMacSystemFont, "SF Pro Display", "SF Pro Text", system-ui, sans-serif;
background:
linear-gradient(135deg, rgba(198, 49, 42, 0.2), rgba(198, 49, 42, 0) 34%),
linear-gradient(180deg, #14161a 0%, #08080a 42%, #050506 100%);
color: var(--text);
overflow: hidden;
}
body::before {
content: "";
position: fixed;
inset: -10%;
pointer-events: none;
background:
repeating-linear-gradient(
90deg,
rgba(255, 255, 255, 0.018) 0,
rgba(255, 255, 255, 0.018) 1px,
transparent 1px,
transparent 44px
),
repeating-linear-gradient(
0deg,
rgba(255, 255, 255, 0.018) 0,
rgba(255, 255, 255, 0.018) 1px,
transparent 1px,
transparent 44px
);
opacity: 0.5;
}
body[data-state="connected"] { --state: #61d58b; }
body[data-state="connecting"] { --state: #ffd05f; }
body[data-state="error"] { --state: #ff6d6d; }
body[data-state="offline"] { --state: #95a3b9; }
canvas {
position: fixed;
inset: 0;
width: 100vw;
height: 100vh;
display: block;
z-index: 1;
}
#openclaw-home {
position: fixed;
inset: 0;
z-index: 2;
display: flex;
align-items: flex-start;
justify-content: center;
padding: calc(var(--safe-top) + 18px) 16px calc(var(--safe-bottom) + 18px);
box-sizing: border-box;
overflow-y: auto;
overflow-x: hidden;
-webkit-overflow-scrolling: touch;
}
.shell {
width: min(100%, 760px);
display: flex;
flex-direction: column;
justify-content: flex-start;
gap: 16px;
min-height: 100%;
box-sizing: border-box;
}
.hero {
position: relative;
overflow: hidden;
border-radius: 24px;
background: linear-gradient(180deg, rgba(31, 33, 38, 0.86), rgba(13, 14, 17, 0.94));
border: 1px solid var(--line);
box-shadow: 0 28px 90px rgba(0, 0, 0, 0.42);
padding: 22px 22px 18px;
}
.hero::before {
content: "";
position: absolute;
inset: 0 auto 0 0;
width: 3px;
background: linear-gradient(180deg, var(--accent-strong), rgba(198, 49, 42, 0.08));
pointer-events: none;
}
.eyebrow {
display: inline-flex;
align-items: center;
gap: 9px;
padding: 8px 12px;
border-radius: 999px;
background: rgba(255, 255, 255, 0.04);
border: 1px solid rgba(255, 255, 255, 0.08);
color: var(--muted);
font-size: 12px;
font-weight: 700;
letter-spacing: 0.06em;
text-transform: uppercase;
max-width: 100%;
box-sizing: border-box;
}
.eyebrow-dot {
flex: 0 0 auto;
width: 9px;
height: 9px;
border-radius: 999px;
background: var(--state);
box-shadow: 0 0 18px color-mix(in srgb, var(--state) 68%, transparent);
}
#openclaw-home-eyebrow {
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.hero h1 {
margin: 18px 0 0;
font-size: clamp(34px, 7vw, 54px);
line-height: 1;
letter-spacing: -0.03em;
}
.hero p {
margin: 14px 0 0;
font-size: 16px;
line-height: 1.5;
color: var(--muted);
max-width: 32rem;
}
.hero-grid {
display: grid;
grid-template-columns: 1.2fr 1fr;
gap: 12px;
margin-top: 22px;
}
.meta-card,
.agent-card {
border-radius: 22px;
background: var(--panel);
border: 1px solid var(--line);
backdrop-filter: blur(18px);
-webkit-backdrop-filter: blur(18px);
}
.meta-card {
padding: 16px 16px 15px;
}
.meta-label {
font-size: 11px;
letter-spacing: 0.08em;
text-transform: uppercase;
color: var(--soft);
}
.meta-value {
margin-top: 8px;
font-size: 24px;
font-weight: 700;
letter-spacing: -0.03em;
overflow-wrap: anywhere;
}
.meta-subtitle {
margin-top: 6px;
color: var(--muted);
font-size: 13px;
line-height: 1.4;
}
.agent-focus {
display: flex;
align-items: flex-start;
gap: 14px;
margin-top: 8px;
}
.agent-badge {
width: 56px;
height: 56px;
border-radius: 18px;
display: inline-flex;
align-items: center;
justify-content: center;
background:
linear-gradient(135deg, rgba(198, 49, 42, 0.28), rgba(239, 62, 82, 0.1)),
rgba(255, 255, 255, 0.04);
border: 1px solid rgba(255, 255, 255, 0.12);
font-size: 24px;
font-weight: 700;
}
.agent-focus .name {
font-size: 22px;
font-weight: 700;
letter-spacing: -0.03em;
overflow-wrap: anywhere;
}
.agent-focus .caption {
margin-top: 4px;
font-size: 13px;
color: var(--muted);
}
.section {
padding: 16px 16px 14px;
}
.section-header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 12px;
}
.section-title {
font-size: 14px;
font-weight: 700;
color: var(--muted);
}
.section-count {
font-size: 12px;
font-weight: 700;
color: var(--soft);
}
.agent-grid {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 10px;
}
.agent-card {
padding: 13px 13px 12px;
}
.agent-card.active {
background: var(--panel-strong);
border-color: var(--line-strong);
box-shadow: inset 0 0 0 1px rgba(198, 49, 42, 0.18);
}
.agent-row {
display: flex;
align-items: center;
gap: 10px;
}
.agent-row .badge {
width: 38px;
height: 38px;
border-radius: 14px;
display: inline-flex;
align-items: center;
justify-content: center;
background: rgba(255, 255, 255, 0.05);
border: 1px solid rgba(255, 255, 255, 0.08);
font-size: 16px;
font-weight: 700;
}
.agent-row .name {
font-size: 15px;
font-weight: 700;
line-height: 1.2;
overflow-wrap: anywhere;
}
.agent-row .caption {
margin-top: 3px;
font-size: 12px;
color: var(--muted);
}
.empty-state {
padding: 18px;
border-radius: 18px;
background: rgba(255, 255, 255, 0.03);
border: 1px dashed rgba(255, 255, 255, 0.12);
color: var(--muted);
font-size: 14px;
line-height: 1.45;
}
.footer-note {
margin-top: 12px;
color: var(--soft);
font-size: 12px;
line-height: 1.45;
}
#openclaw-status {
position: fixed;
inset: 0;
display: none;
align-items: center;
justify-content: flex-start;
flex-direction: column;
padding-top: calc(var(--safe-top) + 18px);
pointer-events: none;
z-index: 3;
}
#openclaw-status .card {
text-align: center;
padding: 16px 18px;
border-radius: 14px;
background: rgba(18, 18, 22, 0.46);
border: 1px solid rgba(255, 255, 255, 0.08);
box-shadow: 0 18px 60px rgba(0, 0, 0, 0.55);
-webkit-backdrop-filter: blur(14px);
backdrop-filter: blur(14px);
}
#openclaw-status .title {
font: 600 20px -apple-system, BlinkMacSystemFont, "SF Pro Display", "SF Pro Text", system-ui, sans-serif;
color: rgba(255, 255, 255, 0.92);
}
#openclaw-status .subtitle {
margin-top: 6px;
font: 500 12px -apple-system, BlinkMacSystemFont, "SF Pro Text", system-ui, sans-serif;
color: rgba(255, 255, 255, 0.58);
}
@media (max-width: 640px) {
#openclaw-home {
padding-left: 12px;
padding-right: 12px;
}
.hero {
border-radius: 24px;
padding: 18px 16px 16px;
}
.hero-grid,
.agent-grid {
grid-template-columns: 1fr;
}
.hero h1 {
font-size: 34px;
}
}
@media (max-height: 760px) {
#openclaw-home {
padding-top: calc(var(--safe-top) + 14px);
padding-bottom: calc(var(--safe-bottom) + 12px);
}
.shell {
gap: 12px;
}
.hero {
border-radius: 24px;
padding: 16px 15px 15px;
}
.hero h1 {
margin-top: 14px;
font-size: clamp(28px, 8vw, 38px);
}
.hero p {
margin-top: 10px;
font-size: 15px;
line-height: 1.42;
}
.hero-grid {
margin-top: 18px;
}
.meta-card {
padding: 14px 14px 13px;
}
.meta-value {
font-size: 22px;
}
.agent-badge {
width: 50px;
height: 50px;
border-radius: 16px;
font-size: 22px;
}
.agent-focus .name {
font-size: 20px;
}
.section {
padding: 14px 14px 12px;
}
.section-header {
margin-bottom: 10px;
}
}
@media (prefers-reduced-motion: reduce) {
body::before,
body::after {
animation: none !important;
}
}
</style>
</head>
<body data-state="offline">
<canvas id="openclaw-canvas"></canvas>
<div id="openclaw-home">
<div class="shell">
<div class="hero">
<div class="eyebrow">
<span class="eyebrow-dot"></span>
<span id="openclaw-home-eyebrow">Welcome to OpenClaw</span>
</div>
<h1 id="openclaw-home-title">Your phone stays quiet until it is needed</h1>
<p id="openclaw-home-subtitle">
Pair this device to your gateway to wake it only for real work, keep a live agent overview handy, and avoid battery-draining background loops.
</p>
<div class="hero-grid">
<div class="meta-card">
<div class="meta-label">Gateway</div>
<div class="meta-value" id="openclaw-home-gateway">Gateway</div>
<div class="meta-subtitle" id="openclaw-home-gateway-caption">Connect to load your agents</div>
</div>
<div class="meta-card">
<div class="meta-label">Active Agent</div>
<div class="agent-focus">
<div class="agent-badge" id="openclaw-home-active-badge">OC</div>
<div>
<div class="name" id="openclaw-home-active-name">Main</div>
<div class="caption" id="openclaw-home-active-caption">Connect to load your agents</div>
</div>
</div>
</div>
</div>
</div>
<div class="meta-card section">
<div class="section-header">
<div class="section-title">Live agents</div>
<div class="section-count" id="openclaw-home-agent-count">0 agents</div>
</div>
<div class="agent-grid" id="openclaw-home-agent-grid"></div>
<div class="footer-note" id="openclaw-home-footer">
When connected, the gateway can wake the phone with a silent push instead of holding an always-on session.
</div>
</div>
</div>
</div>
<div id="openclaw-status">
<div class="card">
<div class="title" id="openclaw-status-title">Ready</div>
<div class="subtitle" id="openclaw-status-subtitle">Waiting for agent</div>
</div>
</div>
<script>
(() => {
const canvas = document.getElementById('openclaw-canvas');
const ctx = canvas.getContext('2d');
const statusEl = document.getElementById('openclaw-status');
const titleEl = document.getElementById('openclaw-status-title');
const subtitleEl = document.getElementById('openclaw-status-subtitle');
const home = {
root: document.getElementById('openclaw-home'),
eyebrow: document.getElementById('openclaw-home-eyebrow'),
title: document.getElementById('openclaw-home-title'),
subtitle: document.getElementById('openclaw-home-subtitle'),
gateway: document.getElementById('openclaw-home-gateway'),
gatewayCaption: document.getElementById('openclaw-home-gateway-caption'),
activeBadge: document.getElementById('openclaw-home-active-badge'),
activeName: document.getElementById('openclaw-home-active-name'),
activeCaption: document.getElementById('openclaw-home-active-caption'),
agentCount: document.getElementById('openclaw-home-agent-count'),
agentGrid: document.getElementById('openclaw-home-agent-grid'),
footer: document.getElementById('openclaw-home-footer')
};
const debugStatusEnabledByQuery = (() => {
try {
const params = new URLSearchParams(window.location.search);
const raw = params.get('debugStatus') ?? params.get('debug');
if (!raw) return false;
const normalized = String(raw).trim().toLowerCase();
return normalized === '1' || normalized === 'true' || normalized === 'yes';
} catch (_) {
return false;
}
})();
let debugStatusEnabled = debugStatusEnabledByQuery;
function resize() {
const dpr = window.devicePixelRatio || 1;
const width = Math.max(1, Math.floor(window.innerWidth * dpr));
const height = Math.max(1, Math.floor(window.innerHeight * dpr));
canvas.width = width;
canvas.height = height;
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
}
function setDebugStatusEnabled(enabled) {
debugStatusEnabled = !!enabled;
if (!debugStatusEnabled) {
statusEl.style.display = 'none';
}
}
function setStatus(title, subtitle) {
if (!debugStatusEnabled) return;
if (!title && !subtitle) {
statusEl.style.display = 'none';
return;
}
statusEl.style.display = 'flex';
if (typeof title === 'string') titleEl.textContent = title;
if (typeof subtitle === 'string') subtitleEl.textContent = subtitle;
}
function clearChildren(node) {
while (node.firstChild) node.removeChild(node.firstChild);
}
function createAgentCard(agent) {
const card = document.createElement('div');
card.className = `agent-card${agent.isActive ? ' active' : ''}`;
const row = document.createElement('div');
row.className = 'agent-row';
const badge = document.createElement('div');
badge.className = 'badge';
badge.textContent = agent.badge || 'OC';
const text = document.createElement('div');
const name = document.createElement('div');
name.className = 'name';
name.textContent = agent.name || agent.id || 'Agent';
const caption = document.createElement('div');
caption.className = 'caption';
caption.textContent = agent.caption || 'Ready';
text.appendChild(name);
text.appendChild(caption);
row.appendChild(badge);
row.appendChild(text);
card.appendChild(row);
return card;
}
function renderHome(state) {
if (!state || typeof state !== 'object') return;
document.body.dataset.state = state.gatewayState || 'offline';
home.root.style.display = 'flex';
home.eyebrow.textContent = state.eyebrow || 'Welcome to OpenClaw';
home.title.textContent = state.title || 'OpenClaw';
home.subtitle.textContent = state.subtitle || '';
home.gateway.textContent = state.gatewayLabel || 'Gateway';
home.gatewayCaption.textContent = state.gatewayState === 'connected'
? `${state.agentCount || 0} agent${state.agentCount === 1 ? '' : 's'} available`
: (state.activeAgentCaption || 'Connect to load your agents');
home.activeBadge.textContent = state.activeAgentBadge || 'OC';
home.activeName.textContent = state.activeAgentName || 'Main';
home.activeCaption.textContent = state.activeAgentCaption || '';
home.agentCount.textContent = `${state.agentCount || 0} agent${state.agentCount === 1 ? '' : 's'}`;
home.footer.textContent = state.footer || '';
clearChildren(home.agentGrid);
const agents = Array.isArray(state.agents) ? state.agents : [];
if (!agents.length) {
const empty = document.createElement('div');
empty.className = 'empty-state';
empty.textContent = state.gatewayState === 'connected'
? 'Your gateway is online. Agents will appear here as soon as the current scope reports them.'
: 'Connect this phone to your gateway and the live agent overview will appear here.';
home.agentGrid.appendChild(empty);
return;
}
agents.forEach((agent) => {
home.agentGrid.appendChild(createAgentCard(agent));
});
}
window.addEventListener('resize', resize);
resize();
if (!debugStatusEnabled) {
statusEl.style.display = 'none';
}
window.__openclaw = {
canvas,
ctx,
setDebugStatusEnabled,
setStatus,
renderHome
};
})();
</script>
</body>
</html>
apps/shared/OpenClawKit/Sources/OpenClawKit/Resources/tool-display.json
-31
View File
@@ -262,37 +262,6 @@
"node",
"nodeId"
]
},
"eval": {
"label": "eval",
"detailKeys": [
"javaScript",
"node",
"nodeId"
]
},
"snapshot": {
"label": "snapshot",
"detailKeys": [
"format",
"node",
"nodeId"
]
},
"a2ui_push": {
"label": "A2UI push",
"detailKeys": [
"jsonlPath",
"node",
"nodeId"
]
},
"a2ui_reset": {
"label": "A2UI reset",
"detailKeys": [
"node",
"nodeId"
]
}
}
},
apps/shared/OpenClawKit/Sources/OpenClawKit/ToolDisplay.swift
+57 -1
View File
@@ -17,6 +17,9 @@ public struct ToolDisplaySummary: Sendable, Equatable {
}
public enum ToolDisplayRegistry {
private static let resourceBundleName = "OpenClawKit_OpenClawKit"
private static let resourceBundle = locateResourceBundle()
private struct ToolDisplayActionSpec: Decodable {
let label: String?
let detailKeys: [String]?
@@ -83,7 +86,7 @@ public enum ToolDisplayRegistry {
}
private static func loadConfig() -> ToolDisplayConfig {
guard let url = OpenClawKitResources.bundle.url(forResource: "tool-display", withExtension: "json") else {
guard let url = self.resourceBundle.url(forResource: "tool-display", withExtension: "json") else {
return self.defaultConfig()
}
do {
@@ -94,6 +97,57 @@ public enum ToolDisplayRegistry {
}
}
private static func locateResourceBundle() -> Bundle {
if let mainResourceURL = Bundle.main.resourceURL,
let bundle = Bundle(
url: mainResourceURL.appendingPathComponent("\(self.resourceBundleName).bundle"))
{
return bundle
}
if Bundle.main.url(forResource: "tool-display", withExtension: "json") != nil {
return Bundle.main
}
return self.loadModuleBundleSafely() ?? Bundle.main
}
private static func loadModuleBundleSafely() -> Bundle? {
let candidates: [URL?] = [
Bundle.main.resourceURL,
Bundle.main.bundleURL,
Bundle(for: ToolDisplayBundleLocator.self).resourceURL,
Bundle(for: ToolDisplayBundleLocator.self).bundleURL,
]
for candidate in candidates {
guard let baseURL = candidate else { continue }
var roots = [
baseURL,
baseURL.appendingPathComponent("Resources"),
baseURL.appendingPathComponent("Contents/Resources"),
]
var current = baseURL
for _ in 0..<5 {
current = current.deletingLastPathComponent()
roots.append(current)
roots.append(current.appendingPathComponent("Resources"))
roots.append(current.appendingPathComponent("Contents/Resources"))
}
for root in roots {
if let bundle = Bundle(
url: root.appendingPathComponent("\(self.resourceBundleName).bundle"))
{
return bundle
}
}
}
return nil
}
private static func defaultConfig() -> ToolDisplayConfig {
ToolDisplayConfig(
version: 1,
@@ -219,3 +273,5 @@ public enum ToolDisplayRegistry {
return value.replacingOccurrences(of: home, with: "~")
}
}
private final class ToolDisplayBundleLocator {}
apps/shared/OpenClawKit/Tests/OpenClawKitTests/CanvasA2UIActionTests.swift
-36
View File
@@ -1,36 +0,0 @@
import OpenClawKit
import Foundation
import Testing
@Suite struct CanvasA2UIActionTests {
@Test func sanitizeTagValueIsStable() {
#expect(OpenClawCanvasA2UIAction.sanitizeTagValue("Hello World!") == "Hello_World_")
#expect(OpenClawCanvasA2UIAction.sanitizeTagValue(" ") == "-")
#expect(OpenClawCanvasA2UIAction.sanitizeTagValue("macOS 26.2") == "macOS_26.2")
}
@Test func extractActionNameAcceptsNameOrAction() {
#expect(OpenClawCanvasA2UIAction.extractActionName(["name": "Hello"]) == "Hello")
#expect(OpenClawCanvasA2UIAction.extractActionName(["action": "Wave"]) == "Wave")
#expect(OpenClawCanvasA2UIAction.extractActionName(["name": " ", "action": "Fallback"]) == "Fallback")
#expect(OpenClawCanvasA2UIAction.extractActionName(["action": " "]) == nil)
}
@Test func formatAgentMessageIsTokenEfficientAndUnambiguous() {
let messageContext = OpenClawCanvasA2UIAction.AgentMessageContext(
actionName: "Get Weather",
session: .init(key: "main", surfaceId: "main"),
component: .init(id: "btnWeather", host: "Peter’s iPad", instanceId: "ipad16,6"),
contextJSON: "{\"city\":\"Vienna\"}")
let msg = OpenClawCanvasA2UIAction.formatAgentMessage(messageContext)
#expect(msg.contains("CANVAS_A2UI "))
#expect(msg.contains("action=Get_Weather"))
#expect(msg.contains("session=main"))
#expect(msg.contains("surface=main"))
#expect(msg.contains("component=btnWeather"))
#expect(msg.contains("host=Peter_s_iPad"))
#expect(msg.contains("instance=ipad16_6 ctx={\"city\":\"Vienna\"}"))
#expect(msg.hasSuffix(" default=update_canvas"))
}
}
apps/shared/OpenClawKit/Tests/OpenClawKitTests/CanvasA2UITests.swift
-42
View File
@@ -1,42 +0,0 @@
import OpenClawKit
import Testing
@Suite struct CanvasA2UITests {
@Test func commandStringsAreStable() {
#expect(OpenClawCanvasA2UICommand.push.rawValue == "canvas.a2ui.push")
#expect(OpenClawCanvasA2UICommand.pushJSONL.rawValue == "canvas.a2ui.pushJSONL")
#expect(OpenClawCanvasA2UICommand.reset.rawValue == "canvas.a2ui.reset")
}
@Test func jsonlDecodesAndValidatesV0_8() throws {
let jsonl = """
{"beginRendering":{"surfaceId":"main","timestamp":1}}
{"surfaceUpdate":{"surfaceId":"main","ops":[]}}
{"dataModelUpdate":{"dataModel":{"title":"Hello"}}}
{"deleteSurface":{"surfaceId":"main"}}
"""
let messages = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(jsonl)
#expect(messages.count == 4)
}
@Test func jsonlRejectsV0_9CreateSurface() {
let jsonl = """
{"createSurface":{"surfaceId":"main"}}
"""
#expect(throws: Error.self) {
_ = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(jsonl)
}
}
@Test func jsonlRejectsUnknownShape() {
let jsonl = """
{"wat":{"nope":1}}
"""
#expect(throws: Error.self) {
_ = try OpenClawCanvasA2UIJSONL.decodeMessagesFromJSONL(jsonl)
}
}
}
apps/shared/OpenClawKit/Tests/OpenClawKitTests/CanvasCommandsTests.swift
+29
View File
@@ -0,0 +1,29 @@
import Foundation
import OpenClawKit
import Testing
struct CanvasCommandsTests {
@Test func `presenter command strings are stable`() {
#expect(OpenClawCanvasCommand.present.rawValue == "canvas.present")
#expect(OpenClawCanvasCommand.hide.rawValue == "canvas.hide")
#expect(OpenClawCanvasCommand.navigate.rawValue == "canvas.navigate")
}
@Test func `presenter params decode shipped wire shapes`() throws {
let presentData = try #require(
"""
{"url":"openclaw-canvas://widget.html","placement":{"x":1,"y":2,"width":3,"height":4}}
""".data(using: .utf8))
let present = try JSONDecoder().decode(OpenClawCanvasPresentParams.self, from: presentData)
#expect(present.url == "openclaw-canvas://widget.html")
#expect(present.placement?.x == 1)
#expect(present.placement?.y == 2)
#expect(present.placement?.width == 3)
#expect(present.placement?.height == 4)
let navigateData = try #require(
"{\"url\":\"openclaw-canvas://next.html\"}".data(using: .utf8))
let navigate = try JSONDecoder().decode(OpenClawCanvasNavigateParams.self, from: navigateData)
#expect(navigate.url == "openclaw-canvas://next.html")
}
}
apps/shared/OpenClawKit/Tests/OpenClawKitTests/CanvasSnapshotFormatTests.swift
-15
View File
@@ -1,15 +0,0 @@
import OpenClawKit
import Foundation
import Testing
@Suite struct CanvasSnapshotFormatTests {
@Test func acceptsJpgAlias() throws {
struct Wrapper: Codable {
var format: OpenClawCanvasSnapshotFormat
}
let data = try #require("{\"format\":\"jpg\"}".data(using: .utf8))
let decoded = try JSONDecoder().decode(Wrapper.self, from: data)
#expect(decoded.format == .jpeg)
}
}
apps/shared/OpenClawKit/Tests/OpenClawKitTests/ToolDisplayRegistryTests.swift
-5
View File
@@ -3,11 +3,6 @@ import Foundation
import Testing
@Suite struct ToolDisplayRegistryTests {
@Test func loadsToolDisplayConfigFromBundle() {
let url = OpenClawKitResources.bundle.url(forResource: "tool-display", withExtension: "json")
#expect(url != nil)
}
@Test func resolvesKnownToolFromConfig() {
let summary = ToolDisplayRegistry.resolve(name: "exec", args: nil)
#expect(summary.emoji == "🛠️")
config/assertion-safety-baseline.txt
+2 -4
View File
@@ -141,10 +141,9 @@ extensions/buzz/src/setup-verify.ts 5
extensions/buzz/src/types.ts 2
extensions/byteplus/video-generation-provider.ts 1
extensions/canvas/src/a2ui-jsonl.ts 1
extensions/canvas/src/cli.ts 4
extensions/canvas/src/cli.ts 2
extensions/canvas/src/config-migration.ts 2
extensions/canvas/src/host/server.ts 1
extensions/canvas/src/tool.ts 3
extensions/canvas/src/tool.ts 1
extensions/chutes/models.ts 1
extensions/chutes/oauth.ts 2
extensions/clawrouter/index.ts 1
@@ -637,7 +636,6 @@ extensions/line/src/setup-core.ts 1
extensions/line/src/webhook-spool.ts 4
extensions/line/src/webhook-utils.ts 1
extensions/line/src/webhook.ts 1
extensions/linux-canvas/src/commands.ts 1
extensions/litellm/onboard.ts 1
extensions/llama-cpp/src/embedding-provider.ts 1
extensions/llm-task/doctor-contract-api.ts 1
docs/.i18n/glossary.zh-CN.json
+16
View File
@@ -1806,5 +1806,21 @@
{
"source": "LM Studio",
"target": "LM Studio"
},
{
"source": "Widget panel",
"target": "小组件面板"
},
{
"source": "Show widget",
"target": "显示小组件"
},
{
"source": "Session dashboards",
"target": "会话仪表板"
},
{
"source": "macOS widget panel",
"target": "macOS 小组件面板"
}
]
docs/cli/index.md
+1 -2
View File
@@ -350,8 +350,7 @@ openclaw [--dev] [--profile <name>] <command>
invoke
notify
push
canvas snapshot|present|hide|navigate|eval
canvas a2ui push|reset
canvas present|hide|navigate
camera list|snap|clip
screen record
location get

Some files were not shown because too many files have changed in this diff Show More