feat(cua-computer): recording family with host-owned resource handles (#124035)

* feat(cua-computer): add recording resource handles

* test(agents): split computer tool coverage
This commit is contained in:
Peter Steinberger
2026-08-15 00:14:13 -07:00
committed by GitHub
parent 6d9ea63a87
commit 079bb34196
30 changed files with 1854 additions and 599 deletions
+5 -2
View File
@@ -38,6 +38,8 @@ Providers with the v2 window/element family can additionally expose `list_apps`,
The CUA provider also exposes the v2 browser family: `get_browser_state`, `browser_prepare`, `browser_navigate`, `browser_click`, `browser_type`, `browser_dialog`, `browser_set_input_files`, `browser_download`, and `browser_pointer`. Bind a discovered native browser window with `get_browser_state`, then use the returned opaque `browserRef`, `pageRef`, observation, and element references. These references belong to one Computer Use execution and driver generation; navigation invalidates page-element observations, and a driver restart invalidates the complete browser reference set.
CUA additionally exposes `get_recording_state`, `start_recording`, `stop_recording`, and `replay_trajectory`. Recording and browser file operations use opaque `openclaw:computer-resource` handles. The node creates and validates the underlying files and directories; agent actions never accept native paths, output roots, or helper executable paths. Handles belong to one Computer Use execution and cannot be reused by another execution.
Modifier keys ride the `text` field on click and scroll actions (`shift`, `ctrl`, `alt`, `cmd`). After an input action the tool returns a fresh screenshot so the model can observe the result. If more than one computer-capable node is connected, pass `node` explicitly.
Screenshots are kept **model-only**: they are never auto-delivered to the chat channel. Treat all on-screen content as untrusted input; the tool warns the model not to follow on-screen instructions that conflict with the user's request.
@@ -54,7 +56,7 @@ The app waits until the private socket accepts connections before advertising CU
The embedded CUA daemon runs in unrestricted mode because bounded CUA grants require exact launch-time resources and cannot represent OpenClaw's runtime-discovered windows and elements. OpenClaw command arming, pairing approval, and tool policy are the authoritative authorization gate, identical to the shipped Peekaboo fulfiller. The app owns the daemon and its macOS TCC identity, and the daemon accepts local connections only through an owner-only socket directory.
The CUA descriptor advertises window, element, and browser targets; background and foreground delivery; and image, accessibility, and browser observations. Peekaboo remains the default in this release and advertises only the action families its native adapter implements.
The CUA descriptor advertises window, element, and browser targets; background and foreground delivery; image, accessibility, and browser observations; and recording. Peekaboo remains the default in this release and does not advertise recording.
#### Browser profiles
@@ -105,7 +107,7 @@ The bundled `cua-computer` plugin provides an experimental fulfiller for Windows
OpenClaw checks the SDK package version, the selected OS/CPU package version, regular-file identity, and the pinned SHA-256 digest of the native library and Node runtime. A clean check prints `no findings`. If it reports a `COMPUTER_DRIVER_*` error, reinstall or update OpenClaw on this node host and run the check again. Do not download a standalone `cua-driver` executable or add one to `PATH`; Windows and Linux use the npm-installed in-process SDK.
3. Start `openclaw node run` from the interactive desktop session. The plugin repeats the artifact verification at startup before it imports native code, creates its configured SDK runtime lazily, then creates separate fixed window- and desktop-scoped trusted sessions for node-host command execution. `escalate_scope` reads the already-desktop session state, so the window identity remains immutable. It closes both sessions and shuts down the runtime when the command host stops or restarts.
3. Start `openclaw node run` from the interactive desktop session. The plugin repeats the artifact verification at startup before it imports native code, creates its configured SDK runtime lazily, then creates separate fixed window- and desktop-scoped trusted sessions for each provider execution. `escalate_scope` reads the already-desktop session state, so the window identity remains immutable. Completion, cancellation, Gateway disconnect, provider switching, local Stop, and command-host shutdown all close that exact execution, finalize or discard its recording resources, close both sessions, and shut down its runtime.
4. Add `computer.act` to the Gateway allowlist. This plugin registers `computer.act` as a dangerous plugin node command, so enabling the plugin alone is not enough; the operator must opt in explicitly:
@@ -181,6 +183,7 @@ On macOS, default-on means a paired gateway can drive pointer and keyboard input
- Every layer (tool policy, gateway command policy, pairing, node-app setting, and platform permissions) must agree. On macOS that includes **Allow Computer Control**, Accessibility, and Screen Recording; the native Peekaboo path also requires Event Posting. Actions execute while those durable controls remain enabled; there is no per-action confirmation.
- The macOS fulfiller posts text one grapheme at a time, so cancellation, disconnect, pause, disable, or endpoint replacement stops it before the next grapheme. The experimental CUA Driver fulfiller passes node cancellation to the SDK for each call.
- CUA recording, replay, browser upload, and browser download paths are node-owned. The model receives only opaque execution-scoped resource handles; traversal, absolute paths, symlink escapes, and helper selection are rejected before driver dispatch.
- Screenshots are model-only and never auto-sent to chat (issue [#44759](https://github.com/openclaw/openclaw/issues/44759)).
- Treat screen content as untrusted; it can carry prompt injection.
@@ -14,6 +14,7 @@ import {
export type CuaComputerActParams = {
action: ComputerActParams["action"];
executionId?: string;
displayFrameId?: string;
x?: number;
y?: number;
@@ -51,8 +52,11 @@ export type CuaComputerActParams = {
dialogAction?: "inspect" | "accept" | "dismiss";
dialogRef?: string;
promptText?: string;
files?: string[];
destinationRoot?: string;
resourceHandle?: string;
resourceHandles?: string[];
recordVideo?: boolean;
delayMs?: number;
stopOnError?: boolean;
pointerAction?: "hover" | "right_click" | "double_click" | "scroll" | "drag";
destinationElementRef?: string;
toX?: number;
@@ -1,4 +1,7 @@
import { describe, expect, it } from "vitest";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { driver, execution } from "./commands.test-helpers.js";
import {
CUA_DRIVER_CONTRACT_FIXTURES,
@@ -6,9 +9,20 @@ import {
} from "./cua-driver-contract.test-fixtures.js";
import type { CuaToolResult } from "./driver-client.js";
const tempRoots: string[] = [];
afterEach(async () => {
await Promise.all(
tempRoots.splice(0).map(async (root) => await fs.rm(root, { recursive: true, force: true })),
);
});
describe("cua-computer browser actions", () => {
it("maps every browser action to the pinned driver tool contract", async () => {
const resourceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-cua-browser-"));
tempRoots.push(resourceRoot);
const { session, callTool } = driver();
let downloadedFile = "";
callTool.mockImplementation(async (name, args) => {
switch (name) {
case "list_windows":
@@ -29,6 +43,8 @@ describe("cua-computer browser actions", () => {
case "browser_set_input_files":
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.browserFiles);
case "browser_download":
downloadedFile = path.join(String(args.destination_root), "download.txt");
await fs.writeFile(downloadedFile, "download");
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.browserDownload);
case "browser_click":
case "browser_type":
@@ -44,7 +60,7 @@ describe("cua-computer browser actions", () => {
return cuaToolResult({});
}
});
const computer = await execution(session);
const computer = await execution(session, { resourceRoot });
const listed = JSON.parse(await computer.act('{"action":"list_windows"}')) as {
details: { windows: Array<{ windowRef: string }> };
};
@@ -119,6 +135,21 @@ describe("cua-computer browser actions", () => {
),
) as { details: { dialogRef: string } };
expect(dialog.details.dialogRef).toMatch(/^cua:v2:dialog:/);
const downloadJson = await computer.act(
JSON.stringify({
action: "browser_download",
browserRef,
pageRef,
observationId,
elementRef: firstElement,
}),
);
expect(downloadJson).not.toContain(resourceRoot);
const download = JSON.parse(downloadJson) as {
details: { fileResourceHandles: string[]; resourceHandle: string };
};
expect(download.details.resourceHandle).toMatch(/^openclaw:computer-resource:v1:/u);
expect(download.details.fileResourceHandles[0]).toMatch(/^openclaw:computer-resource:v1:/u);
await computer.act(
JSON.stringify({
action: "browser_set_input_files",
@@ -126,17 +157,7 @@ describe("cua-computer browser actions", () => {
pageRef,
observationId,
elementRef: secondElement,
files: ["/tmp/input.txt"],
}),
);
await computer.act(
JSON.stringify({
action: "browser_download",
browserRef,
pageRef,
observationId,
elementRef: firstElement,
destinationRoot: "/tmp/downloads",
resourceHandles: download.details.fileResourceHandles,
}),
);
await computer.act(
@@ -213,23 +234,23 @@ describe("cua-computer browser actions", () => {
},
undefined,
],
[
"browser_set_input_files",
{
target_id: "native-browser-target-1",
tab_id: "native-page-1",
ref: "p7:1",
files: ["/tmp/input.txt"],
},
undefined,
],
[
"browser_download",
{
target_id: "native-browser-target-1",
tab_id: "native-page-1",
ref: "p7:0",
destination_root: "/tmp/downloads",
destination_root: path.dirname(downloadedFile),
},
undefined,
],
[
"browser_set_input_files",
{
target_id: "native-browser-target-1",
tab_id: "native-page-1",
ref: "p7:1",
files: [downloadedFile],
},
undefined,
],
+57 -25
View File
@@ -1,6 +1,6 @@
import { browserElement, browserTarget, requireWindowTarget } from "./action-targets.js";
import type { CuaComputerActParams } from "./action-targets.js";
import type { CuaDriverSession } from "./driver-client.js";
import type { CuaDriverSession, CuaToolResult } from "./driver-client.js";
import {
browserBinding,
browserDialogEnvelope,
@@ -8,6 +8,7 @@ import {
browserToolEnvelope,
callWindowTool,
} from "./driver-result.js";
import type { CuaExecutionResources } from "./execution-resources.js";
import {
clearDialogRef,
invalidateBrowserObservation,
@@ -21,6 +22,7 @@ import {
export async function handleBrowserAct(
driver: CuaDriverSession,
state: CuaFrameState,
resources: CuaExecutionResources,
input: CuaComputerActParams,
signal?: AbortSignal,
): Promise<string | undefined> {
@@ -174,36 +176,66 @@ export async function handleBrowserAct(
case "browser_set_input_files": {
const target = browserTarget(driver, state, input);
const ref = browserElement(state, input, target)!;
const result = await callWindowTool(
driver,
state,
"browser_set_input_files",
{
target_id: target.targetId,
tab_id: target.tabId,
ref,
files: input.files,
},
signal,
);
const files = await resources.resolveFiles(input.resourceHandles ?? []);
let result: CuaToolResult;
try {
result = await callWindowTool(
driver,
state,
"browser_set_input_files",
{
target_id: target.targetId,
tab_id: target.tabId,
ref,
files,
},
signal,
);
} catch (error) {
signal?.throwIfAborted();
throw new Error(
"COMPUTER_DRIVER_ERROR: browser_set_input_files failed; inspect node logs and resource state before retrying",
{ cause: error },
);
}
return JSON.stringify(browserToolEnvelope(result, "browser_set_input_files"));
}
case "browser_download": {
const target = browserTarget(driver, state, input);
const ref = browserElement(state, input, target)!;
const result = await callWindowTool(
driver,
state,
"browser_download",
{
target_id: target.targetId,
tab_id: target.tabId,
ref,
destination_root: input.destinationRoot,
const resource = await resources.createDirectory("browser-download");
let result: CuaToolResult;
try {
result = await callWindowTool(
driver,
state,
"browser_download",
{
target_id: target.targetId,
tab_id: target.tabId,
ref,
destination_root: resource.path,
},
signal,
);
} catch (error) {
await resources.discard(resource.handle).catch(() => {});
signal?.throwIfAborted();
throw new Error(
"COMPUTER_DRIVER_ERROR: browser_download failed; inspect node logs and resource state before retrying",
{ cause: error },
);
}
const envelope = browserToolEnvelope(result, "browser_download");
const fileResourceHandles = await resources.captureFiles(resource.handle);
return JSON.stringify({
...envelope,
details: {
...envelope.details,
resourceHandle: resource.handle,
fileResourceHandles,
},
signal,
);
return JSON.stringify(browserToolEnvelope(result, "browser_download"));
});
}
case "browser_pointer": {
const target = browserTarget(driver, state, input);
@@ -1,3 +1,4 @@
import { randomUUID } from "node:crypto";
import { vi } from "vitest";
import { createCuaComputerProvider } from "./commands.js";
import type { CuaDriverSession, CuaToolResult } from "./driver-client.js";
@@ -133,12 +134,16 @@ export function driver(
};
}
export async function execution(session: CuaDriverSession) {
export async function execution(
session: CuaDriverSession,
options: { resourceRoot?: string } = {},
) {
return await createCuaComputerProvider({
platform: "linux",
driver: session,
imageProcessor: {
encode: vi.fn(async () => ({ data: Buffer.from("jpeg"), width: 100, height: 50 })),
},
}).openExecution({});
resourceRoot: options.resourceRoot,
}).openExecution({ executionId: randomUUID() });
}
+18 -4
View File
@@ -66,12 +66,16 @@ describe("cua-computer provider", () => {
"browser_download",
"browser_pointer",
"escalate_scope",
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
"invoke_menu",
],
targets: ["screen", "window", "element", "browser"],
deliveryModes: ["background", "foreground"],
observations: ["image", "accessibility", "browser"],
features: { recording: false, agentCursor: false, multiDisplay: false },
features: { recording: true, agentCursor: false, multiDisplay: false },
});
});
@@ -82,6 +86,14 @@ describe("cua-computer provider", () => {
expect(actions).not.toContain("left_mouse_down");
expect(actions).not.toContain("left_mouse_up");
expect(actions).toContain("get_window_state");
expect(actions).toEqual(
expect.arrayContaining([
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
]),
);
});
it("advertises the macOS mapping only with a valid atomic app-provided endpoint", () => {
@@ -97,7 +109,7 @@ describe("cua-computer provider", () => {
expect(provider.capabilities().actions).toContain("get_window_state");
expect(provider.capabilities().actions).not.toContain("left_mouse_down");
expect(provider.capabilities().features).toEqual({
recording: false,
recording: true,
agentCursor: false,
multiDisplay: false,
});
@@ -140,7 +152,7 @@ describe("cua-computer provider", () => {
imageProcessor: {
encode: vi.fn(async () => ({ data: Buffer.from("png"), width: 100, height: 50 })),
},
}).openExecution({});
}).openExecution({ executionId: "123e4567-e89b-42d3-a456-426614174000" });
const screen = JSON.parse(await computer.snapshot('{"format":"png","maxWidth":100}')) as {
displayFrameId: string;
width: number;
@@ -307,7 +319,9 @@ describe("cua-computer provider", () => {
});
expect(createDriver).not.toHaveBeenCalled();
const computer = await provider.openExecution({});
const computer = await provider.openExecution({
executionId: "123e4567-e89b-42d3-a456-426614174000",
});
await computer.snapshot('{"format":"png","maxWidth":100}');
expect(createDriver).toHaveBeenCalledOnce();
+68 -24
View File
@@ -20,6 +20,7 @@ import {
type CuaToolResult,
} from "./driver-client.js";
import { platformActions } from "./driver-result.js";
import { createLazyCuaExecutionResources } from "./execution-resources.js";
import {
adoptGeneration,
issueFrame,
@@ -31,6 +32,7 @@ import {
type CuaScreenSize,
} from "./frame.js";
import { createCuaMcpDriver } from "./mcp-driver-client.js";
import { closeRecordingExecution } from "./recording-actions.js";
import { handleV2Act, type CuaComputerActParams } from "./v2-actions.js";
const AVAILABILITY_POLL_MS = 5_000;
@@ -82,6 +84,7 @@ type CuaComputerProviderOptions = {
imageProcessor?: ImageProcessor;
setInterval?: typeof setInterval;
clearInterval?: typeof clearInterval;
resourceRoot?: string;
};
function resolveMacOsMcpEndpoint(
@@ -435,26 +438,21 @@ export function createCuaComputerProvider(
const platform = options.platform ?? process.platform;
const env = options.env ?? process.env;
const macOsEndpoint = platform === "darwin" ? resolveMacOsMcpEndpoint(env) : undefined;
let ownedDriver: CuaDriverSession | undefined;
let ownedAvailabilityDriver: CuaDriverSession | undefined;
let stopped = false;
// The node host owns one trusted SDK session for this command execution.
// It is shared by snapshot/act so a frame can only authorize its paired input.
const driver = () => {
const createDriver =
options.createDriver ??
(macOsEndpoint ? () => createCuaMcpDriver({ ...macOsEndpoint, env }) : createCuaDriver);
const availabilityDriver = () => {
if (stopped) {
throw new Error("COMPUTER_DRIVER_UNAVAILABLE: cua-computer is stopping");
}
return (
options.driver ??
(ownedDriver ??= (
options.createDriver ??
(macOsEndpoint ? () => createCuaMcpDriver({ ...macOsEndpoint, env }) : createCuaDriver)
)())
);
return options.driver ?? (ownedAvailabilityDriver ??= createDriver());
};
const disposeOwnedDriver = async () => {
const disposeAvailabilityDriver = async () => {
stopped = true;
const current = ownedDriver;
ownedDriver = undefined;
const current = ownedAvailabilityDriver;
ownedAvailabilityDriver = undefined;
await current?.dispose();
};
const imageProcessor = options.imageProcessor ?? createImageProcessor(env);
@@ -467,7 +465,7 @@ export function createCuaComputerProvider(
// endpoint is the synchronous macOS readiness lease; invocation still
// awaits the MCP initialize handshake and fails visibly if it cannot attach.
const isAvailable = () =>
macOsEndpoint !== undefined || (isSupportedPlatform && driver().isAvailable());
macOsEndpoint !== undefined || (isSupportedPlatform && availabilityDriver().isAvailable());
return {
id: "cua-computer",
@@ -478,20 +476,20 @@ export function createCuaComputerProvider(
id: "cua-computer",
label: "CUA Computer",
generation: isSupportedPlatform
? `cua-computer-v2:${driver().generation}`
? `cua-computer-v2:${availabilityDriver().generation}`
: "cua-computer-v2:unsupported",
},
actions: platformActions(platform),
targets: ["screen", "window", "element", "browser"],
deliveryModes: ["background", "foreground"],
observations: ["image", "accessibility", "browser"],
features: { recording: false, agentCursor: false, multiDisplay: false },
features: { recording: true, agentCursor: false, multiDisplay: false },
}),
isAvailable,
watchAvailability: (_context, onChange) => {
let knownAvailable = isAvailable();
const timer = interval(() => {
driver().resetAvailabilityCache();
availabilityDriver().resetAvailabilityCache();
const available = isAvailable();
if (available !== knownAvailable) {
knownAvailable = available;
@@ -501,15 +499,29 @@ export function createCuaComputerProvider(
timer.unref?.();
return () => {
clear(timer);
void disposeOwnedDriver();
void disposeAvailabilityDriver();
};
},
openExecution: async () => {
if (stopped) {
throw new Error("COMPUTER_DRIVER_UNAVAILABLE: cua-computer is stopping");
}
const executionDriver = options.driver ?? createDriver();
const resources = createLazyCuaExecutionResources({ rootDir: options.resourceRoot });
const executionState = { resources, recording: {} };
const queue = new PromiseQueue();
const frameState: CuaFrameState = { generation: driver().generation };
const frameState: CuaFrameState = { generation: executionDriver.generation };
let closing = false;
let closePromise: Promise<void> | undefined;
const assertOpen = () => {
if (closing) {
throw new Error("COMPUTER_DRIVER_UNAVAILABLE: provider execution is closing");
}
};
return {
snapshot: async (paramsJSON, signal) =>
await queue.run(async () => {
assertOpen();
if (!isSupportedPlatform) {
throw new Error(
platform === "darwin"
@@ -522,7 +534,7 @@ export function createCuaComputerProvider(
const format = params.format ?? "jpeg";
const maxWidth = params.maxWidth ?? (format === "png" ? 900 : 1_600);
const quality = Math.min(1, Math.max(0.05, params.quality ?? 0.72));
const desktop = await driver().getDesktopState(signal);
const desktop = await executionDriver.getDesktopState(signal);
const geometry = desktopGeometry(desktop);
// Windows and Linux report capture and input geometry in the same
// physical-pixel space. macOS intentionally reports logical screen
@@ -551,7 +563,7 @@ export function createCuaComputerProvider(
width = result.width;
height = result.height;
}
adoptGeneration(frameState, driver().generation);
adoptGeneration(frameState, executionDriver.generation);
const displayFrameId = issueFrame(frameState, geometry, { width, height });
return JSON.stringify({
format,
@@ -564,6 +576,7 @@ export function createCuaComputerProvider(
}),
act: async (paramsJSON, signal) =>
await queue.run(async () => {
assertOpen();
if (!isSupportedPlatform) {
throw new Error(
platform === "darwin"
@@ -573,14 +586,45 @@ export function createCuaComputerProvider(
}
return await handleV2Act(
platform,
driver(),
executionDriver,
frameState,
executionState,
parseComputerActParamsJSON(paramsJSON),
handleDesktopAct,
signal,
);
}),
close: async () => await disposeOwnedDriver(),
close: async (reason) => {
if (closePromise) {
return await closePromise;
}
closing = true;
closePromise = queue.run(async () => {
let failure: unknown;
try {
await closeRecordingExecution({
driver: executionDriver,
state: executionState.recording,
resources,
reason,
});
} catch (error) {
failure = error;
}
await resources.dispose(reason !== "completion").catch((error: unknown) => {
failure ??= error;
});
await executionDriver.dispose().catch((error: unknown) => {
failure ??= error;
});
if (failure) {
throw failure instanceof Error
? failure
: new Error("CUA Computer cleanup failed", { cause: failure });
}
});
return await closePromise;
},
};
},
};
@@ -129,6 +129,34 @@ export const CUA_DRIVER_CONTRACT_FIXTURES = {
download_id: "opaque-download-guid",
bytes: 42,
},
recordingActive: {
recording: true,
enabled: true,
output_dir: "/native/recording",
next_turn: 1,
last_error: null,
video_active: false,
last_video_path: null,
owner: "native-session",
},
recordingStopped: {
recording: false,
enabled: false,
output_dir: null,
next_turn: 1,
last_error: null,
video_active: false,
last_video_path: "/native/recording/recording.mp4",
owner: null,
},
replay: {
directory: "/native/recording",
attempted: 1,
succeeded: 1,
failed: 0,
stop_on_error: true,
turns: [{ turn: "turn-00001", tool: "click", ok: true, result_summary: "ok" }],
},
confirmedBackgroundAction: {
effect: 0,
route: 0,
@@ -46,6 +46,10 @@ const CUA_COMMON_ACTION_NAMES = [
"browser_download",
"browser_pointer",
"escalate_scope",
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
"invoke_menu",
] as const;
@@ -0,0 +1,225 @@
import { randomUUID } from "node:crypto";
import path from "node:path";
import {
removePathWithinRoot,
resolvePreferredOpenClawTmpDir,
root,
} from "openclaw/plugin-sdk/file-access-runtime";
const RESOURCE_HANDLE_PREFIX = "openclaw:computer-resource:v1:";
const RESOURCE_ROOT_NAME = "cua-computer-resources";
const MAX_RESOURCE_TREE_ENTRIES = 10_000;
type DirectoryResourceKind = "browser-download" | "recording";
type ResourceKind = DirectoryResourceKind | "file";
type ResourceEntry = {
kind: ResourceKind;
relativePath: string;
};
type SafeRoot = Awaited<ReturnType<typeof root>>;
export type CuaExecutionResources = {
createDirectory(kind: DirectoryResourceKind): Promise<{ handle: string; path: string }>;
resolveFiles(handles: readonly string[]): Promise<string[]>;
validateDirectoryTree(handle: string): Promise<string>;
captureFiles(handle: string): Promise<string[]>;
discard(handle: string): Promise<void>;
dispose(discard: boolean): Promise<void>;
};
export function createLazyCuaExecutionResources(
options: {
rootDir?: string;
} = {},
): CuaExecutionResources {
let resourcesPromise: Promise<CuaExecutionResources> | undefined;
const resources = () => (resourcesPromise ??= createCuaExecutionResources(options));
return {
createDirectory: async (label) => await (await resources()).createDirectory(label),
resolveFiles: async (handles) => await (await resources()).resolveFiles(handles),
validateDirectoryTree: async (handle) =>
await (await resources()).validateDirectoryTree(handle),
captureFiles: async (handle) => await (await resources()).captureFiles(handle),
discard: async (handle) => await (await resources()).discard(handle),
dispose: async (discard) => {
if (resourcesPromise) {
await (await resourcesPromise).dispose(discard);
}
},
};
}
function resourceError(message: string): Error {
return new Error(`COMPUTER_INVALID_RESOURCE: ${message}`);
}
function newHandle(): string {
return `${RESOURCE_HANDLE_PREFIX}${randomUUID()}`;
}
function safeLabel(value: string): string {
return value.replaceAll(/[^a-z0-9-]/giu, "-").slice(0, 32) || "resource";
}
async function requireEntry(
resources: Map<string, ResourceEntry>,
executionRoot: SafeRoot,
handle: string,
kind: ResourceKind,
): Promise<{ entry: ResourceEntry; path: string }> {
if (!handle.startsWith(RESOURCE_HANDLE_PREFIX)) {
throw resourceError("malformed resource handle");
}
const entry = resources.get(handle);
if (!entry || entry.kind !== kind) {
throw resourceError("resource handle is unknown in this provider execution");
}
let stat;
let resolved;
try {
stat = await executionRoot.stat(entry.relativePath);
resolved = await executionRoot.resolve(entry.relativePath);
} catch (error) {
throw resourceError(`resource is no longer a safe ${kind}: ${String(error)}`);
}
if (stat.isSymbolicLink || (kind === "file" ? !stat.isFile : !stat.isDirectory)) {
throw resourceError(`resource is no longer a regular ${kind}`);
}
return { entry, path: resolved };
}
async function createCuaExecutionResources(
options: {
rootDir?: string;
} = {},
): Promise<CuaExecutionResources> {
const baseRoot = await root(
options.rootDir ?? path.join(resolvePreferredOpenClawTmpDir(), RESOURCE_ROOT_NAME),
{ hardlinks: "reject", mode: 0o700, symlinks: "reject" },
);
await baseRoot.ensureRoot();
const executionDirectory = `execution-${randomUUID()}`;
await baseRoot.mkdir(executionDirectory);
const executionPath = await baseRoot.resolve(executionDirectory);
const executionRoot = await root(executionPath, {
hardlinks: "reject",
mode: 0o700,
symlinks: "reject",
});
const resources = new Map<string, ResourceEntry>();
const handlesByPath = new Map<string, string>();
let disposed = false;
const assertActive = () => {
if (disposed) {
throw resourceError("provider execution is closed");
}
};
const register = (kind: ResourceKind, relativePath: string): string => {
const existing = handlesByPath.get(relativePath);
if (existing) {
return existing;
}
const handle = newHandle();
resources.set(handle, { kind, relativePath });
handlesByPath.set(relativePath, handle);
return handle;
};
const removeHandle = async (handle: string) => {
const entry = resources.get(handle);
if (!entry) {
return;
}
await removePathWithinRoot({
rootDir: executionRoot.rootReal,
relativePath: entry.relativePath,
recursive: true,
force: true,
});
resources.delete(handle);
handlesByPath.delete(entry.relativePath);
};
return {
async createDirectory(kind) {
assertActive();
const relativePath = `${safeLabel(kind)}-${randomUUID()}`;
await executionRoot.mkdir(relativePath);
return {
handle: register(kind, relativePath),
path: await executionRoot.resolve(relativePath),
};
},
async resolveFiles(handles) {
assertActive();
return await Promise.all(
handles.map(
async (handle) => (await requireEntry(resources, executionRoot, handle, "file")).path,
),
);
},
async validateDirectoryTree(handle) {
assertActive();
const directory = await requireEntry(resources, executionRoot, handle, "recording");
let visited = 0;
const visit = async (relativePath: string): Promise<void> => {
for (const child of await executionRoot.list(relativePath, { withFileTypes: true })) {
visited += 1;
if (visited > MAX_RESOURCE_TREE_ENTRIES) {
throw resourceError("resource tree is too large");
}
if (child.isSymbolicLink || (!child.isDirectory && !child.isFile)) {
throw resourceError("resource tree contains an unsupported entry");
}
const childPath = path.join(relativePath, child.name);
await executionRoot.resolve(childPath);
if (child.isDirectory) {
await visit(childPath);
}
}
};
await visit(directory.entry.relativePath);
return directory.path;
},
async captureFiles(handle) {
assertActive();
const directory = await requireEntry(resources, executionRoot, handle, "browser-download");
const handles: string[] = [];
for (const child of await executionRoot.list(directory.entry.relativePath, {
withFileTypes: true,
})) {
if (child.isSymbolicLink || !child.isFile) {
continue;
}
const relativePath = path.join(directory.entry.relativePath, child.name);
await executionRoot.resolve(relativePath);
handles.push(register("file", relativePath));
}
return handles;
},
async discard(handle) {
assertActive();
await removeHandle(handle);
},
async dispose(discard) {
if (disposed) {
return;
}
disposed = true;
resources.clear();
handlesByPath.clear();
if (discard) {
await removePathWithinRoot({
rootDir: baseRoot.rootReal,
relativePath: executionDirectory,
recursive: true,
force: true,
});
}
},
};
}
@@ -0,0 +1,7 @@
import type { CuaExecutionResources } from "./execution-resources.js";
import type { CuaRecordingState } from "./recording-actions.js";
export type CuaExecutionState = {
resources: CuaExecutionResources;
recording: CuaRecordingState;
};
@@ -0,0 +1,173 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { driver, execution } from "./commands.test-helpers.js";
import {
CUA_DRIVER_CONTRACT_FIXTURES,
cuaToolResult,
} from "./cua-driver-contract.test-fixtures.js";
const tempRoots: string[] = [];
async function tempRoot(label: string): Promise<string> {
const value = await fs.mkdtemp(path.join(os.tmpdir(), label));
tempRoots.push(value);
return value;
}
afterEach(async () => {
await Promise.all(
tempRoots.splice(0).map(async (root) => await fs.rm(root, { recursive: true, force: true })),
);
});
describe("cua-computer recording actions", () => {
it("maps each recording tool while projecting only opaque resource handles", async () => {
const resourceRoot = await tempRoot("openclaw-cua-recording-");
const active = driver();
let nativeRecordingRoot = "";
active.callTool.mockImplementation(async (name, args) => {
switch (name) {
case "start_recording":
nativeRecordingRoot = String(args.output_dir);
await fs.mkdir(path.join(nativeRecordingRoot, "turn-00001"));
await fs.writeFile(
path.join(nativeRecordingRoot, "turn-00001", "action.json"),
JSON.stringify({ tool: "click", arguments: {} }),
);
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingActive, {
text: `recording at ${nativeRecordingRoot}`,
});
case "get_recording_state":
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingActive);
case "stop_recording":
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingStopped, {
text: `${nativeRecordingRoot}/recording.mp4`,
});
case "replay_trajectory":
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.replay);
default:
return cuaToolResult({});
}
});
const computer = await execution(active.session, { resourceRoot });
const startedJson = await computer.act(
JSON.stringify({ action: "start_recording", recordVideo: false }),
);
expect(startedJson).not.toContain(nativeRecordingRoot);
expect(startedJson).not.toContain("native-session");
const started = JSON.parse(startedJson) as { details: { resourceHandle: string } };
expect(started.details.resourceHandle).toMatch(/^openclaw:computer-resource:v1:/u);
const stateJson = await computer.act(JSON.stringify({ action: "get_recording_state" }));
expect(stateJson).not.toContain("/native/");
expect(stateJson).toContain(started.details.resourceHandle);
const stoppedJson = await computer.act(JSON.stringify({ action: "stop_recording" }));
expect(stoppedJson).not.toContain("recording.mp4");
expect(stoppedJson).toContain(started.details.resourceHandle);
const replayJson = await computer.act(
JSON.stringify({
action: "replay_trajectory",
resourceHandle: started.details.resourceHandle,
delayMs: 25,
stopOnError: true,
}),
);
expect(replayJson).not.toContain("/native/");
expect(replayJson).toContain(started.details.resourceHandle);
expect(active.callTool.mock.calls).toEqual([
["start_recording", { output_dir: nativeRecordingRoot, record_video: false }, undefined],
["get_recording_state", {}, undefined],
["stop_recording", {}, undefined],
[
"replay_trajectory",
{ dir: nativeRecordingRoot, delay_ms: 25, stop_on_error: true },
undefined,
],
]);
});
it("rejects malformed, absolute, traversal, and symlink-escaped replay resources", async () => {
const resourceRoot = await tempRoot("openclaw-cua-resource-security-");
const outside = await tempRoot("openclaw-cua-resource-outside-");
const active = driver();
let nativeRecordingRoot = "";
active.callTool.mockImplementation(async (name, args) => {
if (name === "start_recording") {
nativeRecordingRoot = String(args.output_dir);
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingActive);
}
if (name === "stop_recording") {
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingStopped);
}
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.replay);
});
const computer = await execution(active.session, { resourceRoot });
for (const resourceHandle of [
"../outside",
outside,
"openclaw:computer-resource:v1:unknown",
"openclaw:computer-resource:v1:123e4567-e89b-42d3-a456-426614174000",
]) {
await expect(
computer.act(JSON.stringify({ action: "replay_trajectory", resourceHandle })),
).rejects.toThrow("COMPUTER_");
}
const started = JSON.parse(
await computer.act(JSON.stringify({ action: "start_recording" })),
) as { details: { resourceHandle: string } };
await computer.act(JSON.stringify({ action: "stop_recording" }));
await fs.rm(nativeRecordingRoot, { recursive: true });
await fs.symlink(outside, nativeRecordingRoot, "dir");
const callsBeforeReplay = active.callTool.mock.calls.length;
await expect(
computer.act(
JSON.stringify({
action: "replay_trajectory",
resourceHandle: started.details.resourceHandle,
}),
),
).rejects.toThrow("COMPUTER_INVALID_RESOURCE");
expect(active.callTool).toHaveBeenCalledTimes(callsBeforeReplay);
});
it("finalizes an in-flight recording once on every execution close", async () => {
const resourceRoot = await tempRoot("openclaw-cua-recording-close-");
const active = driver();
let releaseStart: (() => void) | undefined;
const startGate = new Promise<void>((resolve) => {
releaseStart = resolve;
});
active.callTool.mockImplementation(async (name) => {
if (name === "start_recording") {
await startGate;
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingActive);
}
return cuaToolResult(CUA_DRIVER_CONTRACT_FIXTURES.recordingStopped);
});
const computer = await execution(active.session, { resourceRoot });
const start = computer.act(JSON.stringify({ action: "start_recording" }));
await vi.waitFor(() =>
expect(active.callTool).toHaveBeenCalledWith("start_recording", expect.anything(), undefined),
);
const close = computer.close("cancel");
releaseStart?.();
await start;
await close;
await computer.close("cancel");
expect(active.callTool.mock.calls.map(([name]) => name)).toEqual([
"start_recording",
"stop_recording",
]);
expect(active.dispose).toHaveBeenCalledOnce();
expect(await fs.readdir(resourceRoot)).toEqual([]);
});
});
@@ -0,0 +1,246 @@
import { z } from "zod";
import type { CuaComputerActParams } from "./action-targets.js";
import type { CuaDriverSession, CuaToolResult } from "./driver-client.js";
import type { CuaExecutionResources } from "./execution-resources.js";
const RecordingStateSchema = z.object({
recording: z.boolean(),
enabled: z.boolean(),
output_dir: z.string().nullable(),
next_turn: z.number().int().nonnegative(),
last_error: z.string().nullable(),
video_active: z.boolean(),
last_video_path: z.string().nullable(),
owner: z.string().nullable(),
});
const ReplayTurnSchema = z.object({
turn: z.string(),
tool: z.string().optional(),
ok: z.boolean(),
result_summary: z.string().optional(),
parse_error: z.string().optional(),
});
const ReplayResultSchema = z.object({
directory: z.string(),
attempted: z.number().int().nonnegative(),
succeeded: z.number().int().nonnegative(),
failed: z.number().int().nonnegative(),
stop_on_error: z.boolean(),
turns: z.array(ReplayTurnSchema),
first_failure: z.object({ turn: z.string(), tool: z.string(), error: z.string() }).optional(),
});
const MAX_REPLAY_TURNS = 200;
export type CuaRecordingState = {
active?: { resourceHandle: string };
};
function driverError(result: CuaToolResult, tool: string): Error {
const code = result.errorCode ? `COMPUTER_REFUSED_${result.errorCode}` : "COMPUTER_DRIVER_ERROR";
return new Error(`${code}: ${tool} failed; inspect node logs and resource state before retrying`);
}
function structured(result: CuaToolResult, tool: string): unknown {
if (result.isError) {
throw driverError(result, tool);
}
if (!result.structuredJson) {
throw new Error(`COMPUTER_DRIVER_ERROR: ${tool} returned no structuredContent`);
}
try {
return JSON.parse(result.structuredJson) as unknown;
} catch (error) {
throw new Error(`COMPUTER_DRIVER_ERROR: ${tool} returned invalid structuredContent`, {
cause: error,
});
}
}
function projectRecordingState(
native: z.infer<typeof RecordingStateSchema>,
resourceHandle: string | undefined,
) {
return {
recording: native.enabled,
nextTurn: native.next_turn,
videoActive: native.video_active,
...(native.last_error
? { videoError: "video unavailable; per-turn trajectory capture remains active" }
: {}),
...(resourceHandle ? { resourceHandle } : {}),
};
}
async function stopOwnedRecording(params: {
driver: CuaDriverSession;
state: CuaRecordingState;
resources: CuaExecutionResources;
discard: boolean;
}): Promise<void> {
const active = params.state.active;
params.state.active = undefined;
if (!active) {
return;
}
let failure: unknown;
try {
const result = await params.driver.callTool("stop_recording", {});
if (result.isError) {
failure = driverError(result, "stop_recording");
}
} catch (error) {
failure = error;
}
if (params.discard) {
try {
await params.resources.discard(active.resourceHandle);
} catch (error) {
failure ??= error;
}
}
if (failure) {
throw failure instanceof Error
? failure
: new Error("CUA recording cleanup failed", { cause: failure });
}
}
function projectReplayTurn(turn: z.infer<typeof ReplayTurnSchema>) {
const projected: Record<string, unknown> = { turn: turn.turn, ok: turn.ok };
if (turn.tool) {
projected.tool = turn.tool;
}
if (turn.parse_error) {
projected.parseError = true;
}
return projected;
}
export async function closeRecordingExecution(params: {
driver: CuaDriverSession;
state: CuaRecordingState;
resources: CuaExecutionResources;
reason: string;
}): Promise<void> {
await stopOwnedRecording({
...params,
discard: params.reason !== "completion",
});
}
export async function handleRecordingAct(
driver: CuaDriverSession,
state: CuaRecordingState,
resources: CuaExecutionResources,
input: CuaComputerActParams,
signal?: AbortSignal,
): Promise<string | undefined> {
switch (input.action) {
case "get_recording_state": {
if (!state.active) {
return JSON.stringify({ ok: true, details: { recording: false } });
}
const native = RecordingStateSchema.parse(
structured(await driver.callTool("get_recording_state", {}, signal), "get_recording_state"),
);
return JSON.stringify({
ok: true,
details: projectRecordingState(native, state.active.resourceHandle),
});
}
case "start_recording": {
if (state.active) {
throw new Error(
"COMPUTER_RECORDING_ACTIVE: stop the current recording before starting another",
);
}
const resource = await resources.createDirectory("recording");
try {
const result = await driver.callTool(
"start_recording",
{ output_dir: resource.path, record_video: input.recordVideo ?? false },
signal,
);
const native = RecordingStateSchema.parse(structured(result, "start_recording"));
if (!native.enabled) {
throw new Error("COMPUTER_DRIVER_ERROR: start_recording returned disabled state");
}
state.active = { resourceHandle: resource.handle };
return JSON.stringify({
ok: true,
details: projectRecordingState(native, resource.handle),
});
} catch (error) {
// A failed call can still have landed. End this exact trusted driver
// session so upstream stops only its owned recording; the public stop
// tool is unconditional and could tear down another session's work.
await driver.dispose().catch(() => {});
await resources.discard(resource.handle).catch(() => {});
throw error;
}
}
case "stop_recording": {
const active = state.active;
if (!active) {
return JSON.stringify({ ok: true, details: { recording: false } });
}
state.active = undefined;
const native = RecordingStateSchema.parse(
structured(await driver.callTool("stop_recording", {}, signal), "stop_recording"),
);
return JSON.stringify({
ok: true,
details: projectRecordingState(native, active.resourceHandle),
});
}
case "replay_trajectory": {
const resourceHandle = input.resourceHandle;
if (!resourceHandle) {
throw new Error(
"COMPUTER_INVALID_REQUEST: resourceHandle is required for replay_trajectory",
);
}
const directory = await resources.validateDirectoryTree(resourceHandle);
const native = ReplayResultSchema.parse(
structured(
await driver.callTool(
"replay_trajectory",
{
dir: directory,
...(input.delayMs !== undefined ? { delay_ms: input.delayMs } : {}),
...(input.stopOnError !== undefined ? { stop_on_error: input.stopOnError } : {}),
},
signal,
),
"replay_trajectory",
),
);
return JSON.stringify({
ok: true,
details: {
resourceHandle,
attempted: native.attempted,
succeeded: native.succeeded,
failed: native.failed,
stopOnError: native.stop_on_error,
turns: native.turns.slice(0, MAX_REPLAY_TURNS).map(projectReplayTurn),
...(native.turns.length > MAX_REPLAY_TURNS
? { truncatedTurns: native.turns.length - MAX_REPLAY_TURNS }
: {}),
...(native.first_failure
? {
firstFailure: {
turn: native.first_failure.turn,
tool: native.first_failure.tool,
},
}
: {}),
},
});
}
}
return undefined;
}
+14 -1
View File
@@ -21,6 +21,7 @@ import {
projectWindows,
windowObservation,
} from "./driver-result.js";
import type { CuaExecutionState } from "./execution-state.js";
import {
adoptGeneration,
resolveAppRef,
@@ -29,6 +30,7 @@ import {
verifyGeneration,
type CuaFrameState,
} from "./frame.js";
import { handleRecordingAct } from "./recording-actions.js";
const CUA_WIRE_ACTION_NAMES = COMPUTER_USE_V2_ACTION_NAMES.slice(1, 14);
const CUA_TARGETED_ACTION_NAMES = new Set([
@@ -216,6 +218,7 @@ export async function handleV2Act(
platform: NodeJS.Platform,
driver: CuaDriverSession,
state: CuaFrameState,
execution: CuaExecutionState,
params: ComputerActParams,
handleDesktop: (
driver: CuaDriverSession,
@@ -235,7 +238,17 @@ export async function handleV2Act(
if ((CUA_WIRE_ACTION_NAMES as readonly string[]).includes(input.action)) {
return await handleDesktop(driver, state, params, signal);
}
const browserResult = await handleBrowserAct(driver, state, input, signal);
const recordingResult = await handleRecordingAct(
driver,
execution.recording,
execution.resources,
input,
signal,
);
if (recordingResult !== undefined) {
return recordingResult;
}
const browserResult = await handleBrowserAct(driver, state, execution.resources, input, signal);
if (browserResult !== undefined) {
return browserResult;
}
+3
View File
@@ -305,6 +305,8 @@ type OpenClawCodingToolsOptions = {
modelHasVision?: boolean;
/** Mutable model-context generation used to expire screenshot coordinate frames. */
computerContextEpoch?: { value: number };
/** Registers run-owned cleanup for tools that hold node resources. */
registerRunCleanup?: (cleanup: (reason: string) => Promise<void>) => void;
/** Require explicit message targets (no implicit last-route sends). */
requireExplicitMessageTarget?: boolean;
/** Visible source replies must be sent through the message tool when set to message_tool_only. */
@@ -796,6 +798,7 @@ function createOpenClawCodingToolsInternal(options?: OpenClawCodingToolsOptions)
hasRepliedRef: options?.hasRepliedRef,
modelHasVision: options?.modelHasVision,
computerContextEpoch: options?.computerContextEpoch,
registerRunCleanup: options?.registerRunCleanup,
requireExplicitMessageTarget: options?.requireExplicitMessageTarget,
sourceReplyDeliveryMode: options?.sourceReplyDeliveryMode,
sourceReplyOnly,
@@ -122,6 +122,7 @@ export function prepareEmbeddedAttemptToolBase(params: {
const cronCreatorToolAllowlist: CronCreatorToolAllowlistEntry[] = [];
const cronCreatorToolAllowlistCaptureRef: CronToolsAllowCaptureRef = {};
const inheritedToolAllowlist: string[] = [];
const runCleanups: Array<(reason: string) => Promise<void>> = [];
const spawnWorkspaceDir =
params.effectiveCwd !== params.effectiveWorkspace
? params.resolvedWorkspace
@@ -301,6 +302,7 @@ export function prepareEmbeddedAttemptToolBase(params: {
hasRepliedRef: attempt.hasRepliedRef,
modelHasVision: attempt.model.input?.includes("image") ?? false,
computerContextEpoch,
registerRunCleanup: (cleanup) => runCleanups.push(cleanup),
requireExplicitMessageTarget:
attempt.requireExplicitMessageTarget ?? isSubagentSessionKey(attempt.sessionKey),
sourceReplyDeliveryMode: attempt.sourceReplyDeliveryMode,
@@ -361,6 +363,7 @@ export function prepareEmbeddedAttemptToolBase(params: {
localModelLeanPreserveToolNames,
replaySafetyOptions,
runtimeCapabilityProfile,
runCleanups,
toolSearchCatalogRef,
toolSearchConfig,
toolSearchControlsEnabledForRun,
@@ -99,6 +99,7 @@ export async function runEmbeddedAttempt(
let bundleLspRuntime: Awaited<ReturnType<typeof createBundleLspToolRuntime>> | undefined;
let toolSearchCatalogRef: ToolSearchCatalogRef | undefined;
let toolSearchCatalogApplied = false;
let runCleanups: Array<(reason: string) => Promise<void>> = [];
const cleanupEmbeddedPrepResourcesAfterEarlyExit = async () => {
if (toolSearchCatalogApplied) {
clearToolSearchCatalog({
@@ -228,11 +229,13 @@ export async function runEmbeddedAttempt(
computerContextEpoch,
localModelLeanEnabled,
replaySafetyOptions,
runCleanups: preparedRunCleanups,
toolSearchControlsEnabledForRun,
toolSearchRuntimeConfig,
toolsEnabled,
toolsRaw,
} = preparedToolBase;
runCleanups = preparedRunCleanups;
prepStages.mark("core-plugin-tools");
emitCorePluginToolStageSummary("core-plugin-tools", corePluginToolStages.snapshot());
const preparedBootstrap = await measureEmbeddedAgentPreparation(
@@ -536,6 +539,19 @@ export async function runEmbeddedAttempt(
}
throw error;
} finally {
const cleanupTerminal = projectAgentRunAttemptTerminal(executionState.terminal);
const cleanupReason =
cleanupTerminal.timedOut ||
cleanupTerminal.timedOutDuringCompaction ||
cleanupTerminal.timedOutDuringToolExecution
? "timeout"
: cleanupTerminal.aborted
? "cancel"
: cleanupTerminal.failed
? "error"
: "completion";
const cleanups = runCleanups.splice(0);
await Promise.allSettled(cleanups.map(async (cleanup) => await cleanup(cleanupReason)));
externalAbortController.dispose();
clearToolActivityRun(params.runId);
try {
+3
View File
@@ -160,6 +160,8 @@ export function createOpenClawTools(
sameChannelThreadRequired?: boolean;
/** Mutable model-context generation used to expire screenshot coordinate frames. */
computerContextEpoch?: { value: number };
/** Registers run-owned cleanup for tools that hold node resources. */
registerRunCleanup?: (cleanup: (reason: string) => Promise<void>) => void;
/** Internal review-run restrictions and proposal provenance. */
skillWorkshop?: SkillWorkshopRunOptions;
/** If true, nodes action="invoke" can call media-returning commands directly. */
@@ -475,6 +477,7 @@ export function createOpenClawTools(
// Run ids expire before later assistant runs can reuse a provider call id.
idempotencyScope: options?.runId,
contextEpoch: options?.computerContextEpoch,
registerRunCleanup: options?.registerRunCleanup,
}),
]),
createCronTool({
@@ -0,0 +1,92 @@
import { createHash } from "node:crypto";
import { describe, expect, it } from "vitest";
import type { AgentMessage } from "../runtime/index.js";
import { invalidateComputerFrameIfMissing, TINY_PNG_BASE64 } from "./computer-tool.test-helpers.js";
function imageIdentity(data: string, mimeType = "image/png") {
return createHash("sha256")
.update(JSON.stringify([mimeType, data]))
.digest("hex");
}
function computerToolResult(
toolCallId: string,
content: Extract<AgentMessage, { role: "toolResult" }>["content"],
) {
return {
role: "toolResult" as const,
toolCallId,
toolName: "computer",
content,
details: {},
isError: false,
timestamp: 1,
} satisfies AgentMessage;
}
function trackedContextEpoch(value: number) {
return {
value,
frameToolCallId: "shot-1",
frameImageIdentity: imageIdentity(TINY_PNG_BASE64),
};
}
function screenshotToolResult(data = TINY_PNG_BASE64) {
return computerToolResult("shot-1", [{ type: "image", data, mimeType: "image/png" }]);
}
describe("computer screenshot context binding", () => {
it("keeps coordinates valid while the tracked tool result image remains visible", () => {
const contextEpoch = trackedContextEpoch(0);
expect(
invalidateComputerFrameIfMissing({
contextEpoch,
messages: [screenshotToolResult()],
}),
).toBe(false);
expect(contextEpoch).toEqual(trackedContextEpoch(0));
});
it("expires coordinates once the final context drops the tracked image", () => {
const contextEpoch = trackedContextEpoch(0);
expect(
invalidateComputerFrameIfMissing({
contextEpoch,
messages: [computerToolResult("shot-1", [{ type: "text", text: "compacted" }])],
}),
).toBe(true);
expect(contextEpoch).toEqual({ value: 1 });
expect(invalidateComputerFrameIfMissing({ contextEpoch, messages: [] })).toBe(false);
expect(contextEpoch.value).toBe(1);
});
it.each([
[
"expires coordinates when image input is disabled at the model boundary",
trackedContextEpoch(3),
[screenshotToolResult()],
true,
{ value: 4 },
],
[
"expires coordinates when middleware swaps the tracked screenshot",
trackedContextEpoch(5),
[screenshotToolResult("AQ==")],
undefined,
{ value: 6 },
],
[
"cleans up an orphaned image identity",
{ value: 8, frameImageIdentity: imageIdentity(TINY_PNG_BASE64) },
[],
undefined,
{ value: 9 },
],
])("%s", (_name, contextEpoch, messages, imagesBlocked, expected) => {
expect(invalidateComputerFrameIfMissing({ contextEpoch, messages, imagesBlocked })).toBe(true);
expect(contextEpoch).toEqual(expected);
});
});
+17 -30
View File
@@ -1,34 +1,6 @@
/** Computer tool model-schema contract tests. */
import { describe, expect, it } from "vitest";
import type {
ComputerUseCapabilityDescriptor,
ComputerUseV2ActionName,
} from "../../plugins/computer-use-contract.js";
const { createComputerTool } = await import("./computer-tool.js");
type ComputerTool = ReturnType<typeof createComputerTool>;
function v2Descriptor(
actions: ComputerUseV2ActionName[],
overrides: Partial<ComputerUseCapabilityDescriptor> = {},
): ComputerUseCapabilityDescriptor {
return {
contractVersion: 2 as const,
provider: { id: "fixture", label: "Fixture", generation: "generation-1" },
actions,
targets: ["screen", "window", "element", "browser"] as const,
deliveryModes: ["background", "foreground"] as const,
observations: ["image", "accessibility", "browser"] as const,
features: { recording: false, agentCursor: false, multiDisplay: false },
...overrides,
};
}
function readActionEnum(tool: ComputerTool): string[] {
const schema = tool.parameters as { properties?: { action?: { enum?: string[] } } };
return schema.properties?.action?.enum ?? [];
}
import type { ComputerUseV2ActionName } from "../../plugins/computer-use-contract.js";
import { createComputerTool, readActionEnum, v2Descriptor } from "./computer-tool.test-helpers.js";
describe("createComputerTool schema", () => {
it("keeps an undeclared node on the exact v1 action list", () => {
@@ -57,6 +29,21 @@ describe("createComputerTool schema", () => {
expect(readActionEnum(tool)).toEqual(actions);
});
it("advertises resource actions only with an attempt cleanup owner", () => {
const actions: ComputerUseV2ActionName[] = ["browser_download", "start_recording"];
expect(
readActionEnum(createComputerTool({ capabilityDescriptor: v2Descriptor(actions) })),
).toEqual([]);
expect(
readActionEnum(
createComputerTool({
capabilityDescriptor: v2Descriptor(actions),
registerRunCleanup: () => {},
}),
),
).toEqual(actions);
});
it("keeps the v2 guidance provider-neutral and free of host setup instructions", () => {
const description = createComputerTool({
capabilityDescriptor: v2Descriptor([
@@ -0,0 +1,138 @@
import { vi } from "vitest";
import type {
ComputerUseCapabilityDescriptor,
ComputerUseV2ActionName,
} from "../../plugins/computer-use-contract.js";
const mocks = vi.hoisted(() => ({
listNodesMock: vi.fn(),
callGatewayToolMock: vi.fn(),
sleepMock: vi.fn(),
}));
export const listNodesMock = mocks.listNodesMock;
export const callGatewayToolMock = mocks.callGatewayToolMock;
export const sleepMock = mocks.sleepMock;
export const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII=";
export const COMPUTER_ACT_COMMAND = "computer.act";
vi.mock("./nodes-utils.js", async (importOriginal) => {
const actual = await importOriginal<typeof import("./nodes-utils.js")>();
return { ...actual, listNodes: listNodesMock };
});
vi.mock("./gateway.js", async (importOriginal) => {
const actual = await importOriginal<typeof import("./gateway.js")>();
return { ...actual, callGatewayTool: callGatewayToolMock };
});
vi.mock("../../utils/sleep.js", () => ({ sleep: sleepMock }));
export const { createComputerTool, invalidateComputerFrameIfMissing } =
await import("./computer-tool.js");
const { DEFAULT_IMAGE_MAX_DIMENSION_PX } = await import("../image-sanitization.js");
// With no config the reference width is capped at the default sanitization limit.
export const EFFECTIVE_REF_WIDTH = Math.min(1280, DEFAULT_IMAGE_MAX_DIMENSION_PX);
export function macComputerNode(overrides?: Record<string, unknown>) {
return {
nodeId: "mac-1",
displayName: "Studio",
platform: "macos",
connected: true,
commands: ["screen.snapshot", "computer.act"],
...overrides,
};
}
export function v2Descriptor(
actions: ComputerUseV2ActionName[],
overrides: Partial<ComputerUseCapabilityDescriptor> = {},
): ComputerUseCapabilityDescriptor {
return {
contractVersion: 2 as const,
provider: { id: "fixture", label: "Fixture", generation: "generation-1" },
actions,
targets: ["screen", "window", "element", "browser"] as const,
deliveryModes: ["background", "foreground"] as const,
observations: ["image", "accessibility", "browser"] as const,
features: { recording: false, agentCursor: false, multiDisplay: false },
...overrides,
};
}
export type ComputerTool = ReturnType<typeof createComputerTool>;
export type ComputerToolOptions = NonNullable<Parameters<typeof createComputerTool>[0]>;
export type ComputerActBody = {
nodeId?: string;
command?: string;
idempotencyKey?: string;
params?: Record<string, unknown>;
};
export function readActionEnum(tool: ComputerTool): string[] {
const schema = tool.parameters as { properties?: { action?: { enum?: string[] } } };
return schema.properties?.action?.enum ?? [];
}
export function screenshotPayload(screenIndex = 0, base64 = TINY_PNG_BASE64) {
return {
payload: {
format: "png",
base64,
displayFrameId: `display-${screenIndex}-frame`,
width: 1280,
height: 800,
screenIndex,
},
};
}
export function readFrameId(result: { details?: unknown }): string {
const frameId = (result.details as { frameId?: unknown } | undefined)?.frameId;
if (typeof frameId !== "string") {
throw new Error("missing frameId");
}
return frameId;
}
export function readLastComputerActParams(): Record<string, unknown> {
const call = callGatewayToolMock.mock.calls.findLast(
(entry) => (entry[2] as { command?: string }).command === COMPUTER_ACT_COMMAND,
);
const body = call?.[2] as { params?: Record<string, unknown> } | undefined;
if (!body?.params) {
throw new Error("missing computer.act request");
}
const { executionId: _executionId, ...params } = body.params;
return params;
}
export function createVisionComputerTool(options: ComputerToolOptions = {}) {
return createComputerTool({ modelHasVision: true, ...options });
}
export function resetComputerToolMocks() {
listNodesMock.mockReset();
callGatewayToolMock.mockReset();
sleepMock.mockReset();
sleepMock.mockImplementation((ms: number, signal?: AbortSignal) => {
if (signal?.aborted) {
return Promise.reject(new Error("Aborted"));
}
if (ms === 500 || !signal) {
return Promise.resolve();
}
return new Promise<void>((_resolve, reject) => {
signal.addEventListener("abort", () => reject(new Error("Aborted")), { once: true });
});
});
listNodesMock.mockResolvedValue([macComputerNode()]);
callGatewayToolMock.mockImplementation(async (_method, _opts, body) =>
(body as ComputerActBody).command === COMPUTER_ACT_COMMAND
? { payload: { ok: true } }
: screenshotPayload(),
);
}
+28 -419
View File
@@ -1,130 +1,28 @@
/**
* computer tool tests.
*
* Cover the computer.act wire mapping, frame binding, and enablement behavior.
* Node selection lives in computer-tool.node-resolution.test.ts.
*/
import { createHash } from "node:crypto";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type {
ComputerUseCapabilityDescriptor,
ComputerUseV2ActionName,
} from "../../plugins/computer-use-contract.js";
import type { AgentMessage } from "../runtime/index.js";
import {
callGatewayToolMock,
COMPUTER_ACT_COMMAND,
type ComputerActBody,
type ComputerTool,
type ComputerToolOptions,
createVisionComputerTool,
EFFECTIVE_REF_WIDTH,
listNodesMock,
macComputerNode,
readFrameId,
readLastComputerActParams,
resetComputerToolMocks,
screenshotPayload,
TINY_PNG_BASE64,
} from "./computer-tool.test-helpers.js";
const listNodesMock = vi.fn();
const callGatewayToolMock = vi.fn();
const sleepMock = vi.hoisted(() => vi.fn());
const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII=";
const COMPUTER_ACT_COMMAND = "computer.act";
const INVALID_SCROLL_AMOUNT = /scrollAmount must be a positive integer/;
const INVALID_HOLD_DURATION = /duration must be >0 and <=10 seconds/;
function imageIdentity(data: string, mimeType = "image/png") {
return createHash("sha256")
.update(JSON.stringify([mimeType, data]))
.digest("hex");
}
vi.mock("./nodes-utils.js", async (importOriginal) => {
const actual = await importOriginal<typeof import("./nodes-utils.js")>();
return { ...actual, listNodes: listNodesMock };
});
vi.mock("./gateway.js", async (importOriginal) => {
const actual = await importOriginal<typeof import("./gateway.js")>();
return { ...actual, callGatewayTool: callGatewayToolMock };
});
vi.mock("../../utils/sleep.js", () => ({ sleep: sleepMock }));
const { createComputerTool, invalidateComputerFrameIfMissing } = await import("./computer-tool.js");
const { DEFAULT_IMAGE_MAX_DIMENSION_PX } = await import("../image-sanitization.js");
// With no config the reference width is capped at the default sanitization limit.
const EFFECTIVE_REF_WIDTH = Math.min(1280, DEFAULT_IMAGE_MAX_DIMENSION_PX);
function macComputerNode(overrides?: Record<string, unknown>) {
return {
nodeId: "mac-1",
displayName: "Studio",
platform: "macos",
connected: true,
commands: ["screen.snapshot", "computer.act"],
...overrides,
};
}
function v2Descriptor(
actions: ComputerUseV2ActionName[],
overrides: Partial<ComputerUseCapabilityDescriptor> = {},
): ComputerUseCapabilityDescriptor {
return {
contractVersion: 2 as const,
provider: { id: "fixture", label: "Fixture", generation: "generation-1" },
actions,
targets: ["screen", "window", "element", "browser"] as const,
deliveryModes: ["background", "foreground"] as const,
observations: ["image", "accessibility", "browser"] as const,
features: { recording: false, agentCursor: false, multiDisplay: false },
...overrides,
};
}
function readActionEnum(tool: ComputerTool): string[] {
const schema = tool.parameters as { properties?: { action?: { enum?: string[] } } };
return schema.properties?.action?.enum ?? [];
}
function screenshotPayload(screenIndex = 0, base64 = TINY_PNG_BASE64) {
return {
payload: {
format: "png",
base64,
displayFrameId: `display-${screenIndex}-frame`,
width: 1280,
height: 800,
screenIndex,
},
};
}
function readFrameId(result: { details?: unknown }): string {
const frameId = (result.details as { frameId?: unknown } | undefined)?.frameId;
if (typeof frameId !== "string") {
throw new Error("missing frameId");
}
return frameId;
}
function readLastComputerActParams(): Record<string, unknown> {
const call = callGatewayToolMock.mock.calls.findLast(
(entry) => (entry[2] as { command?: string }).command === COMPUTER_ACT_COMMAND,
);
const body = call?.[2] as { params?: Record<string, unknown> } | undefined;
if (!body?.params) {
throw new Error("missing computer.act request");
}
return body.params;
}
function expectedAct(action: string, fields: Record<string, unknown> = {}) {
return { action, screenIndex: 0, refWidth: EFFECTIVE_REF_WIDTH, ...fields };
}
type ComputerTool = ReturnType<typeof createComputerTool>;
type ComputerToolOptions = NonNullable<Parameters<typeof createComputerTool>[0]>;
type ComputerActBody = {
nodeId?: string;
command?: string;
idempotencyKey?: string;
params?: Record<string, unknown>;
};
function createVisionComputerTool(options: ComputerToolOptions = {}) {
return createComputerTool({ modelHasVision: true, ...options });
}
function twoMacComputerNodes() {
return [
macComputerNode({ nodeId: "mac-a" }),
@@ -135,7 +33,16 @@ function twoMacComputerNodes() {
function computerActBodies(): ComputerActBody[] {
return callGatewayToolMock.mock.calls
.map((call) => call[2] as ComputerActBody)
.filter((body) => body.command === COMPUTER_ACT_COMMAND);
.filter((body) => body.command === COMPUTER_ACT_COMMAND)
.map((body) => {
if (!body.params) {
return body;
}
const { executionId: _executionId, ...params } = body.params;
const sanitizedBody = Object.assign({}, body);
sanitizedBody.params = params;
return sanitizedBody;
});
}
async function captureFrame(
@@ -235,306 +142,8 @@ async function executeComputerAction(params: Record<string, unknown>) {
return readLastComputerActParams();
}
function computerToolResult(
toolCallId: string,
content: Extract<AgentMessage, { role: "toolResult" }>["content"],
) {
return {
role: "toolResult" as const,
toolCallId,
toolName: "computer",
content,
details: {},
isError: false,
timestamp: 1,
} satisfies AgentMessage;
}
function trackedContextEpoch(value: number) {
return {
value,
frameToolCallId: "shot-1",
frameImageIdentity: imageIdentity(TINY_PNG_BASE64),
};
}
function screenshotToolResult(data = TINY_PNG_BASE64) {
return computerToolResult("shot-1", [{ type: "image", data, mimeType: "image/png" }]);
}
describe("computer screenshot context binding", () => {
it("keeps coordinates valid while the tracked tool result image remains visible", () => {
const contextEpoch = trackedContextEpoch(0);
expect(
invalidateComputerFrameIfMissing({
contextEpoch,
messages: [screenshotToolResult()],
}),
).toBe(false);
expect(contextEpoch).toEqual(trackedContextEpoch(0));
});
it("expires coordinates once the final context drops the tracked image", () => {
const contextEpoch = trackedContextEpoch(0);
expect(
invalidateComputerFrameIfMissing({
contextEpoch,
messages: [computerToolResult("shot-1", [{ type: "text", text: "compacted" }])],
}),
).toBe(true);
expect(contextEpoch).toEqual({ value: 1 });
expect(invalidateComputerFrameIfMissing({ contextEpoch, messages: [] })).toBe(false);
expect(contextEpoch.value).toBe(1);
});
it.each([
[
"expires coordinates when image input is disabled at the model boundary",
trackedContextEpoch(3),
[screenshotToolResult()],
true,
{ value: 4 },
],
[
"expires coordinates when middleware swaps the tracked screenshot",
trackedContextEpoch(5),
[screenshotToolResult("AQ==")],
undefined,
{ value: 6 },
],
[
"cleans up an orphaned image identity",
{ value: 8, frameImageIdentity: imageIdentity(TINY_PNG_BASE64) },
[],
undefined,
{ value: 9 },
],
])("%s", (_name, contextEpoch, messages, imagesBlocked, expected) => {
expect(invalidateComputerFrameIfMissing({ contextEpoch, messages, imagesBlocked })).toBe(true);
expect(contextEpoch).toEqual(expected);
});
});
describe("createComputerTool execution", () => {
beforeEach(() => {
listNodesMock.mockReset();
callGatewayToolMock.mockReset();
sleepMock.mockReset();
sleepMock.mockImplementation((ms: number, signal?: AbortSignal) => {
if (signal?.aborted) {
return Promise.reject(new Error("Aborted"));
}
if (ms === 500 || !signal) {
return Promise.resolve();
}
return new Promise<void>((_resolve, reject) => {
signal.addEventListener("abort", () => reject(new Error("Aborted")), { once: true });
});
});
listNodesMock.mockResolvedValue([macComputerNode()]);
callGatewayToolMock.mockImplementation(async (_method, _opts, body) =>
(body as ComputerActBody).command === COMPUTER_ACT_COMMAND
? { payload: { ok: true } }
: screenshotPayload(),
);
});
it("rebuilds the visible action enum from the selected node declaration", async () => {
const actions: ComputerUseV2ActionName[] = ["screenshot", "list_apps", "get_window_state"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
const tool = createVisionComputerTool();
expect(readActionEnum(tool)).toHaveLength(15);
expect(tool.description).not.toContain("get_window_state");
await tool.execute("select", { action: "screenshot" });
expect(readActionEnum(tool)).toEqual(actions);
expect(tool.description).toContain("Observe first with `get_window_state`");
});
it("projects a provider observation without taking a duplicate desktop screenshot", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValue({
payload: {
ok: true,
effect: "confirmed",
observation: {
kind: "window",
base64: TINY_PNG_BASE64,
format: "png",
width: 1,
height: 1,
observationId: "observation-1",
elements: [
{
elementRef: "element-1",
role: "button",
label: "Save",
bounds: { x: 0, y: 0, width: 1, height: 1 },
},
],
},
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
const result = await tool.execute("observe", {
action: "get_window_state",
windowRef: "window-1",
});
expect(result.content).toContainEqual(
expect.objectContaining({ type: "image", mimeType: "image/png" }),
);
expect(callGatewayToolMock).toHaveBeenCalledOnce();
expect(readLastComputerActParams()).toEqual({
action: "get_window_state",
windowRef: "window-1",
});
expect(sleepMock).not.toHaveBeenCalledWith(500, expect.anything());
});
it("rejects stale semantic references before dispatch", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state", "set_value"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValue({
payload: {
ok: true,
observation: { kind: "window", observationId: "observation-current" },
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe", { action: "get_window_state", windowRef: "window-1" });
callGatewayToolMock.mockClear();
await expect(
tool.execute("write", {
action: "set_value",
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-stale",
value: "hello",
deliveryMode: "background",
}),
).rejects.toThrow("COMPUTER_STALE_OBSERVATION");
expect(callGatewayToolMock).not.toHaveBeenCalled();
});
it("maps browser observations and opaque refs through the public tool", async () => {
const actions: ComputerUseV2ActionName[] = ["get_browser_state", "browser_pointer"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValueOnce({
payload: {
ok: true,
observation: { kind: "browser", observationId: "browser-observation-1" },
details: {
browserRef: "browser-1",
pageRef: "page-1",
elements: [{ elementRef: "element-1" }, { elementRef: "element-2" }],
},
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe-browser", {
action: "get_browser_state",
browserRef: "browser-1",
pageRef: "page-1",
snapshotFormat: "dom_refs_v1",
includeScreenshot: true,
});
expect(readLastComputerActParams()).toEqual({
action: "get_browser_state",
browserRef: "browser-1",
pageRef: "page-1",
snapshotFormat: "dom_refs_v1",
includeScreenshot: true,
});
callGatewayToolMock.mockImplementation(async (_method, _opts, body) =>
(body as ComputerActBody).command === COMPUTER_ACT_COMMAND
? { payload: { ok: true, effect: "confirmed" } }
: screenshotPayload(),
);
await tool.execute("drag-browser", {
action: "browser_pointer",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "browser-observation-1",
pointerAction: "drag",
inputRoute: "dom_event",
elementRef: "element-1",
destinationElementRef: "element-2",
});
expect(readLastComputerActParams()).toEqual({
action: "browser_pointer",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "browser-observation-1",
pointerAction: "drag",
inputRoute: "dom_event",
elementRef: "element-1",
destinationElementRef: "element-2",
});
});
it("routes an observation-bound element click without requiring coordinates", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state", "left_click"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockImplementation(async (_method, _opts, body) => {
const request = body as ComputerActBody;
if (request.command !== COMPUTER_ACT_COMMAND) {
return screenshotPayload();
}
if (request.params?.action === "get_window_state") {
return {
payload: {
ok: true,
observation: {
kind: "window",
observationId: "observation-1",
},
},
};
}
return { payload: { ok: true, effect: "confirmed" } };
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe", { action: "get_window_state", windowRef: "window-1" });
await expect(
tool.execute("click", {
action: "left_click",
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-1",
deliveryMode: "background",
}),
).resolves.toBeDefined();
expect(readLastComputerActParams()).toEqual({
action: "left_click",
screenIndex: 0,
refWidth: EFFECTIVE_REF_WIDTH,
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-1",
deliveryMode: "background",
});
});
it("rejects recording actions that remain contract-only", async () => {
const actions: ComputerUseV2ActionName[] = ["start_recording"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await expect(tool.execute("record", { action: "start_recording" })).rejects.toThrow(
"COMPUTER_CONTRACT_MISMATCH",
);
expect(callGatewayToolMock).not.toHaveBeenCalled();
});
describe("createComputerTool v1 execution", () => {
beforeEach(resetComputerToolMocks);
it.each([
[
+87 -14
View File
@@ -82,6 +82,14 @@ const LOCAL_ACTIONS = new Set<ComputerUseV2ActionName>(["screenshot", "wait"]);
const CONTRACT_ONLY_ACTIONS = new Set<ComputerUseV2ActionName>(
COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES,
);
const EXECUTION_OWNED_ACTIONS = new Set<ComputerUseV2ActionName>([
"browser_set_input_files",
"browser_download",
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
]);
const INPUT_ACTIONS = new Set<ComputerUseV2ActionName>(
COMPUTER_USE_V2_ACTION_NAMES.filter(
(action) => !LOCAL_ACTIONS.has(action) && !CONTRACT_ONLY_ACTIONS.has(action),
@@ -250,8 +258,15 @@ function createComputerToolSchema(actions: readonly ComputerUseV2ActionName[]) {
dialogAction: optionalStringEnum(["inspect", "accept", "dismiss"] as const),
dialogRef: Type.Optional(Type.String()),
promptText: Type.Optional(Type.String()),
files: Type.Optional(Type.Array(Type.String({ minLength: 1 }), { minItems: 1, maxItems: 32 })),
destinationRoot: Type.Optional(Type.String()),
resourceHandle: Type.Optional(
Type.String({ description: "Opaque node-owned Computer Use resource handle." }),
),
resourceHandles: Type.Optional(
Type.Array(Type.String({ minLength: 1 }), { minItems: 1, maxItems: 32 }),
),
recordVideo: Type.Optional(Type.Boolean()),
delayMs: Type.Optional(Type.Integer({ minimum: 0, maximum: 10_000 })),
stopOnError: Type.Optional(Type.Boolean()),
pointerAction: optionalStringEnum([
"hover",
"right_click",
@@ -367,12 +382,13 @@ function copyBrowserRefs(target: Record<string, unknown>, input: Record<string,
function buildComputerActParams(params: {
action: ComputerToolAction;
input: Record<string, unknown>;
executionId: string;
screenIndex: number;
displayFrameId?: string;
refWidth?: number;
}): ComputerActParams {
const { action, input } = params;
const wire: Record<string, unknown> = { action };
const wire: Record<string, unknown> = { action, executionId: params.executionId };
if ((COMPUTER_ACT_V1_ACTION_NAMES as readonly string[]).includes(action)) {
wire.screenIndex = params.screenIndex;
wire.refWidth = params.refWidth ?? COMPUTER_REF_WIDTH;
@@ -568,21 +584,21 @@ function buildComputerActParams(params: {
for (const key of ["observationId", "elementRef"] as const) {
wire[key] = readToolStringParam(input, key, { required: true });
}
const files = input.files;
const resourceHandles = input.resourceHandles;
if (
!Array.isArray(files) ||
files.length < 1 ||
files.length > 32 ||
files.some((file) => typeof file !== "string" || !file)
!Array.isArray(resourceHandles) ||
resourceHandles.length < 1 ||
resourceHandles.length > 32 ||
resourceHandles.some((handle) => typeof handle !== "string" || !handle)
) {
throw new Error("files must contain 1-32 non-empty paths");
throw new Error("resourceHandles must contain 1-32 opaque resource handles");
}
wire.files = files;
wire.resourceHandles = resourceHandles;
break;
}
case "browser_download": {
copyBrowserRefs(wire, input);
for (const key of ["observationId", "elementRef", "destinationRoot"] as const) {
for (const key of ["observationId", "elementRef"] as const) {
wire[key] = readToolStringParam(input, key, { required: true });
}
break;
@@ -627,6 +643,16 @@ function buildComputerActParams(params: {
wire.reason = reason;
break;
}
case "start_recording": {
copyOptionalBooleanParam(wire, input, "recordVideo");
break;
}
case "replay_trajectory": {
wire.resourceHandle = readToolStringParam(input, "resourceHandle", { required: true });
copyOptionalIntegerParam(wire, input, "delayMs", { min: 0, max: 10_000 });
copyOptionalBooleanParam(wire, input, "stopOnError");
break;
}
default:
break;
}
@@ -692,6 +718,7 @@ const READ_ONLY_COMPUTER_ACT_ACTIONS = new Set<ComputerUseV2ActionName>([
"get_window_state",
"zoom",
"get_browser_state",
"get_recording_state",
]);
function parseComputerActPayload(value: unknown): ComputerActResult {
@@ -794,9 +821,11 @@ async function captureScreenshot(params: {
nodeId: string;
screenIndex: number;
refWidth: number;
executionId: string;
signal?: AbortSignal;
}): Promise<ScreenshotCapture> {
const commandParams: ScreenSnapshotParams = {
executionId: params.executionId,
screenIndex: params.screenIndex,
maxWidth: params.refWidth,
quality: SCREENSHOT_QUALITY,
@@ -1022,11 +1051,18 @@ export function createComputerTool(options?: {
contextEpoch?: ComputerContextEpoch;
/** Preselected node declaration, when tool preparation already resolved one. */
capabilityDescriptor?: ComputerUseCapabilityDescriptor;
/** Attempt owner for deterministic provider-execution cleanup. */
registerRunCleanup?: (cleanup: (reason: string) => Promise<void>) => void;
}): AnyAgentTool {
const executionId = crypto.randomUUID();
const availableActions = (actions: readonly ComputerUseV2ActionName[]) =>
options?.registerRunCleanup
? actions
: actions.filter((action) => !EXECUTION_OWNED_ACTIONS.has(action));
const configuredLimits = resolveImageSanitizationLimits(options?.config);
const referenceWidth = resolveReferenceWidth(configuredLimits);
const parameterSchema = createComputerToolSchema(
options?.capabilityDescriptor?.actions ?? COMPUTER_TOOL_ACTIONS,
availableActions(options?.capabilityDescriptor?.actions ?? COMPUTER_TOOL_ACTIONS),
);
let selectedCapabilities = options?.capabilityDescriptor;
let selectedCapabilityNodeId: string | undefined;
@@ -1048,7 +1084,7 @@ export function createComputerTool(options?: {
selectedCapabilities?.provider.generation !== next?.provider.generation;
selectedCapabilityNodeId = node.nodeId;
selectedCapabilities = next;
replaceParameterSchema(next?.actions ?? COMPUTER_TOOL_ACTIONS);
replaceParameterSchema(availableActions(next?.actions ?? COMPUTER_TOOL_ACTIONS));
tool.description = buildComputerToolDescription(next);
if (changed) {
observationState = undefined;
@@ -1108,6 +1144,36 @@ export function createComputerTool(options?: {
);
return result;
};
const executionNodes = new Map<string, GatewayCallOptions>();
let disposePromise: Promise<void> | undefined;
const dispose = async (reason: string): Promise<void> => {
if (disposePromise) {
return await disposePromise;
}
disposePromise = opQueue
.catch(() => {})
.then(async () => {
const nodes = [...executionNodes.entries()];
executionNodes.clear();
await Promise.allSettled(
nodes.map(async ([nodeId, gatewayOpts]) => {
await invokeNodeCommand({
gatewayOpts,
nodeId,
command: COMPUTER_ACT_COMMAND,
commandParams: {
action: "__close_execution",
executionId,
reason,
},
idempotencyKey: `computer.close:${executionId}:${nodeId}`,
});
}),
);
});
return await disposePromise;
};
options?.registerRunCleanup?.(dispose);
const tool: AnyAgentTool = {
label: "Computer",
name: "computer",
@@ -1165,7 +1231,10 @@ export function createComputerTool(options?: {
}
const capabilitiesForNode =
selectedCapabilityNodeId === nodeId ? selectedCapabilities : undefined;
const advertisedActions = capabilitiesForNode?.actions ?? COMPUTER_TOOL_ACTIONS;
executionNodes.set(nodeId, gatewayOpts);
const advertisedActions = availableActions(
capabilitiesForNode?.actions ?? COMPUTER_TOOL_ACTIONS,
);
if (!advertisedActions.includes(action)) {
throw new Error(
`${COMPUTER_CONTRACT_MISMATCH}: node ${nodeId} does not advertise action ${action}`,
@@ -1365,6 +1434,7 @@ export function createComputerTool(options?: {
nodeId,
screenIndex,
refWidth: referenceWidth,
executionId,
signal,
});
return await screenshotResult(capture, []);
@@ -1383,6 +1453,7 @@ export function createComputerTool(options?: {
nodeId,
screenIndex,
refWidth: referenceWidth,
executionId,
signal,
});
return await screenshotResult(capture, [`waited ${seconds}s`]);
@@ -1397,6 +1468,7 @@ export function createComputerTool(options?: {
const wireParams = buildComputerActParams({
action,
input: params,
executionId,
screenIndex,
displayFrameId: frameForNode?.displayFrameId,
refWidth: referenceWidth,
@@ -1462,6 +1534,7 @@ export function createComputerTool(options?: {
nodeId,
screenIndex,
refWidth: referenceWidth,
executionId,
signal,
});
return await screenshotResult(capture, [computerActResultText(action, actResult)]);
+267
View File
@@ -0,0 +1,267 @@
import { beforeEach, describe, expect, it } from "vitest";
import type { ComputerUseV2ActionName } from "../../plugins/computer-use-contract.js";
import {
callGatewayToolMock,
COMPUTER_ACT_COMMAND,
type ComputerActBody,
createVisionComputerTool,
EFFECTIVE_REF_WIDTH,
listNodesMock,
macComputerNode,
readActionEnum,
readLastComputerActParams,
resetComputerToolMocks,
screenshotPayload,
sleepMock,
TINY_PNG_BASE64,
v2Descriptor,
} from "./computer-tool.test-helpers.js";
describe("createComputerTool v2 execution", () => {
beforeEach(resetComputerToolMocks);
it("rebuilds the visible action enum from the selected node declaration", async () => {
const actions: ComputerUseV2ActionName[] = ["screenshot", "list_apps", "get_window_state"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
const tool = createVisionComputerTool();
expect(readActionEnum(tool)).toHaveLength(15);
expect(tool.description).not.toContain("get_window_state");
await tool.execute("select", { action: "screenshot" });
expect(readActionEnum(tool)).toEqual(actions);
expect(tool.description).toContain("Observe first with `get_window_state`");
});
it("projects a provider observation without taking a duplicate desktop screenshot", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValue({
payload: {
ok: true,
effect: "confirmed",
observation: {
kind: "window",
base64: TINY_PNG_BASE64,
format: "png",
width: 1,
height: 1,
observationId: "observation-1",
elements: [
{
elementRef: "element-1",
role: "button",
label: "Save",
bounds: { x: 0, y: 0, width: 1, height: 1 },
},
],
},
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
const result = await tool.execute("observe", {
action: "get_window_state",
windowRef: "window-1",
});
expect(result.content).toContainEqual(
expect.objectContaining({ type: "image", mimeType: "image/png" }),
);
expect(callGatewayToolMock).toHaveBeenCalledOnce();
expect(readLastComputerActParams()).toEqual({
action: "get_window_state",
windowRef: "window-1",
});
expect(sleepMock).not.toHaveBeenCalledWith(500, expect.anything());
});
it("rejects stale semantic references before dispatch", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state", "set_value"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValue({
payload: {
ok: true,
observation: { kind: "window", observationId: "observation-current" },
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe", { action: "get_window_state", windowRef: "window-1" });
callGatewayToolMock.mockClear();
await expect(
tool.execute("write", {
action: "set_value",
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-stale",
value: "hello",
deliveryMode: "background",
}),
).rejects.toThrow("COMPUTER_STALE_OBSERVATION");
expect(callGatewayToolMock).not.toHaveBeenCalled();
});
it("maps browser observations and opaque refs through the public tool", async () => {
const actions: ComputerUseV2ActionName[] = ["get_browser_state", "browser_pointer"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockResolvedValueOnce({
payload: {
ok: true,
observation: { kind: "browser", observationId: "browser-observation-1" },
details: {
browserRef: "browser-1",
pageRef: "page-1",
elements: [{ elementRef: "element-1" }, { elementRef: "element-2" }],
},
},
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe-browser", {
action: "get_browser_state",
browserRef: "browser-1",
pageRef: "page-1",
snapshotFormat: "dom_refs_v1",
includeScreenshot: true,
});
expect(readLastComputerActParams()).toEqual({
action: "get_browser_state",
browserRef: "browser-1",
pageRef: "page-1",
snapshotFormat: "dom_refs_v1",
includeScreenshot: true,
});
callGatewayToolMock.mockImplementation(async (_method, _opts, body) =>
(body as ComputerActBody).command === COMPUTER_ACT_COMMAND
? { payload: { ok: true, effect: "confirmed" } }
: screenshotPayload(),
);
await tool.execute("drag-browser", {
action: "browser_pointer",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "browser-observation-1",
pointerAction: "drag",
inputRoute: "dom_event",
elementRef: "element-1",
destinationElementRef: "element-2",
});
expect(readLastComputerActParams()).toEqual({
action: "browser_pointer",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "browser-observation-1",
pointerAction: "drag",
inputRoute: "dom_event",
elementRef: "element-1",
destinationElementRef: "element-2",
});
});
it("routes an observation-bound element click without requiring coordinates", async () => {
const actions: ComputerUseV2ActionName[] = ["get_window_state", "left_click"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
callGatewayToolMock.mockImplementation(async (_method, _opts, body) => {
const request = body as ComputerActBody;
if (request.command !== COMPUTER_ACT_COMMAND) {
return screenshotPayload();
}
if (request.params?.action === "get_window_state") {
return {
payload: {
ok: true,
observation: {
kind: "window",
observationId: "observation-1",
},
},
};
}
return { payload: { ok: true, effect: "confirmed" } };
});
const tool = createVisionComputerTool({ capabilityDescriptor: v2Descriptor(actions) });
await tool.execute("observe", { action: "get_window_state", windowRef: "window-1" });
await expect(
tool.execute("click", {
action: "left_click",
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-1",
deliveryMode: "background",
}),
).resolves.toBeDefined();
expect(readLastComputerActParams()).toEqual({
action: "left_click",
screenIndex: 0,
refWidth: EFFECTIVE_REF_WIDTH,
windowRef: "window-1",
elementRef: "element-1",
observationId: "observation-1",
deliveryMode: "background",
});
});
it("maps the recording family through opaque resource parameters", async () => {
const actions: ComputerUseV2ActionName[] = [
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
const tool = createVisionComputerTool({
capabilityDescriptor: v2Descriptor(actions),
registerRunCleanup: () => {},
});
const resourceHandle = "openclaw:computer-resource:v1:123e4567-e89b-42d3-a456-426614174000";
await tool.execute("record", { action: "start_recording", recordVideo: true });
expect(readLastComputerActParams()).toEqual({ action: "start_recording", recordVideo: true });
await tool.execute("replay", {
action: "replay_trajectory",
resourceHandle,
delayMs: 25,
stopOnError: false,
});
expect(readLastComputerActParams()).toEqual({
action: "replay_trajectory",
resourceHandle,
delayMs: 25,
stopOnError: false,
});
});
it("closes the exact host execution through attempt-owned cleanup", async () => {
const actions: ComputerUseV2ActionName[] = ["start_recording"];
listNodesMock.mockResolvedValue([macComputerNode({ computerUse: v2Descriptor(actions) })]);
let cleanup: ((reason: string) => Promise<void>) | undefined;
const tool = createVisionComputerTool({
capabilityDescriptor: v2Descriptor(actions),
registerRunCleanup: (registered) => {
cleanup = registered;
},
});
await tool.execute("record", { action: "start_recording" });
const start = callGatewayToolMock.mock.calls
.map((call) => call[2] as ComputerActBody)
.findLast((body) => body.command === COMPUTER_ACT_COMMAND);
if (!start?.params) {
throw new Error("missing start_recording node invocation");
}
const executionId = start.params.executionId;
expect(executionId).toEqual(expect.any(String));
await cleanup?.("completion");
const close = callGatewayToolMock.mock.calls.at(-1)?.[2] as ComputerActBody;
expect(close.params).toEqual({
action: "__close_execution",
executionId,
reason: "completion",
});
});
});
+17
View File
@@ -6,6 +6,7 @@ import { getPluginRuntimeGatewayRequestScope } from "../plugins/runtime/gateway-
import {
invokeRegisteredNodeHostCommand,
listRegisteredNodeHostCapsAndCommands,
notifyRegisteredNodeHostCommandDisconnect,
watchRegisteredNodeHostCommandAvailability,
} from "./plugin-node-host.js";
@@ -241,6 +242,22 @@ describe("plugin node-host registry", () => {
expect(scopedRegistry).toHaveBeenNthCalledWith(3, registry);
});
it("notifies each shared plugin disconnect owner once", async () => {
const onDisconnect = vi.fn(async () => {});
const registry = createEmptyPluginRegistry();
registry.nodeHostCommands = ["screen.snapshot", "computer.act"].map((command) => ({
pluginId: "computer",
pluginName: "Computer",
command: { command, onDisconnect, handle: vi.fn(async () => "{}") },
source: "test",
}));
setActivePluginRegistry(registry);
await notifyRegisteredNodeHostCommandDisconnect();
expect(onDisconnect).toHaveBeenCalledOnce();
});
it("dispatches plugin-declared node-host commands", async () => {
const handle = vi.fn(async (paramsJSON?: string | null) => {
expect(getPluginRuntimeGatewayRequestScope()?.pluginRegistry).toBe(registry);
+27
View File
@@ -122,6 +122,33 @@ export function watchRegisteredNodeHostCommandAvailability(
});
}
/** Release plugin command state before a reconnected Gateway can invoke it again. */
export async function notifyRegisteredNodeHostCommandDisconnect(): Promise<void> {
const registry = resolveNodeHostPluginRegistry();
const callbacks = new Set(
(registry?.nodeHostCommands ?? [])
.map((entry) => entry.command.onDisconnect)
.filter((callback): callback is () => Promise<void> | void => callback !== undefined),
);
await withPluginRuntimeRegistryScope(registry, async () => {
const results = await Promise.allSettled(
[...callbacks].map(async (callback) => await callback()),
);
const failures = results.flatMap((result) =>
result.status === "rejected" ? [result.reason] : [],
);
if (failures.length === 1) {
const failure = failures[0];
throw failure instanceof Error
? failure
: new Error("node-host plugin disconnect cleanup failed", { cause: failure });
}
if (failures.length > 1) {
throw new AggregateError(failures, "node-host plugin disconnect cleanup failed");
}
});
}
function isProviderSafeToolName(value: string): boolean {
return /^[A-Za-z][A-Za-z0-9_-]{0,63}$/.test(value);
}
+25 -12
View File
@@ -36,6 +36,7 @@ import {
ensureNodeHostPluginRegistry,
isRegisteredNodeHostCommandDuplex,
listRegisteredNodeHostCapsAndCommands,
notifyRegisteredNodeHostCommandDisconnect,
watchRegisteredNodeHostCommandAvailability,
} from "./plugin-node-host.js";
import { scanNodeHostedSkills } from "./skills.js";
@@ -346,6 +347,7 @@ export async function prepareNodeHostRuntime(params?: {
}
const skillBins = new SkillBinsCache(client, pathEnv);
const activeInvokes = new Map<string, ActiveNodeInvoke>();
let pluginDisconnectCleanup: Promise<void> = Promise.resolve();
const pluginCommandContext: OpenClawPluginNodeHostCommandContext = {
sendNodeEvent: async (event, payload) =>
await client.request("node.event", buildNodeEventParams(event, payload)),
@@ -397,6 +399,7 @@ export async function prepareNodeHostRuntime(params?: {
}
return {
async invoke(frame) {
await pluginDisconnectCleanup;
const duplexCommand = duplexEnabled && isRegisteredNodeHostCommandDuplex(frame.command);
const progressEnabled = duplexCommand || frame.command === NODE_DESKTOP_STREAM_COMMAND;
const controller = new AbortController();
@@ -487,6 +490,11 @@ export async function prepareNodeHostRuntime(params?: {
active.controller.abort();
}
activeInvokes.clear();
pluginDisconnectCleanup = pluginDisconnectCleanup
.then(async () => await notifyRegisteredNodeHostCommandDisconnect())
.catch((error: unknown) => {
logDebug(`node-host: plugin disconnect cleanup failed: ${String(error)}`);
});
},
updateGatewayConnection(connection) {
gatewayConnection = connection;
@@ -505,20 +513,25 @@ export async function prepareNodeHostRuntime(params?: {
}
// Startup observes this signal before either independent owner is joined.
mcpAbort.abort();
const disconnectClose = pluginDisconnectCleanup;
const supervisorClose = Promise.resolve().then(() => workerSupervisor?.close());
const mcpClose = startup.then((resolved) => resolved.close());
closePromise = Promise.allSettled([supervisorClose, mcpClose]).then((results) => {
const errors = [
...preludeErrors,
...results.flatMap((result) => (result.status === "rejected" ? [result.reason] : [])),
];
if (errors.length === 1) {
throw errors[0];
}
if (errors.length > 1) {
throw new AggregateError(errors, "node-host runtime close failed");
}
});
closePromise = Promise.allSettled([disconnectClose, supervisorClose, mcpClose]).then(
(results) => {
const errors = [
...preludeErrors,
...results.flatMap((result) =>
result.status === "rejected" ? [result.reason] : [],
),
];
if (errors.length === 1) {
throw errors[0];
}
if (errors.length > 1) {
throw new AggregateError(errors, "node-host runtime close failed");
}
},
);
return closePromise;
},
};
+109 -14
View File
@@ -141,15 +141,56 @@ describe("Computer Use wire contract", () => {
).toThrow("COMPUTER_INVALID_REQUEST");
});
it("keeps only the unimplemented recording family contract-gated", () => {
expect(COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES).toEqual([
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
]);
for (const action of COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES) {
expect(() => parseComputerActParamsJSON(JSON.stringify({ action }))).toThrow(
it("accepts the portable recording family without native path or helper inputs", () => {
const resourceHandle = "openclaw:computer-resource:v1:123e4567-e89b-42d3-a456-426614174000";
expect(COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES).toEqual([]);
for (const input of [
{ action: "get_recording_state" },
{ action: "start_recording", recordVideo: true },
{ action: "stop_recording" },
{ action: "replay_trajectory", resourceHandle, delayMs: 25, stopOnError: false },
{
action: "browser_set_input_files",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "observation-1",
elementRef: "element-1",
resourceHandles: [resourceHandle],
},
{
action: "browser_download",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "observation-1",
elementRef: "element-1",
},
]) {
expect(parseComputerActParamsJSON(JSON.stringify(input))).toEqual(input);
}
for (const input of [
{ action: "start_recording", output_dir: "/tmp/recording" },
{ action: "start_recording", helperPath: "/tmp/ffmpeg" },
{ action: "replay_trajectory", dir: "../outside" },
{ action: "replay_trajectory", ffmpegPath: "/tmp/ffmpeg" },
{
action: "browser_set_input_files",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "observation-1",
elementRef: "element-1",
files: ["/tmp/input.txt"],
},
{
action: "browser_download",
browserRef: "browser-1",
pageRef: "page-1",
observationId: "observation-1",
elementRef: "element-1",
destinationRoot: "/tmp/downloads",
},
]) {
expect(() => parseComputerActParamsJSON(JSON.stringify(input))).toThrow(
"COMPUTER_INVALID_REQUEST",
);
}
@@ -221,6 +262,7 @@ describe("Computer Use wire contract", () => {
describe("Computer Use provider registration", () => {
it("registers one command pair and dispatches both through one execution", async () => {
const executionId = "123e4567-e89b-42d3-a456-426614174000";
const commands: OpenClawPluginNodeHostCommand[] = [];
const snapshot = vi.fn(async () => "snapshot");
const act = vi.fn(async () => "act");
@@ -256,16 +298,69 @@ describe("Computer Use provider registration", () => {
const signal = new AbortController().signal;
const context = { sendNodeEvent: vi.fn(), sessionKey: "session-1", signal };
await expect(commands[0]!.handle("{}", undefined, context)).resolves.toBe("snapshot");
await expect(commands[1]!.handle("{}", undefined, context)).resolves.toBe("act");
const paramsJSON = JSON.stringify({ executionId });
await expect(commands[0]!.handle(paramsJSON, undefined, context)).resolves.toBe("snapshot");
await expect(commands[1]!.handle(paramsJSON, undefined, context)).resolves.toBe("act");
expect(openExecution).toHaveBeenCalledOnce();
expect(openExecution).toHaveBeenCalledWith({ sessionKey: "session-1" });
expect(snapshot).toHaveBeenCalledWith("{}", signal);
expect(act).toHaveBeenCalledWith("{}", signal);
expect(openExecution).toHaveBeenCalledWith({ executionId, sessionKey: "session-1" });
expect(snapshot).toHaveBeenCalledWith(paramsJSON, signal);
expect(act).toHaveBeenCalledWith(paramsJSON, signal);
const stop = commands[0]!.watchAvailability?.({ config: {} as never, env: {} }, vi.fn());
stop?.();
await vi.waitFor(() => expect(close).toHaveBeenCalledWith("node-host-stop"));
expect(stopWatching).toHaveBeenCalledOnce();
});
it("rejects cross-execution control and closes only the exact host execution", async () => {
const firstId = "123e4567-e89b-42d3-a456-426614174000";
const secondId = "223e4567-e89b-42d3-a456-426614174000";
const commands: OpenClawPluginNodeHostCommand[] = [];
const closes: string[] = [];
const openExecution = vi.fn(async () => ({
snapshot: vi.fn(async () => "snapshot"),
act: vi.fn(async () => "act"),
close: vi.fn(async (reason: string) => {
closes.push(reason);
}),
}));
const provider: ComputerUseProvider = {
id: "fixture",
label: "Fixture",
capabilities: () => ({
contractVersion: 2,
provider: { id: "fixture", label: "Fixture", generation: "generation-1" },
actions: ["start_recording", "stop_recording"],
targets: ["screen"],
deliveryModes: ["foreground"],
observations: ["image"],
features: { recording: true, agentCursor: false, multiDisplay: false },
}),
isAvailable: () => true,
openExecution,
};
registerComputerUseProvider(
{ registerNodeHostCommand: (command) => commands.push(command) },
provider,
);
const computer = commands.find((command) => command.command === "computer.act")!;
await expect(
computer.handle(JSON.stringify({ action: "start_recording", executionId: firstId })),
).resolves.toBe("act");
await expect(
computer.handle(JSON.stringify({ action: "stop_recording", executionId: secondId })),
).rejects.toThrow("COMPUTER_HOST_BUSY");
expect(openExecution).toHaveBeenCalledOnce();
await computer.handle(
JSON.stringify({ action: "__close_execution", executionId: firstId, reason: "completion" }),
);
await expect(
computer.handle(JSON.stringify({ action: "start_recording", executionId: secondId })),
).resolves.toBe("act");
expect(openExecution).toHaveBeenCalledTimes(2);
await commands.find((command) => command.command === "screen.snapshot")!.onDisconnect?.();
expect(closes).toEqual(["completion", "gateway-disconnect"]);
});
});
+117 -26
View File
@@ -1,3 +1,4 @@
import { randomUUID } from "node:crypto";
import { type Static, type TSchema, Type } from "typebox";
import { Compile } from "typebox/compile";
import type {
@@ -55,12 +56,8 @@ export const COMPUTER_USE_V1_ACTION_NAMES = COMPUTER_USE_V2_ACTION_NAMES.slice(0
export const COMPUTER_ACT_V1_ACTION_NAMES = COMPUTER_USE_V2_ACTION_NAMES.slice(1, 14);
export const COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES = [
"get_recording_state",
"start_recording",
"stop_recording",
"replay_trajectory",
] as const satisfies readonly ComputerUseV2ActionName[];
export const COMPUTER_USE_CONTRACT_ONLY_ACTION_NAMES =
[] as const satisfies readonly ComputerUseV2ActionName[];
export const COMPUTER_CONTRACT_MISMATCH = "COMPUTER_CONTRACT_MISMATCH";
export const COMPUTER_STALE_OBSERVATION = "COMPUTER_STALE_OBSERVATION";
@@ -74,6 +71,10 @@ const ESCALATION_REASONS = [
"no_window_target",
"other",
] as const;
const COMPUTER_RESOURCE_HANDLE_PATTERN =
"^openclaw:computer-resource:v1:[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$";
const COMPUTER_EXECUTION_ID_PATTERN =
"^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$";
const optionalScreenFields = {
screenIndex: Type.Optional(Type.Integer({ minimum: 0 })),
@@ -94,6 +95,7 @@ function actionObject<const Properties extends object>(
return Type.Object(
{
action: Type.Enum(actions, { type: "string" }),
executionId: Type.Optional(Type.String({ pattern: COMPUTER_EXECUTION_ID_PATTERN })),
...properties,
},
{ additionalProperties: false },
@@ -275,14 +277,16 @@ export const ComputerActParamsSchema = Type.Union([
pageRef: Type.String({ minLength: 1 }),
observationId: Type.String({ minLength: 1 }),
elementRef: Type.String({ minLength: 1 }),
files: Type.Array(Type.String({ minLength: 1 }), { minItems: 1, maxItems: 32 }),
resourceHandles: Type.Array(Type.String({ pattern: COMPUTER_RESOURCE_HANDLE_PATTERN }), {
minItems: 1,
maxItems: 32,
}),
}),
actionObject(["browser_download"], {
browserRef: Type.String({ minLength: 1 }),
pageRef: Type.String({ minLength: 1 }),
observationId: Type.String({ minLength: 1 }),
elementRef: Type.String({ minLength: 1 }),
destinationRoot: Type.String({ minLength: 1 }),
}),
actionObject(["browser_pointer"], {
browserRef: Type.String({ minLength: 1 }),
@@ -304,6 +308,15 @@ export const ComputerActParamsSchema = Type.Union([
actionObject(["escalate_scope"], {
reason: Type.Enum(ESCALATION_REASONS, { type: "string" }),
}),
actionObject(["get_recording_state", "stop_recording"], {}),
actionObject(["start_recording"], {
recordVideo: Type.Optional(Type.Boolean()),
}),
actionObject(["replay_trajectory"], {
resourceHandle: Type.String({ pattern: COMPUTER_RESOURCE_HANDLE_PATTERN }),
delayMs: Type.Optional(Type.Integer({ minimum: 0, maximum: 10_000 })),
stopOnError: Type.Optional(Type.Boolean()),
}),
]);
// Bound provider-controlled result collections before they cross the node-host wire contract.
@@ -418,6 +431,7 @@ export const ComputerUseCapabilityDescriptorSchema = Type.Object(
/** Canonical inner payload accepted by the `screen.snapshot` node command. */
export const ScreenSnapshotParamsSchema = Type.Object(
{
executionId: Type.Optional(Type.String({ pattern: COMPUTER_EXECUTION_ID_PATTERN })),
screenIndex: Type.Optional(Type.Integer({ minimum: 0 })),
maxWidth: Type.Optional(Type.Integer({ minimum: 1 })),
quality: Type.Optional(Type.Number()),
@@ -538,7 +552,10 @@ export type ComputerUseProvider = {
context: OpenClawPluginNodeHostCommandAvailabilityContext,
onChange: () => void,
) => (() => void) | void;
openExecution(context: { sessionKey?: string }): Promise<ComputerUseExecution>;
openExecution(context: {
executionId: string;
sessionKey?: string;
}): Promise<ComputerUseExecution>;
};
// Structural registration surface built from leaf node-host types only: importing
@@ -553,29 +570,69 @@ export function registerComputerUseProvider(
api: ComputerUseRegistrationApi,
provider: ComputerUseProvider,
): void {
let executionPromise: Promise<ComputerUseExecution> | undefined;
let execution: { id: string; promise: Promise<ComputerUseExecution> } | undefined;
let closingPromise: Promise<void> = Promise.resolve();
const getExecution = (context?: OpenClawPluginNodeHostCommandContext) => {
if (!executionPromise) {
const executionEnvelopeFromParams = (paramsJSON: string | null | undefined) => {
let value: unknown;
try {
value = JSON.parse(paramsJSON ?? "{}");
} catch {
throw new Error("COMPUTER_INVALID_REQUEST: params must be valid JSON");
}
const executionId =
value && typeof value === "object" && !Array.isArray(value)
? (value as { executionId?: unknown }).executionId
: undefined;
if (executionId === undefined) {
return { executionId: undefined, value };
}
if (
typeof executionId !== "string" ||
!new RegExp(COMPUTER_EXECUTION_ID_PATTERN, "u").test(executionId)
) {
throw new Error("COMPUTER_INVALID_REQUEST: executionId is required");
}
return { executionId, value };
};
const getExecution = async (
paramsJSON: string | null | undefined,
context?: OpenClawPluginNodeHostCommandContext,
) => {
const { executionId } = executionEnvelopeFromParams(paramsJSON);
if (!executionId) {
throw new Error("COMPUTER_INVALID_REQUEST: executionId is required");
}
await closingPromise;
if (execution && execution.id !== executionId) {
throw new Error("COMPUTER_HOST_BUSY: another provider execution owns this computer");
}
if (!execution) {
const opened = provider.openExecution(
context?.sessionKey ? { sessionKey: context.sessionKey } : {},
context?.sessionKey ? { executionId, sessionKey: context.sessionKey } : { executionId },
);
// A failed open must not wedge the provider behind a cached rejection;
// the next command call retries openExecution instead.
opened.catch(() => {
if (executionPromise === opened) {
executionPromise = undefined;
if (execution?.promise === opened) {
execution = undefined;
}
});
executionPromise = opened;
execution = { id: executionId, promise: opened };
}
return executionPromise;
return execution.promise;
};
const closeExecution = async (reason: string) => {
const current = executionPromise;
executionPromise = undefined;
const closeExecution = async (executionId: string | undefined, reason: string) => {
await closingPromise;
const current = execution;
if (!current || (executionId !== undefined && current.id !== executionId)) {
return;
}
execution = undefined;
if (current) {
await (await current).close(reason);
const close = current.promise.then(async (opened) => await opened.close(reason));
closingPromise = close.catch(() => {});
await close;
}
};
@@ -588,11 +645,27 @@ export function registerComputerUseProvider(
const stopWatching = provider.watchAvailability?.(context, onChange);
return () => {
stopWatching?.();
void closeExecution("node-host-stop");
void closeExecution(undefined, "node-host-stop");
};
},
handle: async (paramsJSON, _io, context) =>
await (await getExecution(context)).snapshot(paramsJSON, context?.signal),
onDisconnect: async () => await closeExecution(undefined, "gateway-disconnect"),
handle: async (paramsJSON, _io, context) => {
const envelope = executionEnvelopeFromParams(paramsJSON);
if (envelope.executionId) {
return await (
await getExecution(paramsJSON, context)
).snapshot(paramsJSON, context?.signal);
}
const executionId = randomUUID();
const opened = await provider.openExecution(
context?.sessionKey ? { executionId, sessionKey: context.sessionKey } : { executionId },
);
try {
return await opened.snapshot(paramsJSON, context?.signal);
} finally {
await opened.close("snapshot-complete");
}
},
});
api.registerNodeHostCommand({
command: "computer.act",
@@ -600,8 +673,26 @@ export function registerComputerUseProvider(
dangerous: true,
computerUse: () => provider.capabilities(),
isAvailable: () => provider.isAvailable(),
handle: async (paramsJSON, _io, context) =>
await (await getExecution(context)).act(paramsJSON, context?.signal),
handle: async (paramsJSON, _io, context) => {
const envelope = executionEnvelopeFromParams(paramsJSON);
if (!envelope.executionId) {
throw new Error("COMPUTER_INVALID_REQUEST: executionId is required");
}
if (
envelope.value &&
typeof envelope.value === "object" &&
!Array.isArray(envelope.value) &&
(envelope.value as { action?: unknown }).action === "__close_execution"
) {
const reason = (envelope.value as { reason?: unknown }).reason;
await closeExecution(
envelope.executionId,
typeof reason === "string" && reason.trim() ? reason.slice(0, 64) : "completion",
);
return JSON.stringify({ ok: true });
}
return await (await getExecution(paramsJSON, context)).act(paramsJSON, context?.signal);
},
});
// The provider plugin must also register its dangerous `computer.act` invoke
// policy with the full plugin API. Forgetting it fails closed: the Gateway
+2
View File
@@ -34,6 +34,8 @@ type OpenClawPluginNodeHostCommandBase = {
context: OpenClawPluginNodeHostCommandAvailabilityContext,
onChange: () => void,
) => (() => void) | void;
/** Release command-owned state when the active Gateway connection closes. */
onDisconnect?: () => Promise<void> | void;
/** Optional Computer Use declaration published with this command's node manifest. */
computerUse?: (context: OpenClawPluginNodeHostCommandAvailabilityContext) => unknown;
agentTool?: {