ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-27 12:56:01 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(workers): run OpenClaw and Codex on the same cloud profile (#127752)

Browse Source 
* feat(workers): run both harnesses on one cloud node profile

* docs(workers): explain shared OpenClaw and Codex cloud profiles

* fix(workers): preserve trusted Codex cloud enrollment

* docs(codex): clarify bundled cloud plugin eligibility

* fix(workers): honor Crabbox bootstrap retries

* fix(workers): allow cloud profile setup to finish

* fix(workers): reject unsupported cloud execution modes

* refactor(workers): split cloud placement modules

* fix(workers): preserve completed container results
This commit is contained in:
Peter Steinberger
2026-08-25 00:13:29 -07:00
committed by GitHub
parent 5ca5fa6f9e
commit 8a1ff3966b
78 changed files with 2576 additions and 827 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gateway/cloud-workers.md
+18 -14
View File
@@ -7,7 +7,7 @@ status: active
doc-schema-version: 1
---
Cloud workers move a session's coding work onto a throwaway cloud machine while the session stays visible in the sidebar and its transcript remains owned by the Gateway. The bundled Crabbox provider boots the box, runs profile setup, and starts `openclaw connect --ephemeral`. In OpenClaw `worker-turn` mode, the enrolled node receives the Gateway's pinned worker bundle and hosts a restricted `openclaw worker` child. Eligible paired devices can instead carry Codex `remote-exec` without launching an OpenClaw worker child.
Cloud workers move a session's coding work onto a throwaway cloud machine while the session stays visible in the sidebar and its transcript remains owned by the Gateway. The bundled Crabbox provider boots the box, runs profile setup, and starts `openclaw connect --ephemeral`. One configured Crabbox profile supports both OpenClaw `worker-turn` and Codex `remote-exec` over the same enrolled outbound-node transport. OpenClaw launches a restricted `openclaw worker` child; Codex runs its managed exec-server on the node while keeping app-server and model authentication on the Gateway.
Enrollment is environment-owned and replay-safe. The Gateway persists one setup identity before provider allocation, binds the first authenticated device identity to that exact environment, and reuses the durable device token when provisioning resumes. Initial enrollment and replay both enable worker hosting only for that node process; they do not change durable worker-host configuration. Reclaim or destroy releases the cloud lease and removes the environment-owned node pairing.
@@ -22,14 +22,14 @@ Cloud workers are opt-in. Until you configure a profile, clients hide the Cloud
| Concern | OpenClaw `worker-turn` mode | Codex `remote-exec` mode |
| ---------------------------------- | ---------------------------------------------------- | ------------------------------------------------------- |
| Agent runtime and turn loop | Cloud box (`openclaw worker`) | Gateway (Codex app-server) |
| Command, filesystem, and HTTP work | Cloud box | Paired device or SSH-backed provider sandbox |
| Command, filesystem, and HTTP work | Cloud box | Cloud node, paired device, or SSH-backed provider |
| Model inference and provider auth | Gateway, proxied by `{provider, model}` reference | Gateway, including ChatGPT subscription or API-key auth |
| Transcript and live session state | Gateway, fed by the worker's replayable event stream | Gateway through the normal local harness path |
| Workspace file state | Changed on the box; reconciled by the Gateway | Changed remotely; reconciled by the Gateway |
The bundled Crabbox cloud provider supports `worker-turn` through the node transport. Codex `remote-exec` supports an explicitly authorized paired device through that device's authenticated duplex node channel, or a cloud provider that explicitly advertises an SSH-backed execution carrier. A Crabbox cloud profile still does not advertise Codex `remote-exec`.
The bundled Crabbox cloud provider advertises both `worker-turn` and `remote-exec` through its enrolled node transport, so the same cloud profile is available to both harnesses. Codex can also use an explicitly authorized paired device or a provider that retains an SSH-backed remote-execution carrier. A profile that advertises only one mode remains unavailable to the other runtime.
After Crabbox setup, the cloud node dials the Gateway's public TLS endpoint over outbound WebSocket. Worker control and workspace transfer use the authenticated node and worker channels, not a Gateway-created reverse tunnel or rsync. Crabbox itself may still require SSH reachability while its CLI runs the provider-owned setup command. Outbound internet access is provider policy; the default AWS profile can reach the internet unless you restrict its network or security group.
After Crabbox setup, the cloud node dials the Gateway's public TLS endpoint over outbound WebSocket. Worker control, Codex remote execution, and workspace transfer use authenticated node or worker channels, not a Gateway-created reverse tunnel or rsync. Crabbox itself may still require SSH reachability while its CLI runs the provider-owned setup command. Outbound internet access is provider policy; the default AWS profile can reach the internet unless you restrict its network or security group.
For a loopback Gateway behind public HTTPS ingress, set `gateway.publicOrigin` to the proxy's bare origin. Node enrollment uses it as the default external pairing endpoint; `plugins.entries.device-pair.config.publicUrl` remains the pairing-specific override. If either URL is behind a reverse proxy, including cloudflared, nginx, or externally managed Tailscale Serve, `gateway.trustedProxies` must include the proxy's source address (typically loopback for a same-host proxy). Otherwise, forwarded client headers cause node enrollment to fail with `proxy_attribution_required`.
@@ -142,6 +142,8 @@ OpenClaw derives one canonical `cbx_...` lease ID from the durable provision ope
The setup bootstrap first reuses an installed `openclaw` binary when its version exactly matches the Gateway, then tries the exact `openclaw@<version>` registry package. For an unreleased source build, install a locally packed candidate with the same version in `settings.setup`; the provider will select it before touching the registry. After the node connects and publishes its session-host inventory, the Gateway pushes one content-addressed worker bundle through the paired channel. The node verifies and publishes those exact bytes without installing the normal OpenClaw package dependency tree. A stale Gateway build retires the environment and reprovisions against the current bundle rather than downgrading the execution-context protocol.
Codex remote execution additionally requires `settings.setup` or the cloud image to install the exact official npm `@openclaw/codex` plugin matching the Gateway version, including its pinned platform-native `@openai/codex` dependency. An exact-version bundled plugin is also accepted. Enrollment preserves the plugin's official npm or bundled provenance in the node's isolated per-lease state; it does not install plugins from npm or accept an untrusted local plugin path. A missing, mismatched, or untrusted plugin fails before the node starts.
### Verify the profile
Validate before restarting the Gateway:
@@ -153,6 +155,8 @@ openclaw plugins inspect crabbox --runtime --json
Changes under `cloudWorkers.profiles` require a Gateway restart. The default `gateway.reload.mode: "hybrid"` watches the config and performs that restart automatically; with reload watching disabled, run `openclaw gateway restart`.
To use the same profile with Codex, enable the Codex plugin on the Gateway, prepare its exact trusted installation in the cloud profile or image, and explicitly add `codex.exec-server.stdio.v1` to `gateway.nodes.commands.allow`. Crabbox activates that prepared plugin inside the cloud node's isolated state only when a Codex session selects the profile. Persistent command enablement does not replace the separate critical allow-once approval required for each exec-server attempt.
After the Gateway is back, prove the profile is advertised and compare it with Crabbox's read-only lease inventory:
```bash
@@ -180,12 +184,9 @@ While a placement is active, OpenClaw automatically samples available space on t
### Runtime support
- **OpenClaw** uses `worker-turn` placement. The restricted `openclaw worker` process runs each turn on the leased node and proxies inference through the Gateway.
- **Codex** uses `remote-exec` placement on an eligible paired device, or with a cloud provider that advertises an SSH-backed execution carrier. The bundled Crabbox cloud profile supports only `worker-turn`, so selecting that profile for Codex still fails before allocation.
- **Codex** uses `remote-exec` placement on the same bundled Crabbox cloud profile, an eligible paired device, or a provider that advertises an SSH-backed execution carrier. The Gateway keeps the Codex app-server and authentication local; an enrolled cloud node runs only the explicitly authorized Codex exec-server and does not start an OpenClaw worker child.
The Control UI disables cloud destinations whose advertised mode does not match
the selected runtime, including when moving an existing session. An
incompatible move is rejected before the active source starts draining or
changes its durable placement.
The Control UI checks each cloud destination's advertised execution modes in both New Session and Move Session. One Crabbox **Cloud · profile** row is selectable for OpenClaw and Codex, while a genuinely single-mode provider stays disabled for the other runtime. An incompatible move is rejected before the active source starts draining or changes its durable placement.
Other runtimes remain unavailable unless their harness explicitly declares a cloud placement mode. Cloud targets are not offered for external CLI session catalogs. Remote-exec fails closed if the selected provider or placement sandbox is unavailable; it never falls back to running the operation on the Gateway host.
@@ -201,9 +202,7 @@ the node's updated pairing surface if needed. Before each exec-server launch,
OpenClaw also requires the normal node invocation approval; denying that
request does not start a process.
Codex launches its exec-server directly, so paired-device placement does not
consume an OpenClaw worker slot and remains eligible when those slots are full.
OpenClaw `worker-turn` placement still requires an available worker slot.
Codex launches its exec-server directly, so paired-device and cloud-node placement do not consume an OpenClaw worker slot and remain eligible when those slots are full. OpenClaw `worker-turn` placement still requires an available worker slot.
Approval permits process execution and filesystem access anywhere the node's
operating system account allows. The exact placement workspace controls the
@@ -232,6 +231,10 @@ reconciliation as worker turns. See
[Run Codex on a paired device](/plugins/codex-harness#run-codex-on-a-paired-device)
for the exact allowlist configuration and lifecycle.
### Codex or OpenClaw on a cloud profile
The same configured Crabbox profile can host either harness. Select its **Cloud · profile** row after choosing an OpenClaw or Codex model; the selected runtime determines whether provisioning prepares a worker child or the managed Codex exec-server. Codex cloud-node execution requires the same explicit Gateway command allowlist and per-attempt allow-once approval as paired-device execution. It never falls back to Gateway-local or SSH execution if the node command is missing, denied, or disconnected.
For cloud-profile placement, the equivalent RPC flow is:
Create a session with a managed worktree, then dispatch it. Profile dispatch requires `operator.admin` and is available only while at least one worker profile is configured:
@@ -257,7 +260,7 @@ openclaw gateway call sessions.dispatch \
The bundled Crabbox provider advertises whatever machine classes the configured Crabbox binary reports for the selected backend, preserving Crabbox's size order. For example, a catalog containing `tiny`, `small`, `standard`, `fast`, `large`, and `beast` produces those six picker rows in that order; if Crabbox reports `standard` as 32 vCPU · 64 GB, that shape appears beside the class. Older binaries that publish no matching class catalog retain the label-only `standard`, `fast`, `large`, and `beast` fallback. You can also pass a provider-native server or instance type such as `c7a.24xlarge`; Crabbox treats any other non-empty class as that exact type. The selected value is fixed for that placement and reused by safe provisioning retries. `machineClass` is valid only with `profileId`, not `deviceId`.
`sessions.dispatch` closes local turn admission, drains active work, validates the eligible Git workspace inventory, provisions the lease, runs setup, enrolls the node, pushes the Gateway bundle when worker hosting requires it, syncs the workspace, and returns once the placement reaches `active` ownership. Inventory validation happens before provider allocation and reports an invalid request with an actionable size or entry limit when the workspace cannot be dispatched. Budget several minutes for the first cloud dispatch; leases and content-addressed bundles are reused where safe. After that, talk to the session as usual. OpenClaw turns route to the worker process; Codex native operations run on the authorized paired device or supported SSH-backed provider.
`sessions.dispatch` closes local turn admission, drains active work, validates the eligible Git workspace inventory, provisions the lease for the selected execution mode, runs setup, enrolls the node, pushes the required pinned Gateway bundle, syncs the workspace, and returns once the placement reaches `active` ownership. Inventory validation happens before provider allocation and reports an invalid request with an actionable size or entry limit when the workspace cannot be dispatched. Budget several minutes for the first cloud dispatch; leases and content-addressed bundles are reused where safe. After that, talk to the session as usual. OpenClaw turns route to the worker process; Codex native operations run on the authorized cloud node, paired device, or supported SSH-backed provider.
Completed cloud turns reconcile eligible, size-bounded workspace files back into the session's managed worktree before the turn claim is released. Worker-turn uses its terminal worker event to create the durable pending-result fence. Remote-exec waits for workspace quiescence and enters the same reconciliation flow after the local Codex attempt. Before applying the result, the Gateway stages complete authenticated base/current manifests plus each changed resulting blob as a Git ref under `refs/openclaw/worker-results/`; deletions are represented by the manifests and need no blob. This keeps the cloud delta recoverable even if the Gateway stops during the apply without duplicating unchanged baseline content. Workspace results use Git file semantics: regular files, executable bits, symlinks, additions, changes, and deletions are retained, while empty directories and other directory modes are not. The resulting file changes remain in the managed worktree for normal review and commit.
@@ -341,7 +344,7 @@ The desktop never gains public ingress. The node reads `/var/lib/crabbox/vnc.pas
- **Gateway-owned tool authority.** In worker-turn mode, the Gateway projects current profile, provider, agent, group, sender, sandbox, delegation, inherited, and runtime-cap policy over the worker's fixed coding-tool catalog before every turn. The launch envelope carries only that final closed-vocabulary subset. Explicitly capped scheduled turns reuse their trusted owner-group context without sending that identity to the box or reapplying a fresh sender overlay. Tools outside the worker catalog remain unavailable; an empty result runs with no tools.
- **Minted credentials, hashed at rest.** Each dispatch mints a worker credential; the Gateway stores only its hash. Credential rotation and owner-epoch fencing guarantee at most one live owner per session — a stale worker that reconnects is fenced, never merged.
- **Environment-bound enrollment.** One short-lived node-only setup credential is bound to the durable environment before allocation. Its first authenticated Ed25519 device identity is recorded atomically with setup completion; replay cannot substitute an unrelated node.
- **Explicit Codex device authorization.** Paired-device remote execution requires an explicitly allowed `codex.exec-server.stdio.v1` command, an approved pairing surface, and normal node invocation approval. The managed exec-server starts with a fresh private home and sanitized environment; allow-once never grants a later launch. Its managed workspace is not an OS sandbox: approved execution can access processes and files allowed to the node account, so use a separate least-privilege account when isolation is required.
- **Explicit Codex node authorization.** Cloud-node and paired-device remote execution require an explicitly allowed `codex.exec-server.stdio.v1` command, an approved pairing surface, and critical node invocation approval for each attempt. The managed exec-server starts with a fresh private home and sanitized environment; allow-once never grants a later launch. Its managed workspace is not an OS sandbox: approved execution can access processes and files allowed to the node account, so use a separate least-privilege account when isolation is required.
- **No standing model, forge, or cloud credentials on the box.** OpenClaw worker turns proxy inference by `{provider, model}` reference. Codex remote-exec keeps the app-server plus ChatGPT subscription or API-key auth on the Gateway and sends only sandbox operations to the box. Remote-exec requires prepared auth and rejects ambient auth fallback. Workspace git commits are authored without forge credentials, and Crabbox AWS lease metadata is checked authoritatively for an instance role before setup. Keep setup commands credential-free too.
- **Gateway-owned GitHub publication.** Publication credentials stay in the effective managed or native GitHub profile on the Gateway. The broker disables repository hooks, refuses configured Git clean filters, uses a temporary index and `git commit-tree`, pushes only a reconstructed public HTTPS URL with a command-local `gh auth git-credential` helper, and never writes a bearer token to argv, a remote URL, `.git/config`, a worker payload, or a transcript.
- **Provider-owned egress.** Gateway-proxied inference removes any OpenClaw need for direct model access, but OpenClaw does not rewrite provider firewalls. Restrict outbound traffic in the worker provider when the task requires it.
@@ -352,6 +355,7 @@ The desktop never gains public ingress. The node reads `/var/lib/crabbox/vnc.pas
- **No cloud profile is advertised** — run the `operator.read`-scoped `openclaw gateway call environments.list --params '{}'`. If the response has no `profiles`, ask an administrator to validate `cloudWorkers.profiles`, inspect the provider plugin, and restart the Gateway. This is a configuration or provider-activation problem, not an authorization result.
- **Cloud destinations are hidden or an RPC is denied** — cloud profile dispatch and profile-target moves require `operator.admin`. `operator.write` can dispatch or move to an eligible paired device, move to the Gateway, and reclaim a placement; `operator.read` alone can discover profiles but cannot start, stop, or move a session. Profile configuration, infrastructure pairing, Connect machine, raw environment lifecycle, direct `execNode` execution, incognito sessions, and arbitrary host or node paths remain `operator.admin`.
- **The selected runtime lacks cloud placement support** — choose a model whose advertised runtime supports cloud placement. The bundled OpenClaw and Codex runtimes are supported; undeclared runtimes remain local-only.
- **Codex cannot use a cloud profile** — verify that the profile advertises `remote-exec`, the Gateway enables the Codex plugin, and `gateway.nodes.commands.allow` includes `codex.exec-server.stdio.v1` without a matching deny rule. Approve the exact node invocation when prompted. Codex does not require an available OpenClaw worker slot; a missing or denied command must be corrected rather than bypassed with Gateway or SSH execution.
- **"Worker bootstrap requires Node.js on the leased host"** — add a Node install to `settings.setup` (see above).
- **AWS instance-role attestation fails** — clear `aws.instanceProfile` (and `CRABBOX_AWS_INSTANCE_PROFILE`, if set). Install Crabbox 0.41.1 or newer; older binaries do not satisfy the fixed-ID and authoritative `providerMetadata.instanceProfileAttached` contracts required for AWS admission.
- **Dispatch or workspace recovery fails** — inspect `environments.list` and `sessions.describe`. A failed environment exposes its bounded environment error. A failed placement exposes `recoveryError` plus its durable per-session `terminalReason`; the selected Control UI chat shows that terminal reason above the composer. When deeper diagnosis is necessary, an operator on the Gateway host can inspect the durable worker state read-only. Do not edit the state database to bypass lifecycle fencing.
docs/gateway/configuration-reference.md
+14 -6
View File
@@ -880,7 +880,7 @@ Gateway or node host and check `openclaw nodes pending` again.
- `gateway.nodes.pairing.autoApproveLocal`: silently approves pairing, role upgrades, and scope upgrades from trusted local connections (default: `true`). Scope upgrades additionally require the connection itself to prove local-grade credentials (auth mode `none`, or the shared token/password); Tailscale, trusted-proxy, and device-token connects keep their paired scopes as a durable cap. Set `false` to require explicit approval for every device; metadata-only reconnect refreshes remain automatic.
- `gateway.nodes.pairing.autoApproveCidrs`: optional CIDR/IP allowlist for auto-approving first-time node device pairing with no requested scopes. It is disabled when unset. This does not auto-approve operator/browser/Control UI/WebChat pairing, and it does not auto-approve role, scope, metadata, or public-key upgrades.
- `gateway.nodes.pairing.sshVerify`: SSH-verified auto-approval for first-time node device pairing (default: enabled). The gateway SSHes back to the pairing host (BatchMode, strict host keys) and approves only on an exact `openclaw node identity` device-key match. Same eligibility floor as `autoApproveCidrs`; probes are limited to private/CGNAT source addresses unless `cidrs` overrides them. Set `false` to disable, or `{ user, identity, timeoutMs, cidrs }` to tune. See [Node pairing](/gateway/pairing#ssh-verified-device-auto-approval-default).
- `gateway.nodes.commands.allow` / `gateway.nodes.commands.deny`: global allow/deny shaping for declared node commands after pairing and platform allowlist evaluation. `commands.allow` is the one-time persistent enable for classified commands such as `camera.snap`, `camera.clip`, `desktop.stream`, `screen.record`, `health.summary`, `sms.search`, and `sms.send`; `commands.deny` removes a command even if a platform default or explicit allow would otherwise include it. Computer and mobile UI control instead rely on default-off node-local enablement plus pairing. iOS Health permission, Android SMS permission, and Gateway command authorization are independent. After a node changes its declared command list, reject and re-approve that device pairing so the gateway stores the updated command snapshot.
- `gateway.nodes.commands.allow` / `gateway.nodes.commands.deny`: global allow/deny shaping for declared node commands after pairing and platform allowlist evaluation. `commands.allow` is the persistent enable for classified commands such as `camera.snap`, `camera.clip`, `codex.exec-server.stdio.v1`, `desktop.stream`, `screen.record`, `health.summary`, `sms.search`, and `sms.send`; `commands.deny` removes a command even if a platform default or explicit allow would otherwise include it. Codex remote execution on a paired device or enrolled cloud node additionally requires a separate critical allow-once approval for every exec-server attempt; persistent allowlisting never grants that approval. Computer and mobile UI control instead rely on default-off node-local enablement plus pairing. iOS Health permission, Android SMS permission, and Gateway command authorization are independent. After a node changes its declared command list, reject and re-approve that device pairing so the gateway stores the updated command snapshot.
- `gateway.tools.deny`: extra tool names blocked for HTTP `POST /tools/invoke` (extends default deny list).
- `gateway.tools.allow`: remove tool names from the default HTTP deny list for
owner/admin callers. This does not upgrade identity-bearing `operator.write`
@@ -967,14 +967,22 @@ Cloud workers are opt-in. If `cloudWorkers` is absent, or `profiles` is empty, O
SSH-backed `remote-exec` providers must return a trusted `hostKey` as exactly `algorithm base64`, without a hostname or comment. Bootstrap writes that key to an isolated `known_hosts` file, uses `StrictHostKeyChecking=yes`, and fails before opening a connection when the provider omits it. There is no trust-on-first-use fallback. These providers also carry workspace traffic over separate pinned SSH connections so rsync cannot block control traffic.
Node-backed `worker-turn` providers instead return an authenticated node device id. The Gateway installs the current worker bundle and transfers the workspace through the node transport; they do not return or resolve OpenClaw SSH endpoint credentials.
Node-backed providers return an authenticated node device id for either `worker-turn` or `remote-exec`. The Gateway installs the current pinned bundle and transfers the workspace through the node transport; these leases do not return or resolve OpenClaw SSH endpoint credentials. `worker-turn` requires a node lease and launches a restricted OpenClaw worker child. `remote-exec` can use either an enrolled node or an existing SSH-backed provider and keeps the harness plus model authentication on the Gateway.
### Crabbox profile
The bundled `crabbox` provider provisions a disposable machine through the local Crabbox CLI, enrolls it as an ephemeral node, and returns a node lease for `worker-turn`. The inner `settings.provider` selects the Crabbox backend; it is separate from the outer OpenClaw provider id.
The bundled `crabbox` provider provisions a disposable machine through the local Crabbox CLI, enrolls it as an ephemeral outbound node, and returns the same node transport for OpenClaw `worker-turn` or Codex `remote-exec`. One configured profile can therefore be selected by both harnesses; the selected session runtime determines its execution semantics. The inner `settings.provider` selects the Crabbox backend; it is separate from the outer OpenClaw provider id.
```json5
{
gateway: {
nodes: {
commands: {
// Required only when this profile also runs Codex remote-exec sessions.
allow: ["codex.exec-server.stdio.v1"],
},
},
},
cloudWorkers: {
profiles: {
production: {
@@ -1002,7 +1010,7 @@ Unknown settings are rejected. Crabbox credentials and backend-specific account
For coordinator-backed AWS, Crabbox's own `aws.sshCIDRs` should include the Gateway host's outbound IPv4 as a `/32`. Verify it with `crabbox config show --json` and `crabbox doctor --provider aws --json` before provisioning; do not place this provider-ingress setting in OpenClaw `settings`. See [Coordinator-backed Crabbox](/gateway/cloud-workers#coordinator-backed-crabbox).
Crabbox setup uses an environment-owned one-use pairing credential and the configured public Gateway URL. The provider returns the exact authenticated node id; the Gateway then installs its current bundle and transfers the workspace through authenticated node routes. OpenClaw does not persist Crabbox SSH endpoint, key, host-key, or fallback-port output.
Crabbox setup uses an environment-owned one-use pairing credential and the configured public Gateway URL. The provider returns the exact authenticated node id; the Gateway then installs its current bundle and transfers the workspace through authenticated node routes. For Codex remote execution, Crabbox prepares the bundled Codex plugin and pinned managed binary in the node's private state, and the Gateway requires the explicitly allowed `codex.exec-server.stdio.v1` command plus critical allow-once approval for each attempt. No OpenClaw worker child or worker slot is used in that mode. OpenClaw does not persist Crabbox SSH endpoint, key, host-key, or fallback-port output.
<Note>
AWS admission requires `providerMetadata.instanceProfileAttached` to be false. Install Crabbox 0.41.1 or newer for the fixed-ID replay and closed inspection contracts.
@@ -1035,13 +1043,13 @@ Crabbox setup uses an environment-owned one-use pairing credential and the confi
- `profiles`: named worker profiles with non-empty, whitespace-trimmed ids. Each profile selects a provider registered by a plugin.
- `provider`: non-empty worker provider id. The examples use the bundled `crabbox` provider and the QA Lab `static-ssh` provider.
- `install`: SSH-backed `remote-exec` worker installation method. `"bundle"` (default) transfers a content-hashed bundle of the gateway's installed build and supports released, development, and unreleased versions. `"npm"` is an opt-in optimization for an unmodified packaged release; it installs `openclaw@<exact gateway version>` from the public npm registry and never installs `latest`. Node-backed `worker-turn` providers install the Gateway bundle through node transport instead.
- `install`: SSH-backed `remote-exec` worker installation method. `"bundle"` (default) transfers a content-hashed bundle of the gateway's installed build and supports released, development, and unreleased versions. `"npm"` is an opt-in optimization for an unmodified packaged release; it installs `openclaw@<exact gateway version>` from the public npm registry and never installs `latest`. Node-backed `worker-turn` and `remote-exec` providers install the pinned Gateway bundle through node transport instead.
- Bundled provider plugins are selected automatically when configured, but explicit disables and `plugins.allow` still apply. Include the provider id (for example, `crabbox`) when an allowlist is configured. External provider plugins must also be installed and explicitly enabled.
- `settings`: provider-owned bounded JSON. The selected plugin defines and validates its keys; use [SecretRef objects](/gateway/secrets) for secret-bearing values. The static SSH provider requires `host`, `user`, `hostKey`, and `keyRef`; `port` defaults to `22`. `hostKey` must be one OpenSSH public host-key line (`algorithm base64`) obtained from the known host or another trusted channel, with no options prefix.
A supported Node runtime (22.22.3+, 24.15+, or 25.9+) with WAL-reset-safe SQLite must already be installed on the worker. The opt-in `"npm"` method also requires `npm` and outbound HTTPS access to the public npm registry. Networked toolchain setup is provider policy; bootstrap reports an actionable error instead of installing toolchains itself.
Node-backed `worker-turn` launches the self-contained worker loop and proxies model inference through the Gateway. SSH-backed `remote-exec` keeps the model loop on the Gateway and routes sandbox operations to the remote host. Both reconcile the session workspace and transcript through the durable placement lifecycle.
Node-backed `worker-turn` launches the self-contained worker loop and proxies model inference through the Gateway. Node-backed or SSH-backed `remote-exec` keeps the model loop on the Gateway and routes sandbox operations to the remote host. Node-backed Codex accepts process, filesystem, capability, and credential-free HTTP operations; authenticated HTTP is rejected before reaching the node. Both modes reconcile the session workspace and transcript through the durable placement lifecycle. A disconnected node-backed Codex attempt is terminal; reconnect permits only a fresh attempt, never process or stream resumption.
Each durable environment record retains its validated provider settings and resolved install method in a creation-time profile snapshot. Changing or removing a named profile affects new creates; existing records continue lifecycle reconciliation with that snapshot, provided the owning plugin remains available.
docs/gateway/protocol.md
+3 -3
View File
@@ -641,8 +641,8 @@ methods. Treat this as feature discovery, not a full enumeration of
- `agents.workspace.list` and `agents.workspace.get` (`operator.read`) expose read-only, paginated browsing of an agent's workspace directory for clients in the trusted operator domain described in [Operator scopes](/gateway/operator-scopes). Requests accept workspace-relative paths only; reads stay confined to the realpathed workspace root (symlink and hardlink escapes rejected), size-capped, and limited to UTF-8 text plus common image types (base64). Responses do not expose the host workspace path. There are no write operations in this namespace.
- `tasks.list`, `tasks.get`, and `tasks.cancel` expose the gateway task ledger to SDK and operator clients. See [Task ledger RPCs](#task-ledger-rpcs) below.
- `artifacts.list`, `artifacts.get`, and `artifacts.download` expose transcript-derived artifact summaries and downloads for an explicit `sessionKey`, `runId`, or `taskId` scope. Run and task queries resolve the owning session server-side and only return transcript media with matching provenance; unsafe or local URL sources return unsupported downloads instead of fetching server-side.
- `environments.list` and `environments.status` (`operator.read`) remain available without cloud-worker profiles and preserve gateway-local and node environment discovery. Node environments include the durable `sessionHost` identity used to keep a known offline host visible, while current connected inventory is authoritative over that history. Missing identity means false. Exact bounded `{ total, available }` worker slots are live-only and omitted offline. Configured cloud workers and durable records left by earlier profiles add `worker` metadata with `providerId`, optional `leaseId`, `state`, `ageMs`, optional `idleMs`, and `attachedSessionIds`. Worker lifecycle states are `requested`, `provisioning`, `bootstrapping`, `ready`, `attached`, `idle`, `draining`, `destroying`, `destroyed`, `failed`, and `orphaned`. A connected node may also include `workerBundle: { status: "installed", version }` or `workerBundle: { status: "missing" }`. This optional observation is reconnect-scoped and reports validation of one Gateway-retained bundle; it is not launch authority. The public result never exposes the bundle hash, Gateway namespace, node filesystem path, receipt, or protocol-feature details.
- `environments.create` (`{ profileId, idempotencyKey }`) provisions a worker from a configured plugin provider profile; retries with the same key reuse the durable operation. `environments.destroy` (`{ environmentId }`) requests idempotent teardown of a durable worker environment. Both require `operator.admin`, are control-plane writes, and return the same environment summary shape used by status responses.
- `environments.list` and `environments.status` (`operator.read`) remain available without cloud-worker profiles and preserve gateway-local and node environment discovery. Node environments include the durable `sessionHost` identity used to keep a known offline host visible, while current connected inventory is authoritative over that history. Missing identity means false. Exact bounded `{ total, available }` worker slots are live-only and omitted offline; worker-turn admission consumes a slot, while node-backed remote-exec does not. Configured profile summaries expose their bounded, canonically ordered `executionModes` array plus the existing singular `executionMode` primary/default display projection. Current clients select profiles only by membership in `executionModes`. Configured cloud workers and durable records left by earlier profiles add `worker` metadata with `providerId`, optional `leaseId`, `state`, `ageMs`, optional `idleMs`, and `attachedSessionIds`. Worker lifecycle states are `requested`, `provisioning`, `bootstrapping`, `ready`, `attached`, `idle`, `draining`, `destroying`, `destroyed`, `failed`, and `orphaned`. A connected node may also include `workerBundle: { status: "installed", version }` or `workerBundle: { status: "missing" }`. This optional observation is reconnect-scoped and reports validation of one Gateway-retained bundle; it is not launch authority. The public result never exposes the bundle hash, Gateway namespace, node filesystem path, receipt, or protocol-feature details.
- `environments.create` (`{ profileId, idempotencyKey }`) provisions an environment from a configured plugin provider profile; retries with the same key reuse the durable operation. Direct creation without a session does not select an execution mode, so the provider uses its intentional default; Crabbox prepares `worker-turn`. `environments.destroy` (`{ environmentId }`) requests idempotent teardown of a durable worker environment. Both require `operator.admin`, are control-plane writes, and return the same environment summary shape used by status responses.
- `worker.desktop.observe` (`{ environmentId, control? }`, `operator.admin`) starts or reuses the environment's desktop forward and returns `{ transport, wsPath, expiresAtMs, control, vncPassword? }`. `wsPath` carries a single-use 60-second token for the Gateway's desktop observer WebSocket; reconnecting requires a fresh observe call. Environments with an observable desktop advertise `worker.desktop: true` in `environments.list`. The method is advertised only when the `cloudWorkers.desktop` lab is enabled. See [Cloud workers](/gateway/cloud-workers#desktop-interactive).
- `agent.identity.get` returns the effective assistant identity for an agent or session.
- `agent.wait` waits for a run to finish and returns the terminal snapshot when available.
@@ -657,7 +657,7 @@ methods. Treat this as feature discovery, not a full enumeration of
- `sessions.describe` returns one gateway session row for an exact session key.
- `sessions.resolve` resolves or canonicalizes a session target by key, raw session ID, label, or Control UI short ID. Ambiguous short IDs return a bounded candidate list as a successful RPC result.
- `sessions.create` creates a new session entry. Optional `model`, `contextWindow`, and `thinkingLevel` values persist the initial model, advertised context-window choice, and reasoning overrides atomically; optional `category` assigns the session to a custom group and registers that group when first used. `worktree: true` provisions a managed worktree; optional `worktreeBaseRef`/`worktreeName` select the base ref and branch name, and `execNode` (`operator.admin`) binds session exec to a node host. Without `worktreeName`, OpenClaw derives a readable name from the session label or generated first-message title, then falls back to a crustacean-themed name; names already occupied by another owner, local branch, or unmanaged path receive a numeric suffix. The created worktree is echoed in the result and persisted on the session row (`worktree: { id, branch, repoRoot }`). When the entry is created but its nested initial `chat.send` is rejected, the successful result includes `runStarted: false` and `runError`; clients can preserve the prompt and retry against the returned session key. A caller that passes `parentSessionKey` with `emitCommandHooks: true` should also declare the lifecycle disposition of a distinct child: `succeedsParent: true` ends the parent with `session_end`, while `false` keeps the parent active and emits only the child's `session_start`. Omitting `succeedsParent` preserves the legacy parent-rollover behavior for existing clients. The disposition requires both parent linkage and command hooks; a fork cannot succeed its parent. Main-session reset-in-place behavior is unchanged because no distinct child is created. New rows are stamped with write-once creation provenance (`createdVia`, `createdActor`, `createdAt`) from the trusted creation seam; adopting an existing key never restamps it. For human profile actors, `createdActor.label` is resolved from the current user profile when the row is projected and is never stored on the session entry, so profile renames do not drift. Session rows also carry `parentSessionKey` (navigation parent, persisted), `controlOwnerSessionKey` (runtime controller when live), `forkSource` (exact source key + transcript generation for forks), and `previousSessionId` (prior transcript generation under the same key).
- `sessions.dispatch` moves an authorized local OpenClaw session with a live, registry-owned session managed worktree to a paired device or configured cloud profile. Pass `{ key, deviceId, agentId? }` for an explicit device, `{ key, autoDevice: true, agentId? }` for automatic paired-device selection, `{ key, profileId, machineClass?, agentId? }` for an explicit profile, or `{ key, agentId? }` to look up the managed worktree's normalized origin in `cloudWorkers.projectProfiles`. These target modes are mutually exclusive and explicit targets take precedence over project-profile lookup. Automatic selection ranks worker-slot runtimes by available slots and then device ID; runtimes without worker slots use device ID order. If a candidate becomes ineligible during dispatch, up to three ranked candidates are attempted; other errors are not retried. Explicit and automatic device dispatch require `operator.write`; explicit-profile and project-profile dispatch require `operator.admin`. A missing origin, unmatched mapping, or mapping to an unconfigured profile returns a typed `INVALID_REQUEST` without provisioning or falling back to another target. Malformed params use the write scope before schema validation. A missing cloud profile hides only cloud targets; eligible paired-device dispatch remains available. Dispatch closes local turn admission before draining active work and returns only after placement reaches `active` worker ownership. Arbitrary plain directories are not dispatchable; after admission, the workspace transport may use manifest mirroring if the managed worktree's Git metadata later becomes unavailable. SSH fallback candidates rotate only for idempotent probes, content-addressed transfers, receipt/lock-guarded artifact installation, convergent managed-worktree mirroring, and tunnel reconnects. Ambiguous unguarded stateful commands fail closed and are not replayed. Dispatch is one-way; worker-to-local pull-back is not part of this RPC.
- `sessions.dispatch` moves an authorized local OpenClaw or Codex session with a live, registry-owned session managed worktree to a paired device or configured cloud profile. Pass `{ key, deviceId, agentId? }` for an explicit device, `{ key, autoDevice: true, agentId? }` for automatic paired-device selection, `{ key, profileId, machineClass?, agentId? }` for an explicit profile, or `{ key, agentId? }` to look up the managed worktree's normalized origin in `cloudWorkers.projectProfiles`. These target modes are mutually exclusive and explicit targets take precedence over project-profile lookup. Automatic selection ranks worker-slot runtimes by available slots and then device ID; runtimes without worker slots use device ID order. If a candidate becomes ineligible during dispatch, up to three ranked candidates are attempted; other errors are not retried. Explicit and automatic device dispatch require `operator.write`; explicit-profile and project-profile dispatch require `operator.admin`. A missing origin, unmatched mapping, or mapping to an unconfigured profile returns a typed `INVALID_REQUEST` without provisioning or falling back to another target. Malformed params use the write scope before schema validation. A missing cloud profile hides only cloud targets; eligible paired-device dispatch remains available. Dispatch closes local turn admission before draining active work and returns only after placement reaches `active`, with worker-child ownership for `worker-turn` or Gateway-owned harness execution for `remote-exec`. Arbitrary plain directories are not dispatchable; after admission, the workspace transport may use manifest mirroring if the managed worktree's Git metadata later becomes unavailable. SSH fallback candidates rotate only for idempotent probes, content-addressed transfers, receipt/lock-guarded artifact installation, convergent managed-worktree mirroring, and tunnel reconnects. Ambiguous unguarded stateful commands fail closed and are not replayed. Dispatch is one-way; worker-to-local pull-back is not part of this RPC.
- `sessions.reclaim` (`operator.write`) safely stops a session placement by key. It waits for an in-flight dispatch, drains admitted work, reconciles active workspace changes, and retries pending failed-environment teardown through the placement owner. Callers never need raw environment-destroy authority.
- `sessions.move` moves an authorized active session to the Gateway, a paired device, or a configured profile. Gateway and device targets require `operator.write`; profile targets require `operator.admin`; malformed targets use the write scope before schema validation. The caller supplies the exact observed generation, environment, and owner epoch; session authorization and those source facts are revalidated before the move commits. Ordinary moves always reconcile the source. Only a Gateway target may add `abandonSource: true`, and only when the exact source is a currently offline paired-device placement. That durable decision force-fences and destroys the remote owner, skips remote workspace reconciliation, and continues from the last Gateway-synced state without replay; unsynced files and in-flight work may be lost. Available, unknown, profile, and other-worker sources reject explicit abandonment.
- `sessions.groups.list`, `sessions.groups.put`, `sessions.groups.rename`, and `sessions.groups.delete` manage the gateway-owned custom session group catalog (names + display order). The read-scoped list result is intentionally path-free. `sessions.groups.defaults` and `sessions.groups.update` require `operator.write` and read or replace one custom group's optional working-directory and worktree defaults. Non-admin callers can save only directories inside a configured agent workspace; other absolute Gateway paths require `operator.admin`. Membership stays on each session's `category` field; rename and delete update member sessions server-side.
docs/plan/cloud-workers.md
+2 -2
View File
@@ -8,7 +8,7 @@ read_when:
## Status
Superseded historical proposal. The implemented architecture is documented in [Runners and execution environments](/plan/runners) and [Cloud workers](/gateway/cloud-workers): Crabbox provisions a node-backed `worker-turn` lease with current reconnect-scoped v6 supervisor proof, the worker child dials the Gateway's authenticated public worker route, and workspace transfer uses the node channel. The former dedicated loopback listener, SSH reverse-forward carrier, and SSH-launched worker-turn path have been removed. SSH remains a separate `remote-exec` workspace transport and desktop carrier.
Superseded historical proposal. The implemented architecture is documented in [Runners and execution environments](/plan/runners) and [Cloud workers](/gateway/cloud-workers): one Crabbox profile provisions enrolled outbound-node leases for OpenClaw `worker-turn` or Codex `remote-exec`. OpenClaw launches a restricted worker child through the Gateway's authenticated public worker route; Codex uses the explicitly authorized node exec-server while its app-server and authentication remain on the Gateway. Workspace transfer uses the node channel. The former dedicated loopback listener, SSH reverse-forward carrier, and SSH-launched worker-turn path remain removed. SSH is retained only for existing SSH-backed `remote-exec` providers and separately owned desktop carriers.
The sections below preserve the pre-convergence design record and are not the current runtime contract.
@@ -97,7 +97,7 @@ Worker mode (`openclaw worker`) is an entry point, not a fork: connection handli
### 3. Historical transport proposal: everything over SSH
This section describes the superseded carrier. Current `worker-turn` environments use authenticated node connectivity; only `remote-exec` workspace operations retain pinned SSH.
This section describes the superseded carrier. Current node-backed `worker-turn` and `remote-exec` environments use authenticated node connectivity; existing SSH-backed `remote-exec` providers retain pinned SSH.
The original proposal had the gateway own connectivity while the worker required nothing but sshd:
docs/plan/runners.md
+51 -32
View File
@@ -1,5 +1,5 @@
---
summary: Everything is a node — one placement model where paired machines and cloud boxes host sessions through the worker admission path; clients attach to sessions, never to runners.
summary: Everything is a node — one placement model where paired machines and cloud boxes run OpenClaw worker turns or Codex remote execution through mode-specific admission; clients attach to sessions, never to runners.
title: Runners plan
read_when:
- Designing or reviewing where sessions run (gateway, device, cloud)
@@ -27,7 +27,7 @@ advances a milestone.
| 6 | Node worker provider (device runners) | in progress | #122683, #122769, #122829, #122939, #123013, #123033, #122966, #123157, #123280, #123612, #123641, #123665, #123673, #123700, #123696, #123785, #123859, #123889, #123901, #125708 |
| 7 | Bundle push consent + runner updates | in progress | #123985, #124037, #124356, #124590 |
| 8 | Stop-and-continue moves and offline device recovery | landed | #125036, #126284 |
| 9 | Node exec-server carrier and contract-preserving cleanup | in progress | #125503, #125524, #125587 |
| 9 | Node exec-server carrier and contract-preserving cleanup | in progress | #125503, #125524, #125587, #127202 |
| 10 | Cloud convergence (provisioners run `openclaw connect`) | landed | #125288, #125384, #125465 |
Revision history: revision 1 (2026-08-08) established the session/runner
@@ -79,12 +79,12 @@ Node a paired machine holding an outbound connection to the gateway
copy says "device". EVERY remote machine is a node — personal
workstations, servers, cloud leases. Phones are nodes that never
advertise session hosting.
Runner anything that can host a session's turn loop: the gateway itself,
or a session-capable node. "Runner" is internal/docs vocabulary;
UI copy says "Runs on …".
Worker the per-turn child process (`openclaw worker`) that hosts a
session's loop under worker admission. On paired devices and cloud
leases alike, it is a supervised child of the node host.
Runner anything that can own a session's turn loop or delegated execution:
the gateway itself or a session-capable node. "Runner" is
internal/docs vocabulary; UI copy says "Runs on …".
Worker the per-turn child process (`openclaw worker`) used only by
`worker-turn` placement. Codex `remote-exec` keeps its turn loop on the
gateway and runs its managed exec-server on the enrolled node.
Isolation a property OF the runner (none | docker | podman), not a place.
Project repo identity: normalized remote.origin.url, with the existing
16-char repo fingerprint as the no-remote fallback. Derived,
@@ -131,7 +131,7 @@ channel.
### Worker ingress on the public endpoint (milestone 5)
Worker admission is exposed only on a path-tagged upgrade route on the public TLS endpoint (`connectionKind = "worker"` is forced by the route). Node-hosted worker children dial that endpoint directly. The former loopback listener and SSH reverse-forward carrier were removed after Crabbox converged onto node-backed worker turns; SSH remains only for `remote-exec` workspace transport and separately owned desktop tunnels.
Worker admission is exposed only on a path-tagged upgrade route on the public TLS endpoint (`connectionKind = "worker"` is forced by the route). Node-hosted `worker-turn` children dial that endpoint directly; node-backed Codex `remote-exec` instead uses the explicitly authorized duplex node carrier and never starts a worker child. The former loopback listener and SSH reverse-forward carrier were removed after Crabbox converged onto enrolled cloud nodes; SSH remains available for explicitly SSH-backed `remote-exec` providers and separately owned desktop tunnels.
Hardening that ships with the exposure, not after it:
@@ -215,6 +215,9 @@ stated honestly (revision 1 undersold this):
visibly. Public node/environment inventory projects the same slot snapshot.
Terminal node launch receipts retain a 24-hour replay window and prune in
bounded batches; `pending` and `running` capacity reservations never age out.
Codex `remote-exec` starts its managed node exec-server directly and does not
consume a worker slot; eligibility instead requires its current, effectively
invocable command and separate per-attempt approval.
- **Multi-gateway safety.** The worker install/workspace root on a node is
namespaced by gateway identity so two gateways pairing one machine cannot
corrupt each other's state.
@@ -376,11 +379,14 @@ speak. Additions:
- **Where picker regrouped** (`ui/src/pages/new-session/device-placement.ts`):
sections "This gateway" / "Devices" / "Cloud". Device rows come only from
node entries in `environments.list`. A device is selectable only when its
current status is available, `sessionHost` is true, and its exact bounded
worker slots are valid with `available > 0`. Offline known session hosts,
connected non-hosts, saturated hosts, hosts without capacity, outdated
hosts, and unavailable hosts stay visible but disabled with a next step.
Cloud profiles remain their separate list.
current status is available, `sessionHost` is true, and the selected runtime
has its required authority. OpenClaw `worker-turn` requires exact bounded
worker slots with `available > 0`; Codex `remote-exec` requires its
effectively invocable exec-server command and can run with zero free worker
slots. Offline known session hosts, connected non-hosts, incompatible or
saturated hosts, outdated hosts, and unavailable hosts stay visible but
disabled with a next step. Cloud profiles remain their separate list and are
filtered by membership in their complete advertised execution-mode set.
- **Remote placement uses one session path.** Device and cloud selections use
a Gateway project or folder, force a managed worktree, create the session
without `execNode`, dispatch by exact `{ deviceId }` or `{ profileId }`, and
@@ -412,18 +418,26 @@ speak. Additions:
### Cloud convergence (milestone 10)
The bundled Crabbox provider now boots the box and runs
The bundled Crabbox provider boots the box and runs
`openclaw connect <setup code> --ephemeral` in an isolated per-lease state
directory. The Gateway persists one replay-safe setup identity, atomically
binds the authenticated device identity to the worker environment, pushes the
current bundle through the node channel, and removes the node role after
provider teardown. `destroy` = release lease plus pairing cleanup. Codex now
supports paired-device `remote-exec` over the approved duplex node carrier;
disconnect ends the attempt, and reconnect starts a fresh attempt without
resume. Crabbox cloud profiles remain `worker-turn` only. The replaced
reverse-tunnel/rsync cloud carrier has been deleted. Distinct stable SSH,
OpenShell, Claude, and exec-host contracts remain intact; broader replacement
and reconnect or resume are later work.
directory. One configured cloud profile advertises both `worker-turn` and
`remote-exec`; the selected session runtime determines provider setup and
placement ownership, while both modes return the same enrolled node transport.
The Gateway persists one replay-safe setup identity, atomically binds the
authenticated device identity to the environment, pushes the current pinned
bundle through the node channel, and removes the node role after provider
teardown. `destroy` releases the lease and cleans up its pairing and children.
OpenClaw launches a restricted node-supervised worker child. Codex instead uses
the bundled, pinned managed exec-server over the approved duplex node carrier;
its app-server, model authentication, and transcript remain Gateway-owned.
The Codex command requires explicit Gateway allowlisting and critical
per-attempt allow-once approval, and consumes no worker slot. Disconnect ends
the attempt; reconnection permits only a fresh attempt without process or
stream resumption. The replaced reverse-tunnel/rsync cloud carrier remains
deleted. Distinct stable SSH, OpenShell, Claude, and exec-host contracts remain
intact; broader replacement, reconnect or resume, node-child isolation, and
device-project checkouts remain later work.
## What the adversarial reviews killed or reshaped
@@ -545,14 +559,19 @@ Independently mergeable PR series; 3–5 can interleave after 1c.
8. **Stop-and-continue moves** (landed): drain + reclaim + re-dispatch to
another runner, plus durable offline-device waiting and explicit destructive
Gateway continuation.
9. **Node exec-server carrier and contract-preserving cleanup**: the missing
node exec-server carrier must first reproduce existing remote-exec and
approval behavior. Keep the stable SSH sandbox, OpenShell, Claude one-shot,
and exec-host contracts until their individual replacements are proved;
none is deletable merely because node-backed session hosting exists.
9. **Node exec-server carrier and contract-preserving cleanup**: the approved
duplex node exec-server carrier supports paired-device and cloud-node
Codex remote execution while retaining explicit command authorization,
per-attempt approval, Gateway-owned authentication, terminal disconnects,
and fresh-attempt-only recovery. Keep the stable SSH sandbox, OpenShell,
Claude one-shot, and exec-host contracts until their individual replacements
are proved; none is deletable merely because node-backed session hosting
exists.
10. **Cloud convergence** (landed): `--ephemeral` enrollment, provisioners run
`openclaw connect`, and the former worker reverse-tunnel/rsync carrier is
removed. Stable SSH-backed remote-exec and desktop contracts remain.
`openclaw connect`, and one bundled Crabbox profile supports both OpenClaw
`worker-turn` and Codex `remote-exec` through enrolled outbound nodes. The
former worker reverse-tunnel/rsync carrier stays removed. Stable
SSH-backed remote-exec and desktop contracts remain.
Net production LOC across the plan is targeted negative: milestones 3–5 are
small additions, 6–7 are mostly a provider + one transport implementation
docs/plugins/codex-harness-reference.md
+19 -16
View File
@@ -482,27 +482,30 @@ This preview path is local-only. A remote WebSocket app-server cannot reach
the loopback exec-server unless it is running on the same host, so OpenClaw
rejects that combination.
Paired-device `remote-exec` placement is a separate, placement-owned execution
path and does not require `appServer.experimental.sandboxExecServer`. The
Gateway keeps Codex app-server and provider auth local, while the authorized
paired device runs the managed Codex exec-server over its existing duplex node
connection. It requires explicit `gateway.nodes.commands.allow` authorization
for `codex.exec-server.stdio.v1`, the approved pairing surface, and normal node
invocation approval. The node receives a fresh private home and sanitized
environments, never Gateway provider, cloud, or GitHub credentials. A lost
node connection terminates the attempt and process instead of resuming it.
Each paired-device attempt uses its own Gateway app-server client because
Codex can register a remote environment but cannot remove one from a running
app-server. The device exec-server does not consume an OpenClaw worker slot.
HTTP requests containing authentication, cookies, API keys, or other
credential-bearing headers are rejected before reaching the device; use a
Gateway-owned authenticated request or a credential-free endpoint instead.
Node-backed `remote-exec` placement on a paired device or enrolled Crabbox
cloud worker is a separate, placement-owned execution path and does not require
`appServer.experimental.sandboxExecServer`. The Gateway keeps Codex
app-server and provider auth local, while the authorized node runs the managed,
pinned Codex exec-server over its existing duplex connection. It requires
explicit `gateway.nodes.commands.allow` authorization for
`codex.exec-server.stdio.v1`, the approved pairing surface, and separate
allow-once node invocation approval for each attempt. The node receives a
fresh private home and sanitized environments, never Gateway provider, cloud,
or GitHub credentials. A lost node connection terminates the attempt and
process instead of resuming it. Each node-backed attempt uses its own Gateway
app-server client because Codex can register a remote environment but cannot
remove one from a running app-server. The node exec-server does not consume an
OpenClaw worker slot. HTTP requests containing authentication, cookies, API
keys, or other credential-bearing headers are rejected before reaching the
node; use a Gateway-owned authenticated request or a credential-free endpoint
instead.
Normal Codex turns are supported, but `/btw` side questions are unavailable
until they can be bound to the active placement.
The managed placement workspace is not an OS sandbox: approved processes and
files have the node account's full access. Use a separate least-privilege node
account when isolation is required.
See [Run Codex on a paired device](/plugins/codex-harness#run-codex-on-a-paired-device).
See [Run Codex on a paired device](/plugins/codex-harness#run-codex-on-a-paired-device)
and [Run Codex on a cloud worker](/plugins/codex-harness#run-codex-on-a-cloud-worker).
## Auth and environment isolation
docs/plugins/codex-harness-runtime.md
+17 -14
View File
@@ -223,20 +223,23 @@ process. Failed environment registration never falls back to host execution.
See [Sandboxed native execution](/plugins/codex-harness-reference#sandboxed-native-execution)
for configuration and local-only transport restrictions.
Paired-device `remote-exec` is separate from the experimental local sandbox
flag: Codex app-server and model auth stay on the Gateway, while an explicitly
authorized managed exec-server on the node owns process, filesystem, capability,
and credential-free HTTP operations. The Gateway rejects authentication,
cookie, API-key, and other sensitive HTTP headers before they reach the node;
authenticated HTTP must run on the Gateway. The existing duplex node channel
carries the Codex JSON-RPC stream without starting an OpenClaw worker child or
consuming a worker slot. Each attempt owns an isolated Gateway app-server
client so its remote environment registration retires with that attempt.
Disconnect ends the active attempt and its remote processes; reconnect allows
only a fresh attempt. Normal Codex turns work, but `/btw` side questions fail
closed because they are not yet placement-bound. The placement workspace does
not confine execution: process and filesystem access remain bounded only by the
node's operating system account.
Node-backed `remote-exec`, whether on a paired device or the same Crabbox cloud
profile used for OpenClaw worker turns, is separate from the experimental
local sandbox flag. Codex app-server and model auth stay on the Gateway, while
an explicitly authorized managed exec-server on the enrolled node owns
process, filesystem, capability, and credential-free HTTP operations. The
Gateway rejects authentication, cookie, API-key, and other sensitive HTTP
headers before they reach the node; authenticated HTTP must run on the
Gateway. The existing duplex node channel carries the Codex JSON-RPC stream
without starting an OpenClaw worker child or consuming a worker slot. Explicit
Gateway command allowlisting and separate per-attempt allow-once approval
remain required. Each attempt owns an isolated Gateway app-server client so its
remote environment registration retires with that attempt. Disconnect ends the
active attempt and its remote processes; reconnect allows only a fresh
attempt. Normal Codex turns work, but `/btw` side questions fail closed because
they are not yet placement-bound. The placement workspace does not confine
execution: process and filesystem access remain bounded only by the node's
operating system account.
## V1 support contract
docs/plugins/codex-harness.md
+35 -3
View File
@@ -46,9 +46,9 @@ with Codex native code mode enabled (code-mode-only stays off by default), so
native workspace/code capabilities remain available alongside OpenClaw
dynamic tools routed through the app-server `item/tool/call` bridge. An
ordinary OpenClaw sandbox or restricted tool policy disables native code mode
unless you opt into the experimental sandbox exec-server path. Paired-device
`remote-exec` instead uses its placement-owned environment without that
experimental flag.
unless you opt into the experimental sandbox exec-server path. Node-backed
`remote-exec` on a paired device or cloud worker instead uses its
placement-owned environment without that experimental flag.
Eligible native-shell turns also retain `gateway_exec` and `gateway_process`
as a distinct OpenClaw execution path. Use `gateway_exec` only when a command
@@ -238,6 +238,38 @@ yet bound to paired-device placement and fail with an actionable explanation.
See [Cloud workers and paired-device placement](/gateway/cloud-workers) and
[Node command policy](/nodes#command-policy).
## Run Codex on a cloud worker
The bundled Crabbox provider supports both OpenClaw `worker-turn` and Codex
`remote-exec`, so one configured cloud-worker profile is selectable for either
harness. Choose the same **Cloud · profile** destination in New Session or
Move Session after selecting a Codex model. Profile placement requires
`operator.admin` and a managed Gateway worktree.
Enable the Codex plugin and explicitly allow
`codex.exec-server.stdio.v1` on the Gateway, as shown in
[Run Codex on a paired device](/plugins/codex-harness#run-codex-on-a-paired-device).
The cloud image may include the exact-version bundled Codex plugin; otherwise,
the profile setup or image must install the matching trusted official npm Codex
plugin and its pinned platform-native Codex binary. Crabbox validates the
bundled or prepared installation and preserves its provenance in the disposable
node's isolated state without installing a plugin during enrollment. The Gateway
checks the cloud node's current pairing and
effectively invocable command before starting a Codex process; approve the
critical allow-once request for each exec-server attempt.
Codex runs its managed exec-server over the enrolled node's authenticated
outbound connection without starting an OpenClaw worker child or consuming a
worker slot. Its app-server, model connection, provider authentication, and
transcript remain Gateway-owned. Process and filesystem access still have the
node operating-system account's permissions, and only credential-free HTTP is
forwarded. Workspace changes reconcile to the Gateway-owned worktree. A failed
or disconnected attempt is terminal and requires a fresh attempt; it never
resumes the remote process or falls back to Gateway-local or SSH execution.
See [Cloud workers](/gateway/cloud-workers) for profile configuration,
placement lifecycle, and cleanup.
## Share threads with Codex Desktop and CLI
The default `appServer.homeScope: "agent"` isolates each OpenClaw agent from
docs/plugins/manifest.md
+1 -1
View File
@@ -750,7 +750,7 @@ Provider plugins that implement both `resolveUsageAuth` and `fetchUsageSnapshot`
Embedding providers must declare `contracts.embeddingProviders` for each adapter registered with `api.registerEmbeddingProvider(...)`. The same generic contract serves reusable vector generation and memory search. The retired `contracts.memoryEmbeddingProviders` key is no longer accepted.
Worker providers must declare each `api.registerWorkerProvider(...)` id in `contracts.workerProviders`. Core persists durable intent before calling `provision`; providers validate their settings and any optional per-dispatch `machineClass` before external allocation, and repeated calls with the same operation id must adopt the same lease. Providers may implement asynchronous `listMachineOptions(profile)` to expose process-stable picker metadata; omit it when machine selection is not meaningful. Machine options contain only `id`, `label`, optional positive-integer `cpu` and `memoryGb`, and optional `default`. Providers used for session placement must declare exactly one `supportedExecutionModes` value: `worker-turn` providers return node leases and `remote-exec` providers return SSH leases. Omission advertises no session-placement modes while leaving direct lifecycle operations available. Providers whose bounded provisioning exceeds core's five-minute default may implement `resolveProvisionTimeoutMs(profile)` and include acquisition, provider-owned setup, and cleanup in the returned positive millisecond budget. Core also persists that validated settings snapshot and passes it with `leaseId` to `inspect({ leaseId, profile })` and `destroy({ leaseId, profile })`, including after the named profile is changed or removed. Destruction is idempotent, inspection returns the closed `active` / `destroyed` / `unknown` status union, and SSH private-key material is referenced only through `SecretRef`. Provisioned SSH endpoints must also include a public `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment, so core can pin the host before connecting. They may include up to 10 ordered, unique `fallbackPorts`, excluding the primary `port`; core persists those candidates and rotates among them only for idempotent probes, content-addressed transfers, receipt/lock-guarded artifact installation, convergent managed-worktree mirroring, and tunnel reconnects. Ambiguous unguarded stateful commands fail closed and are not replayed across candidates. A lease may set `sharedHost: true` when the SSH account also owns unrelated processes; core then avoids host-wide process freezing during workspace reconciliation. Omitted or `false` means a dedicated worker host. Active inspection repeats this fact so core can reconcile provider-owned isolation for leases persisted before the field existed; tunnel startup waits for that first authoritative inspection. Optional desktop metadata may advertise up to eight unique closed apps: `browser` with an absolute `executablePath` and a CDP port from 1 through 65535, or `terminal` with an absolute `executablePath`. Core rejects unknown app ids and fields and persists the validated metadata with the existing desktop record. Providers that mint dynamic identity refs may implement authoritative `resolveSshIdentity({ leaseId, profile, keyRef })`; providers without it use core's generic secret resolver. An authoritative `unknown` orphans an active local record; after a persisted destroy request it confirms teardown.
Worker providers must declare each `api.registerWorkerProvider(...)` id in `contracts.workerProviders`. Core persists durable intent before calling `provision`; providers validate their settings and optional per-dispatch `machineClass` and `executionMode` before external allocation, and repeated calls with the same operation id must adopt the same lease without changing the selected mode. Providers may implement asynchronous `listMachineOptions(profile)` to expose process-stable picker metadata; omit it when machine selection is not meaningful. Machine options contain only `id`, `label`, optional positive-integer `cpu` and `memoryGb`, and optional `default`. Session-placement providers declare a closed, unique, canonically ordered `supportedExecutionModes` tuple: `["worker-turn"]`, `["remote-exec"]`, or `["worker-turn", "remote-exec"]`. Empty lists, duplicates, unknown values, and noncanonical ordering are rejected. `worker-turn` requires a node lease; `remote-exec` accepts a node lease or an SSH lease. Omission advertises no session-placement modes while leaving direct lifecycle operations available. A direct environment create supplies no session execution mode; providers use their documented default, which is `worker-turn` for Crabbox. Providers whose bounded provisioning exceeds core's five-minute default may implement `resolveProvisionTimeoutMs(profile)` and include acquisition, provider-owned setup, and cleanup in the returned positive millisecond budget. Core also persists that validated settings snapshot and passes it with `leaseId` to `inspect({ leaseId, profile })` and `destroy({ leaseId, profile })`, including after the named profile is changed or removed. Destruction is idempotent, inspection returns the closed `active` / `destroyed` / `unknown` status union, and SSH private-key material is referenced only through `SecretRef`. Provisioned SSH endpoints must also include a public `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment, so core can pin the host before connecting. They may include up to 10 ordered, unique `fallbackPorts`, excluding the primary `port`; core persists those candidates and rotates among them only for idempotent probes, content-addressed transfers, receipt/lock-guarded artifact installation, convergent managed-worktree mirroring, and tunnel reconnects. Ambiguous unguarded stateful commands fail closed and are not replayed across candidates. A lease may set `sharedHost: true` when the SSH account also owns unrelated processes; core then avoids host-wide process freezing during workspace reconciliation. Omitted or `false` means a dedicated worker host. Active inspection repeats this fact so core can reconcile provider-owned isolation for leases persisted before the field existed; tunnel startup waits for that first authoritative inspection. Optional desktop metadata may advertise up to eight unique closed apps: `browser` with an absolute `executablePath` and a CDP port from 1 through 65535, or `terminal` with an absolute `executablePath`. Core rejects unknown app ids and fields and persists the validated metadata with the existing desktop record. Providers that mint dynamic identity refs may implement authoritative `resolveSshIdentity({ leaseId, profile, keyRef })`; providers without it use core's generic secret resolver. An authoritative `unknown` orphans an active local record; after a persisted destroy request it confirms teardown.
`contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth. An entitled route remains reachable while Gateway root-work admission is closed only when it also declares `auth: "gateway"` and the route-specific `gatewayRuntimeScopeSurface: "trusted-operator"`; ordinary sibling routes from the same plugin remain behind the admission boundary. This keeps suspension status and resume reachable without granting the whole plugin an admission bypass. Keep parsing and response shaping bounded outside dispatch; substantive or mutating work must go through Gateway method dispatch, which owns admission and scope enforcement.
docs/plugins/sdk-overview.md
+1 -1
View File
@@ -131,7 +131,7 @@ persists the start or invokes the provider. Provider aliases are lookup names
only and must not be used for this declaration.
Worker providers must also declare their id in `contracts.workerProviders`.
Core persists durable intent before `provision(profile, operationId, options?)`. Providers validate settings and any optional `options.machineClass` before external allocation and throw `WorkerProviderError` for permanent profile rejection. `provision` must adopt the same lease when the operation id repeats. If provider-owned setup fails after allocation and cleanup is indeterminate, throw `WorkerProviderError.cleanupIndeterminate(leaseId, provisionError, cleanupError)` so core persists the known lease and reconciles teardown instead of replaying provision. Providers may expose process-stable picker metadata with asynchronous `listMachineOptions(profile)`; omit the hook when the profile has no meaningful machine choice. Machine options contain only `id`, `label`, optional positive-integer `cpu` and `memoryGb`, and optional `default`. Session-placement providers declare exactly one `supportedExecutionModes` value: `worker-turn` providers return node leases, while `remote-exec` providers return SSH leases. Omission advertises no placement modes while preserving direct environment lifecycle calls. Providers whose provisioning can legitimately exceed core's five-minute default may return a positive millisecond budget from `resolveProvisionTimeoutMs(profile)`; include acquisition, provider-owned setup, and cleanup in that bound.
Core persists durable intent before `provision(profile, operationId, options?)`. Providers validate settings and any optional `options.machineClass` and `options.executionMode` before external allocation and throw `WorkerProviderError` for permanent profile rejection. `provision` must adopt the same lease for the same operation id and selected execution mode; a retry cannot silently change modes. If provider-owned setup fails after allocation and cleanup is indeterminate, throw `WorkerProviderError.cleanupIndeterminate(leaseId, provisionError, cleanupError)` so core persists the known lease and reconciles teardown instead of replaying provision. Providers may expose process-stable picker metadata with asynchronous `listMachineOptions(profile)`; omit the hook when the profile has no meaningful machine choice. Machine options contain only `id`, `label`, optional positive-integer `cpu` and `memoryGb`, and optional `default`. Session-placement providers declare one or both current `supportedExecutionModes` values in deterministic canonical order: `["worker-turn"]`, `["remote-exec"]`, or `["worker-turn", "remote-exec"]`. Empty lists, duplicate values, unknown modes, and noncanonical order are rejected. `worker-turn` requires a node lease; `remote-exec` accepts either a node lease or an existing SSH lease. Omission advertises no placement modes while preserving direct environment lifecycle calls. Direct environment creation without a session supplies no execution mode, so providers retain their intentional default setup; the bundled Crabbox provider defaults to `worker-turn`. Providers whose provisioning can legitimately exceed core's five-minute default may return a positive millisecond budget from `resolveProvisionTimeoutMs(profile)`; include acquisition, provider-owned setup, and cleanup in that bound.
Core persists the validated profile settings with the lease and supplies that snapshot to `destroy({ leaseId, profile })`, which must be idempotent, and `inspect({ leaseId, profile })`, which returns `active`, `destroyed`, or `unknown`. This lets providers route lifecycle calls after a gateway restart or named-profile removal. SSH endpoints use a `SecretRef` for `keyRef`, never inline key material, and include a `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment. Core pins `hostKey` and never trusts a key from the first connection. Providers may also return up to 10 ordered, unique `fallbackPorts` (integer ports from 1 through 65535, excluding the primary `port`); core validates and persists those advertised candidates for idempotent probes, content-addressed transfers, receipt/lock-guarded artifact installation, convergent managed-worktree mirroring, and tunnel reconnects. Ambiguous unguarded stateful commands fail closed and are not replayed across candidates. A lease may set `sharedHost: true` when the SSH account also owns unrelated processes; core then avoids host-wide process freezing during workspace reconciliation. Omitted or `false` means a dedicated worker host. Active inspection repeats this fact so core can reconcile provider-owned isolation for leases persisted before the field existed; tunnel startup waits for that first authoritative inspection. A provider that mints a dynamic `keyRef` can implement `resolveSshIdentity({ leaseId, profile, keyRef })`; when present, that resolver is authoritative, while providers without it use the configured generic secret resolver.
`WorkerLease.desktop` is optional and has the shape `{ protocol: "rfb"; port: number; passwordFilePath?: string; apps?: WorkerDesktopApp[] }`; `passwordFilePath`, when present, must be absolute. Providers report this warm-time capability from `provision`; it cannot be retrofitted onto a live lease. The owning SSH or node carrier reads the password on the worker when needed and never persists it in the Gateway store. `WorkerDesktopApp` is a closed union: `{ id: "browser"; executablePath: string; cdpPort: number }` or `{ id: "terminal"; executablePath: string }`. App ids must be unique, executable paths must be absolute, browser CDP ports must be integers from 1 through 65535, and the list accepts at most eight entries. Core rejects unknown ids and fields.
Providers with renewable leases can also implement `renew(leaseId)`.
docs/web/control-ui.md
+2 -2
View File
@@ -248,7 +248,7 @@ The sidebar organizes everything around the agent. The identity row at the top i
### Session placement
A selected session running on a worker shows a quiet **Runs on Cloud** chip in the chat header. Connections with `operator.write` can choose **Move session…** to continue on the Gateway or an eligible paired device, and can use **Stop cloud worker…** through the write-scoped `sessions.reclaim` lifecycle. Moving to a configured cloud profile requires `operator.admin`. Profiles with multiple machine classes show a machine picker; choosing the default omits an override, while choosing a different class on the current profile resizes the session. The confirmation explains that an active turn is interrupted and never replayed; OpenClaw reconciles the workspace before activating the destination. While the durable operation is in progress, the chip shows **Moving to…**. If recovery is blocked, the chip exposes the bounded error after reconnect so the action never fails silently.
A selected session running on a worker shows a quiet **Runs on Cloud** chip in the chat header. Connections with `operator.write` can choose **Move session…** to continue on the Gateway or an eligible paired device, and can use **Stop cloud worker…** through the write-scoped `sessions.reclaim` lifecycle. Moving to a configured cloud profile requires `operator.admin`. Cloud rows are filtered against all execution modes advertised by each profile: the same bundled Crabbox profile is selectable for OpenClaw `worker-turn` and Codex `remote-exec`, while a genuinely single-mode profile stays disabled for the other runtime. Profiles with multiple machine classes show a machine picker; choosing the default omits an override, while choosing a different class on the current profile resizes the session. The confirmation explains that an active turn is interrupted and never replayed; OpenClaw reconciles the workspace before activating the destination. While the durable operation is in progress, the chip shows **Moving to…**. If recovery is blocked, the chip exposes the bounded error after reconnect so the action never fails silently.
### Session icons
@@ -256,7 +256,7 @@ Choose **Set icon** from a single session's context menu to give its sidebar row
## New session page
The **+** in the sidebar's **Sessions** toolbar opens a full-page draft at `/new`: nothing is created until you send the first message. A unified **Place** picker chooses a Gateway project or folder and an execution destination. Connections with `operator.write` can choose **Gateway · local**, **Any available node**, or any paired device returned by `environments.list`; administrators additionally see configured cloud profiles and **Connect a machine…**. Automatic selection chooses the eligible host with the most available worker slots, breaking ties by device ID; runtimes that do not consume worker slots use device ID order. The device list stays authoritative to that environment catalog: worker turns require an available current session host with at least one free slot. Offline known hosts, connected non-hosts, saturated hosts, hosts without current capacity, outdated hosts, and unavailable hosts remain visible with a reason and next step.
The **+** in the sidebar's **Sessions** toolbar opens a full-page draft at `/new`: nothing is created until you send the first message. A unified **Place** picker chooses a Gateway project or folder and an execution destination. Connections with `operator.write` can choose **Gateway · local**, **Any available node**, or any paired device returned by `environments.list`; administrators additionally see configured cloud profiles and **Connect a machine…**. A cloud profile is selectable when its advertised execution modes include the selected runtime, so one Crabbox **Cloud · profile** row supports both OpenClaw and Codex. Automatic selection chooses the eligible host with the most available worker slots, breaking ties by device ID; runtimes that do not consume worker slots use device ID order. Device eligibility remains authoritative to the environment catalog and the selected runtime: OpenClaw `worker-turn` requires an available current session host with valid worker capacity and at least one free slot; Codex `remote-exec` requires its currently invocable, explicitly authorized exec-server command and consumes no worker slot. Offline known hosts, connected non-hosts, incompatible or saturated hosts, hosts missing required capabilities, outdated hosts, and unavailable hosts remain visible with a reason and next step.
The folder defaults to the agent workspace. Write-scoped connections can browse, restore recent Gateway folders, and start sessions anywhere inside a configured agent workspace; another absolute Gateway path requires `operator.admin` but can run directly without being a Git checkout. Local placement keeps the optional **Worktree** control with a base-branch picker backed by `worktrees.branches` (no fetch) and an optional worktree name (the branch becomes `openclaw/<name>`). Choosing either a device or cloud profile forces a managed worktree from the selected Gateway source. **Start in terminal** is available only for local placement.
extensions/codex/src/app-server/dynamic-tool-build.ts
+2 -2
View File
@@ -76,7 +76,7 @@ const CODEX_NATIVE_SANDBOX_TOOL_REQUIREMENTS = [
] as const;
const CODEX_MEMORY_FLUSH_DYNAMIC_TOOL_ALLOW = new Set(["read", "write"]);
/** Keeps paired-device filesystem and process ownership on its native exec-server. */
/** Keeps node filesystem and process ownership on its native exec-server. */
export function resolveCodexNodePlacementToolConstructionPlan(
sandbox: OpenClawSandboxContext | undefined,
nativeToolSurfaceEnabled: boolean | undefined,
@@ -92,7 +92,7 @@ export function resolveCodexNodePlacementToolConstructionPlan(
}
if (!nativeToolSurfaceEnabled) {
throw new Error(
"Codex paired-device remote execution requires its native exec-server tool surface; adjust the session tool policy and start a fresh attempt.",
"Codex node execution requires its native exec-server tool surface; adjust the session tool policy and start a fresh attempt.",
);
}
return {
extensions/codex/src/app-server/sandbox-exec-server-node-relay.ts
+3 -3
View File
@@ -47,7 +47,7 @@ export function createCodexNodeExecServerDisconnectError(reason: string, cause?:
CODEX_NODE_EXEC_SERVER_MAX_FAILURE_DETAIL_CHARS,
)}`;
return new Error(
`Codex paired execution device disconnected; start a fresh attempt. (${reason}${detail})`,
`Codex execution node disconnected; start a fresh attempt. (${reason}${detail})`,
);
}
@@ -92,7 +92,7 @@ export async function startCodexNodeExecServerRelay(params: {
params.lease.onChannelClosed = ({ failed, error }) =>
failUnexpectedly(
failed ? 1011 : 1001,
failed ? "execution device failed" : "execution device disconnected",
failed ? "execution node failed" : "execution node disconnected",
error,
);
socket.once("close", () => failUnexpectedly(1001, "execution socket closed"));
@@ -456,7 +456,7 @@ function createCredentialedCodexNodeHttpRejection(request: Record<string, unknow
error: {
code: -32602,
message:
"Authenticated remote HTTP is unavailable on paired devices; run on Gateway or use an intentionally credential-free endpoint.",
"Authenticated remote HTTP is unavailable on execution nodes; run on Gateway or use an intentionally credential-free endpoint.",
},
}),
);
extensions/codex/src/app-server/sandbox-exec-server.ts
+5 -7
View File
@@ -58,7 +58,7 @@ export async function ensureCodexSandboxExecServerEnvironment(params: {
return undefined;
}
if (placementNodeId && !params.runtime) {
throw new Error("Codex paired-device execution requires its active plugin runtime.");
throw new Error("Codex node execution requires its active plugin runtime.");
}
if (!canExposeLocalExecServerToAppServer(params.appServerStartOptions)) {
throw new Error(
@@ -162,7 +162,7 @@ async function acquireOpenClawExecServer(params: {
}
if (!runtime || !signal) {
await releaseOpenClawExecServer(server);
throw new Error("Codex paired-device execution requires an active runtime and attempt.");
throw new Error("Codex node execution requires an active runtime and attempt.");
}
try {
const placementIdentity = readCodexPlacementWorkspaceIdentity(sandbox);
@@ -183,7 +183,7 @@ async function acquireOpenClawExecServer(params: {
sandboxExecServerRegistry.servers.get(key) !== promise
) {
channel.close();
throw new Error("Codex paired-device execution retired before its channel was ready.");
throw new Error("Codex node execution retired before its channel was ready.");
}
const nodeLease = {
id: randomUUID(),
@@ -381,9 +381,7 @@ function readCodexPlacementWorkspaceIdentity(sandbox: SandboxContext): {
!sandbox.sessionKey ||
sandbox.sessionKey.trim() !== sandbox.sessionKey
) {
throw new Error(
"Codex paired-device execution requires its exact placement workspace identity.",
);
throw new Error("Codex node execution requires its exact placement workspace identity.");
}
return {
environmentId: sandbox.placementEnvironmentId,
@@ -446,7 +444,7 @@ function handleClosedCodexNodeExecServerLease(
try {
lease.onDisconnected?.(
createCodexNodeExecServerDisconnectError(
result.failed ? "execution device failed" : "execution device disconnected",
result.failed ? "execution node failed" : "execution node disconnected",
result.error,
),
);
extensions/codex/src/app-server/side-question.test.ts
+39 -33
View File
@@ -901,41 +901,47 @@ describe("runCodexAppServerSideQuestion", () => {
});
});
it("rejects paired-device side questions before acquiring a client, channel, or approval", async () => {
const client = createFakeClient();
getSharedCodexAppServerClientMock.mockResolvedValue(client);
const openDuplex = vi.fn(async () => {
throw new Error("paired-device side-question channel was opened");
});
const requestApproval = vi.fn(async () => undefined);
const sandbox = {
...createSandboxContext({}),
placementExecutionMode: "remote-exec" as const,
placementNodeId: "paired-device-1",
placementEnvironmentId: "environment-1",
placementSessionId: "session-1",
placementOwnerEpoch: 1,
sessionKey: "agent:main:session-1",
};
it.each([
{ host: "paired device", nodeId: "paired-device-1" },
{ host: "cloud worker", nodeId: "cloud-worker-node-1" },
])(
"rejects $host side questions before acquiring a client, channel, or approval",
async ({ nodeId }) => {
const client = createFakeClient();
getSharedCodexAppServerClientMock.mockResolvedValue(client);
const openDuplex = vi.fn(async () => {
throw new Error("node side-question channel was opened");
});
const requestApproval = vi.fn(async () => undefined);
const sandbox = {
...createSandboxContext({}),
placementExecutionMode: "remote-exec" as const,
placementNodeId: nodeId,
placementEnvironmentId: "environment-1",
placementSessionId: "session-1",
placementOwnerEpoch: 1,
sessionKey: "agent:main:session-1",
};
await expect(
runCodexAppServerSideQuestion(
sideParams({
sandbox,
hostCapabilities: { ...TEST_HOST_CAPABILITIES, requestApproval },
}),
{ runtime: { nodes: { openDuplex } } as never },
),
).rejects.toThrow(
"Normal Codex turns are supported on paired devices, but /btw is not yet bound to the active placement.",
);
await expect(
runCodexAppServerSideQuestion(
sideParams({
sandbox,
hostCapabilities: { ...TEST_HOST_CAPABILITIES, requestApproval },
}),
{ runtime: { nodes: { openDuplex } } as never },
),
).rejects.toThrow(
"Normal Codex turns are supported on nodes, but /btw is not yet bound to the active placement.",
);
expect(getSharedCodexAppServerClientMock).not.toHaveBeenCalled();
expect(openDuplex).not.toHaveBeenCalled();
expect(requestApproval).not.toHaveBeenCalled();
expect(client.request).not.toHaveBeenCalled();
expect(createOpenClawCodingToolsMock).not.toHaveBeenCalled();
});
expect(getSharedCodexAppServerClientMock).not.toHaveBeenCalled();
expect(openDuplex).not.toHaveBeenCalled();
expect(requestApproval).not.toHaveBeenCalled();
expect(client.request).not.toHaveBeenCalled();
expect(createOpenClawCodingToolsMock).not.toHaveBeenCalled();
},
);
it("rebinds side-question handlers when selection retry replaces the client", async () => {
const initialClient = createFakeClient();
extensions/codex/src/app-server/side-question.ts
+1 -1
View File
@@ -210,7 +210,7 @@ export async function runCodexAppServerSideQuestion(
}
if (isCodexPairedNodeRemoteExecPlacementSandbox(params.sandbox)) {
throw new Error(
"Normal Codex turns are supported on paired devices, but /btw is not yet bound to the active placement.",
"Normal Codex turns are supported on nodes, but /btw is not yet bound to the active placement.",
);
}
const pluginConfig = readCodexPluginConfig(options.pluginConfig);
extensions/codex/src/node-exec-server.test.ts
+10 -7
View File
@@ -1,4 +1,4 @@
/** Protects paired-node policy, real pinned Codex stdio framing, and child cleanup. */
/** Protects node policy, real pinned Codex stdio framing, and child cleanup. */
import { once } from "node:events";
import { access, readFile, realpath } from "node:fs/promises";
import { createServer } from "node:http";
@@ -136,8 +136,11 @@ afterEach(() => {
vi.unstubAllEnvs();
});
describe("Codex paired-node exec-server", () => {
it("requires exact one-time approval before the dangerous explicit-allowlist command runs", async () => {
describe("Codex node exec-server", () => {
it.each([
{ host: "paired device", nodeId: "paired-node" },
{ host: "cloud worker", nodeId: "cloud-worker-node" },
])("requires critical one-time approval on a $host", async ({ nodeId }) => {
const policy = createCodexNodeExecServerInvokePolicy();
expect(policy.commands).toEqual([CODEX_NODE_EXEC_SERVER_COMMAND]);
expect(policy.dangerous).toBe(true);
@@ -151,7 +154,7 @@ describe("Codex paired-node exec-server", () => {
const request = vi.fn();
const { placement } = createManagedWorkspaceInvocation(process.cwd());
const context = {
nodeId: "paired-node",
nodeId,
command: CODEX_NODE_EXEC_SERVER_COMMAND,
params: placement,
config: {},
@@ -195,14 +198,14 @@ describe("Codex paired-node exec-server", () => {
expect(invokeNode).toHaveBeenCalledWith({ params: approvedPlacement });
expect(request).toHaveBeenCalledWith(
expect.objectContaining({
title: "Run Codex execution on paired device",
description: expect.stringContaining(`paired-node: ${approvedPlacement.cwd}`),
title: "Run Codex execution on node",
description: expect.stringContaining(`${nodeId}: ${approvedPlacement.cwd}`),
severity: "critical",
allowedDecisions: ["allow-once"],
}),
);
expect(request.mock.lastCall?.[0].description).toContain(
"arbitrary processes and filesystem access across the paired-device account",
"arbitrary processes and filesystem access across the node account",
);
});
extensions/codex/src/node-exec-server.ts
+8 -8
View File
@@ -1,4 +1,4 @@
/** Declares the explicitly approved, lazily loaded paired-node Codex exec-server. */
/** Declares the explicitly approved, lazily loaded node-backed Codex exec-server. */
import type {
OpenClawPluginNodeHostCommand,
OpenClawPluginNodeInvokePolicy,
@@ -104,7 +104,7 @@ export function createCodexNodeExecServerCommand(): OpenClawPluginNodeHostComman
};
}
/** Keeps paired-device exec-server launch behind explicit arming and one-time approval. */
/** Keeps node exec-server launch behind explicit arming and one-time approval. */
export function createCodexNodeExecServerInvokePolicy(): OpenClawPluginNodeInvokePolicy {
return {
commands: [CODEX_NODE_EXEC_SERVER_COMMAND],
@@ -115,7 +115,7 @@ export function createCodexNodeExecServerInvokePolicy(): OpenClawPluginNodeInvok
return {
ok: false,
code: "CODEX_NODE_EXEC_APPROVAL_REQUIRED",
message: "Codex paired-device execution requires an available approval reviewer.",
message: "Codex node execution requires an available approval reviewer.",
};
}
let placement: ReturnType<typeof parseCodexNodePlacementWorkspace>;
@@ -125,13 +125,13 @@ export function createCodexNodeExecServerInvokePolicy(): OpenClawPluginNodeInvok
return {
ok: false,
code: "CODEX_NODE_EXEC_WORKSPACE_INVALID",
message: "Codex paired-device execution requires an exact managed placement workspace.",
message: "Codex node execution requires an exact managed placement workspace.",
};
}
const deviceName = context.node?.displayName ?? context.nodeId;
const nodeName = context.node?.displayName ?? context.nodeId;
const approval = await context.approvals.request({
title: "Run Codex execution on paired device",
description: `${deviceName}: ${placement.cwd}; allows arbitrary processes and filesystem access across the paired-device account, not only this workspace.`,
title: "Run Codex execution on node",
description: `${nodeName}: ${placement.cwd}; allows arbitrary processes and filesystem access across the node account, not only this workspace.`,
severity: "critical",
allowedDecisions: ["allow-once"],
});
@@ -139,7 +139,7 @@ export function createCodexNodeExecServerInvokePolicy(): OpenClawPluginNodeInvok
return {
ok: false,
code: "CODEX_NODE_EXEC_APPROVAL_DENIED",
message: "Codex paired-device execution requires one-time approval.",
message: "Codex node execution requires one-time approval.",
};
}
return await context.invokeNode({ params: placement });
extensions/crabbox/src/crabbox-worker-node-enrollment-diagnostics.ts
+51
View File
@@ -0,0 +1,51 @@
import { redactToolPayloadText } from "openclaw/plugin-sdk/logging-core";
import { truncateUtf8Prefix } from "openclaw/plugin-sdk/text-utility-runtime";
import { crabboxCommandError } from "./crabbox-worker-command-error.js";
import { runCrabboxCommand, type CrabboxCommandRunner } from "./crabbox-worker-command.js";
const NODE_ENROLLMENT_DIAGNOSTIC_TIMEOUT_MS = 60_000;
const MAX_NODE_ENROLLMENT_EVIDENCE_BYTES = 2_048;
export async function collectCrabboxNodeEnrollmentEvidence(params: {
args: string[];
binary: string;
id: string;
runCommand: CrabboxCommandRunner;
signal?: AbortSignal;
}): Promise<string> {
let label = "box evidence";
let detail: string;
try {
const result = await runCrabboxCommand({
action: "enrollment diagnostics",
args: params.args,
binary: params.binary,
input: [
`state_dir="$HOME/.openclaw/cloud-workers/${params.id}"`,
'printf "package-spec="',
'if [ -s "$state_dir/package-spec" ]; then head -c 256 "$state_dir/package-spec"; else printf absent; fi',
'printf " node-pid="',
'if [ -s "$state_dir/node.pid" ] && kill -0 "$(head -c 32 "$state_dir/node.pid")" 2>/dev/null; then printf alive; else printf dead-or-absent; fi',
'printf " node.log tail: "',
'if [ -r "$state_dir/node.log" ]; then tail -c 2000 "$state_dir/node.log"; else printf absent; fi',
].join("\n"),
runCommand: params.runCommand,
...(params.signal ? { signal: params.signal } : {}),
// The enrollment deadline has already elapsed; diagnostics need their own bounded budget.
timeoutMs: NODE_ENROLLMENT_DIAGNOSTIC_TIMEOUT_MS,
});
if (result.termination !== "exit" || result.code !== 0) {
throw crabboxCommandError("enrollment diagnostics", result);
}
detail = result.stdout.trim();
if (!detail) {
throw new Error("diagnostic command returned no output");
}
} catch (error) {
label = "box evidence unavailable";
detail = error instanceof Error ? error.message : "diagnostic command failed";
}
const prefix = `${label}: `;
const safeDetail = redactToolPayloadText(detail).replace(/\s+/gu, " ").trim();
return `${prefix}${truncateUtf8Prefix(safeDetail, MAX_NODE_ENROLLMENT_EVIDENCE_BYTES - prefix.length)}`;
}
extensions/crabbox/src/crabbox-worker-node-enrollment.ts
+42 -1
View File
@@ -14,9 +14,10 @@ function shellQuote(value: string): string {
export function createCrabboxNodeEnrollmentSetup(params: {
enrollment: CrabboxWorkerNodeEnrollment;
executionMode?: NonNullable<WorkerProvider["supportedExecutionModes"]>[number];
leaseId: string;
}): { command: string; forwardedEnv?: Record<string, string> } {
const { enrollment, leaseId } = params;
const { enrollment, executionMode, leaseId } = params;
const stateDir = `.openclaw/cloud-workers/${leaseId}`;
const packageCandidates = enrollment.packageSpecs.map(shellQuote).join(" ");
if (!packageCandidates) {
@@ -30,12 +31,50 @@ export function createCrabboxNodeEnrollmentSetup(params: {
'setup_code_file="$state_dir/setup-code"',
"umask 077",
`printf "%s\\n" "$${CLOUD_SETUP_CODE_ENV}" >"$setup_code_file"`,
`unset ${CLOUD_SETUP_CODE_ENV}`,
]
: [];
const launch =
enrollment.mode === "connect"
? `connect --target-file "$setup_code_file" --ephemeral --display-name ${shellQuote(enrollment.displayName)}`
: `node run --ephemeral --display-name ${shellQuote(enrollment.displayName)}`;
const prepareCodex = (binary: string): string[] => {
if (executionMode !== "remote-exec") {
return [];
}
const inspectPlugin = [
'const fs=require("node:fs"),path=require("node:path"),module=require("node:module");',
'const inspection=JSON.parse(fs.readFileSync(0,"utf8")),plugin=inspection.plugin;',
`const version=${JSON.stringify(enrollment.openclawVersion)};`,
'if(plugin?.id!=="codex"||plugin.packageName!=="@openclaw/codex"||plugin.packageVersion!==version||(plugin.origin!=="bundled"&&(plugin.trustedOfficialInstall!==true||inspection.install?.source!=="npm"))){',
"throw new Error(`Codex remote-exec requires the exact official @openclaw/codex@${version} plugin to be installed by cloudWorkers profile setup`)}",
"const root=fs.realpathSync(plugin.rootDir);",
'const manifest=JSON.parse(fs.readFileSync(path.join(root,"package.json"),"utf8"));',
'const requirePlugin=module.createRequire(path.join(root,"package.json"));',
'const runtime=requirePlugin("@openai/codex/package.json");',
'if(manifest.name!==plugin.packageName||manifest.version!==version||runtime.version!==manifest.dependencies?.["@openai/codex"]){',
'throw new Error("Codex remote-exec requires the plugin and its exact pinned native runtime")}',
'const launcher=requirePlugin.resolve("@openai/codex/bin/codex.js");',
'const probe=require("node:child_process").spawnSync(process.execPath,[launcher,"--version"],{encoding:"utf8",timeout:10000,stdio:["ignore","pipe","pipe"]});',
"if(probe.status!==0||probe.stdout?.trim()!==`codex-cli ${runtime.version}`){",
'throw new Error("Codex remote-exec requires the exact executable platform-native Codex binary")}',
'if(plugin.origin!=="bundled"){',
'const project=path.join(process.argv[1],"npm","projects","codex");',
'const packageRoot=path.join(project,"node_modules","@openclaw");',
"fs.mkdirSync(packageRoot,{recursive:true,mode:0o700});",
'const dependency={"@openclaw/codex":version};',
'fs.writeFileSync(path.join(project,"package.json"),JSON.stringify({name:"openclaw-cloud-codex",private:true,dependencies:dependency})+"\\n",{mode:0o600});',
'const projected=path.join(packageRoot,"codex");',
"try{const existing=fs.lstatSync(projected);",
'if(!existing.isSymbolicLink()||fs.realpathSync(projected)!==root){throw new Error("Codex node plugin path is occupied")}',
'}catch(error){if(error.code!=="ENOENT"){throw error}fs.symlinkSync(root,projected)}',
"}",
].join("");
return [
` ${binary} plugins inspect codex --json | node -e ${shellQuote(inspectPlugin)} "$state_dir"`,
` OPENCLAW_STATE_DIR="$state_dir" ${binary} plugins enable codex`,
];
};
const command = [
"set -eu",
`state_dir="$HOME/${stateDir}"`,
@@ -67,8 +106,10 @@ export function createCrabboxNodeEnrollmentSetup(params: {
"fi",
'package_spec="$(cat "$package_spec_file")"',
'if [ "$package_spec" = "@global" ]; then',
...prepareCodex("openclaw"),
` setsid -f sh -c 'printf "%s\\n" "$$" >"$1"; shift; exec "$@"' sh "$pid_file" env OPENCLAW_STATE_DIR="$state_dir" openclaw ${launch} >"$state_dir/node.log" 2>&1 </dev/null`,
"else",
...prepareCodex('npx --yes --package "$package_spec" -- openclaw'),
` setsid -f sh -c 'printf "%s\\n" "$$" >"$1"; shift; exec "$@"' sh "$pid_file" env OPENCLAW_STATE_DIR="$state_dir" npx --yes --package "$package_spec" -- openclaw ${launch} >"$state_dir/node.log" 2>&1 </dev/null`,
"fi",
'for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s "$pid_file" ] && break; sleep 0.1; done',
extensions/crabbox/src/crabbox-worker-provider.test.ts
+96 -8
View File
@@ -20,6 +20,7 @@ import { createCrabboxWorkerProvider, resolveOpenClawRoot } from "./crabbox-work
import {
CRABBOX_LIFECYCLE_TIMEOUT_MS,
CRABBOX_MACHINE_CATALOG_TIMEOUT_MS,
resolveCrabboxProvisionBaseTimeoutMs,
} from "./crabbox-worker-timeouts.js";
const OPERATION_ID = `provision:v2:${"0".repeat(64)}`;
@@ -174,7 +175,7 @@ describe("Crabbox worker provider", () => {
]),
});
});
expect(provider.supportedExecutionModes).toEqual(["worker-turn"]);
expect(provider.supportedExecutionModes).toEqual(["worker-turn", "remote-exec"]);
expect(await provider.listMachineOptions?.(PROFILE)).toEqual([
{ id: "tiny", label: "Tiny", cpu: 8, memoryGb: 16 },
{ id: "small", label: "Small", cpu: 16, memoryGb: 32 },
@@ -367,9 +368,15 @@ describe("Crabbox worker provider", () => {
expect(() => createCrabboxWorkerProvider({ wallpaperPath })).toThrow(message);
});
it("returns the environment-bound node after enrollment", async () => {
it.each([
{ name: "the direct-environment default", executionMode: undefined },
{ name: "an OpenClaw worker turn", executionMode: "worker-turn" },
{ name: "a Codex remote-exec turn", executionMode: "remote-exec" },
] as const)("returns the same enrolled node transport for $name", async ({ executionMode }) => {
const calls: Array<{ argv: string[]; options: Parameters<CrabboxCommandRunner>[1] }> = [];
let warmed = false;
const provider = providerWithRunner(async (argv) => {
const provider = providerWithRunner(async (argv, options) => {
calls.push({ argv, options });
if (argv[1] === "warmup") {
warmed = true;
return commandResult({ stdout: `leased ${LEASE_ID} slug=test\n` });
@@ -386,11 +393,69 @@ describe("Crabbox worker provider", () => {
: commandResult({ code: 4, stderr: `lease/server not found: ${argv.at(-2)}` });
});
await expect(provider.provision(PROFILE, OPERATION_ID)).resolves.toEqual({
const provision =
executionMode === undefined
? provider.provision(PROFILE, OPERATION_ID)
: provider.provision(PROFILE, OPERATION_ID, { executionMode });
await expect(provision).resolves.toEqual({
leaseId: LEASE_ID,
node: { deviceId: "device-1" },
sharedHost: false,
});
const enrollmentCall = calls.find(
(call) => call.argv[1] === "run" && String(call.options.input).includes("--ephemeral"),
);
expect(enrollmentCall).toBeDefined();
const setup = String(enrollmentCall?.options.input);
const setupCodeCleared = "unset CRABBOX_WORKER_SETUP_CODE";
expect(setup).toContain(setupCodeCleared);
expect(setup.indexOf(setupCodeCleared)).toBeGreaterThan(setup.indexOf('>"$setup_code_file"'));
expect(setup.indexOf(setupCodeCleared)).toBeLessThan(setup.indexOf("setsid -f sh -c"));
if (executionMode === "remote-exec") {
expect(setup).toContain("plugins inspect codex --json");
expect(setup).toContain('require("node:child_process").spawnSync');
expect(setup).toContain('[launcher,"--version"]');
expect(setup).toContain("codex-cli ${runtime.version}");
expect(setup).not.toContain("$state_dir/extensions/codex");
expect(setup).toContain('OPENCLAW_STATE_DIR="$state_dir" openclaw plugins enable codex');
expect(setup).toContain(
'OPENCLAW_STATE_DIR="$state_dir" npx --yes --package "$package_spec" -- openclaw plugins enable codex',
);
expect(setup.indexOf("plugins inspect codex --json")).toBeGreaterThan(
setup.indexOf(setupCodeCleared),
);
expect(setup.indexOf("plugins enable codex")).toBeLessThan(
setup.lastIndexOf("setsid -f sh -c"),
);
} else {
expect(setup).not.toContain("plugins inspect codex");
}
expect(setup).not.toContain("plugins install");
expect(setup).not.toContain("npm:@openclaw/codex");
expect(setup).toContain("connect --target-file");
expect(setup).toContain("--ephemeral");
expect(setup).not.toContain("secret-setup-value");
const commandArguments = calls.flatMap((call) => call.argv);
for (const forbiddenArgument of ["remote-exec", "worker-turn", "ssh", "scp", "rsync"]) {
expect(commandArguments).not.toContain(forbiddenArgument);
}
});
it("rejects an unsupported execution mode before invoking Crabbox", async () => {
const calls: string[][] = [];
const provider = providerWithRunner(async (argv) => {
calls.push(argv);
return commandResult();
});
await expect(
provider.provision(PROFILE, OPERATION_ID, { executionMode: "unsupported" as never }),
).rejects.toMatchObject({
name: "WorkerProviderError",
message: "Crabbox execution mode is unsupported",
});
expect(calls).toEqual([]);
});
it("resumes a bound node without replaying the consumed setup code", async () => {
@@ -1271,6 +1336,29 @@ describe("Crabbox worker provider", () => {
expect(calls.map((argv) => argv[1])).toEqual(["config"]);
});
const provisionTimeoutCases = [
{ name: "normal without setup", profile: { ...PROFILE }, minutes: 67 },
{
name: "normal with setup",
profile: { ...PROFILE, setup: "install-node" },
minutes: 82,
},
{ name: "desktop without setup", profile: { ...PROFILE, desktop: true }, minutes: 132 },
{
name: "desktop with setup",
profile: { ...PROFILE, desktop: true, setup: "install-node" },
minutes: 147,
},
] satisfies Array<{ name: string; profile: WorkerProfile; minutes: number }>;
it.each(provisionTimeoutCases)(
"includes warmup, lifecycle, setup, and node enrollment for $name",
({ profile, minutes }) => {
const provider = providerWithRunner(async () => commandResult());
expect(provider.resolveProvisionTimeoutMs?.(profile)).toBe(minutes * 60_000);
},
);
it.each([
{
name: "direct AWS",
@@ -1349,7 +1437,7 @@ describe("Crabbox worker provider", () => {
});
expect(calls.find((call) => call.argv[1] === "warmup")).toEqual(
expect.objectContaining({
options: expect.objectContaining({ timeoutMs: 50 * 60_000 }),
options: expect.objectContaining({ timeoutMs: 100 * 60_000 }),
}),
);
expect(calls.find((call) => call.argv[1] === "warmup")?.argv.slice(-4)).toEqual([
@@ -1364,7 +1452,7 @@ describe("Crabbox worker provider", () => {
provider: providerId,
desktop: true,
}),
).toBe(72 * 60_000);
).toBe(132 * 60_000);
expect(setupOrder).toEqual(["desktop", "enrollment"]);
});
@@ -1632,7 +1720,7 @@ describe("Crabbox worker provider", () => {
"--keep=true",
]);
expect(calls[0]?.options).toEqual({
timeoutMs: 240_000,
timeoutMs: 50 * 60_000,
maxOutputBytes: 65_536,
killProcessTree: true,
});
@@ -1948,7 +2036,7 @@ describe("Crabbox worker provider", () => {
pathEnv: "",
isExecutable: (candidate) => candidate === SIBLING_BINARY,
sleep: async () => {
nowMs += 290_001;
nowMs += resolveCrabboxProvisionBaseTimeoutMs({}) + 1;
},
wallpaperPath: WORKER_WALLPAPER_PATH,
});
extensions/crabbox/src/crabbox-worker-provider.ts
+19 -47
View File
@@ -1,4 +1,4 @@
import { redactSensitiveText, redactToolPayloadText } from "openclaw/plugin-sdk/logging-core";
import { redactSensitiveText } from "openclaw/plugin-sdk/logging-core";
import {
WorkerProviderError,
type WorkerLease,
@@ -8,7 +8,7 @@ import {
} from "openclaw/plugin-sdk/plugin-entry";
import { runCommandWithTimeout, type SpawnResult } from "openclaw/plugin-sdk/process-runtime";
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
import { truncateUtf16Safe, truncateUtf8Prefix } from "openclaw/plugin-sdk/text-utility-runtime";
import { truncateUtf16Safe } from "openclaw/plugin-sdk/text-utility-runtime";
import {
crabboxCommandError,
permanentCrabboxCommandError,
@@ -27,6 +27,7 @@ import {
import { createCrabboxHeartbeatManager } from "./crabbox-worker-heartbeat.js";
import { parseInspectJson, type ParsedInspect } from "./crabbox-worker-inspect.js";
import { createCrabboxMachineOptionsResolver } from "./crabbox-worker-machine-options.js";
import { collectCrabboxNodeEnrollmentEvidence } from "./crabbox-worker-node-enrollment-diagnostics.js";
import {
createCrabboxNodeEnrollmentSetup,
type CrabboxWorkerNodeEnrollment,
@@ -56,8 +57,6 @@ export { resolveOpenClawRoot } from "./crabbox-worker-profile.js";
const READY_POLL_INTERVAL_MS = 2_000;
const MAX_ERROR_DETAIL_CHARS = 512;
const NODE_ENROLLMENT_DIAGNOSTIC_TIMEOUT_MS = 60_000;
const MAX_NODE_ENROLLMENT_EVIDENCE_BYTES = 2_048;
// Only states that prove the resource is gone or stopped map to `destroyed`. Crabbox also
// treats `deleting` and `failed` as unable to become ready, but those can retain resources
// that still need an explicit stop during teardown.
@@ -334,46 +333,6 @@ async function runProvisionSetupAndWaitReady(
return await waitForProvisionReady({ ...params, refresh: true });
}
async function collectNodeEnrollmentEvidence(
params: LeaseCommandContext & { runCommand: CrabboxCommandRunner; signal?: AbortSignal },
): Promise<string> {
let label = "box evidence";
let detail: string;
try {
const result = await runCrabboxCommand({
action: "enrollment diagnostics",
args: crabboxLeaseRunArgs(params),
binary: params.binary,
input: [
`state_dir="$HOME/.openclaw/cloud-workers/${params.id}"`,
'printf "package-spec="',
'if [ -s "$state_dir/package-spec" ]; then head -c 256 "$state_dir/package-spec"; else printf absent; fi',
'printf " node-pid="',
'if [ -s "$state_dir/node.pid" ] && kill -0 "$(head -c 32 "$state_dir/node.pid")" 2>/dev/null; then printf alive; else printf dead-or-absent; fi',
'printf " node.log tail: "',
'if [ -r "$state_dir/node.log" ]; then tail -c 2000 "$state_dir/node.log"; else printf absent; fi',
].join("\n"),
runCommand: params.runCommand,
...(params.signal ? { signal: params.signal } : {}),
// The enrollment deadline has already elapsed; diagnostics need their own bounded budget.
timeoutMs: NODE_ENROLLMENT_DIAGNOSTIC_TIMEOUT_MS,
});
if (result.termination !== "exit" || result.code !== 0) {
throw crabboxCommandError("enrollment diagnostics", result);
}
detail = result.stdout.trim();
if (!detail) {
throw new Error("diagnostic command returned no output");
}
} catch (error) {
label = "box evidence unavailable";
detail = error instanceof Error ? error.message : "diagnostic command failed";
}
const prefix = `${label}: `;
const safeDetail = redactToolPayloadText(detail).replace(/\s+/gu, " ").trim();
return `${prefix}${truncateUtf8Prefix(safeDetail, MAX_NODE_ENROLLMENT_EVIDENCE_BYTES - prefix.length)}`;
}
async function stopProvisionId(params: {
binary: string;
id: string;
@@ -526,7 +485,7 @@ export function createCrabboxWorkerProvider(
id: CRABBOX_WORKER_PROVIDER_ID,
dispose: () => heartbeats.dispose(),
listMachineOptions,
supportedExecutionModes: ["worker-turn"],
supportedExecutionModes: ["worker-turn", "remote-exec"],
provisionBeforeInstallation: true,
requiresNodeEnrollment: true,
resolveProvisionTimeoutMs(profile) {
@@ -537,6 +496,14 @@ export function createCrabboxWorkerProvider(
operationId: string,
options: Parameters<WorkerProvider["provision"]>[2],
): Promise<WorkerLease> {
const executionMode: unknown = options?.executionMode;
if (
executionMode !== undefined &&
executionMode !== "worker-turn" &&
executionMode !== "remote-exec"
) {
throw new WorkerProviderError("Crabbox execution mode is unsupported");
}
const configured = parseCrabboxProfile(profile);
const requestedClass = nonEmptyString(options?.machineClass);
if (options?.machineClass !== undefined && (!requestedClass || requestedClass.length > 128)) {
@@ -665,7 +632,11 @@ export function createCrabboxWorkerProvider(
}
return await failProvisionAfterCleanup({ ...inspectedParams, id: leaseId }, error);
}
const nodeEnrollmentSetup = createCrabboxNodeEnrollmentSetup({ enrollment, leaseId });
const nodeEnrollmentSetup = createCrabboxNodeEnrollmentSetup({
enrollment,
executionMode,
leaseId,
});
inspectedParams.inspect = await runProvisionSetupAndWaitReady({
...inspectedParams,
setup: nodeEnrollmentSetup.command,
@@ -685,8 +656,9 @@ export function createCrabboxWorkerProvider(
}
const leaseContext = { ...inspectedParams, id: leaseId };
// Read node evidence before cleanup destroys its only copy on the leased machine.
const evidence = await collectNodeEnrollmentEvidence({
const evidence = await collectCrabboxNodeEnrollmentEvidence({
...leaseContext,
args: crabboxLeaseRunArgs(leaseContext),
...(enrollment.signal ? { signal: enrollment.signal } : {}),
});
enrollment.signal?.throwIfAborted();
extensions/crabbox/src/crabbox-worker-timeouts.ts
+17 -10
View File
@@ -3,7 +3,17 @@ type CrabboxProvisionTimeoutProfile = {
setup?: string;
};
export const CRABBOX_WARMUP_TIMEOUT_MS = 240_000;
const CRABBOX_ACQUISITION_ENVELOPE_MS = 5 * 60_000;
const CRABBOX_BOOTSTRAP_TIMEOUT_MS = 20 * 60_000;
const CRABBOX_DESKTOP_BOOTSTRAP_TIMEOUT_MS = 45 * 60_000;
const CRABBOX_WARMUP_ATTEMPTS = 2;
// Crabbox allows 20m Linux / 45m desktop bootstrap plus one fresh-lease retry;
// include acquisition for both attempts so OpenClaw cannot preempt readiness.
export const CRABBOX_WARMUP_TIMEOUT_MS =
CRABBOX_WARMUP_ATTEMPTS * (CRABBOX_ACQUISITION_ENVELOPE_MS + CRABBOX_BOOTSTRAP_TIMEOUT_MS);
export const CRABBOX_DESKTOP_WARMUP_TIMEOUT_MS =
CRABBOX_WARMUP_ATTEMPTS *
(CRABBOX_ACQUISITION_ENVELOPE_MS + CRABBOX_DESKTOP_BOOTSTRAP_TIMEOUT_MS);
export const CRABBOX_LIFECYCLE_TIMEOUT_MS = 60_000;
// AWS coordinator heartbeat latency reached 107.6 seconds in production measurements.
export const CRABBOX_HEARTBEAT_TIMEOUT_MS = 150_000;
@@ -13,21 +23,18 @@ export const CRABBOX_HEARTBEAT_TIMEOUT_MS = 150_000;
// lifecycle budget — a hung binary must fall back to label-only choices
// promptly instead of stalling the whole cloud picker.
export const CRABBOX_MACHINE_CATALOG_TIMEOUT_MS = 5_000;
const CRABBOX_PROVISION_TIMEOUT_MS = 290_000;
// Crabbox starts its 45-minute desktop/browser bootstrap clock after acquisition.
// Preserve OpenClaw's existing five-minute acquisition envelope, then leave one
// lifecycle allowance for post-warmup inspection and cleanup.
export const CRABBOX_DESKTOP_WARMUP_TIMEOUT_MS = 50 * 60_000;
const CRABBOX_DESKTOP_PROVISION_TIMEOUT_MS =
CRABBOX_DESKTOP_WARMUP_TIMEOUT_MS + CRABBOX_LIFECYCLE_TIMEOUT_MS;
// Setup gets its own budget on top of provision so a slow warmup cannot starve it.
export const CRABBOX_SETUP_TIMEOUT_MS = 300_000;
// Setup may install an exact candidate CLI and official plugins on a minimal cloud image.
export const CRABBOX_SETUP_TIMEOUT_MS = 15 * 60_000;
export const CRABBOX_NODE_ENROLLMENT_TIMEOUT_MS = 15 * 60_000;
export function resolveCrabboxProvisionBaseTimeoutMs(
profile: CrabboxProvisionTimeoutProfile,
): number {
return profile.desktop ? CRABBOX_DESKTOP_PROVISION_TIMEOUT_MS : CRABBOX_PROVISION_TIMEOUT_MS;
const warmupTimeoutMs = profile.desktop
? CRABBOX_DESKTOP_WARMUP_TIMEOUT_MS
: CRABBOX_WARMUP_TIMEOUT_MS;
return warmupTimeoutMs + CRABBOX_LIFECYCLE_TIMEOUT_MS;
}
export function countCrabboxProvisionSetupPhases(profile: CrabboxProvisionTimeoutProfile): number {
packages/gateway-protocol/src/schema/environments.test.ts
+23 -2
View File
@@ -254,7 +254,8 @@ describe("worker environment protocol schemas", () => {
id: "aws",
providerId: "crabbox",
trust: "disposable",
executionMode: "remote-exec",
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
machines: [
{
id: "standard",
@@ -265,7 +266,13 @@ describe("worker environment protocol schemas", () => {
},
],
},
{ id: "worker", providerId: "static-ssh", executionMode: "worker-turn" },
{
id: "worker",
providerId: "static-ssh",
executionMode: "remote-exec",
executionModes: ["remote-exec"],
},
{ id: "legacy-primary", providerId: "static-ssh", executionMode: "worker-turn" },
{ id: "legacy", providerId: "static-ssh" },
],
}),
@@ -288,6 +295,20 @@ describe("worker environment protocol schemas", () => {
profiles: [{ id: "aws", providerId: "crabbox", executionMode: "sandbox" }],
}),
).toBe(false);
for (const executionModes of [
[],
["worker-turn", "worker-turn"],
["remote-exec", "worker-turn"],
["worker-turn", "sandbox"],
["worker-turn", "remote-exec", "worker-turn"],
]) {
expect(
Value.Check(EnvironmentsListResultSchema, {
environments: [],
profiles: [{ id: "aws", providerId: "crabbox", executionModes }],
}),
).toBe(false);
}
expect(
Value.Check(EnvironmentsListResultSchema, {
environments: [],
packages/gateway-protocol/src/schema/environments.ts
+6
View File
@@ -146,6 +146,12 @@ const WorkerEnvironmentProfileSummarySchema = closedObject({
providerId: NonEmptyString,
trust: Type.Optional(EnvironmentTrustSchema),
executionMode: Type.Optional(WorkerExecutionModeSchema),
executionModes: Type.Optional(
Type.Union([
Type.Tuple([WorkerExecutionModeSchema]),
Type.Tuple([Type.Literal("worker-turn"), Type.Literal("remote-exec")]),
]),
),
machines: Type.Optional(WorkerMachineOptionsSchema),
});
src/gateway/node-registry-private.ts
+10 -4
View File
@@ -84,7 +84,11 @@ export type NodeWorkerSupervisorTransport = {
node: NodeWorkerSupervisorNodeProof,
observation: NodeWorkerBundleStatusObservation | undefined,
): boolean;
isCurrent(node: NodeWorkerSupervisorNodeProof, requireLaunchEligibility?: boolean): boolean;
isCurrent(
node: NodeWorkerSupervisorNodeProof,
requireLaunchEligibility?: boolean,
requiredCommands?: readonly string[],
): boolean;
invoke(params: {
node: NodeWorkerSupervisorNodeProof;
command: NodeWorkerPrivateCommand;
@@ -193,6 +197,7 @@ function isWorkerSupervisorProofCurrent(
state: NodeRegistryPrivateState,
proof: NodeWorkerSupervisorNodeProof,
requireLaunchEligibility: boolean,
requiredCommands: readonly string[] = [],
): boolean {
const node = state.context.getNode(proof.nodeId);
if (!node || node.client.invalidated === true || node.connId !== proof.connId) {
@@ -205,7 +210,8 @@ function isWorkerSupervisorProofCurrent(
current.clientId === proof.clientId &&
current.clientMode === proof.clientMode &&
current.protocolFeature === proof.protocolFeature &&
(!requireLaunchEligibility || current.workerHost.capacity.available > 0)
(!requireLaunchEligibility || current.workerHost.capacity.available > 0) &&
requiredCommands.every((command) => current.commands.includes(command))
);
}
@@ -489,8 +495,8 @@ export function registerNodeRegistryPrivateRuntime(
}
return true;
},
isCurrent: (node, requireLaunchEligibility = false) =>
isWorkerSupervisorProofCurrent(state, node, requireLaunchEligibility),
isCurrent: (node, requireLaunchEligibility = false, requiredCommands = []) =>
isWorkerSupervisorProofCurrent(state, node, requireLaunchEligibility, requiredCommands),
invoke: async (params) => {
if (!NODE_WORKER_PRIVATE_COMMANDS.includes(params.command)) {
return {
src/gateway/node-registry.test.ts
+8
View File
@@ -881,6 +881,14 @@ describe("gateway/node-registry", () => {
capacity: { total: 2, available: 2 },
bundlePrewarm: 1,
});
expect(
priorProof && nodeWorkerSupervisorTransport.isCurrent(priorProof, true, ["system.run"]),
).toBe(true);
expect(nodeRegistry.updateSurface("node-1", { commands: [] })).not.toBeNull();
expect(priorProof && nodeWorkerSupervisorTransport.isCurrent(priorProof, true)).toBe(true);
expect(
priorProof && nodeWorkerSupervisorTransport.isCurrent(priorProof, true, ["system.run"]),
).toBe(false);
});
it("rejects generation-mismatched lookup and dispatch without invalidating the session", async () => {
src/gateway/server-methods/environments.test.ts
+22 -28
View File
@@ -471,21 +471,22 @@ describe("environment gateway methods", () => {
expect(worker?.worker).not.toHaveProperty("sshEndpoint");
expect(worker?.worker).not.toHaveProperty("keyRef");
expect(service.list).toHaveBeenCalledOnce();
for (const profile of (payload as { profiles: Array<Record<string, unknown>> }).profiles) {
expect(profile).not.toHaveProperty("executionMode");
expect(profile).not.toHaveProperty("executionModes");
}
});
it("adds known provider capabilities to configured profile summaries", async () => {
const standardMachine = {
id: "standard",
label: "Standard",
cpu: 32,
memoryGb: 64,
default: true,
};
const listMachineOptions = vi.fn(async (profileId: string) =>
profileId === "aws"
? [
{
id: "standard",
label: "Standard",
cpu: 32,
memoryGb: 64,
default: true,
},
]
: undefined,
profileId === "aws" ? [standardMachine] : undefined,
);
const [ok, payload] = await callEnvironmentMethod(
"environments.list",
@@ -494,7 +495,7 @@ describe("environment gateway methods", () => {
service: workerService({
listMachineOptions,
supportsExecutionMode: vi.fn(
(profileId, mode) => profileId === "aws" && mode === "remote-exec",
(profileId, mode) => profileId === "aws" || mode === "remote-exec",
),
}),
},
@@ -506,26 +507,19 @@ describe("environment gateway methods", () => {
{
id: "aws",
providerId: "crabbox",
executionMode: "remote-exec",
machines: [
{
id: "standard",
label: "Standard",
cpu: 32,
memoryGb: 64,
default: true,
},
],
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
machines: [standardMachine],
},
{
id: "zeta",
providerId: "static-ssh",
executionMode: "remote-exec",
executionModes: ["remote-exec"],
},
{ id: "zeta", providerId: "static-ssh" },
],
});
expect(listMachineOptions.mock.calls).toEqual([["aws"], ["zeta"]]);
expect(
(payload as { profiles: Array<Record<string, unknown>> }).profiles.find(
(profile) => profile.id === "zeta",
),
).not.toHaveProperty("executionMode");
});
it("projects trust from recorded worker isolation without guessing unknown leases", () => {
src/gateway/server-methods/environments.ts
+6 -2
View File
@@ -239,11 +239,15 @@ async function listWorkerProfilesWithMachines(context: GatewayRequestContext) {
const summaries = listWorkerProfiles(context);
return await Promise.all(
summaries.map(async (summary) => {
const executionMode = (["worker-turn", "remote-exec"] as const).find(
const executionModes = (["worker-turn", "remote-exec"] as const).filter(
(mode) =>
context.workerEnvironmentService?.supportsExecutionMode?.(summary.id, mode) === true,
);
const resolvedSummary = Object.assign(summary, executionMode ? { executionMode } : {});
const executionMode = executionModes[0];
const resolvedSummary = Object.assign(
summary,
executionMode ? { executionMode, executionModes } : {},
);
try {
const options = await context.workerEnvironmentService?.listMachineOptions?.(summary.id);
const machines = options ?? [];
src/gateway/server-methods/sessions-dispatch.ts
+6 -7
View File
@@ -163,18 +163,17 @@ async function validateDispatchExecutionMode(params: {
respondInvalidWorkerSession(params.respond, eligibility.error);
return false;
}
const environmentService = params.context.workerEnvironmentService;
if (
params.executionMode !== "remote-exec" ||
params.context.workerEnvironmentService?.supportsExecutionMode?.(
params.target.profileId,
params.executionMode,
) === true
(params.executionMode === "worker-turn" && !environmentService?.supportsExecutionMode) ||
environmentService?.supportsExecutionMode?.(params.target.profileId, params.executionMode) ===
true
) {
return true;
}
respondInvalidWorkerSession(
params.respond,
`selected cloud worker provider does not support the remote-exec execution mode required by runtime ${params.sessionRuntime}; use an approved paired device or a provider that advertises remote-exec`,
`runtime ${params.sessionRuntime} requires a cloud worker provider that supports ${params.executionMode}; choose a compatible provider, or select an agent/model route with agentRuntime.id "openclaw"`,
);
return false;
}
@@ -446,7 +445,7 @@ export const sessionDispatchHandlers: GatewayRequestHandlers = {
agentId: target.target.agentId,
executionMode,
...dispatchTarget,
...(dispatchTarget.deviceId && devicePlacement ? { devicePlacement } : {}),
...(devicePlacement ? { devicePlacement } : {}),
},
() =>
emitSessionsChanged(context, {
src/gateway/server-methods/sessions.dispatch.device.test.ts
+37
View File
@@ -684,6 +684,43 @@ describe("sessions.dispatch device targets", () => {
}),
);
});
it("carries runtime-owned node command requirements into cloud-profile dispatch", async () => {
useDeviceSession("codex");
const dispatch = vi.fn().mockRejectedValue(new Error("cloud-profile dispatch reached"));
const respond = await invokeSessionDispatch(
makeDispatchTestContext({
getRuntimeConfig: () => ({
cloudWorkers: { profiles: { test: { provider: "multimode-cloud" } } },
gateway: { nodes: { commands: { allow: ["codex.exec-server.stdio.v1"] } } },
}),
workerPlacementDispatchService: { dispatch },
workerSessionPlacementService: { getMany: () => new Map() },
}),
);
expect(dispatch).toHaveBeenCalledWith(
expect.objectContaining({
executionMode: "remote-exec",
profileId: "test",
devicePlacement: {
requiredNodeCommands: ["codex.exec-server.stdio.v1"],
consumesWorkerSlot: false,
},
}),
expect.any(Function),
undefined,
);
expect(respond).toHaveBeenCalledWith(
false,
undefined,
expect.objectContaining({
code: ErrorCodes.UNAVAILABLE,
message: "cloud-profile dispatch reached",
}),
);
});
});
it.each([
src/gateway/server-methods/sessions.dispatch.test.ts
+42 -4
View File
@@ -295,6 +295,36 @@ describe("sessions.dispatch", () => {
);
});
it("rejects worker-turn before allocation when its profile supports only remote-exec", async () => {
mocks.resolveTarget.mockReturnValue(
targetWithEntry({
sessionId,
worktree: { id: "worktree-1", branch: "openclaw/cloud-test", repoRoot: "/repo" },
}),
);
const dispatch = vi.fn();
const respond = await invoke(
makeContext({
workerEnvironmentService: {
supportsExecutionMode: (_profileId: string, mode: "worker-turn" | "remote-exec") =>
mode === "remote-exec",
} as never,
workerPlacementDispatchService: { dispatch },
workerSessionPlacementService: { getMany: () => new Map() },
}),
);
expect(dispatch).not.toHaveBeenCalled();
expect(respond).toHaveBeenCalledWith(
false,
undefined,
expect.objectContaining({
code: ErrorCodes.INVALID_REQUEST,
message: expect.stringContaining("supports worker-turn"),
}),
);
});
it("treats a whitespace-only profile as an omitted dispatch target", async () => {
mocks.resolveTarget.mockReturnValue(targetWithEntry({ sessionId }));
const dispatch = vi.fn();
@@ -371,7 +401,7 @@ describe("sessions.dispatch", () => {
);
});
it("dispatches codex sessions in remote-exec mode", async () => {
it("dispatches codex sessions through SSH without requiring node command allowlisting", async () => {
mocks.resolveTarget.mockReturnValue(
targetWithEntry({
sessionId,
@@ -388,7 +418,8 @@ describe("sessions.dispatch", () => {
const respond = await invoke(
makeContext({
workerEnvironmentService: {
supportsExecutionMode: () => true,
supportsExecutionMode: (_profileId: string, mode: "worker-turn" | "remote-exec") =>
mode === "remote-exec",
} as never,
workerPlacementDispatchService: { dispatch },
workerSessionPlacementService: { getMany: () => new Map() },
@@ -396,7 +427,13 @@ describe("sessions.dispatch", () => {
);
expect(dispatch).toHaveBeenCalledWith(
expect.objectContaining({ executionMode: "remote-exec" }),
expect.objectContaining({
executionMode: "remote-exec",
devicePlacement: {
requiredNodeCommands: ["codex.exec-server.stdio.v1"],
consumesWorkerSlot: false,
},
}),
expect.any(Function),
undefined,
);
@@ -437,7 +474,7 @@ describe("sessions.dispatch", () => {
expect.objectContaining({
code: ErrorCodes.INVALID_REQUEST,
message:
"selected cloud worker provider does not support the remote-exec execution mode required by runtime codex; use an approved paired device or a provider that advertises remote-exec",
'runtime codex requires a cloud worker provider that supports remote-exec; choose a compatible provider, or select an agent/model route with agentRuntime.id "openclaw"',
}),
);
});
@@ -986,6 +1023,7 @@ describe("sessions.dispatch", () => {
agentId: "main",
executionMode: "worker-turn",
profileId: "test",
devicePlacement: { requiredNodeCommands: [], consumesWorkerSlot: true },
}),
expect.any(Function),
undefined,
src/gateway/server-worker-placement-move-destination.test.ts
+49 -29
View File
@@ -35,6 +35,13 @@ describe("worker placement move destination owner", () => {
expectedError: "worker profile incompatible does not support remote-exec placement",
barrierCalls: 0,
},
{
name: "carries runtime-owned node command requirements into a compatible cloud profile",
target: { kind: "profile" as const, profileId: "compatible" },
supported: true,
expectedError: "source placement barrier started",
barrierCalls: 1,
},
])("$name", async ({ target, supported, expectedError, barrierCalls }) => {
const source = Object.freeze({
sessionId: "session-move-source",
@@ -47,10 +54,15 @@ describe("worker placement move destination owner", () => {
const sourceBefore = JSON.stringify(source);
const entry = { sessionId: source.sessionId, worktree: { id: "worktree-recovery" } };
const config = {
cloudWorkers: { profiles: { incompatible: { provider: "worker-only" } } },
cloudWorkers: {
profiles: {
compatible: { provider: "multimode-cloud" },
incompatible: { provider: "worker-only" },
},
},
gateway: { nodes: { commands: { allow: [CODEX_COMMAND] } } },
};
const supportsExecutionMode = vi.fn(() => false);
const supportsExecutionMode = vi.fn((profileId: string) => profileId === "compatible");
const environments = { supportsExecutionMode };
bindDeviceWorkerAvailability(environments, async (deviceId) => ({
available: true,
@@ -66,33 +78,35 @@ describe("worker placement move destination owner", () => {
commands: [CODEX_COMMAND],
},
}));
const resolveDestination = createGatewayWorkerPlacementMoveDestinationResolver({
environments: environments as never,
getConfig: () => config,
loadSessionRuntime: async () =>
({
managedWorktrees: {
findLiveByOwner: () => ({
id: "worktree-recovery",
ownerId: SESSION_KEY,
path: "/gateway/workspace",
const resolveDestination = vi.fn(
createGatewayWorkerPlacementMoveDestinationResolver({
environments: environments as never,
getConfig: () => config,
loadSessionRuntime: async () =>
({
managedWorktrees: {
findLiveByOwner: () => ({
id: "worktree-recovery",
ownerId: SESSION_KEY,
path: "/gateway/workspace",
}),
},
resolveGatewaySessionStoreTargetWithStore: () => ({
agentId: "main",
canonicalKey: SESSION_KEY,
storePath: "/gateway/session.sqlite",
storeKeys: [SESSION_KEY],
store: { [SESSION_KEY]: entry },
}),
},
resolveGatewaySessionStoreTargetWithStore: () => ({
agentId: "main",
canonicalKey: SESSION_KEY,
storePath: "/gateway/session.sqlite",
storeKeys: [SESSION_KEY],
store: { [SESSION_KEY]: entry },
}),
resolveCanonicalSessionEntryFromStoreKeys: () => entry,
resolveWorkerPlacementSessionRuntime: () => "codex",
resolveWorkerPlacementCapabilities: () => ({
executionMode: "remote-exec",
...(supported ? { devicePlacement: DEVICE_REQUIREMENT } : {}),
}),
}) as never,
});
resolveCanonicalSessionEntryFromStoreKeys: () => entry,
resolveWorkerPlacementSessionRuntime: () => "codex",
resolveWorkerPlacementCapabilities: () => ({
executionMode: "remote-exec",
...(supported ? { devicePlacement: DEVICE_REQUIREMENT } : {}),
}),
}) as never,
}),
);
const runMoveBarrier = vi.fn(async () => {
throw new Error("source placement barrier started");
});
@@ -135,8 +149,14 @@ describe("worker placement move destination owner", () => {
expect(dispatch).not.toHaveBeenCalled();
expect(destroy).not.toHaveBeenCalled();
expect(JSON.stringify(source)).toBe(sourceBefore);
if (supported) {
await expect(resolveDestination.mock.results[0]?.value).resolves.toMatchObject({
executionMode: "remote-exec",
devicePlacement: DEVICE_REQUIREMENT,
});
}
if (target.kind === "profile") {
expect(supportsExecutionMode).toHaveBeenCalledWith("incompatible", "remote-exec");
expect(supportsExecutionMode).toHaveBeenCalledWith(target.profileId, "remote-exec");
}
});
});
src/gateway/server-worker-placement-move-destination.ts
+11 -11
View File
@@ -64,17 +64,17 @@ export function createGatewayWorkerPlacementMoveDestinationResolver(params: {
`worker profile ${moveTarget.profileId} does not support ${executionMode} placement; select a compatible worker provider`,
);
}
return { executionMode, ...destination.value };
}
const eligibility = await resolveDevicePlacementEligibility({
environmentService: params.environments,
deviceId: moveTarget.deviceId,
runtimeId: runtime,
requirement: devicePlacement,
config,
});
if (!eligibility.ok) {
throw new Error(eligibility.error);
} else {
const eligibility = await resolveDevicePlacementEligibility({
environmentService: params.environments,
deviceId: moveTarget.deviceId,
runtimeId: runtime,
requirement: devicePlacement,
config,
});
if (!eligibility.ok) {
throw new Error(eligibility.error);
}
}
return { executionMode, ...destination.value, ...(devicePlacement ? { devicePlacement } : {}) };
};
src/gateway/server-worker-placement-startup.ts
+41 -64
View File
@@ -5,7 +5,6 @@ import { getRuntimeConfig } from "../config/config.js";
import { registerSessionMaintenancePreserveKeysProvider } from "../config/sessions/store-maintenance-preserve.js";
import { runExclusiveSessionStoreWrite } from "../config/sessions/store-writer.js";
import { formatErrorMessage } from "../infra/errors.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import {
interruptSessionWorkAdmissions,
runExclusiveSessionLifecycleMutation,
@@ -14,6 +13,7 @@ import {
import { onSessionIdentityMutation } from "../sessions/session-lifecycle-events.js";
import { createLazyRuntimeModule } from "../shared/lazy-runtime.js";
import { createGitHubPublicationRuntime } from "./github-publication-runtime.js";
import { isNodeCommandAllowed, resolveNodeCommandAllowlist } from "./node-command-policy.js";
import type { NodeWorkerSupervisorTransport } from "./node-registry-private.js";
import { emitSessionsChanged } from "./server-methods/session-change-event.js";
import { createGatewayWorkerPlacementChangePublisher } from "./server-worker-placement-change-events.js";
@@ -27,12 +27,12 @@ import {
runWorkerPlacementSessionBarrier,
WorkerDispatchTargetChangedError,
} from "./server-worker-placement-session-target.js";
import { recoverGatewayWorkerPlacementWorkspaces } from "./server-worker-placement-workspace-recovery.js";
import { createNodeWorkspaceRetainCoordinator } from "./worker-environments/node-workspace-retain-coordinator.js";
import { createWorkerPlacementDiskSpaceMonitor } from "./worker-environments/placement-disk-space.js";
import { coordinateWorkerPlacementDispatch } from "./worker-environments/placement-dispatch-coordinator.js";
import type { WorkerDevicePlacementRequirementResolver } from "./worker-environments/placement-dispatch-startup.js";
import { createWorkerPlacementDispatchService } from "./worker-environments/placement-dispatch.js";
import { FORCED_WORKER_ABANDONMENT_ERROR } from "./worker-environments/placement-force-abandon.js";
import { createWorkerPlacementRunnerAvailabilityReader } from "./worker-environments/placement-projector.js";
import { createPlacementSessionRetirement } from "./worker-environments/placement-session-retirement.js";
import type { WorkerSessionPlacementStore } from "./worker-environments/placement-store.js";
@@ -41,11 +41,9 @@ import type { WorkerEnvironmentService } from "./worker-environments/service.js"
import { isFailedWorkerPlacementEnvironmentGone } from "./worker-environments/session-placement-lifecycle.js";
import { createWorkerSessionTurnPlacementProvider } from "./worker-environments/worker-turn-launcher.js";
import { createWorkerWorkspaceOperationCoordinator } from "./worker-environments/workspace-operation-coordinator.js";
import { recoverWorkerWorkspaceReconciliation } from "./worker-environments/workspace-reconcile.js";
import { createWorkerWorkspaceConflictTranscriptHandlers } from "./worker-workspace-conflict-transcript.js";
const WORKER_PLACEMENT_RECONCILE_INTERVAL_MS = 60_000;
const workerPlacementLog = createSubsystemLogger("gateway/worker-placement");
const loadWorkerPlacementSessionRuntimeModule = createLazyRuntimeModule(async () => {
const [placementSessionRuntime, { managedWorktrees }, sessionUtils] = await Promise.all([
@@ -175,7 +173,7 @@ export function createGatewayWorkerPlacementRuntime(
sessionRuntime,
config: getRuntimeConfig(),
...identity,
errorMessage: `Session ${identity.sessionKey} changed before paired-device recovery`,
errorMessage: `Session ${identity.sessionKey} changed before node-backed placement recovery`,
});
const runtime = sessionRuntime.resolveWorkerPlacementSessionRuntime({
cfg: config,
@@ -187,7 +185,7 @@ export function createGatewayWorkerPlacementRuntime(
sessionRuntime.resolveWorkerPlacementCapabilities(runtime);
if (executionMode !== identity.executionMode || !devicePlacement) {
throw new Error(
`runtime ${runtime} no longer supports this paired-device placement; select a compatible runtime or continue on the Gateway`,
`runtime ${runtime} no longer supports this node-backed placement; select a compatible runtime or continue on the Gateway`,
);
}
return devicePlacement;
@@ -225,6 +223,25 @@ export function createGatewayWorkerPlacementRuntime(
environments: params.environments,
runnerAvailability,
resolveDevicePlacementRequirement,
isCurrentNodePlacement: (node, requirement) => {
if (
nodeWorkerSupervisorTransport?.isCurrent(
node,
requirement.consumesWorkerSlot,
requirement.requiredNodeCommands,
) !== true
) {
return false;
}
const declaredCommands = [...node.commands];
const allowlist = resolveNodeCommandAllowlist(getRuntimeConfig(), {
commands: declaredCommands,
approvedCommands: declaredCommands,
});
return requirement.requiredNodeCommands.every(
(command) => isNodeCommandAllowed({ command, declaredCommands, allowlist }).ok,
);
},
...workspaceConflictHandlers,
...reclaimBarriers,
runLocalBarrier: async ({
@@ -375,9 +392,20 @@ export function createGatewayWorkerPlacementRuntime(
await run(worktree.path);
},
}),
onActivated: (request) => {
if (request.deviceId) {
void nodeWorkspaceRetention.schedule(request.deviceId);
onActivated: ({ sessionId }) => {
const placement = params.placements.get(sessionId);
if (placement?.state !== "active") {
return;
}
const environment = params.environments.get(placement.environmentId);
if (
environment?.state === "attached" &&
environment.ownerEpoch === placement.activeOwnerEpoch &&
environment.attachedSessionIds.length === 1 &&
environment.attachedSessionIds[0] === sessionId &&
environment.nodeDeviceId
) {
void nodeWorkspaceRetention.schedule(environment.nodeDeviceId);
}
},
runMoveBarrier,
@@ -434,60 +462,6 @@ export function createGatewayWorkerPlacementRuntime(
prepareAcceptedWorkspacePublication,
publishAcceptedWorkspace,
});
const recoverPendingWorkspaceReconciliations = async (): Promise<void> => {
const orphanedJournals = params.placements.pruneOrphanedWorkspaceReconciliations({
retainFailedOwner: (recoveryError) =>
recoveryError.startsWith(FORCED_WORKER_ABANDONMENT_ERROR),
});
for (const owner of orphanedJournals) {
workerPlacementLog.warn(`discarded orphaned cloud workspace journal for ${owner.sessionId}`);
}
const pendingBySession = new Map(
params.placements
.listPendingWorkspaceResults()
.map((pending) => [pending.sessionId, pending] as const),
);
for (const owner of params.placements.listWorkspaceReconciliationOwners()) {
try {
const placement = params.placements.get(owner.sessionId);
const pending = pendingBySession.get(owner.sessionId);
const ownsCurrentGeneration = placement?.generation === owner.placementGeneration;
const ownsDrainedPendingGeneration =
placement?.state === "draining" &&
placement.generation === owner.placementGeneration + 1 &&
pending?.environmentId === owner.environmentId &&
pending.ownerEpoch === owner.ownerEpoch &&
pending.placementGeneration === owner.placementGeneration;
if (
(placement?.state !== "active" && placement?.state !== "draining") ||
placement.environmentId !== owner.environmentId ||
placement.activeOwnerEpoch !== owner.ownerEpoch ||
(!ownsCurrentGeneration && !ownsDrainedPendingGeneration)
) {
throw new Error(`Cloud workspace journal has no matching owner: ${owner.sessionId}`);
}
const localPath = await resolveWorkspacePath({
sessionId: placement.sessionId,
sessionKey: placement.sessionKey,
agentId: placement.agentId,
});
const journal = params.placements.loadWorkspaceReconciliation(owner);
if (!journal) {
continue;
}
// Recover before placement/environment reconciliation can reclaim the
// owner; otherwise a crashed partial apply loses its final repair path.
await recoverWorkerWorkspaceReconciliation({ root: localPath, journal });
params.placements.abortWorkspaceReconciliation(owner);
} catch (error) {
// A local edit can intentionally block rollback. Leave that journal
// retryable for this session without withholding every cloud worker.
workerPlacementLog.error(
`cloud workspace recovery deferred for ${owner.sessionId}: ${formatErrorMessage(error)}`,
);
}
}
};
const startRuntime = async (hooks: {
isClosePreludeStarted: () => boolean;
registerSidecar: (sidecar: WorkerPlacementSidecar) => void;
@@ -626,7 +600,10 @@ export function createGatewayWorkerPlacementRuntime(
try {
// Track startup reconciliation in the placement slot so a concurrent
// close prelude drains it before uninstalling guards and stopping environments.
const startupRecovery = recoverPendingWorkspaceReconciliations();
const startupRecovery = recoverGatewayWorkerPlacementWorkspaces({
placements: params.placements,
resolveWorkspacePath,
});
placementReconcile.current = startupRecovery;
try {
await startupRecovery;
src/gateway/server-worker-placement-workspace-recovery.ts
+68
View File
@@ -0,0 +1,68 @@
import { formatErrorMessage } from "../infra/errors.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import { FORCED_WORKER_ABANDONMENT_ERROR } from "./worker-environments/placement-force-abandon.js";
import type { WorkerSessionPlacementStore } from "./worker-environments/placement-store.js";
import { recoverWorkerWorkspaceReconciliation } from "./worker-environments/workspace-reconcile.js";
const workerPlacementLog = createSubsystemLogger("gateway/worker-placement");
export async function recoverGatewayWorkerPlacementWorkspaces(params: {
placements: WorkerSessionPlacementStore;
resolveWorkspacePath: (identity: {
sessionId: string;
sessionKey: string;
agentId: string;
}) => Promise<string>;
}): Promise<void> {
const orphanedJournals = params.placements.pruneOrphanedWorkspaceReconciliations({
retainFailedOwner: (recoveryError) => recoveryError.startsWith(FORCED_WORKER_ABANDONMENT_ERROR),
});
for (const owner of orphanedJournals) {
workerPlacementLog.warn(`discarded orphaned cloud workspace journal for ${owner.sessionId}`);
}
const pendingBySession = new Map(
params.placements
.listPendingWorkspaceResults()
.map((pending) => [pending.sessionId, pending] as const),
);
for (const owner of params.placements.listWorkspaceReconciliationOwners()) {
try {
const placement = params.placements.get(owner.sessionId);
const pending = pendingBySession.get(owner.sessionId);
const ownsCurrentGeneration = placement?.generation === owner.placementGeneration;
const ownsDrainedPendingGeneration =
placement?.state === "draining" &&
placement.generation === owner.placementGeneration + 1 &&
pending?.environmentId === owner.environmentId &&
pending.ownerEpoch === owner.ownerEpoch &&
pending.placementGeneration === owner.placementGeneration;
if (
(placement?.state !== "active" && placement?.state !== "draining") ||
placement.environmentId !== owner.environmentId ||
placement.activeOwnerEpoch !== owner.ownerEpoch ||
(!ownsCurrentGeneration && !ownsDrainedPendingGeneration)
) {
throw new Error(`Cloud workspace journal has no matching owner: ${owner.sessionId}`);
}
const localPath = await params.resolveWorkspacePath({
sessionId: placement.sessionId,
sessionKey: placement.sessionKey,
agentId: placement.agentId,
});
const journal = params.placements.loadWorkspaceReconciliation(owner);
if (!journal) {
continue;
}
// Recover before placement/environment reconciliation can reclaim the
// owner; otherwise a crashed partial apply loses its final repair path.
await recoverWorkerWorkspaceReconciliation({ root: localPath, journal });
params.placements.abortWorkspaceReconciliation(owner);
} catch (error) {
// A local edit can intentionally block rollback. Leave that journal
// retryable for this session without withholding every cloud worker.
workerPlacementLog.error(
`cloud workspace recovery deferred for ${owner.sessionId}: ${formatErrorMessage(error)}`,
);
}
}
}
src/gateway/worker-environments/device-placement-eligibility.ts
+9 -7
View File
@@ -1,10 +1,11 @@
import type { DevicePlacementRequirement } from "../../agents/harness/types.js";
import type { OpenClawConfig } from "../../config/types.openclaw.js";
import { isNodeCommandAllowed, resolveNodeCommandAllowlist } from "../node-command-policy.js";
import type { NodeWorkerSupervisorNodeProof } from "../node-registry-private.js";
import { deviceUnavailableText, resolveDeviceWorkerAvailability } from "./device-provider.js";
type DevicePlacementEligibility =
| { ok: true; availableSlots: number }
| { ok: true; availableSlots: number; node: NodeWorkerSupervisorNodeProof }
| { ok: false; error: string };
export async function resolveDevicePlacementEligibility(params: {
@@ -34,11 +35,12 @@ export async function resolveDevicePlacementEligibility(params: {
}
const node = availability.node;
if (
params.currentNode &&
(params.currentNode.nodeId !== node.nodeId ||
(params.currentNode.connId && params.currentNode.connId !== node.connId) ||
(params.currentNode.pairingGeneration &&
params.currentNode.pairingGeneration !== node.pairingGeneration))
node.nodeId !== deviceId ||
(params.currentNode &&
(params.currentNode.nodeId !== node.nodeId ||
(params.currentNode.connId && params.currentNode.connId !== node.connId) ||
(params.currentNode.pairingGeneration &&
params.currentNode.pairingGeneration !== node.pairingGeneration)))
) {
return {
ok: false,
@@ -72,5 +74,5 @@ export async function resolveDevicePlacementEligibility(params: {
}),
};
}
return { ok: true, availableSlots: node.workerHost.capacity.available };
return { ok: true, availableSlots: node.workerHost.capacity.available, node };
}
src/gateway/worker-environments/device-provider.ts
+1 -1
View File
@@ -134,7 +134,7 @@ export function createDeviceWorkerRuntime(options: DeviceWorkerRuntimeOptions) {
...(unavailableReason ? { unavailableReason } : {}),
};
};
const provider: WorkerProvider<"internal"> = {
const provider: WorkerProvider = {
id: DEVICE_WORKER_PROVIDER_ID,
supportedExecutionModes: ["worker-turn", "remote-exec"],
provisionBeforeInstallation: true,
src/gateway/worker-environments/environment-access.ts
+2 -2
View File
@@ -27,12 +27,12 @@ type WorkerEnvironmentAccessOptions = {
now: () => number;
identityResolverFor: (
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
leaseId: string,
) => Parameters<WorkerTunnelManager["start"]>[0]["resolveIdentity"];
inState: (record: WorkerEnvironmentRecord, ...states: WorkerEnvironmentState[]) => boolean;
isStopping: () => boolean;
providerFor: (providerId: string) => WorkerProvider<"internal">;
providerFor: (providerId: string) => WorkerProvider;
serviceError: (
code:
| "desktop_app_not_found"
src/gateway/worker-environments/identity.ts
+1 -1
View File
@@ -31,7 +31,7 @@ function requireIdentity(value: unknown): WorkerSshIdentity {
/** Routes dynamic identities to their provider owner and configured refs to the generic resolver. */
export async function resolveWorkerSshIdentity(params: {
provider: WorkerProvider<"internal">;
provider: WorkerProvider;
leaseId: string;
profile: WorkerProfile;
keyRef: SecretRef;
src/gateway/worker-environments/placement-dispatch-device.test.ts
+330 -5
View File
@@ -21,13 +21,20 @@ import { REQUEST, type PlacementStore } from "./placement-dispatch-test-fixtures
import { createHarness } from "./placement-dispatch-test-harness.js";
import { createWorkerSessionPlacementStore } from "./placement-store.js";
const runtimeNodeCommandPolicy = vi.hoisted(() => ({
commands: { allow: ["codex.exec-server.stdio.v1"] } as {
allow?: string[];
deny?: string[];
},
}));
vi.mock("../../config/config.js", async (importOriginal) => {
const actual = await importOriginal<typeof import("../../config/config.js")>();
return {
...actual,
getRuntimeConfig: () => ({
...actual.getRuntimeConfig(),
gateway: { nodes: { commands: { allow: ["codex.exec-server.stdio.v1"] } } },
gateway: { nodes: { commands: runtimeNodeCommandPolicy.commands } },
}),
};
});
@@ -57,12 +64,52 @@ function deviceProof(
};
}
function prepareCloudNodeDispatch(
harness: ReturnType<typeof createHarness>,
executionMode: "worker-turn" | "remote-exec" = "remote-exec",
) {
const cloudNodeIdentity = {
providerId: "generic-cloud-node",
profileId: "multi-mode-cloud",
nodeDeviceId: "device-1",
sshEndpoint: null,
sharedHost: false,
};
const ready = { ...harness.ready, ...cloudNodeIdentity };
const attached = { ...harness.attached, ...cloudNodeIdentity };
let isAttached = false;
Object.assign(harness.environments, {
requiresNodeEnrollment: (profileId: string) => profileId === ready.profileId,
});
vi.mocked(harness.environments.create).mockResolvedValue(ready);
vi.mocked(harness.environments.get).mockImplementation((environmentId) =>
environmentId === ready.environmentId ? (isAttached ? attached : ready) : undefined,
);
const attachSession = vi.mocked(harness.environments.attachSession).getMockImplementation();
vi.mocked(harness.environments.attachSession).mockImplementation(async (params) => {
if (!attachSession) {
throw new Error("cloud node dispatch fixture has no session attachment owner");
}
const credential = await attachSession(params);
isAttached = true;
return credential;
});
return {
...REQUEST,
profileId: ready.profileId,
executionMode,
devicePlacement:
executionMode === "remote-exec" ? CODEX_DEVICE_REQUIREMENT : OPENCLAW_DEVICE_REQUIREMENT,
};
}
describe("device worker placement dispatch", () => {
let root: string;
let database: OpenClawStateDatabase;
let placementStore: PlacementStore;
beforeEach(() => {
runtimeNodeCommandPolicy.commands = { allow: [CODEX_COMMAND] };
root = tempDirs.make("openclaw-device-dispatch-");
database = openOpenClawStateDatabase({ env: { OPENCLAW_STATE_DIR: root } });
placementStore = createWorkerSessionPlacementStore({ database, now: () => 1_000 });
@@ -83,7 +130,7 @@ describe("device worker placement dispatch", () => {
providerId: "device",
profileId: "device:device-1",
profileSnapshot: { install: "bundle", settings: { device: "device-1" } },
leaseId: "device-lease-1",
leaseId: "lease-1",
sshEndpoint: null,
bootstrapReceipt: {
bundleHash: "a".repeat(64),
@@ -136,15 +183,31 @@ describe("device worker placement dispatch", () => {
available: true,
node: deviceProof(0),
}));
vi.mocked(harness.environments.createFromProfileSnapshot).mockResolvedValue({
const nodeEnvironment = {
...harness.ready,
providerId: "device",
profileId: "device:device-1",
profileSnapshot: { install: "bundle", settings: { device: "device-1" } },
nodeDeviceId: "device-1",
leaseId: "device-lease-1",
leaseId: "lease-1",
sshEndpoint: null,
sharedHost: true,
} as const;
vi.mocked(harness.environments.createFromProfileSnapshot).mockResolvedValue(nodeEnvironment);
vi.mocked(harness.environments.get).mockImplementation((environmentId) => {
if (environmentId !== nodeEnvironment.environmentId) {
return undefined;
}
return vi.mocked(harness.environments.attachSession).mock.calls.length > 0
? {
...harness.attached,
providerId: "device",
profileId: "device:device-1",
nodeDeviceId: "device-1",
sshEndpoint: null,
sharedHost: true,
}
: nodeEnvironment;
});
const request = {
...REQUEST,
@@ -182,6 +245,247 @@ describe("device worker placement dispatch", () => {
expect(workspaceTunnel?.launchTurn).not.toHaveBeenCalled();
});
it("activates non-device node remote-exec without a worker slot or worker child", async () => {
const harness = createHarness(placementStore);
const resolveAvailability = vi.fn(async () => ({
available: true,
node: deviceProof(0),
}));
bindDeviceWorkerAvailability(harness.environments, resolveAvailability);
const request = prepareCloudNodeDispatch(harness);
await expect(harness.service.dispatch(request)).resolves.toMatchObject({
state: "active",
executionMode: "remote-exec",
remoteWorkspaceDir: "/worker/workspace",
});
expect(resolveAvailability).toHaveBeenCalledWith("device-1");
expect(harness.environments.create).toHaveBeenCalledWith(
"multi-mode-cloud",
expect.stringMatching(/^session-dispatch:/u),
undefined,
"remote-exec",
);
const workspaceTunnel = await vi.mocked(harness.environments.startTunnel).mock.results[0]
?.value;
expect(workspaceTunnel?.syncWorkspace).toHaveBeenCalledOnce();
expect(workspaceTunnel?.launchTurn).not.toHaveBeenCalled();
});
it("does not require node command approval for an SSH-only remote-exec profile", async () => {
const harness = createHarness(placementStore);
runtimeNodeCommandPolicy.commands = { deny: [CODEX_COMMAND] };
await expect(
harness.service.dispatch({
...REQUEST,
executionMode: "remote-exec",
devicePlacement: CODEX_DEVICE_REQUIREMENT,
}),
).resolves.toMatchObject({ state: "active", executionMode: "remote-exec" });
expect(harness.environments.create).toHaveBeenCalledOnce();
});
it("rejects a remote-exec-only enrolled node before provider allocation when its command is denied", async () => {
const harness = createHarness(placementStore);
const request = prepareCloudNodeDispatch(harness);
runtimeNodeCommandPolicy.commands = { deny: [CODEX_COMMAND] };
Object.assign(harness.environments, {
supportsExecutionMode: (_profileId: string, mode: string) => mode === "remote-exec",
requiresNodeEnrollment: () => true,
});
await expect(harness.service.dispatch(request)).rejects.toThrow(CODEX_COMMAND);
expect(harness.environments.create).not.toHaveBeenCalled();
expect(harness.environments.attachSession).not.toHaveBeenCalled();
});
it.each([
{
name: "undeclared required node command",
node: deviceProof(0, ["system.run"]),
deniedByGateway: false,
expectedProvisionCalls: 1,
},
{
name: "required node command denied by Gateway policy",
node: deviceProof(0),
deniedByGateway: true,
expectedProvisionCalls: 0,
},
])("rejects a non-device cloud node with an $name before workspace sync", async (scenario) => {
const harness = createHarness(placementStore);
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: scenario.node,
}));
if (scenario.deniedByGateway) {
runtimeNodeCommandPolicy.commands = { deny: [CODEX_COMMAND] };
}
const request = prepareCloudNodeDispatch(harness);
await expect(harness.service.dispatch(request)).rejects.toThrow("codex.exec-server.stdio.v1");
expect(harness.environments.create).toHaveBeenCalledTimes(scenario.expectedProvisionCalls);
expect(harness.environments.attachSession).not.toHaveBeenCalled();
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it("rejects a non-device cloud node whose current proof names a different node", async () => {
const harness = createHarness(placementStore);
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: { ...deviceProof(), nodeId: "replacement-node" },
}));
const request = prepareCloudNodeDispatch(harness);
await expect(harness.service.dispatch(request)).rejects.toThrow();
expect(harness.environments.create).toHaveBeenCalledOnce();
expect(harness.environments.attachSession).not.toHaveBeenCalled();
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it("fences a non-device cloud node replaced inside the activation barrier", async () => {
const harness = createHarness(placementStore);
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: deviceProof(),
}));
const request = prepareCloudNodeDispatch(harness);
let authorizationChecks = 0;
await expect(
harness.service.dispatch(request, undefined, () => {
authorizationChecks += 1;
if (authorizationChecks === 2) {
vi.mocked(harness.environments.get).mockReturnValue({
...harness.attached,
providerId: "generic-cloud-node",
profileId: "multi-mode-cloud",
nodeDeviceId: "replacement-node",
sshEndpoint: null,
sharedHost: false,
});
}
}),
).rejects.toThrow("exact environment owner");
expect(harness.environments.startTunnel).toHaveBeenCalledOnce();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it("rejects a cloud node re-paired while its managed workspace is synchronizing", async () => {
let currentNode = deviceProof(0);
const harness = createHarness(placementStore, {
isCurrentNodePlacement: (node) =>
node.nodeId === currentNode.nodeId &&
node.connId === currentNode.connId &&
node.pairingGeneration === currentNode.pairingGeneration,
});
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: currentNode,
}));
const request = prepareCloudNodeDispatch(harness);
await expect(
harness.service.dispatch(request, (placement) => {
if (placement.state === "starting") {
currentNode = {
...currentNode,
connId: "replacement-connection",
pairingGeneration: "replacement-generation",
};
}
}),
).rejects.toThrow();
expect(harness.environments.startTunnel).toHaveBeenCalledOnce();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it("fences a cloud node re-paired inside the synchronous activation callback", async () => {
let currentNode = deviceProof(0);
const harness = createHarness(placementStore, {
isCurrentNodePlacement: (node) =>
node.nodeId === currentNode.nodeId &&
node.connId === currentNode.connId &&
node.pairingGeneration === currentNode.pairingGeneration,
});
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: currentNode,
}));
const request = prepareCloudNodeDispatch(harness);
let authorizationChecks = 0;
await expect(
harness.service.dispatch(request, undefined, () => {
authorizationChecks += 1;
if (authorizationChecks === 2) {
currentNode = {
...currentNode,
connId: "replacement-connection",
pairingGeneration: "replacement-generation",
};
}
}),
).rejects.toThrow("pairing generation");
expect(harness.environments.startTunnel).toHaveBeenCalledOnce();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it.each([
{ name: "withdraws its required node command", revocation: "command" as const },
{ name: "denies its required command in Gateway policy", revocation: "policy" as const },
])("fences a cloud node that $name inside the activation callback", async (scenario) => {
let currentNode = deviceProof(0);
const harness = createHarness(placementStore, {
isCurrentNodePlacement: (node, requirement) =>
node.nodeId === currentNode.nodeId &&
node.connId === currentNode.connId &&
node.pairingGeneration === currentNode.pairingGeneration &&
requirement.requiredNodeCommands.every(
(command) =>
currentNode.commands.includes(command) &&
runtimeNodeCommandPolicy.commands.allow?.includes(command) === true &&
runtimeNodeCommandPolicy.commands.deny?.includes(command) !== true,
),
});
bindDeviceWorkerAvailability(harness.environments, async () => ({
available: true,
node: currentNode,
}));
const request = prepareCloudNodeDispatch(harness);
let authorizationChecks = 0;
await expect(
harness.service.dispatch(request, undefined, () => {
authorizationChecks += 1;
if (authorizationChecks === 2) {
if (scenario.revocation === "command") {
currentNode = {
...currentNode,
commands: currentNode.commands.filter((command) => command !== CODEX_COMMAND),
};
} else {
runtimeNodeCommandPolicy.commands = { deny: [CODEX_COMMAND] };
}
}
}),
).rejects.toThrow();
expect(harness.environments.startTunnel).toHaveBeenCalledOnce();
expect(harness.placements.current()).toMatchObject({ state: "failed" });
});
it("records an unavailable device dispatch as a durable failed placement", async () => {
const harness = createHarness(placementStore);
bindDeviceWorkerAvailability(harness.environments, async () => ({
@@ -321,21 +625,42 @@ describe("device worker placement dispatch", () => {
name: "saturated worker-turn node",
executionMode: "worker-turn" as const,
node: deviceProof(0),
providerId: "device",
expectedMessage: "at capacity",
},
{
name: "remote-exec node missing its required command",
executionMode: "remote-exec" as const,
node: deviceProof(0, ["system.run"]),
providerId: "device",
expectedMessage: "not enabled or approved",
},
{
name: "non-device remote-exec cloud node missing its required command",
executionMode: "remote-exec" as const,
node: deviceProof(0, ["system.run"]),
providerId: "generic-cloud-node",
expectedMessage: "not enabled or approved",
},
{
name: "saturated non-device worker-turn cloud node",
executionMode: "worker-turn" as const,
node: deviceProof(0),
providerId: "generic-cloud-node",
expectedMessage: "at capacity",
},
])("fences recovery of a $name before workspace sync", async (scenario) => {
const harness = createHarness(placementStore);
const provisioning = harness.placements.seedProvisioning(scenario.executionMode);
if (provisioning.state !== "provisioning") {
throw new Error("paired-device recovery fixture did not enter provisioning");
}
const environment = { ...harness.ready, providerId: "device", nodeDeviceId: "device-1" };
const environment = {
...harness.ready,
providerId: scenario.providerId,
nodeDeviceId: "device-1",
sshEndpoint: null,
};
vi.mocked(harness.environments.get).mockImplementation((environmentId) =>
environmentId === environment.environmentId ? environment : undefined,
);
src/gateway/worker-environments/placement-dispatch-failure.ts
+1
View File
@@ -81,6 +81,7 @@ export type WorkerDispatchEnvironmentService = Pick<
| "reconcileOnce"
| "startTunnel"
| "stopTunnel"
| "supportsProviderExecutionMode"
>;
export type WorkerActivationBarrier = (params: {
src/gateway/worker-environments/placement-dispatch-recovery.test.ts
+156
View File
@@ -23,6 +23,135 @@ import { createWorkerWorkspaceOperationCoordinator } from "./workspace-operation
describe("worker placement restart recovery", () => {
support.setupWorkerEnvironmentServiceSuite();
it.each([
{
failure: "provider no longer supports the persisted execution mode",
executionMode: "remote-exec",
snapshotMode: "remote-exec",
nodeBacked: true,
providerSupportsMode: false,
expectedError: "does not support remote-exec",
sweep: false,
},
{
failure: "persisted provisioning mode does not match placement authority",
executionMode: "remote-exec",
snapshotMode: "worker-turn",
nodeBacked: true,
providerSupportsMode: true,
expectedError: "execution mode does not match",
sweep: false,
},
{
failure: "worker-turn placement has an SSH lease",
executionMode: "worker-turn",
snapshotMode: "worker-turn",
nodeBacked: false,
providerSupportsMode: true,
expectedError: "requires a node lease",
sweep: false,
},
{
failure: "runtime sweep observes a changed provider execution mode",
executionMode: "remote-exec",
snapshotMode: "remote-exec",
nodeBacked: true,
providerSupportsMode: false,
expectedError: "does not support remote-exec",
sweep: true,
},
] as const)("fences active restart recovery when $failure", async (scenario) => {
const placements = createWorkerSessionPlacementStore({
database: support.testState.stateDb,
now: () => 1_000,
});
const harness = createHarness(placements);
await harness.environments.attachSession({
environmentId: harness.ready.environmentId,
ownerEpoch: harness.ready.ownerEpoch,
sessionId: REQUEST.sessionId,
});
const environment = {
...harness.attached,
providerId: "durable-provider",
profileSnapshot: {
...harness.attached.profileSnapshot,
executionMode: scenario.snapshotMode,
},
...(scenario.nodeBacked ? { nodeDeviceId: "durable-node", sshEndpoint: null } : {}),
};
vi.mocked(harness.environments.get).mockReturnValue(environment);
Object.assign(harness.environments, {
supportsProviderExecutionMode: vi.fn(() => scenario.providerSupportsMode),
});
harness.placements.seedActive(environment.ownerEpoch, scenario.executionMode);
if (scenario.sweep) {
await harness.service.reconcileActive(environment.environmentId);
} else {
await harness.service.reconcile();
}
expect(harness.placements.current()).toMatchObject({
state: "failed",
recoveryError: expect.stringContaining(scenario.expectedError),
});
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
expect(harness.environments.destroy).toHaveBeenCalledWith(environment.environmentId);
expect(harness.log).not.toContain("placement:adopted");
});
it.each([
{ transport: "node", nodeBacked: true },
{ transport: "SSH", nodeBacked: false },
] as const)(
"adopts an exact remote-exec $transport lease from its durable provider after profile changes",
async ({ nodeBacked }) => {
const placements = createWorkerSessionPlacementStore({
database: support.testState.stateDb,
now: () => 1_000,
});
const harness = createHarness(placements);
await harness.environments.attachSession({
environmentId: harness.ready.environmentId,
ownerEpoch: harness.ready.ownerEpoch,
sessionId: REQUEST.sessionId,
});
const environment = {
...harness.attached,
providerId: "durable-provider",
profileSnapshot: {
...harness.attached.profileSnapshot,
executionMode: "remote-exec" as const,
},
...(nodeBacked ? { nodeDeviceId: "durable-node", sshEndpoint: null } : {}),
};
vi.mocked(harness.environments.get).mockReturnValue(environment);
const supportsProviderExecutionMode = vi.fn(() => true);
const supportsExecutionMode = vi.fn(() => false);
Object.assign(harness.environments, {
supportsExecutionMode,
supportsProviderExecutionMode,
});
harness.placements.seedActive(environment.ownerEpoch, "remote-exec");
await harness.service.reconcile();
expect(harness.placements.current()).toMatchObject({
state: "active",
executionMode: "remote-exec",
});
expect(supportsProviderExecutionMode).toHaveBeenCalledWith("durable-provider", "remote-exec");
expect(supportsExecutionMode).not.toHaveBeenCalled();
expect(harness.environments.destroy).not.toHaveBeenCalled();
if (nodeBacked) {
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
} else {
expect(harness.environments.startTunnel).toHaveBeenCalledOnce();
}
},
);
it.each([
{ creation: "profile", matchingEnvironmentExists: true },
{ creation: "inherited", matchingEnvironmentExists: true },
@@ -126,6 +255,33 @@ describe("worker placement restart recovery", () => {
expect(harness.log).not.toContain("activation");
});
it("fences provisioning recovery before attachment when its durable execution mode differs", async () => {
const placements = createWorkerSessionPlacementStore({
database: support.testState.stateDb,
now: () => 1_000,
});
const harness = createHarness(placements);
const provisioning = harness.placements.seedProvisioning("remote-exec");
if (provisioning.state !== "provisioning") {
throw new Error("recovery fixture did not produce a provisioning placement");
}
const environment = {
...harness.ready,
profileSnapshot: { ...harness.ready.profileSnapshot, executionMode: "worker-turn" as const },
};
vi.mocked(harness.environments.get).mockReturnValue(environment);
await harness.service.resumeProvisioning(provisioning, async () => {});
expect(harness.placements.current()).toMatchObject({
state: "failed",
recoveryError: expect.stringContaining("execution mode"),
});
expect(harness.environments.attachSession).not.toHaveBeenCalled();
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
expect(harness.environments.destroy).toHaveBeenCalledWith(environment.environmentId);
});
it("fails a placement interrupted before its environment intent and permits redispatch", async () => {
const placements = createWorkerSessionPlacementStore({
database: support.testState.stateDb,
src/gateway/worker-environments/placement-dispatch-recovery.ts
+37 -6
View File
@@ -3,6 +3,7 @@ import {
isCurrentActiveWorkerEnvironment,
isUnavailableEnvironment,
type WorkerActiveDispatchPlacement,
type WorkerDispatchEnvironmentService,
type WorkerDispatchPlacement,
type WorkerFailedDispatchPlacement,
} from "./placement-dispatch-failure.js";
@@ -36,6 +37,28 @@ function workerDisappearanceError(
);
}
function activePlacementExecutionError(
placement: WorkerActiveDispatchPlacement,
environment: NonNullable<ReturnType<WorkerDispatchEnvironmentService["get"]>>,
environments: Pick<WorkerDispatchEnvironmentService, "supportsProviderExecutionMode">,
): Error | undefined {
const provisionedMode = environment.profileSnapshot.executionMode;
if (provisionedMode !== undefined && provisionedMode !== placement.executionMode) {
return new Error("Active worker placement execution mode does not match its environment");
}
if (placement.executionMode === "worker-turn" && !environment.nodeDeviceId) {
return new Error("Active worker-turn placement requires a node lease");
}
if (
!environments.supportsProviderExecutionMode(environment.providerId, placement.executionMode)
) {
return new Error(
`Worker provider ${environment.providerId} does not support ${placement.executionMode} placement`,
);
}
return undefined;
}
function blockingWorkspaceJournalSessions(
placements: PlacementRecoveryDeps["placements"],
): Set<string> {
@@ -71,8 +94,8 @@ export function createPlacementRecoveryActions(deps: PlacementRecoveryDeps) {
const { environments, failure, placements } = deps;
const adoptActive = async (placement: WorkerActiveDispatchPlacement): Promise<void> => {
// Worker turns are one-shot SSH children owned by the previous gateway process. A durable
// claim cannot prove that child remains live after restart, so fence the whole placement.
// Turn claims belong to the previous Gateway lifecycle and cannot prove live authority
// after restart, so fence the whole placement before attempting to adopt it.
if (placement.turnClaim) {
const error = new Error(
"Active worker turn claim cannot be proven live after gateway restart",
@@ -101,9 +124,12 @@ export function createPlacementRecoveryActions(deps: PlacementRecoveryDeps) {
return;
}
try {
// Paired nodes are persistent runners, not one-shot SSH children. Their
// dormant lease remains authoritative while offline; validate and create
// the reconnect-scoped tunnel lazily when the next turn actually launches.
const executionError = activePlacementExecutionError(placement, environment, environments);
if (executionError) {
throw executionError;
}
// Node leases stay authoritative while offline; their reconnect-scoped
// tunnel is validated lazily when the next turn actually launches.
if (!environment.nodeDeviceId) {
await environments.startTunnel({
environmentId: environment.environmentId,
@@ -236,12 +262,17 @@ export function createPlacementRecoveryActions(deps: PlacementRecoveryDeps) {
);
continue;
}
if (!isCurrentActiveWorkerEnvironment(placement, environment)) {
if (!environment || !isCurrentActiveWorkerEnvironment(placement, environment)) {
await failure.reclaimActive(
placement,
environment,
new Error("Active worker placement does not match its environment owner"),
);
continue;
}
const executionError = activePlacementExecutionError(placement, environment, environments);
if (executionError) {
await failure.failActive(placement, executionError, { forceClaimFence: true });
}
}
};
src/gateway/worker-environments/placement-dispatch-startup.ts
+94 -23
View File
@@ -1,5 +1,6 @@
import type { DevicePlacementRequirement } from "../../agents/harness/types.js";
import { getRuntimeConfig } from "../../config/config.js";
import type { NodeWorkerSupervisorNodeProof } from "../node-registry-private.js";
import { supportsWorkerExecutionContextLaunch } from "./admission.js";
import { resolveDevicePlacementEligibility } from "./device-placement-eligibility.js";
import { DEVICE_WORKER_PROVIDER_ID } from "./device-provider-identity.js";
@@ -35,6 +36,11 @@ export type WorkerDevicePlacementRequirementResolver = (
>,
) => Promise<DevicePlacementRequirement>;
export type WorkerNodePlacementAuthority = (
node: NodeWorkerSupervisorNodeProof,
requirement: DevicePlacementRequirement,
) => boolean;
function isPendingProvisioningEnvironment(
environment: ReturnType<WorkerEnvironmentService["get"]>,
environmentId: string | null,
@@ -51,6 +57,8 @@ function isPendingProvisioningEnvironment(
function requireProvisionedEnvironment(
environment: Awaited<ReturnType<WorkerEnvironmentService["create"]>>,
expectedEnvironmentId: string,
executionMode: WorkerPlacementDispatchRequest["executionMode"],
environments: Pick<WorkerDispatchEnvironmentService, "supportsProviderExecutionMode">,
): { environmentId: string; ownerEpoch: number; bundleHash: string } {
if (
(environment.state !== "ready" && environment.state !== "idle") ||
@@ -63,6 +71,16 @@ function requireProvisionedEnvironment(
`Worker environment is not dispatchable with the current execution-context contract: ${environment.state}`,
);
}
if (
(environment.profileSnapshot.executionMode !== undefined &&
environment.profileSnapshot.executionMode !== executionMode) ||
(executionMode === "worker-turn" &&
environment.profileSnapshot.executionMode !== undefined &&
!environment.nodeDeviceId) ||
!environments.supportsProviderExecutionMode(environment.providerId, executionMode)
) {
throw new Error("Worker environment does not support the placement's exact execution mode");
}
return {
environmentId: environment.environmentId,
ownerEpoch: environment.ownerEpoch,
@@ -79,6 +97,7 @@ export function createWorkerPlacementDispatchStartup(options: {
onActivated?: (request: WorkerPlacementDispatchRequest) => void;
resolveGitAuthor?: (agentId: string) => { name?: string; email?: string } | undefined;
resolveDevicePlacementRequirement?: WorkerDevicePlacementRequirementResolver;
isCurrentNodePlacement?: WorkerNodePlacementAuthority;
reportTransition: (
observer: ((placement: WorkerDispatchPlacement) => void) | undefined,
placement: WorkerDispatchPlacement,
@@ -86,6 +105,43 @@ export function createWorkerPlacementDispatchStartup(options: {
}) {
const { environments, failure, placements } = options;
const requireNodePlacementEligibility = async (
request: WorkerPlacementDispatchRequest,
environment: Awaited<ReturnType<WorkerEnvironmentService["create"]>>,
admittedNode?: NodeWorkerSupervisorNodeProof,
): Promise<
{ node: NodeWorkerSupervisorNodeProof; requirement: DevicePlacementRequirement } | undefined
> => {
const deviceId = environment.nodeDeviceId;
if (!deviceId) {
return undefined;
}
const requirement =
request.devicePlacement ??
(options.resolveDevicePlacementRequirement
? await options.resolveDevicePlacementRequirement({
sessionId: request.sessionId,
sessionKey: request.sessionKey,
agentId: request.agentId,
executionMode: request.executionMode,
})
: undefined);
if (!requirement) {
throw new Error("Node-backed cloud placement has no authoritative runtime requirement");
}
const eligibility = await resolveDevicePlacementEligibility({
environmentService: environments,
deviceId,
requirement,
config: getRuntimeConfig(),
...(admittedNode ? { currentNode: admittedNode } : {}),
});
if (!eligibility.ok) {
throw new Error(eligibility.error);
}
return { node: eligibility.node, requirement };
};
const continueProvisionedDispatch = async (params: {
request: WorkerPlacementDispatchRequest;
placement: WorkerDispatchPlacement;
@@ -103,7 +159,10 @@ export function createWorkerPlacementDispatchStartup(options: {
const provisioned = requireProvisionedEnvironment(
params.environment,
params.expectedEnvironmentId,
request.executionMode,
environments,
);
const admittedNode = await requireNodePlacementEligibility(request, params.environment);
let placement = placements.transition({
sessionId: request.sessionId,
from: "provisioning",
@@ -144,18 +203,38 @@ export function createWorkerPlacementDispatchStartup(options: {
});
options.reportTransition(params.onTransition, placement);
const startingPlacement = placement;
const attachedEnvironment = environments.get(provisioned.environmentId);
if (
!attachedEnvironment ||
attachedEnvironment.state !== "attached" ||
attachedEnvironment.ownerEpoch !== ownerEpoch ||
attachedEnvironment.attachedSessionIds.length !== 1 ||
attachedEnvironment.attachedSessionIds[0] !== request.sessionId ||
attachedEnvironment.bootstrapReceipt?.bundleHash !== provisioned.bundleHash
) {
throw new Error("Worker dispatch lost its exact environment owner before activation");
}
const requireAttachedEnvironment = () => {
const attachedEnvironment = environments.get(provisioned.environmentId);
if (
!attachedEnvironment ||
attachedEnvironment.state !== "attached" ||
attachedEnvironment.ownerEpoch !== ownerEpoch ||
attachedEnvironment.attachedSessionIds.length !== 1 ||
attachedEnvironment.attachedSessionIds[0] !== request.sessionId ||
attachedEnvironment.nodeDeviceId !== params.environment.nodeDeviceId ||
attachedEnvironment.leaseId !== params.environment.leaseId ||
attachedEnvironment.bootstrapReceipt?.bundleHash !== provisioned.bundleHash
) {
throw new Error("Worker dispatch lost its exact environment owner before activation");
}
return attachedEnvironment;
};
await requireNodePlacementEligibility(
request,
requireAttachedEnvironment(),
admittedNode?.node,
);
requireAttachedEnvironment();
const activate = (): WorkerActiveDispatchPlacement => {
requireAttachedEnvironment();
if (
admittedNode &&
!options.isCurrentNodePlacement?.(admittedNode.node, admittedNode.requirement)
) {
throw new Error(
"Worker dispatch lost its current node connection, pairing generation, command authorization, or capacity before activation",
);
}
const activated = placements.transition({
sessionId: request.sessionId,
from: "starting",
@@ -266,9 +345,9 @@ export function createWorkerPlacementDispatchStartup(options: {
return;
}
let devicePlacement: DevicePlacementRequirement | undefined;
if (environment.providerId === DEVICE_WORKER_PROVIDER_ID && environment.nodeDeviceId) {
if (environment.nodeDeviceId) {
if (!options.resolveDevicePlacementRequirement) {
throw new Error("Paired-device recovery has no authoritative runtime requirement");
throw new Error("Node-backed recovery has no authoritative runtime requirement");
}
devicePlacement = await options.resolveDevicePlacementRequirement({
sessionId: placement.sessionId,
@@ -276,15 +355,6 @@ export function createWorkerPlacementDispatchStartup(options: {
agentId: placement.agentId,
executionMode: placement.executionMode,
});
const eligibility = await resolveDevicePlacementEligibility({
environmentService: environments,
deviceId: environment.nodeDeviceId,
requirement: devicePlacement,
config: getRuntimeConfig(),
});
if (!eligibility.ok) {
throw new Error(eligibility.error);
}
}
await continueProvisionedDispatch({
request: {
@@ -293,8 +363,9 @@ export function createWorkerPlacementDispatchStartup(options: {
agentId: placement.agentId,
profileId: environment.profileId,
executionMode: placement.executionMode,
...(devicePlacement ? { devicePlacement } : {}),
...(environment.providerId === DEVICE_WORKER_PROVIDER_ID && environment.nodeDeviceId
? { deviceId: environment.nodeDeviceId, devicePlacement }
? { deviceId: environment.nodeDeviceId }
: {}),
},
placement: current,
src/gateway/worker-environments/placement-dispatch-test-harness.ts
+6 -1
View File
@@ -66,6 +66,9 @@ export function createHarness(
afterReconcile?: () => Promise<void> | void;
afterStopTunnel?: () => Promise<void> | void;
deviceRunnerAvailable?: boolean;
isCurrentNodePlacement?: Parameters<
typeof createWorkerPlacementDispatchService
>[0]["isCurrentNodePlacement"];
} = {},
) {
const reconciledManifestRef = MANIFEST_REF.replaceAll("b", "c");
@@ -329,6 +332,7 @@ export function createHarness(
expiresAtMs: 10_000,
};
const environments: WorkerDispatchEnvironmentService = {
supportsProviderExecutionMode: vi.fn(() => true),
create: vi.fn(async () => {
fail("create");
return currentEnvironment ?? ready;
@@ -447,6 +451,7 @@ export function createHarness(
consumesWorkerSlot: false,
}
: { requiredNodeCommands: [], consumesWorkerSlot: true },
isCurrentNodePlacement: options.isCurrentNodePlacement ?? (() => true),
runReclaimBarrier: async ({ authorize, begin, reclaim }) => {
authorize?.();
return await reclaim(options.workspacePath ?? "/gateway/workspace", begin(), authorize);
@@ -513,7 +518,7 @@ export function createHarness(
currentEnvironment = { ...attached, ownerEpoch };
},
markEnvironmentNodeDeviceId: (nodeDeviceId: string) => {
currentEnvironment = { ...attached, providerId: "device", nodeDeviceId };
currentEnvironment = { ...attached, providerId: "device", nodeDeviceId, sshEndpoint: null };
},
markEnvironmentAttachments: (attachedSessionIds: string[]) => {
currentEnvironment = { ...attached, attachedSessionIds };
src/gateway/worker-environments/placement-dispatch.test.ts
+3 -1
View File
@@ -104,6 +104,7 @@ describe("worker placement dispatch", () => {
});
const active = harness.placements.seedActive(2);
harness.markEnvironmentOwnerEpoch(2);
harness.markEnvironmentNodeDeviceId("completed-worker-node");
if (active.state !== "active") {
throw new Error("active placement fixture was not active");
}
@@ -703,7 +704,7 @@ describe("worker placement dispatch", () => {
ownerEpoch: harness.ready.ownerEpoch,
sessionId: REQUEST.sessionId,
});
harness.placements.seedActive(harness.attached.ownerEpoch);
harness.placements.seedActive(harness.attached.ownerEpoch, "remote-exec");
harness.log.length = 0;
await harness.service.reconcile();
@@ -943,6 +944,7 @@ describe("worker placement dispatch", () => {
sessionId: REQUEST.sessionId,
});
harness.placements.seedActive(harness.attached.ownerEpoch);
harness.markEnvironmentNodeDeviceId("live-worker-node");
placementStore.claimTurn({
...REQUEST,
claimId: "claim-1",
src/gateway/worker-environments/placement-dispatch.ts
+25 -1
View File
@@ -1,5 +1,6 @@
import { randomUUID } from "node:crypto";
import { getRuntimeConfig } from "../../config/config.js";
import { resolveNodeCommandAllowlist } from "../node-command-policy.js";
import { resolveDevicePlacementEligibility } from "./device-placement-eligibility.js";
import {
createPlacementFailureActions,
@@ -13,6 +14,7 @@ import { createPlacementRecoveryActions } from "./placement-dispatch-recovery.js
import {
createWorkerPlacementDispatchStartup,
type WorkerDevicePlacementRequirementResolver,
type WorkerNodePlacementAuthority,
type WorkerPlacementRecoveryBarrier,
} from "./placement-dispatch-startup.js";
import { createWorkerPlacementMoveAbandonment } from "./placement-move-abandon.js";
@@ -35,6 +37,7 @@ import type {
WorkerPlacementReclaimRequest,
} from "./service-contract.js";
import { deriveEnvironmentIntent } from "./service-contract.js";
import type { WorkerEnvironmentService } from "./service.js";
import { isFailedWorkerPlacementEnvironmentGone } from "./session-placement-lifecycle.js";
import { WorkerTunnelOwnerDisconnectedError } from "./tunnel-contract.js";
import type {
@@ -94,7 +97,8 @@ export type WorkerPlacementReclaimBarriers = {
type WorkerPlacementDispatchOptions = WorkerPlacementReclaimBarriers & {
placements: WorkerDispatchPlacementStore;
environments: WorkerDispatchEnvironmentService;
environments: WorkerDispatchEnvironmentService &
Partial<Pick<WorkerEnvironmentService, "requiresNodeEnrollment">>;
runnerAvailability: WorkerPlacementRunnerAvailabilityReader;
runLocalBarrier: WorkerLocalDispatchBarrier;
runRecoveryBarrier: WorkerPlacementRecoveryBarrier;
@@ -133,6 +137,7 @@ type WorkerPlacementDispatchOptions = WorkerPlacementReclaimBarriers & {
) => Promise<void>;
resolveGitAuthor?: (agentId: string) => { name?: string; email?: string } | undefined;
resolveDevicePlacementRequirement?: WorkerDevicePlacementRequirementResolver;
isCurrentNodePlacement?: WorkerNodePlacementAuthority;
};
function isExactAttachedEnvironment(
@@ -174,6 +179,7 @@ export function createWorkerPlacementDispatchService(options: WorkerPlacementDis
onActivated: options.onActivated,
resolveGitAuthor: options.resolveGitAuthor,
resolveDevicePlacementRequirement: options.resolveDevicePlacementRequirement,
isCurrentNodePlacement: options.isCurrentNodePlacement,
reportTransition,
});
@@ -235,6 +241,24 @@ export function createWorkerPlacementDispatchService(options: WorkerPlacementDis
return placement;
},
});
if (
!request.deviceId &&
request.devicePlacement?.requiredNodeCommands.length &&
environments.requiresNodeEnrollment?.(
request.profileId,
request.inheritedProfile?.providerId,
)
) {
const allowlist = resolveNodeCommandAllowlist(getRuntimeConfig());
const deniedCommand = request.devicePlacement.requiredNodeCommands.find(
(command) => !allowlist.has(command),
);
if (deniedCommand) {
throw new Error(
`cloud worker node command ${deniedCommand} is not enabled; add it to gateway.nodes.commands.allow and approve the command on the node`,
);
}
}
await validateDevicePlacement();
const localPath = await options.resolveWorkspacePath(request);
// Workspace preparation yields; fence the current paired node again before durable provision.
src/gateway/worker-environments/provider-lifecycle.ts
+46 -23
View File
@@ -6,6 +6,7 @@ import { validateCloudWorkerProfileSettings } from "../../config/zod-schema.clou
import { normalizeCapabilityProviderId } from "../../plugins/provider-registry-shared.js";
import {
WorkerProviderError,
type WorkerExecutionMode,
type WorkerLease,
type WorkerProfile,
type WorkerProvider,
@@ -24,7 +25,7 @@ import {
requireProviderProvisionTimeoutMs,
requireWorkerLease,
requireWorkerLeaseStatus,
resolveWorkerLeaseModeError,
resolveWorkerLeaseTransportError,
} from "./service-validation.js";
import type {
WorkerEnvironmentRecord,
@@ -61,7 +62,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
const identityResolverFor = (
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
leaseId: string,
) => {
const profile = requireWorkerProfile(record.profileSnapshot.settings);
@@ -76,7 +77,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
};
};
const providerFor = (providerId: string): WorkerProvider<"internal"> => {
const providerFor = (providerId: string): WorkerProvider => {
const provider = options.resolveProvider(providerId);
if (provider) {
return provider;
@@ -126,7 +127,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
const failBootstrap = async (
record: WorkerEnvironmentRecord,
leaseId: string,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
error: unknown,
failureCode: "bootstrap_failure" | "invalid_profile" = "bootstrap_failure",
leasePatch?: TransitionPatch,
@@ -199,7 +200,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
const finishBootstrap = async (
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
installation: WorkerInstallationArtifact,
) => {
if (record.state !== "bootstrapping" || !record.leaseId || !record.sshEndpoint) {
@@ -229,12 +230,27 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
const finishProvision = async (
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
preparedInstallation?: WorkerInstallationArtifact,
) => {
let lease: WorkerLease;
let executionMode: WorkerExecutionMode | undefined;
try {
const profile = requireWorkerProfile(record.profileSnapshot.settings);
const requestedExecutionMode = record.profileSnapshot.executionMode;
if (
requestedExecutionMode !== undefined &&
requestedExecutionMode !== "worker-turn" &&
requestedExecutionMode !== "remote-exec"
) {
throw new WorkerProviderError("Worker environment has an invalid placement execution mode");
}
executionMode = requestedExecutionMode;
if (executionMode && !provider.supportedExecutionModes?.includes(executionMode)) {
throw new WorkerProviderError(
`Worker provider ${provider.id} does not support ${executionMode} placement`,
);
}
const providerTimeoutMs =
options.providerCallTimeoutMs === undefined
? requireProviderProvisionTimeoutMs(provider.resolveProvisionTimeoutMs?.(profile))
@@ -248,9 +264,10 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
throw new Error("Worker node enrollment runtime is unavailable");
}
const provisionOptions =
machineClass || provider.requiresNodeEnrollment === true
machineClass || executionMode || provider.requiresNodeEnrollment === true
? {
...(machineClass ? { machineClass } : {}),
...(executionMode ? { executionMode } : {}),
...(provider.requiresNodeEnrollment === true && prepareNodeEnrollment
? { beginNodeEnrollment: async () => await prepareNodeEnrollment(record) }
: {}),
@@ -287,7 +304,11 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
? { nodeDeviceId: lease.node.deviceId, sshEndpoint: null }
: { nodeDeviceId: null, sshEndpoint: lease.ssh }),
};
const leaseModeError = resolveWorkerLeaseModeError(provider, lease);
const leaseModeError = resolveWorkerLeaseTransportError(
provider,
lease.node ? "node" : "ssh",
executionMode,
);
if (leaseModeError) {
return await failBootstrap(
record,
@@ -370,10 +391,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
throw serviceError("invalid_state", `Cannot destroy worker in state: ${record.state}`);
};
const finishDestroy = async (
r: WorkerEnvironmentRecord,
provider?: WorkerProvider<"internal">,
) => {
const finishDestroy = async (r: WorkerEnvironmentRecord, provider?: WorkerProvider) => {
if (!r.leaseId) {
throw serviceError("invalid_state", "Worker environment has no lease");
}
@@ -413,7 +431,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
// Provider inspection and the state-specific path below retain their existing retry policy.
}
}
let provider: WorkerProvider<"internal">;
let provider: WorkerProvider;
try {
provider = providerFor(record.providerId);
} catch (error) {
@@ -583,11 +601,15 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
profileSnapshot: WorkerProfile;
};
machineClass?: string;
executionMode?: WorkerExecutionMode;
} = {},
) => {
const { inherited, machineClass } = createOptions;
let stopping = options.isStopping();
if (stopping) {
const { inherited, machineClass, executionMode } = createOptions;
const provisionSnapshot = {
...(machineClass === undefined ? {} : { machineClass }),
...(executionMode === undefined ? {} : { executionMode }),
};
if (options.isStopping()) {
throw serviceError("invalid_state", "Worker environment service is stopping");
}
const normalizedProfileId = profileId.trim();
@@ -596,8 +618,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
}
const { environmentId, provisionOperationId } = deriveEnvironmentIntent(idempotencyKey);
return withLock(environmentId, async () => {
stopping = options.isStopping();
if (stopping) {
if (options.isStopping()) {
throw serviceError("invalid_state", "Worker environment service is stopping");
}
const existing = store.get(environmentId);
@@ -608,9 +629,11 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
(existing.providerId !== inherited.providerId ||
!isDeepStrictEqual(existing.profileSnapshot, {
...inherited.profileSnapshot,
...(machineClass === undefined ? {} : { machineClass }),
...provisionSnapshot,
}))) ||
(inherited === undefined && existing.profileSnapshot.machineClass !== machineClass)
(inherited === undefined &&
(existing.profileSnapshot.machineClass !== machineClass ||
existing.profileSnapshot.executionMode !== executionMode))
) {
throw serviceError("invalid_profile", "Idempotency key belongs to another profile");
}
@@ -622,7 +645,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
}
return existing;
}
let provider: WorkerProvider<"internal">;
let provider: WorkerProvider;
let providerId: string;
let profileSnapshot: WorkerProfile;
if (inherited) {
@@ -637,7 +660,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
}
profileSnapshot = requireWorkerProfile({
...inherited.profileSnapshot,
...(machineClass === undefined ? {} : { machineClass }),
...provisionSnapshot,
});
} else {
const profiles = options.getConfig().cloudWorkers?.profiles;
@@ -654,7 +677,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
profileSnapshot = requireWorkerProfile({
install: profile.install ?? "bundle",
settings,
...(machineClass === undefined ? {} : { machineClass }),
...provisionSnapshot,
});
}
const intent = store.createIntent({
src/gateway/worker-environments/provider-lifecycle.types.ts
+2 -2
View File
@@ -21,7 +21,7 @@ import type { WorkerTunnelManager } from "./tunnel.js";
export type WorkerProviderLifecycleInputOptions = {
store: WorkerEnvironmentStore;
getConfig: () => OpenClawConfig;
resolveProvider: (providerId: string) => WorkerProvider<"internal"> | undefined;
resolveProvider: (providerId: string) => WorkerProvider | undefined;
prepareInstallation: (
install: WorkerInstallationArtifact["install"],
) => Promise<WorkerInstallationArtifact>;
@@ -33,7 +33,7 @@ export type WorkerProviderLifecycleInputOptions = {
signal: AbortSignal;
}) => Promise<WorkerAdmissionHandshake>;
resolveSshIdentity?: (params: {
provider: WorkerProvider<"internal">;
provider: WorkerProvider;
leaseId: string;
profile: WorkerProfile;
keyRef: SecretRef;
src/gateway/worker-environments/provider-node-provisioning.ts
+3 -3
View File
@@ -20,7 +20,7 @@ type WorkerNodeProvisioningOptions = {
destroyProviderLease: (
record: WorkerEnvironmentRecord,
leaseId: string,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
) => Promise<void>;
finishProvenDestroy: (record: WorkerEnvironmentRecord) => Promise<WorkerEnvironmentRecord>;
saveError: (record: WorkerEnvironmentRecord, error: unknown) => WorkerEnvironmentRecord;
@@ -31,7 +31,7 @@ export function createWorkerNodeProvisioning(options: WorkerNodeProvisioningOpti
const fail = async (
record: WorkerEnvironmentRecord,
lease: NodeLease,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
error: unknown,
): Promise<never> => {
const detail = boundedError(error);
@@ -70,7 +70,7 @@ export function createWorkerNodeProvisioning(options: WorkerNodeProvisioningOpti
return async (
record: WorkerEnvironmentRecord,
lease: NodeLease,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
patch: { leaseId: string; sharedHost: boolean; desktop: WorkerLease["desktop"] | null },
): Promise<WorkerEnvironmentRecord> => {
let nodeBuild: WorkerAdmissionHandshake;
src/gateway/worker-environments/provider-persisted-lease.ts
+6 -10
View File
@@ -1,6 +1,6 @@
import type { WorkerProvider } from "../../plugins/types.js";
import { STALE_WORKER_BUILD_REASON } from "./admission.js";
import { resolveWorkerTransportModeError } from "./service-validation.js";
import { resolveWorkerLeaseTransportError } from "./service-validation.js";
import type { WorkerEnvironmentRecord, WorkerEnvironmentStore } from "./store.js";
export function requestStaleWorkerDestroy(
@@ -20,20 +20,16 @@ export function requestStaleWorkerDestroy(
export async function retireMismatchedWorkerLease(
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
store: WorkerEnvironmentStore,
finishDestroy: (
record: WorkerEnvironmentRecord,
provider: WorkerProvider<"internal">,
provider: WorkerProvider,
) => Promise<WorkerEnvironmentRecord>,
): Promise<boolean> {
const transportMode = record.nodeDeviceId
? "worker-turn"
: record.sshEndpoint
? "remote-exec"
: undefined;
const modeError = transportMode
? resolveWorkerTransportModeError(provider, transportMode)
const transport = record.nodeDeviceId ? "node" : record.sshEndpoint ? "ssh" : undefined;
const modeError = transport
? resolveWorkerLeaseTransportError(provider, transport, record.profileSnapshot.executionMode)
: undefined;
if (!modeError || record.destroyRequestedAtMs !== null) {
return false;
src/gateway/worker-environments/provider-provisioning.replay.test.ts
+25
View File
@@ -3,13 +3,19 @@
import { expectDefined } from "@openclaw/normalization-core";
import { MAX_TIMER_TIMEOUT_MS } from "@openclaw/normalization-core/number-coercion";
import { describe, expect, it, vi } from "vitest";
import {
GATEWAY_CLIENT_IDS,
GATEWAY_CLIENT_MODES,
} from "../../../packages/gateway-protocol/src/client-info.js";
import { WORKER_EXECUTION_CONTEXT_PROTOCOL_FEATURE } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE } from "../../infra/node-runner-inventory.js";
import { WorkerProviderError } from "../../plugins/types.js";
import { createDeferredCore } from "../../shared/deferred.js";
import {
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../../state/openclaw-state-db.js";
import { bindDeviceWorkerAvailability } from "./device-provider.js";
import { REQUEST } from "./placement-dispatch-test-fixtures.js";
import { createWorkerPlacementDispatchService } from "./placement-dispatch.js";
import { createWorkerSessionPlacementStore } from "./placement-store.js";
@@ -240,6 +246,20 @@ describe("worker environment service provision replay", () => {
},
nodeTunnelManager: nodeTunnelManager as never,
});
bindDeviceWorkerAvailability(restarted, async (nodeId) => ({
available: true,
node: {
nodeId,
connId: `conn-${nodeId}`,
pairingIdentity: `identity-${nodeId}`,
pairingGeneration: `generation-${nodeId}`,
clientId: GATEWAY_CLIENT_IDS.NODE_HOST,
clientMode: GATEWAY_CLIENT_MODES.NODE,
protocolFeature: NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
workerHost: { enabled: true, capacity: { total: 1, available: 1 } },
commands: [],
},
}));
const recoveryBarrier = vi.fn(async ({ expectedGeneration, environmentId, run }) => {
expect(placements.get(REQUEST.sessionId)).toMatchObject({
state: "provisioning",
@@ -255,6 +275,11 @@ describe("worker environment service provision replay", () => {
placements,
environments: restarted,
runnerAvailability: { read: () => undefined, version: () => 0 },
resolveDevicePlacementRequirement: async () => ({
requiredNodeCommands: [],
consumesWorkerSlot: true,
}),
isCurrentNodePlacement: () => true,
workspaceOperations: createWorkerWorkspaceOperationCoordinator(),
runLocalBarrier: async ({ startDispatch }) => startDispatch(),
runRecoveryBarrier: recoveryBarrier,
src/gateway/worker-environments/provider-provisioning.shutdown-replay.test.ts
+25
View File
@@ -1,10 +1,16 @@
import { expectDefined } from "@openclaw/normalization-core";
import { describe, expect, it, vi } from "vitest";
import {
GATEWAY_CLIENT_IDS,
GATEWAY_CLIENT_MODES,
} from "../../../packages/gateway-protocol/src/client-info.js";
import { WORKER_EXECUTION_CONTEXT_PROTOCOL_FEATURE } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE } from "../../infra/node-runner-inventory.js";
import {
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../../state/openclaw-state-db.js";
import { bindDeviceWorkerAvailability } from "./device-provider.js";
import { createWorkerNodeEnrollmentManager } from "./node-enrollment.js";
import { REQUEST } from "./placement-dispatch-test-fixtures.js";
import { createWorkerPlacementDispatchService } from "./placement-dispatch.js";
@@ -101,6 +107,11 @@ describe("worker node provisioning shutdown replay", () => {
placements,
environments,
runnerAvailability: { read: () => undefined, version: () => 0 },
resolveDevicePlacementRequirement: async () => ({
requiredNodeCommands: [],
consumesWorkerSlot: true,
}),
isCurrentNodePlacement: () => true,
workspaceOperations: createWorkerWorkspaceOperationCoordinator(),
runLocalBarrier: async ({ startDispatch }) => startDispatch(),
runRecoveryBarrier: async ({ run }) => await run("/gateway/workspace"),
@@ -192,6 +203,20 @@ describe("worker node provisioning shutdown replay", () => {
ensureNodeWorkerBundle: async () => receipt,
nodeTunnelManager: nodeTunnelManager as never,
});
bindDeviceWorkerAvailability(restarted, async (nodeId) => ({
available: true,
node: {
nodeId,
connId: `conn-${nodeId}`,
pairingIdentity: `identity-${nodeId}`,
pairingGeneration: `generation-${nodeId}`,
clientId: GATEWAY_CLIENT_IDS.NODE_HOST,
clientMode: GATEWAY_CLIENT_MODES.NODE,
protocolFeature: NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
workerHost: { enabled: true, capacity: { total: 1, available: 1 } },
commands: [],
},
}));
const restartedDispatch = createDispatch(restarted);
const uninstallRestartedGuard = restarted.installReconcileEnvironmentGuard(
async (environmentId, reconcileCore) => {
src/gateway/worker-environments/provider-provisioning.test.ts
+155 -31
View File
@@ -100,57 +100,181 @@ describe("worker environment service", () => {
state: "ready",
});
expect(provision).toHaveBeenCalledOnce();
expect(provision).toHaveBeenCalledWith(
{ region: "test" },
expect.stringMatching(/^provision:v2:[a-f0-9]{64}$/u),
undefined,
);
});
it("P1: direct creation preserves the default setup of an advertised node provider", async () => {
const provision = vi.fn(async () => ({
leaseId: "lease-direct-default-node",
node: { deviceId: "device-direct-default-node" },
}));
const workerService = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
provision,
}),
{ ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT) },
);
const environment = await workerService.create("development", "request-direct-default-node");
expect(environment).toMatchObject({
state: "ready",
nodeDeviceId: "device-direct-default-node",
});
expect(environment.profileSnapshot).not.toHaveProperty("executionMode");
expect(provision).toHaveBeenCalledWith(
{ region: "test" },
expect.stringMatching(/^provision:v2:[a-f0-9]{64}$/u),
undefined,
);
});
it.each<{
mode: WorkerExecutionMode;
lease: WorkerLease;
expectedTransport: "node" | "SSH";
transport: "node" | "SSH";
inherited?: true;
}>([
{
mode: "worker-turn",
lease: { leaseId: "lease-worker-turn-ssh", ssh: support.SSH_ENDPOINT },
expectedTransport: "node",
lease: { leaseId: "lease-worker-turn-node", node: { deviceId: "worker-turn-device" } },
transport: "node",
},
{
mode: "remote-exec",
lease: { leaseId: "lease-remote-exec-node", node: { deviceId: "device-1" } },
expectedTransport: "SSH",
lease: { leaseId: "lease-remote-exec-node", node: { deviceId: "remote-exec-device" } },
transport: "node",
},
{
mode: "remote-exec",
lease: { leaseId: "lease-remote-exec-ssh", ssh: support.SSH_ENDPOINT },
transport: "SSH",
},
{
mode: "remote-exec",
lease: { leaseId: "lease-inherited-node", node: { deviceId: "inherited-device" } },
transport: "node",
inherited: true,
},
])(
"rejects a $mode provider that returns the wrong lease transport",
async ({ mode, lease, expectedTransport }) => {
const destroy = vi.fn(async () => {});
"forwards $mode placement to its $transport provider transport (inherited: $inherited)",
async ({ mode, lease, transport, inherited }) => {
const provision = vi.fn(async () => lease);
const provider = support.createProvider({
supportedExecutionModes: [mode],
provision: async () => lease,
destroy,
supportedExecutionModes: ["worker-turn", "remote-exec"],
provision,
});
const workerService = support.createService(provider);
await expect(
workerService.create("development", `transport-${mode}`, undefined, mode),
).rejects.toMatchObject({
code: "invalid_profile",
message: expect.stringContaining(
`${mode} providers must return a ${expectedTransport} lease`,
),
const workerService = support.createService(provider, {
ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT),
});
const idempotencyKey = `transport-${mode}-${transport}-${inherited ? "inherited" : "profile"}`;
expect(destroy).toHaveBeenCalledWith({ leaseId: lease.leaseId, profile: { region: "test" } });
expect(support.testState.bootstrapWorker).not.toHaveBeenCalled();
expect(support.testState.store.list()).toEqual([
expect.objectContaining({
state: "failed",
leaseId: null,
nodeDeviceId: null,
sshEndpoint: null,
lastError: `${mode} providers must return a ${expectedTransport} lease`,
}),
]);
const result = inherited
? await workerService.createFromProfileSnapshot(
{
profileId: "development",
providerId: provider.id,
profileSnapshot: { install: "bundle", settings: { region: "test" } },
},
idempotencyKey,
undefined,
mode,
)
: await workerService.create("development", idempotencyKey, undefined, mode);
expect(result).toMatchObject({
state: "ready",
leaseId: lease.leaseId,
profileSnapshot: { executionMode: mode, settings: { region: "test" } },
...(lease.node ? { nodeDeviceId: lease.node.deviceId, sshEndpoint: null } : {}),
});
expect(provision).toHaveBeenCalledWith(
{ region: "test" },
expect.stringMatching(/^provision:v2:[a-f0-9]{64}$/u),
{ executionMode: mode },
);
expect(support.testState.bootstrapWorker).toHaveBeenCalledTimes(transport === "SSH" ? 1 : 0);
},
);
it("rejects an SSH lease for worker-turn placement even when its provider also supports remote-exec", async () => {
const lease = { leaseId: "lease-worker-turn-ssh", ssh: support.SSH_ENDPOINT };
const destroy = vi.fn(async () => {});
const provider = support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
provision: async () => lease,
destroy,
});
const workerService = support.createService(provider);
await expect(
workerService.create("development", "transport-worker-turn-ssh", undefined, "worker-turn"),
).rejects.toMatchObject({
code: "invalid_profile",
message: expect.stringContaining("worker-turn providers must return a node lease"),
});
expect(destroy).toHaveBeenCalledWith({ leaseId: lease.leaseId, profile: { region: "test" } });
expect(support.testState.bootstrapWorker).not.toHaveBeenCalled();
expect(support.testState.store.list()).toEqual([
expect.objectContaining({
state: "failed",
leaseId: null,
nodeDeviceId: null,
sshEndpoint: null,
lastError: "worker-turn providers must return a node lease",
}),
]);
});
it("rejects a repeated operation id when its selected execution mode changes", async () => {
const provision = vi.fn(async () => ({
leaseId: "lease-stable-operation-mode",
node: { deviceId: "device-stable-operation-mode" },
}));
const workerService = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
provision,
}),
{ ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT) },
);
const original = await workerService.create(
"development",
"request-stable-operation-mode",
undefined,
"worker-turn",
);
await expect(
workerService.create(
"development",
"request-stable-operation-mode",
undefined,
"worker-turn",
),
).resolves.toMatchObject({ environmentId: original.environmentId });
await expect(
workerService.create(
"development",
"request-stable-operation-mode",
undefined,
"remote-exec",
),
).rejects.toMatchObject({ code: "invalid_profile" });
expect(provision).toHaveBeenCalledOnce();
expect(support.testState.store.get(original.environmentId)).toMatchObject({
state: "ready",
leaseId: original.leaseId,
});
});
it("delegates configured machine options to the profile provider", async () => {
const listMachineOptions = vi.fn(async () => [
{ id: "standard", label: "Standard", cpu: 32, memoryGb: 64, default: true },
src/gateway/worker-environments/provider-reconciliation.test.ts
+150
View File
@@ -97,6 +97,156 @@ describe("worker environment service", () => {
});
});
it.each([
["SSH", { leaseId: "lease-direct-only-ssh", ssh: support.SSH_ENDPOINT }],
["node", { leaseId: "lease-direct-only-node", node: { deviceId: "device-direct-only-node" } }],
] as const)(
"P1: preserves a direct-only %s lease after profile removal and reconciliation",
async (_transport, lease) => {
const inspect = vi.fn(async () => ({ status: "active" as const, sharedHost: false }));
const destroy = vi.fn(async () => {});
const workerService = support.createService(
support.createProvider({
supportedExecutionModes: undefined,
provision: async () => lease,
inspect,
destroy,
}),
{ ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT) },
);
const environment = await workerService.create("development", `request-${lease.leaseId}`);
support.testState.config.cloudWorkers!.profiles = {};
await workerService.reconcileOnce();
expect(environment.profileSnapshot).not.toHaveProperty("executionMode");
expect(inspect).toHaveBeenCalledWith({ leaseId: lease.leaseId, profile: { region: "test" } });
expect(destroy).not.toHaveBeenCalled();
expect(support.testState.store.get(environment.environmentId)).toMatchObject({
state: "ready",
leaseId: lease.leaseId,
});
},
);
it("P1: destroys a persisted SSH lease when provider capabilities and its profile are removed", async () => {
const leaseId = "lease-unadvertised-persisted-ssh";
const inspect = vi.fn(async () => ({ status: "active" as const }));
const destroy = vi.fn(async () => {});
const provider = support.createProvider({
supportedExecutionModes: ["remote-exec"],
provision: async () => ({ leaseId, ssh: support.SSH_ENDPOINT }),
inspect,
destroy,
});
const workerService = support.createService(provider);
const environment = await workerService.create(
"development",
"request-unadvertised-persisted-ssh",
undefined,
"remote-exec",
);
provider.supportedExecutionModes = undefined;
support.testState.config.cloudWorkers!.profiles = {};
await workerService.reconcileOnce();
expect(inspect).not.toHaveBeenCalled();
expect(destroy).toHaveBeenCalledWith({ leaseId, profile: { region: "test" } });
expect(support.testState.store.get(environment.environmentId)).toMatchObject({
state: "failed",
leaseId: null,
sshEndpoint: null,
lastError: expect.stringContaining("remote-exec"),
});
});
it.each([
{ name: "all placement capabilities", supportedExecutionModes: undefined },
{ name: "its exact remote-exec capability", supportedExecutionModes: ["worker-turn"] as const },
])(
"destroys a persisted node lease when its provider loses $name",
async ({ supportedExecutionModes }) => {
const leaseId = "lease-unadvertised-persisted-node";
const deviceId = "device-unadvertised-persisted-node";
const inspect = vi.fn(async () => ({ status: "active" as const, sharedHost: false }));
const destroy = vi.fn(async () => {});
const provider = support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
provisionBeforeInstallation: true,
provision: async () => ({ leaseId, node: { deviceId } }),
inspect,
destroy,
});
const workerService = support.createService(provider, {
ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT),
});
const environment = await workerService.create(
"development",
"request-unadvertised-persisted-node",
undefined,
"remote-exec",
);
provider.supportedExecutionModes = supportedExecutionModes;
support.getDevelopmentProfile().settings = { region: "edited" };
await workerService.reconcileOnce();
expect(inspect).not.toHaveBeenCalled();
expect(destroy).toHaveBeenCalledWith({ leaseId, profile: { region: "test" } });
expect(support.testState.store.get(environment.environmentId)).toMatchObject({
state: "failed",
leaseId: null,
nodeDeviceId: null,
lastError: expect.stringMatching(/node|remote-exec/u),
});
},
);
it.each([
["remote-exec-only", ["remote-exec"]],
["dual-mode", ["worker-turn", "remote-exec"]],
] as const)(
"preserves a persisted node lease after restarting with a %s provider",
async (_label, supportedExecutionModes) => {
const leaseId = "lease-persisted-multimode-node";
const deviceId = "device-persisted-multimode-node";
const inspect = vi.fn(async () => ({ status: "active" as const, sharedHost: false }));
const destroy = vi.fn(async () => {});
const initial = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
provisionBeforeInstallation: true,
provision: async () => ({ leaseId, node: { deviceId } }),
inspect,
destroy,
}),
{ ensureNodeWorkerBundle: async () => structuredClone(support.BOOTSTRAP_RECEIPT) },
);
const environment = await initial.create(
"development",
"request-persisted-multimode-node",
undefined,
"remote-exec",
);
await initial.stop();
const restarted = support.createService(
support.createProvider({ supportedExecutionModes, inspect, destroy }),
);
await restarted.reconcileOnce();
expect(inspect).toHaveBeenCalledWith({ leaseId, profile: { region: "test" } });
expect(destroy).not.toHaveBeenCalled();
expect(support.testState.store.get(environment.environmentId)).toMatchObject({
state: "ready",
leaseId,
nodeDeviceId: deviceId,
sshEndpoint: null,
});
},
);
it("reconciles one exact environment without sweeping its siblings", async () => {
support.seedReady("worker-target");
support.seedReady("worker-sibling");
src/gateway/worker-environments/reclaimed-placement-redispatch.test.ts
+63 -8
View File
@@ -77,25 +77,80 @@ describe("createReclaimedPlacementRedispatch", () => {
);
});
it("rejects paired-device redispatch without its runtime requirement owner", async () => {
const dispatch = vi.fn();
it("revalidates a reclaimed cloud node without targeting its retired device", async () => {
const remotePlacement = {
...placement,
executionMode: "remote-exec" as const,
} as ReclaimedWorkerPlacement;
const requirement = {
requiredNodeCommands: ["codex.exec-server.stdio.v1"],
consumesWorkerSlot: false,
};
const dispatch = vi.fn(async () => ({ state: "active" }) as never);
const resolveDevicePlacementRequirement = vi.fn(async () => requirement);
const redispatch = createReclaimedPlacementRedispatch({
environments: {
get: () =>
({
profileId: "device:paired-node",
providerId: "device",
nodeDeviceId: "paired-node",
profileSnapshot: { install: "bundle", settings: { device: "paired-node" } },
profileId: "cloud:development",
providerId: "crabbox",
nodeDeviceId: "retired-cloud-node",
profileSnapshot: { machineClass: "large", settings: { region: "parent" } },
}) as never,
},
dispatch,
resolveDevicePlacementRequirement,
});
await expect(redispatch(placement)).rejects.toThrow("authoritative runtime requirement");
expect(dispatch).not.toHaveBeenCalled();
await redispatch(remotePlacement);
expect(resolveDevicePlacementRequirement).toHaveBeenCalledWith({
sessionId: remotePlacement.sessionId,
sessionKey: remotePlacement.sessionKey,
agentId: remotePlacement.agentId,
executionMode: "remote-exec",
});
expect(dispatch).toHaveBeenCalledWith({
sessionId: remotePlacement.sessionId,
sessionKey: remotePlacement.sessionKey,
agentId: remotePlacement.agentId,
profileId: "cloud:development",
executionMode: "remote-exec",
devicePlacement: requirement,
inheritedProfile: {
providerId: "crabbox",
profileSnapshot: { machineClass: "large", settings: { region: "parent" } },
},
});
});
it.each([
{ providerId: "device", executionMode: "worker-turn" },
{ providerId: "crabbox", executionMode: "remote-exec" },
] as const)(
"rejects $providerId node redispatch without its runtime requirement owner",
async ({ providerId, executionMode }) => {
const dispatch = vi.fn();
const redispatch = createReclaimedPlacementRedispatch({
environments: {
get: () =>
({
profileId: "device:paired-node",
providerId,
nodeDeviceId: "paired-node",
profileSnapshot: { install: "bundle", settings: { device: "paired-node" } },
}) as never,
},
dispatch,
});
await expect(
redispatch({ ...placement, executionMode } as ReclaimedWorkerPlacement),
).rejects.toThrow("authoritative runtime requirement");
expect(dispatch).not.toHaveBeenCalled();
},
);
it("fails closed when the prior environment record is unavailable", async () => {
const redispatch = createReclaimedPlacementRedispatch({
environments: { get: () => undefined },
src/gateway/worker-environments/reclaimed-placement-redispatch.ts
+4 -6
View File
@@ -19,12 +19,9 @@ export function createReclaimedPlacementRedispatch(params: {
);
}
let devicePlacement: Awaited<ReturnType<WorkerDevicePlacementRequirementResolver>> | undefined;
if (
previousEnvironment.providerId === DEVICE_WORKER_PROVIDER_ID &&
previousEnvironment.nodeDeviceId
) {
if (previousEnvironment.nodeDeviceId) {
if (!params.resolveDevicePlacementRequirement) {
throw new Error("Paired-device redispatch has no authoritative runtime requirement");
throw new Error("Node-backed redispatch has no authoritative runtime requirement");
}
devicePlacement = await params.resolveDevicePlacementRequirement({
sessionId: placement.sessionId,
@@ -39,9 +36,10 @@ export function createReclaimedPlacementRedispatch(params: {
agentId: placement.agentId,
profileId: previousEnvironment.profileId,
executionMode: placement.executionMode,
...(devicePlacement ? { devicePlacement } : {}),
...(previousEnvironment.providerId === DEVICE_WORKER_PROVIDER_ID &&
previousEnvironment.nodeDeviceId
? { deviceId: previousEnvironment.nodeDeviceId, devicePlacement }
? { deviceId: previousEnvironment.nodeDeviceId }
: {}),
inheritedProfile: {
providerId: previousEnvironment.providerId,
src/gateway/worker-environments/service-validation.ts
+23 -17
View File
@@ -5,7 +5,6 @@ import { WorkerMachineOptionsSchema } from "../../../packages/gateway-protocol/s
import {
WorkerProviderError,
type WorkerDesktopEndpoint,
type WorkerExecutionMode,
type WorkerLease,
type WorkerLeaseStatus,
type WorkerProvider,
@@ -86,24 +85,31 @@ export function requireWorkerLeaseStatus(value: unknown): WorkerLeaseStatus {
return { status };
}
export function resolveWorkerTransportModeError(
provider: WorkerProvider<"internal">,
transportMode: WorkerExecutionMode,
export function resolveWorkerLeaseTransportError(
provider: WorkerProvider,
transport: "node" | "ssh",
executionMode?: unknown,
): WorkerProviderError | undefined {
const modes = provider.supportedExecutionModes;
const executionMode: WorkerExecutionMode | undefined = modes?.length === 1 ? modes[0] : undefined;
return !executionMode || executionMode === transportMode
? undefined
: new WorkerProviderError(
`${executionMode} providers must return a ${executionMode === "worker-turn" ? "node" : "SSH"} lease`,
);
}
export function resolveWorkerLeaseModeError(
provider: WorkerProvider<"internal">,
lease: WorkerLease,
): WorkerProviderError | undefined {
return resolveWorkerTransportModeError(provider, lease.node ? "worker-turn" : "remote-exec");
if (
executionMode !== undefined &&
executionMode !== "worker-turn" &&
executionMode !== "remote-exec"
) {
return new WorkerProviderError("Worker environment has an invalid placement execution mode");
}
if (
transport === "ssh" &&
(executionMode === "worker-turn" || (modes !== undefined && !modes.includes("remote-exec")))
) {
return new WorkerProviderError("worker-turn providers must return a node lease");
}
if (executionMode !== undefined && !modes?.includes(executionMode)) {
return new WorkerProviderError(
`Worker provider ${provider.id} does not advertise ${executionMode} for its ${transport} lease`,
);
}
return undefined;
}
export function requireWorkerLease(value: unknown): WorkerLease {
src/gateway/worker-environments/service.ts
+10 -5
View File
@@ -495,10 +495,15 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService
const service = {
list: environmentAccess.list,
supportsProviderExecutionMode: providerSupportsExecutionMode,
supportsExecutionMode: (profileId: string, mode: WorkerExecutionMode) => {
const profile = options.getConfig().cloudWorkers?.profiles?.[profileId];
return profile ? providerSupportsExecutionMode(profile.provider, mode) : false;
},
requiresNodeEnrollment: (profileId: string, providerId?: string) => {
const id = providerId ?? options.getConfig().cloudWorkers?.profiles?.[profileId]?.provider;
return id ? options.resolveProvider(id)?.requiresNodeEnrollment === true : false;
},
get: environmentAccess.get,
hasPendingNodeEnrollmentSetup: (setupId: string, deviceId: string) =>
store.hasPendingNodeEnrollmentSetup(setupId, deviceId),
@@ -514,11 +519,10 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService
requireProviderExecutionMode(configuredProfileProviderId(profileId), executionMode);
}
return environmentAccess.project(
await providerLifecycle.createWithProfile(
profileId,
idempotencyKey,
machineClass === undefined ? {} : { machineClass },
),
await providerLifecycle.createWithProfile(profileId, idempotencyKey, {
...(machineClass === undefined ? {} : { machineClass }),
...(executionMode === undefined ? {} : { executionMode }),
}),
);
},
createFromProfileSnapshot: async (
@@ -535,6 +539,7 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService
profileSnapshot: profile.profileSnapshot,
},
...(machineClass === undefined ? {} : { machineClass }),
...(executionMode === undefined ? {} : { executionMode }),
}),
);
},
src/gateway/worker-environments/worker-turn-launcher-failure-recovery.test.ts
+1
View File
@@ -185,6 +185,7 @@ describe("worker turn launcher failure recovery", () => {
});
const environments: WorkerTurnEnvironmentService & WorkerDispatchEnvironmentService = {
...unusedEnvironments(),
supportsProviderExecutionMode: vi.fn(() => true),
get: vi.fn(() => environment),
acquireTurnCredential: vi.fn(async () => credential()),
acknowledgeCredentialDelivery: vi.fn(() => true),
src/gateway/worker-environments/worker-turn-launcher.test.ts
+12 -9
View File
@@ -461,11 +461,12 @@ describe("worker turn launcher local placement", () => {
);
it.each([
{ label: "SSH", nodeDeviceId: undefined },
{ label: "paired-device", nodeDeviceId: "paired-node-1" },
{ label: "SSH", nodeDeviceId: undefined, providerId: "fake" },
{ label: "paired-device", nodeDeviceId: "paired-node-1", providerId: "device" },
{ label: "cloud-node", nodeDeviceId: "cloud-node-1", providerId: "crabbox" },
])(
"runs a $label remote-exec placement locally and reconciles without launching a worker child",
async ({ nodeDeviceId }) => {
async ({ nodeDeviceId, providerId }) => {
seedActivePlacement("remote-exec");
const order: string[] = [];
const launchTurn = vi.fn();
@@ -504,7 +505,7 @@ describe("worker turn launcher local placement", () => {
...unusedEnvironments(),
get: vi.fn(() =>
nodeDeviceId
? { ...attachedEnvironment(), providerId: "device", nodeDeviceId, sshEndpoint: null }
? { ...attachedEnvironment(), providerId, nodeDeviceId, sshEndpoint: null }
: attachedEnvironment(),
),
startTunnel: vi.fn(async () => tunnel),
@@ -715,11 +716,13 @@ describe("worker turn launcher local placement", () => {
);
it.each([
{ label: "failed execution", executionFailed: true },
{ label: "successful execution", executionFailed: false },
{ label: "failed paired-device execution", executionFailed: true, providerId: "device" },
{ label: "successful paired-device execution", executionFailed: false, providerId: "device" },
{ label: "failed cloud-node execution", executionFailed: true, providerId: "crabbox" },
{ label: "successful cloud-node execution", executionFailed: false, providerId: "crabbox" },
])(
"preserves a disconnected paired-node placement after $label for a fresh attempt",
async ({ executionFailed }) => {
"preserves a disconnected node-backed placement after $label for a fresh attempt",
async ({ executionFailed, providerId }) => {
seedActivePlacement("remote-exec");
const original = placements.get(SESSION_ID);
if (original?.state !== "active") {
@@ -758,7 +761,7 @@ describe("worker turn launcher local placement", () => {
};
const environment = {
...attachedEnvironment(),
providerId: "device",
providerId,
nodeDeviceId: "paired-node-1",
sshEndpoint: null,
};
src/gateway/worker-environments/workspace-result-finalize.ts
+2 -4
View File
@@ -4,7 +4,6 @@ import type { SessionPlacementTurnParams } from "../../agents/session-placement-
import { SessionManager } from "../../agents/sessions/session-manager.js";
import { formatErrorMessage } from "../../infra/errors.js";
import { redactSensitiveText } from "../../logging/redact.js";
import { DEVICE_WORKER_PROVIDER_ID } from "./device-provider-identity.js";
import type {
WorkerSessionPlacementRecord,
WorkerSessionPlacementStore,
@@ -336,10 +335,9 @@ export async function executeRemoteExecTurn(params: {
}).catch((reconciliationError: unknown) => {
const currentEnvironment = params.environments.get(params.placement.environmentId);
if (
environment.providerId === DEVICE_WORKER_PROVIDER_ID &&
environment.nodeDeviceId &&
currentEnvironment?.state === "attached" &&
currentEnvironment.providerId === DEVICE_WORKER_PROVIDER_ID &&
currentEnvironment.providerId === environment.providerId &&
currentEnvironment.environmentId === environment.environmentId &&
currentEnvironment.ownerEpoch === environment.ownerEpoch &&
currentEnvironment.nodeDeviceId === environment.nodeDeviceId &&
@@ -348,7 +346,7 @@ export async function executeRemoteExecTurn(params: {
reconciliationError instanceof WorkerWorkspaceReconciliationError &&
reconciliationError.cause instanceof WorkerTunnelOwnerDisconnectedError
) {
// Offline paired nodes keep their exact lease; the next turn reconciles its dirty workspace.
// Offline nodes keep their exact lease; the next turn reconciles its dirty workspace.
params.placements.cancelWorkspaceResultAndReleaseTurn(params.turnClaim, {
reason: "node-disconnect",
});
src/gateway/worker-workspace-recovery-transcript.test.ts
+1
View File
@@ -68,6 +68,7 @@ describe("worker workspace recovery transcript reporting", () => {
throw new Error("expected active worker placement");
}
harness.markEnvironmentOwnerEpoch(active.activeOwnerEpoch);
harness.markEnvironmentNodeDeviceId("workspace-recovery-worker-node");
const claim = placements.claimTurn({
...REQUEST,
claimId: "workspace-recovery-claim",
src/node-host/node-worker-supervisor.ts
+9 -8
View File
@@ -210,22 +210,23 @@ class NodeWorkerSupervisor {
}
if (active?.state === "running") {
if (active.container) {
const containerState = await this.requireContainerLifecycle().inspect(
active.container,
active,
);
if (containerState === "unknown") {
const inspection = await this.requireContainerLifecycle().inspect(active.container, active);
if (inspection === "unknown") {
return this.store.get(launchId);
}
if (containerState === "reused") {
if (inspection === "reused") {
throw new Error(`node worker launch ${launchId} lost its container ownership`);
}
if (containerState === "live") {
if (inspection === "live") {
const clientState = inspectNodeWorkerProcessIdentity(active.worker);
if (clientState !== "dead" && clientState !== "reused") {
return this.store.get(launchId);
}
await this.stopChild(active, "interrupted");
// Observe the dead attach client's result before fencing its still-running owner.
await active.done;
if (this.active.get(launchId) === active) {
await this.stopChild(active, "interrupted");
}
} else {
await this.cleanupActiveContainer(active);
await active.done;
src/plugins/capability-provider.types.ts
+7 -6
View File
@@ -188,15 +188,15 @@ export class WorkerProviderError extends Error {
}
}
/** Plugin registrations declare exactly one mode; the internal paired-device owner can carry both. */
export type WorkerProvider<Scope extends "plugin" | "internal" = "plugin"> = {
/** Cloud-worker lifecycle capability shared by plugin and internal providers. */
export type WorkerProvider = {
id: string;
/** Process-stable choices available for this profile; omit the hook to hide machine selection. */
listMachineOptions?: (profile: WorkerProfile) => Promise<readonly WorkerMachineOption[]>;
/** Omission advertises no placement support; external providers declare one transport mode. */
supportedExecutionModes?: Scope extends "internal"
? readonly [WorkerExecutionMode] | readonly ["worker-turn", "remote-exec"]
: readonly [WorkerExecutionMode];
/** Omission advertises no placement support; multiple modes use their canonical order. */
supportedExecutionModes?:
| readonly [WorkerExecutionMode]
| readonly ["worker-turn", "remote-exec"];
/**
* Provision before preparing an installation when the lease transport decides whether an
* installation is needed. Defaults to false so SSH providers retain prepare-before-allocation.
@@ -212,6 +212,7 @@ export type WorkerProvider<Scope extends "plugin" | "internal" = "plugin"> = {
profile: WorkerProfile,
operationId: string,
options?: {
executionMode?: WorkerExecutionMode;
machineClass?: string;
beginNodeEnrollment?: () => Promise<WorkerNodeEnrollment>;
},
src/plugins/worker-provider-registry.test.ts
+22 -3
View File
@@ -122,10 +122,30 @@ describe("worker provider registry", () => {
);
});
it("registers both placement modes in canonical order", () => {
const pluginRegistry = createTestRegistry();
const provider = {
...createWorkerProvider("static-ssh"),
supportedExecutionModes: ["worker-turn", "remote-exec"],
} satisfies WorkerProvider;
pluginRegistry.registerWorkerProvider(createOwner("owner", ["static-ssh"]), provider);
expect(pluginRegistry.registry.workerProviders.get("static-ssh")?.provider).toBe(provider);
expect(pluginRegistry.registry.diagnostics).toEqual([]);
});
it.each([
{ modes: [], label: "no modes" },
{ modes: ["worker-turn", "remote-exec"], label: "multiple modes" },
{ modes: ["remote-exec", "worker-turn"], label: "modes in noncanonical order" },
{ modes: ["worker-turn", "worker-turn"], label: "duplicate worker-turn modes" },
{ modes: ["remote-exec", "remote-exec"], label: "duplicate remote-exec modes" },
{ modes: ["unsupported"], label: "an unknown mode" },
{ modes: ["worker-turn", "unsupported"], label: "an unknown additional mode" },
{
modes: ["worker-turn", "remote-exec", "worker-turn"],
label: "more than two modes",
},
])("rejects $label in a placement declaration", ({ modes }) => {
const pluginRegistry = createTestRegistry();
const provider = {
@@ -138,8 +158,7 @@ describe("worker provider registry", () => {
expect(pluginRegistry.registry.workerProviders.size).toBe(0);
expect(pluginRegistry.registry.diagnostics).toContainEqual(
expect.objectContaining({
message:
"worker provider registration supportedExecutionModes must contain exactly one current mode",
message: expect.stringContaining("worker provider registration supportedExecutionModes"),
}),
);
});
src/plugins/worker-provider-registry.ts
+9 -7
View File
@@ -39,16 +39,18 @@ export function validateWorkerProviderContract(
};
}
const executionModes = provider.supportedExecutionModes;
if (
executionModes !== undefined &&
(!Array.isArray(executionModes) ||
executionModes.length !== 1 ||
(executionModes[0] !== "worker-turn" && executionModes[0] !== "remote-exec"))
) {
const validExecutionModes =
Array.isArray(executionModes) &&
((executionModes.length === 1 &&
(executionModes[0] === "worker-turn" || executionModes[0] === "remote-exec")) ||
(executionModes.length === 2 &&
executionModes[0] === "worker-turn" &&
executionModes[1] === "remote-exec"));
if (executionModes !== undefined && !validExecutionModes) {
return {
ok: false,
message:
"worker provider registration supportedExecutionModes must contain exactly one current mode",
"worker provider registration supportedExecutionModes must contain one current mode or both current modes in canonical order",
};
}
if (
test/e2e/qa-lab/runtime/codex-node-exec-server.e2e.test.ts
+3 -3
View File
@@ -355,7 +355,7 @@ async function resolveNextApproval(
}
expect(pending.request).toMatchObject({
pluginId: "codex",
title: "Run Codex execution on paired device",
title: "Run Codex execution on node",
allowedDecisions: ["allow-once", "deny"],
});
await reviewer.request("plugin.approval.resolve", { id: pending.id, decision });
@@ -728,7 +728,7 @@ describe("Codex paired-device exec-server carrier", () => {
expect(interruptedOutcome).toMatchObject({ status: "error" });
if (
typeof interruptedOutcome.error !== "string" ||
!/paired.*device disconnected.*fresh attempt/iu.test(interruptedOutcome.error)
!/execution node disconnected.*fresh attempt/iu.test(interruptedOutcome.error)
) {
throw new Error(
`Codex node disconnect omitted actionable guidance: ${JSON.stringify({
@@ -743,7 +743,7 @@ describe("Codex paired-device exec-server carrier", () => {
);
}
expect(interruptedOutcome.error).toEqual(
expect.stringMatching(/paired.*device disconnected.*fresh attempt/iu),
expect.stringMatching(/execution node disconnected.*fresh attempt/iu),
);
await vi.waitFor(
() => expect(processIsAlive(interruptedProcess)).toBe(false),
ui/src/e2e/new-session-page.places-live.e2e.test.ts
+69 -1
View File
@@ -47,6 +47,8 @@ suite.define(() => {
{
id: "aws",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes: ["worker-turn"],
machines: [
{ id: "standard", label: "Standard", default: true },
{ id: "fast", label: "Fast" },
@@ -92,7 +94,14 @@ suite.define(() => {
methodResponses: {
"environments.list": {
environments: [],
profiles: [{ id: "aws", providerId: "crabbox", executionMode: "worker-turn" }],
profiles: [
{
id: "aws",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes: ["worker-turn"],
},
],
},
"worktrees.branches": { branches: [], repositoryStatus: "git" },
},
@@ -114,6 +123,65 @@ suite.define(() => {
}
});
it.each([
{ name: "OpenClaw", runtime: "openclaw" },
{ name: "Codex", runtime: "codex" },
] as const)(
"keeps the same multimode Crabbox profile selectable for $name",
async ({ runtime }) => {
const context = await suite.browser.newContext({ locale: "en-US", serviceWorkers: "block" });
const page = await context.newPage();
const gateway = await installMockGateway(page, {
agentModel: "openai/gpt-5.5",
models: [
{
id: "gpt-5.5",
name: "gpt-5.5",
provider: "openai",
agentRuntime: {
id: runtime,
cloudPlacementSupported: true,
cloudPlacementExecutionMode: runtime === "codex" ? "remote-exec" : "worker-turn",
source: "model",
},
},
],
workspace: WORKSPACE,
workspaceGit: true,
methodResponses: {
"environments.list": {
environments: [],
profiles: [
{
id: "aws",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
},
],
},
"worktrees.branches": { branches: [], repositoryStatus: "git" },
},
});
try {
await page.goto(`${suite.server.baseUrl}new`);
await gateway.waitForRequest("environments.list");
await gateway.waitForRequest("chat.metadata");
await page.locator("#new-session-where-trigger").click();
const profile = page.locator('[data-value="cloud:aws"]');
await profile.waitFor();
await expect.poll(() => profile.isEnabled()).toBe(true);
await profile.click();
await expect
.poll(() => page.locator("#new-session-where-trigger").textContent())
.toContain("aws");
} finally {
await context.close();
}
},
);
it("refreshes authoritative device capacity from Gateway topology events", async () => {
const context = await suite.browser.newContext({ locale: "en-US", serviceWorkers: "block" });
const page = await context.newPage();
ui/src/pages/chat/chat-pane-placement.runtime.ts
+2 -2
View File
@@ -15,6 +15,7 @@ import {
projectDevicePlacements,
type DevicePlacementRequirement,
} from "../new-session/device-placement.ts";
import { draftCloudProfileSupportsExecutionMode } from "../new-session/discovery.ts";
async function loadPlacementMoveCatalog(
client: GatewayBrowserClient,
@@ -86,8 +87,7 @@ export async function moveChatPanePlacement(params: {
return t("newSession.cloudRuntimeUnsupported", { runtime: runtime.id });
}
return runtime?.cloudPlacementExecutionMode &&
profile.executionMode &&
profile.executionMode !== runtime.cloudPlacementExecutionMode
!draftCloudProfileSupportsExecutionMode(profile, runtime.cloudPlacementExecutionMode)
? t("newSession.cloudProfileRuntimeUnsupported", { runtime: runtime.id })
: undefined;
},
ui/src/pages/chat/chat-pane-placement.test.ts
+118 -81
View File
@@ -350,89 +350,126 @@ describe("chat pane placement", () => {
expect(refreshReplacement).toHaveBeenCalledWith("main");
});
it("disables incompatible cloud execution modes while preserving compatible machine selection", async () => {
const request = vi.fn(async (method: string) => {
if (method === "environments.list") {
return {
profiles: [
{
id: "worker-only",
providerId: "crabbox",
executionMode: "worker-turn",
},
{
id: "remote-exec",
providerId: "crabbox",
executionMode: "remote-exec",
machines: [
{ id: "standard", label: "Standard", default: true },
{ id: "beast", label: "Beast" },
],
},
],
environments: [],
};
}
return { ok: true };
});
const { pane } = createTestChatPane({
client: { request } as unknown as GatewayBrowserClient,
sessions: {
refreshReplacement: vi.fn(async () => undefined),
} as unknown as SessionCapability,
});
pane.context.gateway.snapshot.hello = {
features: { methods: ["sessions.move"] },
auth: { role: "operator", scopes: ["operator.admin", "operator.write"] },
} as never;
const session = {
...activePlacementSession(),
agentRuntime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec",
source: "model",
},
} satisfies GatewaySessionRow;
const moving = pane.moveHeaderPlacement(session);
try {
await vi.waitFor(() => {
expect(document.body.querySelector('[data-value="cloud:worker-only"]')).not.toBeNull();
it.each([
{
runtimeId: "openclaw",
executionMode: "worker-turn",
compatibleSingleMode: "worker-only",
incompatibleSingleMode: "remote-only",
},
{
runtimeId: "codex",
executionMode: "remote-exec",
compatibleSingleMode: "remote-only",
incompatibleSingleMode: "worker-only",
},
] as const)(
"moves $runtimeId to the same two-mode cloud profile while preserving machine selection",
async ({ runtimeId, executionMode, compatibleSingleMode, incompatibleSingleMode }) => {
const request = vi.fn(async (method: string) => {
if (method === "environments.list") {
return {
profiles: [
{
id: "lifecycle-only",
providerId: "crabbox",
},
{
id: "worker-only",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes: ["worker-turn"],
},
{
id: "remote-only",
providerId: "crabbox",
executionMode: "remote-exec",
executionModes: ["remote-exec"],
},
{
id: "aws",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
machines: [
{ id: "standard", label: "Standard", default: true },
{ id: "beast", label: "Beast" },
],
},
],
environments: [],
};
}
return { ok: true };
});
const incompatible = document.body.querySelector<HTMLButtonElement>(
'[data-value="cloud:worker-only"]',
);
expect(incompatible?.disabled).toBe(true);
expect(incompatible?.title).toMatch(/compatible cloud worker|cannot use/i);
const compatible = document.body.querySelector<HTMLButtonElement>(
'[data-value="cloud:remote-exec"]',
);
expect(compatible?.disabled).toBe(false);
compatible?.click();
document.body.querySelector<HTMLButtonElement>('[data-value="machine:beast"]')?.click();
[...document.body.querySelectorAll<HTMLButtonElement>("button")]
.find((button) => button.textContent?.trim() === "Move session")
?.click();
await moving;
const { pane } = createTestChatPane({
client: { request } as unknown as GatewayBrowserClient,
sessions: {
refreshReplacement: vi.fn(async () => undefined),
} as unknown as SessionCapability,
});
pane.context.gateway.snapshot.hello = {
features: { methods: ["sessions.move"] },
auth: { role: "operator", scopes: ["operator.admin", "operator.write"] },
} as never;
const session = {
...activePlacementSession(),
agentRuntime: {
id: runtimeId,
cloudPlacementSupported: true,
cloudPlacementExecutionMode: executionMode,
source: "model",
},
} satisfies GatewaySessionRow;
expect(request).toHaveBeenCalledWith(
"sessions.move",
expect.objectContaining({
target: {
kind: "profile",
profileId: "remote-exec",
machineClass: "beast",
},
}),
);
} finally {
[...document.body.querySelectorAll<HTMLButtonElement>("button")]
.find((button) => button.textContent?.trim() === "Cancel")
?.click();
await moving;
}
});
const moving = pane.moveHeaderPlacement(session);
try {
await vi.waitFor(() => {
expect(document.body.querySelector('[data-value="cloud:aws"]')).not.toBeNull();
});
const incompatible = document.body.querySelector<HTMLButtonElement>(
`[data-value="cloud:${incompatibleSingleMode}"]`,
);
expect(incompatible?.disabled).toBe(true);
expect(incompatible?.title).toMatch(/compatible cloud worker|cannot use/i);
const lifecycleOnly = document.body.querySelector<HTMLButtonElement>(
'[data-value="cloud:lifecycle-only"]',
);
expect(lifecycleOnly?.disabled).toBe(true);
expect(lifecycleOnly?.title).toMatch(/compatible cloud worker|cannot use/i);
const compatibleSingle = document.body.querySelector<HTMLButtonElement>(
`[data-value="cloud:${compatibleSingleMode}"]`,
);
expect(compatibleSingle?.disabled).toBe(false);
const multiMode = document.body.querySelector<HTMLButtonElement>(
'[data-value="cloud:aws"]',
);
expect(multiMode?.disabled).toBe(false);
multiMode?.click();
document.body.querySelector<HTMLButtonElement>('[data-value="machine:beast"]')?.click();
[...document.body.querySelectorAll<HTMLButtonElement>("button")]
.find((button) => button.textContent?.trim() === "Move session")
?.click();
await moving;
expect(request).toHaveBeenCalledWith(
"sessions.move",
expect.objectContaining({
target: {
kind: "profile",
profileId: "aws",
machineClass: "beast",
},
}),
);
} finally {
[...document.body.querySelectorAll<HTMLButtonElement>("button")]
.find((button) => button.textContent?.trim() === "Cancel")
?.click();
await moving;
}
},
);
it("cancels offline-device continuation without opening a picker or sending an RPC", async () => {
const request = vi.fn(async () => ({ ok: true }));
ui/src/pages/new-session/discovery.test.ts
+66 -3
View File
@@ -1,6 +1,37 @@
// @vitest-environment node
import { describe, expect, it } from "vitest";
import { readDraftCloudProfiles, readDraftEnvironments } from "./discovery.ts";
import {
draftCloudProfileSupportsExecutionMode,
readDraftCloudProfiles,
readDraftEnvironments,
} from "./discovery.ts";
describe("draftCloudProfileSupportsExecutionMode", () => {
it.each([
{ name: "worker turns", executionMode: "worker-turn" },
{ name: "remote execution", executionMode: "remote-exec" },
] as const)(
"rejects $name when the provider advertises no placement modes",
({ executionMode }) => {
expect(
draftCloudProfileSupportsExecutionMode(
{ id: "lifecycle-only", providerId: "crabbox" },
executionMode,
),
).toBe(false);
},
);
it("does not treat the singular display projection as a placement capability", () => {
expect(
draftCloudProfileSupportsExecutionMode(
{ id: "legacy", providerId: "crabbox", executionMode: "worker-turn" },
"worker-turn",
),
).toBe(false);
});
});
describe("readDraftCloudProfiles", () => {
it("keeps closed profile summaries in stable order", () => {
expect(
@@ -18,7 +49,8 @@ describe("readDraftCloudProfiles", () => {
id: "aws",
providerId: "crabbox",
trust: "persistent",
executionMode: "remote-exec",
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
machines: [
{
id: "standard",
@@ -47,7 +79,8 @@ describe("readDraftCloudProfiles", () => {
id: "aws",
providerId: "crabbox",
trust: "persistent",
executionMode: "remote-exec",
executionMode: "worker-turn",
executionModes: ["worker-turn", "remote-exec"],
machines: [
{
id: "standard",
@@ -79,6 +112,36 @@ describe("readDraftCloudProfiles", () => {
},
]);
});
it.each([
{ name: "empty", executionModes: [] },
{ name: "unknown", executionModes: ["sandbox"] },
{ name: "duplicate", executionModes: ["remote-exec", "remote-exec"] },
{ name: "out-of-order", executionModes: ["remote-exec", "worker-turn"] },
{ name: "oversized", executionModes: ["worker-turn", "remote-exec", "worker-turn"] },
])(
"keeps a present $name mode set closed instead of using its primary-mode fallback",
({ executionModes }) => {
expect(
readDraftCloudProfiles([
{
id: "aws",
providerId: "crabbox",
executionMode: "remote-exec",
executionModes,
},
]),
).toEqual([
{
id: "aws",
providerId: "crabbox",
trust: undefined,
executionMode: "remote-exec",
executionModes: [],
},
]);
},
);
});
describe("readDraftEnvironments", () => {
ui/src/pages/new-session/discovery.ts
+31 -1
View File
@@ -30,6 +30,7 @@ export type DraftCloudProfile = {
providerId: string;
trust?: "persistent" | "disposable";
executionMode?: WorkerExecutionMode;
executionModes?: readonly WorkerExecutionMode[];
machines?: DraftMachineOption[];
};
@@ -88,6 +89,25 @@ function readRuntimeTargetIssues(value: unknown): RuntimeTargetIssue[] | undefin
return issues.length > 0 ? issues : undefined;
}
function readDraftCloudProfileExecutionModes(value: unknown): readonly WorkerExecutionMode[] {
if (!Array.isArray(value)) {
return [];
}
if (value.length === 1 && (value[0] === "worker-turn" || value[0] === "remote-exec")) {
return [value[0]];
}
return value.length === 2 && value[0] === "worker-turn" && value[1] === "remote-exec"
? ["worker-turn", "remote-exec"]
: [];
}
export function draftCloudProfileSupportsExecutionMode(
profile: DraftCloudProfile,
executionMode: WorkerExecutionMode,
): boolean {
return profile.executionModes?.includes(executionMode) === true;
}
export function readDraftCloudProfiles(value: unknown): DraftCloudProfile[] {
return (Array.isArray(value) ? value : [])
.flatMap<DraftCloudProfile>((raw) => {
@@ -99,6 +119,7 @@ export function readDraftCloudProfiles(value: unknown): DraftCloudProfile[] {
providerId?: unknown;
trust?: unknown;
executionMode?: unknown;
executionModes?: unknown;
machines?: unknown;
};
const id = normalizeOptionalString(profile.id);
@@ -116,7 +137,16 @@ export function readDraftCloudProfiles(value: unknown): DraftCloudProfile[] {
: undefined;
const machines = readDraftMachineOptions(profile.machines);
return [
{ id, providerId, trust, executionMode, ...(machines.length > 0 ? { machines } : {}) },
{
id,
providerId,
trust,
executionMode,
...(Object.hasOwn(profile, "executionModes")
? { executionModes: readDraftCloudProfileExecutionModes(profile.executionModes) }
: {}),
...(machines.length > 0 ? { machines } : {}),
},
];
})
.toSorted((left, right) => left.id.localeCompare(right.id));
ui/src/pages/new-session/draft-place-state.test.ts
+38 -10
View File
@@ -100,10 +100,30 @@ describe("DraftPlaceState cloud machine selection", () => {
expect(state.machineClass).toBe("");
});
it("clears a selected cloud profile when the runtime switches to an incompatible mode", () => {
it.each([
{
name: "clears a one-mode cloud profile when the runtime becomes incompatible",
executionModes: ["worker-turn"] as const,
compatible: false,
},
{
name: "retains a two-mode cloud profile and its machine when the runtime changes",
executionModes: ["worker-turn", "remote-exec"] as const,
compatible: true,
},
])("$name", ({ executionModes, compatible }) => {
const persistPreference = vi.fn();
const cloudProfiles: DraftCloudProfile[] = [
{ id: "aws", providerId: "crabbox", executionMode: "worker-turn" },
{
id: "aws",
providerId: "crabbox",
executionMode: "worker-turn",
executionModes,
machines: [
{ id: "standard", label: "Standard", default: true },
{ id: "fast", label: "Fast" },
],
},
];
const state = new DraftPlaceState(
{ cloudProfiles, persistPreference } as unknown as DraftGatewayState,
@@ -129,9 +149,10 @@ describe("DraftPlaceState cloud machine selection", () => {
cloudPlacementExecutionMode: "worker-turn",
source: "model",
});
state.applyPendingPlacement({ agentId: "main", profileId: "aws" });
state.applyPendingPlacement({ agentId: "main", profileId: "aws", machineClass: "fast" });
state.restorePreferenceSelections();
expect(state.cloudProfileId).toBe("aws");
expect(state.machineClass).toBe("fast");
resolveRuntime.mockReturnValue({
id: "codex",
@@ -141,12 +162,19 @@ describe("DraftPlaceState cloud machine selection", () => {
});
state.restorePreferenceSelections();
expect(state.cloudProfileId).toBe("");
expect(state.worktree).toBe(false);
expect(persistPreference).toHaveBeenLastCalledWith(
"main",
"",
expect.objectContaining({ where: { kind: "local" }, worktree: false }),
);
if (compatible) {
expect(state.cloudProfileId).toBe("aws");
expect(state.machineClass).toBe("fast");
expect(state.worktree).toBe(true);
expect(persistPreference).not.toHaveBeenCalled();
} else {
expect(state.cloudProfileId).toBe("");
expect(state.worktree).toBe(false);
expect(persistPreference).toHaveBeenLastCalledWith(
"main",
"",
expect.objectContaining({ where: { kind: "local" }, worktree: false }),
);
}
});
});
ui/src/pages/new-session/model-control.runtime-placement.test.ts
+221
View File
@@ -0,0 +1,221 @@
import { describe, expect, it, vi } from "vitest";
import type { GatewayAgentRow } from "../../api/types.ts";
import type { DraftCloudProfile } from "./discovery.ts";
import { contextWith } from "./model-control.test-support.ts";
import { NewSessionModelControl } from "./model-control.ts";
describe("new-session model runtime placement", () => {
it.each([
{
name: "rejects a remote-exec runtime on a worker-turn profile",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
executionModes: ["worker-turn"] as const,
expected:
"The codex runtime cannot use this cloud worker. Choose a compatible cloud worker or run locally.",
},
{
name: "accepts a worker-turn runtime on a worker-turn profile",
runtime: {
id: "openclaw",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "worker-turn" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
executionModes: ["worker-turn"] as const,
expected: undefined,
},
{
name: "accepts a worker-turn runtime on a profile supporting both execution modes",
runtime: {
id: "openclaw",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "worker-turn" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
executionModes: ["worker-turn", "remote-exec"] as const,
expected: undefined,
},
{
name: "accepts a remote-exec runtime on the same worker-turn-primary profile",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
executionModes: ["worker-turn", "remote-exec"] as const,
expected: undefined,
},
{
name: "rejects a remote-exec runtime when the current profile supports only worker turns",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
executionModes: ["worker-turn"] as const,
expected:
"The codex runtime cannot use this cloud worker. Choose a compatible cloud worker or run locally.",
},
{
name: "does not fall back to the primary mode when an explicit mode set is empty",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: "remote-exec" as const,
executionModes: [] as const,
expected:
"The codex runtime cannot use this cloud worker. Choose a compatible cloud worker or run locally.",
},
{
name: "rejects a provider that advertises no placement execution mode",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: undefined,
expected:
"The codex runtime cannot use this cloud worker. Choose a compatible cloud worker or run locally.",
},
{
name: "retains the existing whole-runtime rejection",
runtime: { id: "acpx", cloudPlacementSupported: false, source: "model" as const },
executionMode: "worker-turn" as const,
expected: "The acpx runtime does not support cloud workers.",
},
])("$name", ({ runtime, executionMode, executionModes, expected }) => {
const profile: DraftCloudProfile = {
id: "aws",
providerId: "crabbox",
...(executionMode ? { executionMode } : {}),
...(executionModes === undefined ? {} : { executionModes }),
};
const control = new NewSessionModelControl(() => undefined);
vi.spyOn(control, "resolveAgentRuntime").mockReturnValue(runtime);
expect(control.cloudRuntimeUnsupportedReason(profile)).toBe(expected);
});
it.each([
{
name: "allows opted-in remote execution",
runtimeId: "codex",
devicePlacement: {
requiredNodeCommands: ["codex.exec-server.stdio.v1"],
consumesWorkerSlot: false,
},
},
{
name: "allows embedded execution",
runtimeId: "openclaw",
devicePlacement: { requiredNodeCommands: [], consumesWorkerSlot: true },
},
{ name: "rejects a cloud-only runtime", runtimeId: "cloud-only" },
{
name: "rejects a stale support flag without an owner requirement",
runtimeId: "stale",
devicePlacementSupported: true,
},
])("$name on paired devices", ({ runtimeId, devicePlacement, devicePlacementSupported }) => {
const control = new NewSessionModelControl(() => undefined);
vi.spyOn(control, "resolveAgentRuntime").mockReturnValue({
id: runtimeId,
cloudPlacementSupported: true,
devicePlacementSupported: devicePlacementSupported ?? Boolean(devicePlacement),
...(devicePlacement ? { devicePlacement } : {}),
source: "model",
});
expect(control.devicePlacementUnsupportedReason()).toBe(
devicePlacement ? undefined : "This runtime does not support paired devices",
);
});
it("uses model catalog runtime metadata for an explicit cloud target", async () => {
const { context, request } = contextWith([
{
id: "gpt-5.6-luna",
name: "GPT-5.6 Luna",
provider: "openai",
agentRuntime: { id: "codex", cloudPlacementSupported: true, source: "model" },
},
]);
const control = new NewSessionModelControl(() => undefined);
control.load(context, "main", true);
await vi.waitFor(() => expect(request).toHaveBeenCalledOnce());
await vi.waitFor(() => {
control.selected = "openai/gpt-5.6-luna";
expect(control.resolveAgentRuntime({ context })).toEqual({
id: "codex",
cloudPlacementSupported: true,
source: "model",
});
});
});
it("falls back to the selected agent runtime for its default model", () => {
const { context } = contextWith([]);
const agent = {
id: "main",
agentRuntime: { id: "claude-cli", cloudPlacementSupported: false, source: "agent" },
} satisfies GatewayAgentRow & {
agentRuntime: { id: string; cloudPlacementSupported: boolean; source: "agent" };
};
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ agent, context })).toEqual({
id: "claude-cli",
cloudPlacementSupported: false,
source: "agent",
});
});
it("falls back to the session defaults runtime capability", () => {
const { context } = contextWith([], "codex", [], true);
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ context })).toEqual({
id: "codex",
cloudPlacementSupported: true,
source: "defaults",
});
});
it.each(["auto", "default"])(
"leaves the %s runtime selector unresolved for server-side policy",
(runtime) => {
const { context } = contextWith([], runtime);
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ context })).toBeUndefined();
},
);
it("does not apply default runtime metadata to an explicit model", async () => {
const { context } = contextWith(
[{ id: "sonnet-4.6", name: "Sonnet 4.6", provider: "anthropic" }],
"codex",
);
const control = new NewSessionModelControl(() => undefined);
control.load(context, "main", true);
control.selected = "anthropic/sonnet-4.6";
await vi.waitFor(() => expect(control.resolveAgentRuntime({ context })).toBeUndefined());
});
});
ui/src/pages/new-session/model-control.test.ts
-161
View File
@@ -6,7 +6,6 @@ import {
rememberChatMetadata,
} from "../../lib/chat/chat-metadata-store.ts";
import { waitForFast } from "../../test-helpers/wait-for.ts";
import type { DraftCloudProfile } from "./discovery.ts";
import { contextWith, deferred, renderControl } from "./model-control.test-support.ts";
import { NewSessionModelControl } from "./model-control.ts";
@@ -15,94 +14,6 @@ afterEach(() => {
});
describe("new-session model runtime", () => {
it.each([
{
name: "rejects a remote-exec runtime on a worker-turn profile",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
expected:
"The codex runtime cannot use this cloud worker. Choose a compatible cloud worker or run locally.",
},
{
name: "accepts a worker-turn runtime on a worker-turn profile",
runtime: {
id: "openclaw",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "worker-turn" as const,
source: "model" as const,
},
executionMode: "worker-turn" as const,
expected: undefined,
},
{
name: "preserves an unknown provider mode",
runtime: {
id: "codex",
cloudPlacementSupported: true,
cloudPlacementExecutionMode: "remote-exec" as const,
source: "model" as const,
},
executionMode: undefined,
expected: undefined,
},
{
name: "retains the existing whole-runtime rejection",
runtime: { id: "acpx", cloudPlacementSupported: false, source: "model" as const },
executionMode: "worker-turn" as const,
expected: "The acpx runtime does not support cloud workers.",
},
])("$name", ({ runtime, executionMode, expected }) => {
const profile: DraftCloudProfile = {
id: "aws",
providerId: "crabbox",
...(executionMode ? { executionMode } : {}),
};
const control = new NewSessionModelControl(() => undefined);
vi.spyOn(control, "resolveAgentRuntime").mockReturnValue(runtime);
expect(control.cloudRuntimeUnsupportedReason(profile)).toBe(expected);
});
it.each([
{
name: "allows opted-in remote execution",
runtimeId: "codex",
devicePlacement: {
requiredNodeCommands: ["codex.exec-server.stdio.v1"],
consumesWorkerSlot: false,
},
},
{
name: "allows embedded execution",
runtimeId: "openclaw",
devicePlacement: { requiredNodeCommands: [], consumesWorkerSlot: true },
},
{ name: "rejects a cloud-only runtime", runtimeId: "cloud-only" },
{
name: "rejects a stale support flag without an owner requirement",
runtimeId: "stale",
devicePlacementSupported: true,
},
])("$name on paired devices", ({ runtimeId, devicePlacement, devicePlacementSupported }) => {
const control = new NewSessionModelControl(() => undefined);
vi.spyOn(control, "resolveAgentRuntime").mockReturnValue({
id: runtimeId,
cloudPlacementSupported: true,
devicePlacementSupported: devicePlacementSupported ?? Boolean(devicePlacement),
...(devicePlacement ? { devicePlacement } : {}),
source: "model",
});
expect(control.devicePlacementUnsupportedReason()).toBe(
devicePlacement ? undefined : "This runtime does not support paired devices",
);
});
it("keeps CLI agents hidden and undiscovered while the Labs gate is off", async () => {
const { context, request } = contextWith([
{ id: "gpt-5.6-luna", name: "GPT-5.6 Luna", provider: "openai" },
@@ -957,76 +868,4 @@ describe("new-session model runtime", () => {
thinkingLevel: "",
});
});
it("uses model catalog runtime metadata for an explicit cloud target", async () => {
const { context, request } = contextWith([
{
id: "gpt-5.6-luna",
name: "GPT-5.6 Luna",
provider: "openai",
agentRuntime: { id: "codex", cloudPlacementSupported: true, source: "model" },
},
]);
const control = new NewSessionModelControl(() => undefined);
control.load(context, "main", true);
await vi.waitFor(() => expect(request).toHaveBeenCalledOnce());
await vi.waitFor(() => {
control.selected = "openai/gpt-5.6-luna";
expect(control.resolveAgentRuntime({ context })).toEqual({
id: "codex",
cloudPlacementSupported: true,
source: "model",
});
});
});
it("falls back to the selected agent runtime for its default model", () => {
const { context } = contextWith([]);
const agent = {
id: "main",
agentRuntime: { id: "claude-cli", cloudPlacementSupported: false, source: "agent" },
} satisfies GatewayAgentRow & {
agentRuntime: { id: string; cloudPlacementSupported: boolean; source: "agent" };
};
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ agent, context })).toEqual({
id: "claude-cli",
cloudPlacementSupported: false,
source: "agent",
});
});
it("falls back to the session defaults runtime capability", () => {
const { context } = contextWith([], "codex", [], true);
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ context })).toEqual({
id: "codex",
cloudPlacementSupported: true,
source: "defaults",
});
});
it.each(["auto", "default"])(
"leaves the %s runtime selector unresolved for server-side policy",
(runtime) => {
const { context } = contextWith([], runtime);
const control = new NewSessionModelControl(() => undefined);
expect(control.resolveAgentRuntime({ context })).toBeUndefined();
},
);
it("does not apply default runtime metadata to an explicit model", async () => {
const { context } = contextWith(
[{ id: "sonnet-4.6", name: "Sonnet 4.6", provider: "anthropic" }],
"codex",
);
const control = new NewSessionModelControl(() => undefined);
control.load(context, "main", true);
control.selected = "anthropic/sonnet-4.6";
await vi.waitFor(() => expect(control.resolveAgentRuntime({ context })).toBeUndefined());
});
});
ui/src/pages/new-session/model-control.ts
+3 -3
View File
@@ -24,7 +24,7 @@ import {
type ChatModelCatalogState,
} from "../chat/components/chat-model-controls.ts";
import type { ChatModelPickerTargetGroup } from "../chat/components/chat-model-picker-options.ts";
import type { DraftCloudProfile } from "./discovery.ts";
import { draftCloudProfileSupportsExecutionMode, type DraftCloudProfile } from "./discovery.ts";
import type { NewSessionPreference } from "./preferences.ts";
type NewSessionMetadataClient = NonNullable<ApplicationContext["gateway"]["snapshot"]["client"]>;
@@ -584,9 +584,9 @@ export class NewSessionModelControl {
return t("newSession.cloudRuntimeUnsupported", { runtime: runtime.id });
}
return runtime &&
profile?.executionMode &&
profile &&
runtime.cloudPlacementExecutionMode &&
profile.executionMode !== runtime.cloudPlacementExecutionMode
!draftCloudProfileSupportsExecutionMode(profile, runtime.cloudPlacementExecutionMode)
? t("newSession.cloudProfileRuntimeUnsupported", { runtime: runtime.id })
: undefined;
}