diff --git a/docs/docs.json b/docs/docs.json index fb276f5aa05f..939310ace144 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1684,6 +1684,7 @@ "gateway/gateway-lock", "gateway/background-process", "gateway/restart-recovery", + "gateway/cloud-workers", "gateway/multiple-gateways", "gateway/multi-tenant-hosting" ] diff --git a/docs/gateway/cloud-workers.md b/docs/gateway/cloud-workers.md index 2dbce0ebc092..4b8288ec69f8 100644 --- a/docs/gateway/cloud-workers.md +++ b/docs/gateway/cloud-workers.md @@ -4,6 +4,7 @@ title: "Cloud Workers" sidebarTitle: "Cloud Workers" read_when: "You want agent sessions to run on ephemeral cloud machines instead of the Gateway host, or you are configuring cloudWorkers profiles." status: active +doc-schema-version: 1 --- Cloud workers let a session run its agent loop on a throwaway cloud machine while everything about the session stays where it always was: visible in the sidebar, streaming live, with the transcript owned by the Gateway. The Gateway leases a box, installs a pinned copy of OpenClaw on it, syncs the session's workspace over, and hands the turn loop to a restricted `openclaw worker` process. Model calls are proxied back through the Gateway, so provider credentials never leave your machine, and prompt caching keeps working because the provider sees one continuous stream. @@ -28,11 +29,49 @@ The box needs no inbound ports except `sshd`: the Gateway connects out via pinne ## Requirements -- A worker provider plugin. The bundled `crabbox` plugin drives the [Crabbox](https://github.com/openclaw/crabbox) CLI, which brokers leases across cloud backends (AWS, Hetzner, and others). The `crabbox` binary must be on `PATH` (or set `settings.binary`) with provider credentials already configured. AWS admission requires Crabbox 0.38.1 or newer. +- A worker provider plugin. The bundled `crabbox` plugin drives the [Crabbox](https://github.com/openclaw/crabbox) CLI, which brokers leases across cloud backends (AWS, Hetzner, and others). Install the `crabbox` binary for the operating-system user that runs the Gateway and put it on that user's `PATH`, or set `settings.binary` to its absolute path. AWS admission requires Crabbox 0.38.1 or newer. - For Crabbox AWS workers, the effective `aws.instanceProfile` must be empty. The provider checks `crabbox config show --json` before allocation, then requires `crabbox inspect --json` to report `providerMetadata.instanceProfileAttached: false` from EC2 `DescribeInstances`. Leases with an instance role or without authoritative metadata are stopped and rejected. - Node.js on the leased machine. Bare cloud images usually lack it — install it in the profile's `setup` command. - A session with a session-owned managed worktree (create one with `worktree: true`). Dispatch moves that worktree's contents; plain directories sync as a manifest mirror. +### Coordinator-backed Crabbox + +In managed mode, the Crabbox coordinator owns the cloud-provider credentials and provisions AWS on the Gateway user's behalf. Local AWS keys are not required. Authenticate interactively, then verify the stored coordinator and provider state: + +Before provisioning, determine the Gateway host's outbound IPv4: + +```bash +curl -fsS https://checkip.amazonaws.com +``` + +Add that address as a `/32` to Crabbox's own configuration. For example, if the command prints `203.0.113.10`: + +```yaml +aws: + sshCIDRs: + - 203.0.113.10/32 +``` + +Direct SSH originates from the Gateway host, while the coordinator API may see a reverse-proxy or request-source address. Explicit pinning keeps later Crabbox security-group reconciliation from replacing the actual SSH caller with that API-facing address. + +```bash +crabbox login --url --provider aws +crabbox config show --json +crabbox whoami --json +crabbox doctor --provider aws --json +``` + +Before provisioning, confirm `crabbox config show --json` reports the expected `/32` under `aws.sshCIDRs`, then review `crabbox doctor --provider aws --json` for provider-readiness failures. `doctor` is non-mutating: it checks the coordinator, broker identity, local tools, and AWS provider readiness without creating or changing a lease. Trusted automation can pipe an approved coordinator token through stdin instead of placing it on the command line: + +```bash +printf '%s' "$CRABBOX_COORDINATOR_TOKEN" | crabbox login \ + --url \ + --provider aws \ + --token-stdin +``` + +Keep the token out of repository config and shell arguments. + ## Configuration Add a profile under `cloudWorkers.profiles` in `openclaw.json`: @@ -64,27 +103,54 @@ Profile fields: | `provider` | Worker provider id registered by a plugin (`crabbox` for the bundled plugin). | | `install` | `bundle` (default) ships the running Gateway's build; `npm` installs the exact released Gateway version with pinned integrity. `npm` requires the Gateway to run from a packaged release. | | `settings` | Provider-owned JSON. For crabbox: `provider` (backend), `class` (machine class), `ttl`, `idleTimeout` (Go durations), optional `setup` and absolute `binary` path. OpenClaw forces public SSH and disables managed Tailscale for these leases. | -| `lifetime` | Optional stored policy (`idleTimeoutMinutes`, `maxLifetimeMinutes`). | + +Crabbox inspect reports a primary SSH port and may advertise ordered fallback ports. OpenClaw persists that order across Gateway restarts. Its shared pinned SSH transport uses the current candidate first and retries the remaining advertised ports when a fresh authenticated connection fails at the SSH transport layer. It never invents an unadvertised port. If your network policy pins SSH ingress, allow at least one advertised Crabbox candidate. ### The setup command -`settings.setup` runs on the leased box after it is SSH-ready and before OpenClaw is installed. It runs on **every** provision attempt (including replays after an interrupted dispatch), so it must be idempotent — guard installs with a `command -v`/`test -x` check as in the example. If setup fails, the provider stops the lease and the dispatch fails closed; no half-configured box is left running. +`settings.setup` runs on the leased box after it is SSH-ready and before OpenClaw is installed. After setup succeeds, OpenClaw performs a fresh Crabbox inspect and waits for SSH readiness again before bootstrap, because setup may restart SSH. It runs on **every** provision attempt (including replays after an interrupted dispatch), so it must be idempotent — guard installs with a `command -v`/`test -x` check as in the example. If setup fails, the provider stops the lease and the dispatch fails closed; no half-configured box is left running. ### Install channels - **`bundle`** packs the running Gateway's `dist`, a pruned `package.json`, and any workspace packages the build references, all covered by a content hash. The box verifies the pristine bundle against that hash, then installs production npm dependencies (scripts disabled). This is how you run a dev build on a worker. - **`npm`** proves the release exists on the public registry, pins its SHA-512 integrity, and installs `openclaw@` matching the Gateway exactly. +### Verify the profile + +Validate before restarting the Gateway: + +```bash +openclaw config validate --json +openclaw plugins inspect crabbox --runtime --json +``` + +Changes under `cloudWorkers.profiles` require a Gateway restart. The default `gateway.reload.mode: "hybrid"` watches the config and performs that restart automatically; with reload watching disabled, run `openclaw gateway restart`. + +After the Gateway is back, prove the profile is advertised and compare it with Crabbox's read-only lease inventory: + +```bash +openclaw gateway call environments.list --params '{}' +crabbox list --provider aws --json +``` + +The `environments.list` response must include the configured id under `profiles`. `crabbox list` is non-mutating. By contrast, `crabbox warmup` provisions a lease, and `crabbox stop` or `crabbox release` tears one down; use those mutating commands only when you intend to create or destroy cloud resources. + ## Dispatching a session -In the Control UI, open **New Session**, choose an agent whose configured runtime is OpenClaw, select a configured **Cloud · profile** target from the **Where** menu, and start the task. Cloud selection enables the required managed worktree automatically; the Gateway creates the session, finishes dispatch, and only then sends the first turn. The server badge in the session sidebar shows the durable placement state. Cloud targets are not offered for external CLI session catalogs. +In the Control UI, open **New Session** and use the unified **Place** picker to choose both the working folder and a **Cloud · profile** destination. A cloud destination appears only when all three eligibility gates pass: + +1. The connected operator has `operator.admin` scope. +2. `environments.list` advertises at least one configured profile. +3. The selected Gateway folder is a Git checkout that can use a managed worktree. + +Cloud selection enables that worktree automatically. The Gateway creates the session, finishes dispatch, and only then sends the first turn. The server badge in the session sidebar shows the durable placement state. + +Cloud workers run the OpenClaw agent runtime. Models mapped to an external runtime such as Codex or Claude CLI are disabled in the picker; select a direct model that resolves to the OpenClaw runtime. Cloud targets are not offered for external CLI session catalogs. The equivalent RPC flow is: Create a session with a managed worktree, then dispatch it (the RPC requires `operator.admin` and only exists when profiles are configured): -Cloud workers run the OpenClaw agent runtime. Choose an `openai/*` or other model that resolves to that runtime; sessions configured for an external CLI runtime such as `claude-cli` cannot dispatch. - ```bash openclaw gateway call sessions.create \ --params '{"key":"agent:main:big-refactor","worktree":true,"cwd":"/path/to/repo","worktreeName":"big-refactor"}' @@ -128,16 +194,18 @@ Placement moves through a durable state machine (`local → requested → provis ## Troubleshooting -- **`sessions.dispatch` is an unknown method** — no `cloudWorkers.profiles` are configured, or the caller lacks `operator.admin`. -- **"Cloud worker turns require the OpenClaw runtime"** — choose a model whose configured runtime is OpenClaw. External CLI runtimes such as `claude-cli` do not support worker inference. +- **No cloud profile is advertised** — run `openclaw gateway call environments.list --params '{}'` as an admin. If the response has no `profiles`, validate `cloudWorkers.profiles`, inspect the provider plugin, and restart the Gateway. This is a configuration or provider-activation problem, not an authorization result. +- **Cloud destinations are hidden or an RPC is denied** — the connected operator lacks `operator.admin`. Reconnect with admin scope; configuring a profile does not grant that scope. +- **"Cloud worker turns require the OpenClaw runtime"** — choose a direct model whose configured runtime is OpenClaw. Models mapped to external Codex or Claude CLI runtimes do not support worker inference. - **"Worker bootstrap requires Node.js on the leased host"** — add a Node install to `settings.setup` (see above). - **AWS instance-role attestation fails** — clear `aws.instanceProfile` (and `CRABBOX_AWS_INSTANCE_PROFILE`, if set). Install Crabbox 0.38.1 or newer; older binaries do not expose the authoritative `providerMetadata.instanceProfileAttached` contract required for AWS admission. -- **Dispatch fails with a provider error** — the placement record and `environments.list` keep the last error, including the setup/bootstrap stderr tail. Boxes are destroyed on failure, so that tail is the primary forensic. +- **Dispatch fails with a provider or bootstrap error** — `environments.list` intentionally omits internal `lastError`. Inspect the session with `sessions.describe`; a failed placement may expose a bounded `recoveryError`. When deeper diagnosis is necessary, an operator on the Gateway host can inspect the durable worker state read-only. Do not edit the state database to bypass lifecycle fencing. +- **No SSH candidate is reachable** — compare the Gateway host's current outbound IPv4 with Crabbox's effective `aws.sshCIDRs` in `crabbox config show --json`. If the matching `/32` is absent, correct Crabbox's configuration and rerun `crabbox doctor --provider aws --json` before retrying; the coordinator's reverse-proxy or request-source address is not necessarily the Gateway's direct SSH source. Then ensure the Gateway's outbound route and the worker ingress policy permit at least one advertised candidate. OpenClaw already tries Crabbox's ordered ports with the same identity and pinned host key. - **Client timeout while dispatching** — `openclaw gateway call` defaults to a 10s timeout; pass `--timeout` generously (dispatch keeps running server-side either way, and a retry while provisioning is rejected with `session cannot dispatch from placement provisioning`). - **Worker reclaimed after upgrading from a 2026.7.2 beta** — those betas used the older worker launch contract. On restart, OpenClaw destroys an idle incompatible worker, keeps the session and workspace, marks the placement reclaimed, and provisions a current worker on the next dispatch or turn. A beta worker interrupted while still starting is marked failed after cleanup; retry the dispatch to provision it with the current contract. - **Cloud workspace conflict notice** — the turn completed and kept the local version of each listed path. Use the staged-ref commands in the notice to inspect or take the cloud version; no retry is required for the non-conflicting changes, which are already applied. - **“The previous cloud turn's workspace result is still reconciling”** — the Gateway waited briefly for the prior result's durable fence and could not acquire the session claim. Wait for reconciliation to finish, then retry the turn; restarting the Gateway is safe because recovery preserves staged results before reclaiming a dead worker. -- **Lease housekeeping** — `crabbox list --provider ` shows live leases; `crabbox stop --provider --id ` releases one manually. Idle leases expire on the profile's `idleTimeout`. +- **Lease housekeeping** — `crabbox list --provider --json` is a read-only inventory. `crabbox stop --provider --id ` and `crabbox release --provider --id ` are destructive and release a lease manually. Idle leases expire on the profile's `idleTimeout`. ## Related diff --git a/docs/gateway/configuration-reference.md b/docs/gateway/configuration-reference.md index cb0f4f6a8dbd..7ae561619e0c 100644 --- a/docs/gateway/configuration-reference.md +++ b/docs/gateway/configuration-reference.md @@ -4,6 +4,7 @@ title: "Configuration reference" read_when: - You need exact field-level config semantics or defaults - You are validating channel, model, gateway, or tool config blocks +doc-schema-version: 1 --- Field-level reference for `~/.openclaw/openclaw.json`: keys, defaults, and links to deeper subsystem pages. For task-oriented setup guidance, see [Configuration](/gateway/configuration). Channel- and plugin-owned command catalogs and deep memory/QMD knobs live on their own pages, not here. @@ -813,10 +814,6 @@ The bundled `crabbox` provider provisions an SSH-capable lease through the local // Optional absolute path. Default: sibling ../crabbox/bin/crabbox, then PATH. binary: "/usr/local/bin/crabbox", }, - lifetime: { - idleTimeoutMinutes: 60, - maxLifetimeMinutes: 1440, - }, }, }, }, @@ -825,11 +822,15 @@ The bundled `crabbox` provider provisions an SSH-capable lease through the local - `settings.provider` (required): Crabbox backend passed through `--provider`. Use a backend whose inspect output includes an SSH endpoint; `aws` selects the direct AWS backend. - `settings.class` (required): Crabbox machine class passed to `--class`. -- `settings.ttl` and `settings.idleTimeout` (required): positive Go duration strings passed to `--ttl` and `--idle-timeout`. These provider-side failsafes are distinct from OpenClaw's stored `lifetime` policy below. +- `settings.ttl` and `settings.idleTimeout` (required): positive Go duration strings passed to `--ttl` and `--idle-timeout` as provider-side failsafes. - `settings.binary`: optional absolute Crabbox executable path. Without it, OpenClaw checks the sibling Crabbox checkout, then executable entries on `PATH`, and finally invokes `crabbox` so a missing CLI remains a visible provider error. Unknown settings are rejected. Crabbox credentials and backend-specific account configuration remain owned by Crabbox; do not place them in `settings`. OpenClaw invokes only the local CLI and makes no provider network calls from this plugin. Provisioning always passes `--keep=true`; OpenClaw owns the external lifecycle and destroys the lease with `crabbox stop`. +For coordinator-backed AWS, Crabbox's own `aws.sshCIDRs` should include the Gateway host's outbound IPv4 as a `/32`. Verify it with `crabbox config show --json` and `crabbox doctor --provider aws --json` before provisioning; do not place this provider-ingress setting in OpenClaw `settings`. See [Coordinator-backed Crabbox](/gateway/cloud-workers#coordinator-backed-crabbox). + +Crabbox inspect may expose ordered `sshFallbackPorts` in addition to its primary `sshPort`. OpenClaw persists the advertised order across Gateway restarts. The shared pinned SSH transport uses the current candidate first and retries the remaining advertised ports only when a fresh authenticated SSH or workspace-transfer connection fails at the transport layer. Network policies must allow at least one advertised candidate. + OpenClaw resolves Crabbox's lease-local `sshKey` path through the provider-owned secret resolver and pins the authoritative `sshHostKey` returned by `crabbox inspect --json`. AWS admission also requires `providerMetadata.instanceProfileAttached`. Install Crabbox 0.38.1 or newer for this closed inspection contract. @@ -853,10 +854,6 @@ Unknown settings are rejected. Crabbox credentials and backend-specific account id: "OPENCLAW_WORKER_SSH_KEY", }, }, - lifetime: { - idleTimeoutMinutes: 60, - maxLifetimeMinutes: 1440, - }, }, }, }, @@ -868,16 +865,14 @@ Unknown settings are rejected. Crabbox credentials and backend-specific account - `install`: worker installation method. `"bundle"` (default) transfers a content-hashed bundle of the gateway's installed build and supports released, development, and unreleased versions. `"npm"` is an opt-in optimization for an unmodified packaged release; it installs `openclaw@` from the public npm registry and never installs `latest`. - Bundled provider plugins are selected automatically when configured, but explicit disables and `plugins.allow` still apply. Include the provider id (for example, `crabbox`) when an allowlist is configured. External provider plugins must also be installed and explicitly enabled. - `settings`: provider-owned bounded JSON. The selected plugin defines and validates its keys; use [SecretRef objects](/gateway/secrets) for secret-bearing values. The static SSH provider requires `host`, `user`, `hostKey`, and `keyRef`; `port` defaults to `22`. `hostKey` must be one OpenSSH public host-key line (`algorithm base64`) obtained from the known host or another trusted channel, with no options prefix. -- `lifetime.idleTimeoutMinutes`: positive integer minutes stored for later idle-reclamation policy. -- `lifetime.maxLifetimeMinutes`: positive integer minutes stored for later lifecycle policy. A supported Node runtime (22.22.3+, 24.15+, or 25.9+) with WAL-reset-safe SQLite must already be installed on the worker. The opt-in `"npm"` method also requires `npm` and outbound HTTPS access to the public npm registry. Networked toolchain setup is provider policy; bootstrap reports an actionable error instead of installing toolchains itself. -This foundation installs and verifies the gateway build and provides tunnel start/stop lifecycle, but it does not launch the general OpenClaw CLI. The self-contained worker entry and loop land in the next cloud-worker milestone. +The Gateway installs and verifies the selected OpenClaw build, launches the self-contained worker loop, proxies model inference through the Gateway, and reconciles the session workspace and transcript through the durable placement lifecycle. -Each durable environment record retains its validated provider settings, resolved install method, and lifetime policy in a creation-time profile snapshot. Changing or removing a named profile affects new creates; existing records continue lifecycle reconciliation with that snapshot, provided the owning plugin remains available. +Each durable environment record retains its validated provider settings and resolved install method in a creation-time profile snapshot. Changing or removing a named profile affects new creates; existing records continue lifecycle reconciliation with that snapshot, provided the owning plugin remains available. -Lifetime values are data only in the first cloud-worker release; automatic enforcement lands with later lifecycle work. Profile changes require a gateway restart. +Profile changes require a Gateway restart. With the default `gateway.reload.mode: "hybrid"`, the config watcher performs the restart automatically; `"off"` mode requires a manual restart. The `static-ssh` provider is a source-tree QA Lab development harness and is excluded from packaged distributions. A worker running on its shared host can read unrelated host data, so do not use this provider as a production isolation boundary. diff --git a/docs/plugins/manifest.md b/docs/plugins/manifest.md index 9c513bfdf4de..01daea77929a 100644 --- a/docs/plugins/manifest.md +++ b/docs/plugins/manifest.md @@ -710,7 +710,7 @@ Provider plugins that implement both `resolveUsageAuth` and `fetchUsageSnapshot` General embedding providers should declare `contracts.embeddingProviders` for each adapter registered with `api.registerEmbeddingProvider(...)`. Use the general contract for reusable vector generation, including providers consumed by memory search. `contracts.memoryEmbeddingProviders` is deprecated memory-specific compatibility and remains only while existing providers migrate to the generic embedding provider seam. -Worker providers must declare each `api.registerWorkerProvider(...)` id in `contracts.workerProviders`. Core persists durable intent before calling `provision`; providers validate their settings before external allocation, and repeated calls with the same operation id must adopt the same lease. Core also persists that validated settings snapshot and passes it with `leaseId` to `inspect({ leaseId, profile })` and `destroy({ leaseId, profile })`, including after the named profile is changed or removed. Destruction is idempotent, inspection returns the closed `active` / `destroyed` / `unknown` status union, and SSH private-key material is referenced only through `SecretRef`. Provisioned SSH endpoints must also include a public `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment, so core can pin the host before connecting. Providers that mint dynamic identity refs may implement authoritative `resolveSshIdentity({ leaseId, profile, keyRef })`; providers without it use core's generic secret resolver. An authoritative `unknown` orphans an active local record; after a persisted destroy request it confirms teardown. +Worker providers must declare each `api.registerWorkerProvider(...)` id in `contracts.workerProviders`. Core persists durable intent before calling `provision`; providers validate their settings before external allocation, and repeated calls with the same operation id must adopt the same lease. Core also persists that validated settings snapshot and passes it with `leaseId` to `inspect({ leaseId, profile })` and `destroy({ leaseId, profile })`, including after the named profile is changed or removed. Destruction is idempotent, inspection returns the closed `active` / `destroyed` / `unknown` status union, and SSH private-key material is referenced only through `SecretRef`. Provisioned SSH endpoints must also include a public `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment, so core can pin the host before connecting. They may include up to 10 ordered, unique `fallbackPorts`, excluding the primary `port`; core persists and retries only those advertised candidates. Providers that mint dynamic identity refs may implement authoritative `resolveSshIdentity({ leaseId, profile, keyRef })`; providers without it use core's generic secret resolver. An authoritative `unknown` orphans an active local record; after a persisted destroy request it confirms teardown. `contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth. An entitled route remains reachable while Gateway root-work admission is closed only when it also declares `auth: "gateway"` and the route-specific `gatewayRuntimeScopeSurface: "trusted-operator"`; ordinary sibling routes from the same plugin remain behind the admission boundary. This keeps suspension status and resume reachable without granting the whole plugin an admission bypass. Keep parsing and response shaping bounded outside dispatch; substantive or mutating work must go through Gateway method dispatch, which owns admission and scope enforcement. diff --git a/docs/plugins/sdk-overview.md b/docs/plugins/sdk-overview.md index d5aed43d7feb..3d72c6be50fe 100644 --- a/docs/plugins/sdk-overview.md +++ b/docs/plugins/sdk-overview.md @@ -119,7 +119,7 @@ and external URLs. Registering another provider replaces the current provider. Worker providers must also declare their id in `contracts.workerProviders`. Core persists durable intent before `provision(profile, operationId)`. Providers validate settings before external allocation and throw `WorkerProviderError` for permanent profile rejection. `provision` must adopt the same lease when the operation id repeats. -Core persists the validated profile settings with the lease and supplies that snapshot to `destroy({ leaseId, profile })`, which must be idempotent, and `inspect({ leaseId, profile })`, which returns `active`, `destroyed`, or `unknown`. This lets providers route lifecycle calls after a gateway restart or named-profile removal. SSH endpoints use a `SecretRef` for `keyRef`, never inline key material, and include a `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment. Core pins `hostKey` and never trusts a key from the first connection. A provider that mints a dynamic `keyRef` can implement `resolveSshIdentity({ leaseId, profile, keyRef })`; when present, that resolver is authoritative, while providers without it use the configured generic secret resolver. +Core persists the validated profile settings with the lease and supplies that snapshot to `destroy({ leaseId, profile })`, which must be idempotent, and `inspect({ leaseId, profile })`, which returns `active`, `destroyed`, or `unknown`. This lets providers route lifecycle calls after a gateway restart or named-profile removal. SSH endpoints use a `SecretRef` for `keyRef`, never inline key material, and include a `hostKey` from trusted provisioning output as exactly `algorithm base64`, without a hostname or comment. Core pins `hostKey` and never trusts a key from the first connection. Providers may also return up to 10 ordered, unique `fallbackPorts` (integer ports from 1 through 65535, excluding the primary `port`); core validates and persists those advertised candidates for its shared pinned SSH transport. A provider that mints a dynamic `keyRef` can implement `resolveSshIdentity({ leaseId, profile, keyRef })`; when present, that resolver is authoritative, while providers without it use the configured generic secret resolver. Providers with renewable leases can also implement `renew(leaseId)`. `inspect` must throw on transient or indeterminate failures; return `unknown` only for authoritative absence. Core marks an active local record orphaned, or treats the absence as teardown completion after a persisted destroy request. diff --git a/extensions/crabbox/src/crabbox-worker-inspect.test.ts b/extensions/crabbox/src/crabbox-worker-inspect.test.ts new file mode 100644 index 000000000000..0787d16f7465 --- /dev/null +++ b/extensions/crabbox/src/crabbox-worker-inspect.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from "vitest"; +import { parseInspectJson } from "./crabbox-worker-inspect.js"; + +function inspectJson(overrides: Record = {}): string { + return JSON.stringify({ id: "cbx_012345abcdef", state: "running", ...overrides }); +} + +describe("Crabbox worker inspect", () => { + it("defaults missing SSH fallback ports to an empty list", () => { + expect(parseInspectJson(inspectJson()).sshFallbackPorts).toStrictEqual([]); + }); + + it("normalizes SSH fallback ports in stable order without primary duplicates", () => { + expect( + parseInspectJson( + inspectJson({ + sshPort: "2222", + sshFallbackPorts: [22, "2200", "22", 2222, "2200"], + }), + ).sshFallbackPorts, + ).toStrictEqual([22, 2200]); + }); + + it.each([null, "22", [""], ["22x"], [0], [65_536], [22.5], [null]])( + "rejects invalid SSH fallback ports %#", + (sshFallbackPorts) => { + expect(() => parseInspectJson(inspectJson({ sshFallbackPorts }))).toThrow( + "invalid sshFallbackPorts", + ); + }, + ); + + it("accepts at most ten normalized SSH fallback ports", () => { + const tenPorts = Array.from({ length: 10 }, (_, index) => 2300 + index); + expect( + parseInspectJson(inspectJson({ sshPort: 2222, sshFallbackPorts: tenPorts })).sshFallbackPorts, + ).toEqual(tenPorts); + expect(() => + parseInspectJson( + inspectJson({ + sshPort: 2222, + sshFallbackPorts: Array.from({ length: 11 }, (_, index) => 2400 + index), + }), + ), + ).toThrow("invalid sshFallbackPorts: maximum 10"); + }); +}); diff --git a/extensions/crabbox/src/crabbox-worker-inspect.ts b/extensions/crabbox/src/crabbox-worker-inspect.ts index 60d5df7de419..19acfc2686aa 100644 --- a/extensions/crabbox/src/crabbox-worker-inspect.ts +++ b/extensions/crabbox/src/crabbox-worker-inspect.ts @@ -1,5 +1,7 @@ import { nonEmptyString } from "./crabbox-worker-profile.js"; +const MAX_SSH_FALLBACK_PORTS = 10; + type CrabboxInspect = { host?: unknown; id?: unknown; @@ -8,6 +10,7 @@ type CrabboxInspect = { sshHost?: unknown; sshHostKey?: unknown; sshKey?: unknown; + sshFallbackPorts?: unknown; sshPort?: unknown; sshUser?: unknown; state?: unknown; @@ -21,6 +24,7 @@ export type ParsedInspect = { ready?: boolean; sshHostKey?: string; sshKey?: string; + sshFallbackPorts: number[]; sshPort?: number; sshUser?: string; state: string; @@ -79,10 +83,12 @@ export function parseInspectJson(stdout: string): ParsedInspect { const sshHostKey = inspectString(value.sshHostKey, "sshHostKey"); const sshKey = inspectString(value.sshKey, "sshKey"); const sshPort = inspectPort(value.sshPort); + const sshFallbackPorts = inspectFallbackPorts(value.sshFallbackPorts, sshPort); return { id, state, tailscaleEnabled, + sshFallbackPorts, ...(awsInstanceProfileAttached !== undefined ? { awsInstanceProfileAttached } : {}), ...(host ? { host } : {}), ...(sshUser ? { sshUser } : {}), @@ -107,12 +113,38 @@ function inspectPort(value: unknown): number | undefined { if (value === undefined || value === "") { return undefined; } + return inspectRequiredPort(value, "sshPort"); +} + +function inspectFallbackPorts(value: unknown, primaryPort: number | undefined): number[] { + if (value === undefined) { + return []; + } + if (!Array.isArray(value)) { + throw new Error("Crabbox inspect returned invalid sshFallbackPorts"); + } + const seen = new Set(primaryPort === undefined ? [] : [primaryPort]); + const ports: number[] = []; + for (const entry of value) { + const port = inspectRequiredPort(entry, "sshFallbackPorts"); + if (!seen.has(port)) { + seen.add(port); + ports.push(port); + } + } + if (ports.length > MAX_SSH_FALLBACK_PORTS) { + throw new Error("Crabbox inspect returned invalid sshFallbackPorts: maximum 10"); + } + return ports; +} + +function inspectRequiredPort(value: unknown, field: "sshPort" | "sshFallbackPorts"): number { if (typeof value !== "number" && (typeof value !== "string" || !/^\d+$/u.test(value))) { - throw new Error("Crabbox inspect returned an invalid sshPort"); + throw new Error(`Crabbox inspect returned an invalid ${field}`); } const port = typeof value === "number" ? value : Number(value); if (!Number.isInteger(port) || port < 1 || port > 65_535) { - throw new Error("Crabbox inspect returned an invalid sshPort"); + throw new Error(`Crabbox inspect returned an invalid ${field}`); } return port; } diff --git a/extensions/crabbox/src/crabbox-worker-provider.test.ts b/extensions/crabbox/src/crabbox-worker-provider.test.ts index c55819d5d58e..a4da00d54d1d 100644 --- a/extensions/crabbox/src/crabbox-worker-provider.test.ts +++ b/extensions/crabbox/src/crabbox-worker-provider.test.ts @@ -99,10 +99,14 @@ describe("Crabbox worker provider", () => { return commandResult({ stdout: `leased ${LEASE_ID} slug=test\n` }); } if (argv.includes(LEASE_ID)) { - return commandResult({ stdout: inspectJson({ sshHostKey: HOST_KEY }) }); + return commandResult({ + stdout: inspectJson({ sshFallbackPorts: [22], sshHostKey: HOST_KEY }), + }); } return warmed - ? commandResult({ stdout: inspectJson({ sshHostKey: HOST_KEY }) }) + ? commandResult({ + stdout: inspectJson({ sshFallbackPorts: [22], sshHostKey: HOST_KEY }), + }) : commandResult({ code: 4, stderr: `lease/server not found: ${argv.at(-2)}` }); }); @@ -111,6 +115,7 @@ describe("Crabbox worker provider", () => { ssh: { host: "worker.example.test", port: 2222, + fallbackPorts: [22], user: "openclaw", hostKey: HOST_KEY, keyRef: { @@ -122,6 +127,18 @@ describe("Crabbox worker provider", () => { }); }); + it("preserves ordered SSH fallback ports advertised by Crabbox", async () => { + const provider = providerWithRunner(async () => + commandResult({ + stdout: inspectJson({ sshFallbackPorts: [22, 2200], sshHostKey: HOST_KEY }), + }), + ); + + await expect(provider.provision(PROFILE, "provision:fallback-port")).resolves.toMatchObject({ + ssh: { port: 2222, fallbackPorts: [22, 2200] }, + }); + }); + it("runs the profile setup command on the ready lease and keeps it", async () => { const calls: string[][] = []; let warmed = false; @@ -162,6 +179,155 @@ describe("Crabbox worker provider", () => { ]); }); + it.each([ + { + kind: "newly warmed", + replay: false, + expectedCommands: ["inspect", "warmup", "inspect", "run", "inspect", "inspect"], + }, + { + kind: "replayed", + replay: true, + expectedCommands: ["inspect", "run", "inspect", "inspect"], + }, + ])( + "waits for post-setup SSH readiness on a $kind lease and returns its final endpoint", + async ({ replay, expectedCommands }) => { + const calls: string[][] = []; + let warmed = false; + let leaseInspections = 0; + let resolveFinalInspect!: (result: SpawnResult) => void; + let markFinalInspectStarted!: () => void; + const finalInspect = new Promise((resolve) => { + resolveFinalInspect = resolve; + }); + const finalInspectStarted = new Promise((resolve) => { + markFinalInspectStarted = resolve; + }); + const provider = providerWithRunner(async (argv) => { + calls.push(argv); + if (argv[1] === "warmup") { + warmed = true; + return commandResult({ stdout: `leased ${LEASE_ID} slug=test\n` }); + } + if (argv[1] === "run") { + return commandResult(); + } + const id = argv[argv.indexOf("--id") + 1]; + if (!replay && !warmed && id !== LEASE_ID) { + return commandResult({ code: 4, stderr: `lease/server not found: ${id}` }); + } + leaseInspections += 1; + if (leaseInspections === 1) { + return commandResult({ + stdout: inspectJson({ + sshFallbackPorts: [22], + sshHost: "before-setup.example.test", + sshHostKey: HOST_KEY, + }), + }); + } + if (leaseInspections === 2) { + return commandResult({ + stdout: inspectJson({ + ready: false, + sshFallbackPorts: [22], + sshHost: "restarting.example.test", + sshHostKey: HOST_KEY, + }), + }); + } + markFinalInspectStarted(); + return await finalInspect; + }); + + const provision = provider.provision( + { ...PROFILE, setup: "install-node" }, + `provision:post-setup-${replay ? "replay" : "fresh"}`, + ); + let settled = false; + void provision.then( + () => { + settled = true; + }, + () => { + settled = true; + }, + ); + await finalInspectStarted; + expect(settled).toBe(false); + resolveFinalInspect( + commandResult({ + stdout: inspectJson({ + sshFallbackPorts: [22, 2222], + sshHost: "after-setup.example.test", + sshHostKey: "ssh-ed25519 BBBB", + sshPort: "2200", + }), + }), + ); + + await expect(provision).resolves.toMatchObject({ + leaseId: LEASE_ID, + ssh: { + fallbackPorts: [22, 2222], + host: "after-setup.example.test", + hostKey: "ssh-ed25519 BBBB", + port: 2200, + }, + }); + expect(calls.map((argv) => argv[1])).toEqual(expectedCommands); + }, + ); + + it("stops a lease that disappears after successful setup", async () => { + const calls: string[][] = []; + let inspections = 0; + const provider = providerWithRunner(async (argv) => { + calls.push(argv); + if (argv[1] === "run" || argv[1] === "stop") { + return commandResult(); + } + inspections += 1; + return inspections === 1 + ? commandResult({ stdout: inspectJson({ sshHostKey: HOST_KEY }) }) + : commandResult({ code: 4, stderr: `lease/server not found: ${LEASE_ID}` }); + }); + + await expect( + provider.provision({ ...PROFILE, setup: "install-node" }, "provision:setup-disappeared"), + ).rejects.toThrow("disappeared while waiting for SSH readiness"); + expect(calls.map((argv) => argv[1])).toEqual(["inspect", "run", "inspect", "stop"]); + expect(calls.at(-1)).toEqual([SIBLING_BINARY, "stop", "--provider", "aws", "--id", LEASE_ID]); + }); + + it("re-attests security on the fresh post-setup inspect before polling", async () => { + const calls: string[][] = []; + let inspections = 0; + const provider = providerWithRunner(async (argv) => { + calls.push(argv); + if (argv[1] === "run" || argv[1] === "stop") { + return commandResult(); + } + inspections += 1; + return commandResult({ + stdout: inspectJson({ + providerMetadata: { instanceProfileAttached: inspections > 1 }, + ready: inspections === 1, + sshHostKey: HOST_KEY, + }), + }); + }); + + await expect( + provider.provision({ ...PROFILE, setup: "install-node" }, "provision:setup-reattest"), + ).rejects.toMatchObject({ + code: "invalid_profile", + message: "Crabbox AWS inspect must attest that no instance profile is attached", + }); + expect(calls.map((argv) => argv[1])).toEqual(["inspect", "run", "inspect", "stop"]); + }); + it("stops the lease when the profile setup command fails", async () => { const calls: string[][] = []; let warmed = false; @@ -345,47 +511,67 @@ describe("Crabbox worker provider", () => { expect(calls.some((argv) => argv[1] === "stop" && argv.includes(LEASE_ID))).toBe(true); }); - it("stops a forbidden replay lease before polling for SSH readiness", async () => { - const calls: string[][] = []; - const provider = providerWithRunner(async (argv) => { - calls.push(argv); - if (argv[1] === "inspect") { - return commandResult({ - stdout: inspectJson({ - providerMetadata: { instanceProfileAttached: true }, - ready: false, - }), + it.each([ + { + state: "pending-metadata then ready-safe", + inspections: [ + { providerMetadata: undefined, ready: false }, + { + providerMetadata: { instanceProfileAttached: false }, + ready: true, + sshHostKey: HOST_KEY, + }, + ], + expectedError: null, + expectedCommands: ["inspect", "inspect"], + }, + { + state: "pending-forbidden", + inspections: [ + { + providerMetadata: { instanceProfileAttached: true }, + ready: false, + }, + ], + expectedError: "Crabbox AWS inspect must attest that no instance profile is attached", + expectedCommands: ["inspect", "stop"], + }, + { + state: "ready-metadata-missing", + inspections: [{ providerMetadata: undefined, ready: true, sshHostKey: HOST_KEY }], + expectedError: "Crabbox AWS inspect must attest that no instance profile is attached", + expectedCommands: ["inspect", "stop"], + }, + ])( + "enforces AWS instance-profile attestation across the $state sequence", + async ({ inspections, expectedError, expectedCommands }) => { + const calls: string[][] = []; + let inspectionIndex = 0; + const provider = providerWithRunner(async (argv) => { + calls.push(argv); + if (argv[1] === "inspect") { + const inspection = inspections[inspectionIndex]; + if (!inspection) { + throw new Error("unexpected extra inspection"); + } + inspectionIndex += 1; + return commandResult({ stdout: inspectJson(inspection) }); + } + return commandResult(); + }); + + const provision = provider.provision(PROFILE, "provision:metadata-sequence"); + if (expectedError) { + await expect(provision).rejects.toMatchObject({ + code: "invalid_profile", + message: expectedError, }); + } else { + await expect(provision).resolves.toMatchObject({ leaseId: LEASE_ID }); } - return commandResult(); - }); - - await expect(provider.provision(PROFILE, "provision:forbidden-replay")).rejects.toMatchObject({ - code: "invalid_profile", - message: "Crabbox AWS inspect must attest that no instance profile is attached", - }); - expect(calls.map((argv) => argv[1])).toEqual(["inspect", "stop"]); - }); - - it("stops an AWS lease when provider metadata cannot attest the instance profile state", async () => { - const calls: string[][] = []; - const provider = providerWithRunner(async (argv) => { - calls.push(argv); - if (argv[1] === "inspect") { - return commandResult({ - stdout: inspectJson({ providerMetadata: undefined, sshHostKey: HOST_KEY }), - }); - } - return commandResult(); - }); - - await expect( - provider.provision(PROFILE, "provision:instance-profile-unknown"), - ).rejects.toMatchObject({ - code: "invalid_profile", - }); - expect(calls.at(-1)).toEqual([SIBLING_BINARY, "stop", "--provider", "aws", "--id", LEASE_ID]); - }); + expect(calls.map((argv) => argv[1])).toEqual(expectedCommands); + }, + ); it.each([ { @@ -393,6 +579,12 @@ describe("Crabbox worker provider", () => { overrides: { providerMetadata: { instanceProfileAttached: "no" } }, }, { field: "Tailscale state", overrides: { tailscale: null } }, + { + field: "SSH fallback ports", + overrides: { + sshFallbackPorts: Array.from({ length: 11 }, (_, index) => 2300 + index), + }, + }, ])("stops a replay lease with malformed $field", async ({ overrides }) => { const calls: string[][] = []; const provider = providerWithRunner(async (argv) => { @@ -770,23 +962,6 @@ describe("Crabbox worker provider", () => { ]); }); - it("waits for a replayed operation lease to become SSH-ready", async () => { - let inspections = 0; - const provider = providerWithRunner(async () => { - inspections += 1; - return commandResult({ - stdout: inspectJson({ ready: inspections > 1, sshHostKey: HOST_KEY }), - }); - }); - - await expect(provider.provision(PROFILE, "provision:operation-pending")).resolves.toMatchObject( - { - leaseId: LEASE_ID, - }, - ); - expect(inspections).toBe(2); - }); - it("keeps readiness polling out of the setup timeout budget", async () => { const calls: string[][] = []; let nowMs = 1_000; diff --git a/extensions/crabbox/src/crabbox-worker-provider.ts b/extensions/crabbox/src/crabbox-worker-provider.ts index 39dc5725f16e..734973b1a35b 100644 --- a/extensions/crabbox/src/crabbox-worker-provider.ts +++ b/extensions/crabbox/src/crabbox-worker-provider.ts @@ -60,6 +60,14 @@ type LeaseCommandContext = { provider: string; }; +type ProvisionInspectContext = { + binary: string; + deadline: number; + inspect: ParsedInspect; + provider: string; + runCommand: CrabboxCommandRunner; +}; + type InspectCommandResult = { status: "found"; inspect: ParsedInspect } | { status: "unknown" }; class InvalidInspectResultError extends Error {} @@ -350,6 +358,7 @@ function leaseFromInspect(inspect: ParsedInspect): WorkerLease { ssh: { host: inspect.host, port: inspect.sshPort, + fallbackPorts: inspect.sshFallbackPorts, user: inspect.sshUser, hostKey: requireHostKey(inspect.sshHostKey), keyRef: { @@ -361,13 +370,7 @@ function leaseFromInspect(inspect: ParsedInspect): WorkerLease { }; } -async function leaseFromProvisionInspect(params: { - binary: string; - deadline: number; - inspect: ParsedInspect; - provider: string; - runCommand: CrabboxCommandRunner; -}): Promise { +async function leaseFromProvisionInspect(params: ProvisionInspectContext): Promise { try { assertProvisionSecurityPolicy(params); return leaseFromInspect(params.inspect); @@ -381,40 +384,43 @@ function assertProvisionSecurityPolicy(params: { inspect: ParsedInspect; provide if (params.inspect.tailscaleEnabled) { throw new WorkerProviderError("Crabbox cloud worker lease must not have Tailscale enabled"); } - if (params.provider === "aws" && params.inspect.awsInstanceProfileAttached !== false) { + const attached = params.inspect.awsInstanceProfileAttached; + const pending = !params.inspect.ready && !isUnusableProvisionState(params.inspect.state); + if (params.provider === "aws" && attached !== false && (attached || !pending)) { throw new WorkerProviderError( "Crabbox AWS inspect must attest that no instance profile is attached", ); } } -async function waitForProvisionReady(params: { - binary: string; - deadline: number; - inspect: ParsedInspect; - provider: string; - runCommand: CrabboxCommandRunner; - sleep: (milliseconds: number) => Promise; -}): Promise { +async function waitForProvisionReady( + params: ProvisionInspectContext & { + refresh?: boolean; + sleep: (milliseconds: number) => Promise; + }, +): Promise { let inspect = params.inspect; + const inspectAgain = async (): Promise => { + const replay = await inspectWithContext({ + context: { binary: params.binary, provider: params.provider }, + expectedLeaseId: inspect.id, + id: inspect.id, + runCommand: params.runCommand, + timeoutMs: remainingProvisionTimeout(params.deadline, LIFECYCLE_TIMEOUT_MS), + }); + if (replay.status === "unknown") { + throw new Error("Crabbox operation lease disappeared while waiting for SSH readiness"); + } + return replay.inspect; + }; try { - // Credential and private-network attestation is authoritative before SSH readiness. - // Reject immediately so a forbidden lease cannot remain live during polling. + inspect = params.refresh ? await inspectAgain() : params.inspect; + // Reject forbidden state immediately; omitted AWS metadata is pending only until ready. assertProvisionSecurityPolicy({ inspect, provider: params.provider }); while (inspect.ready !== true && !isUnusableProvisionState(inspect.state)) { const remaining = remainingProvisionTimeout(params.deadline, LIFECYCLE_TIMEOUT_MS); await params.sleep(Math.min(READY_POLL_INTERVAL_MS, remaining)); - const replay = await inspectWithContext({ - context: { binary: params.binary, provider: params.provider }, - expectedLeaseId: inspect.id, - id: inspect.id, - runCommand: params.runCommand, - timeoutMs: remainingProvisionTimeout(params.deadline, LIFECYCLE_TIMEOUT_MS), - }); - if (replay.status === "unknown") { - throw new Error("Crabbox operation lease disappeared while waiting for SSH readiness"); - } - inspect = replay.inspect; + inspect = await inspectAgain(); assertProvisionSecurityPolicy({ inspect, provider: params.provider }); } if (isUnusableProvisionState(inspect.state)) { @@ -430,14 +436,9 @@ async function waitForProvisionReady(params: { // Setup runs on every provision attempt (including replay adoption), so commands // must be idempotent. A failed setup stops the lease before surfacing the error; // otherwise the caller cannot release a box it never learned about. -async function runProvisionSetup(params: { - binary: string; - deadline: number; - inspect: ParsedInspect; - provider: string; - runCommand: CrabboxCommandRunner; - setup: string; -}): Promise { +async function runProvisionSetup( + params: ProvisionInspectContext & { setup: string }, +): Promise { let result: SpawnResult; try { result = await runCrabboxCommand({ @@ -476,13 +477,19 @@ async function runProvisionSetup(params: { throw error; } -async function stopProvisionInspect(params: { - binary: string; - deadline: number; - inspect: ParsedInspect; - provider: string; - runCommand: CrabboxCommandRunner; -}): Promise { +async function runProvisionSetupAndWaitReady( + params: ProvisionInspectContext & { + setup: string; + sleep: (milliseconds: number) => Promise; + }, +): Promise { + await runProvisionSetup(params); + // Setup may restart SSH or change its endpoint. Re-read the authoritative lease before + // returning any endpoint or security attestation to core bootstrap. + return await waitForProvisionReady({ ...params, refresh: true }); +} + +async function stopProvisionInspect(params: ProvisionInspectContext): Promise { await stopProvisionId({ ...params, id: params.inspect.id }); } @@ -607,12 +614,15 @@ export function createCrabboxWorkerProvider( await stopProvisionInspect(existingParams); } else { existingParams.inspect = await waitForProvisionReady({ ...existingParams, sleep }); - const lease = await leaseFromProvisionInspect(existingParams); if (parsed.setup) { existingParams.deadline = setupDeadline; - await runProvisionSetup({ ...existingParams, setup: parsed.setup }); + existingParams.inspect = await runProvisionSetupAndWaitReady({ + ...existingParams, + setup: parsed.setup, + sleep, + }); } - return lease; + return await leaseFromProvisionInspect(existingParams); } } @@ -693,12 +703,15 @@ export function createCrabboxWorkerProvider( throw new Error("Crabbox warmup lease entered a terminal state"); } inspectedParams.inspect = await waitForProvisionReady({ ...inspectedParams, sleep }); - const lease = await leaseFromProvisionInspect(inspectedParams); if (parsed.setup) { inspectedParams.deadline = setupDeadline; - await runProvisionSetup({ ...inspectedParams, setup: parsed.setup }); + inspectedParams.inspect = await runProvisionSetupAndWaitReady({ + ...inspectedParams, + setup: parsed.setup, + sleep, + }); } - return lease; + return await leaseFromProvisionInspect(inspectedParams); }, async inspect(lease): Promise { const context = resolveLeaseContext(lease); diff --git a/src/config/zod-schema.cloud-workers.test.ts b/src/config/zod-schema.cloud-workers.test.ts index 5cd9380dcd96..b79426e9eca2 100644 --- a/src/config/zod-schema.cloud-workers.test.ts +++ b/src/config/zod-schema.cloud-workers.test.ts @@ -97,6 +97,37 @@ describe("OpenClawSchema cloudWorkers config", () => { }); }); + it("defaults a minimal Crabbox profile to bundle installation", () => { + expect( + parseCloudWorkers({ + profiles: { + aws: { + provider: "crabbox", + settings: { + provider: "aws", + class: "standard", + ttl: "8h", + idleTimeout: "45m", + }, + }, + }, + }), + ).toStrictEqual({ + profiles: { + aws: { + provider: "crabbox", + install: "bundle", + settings: { + provider: "aws", + class: "standard", + ttl: "8h", + idleTimeout: "45m", + }, + }, + }, + }); + }); + it("accepts npm as an explicit install method", () => { expect( parseCloudWorkers({ diff --git a/src/docs/cloud-workers-config.test.ts b/src/docs/cloud-workers-config.test.ts new file mode 100644 index 000000000000..b15e34c03373 --- /dev/null +++ b/src/docs/cloud-workers-config.test.ts @@ -0,0 +1,68 @@ +import fs from "node:fs"; +import path from "node:path"; +import JSON5 from "json5"; +import { describe, expect, it } from "vitest"; +import { OpenClawSchema } from "../config/zod-schema.js"; + +const CLOUD_WORKER_DOCS = [ + "docs/gateway/cloud-workers.md", + "docs/gateway/configuration-reference.md", +] as const; +const CLOUD_WORKER_PAGE = "gateway/cloud-workers"; + +type NavigationNode = { + group?: string; + groups?: NavigationNode[]; + pages?: Array; + tab?: string; +}; + +function cloudWorkerConfigExamples(filePath: string): unknown[] { + const markdown = fs.readFileSync(path.join(process.cwd(), filePath), "utf8"); + return Array.from(markdown.matchAll(/```(?:json5|json)\n([\s\S]*?)```/gu)) + .map((match) => match[1] ?? "") + .filter((source) => /["']?cloudWorkers["']?\s*:/u.test(source)) + .map((source) => JSON5.parse(source)); +} + +function countPage(value: unknown, page: string): number { + if (value === page) { + return 1; + } + if (Array.isArray(value)) { + return value.reduce((count, entry) => count + countPage(entry, page), 0); + } + if (value && typeof value === "object") { + return Object.values(value).reduce((count, entry) => count + countPage(entry, page), 0); + } + return 0; +} + +describe("Cloud Workers documentation contract", () => { + it.each(CLOUD_WORKER_DOCS)("keeps %s config examples schema-valid", (filePath) => { + const examples = cloudWorkerConfigExamples(filePath); + expect(examples.length).toBeGreaterThan(0); + for (const example of examples) { + expect(OpenClawSchema.safeParse(example).success).toBe(true); + } + }); + + it("lists Cloud Workers exactly once in English navigation", () => { + const docs = JSON.parse( + fs.readFileSync(path.join(process.cwd(), "docs", "docs.json"), "utf8"), + ) as { + navigation?: { languages?: Array<{ language?: string; tabs?: NavigationNode[] }> }; + }; + const english = docs.navigation?.languages?.find((entry) => entry.language === "en"); + const gatewayOps = english?.tabs?.find((entry) => entry.tab === "Gateway & Ops"); + const gateway = gatewayOps?.groups?.find((entry) => entry.group === "Gateway"); + const scaling = gateway?.pages?.find( + (entry): entry is NavigationNode => + typeof entry === "object" && entry.group === "Scaling and operations", + ); + + expect(english).toBeDefined(); + expect(countPage(english?.tabs, CLOUD_WORKER_PAGE)).toBe(1); + expect(countPage(scaling?.pages, CLOUD_WORKER_PAGE)).toBe(1); + }); +}); diff --git a/src/gateway/config-reload.test.ts b/src/gateway/config-reload.test.ts index 7c15c5e83896..a1968bc22c72 100644 --- a/src/gateway/config-reload.test.ts +++ b/src/gateway/config-reload.test.ts @@ -305,6 +305,7 @@ describe("buildGatewayReloadPlan", () => { it.each([ "gateway.port", "gateway.terminal.enabled", + "cloudWorkers.profiles.aws.settings.class", "browser.enabled", "plugins.installs.telegram.installPath", "plugins.load.paths.0", diff --git a/src/gateway/server-methods/sessions-delete.ts b/src/gateway/server-methods/sessions-delete.ts index 9b5ff1774306..d78c22725940 100644 --- a/src/gateway/server-methods/sessions-delete.ts +++ b/src/gateway/server-methods/sessions-delete.ts @@ -27,6 +27,7 @@ import { handleSessionStateSessionDeleted } from "../../sessions/session-state-e import { resolveRequestedSessionAgentId as resolveRequestedGlobalAgentId } from "../session-request-agent.js"; import { resolveSessionStoreAgentId } from "../session-store-key.js"; import { loadSessionEntry } from "../session-utils.js"; +import type { WorkerSessionPlacementRetirement } from "../worker-environments/placement-store.js"; import { chatHandlers } from "./chat.js"; import { emitSessionsChanged } from "./session-change-event.js"; import { @@ -35,6 +36,7 @@ import { rejectPluginRuntimeSessionOwnershipMismatch, requireSessionKey, resolveGatewaySessionTargetFromKey, + resolveSessionWorkerPlacementMutationGuard, resolveSessionWorkerPlacementMutationError, respondSessionWorkerPlacementMutationError, sessionLog, @@ -231,6 +233,7 @@ export const sessionDeleteHandlers: GatewayRequestHandlers = { let expectedSessionStillCurrent = true; let deleteBlockedByModelLock = false; let deleteBlockedByWorkerPlacement = false; + let placementRetirement: WorkerSessionPlacementRetirement | undefined; const deletion = await runExclusiveSessionLifecycleMutation({ scope: storePath, identities: deleteLifecycleIdentities, @@ -248,17 +251,23 @@ export const sessionDeleteHandlers: GatewayRequestHandlers = { if (!expectedSessionStillCurrent) { return; } - const placementError = resolveSessionWorkerPlacementMutationError({ + const placementGuard = resolveSessionWorkerPlacementMutationGuard({ action: "delete", context, key, sessionId: normalizeOptionalString(preparedEntry?.sessionId), }); - if (placementError) { + if (placementGuard.error) { deleteBlockedByWorkerPlacement = true; - respondSessionWorkerPlacementMutationError(placementError, respond); + respondSessionWorkerPlacementMutationError(placementGuard.error, respond); return; } + if (placementGuard.retirement) { + if (!context.workerSessionPlacementService?.retireSessionPlacement) { + throw new Error("Worker session placement retirement service is unavailable"); + } + placementRetirement = placementGuard.retirement; + } admittedWorkReleased = await interruptSessionWorkAdmissions({ scope: storePath, identities: deleteLifecycleIdentities, @@ -378,6 +387,12 @@ export const sessionDeleteHandlers: GatewayRequestHandlers = { return undefined; } if (result.deleted) { + if (placementRetirement) { + if (result.deletedSessionId !== placementRetirement.sessionId) { + throw new Error("Deleted session id changed before placement retirement"); + } + context.workerSessionPlacementService?.retireSessionPlacement?.(placementRetirement); + } emitGatewaySessionEndPluginHook({ cfg, sessionKey: target.canonicalKey ?? key, diff --git a/src/gateway/server-methods/sessions-shared.ts b/src/gateway/server-methods/sessions-shared.ts index b8d409bfb7ad..f05106c3b7d5 100644 --- a/src/gateway/server-methods/sessions-shared.ts +++ b/src/gateway/server-methods/sessions-shared.ts @@ -27,6 +27,7 @@ import { isWorkerPlacementSessionRuntimeSupported, resolveWorkerPlacementSessionRuntime, } from "../worker-environments/placement-session-runtime.js"; +import type { WorkerSessionPlacementRetirement } from "../worker-environments/placement-store.js"; import type { GatewayClient, GatewayRequestContext, RespondFn } from "./types.js"; export const sessionLog = createSubsystemLogger("gateway/sessions"); @@ -41,34 +42,64 @@ export class SessionWorkerPlacementMutationError extends Error { } } -export function resolveSessionWorkerPlacementMutationError(params: { +type SessionWorkerPlacementMutationGuard = { + error?: SessionWorkerPlacementMutationError; + retirement?: WorkerSessionPlacementRetirement; +}; + +export function resolveSessionWorkerPlacementMutationGuard(params: { action: "delete" | "fork" | "reset" | "restore" | "rewind" | "switch"; context: GatewayRequestContext; key: string; sessionId: string | undefined; -}): SessionWorkerPlacementMutationError | undefined { +}): SessionWorkerPlacementMutationGuard { if (!params.sessionId) { - return undefined; + return {}; } const placement = params.context.workerSessionPlacementService ?.getMany([params.sessionId]) .get(params.sessionId); - // Failed placement normally keeps destructive mutation fenced. Missing worker identity or an - // authoritative destroyed environment proves cleanup cannot orphan a live worker. + const environment = placement?.environmentId + ? params.context.workerEnvironmentService?.get(placement.environmentId) + : undefined; + // finishProvenDestroy clears leaseId only after provider teardown succeeds. Failed environments + // that retain a lease stay fenced because their teardown is pending or indeterminate. const failedPlacementCanDelete = params.action === "delete" && placement?.state === "failed" && (placement.environmentId === null || - params.context.workerEnvironmentService?.get(placement.environmentId)?.state === "destroyed"); - if ( + environment?.state === "destroyed" || + (environment?.state === "failed" && environment.leaseId === null)); + const placementCanMutate = !placement || placement.state === "local" || (params.action === "delete" && placement.state === "reclaimed") || - failedPlacementCanDelete - ) { - return undefined; + failedPlacementCanDelete; + if (!placementCanMutate) { + return { + error: new SessionWorkerPlacementMutationError(placement.state, params.action, params.key), + }; } - return new SessionWorkerPlacementMutationError(placement.state, params.action, params.key); + if ( + params.action === "delete" && + placement && + (placement.state === "local" || placement.state === "reclaimed" || placement.state === "failed") + ) { + return { + retirement: { + sessionId: placement.sessionId, + expectedState: placement.state, + expectedGeneration: placement.generation, + }, + }; + } + return {}; +} + +export function resolveSessionWorkerPlacementMutationError( + params: Parameters[0], +): SessionWorkerPlacementMutationError | undefined { + return resolveSessionWorkerPlacementMutationGuard(params).error; } export function respondSessionWorkerPlacementMutationError( diff --git a/src/gateway/server-methods/shared-types.ts b/src/gateway/server-methods/shared-types.ts index 51b57f4ac30f..18b170215cfe 100644 --- a/src/gateway/server-methods/shared-types.ts +++ b/src/gateway/server-methods/shared-types.ts @@ -49,6 +49,7 @@ import type { SessionObserverService } from "../session-observer-contract.js"; import type { TerminalLaunchResolution } from "../terminal/launch.js"; import type { TerminalSessionManager } from "../terminal/session-manager.js"; import type { WorkerSessionPlacementReader } from "../worker-environments/placement-projector.js"; +import type { WorkerSessionPlacementRetirementService } from "../worker-environments/placement-store.js"; import type { WorkerEnvironmentServiceContract, WorkerPlacementDispatchContract, @@ -275,7 +276,8 @@ export type GatewayRequestContext = { /** Durable cloud-worker lifecycle; absent from lightweight in-process contexts. */ workerEnvironmentService?: WorkerEnvironmentServiceContract; /** Durable per-session worker placement; absent when cloud workers are disabled. */ - workerSessionPlacementService?: WorkerSessionPlacementReader; + workerSessionPlacementService?: WorkerSessionPlacementReader & + Partial; /** One-way local-to-worker dispatch; absent when cloud workers are disabled. */ workerPlacementDispatchService?: WorkerPlacementDispatchContract; // Operator terminal session store. Absent in local/in-process contexts where diff --git a/src/gateway/server-worker-environment-startup.ts b/src/gateway/server-worker-environment-startup.ts index 7a515a5e089a..1d2510fb6526 100644 --- a/src/gateway/server-worker-environment-startup.ts +++ b/src/gateway/server-worker-environment-startup.ts @@ -167,10 +167,17 @@ export async function createGatewayWorkerEnvironmentRuntime(params: { }), }); }, - bootstrapWorker: async ({ sshEndpoint, installation, resolveIdentity, signal }) => { + bootstrapWorker: async ({ + operationId, + sshEndpoint, + installation, + resolveIdentity, + signal, + }) => { const workerRuntime = await loadWorkerEnvironmentRuntimeModule(); return await workerRuntime.bootstrapWorker( { + operationId, ssh: sshEndpoint, artifact: installation, pinnedHostKey: sshEndpoint.hostKey, diff --git a/src/gateway/server.sessions.worker-placement-lifecycle.test.ts b/src/gateway/server.sessions.worker-placement-lifecycle.test.ts index 3a06204e6174..b242ef1ac8d5 100644 --- a/src/gateway/server.sessions.worker-placement-lifecycle.test.ts +++ b/src/gateway/server.sessions.worker-placement-lifecycle.test.ts @@ -1,4 +1,4 @@ -import { afterEach, expect, test } from "vitest"; +import { afterEach, expect, test, vi } from "vitest"; import { installSessionPlacementResetGuard } from "../agents/session-placement-admission.js"; import { closeOpenClawStateDatabaseForTest } from "../state/openclaw-state-db.js"; import { loadSessionEntry } from "./session-utils.js"; @@ -9,7 +9,10 @@ import { setupGatewaySessionsHandlerTestHarness, } from "./test/server-sessions.test-helpers.js"; import type { WorkerSessionPlacementReader } from "./worker-environments/placement-projector.js"; -import type { WorkerSessionPlacementRecord } from "./worker-environments/placement-store.js"; +import type { + WorkerSessionPlacementRecord, + WorkerSessionPlacementRetirementService, +} from "./worker-environments/placement-store.js"; const { createSessionStoreDir, seedActiveMainSession } = setupGatewaySessionsHandlerTestHarness(); let uninstallResetGuard: (() => void) | undefined; @@ -119,6 +122,16 @@ function sequencedPlacementReader( }; } +function sequencedPlacementService( + records: readonly WorkerSessionPlacementRecord[], + retire: WorkerSessionPlacementRetirementService["retireSessionPlacement"] = () => {}, +) { + return { + ...sequencedPlacementReader(records), + retireSessionPlacement: vi.fn(retire), + }; +} + test("sessions.reset rechecks worker placement inside the lifecycle fence", async () => { await seedActiveMainSession(); let resetGuardReadCount = 0; @@ -142,7 +155,7 @@ test("sessions.delete rechecks worker placement before destructive cleanup", asy const sessionKey = "discord:group:worker-session"; const sessionId = "sess-worker-delete"; await writeSessionStore({ entries: { [sessionKey]: sessionStoreEntry(sessionId) } }); - const placementReader = sequencedPlacementReader([ + const placementService = sequencedPlacementService([ placementRecord(sessionId, "local"), placementRecord(sessionId, "active"), ]); @@ -151,7 +164,7 @@ test("sessions.delete rechecks worker placement before destructive cleanup", asy "sessions.delete", { key: sessionKey }, { - context: { workerSessionPlacementService: placementReader }, + context: { workerSessionPlacementService: placementService }, }, ); @@ -159,14 +172,17 @@ test("sessions.delete rechecks worker placement before destructive cleanup", asy expect(deleted.error?.message).toContain("cloud worker placement is active"); expect(loadSessionEntry(sessionKey).entry?.sessionId).toBe(sessionId); expect(embeddedRunMock.abortCalls).toEqual([]); + expect(placementService.retireSessionPlacement).not.toHaveBeenCalled(); }); -test("sessions.delete rejects failed placement with unresolved worker ownership", async () => { +test("sessions.delete rejects failed placement while its worker lease remains", async () => { await createSessionStoreDir(); const sessionKey = "discord:group:failed-worker-session"; const sessionId = "sess-failed-worker-delete"; await writeSessionStore({ entries: { [sessionKey]: sessionStoreEntry(sessionId) } }); - const placementReader = sequencedPlacementReader([terminalPlacementRecord(sessionId, "failed")]); + const placementService = sequencedPlacementService([ + terminalPlacementRecord(sessionId, "failed"), + ]); const deleted = await directSessionReq( "sessions.delete", @@ -174,10 +190,10 @@ test("sessions.delete rejects failed placement with unresolved worker ownership" { context: { workerEnvironmentService: { - get: () => ({ state: "attached" }), + get: () => ({ state: "failed", leaseId: "lease-1" }), resolveInferenceSessionForRunId: () => undefined, } as never, - workerSessionPlacementService: placementReader, + workerSessionPlacementService: placementService, }, }, ); @@ -186,29 +202,59 @@ test("sessions.delete rejects failed placement with unresolved worker ownership" expect(deleted.error?.message).toContain("cloud worker placement is failed"); expect(loadSessionEntry(sessionKey).entry?.sessionId).toBe(sessionId); expect(embeddedRunMock.abortCalls).toEqual([]); + expect(placementService.retireSessionPlacement).not.toHaveBeenCalled(); }); -test("sessions.delete allows failed placement after its worker is destroyed", async () => { +test.each([ + { name: "local", state: "local" as const }, + { name: "reclaimed", state: "reclaimed" as const }, + { + name: "failed after proven bootstrap teardown", + state: "failed" as const, + environment: { state: "failed", leaseId: null }, + }, + { + name: "failed after worker destruction", + state: "failed" as const, + environment: { state: "destroyed" }, + }, + { + name: "failed before acquiring a worker", + state: "failed" as const, + withoutEnvironment: true, + }, +])("sessions.delete retires a $name placement after deleting its session", async (testCase) => { await createSessionStoreDir(); - const sessionKey = "discord:group:destroyed-failed-worker-session"; - const sessionId = "sess-destroyed-failed-worker-delete"; + const caseId = testCase.name.replaceAll(" ", "-"); + const sessionKey = `discord:group:${caseId}`; + const sessionId = `sess-${caseId}`; await writeSessionStore({ entries: { [sessionKey]: sessionStoreEntry(sessionId) } }); - const placementReader = sequencedPlacementReader([terminalPlacementRecord(sessionId, "failed")]); + const placement = + testCase.state === "local" + ? placementRecord(sessionId, "local") + : terminalPlacementRecord(sessionId, testCase.state); + if ("withoutEnvironment" in testCase && placement.state === "failed") { + placement.environmentId = null; + } + const placementService = sequencedPlacementService([placement], () => { + expect(loadSessionEntry(sessionKey).entry).toBeUndefined(); + }); const deleted = await directSessionReq( "sessions.delete", { key: sessionKey }, { context: { - workerEnvironmentService: { - get: (environmentId: string) => { - expect(environmentId).toBe("worker-environment"); - return { state: "destroyed" }; - }, - hasInferenceForSession: () => false, - resolveInferenceSessionForRunId: () => undefined, - } as never, - workerSessionPlacementService: placementReader, + ...("environment" in testCase + ? { + workerEnvironmentService: { + get: () => testCase.environment, + hasInferenceForSession: () => false, + resolveInferenceSessionForRunId: () => undefined, + } as never, + } + : {}), + workerSessionPlacementService: placementService, }, }, ); @@ -216,52 +262,11 @@ test("sessions.delete allows failed placement after its worker is destroyed", as expect(deleted.ok).toBe(true); expect(deleted.payload).toMatchObject({ ok: true, deleted: true }); expect(loadSessionEntry(sessionKey).entry).toBeUndefined(); -}); - -test("sessions.delete allows failed placement that never acquired a worker", async () => { - await createSessionStoreDir(); - const sessionKey = "discord:group:unallocated-failed-worker-session"; - const sessionId = "sess-unallocated-failed-worker-delete"; - await writeSessionStore({ entries: { [sessionKey]: sessionStoreEntry(sessionId) } }); - const placement = terminalPlacementRecord(sessionId, "failed"); - if (placement.state !== "failed") { - throw new Error("expected failed placement fixture"); - } - placement.environmentId = null; - - const deleted = await directSessionReq( - "sessions.delete", - { key: sessionKey }, - { - context: { workerSessionPlacementService: sequencedPlacementReader([placement]) }, - }, - ); - - expect(deleted.ok).toBe(true); - expect(deleted.payload).toMatchObject({ ok: true, deleted: true }); - expect(loadSessionEntry(sessionKey).entry).toBeUndefined(); -}); - -test("sessions.delete allows reclaimed placement with no live worker owner", async () => { - await createSessionStoreDir(); - const sessionKey = "discord:group:reclaimed-worker-session"; - const sessionId = "sess-reclaimed-worker-delete"; - await writeSessionStore({ entries: { [sessionKey]: sessionStoreEntry(sessionId) } }); - const placementReader = sequencedPlacementReader([ - terminalPlacementRecord(sessionId, "reclaimed"), - ]); - - const deleted = await directSessionReq( - "sessions.delete", - { key: sessionKey }, - { - context: { workerSessionPlacementService: placementReader }, - }, - ); - - expect(deleted.ok).toBe(true); - expect(deleted.payload).toMatchObject({ ok: true, deleted: true }); - expect(loadSessionEntry(sessionKey).entry).toBeUndefined(); + expect(placementService.retireSessionPlacement).toHaveBeenCalledWith({ + sessionId, + expectedState: placement.state, + expectedGeneration: placement.generation, + }); }); test("sessions.compaction.restore rechecks worker placement inside the lifecycle fence", async () => { diff --git a/src/gateway/worker-environments/bootstrap.test.ts b/src/gateway/worker-environments/bootstrap.test.ts index 94c9986d76d4..a9ba563498f3 100644 --- a/src/gateway/worker-environments/bootstrap.test.ts +++ b/src/gateway/worker-environments/bootstrap.test.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import fs from "node:fs/promises"; import path from "node:path"; import { describe, expect, it } from "vitest"; @@ -16,7 +17,10 @@ const TARBALL_SHA256 = "b".repeat(64); const VERSION = "2026.7.11"; const NPM_INTEGRITY = `sha512-${Buffer.alloc(64).toString("base64")}`; const OUTPUT_TAG = "OPENCLAW_WORKER_BOOTSTRAP_V1"; -const REMOTE_TARBALL = `/home/worker/.openclaw-worker/.incoming/${BUNDLE_HASH}.tgz.ABCDEFGH`; +const OPERATION_ID = "provision-operation-1"; +const OPERATION_TOKEN = createHash("sha256").update(OPERATION_ID).digest("hex"); +const UPLOAD_FILENAME = `openclaw-upload-${BUNDLE_HASH}.tgz.${OPERATION_TOKEN}`; +const REMOTE_TARBALL = `/home/worker/.openclaw-worker/.incoming/${UPLOAD_FILENAME}`; const HOST_KEY = ["ssh-ed25519", "AAAA"].join(" "); const RECEIPT_JSON = JSON.stringify({ bundleHash: BUNDLE_HASH, @@ -80,11 +84,20 @@ function fakeRunner( return { calls, runCommand }; } +function commandPort(argv: string[]): number { + const portFlag = argv[0] === "scp" ? "-P" : "-p"; + return Number(argv[argv.indexOf(portFlag) + 1]); +} + const resolveIdentity = async () => ({ kind: "path", path: "/keys/worker" }) as const; const bootstrapWorker = ( - request: WorkerBootstrapRequest, + request: Omit & { operationId?: string }, dependencies: WorkerBootstrapDependencies, -) => bootstrapWorkerCore({ pinnedHostKey: request.ssh.hostKey, ...request }, dependencies); +) => + bootstrapWorkerCore( + { operationId: OPERATION_ID, pinnedHostKey: request.ssh.hostKey, ...request }, + dependencies, + ); describe("bootstrapWorker", () => { it("skips a matching installed bundle and uses the pinned host key", async () => { @@ -129,7 +142,7 @@ describe("bootstrapWorker", () => { await expect( bootstrapWorkerCore( - { ssh: SSH, artifact: BUNDLE }, + { ssh: SSH, artifact: BUNDLE, operationId: OPERATION_ID }, { resolveIdentity: async () => { identityResolutionCount += 1; @@ -144,11 +157,12 @@ describe("bootstrapWorker", () => { expect(runner.calls).toHaveLength(0); }); - it("transfers and installs a fresh bundle with a receipt", async () => { + it("transfers and installs a fresh bundle despite terminal cleanup failure", async () => { const runner = fakeRunner([ result({ stdout: tagged("install", REMOTE_TARBALL) }), result(), result({ stdout: tagged("receipt", RECEIPT_JSON) }), + result({ code: 1, stderr: "synthetic cleanup failure" }), ]); await expect( @@ -162,11 +176,13 @@ describe("bootstrapWorker", () => { protocolFeatures: ["admission-v1"], }); - expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "scp", "ssh"]); + expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "scp", "ssh", "ssh"]); expect(runner.calls[0]?.argv).toContain("StrictHostKeyChecking=yes"); expect(runner.calls.flatMap((call) => call.argv)).not.toContain("StrictHostKeyChecking=no"); expect(runner.calls[1]?.argv).toContain(BUNDLE.tarballPath); expect(runner.calls[1]?.argv).toContain(`worker@worker.example.com:${REMOTE_TARBALL}`); + expect(runner.calls.flatMap((call) => call.argv).join(" ")).not.toContain(OPERATION_ID); + expect(runner.calls[0]?.argv.at(-1)).toContain(OPERATION_TOKEN); expect(runner.calls[2]?.options.input).toContain("bootstrap-receipt.json"); expect(runner.calls[2]?.options.input).toContain("lock=$lock_root/$hash"); expect(runner.calls[2]?.options.input).toContain('ln -s "$lock_identity" "$lock"'); @@ -177,12 +193,102 @@ describe("bootstrapWorker", () => { expect(runner.calls[2]?.argv.at(-1)).toContain(VERSION); }); + it.each([ + `/home/worker/other/.incoming/${UPLOAD_FILENAME}`, + `/home/worker/.openclaw-worker/other/${UPLOAD_FILENAME}`, + `/home/worker/.openclaw-worker/.incoming/../.incoming/${UPLOAD_FILENAME}`, + `/home/worker/./.openclaw-worker/.incoming/${UPLOAD_FILENAME}`, + `/home//worker/.openclaw-worker/.incoming/${UPLOAD_FILENAME}`, + `/home/worker/.openclaw-worker/.incoming/${UPLOAD_FILENAME}.other`, + ])("rejects a noncanonical or non-owned upload path: %s", async (remotePath) => { + const runner = fakeRunner([result({ stdout: tagged("install", remotePath) }), result()]); + + await expect( + bootstrapWorker( + { ssh: SSH, artifact: BUNDLE }, + { resolveIdentity, runCommand: runner.runCommand }, + ), + ).rejects.toThrow("preflight returned an invalid upload path"); + expect(runner.calls).toHaveLength(2); + }); + + it("selects an authenticated fallback before transfer and install", async () => { + const runner = fakeRunner([ + result({ code: 255, stderr: "primary transport unavailable" }), + result({ stdout: tagged("install", REMOTE_TARBALL) }), + result(), + result({ stdout: tagged("receipt", RECEIPT_JSON) }), + result(), + ]); + + await expect( + bootstrapWorker( + { ssh: { ...SSH, fallbackPorts: [22] }, artifact: BUNDLE }, + { resolveIdentity, runCommand: runner.runCommand }, + ), + ).resolves.toEqual(JSON.parse(RECEIPT_JSON)); + + expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "ssh", "scp", "ssh", "ssh"]); + expect(runner.calls.map((call) => commandPort(call.argv))).toEqual([2222, 22, 22, 22, 22]); + expect(new Set(runner.calls.map((call) => call.argv[call.argv.indexOf("-i") + 1]))).toEqual( + new Set(["/keys/worker"]), + ); + expect( + new Set( + runner.calls.map((call) => + call.argv.find((value) => value.startsWith("UserKnownHostsFile=")), + ), + ).size, + ).toBe(1); + }); + + it("retries bundle transfer when the selected port changes after preflight", async () => { + const runner = fakeRunner([ + result({ stdout: tagged("install", REMOTE_TARBALL) }), + result({ code: 255, stderr: "primary transport unavailable" }), + result(), + result({ stdout: tagged("receipt", RECEIPT_JSON) }), + result(), + ]); + + await expect( + bootstrapWorker( + { ssh: { ...SSH, fallbackPorts: [22] }, artifact: BUNDLE }, + { resolveIdentity, runCommand: runner.runCommand }, + ), + ).resolves.toEqual(JSON.parse(RECEIPT_JSON)); + + expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "scp", "scp", "ssh", "ssh"]); + expect(runner.calls.map((call) => commandPort(call.argv))).toEqual([2222, 2222, 22, 22, 22]); + }); + + it("retries install when the selected port changes after bundle transfer", async () => { + const runner = fakeRunner([ + result({ stdout: tagged("install", REMOTE_TARBALL) }), + result(), + result({ code: 255, stderr: "primary transport unavailable" }), + result({ stdout: tagged("receipt", RECEIPT_JSON) }), + result(), + ]); + + await expect( + bootstrapWorker( + { ssh: { ...SSH, fallbackPorts: [22] }, artifact: BUNDLE }, + { resolveIdentity, runCommand: runner.runCommand }, + ), + ).resolves.toEqual(JSON.parse(RECEIPT_JSON)); + + expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "scp", "ssh", "ssh", "ssh"]); + expect(runner.calls.map((call) => commandPort(call.argv))).toEqual([2222, 2222, 2222, 22, 22]); + }); + it("fails with provider setup guidance when Node.js is missing", async () => { const runner = fakeRunner([ result({ code: 42, stderr: "OPENCLAW_WORKER_NODE_MISSING\n", }), + result(), ]); await expect( @@ -191,7 +297,7 @@ describe("bootstrapWorker", () => { { resolveIdentity, runCommand: runner.runCommand }, ), ).rejects.toThrow("install Node in the provider setup phase"); - expect(runner.calls).toHaveLength(1); + expect(runner.calls).toHaveLength(2); }); it("fails with provider setup guidance when Node.js is unsupported", async () => { @@ -200,6 +306,7 @@ describe("bootstrapWorker", () => { code: 45, stderr: "OPENCLAW_WORKER_NODE_UNSUPPORTED: v24.14.1\n", }), + result(), ]); await expect( @@ -208,7 +315,7 @@ describe("bootstrapWorker", () => { { resolveIdentity, runCommand: runner.runCommand }, ), ).rejects.toThrow("Node 22.22.3+, 24.15.0+, or 25.9.0+ with WAL-reset-safe SQLite"); - expect(runner.calls).toHaveLength(1); + expect(runner.calls).toHaveLength(2); expect(runner.calls[0]?.options.input).toContain("process.versions.node"); expect(runner.calls[0]?.options.input).toContain("SELECT sqlite_version() AS version"); }); @@ -230,6 +337,7 @@ describe("bootstrapWorker", () => { const npmRunner = fakeRunner([ result({ stdout: tagged("install", REMOTE_TARBALL) }), result({ stdout: tagged("receipt", npmReceipt) }), + result(), ]); await bootstrapWorker( @@ -237,7 +345,7 @@ describe("bootstrapWorker", () => { { resolveIdentity, runCommand: npmRunner.runCommand }, ); - expect(npmRunner.calls.map((call) => call.argv[0])).toEqual(["ssh", "ssh"]); + expect(npmRunner.calls.map((call) => call.argv[0])).toEqual(["ssh", "ssh", "ssh"]); expect(npmRunner.calls[1]?.options.input).toContain("npm pack"); expect(npmRunner.calls[1]?.options.input).toContain("npm install --global"); expect(npmRunner.calls[1]?.options.input).toContain("--registry=https://registry.npmjs.org/"); @@ -329,7 +437,7 @@ describe("bootstrapWorker", () => { openclawVersion: "2026.7.10", protocolFeatures: ["admission-v1"], }); - const runner = fakeRunner([result({ stdout: tagged("current", staleReceipt) })]); + const runner = fakeRunner([result({ stdout: tagged("current", staleReceipt) }), result()]); await expect( bootstrapWorker( @@ -337,32 +445,71 @@ describe("bootstrapWorker", () => { { resolveIdentity, runCommand: runner.runCommand }, ), ).rejects.toThrow("receipt does not match"); - expect(runner.calls).toHaveLength(1); + expect(runner.calls).toHaveLength(2); }); - it("removes a partial remote upload after transfer failure", async () => { - const runner = fakeRunner([ - result({ stdout: tagged("install", REMOTE_TARBALL) }), - result({ code: 1, stderr: "connection reset" }), - result(), - ]); + it.each([ + { + phase: "bundle transfer", + phaseCommand: "scp", + responses: [ + result({ stdout: tagged("install", REMOTE_TARBALL) }), + result({ code: 1, stderr: "transfer rejected" }), + result({ code: 255, stderr: "selected port changed" }), + result(), + ], + commands: ["ssh", "scp", "ssh", "ssh"], + cleanupPorts: [2222, 22], + }, + { + phase: "install", + phaseCommand: "install", + responses: [ + result({ stdout: tagged("install", REMOTE_TARBALL) }), + result(), + result({ code: 1, stderr: "install rejected" }), + result(), + ], + commands: ["ssh", "scp", "ssh", "ssh"], + cleanupPorts: [2222], + }, + ])( + "does not retry fallback after a non-255 $phase failure and cleans up the upload", + async ({ phase, phaseCommand, responses, commands, cleanupPorts }) => { + const runner = fakeRunner(responses); - await expect( - bootstrapWorker( - { ssh: SSH, artifact: BUNDLE }, - { resolveIdentity, runCommand: runner.runCommand }, - ), - ).rejects.toThrow("bundle transfer failed"); + await expect( + bootstrapWorker( + { ssh: { ...SSH, fallbackPorts: [22] }, artifact: BUNDLE }, + { resolveIdentity, runCommand: runner.runCommand }, + ), + ).rejects.toThrow(`Worker bootstrap ${phase} failed`); - expect(runner.calls.map((call) => call.argv[0])).toEqual(["ssh", "scp", "ssh"]); - expect(runner.calls[2]?.options.input).toContain('rm -f -- "$1"'); - expect(runner.calls[2]?.argv.at(-1)).toContain(REMOTE_TARBALL); - expect(runner.calls[2]?.options.signal).toBeUndefined(); - }); + expect(runner.calls.map((call) => call.argv[0])).toEqual(commands); + const phaseCalls = runner.calls.filter((call) => + phaseCommand === "scp" + ? call.argv[0] === "scp" + : typeof call.options.input === "string" && + call.options.input.includes("receipt_json=$5"), + ); + expect(phaseCalls).toHaveLength(1); + expect(commandPort(phaseCalls[0]!.argv)).toBe(2222); + + const cleanupCalls = runner.calls.filter( + (call) => + typeof call.options.input === "string" && + call.options.input.includes("operation_token=$2"), + ); + expect(cleanupCalls.map((call) => commandPort(call.argv))).toEqual(cleanupPorts); + expect(cleanupCalls.every((call) => call.argv.at(-1)?.includes(BUNDLE_HASH))).toBe(true); + expect(cleanupCalls.every((call) => call.argv.at(-1)?.includes(OPERATION_TOKEN))).toBe(true); + expect(cleanupCalls.every((call) => call.options.signal === undefined)).toBe(true); + }, + ); it("keeps bootstrap failure details on a valid UTF-16 boundary", async () => { const prefix = "e".repeat(511); - const runner = fakeRunner([result({ code: 1, stderr: `${prefix}😀 tail` })]); + const runner = fakeRunner([result({ code: 1, stderr: `${prefix}😀 tail` }), result()]); await expect( bootstrapWorker( @@ -385,7 +532,7 @@ describe("bootstrapWorker", () => { }); it.skipIf(process.platform === "win32")( - "verifies the transferred archive and installed manifest before a receipt", + "reuses and finally cleans the operation upload across ambiguous candidate attempts", async () => { await withTempDir({ prefix: "openclaw-worker-bootstrap-script-" }, async (root) => { const packageRoot = path.join(root, "package"); @@ -405,6 +552,9 @@ describe("bootstrapWorker", () => { openclawVersion: VERSION, protocolFeatures: ["admission-v1"], }).prepare(); + const fakeBin = path.join(root, "fake-bin"); + await fs.mkdir(fakeBin); + await fs.writeFile(path.join(fakeBin, "tar"), "#!/bin/sh\nexit 255\n", { mode: 0o755 }); const receiptJson = JSON.stringify({ bundleHash: artifact.bundleHash, openclawVersion: VERSION, @@ -423,6 +573,11 @@ describe("bootstrapWorker", () => { await fs.symlink(`${process.pid}:1`, staleLock); let remoteTarball = ""; let transfers = 0; + let syntheticPreflightFailures = 2; + let installAttempts = 0; + let uploadSurvivedAmbiguousInstall = false; + let cleanupAttempts = 0; + const preflightPaths: string[] = []; const runCommand: WorkerBootstrapCommandRunner = async (argv, options) => { if (argv[0] === "scp") { transfers += 1; @@ -433,31 +588,101 @@ describe("bootstrapWorker", () => { } const isPreflight = typeof options.input === "string" && options.input.includes("expected_receipt=$2"); + const isInstall = + typeof options.input === "string" && options.input.includes("receipt_json=$5"); + const isCleanup = + typeof options.input === "string" && options.input.includes("operation_token=$2"); + if (isCleanup) { + cleanupAttempts += 1; + } const scriptArgs = isPreflight - ? [artifact.bundleHash, receiptJson, "bundle"] - : [ - "bundle", - artifact.bundleHash, - "", - "", - receiptJson, - remoteTarball, - artifact.tarballSha256, - ]; - return await runCommandWithTimeout(["sh", "-s", "--", ...scriptArgs], { + ? [artifact.bundleHash, receiptJson, "bundle", OPERATION_TOKEN] + : isInstall + ? [ + "bundle", + artifact.bundleHash, + "", + "", + receiptJson, + remoteTarball, + artifact.tarballSha256, + ] + : isCleanup + ? [artifact.bundleHash, OPERATION_TOKEN] + : []; + if (isInstall && installAttempts === 1) { + uploadSurvivedAmbiguousInstall = await fs + .stat(remoteTarball) + .then((stats) => stats.isFile()) + .catch(() => false); + } + const useFailingTar = isInstall && installAttempts === 0; + if (isInstall) { + installAttempts += 1; + } + const shellResult = await runCommandWithTimeout(["sh", "-s", "--", ...scriptArgs], { ...options, - baseEnv: { ...options.baseEnv, HOME: remoteHome }, + baseEnv: { + ...options.baseEnv, + HOME: remoteHome, + PATH: useFailingTar + ? `${fakeBin}:${options.baseEnv?.PATH ?? ""}` + : options.baseEnv?.PATH, + }, }); + if (isPreflight) { + const taggedRecord = shellResult.stdout + .split(/\r?\n/u) + .find((line) => line.startsWith(`${OUTPUT_TAG}\tinstall\t`)); + if (taggedRecord) { + preflightPaths.push(taggedRecord.slice(taggedRecord.lastIndexOf("\t") + 1)); + } + if (syntheticPreflightFailures > 0) { + syntheticPreflightFailures -= 1; + return result({ code: 255, stderr: "synthetic ambiguous preflight" }); + } + } + return shellResult; }; + const bootstrapRequest = { ssh: { ...SSH, fallbackPorts: [22] }, artifact }; await expect( - bootstrapWorker({ ssh: SSH, artifact }, { resolveIdentity, runCommand }), - ).resolves.toEqual(JSON.parse(receiptJson)); + bootstrapWorker(bootstrapRequest, { resolveIdentity, runCommand }), + ).rejects.toThrow("Worker bootstrap preflight failed (exit 255)"); + expect(preflightPaths).toHaveLength(2); + await expect(fs.stat(preflightPaths[0]!)).rejects.toMatchObject({ code: "ENOENT" }); + + syntheticPreflightFailures = 1; await expect( - bootstrapWorker({ ssh: SSH, artifact }, { resolveIdentity, runCommand }), + bootstrapWorker(bootstrapRequest, { resolveIdentity, runCommand }), ).resolves.toEqual(JSON.parse(receiptJson)); + const operationUpload = preflightPaths[0]!; + const staleUpload = path.join( + path.dirname(operationUpload), + `openclaw-upload-${"c".repeat(64)}.tgz.${"d".repeat(64)}`, + ); + await fs.writeFile(operationUpload, "ambiguous prior upload"); + await fs.writeFile(staleUpload, "stale upload"); + const staleTime = new Date(Date.now() - 61 * 60_000); + await fs.utimes(staleUpload, staleTime, staleTime); + const cleanupAttemptsBeforeCurrent = cleanupAttempts; + await expect( + bootstrapWorker({ ssh: SSH, artifact }, { resolveIdentity, runCommand }), + ).resolves.toEqual(JSON.parse(receiptJson)); + expect(cleanupAttempts).toBe(cleanupAttemptsBeforeCurrent); + expect(transfers).toBe(1); + expect(preflightPaths).toHaveLength(4); + expect(new Set(preflightPaths).size).toBe(1); + expect(path.basename(preflightPaths[0]!)).toBe( + `openclaw-upload-${artifact.bundleHash}.tgz.${OPERATION_TOKEN}`, + ); + expect(installAttempts).toBe(2); + expect(uploadSurvivedAmbiguousInstall).toBe(true); + await expect(fs.stat(remoteTarball)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.stat(operationUpload)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.stat(staleUpload)).rejects.toMatchObject({ code: "ENOENT" }); await expect(fs.stat(staleStaging)).rejects.toMatchObject({ code: "ENOENT" }); await expect(fs.lstat(staleLock)).rejects.toMatchObject({ code: "ENOENT" }); await expect( @@ -486,11 +711,17 @@ describe("bootstrapWorker", () => { await fs.mkdir(unrelated); await fs.writeFile(path.join(unrelated, "sentinel"), "keep"); await fs.symlink(unrelated, path.join(bootstrapRoot, ".incoming")); - const runCommand: WorkerBootstrapCommandRunner = async (_argv, options) => - await runCommandWithTimeout(["sh", "-s", "--", BUNDLE_HASH, RECEIPT_JSON, "bundle"], { + const runCommand: WorkerBootstrapCommandRunner = async (_argv, options) => { + const isPreflight = + typeof options.input === "string" && options.input.includes("expected_receipt=$2"); + const scriptArgs = isPreflight + ? [BUNDLE_HASH, RECEIPT_JSON, "bundle", OPERATION_TOKEN] + : [BUNDLE_HASH, OPERATION_TOKEN]; + return await runCommandWithTimeout(["sh", "-s", "--", ...scriptArgs], { ...options, baseEnv: { ...options.baseEnv, HOME: remoteHome }, }); + }; await expect( bootstrapWorker({ ssh: SSH, artifact: BUNDLE }, { resolveIdentity, runCommand }), @@ -500,6 +731,39 @@ describe("bootstrapWorker", () => { }, ); + it.skipIf(process.platform === "win32")( + "does not follow a poisoned bootstrap root during terminal cleanup", + async () => { + await withTempDir({ prefix: "openclaw-worker-bootstrap-cleanup-root-" }, async (root) => { + const remoteHome = path.join(root, "remote-home"); + const unrelated = path.join(root, "unrelated"); + const incoming = path.join(unrelated, ".incoming"); + const upload = path.join(incoming, UPLOAD_FILENAME); + await fs.mkdir(remoteHome); + await fs.mkdir(incoming, { recursive: true }); + await fs.writeFile(upload, "keep"); + await fs.symlink(unrelated, path.join(remoteHome, ".openclaw-worker")); + + const runCommand: WorkerBootstrapCommandRunner = async (_argv, options) => { + const isPreflight = + typeof options.input === "string" && options.input.includes("expected_receipt=$2"); + const scriptArgs = isPreflight + ? [BUNDLE_HASH, RECEIPT_JSON, "bundle", OPERATION_TOKEN] + : [BUNDLE_HASH, OPERATION_TOKEN]; + return await runCommandWithTimeout(["sh", "-s", "--", ...scriptArgs], { + ...options, + baseEnv: { ...options.baseEnv, HOME: remoteHome }, + }); + }; + + await expect( + bootstrapWorker({ ssh: SSH, artifact: BUNDLE }, { resolveIdentity, runCommand }), + ).rejects.toThrow("unsafe worker bootstrap directory"); + await expect(fs.readFile(upload, "utf8")).resolves.toBe("keep"); + }); + }, + ); + it.skipIf(process.platform === "win32")( "verifies npm installs from the packaged dist inventory", async () => { @@ -542,11 +806,17 @@ describe("bootstrapWorker", () => { await fs.mkdir(path.dirname(installRoot), { recursive: true }); await fs.cp(packageRoot, installRoot, { recursive: true }); await fs.writeFile(path.join(installRoot, "bootstrap-receipt.json"), `${receiptJson}\n`); - const runCommand: WorkerBootstrapCommandRunner = async (_argv, options) => - await runCommandWithTimeout(["sh", "-s", "--", bundle.bundleHash, receiptJson, "npm"], { + const runCommand: WorkerBootstrapCommandRunner = async (_argv, options) => { + const isPreflight = + typeof options.input === "string" && options.input.includes("expected_receipt=$2"); + const scriptArgs = isPreflight + ? [bundle.bundleHash, receiptJson, "npm", OPERATION_TOKEN] + : [bundle.bundleHash, OPERATION_TOKEN]; + return await runCommandWithTimeout(["sh", "-s", "--", ...scriptArgs], { ...options, baseEnv: { ...options.baseEnv, HOME: remoteHome }, }); + }; await expect( bootstrapWorker({ ssh: SSH, artifact }, { resolveIdentity, runCommand }), diff --git a/src/gateway/worker-environments/bootstrap.ts b/src/gateway/worker-environments/bootstrap.ts index 12689d85571b..78851a1d49d1 100644 --- a/src/gateway/worker-environments/bootstrap.ts +++ b/src/gateway/worker-environments/bootstrap.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice"; import { type WorkerAdmissionHandshake, @@ -18,6 +19,7 @@ import { WORKER_BUNDLE_MANIFEST_VERSION, type WorkerInstallationArtifact } from import { prepareWorkerSsh, type PreparedWorkerSsh, + runWorkerSshCandidates, workerSshCommandOptions, workerSshOptions, workerSshRemoteCommand, @@ -234,10 +236,19 @@ umask 077 hash=$1 expected_receipt=$2 install=$3 +operation_token=$4 root=$HOME/${BOOTSTRAP_ROOT} install_dir=$root/$hash receipt=$install_dir/${BOOTSTRAP_RECEIPT} +case "$operation_token" in + *[!a-f0-9]*|'') printf '%s\n' 'invalid worker bootstrap operation token' >&2; exit 2 ;; +esac +if [ "${"${"}#operation_token}" -ne 64 ]; then + printf '%s\n' 'invalid worker bootstrap operation token' >&2 + exit 2 +fi + ensure_private_directory() { directory=$1 if [ -e "$directory" ] || [ -L "$directory" ]; then @@ -264,20 +275,30 @@ if ! node -e '${NODE_RUNTIME_CHECK_JS}'; then exit ${NODE_UNSUPPORTED_EXIT_CODE} fi +incoming=$root/.incoming +ensure_private_directory "$incoming" +incoming=$(cd "$incoming" && pwd -P) +find "$incoming" -type f -name 'openclaw-upload-*.tgz.*' -mmin +60 -exec rm -f -- {} + 2>/dev/null || true +upload=$incoming/openclaw-upload-$hash.tgz.$operation_token + if [ -d "$install_dir" ] && [ ! -L "$install_dir" ] && [ -f "$receipt" ] && node -e '${RECEIPT_MATCH_JS}' "$receipt" "$expected_receipt" && node -e '${VERIFY_INSTALL_JS}' "$install_dir" "$hash" "$install"; then + rm -f -- "$upload" printf '%s\t%s\t' '${BOOTSTRAP_OUTPUT_TAG}' current cat "$receipt" printf '\n' exit 0 fi -incoming=$root/.incoming -ensure_private_directory "$incoming" -incoming=$(cd "$incoming" && pwd -P) -find "$incoming" -type f -name 'openclaw-upload-*.tgz.*' -mmin +60 -exec rm -f -- {} + 2>/dev/null || true -upload=$(mktemp "$incoming/openclaw-upload-$hash.tgz.XXXXXXXX") +if [ ! -e "$upload" ] && [ ! -L "$upload" ]; then + (set -C; : > "$upload") 2>/dev/null || true +fi +if [ ! -f "$upload" ] || [ -L "$upload" ]; then + printf '%s\n' 'unsafe worker bootstrap upload' >&2 + exit 2 +fi +chmod 600 "$upload" printf '%s\t%s\t%s\n' '${BOOTSTRAP_OUTPUT_TAG}' install "$upload" `; @@ -323,9 +344,6 @@ cleanup() { rm -f "$lock" fi fi - if [ -n "$upload" ]; then - rm -f "$upload" - fi } trap cleanup 0 trap 'exit 1' 1 2 15 @@ -503,6 +521,7 @@ type WorkerBootstrapCommandRunner = ( type WorkerBootstrapRequest = { ssh: WorkerSshEndpoint; artifact: WorkerInstallationArtifact; + operationId: string; /** Provider endpoint host key copied by the gateway bootstrap adapter. */ pinnedHostKey?: string; }; @@ -594,6 +613,7 @@ async function runSshScript(params: { script: string; scriptArgs: readonly string[]; timeoutMs: number; + port?: number; signal?: AbortSignal; }): Promise { return await params.runCommand( @@ -604,7 +624,7 @@ async function runSshScript(params: { "-x", "-T", "-p", - String(params.prepared.port), + String(params.port ?? params.prepared.port), "--", params.prepared.sshTarget, workerSshRemoteCommand(["sh", "-s", "--", ...params.scriptArgs]), @@ -617,22 +637,54 @@ async function runSshScript(params: { ); } +function workerUploadFilename(bundleHash: string, operationToken: string): string { + return `openclaw-upload-${bundleHash}.tgz.${operationToken}`; +} + const CLEANUP_UPLOAD_SCRIPT = String.raw`set -eu -rm -f -- "$1" +hash=$1 +operation_token=$2 +case "$hash" in + *[!a-f0-9]*|'') exit 2 ;; +esac +case "$operation_token" in + *[!a-f0-9]*|'') exit 2 ;; +esac +if [ "${"${"}#hash}" -ne 64 ] || [ "${"${"}#operation_token}" -ne 64 ]; then + exit 2 +fi +root=$HOME/${BOOTSTRAP_ROOT} +if [ ! -e "$root" ] && [ ! -L "$root" ]; then + exit 0 +fi +if [ ! -d "$root" ] || [ -L "$root" ]; then + exit 2 +fi +incoming=$root/.incoming +if [ ! -d "$incoming" ] || [ -L "$incoming" ]; then + exit 0 +fi +incoming=$(cd "$incoming" && pwd -P) +rm -f -- "$incoming/openclaw-upload-$hash.tgz.$operation_token" `; async function cleanupRemoteUpload(params: { prepared: PreparedWorkerSsh; - remotePath: string; + bundleHash: string; + operationToken: string; runCommand: WorkerBootstrapCommandRunner; timeoutMs: number; }): Promise { - await runSshScript({ - prepared: params.prepared, - runCommand: params.runCommand, - script: CLEANUP_UPLOAD_SCRIPT, - scriptArgs: [params.remotePath], - timeoutMs: Math.min(params.timeoutMs, 10_000), + const cleanupTimeoutMs = Math.min(params.timeoutMs, 10_000); + await runWorkerSshCandidates(params.prepared, cleanupTimeoutMs, (port, remainingTimeoutMs) => { + return runSshScript({ + prepared: params.prepared, + runCommand: params.runCommand, + script: CLEANUP_UPLOAD_SCRIPT, + scriptArgs: [params.bundleHash, params.operationToken], + timeoutMs: remainingTimeoutMs, + port, + }); }).catch(() => undefined); } @@ -654,6 +706,7 @@ function parseTaggedOutput(stdout: string): { action: string; payload: string } function parsePreflight( result: SpawnResult, expected: WorkerAdmissionHandshake, + expectedUploadFilename: string, ): { action: "current"; receipt: WorkerAdmissionHandshake } | { action: "install"; path: string } { if ( result.code === NODE_MISSING_EXIT_CODE || @@ -682,7 +735,12 @@ function parsePreflight( } const remotePath = output?.action === "install" ? output.payload : undefined; const normalizedPath = normalizeScpRemotePath(remotePath); - if (!normalizedPath) { + const expectedSuffix = `/${BOOTSTRAP_ROOT}/.incoming/${expectedUploadFilename}`; + const hasCanonicalSegments = normalizedPath + ?.split("/") + .slice(1) + .every((segment) => segment !== "" && segment !== "." && segment !== ".."); + if (!normalizedPath || !hasCanonicalSegments || !normalizedPath.endsWith(expectedSuffix)) { throw new Error("Worker bootstrap preflight returned an invalid upload path"); } return { action: "install", path: normalizedPath }; @@ -693,7 +751,10 @@ export async function bootstrapWorker( request: WorkerBootstrapRequest, dependencies: WorkerBootstrapDependencies, ): Promise { - const receipt = normalizeHandshake(request.artifact); + const artifact = request.artifact; + const receipt = normalizeHandshake(artifact); + const operationToken = createHash("sha256").update(request.operationId).digest("hex"); + const uploadFilename = workerUploadFilename(receipt.bundleHash, operationToken); const timeoutMs = dependencies.timeoutMs ?? DEFAULT_BOOTSTRAP_TIMEOUT_MS; const runCommand = dependencies.runCommand ?? runCommandWithTimeout; const prepared = await prepareWorkerSsh({ @@ -702,84 +763,104 @@ export async function bootstrapWorker( resolveIdentity: dependencies.resolveIdentity, temporaryDirectoryPrefix: "openclaw-worker-bootstrap-", }); + let needsUploadCleanup = true; try { - const preflight = parsePreflight( - await runSshScript({ - prepared, - runCommand, - script: PREFLIGHT_SCRIPT, - scriptArgs: [receipt.bundleHash, JSON.stringify(receipt), request.artifact.install], - timeoutMs, - signal: dependencies.signal, - }), - receipt, + const preflightResult = await runWorkerSshCandidates( + prepared, + timeoutMs, + (port, remainingTimeoutMs) => + runSshScript({ + prepared, + runCommand, + script: PREFLIGHT_SCRIPT, + scriptArgs: [ + receipt.bundleHash, + JSON.stringify(receipt), + artifact.install, + operationToken, + ], + timeoutMs: remainingTimeoutMs, + port, + signal: dependencies.signal, + }), ); + const preflight = parsePreflight(preflightResult, receipt, uploadFilename); if (preflight.action === "current") { + // A validated current response already removed this operation's upload in preflight. + needsUploadCleanup = false; return preflight.receipt; } - try { - if (request.artifact.install === "bundle") { - const transfer = await runCommand( - [ - "scp", - ...workerSshOptions(prepared, { forwarding: "disabled" }), - "-P", - String(prepared.port), - "--", - request.artifact.tarballPath, - `${prepared.scpTarget}:${preflight.path}`, - ], - workerSshCommandOptions({ timeoutMs, signal: dependencies.signal }), - ); - if (!isSuccess(transfer)) { - throw commandFailure("bundle transfer", transfer); - } + if (artifact.install === "bundle") { + const transfer = await runWorkerSshCandidates( + prepared, + timeoutMs, + (port, remainingTimeoutMs) => + runCommand( + [ + "scp", + ...workerSshOptions(prepared, { forwarding: "disabled" }), + "-P", + String(port), + "--", + artifact.tarballPath, + `${prepared.scpTarget}:${preflight.path}`, + ], + workerSshCommandOptions({ timeoutMs: remainingTimeoutMs, signal: dependencies.signal }), + ), + ); + if (!isSuccess(transfer)) { + throw commandFailure("bundle transfer", transfer); } + } - const install = await runSshScript({ + const install = await runWorkerSshCandidates(prepared, timeoutMs, (port, remainingTimeoutMs) => + runSshScript({ prepared, runCommand, script: INSTALL_SCRIPT, scriptArgs: [ - request.artifact.install, + artifact.install, receipt.bundleHash, - request.artifact.install === "npm" ? request.artifact.packageSpec : "", - request.artifact.install === "npm" ? request.artifact.packageIntegrity : "", + artifact.install === "npm" ? artifact.packageSpec : "", + artifact.install === "npm" ? artifact.packageIntegrity : "", JSON.stringify(receipt), preflight.path, - request.artifact.install === "bundle" ? request.artifact.tarballSha256 : "", + artifact.install === "bundle" ? artifact.tarballSha256 : "", ], - timeoutMs, + timeoutMs: remainingTimeoutMs, + port, signal: dependencies.signal, - }); - if ( - install.code === NPM_MISSING_EXIT_CODE || - install.stderr.includes(NPM_MISSING_MARKER) || - install.stdout.includes(NPM_MISSING_MARKER) - ) { - throw new Error( - "Worker npm bootstrap requires npm on the leased host; use bundle install or provide npm in the provider setup phase", - ); - } - if (!isSuccess(install)) { - throw commandFailure("install", install); - } - const output = parseTaggedOutput(install.stdout); - if (output?.action !== "receipt") { - throw new Error("Worker bootstrap install returned an invalid receipt"); - } - return parseReceiptJson(output.payload, receipt); - } catch (error) { + }), + ); + if ( + install.code === NPM_MISSING_EXIT_CODE || + install.stderr.includes(NPM_MISSING_MARKER) || + install.stdout.includes(NPM_MISSING_MARKER) + ) { + throw new Error( + "Worker npm bootstrap requires npm on the leased host; use bundle install or provide npm in the provider setup phase", + ); + } + if (!isSuccess(install)) { + throw commandFailure("install", install); + } + const output = parseTaggedOutput(install.stdout); + if (output?.action !== "receipt") { + throw new Error("Worker bootstrap install returned an invalid receipt"); + } + return parseReceiptJson(output.payload, receipt); + } finally { + if (needsUploadCleanup) { + // One operation reuses this upload across candidate attempts; only terminal cleanup removes it. await cleanupRemoteUpload({ prepared, - remotePath: preflight.path, + bundleHash: receipt.bundleHash, + operationToken, runCommand, timeoutMs, }); - throw error; } - } finally { await prepared.dispose(); } } diff --git a/src/gateway/worker-environments/placement-dispatch-reclaim.test.ts b/src/gateway/worker-environments/placement-dispatch-reclaim.test.ts index 1c68b353e4a1..8a7b4bbc21dc 100644 --- a/src/gateway/worker-environments/placement-dispatch-reclaim.test.ts +++ b/src/gateway/worker-environments/placement-dispatch-reclaim.test.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -159,6 +160,116 @@ describe("worker placement dispatch reclaim", () => { expect(harness.environments.destroy).toHaveBeenCalledOnce(); }); + it("retires only the exact unclaimed safe placement generation", () => { + const claim = placementStore.claimTurn({ + ...REQUEST, + owner: { kind: "local" }, + claimId: "retirement-claim", + runId: "retirement-run", + }); + expect(() => + placementStore.retireSessionPlacement({ + sessionId: REQUEST.sessionId, + expectedState: "local", + expectedGeneration: 0, + }), + ).toThrow("changed before retirement"); + placementStore.releaseTurn(claim); + placementStore.retireSessionPlacement({ + sessionId: REQUEST.sessionId, + expectedState: "local", + expectedGeneration: 0, + }); + expect(placementStore.get(REQUEST.sessionId)).toBeUndefined(); + + const requested = placementStore.startDispatch(REQUEST); + const failed = placementStore.fail({ + sessionId: REQUEST.sessionId, + expectedGeneration: requested.generation, + recoveryError: "dispatch failed", + }); + for (const stale of [ + { expectedState: "local" as const, expectedGeneration: 0 }, + { expectedState: "failed" as const, expectedGeneration: failed.generation - 1 }, + ]) { + expect(() => + placementStore.retireSessionPlacement({ sessionId: REQUEST.sessionId, ...stale }), + ).toThrow("changed before retirement"); + } + expect(placementStore.get(REQUEST.sessionId)).toMatchObject({ + state: "failed", + generation: failed.generation, + }); + }); + + it("retires a reclaimed placement with its child rows and conflict projection", () => { + const harness = createHarness(placementStore); + const active = harness.placements.seedActive(7); + if (active.state !== "active") { + throw new Error("expected active worker placement"); + } + const claim = placementStore.claimTurn({ + ...REQUEST, + owner: { + kind: "worker", + environmentId: active.environmentId, + ownerEpoch: active.activeOwnerEpoch, + }, + claimId: "retirement-worker-claim", + runId: "retirement-worker-run", + }); + placementStore.recordWorkspaceResultConflict(claim, { + paths: ["conflicted.txt"], + stagedResultRef: `refs/openclaw/worker-results/${claim.claimId}`, + }); + placementStore.releaseTurn(claim); + + const basePack = Buffer.from("retirement workspace base pack"); + placementStore.beginWorkspaceReconciliation( + { + sessionId: active.sessionId, + environmentId: active.environmentId, + ownerEpoch: active.activeOwnerEpoch, + placementGeneration: active.generation, + }, + { + version: 1, + temporaryNonce: "c".repeat(32), + baseManifestRef: active.workspaceBaseManifestRef, + currentManifestRef: `sha256:${"d".repeat(64)}`, + baseEntries: [], + appliedEntries: [], + baseTree: "e".repeat(40), + basePackSha256: createHash("sha256").update(basePack).digest("hex"), + basePack, + }, + ); + const reclaimed = placementStore.finishReclaim({ + sessionId: active.sessionId, + environmentId: active.environmentId, + ownerEpoch: active.activeOwnerEpoch, + expectedGeneration: active.generation, + }); + expect(placementStore.listWorkspaceReconciliationOwners()).toHaveLength(1); + expect(placementStore.get(active.sessionId)?.workspaceResultConflict).toBeDefined(); + + placementStore.retireSessionPlacement({ + sessionId: reclaimed.sessionId, + expectedState: "reclaimed", + expectedGeneration: reclaimed.generation, + }); + + expect(placementStore.get(active.sessionId)).toBeUndefined(); + expect(placementStore.listWorkspaceReconciliationOwners()).toEqual([]); + placementStore.claimTurn({ + ...REQUEST, + owner: { kind: "local" }, + claimId: "replacement-local-claim", + runId: "replacement-local-run", + }); + expect(placementStore.get(active.sessionId)).not.toHaveProperty("workspaceResultConflict"); + }); + it("applies a prepared staged result before requiring its manifest commit", async () => { const harness = createHarness(placementStore, { reconcileCommitsManifest: false, diff --git a/src/gateway/worker-environments/placement-store.ts b/src/gateway/worker-environments/placement-store.ts index 20d867374a6e..480d91ec7325 100644 --- a/src/gateway/worker-environments/placement-store.ts +++ b/src/gateway/worker-environments/placement-store.ts @@ -42,6 +42,14 @@ import { } from "./placement-workspace-result.js"; import { projectWorkspaceResultConflict } from "./workspace-conflicts.js"; +const RETIRABLE_PLACEMENT_STATES = ["local", "reclaimed", "failed"] as const; + +export type WorkerSessionPlacementRetirement = { + sessionId: string; + expectedState: (typeof RETIRABLE_PLACEMENT_STATES)[number]; + expectedGeneration: number; +}; + function exactConflictPath(value: string): string { if (typeof value !== "string" || value.length === 0) { throw new Error("Worker placement conflict path is required"); @@ -148,6 +156,32 @@ export function createWorkerSessionPlacementStore( return records; }, + retireSessionPlacement(input: WorkerSessionPlacementRetirement): void { + const sessionId = required(input.sessionId, "session id"); + if (!(RETIRABLE_PLACEMENT_STATES as readonly string[]).includes(input.expectedState)) { + throw new Error(`Cannot retire worker session placement from ${input.expectedState}`); + } + write((db) => { + const result = executeSqliteQuerySync( + db, + query(db) + .deleteFrom("worker_session_placements") + .where("session_id", "=", sessionId) + .where("state", "=", input.expectedState) + .where("transition_generation", "=", input.expectedGeneration) + .where("turn_claim_owner", "is", null) + .where("turn_claim_id", "is", null) + .where("turn_claim_run_id", "is", null) + .where("turn_claim_generation", "is", null) + .where("turn_claim_owner_epoch", "is", null), + ); + if (result.numAffectedRows !== 1n) { + throw new Error(`Worker session placement ${sessionId} changed before retirement`); + } + }); + workspaceResultConflicts.delete(sessionId); + }, + recordWorkspaceResultConflict( claim: WorkerSessionTurnClaim, conflict: WorkerWorkspaceResultConflict | undefined, @@ -528,3 +562,7 @@ export function createWorkerSessionPlacementStore( } export type WorkerSessionPlacementStore = ReturnType; +export type WorkerSessionPlacementRetirementService = Pick< + WorkerSessionPlacementStore, + "retireSessionPlacement" +>; diff --git a/src/gateway/worker-environments/service.test.ts b/src/gateway/worker-environments/service.test.ts index 9fac9f6f3e51..ffef81b4e045 100644 --- a/src/gateway/worker-environments/service.test.ts +++ b/src/gateway/worker-environments/service.test.ts @@ -1504,6 +1504,17 @@ describe("worker environment service", () => { { leaseId: "lease-invalid", ssh: { ...SSH_ENDPOINT, keyRef: "not-a-secret-ref" } }, "SSH key must be a canonical SecretRef", ], + [ + "excessive SSH fallback ports", + { + leaseId: "lease-invalid", + ssh: { + ...SSH_ENDPOINT, + fallbackPorts: Array.from({ length: 11 }, (_, index) => 2300 + index), + }, + }, + "SSH fallback ports cannot exceed 10", + ], ])("keeps %s from a provider retryable", async (_name, result, error) => { const workerService = createService(createProvider({ provision: async () => result as never })); @@ -1873,6 +1884,7 @@ describe("worker environment service", () => { }); expect(prepareInstallation).toHaveBeenCalledWith("npm"); expect(bootstrapWorker).toHaveBeenCalledWith({ + operationId: result.provisionOperationId, sshEndpoint: SSH_ENDPOINT, installation: NPM_ARTIFACT, resolveIdentity: expect.any(Function), diff --git a/src/gateway/worker-environments/service.ts b/src/gateway/worker-environments/service.ts index c5d118d4a1ad..024e640f198f 100644 --- a/src/gateway/worker-environments/service.ts +++ b/src/gateway/worker-environments/service.ts @@ -110,6 +110,7 @@ type WorkerEnvironmentServiceOptions = { install: WorkerInstallationArtifact["install"], ) => Promise; bootstrapWorker: (params: { + operationId: string; sshEndpoint: WorkerSshEndpoint; installation: WorkerInstallationArtifact; resolveIdentity: (keyRef: SecretRef) => Promise; @@ -563,6 +564,7 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService try { receipt = await callBootstrap((signal) => options.bootstrapWorker({ + operationId: record.provisionOperationId, sshEndpoint: record.sshEndpoint, installation, resolveIdentity: identityResolverFor(record, provider, record.leaseId), diff --git a/src/gateway/worker-environments/ssh.test.ts b/src/gateway/worker-environments/ssh.test.ts index 42d95944bd9b..05505074e0b4 100644 --- a/src/gateway/worker-environments/ssh.test.ts +++ b/src/gateway/worker-environments/ssh.test.ts @@ -1,28 +1,47 @@ import fs from "node:fs/promises"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import type { WorkerSshEndpoint } from "../../plugins/types.js"; -import { prepareWorkerSsh, workerSshOptions } from "./ssh.js"; +import { prepareWorkerSsh, runWorkerSshCandidates, workerSshOptions } from "./ssh.js"; const HOST_KEY = [["ssh", "ed25519"].join("-"), "AAAA"].join(" "); const SSH: WorkerSshEndpoint = { host: "worker.example.test", port: 2202, + fallbackPorts: [22, 2200], user: "worker", hostKey: HOST_KEY, keyRef: { source: "file", provider: "workers", id: "/identity" }, }; +function prepareTestWorkerSsh() { + return prepareWorkerSsh({ + ssh: SSH, + pinnedHostKey: SSH.hostKey, + resolveIdentity: async () => ({ kind: "path" as const, path: "/keys/worker" }), + }); +} + describe("worker SSH preparation", () => { it("shares the pinned trust context while disabling only unrequested forwardings", async () => { + let identityResolutions = 0; const prepared = await prepareWorkerSsh({ ssh: SSH, pinnedHostKey: SSH.hostKey, - resolveIdentity: async () => ({ kind: "path", path: "/keys/worker" }), + resolveIdentity: async () => { + identityResolutions += 1; + return { kind: "path", path: "/keys/worker" }; + }, }); try { expect(await fs.readFile(prepared.knownHostsPath, "utf8")).toBe( - `[worker.example.test]:2202 ${HOST_KEY}\n`, + [ + `[worker.example.test]:2202 ${HOST_KEY}`, + `worker.example.test ${HOST_KEY}`, + `[worker.example.test]:2200 ${HOST_KEY}`, + "", + ].join("\n"), ); + expect(identityResolutions).toBe(1); expect(workerSshOptions(prepared, { forwarding: "disabled" })).toContain( "ClearAllForwardings=yes", ); @@ -43,6 +62,116 @@ describe("worker SSH preparation", () => { } }); + it("rotates stable advertised order from the selected authenticated port", async () => { + const prepared = await prepareTestWorkerSsh(); + try { + const attempted: number[] = []; + await runWorkerSshCandidates(prepared, 10_000, async (port) => { + attempted.push(port); + return { + stdout: "", + stderr: "", + code: port === 2202 ? 255 : 0, + signal: null, + killed: false, + termination: "exit" as const, + }; + }); + + expect(attempted).toEqual([2202, 22]); + expect(prepared.port).toBe(22); + + const retryOrder: number[] = []; + await runWorkerSshCandidates(prepared, 10_000, async (port) => { + retryOrder.push(port); + return { + stdout: "", + stderr: "", + code: 255, + signal: null, + killed: false, + termination: "exit" as const, + }; + }); + expect(retryOrder).toEqual([22, 2200, 2202]); + } finally { + await prepared.dispose(); + } + }); + + it("shares a decreasing deadline while preserving fast fallback budget", async () => { + vi.useFakeTimers(); + vi.setSystemTime(1_000); + const prepared = await prepareTestWorkerSsh(); + try { + const remainingTimeouts: number[] = []; + const result = await runWorkerSshCandidates( + prepared, + 1_000, + async (port, remainingTimeoutMs) => { + remainingTimeouts.push(remainingTimeoutMs); + if (port === 2202) { + vi.advanceTimersByTime(1); + return { code: 255, termination: "exit" }; + } + return { code: 0, termination: "exit" }; + }, + ); + + expect(remainingTimeouts).toEqual([1_000, 999]); + expect(result).toEqual({ code: 0, termination: "exit" }); + expect(prepared.port).toBe(22); + } finally { + await prepared.dispose(); + vi.useRealTimers(); + } + }); + + it("does not start a later candidate after the operation deadline", async () => { + vi.useFakeTimers(); + vi.setSystemTime(1_000); + const prepared = await prepareTestWorkerSsh(); + try { + const attempted: number[] = []; + const firstResult = { code: 255, termination: "exit" as const }; + const result = await runWorkerSshCandidates(prepared, 100, async (port) => { + attempted.push(port); + vi.advanceTimersByTime(100); + return firstResult; + }); + + expect(attempted).toEqual([2202]); + expect(result).toBe(firstResult); + expect(prepared.port).toBe(2202); + } finally { + await prepared.dispose(); + vi.useRealTimers(); + } + }); + + it.each(["timeout", "signal"] as const)( + "does not select a candidate after %s termination", + async (termination) => { + const prepared = await prepareTestWorkerSsh(); + prepared.selectPort(22); + try { + const attempted: number[] = []; + const result = await runWorkerSshCandidates(prepared, 1_000, async (port) => { + attempted.push(port); + return port === 22 + ? { code: 255, termination: "exit" as const } + : { code: null, termination }; + }); + + expect(attempted).toEqual([22, 2200]); + expect(result).toEqual({ code: null, termination }); + expect(prepared.port).toBe(22); + } finally { + await prepared.dispose(); + } + }, + ); + it("materializes identity contents once and removes them with the shared context", async () => { const prepared = await prepareWorkerSsh({ ssh: SSH, diff --git a/src/gateway/worker-environments/ssh.ts b/src/gateway/worker-environments/ssh.ts index a9ca7606f1f2..7513fa9beed1 100644 --- a/src/gateway/worker-environments/ssh.ts +++ b/src/gateway/worker-environments/ssh.ts @@ -16,9 +16,11 @@ export type PreparedWorkerSsh = { sshTarget: string; scpTarget: string; host: string; - port: number; + readonly advertisedPorts: readonly number[]; + readonly port: number; identityPath: string; knownHostsPath: string; + selectPort(port: number): void; dispose(): Promise; }; @@ -101,12 +103,18 @@ export async function prepareWorkerSsh(params: { "Worker SSH setup is missing pinnedHostKey; WorkerProvider.provision() must return ssh.hostKey", ); } + const pinnedHostKey = params.pinnedHostKey; const endpoint = normalizeEndpoint(params.ssh); - const knownHosts = pinnedKnownHostsLine({ - host: endpoint.host, - port: endpoint.port, - pinnedHostKey: params.pinnedHostKey, - }); + const advertisedPorts = [endpoint.port, ...(params.ssh.fallbackPorts ?? [])]; + const knownHosts = advertisedPorts + .map((port) => + pinnedKnownHostsLine({ + host: endpoint.host, + port, + pinnedHostKey, + }), + ) + .join(""); const temporaryDir = await fs.mkdtemp( path.join(os.tmpdir(), params.temporaryDirectoryPrefix ?? "openclaw-worker-ssh-"), ); @@ -137,10 +145,23 @@ export async function prepareWorkerSsh(params: { // The isolated file contains only trusted provisioning output; SSH never learns the first key. await fs.writeFile(knownHostsPath, knownHosts, { mode: 0o600 }); let disposed = false; + let selectedPort = endpoint.port; return { - ...endpoint, + sshTarget: endpoint.sshTarget, + scpTarget: endpoint.scpTarget, + host: endpoint.host, + advertisedPorts, + get port() { + return selectedPort; + }, identityPath, knownHostsPath, + selectPort(port) { + if (!advertisedPorts.includes(port)) { + throw new Error("Worker SSH selected an unadvertised port"); + } + selectedPort = port; + }, async dispose() { if (disposed) { return; @@ -155,6 +176,70 @@ export async function prepareWorkerSsh(params: { } } +/** Returns advertised candidates in stable circular order from the lifecycle selection. */ +function workerSshCandidatePorts(prepared: PreparedWorkerSsh): readonly number[] { + const selectedIndex = prepared.advertisedPorts.indexOf(prepared.port); + if (selectedIndex <= 0) { + return prepared.advertisedPorts; + } + return [ + ...prepared.advertisedPorts.slice(selectedIndex), + ...prepared.advertisedPorts.slice(0, selectedIndex), + ]; +} + +type WorkerSshCommandResult = { + termination: string; + code: number | null; +}; + +function isWorkerSshTransportFailure(result: WorkerSshCommandResult): boolean { + return result.termination === "exit" && result.code === 255; +} + +/** Retries only SSH's transport-level exit 255 under one operation deadline. */ +export async function runWorkerSshCandidates( + prepared: PreparedWorkerSsh, + timeoutMs: number, + run: (port: number, remainingTimeoutMs: number) => Promise, +): Promise { + const deadlineMs = Date.now() + timeoutMs; + let lastResult: T | undefined; + for (const port of workerSshCandidatePorts(prepared)) { + const remainingTimeoutMs = deadlineMs - Date.now(); + if (lastResult !== undefined && remainingTimeoutMs <= 0) { + return lastResult; + } + const result = await run(port, Math.max(0, remainingTimeoutMs)); + lastResult = result; + if (result.termination === "exit" && result.code !== null && result.code !== 255) { + prepared.selectPort(port); + return result; + } + if (!isWorkerSshTransportFailure(result)) { + return result; + } + } + return lastResult!; +} + +/** Moves a reconnect to the next candidate without overwriting a newer concurrent selection. */ +export function advanceWorkerSshAfterTransportExit( + prepared: PreparedWorkerSsh, + failedPort: number, + exit: { code: number | null; signal: NodeJS.Signals | null }, +): boolean { + if (exit.code !== 255 || exit.signal !== null || prepared.port !== failedPort) { + return false; + } + const nextPort = workerSshCandidatePorts(prepared)[1]; + if (nextPort === undefined) { + return false; + } + prepared.selectPort(nextPort); + return true; +} + /** Pinned SSH options shared by bootstrap, tunnel control, and workspace transfer. */ export function workerSshOptions( prepared: PreparedWorkerSsh, diff --git a/src/gateway/worker-environments/store.test.ts b/src/gateway/worker-environments/store.test.ts index 1f1a9161b11d..b4700fba6eed 100644 --- a/src/gateway/worker-environments/store.test.ts +++ b/src/gateway/worker-environments/store.test.ts @@ -3,14 +3,21 @@ import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { WorkerAdmissionHandshake } from "../../../packages/gateway-protocol/src/schema/worker-admission.js"; +import { requireNodeSqlite } from "../../infra/node-sqlite.js"; import type { WorkerProfile, WorkerSshEndpoint } from "../../plugins/types.js"; import { + assertOpenClawStateDatabaseForMaintenance, closeOpenClawStateDatabaseForTest, openOpenClawStateDatabase, + OPENCLAW_STATE_SCHEMA_VERSION, type OpenClawStateDatabase, } from "../../state/openclaw-state-db.js"; import { hashWorkerCredential } from "./credential.js"; -import { createWorkerEnvironmentStore, type WorkerEnvironmentStore } from "./store.js"; +import { + createWorkerEnvironmentStore, + normalizeWorkerSshEndpoint, + type WorkerEnvironmentStore, +} from "./store.js"; type WorkerEnvironmentBootstrapReceipt = WorkerAdmissionHandshake; type WorkerEnvironmentProfileSnapshot = WorkerProfile; @@ -19,7 +26,8 @@ type WorkerEnvironmentSshEndpoint = WorkerSshEndpoint; const HOST_KEY = ["ssh-ed25519", "AAAA"].join(" "); const SSH_ENDPOINT: WorkerEnvironmentSshEndpoint = { host: "worker.example.test", - port: 22, + port: 2222, + fallbackPorts: [22, 2200], user: "openclaw", hostKey: HOST_KEY, keyRef: { @@ -69,6 +77,17 @@ describe("worker environment store", () => { }); } + function fallbackPortRows(environmentId: string) { + return database.db + .prepare( + `SELECT position, port + FROM worker_environment_ssh_fallback_ports + WHERE environment_id = ? + ORDER BY position`, + ) + .all(environmentId); + } + function seedBootstrapping(environmentId: string, leaseId: string) { createIntent(environmentId); store.transition({ environmentId, from: "requested", to: "provisioning" }); @@ -184,6 +203,11 @@ describe("worker environment store", () => { protocolFeatures: ["model-proxy-v1", "workspace-sync-v1"], }, }); + expect(store.list()[0]?.sshEndpoint).toEqual(SSH_ENDPOINT); + expect(fallbackPortRows("worker-1")).toEqual([ + { position: 0, port: 22 }, + { position: 1, port: 2200 }, + ]); nowMs = 1_040; expect( store.transition({ @@ -225,10 +249,117 @@ describe("worker environment store", () => { stateChangedAtMs: 1_080, idleSinceAtMs: null, attachedSessionIds: [], + sshEndpoint: SSH_ENDPOINT, }); + expect(fallbackPortRows("worker-1")).toEqual([ + { position: 0, port: 22 }, + { position: 1, port: 2200 }, + ]); expect(store.listForReconcile()).toEqual([]); }); + it("replaces ordered SSH fallback rows when the endpoint changes", () => { + seedBootstrapping("worker-endpoint-change", "lease-endpoint-change"); + const replacement = { ...SSH_ENDPOINT, fallbackPorts: [2201, 22] }; + + expect( + store.transition({ + environmentId: "worker-endpoint-change", + from: "bootstrapping", + to: "ready", + patch: { ...readyPatch(), sshEndpoint: replacement }, + }).sshEndpoint, + ).toEqual(replacement); + expect(fallbackPortRows("worker-endpoint-change")).toEqual([ + { position: 0, port: 2201 }, + { position: 1, port: 22 }, + ]); + }); + + it("lazily ensures the companion table once for a current database", () => { + const databasePath = database.path; + closeOpenClawStateDatabaseForTest(); + const { DatabaseSync } = requireNodeSqlite(); + const current = new DatabaseSync(databasePath); + current.exec("DROP TABLE worker_environment_ssh_fallback_ports;"); + current.close(); + + database = openOpenClawStateDatabase({ env: { OPENCLAW_STATE_DIR: root } }); + expect( + database.db + .prepare("SELECT name FROM sqlite_schema WHERE type = 'table' AND name = ?") + .get("worker_environment_ssh_fallback_ports"), + ).toBeUndefined(); + expect(database.db.prepare("PRAGMA user_version").get()).toEqual({ + user_version: OPENCLAW_STATE_SCHEMA_VERSION, + }); + + store = createWorkerEnvironmentStore({ database, now: () => nowMs }); + createWorkerEnvironmentStore({ database, now: () => nowMs }); + expect( + database.db + .prepare("SELECT name FROM sqlite_schema WHERE type = 'table' AND name = ?") + .get("worker_environment_ssh_fallback_ports"), + ).toEqual({ name: "worker_environment_ssh_fallback_ports" }); + expect(() => + assertOpenClawStateDatabaseForMaintenance(database.db, { + pathname: database.path, + }), + ).not.toThrow(); + }); + + it("enforces canonical companion-table constraints and cascading ownership", () => { + createIntent("worker-constraints"); + expect( + database.db + .prepare( + "SELECT strict FROM pragma_table_list WHERE name = 'worker_environment_ssh_fallback_ports'", + ) + .get(), + ).toEqual({ strict: 1 }); + const insert = database.db.prepare( + `INSERT INTO worker_environment_ssh_fallback_ports (environment_id, position, port) + VALUES (?, ?, ?)`, + ); + expect(() => insert.run("worker-constraints", -1, 22)).toThrow(); + expect(() => insert.run("worker-constraints", 10, 22)).toThrow(); + expect(() => insert.run("worker-constraints", 0, 0)).toThrow(); + expect(() => insert.run("worker-constraints", 0, 65_536)).toThrow(); + expect(() => insert.run("missing-worker", 0, 22)).toThrow(); + + insert.run("worker-constraints", 0, 22); + expect(() => insert.run("worker-constraints", 0, 2200)).toThrow(); + expect(() => insert.run("worker-constraints", 1, 22)).toThrow(); + database.db + .prepare("DELETE FROM worker_environments WHERE environment_id = ?") + .run("worker-constraints"); + expect(fallbackPortRows("worker-constraints")).toEqual([]); + }); + + it("normalizes provider-advertised SSH fallback ports at the durable boundary", () => { + expect( + normalizeWorkerSshEndpoint({ + ...SSH_ENDPOINT, + fallbackPorts: [22, 2200, 22, 2222], + }), + ).toEqual(SSH_ENDPOINT); + }); + + it.each([ + ["non-array", "22"], + ["non-integer", [22.5]], + ["below range", [0]], + ["above range", [65_536]], + ["more than ten", Array.from({ length: 11 }, (_, index) => 2300 + index)], + ])("rejects %s SSH fallback ports", (_name, fallbackPorts) => { + expect(() => + normalizeWorkerSshEndpoint({ + ...SSH_ENDPOINT, + fallbackPorts, + } as unknown as WorkerEnvironmentSshEndpoint), + ).toThrow("SSH fallback ports"); + }); + it("keeps renewal on one owner epoch and fences session replacement", () => { const bootstrapping = seedBootstrapping("worker-owner", "lease-owner"); store.transition({ @@ -570,6 +701,8 @@ describe("worker environment store", () => { leaseId: null, teardownTerminalState: "failed", }); + expect(store.get(pending.environmentId)?.sshEndpoint).toBeNull(); + expect(fallbackPortRows(pending.environmentId)).toEqual([]); }); it("persists retryable errors without a self-transition", () => { diff --git a/src/gateway/worker-environments/store.ts b/src/gateway/worker-environments/store.ts index 0f6eebde8675..0776ee2f5132 100644 --- a/src/gateway/worker-environments/store.ts +++ b/src/gateway/worker-environments/store.ts @@ -17,6 +17,7 @@ import { isValidSecretRef } from "../../secrets/ref-contract.js"; import type { DB as StateDatabase, WorkerEnvironmentCredentials, + WorkerEnvironmentSshFallbackPorts, WorkerEnvironments, } from "../../state/openclaw-state-db.generated.js"; import { @@ -73,10 +74,15 @@ export type WorkerEnvironmentTransitionPatch = { }; type WorkerDb = Pick< StateDatabase, - "worker_environment_credentials" | "worker_environments" | "worker_transcript_commit_heads" + | "worker_environment_credentials" + | "worker_environment_ssh_fallback_ports" + | "worker_environments" + | "worker_transcript_commit_heads" >; type Row = Selectable; +type RowWithFallbackPort = Row & { ssh_fallback_port: number | null }; type RowUpdate = Updateable; +type SshFallbackPortInsert = Insertable; type CredentialRow = Selectable; type CredentialInsert = Insertable; type CredentialInput = { @@ -99,6 +105,18 @@ type TransitionInput = { const TERMINAL_STATES: WorkerEnvironmentState[] = ["destroyed", "failed", "orphaned"]; const WORKER_BUNDLE_HASH_PATTERN = /^[a-f0-9]{64}$/u; const MAX_HOST_KEY_LENGTH = 16_384; +const MAX_SSH_FALLBACK_PORTS = 10; +const ensuredWorkerEnvironmentDatabases = new WeakSet(); +const WORKER_ENVIRONMENT_SSH_FALLBACK_PORTS_SCHEMA_SQL = ` +CREATE TABLE IF NOT EXISTS worker_environment_ssh_fallback_ports ( + environment_id TEXT NOT NULL, + position INTEGER NOT NULL CHECK (position >= 0 AND position <= 9), + port INTEGER NOT NULL CHECK (port >= 1 AND port <= 65535), + PRIMARY KEY (environment_id, position), + UNIQUE (environment_id, port), + FOREIGN KEY (environment_id) REFERENCES worker_environments(environment_id) ON DELETE CASCADE +) STRICT; +`; const WORKER_CREDENTIAL_HASH_PATTERN = /^[A-Za-z0-9_-]{43}$/u; const OPENSSH_HOST_KEY_TYPE_PATTERN = /^(?:ssh|ecdsa-sha2|sk-(?:ssh|ecdsa-sha2))-[A-Za-z0-9@._+-]+$/u; @@ -223,9 +241,37 @@ export function normalizeWorkerSshEndpoint(value: Ssh): Ssh { if (!isValidSecretRef(value.keyRef)) { throw new Error("Worker environment SSH key must be a canonical SecretRef"); } - return { host, port: value.port, user, hostKey, keyRef: { ...value.keyRef } }; + if (value.fallbackPorts !== undefined && !Array.isArray(value.fallbackPorts)) { + throw new Error("Worker environment SSH fallback ports must be an array"); + } + const seen = new Set([value.port]); + const fallbackPorts: number[] = []; + for (const port of value.fallbackPorts ?? []) { + if (!Number.isSafeInteger(port) || port < 1 || port > 65_535) { + throw new Error( + "Worker environment SSH fallback ports must be integers from 1 through 65535", + ); + } + if (!seen.has(port)) { + seen.add(port); + fallbackPorts.push(port); + } + } + if (fallbackPorts.length > MAX_SSH_FALLBACK_PORTS) { + throw new Error( + `Worker environment SSH fallback ports cannot exceed ${MAX_SSH_FALLBACK_PORTS}`, + ); + } + return { + host, + port: value.port, + ...(fallbackPorts.length > 0 ? { fallbackPorts } : {}), + user, + hostKey, + keyRef: { ...value.keyRef }, + }; } -function endpointFrom(row: Row): Ssh | null { +function endpointFrom(row: Row, fallbackPorts: readonly number[]): Ssh | null { const { ssh_host: host, ssh_port: port, @@ -239,6 +285,7 @@ function endpointFrom(row: Row): Ssh | null { return normalizeWorkerSshEndpoint({ host, port, + ...(fallbackPorts.length > 0 ? { fallbackPorts } : {}), user, hostKey, keyRef: JSON.parse(encoded) as Ssh["keyRef"], @@ -315,7 +362,7 @@ function nextGlobalOwnerEpoch(db: DatabaseSync): number { Math.max(latestEnvironment?.owner_epoch ?? 0, latestTranscriptCommit?.run_epoch ?? 0), ); } -function fromRow(row: Row): WorkerEnvironmentRecord { +function fromRow(row: Row, fallbackPorts: readonly number[]): WorkerEnvironmentRecord { const record = { environmentId: row.environment_id, providerId: row.provider_id, @@ -323,7 +370,7 @@ function fromRow(row: Row): WorkerEnvironmentRecord { profileSnapshot: JSON.parse(row.profile_snapshot_json) as WorkerEnvironmentProfileSnapshot, provisionOperationId: row.provision_operation_id, leaseId: row.lease_id, - sshEndpoint: endpointFrom(row), + sshEndpoint: endpointFrom(row, fallbackPorts), bootstrapReceipt: bootstrapReceiptFrom(row), ownerEpoch: row.owner_epoch, teardownTerminalState: teardownTerminalStateFrom(row.teardown_terminal_state), @@ -361,15 +408,42 @@ function credentialFromRow(row: CredentialRow): WorkerCredentialRecord { } const json = (value: unknown) => JSON.stringify(value) as string; const query = (db: DatabaseSync) => getNodeSqliteKysely(db); +function environmentRows(db: DatabaseSync) { + return query(db) + .selectFrom("worker_environments") + .leftJoin( + "worker_environment_ssh_fallback_ports", + "worker_environment_ssh_fallback_ports.environment_id", + "worker_environments.environment_id", + ) + .selectAll("worker_environments") + .select("worker_environment_ssh_fallback_ports.port as ssh_fallback_port"); +} +function recordsFromRows(rows: readonly RowWithFallbackPort[]): WorkerEnvironmentRecord[] { + const grouped = new Map(); + for (const row of rows) { + const current = grouped.get(row.environment_id); + if (current) { + if (row.ssh_fallback_port !== null) { + current.ports.push(row.ssh_fallback_port); + } + continue; + } + grouped.set(row.environment_id, { + ports: row.ssh_fallback_port === null ? [] : [row.ssh_fallback_port], + row, + }); + } + return Array.from(grouped.values(), ({ row, ports }) => fromRow(row, ports)); +} function find(db: DatabaseSync, environmentId: string) { - const row = executeSqliteQueryTakeFirstSync( + const rows = executeSqliteQuerySync( db, - query(db) - .selectFrom("worker_environments") - .selectAll() - .where("environment_id", "=", environmentId), - ); - return row ? fromRow(row) : undefined; + environmentRows(db) + .where("worker_environments.environment_id", "=", environmentId) + .orderBy("worker_environment_ssh_fallback_ports.position"), + ).rows; + return recordsFromRows(rows)[0]; } function findCredential(db: DatabaseSync, environmentId: string) { const row = executeSqliteQueryTakeFirstSync( @@ -398,7 +472,7 @@ function getRequired(db: DatabaseSync, environmentId: string) { } return record; } -function update(db: DatabaseSync, id: string, state: WorkerEnvironmentState, values: RowUpdate) { +function updateRow(db: DatabaseSync, id: string, state: WorkerEnvironmentState, values: RowUpdate) { const result = executeSqliteQuerySync( db, query(db) @@ -410,8 +484,35 @@ function update(db: DatabaseSync, id: string, state: WorkerEnvironmentState, val if (result.numAffectedRows !== 1n) { throw new Error(`Worker environment ${id} changed during update`); } +} +function update(db: DatabaseSync, id: string, state: WorkerEnvironmentState, values: RowUpdate) { + updateRow(db, id, state, values); return getRequired(db, id); } +function replaceSshFallbackPorts( + db: DatabaseSync, + environmentId: string, + ports: readonly number[], +): void { + executeSqliteQuerySync( + db, + query(db) + .deleteFrom("worker_environment_ssh_fallback_ports") + .where("environment_id", "=", environmentId), + ); + if (ports.length === 0) { + return; + } + const rows: SshFallbackPortInsert[] = ports.map((port, position) => ({ + environment_id: environmentId, + position, + port, + })); + executeSqliteQuerySync( + db, + query(db).insertInto("worker_environment_ssh_fallback_ports").values(rows), + ); +} function revokeCredential(db: DatabaseSync, environmentId: string): void { executeSqliteQuerySync( db, @@ -465,13 +566,19 @@ function credentialInsert(params: { }; } function listRows(db: DatabaseSync, reconcile: boolean): WorkerEnvironmentRecord[] { - const base = query(db).selectFrom("worker_environments").selectAll(); - const filtered = reconcile ? base.where("state", "not in", TERMINAL_STATES) : base; - const ordered = reconcile ? filtered.orderBy("provider_id") : filtered; - return executeSqliteQuerySync( + const base = environmentRows(db); + const filtered = reconcile + ? base.where("worker_environments.state", "not in", TERMINAL_STATES) + : base; + const ordered = reconcile ? filtered.orderBy("worker_environments.provider_id") : filtered; + const rows = executeSqliteQuerySync( db, - ordered.orderBy("created_at_ms").orderBy("environment_id"), - ).rows.map(fromRow); + ordered + .orderBy("worker_environments.created_at_ms") + .orderBy("worker_environments.environment_id") + .orderBy("worker_environment_ssh_fallback_ports.position"), + ).rows; + return recordsFromRows(rows); } function compareAttachmentAuthority( @@ -528,7 +635,19 @@ function reconcileAttachedSessionOwners(db: DatabaseSync, nowMs: number): void { export function createWorkerEnvironmentStore( options: { database?: OpenClawStateDatabase; now?: () => number } = {}, ) { - const path = (options.database ?? openOpenClawStateDatabase()).path; + const database = options.database ?? openOpenClawStateDatabase(); + if (!ensuredWorkerEnvironmentDatabases.has(database.db)) { + runOpenClawStateWriteTransaction( + ({ db }) => { + // sqlite-allow-raw -- feature-local additive schema DDL; rows use Kysely below. + db.exec(WORKER_ENVIRONMENT_SSH_FALLBACK_PORTS_SCHEMA_SQL); + }, + { database }, + { operationLabel: "worker-environments.ssh-fallback-ports.schema.ensure" }, + ); + ensuredWorkerEnvironmentDatabases.add(database.db); + } + const path = database.path; const now = options.now ?? Date.now; const read = () => openOpenClawStateDatabase({ path }).db; const write = (operation: (db: DatabaseSync) => T): T => @@ -785,7 +904,7 @@ export function createWorkerEnvironmentStore( : acceptsAttachedCredential || ownerEndingTransition ? nextGlobalOwnerEpoch(db) : current.ownerEpoch; - const record = update(db, environmentId, from, { + updateRow(db, environmentId, from, { lease_id: leaseId, ssh_host: sshEndpoint?.host ?? null, ssh_port: sshEndpoint?.port ?? null, @@ -805,6 +924,9 @@ export function createWorkerEnvironmentStore( idle_since_at_ms: to === "idle" ? updatedAtMs : null, last_error: "lastError" in patch ? patch.lastError?.trim() || null : null, }); + if (patch.sshEndpoint !== undefined) { + replaceSshFallbackPorts(db, environmentId, sshEndpoint?.fallbackPorts ?? []); + } if (revokesCredential) { revokeCredential(db, environmentId); } @@ -821,7 +943,7 @@ export function createWorkerEnvironmentStore( }), ); } - return record; + return getRequired(db, environmentId); }); }, renewCredential( diff --git a/src/gateway/worker-environments/tunnel-contract.ts b/src/gateway/worker-environments/tunnel-contract.ts index 5d0a89a91b87..b054309c9798 100644 --- a/src/gateway/worker-environments/tunnel-contract.ts +++ b/src/gateway/worker-environments/tunnel-contract.ts @@ -20,6 +20,7 @@ export type WorkerTunnelRequest = { export type WorkerWorkspaceCommand = { argv: readonly string[]; + transportRetry: "idempotent" | "never"; input?: string; timeoutMs?: number; signal?: AbortSignal; diff --git a/src/gateway/worker-environments/tunnel.test.ts b/src/gateway/worker-environments/tunnel.test.ts index 30491faeafd1..03af7519aeba 100644 --- a/src/gateway/worker-environments/tunnel.test.ts +++ b/src/gateway/worker-environments/tunnel.test.ts @@ -1,7 +1,8 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { useAutoCleanupTempDirTracker } from "../../../test/helpers/temp-dir.js"; import type { WorkerSshEndpoint } from "../../plugins/types.js"; import { runCommandWithTimeout, @@ -14,6 +15,7 @@ import { type WorkerSshRunner, } from "./tunnel-ssh-runner.js"; import { createWorkerTunnelManager } from "./tunnel.js"; +import { rsyncArgvPort, sshArgvPort } from "./worker-ssh-argv.test-support.js"; import type { WorkerWorkspaceReconciliationJournal, WorkerWorkspaceReconciliationJournalAdapter, @@ -28,6 +30,7 @@ function waitForFast( type WorkerSshProcessExit = Awaited; +const tempDirs = useAutoCleanupTempDirTracker(afterEach); const HOST_KEY = [["ssh", "ed25519"].join("-"), "AAAA"].join(" "); const SSH: WorkerSshEndpoint = { host: "worker.example.test", @@ -36,6 +39,7 @@ const SSH: WorkerSshEndpoint = { hostKey: HOST_KEY, keyRef: { source: "file", provider: "workers", id: "/identity" }, }; +const PWD_COMMAND = { transportRetry: "idempotent", argv: ["pwd"] } as const; function success(stdout = "", stderr = ""): SpawnResult { return { @@ -90,12 +94,13 @@ class FakeProcess implements WorkerSshProcess { this.readyDeferred.resolve(); } - failReady(message = "connect failed") { + failReady(message = "connect failed", code = 1) { this.readyDeferred.reject(new Error(message)); + this.exitDeferred.resolve({ code, signal: null }); } - exit() { - this.exitDeferred.resolve({ code: 1, signal: null }); + exit(code = 1) { + this.exitDeferred.resolve({ code, signal: null }); } blockStopUntil(barrier: Promise) { @@ -205,20 +210,48 @@ async function waitForStarts(starts: unknown[], count: number) { await waitForFast(() => expect(starts).toHaveLength(count)); } +type TunnelTestFake = Pick, "runner" | "starts">; +type TunnelManagerOptions = NonNullable[0]>; +type TunnelManager = ReturnType; + +function startTestTunnel( + manager: TunnelManager, + environmentId: string, + ownerEpoch: number, + ssh: WorkerSshEndpoint = SSH, +) { + return manager.start({ + environmentId, + ownerEpoch, + ssh, + gateway: { host: "127.0.0.1", port: 18789 }, + resolveIdentity, + }); +} + +async function startConnectedTunnel( + fake: TunnelTestFake, + environmentId: string, + ownerEpoch: number, + options: { + ssh?: WorkerSshEndpoint; + manager?: Omit; + beforeReady?: (start: TunnelTestFake["starts"][number]) => void; + } = {}, +) { + const manager = createWorkerTunnelManager({ ...options.manager, runner: fake.runner }); + const starting = startTestTunnel(manager, environmentId, ownerEpoch, options.ssh); + await waitForStarts(fake.starts, 1); + const start = fake.starts[0]!; + options.beforeReady?.(start); + start.process.becomeReady(); + return { manager, handle: await starting, start }; +} + describe("worker tunnel manager", () => { it("establishes a pinned reverse socket with keepalives and a separate workspace connection", async () => { const fake = fakeRunner(); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:one", - ownerEpoch: 3, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - - await waitForStarts(fake.starts, 1); - const tunnel = fake.starts[0]; + const { manager, handle, start: tunnel } = await startConnectedTunnel(fake, "worker:one", 3); expect(tunnel?.argv).toContain("ClearAllForwardings=no"); expect(tunnel?.argv).toContain("ServerAliveInterval=15"); expect(tunnel?.argv).toContain("ServerAliveCountMax=3"); @@ -228,18 +261,14 @@ describe("worker tunnel manager", () => { expect(tunnel?.argv[tunnel.argv.indexOf("-R") + 1]).toMatch( /^\/tmp\/ocw-[a-f0-9]{16}-3\/gateway\.sock:127\.0\.0\.1:18789$/u, ); - tunnel?.process.becomeReady(); - const handle = await starting; expect(manager.status("worker:one")).toBe("connected"); - - await expect(handle.runWorkspaceCommand({ argv: ["pwd"] })).resolves.toEqual(success()); + await expect(handle.runWorkspaceCommand(PWD_COMMAND)).resolves.toEqual(success()); const workspace = fake.runs.at(-1); expect(workspace?.argv).toContain("ClearAllForwardings=yes"); expect(workspace?.argv).toContain("ControlMaster=no"); expect(workspace?.argv).toContain("ControlPath=none"); expect(workspace?.argv.at(-1)).toContain("pwd"); expect(fake.starts).toHaveLength(1); - await handle.stop(); expect(tunnel?.process.stopCount).toBe(1); expect(manager.status("worker:one")).toBe("stopped"); @@ -257,17 +286,7 @@ describe("worker tunnel manager", () => { } return undefined; }); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:quiescence-renewal", - ownerEpoch: 3, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:quiescence-renewal", 3); vi.useFakeTimers(); try { @@ -314,17 +333,7 @@ describe("worker tunnel manager", () => { } return undefined; }); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:sync", - ownerEpoch: 5, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:sync", 5); try { await expect( @@ -372,17 +381,9 @@ describe("worker tunnel manager", () => { } return undefined; }); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:sync-failure", - ownerEpoch: 2, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, + const { handle } = await startConnectedTunnel(fake, "worker:sync-failure", 2, { + ssh: { ...SSH, fallbackPorts: [22] }, }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; await expect( handle.syncWorkspace({ @@ -394,10 +395,71 @@ describe("worker tunnel manager", () => { expect( fake.runs.some((entry) => entry.argv.at(-1)?.includes("worker workspace symlink escapes")), ).toBe(false); + const rsyncCalls = fake.runs.filter((entry) => entry.argv[0] === "rsync"); + expect(rsyncCalls).toHaveLength(1); + expect(rsyncArgvPort(rsyncCalls[0]!.argv)).toBe(2202); await handle.stop(); }); + it("moves a later fresh workspace transfer to an advertised fallback", async () => { + const endpoint = { ...SSH, port: 2222, fallbackPorts: [22] }; + const remoteWorkspaceDir = "/home/worker/.openclaw-worker/workspaces/env/session/1"; + const manifestRef = `sha256:${"c".repeat(64)}`; + const localPath = tempDirs.make("openclaw-worker-fallback-sync-"); + await fs.writeFile(path.join(localPath, "artifact.txt"), "transfer me\n"); + const fake = fakeRunner((argv, options) => { + if ( + typeof options.input === "string" && + options.input.includes("unsafe worker workspace directory") + ) { + return success(`${remoteWorkspaceDir}\n`); + } + if (argv[0] === "rsync") { + return rsyncArgvPort(argv) === 2222 + ? { ...success("", "primary transport unavailable"), code: 255 } + : success(); + } + if (argv.at(-1)?.includes("worker workspace symlink escapes")) { + return success(`${manifestRef}\n`); + } + return undefined; + }); + const { handle } = await startConnectedTunnel(fake, "worker:fallback-sync", 1, { + ssh: endpoint, + beforeReady: (start) => expect(sshArgvPort(start.argv)).toBe(2222), + }); + + try { + await expect( + handle.syncWorkspace({ localPath, sessionId: "session:fallback", generation: 1 }), + ).resolves.toEqual({ mode: "plain", remoteWorkspaceDir, manifestRef }); + await expect(handle.runWorkspaceCommand(PWD_COMMAND)).resolves.toEqual(success()); + + const freshConnections = fake.runs.filter( + (entry) => entry.argv[0] === "ssh" || entry.argv[0] === "rsync", + ); + const ports = freshConnections.map((entry) => + entry.argv[0] === "ssh" ? sshArgvPort(entry.argv) : rsyncArgvPort(entry.argv), + ); + expect(ports).toEqual(expect.arrayContaining([2222, 22])); + expect(new Set(ports)).toEqual(new Set([2222, 22])); + expect(sshArgvPort(fake.runs.at(-1)!.argv)).toBe(22); + + const identityPath = fake.runs[0]!.argv[fake.runs[0]!.argv.indexOf("-i") + 1]!; + const knownHostsOption = fake.runs[0]!.argv.find((value) => + value.startsWith("UserKnownHostsFile="), + )!; + for (const connection of [...freshConnections, ...fake.starts]) { + expect(connection.argv.join(" ")).toContain(identityPath); + expect(connection.argv.join(" ")).toContain(knownHostsOption); + } + } finally { + await handle.stop(); + await fs.rm(localPath, { recursive: true, force: true }); + } + }); + it("does not downgrade an operational HEAD probe failure to plain sync", async () => { const remoteWorkspaceDir = "/home/worker/.openclaw-worker/workspaces/env/session/3"; const localPath = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-worker-head-probe-")); @@ -422,17 +484,7 @@ describe("worker tunnel manager", () => { } return undefined; }); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:head-probe-failure", - ownerEpoch: 3, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:head-probe-failure", 3); try { await expect( @@ -469,17 +521,7 @@ describe("worker tunnel manager", () => { } return undefined; }); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:root-probe-failure", - ownerEpoch: 4, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:root-probe-failure", 4); try { await expect( @@ -541,17 +583,7 @@ describe("worker tunnel manager", () => { ]); const fake = localWorkspaceRunner(remoteHome); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:real-git-sync", - ownerEpoch: 11, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:real-git-sync", 11); try { const result = await handle.syncWorkspace({ @@ -736,17 +768,7 @@ describe("worker tunnel manager", () => { await fs.symlink(path.join(root, "outside"), path.join(gitPath, "escape")); const fake = localWorkspaceRunner(remoteHome); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const starting = manager.start({ - environmentId: "worker:real-sync-modes", - ownerEpoch: 12, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; + const { handle } = await startConnectedTunnel(fake, "worker:real-sync-modes", 12); try { const plain = await handle.syncWorkspace({ @@ -782,23 +804,14 @@ describe("worker tunnel manager", () => { it("reconnects with capped backoff after unexpected exits and failed attempts", async () => { const fake = fakeRunner(); const delays: number[] = []; - const manager = createWorkerTunnelManager({ - runner: fake.runner, - backoff: { initialMs: 5, maxMs: 10, factor: 2, jitter: 0 }, - sleep: async (ms) => { - delays.push(ms); + const { manager, handle } = await startConnectedTunnel(fake, "worker:retry", 1, { + manager: { + backoff: { initialMs: 5, maxMs: 10, factor: 2, jitter: 0 }, + sleep: async (ms) => { + delays.push(ms); + }, }, }); - const starting = manager.start({ - environmentId: "worker:retry", - ownerEpoch: 1, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; fake.starts[0]?.process.exit(); await waitForStarts(fake.starts, 2); @@ -812,33 +825,111 @@ describe("worker tunnel manager", () => { await handle.stop(); }); + it("reconnects on the next advertised port after SSH transport exit 255", async () => { + const fake = fakeRunner(); + const { handle } = await startConnectedTunnel(fake, "worker:port-reconnect", 1, { + ssh: { ...SSH, port: 2222, fallbackPorts: [22] }, + manager: { sleep: async () => {} }, + beforeReady: (start) => expect(sshArgvPort(start.argv)).toBe(2222), + }); + + fake.starts[0]!.process.exit(255); + await waitForStarts(fake.starts, 2); + expect(sshArgvPort(fake.starts[1]!.argv)).toBe(22); + expect(sshArgvPort(fake.runs.at(-1)!.argv)).toBe(22); + fake.starts[1]!.process.becomeReady(); + await handle.stop(); + }); + + it("shares setup and best-effort stop cleanup deadlines across fallback candidates", async () => { + let nowMs = 1_000; + const dateNow = vi.spyOn(Date, "now").mockImplementation(() => nowMs); + const setupAttempts: Array<{ port: number; timeoutMs: number }> = []; + const cleanupAttempts: Array<{ port: number; timeoutMs: number }> = []; + const fake = fakeRunner((argv, options) => { + const port = sshArgvPort(argv); + if (port === undefined) { + throw new Error("missing tunnel SSH port"); + } + if ( + typeof options.input === "string" && + options.input.includes("unsafe worker tunnel directory") + ) { + const timeoutMs = options.timeoutMs; + if (timeoutMs === undefined) { + throw new Error("missing tunnel setup timeout"); + } + setupAttempts.push({ port, timeoutMs }); + if (setupAttempts.length === 1) { + nowMs += 7_000; + return { ...success("", "primary transport unavailable"), code: 255 }; + } + return success(); + } + if (typeof options.input === "string" && options.input.includes('rmdir -- "$directory"')) { + const timeoutMs = options.timeoutMs; + if (timeoutMs === undefined) { + throw new Error("missing tunnel cleanup timeout"); + } + cleanupAttempts.push({ port, timeoutMs }); + if (cleanupAttempts.length === 1) { + nowMs += 5_000; + return { ...success("", "selected transport unavailable"), code: 255 }; + } + return success(); + } + return undefined; + }); + const manager = createWorkerTunnelManager({ runner: fake.runner, sleep: async () => {} }); + try { + const starting = startTestTunnel(manager, "worker:operation-deadline", 1, { + ...SSH, + port: 2222, + fallbackPorts: [22], + }); + await waitForStarts(fake.starts, 1); + expect(sshArgvPort(fake.starts[0]!.argv)).toBe(22); + fake.starts[0]!.process.becomeReady(); + const handle = await starting; + + fake.starts[0]!.process.exit(); + await waitForStarts(fake.starts, 2); + expect(sshArgvPort(fake.starts[1]!.argv)).toBe(22); + fake.starts[1]!.process.becomeReady(); + await handle.stop(); + expect(setupAttempts).toEqual([ + { port: 2222, timeoutMs: 20_000 }, + { port: 22, timeoutMs: 13_000 }, + { port: 22, timeoutMs: 20_000 }, + ]); + expect(cleanupAttempts).toEqual([ + { port: 22, timeoutMs: 20_000 }, + { port: 2222, timeoutMs: 15_000 }, + ]); + expect(manager.status("worker:operation-deadline")).toBe("stopped"); + } finally { + dateNow.mockRestore(); + await manager.stopAll(); + } + }); + it("backs off repeated short-lived connected tunnels", async () => { const fake = fakeRunner(); const delays: number[] = []; - const manager = createWorkerTunnelManager({ - runner: fake.runner, - backoff: { initialMs: 5, maxMs: 10, factor: 2, jitter: 0 }, - sleep: async (ms) => { - delays.push(ms); + const { handle } = await startConnectedTunnel(fake, "worker:flap", 1, { + manager: { + backoff: { initialMs: 5, maxMs: 10, factor: 2, jitter: 0 }, + sleep: async (ms) => { + delays.push(ms); + }, }, }); - const starting = manager.start({ - environmentId: "worker:flap", - ownerEpoch: 1, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; for (let index = 0; index < 3; index += 1) { fake.starts[index]?.process.exit(); await waitForStarts(fake.starts, index + 2); fake.starts[index + 1]?.process.becomeReady(); } - expect(delays).toEqual([5, 10, 10]); await handle.stop(); }); @@ -846,28 +937,19 @@ describe("worker tunnel manager", () => { it("fences reconnect before teardown and ignores a late process readiness signal", async () => { const fake = fakeRunner(); const sleepStarted = deferred(); - const manager = createWorkerTunnelManager({ - runner: fake.runner, - sleep: async (_ms, signal) => { - if (!signal) { - throw new Error("missing reconnect signal"); - } - sleepStarted.resolve(signal); - await new Promise((_resolve, reject) => { - signal.addEventListener("abort", () => reject(new Error("aborted")), { once: true }); - }); + const { manager, handle } = await startConnectedTunnel(fake, "worker:drain", 8, { + manager: { + sleep: async (_ms, signal) => { + if (!signal) { + throw new Error("missing reconnect signal"); + } + sleepStarted.resolve(signal); + await new Promise((_resolve, reject) => { + signal.addEventListener("abort", () => reject(new Error("aborted")), { once: true }); + }); + }, }, }); - const starting = manager.start({ - environmentId: "worker:drain", - ownerEpoch: 8, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await starting; fake.starts[0]?.process.exit(); await sleepStarted.promise; @@ -875,13 +957,7 @@ describe("worker tunnel manager", () => { expect(manager.status("worker:drain")).toBe("stopped"); expect(fake.starts).toHaveLength(1); - const pending = manager.start({ - environmentId: "worker:late", - ownerEpoch: 1, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); + const pending = startTestTunnel(manager, "worker:late", 1); const pendingResult = expect(pending).rejects.toThrow("stopped before connecting"); await waitForStarts(fake.starts, 2); const late = fake.starts[1]?.process; @@ -894,54 +970,20 @@ describe("worker tunnel manager", () => { it("rejects stale owner epochs without replacing the current tunnel", async () => { const fake = fakeRunner(); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const current = manager.start({ - environmentId: "worker:epoch", - ownerEpoch: 4, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - const handle = await current; + const { manager, handle } = await startConnectedTunnel(fake, "worker:epoch", 4); - await expect( - manager.start({ - environmentId: "worker:epoch", - ownerEpoch: 3, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }), - ).rejects.toThrow("epoch is stale"); + await expect(startTestTunnel(manager, "worker:epoch", 3)).rejects.toThrow("epoch is stale"); expect(fake.starts).toHaveLength(1); await handle.stop(); }); it("publishes a replacement epoch before awaiting prior teardown", async () => { const fake = fakeRunner(); - const manager = createWorkerTunnelManager({ runner: fake.runner }); - const current = manager.start({ - environmentId: "worker:replacement", - ownerEpoch: 1, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); - await waitForStarts(fake.starts, 1); - fake.starts[0]?.process.becomeReady(); - await current; + const { manager } = await startConnectedTunnel(fake, "worker:replacement", 1); const releaseStop = deferred(); fake.starts[0]?.process.blockStopUntil(releaseStop.promise); - const replacement = manager.start({ - environmentId: "worker:replacement", - ownerEpoch: 2, - ssh: SSH, - gateway: { host: "127.0.0.1", port: 18789 }, - resolveIdentity, - }); + const replacement = startTestTunnel(manager, "worker:replacement", 2); const rejectedReplacement = expect(replacement).rejects.toThrow("stopped before connecting"); await waitForFast(() => expect(fake.starts[0]?.process.stopCount).toBe(1)); diff --git a/src/gateway/worker-environments/tunnel.ts b/src/gateway/worker-environments/tunnel.ts index c7d7c2227151..3b15779c4c82 100644 --- a/src/gateway/worker-environments/tunnel.ts +++ b/src/gateway/worker-environments/tunnel.ts @@ -3,8 +3,10 @@ import { sleepWithAbort, type BackoffPolicy } from "../../infra/backoff.js"; import type { WorkerSshEndpoint } from "../../plugins/types.js"; import type { SpawnResult } from "../../process/exec.js"; import { + advanceWorkerSshAfterTransportExit, prepareWorkerSsh, type PreparedWorkerSsh, + runWorkerSshCandidates, type WorkerSshIdentityResolver, workerSshCommandOptions, workerSshOptions, @@ -139,7 +141,13 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = const sshCommand = ( prepared: PreparedWorkerSsh, - params: { input: string; remoteArgs: readonly string[]; signal?: AbortSignal }, + params: { + input: string; + port: number; + remoteArgs: readonly string[]; + timeoutMs: number; + signal?: AbortSignal; + }, ) => ({ argv: [ "ssh", @@ -148,14 +156,14 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = "-x", "-T", "-p", - String(prepared.port), + String(params.port), "--", prepared.sshTarget, workerSshRemoteCommand(["sh", "-s", "--", ...params.remoteArgs]), ], options: workerSshCommandOptions({ input: params.input, - timeoutMs: REMOTE_SETUP_TIMEOUT_MS, + timeoutMs: params.timeoutMs, signal: params.signal, }), }); @@ -165,26 +173,43 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = if (!prepared) { throw new Error("Worker tunnel SSH context is unavailable"); } - const command = sshCommand(prepared, { - input: REMOTE_SOCKET_SETUP_SCRIPT, - remoteArgs: [entry.remoteDirectory, entry.remoteSocketPath], - signal: entry.abortController.signal, - }); - const result = await runner.run(command.argv, command.options); + const result = await runWorkerSshCandidates( + prepared, + REMOTE_SETUP_TIMEOUT_MS, + async (port, remainingTimeoutMs) => { + const command = sshCommand(prepared, { + input: REMOTE_SOCKET_SETUP_SCRIPT, + port, + remoteArgs: [entry.remoteDirectory, entry.remoteSocketPath], + timeoutMs: remainingTimeoutMs, + signal: entry.abortController.signal, + }); + return await runner.run(command.argv, command.options); + }, + ); if (!success(result)) { throw workerSshProcessError(result.stderr || result.stdout); } }; const cleanupRemoteSocket = async (entry: TunnelEntry) => { - if (!entry.prepared) { + const prepared = entry.prepared; + if (!prepared) { return; } - const command = sshCommand(entry.prepared, { - input: REMOTE_SOCKET_CLEANUP_SCRIPT, - remoteArgs: [entry.remoteSocketPath, entry.remoteDirectory], - }); - await runner.run(command.argv, command.options).catch(() => undefined); + await runWorkerSshCandidates( + prepared, + REMOTE_SETUP_TIMEOUT_MS, + async (port, remainingTimeoutMs) => { + const command = sshCommand(prepared, { + input: REMOTE_SOCKET_CLEANUP_SCRIPT, + port, + remoteArgs: [entry.remoteSocketPath, entry.remoteDirectory], + timeoutMs: remainingTimeoutMs, + }); + return await runner.run(command.argv, command.options); + }, + ).catch(() => undefined); }; const createHandle = (entry: TunnelEntry): WorkerTunnelHandle => ({ @@ -202,7 +227,9 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = stop: () => stop(entry.environmentId, entry.ownerEpoch), }); - const connect = async (entry: TunnelEntry): Promise => { + const connect = async ( + entry: TunnelEntry, + ): Promise<{ port: number; process: WorkerSshProcess }> => { const prepared = entry.prepared; if (!prepared) { throw new Error("Worker tunnel SSH context is unavailable"); @@ -212,7 +239,8 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = throw new Error("Worker tunnel owner changed during connection"); } const target = `${remoteTargetHost(entry.gateway.host)}:${entry.gateway.port}`; - return runner.start( + const port = prepared.port; + const process = runner.start( [ "ssh", ...workerSshOptions(prepared, { forwarding: "explicit" }), @@ -230,7 +258,7 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = "-R", `${entry.remoteSocketPath}:${target}`, "-p", - String(prepared.port), + String(port), "--", prepared.sshTarget, workerSshRemoteCommand(["sh", "-s", "--", entry.remoteSocketPath]), @@ -241,6 +269,7 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = signal: entry.abortController.signal, }), ); + return { port, process }; }; const reconnectLoop = async (entry: TunnelEntry) => { @@ -248,8 +277,11 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = while (isCurrent(entry)) { entry.status = reconnectSupervisor.attempts === 0 ? "connecting" : "reconnecting"; let child: WorkerSshProcess | undefined; + let childPort: number | undefined; try { - child = await connect(entry); + const connection = await connect(entry); + child = connection.process; + childPort = connection.port; entry.process = child; await child.ready; if (!isCurrent(entry)) { @@ -262,11 +294,20 @@ export function createWorkerTunnelManager(options: WorkerTunnelManagerOptions = entry.resolveReady(createHandle(entry)); } const connectedAtMs = now(); - await child.exited; + const exit = await child.exited; + if (entry.prepared) { + advanceWorkerSshAfterTransportExit(entry.prepared, childPort, exit); + } if (now() - connectedAtMs >= stableConnectionMs) { reconnectSupervisor.reset(); } } catch { + if (child && childPort !== undefined) { + const exit = await child.exited.catch(() => undefined); + if (exit && entry.prepared) { + advanceWorkerSshAfterTransportExit(entry.prepared, childPort, exit); + } + } await child?.stop().catch(() => undefined); } finally { if (entry.process === child) { diff --git a/src/gateway/worker-environments/worker-ssh-argv.test-support.ts b/src/gateway/worker-environments/worker-ssh-argv.test-support.ts new file mode 100644 index 000000000000..e950b0cbf7d9 --- /dev/null +++ b/src/gateway/worker-environments/worker-ssh-argv.test-support.ts @@ -0,0 +1,15 @@ +export function sshArgvPort(argv: readonly string[]): number | undefined { + if (argv[0] !== "ssh") { + return undefined; + } + return Number(argv[argv.indexOf("-p") + 1]); +} + +export function rsyncArgvPort(argv: readonly string[]): number | undefined { + if (argv[0] !== "rsync") { + return undefined; + } + const remoteShell = argv[argv.indexOf("-e") + 1] ?? ""; + const match = /'-p' '(\d+)'/u.exec(remoteShell); + return match ? Number(match[1]) : undefined; +} diff --git a/src/gateway/worker-environments/worker-turn-launcher.test.ts b/src/gateway/worker-environments/worker-turn-launcher.test.ts index 291fb7afb2ea..31d4f4684a26 100644 --- a/src/gateway/worker-environments/worker-turn-launcher.test.ts +++ b/src/gateway/worker-environments/worker-turn-launcher.test.ts @@ -649,6 +649,7 @@ describe("worker turn launcher", () => { ownerEpoch: OWNER_EPOCH, }); descriptor = parseWorkerLaunchDescriptor(JSON.parse(command.input ?? "")); + expect(command.transportRetry).toBe("never"); expect(command.argv).toEqual([ "sh", "-c", diff --git a/src/gateway/worker-environments/worker-turn-launcher.ts b/src/gateway/worker-environments/worker-turn-launcher.ts index 9b8d957be737..17c089c6a6a1 100644 --- a/src/gateway/worker-environments/worker-turn-launcher.ts +++ b/src/gateway/worker-environments/worker-turn-launcher.ts @@ -349,6 +349,7 @@ async function executeWorkerTurn(params: { const handoffAbort = new AbortController(); params.onHandoff(); const processPromise = tunnel.runWorkspaceCommand({ + transportRetry: "never", argv: ["sh", "-c", WORKER_LAUNCH_SCRIPT, "openclaw-worker", placement.workerBundleHash], input: JSON.stringify(descriptor), timeoutMs: turn.timeoutMs, diff --git a/src/gateway/worker-environments/workspace-accepted-sync.ts b/src/gateway/worker-environments/workspace-accepted-sync.ts index 00c883d6b35d..0fc3bbbea474 100644 --- a/src/gateway/worker-environments/workspace-accepted-sync.ts +++ b/src/gateway/worker-environments/workspace-accepted-sync.ts @@ -2,8 +2,7 @@ import { createHash, randomBytes } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import type { CommandOptions, SpawnResult } from "../../process/exec.js"; -import { workerSshCommandOptions } from "./ssh.js"; +import type { SpawnResult } from "../../process/exec.js"; import type { WorkerWorkspaceCommand } from "./tunnel-contract.js"; import { serializeWorkerWorkspaceManifest, @@ -20,13 +19,12 @@ import { REMOTE_WORKSPACE_MANIFEST_JS, } from "./workspace-sync-scripts.js"; -const WORKSPACE_TIMEOUT_MS = 10 * 60_000; - export async function recoverAcceptedWorkspacePublication(params: { runWorkspaceCommand: (command: WorkerWorkspaceCommand) => Promise; remoteWorkspaceDir: string; }) { const recovered = await params.runWorkspaceCommand({ + transportRetry: "never", argv: [ "node", "-e", @@ -43,9 +41,7 @@ export async function recoverAcceptedWorkspacePublication(params: { function createAcceptedWorkspacePublisher(params: { runWorkspaceCommand: (command: WorkerWorkspaceCommand) => Promise; - runTask: (argv: string[], options: CommandOptions) => Promise; - ownerSignal: AbortSignal; - rsyncSsh: string; + runRsync: (argv: (rsyncSsh: string) => string[]) => Promise; scpTarget: string; localPath: string; remoteWorkspaceDir: string; @@ -62,6 +58,7 @@ function createAcceptedWorkspacePublisher(params: { throw new Error("Accepted workspace manifest does not match its reference"); } const published = await params.runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", @@ -79,6 +76,7 @@ function createAcceptedWorkspacePublisher(params: { const verifyAcceptedWorkspace = async () => { const verified = await params.runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", @@ -111,6 +109,7 @@ function createAcceptedWorkspacePublisher(params: { const transactionNonce = randomBytes(16).toString("hex"); const transactionCommand = async (action: "apply" | "rollback" | "commit") => await params.runWorkspaceCommand({ + transportRetry: "never", argv: [ "node", "-e", @@ -123,6 +122,7 @@ function createAcceptedWorkspacePublisher(params: { let transactionBegun = false; try { const begun = await params.runWorkspaceCommand({ + transportRetry: "never", argv: [ "node", "-e", @@ -158,25 +158,19 @@ function createAcceptedWorkspacePublisher(params: { const localSource = params.localPath.endsWith(path.sep) ? params.localPath : `${params.localPath}${path.sep}`; - const transferred = await params.runTask( - [ - "rsync", - "--archive", - "--checksum", - "--no-recursive", - "--from0", - `--files-from=${transferListPath}`, - "-e", - params.rsyncSsh, - "--", - localSource, - `${params.scpTarget}:${remoteStagingRoot}/`, - ], - workerSshCommandOptions({ - timeoutMs: WORKSPACE_TIMEOUT_MS, - signal: params.ownerSignal, - }), - ); + const transferred = await params.runRsync((rsyncSsh) => [ + "rsync", + "--archive", + "--checksum", + "--no-recursive", + "--from0", + `--files-from=${transferListPath}`, + "-e", + rsyncSsh, + "--", + localSource, + `${params.scpTarget}:${remoteStagingRoot}/`, + ]); if (!workerWorkspaceCommandSucceeded(transferred)) { throw workspaceSyncError(transferred); } diff --git a/src/gateway/worker-environments/workspace-sync-helpers.ts b/src/gateway/worker-environments/workspace-sync-helpers.ts index c86a0204f4f5..b72b8da80b1e 100644 --- a/src/gateway/worker-environments/workspace-sync-helpers.ts +++ b/src/gateway/worker-environments/workspace-sync-helpers.ts @@ -57,7 +57,10 @@ export function workspaceSyncError(result: SpawnResult): Error { ); } -export function workerWorkspaceRsyncRemoteCommand(prepared: PreparedWorkerSsh): string { +export function workerWorkspaceRsyncRemoteCommand( + prepared: PreparedWorkerSsh, + port = prepared.port, +): string { return workerSshRemoteCommand([ "ssh", ...workerSshOptions(prepared, { forwarding: "disabled" }), @@ -65,13 +68,14 @@ export function workerWorkspaceRsyncRemoteCommand(prepared: PreparedWorkerSsh): "-x", "-T", "-p", - String(prepared.port), + String(port), ]); } export function workerWorkspaceSshArgv( prepared: PreparedWorkerSsh, remoteArgv: readonly string[], + port = prepared.port, ): string[] { return [ "ssh", @@ -80,7 +84,7 @@ export function workerWorkspaceSshArgv( "-x", "-T", "-p", - String(prepared.port), + String(port), "--", prepared.sshTarget, workerSshRemoteCommand(remoteArgv), @@ -97,6 +101,7 @@ async function resolveRemoteWorkspaceBaseManifest( throw new Error("Worker workspace base manifest reference is invalid"); } const resolved = await runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", @@ -137,6 +142,7 @@ export async function verifyRemoteWorkspaceManifest(params: { }): Promise { const expectedDigest = params.expectedRef.slice("sha256:".length); const verified = await params.runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", diff --git a/src/gateway/worker-environments/workspace-sync-transport.ts b/src/gateway/worker-environments/workspace-sync-transport.ts new file mode 100644 index 000000000000..e10e4df4248f --- /dev/null +++ b/src/gateway/worker-environments/workspace-sync-transport.ts @@ -0,0 +1,56 @@ +import type { CommandOptions, SpawnResult } from "../../process/exec.js"; +import { type PreparedWorkerSsh, runWorkerSshCandidates, workerSshCommandOptions } from "./ssh.js"; +import { + runBoundedInboundRsync as runBoundedInboundRsyncTransfer, + workerWorkspaceRsyncRemoteCommand, +} from "./workspace-sync-helpers.js"; + +type WorkerWorkspaceRsyncTransportOptions = { + ownerSignal: AbortSignal; + runTask: (argv: string[], options: CommandOptions) => Promise; + timeoutMs: number; +}; + +/** Runs fresh workspace transfers through the lifecycle's advertised SSH candidates. */ +export function createWorkerWorkspaceRsyncTransport(options: WorkerWorkspaceRsyncTransportOptions) { + const runRsync = async ( + prepared: PreparedWorkerSsh, + argv: (rsyncSsh: string) => string[], + ): Promise => + await runWorkerSshCandidates( + prepared, + options.timeoutMs, + async (port, remainingTimeoutMs) => + await options.runTask( + argv(workerWorkspaceRsyncRemoteCommand(prepared, port)), + workerSshCommandOptions({ + timeoutMs: remainingTimeoutMs, + signal: options.ownerSignal, + }), + ), + ); + + const runBoundedInboundRsync = async (params: { + prepared: PreparedWorkerSsh; + argv: (rsyncSsh: string) => string[]; + destinationRoot: string; + entryLimit: number; + totalByteLimit: number; + }): Promise => + await runWorkerSshCandidates( + params.prepared, + options.timeoutMs, + async (port, remainingTimeoutMs) => + await runBoundedInboundRsyncTransfer({ + argv: params.argv(workerWorkspaceRsyncRemoteCommand(params.prepared, port)), + destinationRoot: params.destinationRoot, + entryLimit: params.entryLimit, + totalByteLimit: params.totalByteLimit, + ownerSignal: options.ownerSignal, + runTask: options.runTask, + timeoutMs: remainingTimeoutMs, + }), + ); + + return { runBoundedInboundRsync, runRsync }; +} diff --git a/src/gateway/worker-environments/workspace-sync.test.ts b/src/gateway/worker-environments/workspace-sync.test.ts new file mode 100644 index 000000000000..02a9f31c6c8f --- /dev/null +++ b/src/gateway/worker-environments/workspace-sync.test.ts @@ -0,0 +1,190 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { CommandOptions, SpawnResult } from "../../process/exec.js"; +import type { PreparedWorkerSsh } from "./ssh.js"; +import { rsyncArgvPort, sshArgvPort } from "./worker-ssh-argv.test-support.js"; +import { createWorkerWorkspaceRsyncTransport } from "./workspace-sync-transport.js"; +import { createWorkerWorkspaceActions } from "./workspace-sync.js"; + +afterEach(() => vi.restoreAllMocks()); + +function result(code = 0): SpawnResult { + return { + stdout: "", + stderr: "", + code, + signal: null, + killed: false, + termination: "exit", + }; +} + +function createPreparedSsh(): PreparedWorkerSsh { + let selectedPort = 2222; + return { + sshTarget: "worker@example.test", + scpTarget: "worker@example.test", + host: "example.test", + advertisedPorts: [2222, 22], + get port() { + return selectedPort; + }, + identityPath: "/identity", + knownHostsPath: "/known-hosts", + selectPort(port) { + selectedPort = port; + }, + dispose: async () => {}, + }; +} + +function createWorkspaceActions( + run: (argv: string[], options: CommandOptions) => Promise, +) { + const prepared = createPreparedSsh(); + return createWorkerWorkspaceActions({ + environmentId: "worker:test", + ownerSignal: new AbortController().signal, + isConnected: () => true, + getPrepared: () => prepared, + runner: { run }, + tasks: new Set(), + }); +} + +describe("worker workspace command transport retry", () => { + it("runs never commands once without changing the selected port", async () => { + const run = vi.fn(async (argv: string[], _options: CommandOptions) => + argv.at(-1)?.includes("never-command") ? result(255) : result(), + ); + const actions = createWorkspaceActions(run); + + await expect( + actions.runWorkspaceCommand({ + transportRetry: "never", + argv: ["printf", "never-command"], + timeoutMs: 777, + }), + ).resolves.toMatchObject({ code: 255, termination: "exit" }); + expect(run).toHaveBeenCalledOnce(); + expect(sshArgvPort(run.mock.calls[0]![0])).toBe(2222); + expect(run.mock.calls[0]![1].timeoutMs).toBe(777); + + await actions.runWorkspaceCommand({ + transportRetry: "idempotent", + argv: ["printf", "selection-probe"], + }); + expect(sshArgvPort(run.mock.calls[1]![0])).toBe(2222); + }); + + it("retries idempotent commands and records the successful port", async () => { + const run = vi.fn(async (argv: string[]) => + argv.at(-1)?.includes("retry-command") && sshArgvPort(argv) === 2222 ? result(255) : result(), + ); + const actions = createWorkspaceActions(run); + + await expect( + actions.runWorkspaceCommand({ + transportRetry: "idempotent", + argv: ["printf", "retry-command"], + }), + ).resolves.toEqual(result()); + expect(run.mock.calls.slice(0, 2).map(([argv]) => sshArgvPort(argv))).toEqual([2222, 22]); + + await actions.runWorkspaceCommand({ + transportRetry: "idempotent", + argv: ["printf", "selected-port-probe"], + }); + expect(sshArgvPort(run.mock.calls[2]![0])).toBe(22); + }); + + it("gives an idempotent fallback only the remaining operation timeout", async () => { + let now = 1_000; + vi.spyOn(Date, "now").mockImplementation(() => now); + const run = vi.fn(async (argv: string[], _options: CommandOptions) => { + if (sshArgvPort(argv) === 2222) { + now += 175; + return result(255); + } + return result(); + }); + const actions = createWorkspaceActions(run); + + await expect( + actions.runWorkspaceCommand({ + transportRetry: "idempotent", + argv: ["printf", "retry-with-deadline"], + timeoutMs: 1_000, + }), + ).resolves.toEqual(result()); + expect(run.mock.calls.map(([, options]) => options.timeoutMs)).toEqual([1_000, 825]); + expect(run.mock.calls[0]![1]).not.toBe(run.mock.calls[1]![1]); + }); +}); + +describe("worker workspace rsync transport retry", () => { + it("gives an outbound fallback only the remaining operation timeout", async () => { + let now = 2_000; + vi.spyOn(Date, "now").mockImplementation(() => now); + const runTask = vi.fn(async (argv: string[], _options: CommandOptions) => { + if (rsyncArgvPort(argv) === 2222) { + now += 200; + return result(255); + } + return result(); + }); + const transport = createWorkerWorkspaceRsyncTransport({ + ownerSignal: new AbortController().signal, + runTask, + timeoutMs: 1_000, + }); + + await expect( + transport.runRsync(createPreparedSsh(), (rsyncSsh) => [ + "rsync", + "-e", + rsyncSsh, + "source", + "worker:destination", + ]), + ).resolves.toEqual(result()); + expect(runTask.mock.calls.map(([, options]) => options.timeoutMs)).toEqual([1_000, 800]); + expect(runTask.mock.calls[0]![1]).not.toBe(runTask.mock.calls[1]![1]); + }); + + it("gives an inbound fallback only the remaining operation timeout", async () => { + const destinationRoot = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-rsync-budget-")); + try { + let now = 3_000; + vi.spyOn(Date, "now").mockImplementation(() => now); + const runTask = vi.fn(async (argv: string[], _options: CommandOptions) => { + if (rsyncArgvPort(argv) === 2222) { + now += 125; + return result(255); + } + return result(); + }); + const transport = createWorkerWorkspaceRsyncTransport({ + ownerSignal: new AbortController().signal, + runTask, + timeoutMs: 1_000, + }); + + await expect( + transport.runBoundedInboundRsync({ + prepared: createPreparedSsh(), + argv: (rsyncSsh) => ["rsync", "-e", rsyncSsh, "worker:source", destinationRoot], + destinationRoot, + entryLimit: 1, + totalByteLimit: 1, + }), + ).resolves.toEqual(result()); + expect(runTask.mock.calls.map(([, options]) => options.timeoutMs)).toEqual([1_000, 875]); + expect(runTask.mock.calls[0]![1]).not.toBe(runTask.mock.calls[1]![1]); + } finally { + await fs.rm(destinationRoot, { recursive: true, force: true }); + } + }); +}); diff --git a/src/gateway/worker-environments/workspace-sync.ts b/src/gateway/worker-environments/workspace-sync.ts index c1122216fc07..589429a99cba 100644 --- a/src/gateway/worker-environments/workspace-sync.ts +++ b/src/gateway/worker-environments/workspace-sync.ts @@ -2,7 +2,7 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import type { CommandOptions, SpawnResult } from "../../process/exec.js"; -import { type PreparedWorkerSsh, workerSshCommandOptions } from "./ssh.js"; +import { type PreparedWorkerSsh, runWorkerSshCandidates, workerSshCommandOptions } from "./ssh.js"; import { WorkerTunnelOwnerDisconnectedError, type WorkerTunnelHandle, @@ -38,13 +38,11 @@ import { probeWorkspaceGitMode, readTransferredManifest, resolveRemoteWorkspaceManifest, - runBoundedInboundRsync as runBoundedInboundRsyncTransfer, stableWorkerPathComponent, validateWorkspaceSyncRequest, verifyRemoteWorkspaceManifest, waitForQuiescenceRenewal, workerWorkspaceCommandSucceeded as success, - workerWorkspaceRsyncRemoteCommand, workerWorkspaceSshArgv, workspaceSyncError, type WorkerWorkspaceActionsOptions, @@ -61,6 +59,7 @@ import { REMOTE_WORKSPACE_MANIFEST_JS, REMOTE_WORKSPACE_SETUP_SCRIPT, } from "./workspace-sync-scripts.js"; +import { createWorkerWorkspaceRsyncTransport } from "./workspace-sync-transport.js"; const REMOTE_SETUP_TIMEOUT_MS = 20_000; const WORKSPACE_TIMEOUT_MS = 10 * 60_000; @@ -100,31 +99,38 @@ export function createWorkerWorkspaceActions( const runTask = (argv: string[], commandOptions: CommandOptions): Promise => track(options.runner.run(argv, commandOptions)); - const runBoundedInboundRsync = async (params: { - argv: string[]; - destinationRoot: string; - entryLimit: number; - totalByteLimit: number; - }): Promise => { - return await runBoundedInboundRsyncTransfer({ - ...params, - ownerSignal: options.ownerSignal, - runTask, - timeoutMs: WORKSPACE_TIMEOUT_MS, - }); - }; + const { runBoundedInboundRsync, runRsync } = createWorkerWorkspaceRsyncTransport({ + ownerSignal: options.ownerSignal, + runTask, + timeoutMs: WORKSPACE_TIMEOUT_MS, + }); const runWorkspaceCommand = async (command: WorkerWorkspaceCommand): Promise => { const prepared = requirePrepared(); - return await runTask( - workerWorkspaceSshArgv(prepared, command.argv), - workerSshCommandOptions({ - input: command.input, - timeoutMs: command.timeoutMs ?? WORKSPACE_TIMEOUT_MS, - signal: command.signal - ? AbortSignal.any([options.ownerSignal, command.signal]) - : options.ownerSignal, - }), + const timeoutMs = command.timeoutMs ?? WORKSPACE_TIMEOUT_MS; + const signal = command.signal + ? AbortSignal.any([options.ownerSignal, command.signal]) + : options.ownerSignal; + // Exit 255 does not prove whether the remote command was accepted, so stateful + // commands must stay pinned to one transport attempt. + if (command.transportRetry === "never") { + return await runTask( + workerWorkspaceSshArgv(prepared, command.argv), + workerSshCommandOptions({ input: command.input, timeoutMs, signal }), + ); + } + return await runWorkerSshCandidates( + prepared, + timeoutMs, + async (port, remainingTimeoutMs) => + await runTask( + workerWorkspaceSshArgv(prepared, command.argv, port), + workerSshCommandOptions({ + input: command.input, + timeoutMs: remainingTimeoutMs, + signal, + }), + ), ); }; @@ -133,6 +139,7 @@ export function createWorkerWorkspaceActions( throw new Error("Worker workspace quiescence path must be absolute"); } const result = await runWorkspaceCommand({ + transportRetry: "never", argv: [ "node", "-e", @@ -158,6 +165,7 @@ export function createWorkerWorkspaceActions( const renew = (validationMode: "heartbeat" | "final") => { const operation = renewalQueue.then(async () => { const renewedResult = await runWorkspaceCommand({ + transportRetry: "never", argv: [ "node", "-e", @@ -218,6 +226,7 @@ export function createWorkerWorkspaceActions( renewalAbort.abort(); await renewalLoop; const resumedResult = await runWorkspaceCommand({ + transportRetry: "never", argv: ["node", "-e", REMOTE_WORKSPACE_RESUME_JS, remoteWorkspaceDir, nonce], }); if (!success(resumedResult)) { @@ -242,6 +251,7 @@ export function createWorkerWorkspaceActions( String(request.generation), ].join("/"); const setup = await runWorkspaceCommand({ + transportRetry: "never", argv: ["sh", "-s", "--", remoteRelative], input: REMOTE_WORKSPACE_SETUP_SCRIPT, }); @@ -261,7 +271,6 @@ export function createWorkerWorkspaceActions( const temporaryDirectory = await fs.mkdtemp( path.join(os.tmpdir(), "openclaw-worker-workspace-sync-"), ); - const rsyncSsh = workerWorkspaceRsyncRemoteCommand(prepared); try { let fileListPath: string | undefined; if (mode === "git") { @@ -369,22 +378,16 @@ export function createWorkerWorkspaceActions( signal: options.ownerSignal, timeoutMs: WORKSPACE_TIMEOUT_MS, }); - const packTransfer = await runTask( - [ - "rsync", - "--archive", - "--checksum", - "-e", - rsyncSsh, - "--", - packPath, - `${prepared.scpTarget}:${remoteWorkspaceDir}/${REMOTE_GIT_PACK_NAME}`, - ], - workerSshCommandOptions({ - timeoutMs: WORKSPACE_TIMEOUT_MS, - signal: options.ownerSignal, - }), - ); + const packTransfer = await runRsync(prepared, (rsyncSsh) => [ + "rsync", + "--archive", + "--checksum", + "-e", + rsyncSsh, + "--", + packPath, + `${prepared.scpTarget}:${remoteWorkspaceDir}/${REMOTE_GIT_PACK_NAME}`, + ]); if (!success(packTransfer)) { throw workspaceSyncError(packTransfer); } @@ -401,6 +404,7 @@ export function createWorkerWorkspaceActions( }), ); const seeded = await runWorkspaceCommand({ + transportRetry: "never", argv: [ "sh", "-s", @@ -419,30 +423,25 @@ export function createWorkerWorkspaceActions( } const localSource = gitRoot.endsWith(path.sep) ? gitRoot : `${gitRoot}${path.sep}`; - const transfer = await runTask( - [ - "rsync", - "--archive", - "--checksum", - "--exclude=.git", - ...DERIVED_WORKSPACE_RSYNC_EXCLUDES.map((pattern) => `--exclude=${pattern}`), - ...(fileListPath ? ["--recursive", "--from0", `--files-from=${fileListPath}`] : []), - "-e", - rsyncSsh, - "--", - localSource, - `${prepared.scpTarget}:${remoteWorkspaceDir}/`, - ], - workerSshCommandOptions({ - timeoutMs: WORKSPACE_TIMEOUT_MS, - signal: options.ownerSignal, - }), - ); + const transfer = await runRsync(prepared, (rsyncSsh) => [ + "rsync", + "--archive", + "--checksum", + "--exclude=.git", + ...DERIVED_WORKSPACE_RSYNC_EXCLUDES.map((pattern) => `--exclude=${pattern}`), + ...(fileListPath ? ["--recursive", "--from0", `--files-from=${fileListPath}`] : []), + "-e", + rsyncSsh, + "--", + localSource, + `${prepared.scpTarget}:${remoteWorkspaceDir}/`, + ]); if (!success(transfer)) { throw workspaceSyncError(transfer); } const manifest = await runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", @@ -489,12 +488,9 @@ export function createWorkerWorkspaceActions( const manifestRoot = path.join(temporaryDirectory, "manifests"); const baseManifestPath = path.join(manifestRoot, `${baseDigest}.json`); const transferListPath = path.join(temporaryDirectory, "transfer-list"); - const rsyncSsh = workerWorkspaceRsyncRemoteCommand(prepared); const acceptedWorkspacePublisher = createAcceptedWorkspacePublisherFactory({ runWorkspaceCommand, - runTask, - ownerSignal: options.ownerSignal, - rsyncSsh, + runRsync: async (argv) => await runRsync(prepared, argv), scpTarget: prepared.scpTarget, localPath: request.localPath, remoteWorkspaceDir: request.remoteWorkspaceDir, @@ -503,7 +499,8 @@ export function createWorkerWorkspaceActions( await fs.mkdir(stagingRoot, { mode: 0o700 }); await fs.mkdir(manifestRoot, { mode: 0o700 }); const baseManifestTransfer = await runBoundedInboundRsync({ - argv: [ + prepared, + argv: (rsyncSsh) => [ "rsync", "--archive", "--no-recursive", @@ -541,6 +538,7 @@ export function createWorkerWorkspaceActions( expectedRef, }); const currentResult = await runWorkspaceCommand({ + transportRetry: "idempotent", argv: [ "node", "-e", @@ -600,7 +598,8 @@ export function createWorkerWorkspaceActions( const currentDigest = currentRef.slice("sha256:".length); const currentManifestPath = path.join(manifestRoot, `${currentDigest}.json`); const currentManifestTransfer = await runBoundedInboundRsync({ - argv: [ + prepared, + argv: (rsyncSsh) => [ "rsync", "--archive", "--no-recursive", @@ -633,7 +632,8 @@ export function createWorkerWorkspaceActions( mode: 0o600, }); const resultTransfer = await runBoundedInboundRsync({ - argv: [ + prepared, + argv: (rsyncSsh) => [ "rsync", "--archive", "--checksum", diff --git a/src/plugins/capability-provider.types.ts b/src/plugins/capability-provider.types.ts index d0277dcd63e4..ab0d1a534928 100644 --- a/src/plugins/capability-provider.types.ts +++ b/src/plugins/capability-provider.types.ts @@ -52,6 +52,8 @@ export type WorkerProfile = Readonly>; export type WorkerSshEndpoint = { host: string; port: number; + /** Up to 10 ordered unique integer ports (1..65535) after `port`; excludes the primary. */ + fallbackPorts?: readonly number[]; user: string; /** OpenSSH public host-key line obtained from trusted provisioning output. */ hostKey: string; diff --git a/src/state/openclaw-database-maintenance.test.ts b/src/state/openclaw-database-maintenance.test.ts index 98dc311423b3..619cf3e66ab7 100644 --- a/src/state/openclaw-database-maintenance.test.ts +++ b/src/state/openclaw-database-maintenance.test.ts @@ -149,6 +149,40 @@ describe("OpenClaw database maintenance schema validation", () => { } }); + it("allows the lazy worker SSH fallback table to be absent but rejects drift", () => { + const database = createGlobalDatabase(); + try { + const canonicalTable = database + .prepare("SELECT sql FROM sqlite_schema WHERE type = 'table' AND name = ?") + .get("worker_environment_ssh_fallback_ports") as { sql?: unknown } | undefined; + if (typeof canonicalTable?.sql !== "string") { + throw new Error("missing canonical worker SSH fallback port table"); + } + database.exec("DROP TABLE worker_environment_ssh_fallback_ports;"); + + expect(() => + assertOpenClawStateDatabaseForMaintenance(database, { + pathname: "global.sqlite", + }), + ).not.toThrow(); + + const driftedTableSql = canonicalTable.sql.replace( + " PRIMARY KEY (environment_id, position)", + " unexpected TEXT,\n PRIMARY KEY (environment_id, position)", + ); + expect(driftedTableSql).not.toBe(canonicalTable.sql); + database.exec(driftedTableSql); + + expect(() => + assertOpenClawStateDatabaseForMaintenance(database, { + pathname: "global.sqlite", + }), + ).toThrow("column definitions differ for worker_environment_ssh_fallback_ports"); + } finally { + database.close(); + } + }); + it("rejects a current agent database with a missing canonical table", () => { const database = createAgentDatabase(); try { diff --git a/src/state/openclaw-state-db-contract.ts b/src/state/openclaw-state-db-contract.ts index 9563ac5a086b..27a93bc965f0 100644 --- a/src/state/openclaw-state-db-contract.ts +++ b/src/state/openclaw-state-db-contract.ts @@ -20,6 +20,7 @@ export const LAZY_ADDITIVE_STATE_TABLES = [ "skill_workshop_proposal_origin_runs", "skill_workshop_proposal_rollbacks", "skill_workshop_proposals", + "worker_environment_ssh_fallback_ports", ] as const; export const LAZY_ADDITIVE_STATE_INDEXES = [...FIRST_USE_STATE_INDEXES] as const; /** Maximum time one synchronous SQLite call may wait for a lock. */ diff --git a/src/state/openclaw-state-db.generated.d.ts b/src/state/openclaw-state-db.generated.d.ts index 3e90f8cd3393..53f92e6445d5 100644 --- a/src/state/openclaw-state-db.generated.d.ts +++ b/src/state/openclaw-state-db.generated.d.ts @@ -1409,6 +1409,12 @@ export interface WorkerEnvironmentCredentials { session_id: string | null; } +export interface WorkerEnvironmentSshFallbackPorts { + environment_id: string; + port: number; + position: number; +} + export interface WorkerEnvironments { attached_session_ids_json: Generated; bootstrap_bundle_hash: string | null; @@ -1673,6 +1679,7 @@ export interface DB { web_push_subscriptions: WebPushSubscriptions; web_push_vapid_keys: WebPushVapidKeys; worker_environment_credentials: WorkerEnvironmentCredentials; + worker_environment_ssh_fallback_ports: WorkerEnvironmentSshFallbackPorts; worker_environments: WorkerEnvironments; worker_inference_turns: WorkerInferenceTurns; worker_session_placements: WorkerSessionPlacements; diff --git a/src/state/openclaw-state-db.test.ts b/src/state/openclaw-state-db.test.ts index 44377c5ee121..3f2dca5a4f0f 100644 --- a/src/state/openclaw-state-db.test.ts +++ b/src/state/openclaw-state-db.test.ts @@ -15,6 +15,7 @@ import { import { requireNodeSqlite } from "../infra/node-sqlite.js"; import { listOpenFileDescriptorsForPath } from "../infra/open-file-descriptors.test-support.js"; import { readSqliteNumberPragma } from "../infra/sqlite-pragma.test-support.js"; +import { assertSqliteSchemaContains } from "../infra/sqlite-schema-contract.js"; import { loadTaskRegistryStateFromSqlite } from "../tasks/task-registry.store.sqlite.js"; import { withOpenClawTestState } from "../test-utils/openclaw-test-state.js"; import { VERSION } from "../version.js"; @@ -39,6 +40,7 @@ import { withOpenClawStateStartupMigrationCheckpointDatabase, } from "./openclaw-state-db.js"; import { resolveOpenClawStateSqlitePath } from "./openclaw-state-db.paths.js"; +import { OPENCLAW_STATE_SCHEMA_SQL } from "./openclaw-state-schema.js"; import { collectSqliteSchemaShape, createSqliteSchemaShapeFromSql, @@ -68,6 +70,19 @@ function createInitialStateSchemaShape() { return shape; } +function createOlderV6StateSchemaWithoutWorkerSshFallbackPorts(): string { + const startMarker = "CREATE TABLE IF NOT EXISTS worker_environment_ssh_fallback_ports ("; + const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(startMarker); + const endMarker = "\n) STRICT;"; + const end = start >= 0 ? OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start) : -1; + if (start < 0 || end < 0) { + throw new Error("worker SSH fallback port schema block is missing"); + } + return `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice( + end + endMarker.length, + )}`; +} + function expectStateSchemaMigrationRequired( run: () => unknown, expected: { @@ -1120,6 +1135,27 @@ describe("openclaw state database", () => { ).toThrow(); }); + it("keeps the additive worker SSH fallback table compatible with older v6 containment", () => { + const database = openMaterializedCurrentStateDatabase(); + try { + expect( + database + .prepare("SELECT name FROM sqlite_schema WHERE type = 'table' AND name = ?") + .get("worker_environment_ssh_fallback_ports"), + ).toEqual({ name: "worker_environment_ssh_fallback_ports" }); + expect(() => + assertSqliteSchemaContains( + database, + "older v6 state database", + createOlderV6StateSchemaWithoutWorkerSshFallbackPorts(), + { allowedMissingTables: FIRST_USE_STATE_TABLES }, + ), + ).not.toThrow(); + } finally { + database.close(); + } + }); + it("skips exclusive repair when the automatic schema gate is already current", () => { const stateDir = createTempStateDir(); const options = { env: { OPENCLAW_STATE_DIR: stateDir } }; diff --git a/src/state/openclaw-state-schema.sql b/src/state/openclaw-state-schema.sql index a111ae9a9358..07f1414758dd 100644 --- a/src/state/openclaw-state-schema.sql +++ b/src/state/openclaw-state-schema.sql @@ -1883,6 +1883,17 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_worker_environments_provider_lease ON worker_environments(provider_id, lease_id) WHERE lease_id IS NOT NULL; +-- Provider-advertised fallback ports preserve stable retry order separately +-- from the downgrade-sensitive canonical worker environment row. +CREATE TABLE IF NOT EXISTS worker_environment_ssh_fallback_ports ( + environment_id TEXT NOT NULL, + position INTEGER NOT NULL CHECK (position >= 0 AND position <= 9), + port INTEGER NOT NULL CHECK (port >= 1 AND port <= 65535), + PRIMARY KEY (environment_id, position), + UNIQUE (environment_id, port), + FOREIGN KEY (environment_id) REFERENCES worker_environments(environment_id) ON DELETE CASCADE +) STRICT; + -- Session placement lives in the shared state database so local admission, -- worker admission, and environment attachment use one durable authority. CREATE TABLE IF NOT EXISTS worker_session_placements (