openclaw/docs/plan/runners.md

summary, title, read_when

summary	title	read_when
Everything is a node — one placement model where paired machines and cloud boxes host sessions through the worker admission path; clients attach to sessions, never to runners.	Runners plan	Designing or reviewing where sessions run (gateway, device, cloud) Changing the Where picker, device pairing, node onboarding, or worker dispatch surfaces Naming anything around sessions, devices, nodes, or placement

Status

Proposal, revision 2. Supersedes revision 1 in place (2026-08-11, operator decision). Implementation in progress; update this table in every PR that advances a milestone.

#	Milestone	Status	PRs
0	This plan (revision 2)	landed	#122454
1a	Naming: session copy revert	landed	#120667
1b	Naming: devices consolidation	landed	#120689
1c	Cleanup: node-pairing → device-pairing merge	landed	#120726
2	openclaw resume + web Continue in terminal	in progress	#120664
3	openclaw connect one-paste onboarding + /j/ join route	in progress	#120768, #122499
4	Picker: grouping, placement, liveness, enrichment	in progress	#120804, #122531, #122635, #122774
F	Real-wire session boundary harness	landed	#121212
5	Public worker ingress path	landed	#122578, #122643
6	Node worker provider (device runners)	in progress	#122683, #122769, #122829
7	Bundle push consent + runner updates	not started	—
8	Stop-and-continue moves	not started	—
9	Deletions (ssh sandbox, openshell, exec-host clones, …)	not started	—
10	Cloud convergence (provisioners run openclaw connect)	not started	—

Revision history: revision 1 (2026-08-08) established the session/runner vocabulary, the naming rulings, and the milestone skeleton after a code-evidence investigation and three adversarial reviews. Revision 2 (2026-08-11) follows a second round of deep code reads (worker admission, tunnel, sync, node channel, scope model), an industry survey (GitHub/GitLab/ Buildkite/CircleCI runners, Tailscale, VS Code tunnels, Coder, Gitpod Flex, Amp, Cursor/Claude/Codex cloud), a static teardown of Amp's runner transport, and a fresh adversarial review of this revision. The operator decisions that changed the plan:

• Nodes host sessions. Revision 1's "no turn loops on the node role" non-goal is overridden as a conclusion while its facts stand: the node connection is still not an authority boundary, so session-hosting authority lives in the dispatch layer (worker admission, per-dispatch credentials, turn claims, owner epochs) — relocated, not removed.
• openclaw worker becomes a node-supervised child. One machine concept: a paired node can run everything a cloud worker runs today.
• SSH is not the device transport. The gateway never dials devices; the device always dials out. Revision 1's "ship sshd first" for device runners is deleted — it cannot reach a NAT'd machine and no surveyed product uses SSH as control transport. SSH remains only as the legacy cloud-lease transport until milestone 10 retires it.

Problem

Unchanged from revision 1 in substance: OpenClaw has disconnected answers to "where does work run." Nodes receive forwarded exec host=node calls only; a user's always-on workstation is less capable as a session host than a throwaway cloud lease. Cloud workers host full sessions with a durable placement state machine, but only against ephemeral SSH-provisioned leases. The ssh sandbox backend is a third remote-execution path. Placement is chosen once from a flat list mixing ontologies, then becomes invisible; onboarding a new machine takes flags, env vars, and two manual approvals.

The bar, stated as product: an admin clicks "Connect a machine…" in the web picker, pastes one command on any machine, and seconds later that machine is visible in the picker for the whole team and can host full agent sessions.

Model and vocabulary

Session   gateway-owned: transcript, identity, placement, managed worktree.
          Clients (web, TUI, macOS app, channels) attach to sessions,
          never to runners. One noun, everywhere: session.
Node      a paired machine holding an outbound connection to the gateway
          (Ed25519 device identity). Protocol/internal vocabulary; user-facing
          copy says "device". EVERY remote machine is a node — personal
          workstations, servers, cloud leases. Phones are nodes that never
          advertise session hosting.
Runner    anything that can host a session's turn loop: the gateway itself,
          or a session-capable node. "Runner" is internal/docs vocabulary;
          UI copy says "Runs on …".
Worker    the per-turn child process (`openclaw worker`) that hosts a
          session's loop under worker admission. On cloud leases it is
          launched over SSH today; on nodes it is a supervised child of the
          node host. Same admission, same protocol, either way.
Isolation a property OF the runner (none | docker | podman), not a place.
Project   repo identity: normalized remote.origin.url, with the existing
          16-char repo fingerprint as the no-remote fallback. Derived,
          never registered.
Checkout  project × runner = { runnerId, path }.
Turn      one prompt-to-response work attempt inside a session.

Naming rulings (operator-decided, carried from revision 1): session is the only product noun for a conversation; devices is the user-facing word for paired hardware; new CLI ergonomics ship as verbs (openclaw resume, openclaw connect); "runner" never appears in UI copy. Milestone 1c (nodes → devices route/i18n consolidation) lands before any new placement copy ships.

Architecture

The two-connection shape

Every surveyed production system (GitHub Actions runners, GitLab, Buildkite, CircleCI, Tailscale, VS Code tunnels, Coder, Gitpod, Amp) uses outbound-only connections from the machine to the control plane, and the mature ones split a persistent presence/control channel from per-job work channels. OpenClaw already has both halves; this plan connects them:

1. Node connection (exists): the outbound gateway WebSocket. Carries identity, presence, capability manifest, and bounded command invocation (node.invoke). This is the control channel: registration, liveness, and the transport for workspace operations.
2. Worker connection (exists): the per-dispatch WebSocket speaking the closed worker protocol (heartbeat, transcript CAS commits, resumable live events, gateway-proxied inference, gateway-side session tools). Admission is store-backed and transport-free: per-dispatch 32-byte credential (10-minute TTL, hashed at rest), environment binding, owner epochs, exact bundle hash, per-RPC identity revalidation. On a node runner the worker child dials the gateway's public TLS endpoint directly — a connected node proves the outbound path exists.

What is deliberately NOT the transport: node.invoke as a byte pipe for the worker connection. Measured constraints (16 KiB string chunks, an awaited RPC round-trip per chunk, no idempotency dedupe, reconnect kills in-flight invokes, one-session-per-nodeId eviction, 50 MB buffer hard-close) make it unsuitable for hours-long streams. It stays what it is: a bounded command channel.

Worker ingress on the public endpoint (milestone 5)

Today the worker ingress is a dedicated loopback-only listener reached via ssh -R; the main ingress rejects worker frames. For node runners the same admission is exposed on a path-tagged upgrade route on the public TLS endpoint (connectionKind = "worker" forced by route instead of listener). The loopback listener stays for SSH-provisioned cloud workers until milestone 10.

Hardening that ships with the exposure, not after it:

• Admission failures collapse into one opaque reason. The current invalid-credential vs environment-mismatch distinction is an environment-id enumeration oracle and must not be publicly observable.
• The worker path shares the gateway's preauth budgets and rate limits; a pre-credential connection gets the same cheap rejection as any other unauthenticated client.
• Credential strength is already sufficient (32 random bytes, constant-time hashed compare, 10-minute TTL, single environment binding).

Node worker provider (milestone 6)

WorkerLease grows a union: { ssh: … } | { node: { deviceId } }. The admission/placement machinery (environment store, credential broker, placement state machine, turn claims, transcript/live-event/inference protocols) is reused unchanged — that is the hard-won part. What is net-new, stated honestly (revision 1 undersold this):

• Node tunnel handle. A second WorkerTunnelHandle implementation: runWorkspaceCommand maps to a bounded node command (argv + stdin → SpawnResult; the remote-side sync/manifest/quiesce scripts already ship in the bundle and are transport-agnostic). remoteSocketPath is replaced by the descriptor carrying the gateway worker URL.
• Durable launch. In the SSH flow the launch exec stream is the worker lifetime and its death destroys the environment. On a node, launch is a supervised node-host command: the node host spawns the worker child decoupled from the invoke lifetime, persists the one-line result, and the gateway re-collects it idempotently. A node WS blip must not kill a turn.
• Credential delivery. The launch descriptor (including the per-turn credential) travels over the authenticated node channel instead of SSH stdin. Same trust domain: the node host is the machine-side agent either way.
• Workspace sync without rsync. Manifest-driven delta blob transfer over authenticated HTTPS against the gateway (the manifest machinery already computes exact changed-blob lists; rsync was only the carrier), with git-mode base fetch from origin when the project has one. Existing bounds (inventory entries, manifest bytes, reconcile caps) carry over. Nodes with an advertised local checkout skip gateway push entirely (the Amp model: runner identity = host + workdir + repo).
• Persistent-machine lifecycle. destroy = logical lease release. Provider inspect is tri-state against pairing + presence: present, dormant (paired but offline, within a dormancy ceiling — must NOT be driven to orphaned by the reconcile sweep), gone (unpaired or ceiling elapsed → normal orphan/reap path). A device-environment reaper keyed on unpair/dormancy — not on provider teardown proof — cleans rows, credentials, and staged refs. Device-side GC of per-session workspace dirs and superseded bundles is a milestone exit gate, not an open question: persistent machines otherwise leak the user's own disk.
• Placement runner-offline. Heartbeat/presence loss marks the placement with a recorded, operator-visible reason; staged results are preserved by the existing fence machinery; the session offers "continue on gateway" (reclaim) or "wait for device". Never a silent non-outcome.
• Dispatch target union. sessions.dispatch accepts { profileId } | { deviceId }; the device → environment mapping resolves server-side. Devices are not smuggled through synthesized cloudWorkers.profiles entries.
• Concurrency slots. A node declares a session-slot count (default small); the picker shows busy state; a dispatch that no live runner can satisfy fails visibly after a bounded wait instead of queuing forever.
• Multi-gateway safety. The worker install/workspace root on a node is namespaced by gateway identity so two gateways pairing one machine cannot corrupt each other's state.

Isolation on node runners: optional worker-in-docker/podman, same sandbox axis as gateway-local sessions. Cloud leases keep full-permission-within-the- box (the machine is the boundary).

Trust model (operator-decided, v1)

Cloud workers run full-permission because the box is disposable and credential-free. A paired personal machine is neither. The v1 resolution:

• Only admins pair nodes (already enforced: role: node device approval requires operator.admin; the join-code mint is admin-scoped). Pairing a node is the admin declaring it shared team infrastructure — a server, a build box, a dedicated workstation. That is the consent boundary for "everyone on the gateway may dispatch to it and session content lands on it."
• Personal-device runners are out of scope for v1. They arrive together with per-person node ownership (visibility + dispatch policy keyed on a recorded owner), not before. Approver identity is recorded at pairing time from day one as provenance, never authorization (additive nullable column), so the later policy has data to stand on.
• Phones and low-trust devices never advertise session hosting. Capability gating, not ontology: the picker never offers them.
• Non-interactive approval side doors (trusted-CIDR, SSH-verify, trusted-proxy browser auto-approve) remain scoped to their current presence-level grants and are reviewed for the hosted-gateway class; none may mint a session-capable node without an admin.
• Inference stays gateway-proxied; provider keys never reach nodes. If nodes ever fetch private repos from origin directly, the gateway mints short-lived scoped git credentials per dispatch; no standing PATs on nodes.

Onboarding (milestone 3)

Copying the industry-standard split (short-lived enrollment secret → long-lived device identity; GitLab deprecated reusable registration tokens to get here, Tailscale's key/device revocation split is the documented model):

• Admin mints a single-use, ~10-minute join code (≥128-bit entropy) from the picker's "Connect a machine…" foot or openclaw devices CLI. The existing device.pair.setupCode RPC and node bootstrap profile are the substrate; the code pre-approves exactly the node role with zero operator scopes.
• The pasted one-liner is npx openclaw connect <url-or-code> (top-level verb; openclaw node run stays as the plumbing command). It accepts the full oc-pair:// payload (offline form, carries gateway URL + bootstrap token + optional TLS pin for self-signed gateways) or an https://<gateway-host>/j/<shortcode> URL whose payload is fetched over TLS. --service installs the OS service instead of running foreground. A curl installer wrapper on the public website installs the CLI and execs the same verb; the public site never sees tokens.
• The gateway serves /j/<shortcode> (reserved prefix in Control UI routing, single-use burn, strict per-IP rate limiting).
• Revocation split, documented: revoking a join code never unpairs nodes; removing/banning a node is a first-class devices-page action that also fences in-flight placements. Node auto-cleanup after a long dead period mirrors runner-industry practice.

Bundle and updates (milestone 7)

Exact-hash admission stays. The pinned, content-hashed bundle is pushed to the node over the already-authenticated paired channel. Consent is split so it cannot rot into approval fatigue or silent surprise:

• Consent to be a runner: one-time, per-device, at pairing/enablement.
• Consent to run a build: satisfied by the channel — bundles arrive only from the gateway this admin paired, and updates on dispatch are the normal managed-runner behavior (GitHub runners self-update the same way). The devices page shows the installed runner version; the gateway refuses dispatch to stale nodes with a doctor-style hint instead of failing silently.

Projects read model (milestone 4 foundation)

OpenClaw already computes project identity twice without naming it: the worktree service derives originUrl + a 16-char repo fingerprint (src/agents/worktrees/service.ts:199-205), and the sessions catalog groups Codex/Claude rows by project folder, folding .claude/worktrees/<name> into its origin repo. This component promotes that to a first-class observed read model alongside the registered projects already returned by projects.list, following the same computed pattern as environments.list:

• projects.list.observedProjects read model (computed for write-capable callers, no new store): group known checkouts by repo fingerprint → { name, originUrl, checkouts: [{runnerId, path}], lastUsedAt }. Sources: session rows (execCwd/execNode) and the managed-worktree registry. The observed paths and sanitized origins are returned only to operator.write callers; read-only callers keep the registered project catalog and project-only recents. Device-advertised checkouts remain milestone 6 work.

UI (milestone 4)

Revision 1's design rule stands: normal state is silent; only exceptions speak. Additions:

• Use the existing environment type discriminant for picker grouping: local gateway, connected execution-capable nodes, worker environments, and the separate cloud profiles list. sessionHost is deferred to milestone 6, where device runners introduce the capability fact that needs it.
• Where picker regrouped (ui/src/pages/new-session/place-picker.ts): sections "This gateway" / "Devices" / "Cloud". Device rows intersect the environment catalog with connected, execution-capable nodes; cloud profiles remain their separate list. Folder and destination stay orthogonal.
• Placement chip on the session header: shows quiet current placement; active cloud placements reclaim through sessions.reclaim with "Bring home". Stop-and-continue moves arrive with milestone 8.
• Remaining milestone work: live presence and pairing subscriptions, the admin-gated "Connect a machine…" foot, busy and never-connected states, and additive EnvironmentSummary platform, session-host, trust, and runner version facts. runner-offline then shows a banner with the recorded reason and its recovery verbs.

Cloud convergence (milestone 10)

A cloud provider's job collapses to: boot box, run openclaw connect <one-shot code> --ephemeral in setup. Ephemeral enrollment (industry: GitHub --ephemeral/JIT, Buildkite --acquire-job, Tailscale ephemeral keys) auto-deregisters after the run and auto-purges the node record when it goes offline. destroy = release lease. After soak, the SSH reverse-tunnel stack, PreparedWorkerSsh, and the rsync transport are deleted; cloud leases and paired machines become the same runner with different lifecycles.

What the adversarial reviews killed or reshaped

Carried forward from revision 1 (still true): no Places registry (environments.list stays the read model, enriched additively); no dispatch into a live checkout without exclusive ownership; exec host=node stays untouched (different product, different policy domain); no sandbox-as-a-place picker row; no fake mobility verbs; no live migration; no multi-gateway federation; no phones as runners.

Revised or new in revision 2:

• Revision 1's "device runners are the existing worker stack with essentially no changes" was overstated: admission, placement, claims, stores, and the worker protocols are reused; transport, credential delivery, sync carrier, and launch durability are net-new. Scope milestone 6 accordingly.
• Revision 1's "ship sshd first" transport is deleted (unreachable target machines; industry-divergent).
• "Everyone dispatches" is bounded by the trust model above — shared infrastructure only, until per-person ownership ships.
• The node.invoke byte-pipe idea (this revision's own first draft) was killed by measured protocol constraints; the direct-dial worker connection replaced it.

Prior art (what we copy, what we skip)

• Amp (verified by static CLI teardown + manual): outbound WSS only via actor framework; per-user control channel carries registration, heartbeat, presence, and dispatch intents in heartbeat responses; per-thread WS for live sessions; agent loop local on the runner in an existing checkout (no file sync; identity = host + workdir + repo URL); inference centralized server-side; per-workdir PID claim prevents double-serving. We copy the two-channel shape, dispatch-over-control-channel, and checkout advertisement; we keep inference gateway-proxied (their centralization is a billing choice, not architecture); we scope enrollment tighter than their single long-lived API key.
• GitHub Actions runners: registration token → device keypair; JIT/ ephemeral single-job runners; self-update with a staleness ceiling and dispatch refusal; blunt security docs about persistent runners running untrusted code. All copied in spirit above.
• Tailscale: auth-key vs node-key split and the revocation split warning. Copied, documented.
• VS Code tunnels: the gold-standard enrollment UX (run one command, browser confirms); device-code-style confirmation is a candidate alternative to pasted codes later. Their 10-tunnel account cap validates bounded per-gateway node counts.
• Coder / Gitpod Flex: control/data plane split with customer-side execution and orchestration-only control plane — the closest analog to "inference on gateway, execution on node," validating it as a coherent residency story. Gitpod's ~30s registration renewal is the liveness-lease reference if presence needs tightening.
• Cursor / Claude Code / Codex cloud: managed-VM-only execution with git-based handoff; Claude Code's proxy-minted scoped git credentials inform the scoped-git-token rule above; teleport-style continuation validates attach-only sessions (which OpenClaw gets for free).

Milestones

Independently mergeable PR series; 3–5 can interleave after 1c.

1. 1c naming cleanup: finish nodes → devices in route ids, i18n keys, labels; node-pairing.ts facade merge. Before any new placement copy.
2. Continuation ergonomics (in progress): openclaw resume, web "Continue in terminal".
3. openclaw connect: verb + oc-pair:// decoder + TLS pin in payload + /j/<shortcode> join route (reserved prefix, single-use, rate-limited) + shortcode mint + curl wrapper on the public site. Exit: a fresh machine pairs against a remote gateway with one pasted command and one admin click, no manual approval steps.
4. Picker (in progress): regrouped sections, quiet placement + reclaim, and the observed projects read model land first; live presence subscription, the admin-gated "Connect a machine…" foot, additive EnvironmentSummary enrichment, and never-connected vs lost states complete the milestone.
5. Public worker ingress: path-tagged worker upgrade on the main TLS endpoint; opaque admission failure; shared preauth budgets. Exit: a worker process on any internet host with a valid dispatch credential completes admission; invalid attempts are cheap and unenumerable.
6. Node worker provider: lease union, dispatch target union, node tunnel handle, durable supervised launch, HTTPS delta sync + origin fetch, tri-state inspect + reaper + GC, concurrency slots, runner-offline placement semantics, gateway-namespaced install root, approver-provenance column. Fault-injection tests gate exit: device sleep mid-turn, node WS blip mid-turn (turn survives), gateway restart with offline device, credential expiry, slot saturation, dispatch-with-no-live-runner timeout.
7. Bundle push + updates: consent split, push over paired channel, version surfacing, stale-node dispatch refusal.
8. Stop-and-continue moves: drain + reclaim + re-dispatch to another runner, reusing the migration barrier.
9. Deletions: ssh sandbox backend + remote-fs bridge (~2.35k LOC), openshell overlap (~3.4k LOC, verify usage first), exec-host structural clones (~3k of ~5k LOC), one-shot agent.cli.claude.run node path (superseded by full session hosting), node/device pairing merge remainder. Each gated on its replacement, each its own PR with proof.
10. Cloud convergence: --ephemeral enrollment, provisioners run openclaw connect, then delete the SSH tunnel/rsync transport stack.

Net production LOC across the plan is targeted negative: milestones 3–5 are small additions, 6–7 are mostly a provider + one transport implementation against reused machinery, and 9–10 delete more than everything before them adds.

Open questions

• Dormancy ceiling default (how long a sleeping device stays dormant before its environments reap) — proposal: 14 days, config-free, revisit with usage.
• Slot count default for node runners — proposal: 2 for interactive-class devices, higher for server-class; needs a capability signal or a connect flag.
• Device-code-style browser confirmation (VS Code model) as an alternative to pasted codes — later, once /j/ exists.
• Repo-owned environment setup (devcontainer.json) for worker profiles — unchanged from revision 1: adopt the spec if/when it lands, separate plan.
• Forge integration (repo lists, clone-anywhere, PR status) — explicitly out, follow-up once the derived project model has usage.