At high completion concurrency, 12/50 turns failed with 'Cloud worker
finished, but its workspace result could not be reconciled' (#129979):
between a worker's manifest upload and its verification, a sibling
completion could trigger retention with a snapshot holding only the old
base manifest — the terminal worker no longer protected the workspace, so
cleanup deleted the just-uploaded manifest and verification died on ENOENT.
The node workspace runtime now pins the latest transferred manifest per
generation and includes it in every cleanup's reachable set (cleared with
its generation), and gateway retention snapshots treat unsettled results —
an active claim or a pending-result fence — as protection, which also
preserves it across restarts when node-local pins are gone. Capture errors
retain bounded redacted stderr so this failure class can never be
faceless again. No deadlines, protocol, or completion-semantics changes;
deterministic 1-turn and 50-turn reproductions fail pre-fix.
* feat(portals): expose portals to sessions on node-backed cloud workers
Worker-turn sessions placed on cloud workers (bundled Crabbox) can now open
portals. The portal tool joins the closed worker session-tool catalog and
executes on the Gateway, which mints a single-use ticket per proxied
connection; the enrolled node redeems it over a TLS-pinned WebSocket via the
new private worker.portal.stream.v1 command and splices to the worker's
loopback port. The desktop stream broker is generalized with discriminated
desktop/portal tickets, and shared node stream transport is extracted from
the desktop stream command. Portal availability is negotiated via a new
gateway hello capability plus the node's reconnect-scoped
workerHost.portalStream declaration, so old node bundles simply do not
receive the tool. Environment teardown and owner-epoch replacement close a
worker's portals; node disconnects destroy live streams while the listener
keeps serving the waiting page.
Live-proven on a Crabbox AWS worker: agent opened a portal over the worker
protocol, python3 -m http.server on the box served through the Gateway
portal URL with token auth intact.
* fix(portals): break portal proxy/service type cycle; regenerate Swift protocol models
* fix(portals): drop dead PortalTarget re-export; fix test import
* fix(portals): only tear down turn-created portals on post-open authority loss
ClawSweeper finding: a revoked turn's duplicate open could close the live
portal a still-authorized predecessor established, because service.open
returns the existing same-environment portal and the executor's error path
closed whatever id it received. portal.open now reports created-vs-reused;
the executor tears down only portals it created and closes its redundant
carrier handle whenever an existing portal is reused.
* feat(portals): fence portal reuse behind the caller's live-authority assert
Rank-up move from review: portal.open revalidates the caller inside the
serialized operation before a reuse can mutate a live portal's metadata,
mirroring close(assertCurrent). Regression test proves a lapsed caller
cannot rewrite a reused portal's title.
* test(portals): expect assertCurrent in worker portal open call
* refactor(gateway): extract worker request dispatch into its own module
worker-connection.ts crossed the 700-line ratchet after rebasing onto
current main; dispatchWorkerRequest and its rejection helper move to
worker-connection-dispatch.ts unchanged (434 + 329 lines).
Crabbox already ships a Daytona backend; add the profile example and
provider-specific notes (no crabbox login step, default snapshot ships
Node, default machine class) confirmed by a live lease-and-exec proof.
* docs(plan): add worker hibernation design note
Evaluates dormant leases, checkpoint-based warm start, and serverless
backends for a worker-environment hibernation tier; proposes a plugin-only
phase 1 (Crabbox checkpoint fork warm images) with SDK/store phases gated
on maintainer acceptance.
* feat(crabbox): add opt-in profile warm images
* fix(crabbox): preserve warm image machine class
* refactor(crabbox): tighten warm-image seams after review
* fix(crabbox): budget warm-image capture for real snapshot round trips
The 60s command bound starved the scrub and checkpoint-create phases under
coordinator latency, silently degrading every teardown to cold-only.
Live-measured on AWS.
* docs(plan): record worker-hibernation acceptance and live proof
Phase 1 live-proven: local-container full loop (cold 101s vs warm fork 40s,
scrub boundary verified) and AWS cold provisioning with graceful degrade at
the coordinator admin gate; named upstream follow-ups.
* fix(crabbox): make warm-image lifecycle recovery-safe
Delete least-recently-used provider snapshots before image index admission.
Recover stale empty capture reservations without breaking single-flight.
Persist effective machine classes across restarts and revert heartbeat coupling.
Align the hibernation plan with the exact-class warm-image key policy.
The cloud machine picker described each class in prose ("Cheap smoke checks
and small repos"), which was the widest thing in the row, got ellipsized, and
did not say what the operator actually picks on. Machine options now carry the
class shape and the picker renders "32 vCPU · 64 GB" instead.
Crabbox reports per-provider class shapes from `providers --json`; the plugin
reads that catalog once per lifecycle and attaches cpu/memoryGb to each option.
A missing binary, failed command, unparseable output, absent classes, or an
unmatched provider all degrade to label-only rows, so an older Crabbox keeps
working.
`description` was never released, so it is removed rather than deprecated, and
listMachineOptions becomes async for the catalog read. Deletes the duplicate
machine-option projector in the environments server method.
The Control UI settings page (#124864) and the New Session place
picker's machine section had no doc coverage — the configuration and
dispatch walkthroughs only described the JSON/RPC paths.
* feat(gateway): carry machine class through session moves
Profile move targets accept an optional machineClass with new-session
parity: validated in the protocol, persisted on the durable move intent
(bare nullable column, lazy same-version ALTER), replayed exactly by
restart recovery, and selectable from the Control UI move dialog.
Moving to the current profile with a different class resizes the
session's worker.
* fix(gateway): project placement facts on chat startup
chat.startup and chat.history built sessionInfo without the placement
projection sessions.list applies, so clients merging that row erased a
live worker placement and its move intent — the Runs on Cloud chip
disappeared after any turn until a full reload.
Reuse the canonical ensureColumn helper for the move table's additive
column instead of a hand-rolled PRAGMA read.
* feat(workers): run device sessions from Gateway bundles
Install the current Gateway bundle before a device environment becomes ready, verify it at attach and tunnel boundaries, launch only from the immutable namespaced bundle directory, and retire stale environments for idempotent reprovisioning. Remove the local execution mode and preserve the node-local build claim only as temporary inventory metadata for the final projection/cleanup slice.
* docs(runners): record Gateway bundle cutover
* test(ci): repair runner validation fixtures
# Conflicts:
# src/scripts/test-projects.test.ts
* fix(workers): surface outdated node recovery
Keep legacy runner inventory diagnostic-only while exposing the update-and-reconnect action through node, environment, provider, placement, and Control UI surfaces.
* fix(workers): reject legacy inventory with recovery
* fix(workers): bundle worker deploy closure
* test(workers): close bundle cutover gates
* fix(workers): compose browser runtime at build
* fix(workers): satisfy bundle cutover gates
* fix(workers): route temp runtime through infra
* docs(workers): align bundle host guidance
* fix(ui): fence outdated session destinations
* feat(gateway): add remote-exec cloud placements
* feat(codex): run cloud turns through remote exec
* fix(sandbox): quote ssh_config path directives containing whitespace
Crabbox lease keys default to ~/Library/Application Support/... on macOS;
unquoted IdentityFile/UserKnownHostsFile/CertificateFile arguments tokenize
on the space and fail as 'extra arguments'. Found via live remote-exec
cloud-worker proof.
* test: consolidate gateway maintenance schedule coverage
* fix(ci): invalidate plugin sdk declarations on state changes
* feat(ui): click the desktop to take control
View-only desktop connections now escalate to control by clicking
anywhere on the desktop stage (transparent overlay button, keyboard
accessible, same 'Take control' accessible name). The toolbar button is
removed; the connecting status overlay becomes click-through so control
can still be requested mid-connect.
* fix(ui): use cursor-action token for desktop take-control overlay
* test(ui): prove take-control click above a real noVNC surface
In-page RFB 3.8 fake server (security None) lets the production
DesktopClient drive the real noVNC client; the overlay click must
hit-test above the mounted canvas and reconnect with control.
* fix(ui): narrow fake RFB socket payload to ArrayBuffer-backed bytes
* fix(workers): persist placement terminal failures
* fix(workers): refresh placement protocol clients
* refactor(workers): isolate error formatting
* fix: integrate cloud terminal state with current main
* chore(plugin-sdk): refresh API baseline
* refactor(ui): inline one-use cloud terminal-reason banner helper
Keeps the Control UI startup JS bundle inside its 317 KiB gzip budget
(the helper + type-only import tipped it by 16 bytes).
* refactor(ui): trim terminal-reason lookup to type-erased optional access
Recovers the last gzip byte of the Control UI startup budget
(324609 B vs the 324608 B limit).
* feat(gateway): live desktop observer for cloud workers
Adds live observation for cloud worker desktops through the gateway and Crabbox plugin, including desktop provisioning, persisted desktop metadata, tunneled WebSocket proxying, and the worker.desktop.observe protocol method.
The gateway, Crabbox plugin, and gateway protocol surfaces remain off by default behind the cloudWorkers.desktop Labs flag.
* feat(ui): Desktop panel for cloud worker observation
* docs(gateway): document cloud worker desktop lab
* fix(ci): regenerate contract baselines after rebase
* fix(protocol): regenerate Android gateway methods
* fix(ci): align rebased SDK and lint baselines
* fix(gateway): enforce view-only RFB boundary and fence desktop teardown
* fix(gateway): tighten RFB filter surface
* fix(state): keep pre-desktop databases readable and harden view-only RFB
* fix(gateway): fence desktop observer upgrades behind work admission
* fix(gateway): bind desktop observer tokens to their owner epoch
* fix(ci): regenerate config and SDK baselines after rebase
* fix(ci): regenerate native protocol and SDK baselines
* fix(ci): regenerate contracts after main rebase
* fix(state): register desktop metadata as lazy additive
* fix(ci): regenerate SDK baseline after final direct-merge rebase
* fix(cloud-workers): close lifecycle ownership gaps
Own bootstrap cleanup at the operation boundary and make fallback workspace sync converge across retries. Re-establish tunnel readiness per connection, retire placements before destructive session mutation, and keep operator diagnostics lightweight and redacted. Cover destructive lifecycle paths in their original execution order.
* fix(cloud-workers): drain local claims before retirement
delete/reset drain admitted local work, re-read exact identity, retire before destructive cleanup; active-claim/race tests.
* fix(cloud-workers): bind retry cleanup to workspace owner
Attest canonical HOME and the exact managed path.
Revalidate ownership before recursive fallback cleanup.
Cover malicious paths and ownership drift with tests.
* fix(cloud-workers): fence fallback workspace receivers
* fix(cloud-workers): honor Crabbox SSH fallback ports
Select a currently reachable advertised SSH endpoint before returning a Crabbox worker lease, while preserving fail-closed teardown. Allow deletion of failed cloud sessions only after the environment record proves the provider lease is gone, and align the Cloud Workers docs with the shipped configuration and Control UI flow.
* fix(cloud-workers): retry advertised SSH endpoints
Persist provider-advertised SSH fallback ports and retry them through the shared pinned transport used by bootstrap, tunnels, reconnects, and workspace transfers. Keep candidate selection provider-neutral, preserve identity and host-key fencing, and add the same-version SQLite column through the existing additive migration path.
* fix(crabbox): recheck SSH readiness after setup
Treat provider setup as a mutating lifecycle phase: refresh authoritative inspect state and wait for SSH readiness again before returning the lease. This prevents bootstrap from racing setup-triggered SSH restarts on fresh and replayed workers while preserving security attestation and fail-closed teardown.
* fix(cloud-workers): retry bootstrap transfers across ports
Use the shared advertised-port candidate runner for bundle transfer, install, and best-effort upload cleanup as well as preflight. This keeps fresh bootstrap connections on pinned identity and host-key semantics when a previously selected SSH port disappears between phases.
Proof passed: 517 focused tests (1 skipped), final autoreview/TruffleHog clean, targeted lint/format. Full check-changed reached the unrelated max-lines baseline drift on current origin/main: this branch's merge base contains ui/src/pages/chat/components/chat-model-controls.ts while current origin/main removed that baseline entry; do not touch the unrelated baseline.
* fix(cloud-workers): persist fallback ports compatibly
Create the fallback-port companion table lazily as an additive surface, preserving downgrade tolerance for binaries that still expect the older shared-state schema.
* fix(cloud-workers): preserve bundle artifact narrowing
Bind the immutable discriminated artifact before the SSH retry callbacks so bundle-only fields remain narrowed across fresh candidate invocations. Use the shared temp-directory owner in the tunnel fallback test for deterministic cleanup.
* fix(cloud-workers): retire deleted placements
Terminal placement rows retire by exact CAS after session deletion, and managed AWS docs now pin the Gateway SSH CIDR.
* fix(crabbox): await pending AWS attestation
missing authoritative metadata is pending only while the lease is non-ready; ready leases still require explicit false.
* fix(cloud-workers): fence remote command retries
require explicit idempotent/never classification; remote worker launch never retries after ambiguous SSH loss.
* fix(cloud-workers): bound SSH fallback operations
Peter Steinberger