fix: prevent live updater from leaving Gateway offline indefinitely (#119116)

* fix(updater): bound maintenance commands

* fix(updater): bound post-kill cleanup

* fix(updater): fail closed on unverified Windows cleanup

* fix(updater): constrain unverifiable Windows commands

* fix(scripts): satisfy managed child return contract
This commit is contained in:
Jason (Json)
2026-08-04 01:59:59 -06:00
committed by GitHub
parent fd1cce2826
commit b1f2aae303
7 changed files with 1469 additions and 253 deletions
@@ -30,7 +30,7 @@ Keep `/Users/steipete/openclaw` a read-only-to-the-agent deployment mirror: clea
- Every successful update sets `actions.gatewayBuild` and rebuilds exact new `main` before any restart.
- Missing, invalid, or stale build output also forces a build, even when Git did not move.
- A dependency-input change, absent `node_modules`, or missing/invalid build provenance requires `pnpm install --frozen-lockfile`. When a build is required, do not install before acquiring the maintenance suspension and stopping the managed Gateway.
- Before mutating any entrypoint currently executed by the Gateway, invoke an exact trusted source CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-<sha>` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, first accept native proof that the snapshot job is already booted out with its port free; when the isolated snapshot is still running, build the verified clean source checkout to obtain an exact suspension client, then use only that source client for prepare and failure-resume. Never use this recovery build while launchd targets the source checkout. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers further mutation to the next heartbeat; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build unless the exact recovery build already produced the deployment artifact; source launchers can auto-build stale output before dispatching the stop. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart.
- Before mutating any entrypoint currently executed by the Gateway, invoke an exact trusted source CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-<sha>` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, first accept native proof that the snapshot job is already booted out with its port free; when the isolated snapshot is still running, build the verified clean source checkout to obtain an exact suspension client, then use only that source client for prepare and failure-resume. Never use this recovery build while launchd targets the source checkout. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers further mutation to the next heartbeat; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build unless the exact recovery build already produced the deployment artifact; source launchers can auto-build stale output before dispatching the stop. Every external phase has a wall-clock budget: Git fetch 5 minutes, Git merge 2 minutes, dependency install 15 minutes, Gateway build 20 minutes, service control 60 seconds, probes 30 seconds, and Mac rebuild 30 minutes. Progress output never extends a budget. A timeout must drain the task-owned process tree before rollback, previous-service bootstrap, lock release, or the final `command_timeout` JSON result. If POSIX cleanup cannot prove the process group gone, skip service recovery, retain the maintenance lock against that process group, and emit `command_cleanup_failed`; after an operator terminates the group, the next heartbeat reclaims the stale lock. Strict tree verification is unavailable on Windows because `taskkill /T` loses ownership after a normal root exit; refuse before spawning the pre-stop fetch, leave the Gateway untouched, release the lock, and emit `unsupported_process_tree_verification` until a Windows Job Object owner exists. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart.
- Snapshot ownership validation must never invoke Git inside the snapshot. Treat its worktree, local Git configuration, filters, hooks, attributes, and build artifacts as untrusted; prove only that the regular owned LaunchAgent entrypoint is under the canonical detached ancestor snapshot path. Snapshot validation authorizes native bootout and retargeting only. Every CLI path must reject snapshot execution and use the trusted source checkout build instead.
- Only after exact-SHA build proof may it restart the managed Gateway and require `gateway status --deep --require-rpc --json` plus `health --verbose --json`. A validated ancestor snapshot is suspension-only: prove the old launchd job is booted out with its port free, allowing bounded retries while launchd and the listener finish teardown, then atomically retarget only the owned LaunchAgent entrypoint to this checkout's exact `dist/index.js`, including on a retry where the build is already current. Replace the verified `ProgramArguments` array as one value; never use array-index plist mutation that can insert a duplicate argument. Preserve all other service arguments and environment unchanged. After restart, prove the loaded launchd PID owns the configured listener.
- After every managed restart, query Gateway logs through RPC, restrict the audit to entries emitted since that restart began, report warning summaries, and fail the pass on any error/fatal entry. If RPC verification or log retrieval fails, still inspect the local structured log for that restart window. Never accept supervisor or RPC health without this restart-window log audit.
File diff suppressed because it is too large Load Diff