diff --git a/.agents/skills/openclaw-live-updater/SKILL.md b/.agents/skills/openclaw-live-updater/SKILL.md index 217d961b4e1b..529b5eeb5778 100644 --- a/.agents/skills/openclaw-live-updater/SKILL.md +++ b/.agents/skills/openclaw-live-updater/SKILL.md @@ -30,7 +30,7 @@ Keep `/Users/steipete/openclaw` a read-only-to-the-agent deployment mirror: clea - Every successful update sets `actions.gatewayBuild` and rebuilds exact new `main` before any restart. - Missing, invalid, or stale build output also forces a build, even when Git did not move. - A dependency-input change, absent `node_modules`, or missing/invalid build provenance requires `pnpm install --frozen-lockfile`. When a build is required, do not install before acquiring the maintenance suspension and stopping the managed Gateway. - - Before mutating any entrypoint currently executed by the Gateway, invoke an exact trusted source CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, first accept native proof that the snapshot job is already booted out with its port free; when the isolated snapshot is still running, build the verified clean source checkout to obtain an exact suspension client, then use only that source client for prepare and failure-resume. Never use this recovery build while launchd targets the source checkout. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers further mutation to the next heartbeat; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build unless the exact recovery build already produced the deployment artifact; source launchers can auto-build stale output before dispatching the stop. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart. + - Before mutating any entrypoint currently executed by the Gateway, invoke an exact trusted source CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, first accept native proof that the snapshot job is already booted out with its port free; when the isolated snapshot is still running, build the verified clean source checkout to obtain an exact suspension client, then use only that source client for prepare and failure-resume. Never use this recovery build while launchd targets the source checkout. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers further mutation to the next heartbeat; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build unless the exact recovery build already produced the deployment artifact; source launchers can auto-build stale output before dispatching the stop. Every external phase has a wall-clock budget: Git fetch 5 minutes, Git merge 2 minutes, dependency install 15 minutes, Gateway build 20 minutes, service control 60 seconds, probes 30 seconds, and Mac rebuild 30 minutes. Progress output never extends a budget. A timeout must drain the task-owned process tree before rollback, previous-service bootstrap, lock release, or the final `command_timeout` JSON result. If POSIX cleanup cannot prove the process group gone, skip service recovery, retain the maintenance lock against that process group, and emit `command_cleanup_failed`; after an operator terminates the group, the next heartbeat reclaims the stale lock. Strict tree verification is unavailable on Windows because `taskkill /T` loses ownership after a normal root exit; refuse before spawning the pre-stop fetch, leave the Gateway untouched, release the lock, and emit `unsupported_process_tree_verification` until a Windows Job Object owner exists. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart. - Snapshot ownership validation must never invoke Git inside the snapshot. Treat its worktree, local Git configuration, filters, hooks, attributes, and build artifacts as untrusted; prove only that the regular owned LaunchAgent entrypoint is under the canonical detached ancestor snapshot path. Snapshot validation authorizes native bootout and retargeting only. Every CLI path must reject snapshot execution and use the trusted source checkout build instead. - Only after exact-SHA build proof may it restart the managed Gateway and require `gateway status --deep --require-rpc --json` plus `health --verbose --json`. A validated ancestor snapshot is suspension-only: prove the old launchd job is booted out with its port free, allowing bounded retries while launchd and the listener finish teardown, then atomically retarget only the owned LaunchAgent entrypoint to this checkout's exact `dist/index.js`, including on a retry where the build is already current. Replace the verified `ProgramArguments` array as one value; never use array-index plist mutation that can insert a duplicate argument. Preserve all other service arguments and environment unchanged. After restart, prove the loaded launchd PID owns the configured listener. - After every managed restart, query Gateway logs through RPC, restrict the audit to entries emitted since that restart began, report warning summaries, and fail the pass on any error/fatal entry. If RPC verification or log retrieval fails, still inspect the local structured log for that restart window. Never accept supervisor or RPC health without this restart-window log audit. diff --git a/.agents/skills/openclaw-live-updater/scripts/update-main.mjs b/.agents/skills/openclaw-live-updater/scripts/update-main.mjs index 709420081e95..53b5b963a024 100644 --- a/.agents/skills/openclaw-live-updater/scripts/update-main.mjs +++ b/.agents/skills/openclaw-live-updater/scripts/update-main.mjs @@ -17,6 +17,7 @@ import { import { tmpdir } from "node:os"; import path from "node:path"; import process from "node:process"; +import { setTimeout as delay } from "node:timers/promises"; import { fileURLToPath } from "node:url"; import { detectChangedScope } from "../../../../scripts/ci-changed-scope.mjs"; import { isDirectRunUrl } from "../../../../scripts/lib/direct-run.mjs"; @@ -24,6 +25,7 @@ import { BUILD_STAMP_FILE, RUNTIME_POSTBUILD_STAMP_FILE, } from "../../../../scripts/lib/local-build-metadata.mjs"; +import { runManagedCommand } from "../../../../scripts/lib/managed-child-process.mjs"; import { runNodeConfigFiles, runNodeSourceRoots, @@ -46,15 +48,32 @@ const GATEWAY_STOP_PROOF_RETRY_DELAY_MS = 250; const GATEWAY_PROCESS_START_TIMEOUT_MS = 20_000; const GATEWAY_PROCESS_START_RETRY_DELAY_MS = 250; const GATEWAY_SUSPEND_TIMEOUT_MS = 10_000; +const LEAF_COMMAND_TIMEOUT_MS = 30_000; +const COMMAND_TIMEOUT_MS = Object.freeze({ + dependencyInstall: 15 * 60_000, + gatewayBuild: 20 * 60_000, + gatewayProbe: GATEWAY_CLI_TIMEOUT_MS, + gatewayService: 60_000, + gitFetch: 5 * 60_000, + gitMerge: 2 * 60_000, + macAppRebuild: 30 * 60_000, +}); const GATEWAY_STARTUP_TRACE_ENV = "OPENCLAW_GATEWAY_STARTUP_TRACE"; const SYSTEM_LAUNCH_DAEMON_DIR = "/Library/LaunchDaemons"; const MAX_FAILURE_DIAGNOSTIC_DEPTH = 4; const MAX_FAILURE_DIAGNOSTIC_MEMBERS = 8; const SAFE_INVARIANT_DETAIL_KEYS = [ + "elapsedMs", "exitTimeoutSeconds", "listenerClosed", + "lockRetained", + "manualRecoveryRequired", + "phase", "processExited", + "processTreeState", "serviceBootedOut", + "serviceState", + "timeoutMs", ]; // CLI diagnostics stay typed and bounded because child-process errors can // retain argv, environment, and output that must never enter the JSON result. @@ -177,9 +196,9 @@ function gatewayCliOperation(args) { return "gateway.cli"; } -function runUpdateCommand(runCommand, operation, command, args, checkout) { +async function runUpdateCommand(runCommand, operation, command, args, checkout, options) { try { - return runCommand(command, args, checkout); + return await runCommand(command, args, checkout, options); } catch (error) { if ( error instanceof UpdateInvariantError || @@ -198,7 +217,19 @@ function formatInvariantDetails(details) { const value = ownDataProperty(details, key); if (typeof value === "boolean") { formatted[key] = value; - } else if (key === "exitTimeoutSeconds" && Number.isInteger(value)) { + } else if ( + ["elapsedMs", "exitTimeoutSeconds", "timeoutMs"].includes(key) && + Number.isInteger(value) + ) { + formatted[key] = value; + } else if (key === "phase" && typeof value === "string") { + formatted[key] = value; + } else if ( + key === "processTreeState" && + ["indeterminate", "live", "terminated"].includes(value) + ) { + formatted[key] = value; + } else if (key === "serviceState" && ["running", "stopped", "stopping"].includes(value)) { formatted[key] = value; } } @@ -290,8 +321,52 @@ export function formatUpdateFailure(error) { }; } +function findUpdateInvariantError(value, code) { + const pending = [value]; + const seen = new Set(); + while (pending.length > 0) { + const current = pending.shift(); + if (!current || (typeof current !== "object" && typeof current !== "function")) { + continue; + } + if (seen.has(current)) { + continue; + } + seen.add(current); + if (current instanceof UpdateInvariantError && current.code === code) { + return current; + } + if (current instanceof AggregateError) { + pending.push(...current.errors); + } + if ("cause" in current) { + pending.push(current.cause); + } + } + return null; +} + +function findUnsafeCommandCleanupFailure(value) { + const failure = findUpdateInvariantError(value, "command_cleanup_failed"); + const unsafeTreeState = ["indeterminate", "live"].includes(failure?.details?.processTreeState); + const retainedByProcessGroup = Number.isSafeInteger(failure?.details?.processGroupId); + const retainedForManualRecovery = failure?.details?.manualRecoveryRequired === true; + return failure && unsafeTreeState && (retainedByProcessGroup || retainedForManualRecovery) + ? failure + : null; +} + +function boundedSyncOptions(options = {}, timeoutMs = LEAF_COMMAND_TIMEOUT_MS) { + return { + ...options, + killSignal: "SIGKILL", + timeout: timeoutMs, + }; +} + function git(checkout, args, options = {}) { return execFileSync("git", ["-C", checkout, ...args], { + ...boundedSyncOptions({}, options.timeoutMs), encoding: options.encoding ?? "utf8", stdio: options.stdio ?? ["ignore", "pipe", "pipe"], }); @@ -416,7 +491,7 @@ function canonicalBuildRequirements(checkout) { cwd: checkout, env: process.env, fs: fsImpl, - spawnSync, + spawnSync: (command, args, options) => spawnSync(command, args, boundedSyncOptions(options)), distRoot, distEntry: path.join(distRoot, "entry.js"), buildStampPath: path.join(distRoot, BUILD_STAMP_FILE), @@ -600,19 +675,31 @@ function verifyCheckout(checkout, { remote }) { }; } -function updateMain({ checkout, remote }, dependencies = {}) { +async function updateMain({ checkout, remote }, runCommand, dependencies = {}) { const before = verifyCheckout(checkout, { remote }); const fetchMain = dependencies.fetchMain ?? ((target, remoteName) => - git( + runUpdateCommand( + runCommand, + "git.fetch", + "git", + [ + "-C", + target, + "fetch", + "--prune", + remoteName, + `refs/heads/main:refs/remotes/${remoteName}/main`, + ], target, - ["fetch", "--prune", remoteName, `refs/heads/main:refs/remotes/${remoteName}/main`], { - stdio: ["ignore", "ignore", "inherit"], + phase: "Git fetch", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.gitFetch, }, )); - fetchMain(before.checkout, remote); + await fetchMain(before.checkout, remote); const afterFetch = verifyCheckout(before.checkout, { remote }); if (afterFetch.headSha !== before.headSha) { throw new UpdateInvariantError( @@ -622,9 +709,18 @@ function updateMain({ checkout, remote }, dependencies = {}) { } const remoteSha = gitText(before.checkout, ["rev-parse", `${remote}/main`]); - git(before.checkout, ["merge", "--ff-only", `${remote}/main`], { - stdio: ["ignore", "ignore", "inherit"], - }); + await runUpdateCommand( + runCommand, + "git.merge", + "git", + ["-C", before.checkout, "merge", "--ff-only", `${remote}/main`], + before.checkout, + { + phase: "Git fast-forward merge", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.gitMerge, + }, + ); const after = verifyCheckout(before.checkout, { remote }); if (after.headSha !== remoteSha) { throw new UpdateInvariantError( @@ -695,6 +791,22 @@ function processAlive(pid) { } } +function processGroupAlive(processGroupId) { + if ( + process.platform === "win32" || + !Number.isSafeInteger(processGroupId) || + processGroupId <= 1 + ) { + return false; + } + try { + process.kill(-processGroupId, 0); + return true; + } catch (error) { + return error?.code !== "ESRCH"; + } +} + export function acquireMaintenanceLock(checkout, requestedPath) { const lockPath = requestedPath ?? defaultLockPath(checkout); let incompleteLockRetries = 0; @@ -729,13 +841,17 @@ export function acquireMaintenanceLock(checkout, requestedPath) { // budget and only then declare the lock invalid. if (incompleteLockRetries < 20) { incompleteLockRetries += 1; - spawnSync("sleep", ["0.01"]); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 10); continue; } throw new UpdateInvariantError("invalid_lock", `lock owner is unreadable: ${lockPath}`); } incompleteLockRetries = 0; - if (Number.isInteger(owner.pid) && processAlive(owner.pid)) { + const ownerProcessAlive = Number.isInteger(owner.pid) && processAlive(owner.pid); + const blockedProcessGroupAlive = + Number.isInteger(owner.processGroupId) && processGroupAlive(owner.processGroupId); + const manualRecoveryRequired = owner.manualRecoveryRequired === true; + if (ownerProcessAlive || blockedProcessGroupAlive || manualRecoveryRequired) { return { acquired: false, lockPath, owner }; } const staleClaim = `${lockPath}.stale-${process.pid}-${randomUUID()}`; @@ -761,6 +877,38 @@ export function acquireMaintenanceLock(checkout, requestedPath) { acquired: true, lockPath, owner, + retainForCleanupFailure(details = {}) { + const processGroupId = details.processGroupId; + const manualRecoveryRequired = details.manualRecoveryRequired === true; + if ( + (!Number.isSafeInteger(processGroupId) || processGroupId <= 1) && + !manualRecoveryRequired + ) { + throw new UpdateInvariantError( + "invalid_cleanup_lock", + "refusing to retain maintenance lock without a process group or manual recovery state", + ); + } + const ownerPath = path.join(lockPath, "owner.json"); + const current = JSON.parse(readFileSync(ownerPath, "utf8")); + if (current.pid !== process.pid) { + throw new UpdateInvariantError("lock_owner_changed", "maintenance lock ownership changed"); + } + const retainedOwner = { + ...current, + blockedAt: new Date().toISOString(), + reason: "command_cleanup_failed", + ...(Number.isSafeInteger(processGroupId) ? { processGroupId } : {}), + ...(manualRecoveryRequired ? { manualRecoveryRequired: true } : {}), + ...(typeof details.phase === "string" ? { phase: details.phase } : {}), + ...(typeof details.processTreeState === "string" + ? { processTreeState: details.processTreeState } + : {}), + ...(typeof details.serviceState === "string" ? { serviceState: details.serviceState } : {}), + }; + writeFileSync(ownerPath, `${JSON.stringify(retainedOwner)}\n`, { mode: 0o600 }); + return retainedOwner; + }, release() { const current = JSON.parse(readFileSync(path.join(lockPath, "owner.json"), "utf8")); if (current.pid !== process.pid) { @@ -771,11 +919,63 @@ export function acquireMaintenanceLock(checkout, requestedPath) { }; } -function defaultRunCommand(command, args, checkout) { - execFileSync(command, args, { - cwd: checkout, - stdio: ["ignore", process.stderr, process.stderr], - }); +async function defaultRunCommand(command, args, checkout, options = {}) { + const phase = options.phase ?? "command"; + const timeoutMs = options.timeoutMs ?? LEAF_COMMAND_TIMEOUT_MS; + const startedAt = Date.now(); + try { + const managedRunner = options.runManagedCommand ?? runManagedCommand; + const status = await managedRunner({ + bin: command, + args, + cwd: checkout, + requireProcessTreeExit: true, + stdio: ["ignore", 2, 2], + timeoutMs, + }); + if (status !== 0) { + throw new Error(`${phase} command exited with status ${status}`); + } + } catch (error) { + if (error?.code === "EPROCESS_TREE_VERIFICATION_UNSUPPORTED") { + throw new UpdateInvariantError( + "unsupported_process_tree_verification", + `${phase} requires process-tree verification unavailable on this platform`, + { + phase, + serviceState: options.serviceState ?? "running", + }, + ); + } + if (error?.code === "EPROCESSGROUP_CLEANUP_FAILED") { + throw new UpdateInvariantError( + "command_cleanup_failed", + `${phase} process tree could not be cleaned up`, + { + elapsedMs: Date.now() - startedAt, + phase, + ...(Number.isSafeInteger(error.processGroupId) + ? { processGroupId: error.processGroupId } + : {}), + ...(typeof error.processTreeState === "string" + ? { processTreeState: error.processTreeState } + : {}), + ...(error.manualRecoveryRequired === true ? { manualRecoveryRequired: true } : {}), + serviceState: options.serviceState ?? "running", + timeoutMs, + }, + ); + } + if (error?.code !== "ETIMEDOUT") { + throw error; + } + throw new UpdateInvariantError("command_timeout", `${phase} timed out after ${timeoutMs}ms`, { + elapsedMs: Date.now() - startedAt, + phase, + serviceState: options.serviceState ?? "running", + timeoutMs, + }); + } } function readSnapshotMetadata(snapshotRoot) { @@ -1005,7 +1205,11 @@ export function assertNoSystemLaunchDaemonOwnership(label, dependencies = {}) { const readDirectory = dependencies.readdirSync ?? readdirSync; const serviceTarget = `system/${label}`; const inspectLoadedService = () => { - const result = run("/bin/launchctl", ["print", serviceTarget], { encoding: "utf8" }); + const result = run( + "/bin/launchctl", + ["print", serviceTarget], + boundedSyncOptions({ encoding: "utf8" }), + ); if (result.status === 0) { throw new UpdateInvariantError( "gateway_system_launchdaemon_conflict", @@ -1036,9 +1240,11 @@ export function assertNoSystemLaunchDaemonOwnership(label, dependencies = {}) { } for (const entry of entries.filter((candidate) => candidate.endsWith(".plist")).toSorted()) { const plistPath = path.join(SYSTEM_LAUNCH_DAEMON_DIR, entry); - const result = run("/usr/bin/plutil", ["-convert", "json", "-o", "-", "--", plistPath], { - encoding: "utf8", - }); + const result = run( + "/usr/bin/plutil", + ["-convert", "json", "-o", "-", "--", plistPath], + boundedSyncOptions({ encoding: "utf8" }), + ); if (result.status !== 0) { throw new UpdateInvariantError( "gateway_system_launchdaemon_unverifiable", @@ -1092,9 +1298,11 @@ function readManagedGatewayLaunchAgent(checkout) { "managed Gateway LaunchAgent is not a regular owned plist file", ); } - const plistResult = spawnSync("/usr/bin/plutil", ["-convert", "json", "-o", "-", plistPath], { - encoding: "utf8", - }); + const plistResult = spawnSync( + "/usr/bin/plutil", + ["-convert", "json", "-o", "-", plistPath], + boundedSyncOptions({ encoding: "utf8" }), + ); if (plistResult.status !== 0) { throw new UpdateInvariantError( "gateway_launchagent_failed", @@ -1225,7 +1433,7 @@ function prepareLaunchAgentEntrypointReplacement(deployment, entrypoint, options const plistResult = spawnSync( "/usr/bin/plutil", ["-convert", "json", "-o", "-", temporaryPath], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); if (plistResult.status !== 0) { throw new UpdateInvariantError( @@ -1242,15 +1450,15 @@ function prepareLaunchAgentEntrypointReplacement(deployment, entrypoint, options execFileSync( "/usr/bin/plutil", ["-replace", "ProgramArguments", "-json", JSON.stringify(programArguments), temporaryPath], - { stdio: ["ignore", "ignore", "pipe"] }, + boundedSyncOptions({ stdio: ["ignore", "ignore", "pipe"] }), ); execFileSync("/usr/bin/plutil", ["-lint", temporaryPath], { - stdio: ["ignore", "ignore", "pipe"], + ...boundedSyncOptions({ stdio: ["ignore", "ignore", "pipe"] }), }); const validatedResult = spawnSync( "/usr/bin/plutil", ["-convert", "json", "-o", "-", temporaryPath], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); if ( validatedResult.status !== 0 || @@ -1364,7 +1572,7 @@ function verifyManagedGatewayRuntime(checkout, expectedSha) { const launchctl = spawnSync( "/bin/launchctl", ["print", `gui/${process.getuid()}/${deployment.label}`], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); const pidMatch = launchctl.status === 0 ? launchctl.stdout.match(/\bpid = (\d+)\b/u) : null; const pid = Number(pidMatch?.[1] ?? Number.NaN); @@ -1399,7 +1607,7 @@ function verifyManagedGatewayRuntime(checkout, expectedSha) { const listeners = spawnSync( "/usr/sbin/lsof", ["-nP", `-iTCP:${deployment.port}`, "-sTCP:LISTEN", "-t"], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); const listenerPids = listeners.stdout.trim().split(/\s+/u).filter(Boolean).map(Number); // The Gateway overwrites process.title, so ps cannot prove argv. The owned @@ -1471,7 +1679,11 @@ export function runBuiltGatewayCli(checkout, args, deployment, options = {}) { const configuredPath = execFileSync( executable, [...wrapperPrefix, "/usr/bin/printenv", "OPENCLAW_CONFIG_PATH"], - { encoding: "utf8", env: baseEnv, stdio: ["ignore", "pipe", "ignore"] }, + boundedSyncOptions({ + encoding: "utf8", + env: baseEnv, + stdio: ["ignore", "pipe", "ignore"], + }), ).trim(); if (configuredPath) { effectiveConfigPath = configuredPath; @@ -1579,23 +1791,33 @@ function defaultResumeGatewaySuspension(checkout, suspensionId, deployment) { runBuiltGatewayCall(checkout, "gateway.suspend.resume", { suspensionId }, deployment); } -function stopManagedGateway(runCommand, checkout, deployment) { +async function stopManagedGateway(runCommand, checkout, deployment) { if (!deployment) { - runUpdateCommand( + await runUpdateCommand( runCommand, "gateway.stop", process.execPath, ["dist/index.js", "gateway", "stop"], checkout, + { + phase: "gateway stop", + serviceState: "stopping", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); return; } - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.bootout", "/bin/launchctl", ["bootout", `gui/${process.getuid()}/${deployment.label}`], checkout, + { + phase: "Gateway LaunchAgent bootout", + serviceState: "stopping", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); } @@ -1614,7 +1836,7 @@ function recordStoppedMilestones(timing, observation, now) { } } -function stopManagedGatewayAndProve( +async function stopManagedGatewayAndProve( runCommand, checkout, deployment, @@ -1631,7 +1853,7 @@ function stopManagedGatewayAndProve( }; let stopError; try { - stopManagedGateway(runCommand, checkout, deployment); + await stopManagedGateway(runCommand, checkout, deployment); } catch (error) { stopError = error; } finally { @@ -1648,7 +1870,7 @@ function stopManagedGatewayAndProve( let proofError; for (let attempt = 0; attempt < proofAttempts; attempt += 1) { try { - const proof = proveGatewayStopped(checkout); + const proof = await proveGatewayStopped(checkout); recordStoppedMilestones(timing, proof, now); recordGatewayTimestamp(timing, "processExitedAt", timestampAt(now)); recordGatewayTimestamp(timing, "listenerClosedAt", timestampAt(now)); @@ -1657,7 +1879,7 @@ function stopManagedGatewayAndProve( proofError = error; recordStoppedMilestones(timing, error, now); if (attempt + 1 < proofAttempts) { - sleep(GATEWAY_STOP_PROOF_RETRY_DELAY_MS); + await sleep(GATEWAY_STOP_PROOF_RETRY_DELAY_MS); } } } @@ -1719,17 +1941,21 @@ function resolveGatewayControlDeployment(checkout, deployment, buildBefore, curr function proveMacLaunchdGatewayStopped(checkout) { const { label, port } = readManagedGatewayLaunchAgent(checkout); - const launchctl = spawnSync("/bin/launchctl", ["print", `gui/${process.getuid()}/${label}`], { - encoding: "utf8", - }); + const launchctl = spawnSync( + "/bin/launchctl", + ["print", `gui/${process.getuid()}/${label}`], + boundedSyncOptions({ encoding: "utf8" }), + ); const launchctlOutput = `${launchctl.stdout ?? ""}\n${launchctl.stderr ?? ""}`; const serviceBootedOut = launchctl.status !== 0 && /could not find service|service not found/iu.test(launchctlOutput); const processExited = serviceBootedOut || (launchctl.status === 0 && !/\bpid\s*=\s*\d+\b/iu.test(launchctlOutput)); - const listeners = spawnSync("/usr/sbin/lsof", ["-nP", `-iTCP:${port}`, "-sTCP:LISTEN", "-t"], { - encoding: "utf8", - }); + const listeners = spawnSync( + "/usr/sbin/lsof", + ["-nP", `-iTCP:${port}`, "-sTCP:LISTEN", "-t"], + boundedSyncOptions({ encoding: "utf8" }), + ); const listenerClosed = listeners.status === 1 && !listeners.stdout.trim() && !listeners.stderr.trim(); const details = { listenerClosed, processExited, serviceBootedOut }; @@ -1766,7 +1992,9 @@ function defaultProveGatewayStopped(checkout) { execFileSync(process.execPath, ["dist/index.js", "gateway", "status", "--json"], { cwd: checkout, encoding: "utf8", + killSignal: "SIGKILL", stdio: ["ignore", "pipe", "inherit"], + timeout: COMMAND_TIMEOUT_MS.gatewayProbe, }), ); } catch (error) { @@ -1832,10 +2060,19 @@ function isOriginalMacBundle(bundlePath, originalStat) { } } -function runBuildWithPreservedMacApp(runCommand, checkout, sleep = defaultSleep) { +async function runBuildWithPreservedMacApp( + runCommand, + checkout, + sleep = defaultSleep, + serviceState = "stopped", +) { const appBundle = path.join(checkout, "dist/OpenClaw.app"); if (!existsSync(appBundle)) { - runUpdateCommand(runCommand, "build", "pnpm", ["build"], checkout); + await runUpdateCommand(runCommand, "build", "pnpm", ["build"], checkout, { + phase: "Gateway build", + serviceState, + timeoutMs: COMMAND_TIMEOUT_MS.gatewayBuild, + }); return; } const appStat = lstatSync(appBundle); @@ -1854,7 +2091,11 @@ function runBuildWithPreservedMacApp(runCommand, checkout, sleep = defaultSleep) let buildFailed = false; let buildError; try { - runUpdateCommand(runCommand, "build", "pnpm", ["build"], checkout); + await runUpdateCommand(runCommand, "build", "pnpm", ["build"], checkout, { + phase: "Gateway build", + serviceState, + timeoutMs: COMMAND_TIMEOUT_MS.gatewayBuild, + }); } catch (error) { buildFailed = true; buildError = error; @@ -1865,7 +2106,7 @@ function runBuildWithPreservedMacApp(runCommand, checkout, sleep = defaultSleep) if (existsSync(preservedBundle) || existsSync(appBundle)) { break; } - sleep(100); + await sleep(100); } const alreadyRestored = isOriginalMacBundle(appBundle, appStat); if (!alreadyRestored && existsSync(appBundle)) { @@ -1916,7 +2157,7 @@ function runBuildWithPreservedMacApp(runCommand, checkout, sleep = defaultSleep) } } -function restartGateway( +async function restartGateway( runCommand, checkout, expectedSha, @@ -1927,38 +2168,48 @@ function restartGateway( ) { assertExactBuild(checkout, expectedSha); if (!deployment) { - runUpdateCommand( + await runUpdateCommand( runCommand, "gateway.restart", "pnpm", ["openclaw", "gateway", "restart"], checkout, + { + phase: "Gateway restart", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); return { processStartedAt: null, restartStartedAtMs: startedAtMs }; } if (bootstrap) { return { - ...bootstrapManagedGateway(runCommand, checkout, deployment, { + ...(await bootstrapManagedGateway(runCommand, checkout, deployment, { ...options, startupTrace: true, - }), + })), restartStartedAtMs: startedAtMs, }; } const assertOwnership = options.assertNoSystemLaunchDaemonOwnership ?? assertNoSystemLaunchDaemonOwnership; - assertOwnership(deployment.label); - runUpdateCommand( + await assertOwnership(deployment.label); + await runUpdateCommand( runCommand, "gateway.restart", deployment.executable, [...deployment.invocationPrefix, "gateway", "restart"], path.dirname(path.dirname(deployment.entrypoint)), + { + phase: "managed Gateway restart", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); return { processStartedAt: null, restartStartedAtMs: startedAtMs }; } -function bootstrapManagedGateway(runCommand, checkout, deployment, options = {}) { +async function bootstrapManagedGateway(runCommand, checkout, deployment, options = {}) { const plistStat = lstatSync(deployment.plistPath); if (!isTrustedOwnedRegularFile(plistStat)) { throw new UpdateInvariantError( @@ -1968,59 +2219,84 @@ function bootstrapManagedGateway(runCommand, checkout, deployment, options = {}) } const assertOwnership = options.assertNoSystemLaunchDaemonOwnership ?? assertNoSystemLaunchDaemonOwnership; - assertOwnership(deployment.label); + await assertOwnership(deployment.label); const domain = `gui/${process.getuid()}`; const serviceTarget = `${domain}/${deployment.label}`; const waitForProcess = options.waitForProcess ?? waitForManagedGatewayProcess; const now = options.now ?? Date.now; if (!options.startupTrace) { - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.enable", "/bin/launchctl", ["enable", serviceTarget], checkout, + { + phase: "Gateway LaunchAgent enable", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.bootstrap", "/bin/launchctl", ["bootstrap", domain, deployment.plistPath], checkout, + { + phase: "Gateway LaunchAgent bootstrap", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); - waitForProcess(deployment, options.sleep ?? defaultSleep); + await waitForProcess(deployment, options.sleep ?? defaultSleep); return { processStartedAt: timestampAt(now) }; } const readLaunchdEnvironment = options.readLaunchdEnvironment ?? readLaunchdEnvironmentVariable; const armEnvironmentRestore = options.armEnvironmentRestore ?? armLaunchdEnvironmentRestore; - const previousTraceValue = readLaunchdEnvironment(GATEWAY_STARTUP_TRACE_ENV); + const previousTraceValue = await readLaunchdEnvironment(GATEWAY_STARTUP_TRACE_ENV); const environmentRestore = armEnvironmentRestore(GATEWAY_STARTUP_TRACE_ENV, previousTraceValue); let restartError; let processStartedAt = null; - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.setenv", "/bin/launchctl", ["setenv", GATEWAY_STARTUP_TRACE_ENV, "1"], checkout, + { + phase: "Gateway startup trace enable", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); try { - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.enable", "/bin/launchctl", ["enable", serviceTarget], checkout, + { + phase: "Gateway LaunchAgent enable", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); - runUpdateCommand( + await runUpdateCommand( runCommand, "launchd.bootstrap", "/bin/launchctl", ["bootstrap", domain, deployment.plistPath], checkout, + { + phase: "Gateway LaunchAgent bootstrap", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); - waitForProcess(deployment, options.sleep ?? defaultSleep); + await waitForProcess(deployment, options.sleep ?? defaultSleep); processStartedAt = timestampAt(now); } catch (error) { restartError = error; @@ -2028,7 +2304,7 @@ function bootstrapManagedGateway(runCommand, checkout, deployment, options = {}) try { // The booted process already inherited the trace flag. Restore launchd's // previous value immediately so later starts keep the host's normal config. - runUpdateCommand( + await runUpdateCommand( runCommand, previousTraceValue === null ? "launchd.unsetenv" : "launchd.setenv", "/bin/launchctl", @@ -2036,6 +2312,11 @@ function bootstrapManagedGateway(runCommand, checkout, deployment, options = {}) ? ["unsetenv", GATEWAY_STARTUP_TRACE_ENV] : ["setenv", GATEWAY_STARTUP_TRACE_ENV, previousTraceValue], checkout, + { + phase: "Gateway startup trace restore", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayService, + }, ); } catch (cleanupError) { if (restartError) { @@ -2105,7 +2386,11 @@ fi } function readLaunchdEnvironmentVariable(name) { - const result = spawnSync("/bin/launchctl", ["getenv", name], { encoding: "utf8" }); + const result = spawnSync( + "/bin/launchctl", + ["getenv", name], + boundedSyncOptions({ encoding: "utf8" }), + ); if (result.error || result.status !== 0) { throw new UpdateInvariantError( "gateway_restart_failed", @@ -2118,17 +2403,21 @@ function readLaunchdEnvironmentVariable(name) { return value || null; } -function waitForManagedGatewayProcess(deployment, sleep = defaultSleep) { +async function waitForManagedGatewayProcess(deployment, sleep = defaultSleep) { const target = `gui/${process.getuid()}/${deployment.label}`; const attempts = Math.ceil(GATEWAY_PROCESS_START_TIMEOUT_MS / GATEWAY_PROCESS_START_RETRY_DELAY_MS) + 1; for (let attempt = 0; attempt < attempts; attempt += 1) { - const result = spawnSync("/bin/launchctl", ["print", target], { encoding: "utf8" }); + const result = spawnSync( + "/bin/launchctl", + ["print", target], + boundedSyncOptions({ encoding: "utf8" }), + ); if (result.status === 0 && /\bpid\s*=\s*\d+\b/iu.test(result.stdout)) { return; } if (attempt + 1 < attempts) { - sleep(GATEWAY_PROCESS_START_RETRY_DELAY_MS); + await sleep(GATEWAY_PROCESS_START_RETRY_DELAY_MS); } } throw new UpdateInvariantError( @@ -2141,22 +2430,22 @@ function isManagedGatewayLoaded(deployment) { const result = spawnSync( "/bin/launchctl", ["print", `gui/${process.getuid()}/${deployment.label}`], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); return result.status === 0; } -function waitForManagedGatewayReadiness( +async function waitForManagedGatewayReadiness( deployment, probeMilestones = probeGatewayMilestones, sleep = defaultSleep, ) { for (let attempt = 1; attempt <= GATEWAY_READINESS_ATTEMPTS; attempt += 1) { - if (probeMilestones(deployment)?.readyzReady) { + if ((await Promise.resolve(probeMilestones(deployment)))?.readyzReady) { return; } if (attempt < GATEWAY_READINESS_ATTEMPTS) { - sleep(GATEWAY_READINESS_RETRY_DELAY_MS); + await sleep(GATEWAY_READINESS_RETRY_DELAY_MS); } } throw new UpdateInvariantError( @@ -2184,7 +2473,7 @@ function probeGatewayHttp(port, route) { "1", `${scheme}://127.0.0.1:${port}${route}`, ], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }, 5_000), ); if (result.status !== 0) { continue; @@ -2205,7 +2494,7 @@ function probeGatewayMilestones(deployment) { const listeners = spawnSync( "/usr/sbin/lsof", ["-nP", `-iTCP:${deployment.port}`, "-sTCP:LISTEN", "-t"], - { encoding: "utf8" }, + boundedSyncOptions({ encoding: "utf8" }), ); const listenerReady = listeners.status === 0 && Boolean(listeners.stdout.trim()); if (!listenerReady) { @@ -2264,7 +2553,7 @@ function markGatewayMilestones(timing, observation, observedAt, deepRpcUpperBoun } } -function verifyGatewayDeepRpc(runCommand, checkout, expectedSha, deployment, now) { +async function verifyGatewayDeepRpc(runCommand, checkout, expectedSha, deployment, now) { assertExactBuild(checkout, expectedSha); if (deployment) { runBuiltGatewayCli( @@ -2273,18 +2562,23 @@ function verifyGatewayDeepRpc(runCommand, checkout, expectedSha, deployment, now deployment, ); } else { - runUpdateCommand( + await runUpdateCommand( runCommand, "gateway.status", "pnpm", ["openclaw", "gateway", "status", "--deep", "--require-rpc", "--json"], checkout, + { + phase: "Gateway deep status probe", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayProbe, + }, ); } return timestampAt(now); } -function readGatewayHealth(runCommand, checkout, deployment) { +async function readGatewayHealth(runCommand, checkout, deployment) { if (deployment) { const healthOutput = runBuiltGatewayCli( checkout, @@ -2302,29 +2596,40 @@ function readGatewayHealth(runCommand, checkout, deployment) { } return healthSummary; } - runUpdateCommand( + await runUpdateCommand( runCommand, "gateway.health", "pnpm", ["openclaw", "health", "--verbose", "--json"], checkout, + { + phase: "Gateway health probe", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.gatewayProbe, + }, ); return null; } -function verifyGateway(runCommand, checkout, expectedSha, deployment = null, now = Date.now) { - const deepRpcReadyAt = verifyGatewayDeepRpc(runCommand, checkout, expectedSha, deployment, now); +async function verifyGateway(runCommand, checkout, expectedSha, deployment = null, now = Date.now) { + const deepRpcReadyAt = await verifyGatewayDeepRpc( + runCommand, + checkout, + expectedSha, + deployment, + now, + ); return { deepRpcReadyAt, - healthSummary: readGatewayHealth(runCommand, checkout, deployment), + healthSummary: await readGatewayHealth(runCommand, checkout, deployment), }; } function defaultSleep(ms) { - execFileSync("sleep", [String(ms / 1_000)]); + return delay(ms); } -export function verifyGatewayReadiness( +export async function verifyGatewayReadiness( runCommand, checkout, expectedSha, @@ -2347,9 +2652,9 @@ export function verifyGatewayReadiness( for (let attempt = 1; attempt <= GATEWAY_READINESS_ATTEMPTS; attempt += 1) { try { if (deployment) { - markGatewayMilestones(timing, probeMilestones(deployment), timestampAt(now)); + markGatewayMilestones(timing, await probeMilestones(deployment), timestampAt(now)); } - const deepRpcReadyAt = verifyGatewayDeepRpc( + const deepRpcReadyAt = await verifyGatewayDeepRpc( runCommand, checkout, expectedSha, @@ -2360,12 +2665,12 @@ export function verifyGatewayReadiness( if (deployment) { markGatewayMilestones( timing, - probeMilestones(deployment), + await probeMilestones(deployment), timestampAt(now), deepRpcReadyAt, ); } - const healthSummary = readGatewayHealth(runCommand, checkout, deployment); + const healthSummary = await readGatewayHealth(runCommand, checkout, deployment); if (channelConnected(healthSummary, "discord")) { recordGatewayTimestamp(timing, "discordConnectedAt", timestampAt(now)); } @@ -2376,7 +2681,7 @@ export function verifyGatewayReadiness( } catch (error) { lastError = error; if (attempt < GATEWAY_READINESS_ATTEMPTS) { - sleep(GATEWAY_READINESS_RETRY_DELAY_MS); + await sleep(GATEWAY_READINESS_RETRY_DELAY_MS); } } } @@ -2605,7 +2910,11 @@ function defaultAuditGatewayLogs(checkout, sinceMs, deployment = null) { "--timeout", "10000", ], - { cwd: checkout, encoding: "utf8", maxBuffer: 4 * 1024 * 1024 }, + boundedSyncOptions({ + cwd: checkout, + encoding: "utf8", + maxBuffer: 4 * 1024 * 1024, + }), ); } catch (error) { output = readFallbackGatewayLogs(sinceMs); @@ -2628,7 +2937,7 @@ function defaultAuditGatewayLogs(checkout, sinceMs, deployment = null) { return audit; } -function verifyAndAuditGateway({ +async function verifyAndAuditGateway({ runCommand, auditGatewayLogs, checkout, @@ -2643,15 +2952,22 @@ function verifyAndAuditGateway({ let verificationError; let gatewayTiming = timing; try { - gatewayTiming = verifyGatewayReadiness(runCommand, checkout, expectedSha, sleep, deployment, { - timing, - now, - probeMilestones, - }); + gatewayTiming = await verifyGatewayReadiness( + runCommand, + checkout, + expectedSha, + sleep, + deployment, + { + timing, + now, + probeMilestones, + }, + ); } catch (error) { verificationError = error; } - const audit = auditGatewayLogs(checkout, sinceMs, deployment); + const audit = await auditGatewayLogs(checkout, sinceMs, deployment); if (verificationError) { throwPreservingValue(verificationError); } @@ -2692,12 +3008,14 @@ export function findExactMacTarget(processes, executable) { return target ? { executable, pid: Number(target[1]) } : null; } -function defaultVerifyMacTarget(checkout) { - execFileSync("sleep", ["10"]); +async function defaultVerifyMacTarget(checkout) { + await delay(10_000); const executable = path.join(checkout, "dist/OpenClaw.app/Contents/MacOS/OpenClaw"); - const processes = execFileSync("ps", ["axww", "-o", "pid=,command="], { - encoding: "utf8", - }); + const processes = execFileSync( + "ps", + ["axww", "-o", "pid=,command="], + boundedSyncOptions({ encoding: "utf8" }), + ); const target = findExactMacTarget(processes, executable); if (!target) { throw new UpdateInvariantError( @@ -2708,7 +3026,7 @@ function defaultVerifyMacTarget(checkout) { return target; } -export function maintainMain(options, dependencies = {}) { +export async function maintainMain(options, dependencies = {}) { const lock = acquireMaintenanceLock(options.checkout, options.lockPath); if (!lock.acquired) { return { @@ -2716,14 +3034,32 @@ export function maintainMain(options, dependencies = {}) { ok: true, skipped: true, reason: "overlap", - lock: { path: lock.lockPath, ownerPid: lock.owner.pid, startedAt: lock.owner.startedAt }, + lock: { + path: lock.lockPath, + ownerPid: lock.owner.pid, + startedAt: lock.owner.startedAt, + ...(Number.isInteger(lock.owner.processGroupId) + ? { processGroupId: lock.owner.processGroupId } + : {}), + ...(lock.owner.manualRecoveryRequired === true ? { manualRecoveryRequired: true } : {}), + ...(typeof lock.owner.processTreeState === "string" + ? { processTreeState: lock.owner.processTreeState } + : {}), + ...(typeof lock.owner.reason === "string" ? { reason: lock.owner.reason } : {}), + }, }; } let preparedGatewayReplacement = null; + let retainMaintenanceLock = false; try { const verifiedBefore = verifyCheckout(options.checkout, { remote: options.remote }); - const runCommand = dependencies.runCommand ?? defaultRunCommand; + const managedCommand = (command, args, checkout, commandOptions = {}) => + defaultRunCommand(command, args, checkout, { + ...commandOptions, + runManagedCommand: dependencies.runManagedCommand, + }); + const runCommand = dependencies.runCommand ?? managedCommand; const inspectGatewayDeployment = dependencies.inspectGatewayDeployment ?? inspectManagedGatewayDeployment; const repointGatewayDeployment = @@ -2765,7 +3101,7 @@ export function maintainMain(options, dependencies = {}) { dependencies.readLaunchdEnvironment ?? readLaunchdEnvironmentVariable; const armEnvironmentRestore = dependencies.armEnvironmentRestore ?? armLaunchdEnvironmentRestore; - const gatewayDeploymentBefore = inspectGatewayDeployment(verifiedBefore.checkout); + const gatewayDeploymentBefore = await inspectGatewayDeployment(verifiedBefore.checkout); const sourceBuildBeforeUpdate = inspectBuildState( verifiedBefore.checkout, verifiedBefore.headSha, @@ -2776,7 +3112,7 @@ export function maintainMain(options, dependencies = {}) { sourceBuildBeforeUpdate, verifiedBefore.headSha, ); - const update = updateMain(options, dependencies); + const update = await updateMain(options, managedCommand, dependencies); const statePath = options.statePath ?? defaultStatePath(update.checkout); const maintenanceState = readMaintenanceState(statePath); const buildBefore = inspectBuildState(update.checkout, update.afterSha); @@ -2832,7 +3168,7 @@ export function maintainMain(options, dependencies = {}) { try { gatewaySuspension = { status: "offline", - proof: proveGatewayStopped(update.checkout), + proof: await proveGatewayStopped(update.checkout), }; } catch (proofError) { try { @@ -2846,12 +3182,17 @@ export function maintainMain(options, dependencies = {}) { // clean source build cannot mutate its code. Build only to obtain // an exact trusted client for the suspension RPC. if (actions.dependencyInstall) { - runUpdateCommand( + await runUpdateCommand( runCommand, "dependencies.install", "pnpm", ["install", "--frozen-lockfile"], update.checkout, + { + phase: "dependency install", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.dependencyInstall, + }, ); controlDependenciesInstalled = true; } @@ -2861,7 +3202,7 @@ export function maintainMain(options, dependencies = {}) { "managed Gateway snapshot has no exact trusted source control build", ); } - runBuildWithPreservedMacApp(runCommand, update.checkout, sleep); + await runBuildWithPreservedMacApp(runCommand, update.checkout, sleep, "running"); assertExactBuild(update.checkout, update.afterSha); controlBuildPrepared = true; gatewayControlDeployment = resolveGatewayControlDeployment( @@ -2876,7 +3217,7 @@ export function maintainMain(options, dependencies = {}) { "source build did not produce an exact trusted Gateway control client", ); } - gatewaySuspension = prepareSuspension(update.checkout, gatewayControlDeployment); + gatewaySuspension = await prepareSuspension(update.checkout, gatewayControlDeployment); } catch (controlError) { throw aggregateErrorWithCause( [ @@ -2897,12 +3238,12 @@ export function maintainMain(options, dependencies = {}) { } } else { try { - gatewaySuspension = prepareSuspension(update.checkout, gatewayControlDeployment); + gatewaySuspension = await prepareSuspension(update.checkout, gatewayControlDeployment); } catch (prepareError) { try { gatewaySuspension = { status: "offline", - proof: proveGatewayStopped(update.checkout), + proof: await proveGatewayStopped(update.checkout), }; } catch (proofError) { throw aggregateErrorWithCause( @@ -2935,7 +3276,7 @@ export function maintainMain(options, dependencies = {}) { // and avoids source launchers that can rebuild stale dist before stopping. try { if (gatewayDeploymentBefore) { - assertSystemOwnership(gatewayDeploymentBefore.label); + await assertSystemOwnership(gatewayDeploymentBefore.label); } if (gatewayRuntimeRepointRequired) { // Complete every fallible plist rewrite and validation while the @@ -2947,7 +3288,7 @@ export function maintainMain(options, dependencies = {}) { } // launchctl can return before the job and listener have disappeared. // Retarget only after bounded native proof prevents cached snapshot revival. - const stopped = stopManagedGatewayAndProve( + const stopped = await stopManagedGatewayAndProve( runCommand, update.checkout, gatewayDeploymentBefore, @@ -2959,7 +3300,7 @@ export function maintainMain(options, dependencies = {}) { gatewayStoppedForMaintenance = true; } catch (error) { try { - resumeSuspension( + await resumeSuspension( update.checkout, gatewaySuspension.suspensionId, gatewayControlDeployment, @@ -2979,21 +3320,26 @@ export function maintainMain(options, dependencies = {}) { } try { if (actions.dependencyInstall && !controlDependenciesInstalled) { - runUpdateCommand( + await runUpdateCommand( runCommand, "dependencies.install", "pnpm", ["install", "--frozen-lockfile"], update.checkout, + { + phase: "dependency install", + serviceState: "stopped", + timeoutMs: COMMAND_TIMEOUT_MS.dependencyInstall, + }, ); } if (actions.gatewayBuild && !controlBuildPrepared) { - runBuildWithPreservedMacApp(runCommand, update.checkout, sleep); + await runBuildWithPreservedMacApp(runCommand, update.checkout, sleep); } assertExactBuild(update.checkout, update.afterSha); const restartStartedAt = now(); if (gatewayDeploymentBefore) { - assertSystemOwnership(gatewayDeploymentBefore.label); + await assertSystemOwnership(gatewayDeploymentBefore.label); } gatewayDeployment = gatewayDeploymentBefore ? repointGatewayDeployment( @@ -3022,7 +3368,7 @@ export function maintainMain(options, dependencies = {}) { telegramConnectedAt: null, ...gatewayTiming, }; - const restart = restartManagedGateway( + const restart = await restartManagedGateway( runCommand, update.checkout, update.afterSha, @@ -3041,7 +3387,7 @@ export function maintainMain(options, dependencies = {}) { if (typeof restart?.processStartedAt === "string") { recordGatewayTimestamp(gatewayTiming, "processStartedAt", restart.processStartedAt); } - const verification = verifyGatewayAfterRestart({ + const verification = await verifyGatewayAfterRestart({ runCommand, auditGatewayLogs, checkout: update.checkout, @@ -3055,8 +3401,11 @@ export function maintainMain(options, dependencies = {}) { }); gatewayLogAudit = verification?.audit ?? verification; gatewayTiming = finalizeGatewayTiming(verification?.timing ?? gatewayTiming); - gatewayRuntime = verifyGatewayRuntime(update.checkout, update.afterSha); + gatewayRuntime = await verifyGatewayRuntime(update.checkout, update.afterSha); } catch (error) { + if (findUnsafeCommandCleanupFailure(error)) { + throwPreservingValue(error); + } if (!gatewayStoppedForMaintenance || !gatewayDeploymentBefore) { throw error; } @@ -3064,7 +3413,7 @@ export function maintainMain(options, dependencies = {}) { // A failed bootstrap may still have registered or started the // replacement. Bootout is allowed to fail only when native proof // independently confirms that no job or listener remains. - stopManagedGatewayAndProve( + await stopManagedGatewayAndProve( runCommand, update.checkout, gatewayDeploymentBefore, @@ -3073,13 +3422,13 @@ export function maintainMain(options, dependencies = {}) { now, ); preparedGatewayReplacement?.restore?.(); - bootstrapManagedGateway(runCommand, update.checkout, gatewayDeploymentBefore, { + await bootstrapManagedGateway(runCommand, update.checkout, gatewayDeploymentBefore, { now, sleep, waitForProcess: waitForGatewayProcess, assertNoSystemLaunchDaemonOwnership: assertSystemOwnership, }); - waitForManagedGatewayReadiness(gatewayDeploymentBefore, probeMilestones, sleep); + await waitForManagedGatewayReadiness(gatewayDeploymentBefore, probeMilestones, sleep); } catch (recoveryError) { throw aggregateErrorWithCause( [ @@ -3094,15 +3443,20 @@ export function maintainMain(options, dependencies = {}) { } } else { try { - verifyGatewayProbe(runCommand, update.checkout, update.afterSha, gatewayControlDeployment); - gatewayRuntime = verifyGatewayRuntime(update.checkout, update.afterSha); + await verifyGatewayProbe( + runCommand, + update.checkout, + update.afterSha, + gatewayControlDeployment, + ); + gatewayRuntime = await verifyGatewayRuntime(update.checkout, update.afterSha); } catch { actions.gatewayRestart = true; actions.gatewaySelfHeal = true; const bootstrap = - gatewayControlDeployment !== null && !isGatewayLoaded(gatewayControlDeployment); + gatewayControlDeployment !== null && !(await isGatewayLoaded(gatewayControlDeployment)); const restartStartedAt = now(); - const restart = restartManagedGateway( + const restart = await restartManagedGateway( runCommand, update.checkout, update.afterSha, @@ -3134,7 +3488,7 @@ export function maintainMain(options, dependencies = {}) { if (typeof restart?.processStartedAt === "string") { recordGatewayTimestamp(gatewayTiming, "processStartedAt", restart.processStartedAt); } - const verification = verifyGatewayAfterRestart({ + const verification = await verifyGatewayAfterRestart({ runCommand, auditGatewayLogs, checkout: update.checkout, @@ -3148,7 +3502,7 @@ export function maintainMain(options, dependencies = {}) { }); gatewayLogAudit = verification?.audit ?? verification; gatewayTiming = finalizeGatewayTiming(verification?.timing ?? gatewayTiming); - gatewayRuntime = verifyGatewayRuntime(update.checkout, update.afterSha); + gatewayRuntime = await verifyGatewayRuntime(update.checkout, update.afterSha); } } if (actions.macAppRebuild) { @@ -3162,7 +3516,7 @@ export function maintainMain(options, dependencies = {}) { // The exact-SHA JS build above already produced dist/control-ui. Letting // Mac packaging rebuild it can empty dist while the live app bundle is // there, defeating the staged-swap guarantee. - runUpdateCommand( + await runUpdateCommand( runCommand, "mac.restart", "env", @@ -3176,9 +3530,14 @@ export function maintainMain(options, dependencies = {}) { "--target-only", ], update.checkout, + { + phase: "Mac app rebuild", + serviceState: "running", + timeoutMs: COMMAND_TIMEOUT_MS.macAppRebuild, + }, ); - const macTarget = verifyMacTarget(update.checkout); - verifyGatewayProbe( + const macTarget = await verifyMacTarget(update.checkout); + await verifyGatewayProbe( runCommand, update.checkout, update.afterSha, @@ -3220,9 +3579,23 @@ export function maintainMain(options, dependencies = {}) { ...(gatewayRuntime ? { gatewayRuntime } : {}), ...(maintenanceState.macTarget ? { macTarget: maintenanceState.macTarget } : {}), }; + } catch (error) { + const cleanupFailure = findUnsafeCommandCleanupFailure(error); + if (cleanupFailure) { + retainMaintenanceLock = true; + lock.retainForCleanupFailure(cleanupFailure.details); + cleanupFailure.details = { + ...cleanupFailure.details, + lockPath: lock.lockPath, + lockRetained: true, + }; + } + return throwPreservingValue(error); } finally { preparedGatewayReplacement?.discard(); - lock.release(); + if (!retainMaintenanceLock) { + lock.release(); + } } } @@ -3247,9 +3620,9 @@ function parseArgs(argv) { return options; } -function main(argv = process.argv.slice(2)) { +export async function runLiveUpdaterMain(argv = process.argv.slice(2), dependencies = {}) { try { - console.log(JSON.stringify(maintainMain(parseArgs(argv)))); + console.log(JSON.stringify(await maintainMain(parseArgs(argv), dependencies))); } catch (error) { console.log(JSON.stringify(formatUpdateFailure(error))); process.exitCode = 1; @@ -3257,5 +3630,5 @@ function main(argv = process.argv.slice(2)) { } if (isDirectRunUrl(process.argv[1], import.meta.url)) { - main(); + void runLiveUpdaterMain(); } diff --git a/scripts/lib/managed-child-process.d.mts b/scripts/lib/managed-child-process.d.mts index 81ac94fcf725..ff4d54a62038 100644 --- a/scripts/lib/managed-child-process.d.mts +++ b/scripts/lib/managed-child-process.d.mts @@ -18,9 +18,17 @@ export function terminateManagedChild( runTaskkill, }?: { platform?: NodeJS.Platform; - runTaskkill?: (command: string, args?: string[]) => { error?: Error; status: number | null }; + runTaskkill?: ( + command: string, + args?: string[], + options?: { + killSignal?: NodeJS.Signals; + stdio?: import("node:child_process").StdioOptions; + timeout?: number; + }, + ) => { error?: Error; status: number | null }; }, -): void; +): { processTreeState: "indeterminate" | "signaled" | "terminated" } | undefined; /** * Run a child command while forwarding termination signals to the managed process group. * @@ -35,6 +43,8 @@ export function terminateManagedChild( * platform?: NodeJS.Platform; * comSpec?: string; * timeoutMs?: number; + * requireProcessTreeExit?: boolean; + * runTaskkill?: typeof spawnSync; * onReady?: (child: import("node:child_process").ChildProcess) => void; * }} options * @returns {Promise} @@ -50,6 +60,8 @@ export function runManagedCommand({ windowsVerbatimArguments, comSpec, timeoutMs, + requireProcessTreeExit, + runTaskkill, onReady, }: { bin: string; @@ -62,6 +74,16 @@ export function runManagedCommand({ platform?: NodeJS.Platform; comSpec?: string; timeoutMs?: number; + requireProcessTreeExit?: boolean; + runTaskkill?: ( + command: string, + args?: string[], + options?: { + killSignal?: NodeJS.Signals; + stdio?: import("node:child_process").StdioOptions; + timeout?: number; + }, + ) => { error?: Error; status: number | null }; onReady?: (child: import("node:child_process").ChildProcess) => void; }): Promise; /** diff --git a/scripts/lib/managed-child-process.mjs b/scripts/lib/managed-child-process.mjs index b2b2e3d22f03..b6f6aa3e6061 100644 --- a/scripts/lib/managed-child-process.mjs +++ b/scripts/lib/managed-child-process.mjs @@ -6,6 +6,9 @@ import { resolveWindowsTaskkillPath } from "./windows-taskkill.mjs"; const FORWARDED_SIGNALS = ["SIGINT", "SIGTERM", "SIGHUP"]; const FORCE_KILL_DELAY_MS = 5_000; +const PROCESS_GROUP_DRAIN_TIMEOUT_MS = 5_000; +const PROCESS_GROUP_POLL_MS = 25; +const TASKKILL_TIMEOUT_MS = 10_000; const managedChildren = new Set(); const signalHandlers = new Map(); @@ -24,6 +27,7 @@ export function signalExitCode(signal) { * @param {import("node:child_process").ChildProcess} child * @param {NodeJS.Signals} [signal] * @param {{ platform?: NodeJS.Platform; runTaskkill?: typeof spawnSync }} [options] + * @returns {{ processTreeState: "indeterminate" | "signaled" | "terminated" } | undefined} */ export function terminateManagedChild( child, @@ -31,13 +35,13 @@ export function terminateManagedChild( { platform = process.platform, runTaskkill = spawnSync } = {}, ) { if (!child.pid) { - return; + return platform === "win32" ? { processTreeState: "indeterminate" } : undefined; } try { if (platform !== "win32") { process.kill(-child.pid, signal); - return; + return { processTreeState: "signaled" }; } } catch (error) { if (!isMissingProcessError(error)) { @@ -47,7 +51,7 @@ export function terminateManagedChild( // The process may have already exited between the group kill and fallback kill. } } - return; + return isMissingProcessError(error) ? { processTreeState: "terminated" } : undefined; } if (platform === "win32") { @@ -56,19 +60,29 @@ export function terminateManagedChild( if (signal === "SIGKILL") { args.push("/F"); } - const result = runTaskkill(taskkillPath, args, { stdio: "ignore" }); + const taskkillOptions = { + killSignal: "SIGKILL", + stdio: "ignore", + timeout: TASKKILL_TIMEOUT_MS, + }; + const result = runTaskkill(taskkillPath, args, taskkillOptions); if (!result?.error && result?.status === 0) { - return; + return { processTreeState: "terminated" }; } if (signal !== "SIGKILL") { - const forceResult = runTaskkill(taskkillPath, [...args, "/F"], { stdio: "ignore" }); + const forceResult = runTaskkill(taskkillPath, [...args, "/F"], taskkillOptions); if (!forceResult?.error && forceResult?.status === 0) { - return; + return { processTreeState: "terminated" }; } } + try { + child.kill(signal); + } catch { + // The leader may already be gone, but failed taskkill leaves descendants unverified. + } + return { processTreeState: "indeterminate" }; } - - child.kill(signal); + return undefined; } /** @@ -85,6 +99,8 @@ export function terminateManagedChild( * platform?: NodeJS.Platform; * comSpec?: string; * timeoutMs?: number; + * requireProcessTreeExit?: boolean; + * runTaskkill?: typeof spawnSync; * onReady?: (child: import("node:child_process").ChildProcess) => void; * }} options * @returns {Promise} @@ -100,8 +116,13 @@ export async function runManagedCommand({ windowsVerbatimArguments, comSpec, timeoutMs, + requireProcessTreeExit = false, + runTaskkill = spawnSync, onReady, }) { + if (platform === "win32" && requireProcessTreeExit) { + throw createManagedCommandUnsupportedTreeVerificationError(); + } const spawnSpec = createManagedCommandSpawnSpec({ bin, args, @@ -120,14 +141,23 @@ export async function runManagedCommand({ receivedSignal: null, }; addManagedChild(managedChild); - onReady?.(child); let timeoutTimer = null; + let signalTimeout; let timedOut = false; + let childResult; + let timeoutTermination; + const timeoutTriggered = new Promise((resolve) => { + signalTimeout = resolve; + }); try { - return await new Promise((resolve, reject) => { + const childCompletion = new Promise((resolve, reject) => { child.once("error", reject); child.once("close", (status, signal) => { + if (timeoutTimer) { + clearTimeout(timeoutTimer); + timeoutTimer = null; + } if (managedChild.forceKillTimer) { clearTimeout(managedChild.forceKillTimer); } @@ -146,10 +176,60 @@ export async function runManagedCommand({ timeoutTimer = setTimeout(() => { timedOut = true; // Shell commands may spawn grandchildren, so timeout cleanup owns the whole tree. - terminateManagedChild(child, "SIGKILL", { platform }); + timeoutTermination = terminateManagedChild(child, "SIGKILL", { + platform, + runTaskkill, + }); + signalTimeout(); }, timeoutMs); } }); + const childOutcome = childCompletion.then( + (status) => ({ status, type: "completed" }), + (/** @type {unknown} */ error) => ({ error, type: "failed" }), + ); + try { + onReady?.(child); + } catch (error) { + const setupTermination = terminateManagedChild(child, "SIGKILL", { + platform, + runTaskkill, + }); + try { + await ensureManagedProcessTreeExit(child, platform, { + windowsTermination: setupTermination, + }); + } catch (cleanupError) { + throw createManagedCommandSetupCleanupError(error, cleanupError); + } + throw error; + } + const outcome = + timeoutMs === undefined + ? await childOutcome + : await Promise.race([childOutcome, timeoutTriggered.then(() => ({ type: "timeout" }))]); + if (outcome.type === "timeout") { + await ensureManagedProcessTreeExit(child, platform, { + windowsTermination: timeoutTermination, + }); + throw createManagedCommandTimeoutError(timeoutMs); + } + if (outcome.type === "failed") { + if (timedOut) { + await ensureManagedProcessTreeExit(child, platform, { + windowsTermination: timeoutTermination, + }); + } + throw outcome.error; + } + childResult = outcome.status; + if (requireProcessTreeExit) { + await ensureManagedProcessTreeExit(child, platform, { + rejectIfLive: true, + terminateIfLive: true, + }); + } + return childResult; } finally { if (timeoutTimer) { clearTimeout(timeoutTimer); @@ -164,6 +244,105 @@ function createManagedCommandTimeoutError(timeoutMs) { }); } +function createManagedCommandUnsupportedTreeVerificationError() { + return Object.assign( + new Error("Strict managed process-tree verification is not supported on Windows"), + { + code: "EPROCESS_TREE_VERIFICATION_UNSUPPORTED", + }, + ); +} + +function createManagedCommandSetupCleanupError(error, cleanupError) { + return new AggregateError( + [error, cleanupError], + "Managed command setup failed and its process tree could not be cleaned up", + { cause: cleanupError }, + ); +} + +function processGroupStatus(pid) { + if (!Number.isSafeInteger(pid) || pid <= 1 || pid > 0x7fffffff) { + return "indeterminate"; + } + try { + process.kill(-pid, 0); + return "live"; + } catch (error) { + return error?.code === "ESRCH" ? "dead" : "indeterminate"; + } +} + +async function ensureManagedProcessTreeExit( + child, + platform, + { rejectIfLive = false, terminateIfLive = false, windowsTermination } = {}, +) { + if (platform === "win32") { + if (windowsTermination?.processTreeState === "indeterminate") { + throw createManagedCommandCleanupError( + "Windows taskkill could not verify managed process tree exit", + child, + platform, + "indeterminate", + ); + } + return; + } + const initialStatus = processGroupStatus(child.pid); + if (initialStatus === "dead") { + return; + } + let status = initialStatus; + let sawLive = initialStatus === "live"; + if (terminateIfLive) { + terminateManagedChild(child, "SIGKILL", { platform }); + } + const deadline = Date.now() + PROCESS_GROUP_DRAIN_TIMEOUT_MS; + while (Date.now() < deadline) { + await new Promise((resolve) => { + setTimeout(resolve, PROCESS_GROUP_POLL_MS); + }); + status = processGroupStatus(child.pid); + if (status === "dead") { + if (rejectIfLive && sawLive) { + throw createManagedCommandCleanupError( + "Managed command exited while its process group remained active", + child, + platform, + "terminated", + ); + } + return; + } + if (status === "live") { + sawLive = true; + } + } + const processTreeState = status === "indeterminate" ? "indeterminate" : "live"; + throw createManagedCommandCleanupError( + processTreeState === "indeterminate" + ? `Managed process-group state remained indeterminate for ${PROCESS_GROUP_DRAIN_TIMEOUT_MS}ms` + : `Managed process group did not exit within ${PROCESS_GROUP_DRAIN_TIMEOUT_MS}ms`, + child, + platform, + processTreeState, + ); +} + +function createManagedCommandCleanupError(message, child, platform, processTreeState) { + const processGroupId = + platform !== "win32" && Number.isSafeInteger(child.pid) && child.pid > 1 + ? child.pid + : undefined; + return Object.assign(new Error(message), { + code: "EPROCESSGROUP_CLEANUP_FAILED", + ...(platform === "win32" ? { manualRecoveryRequired: true } : {}), + ...(processGroupId === undefined ? {} : { processGroupId }), + processTreeState, + }); +} + /** * Build the spawn command, args, and options used by managed command execution. * diff --git a/test/external-script-modules.d.ts b/test/external-script-modules.d.ts index 3fe626d8b546..ef47a9bac998 100644 --- a/test/external-script-modules.d.ts +++ b/test/external-script-modules.d.ts @@ -250,7 +250,23 @@ declare module "*openclaw-live-updater/scripts/update-main.mjs" { requestedPath?: string, ): { acquired: boolean; - owner: { pid: number; checkout?: string; startedAt?: string }; + owner: { + pid: number; + checkout?: string; + manualRecoveryRequired?: boolean; + processGroupId?: number; + processTreeState?: string; + reason?: string; + serviceState?: string; + startedAt?: string; + }; + retainForCleanupFailure?: (details?: { + manualRecoveryRequired?: boolean; + phase?: string; + processGroupId?: number; + processTreeState?: string; + serviceState?: string; + }) => Record; release?: () => void; }; export function parseGatewayLogAudit( @@ -289,10 +305,15 @@ declare module "*openclaw-live-updater/scripts/update-main.mjs" { options?: { stderr?: "inherit" | "pipe"; timeoutMs?: number }, ): string; export function verifyGatewayReadiness( - runCommand: (command: string, args: string[], checkout: string) => unknown, + runCommand: ( + command: string, + args: string[], + checkout: string, + options?: Record, + ) => unknown | Promise, checkout: string, expectedSha: string, - sleep?: (ms: number) => void, + sleep?: (ms: number) => void | Promise, deployment?: GatewayDeployment | null, options?: { now?: () => number; @@ -303,7 +324,7 @@ declare module "*openclaw-live-updater/scripts/update-main.mjs" { }; timing?: Record; }, - ): Record; + ): Promise>; export function findExactMacTarget( processes: string, executable: string, @@ -311,5 +332,9 @@ declare module "*openclaw-live-updater/scripts/update-main.mjs" { export function maintainMain( options: Record, dependencies?: Record, - ): UpdateResult; + ): Promise; + export function runLiveUpdaterMain( + argv?: string[], + dependencies?: Record, + ): Promise; } diff --git a/test/scripts/managed-child-process.test.ts b/test/scripts/managed-child-process.test.ts index c3cb0f50ebff..81f6528b2945 100644 --- a/test/scripts/managed-child-process.test.ts +++ b/test/scripts/managed-child-process.test.ts @@ -146,7 +146,9 @@ describe("managed-child-process", () => { runTaskkill, }); expect(runTaskkill).toHaveBeenNthCalledWith(1, taskkillPath, ["/PID", "12345", "/T"], { + killSignal: "SIGKILL", stdio: "ignore", + timeout: 10_000, }); terminateManagedChild(child, "SIGKILL", { @@ -154,7 +156,9 @@ describe("managed-child-process", () => { runTaskkill, }); expect(runTaskkill).toHaveBeenNthCalledWith(2, taskkillPath, ["/PID", "12345", "/T", "/F"], { + killSignal: "SIGKILL", stdio: "ignore", + timeout: 10_000, }); expect(child.kill).not.toHaveBeenCalled(); }); @@ -177,10 +181,14 @@ describe("managed-child-process", () => { }); expect(runTaskkill).toHaveBeenNthCalledWith(1, taskkillPath, ["/PID", "12345", "/T"], { + killSignal: "SIGKILL", stdio: "ignore", + timeout: 10_000, }); expect(runTaskkill).toHaveBeenNthCalledWith(2, taskkillPath, ["/PID", "12345", "/T", "/F"], { + killSignal: "SIGKILL", stdio: "ignore", + timeout: 10_000, }); expect(child.kill).not.toHaveBeenCalled(); }); @@ -259,8 +267,8 @@ setInterval(() => {}, 1_000); childPid = Number(fs.readFileSync(childPidPath, "utf8")); descendantPid = Number(fs.readFileSync(descendantPidPath, "utf8")); - await waitFor(() => !isProcessAlive(childPid), 1_500); - await waitFor(() => !isProcessAlive(descendantPid), 1_500); + expect(isProcessAlive(childPid)).toBe(false); + expect(isProcessAlive(descendantPid)).toBe(false); } finally { if (childPid && isProcessAlive(childPid)) { process.kill(childPid, "SIGKILL"); @@ -271,6 +279,221 @@ setInterval(() => {}, 1_000); } }); + it("uses a wall timeout even while the child emits progress", async () => { + const startedAt = Date.now(); + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setInterval(() => process.stderr.write('retrying\\n'), 20)"], + shell: false, + stdio: "ignore", + timeoutMs: 200, + }), + ).rejects.toMatchObject({ code: "ETIMEDOUT" }); + expect(Date.now() - startedAt).toBeLessThan(2_000); + }); + + it("refuses strict Windows commands before spawning an unverifiable tree", async () => { + const onReady = vi.fn(); + const runTaskkill = vi.fn(); + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "process.exit(0)"], + onReady, + platform: "win32", + requireProcessTreeExit: true, + runTaskkill, + shell: false, + stdio: "ignore", + timeoutMs: 1_000, + }), + ).rejects.toMatchObject({ + code: "EPROCESS_TREE_VERIFICATION_UNSUPPORTED", + }); + expect(onReady).not.toHaveBeenCalled(); + expect(runTaskkill).not.toHaveBeenCalled(); + }); + + it("fails closed when Windows taskkill cannot verify timeout cleanup", async () => { + const originalSystemRoot = process.env.SystemRoot; + const originalWindir = process.env.WINDIR; + let childPid = 0; + const runTaskkill = vi.fn(() => ({ + error: Object.assign(new Error("taskkill timed out"), { code: "ETIMEDOUT" }), + status: null, + })); + try { + process.env.SystemRoot = "C:\\Windows"; + delete process.env.WINDIR; + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setInterval(() => {}, 1_000)"], + onReady: (child) => { + childPid = expectProcessPid(child.pid); + }, + platform: "win32", + runTaskkill, + shell: false, + stdio: "ignore", + timeoutMs: 200, + }), + ).rejects.toMatchObject({ + code: "EPROCESSGROUP_CLEANUP_FAILED", + manualRecoveryRequired: true, + processTreeState: "indeterminate", + }); + + expect(runTaskkill).toHaveBeenCalledWith( + taskkillPath, + ["/PID", String(childPid), "/T", "/F"], + { + killSignal: "SIGKILL", + stdio: "ignore", + timeout: 10_000, + }, + ); + await waitFor(() => !isProcessAlive(childPid)); + } finally { + restoreEnvValue("SystemRoot", originalSystemRoot); + restoreEnvValue("WINDIR", originalWindir); + if (childPid && isProcessAlive(childPid)) { + process.kill(childPid, "SIGKILL"); + } + } + }); + + posixIt("does not wait indefinitely when a timed-out child omits close", async () => { + const startedAt = Date.now(); + let childPid = 0; + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setInterval(() => {}, 1_000)"], + onReady: (child) => { + childPid = expectProcessPid(child.pid); + child.removeAllListeners("close"); + }, + shell: false, + stdio: "ignore", + timeoutMs: 200, + }), + ).rejects.toMatchObject({ code: "ETIMEDOUT" }); + + expect(Date.now() - startedAt).toBeLessThan(2_000); + expect(isProcessAlive(childPid)).toBe(false); + }); + + posixIt("waits through transient indeterminate process-group state", async () => { + const originalKill = process.kill; + let childPid = 0; + let injectedIndeterminate = false; + process.kill = ((pid: number, signal?: NodeJS.Signals | number) => { + if (pid === -childPid && signal === 0 && !injectedIndeterminate) { + injectedIndeterminate = true; + throw Object.assign(new Error("transient process-group state"), { code: "EPERM" }); + } + return originalKill(pid, signal); + }) as typeof process.kill; + + try { + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setInterval(() => {}, 1_000)"], + onReady: (child) => { + childPid = expectProcessPid(child.pid); + }, + shell: false, + stdio: "ignore", + timeoutMs: 200, + }), + ).rejects.toMatchObject({ code: "ETIMEDOUT" }); + } finally { + process.kill = originalKill; + } + + expect(injectedIndeterminate).toBe(true); + expect(isProcessAlive(childPid)).toBe(false); + }); + + it("allows bounded retry output and normal long-running work to complete", async () => { + await expect( + runManagedCommand({ + bin: process.execPath, + args: [ + "-e", + "process.stderr.write('network retry 1\\n'); setTimeout(() => process.exit(0), 100)", + ], + shell: false, + stdio: "ignore", + timeoutMs: 1_000, + }), + ).resolves.toBe(0); + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setTimeout(() => process.exit(0), 200)"], + requireProcessTreeExit: true, + shell: false, + stdio: "ignore", + timeoutMs: 1_000, + }), + ).resolves.toBe(0); + }); + + it("cleans up the child when onReady throws", async () => { + let childPid = 0; + await expect( + runManagedCommand({ + bin: process.execPath, + args: ["-e", "setInterval(() => {}, 1_000)"], + onReady: (child) => { + childPid = expectProcessPid(child.pid); + throw new Error("setup failed"); + }, + shell: false, + stdio: "ignore", + }), + ).rejects.toThrow("setup failed"); + expect(isProcessAlive(childPid)).toBe(false); + }); + + posixIt("rejects and drains descendants left after a successful leader exit", async () => { + const dir = createTempDir("openclaw-managed-lingering-"); + const descendantPidPath = path.join(dir, "descendant.pid"); + let descendantPid = 0; + try { + await expect( + runManagedCommand({ + bin: process.execPath, + args: [ + "-e", + ` +const { spawn } = require("node:child_process"); +const fs = require("node:fs"); +const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); +child.unref(); +fs.writeFileSync(process.argv[1], String(child.pid)); +`, + descendantPidPath, + ], + requireProcessTreeExit: true, + shell: false, + stdio: "ignore", + timeoutMs: 1_000, + }), + ).rejects.toMatchObject({ code: "EPROCESSGROUP_CLEANUP_FAILED" }); + descendantPid = Number(fs.readFileSync(descendantPidPath, "utf8")); + expect(isProcessAlive(descendantPid)).toBe(false); + } finally { + if (descendantPid && isProcessAlive(descendantPid)) { + process.kill(descendantPid, "SIGKILL"); + } + } + }); + posixIt( "kills managed child process group descendants when the runner is terminated", async () => { diff --git a/test/scripts/openclaw-live-updater.test.ts b/test/scripts/openclaw-live-updater.test.ts index 90c9a41ab546..6e84a24aa231 100644 --- a/test/scripts/openclaw-live-updater.test.ts +++ b/test/scripts/openclaw-live-updater.test.ts @@ -17,7 +17,7 @@ import { import { tmpdir } from "node:os"; import path from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; -import { afterAll, beforeAll, describe, expect, test } from "vitest"; +import { afterAll, beforeAll, describe, expect, test, vi } from "vitest"; import { acquireMaintenanceLock, assertNoSystemLaunchDaemonOwnership, @@ -40,6 +40,7 @@ import { resolveManagedGatewayEntrypoint, runBuiltGatewayCall, runBuiltGatewayCli, + runLiveUpdaterMain, verifyGatewayReadiness, } from "../../.agents/skills/openclaw-live-updater/scripts/update-main.mjs"; import { @@ -52,6 +53,7 @@ const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../ const script = path.join(repoRoot, ".agents/skills/openclaw-live-updater/scripts/update-main.mjs"); const fixtureOrigins = new Map(); let fixtureTemplate: ReturnType | undefined; +const posixTest = process.platform === "win32" ? test.skip : test; function git(cwd: string, ...args: string[]) { return execFileSync("git", args, { cwd, encoding: "utf8" }).trim(); @@ -220,6 +222,10 @@ function passGatewayRestartVerification({ timing }: { timing: Record { beforeAll(() => { const root = realpathSync(mkdtempSync(path.join(tmpdir(), "openclaw-live-updater-template-"))); @@ -746,14 +752,14 @@ describe("openclaw live updater", () => { ).toBe("/srv/runtime/gateway-abc/dist"); }); - test("retries bounded Gateway readiness after restart", () => { + test("retries bounded Gateway readiness after restart", async () => { const { mirror } = makeFixture(); writeBuild(mirror); const calls: string[] = []; const delays: number[] = []; let statusAttempts = 0; - verifyGatewayReadiness( + await verifyGatewayReadiness( (command: string, args: string[]) => { const call = [command, ...args].join(" "); calls.push(call); @@ -763,7 +769,9 @@ describe("openclaw live updater", () => { }, mirror, git(mirror, "rev-parse", "HEAD"), - (ms: number) => delays.push(ms), + (ms: number) => { + delays.push(ms); + }, ); expect(delays).toEqual([5_000, 5_000, 5_000, 5_000, 5_000, 5_000]); @@ -779,7 +787,7 @@ describe("openclaw live updater", () => { ]); }); - test("records listener, probe, RPC, and channel readiness timestamps", () => { + test("records listener, probe, RPC, and channel readiness timestamps", async () => { const { root, mirror } = makeFixture(); writeBuild(mirror); const entrypoint = path.join(mirror, "dist/index.js"); @@ -797,7 +805,7 @@ console.log(JSON.stringify(command === "health" ? { ); const configPath = path.join(root, "openclaw.json"); writeFileSync(configPath, "{}\n"); - const timing = verifyGatewayReadiness( + const timing = await verifyGatewayReadiness( () => {}, mirror, git(mirror, "rev-parse", "HEAD"), @@ -839,7 +847,7 @@ console.log(JSON.stringify(command === "health" ? { expect(isGatewayProbeResponse("/readyz", { ok: true, status: "ready" })).toBe(false); }); - test("bounds milestones first observed during the deep RPC probe", () => { + test("bounds milestones first observed during the deep RPC probe", async () => { const { root, mirror } = makeFixture(); writeBuild(mirror); const entrypoint = path.join(mirror, "dist/index.js"); @@ -858,7 +866,7 @@ console.log(JSON.stringify(command === "health" ? { ok: true, channels: {} } : { ].map(Date.parse); let probeCalls = 0; - const timing = verifyGatewayReadiness( + const timing = await verifyGatewayReadiness( () => {}, mirror, git(mirror, "rev-parse", "HEAD"), @@ -899,7 +907,7 @@ console.log(JSON.stringify(command === "health" ? { ok: true, channels: {} } : { }); }); - test("does not fail readiness for a present but disconnected channel record", () => { + test("does not fail readiness for a present but disconnected channel record", async () => { const { root, mirror } = makeFixture(); writeBuild(mirror); const entrypoint = path.join(mirror, "dist/index.js"); @@ -916,7 +924,7 @@ console.log(JSON.stringify(command === "health" ? { writeFileSync(configPath, "{}\n"); expect( - verifyGatewayReadiness( + await verifyGatewayReadiness( () => {}, mirror, git(mirror, "rev-parse", "HEAD"), @@ -941,7 +949,7 @@ console.log(JSON.stringify(command === "health" ? { ).toMatchObject({ discordConnectedAt: null }); }); - test("routes managed Gateway health through the injected port", () => { + test("routes managed Gateway health through the injected port", async () => { const { root, mirror } = makeFixture(); writeBuild(mirror); const entrypoint = path.join(mirror, "dist/index.js"); @@ -959,7 +967,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); `, ); - verifyGatewayReadiness( + await verifyGatewayReadiness( () => { throw new Error("managed probes must use the exact built Gateway CLI"); }, @@ -1422,7 +1430,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }); }); - test("fast-forwards, builds exact SHA, restarts Gateway, then proves exact Mac target", () => { + test("fast-forwards, builds exact SHA, restarts Gateway, then proves exact Mac target", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(seed, "apps/macos/Sources/OpenClaw"), { recursive: true }); writeFileSync(path.join(seed, "apps/macos/Sources/OpenClaw/App.swift"), "// changed\n"); @@ -1431,7 +1439,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); git(seed, "push"); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", @@ -1482,7 +1490,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ); }); - test("rejects a local main that is ahead of origin main", () => { + test("rejects a local main that is ahead of origin main", async () => { const { root, mirror } = makeFixture(); git(mirror, "config", "user.name", "Test"); git(mirror, "config", "user.email", "test@example.com"); @@ -1490,21 +1498,21 @@ console.log(JSON.stringify({ ok: true, channels: {} })); git(mirror, "add", "local-commit.txt"); git(mirror, "commit", "-m", "local commit"); - expect(() => + await expect( maintainFixture({ checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock"), }), - ).toThrow(/does not equal origin\/main/u); + ).rejects.toThrow(/does not equal origin\/main/u); }); - test("builds and restarts when build output is missing without a new commit", () => { + test("builds and restarts when build output is missing without a new commit", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand }, ); @@ -1523,12 +1531,12 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("defers a stale build without stopping Gateway when atomic suspension reports active work", () => { + test("defers a stale build without stopping Gateway when atomic suspension reports active work", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -1556,13 +1564,13 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(inspectBuildState(mirror, git(mirror, "rev-parse", "HEAD")).current).toBe(false); }); - test("accepts native stopped proof when the stop command reports an error", () => { + test("accepts native stopped proof when the stop command reports an error", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); const resumed: string[] = []; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand(command: string, args: string[]) { @@ -1581,14 +1589,400 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(resumed).toEqual([]); }); - test("resumes a prepared suspension when stopped proof never converges", () => { + test("reports a pre-stop fetch timeout without stopping Gateway and releases the lock", async () => { + const { root, mirror } = makeFixture(); + const lockPath = path.join(root, "maintenance.lock"); + const calls: string[] = []; + + await expect( + maintainFixture( + { checkout: mirror, remote: "origin", lockPath }, + { + fetchMain: undefined, + runManagedCommand: async ({ + args, + requireProcessTreeExit, + timeoutMs, + }: { + args: string[]; + requireProcessTreeExit: boolean; + timeoutMs: number; + }) => { + calls.push(`${args.join(" ")} timeout=${timeoutMs} strict=${requireProcessTreeExit}`); + throw managedTimeoutError(); + }, + }, + ), + ).rejects.toMatchObject({ + code: "command_timeout", + details: { + phase: "Git fetch", + serviceState: "running", + timeoutMs: 5 * 60_000, + }, + }); + expect(calls).toEqual([ + `-C ${mirror} fetch --prune origin refs/heads/main:refs/remotes/origin/main timeout=300000 strict=true`, + ]); + expect(existsSync(lockPath)).toBe(false); + }); + + test("refuses unsupported Windows tree verification before stopping Gateway", async () => { + const { root, mirror } = makeFixture(); + const lockPath = path.join(root, "maintenance.lock"); + const calls: string[] = []; + + await expect( + maintainFixture( + { checkout: mirror, remote: "origin", lockPath }, + { + fetchMain: undefined, + runManagedCommand: async ({ + args, + requireProcessTreeExit, + }: { + args: string[]; + requireProcessTreeExit: boolean; + }) => { + calls.push(`${args.join(" ")} strict=${requireProcessTreeExit}`); + throw Object.assign(new Error("Windows tree verification is unavailable"), { + code: "EPROCESS_TREE_VERIFICATION_UNSUPPORTED", + }); + }, + }, + ), + ).rejects.toMatchObject({ + code: "unsupported_process_tree_verification", + details: { + phase: "Git fetch", + serviceState: "running", + }, + }); + + expect(calls).toEqual([ + `-C ${mirror} fetch --prune origin refs/heads/main:refs/remotes/origin/main strict=true`, + ]); + expect(existsSync(lockPath)).toBe(false); + }); + + test("emits one machine-readable timeout result with phase details", async () => { + const { mirror } = makeFixture(); + const output: string[] = []; + const log = vi.spyOn(console, "log").mockImplementation((line) => output.push(String(line))); + const previousExitCode = process.exitCode; + process.exitCode = undefined; + try { + await runLiveUpdaterMain(["--checkout", mirror], { + inspectGatewayDeployment: () => null, + runManagedCommand: async () => { + throw managedTimeoutError(); + }, + }); + expect(process.exitCode).toBe(1); + } finally { + process.exitCode = previousExitCode; + log.mockRestore(); + } + + expect(output).toHaveLength(1); + expect(JSON.parse(output[0]!)).toMatchObject({ + schemaVersion: 1, + ok: false, + error: { + code: "command_timeout", + diagnostics: { + kind: "invariant", + code: "command_timeout", + details: { + phase: "Git fetch", + serviceState: "running", + timeoutMs: 5 * 60_000, + }, + }, + }, + }); + }); + + test("recovers the previous service after a post-stop install timeout", async () => { + const { root, mirror } = makeFixture(); + writeBuild(mirror); + const lockPath = path.join(root, "maintenance.lock"); + const plistPath = path.join(root, "ai.openclaw.gateway.plist"); + writeFileSync(plistPath, "plist\n", { mode: 0o600 }); + const deployment = { + configPath: path.join(root, "openclaw.json"), + entrypoint: path.join(mirror, "dist/index.js"), + entrypointIndex: 1, + executable: process.execPath, + invocationPrefix: [path.join(mirror, "dist/index.js")], + label: "ai.openclaw.gateway", + plistPath, + port: 18789, + runtime: process.execPath, + }; + const events: string[] = []; + + await expect( + maintainFixture( + { checkout: mirror, remote: "origin", lockPath }, + { + inspectGatewayDeployment: () => deployment, + runManagedCommand: async ({ + args, + requireProcessTreeExit, + timeoutMs, + }: { + args: string[]; + requireProcessTreeExit: boolean; + timeoutMs: number; + }) => { + const command = args.join(" "); + events.push(`${command} timeout=${timeoutMs} strict=${requireProcessTreeExit}`); + if (command === "install --frozen-lockfile") { + await Promise.resolve(); + events.push("install process tree drained"); + throw managedTimeoutError(); + } + return 0; + }, + proveGatewayStopped: () => { + events.push("prove stopped"); + return { + runtimeStatus: "stopped", + port: 18789, + portStatus: "free", + proofSource: "fixture", + }; + }, + waitForGatewayProcess: () => { + events.push("previous process started"); + }, + }, + ), + ).rejects.toMatchObject({ + code: "command_timeout", + details: { + phase: "dependency install", + serviceState: "stopped", + timeoutMs: 15 * 60_000, + }, + }); + + const timeoutIndex = events.indexOf("install process tree drained"); + const recoveryIndex = events.findIndex((event) => event.startsWith("enable gui/")); + expect(timeoutIndex).toBeGreaterThan(-1); + expect(recoveryIndex).toBeGreaterThan(timeoutIndex); + expect(events.filter((event) => event === "prove stopped")).toHaveLength(2); + expect(events).toContain( + `bootstrap gui/${process.getuid?.() ?? 501} ${plistPath} timeout=60000 strict=true`, + ); + expect(events).toContain("previous process started"); + expect(existsSync(lockPath)).toBe(false); + + const reacquired = acquireMaintenanceLock(mirror, lockPath); + try { + expect(reacquired.acquired).toBe(true); + } finally { + reacquired.release?.(); + } + }); + + posixTest( + "retains the maintenance lock and skips recovery when a timed-out process group stays live", + async () => { + const { root, mirror } = makeFixture(); + writeBuild(mirror); + const lockPath = path.join(root, "maintenance.lock"); + const plistPath = path.join(root, "ai.openclaw.gateway.plist"); + writeFileSync(plistPath, "plist\n", { mode: 0o600 }); + const deployment = { + configPath: path.join(root, "openclaw.json"), + entrypoint: path.join(mirror, "dist/index.js"), + entrypointIndex: 1, + executable: process.execPath, + invocationPrefix: [path.join(mirror, "dist/index.js")], + label: "ai.openclaw.gateway", + plistPath, + port: 18789, + runtime: process.execPath, + }; + const blocker = spawn(process.execPath, ["-e", "setInterval(() => {}, 1_000)"], { + detached: true, + stdio: "ignore", + }); + if (!blocker.pid) { + throw new Error("cleanup blocker did not expose a process group id"); + } + const processGroupId = blocker.pid; + const blockerClosed = new Promise((resolve) => blocker.once("close", () => resolve())); + const events: string[] = []; + + try { + await expect( + maintainFixture( + { checkout: mirror, remote: "origin", lockPath }, + { + inspectGatewayDeployment: () => deployment, + runManagedCommand: async ({ args }: { args: string[] }) => { + const command = args.join(" "); + events.push(command); + if (command === "install --frozen-lockfile") { + throw Object.assign(new Error("process group remained live"), { + code: "EPROCESSGROUP_CLEANUP_FAILED", + processGroupId, + processTreeState: "live", + }); + } + return 0; + }, + proveGatewayStopped: () => ({ + runtimeStatus: "stopped", + port: 18789, + portStatus: "free", + proofSource: "fixture", + }), + waitForGatewayProcess: () => { + events.push("previous process started"); + }, + }, + ), + ).rejects.toMatchObject({ + code: "command_cleanup_failed", + details: { + lockPath, + lockRetained: true, + phase: "dependency install", + processGroupId, + processTreeState: "live", + serviceState: "stopped", + }, + }); + + expect(events.some((event) => event.startsWith("enable gui/"))).toBe(false); + expect(events).not.toContain("previous process started"); + expect(existsSync(lockPath)).toBe(true); + expect(acquireMaintenanceLock(mirror, lockPath)).toMatchObject({ + acquired: false, + owner: { + processGroupId, + reason: "command_cleanup_failed", + serviceState: "stopped", + }, + }); + } finally { + try { + process.kill(-processGroupId, "SIGKILL"); + } catch {} + await blockerClosed; + } + + const ownerPath = path.join(lockPath, "owner.json"); + const retainedOwner = JSON.parse(readFileSync(ownerPath, "utf8")); + writeFileSync(ownerPath, `${JSON.stringify({ ...retainedOwner, pid: processGroupId })}\n`, { + mode: 0o600, + }); + const reacquired = acquireMaintenanceLock(mirror, lockPath); + try { + expect(reacquired.acquired).toBe(true); + } finally { + reacquired.release?.(); + } + }, + ); + + test("retains a manual-recovery lock when Windows timeout cleanup is unverified", async () => { + const { root, mirror } = makeFixture(); + writeBuild(mirror); + const lockPath = path.join(root, "maintenance.lock"); + const plistPath = path.join(root, "ai.openclaw.gateway.plist"); + writeFileSync(plistPath, "plist\n", { mode: 0o600 }); + const deployment = { + configPath: path.join(root, "openclaw.json"), + entrypoint: path.join(mirror, "dist/index.js"), + entrypointIndex: 1, + executable: process.execPath, + invocationPrefix: [path.join(mirror, "dist/index.js")], + label: "ai.openclaw.gateway", + plistPath, + port: 18789, + runtime: process.execPath, + }; + const events: string[] = []; + + await expect( + maintainFixture( + { checkout: mirror, remote: "origin", lockPath }, + { + inspectGatewayDeployment: () => deployment, + runManagedCommand: async ({ args }: { args: string[] }) => { + const command = args.join(" "); + events.push(command); + if (command === "install --frozen-lockfile") { + throw Object.assign(new Error("Windows process tree remained unverified"), { + code: "EPROCESSGROUP_CLEANUP_FAILED", + manualRecoveryRequired: true, + processTreeState: "indeterminate", + }); + } + return 0; + }, + proveGatewayStopped: () => ({ + runtimeStatus: "stopped", + port: 18789, + portStatus: "free", + proofSource: "fixture", + }), + waitForGatewayProcess: () => { + events.push("previous process started"); + }, + }, + ), + ).rejects.toMatchObject({ + code: "command_cleanup_failed", + details: { + lockPath, + lockRetained: true, + manualRecoveryRequired: true, + phase: "dependency install", + processTreeState: "indeterminate", + serviceState: "stopped", + }, + }); + + expect(events.some((event) => event.startsWith("enable gui/"))).toBe(false); + expect(events).not.toContain("previous process started"); + expect(existsSync(lockPath)).toBe(true); + expect(acquireMaintenanceLock(mirror, lockPath)).toMatchObject({ + acquired: false, + owner: { + manualRecoveryRequired: true, + reason: "command_cleanup_failed", + serviceState: "stopped", + }, + }); + + const ownerPath = path.join(lockPath, "owner.json"); + const retainedOwner = JSON.parse(readFileSync(ownerPath, "utf8")); + writeFileSync(ownerPath, `${JSON.stringify({ ...retainedOwner, pid: 2_147_483_647 })}\n`, { + mode: 0o600, + }); + expect(acquireMaintenanceLock(mirror, lockPath)).toMatchObject({ + acquired: false, + owner: { + manualRecoveryRequired: true, + }, + }); + rmSync(lockPath, { recursive: true }); + }); + + test("resumes a prepared suspension when stopped proof never converges", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const resumed: string[] = []; let proofAttempts = 0; let sleepAttempts = 0; - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -1604,20 +1998,20 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("native stopped proof did not converge"); + ).rejects.toThrow("native stopped proof did not converge"); expect(proofAttempts).toBe(141); expect(sleepAttempts).toBe(140); expect(resumed).toEqual(["fixture-suspension"]); }); - test("allows launchd teardown to converge after the old ten-second proof window", () => { + test("allows launchd teardown to converge after the old ten-second proof window", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); let elapsedMs = 0; let proofAttempts = 0; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -1645,12 +2039,12 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(proofAttempts).toBe(49); }); - test("recovers a stale build only after proving an unavailable Gateway is stopped", () => { + test("recovers a stale build only after proving an unavailable Gateway is stopped", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -1675,7 +2069,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("preserves typed suspension command diagnostics when stopped proof also fails", () => { + test("preserves typed suspension command diagnostics when stopped proof also fails", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const configPath = path.join(root, "openclaw.json"); @@ -1701,7 +2095,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); try { Object.defineProperty(process, "platform", { value: "linux" }); - maintainFixture( + await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { prepareGatewaySuspension: (checkout: string) => @@ -1762,7 +2156,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(JSON.stringify(formatted)).not.toContain(entrypoint); }); - test("preserves the signed Mac bundle while a Gateway build replaces dist", () => { + test("preserves the signed Mac bundle while a Gateway build replaces dist", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const appBundle = path.join(mirror, "dist/OpenClaw.app"); @@ -1771,7 +2165,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); writeFileSync(appMarker, "signed\n"); const commands = fakeCommands(mirror); - maintainFixture( + await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand(command: string, args: string[]) { @@ -1791,14 +2185,14 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ).toEqual([]); }); - test("restores the Mac bundle when the Gateway build fails", () => { + test("restores the Mac bundle when the Gateway build fails", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const appMarker = path.join(mirror, "dist/OpenClaw.app/Contents/signature-marker"); mkdirSync(path.dirname(appMarker), { recursive: true }); writeFileSync(appMarker, "signed\n"); - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -1809,17 +2203,17 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("build failed"); + ).rejects.toThrow("build failed"); expect(readFileSync(appMarker, "utf8")).toBe("signed\n"); }); - test("reports a standalone command operation without command output", () => { + test("reports a standalone command operation without command output", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); let failure: unknown; try { - maintainFixture( + await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand(command: string, args: string[]) { @@ -1849,7 +2243,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(JSON.stringify(formatted)).not.toContain("secret build message"); }); - test("accepts a delayed external restore of the exact preserved Mac bundle", () => { + test("accepts a delayed external restore of the exact preserved Mac bundle", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const appBundle = path.join(mirror, "dist/OpenClaw.app"); @@ -1860,7 +2254,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); const delayedBundle = path.join(root, "delayed-openclaw.app"); let restored = false; - maintainFixture( + await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand(command: string, args: string[]) { @@ -1895,7 +2289,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ).toEqual([]); }); - test("preserves a build failure after an external Mac bundle restore", () => { + test("preserves a build failure after an external Mac bundle restore", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const appBundle = path.join(mirror, "dist/OpenClaw.app"); @@ -1903,7 +2297,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); mkdirSync(path.dirname(appMarker), { recursive: true }); writeFileSync(appMarker, "signed\n"); - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -1919,17 +2313,17 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("build failed after external restore"); + ).rejects.toThrow("build failed after external restore"); expect(readFileSync(appMarker, "utf8")).toBe("signed\n"); }); - test("proves a current exact-SHA Gateway on a no-op heartbeat", () => { + test("proves a current exact-SHA Gateway on a no-op heartbeat", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand }, ); @@ -1947,7 +2341,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("repoints an ancestor snapshot across the next source update", () => { + test("repoints an ancestor snapshot across the next source update", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); @@ -1978,7 +2372,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); serviceEnvironment: { PRIVATE_MARKER: "not-serialized" }, }); - const deferred = maintainFixture( + const deferred = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -1996,7 +2390,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(commands.calls).toEqual([]); const resumedSuspensions: string[] = []; - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -2014,11 +2408,11 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("replacement plist lint failed"); + ).rejects.toThrow("replacement plist lint failed"); expect(resumedSuspensions).toEqual(["failed-preparation"]); expect(commands.calls).toEqual([]); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -2125,7 +2519,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("restores the previous LaunchAgent after replacement readiness fails", () => { + test("restores the previous LaunchAgent after replacement readiness fails", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); @@ -2152,7 +2546,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); runtime: process.execPath, }); - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -2205,7 +2599,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("replacement readiness failed"); + ).rejects.toThrow("replacement readiness failed"); const uid = process.getuid?.() ?? 501; expect(deployedEntrypoint).toBe(snapshot); @@ -2232,7 +2626,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("reports primary invariant and rollback command diagnostics", () => { + test("reports primary invariant and rollback command diagnostics", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const source = path.join(mirror, "dist/index.js"); @@ -2241,7 +2635,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); let failure: unknown; try { - maintainFixture( + await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { inspectGatewayDeployment: () => ({ @@ -2308,7 +2702,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(JSON.stringify(formatted)).not.toContain("secret rollback"); }); - test("restores an absent managed service past bootout exit 3 and an unlabeled vendor plist", () => { + test("restores an absent managed service past bootout exit 3 and an unlabeled vendor plist", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); @@ -2366,7 +2760,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }); - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -2408,7 +2802,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("replacement readiness failed"); + ).rejects.toThrow("replacement readiness failed"); const uid = process.getuid?.() ?? 501; expect(serviceLoaded).toBe(true); @@ -2420,7 +2814,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("resumes suspension when system ownership appears before bootout", () => { + test("resumes suspension when system ownership appears before bootout", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); @@ -2433,7 +2827,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); writeFileSync(plistPath, "plist\n", { mode: 0o600 }); const resumed: string[] = []; - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -2459,11 +2853,11 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("same-label system owner"); + ).rejects.toThrow("same-label system owner"); expect(resumed).toEqual(["fixture-suspension"]); }); - test("builds a trusted source control client while a snapshot is still running", () => { + test("builds a trusted source control client while a snapshot is still running", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); @@ -2477,7 +2871,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); let controlEntrypoint = ""; let stoppedProofAttempts = 0; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -2558,7 +2952,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("recovers a stopped snapshot when the source control build is missing", () => { + test("recovers a stopped snapshot when the source control build is missing", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const commands = fakeCommands(mirror); @@ -2571,7 +2965,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); let deployedEntrypoint = snapshot; let prepareCalled = false; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -2636,12 +3030,12 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("restores missing dependencies before probing a current build", () => { + test("restores missing dependencies before probing a current build", async () => { const { root, mirror } = makeFixture(); writeBuild(mirror); const commands = fakeCommands(mirror); - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand }, ); @@ -2661,14 +3055,14 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("restarts once when a current exact-SHA Gateway probe fails", () => { + test("restarts once when a current exact-SHA Gateway probe fails", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); const calls: string[] = []; let failed = false; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand(command: string, args: string[]) { @@ -2699,7 +3093,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("bootstraps an owned LaunchAgent left unloaded by a failed restart", () => { + test("bootstraps an owned LaunchAgent left unloaded by a failed restart", async () => { const uid = process.getuid?.() ?? 501; const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); @@ -2720,7 +3114,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); runtime: process.execPath, }; - const output = maintainFixture( + const output = await maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: commands.runCommand, @@ -2786,7 +3180,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); env: { ...process.env, PATH: `${binDir}:${process.env.PATH}` }, }); - expect(result.status).toBe(0); + expect(result.status, result.stderr).toBe(0); expect(result.stdout.trim().split("\n")).toHaveLength(1); expect(JSON.parse(result.stdout)).toMatchObject({ ok: true, updated: false }); expect(result.stderr).toContain("child-output"); @@ -2814,19 +3208,19 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(result.stderr).toBe(""); }); - test("does not restart Gateway when build provenance misses the exact SHA", () => { + test("does not restart Gateway when build provenance misses the exact SHA", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); const calls: string[] = []; - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { runCommand: (command: string, args: string[]) => calls.push([command, ...args].join(" ")), }, ), - ).toThrow(/build output does not match/u); + ).rejects.toThrow(/build output does not match/u); expect(calls).toEqual([ `${process.execPath} dist/index.js gateway stop`, "pnpm install --frozen-lockfile", @@ -2834,14 +3228,14 @@ console.log(JSON.stringify({ ok: true, channels: {} })); ]); }); - test("audits restart-window logs even when deep Gateway verification fails", () => { + test("audits restart-window logs even when deep Gateway verification fails", async () => { const { root, mirror } = makeFixture(); mkdirSync(path.join(mirror, "node_modules")); writeBuild(mirror); let auditCalls = 0; let statusCalls = 0; - expect(() => + await expect( maintainMain( { checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") }, { @@ -2861,12 +3255,12 @@ console.log(JSON.stringify({ ok: true, channels: {} })); verifyGatewayRuntime: () => null, }, ), - ).toThrow("RPC unavailable"); + ).rejects.toThrow("RPC unavailable"); expect(statusCalls).toBe(8); expect(auditCalls).toBe(1); }); - test("retains failed exact-bundle Mac proof for the next heartbeat", () => { + test("retains failed exact-bundle Mac proof for the next heartbeat", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(seed, "apps/macos/Sources/OpenClaw"), { recursive: true }); writeFileSync(path.join(seed, "apps/macos/Sources/OpenClaw/App.swift"), "// changed\n"); @@ -2877,7 +3271,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); const statePath = path.join(root, "maintenance-state.json"); const firstCommands = fakeCommands(mirror); - expect(() => + await expect( maintainFixture( { checkout: mirror, remote: "origin", lockPath, statePath }, { @@ -2887,7 +3281,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("exact target exited"); + ).rejects.toThrow("exact target exited"); expect(JSON.parse(readFileSync(statePath, "utf8"))).toMatchObject({ macPending: true, attempts: 1, @@ -2895,7 +3289,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }); const retryCommands = fakeCommands(mirror); - const retry = maintainFixture( + const retry = await maintainFixture( { checkout: mirror, remote: "origin", lockPath, statePath }, { runCommand: retryCommands.runCommand, @@ -2913,7 +3307,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); expect(existsSync(statePath)).toBe(false); }); - test("records pending Mac work before Gateway maintenance can fail", () => { + test("records pending Mac work before Gateway maintenance can fail", async () => { const { root, mirror, seed } = makeFixture({ includeSeed: true }); mkdirSync(path.join(seed, "apps/macos/Sources/OpenClaw"), { recursive: true }); writeFileSync(path.join(seed, "apps/macos/Sources/OpenClaw/App.swift"), "// changed\n"); @@ -2923,7 +3317,7 @@ console.log(JSON.stringify({ ok: true, channels: {} })); const statePath = path.join(root, "maintenance-state.json"); const commands = fakeCommands(mirror); - expect(() => + await expect( maintainFixture( { checkout: mirror, @@ -2941,37 +3335,37 @@ console.log(JSON.stringify({ ok: true, channels: {} })); }, }, ), - ).toThrow("Gateway failed"); + ).rejects.toThrow("Gateway failed"); expect(JSON.parse(readFileSync(statePath, "utf8"))).toMatchObject({ macPending: true, attempts: 0, }); }); - test("refuses a symlinked maintenance state file without touching its target", () => { + test("refuses a symlinked maintenance state file without touching its target", async () => { const { root, mirror } = makeFixture(); const statePath = path.join(root, "maintenance-state.json"); const victimPath = path.join(root, "victim.txt"); writeFileSync(victimPath, "untouched\n"); symlinkSync(victimPath, statePath); - expect(() => + await expect( maintainFixture({ checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock"), statePath, }), - ).toThrow(/maintenance state is unreadable/u); + ).rejects.toThrow(/maintenance state is unreadable/u); expect(readFileSync(victimPath, "utf8")).toBe("untouched\n"); }); - test("skips an overlapping heartbeat while the owner process is alive", () => { + test("skips an overlapping heartbeat while the owner process is alive", async () => { const { root, mirror } = makeFixture(); const lockPath = path.join(root, "maintenance.lock"); const held = acquireMaintenanceLock(mirror, lockPath); try { - const output = maintainFixture({ checkout: mirror, remote: "origin", lockPath }); + const output = await maintainFixture({ checkout: mirror, remote: "origin", lockPath }); expect(output).toMatchObject({ ok: true, skipped: true, reason: "overlap" }); } finally { held.release?.();