fix(worker): keep background servers alive after replies (#130733)

* fix(worker): keep background servers alive between turns

Retain the supervised worker environment while background execs are live,
while recreating turn admission, tools, connections, and authorization for
each turn. Keep physical worker ownership separate from bounded turn
receipts so cancellation, restart recovery, and environment retirement
cannot discard a still-running worker.

Preserve process polling across turns, release turn callbacks, join exec
finalization before environment cleanup, and require exact worker stop
confirmation unless a dedicated provider proves the machine is gone.

Fixes #130450.

* fix(worker): fence revoked placements and verify retained lifetimes

Record provider-loss teardown intent before remote stop, prevent pending cleanup
from being recovered as an active placement, and preserve exact physical ownership
until stop is confirmed. Keep draining state when the provider becomes unavailable.

Remove the ownership/lifecycle import cycle and align workspace-retention proof
with physical teardown. Exercise missing-capability rejection and reconnect on
real Gateway/node wire, preserving scenario and cleanup failures in the harness.

* test(worker): verify durable startup revocation cleanup
This commit is contained in:
Peter Steinberger
2026-08-27 00:35:19 -07:00
committed by GitHub
parent f66c162ba6
commit e4d602c6f3
82 changed files with 6320 additions and 1867 deletions
+1 -4
View File
@@ -1709,7 +1709,7 @@ src/agents/bash-tools.exec-approval-request.ts 2
src/agents/bash-tools.exec-host-node-phases.ts 3
src/agents/bash-tools.exec-host-node.ts 2
src/agents/bash-tools.exec-request-preparation.ts 7
src/agents/bash-tools.exec-run.ts 2
src/agents/bash-tools.exec-run.ts 1
src/agents/bash-tools.exec-runtime.ts 2
src/agents/bash-tools.exec-script-preflight.ts 2
src/agents/bash-tools.process.ts 6
@@ -3081,7 +3081,6 @@ src/gateway/worker-environments/inference-control.ts 1
src/gateway/worker-environments/inference-runtime.ts 2
src/gateway/worker-environments/inference-store.ts 1
src/gateway/worker-environments/live-event-projection.ts 1
src/gateway/worker-environments/node-worker-tunnel.ts 6
src/gateway/worker-environments/node-worker-workspace-fallback.ts 2
src/gateway/worker-environments/node-workspace-transfer-service.ts 1
src/gateway/worker-environments/placement-state.ts 3
@@ -3094,7 +3093,6 @@ src/gateway/worker-environments/store.ts 9
src/gateway/worker-environments/transcript-commit.ts 6
src/gateway/worker-environments/worker-session-tool-executor.ts 1
src/gateway/worker-environments/worker-turn-admission.ts 3
src/gateway/worker-environments/worker-turn-payload.ts 4
src/gateway/worker-environments/workspace-accepted-publication.ts 1
src/gateway/worker-environments/workspace-manifest.ts 14
src/gateway/worker-environments/workspace-path-exclusions.ts 2
@@ -3415,7 +3413,6 @@ src/node-host/invoke-payload.ts 5
src/node-host/invoke.ts 7
src/node-host/mcp.ts 3
src/node-host/node-worker-bundle-installer.ts 1
src/node-host/node-worker-supervisor.ts 1
src/node-host/node-worker-transfer-client.ts 3
src/node-host/node-worker-transfer-http.ts 2
src/node-host/node-worker-tree-control.ts 2
+5
View File
@@ -28,6 +28,11 @@ scheduled-owner identity never enter the worker envelope.
The credential is never accepted through command-line arguments, and this page
intentionally provides no credential or hand-authored envelope example.
The node supervisor uses a private managed entry point that can admit successive
turns into the same environment while background processes remain. Each turn
still receives a fresh bounded envelope, Gateway connection, and tool authority.
The standalone command above remains a single-turn entry point.
Admission fails closed if the envelope is invalid, the credential is rejected,
the bundle or protocol features do not match, or the session and owner epoch are
no longer current. Missing, duplicate, or unknown tool names also invalidate the
+23
View File
@@ -58,6 +58,29 @@ Behavior:
| `tools.exec.notifyOnExit` | true | Enqueue a system event + request heartbeat when a backgrounded exec exits. |
| `tools.exec.notifyOnExitEmptySuccess` | false | Also enqueue completion events for successful backgrounded runs with no output. |
## Worker environments
On a paired-node or node-backed cloud worker, background processes belong to the
session's environment. Finishing or cancelling a turn leaves already-backgrounded
commands running. A later turn in the same environment can use `process` to poll,
send input, or stop them; foreground commands still stop when their turn is cancelled.
The retained worker occupies one node worker slot. Reusing it needs no additional
slot. If a command finishes between turns, its retained output remains available
to the next turn, subject to the normal process output limits and TTL. Once a turn
finishes with no live background commands, the worker exits. Moving or retiring
the environment, replacing its ownership, or stopping the node also stops its
processes. Process handles do not survive a worker or node restart.
If the node's pairing is revoked or its provider no longer recognizes the lease,
the session placement fails. Physical cleanup can remain pending until OpenClaw
confirms that the exact worker has stopped; an unconfirmed stop does not release
its ownership record.
Worker completion does not currently wake the Gateway session automatically;
use `process poll` in a later turn to inspect the result. Closing a portal closes
its proxy, not the development server: stop the server with `process kill`.
## Child process bridging
When spawning long-running child processes outside the exec/process tools (CLI respawns, gateway helpers), attach the child-process bridge helper so termination signals forward and listeners detach on exit/close. This avoids orphaned processes on systemd and keeps shutdown consistent across platforms.
+6
View File
@@ -20,6 +20,12 @@ The agent opens a portal for the application's port, then starts the development
For a session on a node-backed cloud worker, including the bundled Crabbox provider, the development server runs on the worker. Each portal connection receives its own single-use ticket, which the enrolled node redeems over a TLS-pinned WebSocket to the Gateway before connecting to the selected loopback port. This uses the existing authenticated node channel without exposing the worker to inbound traffic or creating an SSH tunnel. Stopping or replacing the worker closes its portals.
A background development server continues running after the agent finishes its
reply. Later turns in the same worker environment can inspect or stop it with
`process`. Closing the portal only closes the proxy; it does not stop the server.
See [Worker background processes](/gateway/background-process#worker-environments)
for process lifetime and capacity details.
## Declare development servers
Optionally commit `.openclaw/portals.json` to the workspace repository so the agent can discover the available development servers:
@@ -5,6 +5,7 @@ export const GATEWAY_SERVER_CAPS = {
GATEWAY_RESTART_TARGET_SAFE: "gateway-restart-target-safe-v1",
NODE_WORKER_BUNDLE_RETENTION: "node-worker-bundle-retention-v1",
NODE_WORKER_BUNDLE_STATUS: "node-worker-bundle-status-v1",
NODE_WORKER_ENVIRONMENT_SESSION: "node-worker-environment-session-v1",
NODE_WORKER_PORTAL_STREAM: "node-worker-portal-stream-v1",
SESSION_UNREAD_ACK_CONTRACT: "session-unread-ack-contract",
SYSTEM_AGENT_WIZARD_CANCEL: "openclaw-chat-wizard-cancel",
+21 -15
View File
@@ -353,6 +353,7 @@ describe("bash process registry", () => {
addSession(session);
markBackgrounded(session);
session.backgrounded = false;
deleteSession(session.id);
expect(listRunningSessions()).toHaveLength(0);
@@ -362,23 +363,28 @@ describe("bash process registry", () => {
expect(getActiveBackgroundExecSessionCount()).toBe(0);
});
it("keeps a hidden active session id reserved until exit", () => {
const session = createRegistrySession({
id: "amber-atlas",
maxOutputChars: 100,
pendingMaxOutputChars: 30_000,
backgrounded: false,
});
it.each([false, true])(
"keeps a hidden active session id reserved until exit (backgrounded=%s)",
(backgrounded) => {
const session = createRegistrySession({
id: "amber-atlas",
maxOutputChars: 100,
pendingMaxOutputChars: 30_000,
backgrounded: false,
});
addSession(session);
markBackgrounded(session);
deleteSession(session.id);
expect(createSessionSlug(isProcessSessionIdTaken)).toBe("amber-atlas-2");
addSession(session);
if (backgrounded) {
markBackgrounded(session);
}
deleteSession(session.id);
expect(createSessionSlug(isProcessSessionIdTaken)).toBe("amber-atlas-2");
session.backgrounded = false;
markExited(session, 0, null, "completed");
expect(createSessionSlug(isProcessSessionIdTaken)).toBe("amber-atlas");
});
session.backgrounded = false;
markExited(session, 0, null, "completed");
expect(createSessionSlug(isProcessSessionIdTaken)).toBe("amber-atlas");
},
);
it("clears background activity in the test reset", () => {
const session = createRegistrySession({
+43 -10
View File
@@ -7,6 +7,7 @@ import type { ChildProcessWithoutNullStreams } from "node:child_process";
import { sliceUtf16Safe } from "@openclaw/normalization-core/utf16-slice";
import type { EventSessionRoutingPolicy } from "../infra/event-session-routing.js";
import type { TerminationReason } from "../process/supervisor/types.js";
import { createDeferredCore, type Deferred } from "../shared/deferred.js";
import type { DeliveryContext } from "../utils/delivery-context.types.js";
import { readEnvInt } from "./bash-tools.shared.js";
@@ -118,22 +119,25 @@ const finishedSessions = new Map<string, ProcessSession & { endedAt: number }>()
// Display uses start chronology; retained records are evicted in completion order.
let processSessionStartOrders = new WeakMap<object, number>();
let nextProcessSessionStartOrder = 0;
const activeBackgroundExecSessionIds = new Set<string>();
// Promotion stays live when process removal clears its presentation state.
const activeExecSessions = new Map<
string,
{ session: ProcessSession; promoted: boolean; settled?: Deferred }
>();
let finishedSessionOutputChars = 0;
let sweeper: NodeJS.Timeout | null = null;
/** Return whether a process session id is live, retained, or reserved for notification. */
export function isProcessSessionIdTaken(id: string): boolean {
return (
runningSessions.has(id) || finishedSessions.has(id) || activeBackgroundExecSessionIds.has(id)
);
return runningSessions.has(id) || finishedSessions.has(id) || activeExecSessions.has(id);
}
/** Adds a running session and starts retention sweeping if needed. */
export function addSession(session: ProcessSession) {
processSessionStartOrders.set(session, nextProcessSessionStartOrder++);
runningSessions.set(session.id, session);
activeExecSessions.set(session.id, { session, promoted: session.backgrounded });
startSweeper();
}
@@ -248,7 +252,6 @@ export function markExited(
) {
// Visibility can be cleared before process termination. Keep suspension
// blocked until the process owner reports the actual terminal transition.
activeBackgroundExecSessionIds.delete(session.id);
session.terminalStatus = status;
session.exited = true;
session.exitCode = exitCode;
@@ -262,13 +265,21 @@ export function markExited(
session.pendingOutput = pending.output;
session.pendingOutputDropped = pending.outputDropped;
moveToFinished(session);
const active = activeExecSessions.get(session.id);
if (active?.session === session) {
activeExecSessions.delete(session.id);
// The exec owner's synchronous task/notification callbacks run before
// these promise continuations resume and release the environment state.
active.settled?.resolve();
}
}
/** Marks a running session as reconnectable after the exec call returns. */
export function markBackgrounded(session: ProcessSession) {
session.backgrounded = true;
if (!session.exited) {
activeBackgroundExecSessionIds.add(session.id);
const active = activeExecSessions.get(session.id);
if (active?.session === session) {
active.promoted = true;
}
}
@@ -303,12 +314,31 @@ export function acknowledgeNotifyOnExit(record: {
/** Reports owner-tracked process liveness even after visibility is removed. */
export function hasActiveBackgroundExecSession(sessionId: string): boolean {
return activeBackgroundExecSessionIds.has(sessionId);
return activeExecSessions.get(sessionId)?.promoted === true;
}
/** Returns the number of live background exec sessions without exposing process details. */
export function getActiveBackgroundExecSessionCount(): number {
return activeBackgroundExecSessionIds.size;
let count = 0;
for (const { promoted } of activeExecSessions.values()) {
if (promoted) {
count += 1;
}
}
return count;
}
/** Joins registered exec cleanup, including foreground and hidden processes. */
export async function waitForExecScope(scopeKey: string): Promise<void> {
while (true) {
const pending = Array.from(activeExecSessions.values())
.filter(({ session }) => session.scopeKey === scopeKey)
.map((active) => (active.settled ??= createDeferredCore()).promise);
if (pending.length === 0) {
return;
}
await Promise.all(pending);
}
}
function moveToFinished(session: ProcessSession) {
@@ -405,7 +435,10 @@ function resetProcessRegistryForTests() {
processSessionStartOrders = new WeakMap();
nextProcessSessionStartOrder = 0;
finishedSessionOutputChars = 0;
activeBackgroundExecSessionIds.clear();
for (const active of activeExecSessions.values()) {
active.settled?.resolve();
}
activeExecSessions.clear();
stopSweeper();
}
+95 -95
View File
@@ -26,6 +26,7 @@ import {
registerSecretEgressProxyRun,
} from "../secrets/egress-proxy/registry.js";
import type { SecretStoreExecEnvironment } from "../secrets/store/secret-store.js";
import { createDeferredCore } from "../shared/deferred.js";
import { normalizeDeliveryContext } from "../utils/delivery-context.shared.js";
import { markBackgrounded } from "./bash-process-registry.js";
import { describeExecTool } from "./bash-tools.descriptions.js";
@@ -75,11 +76,27 @@ import type {
import { formatUnavailableWorkdirFailure, resolveExecWorkdir } from "./bash-tools.exec-workdir.js";
import { clampWithDefault, readEnvInt, truncateMiddle } from "./bash-tools.shared.js";
import { createModelExecAutoReviewer } from "./exec-auto-reviewer.js";
import type { AgentToolResult } from "./runtime/index.js";
import { EXEC_TOOL_DISPLAY_SUMMARY } from "./tool-description-presets.js";
import type { AgentToolWithMeta } from "./tools/common.js";
import { withoutGatewayToolCallerIdentity } from "./tools/gateway-caller-context.js";
type GatewayApprovalRevalidator = () => Promise<AgentToolResult<ExecToolDetails> | undefined>;
type GatewayApprovalResult = Awaited<ReturnType<typeof processGatewayAllowlist>>;
function createExecProcessSettlement() {
const settlement: {
outcome: ExecProcessOutcome | null;
backgroundTask: BackgroundExecTaskHandle | null;
settle: (outcome: ExecProcessOutcome) => void;
} = {
outcome: null,
backgroundTask: null,
settle(outcome: ExecProcessOutcome) {
settlement.outcome = outcome;
finalizeBackgroundExecTask({ handle: settlement.backgroundTask, outcome });
},
};
return settlement;
}
/** Creates an exec tool instance with runtime defaults and approval policy wiring. */
export function createExecTool(
@@ -211,7 +228,7 @@ export function createExecTool(
}
const startedAt = Date.now();
let execCommandOverride: string | undefined;
let revalidateGatewayApproval: GatewayApprovalRevalidator | undefined;
let revalidateGatewayApproval: GatewayApprovalResult["revalidateBeforeExecution"];
let approvalReview: ExecToolApprovalReview | undefined;
const foregroundFallbackWarning =
!allowBackground && (params.background === true || typeof params.yieldMs === "number")
@@ -400,8 +417,7 @@ export function createExecTool(
});
}
let run: ExecProcessHandle;
let backgroundTask: BackgroundExecTaskHandle | null = null;
let settledOutcome: ExecProcessOutcome | null = null;
const settlement = createExecProcessSettlement();
let effectiveTimeout: number;
try {
if (elevatedRequested) {
@@ -589,11 +605,13 @@ export function createExecTool(
timeoutSec: effectiveTimeout,
processContinuationAvailable: allowBackground,
onUpdate,
beforeSpawn: revalidateGatewayApproval,
onSettledBeforeNotify: (outcome) => {
settledOutcome = outcome;
finalizeBackgroundExecTask({ handle: backgroundTask, outcome });
beforeSpawn: async () => {
signal?.throwIfAborted();
const denied = await revalidateGatewayApproval?.();
signal?.throwIfAborted();
return denied;
},
onSettledBeforeNotify: settlement.settle,
});
discardPreparedSandboxWorkdir = null;
} catch (error) {
@@ -630,6 +648,7 @@ export function createExecTool(
};
const cleanupToolRunListeners = () => {
run.disableUpdates();
if (registeredAbortSignal) {
registeredAbortSignal.removeEventListener("abort", onAbortSignal);
registeredAbortSignal = null;
@@ -647,103 +666,84 @@ export function createExecTool(
registeredAbortSignal = signal;
}
return new Promise<AgentToolResult<ExecToolDetails>>((resolve, reject) => {
const resolveReviewed = (result: AgentToolResult<ExecToolDetails>) =>
resolve(attachExecApprovalReview(result, approvalReview));
const rejectIfAborted = () => {
if (!toolAborted) {
return false;
}
reject(createAbortError("Tool execution was aborted", { cause: signal?.reason }));
return true;
};
const resolveRunning = () => {
cleanupToolRunListeners();
resolveReviewed({
content: [
{
type: "text",
text: `${getWarningText()}Command still running (session ${run.session.id}, pid ${
run.session.pid ?? "n/a"
}). Use process (list/poll/log/write/send-keys/submit/paste/kill/clear/remove) for follow-up.`,
},
],
details: {
status: "running",
sessionId: run.session.id,
pid: run.session.pid ?? undefined,
startedAt: run.startedAt,
cwd: run.session.cwd,
tail: run.session.tail,
},
});
};
const onYieldNow = () => {
if (yielded || toolAborted || run.session.finalizing) {
return;
}
if (settledOutcome) {
cleanupToolRunListeners();
resolveReviewed(
buildExecForegroundResult({
outcome: settledOutcome,
cwd: run.session.cwd,
warningText: getWarningText(),
aggregateOutputDropped:
run.session.totalOutputChars > run.session.aggregated.length,
}),
);
return;
}
yielded = true;
markBackgrounded(run.session);
// Only the guarded yield transition owns task registration. A process
// that settles before this timer fires must stay out of the task ledger.
backgroundTask = createBackgroundExecTask({
processSessionId: run.session.id,
sessionKey: notifySessionKey,
agentId,
startedAt: run.startedAt,
});
resolveRunning();
};
// Neither the race nor its losing process promise may retain this turn's
// caller context after a background result has returned.
const backgrounded = withoutGatewayToolCallerIdentity(() =>
createDeferredCore<{ status: "backgrounded" }>(),
);
const result = withoutGatewayToolCallerIdentity(() =>
Promise.race([
run.promise.then((outcome) => ({ status: "settled" as const, outcome })),
backgrounded.promise,
]),
);
const onYieldNow = () => {
if (yielded || toolAborted || run.session.finalizing || settlement.outcome) {
return;
}
yielded = true;
run.disableUpdates();
markBackgrounded(run.session);
// Only the guarded yield transition owns task registration. A process
// that settles before this timer fires must stay out of the task ledger.
settlement.backgroundTask = createBackgroundExecTask({
processSessionId: run.session.id,
sessionKey: notifySessionKey,
agentId,
startedAt: run.startedAt,
});
backgrounded.resolve({ status: "backgrounded" });
};
try {
if (!toolAborted && allowBackground && yieldWindow !== null) {
if (yieldWindow === 0) {
onYieldNow();
} else {
yieldTimer = setTimeout(() => {
onYieldNow();
}, yieldWindow);
yieldTimer = setTimeout(onYieldNow, yieldWindow);
}
}
run.promise
.then((outcome) => {
cleanupToolRunListeners();
if (rejectIfAborted() || yielded || run.session.backgrounded) {
return;
}
resolveReviewed(
buildExecForegroundResult({
outcome,
const completed = await result;
if (toolAborted) {
throw createAbortError("Tool execution was aborted", { cause: signal?.reason });
}
return attachExecApprovalReview(
completed.status === "settled"
? buildExecForegroundResult({
outcome: completed.outcome,
cwd: run.session.cwd,
warningText: getWarningText(),
aggregateOutputDropped:
run.session.totalOutputChars > run.session.aggregated.length,
}),
);
})
.catch((err: unknown) => {
cleanupToolRunListeners();
if (rejectIfAborted() || yielded || run.session.backgrounded) {
return;
}
reject(err as Error);
});
});
})
: {
content: [
{
type: "text",
text: `${getWarningText()}Command still running (session ${run.session.id}, pid ${
run.session.pid ?? "n/a"
}). Use process (list/poll/log/write/send-keys/submit/paste/kill/clear/remove) for follow-up.`,
},
],
details: {
status: "running",
sessionId: run.session.id,
pid: run.session.pid ?? undefined,
startedAt: run.startedAt,
cwd: run.session.cwd,
tail: run.session.tail,
},
},
approvalReview,
);
} catch (error) {
if (toolAborted) {
throw createAbortError("Tool execution was aborted", { cause: signal?.reason });
}
throw error;
} finally {
cleanupToolRunListeners();
}
},
};
}
+149 -1
View File
@@ -16,8 +16,16 @@ import {
import type { GatewayActiveWorkInspectors } from "../infra/gateway-active-work.js";
import type { ManagedRun } from "../process/supervisor/index.js";
import type { RunExit, SpawnInput } from "../process/supervisor/types.js";
import { getFinishedSession, markTerminalPollObserved } from "./bash-process-registry.js";
import {
getFinishedSession,
markTerminalPollObserved,
waitForExecScope,
} from "./bash-process-registry.js";
import type { BashSandboxConfig } from "./bash-tools.shared.js";
import {
getGatewayToolCallerIdentity,
withGatewayToolCallerIdentity,
} from "./tools/gateway-caller-context.js";
const requestHeartbeatMock = vi.hoisted(() => vi.fn());
const enqueueSystemEventWithReceiptMock = vi.hoisted(() => vi.fn());
@@ -192,6 +200,65 @@ describe("runExecProcess cursor tracking", () => {
});
describe("sandbox exec preparation failures", () => {
it("keeps turn authority out of process lifetime while preserving foreground updates", async () => {
const exit = createDeferred<RunExit>();
const identity = {
agentId: "main",
sessionKey: "agent:main:exec-lifetime",
signedAgentRuntimeIdentityToken: "synthetic-turn-identity",
};
const spawnIdentity = vi.fn();
const updateIdentity = vi.fn();
const settledIdentity = vi.fn();
const beforeSpawn = vi.fn(async () => {
expect(getGatewayToolCallerIdentity()).toMatchObject(identity);
return undefined;
});
let stdout: SpawnInput["onStdout"];
supervisorMock.spawn.mockImplementationOnce(async (input: SpawnInput) => {
spawnIdentity(getGatewayToolCallerIdentity());
stdout = input.onStdout;
stdout?.("foreground output\n");
return { ...runtimeManagedRun(input), wait: () => exit.promise };
});
const run = await withGatewayToolCallerIdentity(identity, () =>
runExecProcess({
command: "test-command",
workdir: "/tmp",
env: {},
usePty: false,
warnings: [],
maxOutput: 1000,
pendingMaxOutput: 1000,
notifyOnExit: false,
timeoutSec: null,
beforeSpawn,
onUpdate: () => updateIdentity(getGatewayToolCallerIdentity()),
onSettledBeforeNotify: () => settledIdentity(getGatewayToolCallerIdentity()),
}),
);
run.disableUpdates();
stdout?.("background output\n");
exit.resolve({
reason: "exit",
exitCode: 0,
exitSignal: null,
durationMs: 1,
stdout: "",
stderr: "",
timedOut: false,
noOutputTimedOut: false,
});
const outcome = await run.promise;
expect(beforeSpawn).toHaveBeenCalledOnce();
expect(updateIdentity).toHaveBeenCalledExactlyOnceWith(expect.objectContaining(identity));
expect(outcome.aggregated).toBe("foreground output\nbackground output");
expect(spawnIdentity).toHaveBeenCalledExactlyOnceWith(undefined);
expect(settledIdentity).toHaveBeenCalledExactlyOnceWith(undefined);
});
it("runs the final authorization check after async preparation and before spawn", async () => {
const preparation =
createDeferred<Awaited<ReturnType<NonNullable<BashSandboxConfig["buildExecSpec"]>>>>();
@@ -568,6 +635,87 @@ describe("terminal execution-context release", () => {
);
});
describe("exec settlement recovery", () => {
it.each([
{ boundary: "task", trace: ["task:completed", "task:failed", "scope-released"] },
{
boundary: "enqueue",
trace: ["task:completed", "enqueue", "task:failed", "scope-released"],
},
{
boundary: "wake",
trace: ["task:completed", "enqueue", "wake", "task:failed", "scope-released"],
},
])("retries $boundary failure before releasing the exec scope", async ({ boundary, trace }) => {
const exit = createDeferred<RunExit>();
const observed: string[] = [];
const identities: Array<ReturnType<typeof getGatewayToolCallerIdentity>> = [];
const failure = new Error("process settlement failed");
const scopeKey = `settlement-recovery:${boundary}`;
enqueueSystemEventWithReceiptMock.mockImplementation(() => {
observed.push("enqueue");
if (boundary === "enqueue") {
throw failure;
}
return vi.fn(() => true);
});
requestHeartbeatMock.mockImplementation(() => {
observed.push("wake");
if (boundary === "wake") {
throw failure;
}
});
supervisorMock.spawn.mockImplementationOnce(async (input: SpawnInput) => ({
...runtimeManagedRun(input, "process output\n"),
wait: () => exit.promise,
}));
const run = await withGatewayToolCallerIdentity(
{ agentId: "main", sessionKey: "agent:main:settlement-recovery" },
() =>
runExecProcess({
command: "settlement-recovery",
workdir: "/tmp",
env: {},
usePty: false,
warnings: [],
maxOutput: 1000,
pendingMaxOutput: 1000,
scopeKey,
sessionKey: "agent:main:settlement-recovery",
notifyOnExit: true,
timeoutSec: null,
onSettledBeforeNotify: (outcome) => {
observed.push(`task:${outcome.status}`);
identities.push(getGatewayToolCallerIdentity());
if (boundary === "task" && observed.length === 1) {
throw failure;
}
},
}),
);
markBackgrounded(run.session);
const joined = waitForExecScope(scopeKey).then(() => {
observed.push("scope-released");
});
exit.resolve({
reason: "exit",
exitCode: 0,
exitSignal: null,
durationMs: 1,
stdout: "",
stderr: "",
timedOut: false,
noOutputTimedOut: false,
});
const outcome = await run.promise;
await joined;
expect(outcome.status).toBe("failed");
expect(observed).toEqual(trace);
expect(identities).toEqual([undefined, undefined]);
});
});
describe("runExecProcess exit outcomes", () => {
it("keeps non-zero normal exits in the completed path", async () => {
const { outcome } = await runExecWithExit({
+68 -67
View File
@@ -1,3 +1,4 @@
import { AsyncLocalStorage } from "node:async_hooks";
import path from "node:path";
import { normalizeStringEntries } from "@openclaw/normalization-core/string-normalization";
import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice";
@@ -23,7 +24,7 @@ import { logWarn } from "../logger.js";
import { redactToolPayloadText } from "../logging/redact.js";
import type { ManagedRun } from "../process/supervisor/index.js";
import { getProcessSupervisor } from "../process/supervisor/index.js";
import type { RunExit, TerminationReason } from "../process/supervisor/types.js";
import type { RunExit, SpawnInput, TerminationReason } from "../process/supervisor/types.js";
import { isSubagentSessionKey } from "../sessions/session-key-utils.js";
/**
* Bash exec runtime.
@@ -58,6 +59,7 @@ import type { AgentToolResult } from "./runtime/index.js";
import { createSessionSlug } from "./session-slug.js";
import { maybeWrapCommandWithShellSnapshot } from "./shell-snapshot.js";
import { createStreamingBinaryOutputSanitizer, getShellConfig } from "./shell-utils.js";
import { withoutGatewayToolCallerIdentity } from "./tools/gateway-caller-context.js";
export { applyPathPrepend, normalizePathPrepend } from "../infra/path-prepend.js";
export { execSchema } from "./bash-tools.schemas.js";
@@ -642,7 +644,12 @@ function wrapPosixCommandWithPathPrepend(
}
/** Starts a host or sandbox exec process and registers it for polling/backgrounding. */
export async function runExecProcess(opts: {
export async function runExecProcess({
onUpdate: initialOnUpdate,
beforeSpawn: initialBeforeSpawn,
onSettledBeforeNotify: initialOnSettledBeforeNotify,
...opts
}: {
command: string;
// Execute this instead of `command` (which is kept for display/session/logging).
// Used to sanitize safeBins execution while preserving the original user input.
@@ -725,33 +732,20 @@ export async function runExecProcess(opts: {
backgrounded: false,
cursorKeyMode: opts.usePty ? "unknown" : "normal",
};
addSession(session);
withoutGatewayToolCallerIdentity(() => addSession(session));
// Tracks whether the exec run's promise has settled (process exited or
// spawn failed). Once settled the agent-loop no longer expects
// tool_execution_update events, so emitUpdate must become a no-op to
// prevent calling into a disposed agent run (the "Agent listener invoked
// outside active run" crash — see #62520).
let updatesDisabled = false;
// Foreground delivery keeps its caller context only until yield, abort, or exit.
// Clearing the callback also releases the completed turn's captured authority.
let onUpdate = initialOnUpdate && AsyncLocalStorage.bind(initialOnUpdate);
let beforeSpawn = initialBeforeSpawn;
let onSettledBeforeNotify = initialOnSettledBeforeNotify;
const emitUpdate = () => {
if (!opts.onUpdate) {
return;
}
if (session.backgrounded || session.exited || updatesDisabled) {
if (!onUpdate || session.backgrounded || session.exited) {
return;
}
const tailText = session.tail || session.aggregated;
// Note: opts.onUpdate() is provided by agent runtime's agent-loop and
// internally pushes Promise.resolve(emit(event)) into an updateEvents
// array. Because emit → processEvents is async, any failure (e.g.
// activeRun cleared) produces a *rejected Promise*, not a synchronous
// throw — so a try-catch here would be ineffective. Instead we rely
// on the `updatesDisabled` flag being set proactively: by the promise
// chain on process exit (Layer 1) and by `disableUpdates()` on abort
// signal (Layer 2) — both of which prevent this call from ever being
// reached after the agent run has ended.
opts.onUpdate({
onUpdate({
content: [
{ type: "text", text: renderExecUpdateText({ tailText, warnings: opts.warnings }) },
],
@@ -851,10 +845,19 @@ export async function runExecProcess(opts: {
finalOutcome.noOutputTimedOut,
);
}
opts.onSettledBeforeNotify?.(finalOutcome);
onSettledBeforeNotify?.(finalOutcome);
if (shouldNotify) {
maybeNotifyOnExit(session, finalOutcome.status);
}
} catch (error) {
// Recover before yielding: scope joins queued by markExited must not
// outrun the task's failed outcome or restore its environment state.
finalOutcome = buildExecRuntimeErrorOutcome({
error,
aggregated: session.aggregated.trim(),
durationMs: Date.now() - startedAt,
});
onSettledBeforeNotify?.(finalOutcome);
} finally {
// Notifications need start-time routing, but completed logs must not
// retain it, including when a task callback or notification throws.
@@ -946,11 +949,13 @@ export async function runExecProcess(opts: {
};
const assertPreSpawnAuthorized = async () => {
const denied = await opts.beforeSpawn?.();
const denied = await beforeSpawn?.();
if (denied) {
throw new ExecProcessPreflightError(denied);
}
};
const spawn = (input: SpawnInput) =>
withoutGatewayToolCallerIdentity(() => supervisor.spawn(input));
try {
const spawnSpec = await prepareSpawnSpec();
@@ -967,10 +972,10 @@ export async function runExecProcess(opts: {
onStdout: onSupervisorStdout,
onStderr: handleStderr,
};
await assertPreSpawnAuthorized();
if (spawnSpec.mode === "pty") {
try {
await assertPreSpawnAuthorized();
managedRun = await supervisor.spawn({
managedRun = await spawn({
...spawnBase,
mode: "pty",
ptyCommand: spawnSpec.ptyCommand,
@@ -983,7 +988,7 @@ export async function runExecProcess(opts: {
opts.warnings.push(warning);
usingPty = false;
await assertPreSpawnAuthorized();
managedRun = await supervisor.spawn({
managedRun = await spawn({
...spawnBase,
mode: "child",
argv: spawnSpec.childFallbackArgv,
@@ -992,8 +997,7 @@ export async function runExecProcess(opts: {
});
}
} else {
await assertPreSpawnAuthorized();
managedRun = await supervisor.spawn({
managedRun = await spawn({
...spawnBase,
mode: "child",
argv: spawnSpec.argv,
@@ -1001,13 +1005,16 @@ export async function runExecProcess(opts: {
});
}
} catch (error) {
onUpdate = undefined;
const outcome = await finalizeAndSettleSession(
buildExecRuntimeErrorOutcome({
error,
aggregated: session.aggregated.trim(),
durationMs: Date.now() - startedAt,
}),
);
).finally(() => {
onSettledBeforeNotify = undefined;
});
emitExecProcessCompleted({
command: opts.command,
mode: usingPty ? "pty" : "child",
@@ -1016,27 +1023,35 @@ export async function runExecProcess(opts: {
target: diagnosticTarget,
});
throw error;
} finally {
beforeSpawn = undefined;
}
session.stdin = managedRun.stdin;
session.pid = managedRun.pid;
const promise = managedRun
.wait()
.then(async (exit): Promise<ExecProcessOutcome> => {
// Disable updates *before* markExited so that any late stdout/stderr
// data events queued in the same event-loop tick cannot sneak through
// the `session.exited` guard before it flips to true.
updatesDisabled = true;
const durationMs = Date.now() - startedAt;
const outcome = buildExecExitOutcome({
exit,
aggregated: session.aggregated.trim(),
durationMs,
timeoutSec: opts.timeoutSec,
processContinuationAvailable: opts.processContinuationAvailable !== false,
});
const startedRun = managedRun;
const promise = withoutGatewayToolCallerIdentity(async (): Promise<ExecProcessOutcome> => {
try {
let outcome: ExecProcessOutcome;
try {
const exit = await startedRun.wait();
outcome = buildExecExitOutcome({
exit,
aggregated: session.aggregated.trim(),
durationMs: Date.now() - startedAt,
timeoutSec: opts.timeoutSec,
processContinuationAvailable: opts.processContinuationAvailable !== false,
});
} catch (error) {
outcome = buildExecRuntimeErrorOutcome({
error,
aggregated: session.aggregated.trim(),
durationMs: Date.now() - startedAt,
});
} finally {
// Release foreground delivery before finalization marks the record exited.
onUpdate = undefined;
}
const finalOutcome = await finalizeAndSettleSession(outcome);
emitExecProcessCompleted({
command: opts.command,
@@ -1046,24 +1061,10 @@ export async function runExecProcess(opts: {
target: diagnosticTarget,
});
return finalOutcome;
})
.catch(async (err: unknown): Promise<ExecProcessOutcome> => {
updatesDisabled = true;
const outcome = buildExecRuntimeErrorOutcome({
error: err,
aggregated: session.aggregated.trim(),
durationMs: Date.now() - startedAt,
});
const finalOutcome = await finalizeAndSettleSession(outcome);
emitExecProcessCompleted({
command: opts.command,
mode: usingPty ? "pty" : "child",
outcome: finalOutcome,
sessionKey: opts.sessionKey,
target: diagnosticTarget,
});
return finalOutcome;
});
} finally {
onSettledBeforeNotify = undefined;
}
});
return {
session,
@@ -1074,7 +1075,7 @@ export async function runExecProcess(opts: {
managedRun?.cancel("manual-cancel");
},
disableUpdates: () => {
updatesDisabled = true;
onUpdate = undefined;
},
};
}
+61 -4
View File
@@ -1,5 +1,6 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { createDeferred } from "../../test/helpers/promise.js";
import { getProcessSupervisor } from "../process/supervisor/index.js";
const taskTracking = vi.hoisted(() => ({
createBackgroundExecTask: vi.fn(),
@@ -11,6 +12,10 @@ vi.mock("./bash-tools.exec-task-tracking.js", () => taskTracking);
import { getFinishedSession } from "./bash-process-registry.js";
import { createExecTool } from "./bash-tools.exec-run.js";
import type { BashSandboxConfig } from "./bash-tools.shared.js";
import {
getGatewayToolCallerIdentity,
withGatewayToolCallerIdentity,
} from "./tools/gateway-caller-context.js";
describe("exec background task wiring", () => {
beforeEach(() => {
@@ -18,6 +23,49 @@ describe("exec background task wiring", () => {
taskTracking.finalizeBackgroundExecTask.mockReset();
});
it("does not spawn when the turn closes during asynchronous process preparation", async () => {
const abortController = new AbortController();
const preparationStarted = createDeferred();
const preparation =
createDeferred<Awaited<ReturnType<NonNullable<BashSandboxConfig["buildExecSpec"]>>>>();
const spawn = vi.spyOn(getProcessSupervisor(), "spawn");
const tool = createExecTool({
host: "sandbox",
security: "full",
ask: "off",
sandbox: {
containerName: "sandbox",
workspaceDir: process.cwd(),
containerWorkdir: process.cwd(),
buildExecSpec: async () => {
preparationStarted.resolve();
return await preparation.promise;
},
},
});
try {
const execution = tool.execute(
"closed-before-spawn",
{ command: "sandbox-command" },
abortController.signal,
);
const settled = Promise.allSettled([execution]);
await preparationStarted.promise;
abortController.abort(new Error("turn closed during preparation"));
preparation.resolve({
argv: [process.execPath, "-e", ""],
env: process.env,
stdinMode: "pipe-closed",
});
await settled;
expect(spawn.mock.calls.length).toBe(0);
await expect(execution).rejects.toThrow("turn closed during preparation");
expect(taskTracking.createBackgroundExecTask).not.toHaveBeenCalled();
} finally {
spawn.mockRestore();
}
});
it("does not register a foreground command that settles before the yield timer", async () => {
const tool = createExecTool({
host: "gateway",
@@ -139,6 +187,10 @@ describe("exec background task wiring", () => {
it("keeps a real background process running after its tool signal aborts", async () => {
const abortController = new AbortController();
const settledIdentity = vi.fn();
taskTracking.finalizeBackgroundExecTask.mockImplementation(() =>
settledIdentity(getGatewayToolCallerIdentity()),
);
const tool = createExecTool({
host: "gateway",
security: "full",
@@ -149,10 +201,14 @@ describe("exec background task wiring", () => {
const command =
`${JSON.stringify(process.execPath)} -e ` +
`"setTimeout(() => process.stdout.write('background-survived\\n'), 30)"`;
const result = await tool.execute(
"abort-real-background",
{ command, background: true },
abortController.signal,
const result = await withGatewayToolCallerIdentity(
{ agentId: "main", sessionKey: "agent:main:background-lifetime" },
() =>
tool.execute(
"abort-real-background",
{ command, background: true },
abortController.signal,
),
);
expect(result.details.status).toBe("running");
@@ -168,5 +224,6 @@ describe("exec background task wiring", () => {
interval: 10,
})
.toBe("completed");
expect(settledIdentity).toHaveBeenCalledExactlyOnceWith(undefined);
});
});
@@ -160,6 +160,11 @@ export function getGatewayToolCallerIdentity(): GatewayToolCallerIdentity | unde
return gatewayToolCallerStorage.getStore();
}
/** Process-owned work must not retain the turn that authorized its launch. */
export function withoutGatewayToolCallerIdentity<T>(run: () => T): T {
return gatewayToolCallerStorage.exit(run);
}
export async function withGatewayToolCallerIdentity<T>(
identity: GatewayToolCallerIdentity | undefined,
run: () => Promise<T> | T,
+9 -1
View File
@@ -4,12 +4,14 @@ import { normalizeOptionalString } from "@openclaw/normalization-core/string-coe
import { GATEWAY_CLIENT_IDS } from "../../packages/gateway-protocol/src/client-info.js";
import {
isPrivateNodeInvokeCommand,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_PRIVATE_COMMANDS,
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
} from "../infra/node-commands.js";
import {
NODE_WORKER_BUNDLE_RETENTION_VERSION,
NODE_WORKER_BUNDLE_STATUS_VERSION,
NODE_WORKER_ENVIRONMENT_SESSION_VERSION,
type NodeRunnerInventoryIssue,
type NodeRunnerInventoryDeclaration,
type NodeWorkerCapacitySnapshot,
@@ -154,6 +156,7 @@ function isWorkerSupervisorProofCurrent(
proof: NodeWorkerSupervisorNodeProof,
requireLaunchEligibility: boolean,
requiredCommands: readonly string[] = [],
requireEnvironmentSession = false,
): boolean {
const node = state.context.getNode(proof.nodeId);
if (!node || node.client.invalidated === true || node.connId !== proof.connId) {
@@ -167,6 +170,8 @@ function isWorkerSupervisorProofCurrent(
current.clientMode === proof.clientMode &&
current.protocolFeature === proof.protocolFeature &&
(!requireLaunchEligibility || current.workerHost.capacity.available > 0) &&
(!requireEnvironmentSession ||
current.workerHost.environmentSession === NODE_WORKER_ENVIRONMENT_SESSION_VERSION) &&
requiredCommands.every((command) => current.commands.includes(command))
);
}
@@ -485,7 +490,10 @@ export function registerNodeRegistryPrivateRuntime(
isWorkerSupervisorProofCurrent(
state,
params.node,
params.command === NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
false,
[],
params.command === NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND ||
params.command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
);
if (!isProofCurrent()) {
return {
+33 -1
View File
@@ -15,6 +15,7 @@ import { getCurrentActiveNodeContext, setActiveNodeContext } from "../infra/acti
import { onDiagnosticEvent, resetDiagnosticEventsForTest } from "../infra/diagnostic-events.js";
import {
NODE_MCP_TOOLS_CALL_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
NODE_WORKER_PRIVATE_COMMANDS,
NODE_WORKER_SUPERVISOR_STATUS_COMMAND,
@@ -751,7 +752,7 @@ describe("gateway/node-registry", () => {
});
});
it("keeps status proof current across capacity changes while fencing launches", async () => {
it("keeps capacity admission separate from negotiated environment turn reuse", async () => {
const frames: string[] = [];
const { nodeRegistry, nodeWorkerSupervisorTransport } = createPrivateNodeRegistryRuntime();
registerNodeSession(
@@ -825,6 +826,37 @@ describe("gateway/node-registry", () => {
ok: false,
error: { code: "PRIVATE_DIALECT_UNAVAILABLE" },
});
updateNodeRunnerInventory({
registry: nodeRegistry,
nodeId: "node-1",
connId: "conn-1",
declaration: {
protocolFeatures: [NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE],
workerHost: { enabled: true, capacity: { total: 2, available: 0 }, environmentSession: 1 },
},
});
expect(nodeWorkerSupervisorTransport.isCurrent(proof, true)).toBe(false);
for (const command of [
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
] as const) {
const invocation = nodeWorkerSupervisorTransport.invoke({
node: proof,
command,
isDispatchAuthorized: () => true,
});
await vi.waitFor(() => expect(frames.at(-1)).toContain(command));
const dispatched = JSON.parse(frames.at(-1) ?? "{}") as { payload: { id: string } };
nodeRegistry.handleInvokeResult({
id: dispatched.payload.id,
nodeId: "node-1",
connId: "conn-1",
ok: true,
payloadJSON: "null",
});
await expect(invocation).resolves.toMatchObject({ ok: true });
}
});
it("fences retained proofs when runner consent is disabled", async () => {
+2 -1
View File
@@ -111,7 +111,8 @@ export function sameNodeWorkerHostDeclaration(
left.bundlePrewarm === right.bundlePrewarm &&
left.bundleRetention === right.bundleRetention &&
left.bundleStatus === right.bundleStatus &&
left.portalStream === right.portalStream))
left.portalStream === right.portalStream &&
left.environmentSession === right.environmentSession))
);
}
@@ -292,41 +292,47 @@ describe("nodeHandlers node.runnerInventory.update", () => {
runtime.nodeRegistry.unregister("conn-1");
});
it("publishes and retires negotiated portal-stream capability without exposing a private command", async () => {
const inventoryChanged = vi.fn();
const runtime = createNodeRegistryRuntime(() => new NodeRegistry());
setNodeRunnerStateChangedListener(runtime.nodeRegistry, inventoryChanged);
const client = createWorkerSupervisorNodeClient();
runtime.nodeRegistry.register(client, {
pairingIdentity: "identity-1",
pairingGeneration: "generation-1",
});
const publish = async (portalStream: boolean) => {
await runnerInventoryHandler(
runnerInventoryOptions({
nodeRegistry: runtime.nodeRegistry,
client,
declaration: {
...availableHost,
workerHost: {
...availableHost.workerHost,
...(portalStream ? { portalStream: 1 } : {}),
it.each([
["portalStream", "worker.portal.stream.v1"],
["environmentSession", "worker.environment.stop.v1"],
] as const)(
"publishes and retires negotiated %s without exposing %s",
async (capability, command) => {
const inventoryChanged = vi.fn();
const runtime = createNodeRegistryRuntime(() => new NodeRegistry());
setNodeRunnerStateChangedListener(runtime.nodeRegistry, inventoryChanged);
const client = createWorkerSupervisorNodeClient();
runtime.nodeRegistry.register(client, {
pairingIdentity: "identity-1",
pairingGeneration: "generation-1",
});
const publish = async (supported: boolean) => {
await runnerInventoryHandler(
runnerInventoryOptions({
nodeRegistry: runtime.nodeRegistry,
client,
declaration: {
...availableHost,
workerHost: {
...availableHost.workerHost,
...(supported ? { [capability]: 1 } : {}),
},
},
},
}),
);
const [proof] = await runtime.nodeWorkerSupervisorTransport.listCurrentNodes();
return proof;
};
}),
);
const [proof] = await runtime.nodeWorkerSupervisorTransport.listCurrentNodes();
return proof;
};
expect((await publish(false))?.workerHost.portalStream).toBeUndefined();
const supported = await publish(true);
expect(supported?.workerHost.portalStream).toBe(1);
expect(supported?.commands).not.toContain("worker.portal.stream.v1");
expect((await publish(false))?.workerHost.portalStream).toBeUndefined();
expect(inventoryChanged).toHaveBeenCalledTimes(3);
runtime.nodeRegistry.unregister("conn-1");
});
expect((await publish(false))?.workerHost[capability]).toBeUndefined();
const supported = await publish(true);
expect(supported?.workerHost[capability]).toBe(1);
expect(supported?.commands).not.toContain(command);
expect((await publish(false))?.workerHost[capability]).toBeUndefined();
expect(inventoryChanged).toHaveBeenCalledTimes(3);
runtime.nodeRegistry.unregister("conn-1");
},
);
it("requires a fresh current-generation publication after same-connection promotion", async () => {
const runtime = createNodeRegistryRuntime(() => new NodeRegistry());
@@ -121,7 +121,15 @@ describe("gateway worker environment startup", () => {
"device-environment",
]);
expect(startup.store.getCredential("device-environment")).toBeUndefined();
expect(startup.store.get("device-environment")?.state).toBe("orphaned");
expect(startup.store.get("device-environment")).toMatchObject({
state: "failed",
leaseId: null,
nodeDeviceId: null,
attachedSessionIds: [],
destroyRequestedAtMs: expect.any(Number),
teardownTerminalState: "failed",
lastError: "Worker provider no longer recognizes the lease",
});
} finally {
await service.stop();
}
@@ -246,6 +246,7 @@ export async function createGatewayWorkerEnvironmentRuntime(params: {
const nodeWorkerTunnelManager = createNodeWorkerTunnelManager({
gatewayDeviceId,
getEnvironment: (environmentId) => params.startup.store.get(environmentId),
listEnvironments: () => params.startup.store.list(),
getTransport: () => deviceRuntime.getNodeTransport(),
launchNodeWorker: async (request) => await deviceRuntime.launchNodeWorker(request),
validateWorkerTurn: (binding) => placementGate.validateWorkerTurn(binding),
@@ -141,6 +141,7 @@ export async function sendGatewayHello(
GATEWAY_SERVER_CAPS.GATEWAY_RESTART_TARGET_SAFE,
GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_RETENTION,
GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_STATUS,
GATEWAY_SERVER_CAPS.NODE_WORKER_ENVIRONMENT_SESSION,
GATEWAY_SERVER_CAPS.NODE_WORKER_PORTAL_STREAM,
GATEWAY_SERVER_CAPS.SESSION_UNREAD_ACK_CONTRACT,
GATEWAY_SERVER_CAPS.SYSTEM_AGENT_WIZARD_CANCEL,
@@ -61,7 +61,7 @@ export function deviceUnavailableText(deviceId: string, availability: DeviceWork
case "disconnected":
return `device worker node is not connected: ${deviceId}; reconnect it before retrying`;
case "at-capacity":
return `device worker is at capacity (all worker slots in use): ${deviceId}; retry after a running turn completes`;
return `device worker is at capacity (all worker slots in use): ${deviceId}; stop an existing worker environment or retry when a slot is free`;
default:
return `device worker availability is unknown: ${deviceId}; verify the node host is paired and connected, then retry`;
}
@@ -144,6 +144,8 @@ export function createWorkerEnvironmentAccess(options: WorkerEnvironmentAccessOp
throw serviceError("invalid_state", "Node worker tunnel runtime is unavailable");
}
startup = nodeTunnels.start({
executionMode:
record.profileSnapshot.executionMode === "remote-exec" ? "remote-exec" : "worker-turn",
environmentId: record.environmentId,
ownerEpoch: record.ownerEpoch,
deviceId: nodeDeviceId,
@@ -36,13 +36,14 @@ function nodeProof(connId = "conn-1", available = 2): NodeWorkerSupervisorNodePr
clientId: GATEWAY_CLIENT_IDS.NODE_HOST,
clientMode: GATEWAY_CLIENT_MODES.NODE,
protocolFeature: NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
workerHost: { enabled: true, capacity: { total: 2, available } },
workerHost: { enabled: true, capacity: { total: 2, available }, environmentSession: 1 },
commands: ["system.run"],
};
}
function launchInput(): NodeWorkerLaunchInput {
return {
environmentSession: 1,
launchId: "turn-1",
gatewayNamespace: "gateway-1",
expectedBundleHash: WORKER_RUNS.bundleHash,
@@ -322,26 +323,12 @@ describe("node worker launch adapter", () => {
},
);
it.each([
{ label: "offline", slots: [undefined], code: "runner-offline" },
{ label: "full", slots: [0], code: NODE_WORKER_CAPACITY_EXHAUSTED_ERROR_CODE },
{ label: "full then offline", slots: [0, undefined], code: "runner-offline" },
{
label: "offline then full",
slots: [undefined, 0],
code: NODE_WORKER_CAPACITY_EXHAUSTED_ERROR_CODE,
},
])("reports $label availability when the dispatch grace expires", async ({ slots, code }) => {
it("reports offline availability when the dispatch grace expires", async () => {
vi.useFakeTimers();
const onDispatchReady = vi.fn();
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>();
let reads = 0;
const adapter = createNodeWorkerLaunchAdapter({
getTransport: () =>
transportWith(invoke, async () => {
const available = slots[Math.min(reads++, slots.length - 1)];
return available === undefined ? [] : [nodeProof("conn-1", available)];
}),
getTransport: () => transportWith(invoke, async () => []),
});
try {
const launch = adapter
@@ -349,7 +336,7 @@ describe("node worker launch adapter", () => {
.catch((error: unknown) => error);
await vi.runAllTimersAsync();
expect(await launch).toMatchObject({ code });
expect(await launch).toMatchObject({ code: "runner-offline" });
expect(invoke).not.toHaveBeenCalled();
expect(onDispatchReady).not.toHaveBeenCalled();
} finally {
@@ -357,38 +344,32 @@ describe("node worker launch adapter", () => {
}
});
it.each(["available", "abort"] as const)(
"waits at capacity until %s without dispatching early",
async (outcome) => {
const input = launchInput();
const controller = new AbortController();
let available = 0;
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>(async () =>
wire(receipt(input, "completed")),
);
const sleep = vi.fn(async () => {
expect(invoke).not.toHaveBeenCalled();
if (outcome === "abort") {
controller.abort(new Error("operator cancelled capacity wait"));
} else {
available = 1;
}
});
const adapter = createNodeWorkerLaunchAdapter({
getTransport: () => transportWith(invoke, async () => [nodeProof("conn-1", available)]),
sleep,
});
const launch = adapter.launch({ ...launchRequest(input), signal: controller.signal });
if (outcome === "abort") {
await expect(launch).rejects.toThrow("operator cancelled capacity wait");
expect(invoke).not.toHaveBeenCalled();
} else {
await expect(launch).resolves.toEqual(receipt(input, "completed"));
expect(invoke).toHaveBeenCalledOnce();
}
expect(sleep).toHaveBeenCalledOnce();
},
);
it("dispatches a bound environment at capacity so its retained worker can reuse the slot", async () => {
const input = launchInput();
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>(async () =>
wire(receipt(input, "completed")),
);
const adapter = createNodeWorkerLaunchAdapter({
getTransport: () => transportWith(invoke, async () => [nodeProof("conn-1", 0)]),
});
await expect(adapter.launch(launchRequest(input))).resolves.toEqual(
receipt(input, "completed"),
);
expect(invoke).toHaveBeenCalledOnce();
});
it("requires environment lifetime support before dispatching a turn", async () => {
const node = nodeProof();
delete node.workerHost.environmentSession;
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>();
const adapter = createNodeWorkerLaunchAdapter({
getTransport: () => transportWith(invoke, async () => [node]),
});
await expect(adapter.launch(launchRequest())).rejects.toThrow("openclaw update");
expect(invoke).not.toHaveBeenCalled();
});
it("launches once, polls status, and returns the exact completed receipt", async () => {
const input = launchInput();
@@ -6,6 +6,11 @@ import {
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
NODE_WORKER_SUPERVISOR_STATUS_COMMAND,
} from "../../infra/node-commands.js";
import {
formatNodeRunnerUpdateRequired,
NODE_RUNNER_UPDATE_REQUIRED_ISSUE,
NODE_WORKER_ENVIRONMENT_SESSION_VERSION,
} from "../../infra/node-runner-inventory.js";
import {
nodeWorkerPlanHash,
parseNodeWorkerLaunchInput,
@@ -35,7 +40,6 @@ const MAX_ADMISSION_ATTEMPTS = 5;
const ADMISSION_REARM_BACKOFF = { initialMs: 1_000, maxMs: 30_000, factor: 2, jitter: 0.1 };
const RETRYABLE_TRANSPORT_CODES = new Set([
"AT_CAPACITY",
"DISCONNECTED",
"NOT_CONNECTED",
"PAIRING_CHANGED",
@@ -220,7 +224,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
const findNode = async (params: {
transport: NodeWorkerSupervisorTransport;
deviceId: string;
requireLaunchAvailability?: boolean;
signal: AbortSignal;
}): Promise<NodeWorkerSupervisorNodeProof> => {
let nodes: readonly NodeWorkerSupervisorNodeProof[];
@@ -242,9 +245,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
"device worker node is not currently connected",
);
}
if (params.requireLaunchAvailability && node.workerHost.capacity.available === 0) {
throw new NodeWorkerLaunchTransportError("AT_CAPACITY", "device worker capacity is full");
}
return node;
};
@@ -255,7 +255,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
| typeof NODE_WORKER_SUPERVISOR_STATUS_COMMAND
| typeof NODE_WORKER_SUPERVISOR_CANCEL_COMMAND;
payload: unknown;
requireLaunchAvailability?: boolean;
isAuthorized: () => boolean;
deadline: OperationDeadline;
onDispatchReady?: () => void;
@@ -294,9 +293,18 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
const node = await findNode({
transport,
deviceId: params.deviceId,
requireLaunchAvailability: params.requireLaunchAvailability,
signal,
});
if (
params.command === NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND &&
node.workerHost.environmentSession !== NODE_WORKER_ENVIRONMENT_SESSION_VERSION
) {
throw new Error(
formatNodeRunnerUpdateRequired(node.nodeId, NODE_RUNNER_UPDATE_REQUIRED_ISSUE),
);
}
// A retained environment already owns its slot. The node arbitrates new physical
// launches atomically; its advertised free-slot count cannot reject turn reuse.
const operation = transport.invoke({
node,
command: params.command,
@@ -435,7 +443,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
let dispatchReady = false;
let pollStatus = false;
let delayMs = pollIntervalMs;
let availabilityCode: string | undefined;
const markDispatchReady = () => {
mayHaveLaunched = true;
if (!dispatchReady) {
@@ -459,7 +466,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
? NODE_WORKER_SUPERVISOR_STATUS_COMMAND
: NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
payload: pollStatus ? { launchId: input.launchId } : input,
...(!pollStatus ? { requireLaunchAvailability: true } : {}),
isAuthorized: stableRequest.isDispatchAuthorized,
deadline: attemptDeadline,
...(!pollStatus
@@ -538,7 +544,6 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
) {
throw error;
}
availabilityCode = error.code;
pollStatus = false;
}
delayMs = await waitBeforeRetry({
@@ -548,11 +553,7 @@ export function createNodeWorkerLaunchAdapter(options: NodeWorkerLaunchAdapterOp
}
} catch (error) {
if (!dispatchReady && availabilityDeadline.signal.aborted && !deadline.signal.aborted) {
// Keep the latest discovery reason through the grace; a connected full
// node is retryable capacity, not an instruction to reconnect the device.
throw availabilityCode === "AT_CAPACITY"
? new WorkerRunnerCapacityError()
: new WorkerRunnerUnavailableError();
throw new WorkerRunnerUnavailableError();
}
// The node authors this result only after its durable claim stayed absent.
// Transport dispatch is therefore not launch ambiguity and needs no cancel.
@@ -0,0 +1,394 @@
import { describe, expect, it, vi } from "vitest";
import { WORKER_RPC_SET_VERSION } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { createDeferred } from "../../../test/helpers/promise.js";
import { NODE_WORKER_ENVIRONMENT_STOP_COMMAND } from "../../infra/node-commands.js";
import { parseWorkerLaunchPlan } from "../../worker/launch-descriptor.js";
import type { NodeWorkerSupervisorReceipt } from "../../worker/node-supervisor-protocol.js";
import type { NodeWorkerSupervisorTransport } from "../node-registry-private.js";
import type { createDeviceWorkerRuntime } from "./device-provider.js";
import { createNodeWorkerTunnelManager } from "./node-worker-tunnel.js";
import {
BUILD,
environment,
startRequest,
transport,
workspaceTransfer,
} from "./node-worker-tunnel.test-support.js";
import { sameWorkerSessionTurnClaim } from "./placement-record.js";
type NodeWorkerLaunch = ReturnType<typeof createDeviceWorkerRuntime>["launchNodeWorker"];
type TerminalReceipt = Extract<
NodeWorkerSupervisorReceipt,
{ state: "completed" | "failed" | "interrupted" | "cancelled" }
>;
function plan() {
return parseWorkerLaunchPlan({
version: 4,
admission: {
environmentId: "environment-1",
credential: "worker-credential-fixture",
sessionId: "session-1",
ownerEpoch: 2,
rpcSetVersion: WORKER_RPC_SET_VERSION,
handshake: BUILD,
},
assignment: {
agentId: "main",
operationalRunInstance: { instanceId: "instance-1", runId: "run-1" },
agentRuntimeIdentityToken: "runtime-token",
runId: "run-1",
turnId: "turn-1",
prompt: "inspect",
suppressPromptTranscript: true,
workspaceDir: "/node/workspace",
modelRef: { provider: "openai", model: "gpt-5.6-luna" },
inferenceOptions: {},
initialMessages: [],
transcript: { baseLeafId: null, nextSeq: 1 },
liveEvents: { ackedSeq: 0, nextSeq: 1 },
toolAuthority: { allowedToolNames: [] },
},
});
}
function turnClaim() {
return {
sessionId: "session-1",
claimId: "claim-1",
runId: "run-1",
placementGeneration: 4,
owner: { kind: "worker" as const, environmentId: "environment-1", ownerEpoch: 2 },
};
}
describe("node worker tunnel lifetime", () => {
it("revalidates the exact claim when a same-run replacement launches", async () => {
const record = environment();
let currentClaim = turnClaim();
const authorizations: boolean[] = [];
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn<NodeWorkerLaunch>(async (request) => {
authorizations.push(request.isDispatchAuthorized());
return {
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "test launch finished",
};
}),
validateWorkerTurn: (claim) => sameWorkerSessionTurnClaim(claim, currentClaim),
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
const staleClaim = currentClaim;
currentClaim = { ...staleClaim, claimId: "claim-2", placementGeneration: 5 };
await handle.launchTurn({ plan: plan(), turnClaim: staleClaim });
await handle.launchTurn({ plan: plan(), turnClaim: currentClaim });
expect(authorizations).toEqual([false, true]);
});
it("projects a terminal gateway connection failure into the launch result", async () => {
const record = environment();
const errorText =
"worker admission deadline exceeded after 3 attempts to gateway.example:18789: connect failed: Opening handshake has timed out";
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn<NodeWorkerLaunch>(async (request) => ({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText,
})),
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
await expect(
handle.launchTurn({ plan: plan(), turnClaim: turnClaim() }),
).resolves.toMatchObject({
code: 1,
killed: true,
stderr: errorText,
});
});
it("reuses only the exact same epoch binding", async () => {
const record = environment();
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const first = await manager.start(startRequest());
await expect(manager.start(startRequest())).resolves.toBe(first);
await expect(manager.start({ ...startRequest(), sessionId: "session-other" })).rejects.toThrow(
"binding changed",
);
});
it("closes remote-exec workspaces without requiring an embedded worker lifetime command", async () => {
const record = environment();
record.profileSnapshot = { ...record.profileSnapshot, executionMode: "remote-exec" };
const nodeTransport = transport();
const nodes = await nodeTransport.listCurrentNodes();
for (const node of nodes) {
delete node.workerHost.environmentSession;
}
nodeTransport.listCurrentNodes = async () => nodes;
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>();
nodeTransport.invoke = invoke;
const transfer = workspaceTransfer();
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
workspaceTransfer: transfer,
});
const handle = await manager.start({ ...startRequest(), executionMode: "remote-exec" });
await expect(handle.launchTurn({ plan: plan(), turnClaim: turnClaim() })).rejects.toThrow(
"remote-exec",
);
// Later durable changes cannot widen the retiring handle's process ownership.
record.profileSnapshot = { ...record.profileSnapshot, executionMode: "worker-turn" };
await handle.stop();
expect(invoke).not.toHaveBeenCalled();
expect(transfer.close).toHaveBeenCalledExactlyOnceWith(record.environmentId);
});
it("stops a completed turn's environment before exposing its replacement", async () => {
const record = environment();
const stopped = createDeferred();
const nodeTransport = transport();
const invoke = vi.fn<NodeWorkerSupervisorTransport["invoke"]>(async (request) => {
expect(request.isDispatchAuthorized()).toBe(true);
await stopped.promise;
return { ok: true, payloadJSON: "null" };
});
nodeTransport.invoke = invoke;
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: async (request) => ({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: record.environmentId,
sessionId: "session-1",
ownerEpoch: 2,
placementGeneration: 4,
runId: "run-1",
state: "completed",
resultJson: '{"status":"completed"}',
}),
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const first = await manager.start(startRequest());
await expect(first.launchTurn({ plan: plan(), turnClaim: turnClaim() })).resolves.toMatchObject(
{ code: 0 },
);
record.ownerEpoch = 3;
const replacing = manager.start({ ...startRequest(), ownerEpoch: 3 });
await vi.waitFor(() => expect(invoke).toHaveBeenCalledOnce());
expect(manager.status(record.environmentId)).toBe("connecting");
expect(invoke.mock.calls[0]?.[0]).toMatchObject({
command: NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
params: { environmentId: record.environmentId, sessionId: "session-1", ownerEpoch: 2 },
});
stopped.resolve();
await replacing;
expect(invoke.mock.calls[0]?.[0].isDispatchAuthorized()).toBe(false);
await manager.stop(record.environmentId, 2);
expect(invoke).toHaveBeenCalledOnce();
expect(manager.status(record.environmentId)).toBe("connected");
await manager.stop(record.environmentId, 3);
expect(invoke.mock.calls[1]?.[0].params).toMatchObject({ ownerEpoch: 3 });
});
it.each(["stop", "stopAll"] as const)(
"%s recovers exact durable owners after restart and retries an unconfirmed stop",
async (operation) => {
const record = environment();
record.bootstrapReceipt = null;
const nodeTransport = transport();
const invoke = vi
.fn<NodeWorkerSupervisorTransport["invoke"]>()
.mockResolvedValueOnce({ ok: false, error: { code: "DISCONNECTED" } })
.mockResolvedValue({ ok: true, payloadJSON: "null" });
nodeTransport.invoke = invoke;
const transfer = { ...workspaceTransfer(), closeAll: vi.fn(async () => {}) };
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
workspaceTransfer: transfer,
});
const stop = () =>
operation === "stop" ? manager.stop(record.environmentId, 2) : manager.stopAll();
await expect(stop()).rejects.toThrow("DISCONNECTED");
expect(invoke.mock.calls[0]?.[0].isDispatchAuthorized()).toBe(false);
await stop();
expect(invoke.mock.calls.map(([request]) => request.params)).toEqual([
expect.objectContaining({
environmentId: record.environmentId,
sessionId: "session-1",
ownerEpoch: 2,
}),
expect.objectContaining({
environmentId: record.environmentId,
sessionId: "session-1",
ownerEpoch: 2,
}),
]);
},
);
it("cancels a replacement start before it can install a late handle", async () => {
const record = environment();
const releaseLaunch = createDeferred();
const launch: NodeWorkerLaunch = async (request): Promise<TerminalReceipt> =>
await new Promise<TerminalReceipt>((resolve) => {
request.signal?.addEventListener(
"abort",
() => {
void releaseLaunch.promise.then(() => {
resolve({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "node worker cancelled",
});
});
},
{ once: true },
);
});
const launchNodeWorker = vi.fn(launch);
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker,
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const first = await manager.start(startRequest());
const launched = first.launchTurn({
plan: plan(),
turnClaim: turnClaim(),
timeoutMs: 5_000,
});
await vi.waitFor(() => expect(launchNodeWorker).toHaveBeenCalledOnce());
record.ownerEpoch = 3;
const replacement = manager.start({ ...startRequest(), ownerEpoch: 3 });
const stopping = manager.stop("environment-1", 3);
const stopSettled = vi.fn();
void stopping.then(stopSettled, stopSettled);
await expect(replacement).rejects.toThrow("stopped before connecting");
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(stopSettled).not.toHaveBeenCalled();
releaseLaunch.resolve();
await stopping;
await expect(launched).resolves.toMatchObject({ code: 1, killed: true });
expect(manager.status("environment-1")).toBe("stopped");
});
it("keeps cancellation authorized until an active launch settles", async () => {
const record = environment();
let cancellationWasAuthorized = false;
const onDispatchReady = vi.fn();
const launch: NodeWorkerLaunch = async (request): Promise<TerminalReceipt> => {
request.onDispatchReady?.();
return await new Promise<TerminalReceipt>((resolve) => {
request.signal?.addEventListener(
"abort",
() => {
cancellationWasAuthorized = request.isCancellationAuthorized();
resolve({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "node worker cancelled",
});
},
{ once: true },
);
});
};
const launchNodeWorker = vi.fn(launch);
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker,
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
const launched = handle.launchTurn({
plan: plan(),
turnClaim: turnClaim(),
timeoutMs: 5_000,
onDispatchReady,
});
await vi.waitFor(() => expect(launchNodeWorker).toHaveBeenCalledOnce());
expect(onDispatchReady).toHaveBeenCalledOnce();
await handle.stop();
await expect(launched).resolves.toMatchObject({ code: 1, killed: true });
expect(cancellationWasAuthorized).toBe(true);
expect(manager.status("environment-1")).toBe("stopped");
});
});
@@ -4,6 +4,7 @@ import {
GATEWAY_CLIENT_MODES,
} from "../../../packages/gateway-protocol/src/client-info.js";
import { WORKER_PROTOCOL_FEATURES } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { NODE_WORKER_ENVIRONMENT_STOP_COMMAND } from "../../infra/node-commands.js";
import { NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE } from "../../infra/node-runner-inventory.js";
import type { SpawnResult } from "../../process/exec.js";
import type { NodeWorkerSupervisorTransport } from "../node-registry-private.js";
@@ -55,17 +56,21 @@ export function transport(): NodeWorkerSupervisorTransport {
clientId: GATEWAY_CLIENT_IDS.NODE_HOST,
clientMode: GATEWAY_CLIENT_MODES.NODE,
protocolFeature: NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
workerHost: { enabled: true, capacity: { total: 2, available: 2 } },
workerHost: { enabled: true, capacity: { total: 2, available: 2 }, environmentSession: 1 },
commands: ["system.run"],
},
],
isCurrent: () => true,
invoke: async () => ({ ok: false, error: { code: "UNAVAILABLE" } }),
invoke: async ({ command }) =>
command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND
? { ok: true, payloadJSON: "null" }
: { ok: false, error: { code: "UNAVAILABLE" } },
};
}
export function startRequest() {
return {
executionMode: "worker-turn" as const,
environmentId: "environment-1",
ownerEpoch: 2,
deviceId: "node-1",
@@ -3,21 +3,17 @@ import { createHash } from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { WORKER_RPC_SET_VERSION } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { createDeferred } from "../../../test/helpers/promise.js";
import { useAutoCleanupTempDirTracker } from "../../../test/helpers/temp-dir.js";
import { parseWorkerLaunchPlan } from "../../worker/launch-descriptor.js";
import type { NodeWorkerSupervisorReceipt } from "../../worker/node-supervisor-protocol.js";
import { NODE_WORKER_ENVIRONMENT_STOP_COMMAND } from "../../infra/node-commands.js";
import type { NodeWorkerWorkspaceExecInput } from "../../worker/node-workspace-protocol.js";
import {
NODE_WORKSPACE_TRANSFER_ERROR_CODE,
NodeWorkerWorkspaceTransferError,
} from "../../worker/node-workspace-transfer-protocol.js";
import type { NodeWorkerSupervisorTransport } from "../node-registry-private.js";
import type { createDeviceWorkerRuntime } from "./device-provider.js";
import { createNodeWorkerTunnelManager } from "./node-worker-tunnel.js";
import {
BUILD,
environment,
startRequest,
transport,
@@ -25,7 +21,6 @@ import {
workspaceTransfer,
} from "./node-worker-tunnel.test-support.js";
import type { NodeWorkspaceTransferService } from "./node-workspace-transfer-service.js";
import { sameWorkerSessionTurnClaim } from "./placement-record.js";
import { serializeWorkerWorkspaceManifest } from "./workspace-manifest.js";
const workspaceInfo = vi.hoisted(() => vi.fn());
@@ -45,51 +40,6 @@ vi.mock("../../logging/subsystem.js", async (importOriginal) => {
});
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
type NodeWorkerLaunch = ReturnType<typeof createDeviceWorkerRuntime>["launchNodeWorker"];
type TerminalReceipt = Extract<
NodeWorkerSupervisorReceipt,
{ state: "completed" | "failed" | "interrupted" | "cancelled" }
>;
function plan() {
return parseWorkerLaunchPlan({
version: 4,
admission: {
environmentId: "environment-1",
credential: "worker-credential-fixture",
sessionId: "session-1",
ownerEpoch: 2,
rpcSetVersion: WORKER_RPC_SET_VERSION,
handshake: BUILD,
},
assignment: {
agentId: "main",
operationalRunInstance: { instanceId: "instance-1", runId: "run-1" },
agentRuntimeIdentityToken: "runtime-token",
runId: "run-1",
turnId: "turn-1",
prompt: "inspect",
suppressPromptTranscript: true,
workspaceDir: "/node/workspace",
modelRef: { provider: "openai", model: "gpt-5.6-luna" },
inferenceOptions: {},
initialMessages: [],
transcript: { baseLeafId: null, nextSeq: 1 },
liveEvents: { ackedSeq: 0, nextSeq: 1 },
toolAuthority: { allowedToolNames: [] },
},
});
}
function turnClaim() {
return {
sessionId: "session-1",
claimId: "claim-1",
runId: "run-1",
placementGeneration: 4,
owner: { kind: "worker" as const, environmentId: "environment-1", ownerEpoch: 2 },
};
}
describe("node worker tunnel manager", () => {
it.each([
@@ -124,7 +74,10 @@ describe("node worker tunnel manager", () => {
const rawManifest = serializeWorkerWorkspaceManifest(manifest);
const manifestRef = `sha256:${createHash("sha256").update(rawManifest).digest("hex")}`;
const nodeTransport = transport();
nodeTransport.invoke = vi.fn(async ({ params }) => {
nodeTransport.invoke = vi.fn(async ({ command, params }) => {
if (command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND) {
return { ok: true, payloadJSON: "null" };
}
const input = params as NodeWorkerWorkspaceExecInput;
let stdout = "";
let stderr = "";
@@ -173,6 +126,7 @@ describe("node worker tunnel manager", () => {
const handle = await createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: environment,
listEnvironments: () => [environment()],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -212,92 +166,6 @@ describe("node worker tunnel manager", () => {
await handle.stop();
});
it("revalidates the exact claim when a same-run replacement launches", async () => {
const record = environment();
let currentClaim = turnClaim();
const authorizations: boolean[] = [];
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
getTransport: transport,
launchNodeWorker: vi.fn<NodeWorkerLaunch>(async (request) => {
authorizations.push(request.isDispatchAuthorized());
return {
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "test launch finished",
};
}),
validateWorkerTurn: (claim) => sameWorkerSessionTurnClaim(claim, currentClaim),
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
const staleClaim = currentClaim;
currentClaim = { ...staleClaim, claimId: "claim-2", placementGeneration: 5 };
await handle.launchTurn({ plan: plan(), turnClaim: staleClaim });
await handle.launchTurn({ plan: plan(), turnClaim: currentClaim });
expect(authorizations).toEqual([false, true]);
});
it("projects a terminal gateway connection failure into the launch result", async () => {
const record = environment();
const errorText =
"worker admission deadline exceeded after 3 attempts to gateway.example:18789: connect failed: Opening handshake has timed out";
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
getTransport: transport,
launchNodeWorker: vi.fn<NodeWorkerLaunch>(async (request) => ({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText,
})),
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
await expect(
handle.launchTurn({ plan: plan(), turnClaim: turnClaim() }),
).resolves.toMatchObject({
code: 1,
killed: true,
stderr: errorText,
});
});
it("reuses only the exact same epoch binding", async () => {
const record = environment();
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
getTransport: transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const first = await manager.start(startRequest());
await expect(manager.start(startRequest())).resolves.toBe(first);
await expect(manager.start({ ...startRequest(), sessionId: "session-other" })).rejects.toThrow(
"binding changed",
);
});
it("joins same-owner starts while workspace binding resolution is pending", async () => {
const record = environment();
const workspaceBinding = createDeferred<undefined>();
@@ -305,6 +173,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -337,6 +206,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -369,6 +239,7 @@ describe("node worker tunnel manager", () => {
const closeAll = vi.fn(async () => {});
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
listEnvironments: () => [],
getEnvironment: (environmentId) => ({
...environment(),
environmentId,
@@ -421,6 +292,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -481,6 +353,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -563,6 +436,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => {
const nodeTransport = transport();
return {
@@ -682,6 +556,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -741,6 +616,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -764,118 +640,6 @@ describe("node worker tunnel manager", () => {
});
});
it("cancels a replacement start before it can install a late handle", async () => {
const record = environment();
const releaseLaunch = createDeferred();
const launch: NodeWorkerLaunch = async (request): Promise<TerminalReceipt> =>
await new Promise<TerminalReceipt>((resolve) => {
request.signal?.addEventListener(
"abort",
() => {
void releaseLaunch.promise.then(() => {
resolve({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "node worker cancelled",
});
});
},
{ once: true },
);
});
const launchNodeWorker = vi.fn(launch);
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
getTransport: transport,
launchNodeWorker,
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const first = await manager.start(startRequest());
const launched = first.launchTurn({
plan: plan(),
turnClaim: turnClaim(),
timeoutMs: 5_000,
});
await vi.waitFor(() => expect(launchNodeWorker).toHaveBeenCalledOnce());
record.ownerEpoch = 3;
const replacement = manager.start({ ...startRequest(), ownerEpoch: 3 });
const stopping = manager.stop("environment-1", 3);
const stopSettled = vi.fn();
void stopping.then(stopSettled, stopSettled);
await expect(replacement).rejects.toThrow("stopped before connecting");
await new Promise<void>((resolve) => {
setImmediate(resolve);
});
expect(stopSettled).not.toHaveBeenCalled();
releaseLaunch.resolve();
await stopping;
await expect(launched).resolves.toMatchObject({ code: 1, killed: true });
expect(manager.status("environment-1")).toBe("stopped");
});
it("keeps cancellation authorized until an active launch settles", async () => {
const record = environment();
let cancellationWasAuthorized = false;
const onDispatchReady = vi.fn();
const launch: NodeWorkerLaunch = async (request): Promise<TerminalReceipt> => {
request.onDispatchReady?.();
return await new Promise<TerminalReceipt>((resolve) => {
request.signal?.addEventListener(
"abort",
() => {
cancellationWasAuthorized = request.isCancellationAuthorized();
resolve({
launchId: request.input.launchId,
planHash: "b".repeat(64),
environmentId: request.input.descriptor.admission.environmentId,
sessionId: request.input.descriptor.admission.sessionId,
ownerEpoch: request.input.descriptor.admission.ownerEpoch,
placementGeneration: request.input.placementGeneration,
runId: request.input.descriptor.assignment.runId,
state: "cancelled",
errorText: "node worker cancelled",
});
},
{ once: true },
);
});
};
const launchNodeWorker = vi.fn(launch);
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
getTransport: transport,
launchNodeWorker,
validateWorkerTurn: () => true,
workspaceTransfer: workspaceTransfer(),
});
const handle = await manager.start(startRequest());
const launched = handle.launchTurn({
plan: plan(),
turnClaim: turnClaim(),
timeoutMs: 5_000,
onDispatchReady,
});
await vi.waitFor(() => expect(launchNodeWorker).toHaveBeenCalledOnce());
expect(onDispatchReady).toHaveBeenCalledOnce();
await handle.stop();
await expect(launched).resolves.toMatchObject({ code: 1, killed: true });
expect(cancellationWasAuthorized).toBe(true);
expect(manager.status("environment-1")).toBe("stopped");
});
it.each([
{
name: "divergence",
@@ -944,6 +708,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -1008,6 +773,7 @@ describe("node worker tunnel manager", () => {
const manager = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-device-1",
getEnvironment: () => record,
listEnvironments: () => [record],
getTransport: () => nodeTransport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => true,
@@ -1,8 +1,15 @@
import fsp from "node:fs/promises";
import { addTimerTimeoutGraceMs } from "@openclaw/normalization-core/number-coercion";
import type { WorkerAdmissionHandshake } from "../../../packages/gateway-protocol/src/schema/worker-admission.js";
import { sleepWithAbort } from "../../infra/backoff.js";
import { NODE_WORKER_WORKSPACE_EXEC_COMMAND } from "../../infra/node-commands.js";
import {
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_WORKSPACE_EXEC_COMMAND,
} from "../../infra/node-commands.js";
import {
formatNodeRunnerUpdateRequired,
NODE_RUNNER_UPDATE_REQUIRED_ISSUE,
NODE_WORKER_ENVIRONMENT_SESSION_VERSION,
} from "../../infra/node-runner-inventory.js";
import { createSubsystemLogger } from "../../logging/subsystem.js";
import type { SpawnResult } from "../../process/exec.js";
import { createDeferredCore, type Deferred } from "../../shared/deferred.js";
@@ -23,29 +30,21 @@ import type {
} from "../node-registry-private.js";
import { nodeWorkerGatewayNamespace } from "./node-worker-gateway-namespace.js";
import {
createNodeWorkerWorkspaceFallback,
recordNodeSyncPath,
} from "./node-worker-workspace-fallback.js";
createNodeWorkerWorkspaceActions,
type NodeWorkerWorkspaceBinding,
} from "./node-worker-workspace-actions.js";
import type { NodeWorkspaceTransferService } from "./node-workspace-transfer-service.js";
import type { WorkerSessionTurnClaim } from "./placement-record.js";
import type { WorkerEnvironmentRecord } from "./store.js";
import {
WorkerTunnelOwnerDisconnectedError,
type WorkerTunnelStopReason,
type WorkerTunnelStatus,
type WorkerTurnLaunchRequest,
type WorkerTurnTunnelHandle,
type WorkerWorkspaceCommand,
} from "./tunnel-contract.js";
import { boundedWorkerError } from "./worker-error.js";
import { serializeWorkerWorkspaceManifest } from "./workspace-manifest.js";
import { createWorkerWorkspaceQuiescence } from "./workspace-quiescence.js";
import {
applyStagedWorkerWorkspace,
assertWorkspaceResultStable,
recoverWorkerWorkspaceReconciliation,
type WorkerWorkspaceApplyResult,
} from "./workspace-reconcile.js";
import { workerWorkspaceResultStaging } from "./workspace-result-staging.js";
const DEFAULT_COMMAND_TIMEOUT_MS = 60_000;
const COMMAND_RESULT_GRACE_MS = 5_000;
@@ -64,6 +63,7 @@ const RETRYABLE_TRANSPORT_CODES = new Set([
type NodeWorkerLaunch = (request: {
deviceId: string;
input: {
environmentSession: 1;
launchId: string;
gatewayNamespace: string;
expectedBundleHash: string;
@@ -78,12 +78,6 @@ type NodeWorkerLaunch = (request: {
onDispatchReady?: () => void;
}) => Promise<Exclude<NodeWorkerSupervisorReceipt, { state: "pending" | "running" }>>;
type NodeWorkerWorkspaceBinding = {
localPath: string;
manifestRef: string;
remoteWorkspaceDir: string;
};
export type NodeWorkerWorkspaceBindingResolver = (binding: {
environmentId: string;
ownerEpoch: number;
@@ -93,6 +87,7 @@ export type NodeWorkerWorkspaceBindingResolver = (binding: {
type NodeWorkerTunnelManagerOptions = {
gatewayDeviceId: string;
getEnvironment: (environmentId: string) => WorkerEnvironmentRecord | undefined;
listEnvironments: () => readonly WorkerEnvironmentRecord[];
getTransport: () => NodeWorkerSupervisorTransport | undefined;
launchNodeWorker: NodeWorkerLaunch;
validateWorkerTurn: (claim: WorkerSessionTurnClaim) => boolean;
@@ -100,6 +95,7 @@ type NodeWorkerTunnelManagerOptions = {
};
type NodeWorkerTunnelStartRequest = {
executionMode: "worker-turn" | "remote-exec";
environmentId: string;
ownerEpoch: number;
deviceId: string;
@@ -107,14 +103,19 @@ type NodeWorkerTunnelStartRequest = {
expectedBuild: WorkerAdmissionHandshake;
};
type NodeTunnelEntry = NodeWorkerTunnelStartRequest & {
type NodeEnvironmentOwner = Omit<NodeWorkerTunnelStartRequest, "expectedBuild"> & {
stopPromise?: Promise<void>;
stopReason?: WorkerTunnelStopReason;
};
type NodeTunnelEntry = NodeEnvironmentOwner & {
expectedBuild: WorkerAdmissionHandshake;
abortController: AbortController;
gatewayNamespace: string;
handle?: WorkerTurnTunnelHandle;
initialization?: Promise<void>;
launchTasks: Set<Promise<unknown>>;
readiness: Deferred<WorkerTurnTunnelHandle>;
stopPromise?: Promise<void>;
};
function spawnResultFromReceipt(receipt: NodeWorkerSupervisorReceipt): SpawnResult {
@@ -181,6 +182,7 @@ function raceWithSignal<T>(operation: Promise<T>, signal: AbortSignal): Promise<
/** Owns node-channel handles without treating the persistent machine as a disposable lease. */
export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOptions) {
const entries = new Map<string, NodeTunnelEntry>();
const retiredEntries = new Set<NodeEnvironmentOwner>();
let resolveWorkspaceBinding: NodeWorkerWorkspaceBindingResolver | undefined;
const gatewayNamespace = nodeWorkerGatewayNamespace(options.gatewayDeviceId);
@@ -204,7 +206,7 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
hasDurableBinding(entry) && isLiveEntry(entry);
const findNode = async (
entry: NodeTunnelEntry,
entry: NodeEnvironmentOwner,
signal: AbortSignal,
): Promise<{ transport: NodeWorkerSupervisorTransport; node: NodeWorkerSupervisorNodeProof }> => {
const transport = options.getTransport();
@@ -305,200 +307,31 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
};
const createHandle = (
entry: Omit<NodeTunnelEntry, "handle" | "readiness" | "initialization">,
entry: NodeTunnelEntry,
restoredWorkspace: NodeWorkerWorkspaceBinding | undefined,
): { handle: WorkerTurnTunnelHandle; validateRestoredWorkspace: () => Promise<void> } => {
let workspaceReady = restoredWorkspace !== undefined;
const exec = async (command: Parameters<typeof runWorkspaceCommand>[2]) => {
if (!workspaceReady) {
throw new Error("node worker workspace is unavailable before sync");
}
return await runWorkspaceCommand(entry as NodeTunnelEntry, entry.ownerEpoch, command);
};
const workspace = createNodeWorkerWorkspaceFallback(exec);
const quiesceWorkspace = createWorkerWorkspaceQuiescence({
const { validateRestoredWorkspace, ...workspaceActions } = createNodeWorkerWorkspaceActions({
environmentId: entry.environmentId,
ownerEpoch: entry.ownerEpoch,
sessionId: entry.sessionId,
ownerSignal: entry.abortController.signal,
sharedHost: true,
runWorkspaceCommand: async (command) => await exec(command),
isOwnerCurrent: () => isLiveEntry(entry),
restoredWorkspace,
workspaceTransfer: options.workspaceTransfer,
runWorkspaceCommand: (command) => runWorkspaceCommand(entry, entry.ownerEpoch, command),
});
const validateRestoredWorkspace = async (): Promise<void> => {
if (!restoredWorkspace) {
return;
}
const prepared = await options.workspaceTransfer.prepareSync({
environmentId: entry.environmentId,
ownerEpoch: entry.ownerEpoch,
sessionId: entry.sessionId,
generation: entry.ownerEpoch,
localPath: restoredWorkspace.localPath,
// The transfer service re-reads the durable environment and credential together.
// This closure fences the exact in-memory tunnel instance without duplicating that read.
isAuthorized: () => isLiveEntry(entry as NodeTunnelEntry),
signal: entry.abortController.signal,
});
options.workspaceTransfer.revoke(entry.environmentId, prepared.token);
if (prepared.snapshot.manifestRef !== restoredWorkspace.manifestRef) {
throw new Error("Gateway workspace changed before node tunnel recovery");
}
const quiescence = await quiesceWorkspace(restoredWorkspace.remoteWorkspaceDir);
try {
const remoteManifestRef = await workspace.captureManifest(
restoredWorkspace.remoteWorkspaceDir,
prepared.snapshot.manifest.baseCommit,
restoredWorkspace.manifestRef,
);
if (remoteManifestRef !== restoredWorkspace.manifestRef) {
throw new Error("Node workspace changed before tunnel recovery");
}
} finally {
await quiescence.resume();
}
};
const reconcileWorkspace = async (
request: Parameters<WorkerTurnTunnelHandle["reconcileWorkspace"]>[0],
) => {
const pending = request.journal.load();
if (pending) {
await recoverWorkerWorkspaceReconciliation({ root: request.localPath, journal: pending });
request.journal.abort();
}
const uploadToken = options.workspaceTransfer.prepareUpload(
entry.environmentId,
request.baseManifestRef,
);
let uploadedResult: Awaited<ReturnType<typeof exec>>;
try {
uploadedResult = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: {
direction: "upload",
token: uploadToken,
baseManifestRef: request.baseManifestRef,
},
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
} finally {
options.workspaceTransfer.revoke(entry.environmentId, uploadToken);
}
if (uploadedResult.termination !== "exit" || uploadedResult.code !== 0) {
throw new Error("Node workspace reconcile upload failed");
}
const uploaded = options.workspaceTransfer.takeUpload(
entry.environmentId,
request.baseManifestRef,
);
try {
const changed = uploaded.currentManifestRef !== request.baseManifestRef;
let expectedRemoteRef = uploaded.currentManifestRef;
const verifyStable = async () => {
const observed = await workspace.captureManifest(
request.remoteWorkspaceDir,
uploaded.base.baseCommit,
expectedRemoteRef,
);
if (observed !== expectedRemoteRef) {
throw new Error("Cloud workspace changed during final reconciliation");
}
};
await verifyStable();
const publishAcceptedManifest = async (accepted: {
manifestRef: string;
manifest: typeof uploaded.current;
conflictPaths: string[];
}) => {
if (accepted.manifestRef === expectedRemoteRef) {
return;
}
const baseSnapshot = options.workspaceTransfer.getSnapshot(
entry.environmentId,
request.baseManifestRef,
);
const token = options.workspaceTransfer.publishSnapshot(entry.environmentId, {
manifest: accepted.manifest,
manifestRef: accepted.manifestRef,
rawManifest: serializeWorkerWorkspaceManifest(accepted.manifest),
root: await fsp.realpath(request.localPath),
...(baseSnapshot?.packPath ? { packPath: baseSnapshot.packPath } : {}),
});
try {
const published = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: { direction: "download", token, manifestRef: accepted.manifestRef },
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
if (
published.termination !== "exit" ||
published.code !== 0 ||
published.stdout.trim() !== accepted.manifestRef
) {
throw new Error("Node workspace accepted manifest publication failed");
}
expectedRemoteRef = accepted.manifestRef;
} finally {
options.workspaceTransfer.revoke(entry.environmentId, token);
}
};
const preparedStagedResult = request.stagedResult
? await workerWorkspaceResultStaging.prepareRequestedWorkerWorkspaceResult({
request,
stagingRoot: uploaded.stagingRoot,
currentManifestRef: uploaded.currentManifestRef,
baseManifestRaw: uploaded.baseRaw,
currentManifestRaw: uploaded.currentRaw,
publishAcceptedManifest,
})
: undefined;
let appliedWorkspaceResult: WorkerWorkspaceApplyResult | undefined;
if (!preparedStagedResult) {
appliedWorkspaceResult = await applyStagedWorkerWorkspace({
root: request.localPath,
stagingRoot: uploaded.stagingRoot,
baseManifestRef: request.baseManifestRef,
currentManifestRef: uploaded.currentManifestRef,
base: uploaded.base,
current: uploaded.current,
journal: request.journal,
publishAcceptedManifest,
});
}
return {
get manifestRef() {
return expectedRemoteRef;
},
changed,
verifyStable,
verifyLocalStable: async () =>
await (appliedWorkspaceResult?.verifyLocalStable() ??
assertWorkspaceResultStable({
root: request.localPath,
base: uploaded.base,
current: uploaded.current,
})),
getAppliedWorkspaceResult: () => appliedWorkspaceResult,
...(preparedStagedResult
? {
...preparedStagedResult,
applyPreparedStagedResult: async () => {
await preparedStagedResult.applyPreparedStagedResult();
appliedWorkspaceResult = preparedStagedResult.getAppliedWorkspaceResult();
},
}
: {}),
};
} finally {
await fsp.rm(uploaded.stagingRoot, { recursive: true, force: true });
}
};
const handle: WorkerTurnTunnelHandle = {
...workspaceActions,
environmentId: entry.environmentId,
ownerEpoch: entry.ownerEpoch,
launchTurn: async (request) => {
if (entry.executionMode !== "worker-turn") {
throw new Error("remote-exec environments do not launch embedded worker turns");
}
const plan = request.plan;
const claim = request.turnClaim;
const isDispatchAuthorized = () =>
isEnvironmentOwner(entry as NodeTunnelEntry) &&
isEnvironmentOwner(entry) &&
claim.owner.kind === "worker" &&
claim.owner.environmentId === entry.environmentId &&
claim.owner.ownerEpoch === entry.ownerEpoch &&
@@ -508,6 +341,7 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
const operation = options.launchNodeWorker({
deviceId: entry.deviceId,
input: {
environmentSession: 1,
launchId: plan.assignment.turnId,
gatewayNamespace,
expectedBundleHash: entry.expectedBuild.bundleHash,
@@ -515,7 +349,7 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
descriptor: plan,
},
isDispatchAuthorized,
isCancellationAuthorized: () => hasDurableBinding(entry as NodeTunnelEntry),
isCancellationAuthorized: () => hasDurableBinding(entry),
timeoutMs: request.timeoutMs ?? DEFAULT_COMMAND_TIMEOUT_MS,
...(request.credentialExpiresAtMs === undefined
? {}
@@ -532,89 +366,163 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
entry.launchTasks.delete(operation);
}
},
runWorkspaceCommand: async (command) => await exec(command),
syncWorkspace: async (request) => {
workspaceReady = true;
try {
const prepared = await options.workspaceTransfer.prepareSync({
environmentId: entry.environmentId,
ownerEpoch: entry.ownerEpoch,
sessionId: entry.sessionId,
generation: entry.ownerEpoch,
localPath: request.localPath,
// Durable owner state is revalidated by the transfer service after every awaited I/O.
isAuthorized: () => isLiveEntry(entry as NodeTunnelEntry),
signal: entry.abortController.signal,
});
try {
const originStartedAt = performance.now();
const origin = await workspace.trySyncWorkspace(request, prepared.snapshot.manifestRef);
recordNodeSyncPath(entry.environmentId, entry.sessionId, origin, originStartedAt);
if (origin.kind === "synced") {
return await workspace.finalizeSync(request, origin.result);
}
const transferred = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: {
direction: "download",
token: prepared.token,
manifestRef: prepared.snapshot.manifestRef,
},
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
if (
transferred.termination !== "exit" ||
transferred.code !== 0 ||
transferred.stdout.trim() !== prepared.snapshot.manifestRef
) {
throw new Error("Node workspace transfer failed");
}
return await workspace.finalizeSync(request, {
mode: prepared.snapshot.manifest.baseCommit ? ("git" as const) : ("plain" as const),
remoteWorkspaceDir: transferred.workspaceDir,
manifestRef: prepared.snapshot.manifestRef,
});
} finally {
options.workspaceTransfer.revoke(entry.environmentId, prepared.token);
}
} catch (error) {
workspaceReady = restoredWorkspace !== undefined;
throw error;
}
},
quiesceWorkspace,
reconcileWorkspace,
stop: async () => {
await stopEntry(entry as NodeTunnelEntry);
await stopEntry(entry);
},
};
return { handle, validateRestoredWorkspace };
};
function stopEntry(entry: NodeTunnelEntry): Promise<void> {
if (entry.stopPromise) {
return entry.stopPromise;
}
function stopEntry(entry: NodeTunnelEntry, reason?: WorkerTunnelStopReason): Promise<void> {
if (entries.get(entry.environmentId) === entry) {
entries.delete(entry.environmentId);
}
entry.abortController.abort(new Error("node worker tunnel owner stopped"));
entry.readiness.reject(new Error("node worker tunnel stopped before connecting"));
entry.stopPromise = (async () => {
return stopEnvironmentOwner(entry, reason, async () => {
await entry.initialization?.catch(() => undefined);
await Promise.allSettled(entry.launchTasks);
await options.workspaceTransfer.close(entry.environmentId);
})();
});
}
function stopEnvironmentOwner(
entry: NodeEnvironmentOwner,
reason?: WorkerTunnelStopReason,
drain?: () => Promise<void>,
): Promise<void> {
if (entry.stopPromise) {
if (entry.stopReason === reason || !retiredEntries.has(entry)) {
return entry.stopPromise;
}
// Shutdown and provider reconciliation can overlap. Drain the earlier operation,
// then apply the stronger proof without treating local fencing as physical cleanup.
return entry.stopPromise
.catch((error: unknown) => {
if (!reason) {
throw error;
}
})
.then(() => (retiredEntries.has(entry) ? stopEnvironmentOwner(entry, reason) : undefined));
}
retiredEntries.add(entry);
entry.stopReason = reason;
entry.stopPromise = (async () => {
await drain?.();
let stopping = true;
try {
// Remote-exec runtimes own their processes separately; this is only the embedded
// worker's environment lifetime, not a new requirement on the workspace transport.
if (entry.executionMode === "worker-turn" && reason === undefined) {
const signal = AbortSignal.timeout(DEFAULT_COMMAND_TIMEOUT_MS);
const { transport, node } = await findNode(entry, signal);
if (node.workerHost.environmentSession !== NODE_WORKER_ENVIRONMENT_SESSION_VERSION) {
throw new Error(
formatNodeRunnerUpdateRequired(node.nodeId, NODE_RUNNER_UPDATE_REQUIRED_ISSUE),
);
}
// Retirement retains only authority to stop this exact old scope, including after
// replacement. The node must match the tuple before touching any physical worker.
const operation = transport.invoke({
node,
command: NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
params: {
gatewayNamespace,
environmentId: entry.environmentId,
sessionId: entry.sessionId,
ownerEpoch: entry.ownerEpoch,
},
timeoutMs: DEFAULT_COMMAND_TIMEOUT_MS,
signal,
isDispatchAuthorized: () => stopping && retiredEntries.has(entry),
});
const result = await raceWithSignal(operation, signal);
if (!result.ok) {
throw new Error(
`node worker environment stop failed (${result.error?.code ?? "UNAVAILABLE"})`,
);
}
}
} finally {
stopping = false;
await options.workspaceTransfer.close(entry.environmentId);
}
if (reason !== "provider-destroying") {
retiredEntries.delete(entry);
}
})().finally(() => {
// Failed or unconfirmed provider teardown keeps the exact owner retryable. Only
// physical-stop proof may release it and make subsequent stops idempotent.
if (retiredEntries.has(entry)) {
entry.stopPromise = undefined;
}
});
return entry.stopPromise;
}
async function stop(
environmentId: string,
ownerEpoch?: number,
reason?: WorkerTunnelStopReason,
): Promise<void> {
const matches = (entry: NodeEnvironmentOwner) =>
entry.environmentId === environmentId &&
(ownerEpoch === undefined || ownerEpoch === entry.ownerEpoch);
const live = [...entries.values()].filter(matches);
const retired = [...retiredEntries].filter(matches);
const operations = [
...live.map((entry) => stopEntry(entry, reason)),
...retired.map((entry) => stopEnvironmentOwner(entry, reason)),
];
if (operations.length === 0) {
// A restarted Gateway has no tunnel object. The durable attachment is the only
// source of the retired scope; bundle metadata is not cleanup authority.
const record = options.getEnvironment(environmentId);
if (record?.nodeDeviceId && (ownerEpoch === undefined || record.ownerEpoch === ownerEpoch)) {
if (reason) {
// Provider teardown owns the whole dedicated machine. No remote session tuple is
// needed for local transfer cleanup; durable ownership remains until its proof.
operations.push(options.workspaceTransfer.close(environmentId));
} else {
if (record.attachedSessionIds.length > 1) {
throw new Error("node worker environment teardown has an ambiguous session owner");
}
const sessionId = record.attachedSessionIds[0];
if (sessionId) {
operations.push(
stopEnvironmentOwner({
deviceId: record.nodeDeviceId,
environmentId,
ownerEpoch: record.ownerEpoch,
sessionId,
executionMode:
record.profileSnapshot.executionMode === "remote-exec"
? "remote-exec"
: "worker-turn",
}),
);
}
}
}
}
const outcomes = await Promise.allSettled(operations);
const failure = outcomes.find((outcome) => outcome.status === "rejected");
if (failure) {
throw failure.reason;
}
}
return {
bindWorkspaceBindingResolver(resolver: NodeWorkerWorkspaceBindingResolver): void {
resolveWorkspaceBinding = resolver;
},
async start(request: NodeWorkerTunnelStartRequest): Promise<WorkerTurnTunnelHandle> {
const current = entries.get(request.environmentId);
const retiring = [...retiredEntries].filter(
(entry) => entry.environmentId === request.environmentId,
);
if (retiring.some((entry) => entry.ownerEpoch > request.ownerEpoch)) {
throw new Error("node worker tunnel owner epoch is stale");
}
if (current) {
if (request.ownerEpoch < current.ownerEpoch) {
throw new Error("node worker tunnel owner epoch is stale");
@@ -622,6 +530,7 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
if (request.ownerEpoch === current.ownerEpoch) {
if (
current.abortController.signal.aborted ||
current.executionMode !== request.executionMode ||
current.deviceId !== request.deviceId ||
current.sessionId !== request.sessionId ||
!sameWorkerBuild(current.expectedBuild, request.expectedBuild)
@@ -647,6 +556,7 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
if (current) {
await stopEntry(current);
}
await Promise.all(retiring.map((owner) => stopEnvironmentOwner(owner)));
if (!isLiveEntry(entry)) {
return;
}
@@ -685,14 +595,19 @@ export function createNodeWorkerTunnelManager(options: NodeWorkerTunnelManagerOp
});
return await readiness.promise;
},
async stop(environmentId: string, ownerEpoch?: number): Promise<void> {
const entry = entries.get(environmentId);
if (entry && (ownerEpoch === undefined || ownerEpoch === entry.ownerEpoch)) {
await stopEntry(entry);
}
},
stop,
async stopAll(): Promise<void> {
const stopped = await Promise.allSettled([...entries.values()].map(stopEntry));
const environmentIds = new Set([
...entries.keys(),
...[...retiredEntries].map((entry) => entry.environmentId),
...options
.listEnvironments()
.filter((record) => record.nodeDeviceId)
.map((record) => record.environmentId),
]);
const stopped = await Promise.allSettled(
[...environmentIds].map((environmentId) => stop(environmentId)),
);
// Shared transfer state outlives every tunnel, even when a sibling's cleanup fails.
stopped.push(...(await Promise.allSettled([options.workspaceTransfer.closeAll()])));
const failure = stopped.find((result) => result.status === "rejected");
@@ -0,0 +1,282 @@
import fsp from "node:fs/promises";
import type { NodeWorkerWorkspaceExecResult } from "../../worker/node-workspace-protocol.js";
import {
createNodeWorkerWorkspaceFallback,
recordNodeSyncPath,
} from "./node-worker-workspace-fallback.js";
import type { NodeWorkspaceTransferService } from "./node-workspace-transfer-service.js";
import type { WorkerWorkspaceCommand, WorkerWorkspaceTunnelHandle } from "./tunnel-contract.js";
import { serializeWorkerWorkspaceManifest } from "./workspace-manifest.js";
import { createWorkerWorkspaceQuiescence } from "./workspace-quiescence.js";
import {
applyStagedWorkerWorkspace,
assertWorkspaceResultStable,
recoverWorkerWorkspaceReconciliation,
type WorkerWorkspaceApplyResult,
} from "./workspace-reconcile.js";
import { workerWorkspaceResultStaging } from "./workspace-result-staging.js";
export type NodeWorkerWorkspaceBinding = {
localPath: string;
manifestRef: string;
remoteWorkspaceDir: string;
};
type NodeWorkerWorkspaceActions = Pick<
WorkerWorkspaceTunnelHandle,
"runWorkspaceCommand" | "syncWorkspace" | "quiesceWorkspace" | "reconcileWorkspace"
> & { validateRestoredWorkspace: () => Promise<void> };
export function createNodeWorkerWorkspaceActions(params: {
environmentId: string;
ownerEpoch: number;
sessionId: string;
ownerSignal: AbortSignal;
isOwnerCurrent: () => boolean;
restoredWorkspace?: NodeWorkerWorkspaceBinding;
workspaceTransfer: NodeWorkspaceTransferService;
runWorkspaceCommand: (
command: WorkerWorkspaceCommand & { resetWorkspace?: boolean },
) => Promise<NodeWorkerWorkspaceExecResult>;
}): NodeWorkerWorkspaceActions {
const { restoredWorkspace } = params;
let workspaceReady = restoredWorkspace !== undefined;
const exec = async (command: WorkerWorkspaceCommand & { resetWorkspace?: boolean }) => {
if (!workspaceReady) {
throw new Error("node worker workspace is unavailable before sync");
}
return await params.runWorkspaceCommand(command);
};
const workspace = createNodeWorkerWorkspaceFallback(exec);
const quiesceWorkspace = createWorkerWorkspaceQuiescence({
ownerSignal: params.ownerSignal,
sharedHost: true,
runWorkspaceCommand: exec,
});
const validateRestoredWorkspace = async (): Promise<void> => {
if (!restoredWorkspace) {
return;
}
const prepared = await params.workspaceTransfer.prepareSync({
environmentId: params.environmentId,
ownerEpoch: params.ownerEpoch,
sessionId: params.sessionId,
generation: params.ownerEpoch,
localPath: restoredWorkspace.localPath,
// The transfer service re-reads the durable environment and credential together.
// This closure fences the exact in-memory tunnel instance without duplicating that read.
isAuthorized: params.isOwnerCurrent,
signal: params.ownerSignal,
});
params.workspaceTransfer.revoke(params.environmentId, prepared.token);
if (prepared.snapshot.manifestRef !== restoredWorkspace.manifestRef) {
throw new Error("Gateway workspace changed before node tunnel recovery");
}
const quiescence = await quiesceWorkspace(restoredWorkspace.remoteWorkspaceDir);
try {
const remoteManifestRef = await workspace.captureManifest(
restoredWorkspace.remoteWorkspaceDir,
prepared.snapshot.manifest.baseCommit,
restoredWorkspace.manifestRef,
);
if (remoteManifestRef !== restoredWorkspace.manifestRef) {
throw new Error("Node workspace changed before tunnel recovery");
}
} finally {
await quiescence.resume();
}
};
const reconcileWorkspace = async (
request: Parameters<WorkerWorkspaceTunnelHandle["reconcileWorkspace"]>[0],
) => {
const pending = request.journal.load();
if (pending) {
await recoverWorkerWorkspaceReconciliation({ root: request.localPath, journal: pending });
request.journal.abort();
}
const uploadToken = params.workspaceTransfer.prepareUpload(
params.environmentId,
request.baseManifestRef,
);
let uploadedResult: Awaited<ReturnType<typeof exec>>;
try {
uploadedResult = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: {
direction: "upload",
token: uploadToken,
baseManifestRef: request.baseManifestRef,
},
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
} finally {
params.workspaceTransfer.revoke(params.environmentId, uploadToken);
}
if (uploadedResult.termination !== "exit" || uploadedResult.code !== 0) {
throw new Error("Node workspace reconcile upload failed");
}
const uploaded = params.workspaceTransfer.takeUpload(
params.environmentId,
request.baseManifestRef,
);
try {
const changed = uploaded.currentManifestRef !== request.baseManifestRef;
let expectedRemoteRef = uploaded.currentManifestRef;
const verifyStable = async () => {
const observed = await workspace.captureManifest(
request.remoteWorkspaceDir,
uploaded.base.baseCommit,
expectedRemoteRef,
);
if (observed !== expectedRemoteRef) {
throw new Error("Cloud workspace changed during final reconciliation");
}
};
await verifyStable();
const publishAcceptedManifest = async (accepted: {
manifestRef: string;
manifest: typeof uploaded.current;
conflictPaths: string[];
}) => {
if (accepted.manifestRef === expectedRemoteRef) {
return;
}
const baseSnapshot = params.workspaceTransfer.getSnapshot(
params.environmentId,
request.baseManifestRef,
);
const token = params.workspaceTransfer.publishSnapshot(params.environmentId, {
manifest: accepted.manifest,
manifestRef: accepted.manifestRef,
rawManifest: serializeWorkerWorkspaceManifest(accepted.manifest),
root: await fsp.realpath(request.localPath),
...(baseSnapshot?.packPath ? { packPath: baseSnapshot.packPath } : {}),
});
try {
const published = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: { direction: "download", token, manifestRef: accepted.manifestRef },
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
if (
published.termination !== "exit" ||
published.code !== 0 ||
published.stdout.trim() !== accepted.manifestRef
) {
throw new Error("Node workspace accepted manifest publication failed");
}
expectedRemoteRef = accepted.manifestRef;
} finally {
params.workspaceTransfer.revoke(params.environmentId, token);
}
};
const preparedStagedResult = request.stagedResult
? await workerWorkspaceResultStaging.prepareRequestedWorkerWorkspaceResult({
request,
stagingRoot: uploaded.stagingRoot,
currentManifestRef: uploaded.currentManifestRef,
baseManifestRaw: uploaded.baseRaw,
currentManifestRaw: uploaded.currentRaw,
publishAcceptedManifest,
})
: undefined;
let appliedWorkspaceResult: WorkerWorkspaceApplyResult | undefined;
if (!preparedStagedResult) {
appliedWorkspaceResult = await applyStagedWorkerWorkspace({
root: request.localPath,
stagingRoot: uploaded.stagingRoot,
baseManifestRef: request.baseManifestRef,
currentManifestRef: uploaded.currentManifestRef,
base: uploaded.base,
current: uploaded.current,
journal: request.journal,
publishAcceptedManifest,
});
}
return {
get manifestRef() {
return expectedRemoteRef;
},
changed,
verifyStable,
verifyLocalStable: async () =>
await (appliedWorkspaceResult?.verifyLocalStable() ??
assertWorkspaceResultStable({
root: request.localPath,
base: uploaded.base,
current: uploaded.current,
})),
getAppliedWorkspaceResult: () => appliedWorkspaceResult,
...(preparedStagedResult
? {
...preparedStagedResult,
applyPreparedStagedResult: async () => {
await preparedStagedResult.applyPreparedStagedResult();
appliedWorkspaceResult = preparedStagedResult.getAppliedWorkspaceResult();
},
}
: {}),
};
} finally {
await fsp.rm(uploaded.stagingRoot, { recursive: true, force: true });
}
};
return {
validateRestoredWorkspace,
runWorkspaceCommand: exec,
syncWorkspace: async (request) => {
workspaceReady = true;
try {
const prepared = await params.workspaceTransfer.prepareSync({
environmentId: params.environmentId,
ownerEpoch: params.ownerEpoch,
sessionId: params.sessionId,
generation: params.ownerEpoch,
localPath: request.localPath,
// Durable owner state is revalidated by the transfer service after every awaited I/O.
isAuthorized: params.isOwnerCurrent,
signal: params.ownerSignal,
});
try {
const originStartedAt = performance.now();
const origin = await workspace.trySyncWorkspace(request, prepared.snapshot.manifestRef);
recordNodeSyncPath(params.environmentId, params.sessionId, origin, originStartedAt);
if (origin.kind === "synced") {
return await workspace.finalizeSync(request, origin.result);
}
const transferred = await exec({
argv: ["openclaw-internal-workspace-transfer"],
transfer: {
direction: "download",
token: prepared.token,
manifestRef: prepared.snapshot.manifestRef,
},
timeoutMs: 10 * 60_000,
transportRetry: "never",
});
if (
transferred.termination !== "exit" ||
transferred.code !== 0 ||
transferred.stdout.trim() !== prepared.snapshot.manifestRef
) {
throw new Error("Node workspace transfer failed");
}
return await workspace.finalizeSync(request, {
mode: prepared.snapshot.manifest.baseCommit ? ("git" as const) : ("plain" as const),
remoteWorkspaceDir: transferred.workspaceDir,
manifestRef: prepared.snapshot.manifestRef,
});
} finally {
params.workspaceTransfer.revoke(params.environmentId, prepared.token);
}
} catch (error) {
workspaceReady = restoredWorkspace !== undefined;
throw error;
}
},
quiesceWorkspace,
reconcileWorkspace,
};
}
@@ -115,6 +115,7 @@ export function isCurrentActiveWorkerEnvironment(
return Boolean(
environment &&
environment.state === "attached" &&
environment.destroyRequestedAtMs === null &&
placement.environmentId &&
environment.environmentId === placement.environmentId &&
placement.activeOwnerEpoch !== null &&
@@ -23,6 +23,47 @@ import { createWorkerWorkspaceOperationCoordinator } from "./workspace-operation
describe("worker placement restart recovery", () => {
support.setupWorkerEnvironmentServiceSuite();
it.each(["startup", "active"] as const)(
"fences a destroy-requested attachment during %s recovery even when physical cleanup fails",
async (mode) => {
const placements = createWorkerSessionPlacementStore({
database: support.testState.stateDb,
now: () => 1_000,
});
const harness = createHarness(placements, { destroyFails: true });
await harness.environments.attachSession({
environmentId: harness.ready.environmentId,
ownerEpoch: harness.ready.ownerEpoch,
sessionId: REQUEST.sessionId,
});
const environment = {
...harness.attached,
nodeDeviceId: "revoked-node",
destroyRequestedAtMs: 1_000,
};
vi.mocked(harness.environments.get).mockReturnValue(environment);
vi.mocked(harness.environments.stopTunnel).mockRejectedValue(
new Error("node role revoked before stop confirmation"),
);
harness.placements.seedActive(environment.ownerEpoch);
if (mode === "startup") {
await harness.service.reconcile("startup");
} else {
await harness.service.reconcileActive(environment.environmentId);
}
expect(harness.placements.current()).toMatchObject({
state: "failed",
environmentId: environment.environmentId,
activeOwnerEpoch: environment.ownerEpoch,
recoveryError: expect.stringContaining("node role revoked before stop confirmation"),
});
expect(harness.environments.destroy).toHaveBeenCalledWith(environment.environmentId);
expect(harness.environments.startTunnel).not.toHaveBeenCalled();
},
);
it.each([
{
failure: "provider no longer supports the persisted execution mode",
@@ -14,6 +14,7 @@ import { verifyWorkerAdmissionHandshake } from "./admission.js";
import type { WorkerInstallationArtifact } from "./bundle.js";
import type { WorkerProviderLifecycleOptions } from "./provider-lifecycle.types.js";
import { createWorkerNodeProvisioning } from "./provider-node-provisioning.js";
import { createWorkerProviderOwnerLifecycle } from "./provider-owner-lifecycle.js";
import {
requestStaleWorkerDestroy,
retireMismatchedWorkerLease,
@@ -36,17 +37,12 @@ import { boundedWorkerError as boundedError } from "./worker-error.js";
const ORPHANED_LEASE_ERROR = "Worker provider no longer recognizes the lease";
export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOptions) {
const { store } = options;
const tunnels = options.tunnelManager;
const callBootstrap = options.callBootstrap;
const callProvider = options.callProvider;
const inState = options.inState;
const move = options.move;
const saveError = options.saveError;
const serviceError = options.serviceError;
const withLock = options.withLock;
const { store, callBootstrap, callProvider, inState, move, saveError, serviceError, withLock } =
options;
const { commitReady, ensurePendingCredential } = options.credentialBroker;
const { requireCurrentOwner, stopOwner } = createWorkerProviderOwnerLifecycle(options);
function requireWorkerProfile(value: unknown): WorkerProfile {
const error = validateCloudWorkerProfileSettings(value);
if (error) {
@@ -145,8 +141,8 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
terminalState: "failed",
lastError: detail,
});
const draining = move(requested, "draining", { ...leasePatch, lastError: detail });
await tunnels?.stop(record.environmentId);
const stopped = await stopOwner(requested);
const draining = move(stopped, "draining", { ...leasePatch, lastError: detail });
const destroying = move(draining, "destroying", { lastError: detail });
try {
await callProvider(record.environmentId, () =>
@@ -390,17 +386,25 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
throw serviceError("invalid_state", "Worker environment has no lease");
}
const leaseId = r.leaseId;
const draining = beginDrain(r);
await tunnels?.stop(r.environmentId);
// A dedicated provider's destroy result proves physical teardown even if its node is
// offline. Shared hosts retain the machine, so they still require the exact worker stop.
const providerOwnsMachine = r.nodeDeviceId !== null && r.sharedHost === false;
const stopped = await stopOwner(r, providerOwnsMachine ? "provider-destroying" : undefined);
const draining = providerOwnsMachine ? stopped : beginDrain(stopped);
const owningProvider = provider ?? providerFor(r.providerId);
const destroying = beginDestroy(draining);
const destroying = providerOwnsMachine ? draining : beginDestroy(draining);
try {
await callProvider(r.environmentId, () => owningProvider.destroy(lifecycleLease(r, leaseId)));
await callProvider(r.environmentId, () => {
requireCurrentOwner(destroying);
return owningProvider.destroy(lifecycleLease(r, leaseId));
});
} catch (error) {
saveError(destroying, error);
throw serviceError("provider_failure", "Worker provider operation failed");
}
return await finishProvenDestroy(destroying);
return await finishProvenDestroy(
providerOwnsMachine ? await stopOwner(destroying, "provider-destroyed") : destroying,
);
};
const reconcileRecord = async (initialRecord: WorkerEnvironmentRecord): Promise<void> => {
@@ -456,13 +460,14 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
}
const { status } = inspection;
const teardownExpected = record.destroyRequestedAtMs !== null || record.state === "destroying";
if (status === "destroyed" || (status === "unknown" && teardownExpected)) {
if (status === "destroyed") {
requireCurrentOwner(record);
const requested =
record.destroyRequestedAtMs === null
? store.requestDestroy({
environmentId: record.environmentId,
state: record.state,
...(status === "destroyed" && !teardownExpected
...(!teardownExpected
? {
terminalState: "failed",
lastError: "Worker environment disappeared before teardown was requested",
@@ -470,20 +475,26 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
: {}),
})
: record;
const draining = beginDrain(requested);
await tunnels?.stop(record.environmentId);
const stopped = await stopOwner(requested, "provider-destroyed");
const draining = beginDrain(stopped);
await finishProvenDestroy(draining).catch((error: unknown) => {
saveError(draining, error);
});
return;
}
if (status === "unknown") {
const draining =
record.state === "draining"
? record
: move(record, "draining", { lastError: ORPHANED_LEASE_ERROR });
await tunnels?.stop(record.environmentId);
move(draining, "orphaned", { lastError: ORPHANED_LEASE_ERROR });
requireCurrentOwner(record);
// Provider loss fences placement authority before remote cleanup, which may remain
// unreachable after node revocation. Preserve its exact attachment until stop is proven.
const requested = teardownExpected
? record
: store.requestDestroy({
environmentId: record.environmentId,
state: record.state,
terminalState: "failed",
lastError: ORPHANED_LEASE_ERROR,
});
await finishDestroy(requested, provider).catch(() => undefined);
return;
}
if (status === "dormant") {
@@ -498,7 +509,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
if (record.sharedHost !== null && record.sharedHost !== inspectedSharedHost) {
// Workspace actions capture isolation at tunnel creation. Fence the old actions before
// committing a provider-owned change so no reconciliation can use stale host scope.
await tunnels?.stop(record.environmentId);
record = await stopOwner(record);
}
record = store.reconcileSharedHost({
environmentId: record.environmentId,
@@ -542,7 +553,7 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
}
if (record.state === "draining" && record.destroyRequestedAtMs === null) {
// Draining without destroy intent is durable provider-loss cleanup.
await tunnels?.stop(record.environmentId);
record = await stopOwner(record);
move(record, "orphaned", { lastError: record.lastError ?? ORPHANED_LEASE_ERROR });
return;
}
@@ -575,9 +586,9 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
return;
}
}
record = await stopOwner(record);
const bootstrapping =
record.state === "bootstrapping" ? record : move(record, "bootstrapping");
await tunnels?.stop(record.environmentId, record.ownerEpoch);
await finishBootstrap(bootstrapping, provider, installation).catch(() => undefined);
return;
}
@@ -718,9 +729,6 @@ export function createWorkerProviderLifecycle(options: WorkerProviderLifecycleOp
if (record.state === "requested") {
return cancelRequested(record);
}
if (record.leaseId) {
record = beginDrain(record);
}
if (!record.leaseId) {
const provider = providerFor(record.providerId);
record = await resumeProvision(record, provider);
@@ -16,7 +16,7 @@ import type {
WorkerEnvironmentStore,
WorkerEnvironmentTransitionPatch,
} from "./store.js";
import type { WorkerTunnelManager } from "./tunnel.js";
import type { WorkerTunnelStopReason } from "./tunnel-contract.js";
export type WorkerProviderLifecycleInputOptions = {
store: WorkerEnvironmentStore;
@@ -45,7 +45,13 @@ export type WorkerProviderLifecycleInputOptions = {
};
export type WorkerProviderLifecycleOptions = WorkerProviderLifecycleInputOptions & {
tunnelManager?: Pick<WorkerTunnelManager, "stop">;
tunnelManager?: {
stop(
environmentId: string,
ownerEpoch?: number,
reason?: WorkerTunnelStopReason,
): Promise<void>;
};
credentialBroker: WorkerCredentialBroker;
callBootstrap: <T>(
installation: WorkerInstallationArtifact,
@@ -0,0 +1,221 @@
import { describe, expect, it, vi } from "vitest";
import { createNodeWorkerTunnelManager } from "./node-worker-tunnel.js";
import * as nodeTunnelSupport from "./node-worker-tunnel.test-support.js";
import * as support from "./service.test-support.js";
async function disconnectedNodeOwner(environmentId: string, sharedHost: boolean | null = false) {
const deviceId = `node:${environmentId}`;
support.seedReadyNodeDesktop(environmentId);
const attached = support.testState.store.transition({
environmentId,
from: "ready",
to: "attached",
patch: {
...support.attachedPatch(environmentId, "session-destroyed"),
...(sharedHost === null ? {} : { sharedHost }),
},
});
const transport = nodeTunnelSupport.transport();
const connectedNodes = await transport.listCurrentNodes();
for (const node of connectedNodes) {
node.nodeId = deviceId;
}
const listNodes = vi.fn<typeof transport.listCurrentNodes>(async () => []);
transport.listCurrentNodes = listNodes;
const workspaceTransfer = nodeTunnelSupport.workspaceTransfer();
workspaceTransfer.closeAll = vi.fn(async () => {});
const nodeTunnels = createNodeWorkerTunnelManager({
gatewayDeviceId: "gateway-1",
getEnvironment: (id) => support.testState.store.get(id),
listEnvironments: () => support.testState.store.list(),
getTransport: () => transport,
launchNodeWorker: vi.fn(),
validateWorkerTurn: () => false,
workspaceTransfer,
});
return {
attached,
nodeTunnels,
listNodes,
workspaceTransfer,
start: () =>
nodeTunnels.start({
environmentId,
ownerEpoch: attached.ownerEpoch,
deviceId,
sessionId: "session-destroyed",
executionMode: "worker-turn",
expectedBuild: support.BOOTSTRAP_RECEIPT,
}),
reconnect: () => listNodes.mockResolvedValue(connectedNodes),
};
}
describe("worker provider node teardown", () => {
support.setupWorkerEnvironmentServiceSuite();
it.each(["retained", "restarted"] as const)(
"cleans the %s node owner after the provider proves the machine is destroyed",
async (owner) => {
const environmentId = `worker-destroyed-node-${owner}`;
const { attached, nodeTunnels, listNodes, workspaceTransfer, start, reconnect } =
await disconnectedNodeOwner(environmentId);
const provider = support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
inspect: async () => ({ status: "destroyed" }),
});
const service = support.createService(provider, { nodeTunnelManager: nodeTunnels });
try {
if (owner === "retained") {
await start();
await expect(nodeTunnels.stop(environmentId, attached.ownerEpoch)).rejects.toThrow(
"not connected",
);
}
listNodes.mockClear();
vi.mocked(workspaceTransfer.close).mockClear();
await service.reconcileOnce();
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "failed",
attachedSessionIds: [],
leaseId: null,
nodeDeviceId: null,
lastError: "Worker environment disappeared before teardown was requested",
});
await nodeTunnels.stopAll();
expect(listNodes).not.toHaveBeenCalled();
expect(workspaceTransfer.close).toHaveBeenCalledWith(environmentId);
} finally {
reconnect();
}
},
);
it.each(["destroy", "reconcile"] as const)(
"destroys a disconnected dedicated lease without losing retry ownership via %s",
async (retry) => {
const environmentId = "worker-offline-destroy";
const { attached, nodeTunnels, listNodes, start, reconnect } =
await disconnectedNodeOwner(environmentId);
const destroy = vi
.fn(async () => {})
.mockRejectedValueOnce(new Error("provider destruction is indeterminate"));
const service = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
inspect: async () => ({ status: "unknown" }),
destroy,
}),
{ nodeTunnelManager: nodeTunnels },
);
try {
await start();
await expect(service.destroy(environmentId)).rejects.toMatchObject({
code: "provider_failure",
});
expect(nodeTunnels.status(environmentId)).toBe("stopped");
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "attached",
ownerEpoch: attached.ownerEpoch,
attachedSessionIds: ["session-destroyed"],
destroyRequestedAtMs: support.testState.nowMs,
});
expect(support.testState.store.getCredential(environmentId)).toBeUndefined();
if (retry === "destroy") {
await service.destroy(environmentId);
} else {
await service.reconcileOnce();
}
expect(support.testState.store.get(environmentId)?.state).toBe("destroyed");
expect(destroy).toHaveBeenCalledTimes(2);
await nodeTunnels.stopAll();
expect(listNodes).not.toHaveBeenCalled();
} finally {
reconnect();
}
},
);
it("requires confirmed worker stop before retiring a disconnected shared lease", async () => {
const environmentId = "worker-offline-shared-destroy";
const { nodeTunnels, reconnect } = await disconnectedNodeOwner(environmentId, true);
const destroy = vi.fn(async () => {});
const service = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
destroy,
}),
{ nodeTunnelManager: nodeTunnels },
);
try {
await expect(service.destroy(environmentId)).rejects.toThrow("not connected");
expect(destroy).not.toHaveBeenCalled();
expect(support.testState.store.get(environmentId)?.state).toBe("attached");
} finally {
reconnect();
}
});
it.each([
{ status: "destroyed", isolation: "shared", sharedHost: true },
{ status: "destroyed", isolation: "unknown", sharedHost: null },
{ status: "unknown", isolation: "shared", sharedHost: true },
{ status: "unknown", isolation: "unknown", sharedHost: null },
] as const)(
"requires exact worker stop for a $status lease with $isolation host isolation",
async ({ status, sharedHost }) => {
const environmentId = "worker-inspected-destroyed-shared";
const { attached, nodeTunnels, start, reconnect } = await disconnectedNodeOwner(
environmentId,
sharedHost,
);
const destroy = vi.fn(async () => {});
const lastError =
status === "unknown"
? "Worker provider no longer recognizes the lease"
: "Worker environment disappeared before teardown was requested";
const service = support.createService(
support.createProvider({
supportedExecutionModes: ["worker-turn", "remote-exec"],
inspect: async () => ({ status }),
destroy,
}),
{ nodeTunnelManager: nodeTunnels },
);
try {
await start();
await service.reconcileOnce();
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "attached",
ownerEpoch: attached.ownerEpoch,
attachedSessionIds: ["session-destroyed"],
nodeDeviceId: attached.nodeDeviceId,
leaseId: attached.leaseId,
destroyRequestedAtMs: support.testState.nowMs,
teardownTerminalState: "failed",
lastError,
});
expect(support.testState.store.getCredential(environmentId)).toBeUndefined();
expect(destroy).not.toHaveBeenCalled();
reconnect();
await service.reconcileOnce();
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "failed",
attachedSessionIds: [],
nodeDeviceId: null,
leaseId: null,
lastError,
});
expect(destroy).toHaveBeenCalledTimes(status === "unknown" ? 1 : 0);
} finally {
reconnect();
}
},
);
});
@@ -0,0 +1,47 @@
import { isDeepStrictEqual } from "node:util";
import type { WorkerProviderLifecycleOptions } from "./provider-lifecycle.types.js";
import type { WorkerEnvironmentRecord } from "./store.js";
import type { WorkerTunnelStopReason } from "./tunnel-contract.js";
export function createWorkerProviderOwnerLifecycle(
options: Pick<WorkerProviderLifecycleOptions, "store" | "tunnelManager" | "serviceError">,
) {
const { store, serviceError } = options;
const tunnels = options.tunnelManager;
const requireCurrentOwner = (record: WorkerEnvironmentRecord): WorkerEnvironmentRecord => {
const current = store.get(record.environmentId);
if (
!current ||
current.ownerEpoch !== record.ownerEpoch ||
current.state !== record.state ||
current.leaseId !== record.leaseId ||
current.nodeDeviceId !== record.nodeDeviceId ||
current.sharedHost !== record.sharedHost ||
!isDeepStrictEqual(current.attachedSessionIds, record.attachedSessionIds)
) {
throw serviceError("invalid_state", "Worker environment owner changed during teardown");
}
return current;
};
const stopOwner = async (
record: WorkerEnvironmentRecord,
reason?: WorkerTunnelStopReason,
): Promise<WorkerEnvironmentRecord> => {
requireCurrentOwner(record);
// Fence admission without erasing the attachment needed to stop a retained node worker.
// A crash or failed stop leaves the exact scope available for teardown replay.
store.revokeEnvironmentCredential(record.environmentId);
// Only a dedicated node lease makes provider teardown proof of worker termination.
// Shared or unknown host isolation still requires the exact worker's stop acknowledgement.
await tunnels?.stop(
record.environmentId,
record.ownerEpoch,
record.nodeDeviceId !== null && record.sharedHost === false ? reason : undefined,
);
return requireCurrentOwner(record);
};
return { requireCurrentOwner, stopOwner };
}
@@ -1,4 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import { createDeferred } from "../../../test/helpers/promise.js";
import { STALE_WORKER_BUILD_REASON } from "./admission.js";
import * as support from "./service.test-support.js";
import type { WorkerTunnelManager } from "./tunnel.js";
@@ -278,7 +279,12 @@ describe("worker environment service", () => {
await workerService.reconcileEnvironment(environmentId);
expect(support.testState.store.get(environmentId)?.state).toBe("orphaned");
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "failed",
leaseId: null,
destroyRequestedAtMs: support.testState.nowMs,
lastError: "Worker provider no longer recognizes the lease",
});
expect(workerService.validateWorkerConnection(admitted.identity)).toBe("credential-replaced");
});
@@ -588,8 +594,8 @@ describe("worker environment service", () => {
});
});
it("orphans unknown active leases and adopts unknown expected teardown", async () => {
support.seedReady("worker-unknown");
it("fences unknown leases before stop and retries their durable teardown", async () => {
const originalOwner = support.seedReady("worker-unknown");
support.seedReady("worker-transient");
support.seedReady("worker-destroyed-unknown");
support.testState.store.requestDestroy({
@@ -634,7 +640,15 @@ describe("worker environment service", () => {
await workerService.reconcileOnce();
expect(support.testState.store.get("worker-unknown")?.state).toBe("draining");
expect(support.testState.store.get("worker-unknown")).toMatchObject({
state: "ready",
ownerEpoch: originalOwner.ownerEpoch,
leaseId: originalOwner.leaseId,
attachedSessionIds: originalOwner.attachedSessionIds,
destroyRequestedAtMs: support.testState.nowMs,
teardownTerminalState: "failed",
lastError: "Worker provider no longer recognizes the lease",
});
expect(support.testState.store.get("worker-destroyed-unknown")?.state).toBe("destroying");
expect(workerService.validateWorkerConnection(admitted.identity)).toBe("credential-replaced");
expect(support.testState.store.get("worker-transient")).toMatchObject({
@@ -643,7 +657,11 @@ describe("worker environment service", () => {
});
await workerService.reconcileOnce();
expect(tunnelManager.stop).toHaveBeenCalledTimes(4);
expect(support.testState.store.get("worker-unknown")?.state).toBe("orphaned");
expect(support.testState.store.get("worker-unknown")).toMatchObject({
state: "failed",
leaseId: null,
lastError: "Worker provider no longer recognizes the lease",
});
expect(support.testState.store.get("worker-destroyed-unknown")).toMatchObject({
state: "destroyed",
});
@@ -807,6 +825,100 @@ describe("worker environment service", () => {
]);
});
it.each(["destroy", "reconcile"] as const)(
"%s preserves the exact attached owner until remote stop is confirmed across restart",
async (operation) => {
const environmentId = "worker-retained-teardown";
support.seedReady(environmentId);
const attached = support.testState.store.transition({
environmentId,
from: "ready",
to: "attached",
patch: support.attachedPatch(environmentId, "session-retained"),
});
let disconnected = true;
const stop = vi.fn(async (id: string, epoch?: number) => {
expect(support.testState.store.get(id)).toMatchObject({
state: "attached",
attachedSessionIds: ["session-retained"],
ownerEpoch: attached.ownerEpoch,
destroyRequestedAtMs: expect.any(Number),
});
expect(epoch).toBe(attached.ownerEpoch);
if (disconnected) {
throw new Error("node disconnected before stop confirmation");
}
});
const tunnelManager = {
stop,
stopAll: vi.fn(async () => {}),
} as unknown as WorkerTunnelManager;
const destroy = vi.fn(async () => {});
const provider = support.createProvider({ destroy });
const first = support.createService(provider, { tunnelManager });
if (operation === "destroy") {
await expect(first.destroy(environmentId)).rejects.toThrow("node disconnected");
} else {
support.testState.store.requestDestroy({ environmentId, state: "attached" });
await first.reconcileOnce();
}
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "attached",
ownerEpoch: attached.ownerEpoch,
attachedSessionIds: ["session-retained"],
});
expect(support.testState.store.getCredential(environmentId)).toBeUndefined();
expect(destroy).not.toHaveBeenCalled();
await first.stop();
disconnected = false;
const restarted = support.createService(provider, { tunnelManager });
await restarted.reconcileOnce();
expect(stop).toHaveBeenCalledTimes(2);
expect(destroy).toHaveBeenCalledOnce();
expect(support.testState.store.get(environmentId)).toMatchObject({ state: "destroyed" });
},
);
it("does not let an awaited old-owner stop retire a replacement attachment", async () => {
const environmentId = "worker-replaced-during-stop";
support.seedReady(environmentId);
const attached = support.testState.store.transition({
environmentId,
from: "ready",
to: "attached",
patch: support.attachedPatch(environmentId, "session-old"),
});
const stopReturned = createDeferred();
const stop = vi.fn(async () => await stopReturned.promise);
const tunnelManager = {
stop,
stopAll: vi.fn(async () => {}),
} as unknown as WorkerTunnelManager;
const service = support.createService(
support.createProvider({ inspect: async () => ({ status: "active", sharedHost: true }) }),
{ tunnelManager },
);
const reconciling = service.reconcileOnce();
await vi.waitFor(() => expect(stop).toHaveBeenCalledWith(environmentId, attached.ownerEpoch));
support.testState.store.transition({ environmentId, from: "attached", to: "idle" });
const replacement = support.testState.store.transition({
environmentId,
from: "idle",
to: "attached",
patch: support.attachedPatch(environmentId, "session-new"),
});
stopReturned.resolve();
await reconciling;
expect(support.testState.store.get(environmentId)).toMatchObject({
state: "attached",
ownerEpoch: replacement.ownerEpoch,
attachedSessionIds: ["session-new"],
});
expect(support.testState.store.getCredential(environmentId)?.sessionId).toBe("session-new");
});
it("adopts an unpersisted provision result before destroying", async () => {
const intent = support.testState.store.createIntent({
environmentId: "worker-pending-destroy",
+8 -2
View File
@@ -37,6 +37,7 @@ import type {
WorkerEnvironmentRecord,
WorkerEnvironmentTransitionPatch as TransitionPatch,
} from "./store.js";
import type { WorkerTunnelStopReason } from "./tunnel-contract.js";
import type { WorkerTunnelManager } from "./tunnel.js";
import { boundedWorkerError as boundedError } from "./worker-error.js";
import { createWorkerTurnRpc } from "./worker-turn-rpc.js";
@@ -141,10 +142,14 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService
options.nodeDesktopCarrier ||
options.nodePortalCarrier
? {
stop: async (environmentId: string, ownerEpoch?: number) => {
stop: async (
environmentId: string,
ownerEpoch?: number,
reason?: WorkerTunnelStopReason,
) => {
await Promise.all([
options.tunnelManager?.stop(environmentId, ownerEpoch),
options.nodeTunnelManager?.stop(environmentId, ownerEpoch),
options.nodeTunnelManager?.stop(environmentId, ownerEpoch, reason),
options.nodeDesktopCarrier?.stop(environmentId, ownerEpoch),
options.nodePortalCarrier?.stop(environmentId, ownerEpoch),
options.closeWorkerPortals?.(environmentId, ownerEpoch),
@@ -239,6 +244,7 @@ export function createWorkerEnvironmentService(options: WorkerEnvironmentService
const next = store.transition({
environmentId: record.environmentId,
from: record.state,
expectedOwnerEpoch: record.ownerEpoch,
to,
patch,
});
@@ -43,6 +43,9 @@ export type WorkerTunnelRequest = {
ownerEpoch: number;
};
/** Provider teardown fences local work first; only its confirmed result releases physical ownership. */
export type WorkerTunnelStopReason = "provider-destroying" | "provider-destroyed";
export type WorkerWorkspaceCommand = {
argv: readonly string[];
transportRetry: "idempotent" | "never";
@@ -35,7 +35,7 @@ import {
toWorkerTranscriptMessage,
type WorkerProviderReplayUnavailable,
} from "../../worker/transcript-message.js";
import { parseWorkerAdmissionDeadlineResult } from "../../worker/worker-connection-contract.js";
import { parseWorkerRuntimeResult } from "../../worker/worker-process-protocol.js";
import type { WorkerRuntimeResult } from "../../worker/worker.runtime.js";
import {
measureAgentRuntimeIdentityTokenBytes,
@@ -248,47 +248,14 @@ export function parseRuntimeResult(stdout: string): StartedWorkerRuntimeResult {
} catch (error) {
throw new Error("Worker process returned invalid output", { cause: error });
}
const admissionFailure = parseWorkerAdmissionDeadlineResult(value);
if (admissionFailure) {
throw new Error(admissionFailure.errorText);
}
if (!value || typeof value !== "object" || Array.isArray(value)) {
const result = parseWorkerRuntimeResult(value);
if (!result) {
throw new Error("Worker process returned invalid output");
}
const result = value as Record<string, unknown>;
if (
result.status === "failed" &&
result.reason === "turn-failed" &&
(result.transcriptLeafId === null || typeof result.transcriptLeafId === "string") &&
typeof result.transcriptNextSeq === "number" &&
Number.isSafeInteger(result.transcriptNextSeq) &&
result.transcriptNextSeq >= 1 &&
Object.keys(result).every((key) =>
["status", "reason", "transcriptLeafId", "transcriptNextSeq"].includes(key),
)
) {
return result as StartedWorkerRuntimeResult;
if (result.status === "not-started") {
throw new Error(result.errorText);
}
if (
result.status === "completed" &&
(result.transcriptLeafId === null || typeof result.transcriptLeafId === "string") &&
typeof result.transcriptNextSeq === "number" &&
Number.isSafeInteger(result.transcriptNextSeq) &&
result.transcriptNextSeq >= 1 &&
Object.keys(result).every((key) =>
["status", "transcriptLeafId", "transcriptNextSeq"].includes(key),
)
) {
return result as StartedWorkerRuntimeResult;
}
if (
result.status === "fenced" &&
(result.reason === "credential-replaced" || result.reason === "owner-epoch-mismatch") &&
Object.keys(result).every((key) => ["status", "reason"].includes(key))
) {
return result as StartedWorkerRuntimeResult;
}
throw new Error("Worker process returned invalid output");
return result;
}
export function assistantText(message: AgentMessage): string {
+2
View File
@@ -23,6 +23,7 @@ export const NODE_WORKER_BUNDLE_INSTALL_COMMAND = "worker.bundle.install.v1";
export const NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND = "worker.launch.v1";
export const NODE_WORKER_SUPERVISOR_STATUS_COMMAND = "worker.status.v1";
export const NODE_WORKER_SUPERVISOR_CANCEL_COMMAND = "worker.cancel.v1";
export const NODE_WORKER_ENVIRONMENT_STOP_COMMAND = "worker.environment.stop.v1";
export const NODE_WORKER_WORKSPACE_EXEC_COMMAND = "worker.workspace.exec.v1";
export const NODE_WORKER_WORKSPACE_RETAIN_COMMAND = "worker.workspace.retain.v1";
export const NODE_WORKER_DESKTOP_STREAM_COMMAND = "worker.desktop.stream.v1";
@@ -34,6 +35,7 @@ export const NODE_WORKER_PRIVATE_COMMANDS = [
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
NODE_WORKER_SUPERVISOR_STATUS_COMMAND,
NODE_WORKER_SUPERVISOR_CANCEL_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_WORKSPACE_EXEC_COMMAND,
NODE_WORKER_WORKSPACE_RETAIN_COMMAND,
NODE_WORKER_DESKTOP_STREAM_COMMAND,
+10 -2
View File
@@ -13,6 +13,7 @@ const RETIRED_NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURES = [
export const NODE_WORKER_BUNDLE_RETENTION_VERSION = 1;
export const NODE_WORKER_BUNDLE_STATUS_VERSION = 1;
export const NODE_WORKER_PORTAL_STREAM_VERSION = 1;
export const NODE_WORKER_ENVIRONMENT_SESSION_VERSION = 1;
export const NODE_WORKER_CAPACITY_MAX = 1_024;
export const NODE_RUNNER_UPDATE_REQUIRED_ISSUE = {
@@ -37,6 +38,7 @@ export type NodeWorkerHostDeclaration =
bundleRetention?: typeof NODE_WORKER_BUNDLE_RETENTION_VERSION;
bundleStatus?: typeof NODE_WORKER_BUNDLE_STATUS_VERSION;
portalStream?: typeof NODE_WORKER_PORTAL_STREAM_VERSION;
environmentSession?: typeof NODE_WORKER_ENVIRONMENT_SESSION_VERSION;
};
export type NodeRunnerInventoryDeclaration =
@@ -85,7 +87,7 @@ function parseWorkerHostDeclaration(value: unknown): NodeWorkerHostDeclaration |
if (
!capacity ||
keys.length < 2 ||
keys.length > 6 ||
keys.length > 7 ||
!keys.includes("enabled") ||
!keys.includes("capacity") ||
keys.some(
@@ -95,7 +97,8 @@ function parseWorkerHostDeclaration(value: unknown): NodeWorkerHostDeclaration |
key !== "bundlePrewarm" &&
key !== "bundleRetention" &&
key !== "bundleStatus" &&
key !== "portalStream",
key !== "portalStream" &&
key !== "environmentSession",
) ||
(value.bundlePrewarm !== undefined && value.bundlePrewarm !== WORKER_BUNDLE_PREWARM_VERSION) ||
(value.bundleRetention !== undefined &&
@@ -104,6 +107,8 @@ function parseWorkerHostDeclaration(value: unknown): NodeWorkerHostDeclaration |
value.bundleStatus !== NODE_WORKER_BUNDLE_STATUS_VERSION) ||
(value.portalStream !== undefined &&
value.portalStream !== NODE_WORKER_PORTAL_STREAM_VERSION) ||
(value.environmentSession !== undefined &&
value.environmentSession !== NODE_WORKER_ENVIRONMENT_SESSION_VERSION) ||
(value.bundleStatus !== undefined && value.bundleRetention === undefined)
) {
return null;
@@ -123,6 +128,9 @@ function parseWorkerHostDeclaration(value: unknown): NodeWorkerHostDeclaration |
...(value.portalStream === NODE_WORKER_PORTAL_STREAM_VERSION
? { portalStream: NODE_WORKER_PORTAL_STREAM_VERSION }
: {}),
...(value.environmentSession === NODE_WORKER_ENVIRONMENT_SESSION_VERSION
? { environmentSession: NODE_WORKER_ENVIRONMENT_SESSION_VERSION }
: {}),
};
}
@@ -8,6 +8,7 @@ import {
NODE_WORKER_CAPACITY_EXHAUSTED_ERROR_CODE,
NODE_WORKER_DESKTOP_LAUNCH_COMMAND,
NODE_WORKER_DESKTOP_STREAM_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_PORTAL_STREAM_COMMAND,
NODE_WORKER_SUPERVISOR_CANCEL_COMMAND,
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
@@ -87,6 +88,7 @@ function supervisorWith(receipt: NodeWorkerLaunchReceipt): NodeWorkerSupervisorC
status: vi.fn(async () => receipt),
retainWorkspaces: vi.fn(async () => ({ applied: true, deleted: 0, hasMore: false })),
cancel: vi.fn(async () => receipt),
stopEnvironment: vi.fn(async () => {}),
};
}
@@ -95,6 +97,7 @@ type SupervisorMocks = {
status: ReturnType<typeof vi.fn>;
retainWorkspaces: ReturnType<typeof vi.fn>;
cancel: ReturnType<typeof vi.fn>;
stopEnvironment: ReturnType<typeof vi.fn>;
};
function supervisorMocks(supervisor: NodeWorkerSupervisorControl): SupervisorMocks {
@@ -146,6 +149,32 @@ async function invokePrivate(params: {
}
describe("node-host worker supervisor commands", () => {
it("settles environment teardown only after the exact owner has stopped", async () => {
const receipt = fullReceipt();
const supervisor = supervisorWith(receipt);
const owner = {
gatewayNamespace: receipt.gatewayNamespace,
environmentId: receipt.environmentId,
sessionId: receipt.sessionId,
ownerEpoch: receipt.ownerEpoch,
};
const { result } = await invokePrivate({
command: NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
paramsJSON: JSON.stringify(owner),
supervisor,
});
expect(supervisorMocks(supervisor).stopEnvironment).toHaveBeenCalledExactlyOnceWith(owner);
expect(result).toMatchObject({ ok: true, payloadJSON: "null" });
supervisorMocks(supervisor).stopEnvironment.mockRejectedValueOnce(new Error("still running"));
const failed = await invokePrivate({
command: NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
paramsJSON: JSON.stringify(owner),
supervisor,
});
expect(failed.result).toMatchObject({ ok: false, error: { code: "UNAVAILABLE" } });
});
it.each([
{ command: NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND, method: "launch" as const },
{ command: NODE_WORKER_SUPERVISOR_STATUS_COMMAND, method: "status" as const },
@@ -212,6 +241,7 @@ describe("node-host worker supervisor commands", () => {
NODE_WORKER_DESKTOP_STREAM_COMMAND,
NODE_WORKER_DESKTOP_LAUNCH_COMMAND,
NODE_WORKER_PORTAL_STREAM_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
])("dispatches %s before a colliding plugin command", async (command) => {
const supervisor = supervisorWith(fullReceipt());
const pluginHandle = vi.fn(async () => '{"plugin":true}');
@@ -322,6 +322,7 @@ export async function createNodeWorkerContainer(
CONTAINER_NODE_EXECUTABLE,
params.image ?? DEFAULT_NODE_WORKER_CONTAINER_IMAGE,
params.bundleEntry,
"--internal-worker-session",
);
const current = await resolveContainerEngineTarget(engine, { pinned: true });
if (current.target !== engine.target) {
+82 -25
View File
@@ -3,12 +3,16 @@ import {
createCapturedOutputBuffers,
finalizeCapturedOutput,
} from "../process/exec-output.js";
import {
parseWorkerProcessResult,
type WorkerProcessResult,
} from "../worker/worker-process-protocol.js";
import type { NodeWorkerTerminalState } from "./node-worker-launch-store.js";
import type { NodeWorkerChildAdapter } from "./node-worker-launch-transport.js";
import {
NODE_WORKER_STDERR_MAX_BYTES,
NODE_WORKER_STDOUT_MAX_BYTES,
parseNodeWorkerSuccessfulResult,
parseNodeWorkerOutputJson,
sanitizeNodeWorkerDiagnostic,
type NodeWorkerCredentialScrubber,
} from "./node-worker-output.js";
@@ -27,24 +31,81 @@ type NodeWorkerChildObservation = {
stopState?: Extract<NodeWorkerTerminalState, "cancelled" | "interrupted">;
};
/** Decode one bounded, credential-scrubbed worker result after durable admission. */
/** Turn results settle independently; process exit alone releases the physical owner. */
export async function observeNodeWorkerChildOutput(
active: NodeWorkerChildObservation,
onResult: (frame: WorkerProcessResult) => void,
currentTurnId: () => string | undefined,
): Promise<NodeWorkerTerminalOutcome> {
const stdout = createCapturedOutputBuffers();
const stderr = createCapturedOutputBuffers();
active.adapter.onStdout((chunk) =>
appendCapturedOutput(stdout, chunk, NODE_WORKER_STDOUT_MAX_BYTES, "head"),
);
let stdout = "";
let lastResult: string | undefined;
let outputError: unknown;
let journaled = false;
const drain = () => {
if (!journaled || outputError) {
return;
}
try {
let newline: number;
while ((newline = stdout.indexOf("\n")) >= 0) {
const line = stdout.slice(0, newline);
stdout = stdout.slice(newline + 1);
if (Buffer.byteLength(line, "utf8") > NODE_WORKER_STDOUT_MAX_BYTES) {
throw new Error(`worker stdout exceeded ${NODE_WORKER_STDOUT_MAX_BYTES} bytes`);
}
const frame = parseWorkerProcessResult(
JSON.parse(parseNodeWorkerOutputJson(line, active.scrubber.scrub)),
);
if (!frame) {
throw new Error("worker returned an invalid turn result");
}
onResult(frame);
lastResult = JSON.stringify(frame.result);
}
if (Buffer.byteLength(stdout, "utf8") > NODE_WORKER_STDOUT_MAX_BYTES) {
throw new Error(`worker stdout exceeded ${NODE_WORKER_STDOUT_MAX_BYTES} bytes`);
}
} catch (error) {
outputError = error;
stdout = "";
active.adapter.kill("SIGKILL");
}
};
let stderr = createCapturedOutputBuffers();
let diagnosticTurnId = currentTurnId();
const currentStderr = () => {
if (diagnosticTurnId !== currentTurnId()) {
// Old raw diagnostics must not outlive the credential scrubber that owns them.
stderr = createCapturedOutputBuffers();
diagnosticTurnId = currentTurnId();
}
return stderr;
};
active.adapter.onStdout((chunk) => {
if (outputError) {
return;
}
stdout += chunk;
if (!journaled && Buffer.byteLength(stdout, "utf8") > NODE_WORKER_STDOUT_MAX_BYTES) {
outputError = new Error(`worker stdout exceeded ${NODE_WORKER_STDOUT_MAX_BYTES} bytes`);
stdout = "";
active.adapter.kill("SIGKILL");
}
drain();
});
active.adapter.onStderr((chunk) =>
appendCapturedOutput(
stderr,
currentStderr(),
chunk,
NODE_WORKER_STDERR_MAX_BYTES + active.scrubber.maxRepresentationBytes,
"tail",
),
);
try {
void active.journalReady.then(() => {
journaled = true;
drain();
});
const exit = await active.adapter.wait();
await active.journalReady;
if (active.stopState) {
@@ -57,24 +118,20 @@ export async function observeNodeWorkerChildOutput(
: "node worker launch interrupted during node-host shutdown"),
});
}
if (exit.code === 0 && exit.signal === null) {
try {
return Object.freeze({
state: "completed",
resultJson: parseNodeWorkerSuccessfulResult(stdout, active.scrubber.scrub),
});
} catch (error) {
return Object.freeze({
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
error,
"invalid worker result",
active.scrubber.scrub,
),
});
}
if (outputError || stdout.length > 0 || (exit.code === 0 && !lastResult)) {
return Object.freeze({
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
outputError ?? new Error("worker exited without a complete turn result"),
"invalid worker result",
active.scrubber.scrub,
),
});
}
const detail = finalizeCapturedOutput(stderr, "tail", true).toString("utf8");
if (exit.code === 0 && exit.signal === null && lastResult) {
return Object.freeze({ state: "completed", resultJson: lastResult });
}
const detail = finalizeCapturedOutput(currentStderr(), "tail", true).toString("utf8");
const exitLabel = exit.signal ? `signal ${exit.signal}` : `exit code ${String(exit.code)}`;
return Object.freeze({
state: "failed",
+122
View File
@@ -0,0 +1,122 @@
import { isRecord } from "@openclaw/normalization-core/record-coerce";
import type { Selectable } from "kysely";
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
import type { NodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
type NodeWorkerLaunchState =
| "pending"
| "running"
| "completed"
| "failed"
| "interrupted"
| "cancelled";
export type NodeWorkerTerminalState = Exclude<NodeWorkerLaunchState, "pending" | "running">;
export type NodeWorkerContainerIdentity = {
engine: "docker" | "podman";
containerId: string;
engineTarget: string;
};
export type NodeWorkerLaunchRow = Selectable<OpenClawStateDatabase["node_worker_launches"]> & {
container_json?: string | null;
};
export type NodeWorkerLaunchReceipt = {
launchId: string;
planHash: string;
gatewayNamespace: string;
environmentId: string;
sessionId: string;
ownerEpoch: number;
placementGeneration: number;
runId: string;
state: NodeWorkerLaunchState;
supervisor: NodeWorkerProcessIdentity;
worker: NodeWorkerProcessIdentity | null;
container?: NodeWorkerContainerIdentity;
resultJson: string | null;
errorText: string | null;
completedAtMs: number | null;
createdAtMs: number;
updatedAtMs: number;
};
export function isNodeWorkerTerminalState(value: string): value is NodeWorkerTerminalState {
return (
value === "completed" || value === "failed" || value === "interrupted" || value === "cancelled"
);
}
export function validateNodeWorkerContainerIdentity(identity: NodeWorkerContainerIdentity): void {
if (identity.engine !== "docker" && identity.engine !== "podman") {
throw new Error("node worker container engine must be docker or podman");
}
if (!/^[a-f0-9]{64}$/u.test(identity.containerId)) {
throw new Error(
"node worker container id must contain exactly 64 lowercase hexadecimal digits",
);
}
if (!/^[a-f0-9]{64}$/u.test(identity.engineTarget)) {
throw new Error(
"node worker container engine target must contain exactly 64 lowercase hexadecimal digits",
);
}
}
function containerIdentity(value: string | null | undefined): NodeWorkerContainerIdentity | null {
if (value == null) {
return null;
}
let parsed: unknown;
try {
parsed = JSON.parse(value) as unknown;
} catch {
throw new Error("invalid node worker container identity");
}
if (
!isRecord(parsed) ||
Object.keys(parsed).length !== 3 ||
(parsed.engine !== "docker" && parsed.engine !== "podman") ||
typeof parsed.containerId !== "string" ||
typeof parsed.engineTarget !== "string"
) {
throw new Error("invalid node worker container identity");
}
const identity: NodeWorkerContainerIdentity = {
engine: parsed.engine,
containerId: parsed.containerId,
engineTarget: parsed.engineTarget,
};
validateNodeWorkerContainerIdentity(identity);
return identity;
}
export function nodeWorkerLaunchReceiptFromRow(row: NodeWorkerLaunchRow): NodeWorkerLaunchReceipt {
if (row.state !== "pending" && row.state !== "running" && !isNodeWorkerTerminalState(row.state)) {
throw new Error(`invalid node worker launch state ${row.state}`);
}
const container = containerIdentity(row.container_json);
return {
launchId: row.launch_id,
planHash: row.plan_hash,
gatewayNamespace: row.gateway_namespace,
environmentId: row.environment_id,
sessionId: row.session_id,
ownerEpoch: row.owner_epoch,
placementGeneration: row.placement_generation,
runId: row.run_id,
state: row.state,
supervisor: { pid: row.supervisor_pid, startTime: row.supervisor_start_time },
worker:
row.worker_pid === null || row.worker_start_time === null
? null
: { pid: row.worker_pid, startTime: row.worker_start_time },
...(container ? { container } : {}),
resultJson: row.result_json,
errorText: row.error_text,
completedAtMs: row.completed_at_ms,
createdAtMs: row.created_at_ms,
updatedAtMs: row.updated_at_ms,
};
}
+99 -146
View File
@@ -1,6 +1,4 @@
import type { DatabaseSync } from "node:sqlite";
import { isRecord } from "@openclaw/normalization-core/record-coerce";
import type { Selectable } from "kysely";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
@@ -14,53 +12,30 @@ import {
} from "../state/openclaw-state-db.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
import type { NodeWorkerSupervisorIdentity } from "../worker/node-supervisor-protocol.js";
import {
isNodeWorkerTerminalState,
nodeWorkerLaunchReceiptFromRow,
validateNodeWorkerContainerIdentity,
type NodeWorkerContainerIdentity,
type NodeWorkerLaunchReceipt,
type NodeWorkerLaunchRow,
type NodeWorkerTerminalState,
} from "./node-worker-launch-receipt.js";
import {
inspectNodeWorkerProcessIdentity,
type NodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
type NodeWorkerLaunchState =
| "pending"
| "running"
| "completed"
| "failed"
| "interrupted"
| "cancelled";
export type NodeWorkerTerminalState = Exclude<NodeWorkerLaunchState, "pending" | "running">;
export type NodeWorkerContainerIdentity = {
engine: "docker" | "podman";
containerId: string;
engineTarget: string;
};
export type {
NodeWorkerContainerIdentity,
NodeWorkerLaunchReceipt,
NodeWorkerTerminalState,
} from "./node-worker-launch-receipt.js";
type NodeWorkerLaunchDatabase = Pick<
OpenClawStateDatabase,
"node_worker_launch_containers" | "node_worker_launches"
"node_worker_launch_containers" | "node_worker_launches" | "node_worker_turns"
>;
type NodeWorkerLaunchRow = Selectable<NodeWorkerLaunchDatabase["node_worker_launches"]> & {
container_json?: string | null;
};
export type NodeWorkerLaunchReceipt = {
launchId: string;
planHash: string;
gatewayNamespace: string;
environmentId: string;
sessionId: string;
ownerEpoch: number;
placementGeneration: number;
runId: string;
state: NodeWorkerLaunchState;
supervisor: NodeWorkerProcessIdentity;
worker: NodeWorkerProcessIdentity | null;
container?: NodeWorkerContainerIdentity;
resultJson: string | null;
errorText: string | null;
completedAtMs: number | null;
createdAtMs: number;
updatedAtMs: number;
};
export type NodeWorkerLaunchClaim = Pick<
NodeWorkerLaunchReceipt,
@@ -91,12 +66,6 @@ const NODE_WORKER_LAUNCH_CONTAINER_SCHEMA_START =
"CREATE TABLE IF NOT EXISTS node_worker_launch_containers (";
const NODE_WORKER_LAUNCH_CONTAINER_SCHEMA_END = "\n) STRICT;";
const initializedDatabases = new WeakSet<DatabaseSync>();
const TERMINAL_STATES: ReadonlySet<string> = new Set([
"completed",
"failed",
"interrupted",
"cancelled",
]);
const TERMINAL_RECEIPT_RETENTION_MS = 24 * 60 * 60 * 1_000;
const TERMINAL_PRUNE_BATCH_LIMIT = 256;
@@ -207,69 +176,51 @@ function pruneTerminalRows(params: {
return Number(result.numAffectedRows ?? 0n);
}
function processIdentity(pid: number, startTime: number): NodeWorkerProcessIdentity {
return { pid, startTime };
/** Read the authoritative physical owner within an already-open journal transaction. */
export function readNodeWorkerLaunchReceipt(
database: DatabaseSync,
launchId: string,
): NodeWorkerLaunchReceipt | undefined {
if (!tableExists(database, "node_worker_launches")) {
return undefined;
}
const row = readRow(database, launchId);
return row ? nodeWorkerLaunchReceiptFromRow(row) : undefined;
}
function containerIdentity(value: string | null | undefined): NodeWorkerContainerIdentity | null {
if (value == null) {
return null;
}
let parsed: unknown;
try {
parsed = JSON.parse(value) as unknown;
} catch {
throw new Error("invalid node worker container identity");
}
/** Physical extinction closes unfinished turns, never a result already recorded by the worker. */
export function settleNodeWorkerActiveTurns(
database: DatabaseSync,
owner: NodeWorkerLaunchReceipt,
): void {
if (
!isRecord(parsed) ||
Object.keys(parsed).length !== 3 ||
(parsed.engine !== "docker" && parsed.engine !== "podman") ||
typeof parsed.containerId !== "string" ||
typeof parsed.engineTarget !== "string"
owner.state === "pending" ||
owner.state === "running" ||
!tableExists(database, "node_worker_turns")
) {
throw new Error("invalid node worker container identity");
return;
}
const identity: NodeWorkerContainerIdentity = {
engine: parsed.engine,
containerId: parsed.containerId,
engineTarget: parsed.engineTarget,
};
validateContainerIdentity(identity);
return identity;
}
function receiptFromRow(row: NodeWorkerLaunchRow): NodeWorkerLaunchReceipt {
if (!isNodeWorkerLaunchState(row.state)) {
throw new Error(`invalid node worker launch state ${row.state}`);
}
const container = containerIdentity(row.container_json);
return {
launchId: row.launch_id,
planHash: row.plan_hash,
gatewayNamespace: row.gateway_namespace,
environmentId: row.environment_id,
sessionId: row.session_id,
ownerEpoch: row.owner_epoch,
placementGeneration: row.placement_generation,
runId: row.run_id,
state: row.state,
supervisor: processIdentity(row.supervisor_pid, row.supervisor_start_time),
worker:
row.worker_pid === null || row.worker_start_time === null
? null
: processIdentity(row.worker_pid, row.worker_start_time),
...(container ? { container } : {}),
resultJson: row.result_json,
errorText: row.error_text,
completedAtMs: row.completed_at_ms,
createdAtMs: row.created_at_ms,
updatedAtMs: row.updated_at_ms,
};
}
function isNodeWorkerLaunchState(value: string): value is NodeWorkerLaunchState {
return value === "pending" || value === "running" || TERMINAL_STATES.has(value);
executeSqliteQuerySync(
database,
query(database)
.updateTable("node_worker_turns")
.set((expression) => {
const completedAt = expression.fn<number>("max", [
"created_at_ms",
"updated_at_ms",
expression.val(owner.updatedAtMs),
]);
return {
state: owner.state === "completed" ? "interrupted" : owner.state,
result_json: null,
error_text: owner.errorText ?? "node worker stopped before its turn completed",
completed_at_ms: completedAt,
updated_at_ms: completedAt,
};
})
.where("owner_launch_id", "=", owner.launchId)
.where("state", "=", "running"),
);
}
function validateIdentifier(value: string, label: string): void {
@@ -308,22 +259,6 @@ function validateProcessIdentity(identity: NodeWorkerProcessIdentity): void {
}
}
function validateContainerIdentity(identity: NodeWorkerContainerIdentity): void {
if (identity.engine !== "docker" && identity.engine !== "podman") {
throw new Error("node worker container engine must be docker or podman");
}
if (!/^[a-f0-9]{64}$/u.test(identity.containerId)) {
throw new Error(
"node worker container id must contain exactly 64 lowercase hexadecimal digits",
);
}
if (!/^[a-f0-9]{64}$/u.test(identity.engineTarget)) {
throw new Error(
"node worker container engine target must contain exactly 64 lowercase hexadecimal digits",
);
}
}
function requireMatchingRow(
database: DatabaseSync,
launchId: string,
@@ -427,9 +362,10 @@ export class NodeWorkerLaunchStore {
throw new Error(`node worker launch ${claim.launchId} was replayed with a different plan`);
}
const observedSupervisorState = observed
? inspectNodeWorkerProcessIdentity(
processIdentity(observed.supervisor_pid, observed.supervisor_start_time),
)
? inspectNodeWorkerProcessIdentity({
pid: observed.supervisor_pid,
startTime: observed.supervisor_start_time,
})
: undefined;
return this.write("node-worker-launch.claim", (database) => {
@@ -477,7 +413,9 @@ export class NodeWorkerLaunchStore {
);
return finalize({
action: "start",
receipt: receiptFromRow(requireMatchingRow(database, claim.launchId, claim.planHash)),
receipt: nodeWorkerLaunchReceiptFromRow(
requireMatchingRow(database, claim.launchId, claim.planHash),
),
nonterminalCount: readNonterminalCount(database),
});
}
@@ -513,7 +451,7 @@ export class NodeWorkerLaunchStore {
current = requireMatchingRow(database, claim.launchId, claim.planHash);
return finalize({
action: rowHasSupervisor(current, supervisor) ? "start" : "replay",
receipt: receiptFromRow(current),
receipt: nodeWorkerLaunchReceiptFromRow(current),
nonterminalCount: readNonterminalCount(database),
});
}
@@ -525,13 +463,13 @@ export class NodeWorkerLaunchStore {
) {
return finalize({
action: "recover",
receipt: receiptFromRow(current),
receipt: nodeWorkerLaunchReceiptFromRow(current),
nonterminalCount: readNonterminalCount(database),
});
}
return finalize({
action: "replay",
receipt: receiptFromRow(current),
receipt: nodeWorkerLaunchReceiptFromRow(current),
nonterminalCount: readNonterminalCount(database),
});
});
@@ -539,7 +477,7 @@ export class NodeWorkerLaunchStore {
listNonterminal(): NodeWorkerLaunchReceipt[] {
return this.write("node-worker-launch.list-nonterminal", (database) =>
readNonterminalRows(database).map(receiptFromRow),
readNonterminalRows(database).map(nodeWorkerLaunchReceiptFromRow),
);
}
@@ -565,7 +503,7 @@ export class NodeWorkerLaunchStore {
validateIdentifier(launchId, "node worker launch id");
return this.write("node-worker-launch.get", (database) => {
const row = readRow(database, launchId);
return row ? receiptFromRow(row) : undefined;
return row ? nodeWorkerLaunchReceiptFromRow(row) : undefined;
});
}
@@ -574,7 +512,9 @@ export class NodeWorkerLaunchStore {
validatePlanHash(expected.planHash);
return this.write("node-worker-launch.get-matching", (database) => {
const row = readRow(database, expected.launchId);
return row && rowMatchesImmutableIdentity(row, expected) ? receiptFromRow(row) : undefined;
return row && rowMatchesImmutableIdentity(row, expected)
? nodeWorkerLaunchReceiptFromRow(row)
: undefined;
});
}
@@ -595,11 +535,13 @@ export class NodeWorkerLaunchStore {
if (!current || !rowMatchesImmutableIdentity(current, params.expected)) {
return undefined;
}
if (TERMINAL_STATES.has(current.state)) {
return receiptFromRow(current);
if (isNodeWorkerTerminalState(current.state)) {
const receipt = nodeWorkerLaunchReceiptFromRow(current);
settleNodeWorkerActiveTurns(database, receipt);
return receipt;
}
if (!rowHasSupervisor(current, params.supervisor) || !rowHasWorker(current, params.worker)) {
return receiptFromRow(current);
return nodeWorkerLaunchReceiptFromRow(current);
}
const completedAtMs = Math.max(nowMs, current.created_at_ms, current.updated_at_ms);
let update = query(database)
@@ -628,9 +570,12 @@ export class NodeWorkerLaunchStore {
: update.where("worker_pid", "is", null).where("worker_start_time", "is", null);
executeSqliteQuerySync(database, update);
const settled = readRow(database, params.expected.launchId);
return settled && rowMatchesImmutableIdentity(settled, params.expected)
? receiptFromRow(settled)
: undefined;
if (!settled || !rowMatchesImmutableIdentity(settled, params.expected)) {
return undefined;
}
const receipt = nodeWorkerLaunchReceiptFromRow(settled);
settleNodeWorkerActiveTurns(database, receipt);
return receipt;
});
}
@@ -647,18 +592,18 @@ export class NodeWorkerLaunchStore {
validateProcessIdentity(params.supervisor);
validateProcessIdentity(params.worker);
if (params.container) {
validateContainerIdentity(params.container);
validateNodeWorkerContainerIdentity(params.container);
}
return this.write("node-worker-launch.mark-running", (database) => {
const current = requireMatchingRow(database, params.launchId, params.planHash);
if (TERMINAL_STATES.has(current.state)) {
return receiptFromRow(current);
if (isNodeWorkerTerminalState(current.state)) {
return nodeWorkerLaunchReceiptFromRow(current);
}
if (current.state === "running") {
return receiptFromRow(current);
return nodeWorkerLaunchReceiptFromRow(current);
}
if (!rowHasSupervisor(current, params.supervisor) || !rowHasWorker(current, null)) {
return receiptFromRow(current);
return nodeWorkerLaunchReceiptFromRow(current);
}
if (params.container) {
ensureNodeWorkerLaunchSchema(database, "container");
@@ -695,7 +640,9 @@ export class NodeWorkerLaunchStore {
.where("worker_pid", "is", null)
.where("worker_start_time", "is", null),
);
return receiptFromRow(requireMatchingRow(database, params.launchId, params.planHash));
return nodeWorkerLaunchReceiptFromRow(
requireMatchingRow(database, params.launchId, params.planHash),
);
});
}
@@ -717,11 +664,13 @@ export class NodeWorkerLaunchStore {
}
return this.write("node-worker-launch.finish", (database) => {
const current = requireMatchingRow(database, params.launchId, params.planHash);
if (TERMINAL_STATES.has(current.state)) {
return receiptFromRow(current);
if (isNodeWorkerTerminalState(current.state)) {
const receipt = nodeWorkerLaunchReceiptFromRow(current);
settleNodeWorkerActiveTurns(database, receipt);
return receipt;
}
if (!rowHasSupervisor(current, params.supervisor) || !rowHasWorker(current, params.worker)) {
return receiptFromRow(current);
return nodeWorkerLaunchReceiptFromRow(current);
}
const completedAtMs = Math.max(nowMs, current.created_at_ms, current.updated_at_ms);
let update = query(database)
@@ -744,7 +693,11 @@ export class NodeWorkerLaunchStore {
.where("worker_start_time", "=", params.worker.startTime)
: update.where("worker_pid", "is", null).where("worker_start_time", "is", null);
executeSqliteQuerySync(database, update);
return receiptFromRow(requireMatchingRow(database, params.launchId, params.planHash));
const receipt = nodeWorkerLaunchReceiptFromRow(
requireMatchingRow(database, params.launchId, params.planHash),
);
settleNodeWorkerActiveTurns(database, receipt);
return receipt;
});
}
}
+25 -8
View File
@@ -2,6 +2,7 @@ import { isGatewayLoopbackHost } from "../../packages/gateway-client/src/websock
import { createChildAdapter } from "../process/supervisor/adapters/child.js";
import type { WorkerLaunchDescriptor } from "../worker/launch-descriptor.js";
import { parseNodeWorkerConnectionFailureMessage } from "../worker/node-supervisor-protocol.js";
import type { WorkerProcessInput } from "../worker/worker-process-protocol.js";
import {
buildNodeWorkerContainerStartArgv,
createNodeWorkerContainer,
@@ -57,7 +58,7 @@ export async function prepareNodeWorkerLaunchTransport(
return {
kind: "started",
adapter: await createChildAdapter({
argv: [process.execPath, entry, "--internal-worker-ipc"],
argv: [process.execPath, entry, "--internal-worker-ipc", "--internal-worker-session"],
env: options.workerEnv,
exactEnv: true,
ownedWorker: true,
@@ -74,7 +75,7 @@ export async function prepareNodeWorkerLaunchTransport(
)
: undefined;
},
input: JSON.stringify(options.descriptor),
stdinMode: "pipe-open",
}),
};
}
@@ -132,27 +133,43 @@ export async function prepareNodeWorkerLaunchTransport(
}
}
/** Plain stdio workers cannot run until their journaled descriptor reaches EOF. */
/** Both transports admit turns only after the physical owner has been journaled. */
export async function startNodeWorkerLaunchTransport(params: {
adapter: NodeWorkerChildAdapter;
descriptor: WorkerLaunchDescriptor;
container?: NodeWorkerContainerIdentity;
isCurrent: () => boolean;
}): Promise<void> {
if (!params.isCurrent()) {
throw new Error("node worker admission closed before startup");
}
if (!params.container) {
await params.adapter.openStartGate?.();
return;
}
const stdin = params.adapter.stdin;
if (!params.isCurrent()) {
throw new Error("node worker admission closed before descriptor dispatch");
}
await sendNodeWorkerInput(params.adapter, {
type: "turn",
turnId: params.descriptor.assignment.turnId,
descriptor: params.descriptor,
});
}
export async function sendNodeWorkerInput(
adapter: NodeWorkerChildAdapter,
message: WorkerProcessInput,
): Promise<void> {
const stdin = adapter.stdin;
if (!stdin) {
throw new Error("node worker container launch did not provide a writable stdin pipe");
throw new Error("node worker did not provide a writable stdin pipe");
}
await new Promise<void>((resolve, reject) => {
stdin.write(JSON.stringify(params.descriptor), (error) => {
stdin.write(`${JSON.stringify(message)}\n`, (error) => {
if (error) {
reject(error);
return;
}
stdin.end();
resolve();
});
});
+228
View File
@@ -0,0 +1,228 @@
import { registerSecretValueForRedaction } from "../logging/secret-redaction-registry.js";
import type { WorkerLaunchDescriptor } from "../worker/launch-descriptor.js";
import type { NodeWorkerCapacity } from "./node-worker-capacity.js";
import type { NodeWorkerContainerEngine } from "./node-worker-container-engine.js";
import type { NodeWorkerContainerLifecycle } from "./node-worker-container-lifecycle.js";
import type {
NodeWorkerLaunchClaim,
NodeWorkerLaunchReceipt,
NodeWorkerLaunchStore,
NodeWorkerContainerIdentity,
} from "./node-worker-launch-store.js";
import {
prepareNodeWorkerLaunchTransport,
startNodeWorkerLaunchTransport,
type NodeWorkerChildAdapter,
} from "./node-worker-launch-transport.js";
import {
createNodeWorkerCredentialScrubber,
sanitizeNodeWorkerDiagnostic,
} from "./node-worker-output.js";
import {
requireNodeWorkerProcessIdentity,
type NodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
import type { NodeWorkerLaunchInput } from "./node-worker-supervisor-contract.js";
import {
createNodeWorkerActiveTurn,
createNodeWorkerJournalGate,
nodeWorkerEnvironmentBinding,
type NodeWorkerActiveOwnership,
type NodeWorkerRunningChild,
type NodeWorkerStopState,
} from "./node-worker-supervisor-ownership.js";
import { nodeWorkerDescriptorSecrets } from "./node-worker-turn-lifecycle.js";
import type { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
type NodeWorkerLaunchContext = {
bundleRoot: string;
workerEnv: NodeJS.ProcessEnv;
engineEnv: NodeJS.ProcessEnv;
store: NodeWorkerLaunchStore;
turns: NodeWorkerTurnStore;
capacity: NodeWorkerCapacity;
containerEngine?: NodeWorkerContainerEngine;
containerImage?: string;
containerLifecycle?: NodeWorkerContainerLifecycle;
requireContainerLifecycle: () => NodeWorkerContainerLifecycle;
active: Map<string, NodeWorkerActiveOwnership>;
isClosed: () => boolean;
observeChild: (active: NodeWorkerRunningChild) => Promise<void>;
stopChild: (active: NodeWorkerRunningChild, state: NodeWorkerStopState) => Promise<void>;
};
/** Starts one physical owner behind the durable journal gate, independent of turn reuse. */
export async function startNodeWorkerChild(
context: NodeWorkerLaunchContext,
params: {
input: NodeWorkerLaunchInput;
descriptor: WorkerLaunchDescriptor;
planHash: string;
supervisor: NodeWorkerProcessIdentity;
claim: NodeWorkerLaunchClaim;
signal?: AbortSignal;
},
): Promise<NodeWorkerLaunchReceipt> {
const sensitiveValues = nodeWorkerDescriptorSecrets(params.descriptor);
const scrubber = createNodeWorkerCredentialScrubber(sensitiveValues);
// Turn cancellation can beat the child's admission retry deadline. Retain the
// producer's latest cause so the durable terminal receipt does not become generic.
const connectionFailure: { errorText?: string } = {};
for (const value of sensitiveValues) {
registerSecretValueForRedaction(value);
}
let adapter: NodeWorkerChildAdapter;
let container: NodeWorkerContainerIdentity | undefined;
try {
const prepared = await prepareNodeWorkerLaunchTransport({
bundleRoot: context.bundleRoot,
workerEnv: context.workerEnv,
engineEnv: context.engineEnv,
input: params.input,
descriptor: params.descriptor,
connectionFailure,
scrubber,
store: context.store,
containerEngine: context.containerEngine,
containerLifecycle: context.containerLifecycle,
containerImage: context.containerImage,
});
if (prepared.kind === "terminal") {
return prepared.receipt;
}
adapter = prepared.adapter;
container = prepared.container;
} catch (error) {
return context.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(error, "node worker spawn failed", scrubber.scrub),
});
}
if (!adapter.pid) {
if (container) {
await context.requireContainerLifecycle().remove(container, params.input);
}
adapter.kill("SIGKILL");
adapter.dispose();
return context.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: "node worker spawn did not return a process id",
});
}
let worker: NodeWorkerProcessIdentity;
try {
worker = requireNodeWorkerProcessIdentity(adapter.pid);
} catch (error) {
if (container) {
await context.requireContainerLifecycle().remove(container, params.input);
}
adapter.kill("SIGKILL");
await adapter.wait().catch(() => undefined);
adapter.dispose();
return context.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
error,
"node worker process identity unavailable",
scrubber.scrub,
),
});
}
const { journalReady, releaseJournal } = createNodeWorkerJournalGate();
const active = {
state: "running",
binding: nodeWorkerEnvironmentBinding(params.input),
turn: createNodeWorkerActiveTurn(params.claim),
retiring: false,
adapter,
journalReady,
gatewayNamespace: params.input.gatewayNamespace,
launchId: params.input.launchId,
planHash: params.planHash,
releaseJournal,
scrubber,
connectionFailure,
supervisor: params.supervisor,
worker,
...(container ? { container } : {}),
} as NodeWorkerRunningChild; // SAFETY: done is assigned synchronously below; observation waits on journalReady before publishing state.
active.done = context.observeChild(active);
context.active.set(active.launchId, active);
void active.done.catch(() => undefined);
let running: NodeWorkerLaunchReceipt;
try {
running = context.store.markRunning({
launchId: active.launchId,
planHash: active.planHash,
supervisor: params.supervisor,
worker,
...(container ? { container } : {}),
});
} catch (error) {
active.releaseJournal();
if (container) {
await context.stopChild(active, "interrupted");
context.active.delete(active.launchId);
context.capacity.finish({
launchId: active.launchId,
planHash: active.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
error,
"node worker container identity could not be persisted",
scrubber.scrub,
),
});
} else {
await context.stopChild(active, "interrupted").catch(() => undefined);
}
throw error;
}
active.releaseJournal();
if (running.state === "cancelled" || running.state === "interrupted") {
await context.stopChild(active, running.state);
return context.store.get(active.launchId) ?? running;
}
if (running.state !== "running") {
if (container) {
await context.stopChild(active, "interrupted");
} else {
adapter.closeStartGate?.();
}
return running;
}
if (context.isClosed() || params.signal?.aborted || active.turn?.cancelled) {
await context.stopChild(active, context.isClosed() ? "interrupted" : "cancelled");
return context.store.get(active.launchId) ?? running;
}
try {
await startNodeWorkerLaunchTransport({
adapter,
descriptor: params.descriptor,
container,
isCurrent: () =>
context.active.get(active.launchId) === active &&
!context.isClosed() &&
!params.signal?.aborted &&
active.turn?.cancelled === false,
});
} catch {
await context.stopChild(active, active.turn?.cancelled ? "cancelled" : "interrupted");
return context.store.get(active.launchId) ?? running;
}
return context.turns.get(params.input.launchId) ?? running;
}
+2 -7
View File
@@ -1,7 +1,6 @@
import { formatErrorMessage } from "../infra/errors.js";
import { redactToolPayloadText } from "../logging/redact.js";
import { redactRegisteredSecretValues } from "../logging/secret-redaction-registry.js";
import { finalizeCapturedOutput, type CapturedOutputBuffers } from "../process/exec-output.js";
import { truncateUtf8Suffix } from "../utils/utf8-truncate.js";
export const NODE_WORKER_STDOUT_MAX_BYTES = 64 * 1024;
@@ -55,14 +54,10 @@ export function sanitizeNodeWorkerDiagnostic(
return truncateUtf8Suffix(oneLine || fallback, STDERR_MAX_BYTES);
}
export function parseNodeWorkerSuccessfulResult(
stdout: CapturedOutputBuffers,
export function parseNodeWorkerOutputJson(
raw: string,
scrubCredential: (text: string) => string,
): string {
if (stdout.truncatedBytes > 0) {
throw new Error(`worker stdout exceeded ${NODE_WORKER_STDOUT_MAX_BYTES} bytes`);
}
const raw = finalizeCapturedOutput(stdout, "head", true).toString("utf8").trim();
const redacted = redactLaunchText(raw, scrubCredential);
let parsed: unknown;
try {
@@ -5,6 +5,7 @@ import {
NODE_WORKER_CAPACITY_EXHAUSTED_ERROR_CODE,
NODE_WORKER_DESKTOP_LAUNCH_COMMAND,
NODE_WORKER_DESKTOP_STREAM_COMMAND,
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_PORTAL_STREAM_COMMAND,
NODE_WORKER_SUPERVISOR_CANCEL_COMMAND,
NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND,
@@ -40,6 +41,7 @@ import type { NodeWorkerBundleInstallerControl } from "./node-worker-bundle-inst
import { NodeWorkerCapacityExhaustedError } from "./node-worker-capacity.js";
import {
parseNodeWorkerCancelInput,
parseNodeWorkerEnvironmentStopInput,
parseNodeWorkerLaunchInput,
parseNodeWorkerLookupInput,
projectNodeWorkerSupervisorReceipt,
@@ -125,6 +127,7 @@ export async function invokeNodeWorkerSupervisorCommand(params: {
params.command === NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND ||
params.command === NODE_WORKER_SUPERVISOR_STATUS_COMMAND ||
params.command === NODE_WORKER_SUPERVISOR_CANCEL_COMMAND ||
params.command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND ||
params.command === NODE_WORKER_WORKSPACE_EXEC_COMMAND ||
params.command === NODE_WORKER_WORKSPACE_RETAIN_COMMAND ||
params.command === NODE_WORKER_DESKTOP_STREAM_COMMAND ||
@@ -268,6 +271,12 @@ export async function invokeNodeWorkerSupervisorCommand(params: {
}),
};
}
if (params.command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND) {
await params.supervisor!.stopEnvironment(
parseNodeWorkerEnvironmentStopInput(params.paramsJSON),
);
return { handled: true, ok: true, payload: null };
}
const receipt =
params.command === NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND
? await params.supervisor!.launch(
@@ -1,5 +1,6 @@
import {
parseNodeWorkerSupervisorReceipt,
type NodeWorkerEnvironmentStopInput,
type NodeWorkerLaunchInput,
type NodeWorkerSupervisorIdentity,
type NodeWorkerSupervisorReceipt,
@@ -12,9 +13,9 @@ import type { WorkerConnectionEndpoint } from "../worker/worker-connection-endpo
import type { NodeWorkerLaunchReceipt } from "./node-worker-launch-store.js";
export {
assertNodeWorkerLaunchIdentity,
nodeWorkerPlanHash,
parseNodeWorkerCancelInput,
parseNodeWorkerEnvironmentStopInput,
parseNodeWorkerLaunchInput,
parseNodeWorkerLookupInput,
} from "../worker/node-supervisor-protocol.js";
@@ -36,6 +37,7 @@ export type NodeWorkerSupervisorControl = {
signal?: AbortSignal,
): Promise<NodeWorkerWorkspaceRetainResult>;
cancel(expected: NodeWorkerSupervisorIdentity): Promise<NodeWorkerLaunchReceipt | undefined>;
stopEnvironment(input: NodeWorkerEnvironmentStopInput): Promise<void>;
};
export function projectNodeWorkerSupervisorReceipt(
@@ -1,19 +1,73 @@
import type { NodeWorkerCapacitySnapshot } from "../infra/node-runner-inventory.js";
import { createDeferredCore } from "../shared/deferred.js";
import type { NodeWorkerContainerEngine } from "./node-worker-container-engine.js";
import type { NodeWorkerTerminalOutcome } from "./node-worker-launch-observation.js";
import type {
NodeWorkerContainerIdentity,
NodeWorkerLaunchClaim,
NodeWorkerLaunchReceipt,
NodeWorkerTerminalState,
} from "./node-worker-launch-store.js";
import type { NodeWorkerChildAdapter } from "./node-worker-launch-transport.js";
import type { NodeWorkerCredentialScrubber } from "./node-worker-output.js";
import type { NodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
import type { NodeWorkerLaunchInput } from "./node-worker-supervisor-contract.js";
import type { NodeWorkerWorkspaceRuntime } from "./node-worker-workspace.js";
export type NodeWorkerStopState = Extract<NodeWorkerTerminalState, "cancelled" | "interrupted">;
export type NodeWorkerEnvironmentBinding = ReturnType<typeof nodeWorkerEnvironmentBinding>;
/** Only environment facts survive a turn; descriptors contain disposable admission authority. */
export function nodeWorkerEnvironmentBinding(input: NodeWorkerLaunchInput) {
const { admission, assignment } = input.descriptor;
return {
gatewayNamespace: input.gatewayNamespace,
environmentId: admission.environmentId,
sessionId: admission.sessionId,
ownerEpoch: admission.ownerEpoch,
placementGeneration: input.placementGeneration,
bundleHash: input.expectedBundleHash,
agentId: assignment.agentId,
workspaceDir: assignment.workspaceDir,
containmentRoot: assignment.workerContainmentRoot,
permissionMode: assignment.permissionMode,
};
}
export function nodeWorkerEnvironmentKey(
binding: Pick<NodeWorkerEnvironmentBinding, "gatewayNamespace" | "environmentId">,
): string {
return JSON.stringify([binding.gatewayNamespace, binding.environmentId]);
}
export function nodeWorkerEnvironmentMatches(
binding: Pick<
NodeWorkerEnvironmentBinding,
"gatewayNamespace" | "environmentId" | "sessionId" | "ownerEpoch"
>,
expected: Pick<
NodeWorkerEnvironmentBinding,
"gatewayNamespace" | "environmentId" | "sessionId" | "ownerEpoch"
>,
): boolean {
return (
binding.gatewayNamespace === expected.gatewayNamespace &&
binding.environmentId === expected.environmentId &&
binding.sessionId === expected.sessionId &&
binding.ownerEpoch === expected.ownerEpoch
);
}
export function createNodeWorkerActiveTurn(claim: NodeWorkerLaunchClaim) {
const { promise, resolve } = createDeferredCore();
return { claim, done: promise, settle: resolve, cancelled: false };
}
type NodeWorkerActiveTurn = ReturnType<typeof createNodeWorkerActiveTurn>;
type NodeWorkerActiveBase = {
binding: NodeWorkerEnvironmentBinding;
gatewayNamespace: string;
launchId: string;
planHash: string;
@@ -30,6 +84,8 @@ export type NodeWorkerRunningChild = NodeWorkerActiveBase & {
releaseJournal: () => void;
scrubber: NodeWorkerCredentialScrubber;
connectionFailure: { errorText?: string };
turn?: NodeWorkerActiveTurn;
retiring: boolean;
stopState?: NodeWorkerStopState;
containerCleanup?: Promise<void>;
deferredOutcome?: NodeWorkerTerminalOutcome;
@@ -2,6 +2,7 @@ import type { NodeWorkerCapacity } from "./node-worker-capacity.js";
import type { NodeWorkerContainerLifecycle } from "./node-worker-container-lifecycle.js";
import type { NodeWorkerLaunchReceipt, NodeWorkerLaunchStore } from "./node-worker-launch-store.js";
import { inspectNodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
import { nodeWorkerReceiptMatchesOwner } from "./node-worker-supervisor-ownership.js";
import {
inspectOwnedNodeWorkerTree,
signalOwnedNodeWorkerTree,
@@ -18,54 +19,96 @@ export async function recoverNodeWorkerLaunch(params: {
capacity: NodeWorkerCapacity;
containerLifecycle?: NodeWorkerContainerLifecycle;
notifyCapacity: boolean;
state?: "cancelled" | "interrupted";
}): Promise<NodeWorkerLaunchReceipt> {
const { receipt } = params;
if (receipt.state !== "running" || !receipt.worker) {
return receipt;
const state = params.state ?? "interrupted";
const latest = () => params.store.get(receipt.launchId) ?? receipt;
const stillOwned = () => {
const current = params.store.getMatching(receipt);
return (
current?.state === receipt.state &&
current.gatewayNamespace === receipt.gatewayNamespace &&
nodeWorkerReceiptMatchesOwner(current, receipt.supervisor, receipt.worker, receipt.container)
);
};
if ((receipt.state !== "pending" && receipt.state !== "running") || !stillOwned()) {
return latest();
}
const previousSupervisor = inspectNodeWorkerProcessIdentity(receipt.supervisor);
if (previousSupervisor !== "dead" && previousSupervisor !== "reused") {
return params.store.get(receipt.launchId) ?? receipt;
return latest();
}
if (!receipt.worker && params.containerLifecycle) {
// A pending container can exist before its identity reaches the journal.
// Sweep it before releasing the reservation, then revalidate any pending adoption.
await params.containerLifecycle.initialize();
if (!stillOwned()) {
return latest();
}
}
if (receipt.container) {
if (!params.containerLifecycle) {
throw new Error("node worker container isolation has no lifecycle owner");
}
const containerState = await params.containerLifecycle.inspect(receipt.container, receipt);
if (!stillOwned()) {
return latest();
}
if (containerState === "unknown") {
if (state === "cancelled") {
return latest();
}
throw new Error(
`node worker container ${receipt.container.containerId} could not be inspected; restore its ${receipt.container.engine} engine before enabling worker hosting`,
);
}
if (containerState === "reused") {
if (state === "cancelled") {
return latest();
}
throw new Error(`node worker launch ${receipt.launchId} lost its container ownership`);
}
await params.containerLifecycle.remove(receipt.container, receipt);
} else {
} else if (receipt.worker) {
let workerState = inspectOwnedNodeWorkerTree(receipt.worker);
if (workerState === "unknown") {
return params.store.get(receipt.launchId) ?? receipt;
return latest();
}
if (workerState === "live") {
if (!stillOwned()) {
return latest();
}
await signalOwnedNodeWorkerTree(receipt.worker, "SIGTERM");
workerState = await waitForOwnedNodeWorkerTreeDeath(receipt.worker, STOP_GRACE_MS);
}
if (workerState === "live") {
if (!stillOwned()) {
return latest();
}
await signalOwnedNodeWorkerTree(receipt.worker, "SIGKILL");
workerState = await waitForOwnedNodeWorkerTreeDeath(receipt.worker, FORCE_STOP_WAIT_MS);
}
if (workerState !== "dead") {
return params.store.get(receipt.launchId) ?? receipt;
return latest();
}
}
if (!stillOwned()) {
return latest();
}
return params.capacity.finish(
{
launchId: receipt.launchId,
planHash: receipt.planHash,
supervisor: receipt.supervisor,
worker: receipt.worker,
state: "interrupted",
errorText: "node host stopped before the worker launch completed",
state,
errorText:
state === "cancelled"
? "node worker launch cancelled"
: receipt.worker
? "node host stopped before the worker launch completed"
: "node host stopped before the worker launch started",
},
params.notifyCapacity,
);
@@ -41,7 +41,11 @@ describe("node worker admission re-arm journal", () => {
clientId: GATEWAY_CLIENT_IDS.NODE_HOST,
clientMode: GATEWAY_CLIENT_MODES.NODE,
protocolFeature: NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
workerHost: { enabled: true, capacity: { total: 1, available: 1 } },
workerHost: {
enabled: true,
environmentSession: 1,
capacity: { total: 1, available: 1 },
},
commands: [],
},
],
@@ -0,0 +1,272 @@
export const stdioWorkerSource = String.raw`
import fs from "node:fs";
import { createInterface } from "node:readline";
let active;
let retained = false;
if (process.connected || process.argv.includes("--internal-worker-ipc") ||
!process.argv.includes("--internal-worker-session")) {
process.stderr.write("container worker unexpectedly received Node IPC");
process.exit(24);
}
const completedResult = { status: "completed", transcriptLeafId: "leaf-1", transcriptNextSeq: 2 };
const finish = (descriptor, result = completedResult) => {
if (active !== descriptor) return;
fs.writeSync(1, JSON.stringify({
type: "result", turnId: descriptor.assignment.turnId, result, retainWorker: retained,
}) + "\n");
active = undefined;
if (!retained) process.exit(0);
};
const runTurn = async (descriptor) => {
fs.writeFileSync(descriptor.assignment.workspaceDir + "/" + descriptor.assignment.turnId + ".fixture.json", JSON.stringify({
pid: process.pid, argv: process.argv.slice(2), endpoint: descriptor.connectionEndpoint,
}));
if (descriptor.assignment.prompt === "admission-failure") {
throw new Error("worker admission deadline exceeded after 9 attempts to gateway.example:443: connect failed: Opening handshake has timed out " + descriptor.admission.credential);
} else if (descriptor.assignment.prompt === "wait") {
fs.writeFileSync(descriptor.assignment.workspaceDir + "/worker-started", "started");
return;
} else {
retained ||= descriptor.assignment.prompt === "retain";
await new Promise((resolve) => setTimeout(resolve, 35));
finish(descriptor);
}
};
const lines = createInterface({ input: process.stdin, crlfDelay: Infinity });
lines.on("line", (line) => {
const request = JSON.parse(line);
if (request.type === "cancel") {
if (active?.assignment.turnId === request.turnId) {
finish(active, { status: "failed", reason: "turn-failed", transcriptLeafId: null, transcriptNextSeq: 1 });
}
return;
}
if (active || request.type !== "turn" || request.turnId !== request.descriptor.assignment.turnId) process.exit(25);
active = request.descriptor;
void runTurn(active).catch((error) => { process.stderr.write(error.message + "\n"); process.exit(1); });
});
lines.once("close", () => process.exit(0));
`;
export const fakeEngineSource = String.raw`
const { spawn } = require("node:child_process");
const { createHash } = require("node:crypto");
const { DatabaseSync } = require("node:sqlite");
const fs = require("node:fs");
const path = require("node:path");
const args = process.argv.slice(2);
const command = args[0];
const statePath = (id) => path.join(engineRoot, id + ".container.json");
const load = (id) => JSON.parse(fs.readFileSync(statePath(id), "utf8"));
// Sibling shim invocations (rm/inspect/wait) read this state while another
// writes it. A truncating write exposes a zero-length window, so a reader
// parses partial JSON and the shim exits 1; rename is atomic, so readers
// always see either the previous or the next complete state.
const save = (container) => {
const target = statePath(container.id);
const pending = target + "." + process.pid + ".pending";
fs.writeFileSync(pending, JSON.stringify(container));
fs.renameSync(pending, target);
};
const launchIdFor = (container) =>
Buffer.from(container.labels["openclaw.node-worker.launch"], "base64url").toString("utf8");
const journalState = (launchId) => {
try {
const database = new DatabaseSync(path.join(stateRoot, "state", "openclaw.sqlite"), { readOnly: true });
const row = database.prepare("SELECT state FROM node_worker_launches WHERE launch_id = ?").get(launchId);
const hasContainerTable = database.prepare(
"SELECT 1 FROM sqlite_schema WHERE type = 'table' AND name = 'node_worker_launch_containers'",
).get();
const container = hasContainerTable
? database.prepare(
"SELECT container_json FROM node_worker_launch_containers WHERE launch_id = ?",
).get(launchId)
: undefined;
database.close();
return row && { state: row.state, container_json: container?.container_json ?? null };
} catch {
return undefined;
}
};
const record = (entry) => fs.appendFileSync(commandLog, JSON.stringify(entry) + "\n");
const waitForRunningJournal = async (container) => {
let journal;
for (let attempt = 0; attempt < 100; attempt += 1) {
journal = journalState(launchIdFor(container));
const persisted = journal?.container_json && JSON.parse(journal.container_json);
if (
journal?.state === "running" &&
persisted?.containerId === container.id &&
persisted?.engineTarget === expectedEngineTarget
) {
return journal;
}
await new Promise((resolve) => setTimeout(resolve, 10));
}
return journal;
};
const releaseAfterMarker = (marker, operation) => {
const markerPath = path.join(engineRoot, marker);
if (!fs.existsSync(markerPath)) {
operation();
return;
}
const timer = setInterval(() => {
if (!fs.existsSync(markerPath)) {
clearInterval(timer);
operation();
}
}, 10);
};
const missing = (id) => {
process.stderr.write("Error: No such object: " + id + "\n");
process.exit(1);
};
const readContainer = (id) => {
if (!fs.existsSync(statePath(id))) missing(id);
return load(id);
};
if (command === "version") {
record({ argv: args });
process.stdout.write("27.0.0\n");
} else if (command === "info") {
const daemonId = fs.readFileSync(path.join(engineRoot, "daemon-id"), "utf8");
record({ argv: args, daemonId });
const delayFile = path.join(engineRoot, "info-delay-ms");
const delayMs = fs.existsSync(delayFile) ? Number(fs.readFileSync(delayFile, "utf8")) : 0;
setTimeout(() => process.stdout.write(daemonId + "\n"), delayMs);
} else if (command === "create") {
const id = createHash("sha256").update(JSON.stringify(args)).digest("hex");
const labels = {};
const env = {};
const mounts = [];
for (let index = 1; index < args.length; index += 1) {
if (args[index] === "--label" || args[index] === "--env") {
const value = args[++index];
const separator = value.indexOf("=");
(args[index - 1] === "--label" ? labels : env)[value.slice(0, separator)] = value.slice(separator + 1);
} else if (args[index] === "--mount") {
mounts.push(args[++index]);
}
}
const entrypoint = args.indexOf("--entrypoint");
const image = args[entrypoint + 2];
const entry = args[entrypoint + 3];
const workerArgs = args.slice(entrypoint + 4);
const container = { id, labels, env, mounts, image, entry, workerArgs, status: "created", pid: null };
save(container);
record({ argv: args, container, journal: journalState(launchIdFor(container)) });
releaseAfterMarker("hold-create", () => process.stdout.write(id + "\n"));
} else if (command === "start") {
void (async () => {
const container = readContainer(args.at(-1));
const journal = await waitForRunningJournal(container);
record({ argv: args, journal });
const persisted = journal?.container_json && JSON.parse(journal.container_json);
if (
journal?.state !== "running" ||
persisted?.containerId !== container.id ||
persisted?.engineTarget !== expectedEngineTarget
) {
process.stderr.write("container worker executed before its exact identity was journaled\n");
process.exit(67);
}
// A real engine leaves the container "created" until its start request lands,
// so the marker lets a test hold the launch inside that startup window.
await new Promise((resolve) => releaseAfterMarker("hold-start", resolve));
const child = spawn(process.execPath, [container.entry, ...container.workerArgs], {
detached: process.platform !== "win32",
env: container.env,
stdio: ["pipe", "inherit", "inherit"],
});
container.status = "running";
container.pid = child.pid;
save(container);
process.stdin.pipe(child.stdin);
child.stdin.on("error", (error) => {
if (error.code !== "EPIPE") throw error;
});
child.once("error", (error) => {
process.stderr.write(error.message + "\n");
process.exitCode = 1;
});
child.once("exit", (code, signal) => {
if (fs.existsSync(statePath(container.id))) {
const current = load(container.id);
current.status = "exited";
current.pid = null;
save(current);
}
process.exit(code ?? (signal ? 137 : 0));
});
})().catch((error) => {
process.stderr.write(error.message + "\n");
process.exitCode = 1;
});
} else if (command === "inspect") {
const id = args.at(-1);
record({ argv: args });
const container = readContainer(id);
const format = args[args.indexOf("--format") + 1];
const columns = [container.status];
// Releasing the startup hold here proves the supervisor observed the container
// while it was still created: the launch only proceeds after that observation.
const startHold = path.join(engineRoot, "hold-start");
if (fs.existsSync(startHold)) fs.unlinkSync(startHold);
if (format.includes("openclaw.node-worker.host")) {
columns.push(
container.labels["openclaw.node-worker.host"] ?? "",
container.labels["openclaw.node-worker.gateway"] ?? "",
container.labels["openclaw.node-worker.launch"] ?? "",
);
}
process.stdout.write(columns.join("\t") + "\n");
} else if (command === "kill") {
const container = readContainer(args.at(-1));
record({ argv: args, journal: journalState(launchIdFor(container)) });
if (container.status !== "running") {
process.stderr.write("container is not running\n");
process.exit(1);
}
container.status = "exited";
save(container);
if (container.pid) {
try {
process.kill(process.platform === "win32" ? container.pid : -container.pid, "SIGKILL");
} catch (error) {
if (error.code !== "ESRCH") throw error;
}
}
process.stdout.write(container.id + "\n");
} else if (command === "rm") {
const container = readContainer(args.at(-1));
record({ argv: args, journal: journalState(launchIdFor(container)) });
if (fs.existsSync(path.join(engineRoot, "fail-removal"))) {
process.stderr.write("injected container removal failure\n");
process.exit(1);
}
releaseAfterMarker("hold-removal", () => {
fs.unlinkSync(statePath(container.id));
process.stdout.write(container.id + "\n");
});
} else if (command === "ps") {
record({ argv: args });
const ownerFilter = args.find((arg) => arg.startsWith("label=openclaw.node-worker.host="));
const owner = ownerFilter?.slice("label=openclaw.node-worker.host=".length);
for (const file of fs.readdirSync(engineRoot).sort()) {
if (!file.endsWith(".container.json")) continue;
const container = JSON.parse(fs.readFileSync(path.join(engineRoot, file), "utf8"));
if (container.labels["openclaw.node-worker.host"] !== owner) continue;
process.stdout.write([
container.id,
container.labels["openclaw.node-worker.gateway"],
container.labels["openclaw.node-worker.launch"],
].join("\t") + "\n");
}
} else {
record({ argv: args });
process.stderr.write("unsupported fake engine command: " + command + "\n");
process.exit(2);
}
`;
@@ -7,13 +7,22 @@ import { closeOpenClawStateDatabaseForTest } from "../state/openclaw-state-db.js
import type { WorkerConnectionEndpoint } from "../worker/worker-connection-endpoint.js";
import { NodeWorkerContainerLifecycle } from "./node-worker-container-lifecycle.js";
import { NodeWorkerLaunchStore } from "./node-worker-launch-store.js";
import { requireNodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
import {
inspectNodeWorkerProcessIdentity,
requireNodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
import {
fakeEngineSource,
stdioWorkerSource,
} from "./node-worker-supervisor.container.test-support.js";
import { createNodeWorkerSupervisor } from "./node-worker-supervisor.js";
import {
testNodeWorkerEnvironmentIdentity,
testNodeWorkerLaunchIdentity,
testWorkerLaunchInput,
writeNodeWorkerFixture,
} from "./node-worker-supervisor.test-support.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
const endpoint: WorkerConnectionEndpoint = {
@@ -24,250 +33,6 @@ const hostLabel = "openclaw.node-worker.host";
const gatewayLabel = "openclaw.node-worker.gateway";
const launchLabel = "openclaw.node-worker.launch";
const stdioWorkerSource = String.raw`
import fs from "node:fs";
let input = "";
for await (const chunk of process.stdin) input += chunk;
const descriptor = JSON.parse(input);
if (process.connected || process.argv.includes("--internal-worker-ipc")) {
process.stderr.write("container worker unexpectedly received Node IPC");
process.exit(24);
}
if (descriptor.assignment.prompt === "admission-failure") {
throw new Error("worker admission deadline exceeded after 9 attempts to gateway.example:443: connect failed: Opening handshake has timed out " + descriptor.admission.credential);
} else if (descriptor.assignment.prompt === "wait") {
fs.writeFileSync(descriptor.assignment.workspaceDir + "/worker-started", "started");
setInterval(() => {}, 1000);
} else {
await new Promise((resolve) => setTimeout(resolve, 35));
process.stdout.write(JSON.stringify({
status: "completed",
argv: process.argv.slice(2),
endpoint: descriptor.connectionEndpoint,
}) + "\n");
}
`;
const fakeEngineSource = String.raw`
const { spawn } = require("node:child_process");
const { createHash } = require("node:crypto");
const { DatabaseSync } = require("node:sqlite");
const fs = require("node:fs");
const path = require("node:path");
const args = process.argv.slice(2);
const command = args[0];
const statePath = (id) => path.join(engineRoot, id + ".container.json");
const load = (id) => JSON.parse(fs.readFileSync(statePath(id), "utf8"));
// Sibling shim invocations (rm/inspect/wait) read this state while another
// writes it. A truncating write exposes a zero-length window, so a reader
// parses partial JSON and the shim exits 1; rename is atomic, so readers
// always see either the previous or the next complete state.
const save = (container) => {
const target = statePath(container.id);
const pending = target + "." + process.pid + ".pending";
fs.writeFileSync(pending, JSON.stringify(container));
fs.renameSync(pending, target);
};
const launchIdFor = (container) =>
Buffer.from(container.labels["openclaw.node-worker.launch"], "base64url").toString("utf8");
const journalState = (launchId) => {
try {
const database = new DatabaseSync(path.join(stateRoot, "state", "openclaw.sqlite"), { readOnly: true });
const row = database.prepare("SELECT state FROM node_worker_launches WHERE launch_id = ?").get(launchId);
const hasContainerTable = database.prepare(
"SELECT 1 FROM sqlite_schema WHERE type = 'table' AND name = 'node_worker_launch_containers'",
).get();
const container = hasContainerTable
? database.prepare(
"SELECT container_json FROM node_worker_launch_containers WHERE launch_id = ?",
).get(launchId)
: undefined;
database.close();
return row && { state: row.state, container_json: container?.container_json ?? null };
} catch {
return undefined;
}
};
const record = (entry) => fs.appendFileSync(commandLog, JSON.stringify(entry) + "\n");
const waitForRunningJournal = async (container) => {
let journal;
for (let attempt = 0; attempt < 100; attempt += 1) {
journal = journalState(launchIdFor(container));
const persisted = journal?.container_json && JSON.parse(journal.container_json);
if (
journal?.state === "running" &&
persisted?.containerId === container.id &&
persisted?.engineTarget === expectedEngineTarget
) {
return journal;
}
await new Promise((resolve) => setTimeout(resolve, 10));
}
return journal;
};
const releaseAfterMarker = (marker, operation) => {
const markerPath = path.join(engineRoot, marker);
if (!fs.existsSync(markerPath)) {
operation();
return;
}
const timer = setInterval(() => {
if (!fs.existsSync(markerPath)) {
clearInterval(timer);
operation();
}
}, 10);
};
const missing = (id) => {
process.stderr.write("Error: No such object: " + id + "\n");
process.exit(1);
};
const readContainer = (id) => {
if (!fs.existsSync(statePath(id))) missing(id);
return load(id);
};
if (command === "version") {
record({ argv: args });
process.stdout.write("27.0.0\n");
} else if (command === "info") {
const daemonId = fs.readFileSync(path.join(engineRoot, "daemon-id"), "utf8");
record({ argv: args, daemonId });
const delayFile = path.join(engineRoot, "info-delay-ms");
const delayMs = fs.existsSync(delayFile) ? Number(fs.readFileSync(delayFile, "utf8")) : 0;
setTimeout(() => process.stdout.write(daemonId + "\n"), delayMs);
} else if (command === "create") {
const id = createHash("sha256").update(JSON.stringify(args)).digest("hex");
const labels = {};
const env = {};
const mounts = [];
for (let index = 1; index < args.length; index += 1) {
if (args[index] === "--label" || args[index] === "--env") {
const value = args[++index];
const separator = value.indexOf("=");
(args[index - 1] === "--label" ? labels : env)[value.slice(0, separator)] = value.slice(separator + 1);
} else if (args[index] === "--mount") {
mounts.push(args[++index]);
}
}
const entry = args.at(-1);
const image = args.at(-2);
const container = { id, labels, env, mounts, image, entry, status: "created", pid: null };
save(container);
record({ argv: args, container, journal: journalState(launchIdFor(container)) });
releaseAfterMarker("hold-create", () => process.stdout.write(id + "\n"));
} else if (command === "start") {
void (async () => {
let descriptor = "";
for await (const chunk of process.stdin) descriptor += chunk;
const container = readContainer(args.at(-1));
const journal = await waitForRunningJournal(container);
record({ argv: args, journal });
const persisted = journal?.container_json && JSON.parse(journal.container_json);
if (
journal?.state !== "running" ||
persisted?.containerId !== container.id ||
persisted?.engineTarget !== expectedEngineTarget
) {
process.stderr.write("container worker executed before its exact identity was journaled\n");
process.exit(67);
}
// A real engine leaves the container "created" until its start request lands,
// so the marker lets a test hold the launch inside that startup window.
await new Promise((resolve) => releaseAfterMarker("hold-start", resolve));
const child = spawn(process.execPath, [container.entry], {
detached: process.platform !== "win32",
env: container.env,
stdio: ["pipe", "inherit", "inherit"],
});
container.status = "running";
container.pid = child.pid;
save(container);
child.stdin.end(descriptor);
child.once("error", (error) => {
process.stderr.write(error.message + "\n");
process.exitCode = 1;
});
child.once("exit", (code, signal) => {
if (fs.existsSync(statePath(container.id))) {
const current = load(container.id);
current.status = "exited";
current.pid = null;
save(current);
}
process.exitCode = code ?? (signal ? 137 : 0);
});
})().catch((error) => {
process.stderr.write(error.message + "\n");
process.exitCode = 1;
});
} else if (command === "inspect") {
const id = args.at(-1);
record({ argv: args });
const container = readContainer(id);
const format = args[args.indexOf("--format") + 1];
const columns = [container.status];
// Releasing the startup hold here proves the supervisor observed the container
// while it was still created: the launch only proceeds after that observation.
const startHold = path.join(engineRoot, "hold-start");
if (fs.existsSync(startHold)) fs.unlinkSync(startHold);
if (format.includes("openclaw.node-worker.host")) {
columns.push(
container.labels["openclaw.node-worker.host"] ?? "",
container.labels["openclaw.node-worker.gateway"] ?? "",
container.labels["openclaw.node-worker.launch"] ?? "",
);
}
process.stdout.write(columns.join("\t") + "\n");
} else if (command === "kill") {
const container = readContainer(args.at(-1));
record({ argv: args, journal: journalState(launchIdFor(container)) });
if (container.status !== "running") {
process.stderr.write("container is not running\n");
process.exit(1);
}
container.status = "exited";
save(container);
if (container.pid) {
try {
process.kill(process.platform === "win32" ? container.pid : -container.pid, "SIGKILL");
} catch (error) {
if (error.code !== "ESRCH") throw error;
}
}
process.stdout.write(container.id + "\n");
} else if (command === "rm") {
const container = readContainer(args.at(-1));
record({ argv: args, journal: journalState(launchIdFor(container)) });
if (fs.existsSync(path.join(engineRoot, "fail-removal"))) {
process.stderr.write("injected container removal failure\n");
process.exit(1);
}
releaseAfterMarker("hold-removal", () => {
fs.unlinkSync(statePath(container.id));
process.stdout.write(container.id + "\n");
});
} else if (command === "ps") {
record({ argv: args });
const ownerFilter = args.find((arg) => arg.startsWith("label=openclaw.node-worker.host="));
const owner = ownerFilter?.slice("label=openclaw.node-worker.host=".length);
for (const file of fs.readdirSync(engineRoot).sort()) {
if (!file.endsWith(".container.json")) continue;
const container = JSON.parse(fs.readFileSync(path.join(engineRoot, file), "utf8"));
if (container.labels["openclaw.node-worker.host"] !== owner) continue;
process.stdout.write([
container.id,
container.labels["openclaw.node-worker.gateway"],
container.labels["openclaw.node-worker.launch"],
].join("\t") + "\n");
}
} else {
record({ argv: args });
process.stderr.write("unsupported fake engine command: " + command + "\n");
process.exit(2);
}
`;
type FakeContainer = {
id: string;
labels: Record<string, string>;
@@ -275,6 +40,7 @@ type FakeContainer = {
mounts: string[];
image: string;
entry: string;
workerArgs: string[];
status: "created" | "running" | "exited";
pid: number | null;
};
@@ -368,6 +134,7 @@ function containerFixture(
mounts: [],
image: "node:22-slim",
entry: bundleEntry,
workerArgs: ["--internal-worker-session"],
status: params.status ?? "running",
pid: null,
};
@@ -413,7 +180,13 @@ function claimFixtureLaunch(
const supervisor = { pid: 2_147_483_647, startTime: 1 };
const worker = { pid: 2_147_483_646, startTime: 1 };
const store = new NodeWorkerLaunchStore({ env: fixture.env });
store.claim({ ...identity, gatewayNamespace: input.gatewayNamespace }, supervisor, 8);
const claim = { ...identity, gatewayNamespace: input.gatewayNamespace };
store.claim(claim, supervisor, 8);
new NodeWorkerTurnStore({ env: fixture.env }).claim({
claim,
ownerLaunchId: launchId,
supervisor,
});
if (containerId) {
store.markRunning({
launchId,
@@ -455,7 +228,19 @@ describe("node worker supervisor container isolation", () => {
expect(completed).toMatchObject({ state: "completed" });
expect(JSON.parse(completed?.resultJson ?? "null")).toEqual({
status: "completed",
argv: [],
transcriptLeafId: "leaf-1",
transcriptNextSeq: 2,
});
expect(
JSON.parse(
fs.readFileSync(
path.join(fixture.workspaceDir, `${input.launchId}.fixture.json`),
"utf8",
),
),
).toEqual({
pid: expect.any(Number),
argv: ["--internal-worker-session"],
endpoint,
});
@@ -523,6 +308,77 @@ describe("node worker supervisor container isolation", () => {
}
});
it("keeps one container and capacity slot across completed and cancelled turns until environment teardown", async () => {
const capacities: Array<{ total: number; available: number }> = [];
const fixture = containerFixture({
capacity: 1,
onCapacityChanged: (capacity) => capacities.push(capacity),
});
const first = testWorkerLaunchInput(fixture.workspaceDir, "container-retained-first", "retain");
const next = testWorkerLaunchInput(fixture.workspaceDir, "container-retained-next");
const waiting = testWorkerLaunchInput(
fixture.workspaceDir,
"container-retained-cancel",
"wait",
);
const store = new NodeWorkerLaunchStore({ env: fixture.env });
try {
const running = await fixture.supervisor.launch(first, endpoint);
const completed = await waitForTerminal(fixture.supervisor, first.launchId);
const originalWorker = JSON.parse(
fs.readFileSync(path.join(fixture.workspaceDir, `${first.launchId}.fixture.json`), "utf8"),
) as { pid: number };
const worker = requireNodeWorkerProcessIdentity(originalWorker.pid);
expect(completed?.state).toBe("completed");
expect(store.get(first.launchId)).toMatchObject({
state: "running",
container: running.container,
});
expect(fixture.exists(running.container!.containerId)).toBe(true);
expect(capacities.at(-1)).toEqual({ total: 1, available: 0 });
expect(await fixture.supervisor.launch(first, endpoint)).toEqual(completed);
expect(await fixture.supervisor.launch(next, endpoint)).toMatchObject({
state: "running",
worker: running.worker,
container: running.container,
});
expect((await waitForTerminal(fixture.supervisor, next.launchId))?.state).toBe("completed");
expect(
JSON.parse(
fs.readFileSync(path.join(fixture.workspaceDir, `${next.launchId}.fixture.json`), "utf8"),
),
).toMatchObject({ pid: worker.pid });
await fixture.supervisor.launch(waiting, endpoint);
await waitForWorkerStarted(fixture.workspaceDir);
expect(await fixture.supervisor.cancel(testNodeWorkerLaunchIdentity(waiting))).toMatchObject({
state: "cancelled",
});
expect(inspectNodeWorkerProcessIdentity(worker)).toBe("live");
expect(fixture.events().filter((event) => event.argv[0] === "create")).toHaveLength(1);
expect(fixture.events().filter((event) => event.argv[0] === "start")).toHaveLength(1);
expect(fixture.events().filter((event) => event.argv[0] === "rm")).toHaveLength(0);
expect(store.listNonterminal()).toHaveLength(1);
await fixture.supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(first));
expect(fixture.exists(running.container!.containerId)).toBe(false);
expect(fixture.events().find((event) => event.argv[0] === "kill")?.argv).toEqual([
"kill",
running.container!.containerId,
]);
expect(fixture.events().find((event) => event.argv[0] === "rm")?.journal?.state).toBe(
"running",
);
await vi.waitFor(() => expect(inspectNodeWorkerProcessIdentity(worker)).not.toBe("live"));
expect(await fixture.supervisor.status(first.launchId)).toEqual(completed);
expect(capacities.at(-1)).toEqual({ total: 1, available: 1 });
} finally {
await fixture.supervisor.close();
}
});
it("keeps a launch running while its container is still starting", async () => {
const fixture = containerFixture();
const input = testWorkerLaunchInput(fixture.workspaceDir, "container-startup-poll");
@@ -735,7 +591,7 @@ describe("node worker supervisor container isolation", () => {
});
it.each([
["cancel", "cancelled"],
["stopEnvironment", "interrupted"],
["close", "interrupted"],
] as const)(
"%s kills and removes the container before terminal persistence",
@@ -746,8 +602,8 @@ describe("node worker supervisor container isolation", () => {
try {
const running = await fixture.supervisor.launch(input, endpoint);
await waitForWorkerStarted(fixture.workspaceDir);
if (operation === "cancel") {
await fixture.supervisor.cancel(testNodeWorkerLaunchIdentity(input));
if (operation === "stopEnvironment") {
await fixture.supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(input));
} else {
await fixture.supervisor.close();
}
@@ -917,7 +773,7 @@ describe("node worker supervisor container isolation", () => {
}
});
it("keeps the launch and capacity occupied when removal fails until cancellation can retry", async () => {
it("keeps the launch and capacity occupied when removal fails until environment teardown can retry", async () => {
const capacitySnapshots: Array<{ total: number; available: number }> = [];
const fixture = containerFixture({
capacity: 1,
@@ -934,9 +790,9 @@ describe("node worker supervisor container isolation", () => {
);
fs.writeFileSync(failureMarker, "fail");
await expect(fixture.supervisor.cancel(testNodeWorkerLaunchIdentity(input))).rejects.toThrow(
/removal|failed|injected/iu,
);
await expect(
fixture.supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(input)),
).rejects.toThrow(/removal|failed|injected/iu);
expect(store.get(input.launchId)).toMatchObject({
state: "running",
container: running.container,
@@ -945,9 +801,10 @@ describe("node worker supervisor container isolation", () => {
expect(capacitySnapshots.at(-1)).toEqual({ total: 1, available: 0 });
fs.unlinkSync(failureMarker);
await expect(
fixture.supervisor.cancel(testNodeWorkerLaunchIdentity(input)),
).resolves.toMatchObject({ state: "cancelled" });
await fixture.supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(input));
expect(await fixture.supervisor.status(input.launchId)).toMatchObject({
state: "interrupted",
});
expect(fixture.exists(running.container!.containerId)).toBe(false);
expect(capacitySnapshots.at(-1)).toEqual({ total: 1, available: 1 });
} finally {
@@ -962,6 +819,8 @@ describe("node worker supervisor container isolation", () => {
const fixture = containerFixture({ capacity: 2 });
const first = testWorkerLaunchInput(fixture.workspaceDir, "container-close-failed", "wait");
const sibling = testWorkerLaunchInput(fixture.workspaceDir, "container-close-sibling", "wait");
sibling.descriptor.admission.environmentId = "sibling-environment";
sibling.descriptor.admission.sessionId = "sibling-session";
const removalMarker = path.join(fixture.engineRoot, "hold-removal");
const store = new NodeWorkerLaunchStore({ env: fixture.env });
const removalFailure = new Error("injected first container removal failure");
@@ -0,0 +1,425 @@
import fs from "node:fs";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
import { resetSecretRedactionRegistryForTest } from "../logging/secret-redaction-registry.test-support.js";
import { closeOpenClawStateDatabaseForTest } from "../state/openclaw-state-db.js";
import { NodeWorkerLaunchStore } from "./node-worker-launch-store.js";
import {
inspectNodeWorkerProcessIdentity,
requireNodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
import { createNodeWorkerSupervisor } from "./node-worker-supervisor.js";
import {
TEST_WORKER_ENDPOINT,
TEST_WORKER_SOURCE,
testNodeWorkerEnvironmentIdentity,
testNodeWorkerLaunchIdentity,
testWorkerLaunchInput,
writeNodeWorkerFixture,
} from "./node-worker-supervisor.test-support.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
type NodeWorkerSupervisor = ReturnType<typeof createNodeWorkerSupervisor>;
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
afterEach(() => {
vi.restoreAllMocks();
resetSecretRedactionRegistryForTest();
closeOpenClawStateDatabaseForTest();
});
function fixture(
options: {
capacity?: number;
capacityWaitMs?: number;
onCapacityChanged?: (capacity: { total: number; available: number }) => void;
} = {},
) {
const root = tempDirs.make("node-worker-supervisor-");
const { bundleRoot, env, stateDir, workspaceDir } = writeNodeWorkerFixture(root);
const supervisor = createNodeWorkerSupervisor({ bundleRoot, env, ...options });
return { bundleRoot, env, root, stateDir, supervisor, workspaceDir };
}
function launchInput(workspaceDir: string, launchId: string, prompt = "success") {
const input = testWorkerLaunchInput(workspaceDir, launchId, prompt);
input.descriptor.admission.environmentId = `environment-${launchId}`;
input.descriptor.admission.sessionId = `session-${launchId}`;
return input;
}
async function waitForTerminal(supervisor: NodeWorkerSupervisor, launchId: string) {
await vi.waitFor(
async () => {
expect((await supervisor.status(launchId))?.state).not.toMatch(/^(?:pending|running)$/u);
},
{ timeout: 5_000 },
);
const receipt = await supervisor.status(launchId);
if (!receipt) {
throw new Error(`missing launch receipt ${launchId}`);
}
return receipt;
}
describe("node worker environment lifetime", () => {
it("reuses a retained worker at capacity across turns and cancellation until its environment stops", async () => {
const capacitySnapshots: Array<{ total: number; available: number }> = [];
const { env, supervisor, workspaceDir } = fixture({
capacity: 1,
capacityWaitMs: 25,
onCapacityChanged: (capacity) => capacitySnapshots.push(capacity),
});
const first = testWorkerLaunchInput(workspaceDir, "preview-start", "background-start");
const nextTurn = (turnId: string, prompt: string) => {
const input = testWorkerLaunchInput(workspaceDir, turnId, prompt);
input.descriptor.admission.credential = `credential-${turnId}`;
input.descriptor.assignment.runId = `run-${turnId}`;
input.descriptor.assignment.operationalRunInstance = {
instanceId: `instance-${turnId}`,
runId: input.descriptor.assignment.runId,
};
input.descriptor.assignment.agentRuntimeIdentityToken = `signed-token-${turnId}`;
return input;
};
const environment = testNodeWorkerEnvironmentIdentity(first);
const store = new NodeWorkerLaunchStore({ env });
try {
const running = await supervisor.launch(first, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, first.launchId);
const background = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${first.launchId}.background.json`), "utf8"),
) as { pid: number; url: string };
const server = requireNodeWorkerProcessIdentity(background.pid);
expect(completed.state).toBe("completed");
expect(store.get(first.launchId)).toMatchObject({ state: "running", worker: running.worker });
expect(capacitySnapshots.at(-1)).toEqual({ total: 1, available: 0 });
expect(await (await fetch(background.url)).text()).toBe("preview-ready");
expect(await supervisor.launch(first, TEST_WORKER_ENDPOINT)).toEqual(completed);
expect(
JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${first.launchId}.started.json`), "utf8"),
),
).toEqual({ pid: running.worker!.pid, starts: 1 });
const poll = nextTurn("preview-poll", "background-poll");
expect(await supervisor.launch(poll, TEST_WORKER_ENDPOINT)).toMatchObject({
state: "running",
worker: running.worker,
});
expect((await waitForTerminal(supervisor, poll.launchId)).state).toBe("completed");
expect(
JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${poll.launchId}.background.json`), "utf8"),
),
).toEqual({ ...background, response: "preview-ready" });
const waiting = nextTurn("preview-cancel", "background-wait");
await supervisor.launch(waiting, TEST_WORKER_ENDPOINT);
await vi.waitFor(() =>
expect(fs.existsSync(path.join(workspaceDir, `${waiting.launchId}.started.json`))).toBe(
true,
),
);
expect(await supervisor.cancel(testNodeWorkerLaunchIdentity(first))).toEqual(completed);
expect((await supervisor.status(waiting.launchId))?.state).toBe("running");
await expect(
supervisor.launch(nextTurn("preview-concurrent", "background-poll"), TEST_WORKER_ENDPOINT),
).rejects.toThrow("already has an active turn");
expect(await supervisor.cancel(testNodeWorkerLaunchIdentity(waiting))).toMatchObject({
state: "cancelled",
worker: running.worker,
});
expect(inspectNodeWorkerProcessIdentity(server)).toBe("live");
expect(await (await fetch(background.url)).text()).toBe("preview-ready");
const afterCancel = nextTurn("preview-after-cancel", "background-poll");
expect(await supervisor.launch(afterCancel, TEST_WORKER_ENDPOINT)).toMatchObject({
worker: running.worker,
});
expect((await waitForTerminal(supervisor, afterCancel.launchId)).state).toBe("completed");
expect(store.listNonterminal()).toHaveLength(1);
expect(capacitySnapshots.at(-1)).toEqual({ total: 1, available: 0 });
for (const mismatch of [
{ ...environment, gatewayNamespace: "other-gateway" },
{ ...environment, sessionId: "other-session" },
{ ...environment, ownerEpoch: environment.ownerEpoch + 1 },
]) {
await supervisor.stopEnvironment(mismatch);
expect(inspectNodeWorkerProcessIdentity(running.worker!)).toBe("live");
}
await supervisor.stopEnvironment(environment);
await vi.waitFor(() => {
expect(inspectNodeWorkerProcessIdentity(running.worker!)).not.toBe("live");
expect(inspectNodeWorkerProcessIdentity(server)).not.toBe("live");
});
await expect(fetch(background.url)).rejects.toThrow();
expect(capacitySnapshots.at(-1)).toEqual({ total: 1, available: 1 });
expect(await supervisor.status(first.launchId)).toEqual(completed);
expect((await supervisor.status(waiting.launchId))?.state).toBe("cancelled");
} finally {
await supervisor.close();
}
});
it("closes a retained worker and its server after its turn receipt is already complete", async () => {
const { supervisor, workspaceDir } = fixture({ capacity: 1 });
const input = testWorkerLaunchInput(workspaceDir, "preview-close", "background-start");
try {
const running = await supervisor.launch(input, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, input.launchId);
const background = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${input.launchId}.background.json`), "utf8"),
) as { pid: number; url: string };
const server = requireNodeWorkerProcessIdentity(background.pid);
expect(await (await fetch(background.url)).text()).toBe("preview-ready");
await supervisor.close();
expect(await supervisor.status(input.launchId)).toEqual(completed);
await vi.waitFor(() => {
expect(inspectNodeWorkerProcessIdentity(running.worker!)).not.toBe("live");
expect(inspectNodeWorkerProcessIdentity(server)).not.toBe("live");
});
await expect(fetch(background.url)).rejects.toThrow();
} finally {
await supervisor.close();
}
});
it("does not use a pruned first-turn receipt as authority over a later retained turn", async () => {
const { env, supervisor, workspaceDir } = fixture({ capacity: 1 });
const first = testWorkerLaunchInput(workspaceDir, "pruned-first", "background-start");
const next = testWorkerLaunchInput(workspaceDir, "current-second", "background-wait");
const turns = new NodeWorkerTurnStore({ env });
try {
await supervisor.launch(first, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, first.launchId);
const running = await supervisor.launch(next, TEST_WORKER_ENDPOINT);
await vi.waitFor(() =>
expect(fs.existsSync(path.join(workspaceDir, `${next.launchId}.started.json`))).toBe(true),
);
turns.claim({
claim: { ...testNodeWorkerLaunchIdentity(next), gatewayNamespace: next.gatewayNamespace },
ownerLaunchId: first.launchId,
supervisor: running.supervisor,
worker: running.worker,
nowMs: completed.completedAtMs! + 24 * 60 * 60 * 1_000 + 1,
});
expect(turns.get(first.launchId)).toBeUndefined();
expect(new NodeWorkerLaunchStore({ env }).get(first.launchId)).toMatchObject({
state: "running",
worker: running.worker,
});
expect(await supervisor.status(first.launchId)).toBeUndefined();
expect(await supervisor.cancel(testNodeWorkerLaunchIdentity(first))).toBeUndefined();
expect(await supervisor.status(next.launchId)).toMatchObject({ state: "running" });
expect(inspectNodeWorkerProcessIdentity(running.worker!)).toBe("live");
const background = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${first.launchId}.background.json`), "utf8"),
) as { url: string };
expect(await (await fetch(background.url)).text()).toBe("preview-ready");
await supervisor.cancel(testNodeWorkerLaunchIdentity(next));
await expect(supervisor.launch(first, TEST_WORKER_ENDPOINT)).rejects.toThrow();
expect(turns.get(first.launchId)).toBeUndefined();
expect(inspectNodeWorkerProcessIdentity(running.worker!)).toBe("live");
} finally {
await supervisor.close();
}
});
it.each([
"session",
"owner epoch",
"placement generation",
"agent",
"workspace",
"containment",
"permissions",
"bundle",
] as const)("retires a retained worker before replacing its %s binding", async (binding) => {
const { bundleRoot, root, supervisor, workspaceDir } = fixture({ capacity: 1 });
const first = testWorkerLaunchInput(workspaceDir, "binding-first", "background-start");
first.descriptor.assignment = {
...first.descriptor.assignment,
permissionMode: "full",
workerContainmentRoot: workspaceDir,
};
const next = structuredClone(first);
next.launchId = "binding-next";
next.descriptor.assignment.turnId = next.launchId;
switch (binding) {
case "session":
next.descriptor.admission.sessionId = "replacement-session";
break;
case "owner epoch":
next.descriptor.admission.ownerEpoch += 1;
break;
case "placement generation":
next.placementGeneration += 1;
break;
case "agent":
next.descriptor.assignment.agentId = "replacement-agent";
break;
case "workspace": {
const replacement = path.join(workspaceDir, "replacement");
fs.mkdirSync(replacement);
next.descriptor.assignment.workspaceDir = replacement;
break;
}
case "containment":
next.descriptor.assignment.workerContainmentRoot = root;
break;
case "permissions":
next.descriptor.assignment.permissionMode = "guarded";
break;
case "bundle": {
const hash = "b".repeat(64);
const bundle = path.join(bundleRoot, first.gatewayNamespace, "bundles", hash);
fs.mkdirSync(bundle);
fs.writeFileSync(path.join(bundle, "worker.mjs"), TEST_WORKER_SOURCE);
next.expectedBundleHash = hash;
next.descriptor.admission.handshake.bundleHash = hash;
break;
}
}
try {
const original = await supervisor.launch(first, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, first.launchId);
const background = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${first.launchId}.background.json`), "utf8"),
) as { pid: number; url: string };
const server = requireNodeWorkerProcessIdentity(background.pid);
const replacement = await supervisor.launch(next, TEST_WORKER_ENDPOINT);
expect(replacement.worker).not.toEqual(original.worker);
expect((await waitForTerminal(supervisor, next.launchId)).state).toBe("completed");
await vi.waitFor(() => {
expect(inspectNodeWorkerProcessIdentity(original.worker!)).not.toBe("live");
expect(inspectNodeWorkerProcessIdentity(server)).not.toBe("live");
});
await expect(fetch(background.url)).rejects.toThrow();
expect(await supervisor.status(first.launchId)).toEqual(completed);
if (binding === "owner epoch" || binding === "session") {
await supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(first));
expect(inspectNodeWorkerProcessIdentity(replacement.worker!)).toBe("live");
}
} finally {
await supervisor.close();
}
});
it.each(["owner epoch", "placement generation"] as const)(
"rejects an older %s without disturbing the retained worker",
async (binding) => {
const { supervisor, workspaceDir } = fixture({ capacity: 1 });
const first = testWorkerLaunchInput(workspaceDir, "current-owner", "background-start");
const stale = testWorkerLaunchInput(workspaceDir, "stale-owner", "background-poll");
if (binding === "owner epoch") {
stale.descriptor.admission.ownerEpoch -= 1;
} else {
stale.placementGeneration -= 1;
}
try {
const running = await supervisor.launch(first, TEST_WORKER_ENDPOINT);
await waitForTerminal(supervisor, first.launchId);
await expect(supervisor.launch(stale, TEST_WORKER_ENDPOINT)).rejects.toThrow(
"belongs to a replaced environment",
);
expect(inspectNodeWorkerProcessIdentity(running.worker!)).toBe("live");
const background = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${first.launchId}.background.json`), "utf8"),
) as { url: string };
expect(await (await fetch(background.url)).text()).toBe("preview-ready");
} finally {
await supervisor.close();
}
},
);
it.each(["environment stop", "supervisor close"] as const)(
"%s aborts admission behind a stalled retiring worker",
async (operation) => {
const { env, supervisor, workspaceDir } = fixture({ capacity: 2 });
const first = testWorkerLaunchInput(workspaceDir, "retiring-owner", "retire-stall");
const next = testWorkerLaunchInput(workspaceDir, "waiting-for-retirement", "wait");
const sibling = launchInput(workspaceDir, "outside-retiring-environment", "wait");
const store = new NodeWorkerLaunchStore({ env });
let owner: Awaited<ReturnType<NodeWorkerSupervisor["launch"]>> | undefined;
let admission: Promise<unknown> | undefined;
let shutdown: Promise<unknown> | undefined;
let admissionError: unknown;
let shutdownError: unknown;
let stopped = false;
try {
owner = await supervisor.launch(first, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, first.launchId);
const unrelated = await supervisor.launch(sibling, TEST_WORKER_ENDPOINT);
expect(store.get(first.launchId)?.state).toBe("running");
expect(inspectNodeWorkerProcessIdentity(owner.worker!)).toBe("live");
const readOwner = vi.spyOn(NodeWorkerLaunchStore.prototype, "get");
admission = supervisor.launch(next, TEST_WORKER_ENDPOINT).catch((error: unknown) => {
admissionError = error;
});
await vi.waitFor(() => expect(readOwner).toHaveBeenCalledWith(first.launchId));
readOwner.mockRestore();
const stopping =
operation === "environment stop"
? supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(first))
: supervisor.close();
shutdown = stopping.then(
() => {
stopped = true;
},
(error: unknown) => {
shutdownError = error;
},
);
await vi.waitFor(
() => {
expect(admissionError).toMatchObject({
message:
operation === "environment stop"
? "node worker environment stopped"
: "node worker supervisor is closed",
});
expect(shutdownError).toBeUndefined();
expect(stopped).toBe(true);
expect(inspectNodeWorkerProcessIdentity(owner!.worker!)).not.toBe("live");
},
{ timeout: 3_000 },
);
expect(store.get(first.launchId)?.state).toBe("interrupted");
expect(await supervisor.status(first.launchId)).toEqual(completed);
expect(await supervisor.status(next.launchId)).toBeUndefined();
expect(fs.existsSync(path.join(workspaceDir, `${next.launchId}.started.json`))).toBe(false);
if (operation === "environment stop") {
expect(inspectNodeWorkerProcessIdentity(unrelated.worker!)).toBe("live");
expect(await supervisor.status(sibling.launchId)).toMatchObject({ state: "running" });
await expect(supervisor.launch(next, TEST_WORKER_ENDPOINT)).resolves.toMatchObject({
state: "running",
});
} else {
expect(inspectNodeWorkerProcessIdentity(unrelated.worker!)).not.toBe("live");
expect(store.listNonterminal()).toEqual([]);
}
} finally {
// Break the injected retirement stall even when the pre-fix admission never aborts.
if (owner?.worker && inspectNodeWorkerProcessIdentity(owner.worker) === "live") {
process.kill(owner.worker.pid, "SIGKILL");
}
await Promise.allSettled([admission, shutdown]);
await supervisor.close();
}
},
);
});
@@ -10,12 +10,15 @@ import {
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../state/openclaw-state-db.js";
import { NodeWorkerCapacity } from "./node-worker-capacity.js";
import { NodeWorkerContainerLifecycle } from "./node-worker-container-lifecycle.js";
import { NodeWorkerLaunchStore, type NodeWorkerLaunchReceipt } from "./node-worker-launch-store.js";
import {
inspectNodeWorkerProcessIdentity,
requireNodeWorkerProcessIdentity,
type NodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
import { recoverNodeWorkerLaunch } from "./node-worker-supervisor-recovery.js";
import { createNodeWorkerSupervisor } from "./node-worker-supervisor.js";
import {
testNodeWorkerLaunchIdentity,
@@ -23,6 +26,7 @@ import {
testWorkerLaunchInput,
writeNodeWorkerFixture,
} from "./node-worker-supervisor.test-support.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
const spawned = new Set<ChildProcess>();
@@ -76,8 +80,10 @@ function insertLaunch(params: {
state: "pending" | "running";
supervisor: NodeWorkerProcessIdentity;
worker?: NodeWorkerProcessIdentity;
turn?: true;
}) {
const database = openOpenClawStateDatabase({ env: params.env }).db;
const state = params.turn ? "pending" : params.state;
database
.prepare(
`INSERT INTO node_worker_launches (
@@ -96,12 +102,30 @@ function insertLaunch(params: {
params.input.descriptor.admission.ownerEpoch,
params.input.placementGeneration,
params.input.descriptor.assignment.runId,
params.state,
state,
params.supervisor.pid,
params.supervisor.startTime,
params.worker?.pid ?? null,
params.worker?.startTime ?? null,
state === "running" ? (params.worker?.pid ?? null) : null,
state === "running" ? (params.worker?.startTime ?? null) : null,
);
if (params.turn) {
new NodeWorkerTurnStore({ env: params.env }).claim({
claim: {
...testNodeWorkerLaunchIdentity(params.input),
gatewayNamespace: params.input.gatewayNamespace,
},
ownerLaunchId: params.input.launchId,
supervisor: params.supervisor,
});
if (params.state === "running") {
new NodeWorkerLaunchStore({ env: params.env }).markRunning({
launchId: params.input.launchId,
planHash: planHash(params.input),
supervisor: params.supervisor,
worker: params.worker!,
});
}
}
}
function waitForChildLine(child: ChildProcess): Promise<string> {
@@ -261,6 +285,7 @@ describe("node worker supervisor recovery", () => {
input,
state: "pending",
supervisor: { pid: 2_147_483_647, startTime: 1 },
turn: true,
});
const capacitySnapshots: Array<{ total: number; available: number }> = [];
const supervisor = createNodeWorkerSupervisor({
@@ -316,6 +341,7 @@ describe("node worker supervisor recovery", () => {
state: "running",
supervisor: { pid: 2_147_483_647, startTime: 1 },
worker,
turn: true,
});
if (operation === "cancel") {
@@ -425,6 +451,7 @@ describe("node worker supervisor recovery", () => {
runId: input.descriptor.assignment.runId,
};
const storeUrl = pathToFileURL(path.resolve("src/node-host/node-worker-launch-store.ts")).href;
const turnsUrl = pathToFileURL(path.resolve("src/node-host/node-worker-turn-store.ts")).href;
const identityUrl = pathToFileURL(
path.resolve("src/node-host/node-worker-process-identity.ts"),
).href;
@@ -436,15 +463,22 @@ describe("node worker supervisor recovery", () => {
`
import fs from "node:fs";
import { NodeWorkerLaunchStore } from ${JSON.stringify(storeUrl)};
import { NodeWorkerTurnStore } from ${JSON.stringify(turnsUrl)};
import { requireNodeWorkerProcessIdentity } from ${JSON.stringify(identityUrl)};
const [stateDir, claimPath] = process.argv.slice(2);
const store = new NodeWorkerLaunchStore({ env: { ...process.env, OPENCLAW_STATE_DIR: stateDir } });
const env = { ...process.env, OPENCLAW_STATE_DIR: stateDir };
const store = new NodeWorkerLaunchStore({ env });
const claim = JSON.parse(fs.readFileSync(claimPath, "utf8"));
const supervisor = requireNodeWorkerProcessIdentity(process.pid);
const result = store.claim(
JSON.parse(fs.readFileSync(claimPath, "utf8")),
requireNodeWorkerProcessIdentity(process.pid),
claim,
supervisor,
2,
);
process.stdout.write(JSON.stringify(result.receipt) + "\\n");
const turn = new NodeWorkerTurnStore({ env }).claim({
claim, ownerLaunchId: result.receipt.launchId, supervisor,
});
process.stdout.write(JSON.stringify(turn.receipt) + "\\n");
setInterval(() => {}, 1000);
`,
);
@@ -464,4 +498,66 @@ describe("node worker supervisor recovery", () => {
await waitForChildExit(owner);
await second.close();
});
it.each(["pending", "running"] as const)(
"revalidates the %s physical owner after awaited container cleanup work",
async (state) => {
const { bundleRoot, env, workspaceDir } = fixture("node-worker-recovery-reread-");
const store = new NodeWorkerLaunchStore({ env });
store.get("schema-probe");
const input = testWorkerLaunchInput(workspaceDir, "recovery-reread");
const stale = { pid: 2_147_483_647, startTime: 1 };
const current = requireNodeWorkerProcessIdentity(process.pid);
insertLaunch({ env, input, state: "pending", supervisor: stale });
const engine = { id: "docker", command: process.execPath, target: "b".repeat(64) } as const;
const container = {
engine: engine.id,
containerId: "c".repeat(64),
engineTarget: engine.target,
} as const;
if (state === "running") {
store.markRunning({
launchId: input.launchId,
planHash: planHash(input),
supervisor: stale,
worker: current,
container,
});
}
const receipt = store.get(input.launchId)!;
const lifecycle = new NodeWorkerContainerLifecycle(engine, bundleRoot, store);
const replaceOwner = async () => {
await Promise.resolve();
openOpenClawStateDatabase({ env })
.db.prepare(
"UPDATE node_worker_launches SET supervisor_pid = ?, supervisor_start_time = ? WHERE launch_id = ?",
)
.run(current.pid, current.startTime, input.launchId);
};
const initialize = vi.spyOn(lifecycle, "initialize").mockImplementation(replaceOwner);
const inspect = vi.spyOn(lifecycle, "inspect").mockImplementation(async () => {
await replaceOwner();
return "live";
});
const remove = vi.spyOn(lifecycle, "remove").mockResolvedValue(undefined);
try {
await expect(
recoverNodeWorkerLaunch({
receipt,
store,
capacity: new NodeWorkerCapacity(store, { capacity: 1 }),
containerLifecycle: lifecycle,
notifyCapacity: true,
state: "cancelled",
}),
).resolves.toMatchObject({ state, supervisor: current });
expect(remove).not.toHaveBeenCalled();
expect(store.nonterminalCount()).toBe(1);
} finally {
initialize.mockRestore();
inspect.mockRestore();
remove.mockRestore();
}
},
);
});
@@ -23,17 +23,16 @@ export const TEST_WORKER_SOURCE = String.raw`
import fs from "node:fs";
import path from "node:path";
import { spawn } from "node:child_process";
let input = "";
for await (const chunk of process.stdin) input += chunk;
const descriptor = JSON.parse(input);
if (descriptor.assignment.prompt === "exit-before-start") {
fs.writeFileSync(path.join(descriptor.assignment.workspaceDir, "prestart-exited"), "exited");
process.exit(23);
}
if (!process.connected || !process.channel || !process.argv.includes("--internal-worker-ipc")) {
import { createInterface } from "node:readline";
if (!process.connected || !process.channel ||
!process.argv.includes("--internal-worker-ipc") ||
!process.argv.includes("--internal-worker-session")) {
process.exit(24);
}
let grandchild;
let background;
let retained = false;
let currentTurn;
let disposed = false;
let started = false;
let resolveStart;
@@ -79,18 +78,32 @@ const exitWorker = (code) => {
if (process.connected) process.disconnect();
process.exit(code);
};
const writeResultAndExit = (value) => {
fs.writeSync(1, value);
exitWorker(0);
const completedResult = { status: "completed", transcriptLeafId: "leaf-1", transcriptNextSeq: 2 };
const finish = (descriptor, result = completedResult, retainWorker = retained) => {
if (currentTurn !== descriptor) return;
fs.writeSync(1, JSON.stringify({
type: "result", turnId: descriptor.assignment.turnId, result, retainWorker,
}) + "\n");
currentTurn = undefined;
retained = retainWorker;
if (!retainWorker && descriptor.assignment.prompt !== "retire-stall") exitWorker(0);
};
const writeArtifact = (descriptor, name, value) => fs.writeFileSync(
path.join(descriptor.assignment.workspaceDir, descriptor.assignment.turnId + "." + name + ".json"),
JSON.stringify(value),
);
const runTurn = async (descriptor) => {
const startedPath = path.join(descriptor.assignment.workspaceDir, descriptor.assignment.turnId + ".started.json");
const starts = fs.existsSync(startedPath) ? JSON.parse(fs.readFileSync(startedPath, "utf8")).starts + 1 : 1;
writeArtifact(descriptor, "started", { pid: process.pid, starts });
const mode = descriptor.assignment.prompt;
if (mode === "admission-rearm") {
const marker = path.join(descriptor.assignment.workspaceDir, "admission-attempt");
const first = !fs.existsSync(marker);
fs.writeFileSync(marker, descriptor.assignment.turnId);
writeResultAndExit(JSON.stringify(first
finish(descriptor, first
? { status: "not-started", reason: "admission-deadline", errorText: "gateway unreachable " + descriptor.admission.credential }
: { status: "completed", transcriptLeafId: "leaf-1", transcriptNextSeq: 2 }) + "\n");
: completedResult);
} else if (mode === "connection-failure" || mode === "connection-deadline") {
const target = new URL(descriptor.connectionEndpoint.url).host;
const report = (cause) => new Promise((resolve) => process.send(
@@ -108,11 +121,41 @@ if (mode === "admission-rearm") {
setInterval(() => {}, 1000);
}
} else if (mode === "wait") {
setInterval(() => {}, 1000);
return;
} else if (mode === "tree") {
grandchild = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" });
fs.writeFileSync(path.join(descriptor.assignment.workspaceDir, "grandchild.pid"), String(grandchild.pid));
setInterval(() => {}, 1000);
} else if (mode === "background-start") {
grandchild = spawn(process.execPath, ["-e", [
"const server = require('node:http').createServer((req, res) => res.end('preview-ready'));",
"server.listen(0, '127.0.0.1', () => process.stdout.write(JSON.stringify({",
"pid: process.pid, url: 'http://127.0.0.1:' + server.address().port,",
"}) + '\\n'));",
].join("\n")], { stdio: ["ignore", "pipe", "inherit"] });
background = await new Promise((resolve, reject) => {
grandchild.once("error", reject);
const output = createInterface({ input: grandchild.stdout });
output.once("line", (line) => { output.close(); resolve(JSON.parse(line)); });
});
writeArtifact(descriptor, "background", background);
finish(descriptor, completedResult, true);
} else if (mode === "background-poll") {
if (!background) throw new Error("background process was not retained");
const response = await fetch(background.url);
writeArtifact(descriptor, "background", { ...background, response: await response.text() });
finish(descriptor, completedResult, true);
} else if (mode === "background-wait") {
if (!background) throw new Error("background process was not retained");
return;
} else if (mode === "diagnostic-retain") {
fs.writeSync(2, "previous turn stderr " + descriptor.admission.credential + "\n");
await new Promise((resolve) => process.send({
type: "openclaw-worker-connection-failure-v1",
cause: "previous turn connection " + descriptor.admission.credential,
}, resolve));
finish(descriptor, completedResult, true);
} else if (mode === "quiet-fail") {
exitWorker(7);
} else if (mode === "secret-fail") {
await new Promise((resolve) => setTimeout(resolve, 500));
const credential = descriptor.admission.credential;
@@ -135,11 +178,13 @@ if (mode === "admission-rearm") {
} else if (mode === "secret-success") {
await new Promise((resolve) => setTimeout(resolve, 500));
const credential = descriptor.admission.credential;
writeResultAndExit(
JSON.stringify({ raw: credential, encoded: encodeURIComponent(credential), status: "completed" }) + "\n",
);
finish(descriptor, {
...completedResult,
transcriptLeafId: "raw " + credential + " encoded " + encodeURIComponent(credential),
});
} else if (mode === "overflow") {
writeResultAndExit("x".repeat(70 * 1024));
fs.writeSync(1, "x".repeat(70 * 1024));
exitWorker(0);
} else if (mode === "fast-terminal") {
const marker = path.join(descriptor.assignment.workspaceDir, "fast-terminal-marker");
process.once("SIGTERM", () => {
@@ -148,13 +193,45 @@ if (mode === "admission-rearm") {
});
await new Promise((resolve) => setTimeout(resolve, 100));
fs.writeFileSync(marker, "normal");
writeResultAndExit(JSON.stringify({ status: "completed" }) + "\n");
finish(descriptor);
} else if (mode === "env") {
writeResultAndExit(JSON.stringify(process.env) + "\n");
writeArtifact(descriptor, "env", process.env);
finish(descriptor);
} else {
await new Promise((resolve) => setTimeout(resolve, 25));
writeResultAndExit(JSON.stringify({ argv: process.argv.slice(2), status: "completed" }) + "\n");
writeArtifact(descriptor, "argv", process.argv.slice(2));
finish(descriptor);
}
};
const lines = createInterface({ input: process.stdin, crlfDelay: Infinity });
lines.on("line", (line) => {
const request = JSON.parse(line);
if (request.type === "cancel") {
const descriptor = currentTurn;
if (!descriptor || descriptor.assignment.turnId !== request.turnId) return;
const settle = () => finish(descriptor, {
status: "failed", reason: "turn-failed", transcriptLeafId: null, transcriptNextSeq: 1,
}, Boolean(background));
if (grandchild && !background) {
grandchild.once("exit", settle);
grandchild.kill("SIGKILL");
} else {
settle();
}
return;
}
if (currentTurn || request.type !== "turn" ||
request.turnId !== request.descriptor.assignment.turnId) {
process.stderr.write("invalid or concurrent managed turn\n");
exitWorker(25);
}
currentTurn = request.descriptor;
void runTurn(currentTurn).catch((error) => {
process.stderr.write(error.message + "\n");
hardTerminate();
});
});
lines.once("close", () => { if (!disposed) hardTerminate(); });
`;
export function testWorkerDescriptor(
@@ -209,6 +286,15 @@ export function testNodeWorkerLaunchIdentity(
};
}
export function testNodeWorkerEnvironmentIdentity(input: NodeWorkerLaunchInput) {
return {
gatewayNamespace: input.gatewayNamespace,
environmentId: input.descriptor.admission.environmentId,
sessionId: input.descriptor.admission.sessionId,
ownerEpoch: input.descriptor.admission.ownerEpoch,
};
}
export function writeNodeWorkerFixture(root: string) {
const stateDir = path.join(root, "state-root");
const bundleRoot = path.join(root, "bundles-root");
@@ -226,6 +312,7 @@ export function testWorkerLaunchInput(
prompt = "success",
): NodeWorkerLaunchInput {
return {
environmentSession: 1,
launchId,
gatewayNamespace: "gateway-1",
expectedBundleHash: TEST_BUNDLE_HASH,
+184 -39
View File
@@ -1,4 +1,4 @@
import childProcess from "node:child_process";
import childProcess, { type ChildProcess } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
@@ -28,6 +28,7 @@ import {
testWorkerLaunchInput,
writeNodeWorkerFixture,
} from "./node-worker-supervisor.test-support.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
type NodeWorkerSupervisor = ReturnType<typeof createNodeWorkerSupervisor>;
@@ -53,7 +54,10 @@ function fixture(
}
function launchInput(workspaceDir: string, launchId: string, prompt = "success") {
return testWorkerLaunchInput(workspaceDir, launchId, prompt);
const input = testWorkerLaunchInput(workspaceDir, launchId, prompt);
input.descriptor.admission.environmentId = `environment-${launchId}`;
input.descriptor.admission.sessionId = `session-${launchId}`;
return input;
}
async function waitForTerminal(supervisor: NodeWorkerSupervisor, launchId: string) {
@@ -152,36 +156,23 @@ describe("node worker supervisor", () => {
});
it("keeps pending and running launches owned by a live supervisor unchanged", async () => {
const { bundleRoot, env, supervisor } = fixture();
const { bundleRoot, env, supervisor, workspaceDir } = fixture();
await supervisor.status("schema-probe");
const database = openOpenClawStateDatabase({ env }).db;
const supervisorIdentity = requireNodeWorkerProcessIdentity(process.pid);
const insert = database.prepare(`
INSERT INTO node_worker_launches (
launch_id, plan_hash, gateway_namespace, environment_id, session_id,
owner_epoch, placement_generation, run_id, state,
supervisor_pid, supervisor_start_time, worker_pid, worker_start_time,
result_json, error_text, completed_at_ms, created_at_ms, updated_at_ms
) VALUES (?, ?, 'gateway-1', 'environment-1', 'session-1', 3, 4, 'run-1', ?, ?, ?, ?, ?, NULL, NULL, NULL, 1, 1)
`);
insert.run(
"pending-launch",
"b".repeat(64),
"pending",
supervisorIdentity.pid,
supervisorIdentity.startTime,
null,
null,
);
insert.run(
"running-launch",
"c".repeat(64),
"running",
supervisorIdentity.pid,
supervisorIdentity.startTime,
process.pid,
supervisorIdentity.startTime,
);
const store = new NodeWorkerLaunchStore({ env });
const turns = new NodeWorkerTurnStore({ env });
for (const launchId of ["pending-launch", "running-launch"]) {
const input = launchInput(workspaceDir, launchId, "wait");
const claim = {
...testNodeWorkerLaunchIdentity(input),
gatewayNamespace: input.gatewayNamespace,
};
store.claim(claim, supervisorIdentity, 2);
turns.claim({ claim, ownerLaunchId: launchId, supervisor: supervisorIdentity });
if (launchId === "running-launch") {
store.markRunning({ ...claim, supervisor: supervisorIdentity, worker: supervisorIdentity });
}
}
const capacitySnapshots: Array<{ total: number; available: number }> = [];
const sameHandle = createNodeWorkerSupervisor({
@@ -234,6 +225,37 @@ describe("node worker supervisor", () => {
}
});
it("releases the physical capacity claim when the first turn cannot be journaled", async () => {
const capacities: Array<{ total: number; available: number }> = [];
const { env, supervisor, workspaceDir } = fixture({
capacity: 1,
capacityWaitMs: 25,
onCapacityChanged: (capacity) => capacities.push(capacity),
});
const input = launchInput(workspaceDir, "turn-claim-failure");
const claim = vi.spyOn(NodeWorkerTurnStore.prototype, "claim").mockImplementationOnce(() => {
throw new Error("injected turn claim failure");
});
try {
await expect(supervisor.launch(input, TEST_WORKER_ENDPOINT)).rejects.toThrow(
"injected turn claim failure",
);
expect(new NodeWorkerLaunchStore({ env }).get(input.launchId)).toMatchObject({
state: "failed",
worker: null,
});
expect(capacities.at(-1)).toEqual({ total: 1, available: 1 });
expect(fs.existsSync(path.join(workspaceDir, `${input.launchId}.started.json`))).toBe(false);
const next = launchInput(workspaceDir, "turn-claim-recovered");
await supervisor.launch(next, TEST_WORKER_ENDPOINT);
expect((await waitForTerminal(supervisor, next.launchId)).state).toBe("completed");
} finally {
claim.mockRestore();
await supervisor.close();
}
});
it("launches idempotently and persists only bounded non-secret facts", async () => {
const { env, supervisor, workspaceDir } = fixture();
const input = launchInput(workspaceDir, "success-launch");
@@ -241,8 +263,8 @@ describe("node worker supervisor", () => {
expect(await supervisor.launch(input, TEST_WORKER_ENDPOINT)).toMatchObject({
launchId: "success-launch",
state: "running",
environmentId: "environment-1",
sessionId: "session-1",
environmentId: input.descriptor.admission.environmentId,
sessionId: input.descriptor.admission.sessionId,
ownerEpoch: 3,
placementGeneration: 4,
runId: "run-1",
@@ -250,9 +272,13 @@ describe("node worker supervisor", () => {
const completed = await waitForTerminal(supervisor, input.launchId);
expect(completed).toMatchObject({ state: "completed", errorText: null });
expect(JSON.parse(completed.resultJson ?? "null")).toEqual({
argv: ["--internal-worker-ipc"],
status: "completed",
transcriptLeafId: "leaf-1",
transcriptNextSeq: 2,
});
expect(
JSON.parse(fs.readFileSync(path.join(workspaceDir, `${input.launchId}.argv.json`), "utf8")),
).toEqual(["--internal-worker-ipc", "--internal-worker-session"]);
expect(await supervisor.launch(input, TEST_WORKER_ENDPOINT)).toEqual(completed);
await expect(
supervisor.launch(
@@ -484,8 +510,10 @@ describe("node worker supervisor", () => {
suppliedEnv.LANG = "mutated-locale";
const input = launchInput(workspaceDir, "env-launch", "env");
await supervisor.launch(input, TEST_WORKER_ENDPOINT);
const completed = await waitForTerminal(supervisor, input.launchId);
const workerEnv = JSON.parse(completed.resultJson ?? "null") as Record<string, string>;
await waitForTerminal(supervisor, input.launchId);
const workerEnv = JSON.parse(
fs.readFileSync(path.join(workspaceDir, `${input.launchId}.env.json`), "utf8"),
) as Record<string, string>;
expect(workerEnv).toMatchObject(expectedWorkerEnv);
expect(workerEnv).not.toHaveProperty("AMBIENT_SECRET");
@@ -538,9 +566,9 @@ describe("node worker supervisor", () => {
];
expect(success.state).toBe("completed");
expect(JSON.parse(success.resultJson ?? "null")).toEqual({
raw: "[REDACTED]",
encoded: "[REDACTED]",
status: "completed",
transcriptLeafId: "raw [REDACTED] encoded [REDACTED]",
transcriptNextSeq: 2,
});
expect(failure.state).toBe("failed");
expect(Buffer.byteLength(failure.errorText ?? "", "utf8")).toBeLessThanOrEqual(4 * 1024);
@@ -576,6 +604,43 @@ describe("node worker supervisor", () => {
},
);
it("rotates credential scrubbing and drops prior-turn diagnostics when a worker is reused", async () => {
const { supervisor, workspaceDir } = fixture({ capacity: 1 });
const first = testWorkerLaunchInput(workspaceDir, "previous-diagnostic", "diagnostic-retain");
const second = testWorkerLaunchInput(workspaceDir, "rotated-credential", "secret-success");
second.descriptor.admission.credential = 'fresh worker/"credential\\secret?';
const last = testWorkerLaunchInput(workspaceDir, "fresh-failure", "quiet-fail");
last.descriptor.admission.credential = "final-worker-credential";
try {
const original = await supervisor.launch(first, TEST_WORKER_ENDPOINT);
await waitForTerminal(supervisor, first.launchId);
expect(await supervisor.launch(second, TEST_WORKER_ENDPOINT)).toMatchObject({
worker: original.worker,
});
for (let index = 0; index < 600; index += 1) {
registerSecretValueForRedaction(`rotated-eviction-secret-${index}`);
}
const completed = await waitForTerminal(supervisor, second.launchId);
expect(JSON.parse(completed.resultJson ?? "null")).toEqual({
status: "completed",
transcriptLeafId: "raw [REDACTED] encoded [REDACTED]",
transcriptNextSeq: 2,
});
await supervisor.launch(last, TEST_WORKER_ENDPOINT);
const failed = await waitForTerminal(supervisor, last.launchId);
expect(failed).toMatchObject({
state: "failed",
errorText: "node worker failed with exit code 7",
});
for (const input of [first, second, last]) {
expect(JSON.stringify(failed)).not.toContain(input.descriptor.admission.credential);
}
} finally {
await supervisor.close();
}
});
it("does not open or signal a child after markRunning observes its terminal receipt", async () => {
const { supervisor, workspaceDir } = fixture();
const input = launchInput(workspaceDir, "fast-terminal-launch", "fast-terminal");
@@ -604,9 +669,19 @@ describe("node worker supervisor", () => {
});
it("records a gated child that exits before journal readiness as terminal", async () => {
const { supervisor, workspaceDir } = fixture();
const input = launchInput(workspaceDir, "prestart-exit-launch", "exit-before-start");
const { bundleRoot, supervisor, workspaceDir } = fixture();
const input = launchInput(workspaceDir, "prestart-exit-launch");
const exitedPath = path.join(workspaceDir, "prestart-exited");
fs.writeFileSync(
path.join(
bundleRoot,
input.gatewayNamespace,
"bundles",
input.expectedBundleHash,
"worker.mjs",
),
`import fs from "node:fs"; fs.writeFileSync(${JSON.stringify(exitedPath)}, "exited"); process.exit(23);`,
);
await supervisor.launch(input, TEST_WORKER_ENDPOINT);
const terminal = await waitForTerminal(supervisor, input.launchId);
@@ -639,6 +714,76 @@ describe("node worker supervisor", () => {
await supervisor.close();
});
it("bounds a blocked cancellation write and stops only its physical owner", async () => {
const capacities: Array<{ total: number; available: number }> = [];
const { env, supervisor, workspaceDir } = fixture({
capacity: 2,
capacityWaitMs: 25,
onCapacityChanged: (capacity) => capacities.push(capacity),
});
const input = launchInput(workspaceDir, "blocked-cancel", "wait");
const sibling = launchInput(workspaceDir, "unrelated-worker", "wait");
const captureSpawn = vi.spyOn(childProcess.ChildProcess.prototype, "emit");
let heldWrite: { data: unknown; callback?: (error?: Error | null) => void } | undefined;
let restoreWrite: (() => void) | undefined;
let cancellation: ReturnType<NodeWorkerSupervisor["cancel"]> | undefined;
let cancelled: Awaited<ReturnType<NodeWorkerSupervisor["cancel"]>>;
try {
const running = await supervisor.launch(input, TEST_WORKER_ENDPOINT);
const unrelated = await supervisor.launch(sibling, TEST_WORKER_ENDPOINT);
const child = captureSpawn.mock.contexts.find(
(context): context is ChildProcess =>
context instanceof childProcess.ChildProcess && context.pid === running.worker!.pid,
);
captureSpawn.mockRestore();
const stdin = child?.stdin;
if (!stdin) {
throw new Error("missing spawned worker stdin");
}
// Model a pipe that cannot drain: neither frame delivery nor write completion occurs.
const write = vi.spyOn(stdin, "write").mockImplementation((data, encoding, callback) => {
heldWrite = { data, callback: typeof encoding === "function" ? encoding : callback };
return false;
});
restoreWrite = () => write.mockRestore();
cancellation = supervisor.cancel(testNodeWorkerLaunchIdentity(input)).then((receipt) => {
cancelled = receipt;
return receipt;
});
await vi.waitFor(() => {
expect(heldWrite?.data).toBe(
`${JSON.stringify({ type: "cancel", turnId: input.launchId })}\n`,
);
expect(heldWrite?.callback).toEqual(expect.any(Function));
});
await vi.waitFor(
() => {
expect(cancelled).toMatchObject({ state: "cancelled", worker: running.worker });
expect(inspectNodeWorkerProcessIdentity(running.worker!)).not.toBe("live");
expect(capacities.at(-1)).toEqual({ total: 2, available: 1 });
},
{ timeout: 7_000, interval: 25 },
);
expect(new NodeWorkerLaunchStore({ env }).get(input.launchId)?.state).toBe("cancelled");
expect(await supervisor.status(sibling.launchId)).toMatchObject({ state: "running" });
expect(inspectNodeWorkerProcessIdentity(unrelated.worker!)).toBe("live");
await expect(
supervisor.launch(
launchInput(workspaceDir, "after-blocked-cancel", "wait"),
TEST_WORKER_ENDPOINT,
),
).resolves.toMatchObject({ state: "running" });
} finally {
captureSpawn.mockRestore();
restoreWrite?.();
// Release the injected write even on the pre-fix failure, so cleanup cannot inherit its hang.
heldWrite?.callback?.(new Error("released blocked test stdin"));
await cancellation?.catch(() => undefined);
await supervisor.close();
}
}, 15_000);
it.each([
[
"connection-failure",
+298 -291
View File
@@ -1,11 +1,14 @@
import path from "node:path";
import { resolveStateDir } from "../config/paths.js";
import { registerSecretValueForRedaction } from "../logging/secret-redaction-registry.js";
import { withTimeout } from "../infra/fs-safe.js";
import {
completeWorkerLaunchDescriptor,
parseWorkerLaunchPlan,
type WorkerLaunchDescriptor,
} from "../worker/launch-descriptor.js";
import {
validateNodeWorkerLaunchInput,
type NodeWorkerEnvironmentStopInput,
} from "../worker/node-supervisor-protocol.js";
import type {
NodeWorkerWorkspaceRetainInput,
NodeWorkerWorkspaceRetainResult,
@@ -19,35 +22,26 @@ import {
observeNodeWorkerChildOutput,
type NodeWorkerTerminalOutcome,
} from "./node-worker-launch-observation.js";
import {
NodeWorkerLaunchStore,
type NodeWorkerContainerIdentity,
type NodeWorkerLaunchReceipt,
} from "./node-worker-launch-store.js";
import {
prepareNodeWorkerLaunchTransport,
startNodeWorkerLaunchTransport,
type NodeWorkerChildAdapter,
} from "./node-worker-launch-transport.js";
import {
createNodeWorkerCredentialScrubber,
sanitizeNodeWorkerDiagnostic,
} from "./node-worker-output.js";
import { NodeWorkerLaunchStore, type NodeWorkerLaunchReceipt } from "./node-worker-launch-store.js";
import { sendNodeWorkerInput } from "./node-worker-launch-transport.js";
import { startNodeWorkerChild } from "./node-worker-launch.js";
import {
inspectNodeWorkerProcessIdentity,
requireNodeWorkerProcessIdentity,
type NodeWorkerProcessIdentity,
} from "./node-worker-process-identity.js";
import {
assertNodeWorkerLaunchIdentity,
nodeWorkerPlanHash,
type NodeWorkerLaunchInput,
type NodeWorkerSupervisorIdentity,
} from "./node-worker-supervisor-contract.js";
import {
createNodeWorkerJournalGate,
nodeWorkerEnvironmentBinding,
nodeWorkerEnvironmentKey,
nodeWorkerEnvironmentMatches,
nodeWorkerReceiptMatchesOwner,
type NodeWorkerActiveOwnership,
type NodeWorkerEnvironmentBinding,
type NodeWorkerObservedTerminal,
type NodeWorkerRunningChild,
type NodeWorkerStopState,
@@ -59,12 +53,16 @@ import {
signalOwnedNodeWorkerTree,
waitForOwnedNodeWorkerTreeDeath,
} from "./node-worker-tree-control.js";
import {
settleNodeWorkerTurn,
startNodeWorkerTurn,
waitForNodeWorkerRetirement,
} from "./node-worker-turn-lifecycle.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
import { NodeWorkerWorkspaceRuntime } from "./node-worker-workspace.js";
const STOP_GRACE_MS = 1_000;
const FORCE_STOP_WAIT_MS = 4_000;
const GATEWAY_NAMESPACE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/u;
const BUNDLE_HASH_PATTERN = /^[a-f0-9]{64}$/u;
/** Owns worker process groups, lifetime gates, and the durable node-host launch journal. */
class NodeWorkerSupervisor {
@@ -72,6 +70,18 @@ class NodeWorkerSupervisor {
private readonly starting = new Map<string, Promise<NodeWorkerLaunchReceipt>>();
private readonly bundleRoot: string;
private readonly store: NodeWorkerLaunchStore;
private readonly turns: NodeWorkerTurnStore;
private readonly admissions = new Map<
string,
{
binding: NodeWorkerEnvironmentBinding;
launchId: string;
planHash: string;
abort: AbortController;
done: Promise<NodeWorkerLaunchReceipt>;
}
>();
private readonly stoppingEnvironments = new Map<string, number>();
private readonly workerEnv: NodeJS.ProcessEnv;
private readonly engineEnv: NodeJS.ProcessEnv;
private readonly capacity: NodeWorkerCapacity;
@@ -90,6 +100,7 @@ class NodeWorkerSupervisor {
options.bundleRoot ?? path.join(resolveStateDir(env), "node-host"),
);
this.store = new NodeWorkerLaunchStore({ env });
this.turns = new NodeWorkerTurnStore({ env });
this.workerEnv = snapshotNodeWorkerEnv(env);
this.engineEnv = { ...process.env, ...env };
this.containerEngine = options.containerEngine;
@@ -131,40 +142,53 @@ class NodeWorkerSupervisor {
}
async launch(
input: NodeWorkerLaunchInput,
rawInput: NodeWorkerLaunchInput,
connectionEndpoint: WorkerConnectionEndpoint,
signal?: AbortSignal,
): Promise<NodeWorkerLaunchReceipt> {
if (!GATEWAY_NAMESPACE_PATTERN.test(input.gatewayNamespace)) {
throw new Error("gateway namespace must be a safe bounded path component");
}
if (!BUNDLE_HASH_PATTERN.test(input.expectedBundleHash)) {
throw new Error("node worker bundle hash must be 64 lowercase hexadecimal characters");
}
if (!Number.isSafeInteger(input.placementGeneration) || input.placementGeneration < 0) {
throw new Error("node worker placement generation must be a non-negative safe integer");
}
const plan = parseWorkerLaunchPlan(structuredClone(input.descriptor));
const descriptor = completeWorkerLaunchDescriptor(plan, connectionEndpoint);
assertNodeWorkerLaunchIdentity(input, descriptor);
const input = validateNodeWorkerLaunchInput(structuredClone(rawInput));
const descriptor = completeWorkerLaunchDescriptor(input.descriptor, connectionEndpoint);
const planHash = nodeWorkerPlanHash(input);
if (this.closed) {
throw new Error("node worker supervisor is closed");
}
await this.initialize();
const local = this.active.get(input.launchId);
if (local) {
if (local.planHash !== planHash) {
throw new Error(`node worker launch ${input.launchId} was replayed with a different plan`);
const binding = nodeWorkerEnvironmentBinding(input);
const key = nodeWorkerEnvironmentKey(binding);
if (this.stoppingEnvironments.has(key)) {
throw new Error("node worker environment is stopping");
}
const admission = this.admissions.get(key);
if (admission) {
if (admission.launchId !== input.launchId || admission.planHash !== planHash) {
throw new Error("node worker environment already has a turn being admitted");
}
if (local.state === "observed") {
return this.reconcileActiveTerminal(local);
}
const receipt = this.store.get(input.launchId);
if (receipt) {
return receipt;
return await admission.done;
}
const abort = new AbortController();
const done = this.launchAdmitted(
input,
descriptor,
planHash,
signal ? AbortSignal.any([signal, abort.signal]) : abort.signal,
);
const pending = { binding, launchId: input.launchId, planHash, abort, done };
this.admissions.set(key, pending);
try {
return await done;
} finally {
if (this.admissions.get(key) === pending) {
this.admissions.delete(key);
}
}
}
private async launchAdmitted(
input: NodeWorkerLaunchInput,
descriptor: WorkerLaunchDescriptor,
planHash: string,
signal: AbortSignal,
): Promise<NodeWorkerLaunchReceipt> {
await this.initialize();
const supervisor = (this.supervisorIdentity ??= requireNodeWorkerProcessIdentity(process.pid));
const claimInput = {
launchId: input.launchId,
@@ -179,17 +203,83 @@ class NodeWorkerSupervisor {
if (this.closed) {
throw new Error("node worker supervisor is closed");
}
signal.throwIfAborted();
const previous = this.turns.get(input.launchId);
if (previous) {
this.turns.claim({
claim: claimInput,
ownerLaunchId: previous.ownerLaunchId,
supervisor: previous.supervisor,
worker: previous.worker,
});
return (await this.status(input.launchId)) ?? previous;
}
const binding = nodeWorkerEnvironmentBinding(input);
for (const owner of this.active.values()) {
if (nodeWorkerEnvironmentKey(owner.binding) !== nodeWorkerEnvironmentKey(binding)) {
continue;
}
if (owner.state === "observed") {
this.reconcileActiveTerminal(owner);
continue;
}
await this.statusOwner(owner.launchId);
await waitForNodeWorkerRetirement(owner, signal);
signal.throwIfAborted();
if (this.active.get(owner.launchId) !== owner) {
continue;
}
if (owner.turn) {
throw new Error("node worker environment already has an active turn");
}
if (owner.stopState || owner.retiring) {
throw new Error("node worker environment cleanup is incomplete");
}
if (JSON.stringify(owner.binding) !== JSON.stringify(binding)) {
if (
binding.ownerEpoch < owner.binding.ownerEpoch ||
(binding.ownerEpoch === owner.binding.ownerEpoch &&
binding.placementGeneration < owner.binding.placementGeneration)
) {
throw new Error("node worker launch belongs to a replaced environment");
}
await this.stopChild(owner, "interrupted");
if (this.active.get(owner.launchId) === owner) {
throw new Error("node worker environment cleanup is incomplete");
}
signal.throwIfAborted();
continue;
}
return await startNodeWorkerTurn({
active: owner,
descriptor,
claim: claimInput,
signal,
store: this.turns,
cancel: (expected) => this.cancel(expected),
stopChild: (active, state) => this.stopChild(active, state),
});
}
const claim = await this.capacity.claim(claimInput, supervisor, signal);
if (claim.action === "recover") {
return await this.recoverRunning(claim.receipt);
await this.recoverRunning(claim.receipt);
}
if (claim.action === "replay") {
const replay = this.active.get(input.launchId);
if (replay?.planHash === planHash && replay.state === "observed") {
return this.reconcileActiveTerminal(replay);
}
const startup = this.starting.get(input.launchId);
return startup && claim.receipt.state === "pending" ? await startup : claim.receipt;
if (claim.action !== "start") {
// A pruned turn can share the first launch's ID. Its physical anchor is
// cleanup authority, never a substitute receipt for that expired turn.
throw new Error("node worker turn receipt expired; request a fresh turn");
}
try {
this.turns.claim({ claim: claimInput, ownerLaunchId: input.launchId, supervisor });
} catch (error) {
this.capacity.finish({
...claimInput,
supervisor,
worker: null,
state: "failed",
errorText: "node worker turn could not be journaled",
});
throw error;
}
let cancellation: Promise<NodeWorkerLaunchReceipt | undefined> | undefined;
const cancelClaimed = () => {
@@ -197,7 +287,32 @@ class NodeWorkerSupervisor {
void cancellation.catch(() => undefined);
};
signal?.addEventListener("abort", cancelClaimed, { once: true });
const startup = this.startClaimed({ input, descriptor, planHash, supervisor, signal });
const startup = startNodeWorkerChild(
{
bundleRoot: this.bundleRoot,
workerEnv: this.workerEnv,
engineEnv: this.engineEnv,
store: this.store,
turns: this.turns,
capacity: this.capacity,
containerEngine: this.containerEngine,
containerImage: this.containerImage,
containerLifecycle: this.containerLifecycle,
requireContainerLifecycle: () => this.requireContainerLifecycle(),
active: this.active,
isClosed: () => this.closed,
observeChild: (active) => this.observeChild(active),
stopChild: (active, state) => this.stopChild(active, state),
},
{
input,
descriptor,
planHash,
supervisor,
signal,
claim: claimInput,
},
);
this.starting.set(input.launchId, startup);
if (signal?.aborted) {
cancelClaimed();
@@ -214,6 +329,22 @@ class NodeWorkerSupervisor {
}
async status(launchId: string): Promise<NodeWorkerLaunchReceipt | undefined> {
await this.initialize();
const turn = this.turns.get(launchId);
if (turn) {
if (
this.active.get(turn.ownerLaunchId)?.state === "observed" ||
turn.state === "pending" ||
turn.state === "running"
) {
await this.statusOwner(turn.ownerLaunchId);
}
return this.turns.get(launchId);
}
return undefined;
}
private async statusOwner(launchId: string): Promise<NodeWorkerLaunchReceipt | undefined> {
await this.initialize();
const active = this.active.get(launchId);
if (active?.state === "observed") {
@@ -287,6 +418,97 @@ class NodeWorkerSupervisor {
async cancel(
expected: NodeWorkerSupervisorIdentity,
): Promise<NodeWorkerLaunchReceipt | undefined> {
const claimed = this.turns.getMatching(expected);
const claimedOwner = claimed && this.active.get(claimed.ownerLaunchId);
if (
claimedOwner?.state === "running" &&
claimedOwner.turn?.claim.launchId === expected.launchId
) {
// Close admission synchronously: cancellation can arrive inside markRunning,
// before its continuation opens the child's start gate.
claimedOwner.turn.cancelled = true;
}
await this.initialize();
const receipt = this.turns.getMatching(expected);
if (!receipt) {
return undefined;
}
if (receipt.state !== "pending" && receipt.state !== "running") {
return await this.status(receipt.launchId);
}
const active = this.active.get(receipt.ownerLaunchId);
if (active?.state !== "running" || active.turn?.claim.launchId !== expected.launchId) {
const owner = this.store.get(receipt.ownerLaunchId);
if (owner) {
await this.cancelOwner(owner);
}
return this.turns.getMatching(expected);
}
const turn = active.turn;
turn.cancelled = true;
try {
// A worker that stopped reading can block the write as well as the reply.
await withTimeout(
sendNodeWorkerInput(active.adapter, { type: "cancel", turnId: expected.launchId }).then(
() => turn.done,
),
STOP_GRACE_MS + FORCE_STOP_WAIT_MS,
{ message: "node worker turn cancellation did not settle" },
);
} catch {
if (this.active.get(active.launchId) === active && active.turn === turn) {
await this.stopChild(active, "cancelled");
}
}
return this.turns.getMatching(expected);
}
async stopEnvironment(expected: NodeWorkerEnvironmentStopInput): Promise<void> {
const key = nodeWorkerEnvironmentKey(expected);
this.stoppingEnvironments.set(key, (this.stoppingEnvironments.get(key) ?? 0) + 1);
try {
const admission = this.admissions.get(key);
if (admission && nodeWorkerEnvironmentMatches(admission.binding, expected)) {
admission.abort.abort(new Error("node worker environment stopped"));
await admission.done.catch(() => undefined);
}
await this.initialize();
for (const owner of this.active.values()) {
if (!nodeWorkerEnvironmentMatches(owner.binding, expected)) {
continue;
}
if (owner.state === "running") {
await this.stopChild(owner, "interrupted");
}
const observed = this.active.get(owner.launchId);
if (observed?.state === "observed") {
this.reconcileActiveTerminal(observed);
} else if (observed) {
throw new Error("node worker environment cleanup is incomplete");
}
}
for (const owner of this.store.listNonterminal()) {
if (nodeWorkerEnvironmentMatches(owner, expected)) {
await this.cancelOwner(owner);
const remaining = this.store.get(owner.launchId);
if (remaining?.state === "pending" || remaining?.state === "running") {
throw new Error("node worker environment is still owned by another supervisor");
}
}
}
} finally {
const remaining = this.stoppingEnvironments.get(key)! - 1;
if (remaining === 0) {
this.stoppingEnvironments.delete(key);
} else {
this.stoppingEnvironments.set(key, remaining);
}
}
}
private async cancelOwner(
expected: NodeWorkerSupervisorIdentity,
): Promise<NodeWorkerLaunchReceipt | undefined> {
await this.initialize();
const receipt = this.store.getMatching(expected);
@@ -316,7 +538,7 @@ class NodeWorkerSupervisor {
// Startup may already own a container while its create/start client is
// in flight; retain the durable slot until normal cancellation fences it.
await startup;
return await this.cancel(expected);
return await this.cancelOwner(expected);
}
const cancelled = this.capacity.finishCancelled({
expected,
@@ -328,79 +550,15 @@ class NodeWorkerSupervisor {
}
if (startup && receipt.container && receipt.supervisor.pid === process.pid) {
await startup;
return await this.cancel(expected);
return await this.cancelOwner(expected);
}
const supervisorState = inspectNodeWorkerProcessIdentity(receipt.supervisor);
if (supervisorState === "live" || supervisorState === "unknown") {
return receipt;
}
if (!receipt.worker) {
return this.capacity.finishCancelled({
expected,
supervisor: receipt.supervisor,
worker: null,
});
}
if (receipt.container) {
const containerState = await this.requireContainerLifecycle().inspect(
receipt.container,
receipt,
);
if (containerState === "unknown" || containerState === "reused") {
return receipt;
}
const beforeSignal = this.store.getMatching(expected);
if (
beforeSignal?.state !== "running" ||
!nodeWorkerReceiptMatchesOwner(
beforeSignal,
receipt.supervisor,
receipt.worker,
receipt.container,
)
) {
return beforeSignal;
}
await this.requireContainerLifecycle().remove(receipt.container, receipt);
return this.capacity.finishCancelled({
expected,
supervisor: receipt.supervisor,
worker: receipt.worker,
});
}
let workerState = inspectOwnedNodeWorkerTree(receipt.worker);
if (workerState === "unknown") {
return receipt;
}
if (workerState === "live") {
const beforeSignal = this.store.getMatching(expected);
if (
beforeSignal?.state !== "running" ||
!nodeWorkerReceiptMatchesOwner(beforeSignal, receipt.supervisor, receipt.worker)
) {
return beforeSignal;
}
await signalOwnedNodeWorkerTree(receipt.worker, "SIGTERM");
workerState = await waitForOwnedNodeWorkerTreeDeath(receipt.worker, STOP_GRACE_MS);
}
if (workerState === "live") {
const beforeSignal = this.store.getMatching(expected);
if (
beforeSignal?.state !== "running" ||
!nodeWorkerReceiptMatchesOwner(beforeSignal, receipt.supervisor, receipt.worker)
) {
return beforeSignal;
}
await signalOwnedNodeWorkerTree(receipt.worker, "SIGKILL");
workerState = await waitForOwnedNodeWorkerTreeDeath(receipt.worker, FORCE_STOP_WAIT_MS);
}
if (workerState !== "dead") {
return this.store.getMatching(expected);
}
return this.capacity.finishCancelled({
expected,
supervisor: receipt.supervisor,
worker: receipt.worker,
return await recoverNodeWorkerLaunch({
receipt,
store: this.store,
capacity: this.capacity,
containerLifecycle: this.containerLifecycle,
notifyCapacity: true,
state: "cancelled",
});
}
@@ -410,9 +568,13 @@ class NodeWorkerSupervisor {
}
this.closed = true;
this.capacity.close();
for (const admission of this.admissions.values()) {
admission.abort.abort(new Error("node worker supervisor is closed"));
}
const operation = (async () => {
const errors: unknown[] = [];
await this.initializationPromise?.catch((error: unknown) => errors.push(error));
await Promise.allSettled([...this.admissions.values()].map((admission) => admission.done));
await Promise.allSettled(this.starting.values());
const stopped = await Promise.allSettled(
[...this.active.values()]
@@ -475,172 +637,14 @@ class NodeWorkerSupervisor {
});
}
private async startClaimed(params: {
input: NodeWorkerLaunchInput;
descriptor: WorkerLaunchDescriptor;
planHash: string;
supervisor: NodeWorkerProcessIdentity;
signal?: AbortSignal;
}): Promise<NodeWorkerLaunchReceipt> {
const credential = params.descriptor.admission.credential;
const endpoint = params.descriptor.connectionEndpoint;
const cloudflareAccess = endpoint.kind === "websocket" ? endpoint.cloudflareAccess : undefined;
const sensitiveValues = cloudflareAccess
? [credential, cloudflareAccess.clientId, cloudflareAccess.clientSecret]
: [credential];
const scrubber = createNodeWorkerCredentialScrubber(sensitiveValues);
// Turn cancellation can beat the child's admission retry deadline. Retain the
// producer's latest cause so the durable terminal receipt does not become generic.
const connectionFailure: { errorText?: string } = {};
for (const value of sensitiveValues) {
registerSecretValueForRedaction(value);
}
let adapter: NodeWorkerChildAdapter;
let container: NodeWorkerContainerIdentity | undefined;
try {
const prepared = await prepareNodeWorkerLaunchTransport({
bundleRoot: this.bundleRoot,
workerEnv: this.workerEnv,
engineEnv: this.engineEnv,
input: params.input,
descriptor: params.descriptor,
connectionFailure,
scrubber,
store: this.store,
containerEngine: this.containerEngine,
containerLifecycle: this.containerLifecycle,
containerImage: this.containerImage,
});
if (prepared.kind === "terminal") {
return prepared.receipt;
}
adapter = prepared.adapter;
container = prepared.container;
} catch (error) {
return this.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(error, "node worker spawn failed", scrubber.scrub),
});
}
if (!adapter.pid) {
if (container) {
await this.requireContainerLifecycle().remove(container, params.input);
}
adapter.kill("SIGKILL");
adapter.dispose();
return this.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: "node worker spawn did not return a process id",
});
}
let worker: NodeWorkerProcessIdentity;
try {
worker = requireNodeWorkerProcessIdentity(adapter.pid);
} catch (error) {
if (container) {
await this.requireContainerLifecycle().remove(container, params.input);
}
adapter.kill("SIGKILL");
await adapter.wait().catch(() => undefined);
adapter.dispose();
return this.capacity.finish({
launchId: params.input.launchId,
planHash: params.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
error,
"node worker process identity unavailable",
scrubber.scrub,
),
});
}
const { journalReady, releaseJournal } = createNodeWorkerJournalGate();
const active = {
state: "running",
adapter,
journalReady,
gatewayNamespace: params.input.gatewayNamespace,
launchId: params.input.launchId,
planHash: params.planHash,
releaseJournal,
scrubber,
connectionFailure,
supervisor: params.supervisor,
worker,
...(container ? { container } : {}),
} as NodeWorkerRunningChild;
active.done = this.observeChild(active);
this.active.set(active.launchId, active);
void active.done.catch(() => undefined);
let running: NodeWorkerLaunchReceipt;
try {
running = this.store.markRunning({
launchId: active.launchId,
planHash: active.planHash,
supervisor: params.supervisor,
worker,
...(container ? { container } : {}),
});
} catch (error) {
active.releaseJournal();
if (container) {
await this.stopChild(active, "interrupted");
this.active.delete(active.launchId);
this.capacity.finish({
launchId: active.launchId,
planHash: active.planHash,
supervisor: params.supervisor,
worker: null,
state: "failed",
errorText: sanitizeNodeWorkerDiagnostic(
error,
"node worker container identity could not be persisted",
scrubber.scrub,
),
});
} else {
await this.stopChild(active, "interrupted").catch(() => undefined);
}
throw error;
}
active.releaseJournal();
if (running.state === "cancelled" || running.state === "interrupted") {
await this.stopChild(active, running.state);
return this.store.get(active.launchId) ?? running;
}
if (running.state !== "running") {
if (container) {
await this.stopChild(active, "interrupted");
} else {
adapter.closeStartGate?.();
}
return running;
}
if (this.closed || params.signal?.aborted) {
await this.stopChild(active, this.closed ? "interrupted" : "cancelled");
return this.store.get(active.launchId) ?? running;
}
try {
await startNodeWorkerLaunchTransport({ adapter, descriptor: params.descriptor, container });
} catch {
await this.stopChild(active, "interrupted");
return this.store.get(active.launchId) ?? running;
}
return running;
}
private async observeChild(active: NodeWorkerRunningChild): Promise<void> {
const outcome = await observeNodeWorkerChildOutput(active);
const outcome = await observeNodeWorkerChildOutput(
active,
(frame) => {
settleNodeWorkerTurn(active, frame, this.turns);
},
() => active.turn?.claim.launchId,
);
if (active.container) {
try {
await this.cleanupActiveContainer(active);
@@ -660,6 +664,7 @@ class NodeWorkerSupervisor {
): void {
const observed: NodeWorkerObservedTerminal = {
state: "observed",
binding: active.binding,
gatewayNamespace: active.gatewayNamespace,
launchId: active.launchId,
planHash: active.planHash,
@@ -677,6 +682,8 @@ class NodeWorkerSupervisor {
} catch {
// The observed outcome stays owned in memory for the next supervisor operation.
}
active.turn?.settle();
active.turn = undefined;
}
private async cleanupActiveContainer(active: NodeWorkerRunningChild): Promise<void> {
+131
View File
@@ -0,0 +1,131 @@
import { addAbortListener } from "node:events";
import { registerSecretValueForRedaction } from "../logging/secret-redaction-registry.js";
import { createDeferredCore } from "../shared/deferred.js";
import type { WorkerLaunchDescriptor } from "../worker/launch-descriptor.js";
import type { WorkerProcessResult } from "../worker/worker-process-protocol.js";
import type { NodeWorkerLaunchClaim, NodeWorkerLaunchReceipt } from "./node-worker-launch-store.js";
import { sendNodeWorkerInput } from "./node-worker-launch-transport.js";
import { createNodeWorkerCredentialScrubber } from "./node-worker-output.js";
import type { NodeWorkerSupervisorIdentity } from "./node-worker-supervisor-contract.js";
import {
createNodeWorkerActiveTurn,
type NodeWorkerRunningChild,
type NodeWorkerStopState,
} from "./node-worker-supervisor-ownership.js";
import type { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
/** Shutdown must be able to abort admission before it stops the retiring physical owner. */
export async function waitForNodeWorkerRetirement(
active: NodeWorkerRunningChild,
signal: AbortSignal,
): Promise<void> {
signal.throwIfAborted();
if (!active.retiring) {
return;
}
const aborted = createDeferredCore();
const listener = addAbortListener(signal, () => aborted.resolve());
try {
await Promise.race([active.done, aborted.promise]);
} finally {
listener[Symbol.dispose]();
}
}
export function nodeWorkerDescriptorSecrets(descriptor: WorkerLaunchDescriptor): string[] {
const endpoint = descriptor.connectionEndpoint;
const access = endpoint.kind === "websocket" ? endpoint.cloudflareAccess : undefined;
return [
descriptor.admission.credential,
...(access ? [access.clientId, access.clientSecret] : []),
];
}
/** Persist completion before releasing the turn; the physical launch still owns cleanup. */
export function settleNodeWorkerTurn(
active: NodeWorkerRunningChild,
frame: WorkerProcessResult,
store: NodeWorkerTurnStore,
): void {
if (active.stopState) {
return;
}
const turn = active.turn;
if (!turn || turn.claim.launchId !== frame.turnId || active.retiring) {
throw new Error("node worker returned a result outside its active turn");
}
const receipt = store.finish({
expected: turn.claim,
ownerLaunchId: active.launchId,
supervisor: active.supervisor,
worker: active.worker,
...(turn.cancelled
? ({
state: "cancelled",
errorText: active.connectionFailure.errorText ?? "node worker turn cancelled",
} as const)
: ({ state: "completed", resultJson: JSON.stringify(frame.result) } as const)),
});
if (!receipt || receipt.state === "pending" || receipt.state === "running") {
throw new Error("node worker turn completion lost its physical owner");
}
active.turn = undefined;
active.retiring = !frame.retainWorker;
turn.settle();
}
export async function startNodeWorkerTurn({
active,
descriptor,
claim,
signal,
store,
cancel,
stopChild,
}: {
active: NodeWorkerRunningChild;
descriptor: WorkerLaunchDescriptor;
claim: NodeWorkerLaunchClaim;
signal: AbortSignal;
store: NodeWorkerTurnStore;
cancel: (expected: NodeWorkerSupervisorIdentity) => Promise<NodeWorkerLaunchReceipt | undefined>;
stopChild: (active: NodeWorkerRunningChild, state: NodeWorkerStopState) => Promise<void>;
}): Promise<NodeWorkerLaunchReceipt> {
signal.throwIfAborted();
const admitted = store.claim({
claim,
ownerLaunchId: active.launchId,
supervisor: active.supervisor,
worker: active.worker,
});
if (admitted.action === "replay") {
return admitted.receipt;
}
active.turn = createNodeWorkerActiveTurn(claim);
const secrets = nodeWorkerDescriptorSecrets(descriptor);
for (const value of secrets) {
registerSecretValueForRedaction(value);
}
// The IPC diagnostic handler shares this object, so rotate its contents rather than its owner.
Object.assign(active.scrubber, createNodeWorkerCredentialScrubber(secrets));
active.connectionFailure.errorText = undefined;
const onAbort = () => {
void cancel(claim).catch(() => undefined);
};
signal.addEventListener("abort", onAbort, { once: true });
try {
await sendNodeWorkerInput(active.adapter, {
type: "turn",
turnId: claim.launchId,
descriptor,
});
if (signal.aborted) {
await cancel(claim);
}
} catch {
await stopChild(active, "interrupted");
} finally {
signal.removeEventListener("abort", onAbort);
}
return store.get(claim.launchId) ?? admitted.receipt;
}
@@ -0,0 +1,373 @@
import { DatabaseSync } from "node:sqlite";
import { afterEach, describe, expect, it } from "vitest";
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
import { assertSqliteSchemaContains } from "../infra/sqlite-schema-contract.js";
import {
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../state/openclaw-state-db.js";
import { OPENCLAW_STATE_MAINTENANCE_SCHEMA_COMPATIBILITY } from "../state/openclaw-state-schema-compatibility.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
import {
NodeWorkerLaunchStore,
type NodeWorkerContainerIdentity,
type NodeWorkerLaunchClaim,
type NodeWorkerTerminalState,
} from "./node-worker-launch-store.js";
import { requireNodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
import { NodeWorkerTurnStore } from "./node-worker-turn-store.js";
const DAY_MS = 24 * 60 * 60 * 1_000;
const NOW_MS = 10 * DAY_MS;
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
afterEach(() => closeOpenClawStateDatabaseForTest());
function fixture() {
const env = { OPENCLAW_STATE_DIR: tempDirs.make("node-worker-turn-store-") };
const launches = new NodeWorkerLaunchStore({ env });
const turns = new NodeWorkerTurnStore({ env });
const supervisor = requireNodeWorkerProcessIdentity(process.pid);
const first: NodeWorkerLaunchClaim = {
launchId: "first-turn",
planHash: "a".repeat(64),
gatewayNamespace: "gateway-1",
environmentId: "environment-1",
sessionId: "session-1",
ownerEpoch: 3,
placementGeneration: 4,
runId: "first-run",
};
const next: NodeWorkerLaunchClaim = {
...first,
launchId: "second-turn",
planHash: "b".repeat(64),
runId: "second-run",
};
const owner = { ownerLaunchId: first.launchId, supervisor, worker: supervisor };
launches.claim(first, supervisor, 1, NOW_MS);
return {
env,
launches,
turns,
supervisor,
first,
next,
owner,
start(container?: NodeWorkerContainerIdentity) {
turns.claim({ claim: first, ownerLaunchId: first.launchId, supervisor, nowMs: NOW_MS });
launches.markRunning({
...first,
supervisor,
worker: supervisor,
container,
nowMs: NOW_MS,
});
},
finish(claim = first) {
return turns.finish({
...owner,
expected: claim,
state: "completed",
resultJson: JSON.stringify({ turnId: claim.launchId }),
nowMs: NOW_MS,
});
},
};
}
describe("node worker turn journal", () => {
it("keeps completed turn receipts independent of the running physical slot and later turns", () => {
const f = fixture();
expect(
f.turns.claim({
claim: f.first,
ownerLaunchId: f.first.launchId,
supervisor: f.supervisor,
nowMs: NOW_MS,
}),
).toMatchObject({ action: "start", receipt: { state: "pending", worker: null } });
f.start();
expect(f.turns.get(f.first.launchId)).toMatchObject({ state: "running", worker: f.supervisor });
const completed = f.finish();
expect(f.launches.get(f.first.launchId)?.state).toBe("running");
expect(f.launches.nonterminalCount()).toBe(1);
expect(f.turns.claim({ claim: f.next, ...f.owner, nowMs: NOW_MS })).toMatchObject({
action: "start",
receipt: { launchId: f.next.launchId, ownerLaunchId: f.first.launchId, state: "running" },
});
expect(f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS })).toEqual({
action: "replay",
receipt: completed,
});
expect(
f.turns.finish({ ...f.owner, expected: f.first, state: "failed", errorText: "late failure" }),
).toEqual(completed);
expect(f.turns.get(f.next.launchId)?.state).toBe("running");
expect(
f.launches.claim({ ...f.next, launchId: "another-worker" }, f.supervisor, 1, NOW_MS),
).toMatchObject({
action: "at-capacity",
});
});
it.each([
["gateway namespace", { gatewayNamespace: "gateway-2" }],
["environment", { environmentId: "environment-2" }],
["session", { sessionId: "session-2" }],
["owner epoch", { ownerEpoch: 4 }],
["placement generation", { placementGeneration: 5 }],
] satisfies Array<[string, Partial<NodeWorkerLaunchClaim>]>)(
"rejects a turn bound to another %s",
(_label, patch) => {
const f = fixture();
f.start();
expect(() => f.turns.claim({ claim: { ...f.next, ...patch }, ...f.owner })).toThrow(
"live physical owner",
);
expect(f.turns.get(f.next.launchId)).toBeUndefined();
},
);
it("requires the exact supervisor and worker even when the placement matches", () => {
const f = fixture();
f.start();
for (const field of ["supervisor", "worker"] as const) {
expect(() =>
f.turns.claim({
claim: f.next,
...f.owner,
[field]: { ...f.supervisor, startTime: f.supervisor.startTime + 1 },
}),
).toThrow("live physical owner");
}
expect(() =>
f.turns.claim({ claim: f.next, ownerLaunchId: f.first.launchId, supervisor: f.supervisor }),
).toThrow("live physical owner");
});
it("rejects conflicting retries and serializes different turns across store handles", () => {
const f = fixture();
f.start();
f.turns.claim({ claim: f.first, ...f.owner });
const other = new NodeWorkerTurnStore({ env: f.env });
expect(other.claim({ claim: f.first, ...f.owner }).action).toBe("replay");
for (const patch of [
{ planHash: f.next.planHash },
{ runId: f.next.runId },
{ sessionId: f.next.sessionId + "-other" },
]) {
expect(() => other.claim({ claim: { ...f.first, ...patch }, ...f.owner })).toThrow(
"different plan or owner",
);
}
expect(() =>
other.claim({ claim: f.first, ...f.owner, ownerLaunchId: "different-worker" }),
).toThrow("different plan or owner");
expect(() => other.claim({ claim: f.next, ...f.owner })).toThrow("UNIQUE constraint failed");
f.finish();
expect(other.claim({ claim: f.next, ...f.owner }).action).toBe("start");
});
it("rejects stale result writers and immutable identity mismatches", () => {
const f = fixture();
f.start();
f.turns.claim({ claim: f.first, ...f.owner });
expect(
f.turns.finish({
...f.owner,
expected: { ...f.first, runId: "wrong-run" },
state: "completed",
resultJson: "{}",
}),
).toBeUndefined();
expect(f.turns.getMatching({ ...f.first, ownerEpoch: 4 })).toBeUndefined();
expect(
f.turns.finish({
...f.owner,
expected: f.first,
worker: { ...f.supervisor, startTime: f.supervisor.startTime + 1 },
state: "completed",
resultJson: "{}",
}),
).toMatchObject({ state: "running" });
expect(f.turns.get(f.first.launchId)?.state).toBe("running");
});
it.each(["completed", "failed", "interrupted", "cancelled"] satisfies NodeWorkerTerminalState[])(
"closes unfinished turns atomically when their physical owner becomes %s",
(state) => {
const f = fixture();
f.start();
f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS });
const completed = f.finish();
f.turns.claim({ claim: f.next, ...f.owner, nowMs: NOW_MS + 2 });
f.launches.finish({
...f.first,
supervisor: f.supervisor,
worker: f.supervisor,
state,
...(state === "completed"
? { resultJson: "{}" }
: { errorText: "physical worker stopped" }),
nowMs: NOW_MS + 1,
});
const database = openOpenClawStateDatabase({ env: f.env }).db;
expect(
database
.prepare("SELECT state, completed_at_ms FROM node_worker_turns WHERE turn_id = ?")
.get(f.next.launchId),
).toEqual({
state: state === "completed" ? "interrupted" : state,
completed_at_ms: NOW_MS + 2,
});
expect(f.turns.get(f.first.launchId)).toEqual(completed);
expect(f.launches.nonterminalCount()).toBe(0);
},
);
it("keeps bare physical cleanup inert and cancels a pending first turn once admitted", () => {
const untracked = fixture();
untracked.launches.finishCancelled({
expected: untracked.first,
supervisor: untracked.supervisor,
worker: null,
});
expect(
openOpenClawStateDatabase({ env: untracked.env })
.db.prepare("SELECT name FROM sqlite_schema WHERE name = 'node_worker_turns'")
.get(),
).toBeUndefined();
const f = fixture();
f.turns.claim({ claim: f.first, ownerLaunchId: f.first.launchId, supervisor: f.supervisor });
f.launches.finishCancelled({ expected: f.first, supervisor: f.supervisor, worker: null });
expect(f.turns.get(f.first.launchId)).toMatchObject({
state: "cancelled",
errorText: "node worker launch cancelled",
});
});
it("prunes bounded old turn receipts without releasing the warm owner or losing the current replay", () => {
const f = fixture();
f.start();
f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS });
f.finish();
const database = openOpenClawStateDatabase({ env: f.env }).db;
const insert = database.prepare(`
INSERT INTO node_worker_turns (turn_id, owner_launch_id, plan_hash, run_id, state,
result_json, error_text, completed_at_ms, created_at_ms, updated_at_ms)
VALUES (?, ?, ?, 'historical-run', 'completed', '{}', NULL, 1, 1, 1)
`);
for (let index = 0; index < 258; index += 1) {
insert.run(`old-${index}`, f.first.launchId, f.first.planHash);
}
expect(f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS + DAY_MS + 1 }).action).toBe(
"replay",
);
expect(database.prepare("SELECT count(*) AS count FROM node_worker_turns").get()).toEqual({
count: 3,
});
expect(f.turns.get(f.first.launchId)?.state).toBe("completed");
f.turns.claim({ claim: f.next, ...f.owner, nowMs: NOW_MS + DAY_MS + 1 });
expect(database.prepare("SELECT turn_id FROM node_worker_turns").all()).toEqual([
{ turn_id: f.next.launchId },
]);
expect(f.launches.get(f.first.launchId)?.state).toBe("running");
expect(f.launches.nonterminalCount()).toBe(1);
f.finish(f.next);
expect(() => f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS + DAY_MS + 1 })).toThrow(
"live physical owner",
);
});
it("lets the predecessor preserve live capacity, finish and prune the owner, then reopens the candidate", () => {
const f = fixture();
const container = {
engine: "docker",
containerId: "c".repeat(64),
engineTarget: "d".repeat(64),
} as const;
f.start(container);
f.turns.claim({ claim: f.first, ...f.owner, nowMs: NOW_MS });
const completed = f.finish();
f.turns.claim({ claim: f.next, ...f.owner, nowMs: NOW_MS });
const opened = openOpenClawStateDatabase({ env: f.env });
const initialVersion = opened.db.prepare("PRAGMA user_version").get();
closeOpenClawStateDatabaseForTest();
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(
"CREATE TABLE IF NOT EXISTS node_worker_turns (",
);
const endMarker = "\n WHERE state = 'running';";
const end = OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start) + endMarker.length;
const predecessorSchema =
OPENCLAW_STATE_SCHEMA_SQL.slice(0, start) + OPENCLAW_STATE_SCHEMA_SQL.slice(end);
const predecessor = new DatabaseSync(opened.path);
try {
predecessor.exec("PRAGMA foreign_keys = ON");
expect(() =>
assertSqliteSchemaContains(predecessor, "predecessor shared state", predecessorSchema, {
...OPENCLAW_STATE_MAINTENANCE_SCHEMA_COMPATIBILITY,
allowedMissingTables:
OPENCLAW_STATE_MAINTENANCE_SCHEMA_COMPATIBILITY.allowedMissingTables?.filter(
(table) => table !== "node_worker_turns",
),
}),
).not.toThrow();
expect(predecessor.prepare("PRAGMA user_version").get()).toEqual(initialVersion);
expect(
predecessor
.prepare(
"SELECT count(*) AS count FROM node_worker_launches WHERE state IN ('pending', 'running')",
)
.get(),
).toEqual({ count: 1 });
expect(
predecessor
.prepare("SELECT container_json FROM node_worker_launch_containers WHERE launch_id = ?")
.get(f.first.launchId),
).toEqual({ container_json: JSON.stringify(container) });
predecessor
.prepare("DELETE FROM node_worker_launches WHERE completed_at_ms <= ?")
.run(NOW_MS + DAY_MS);
expect(predecessor.prepare("SELECT count(*) AS count FROM node_worker_turns").get()).toEqual({
count: 2,
});
predecessor
.prepare(
"UPDATE node_worker_launches SET state = 'interrupted', error_text = 'predecessor cleanup', completed_at_ms = ?, updated_at_ms = ? WHERE launch_id = ?",
)
.run(NOW_MS + 1, NOW_MS + 1, f.first.launchId);
} finally {
predecessor.close();
}
const candidate = new NodeWorkerTurnStore({ env: f.env });
expect(candidate.get(f.first.launchId)).toEqual(completed);
expect(candidate.get(f.next.launchId)).toMatchObject({
state: "interrupted",
errorText: "predecessor cleanup",
});
closeOpenClawStateDatabaseForTest();
const pruningPredecessor = new DatabaseSync(opened.path);
try {
pruningPredecessor.exec("PRAGMA foreign_keys = ON");
pruningPredecessor
.prepare(
"DELETE FROM node_worker_launch_containers WHERE launch_id IN (SELECT launch_id FROM node_worker_launches WHERE completed_at_ms <= ?)",
)
.run(NOW_MS + DAY_MS);
pruningPredecessor
.prepare("DELETE FROM node_worker_launches WHERE completed_at_ms <= ?")
.run(NOW_MS + DAY_MS);
expect(
pruningPredecessor.prepare("SELECT count(*) AS count FROM node_worker_turns").get(),
).toEqual({ count: 0 });
} finally {
pruningPredecessor.close();
}
expect(new NodeWorkerTurnStore({ env: f.env }).get(f.first.launchId)).toBeUndefined();
});
});
+288
View File
@@ -0,0 +1,288 @@
import type { DatabaseSync } from "node:sqlite";
import type { Selectable } from "kysely";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../infra/kysely-sync.js";
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
import {
runOpenClawStateWriteTransaction,
type OpenClawStateDatabaseOptions,
} from "../state/openclaw-state-db.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
import type { NodeWorkerSupervisorIdentity } from "../worker/node-supervisor-protocol.js";
import {
readNodeWorkerLaunchReceipt,
settleNodeWorkerActiveTurns,
type NodeWorkerLaunchClaim,
type NodeWorkerLaunchReceipt,
type NodeWorkerTerminalState,
} from "./node-worker-launch-store.js";
import type { NodeWorkerProcessIdentity } from "./node-worker-process-identity.js";
type TurnDatabase = Pick<OpenClawStateDatabase, "node_worker_turns">;
type TurnRow = Selectable<TurnDatabase["node_worker_turns"]>;
export type NodeWorkerTurnReceipt = NodeWorkerLaunchReceipt & { ownerLaunchId: string };
const initializedDatabases = new WeakSet<DatabaseSync>();
const TERMINAL_RECEIPT_RETENTION_MS = 24 * 60 * 60 * 1_000;
const TERMINAL_PRUNE_BATCH_LIMIT = 256;
function query(database: DatabaseSync) {
return getNodeSqliteKysely<TurnDatabase>(database);
}
function ensureTurnSchema(database: DatabaseSync): void {
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf("CREATE TABLE IF NOT EXISTS node_worker_turns (");
const endMarker = "\n WHERE state = 'running';";
const end = OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start);
if (start < 0 || end < start) {
throw new Error("OpenClaw node worker turn schema marker is missing.");
}
database.exec(OPENCLAW_STATE_SCHEMA_SQL.slice(start, end + endMarker.length)); // sqlite-allow-raw -- Canonical feature-local additive DDL only.
}
function readRow(database: DatabaseSync, turnId: string): TurnRow | undefined {
return executeSqliteQueryTakeFirstSync(
database,
query(database).selectFrom("node_worker_turns").selectAll().where("turn_id", "=", turnId),
);
}
function readReceipt(database: DatabaseSync, turnId: string): NodeWorkerTurnReceipt | undefined {
let turn = readRow(database, turnId);
if (!turn) {
return undefined;
}
const owner = readNodeWorkerLaunchReceipt(database, turn.owner_launch_id);
if (!owner) {
throw new Error(`node worker turn ${turnId} has no physical owner`);
}
if (turn.state === "running" && owner.state !== "pending" && owner.state !== "running") {
// A predecessor can finish the physical journal without knowing about turn receipts.
settleNodeWorkerActiveTurns(database, owner);
turn = readRow(database, turnId)!;
}
const state = turn.state === "running" && owner.state === "pending" ? "pending" : turn.state;
if (
state !== "pending" &&
state !== "running" &&
state !== "completed" &&
state !== "failed" &&
state !== "interrupted" &&
state !== "cancelled"
) {
throw new Error(`invalid node worker turn state ${state}`);
}
return {
...owner,
ownerLaunchId: owner.launchId,
launchId: turn.turn_id,
planHash: turn.plan_hash,
runId: turn.run_id,
state,
resultJson: turn.result_json,
errorText: turn.error_text,
completedAtMs: turn.completed_at_ms,
createdAtMs: turn.created_at_ms,
updatedAtMs: turn.updated_at_ms,
};
}
function matchesIdentity(
receipt: NodeWorkerTurnReceipt,
expected: NodeWorkerSupervisorIdentity,
): boolean {
return (
receipt.launchId === expected.launchId &&
receipt.planHash === expected.planHash &&
receipt.environmentId === expected.environmentId &&
receipt.sessionId === expected.sessionId &&
receipt.ownerEpoch === expected.ownerEpoch &&
receipt.placementGeneration === expected.placementGeneration &&
receipt.runId === expected.runId
);
}
function matchesProcess(
receipt: NodeWorkerLaunchReceipt,
supervisor: NodeWorkerProcessIdentity,
worker: NodeWorkerProcessIdentity | null,
): boolean {
return (
receipt.supervisor.pid === supervisor.pid &&
receipt.supervisor.startTime === supervisor.startTime &&
receipt.worker?.pid === worker?.pid &&
receipt.worker?.startTime === worker?.startTime
);
}
function pruneTerminal(database: DatabaseSync, nowMs: number, excludeTurnId: string): void {
// A warm worker may live indefinitely; its finished turns must not accumulate with it.
executeSqliteQuerySync(
database,
query(database)
.deleteFrom("node_worker_turns")
.where(
"turn_id",
"in",
query(database)
.selectFrom("node_worker_turns")
.select("turn_id")
.where("completed_at_ms", "<=", Math.max(0, nowMs - TERMINAL_RECEIPT_RETENTION_MS))
.where("turn_id", "!=", excludeTurnId)
.orderBy("completed_at_ms", "asc")
.orderBy("turn_id", "asc")
.limit(TERMINAL_PRUNE_BATCH_LIMIT),
),
);
}
/** Immutable turn outcomes attached to a separately supervised physical worker. */
export class NodeWorkerTurnStore {
private readonly databaseOptions: OpenClawStateDatabaseOptions;
constructor(options: { env?: NodeJS.ProcessEnv } = {}) {
this.databaseOptions = options.env ? { env: options.env } : {};
}
private write<T>(operationLabel: string, operation: (database: DatabaseSync) => T): T {
let initialized: DatabaseSync | undefined;
const result = runOpenClawStateWriteTransaction(
({ db }) => {
if (!initializedDatabases.has(db)) {
ensureTurnSchema(db);
initialized = db;
}
return operation(db);
},
this.databaseOptions,
{ operationLabel },
);
if (initialized) {
initializedDatabases.add(initialized);
}
return result;
}
claim(params: {
claim: NodeWorkerLaunchClaim;
ownerLaunchId: string;
supervisor: NodeWorkerProcessIdentity;
worker?: NodeWorkerProcessIdentity | null;
nowMs?: number;
}): { action: "start" | "replay"; receipt: NodeWorkerTurnReceipt } {
const { claim, ownerLaunchId, supervisor } = params;
const nowMs = params.nowMs ?? Date.now();
return this.write("node-worker-turn.claim", (database) => {
const existing = readReceipt(database, claim.launchId);
if (existing) {
if (
!matchesIdentity(existing, claim) ||
existing.gatewayNamespace !== claim.gatewayNamespace ||
existing.ownerLaunchId !== ownerLaunchId
) {
throw new Error(
`node worker turn ${claim.launchId} was replayed with a different plan or owner`,
);
}
pruneTerminal(database, nowMs, claim.launchId);
return { action: "replay", receipt: existing };
}
const owner = readNodeWorkerLaunchReceipt(database, ownerLaunchId);
if (
!owner ||
(owner.state !== "pending" && owner.state !== "running") ||
!matchesProcess(owner, supervisor, params.worker ?? null) ||
owner.gatewayNamespace !== claim.gatewayNamespace ||
owner.environmentId !== claim.environmentId ||
owner.sessionId !== claim.sessionId ||
owner.ownerEpoch !== claim.ownerEpoch ||
owner.placementGeneration !== claim.placementGeneration ||
(owner.state === "pending" && owner.launchId !== claim.launchId) ||
(owner.launchId === claim.launchId &&
(owner.state !== "pending" ||
owner.planHash !== claim.planHash ||
owner.runId !== claim.runId))
) {
throw new Error(
`node worker turn ${claim.launchId} does not match its live physical owner`,
);
}
executeSqliteQuerySync(
database,
query(database).insertInto("node_worker_turns").values({
turn_id: claim.launchId,
owner_launch_id: ownerLaunchId,
plan_hash: claim.planHash,
run_id: claim.runId,
state: "running",
result_json: null,
error_text: null,
completed_at_ms: null,
created_at_ms: nowMs,
updated_at_ms: nowMs,
}),
);
pruneTerminal(database, nowMs, claim.launchId);
return { action: "start", receipt: readReceipt(database, claim.launchId)! };
});
}
get(turnId: string): NodeWorkerTurnReceipt | undefined {
return this.write("node-worker-turn.get", (database) => readReceipt(database, turnId));
}
getMatching(expected: NodeWorkerSupervisorIdentity): NodeWorkerTurnReceipt | undefined {
const receipt = this.get(expected.launchId);
return receipt && matchesIdentity(receipt, expected) ? receipt : undefined;
}
finish(params: {
expected: NodeWorkerSupervisorIdentity;
ownerLaunchId: string;
supervisor: NodeWorkerProcessIdentity;
worker: NodeWorkerProcessIdentity | null;
state: NodeWorkerTerminalState;
resultJson?: string;
errorText?: string;
nowMs?: number;
}): NodeWorkerTurnReceipt | undefined {
return this.write("node-worker-turn.finish", (database) => {
const receipt = readReceipt(database, params.expected.launchId);
if (
!receipt ||
!matchesIdentity(receipt, params.expected) ||
receipt.ownerLaunchId !== params.ownerLaunchId
) {
return undefined;
}
if (
(receipt.state !== "pending" && receipt.state !== "running") ||
!matchesProcess(receipt, params.supervisor, params.worker)
) {
return receipt;
}
const nowMs = params.nowMs ?? Date.now();
const completedAtMs = Math.max(nowMs, receipt.createdAtMs, receipt.updatedAtMs);
executeSqliteQuerySync(
database,
query(database)
.updateTable("node_worker_turns")
.set({
state: params.state,
result_json: params.state === "completed" ? (params.resultJson ?? null) : null,
error_text: params.state === "completed" ? null : (params.errorText ?? null),
completed_at_ms: completedAtMs,
updated_at_ms: completedAtMs,
})
.where("turn_id", "=", receipt.launchId)
.where("state", "=", "running"),
);
pruneTerminal(database, nowMs, receipt.launchId);
return readReceipt(database, receipt.launchId);
});
}
}
@@ -9,7 +9,7 @@ import type { NodeWorkerWorkspaceRetainInput } from "../worker/node-workspace-re
import { createNodeWorkerSupervisor } from "./node-worker-supervisor.js";
import {
TEST_WORKER_ENDPOINT,
testNodeWorkerLaunchIdentity,
testNodeWorkerEnvironmentIdentity,
testWorkerLaunchInput,
writeNodeWorkerFixture,
} from "./node-worker-supervisor.test-support.js";
@@ -81,18 +81,6 @@ function retainInput(
};
}
async function waitForTerminal(
supervisor: ReturnType<typeof createNodeWorkerSupervisor>,
launchId: string,
): Promise<void> {
await vi.waitFor(
async () => {
expect((await supervisor.status(launchId))?.state).not.toMatch(/^(?:pending|running)$/u);
},
{ timeout: 5_000 },
);
}
afterEach(() => {
vi.restoreAllMocks();
closeOpenClawStateDatabaseForTest();
@@ -388,22 +376,28 @@ describe("node worker workspace retention", () => {
},
);
it("keeps a nonterminal launch until a later authoritative snapshot", async () => {
it("keeps a retained worker workspace after turn completion until environment teardown", async () => {
const root = tempDirs.make("node-worker-workspace-retention-active-");
const { bundleRoot, env, workspaceDir } = writeNodeWorkerFixture(root);
const input = testWorkerLaunchInput(workspaceDir, "active-retention", "wait");
const input = testWorkerLaunchInput(workspaceDir, "active-retention", "background-start");
const active = seedGeneration(bundleRoot, input, input.descriptor.admission.ownerEpoch);
const supervisor = createNodeWorkerSupervisor({ bundleRoot, env });
await supervisor.launch(input, TEST_WORKER_ENDPOINT);
await supervisor.retainWorkspaces(retainInput(input, 1, []));
expect(fs.existsSync(active)).toBe(true);
try {
await supervisor.launch(input, TEST_WORKER_ENDPOINT);
await vi.waitFor(
async () => expect((await supervisor.status(input.launchId))?.state).toBe("completed"),
{ timeout: 5_000 },
);
await supervisor.retainWorkspaces(retainInput(input, 1, []));
expect(fs.existsSync(active)).toBe(true);
await supervisor.cancel(testNodeWorkerLaunchIdentity(input));
await waitForTerminal(supervisor, input.launchId);
await supervisor.retainWorkspaces(retainInput(input, 2, []));
expect(fs.existsSync(active)).toBe(false);
await supervisor.close();
await supervisor.stopEnvironment(testNodeWorkerEnvironmentIdentity(input));
await supervisor.retainWorkspaces(retainInput(input, 2, []));
expect(fs.existsSync(active)).toBe(false);
} finally {
await supervisor.close();
}
});
it("rereads a launch reservation immediately before deleting", async () => {
+11 -39
View File
@@ -761,6 +761,7 @@ describe("runNodeHost", () => {
bundleRetention: 1,
bundleStatus: 1,
portalStream: 1,
environmentSession: 1,
};
options?.onHelloOk?.({
protocol: 4,
@@ -771,6 +772,7 @@ describe("runNodeHost", () => {
GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_RETENTION,
GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_STATUS,
GATEWAY_SERVER_CAPS.NODE_WORKER_PORTAL_STREAM,
GATEWAY_SERVER_CAPS.NODE_WORKER_ENVIRONMENT_SESSION,
],
},
} as unknown as Parameters<NonNullable<GatewayClientOptions["onHelloOk"]>>[0]);
@@ -1007,52 +1009,22 @@ describe("runNodeHost", () => {
}
});
it("appends context path to the Gateway WebSocket URL", async () => {
it.each([
["/gws", "ws://127.0.0.1:18789/gws"],
["/gws/", "ws://127.0.0.1:18789/gws/"],
["gws", "ws://127.0.0.1:18789/gws"],
["", "ws://127.0.0.1:18789"],
[undefined, "ws://127.0.0.1:18789"],
])("builds the Gateway URL for context path %s", async (gatewayContextPath, expectedUrl) => {
await expect(
runNodeHost({
gatewayHost: "127.0.0.1",
gatewayPort: 18789,
gatewayContextPath: "/gws",
gatewayContextPath,
}),
).rejects.toThrow("event loop readiness timeout");
expect(lastCapturedOptions()?.url).toBe("ws://127.0.0.1:18789/gws");
});
it("preserves trailing slash in context path as-is", async () => {
await expect(
runNodeHost({
gatewayHost: "127.0.0.1",
gatewayPort: 18789,
gatewayContextPath: "/gws/",
}),
).rejects.toThrow("event loop readiness timeout");
expect(lastCapturedOptions()?.url).toBe("ws://127.0.0.1:18789/gws/");
});
it("prepends leading slash when context path is missing one", async () => {
await expect(
runNodeHost({
gatewayHost: "127.0.0.1",
gatewayPort: 18789,
gatewayContextPath: "gws",
}),
).rejects.toThrow("event loop readiness timeout");
expect(lastCapturedOptions()?.url).toBe("ws://127.0.0.1:18789/gws");
});
it("omits context path when empty or undefined", async () => {
await expect(
runNodeHost({
gatewayHost: "127.0.0.1",
gatewayPort: 18789,
gatewayContextPath: "",
}),
).rejects.toThrow("event loop readiness timeout");
expect(lastCapturedOptions()?.url).toBe("ws://127.0.0.1:18789");
expect(lastCapturedOptions()?.url).toBe(expectedUrl);
});
it("configures the SQLite gateway snapshot with contextPath", async () => {
+11 -18
View File
@@ -20,6 +20,7 @@ import {
NODE_RUNNER_INVENTORY_UPDATE_METHOD,
NODE_WORKER_BUNDLE_RETENTION_VERSION,
NODE_WORKER_BUNDLE_STATUS_VERSION,
NODE_WORKER_ENVIRONMENT_SESSION_VERSION,
NODE_WORKER_PORTAL_STREAM_VERSION,
NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
type NodeWorkerCapacitySnapshot,
@@ -291,9 +292,7 @@ export async function runNodeHost(opts: NodeHostRunOptions): Promise<void> {
let consecutivePermanentGatewayRejections = 0;
let gatewayConnectionGeneration = 0;
let connectedGatewayProtocol = 0;
let gatewaySupportsBundleRetention = false;
let gatewaySupportsBundleStatus = false;
let gatewaySupportsPortalStream = false;
let gatewayCapabilities: ReadonlySet<string> = new Set();
let optionalPublicationStates = new Map<
NodeOptionalPublicationMethod,
NodeOptionalPublicationState
@@ -310,9 +309,7 @@ export async function runNodeHost(opts: NodeHostRunOptions): Promise<void> {
gatewayConnectionGeneration += 1;
gatewayHelloReceived = false;
connectedGatewayProtocol = 0;
gatewaySupportsBundleRetention = false;
gatewaySupportsBundleStatus = false;
gatewaySupportsPortalStream = false;
gatewayCapabilities = new Set();
retireOptionalPublications();
};
@@ -511,15 +508,19 @@ export async function runNodeHost(opts: NodeHostRunOptions): Promise<void> {
enabled: true,
capacity: workerCapacity,
bundlePrewarm: WORKER_BUNDLE_PREWARM_VERSION,
...(gatewaySupportsBundleRetention
...(gatewayCapabilities.has(GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_RETENTION)
? { bundleRetention: NODE_WORKER_BUNDLE_RETENTION_VERSION }
: {}),
...(gatewaySupportsBundleRetention && gatewaySupportsBundleStatus
...(gatewayCapabilities.has(GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_RETENTION) &&
gatewayCapabilities.has(GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_STATUS)
? { bundleStatus: NODE_WORKER_BUNDLE_STATUS_VERSION }
: {}),
...(gatewaySupportsPortalStream
...(gatewayCapabilities.has(GATEWAY_SERVER_CAPS.NODE_WORKER_PORTAL_STREAM)
? { portalStream: NODE_WORKER_PORTAL_STREAM_VERSION }
: {}),
...(gatewayCapabilities.has(GATEWAY_SERVER_CAPS.NODE_WORKER_ENVIRONMENT_SESSION)
? { environmentSession: NODE_WORKER_ENVIRONMENT_SESSION_VERSION }
: {}),
}
: { enabled: false },
},
@@ -598,15 +599,7 @@ export async function runNodeHost(opts: NodeHostRunOptions): Promise<void> {
gatewayConnectionGeneration += 1;
gatewayHelloReceived = true;
connectedGatewayProtocol = hello.protocol;
gatewaySupportsBundleRetention =
hello.features?.capabilities?.includes(GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_RETENTION) ===
true;
gatewaySupportsBundleStatus =
hello.features?.capabilities?.includes(GATEWAY_SERVER_CAPS.NODE_WORKER_BUNDLE_STATUS) ===
true;
gatewaySupportsPortalStream =
hello.features?.capabilities?.includes(GATEWAY_SERVER_CAPS.NODE_WORKER_PORTAL_STREAM) ===
true;
gatewayCapabilities = new Set(hello.features?.capabilities);
retireOptionalPublications();
optionalPublicationStates = new Map();
if (opts.stopAfterFirstConnect) {
+3
View File
@@ -20,6 +20,7 @@ export const FIRST_USE_STATE_TABLES = [
"mcp_oauth_pending_authorizations",
"node_worker_launch_containers",
"node_worker_launches",
"node_worker_turns",
"operator_approval_execution_identities",
"operator_approval_standing_grants",
"execution_decision_facts",
@@ -29,6 +30,8 @@ export const FIRST_USE_STATE_TABLES = [
] as const;
export const FIRST_USE_STATE_INDEXES = [
"idx_node_worker_launches_terminal_completed",
"idx_node_worker_turns_terminal_completed",
"idx_node_worker_turns_active_owner",
"idx_operator_approval_standing_grants_binding",
"execution_identity_contexts_run_created_idx",
"execution_decision_facts_context_occurred_idx",
+14
View File
@@ -867,6 +867,19 @@ export interface NodeWorkerLaunches {
worker_start_time: number | null;
}
export interface NodeWorkerTurns {
completed_at_ms: number | null;
created_at_ms: number;
error_text: string | null;
owner_launch_id: string;
plan_hash: string;
result_json: string | null;
run_id: string;
state: string;
turn_id: string;
updated_at_ms: number;
}
export interface OfficialExternalPluginCatalogSnapshots {
body: string;
checksum: string;
@@ -1562,6 +1575,7 @@ export interface DB {
native_hook_relay_bridges: NativeHookRelayBridges;
node_worker_launch_containers: NodeWorkerLaunchContainers;
node_worker_launches: NodeWorkerLaunches;
node_worker_turns: NodeWorkerTurns;
official_external_plugin_catalog_snapshots: OfficialExternalPluginCatalogSnapshots;
operator_approval_execution_identities: OperatorApprovalExecutionIdentities;
operator_approval_standing_grants: OperatorApprovalStandingGrants;
+59
View File
@@ -918,6 +918,65 @@ CREATE TABLE IF NOT EXISTS node_worker_launch_containers (
container_json TEXT
) STRICT;
-- Turn receipts have a shorter lifetime than their physical worker owner.
-- Keeping the launch running preserves capacity and predecessor cleanup semantics.
CREATE TABLE IF NOT EXISTS node_worker_turns (
turn_id TEXT NOT NULL PRIMARY KEY
CHECK (length(turn_id) BETWEEN 1 AND 256 AND instr(turn_id, char(0)) = 0),
owner_launch_id TEXT NOT NULL
REFERENCES node_worker_launches(launch_id) ON DELETE CASCADE,
plan_hash TEXT NOT NULL
CHECK (length(plan_hash) = 64 AND plan_hash NOT GLOB '*[^0-9a-f]*'),
run_id TEXT NOT NULL
CHECK (length(run_id) BETWEEN 1 AND 256 AND instr(run_id, char(0)) = 0),
state TEXT NOT NULL
CHECK (state IN ('running', 'completed', 'failed', 'interrupted', 'cancelled')),
result_json TEXT CHECK (
result_json IS NULL
OR (
length(CAST(result_json AS BLOB)) BETWEEN 1 AND 65536
AND instr(result_json, char(0)) = 0
AND json_valid(result_json)
)
),
error_text TEXT CHECK (
error_text IS NULL
OR (
length(CAST(error_text AS BLOB)) BETWEEN 1 AND 4096
AND instr(error_text, char(0)) = 0
AND instr(error_text, char(10)) = 0
AND instr(error_text, char(13)) = 0
)
),
completed_at_ms INTEGER CHECK (
completed_at_ms IS NULL OR completed_at_ms BETWEEN 0 AND 9007199254740991
),
created_at_ms INTEGER NOT NULL CHECK (created_at_ms BETWEEN 0 AND 9007199254740991),
updated_at_ms INTEGER NOT NULL CHECK (
updated_at_ms BETWEEN created_at_ms AND 9007199254740991
),
CHECK (
(state = 'running'
AND result_json IS NULL AND error_text IS NULL AND completed_at_ms IS NULL)
OR
(state = 'completed'
AND result_json IS NOT NULL AND error_text IS NULL
AND completed_at_ms BETWEEN created_at_ms AND updated_at_ms)
OR
(state IN ('failed', 'interrupted', 'cancelled')
AND result_json IS NULL AND error_text IS NOT NULL
AND completed_at_ms BETWEEN created_at_ms AND updated_at_ms)
)
) STRICT;
CREATE INDEX IF NOT EXISTS idx_node_worker_turns_terminal_completed
ON node_worker_turns(completed_at_ms, turn_id)
WHERE completed_at_ms IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_node_worker_turns_active_owner
ON node_worker_turns(owner_launch_id)
WHERE state = 'running';
CREATE TABLE IF NOT EXISTS config_health_entries (
config_path TEXT NOT NULL PRIMARY KEY,
last_known_good_json TEXT,
+11 -3
View File
@@ -9,6 +9,7 @@ import type {
} from "../../packages/gateway-protocol/src/schema/worker-inference.js";
import type { OperationalRunInstanceRef } from "../agents/admitted-run-context.js";
import { toToolDefinitions } from "../agents/agent-tool-definition-adapter.js";
import { wrapToolWithAbortSignal } from "../agents/agent-tools.abort.js";
import { finalizeAgentTools } from "../agents/agent-tools.finalize.js";
import { isApplyPatchAllowedForModel } from "../agents/apply-patch-model-policy.js";
import { buildBootstrapContextForFiles } from "../agents/bootstrap-files.js";
@@ -27,7 +28,6 @@ import { wrapToolWithGatewayCallerIdentity } from "../agents/tools/gateway-calle
import { DEFAULT_AGENTS_FILENAME, loadWorkspaceBootstrapFiles } from "../agents/workspace.js";
import type { OpenClawConfig } from "../config/types.openclaw.js";
import type { AssistantMessage, AssistantMessageEventStreamLike } from "../llm/types.js";
import { getProcessSupervisor } from "../process/supervisor/index.js";
import { createWorkerBrowserToolRuntime, type WorkerBrowserRuntime } from "./browser-runtime.js";
import { createWorkerLiveRuntime } from "./embedded-agent-live.runtime.js";
import {
@@ -209,6 +209,10 @@ export async function runWorkerEmbeddedTurn(params: RunWorkerEmbeddedTurnParams)
...(params.browserRuntime ? { runtime: params.browserRuntime } : {}),
})
: undefined;
const turnLifetime = new AbortController();
const toolSignal = params.signal
? AbortSignal.any([params.signal, turnLifetime.signal])
: turnLifetime.signal;
const { session } = await (async () => {
try {
const unboundLocalTools = finalizeAgentTools({
@@ -230,6 +234,7 @@ export async function runWorkerEmbeddedTurn(params: RunWorkerEmbeddedTurnParams)
}),
},
agentId: params.agentId,
abortSignal: toolSignal,
}).filter((tool) => localToolNameSet.has(tool.name));
const localTools = unboundLocalTools.map((tool) =>
wrapToolWithGatewayCallerIdentity(tool, {
@@ -270,7 +275,7 @@ export async function runWorkerEmbeddedTurn(params: RunWorkerEmbeddedTurnParams)
tools: [...activeToolNames],
customTools: toToolDefinitions([
...localTools.filter((tool) => allowedToolNameSet.has(tool.name)),
...sessionTools,
...sessionTools.map((tool) => wrapToolWithAbortSignal(tool, toolSignal)),
]),
noTools: "all",
sessionManager,
@@ -279,6 +284,7 @@ export async function runWorkerEmbeddedTurn(params: RunWorkerEmbeddedTurnParams)
withSessionWriteSettlement: transcriptRuntime.withSessionWriteSettlement,
});
} catch (error) {
turnLifetime.abort();
await browserRuntime?.dispose();
throw error;
}
@@ -350,9 +356,11 @@ export async function runWorkerEmbeddedTurn(params: RunWorkerEmbeddedTurnParams)
await liveRuntime.emitTerminal();
}
} finally {
// Tools and prepared calls belong to this turn; promoted processes belong
// to the enclosing environment and remain reachable through fresh tools.
turnLifetime.abort();
params.signal?.removeEventListener("abort", abortTurn);
unsubscribe();
getProcessSupervisor().cancelScope(params.sessionKey, "manual-cancel");
session.dispose();
await browserRuntime?.dispose();
}
@@ -3,6 +3,7 @@ import { testWorkerDescriptor } from "../node-host/node-worker-supervisor.test-s
import {
NODE_WORKER_CONNECTION_FAILURE_MESSAGE_TYPE,
parseNodeWorkerConnectionFailureMessage,
parseNodeWorkerEnvironmentStopInput,
parseNodeWorkerLaunchInput,
parseNodeWorkerSupervisorReceipt,
type NodeWorkerSupervisorIdentity,
@@ -22,12 +23,32 @@ const identity: NodeWorkerSupervisorIdentity = {
};
describe("node worker supervisor launch request", () => {
it.each([undefined, 2])(
"rejects a Gateway without the negotiated environment lifetime marker %s",
(environmentSession) => {
const descriptor = testWorkerDescriptor("/tmp/worker", "success", "turn-1");
expect(() =>
parseNodeWorkerLaunchInput(
JSON.stringify({
environmentSession,
launchId: "turn-1",
gatewayNamespace: "gateway-1",
expectedBundleHash: descriptor.admission.handshake.bundleHash,
placementGeneration: 4,
descriptor,
}),
),
).toThrow("INVALID_REQUEST");
},
);
it("rejects mismatched launch and turn ids", () => {
const descriptor = testWorkerDescriptor("/tmp/worker", "success", "turn-1");
expect(() =>
parseNodeWorkerLaunchInput(
JSON.stringify({
environmentSession: 1,
launchId: "other-launch",
gatewayNamespace: "gateway-1",
expectedBundleHash: descriptor.admission.handshake.bundleHash,
@@ -39,6 +60,32 @@ describe("node worker supervisor launch request", () => {
});
});
describe("node worker environment stop request", () => {
const scope = {
gatewayNamespace: "gateway-1",
environmentId: "environment-1",
sessionId: "session-1",
ownerEpoch: 3,
};
it("preserves the exact environment owner independently of its completed turn", () => {
expect(parseNodeWorkerEnvironmentStopInput(JSON.stringify(scope))).toEqual(scope);
});
it.each([
{ ...scope, ownerEpoch: undefined },
{ ...scope, ownerEpoch: -1 },
{ ...scope, sessionId: "" },
{ ...scope, gatewayNamespace: "../gateway" },
{ ...scope, launchId: "turn-1" },
{ ...scope, environmentId: "x".repeat(4096) },
])("rejects an incomplete or unbounded environment owner: %j", (input) => {
expect(() => parseNodeWorkerEnvironmentStopInput(JSON.stringify(input))).toThrow(
"INVALID_REQUEST",
);
});
});
describe("node worker supervisor wire receipt", () => {
it("accepts only bounded worker connection diagnostics", () => {
expect(
+45 -4
View File
@@ -5,13 +5,14 @@ import { parseWorkerLaunchPlan, type WorkerLaunchPlan } from "./launch-descripto
const IDENTIFIER_MAX_CHARS = 256;
const GATEWAY_NAMESPACE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/u;
const NODE_WORKER_SUPERVISOR_CANCEL_REQUEST_MAX_BYTES = 4 * 1024;
const NODE_WORKER_SUPERVISOR_CONTROL_REQUEST_MAX_BYTES = 4 * 1024;
const NODE_WORKER_RESULT_JSON_MAX_BYTES = 64 * 1024;
const NODE_WORKER_ERROR_TEXT_MAX_BYTES = 4 * 1024;
const NODE_WORKER_CONNECTION_FAILURE_CAUSE_MAX_BYTES = 64 * 1024;
export const NODE_WORKER_CONNECTION_FAILURE_MESSAGE_TYPE = "openclaw-worker-connection-failure-v1";
export type NodeWorkerLaunchInput = {
environmentSession: 1;
launchId: string;
gatewayNamespace: string;
expectedBundleHash: string;
@@ -19,6 +20,13 @@ export type NodeWorkerLaunchInput = {
descriptor: WorkerLaunchPlan;
};
export type NodeWorkerEnvironmentStopInput = {
gatewayNamespace: string;
environmentId: string;
sessionId: string;
ownerEpoch: number;
};
export type NodeWorkerSupervisorIdentity = {
launchId: string;
planHash: string;
@@ -102,7 +110,7 @@ function decodeRequest(raw?: string | null): unknown {
}
}
export function assertNodeWorkerLaunchIdentity(
function assertNodeWorkerLaunchIdentity(
input: Pick<NodeWorkerLaunchInput, "launchId" | "expectedBundleHash">,
descriptor: WorkerLaunchPlan,
): void {
@@ -115,10 +123,14 @@ export function assertNodeWorkerLaunchIdentity(
}
export function parseNodeWorkerLaunchInput(raw?: string | null): NodeWorkerLaunchInput {
const value = decodeRequest(raw);
return validateNodeWorkerLaunchInput(decodeRequest(raw));
}
export function validateNodeWorkerLaunchInput(value: unknown): NodeWorkerLaunchInput {
if (
!isRecord(value) ||
!hasExactKeys(value, [
"environmentSession",
"launchId",
"gatewayNamespace",
"expectedBundleHash",
@@ -128,6 +140,9 @@ export function parseNodeWorkerLaunchInput(raw?: string | null): NodeWorkerLaunc
) {
throw new Error("INVALID_REQUEST: invalid node worker launch request");
}
if (value.environmentSession !== 1) {
throw new Error("INVALID_REQUEST: node worker environment lifetime support required");
}
const launchId = requireIdentifier(value.launchId, "launchId");
const gatewayNamespace = requireIdentifier(value.gatewayNamespace, "gatewayNamespace");
if (!GATEWAY_NAMESPACE_PATTERN.test(gatewayNamespace)) {
@@ -149,6 +164,7 @@ export function parseNodeWorkerLaunchInput(raw?: string | null): NodeWorkerLaunc
descriptor,
);
return {
environmentSession: 1,
launchId,
gatewayNamespace,
expectedBundleHash: value.expectedBundleHash,
@@ -169,7 +185,7 @@ export function parseNodeWorkerLookupInput(raw?: string | null): { launchId: str
}
export function parseNodeWorkerCancelInput(raw?: string | null): NodeWorkerSupervisorIdentity {
if (!raw || Buffer.byteLength(raw, "utf8") > NODE_WORKER_SUPERVISOR_CANCEL_REQUEST_MAX_BYTES) {
if (!raw || Buffer.byteLength(raw, "utf8") > NODE_WORKER_SUPERVISOR_CONTROL_REQUEST_MAX_BYTES) {
throw new Error("INVALID_REQUEST: invalid node worker cancel request");
}
const value = decodeRequest(raw);
@@ -204,6 +220,31 @@ export function parseNodeWorkerCancelInput(raw?: string | null): NodeWorkerSuper
};
}
export function parseNodeWorkerEnvironmentStopInput(
raw?: string | null,
): NodeWorkerEnvironmentStopInput {
if (!raw || Buffer.byteLength(raw, "utf8") > NODE_WORKER_SUPERVISOR_CONTROL_REQUEST_MAX_BYTES) {
throw new Error("INVALID_REQUEST: invalid node worker environment stop request");
}
const value = decodeRequest(raw);
if (
!isRecord(value) ||
!hasExactKeys(value, ["gatewayNamespace", "environmentId", "sessionId", "ownerEpoch"])
) {
throw new Error("INVALID_REQUEST: invalid node worker environment stop request");
}
const gatewayNamespace = requireIdentifier(value.gatewayNamespace, "gatewayNamespace");
if (!GATEWAY_NAMESPACE_PATTERN.test(gatewayNamespace)) {
throw new Error("INVALID_REQUEST: gatewayNamespace must be a safe bounded path component");
}
return {
gatewayNamespace,
environmentId: requireIdentifier(value.environmentId, "environmentId"),
sessionId: requireIdentifier(value.sessionId, "sessionId"),
ownerEpoch: requireNonNegativeInteger(value.ownerEpoch, "ownerEpoch"),
};
}
export function nodeWorkerPlanHash(
input: Pick<
NodeWorkerLaunchInput,
+227 -2
View File
@@ -1,21 +1,32 @@
import { Console } from "node:console";
import { PassThrough } from "node:stream";
import path from "node:path";
import { PassThrough, Writable } from "node:stream";
import { beforeEach, describe, expect, it, vi } from "vitest";
import {
WORKER_PROTOCOL_FEATURES,
WORKER_RPC_SET_VERSION,
} from "../../packages/gateway-protocol/src/schema/worker-admission.js";
import { WORKER_PROTOCOL_MAX_INFERENCE_PAYLOAD_BYTES } from "../../packages/gateway-protocol/src/schema/worker-inference.js";
import { createDeferred } from "../../test/helpers/promise.js";
import { setLoggerOverride } from "../logging/logger.js";
import { loggingState } from "../logging/state.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import type { WorkerBrowserRuntime } from "./browser-runtime.js";
import type { WorkerLaunchDescriptor } from "./launch-descriptor.js";
import { runWorkerCommand } from "./worker-command.runtime.js";
import { parseWorkerProcessResult, type WorkerProcessResult } from "./worker-process-protocol.js";
import { runWorkerProcess } from "./worker-process.js";
import { runWorkerDescriptor } from "./worker.runtime.js";
import { createWorkerRuntimeEnvironment, runWorkerDescriptor } from "./worker.runtime.js";
const managedRuntime = vi.hoisted(() => ({ backgroundCount: 0, close: vi.fn() }));
vi.mock("../agents/bash-process-registry.js", () => ({
getActiveBackgroundExecSessionCount: () => managedRuntime.backgroundCount,
}));
vi.mock("./worker.runtime.js", () => ({
runWorkerDescriptor: vi.fn(),
createWorkerRuntimeEnvironment: vi.fn(),
}));
const descriptor = {
@@ -88,6 +99,35 @@ function lifetimeHarness() {
};
}
function managedHarness() {
const input = new PassThrough();
const output = new PassThrough();
const results: WorkerProcessResult[] = [];
output.on("data", (chunk: Buffer) => {
const result = parseWorkerProcessResult(JSON.parse(chunk.toString("utf8")));
if (result) {
results.push(result);
}
});
const launch: WorkerLaunchDescriptor = structuredClone(descriptor);
launch.assignment = {
...launch.assignment,
workspaceDir: process.cwd(),
permissionMode: "full",
workerContainmentRoot: process.cwd(),
};
const send = (value: unknown) => input.write(`${JSON.stringify(value)}\n`);
return {
input,
output,
results,
launch,
send,
turn: (value: WorkerLaunchDescriptor = launch) =>
send({ type: "turn", turnId: value.assignment.turnId, descriptor: value }),
};
}
describe("worker command lifetime gate", () => {
beforeEach(() => {
vi.mocked(runWorkerDescriptor).mockReset();
@@ -96,6 +136,14 @@ describe("worker command lifetime gate", () => {
transcriptLeafId: null,
transcriptNextSeq: 1,
});
managedRuntime.backgroundCount = 0;
managedRuntime.close.mockReset();
managedRuntime.close.mockResolvedValue(undefined);
vi.mocked(createWorkerRuntimeEnvironment).mockReset();
vi.mocked(createWorkerRuntimeEnvironment).mockResolvedValue({
stateDir: "/tmp/openclaw-managed-worker-state",
close: managedRuntime.close,
});
});
it("keeps the ordinary worker command path ungated", async () => {
@@ -248,4 +296,181 @@ describe("worker command lifetime gate", () => {
expect(lifetime.terminateOwnedTree).toHaveBeenCalledOnce();
expect(lifetime.dispose).toHaveBeenCalledOnce();
});
it("retains state across turns and cancels only the exact active turn", async () => {
const harness = managedHarness();
const lifetime = lifetimeHarness();
managedRuntime.backgroundCount = 1;
const secondStarted = createDeferred<AbortSignal>();
vi.mocked(runWorkerDescriptor)
.mockImplementationOnce(async () => ({
status: "completed",
transcriptLeafId: "first-leaf",
transcriptNextSeq: 2,
}))
.mockImplementationOnce(async (_launch, options) => {
const signal = options!.signal!;
secondStarted.resolve(signal);
await new Promise<void>((resolve) => {
signal.addEventListener("abort", () => resolve(), { once: true });
});
managedRuntime.backgroundCount = 0;
return {
status: "failed",
reason: "turn-failed",
transcriptLeafId: "second-leaf",
transcriptNextSeq: 3,
};
});
const running = runWorkerCommand({ ...harness, managed: true, lifetime: lifetime.contract });
lifetime.open();
harness.turn();
await vi.waitFor(() => expect(harness.results).toHaveLength(1));
expect(harness.results[0]).toMatchObject({ turnId: "turn-1", retainWorker: true });
expect(lifetime.dispose).not.toHaveBeenCalled();
const next = structuredClone(harness.launch);
next.assignment.turnId = "turn-2";
next.assignment.runId = "run-2";
next.assignment.operationalRunInstance = { instanceId: "instance-run-2", runId: "run-2" };
harness.turn(next);
const secondSignal = await secondStarted.promise;
harness.send({ type: "cancel", turnId: "turn-1" });
expect(secondSignal.aborted).toBe(false);
harness.send({ type: "cancel", turnId: "turn-2" });
await running;
expect(harness.results[1]).toMatchObject({
turnId: "turn-2",
result: { status: "failed", reason: "turn-failed" },
retainWorker: false,
});
expect(createWorkerRuntimeEnvironment).toHaveBeenCalledOnce();
expect(
vi.mocked(runWorkerDescriptor).mock.calls.map(([, options]) => options?.environmentStateDir),
).toEqual(["/tmp/openclaw-managed-worker-state", "/tmp/openclaw-managed-worker-state"]);
expect(managedRuntime.close).toHaveBeenCalledOnce();
expect(lifetime.dispose).toHaveBeenCalledOnce();
});
it("closes a retained environment when the managed input ends", async () => {
const harness = managedHarness();
managedRuntime.backgroundCount = 1;
const running = runWorkerCommand({ ...harness, managed: true });
harness.turn();
await vi.waitFor(() => expect(harness.results).toHaveLength(1));
harness.input.end();
await running;
expect(managedRuntime.close).toHaveBeenCalledOnce();
});
it.each(["owner", "output"] as const)(
"closes state when %s ends during a pending result write",
async (ending) => {
const harness = managedHarness();
const writing = createDeferred<(error?: Error | null) => void>();
const output = new Writable({
write: (_chunk, _encoding, callback) => {
writing.resolve(callback);
},
});
managedRuntime.backgroundCount = 1;
const running = runWorkerCommand({ ...harness, output, managed: true });
harness.turn();
const completeWrite = await writing.promise;
if (ending === "owner") {
harness.input.end();
await running;
completeWrite();
} else {
const rejected = expect(running).rejects.toThrow("fixture result output failed");
completeWrite(new Error("fixture result output failed"));
await rejected;
}
expect(managedRuntime.close).toHaveBeenCalledOnce();
},
);
it.each([
"duplicate",
"environment",
"session",
"epoch",
"agent",
"permission",
"workspace",
"containment",
] as const)("refuses a retained worker's %s identity change before admission", async (change) => {
const harness = managedHarness();
managedRuntime.backgroundCount = 1;
const running = runWorkerCommand({ ...harness, managed: true });
harness.turn();
await vi.waitFor(() => expect(harness.results).toHaveLength(1));
const next = structuredClone(harness.launch);
if (change !== "duplicate") {
next.assignment.turnId = "turn-2";
}
if (change === "environment") {
next.admission.environmentId = "environment-2";
}
if (change === "session") {
next.admission.sessionId = "session-2";
}
if (change === "epoch") {
next.admission.ownerEpoch = 2;
}
if (change === "agent") {
next.assignment.agentId = "agent-2";
}
if (change === "permission") {
next.assignment.permissionMode = "read-only";
}
if (change === "workspace") {
next.assignment.workspaceDir = path.dirname(process.cwd());
}
if (change === "containment") {
next.assignment.workerContainmentRoot = path.dirname(process.cwd());
}
const rejected = expect(running).rejects.toThrow(
change === "duplicate" ? "already executed" : "binding changed",
);
harness.turn(next);
await rejected;
expect(runWorkerDescriptor).toHaveBeenCalledOnce();
expect(managedRuntime.close).toHaveBeenCalledOnce();
});
it("rejects concurrent turns and aborts the admitted turn before closing state", async () => {
const harness = managedHarness();
const started = createDeferred<AbortSignal>();
vi.mocked(runWorkerDescriptor).mockImplementationOnce(async (_launch, options) => {
const signal = options!.signal!;
started.resolve(signal);
await new Promise<void>((resolve) => {
signal.addEventListener("abort", () => resolve(), { once: true });
});
return { status: "completed", transcriptLeafId: null, transcriptNextSeq: 1 };
});
const running = runWorkerCommand({ ...harness, managed: true });
harness.turn();
const signal = await started.promise;
const rejected = expect(running).rejects.toThrow("already active");
const next = structuredClone(harness.launch);
next.assignment.turnId = "turn-2";
harness.turn(next);
await rejected;
expect(signal.aborted).toBe(true);
expect(harness.results).toEqual([]);
expect(managedRuntime.close).toHaveBeenCalledOnce();
});
it("bounds an unterminated managed input line before attempting admission", async () => {
const harness = managedHarness();
const running = runWorkerCommand({ ...harness, managed: true });
const rejected = expect(running).rejects.toThrow("exceeds the protocol payload limit");
harness.input.write(Buffer.alloc(WORKER_PROTOCOL_MAX_INFERENCE_PAYLOAD_BYTES + 1, 120));
await rejected;
expect(runWorkerDescriptor).not.toHaveBeenCalled();
expect(createWorkerRuntimeEnvironment).not.toHaveBeenCalled();
});
});
+226 -1
View File
@@ -1,14 +1,19 @@
import { realpath } from "node:fs/promises";
import type { Readable, Writable } from "node:stream";
import { WORKER_PROTOCOL_MAX_INFERENCE_PAYLOAD_BYTES } from "../../packages/gateway-protocol/src/schema/worker-inference.js";
import { getActiveBackgroundExecSessionCount } from "../agents/bash-process-registry.js";
import { toErrorObject } from "../infra/errors.js";
import type { WorkerBrowserRuntime } from "./browser-runtime.js";
import { parseWorkerLaunchDescriptor, type WorkerLaunchDescriptor } from "./launch-descriptor.js";
import { runWorkerDescriptor } from "./worker.runtime.js";
import { parseWorkerProcessRequest, type WorkerProcessResult } from "./worker-process-protocol.js";
import { createWorkerRuntimeEnvironment, runWorkerDescriptor } from "./worker.runtime.js";
type RunWorkerCommandOptions = {
input: Readable;
lifetime?: WorkerCommandLifetime;
output: Writable;
browserRuntime?: WorkerBrowserRuntime;
managed?: boolean;
};
export type WorkerCommandLifetime = {
@@ -19,6 +24,213 @@ export type WorkerCommandLifetime = {
terminateOwnedTree: () => void;
};
async function runManagedWorkerCommand(
options: RunWorkerCommandOptions,
signal: AbortSignal,
): Promise<void> {
let environment: Awaited<ReturnType<typeof createWorkerRuntimeEnvironment>> | undefined;
let binding: string | undefined;
let lastTurnId: string | undefined;
let active: { turnId: string; controller: AbortController } | undefined;
let running: Promise<void> | undefined;
let closed = false;
let chunks: Buffer[] = [];
let byteLength = 0;
let removeListeners = () => {};
try {
await new Promise<void>((resolve, reject) => {
const finish = (error?: unknown) => {
if (closed) {
return;
}
const failure =
error === undefined ? undefined : toErrorObject(error, "managed worker command failed");
closed = true;
active?.controller.abort(failure ?? new Error("worker supervisor input closed"));
options.input.destroy();
options.output.destroy();
if (failure) {
reject(failure);
} else {
resolve();
}
};
const onLine = (line: Buffer) => {
let value: unknown;
try {
value = JSON.parse(line.toString("utf8"));
} catch {
throw new Error("managed worker request is not valid JSON");
}
const request = parseWorkerProcessRequest(value);
if (request.type === "cancel") {
if (active?.turnId === request.turnId) {
active.controller.abort(new Error("worker turn cancelled"));
}
return;
}
if (active || lastTurnId === request.turnId) {
throw new Error("managed worker turn is already active or was already executed");
}
// The node turn journal owns replay history; retain only the immediate
// transport duplicate here, then require fresh Gateway admission.
lastTurnId = request.turnId;
const current = { turnId: request.turnId, controller: new AbortController() };
active = current;
running = (async () => {
const descriptor = request.descriptor;
const workspaceDir = await realpath(descriptor.assignment.workspaceDir);
const workerContainmentRoot = await realpath(
descriptor.assignment.workerContainmentRoot ?? workspaceDir,
);
const nextBinding = JSON.stringify({
environmentId: descriptor.admission.environmentId,
sessionId: descriptor.admission.sessionId,
ownerEpoch: descriptor.admission.ownerEpoch,
agentId: descriptor.assignment.agentId,
permissionMode: descriptor.assignment.permissionMode,
workspaceDir,
workerContainmentRoot,
});
if (binding !== undefined && binding !== nextBinding) {
throw new Error("managed worker environment binding changed; relaunch required");
}
binding = nextBinding;
if (closed) {
return;
}
environment ??= await createWorkerRuntimeEnvironment(descriptor.admission.sessionId);
if (closed) {
return;
}
const result = await runWorkerDescriptor(
{
...descriptor,
assignment:
descriptor.assignment.permissionMode === undefined
? { ...descriptor.assignment, workspaceDir }
: {
...descriptor.assignment,
workspaceDir,
permissionMode: descriptor.assignment.permissionMode,
workerContainmentRoot,
},
},
{
environmentStateDir: environment.stateDir,
signal: current.controller.signal,
...(options.lifetime
? { onConnectionFailure: options.lifetime.reportConnectionFailure }
: {}),
...(options.browserRuntime ? { browserRuntime: options.browserRuntime } : {}),
},
);
if (closed) {
return;
}
const retainWorker =
(result.status === "completed" || result.status === "failed") &&
getActiveBackgroundExecSessionCount() > 0;
const response: WorkerProcessResult = {
type: "result",
turnId: current.turnId,
result,
retainWorker,
};
if (retainWorker) {
active = undefined;
}
await new Promise<void>((resolveWrite, rejectWrite) => {
const onClose = () => rejectWrite(new Error("managed worker result output closed"));
options.output.once("close", onClose);
options.output.write(`${JSON.stringify(response)}\n`, (error) => {
options.output.off("close", onClose);
if (error) {
rejectWrite(error);
} else {
resolveWrite();
}
});
});
if (!retainWorker) {
active = undefined;
finish();
}
})().catch(finish);
};
const onData = (raw: unknown) => {
if (closed) {
return;
}
try {
const chunk =
typeof raw === "string"
? Buffer.from(raw)
: raw instanceof Uint8Array
? Buffer.from(raw)
: undefined;
if (!chunk) {
throw new Error("managed worker input must be bytes");
}
let offset = 0;
while (offset < chunk.length) {
if (closed) {
break;
}
const newline = chunk.indexOf(10, offset);
const end = newline === -1 ? chunk.length : newline;
byteLength += end - offset;
if (byteLength > WORKER_PROTOCOL_MAX_INFERENCE_PAYLOAD_BYTES) {
throw new Error("managed worker request exceeds the protocol payload limit");
}
chunks.push(chunk.subarray(offset, end));
if (newline === -1) {
break;
}
const line = Buffer.concat(chunks, byteLength);
chunks = [];
byteLength = 0;
onLine(line);
offset = newline + 1;
}
} catch (error) {
finish(error);
}
};
const onEnd = () => finish();
const onAbort = () => finish(signal.reason);
options.input.on("data", onData);
options.input.once("end", onEnd);
options.input.once("close", onEnd);
options.input.once("error", finish);
options.output.once("error", finish);
signal.addEventListener("abort", onAbort, { once: true });
removeListeners = () => {
options.input.off("data", onData);
options.input.off("end", onEnd);
options.input.off("close", onEnd);
options.input.off("error", finish);
options.output.off("error", finish);
signal.removeEventListener("abort", onAbort);
};
if (signal.aborted) {
onAbort();
} else if (options.input.readableEnded || options.input.destroyed) {
onEnd();
}
});
} finally {
chunks = [];
try {
await running;
await environment?.close();
} finally {
removeListeners();
}
}
}
async function readLaunchDescriptor(input: Readable): Promise<WorkerLaunchDescriptor> {
const chunks: Buffer[] = [];
let byteLength = 0;
@@ -66,6 +278,19 @@ export async function runWorkerCommand(options: RunWorkerCommandOptions): Promis
options.lifetime.terminateOwnedTree();
};
try {
if (options.managed) {
if (!(await (options.lifetime?.started ?? Promise.resolve(true)))) {
return;
}
options.lifetime?.signal.addEventListener("abort", stopForLifetime, { once: true });
if (options.lifetime?.signal.aborted) {
stopForLifetime();
}
process.once("SIGINT", stop);
process.once("SIGTERM", stop);
await runManagedWorkerCommand(options, abortController.signal);
return;
}
const [descriptor, started] = await Promise.all([
readLaunchDescriptor(options.input),
options.lifetime?.started ?? Promise.resolve(true),
+14 -3
View File
@@ -4,9 +4,19 @@ import workerDeployBrowserRuntime from "./worker-deploy-browser-runtime.js";
import { runWorkerProcess } from "./worker-process.js";
const args = process.argv.slice(2);
const internalWorkerIpc = args[0] === "--internal-worker-ipc";
const internalWorkerPrewarm = args[0] === "--internal-worker-prewarm";
if (args.length > 1 || (args.length === 1 && !internalWorkerIpc && !internalWorkerPrewarm)) {
const internalWorkerIpc = args.includes("--internal-worker-ipc");
const internalWorkerPrewarm = args.includes("--internal-worker-prewarm");
const managed = args.includes("--internal-worker-session");
if (
new Set(args).size !== args.length ||
args.some(
(arg) =>
!["--internal-worker-ipc", "--internal-worker-prewarm", "--internal-worker-session"].includes(
arg,
),
) ||
(internalWorkerPrewarm && args.length !== 1)
) {
throw new Error("worker deploy entry received unsupported arguments");
}
@@ -15,6 +25,7 @@ if (internalWorkerPrewarm) {
} else {
await runWorkerProcess({
internalWorkerIpc,
managed,
browserRuntime: workerDeployBrowserRuntime,
});
}
+100
View File
@@ -0,0 +1,100 @@
import { isRecord } from "@openclaw/normalization-core/record-coerce";
import { WORKER_PROTOCOL_MAX_IDENTIFIER_LENGTH } from "../../packages/gateway-protocol/src/schema/worker-protocol-primitives.js";
import { parseWorkerLaunchDescriptor, type WorkerLaunchDescriptor } from "./launch-descriptor.js";
import { parseWorkerAdmissionDeadlineResult } from "./worker-connection-contract.js";
import type { WorkerRuntimeResult } from "./worker.runtime.js";
/** Private JSONL protocol between one node supervisor and its environment-owned worker. */
export type WorkerProcessInput =
| { type: "turn"; turnId: string; descriptor: WorkerLaunchDescriptor }
| { type: "cancel"; turnId: string };
export type WorkerProcessResult = {
type: "result";
turnId: string;
result: WorkerRuntimeResult;
retainWorker: boolean;
};
export function parseWorkerProcessRequest(value: unknown): WorkerProcessInput {
if (
!isRecord(value) ||
typeof value.turnId !== "string" ||
!value.turnId.trim() ||
value.turnId.length > WORKER_PROTOCOL_MAX_IDENTIFIER_LENGTH
) {
throw new Error("invalid managed worker request");
}
if (value.type === "cancel" && Object.keys(value).length === 2) {
return { type: "cancel", turnId: value.turnId };
}
if (value.type === "turn" && Object.keys(value).length === 3) {
const descriptor = parseWorkerLaunchDescriptor(value.descriptor);
if (descriptor.assignment.turnId !== value.turnId) {
throw new Error("managed worker request disagrees with its assigned turn");
}
return { type: "turn", turnId: value.turnId, descriptor };
}
throw new Error("invalid managed worker request");
}
export function parseWorkerRuntimeResult(value: unknown): WorkerRuntimeResult | null {
const admissionFailure = parseWorkerAdmissionDeadlineResult(value);
if (admissionFailure) {
return admissionFailure;
}
if (!isRecord(value)) {
return null;
}
if (
value.status === "fenced" &&
(value.reason === "credential-replaced" || value.reason === "owner-epoch-mismatch") &&
Object.keys(value).length === 2
) {
return { status: value.status, reason: value.reason };
}
if (
(value.transcriptLeafId === null || typeof value.transcriptLeafId === "string") &&
typeof value.transcriptNextSeq === "number" &&
Number.isSafeInteger(value.transcriptNextSeq) &&
value.transcriptNextSeq >= 1
) {
const transcript = {
transcriptLeafId: value.transcriptLeafId,
transcriptNextSeq: value.transcriptNextSeq,
};
if (value.status === "completed" && Object.keys(value).length === 3) {
return { status: value.status, ...transcript };
}
if (
value.status === "failed" &&
value.reason === "turn-failed" &&
Object.keys(value).length === 4
) {
return { status: value.status, reason: value.reason, ...transcript };
}
}
return null;
}
export function parseWorkerProcessResult(value: unknown): WorkerProcessResult | null {
if (
!isRecord(value) ||
Object.keys(value).length !== 4 ||
value.type !== "result" ||
typeof value.turnId !== "string" ||
!value.turnId.trim() ||
value.turnId.length > WORKER_PROTOCOL_MAX_IDENTIFIER_LENGTH ||
typeof value.retainWorker !== "boolean"
) {
return null;
}
const result = parseWorkerRuntimeResult(value.result);
if (
!result ||
(value.retainWorker && result.status !== "completed" && result.status !== "failed")
) {
return null;
}
return { type: "result", turnId: value.turnId, result, retainWorker: value.retainWorker };
}
+2
View File
@@ -114,6 +114,7 @@ function createWorkerIpcLifetime(): WorkerCommandLifetime {
export async function runWorkerProcess(
options: {
internalWorkerIpc?: boolean;
managed?: boolean;
browserRuntime?: WorkerBrowserRuntime;
} = {},
): Promise<void> {
@@ -123,6 +124,7 @@ export async function runWorkerProcess(
await runWorkerCommand({
input: process.stdin,
output: process.stdout,
...(options.managed ? { managed: true } : {}),
...(options.internalWorkerIpc ? { lifetime: createWorkerIpcLifetime() } : {}),
...(options.browserRuntime ? { browserRuntime: options.browserRuntime } : {}),
});
+291 -5
View File
@@ -1,7 +1,8 @@
import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import { mkdir, mkdtemp, readFile, realpath, rm, stat, writeFile } from "node:fs/promises";
import { createServer, type Server } from "node:http";
import { tmpdir } from "node:os";
import path from "node:path";
import { PassThrough } from "node:stream";
import { rawDataToString } from "@openclaw/gateway-client/websocket-data";
import { isRecord } from "@openclaw/normalization-core/record-coerce";
import { Type } from "typebox";
@@ -47,21 +48,29 @@ import {
} from "../../packages/gateway-protocol/src/schema/worker-inference.js";
import { createDeferred, withTestTimeout } from "../../test/helpers/promise.js";
import { createOperationalRunInstanceRef } from "../agents/admitted-run-context.js";
import { listRunningSessions } from "../agents/bash-process-registry.js";
import {
deleteSession,
listRunningSessions,
markBackgrounded,
} from "../agents/bash-process-registry.js";
import { runExecProcess } from "../agents/bash-tools.exec-runtime.js";
import { saveExecApprovals, type ExecApprovalsFile } from "../infra/exec-approvals.js";
import { getProcessSupervisor } from "../process/supervisor/index.js";
import { buildWorkerConnectParams, type WorkerLaunchDescriptor } from "./launch-descriptor.js";
import { WORKER_PROVIDER_REPLAY_LOCAL_RETRY_MESSAGE } from "./transcript-message.js";
import { runWorkerCommand } from "./worker-command.runtime.js";
import {
WorkerAdmissionDeadlineExceededError,
WorkerConnectionStoppedError,
} from "./worker-connection-contract.js";
import { createWorkerConnection, type WorkerConnectionState } from "./worker-connection.js";
import { parseWorkerProcessResult, type WorkerProcessResult } from "./worker-process-protocol.js";
import {
WorkerInferenceProxyClient,
WorkerLiveEventClient,
WorkerTranscriptCommitClient,
} from "./worker-rpc-clients.js";
import { runWorkerDescriptor } from "./worker.runtime.js";
import { createWorkerRuntimeEnvironment, runWorkerDescriptor } from "./worker.runtime.js";
const browserRuntimeMocks = vi.hoisted(() => ({
createWorkerBrowserToolRuntime: vi.fn(),
@@ -111,6 +120,8 @@ type InferencePlan =
| "tool"
| "safe-tool"
| "background-tool"
| "process-poll"
| "process-kill"
| "session-tool"
| "hold"
| "fence"
@@ -125,6 +136,7 @@ type WorkerDoneMessage = Extract<WorkerInferenceTerminalOutcome, { type: "done"
type FakeGatewayOptions = {
admissionFailure?: "gateway-unavailable" | "invalid-credential" | "owner-epoch-mismatch";
backgroundCommand?: string;
execApprovals?: ExecApprovalsFile;
inferencePlans?: InferencePlan[];
outageOnInferenceCancel?: boolean;
@@ -569,6 +581,21 @@ class FakeWorkerGateway {
});
return;
}
if (plan === "process-poll" || plan === "process-kill") {
const processResult = this.acceptedTranscriptRequests
.flatMap((request) => request.messages)
.find((message) => message.role === "toolResult" && message.toolName === "exec");
const details = processResult?.role === "toolResult" ? processResult.details : undefined;
this.sendToolCallTurn(socket, frame.params, {
args: {
action: plan === "process-poll" ? "poll" : "kill",
sessionId: isRecord(details) ? details.sessionId : undefined,
},
toolCallId: plan,
toolName: "process",
});
return;
}
if (plan === "session-tool") {
this.sendSessionToolTurn(socket, frame.params);
return;
@@ -752,11 +779,12 @@ class FakeWorkerGateway {
? {
// POSIX sleep avoids Node startup; Windows keeps the portable Node fixture.
command:
process.platform === "win32"
this.options.backgroundCommand ??
(process.platform === "win32"
? `${JSON.stringify(process.execPath)} -e ${JSON.stringify(
"setInterval(() => undefined, 1000)",
)}`
: "exec sleep 60",
: "exec sleep 60"),
background: true,
}
: {
@@ -1533,6 +1561,264 @@ describe("worker runtime", () => {
});
});
it.each(["running", "completed", "cancelled"] as const)(
"keeps completed-turn background processes controllable in the managed environment (%s)",
async (processState) => {
const { gateway, launch, workspaceDir } = await setup({
inferencePlans: [
"background-tool",
"text",
...(processState === "cancelled" ? ["hold" as const] : []),
"process-poll",
...(processState === "completed" ? [] : ["process-kill" as const]),
"text",
],
...(processState === "completed"
? { backgroundCommand: `${JSON.stringify(process.execPath)} finish-on-file.cjs` }
: {}),
});
if (processState === "completed") {
await writeFile(
path.join(workspaceDir, "finish-on-file.cjs"),
"const fs = require('node:fs'); const finish = () => { if (fs.existsSync('finish-marker')) { process.stdout.write('background-finished'); watcher.close(); } }; const watcher = fs.watch('.', finish); finish();",
);
}
const scopeKey = `worker:${SESSION_ID}`;
const supervisor = getProcessSupervisor();
const input = new PassThrough();
const output = new PassThrough();
const results: WorkerProcessResult[] = [];
output.on("data", (chunk: Buffer) => {
const result = parseWorkerProcessResult(JSON.parse(chunk.toString("utf8")));
if (result) {
results.push(result);
}
});
const command = runWorkerCommand({ managed: true, input, output });
const settled = vi.fn();
void command.then(settled, settled);
try {
input.write(
JSON.stringify({ type: "turn", turnId: launch.assignment.turnId, descriptor: launch }) +
"\n",
);
await waitForFast(() => expect(results).toHaveLength(1), { timeout: 30_000 });
expect(results[0]).toMatchObject({
turnId: launch.assignment.turnId,
result: { status: "completed" },
retainWorker: true,
});
const running = listRunningSessions().filter((session) => session.scopeKey === scopeKey);
expect(running).toHaveLength(1);
const sessionId = running[0]!.id;
expect(settled).not.toHaveBeenCalled();
if (processState === "completed") {
await writeFile(path.join(workspaceDir, "finish-marker"), "finish");
await supervisor.waitForScope?.(scopeKey);
await waitForFast(() =>
expect(
listRunningSessions().filter((session) => session.scopeKey === scopeKey),
).toHaveLength(0),
);
expect(settled).not.toHaveBeenCalled();
}
const sendNextTurn = (index: number) => {
const next = structuredClone(launch);
next.assignment.runId = `worker-next-run-${index}`;
next.assignment.turnId = `worker-next-turn-${index}`;
next.assignment.operationalRunInstance = createOperationalRunInstanceRef(
next.assignment.runId,
);
next.assignment.agentRuntimeIdentityToken = `next-test-runtime-token-${index}`;
next.admission.credential = `next-test-worker-credential-${index}`;
next.assignment.initialMessages = gateway.acceptedTranscriptRequests.flatMap(
(request) => request.messages,
);
input.write(
`${JSON.stringify({ type: "turn", turnId: next.assignment.turnId, descriptor: next })}\n`,
);
return next.assignment.turnId;
};
const nextTurnId = sendNextTurn(2);
if (processState === "cancelled") {
await waitForFast(() => expect(gateway.inferenceRequests).toHaveLength(3), {
timeout: 30_000,
});
input.write(`${JSON.stringify({ type: "cancel", turnId: nextTurnId })}\n`);
await waitForFast(() => expect(results).toHaveLength(2), { timeout: 30_000 });
expect(results[1]).toMatchObject({
result: { status: "failed", reason: "turn-failed" },
retainWorker: true,
});
sendNextTurn(3);
}
const turnCount = processState === "cancelled" ? 3 : 2;
await waitForFast(() => expect(results).toHaveLength(turnCount), { timeout: 30_000 });
expect(gateway.connectionCount).toBe(turnCount);
const processResults = gateway.acceptedTranscriptRequests
.flatMap((request) => request.messages)
.filter((message) => message.role === "toolResult" && message.toolName === "process");
if (processState !== "completed") {
expect(processResults).toMatchObject([
{ details: { status: "running", sessionId } },
{ details: { status: "completed" } },
]);
} else {
expect(processResults).toMatchObject([
{
content: [{ type: "text", text: expect.stringContaining("background-finished") }],
details: { status: "completed", exitCode: 0 },
},
]);
expect(results[1]?.retainWorker).toBe(false);
}
} finally {
input.end();
try {
await command;
} finally {
supervisor.cancelScope(scopeKey, "manual-cancel");
await supervisor.waitForScope?.(scopeKey);
}
}
expect(listRunningSessions().filter((session) => session.scopeKey === scopeKey)).toHaveLength(
0,
);
},
);
it("joins retained background processes before closing the managed owner on EOF", async () => {
const { launch } = await setup({ inferencePlans: ["background-tool", "text"] });
const input = new PassThrough();
const output = new PassThrough();
const result = createDeferred<WorkerProcessResult>();
output.on("data", (chunk: Buffer) => {
const parsed = parseWorkerProcessResult(JSON.parse(chunk.toString("utf8")));
if (parsed) {
result.resolve(parsed);
}
});
const command = runWorkerCommand({ managed: true, input, output });
const scopeKey = `worker:${SESSION_ID}`;
const supervisor = getProcessSupervisor();
try {
input.write(
`${JSON.stringify({ type: "turn", turnId: launch.assignment.turnId, descriptor: launch })}\n`,
);
await expect(result.promise).resolves.toMatchObject({ retainWorker: true });
const running = listRunningSessions().filter((session) => session.scopeKey === scopeKey);
expect(running).toHaveLength(1);
const pid = running[0]!.pid!;
expect(pid).toBeGreaterThan(0);
input.end();
await command;
expect(() => process.kill(pid, 0)).toThrow();
expect(listRunningSessions().filter((session) => session.scopeKey === scopeKey)).toHaveLength(
0,
);
} finally {
input.end();
try {
await command;
} finally {
supervisor.cancelScope(scopeKey, "manual-cancel");
await supervisor.waitForScope?.(scopeKey);
}
}
});
it.each(["foreground", "hidden-background"] as const)(
"keeps environment state until %s exec finalization settles",
async (visibility) => {
const sessionId = `worker-finalizer-${visibility}`;
const scopeKey = `worker:${sessionId}`;
const environment = await createWorkerRuntimeEnvironment(sessionId);
const finalizing = createDeferred();
const releaseFinalizer = createDeferred();
const settledStateDirs: Array<string | undefined> = [];
let run: Awaited<ReturnType<typeof runExecProcess>> | undefined;
try {
run = await runExecProcess({
command: "worker-finalizer-fixture",
workdir: environment.stateDir,
env: {},
sandbox: {
containerName: "worker-finalizer-fixture",
workspaceDir: environment.stateDir,
containerWorkdir: environment.stateDir,
buildExecSpec: async () => ({
argv: [process.execPath, "-e", "process.stdout.write('worker-finalizer-output')"],
env: {},
stdinMode: "pipe-closed",
}),
finalizeExec: async () => {
finalizing.resolve();
await releaseFinalizer.promise;
},
},
usePty: false,
warnings: [],
maxOutput: 1000,
pendingMaxOutput: 1000,
notifyOnExit: false,
scopeKey,
timeoutSec: null,
onSettledBeforeNotify: () => {
settledStateDirs.push(process.env.OPENCLAW_STATE_DIR);
},
});
if (visibility === "hidden-background") {
markBackgrounded(run.session);
deleteSession(run.session.id);
}
await finalizing.promise;
await getProcessSupervisor().waitForScope?.(scopeKey);
const closing = environment.close();
await Promise.resolve();
expect(process.env.OPENCLAW_STATE_DIR).toBe(environment.stateDir);
await expect(stat(environment.stateDir)).resolves.toBeDefined();
releaseFinalizer.resolve();
await run.promise;
await closing;
expect(settledStateDirs).toEqual([environment.stateDir]);
await expect(stat(environment.stateDir)).rejects.toMatchObject({ code: "ENOENT" });
} finally {
releaseFinalizer.resolve();
await run?.promise;
await environment.close();
}
},
);
it("revokes local tool handles when their worker turn closes", async () => {
const { launch } = await setup();
const toolFactory = await import("../agents/agent-tools.finalize.js");
const finalize = vi.spyOn(toolFactory, "finalizeAgentTools");
try {
await expect(runWorkerDescriptor(launch)).resolves.toMatchObject({ status: "completed" });
const tools = finalize.mock.results[0]?.value as ReturnType<
typeof toolFactory.finalizeAgentTools
>;
const processTool = tools.find((tool) => tool.name === "process")!;
await expect(
processTool.execute("retained-process", { action: "list" }),
).rejects.toMatchObject({
name: "AbortError",
});
const execTool = tools.find((tool) => tool.name === "exec")!;
await expect(
execTool.execute("retained-exec", { command: "echo stale-worker" }),
).rejects.toMatchObject({
name: "AbortError",
});
} finally {
finalize.mockRestore();
}
});
it("stops worker-scoped background processes when fenced", async () => {
const { gateway, launch } = await setup({
inferencePlans: ["background-tool", "fence"],
+46 -19
View File
@@ -5,8 +5,10 @@ import {
WORKER_PORTAL_PROTOCOL_FEATURE,
type WorkerHelloOk,
} from "../../packages/gateway-protocol/src/schema/worker-admission.js";
import { waitForExecScope } from "../agents/bash-process-registry.js";
import { isPathInside } from "../infra/path-guards.js";
import { registerSecretValueForRedaction } from "../logging/secret-redaction-registry.js";
import { getProcessSupervisor } from "../process/supervisor/index.js";
import type { WorkerBrowserRuntime } from "./browser-runtime.js";
import { buildWorkerConnectParams, type WorkerLaunchDescriptor } from "./launch-descriptor.js";
import {
@@ -58,12 +60,49 @@ async function assertWorkerDirectory(pathname: string, label: string): Promise<s
return resolved;
}
/** Holds process-local state until every command owned by this environment has exited. */
export async function createWorkerRuntimeEnvironment(sessionId: string) {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-worker-"));
await chmod(stateDir, 0o700);
const previousStateDir = process.env.OPENCLAW_STATE_DIR;
const previousConfigPath = process.env.OPENCLAW_CONFIG_PATH;
process.env.OPENCLAW_STATE_DIR = stateDir;
process.env.OPENCLAW_CONFIG_PATH = path.join(stateDir, "openclaw.json");
let closing: Promise<void> | undefined;
return {
stateDir,
close: () =>
(closing ??= (async () => {
const supervisor = getProcessSupervisor();
const scopeKey = `worker:${sessionId}`;
supervisor.cancelScope(scopeKey, "manual-cancel");
await supervisor.waitForScope?.(scopeKey);
await waitForExecScope(scopeKey);
// Process completion writes its task outcome into this environment's state.
// Restore the ambient directory only after those callbacks have settled.
if (previousStateDir === undefined) {
delete process.env.OPENCLAW_STATE_DIR;
} else {
process.env.OPENCLAW_STATE_DIR = previousStateDir;
}
if (previousConfigPath === undefined) {
delete process.env.OPENCLAW_CONFIG_PATH;
} else {
process.env.OPENCLAW_CONFIG_PATH = previousConfigPath;
}
await rm(stateDir, { recursive: true, force: true });
})()),
};
}
export async function runWorkerDescriptor(
descriptor: WorkerLaunchDescriptor,
options: {
signal?: AbortSignal;
onConnectionFailure?: (cause: string | undefined) => void;
browserRuntime?: WorkerBrowserRuntime;
/** Supplied by the managed process owner, which closes state after its final turn. */
environmentStateDir?: string;
} = {},
): Promise<WorkerRuntimeResult> {
if (
@@ -86,12 +125,10 @@ export async function runWorkerDescriptor(
"worker workspace path escapes its assigned containment root; reprovision the worker workspace and retry",
);
}
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-worker-"));
await chmod(stateDir, 0o700);
const previousStateDir = process.env.OPENCLAW_STATE_DIR;
const previousConfigPath = process.env.OPENCLAW_CONFIG_PATH;
process.env.OPENCLAW_STATE_DIR = stateDir;
process.env.OPENCLAW_CONFIG_PATH = path.join(stateDir, "openclaw.json");
const environment = options.environmentStateDir
? undefined
: await createWorkerRuntimeEnvironment(descriptor.admission.sessionId);
const stateDir = options.environmentStateDir ?? environment!.stateDir;
const abortController = new AbortController();
let turnStarted = false;
@@ -214,7 +251,7 @@ export async function runWorkerDescriptor(
sessions: connection,
signal: abortController.signal,
});
if (options.signal?.aborted) {
if (options.signal?.aborted && !options.environmentStateDir) {
throw toWorkerRuntimeError(options.signal.reason, "worker interrupted");
}
} catch (error) {
@@ -222,7 +259,7 @@ export async function runWorkerDescriptor(
if (fenced) {
return fenced;
}
if (options.signal?.aborted) {
if (options.signal?.aborted && !options.environmentStateDir) {
throw toWorkerRuntimeError(options.signal.reason, "worker interrupted");
}
if (resultFenceAcked && connection.state.kind === "ready") {
@@ -256,16 +293,6 @@ export async function runWorkerDescriptor(
inference.dispose();
live.dispose();
await connection.stop();
if (previousStateDir === undefined) {
delete process.env.OPENCLAW_STATE_DIR;
} else {
process.env.OPENCLAW_STATE_DIR = previousStateDir;
}
if (previousConfigPath === undefined) {
delete process.env.OPENCLAW_CONFIG_PATH;
} else {
process.env.OPENCLAW_CONFIG_PATH = previousConfigPath;
}
await rm(stateDir, { recursive: true, force: true });
await environment?.close();
}
}
@@ -2,7 +2,9 @@ import { execFile } from "node:child_process";
import fs from "node:fs/promises";
import path from "node:path";
import { performance } from "node:perf_hooks";
import { setTimeout as delay } from "node:timers/promises";
import { promisify } from "node:util";
import { isRecord } from "@openclaw/normalization-core/record-coerce";
import { GatewayClient } from "openclaw/plugin-sdk/gateway-runtime";
import { afterEach, describe, expect, it } from "vitest";
import type { AuditRunInspectResult } from "../../../../packages/gateway-protocol/src/index.js";
@@ -91,6 +93,8 @@ describe("node worker launch wire", () => {
let finalizationStartedAt: number | undefined;
let resolveWaveFinalizationStarted: ((startedAt: number) => void) | undefined;
let workerAuditBeforeRestart: string | undefined;
let testFailure: { error: unknown } | undefined;
let cleanupFailures: unknown[];
try {
gateway = await startPairedNodeWorkerGateway({
@@ -166,7 +170,10 @@ describe("node worker launch wire", () => {
state: "active",
workerBundleHash: expect.stringMatching(/^[a-f0-9]{64}$/u),
});
const remoteWorkspaceDir = String(placement?.remoteWorkspaceDir ?? "");
const remoteWorkspaceDir = placement?.remoteWorkspaceDir;
if (typeof remoteWorkspaceDir !== "string" || !remoteWorkspaceDir) {
throw new Error("active worker placement did not expose a remote workspace directory");
}
const baseManifestRef = placement?.workspaceBaseManifestRef;
await expect(
fs.readFile(path.join(remoteWorkspaceDir, "gateway-push.txt"), "utf8"),
@@ -304,11 +311,14 @@ describe("node worker launch wire", () => {
fs.readFile(path.join(permissionLocalDir!, "worker-permission-in-root.txt"), "utf8"),
).resolves.toBe("worker permission proof\n");
// Simulate the old capability declaration with the current supervisor over real wire.
// This proves negotiation and same-identity reconnect, not an older binary upgrade.
legacyWorkerNode = await createPairedNodeWorkerHost({
gateway,
operator,
root,
label: "legacy-node",
environmentSession: false,
onInvoke: (frame) => {
if (frame.command === NODE_WORKER_BUNDLE_INSTALL_COMMAND && frame.paramsJSON) {
legacyBundlePrewarm = (JSON.parse(frame.paramsJSON) as { bundlePrewarm?: unknown })
@@ -330,6 +340,35 @@ describe("node worker launch wire", () => {
{ key: legacySessionKey, deviceId: legacyWorkerNode.identity.deviceId },
{ timeoutMs: PROOF_TIMEOUT_MS },
);
const unsupportedRunId = `node-worker-lifetime-unsupported-${Date.now()}`;
await expect(
operator.request("chat.send", {
sessionKey: legacySessionKey,
message: BASELINE_PROMPT,
deliver: false,
idempotencyKey: unsupportedRunId,
}),
).resolves.toMatchObject({ runId: unsupportedRunId, status: "started" });
await expect(
operator.request(
"agent.wait",
{ runId: unsupportedRunId, timeoutMs: PROOF_TIMEOUT_MS },
{ timeoutMs: PROOF_TIMEOUT_MS + 5_000 },
),
).resolves.toMatchObject({
status: "error",
error: expect.stringMatching(
/requires an update.*openclaw update.*reconnect.*openclaw node restart/su,
),
});
await legacyWorkerNode.waitForInvokes();
expect(legacyWorkerNode.commands).not.toContain(NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND);
await expect(
gateway.call("sessions.describe", { key: legacySessionKey }),
).resolves.toMatchObject({ session: { placement: { state: "active" } } });
await legacyWorkerNode.disconnect();
await legacyWorkerNode.connect({ environmentSession: true });
const legacyRunId = `node-worker-launch-wire-legacy-${Date.now()}`;
await operator.request("chat.send", {
sessionKey: legacySessionKey,
@@ -348,6 +387,18 @@ describe("node worker launch wire", () => {
expect(legacyWorkerNode.invokeErrors).toEqual([]);
expect(legacyWorkerNode.commands).toContain(NODE_WORKER_SUPERVISOR_LAUNCH_COMMAND);
expect(legacyBundlePrewarm).toBeUndefined();
const legacyHistory = await operator.request<{ messages?: unknown[] }>("chat.history", {
sessionKey: legacySessionKey,
limit: 20,
});
expect(
legacyHistory.messages?.filter(
(message) =>
isRecord(message) &&
message.role === "assistant" &&
wireMessageText(message).includes(BASELINE_REPLY),
),
).toHaveLength(1);
const loadSessions: string[] = [];
for (let index = 0; index < FINALIZATION_LOAD_CONCURRENCY; index += 1) {
@@ -369,10 +420,10 @@ describe("node worker launch wire", () => {
}
observeFinalizationLoad = true;
const readyzSamples: Array<{ atMs: number; latencyMs: number; status: number }> = [];
let loadSettled = false;
const samplerAbort = new AbortController();
const httpOrigin = gateway.wsUrl.replace(/^ws/u, "http");
const sampler = (async () => {
while (!loadSettled) {
while (!samplerAbort.signal.aborted) {
const startedAt = performance.now();
try {
const response = await fetch(`${httpOrigin}/readyz`, {
@@ -390,7 +441,7 @@ describe("node worker launch wire", () => {
status: 0,
});
}
await new Promise((resolve) => setTimeout(resolve, 50));
await delay(50);
}
})();
const freshConnectionSamples: number[] = [];
@@ -401,25 +452,25 @@ describe("node worker launch wire", () => {
});
const loadRunIds = await Promise.all(
loadSessions.map(async (sessionKey, index) => {
const runId = `node-worker-finalization-load-${wave}-${index}-${Date.now()}`;
const started = await operator!.request<{ runId?: string; status?: string }>(
const loadRunId = `node-worker-finalization-load-${wave}-${index}-${Date.now()}`;
const loadStarted = await operator!.request<{ runId?: string; status?: string }>(
"chat.send",
{
sessionKey,
message: BASELINE_PROMPT,
deliver: false,
idempotencyKey: runId,
idempotencyKey: loadRunId,
},
);
expect(started).toMatchObject({ runId, status: "started" });
return runId;
expect(loadStarted).toMatchObject({ runId: loadRunId, status: "started" });
return loadRunId;
}),
);
const waits = Promise.all(
loadRunIds.map(async (runId) => {
loadRunIds.map(async (loadRunId) => {
const completedLoad = await operator!.request<{ status?: string }>(
"agent.wait",
{ runId, timeoutMs: PROOF_TIMEOUT_MS },
{ runId: loadRunId, timeoutMs: PROOF_TIMEOUT_MS },
{ timeoutMs: PROOF_TIMEOUT_MS + 5_000 },
);
expect(completedLoad.status).toBe("ok");
@@ -438,7 +489,7 @@ describe("node worker launch wire", () => {
await waits;
}
} finally {
loadSettled = true;
samplerAbort.abort();
await Promise.allSettled([sampler]);
}
const finalizationSamples = readyzSamples.filter(
@@ -470,6 +521,8 @@ describe("node worker launch wire", () => {
JSON.parse(workerAuditAfterRestart) as AuditRunInspectResult,
);
expect(workerAuditAfterRestart).toBe(workerAuditBeforeRestart);
} catch (error) {
testFailure = { error };
} finally {
const cleanup = await Promise.allSettled([
workerNode?.stop() ?? Promise.resolve(),
@@ -479,15 +532,16 @@ describe("node worker launch wire", () => {
provider.stop(),
closeWireServer(published.server),
]);
const failures = cleanup.flatMap((result) =>
cleanupFailures = cleanup.flatMap((result) =>
result.status === "rejected" ? [result.reason] : [],
);
if (failures.length === 1) {
throw failures[0];
}
if (failures.length > 1) {
throw new AggregateError(failures, "node worker launch wire cleanup failed");
}
}
const failures = [...(testFailure ? [testFailure.error] : []), ...cleanupFailures];
if (failures.length === 1) {
throw failures[0];
}
if (failures.length > 1) {
throw new AggregateError(failures, "node worker launch wire test failed");
}
},
);
@@ -1,7 +1,16 @@
import fs from "node:fs/promises";
import { GatewayClient } from "openclaw/plugin-sdk/gateway-runtime";
import { afterEach, describe, expect, it, vi } from "vitest";
import { NODE_WORKER_WORKSPACE_RETAIN_COMMAND } from "../../../../src/infra/node-commands.js";
import {
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../../../../src/infra/kysely-sync.js";
import {
NODE_WORKER_ENVIRONMENT_STOP_COMMAND,
NODE_WORKER_WORKSPACE_RETAIN_COMMAND,
} from "../../../../src/infra/node-commands.js";
import { withOpenClawStateDatabaseReadOnly } from "../../../../src/state/openclaw-state-db-readonly.js";
import type { DB as StateDatabase } from "../../../../src/state/openclaw-state-db.generated.js";
import { useAutoCleanupTempDirTracker } from "../../../helpers/temp-dir.js";
import { PROOF_TIMEOUT_MS } from "./cloud-worker-midturn-loss-fixture.js";
import { startPairedNodeWorkerLifecycleProvider } from "./paired-node-worker-lifecycle-provider.js";
@@ -29,6 +38,7 @@ type Placement = {
activeOwnerEpoch?: number;
environmentId?: string;
generation?: number;
recoveryError?: string;
state?: string;
workerBundleHash?: string;
};
@@ -36,7 +46,7 @@ type EnvironmentRead = {
id: string;
type: string;
workerBundle?: WireNodeRead["workerBundle"];
worker?: { state?: string };
worker?: { state?: string; attachedSessionIds?: string[]; tunnelStatus?: string };
};
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
@@ -241,6 +251,17 @@ describe("paired node worker lifecycle wire", () => {
)) as { placement?: Placement };
expect(movedLocal.placement).toMatchObject({ state: "local" });
expect(await describePlacement(gateway, retainedKey)).toMatchObject({ state: "local" });
await workerNode.waitForInvokes();
expect(
workerNode.frames
.filter((frame) => frame.command === NODE_WORKER_ENVIRONMENT_STOP_COMMAND)
.map((frame) => JSON.parse(frame.paramsJSON!)),
).toContainEqual(
expect.objectContaining({
environmentId: sourcePlacement.environmentId,
ownerEpoch: sourcePlacement.activeOwnerEpoch,
}),
);
await expectSuccessfulTurn({ operator, key: retainedKey, marker: "WIRE-MOVED-LOCAL" });
await dispatchNodeSession({ gateway, key: retainedKey, nodeId });
await expectSuccessfulTurn({ operator, key: retainedKey, marker: "WIRE-MOVED-BACK" });
@@ -347,10 +368,22 @@ describe("paired node worker lifecycle wire", () => {
provider.release(HOLD_B);
await expect(waitForTurn(operator, holdBRunId)).resolves.toMatchObject({ status: "ok" });
// Node-role removal waits for environment and placement cleanup before returning,
// invalidates the old connection, and leaves local execution available.
// Revocation fences placement before responding but cannot prove remote extinction.
// Keep the exact attachment pending cleanup rather than invent a terminal environment.
const removalKey = await createSession({ operator, published, suffix: "role-removal" });
const removalPlacement = await dispatchNodeSession({ gateway, key: removalKey, nodeId });
const removalEnvironment = (await readEnvironments(operator)).find(
(entry) => entry.id === removalPlacement.environmentId,
);
const attachedSessionIds = removalEnvironment?.worker?.attachedSessionIds;
if (
!removalPlacement.environmentId ||
typeof removalPlacement.activeOwnerEpoch !== "number" ||
attachedSessionIds?.length !== 1
) {
throw new Error("role-removal placement did not expose exact ownership");
}
const removalEnvironmentId = removalPlacement.environmentId;
expect(workerNode.client).toBeTruthy();
await expect(operator.request("node.pair.remove", { nodeId })).resolves.toMatchObject({
nodeId,
@@ -358,8 +391,51 @@ describe("paired node worker lifecycle wire", () => {
const removedEnvironment = (await readEnvironments(operator)).find(
(entry) => entry.id === removalPlacement.environmentId,
);
expect(["destroyed", "failed", "orphaned"]).toContain(removedEnvironment?.worker?.state);
expect(await describePlacement(gateway, removalKey)).toMatchObject({ state: "failed" });
expect(removedEnvironment?.worker).toMatchObject({
state: "attached",
attachedSessionIds,
tunnelStatus: "stopped",
});
withOpenClawStateDatabaseReadOnly(
({ db }) => {
const query = getNodeSqliteKysely<StateDatabase>(db);
expect(
executeSqliteQueryTakeFirstSync(
db,
query
.selectFrom("worker_environments")
.select([
"owner_epoch",
"attached_session_ids_json",
"destroy_requested_at_ms",
"teardown_terminal_state",
"last_error",
])
.where("environment_id", "=", removalEnvironmentId),
),
).toMatchObject({
owner_epoch: removalPlacement.activeOwnerEpoch,
attached_session_ids_json: JSON.stringify(attachedSessionIds),
destroy_requested_at_ms: expect.any(Number),
teardown_terminal_state: "failed",
last_error: "Worker provider no longer recognizes the lease",
});
expect(
executeSqliteQueryTakeFirstSync(
db,
query
.selectFrom("worker_environment_credentials")
.select("environment_id")
.where("environment_id", "=", removalEnvironmentId),
),
).toBeUndefined();
},
{ env: gateway.runtimeEnv },
);
expect(await describePlacement(gateway, removalKey)).toMatchObject({
state: "failed",
recoveryError: expect.stringContaining("not connected"),
});
await expect(workerNode.publishInventory()).rejects.toBeTruthy();
await vi.waitFor(
async () => {
@@ -21,6 +21,7 @@ import {
NODE_RUNNER_INVENTORY_UPDATE_METHOD,
NODE_WORKER_BUNDLE_RETENTION_VERSION,
NODE_WORKER_BUNDLE_STATUS_VERSION,
NODE_WORKER_ENVIRONMENT_SESSION_VERSION,
NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE,
} from "../../../../src/infra/node-runner-inventory.js";
import { handleInvoke, type NodeInvokeRequestPayload } from "../../../../src/node-host/invoke.js";
@@ -277,6 +278,7 @@ type WireWorkerHostOptions = {
bundlePrewarm?: boolean;
bundleRetention?: boolean;
bundleStatus?: boolean;
environmentSession?: boolean;
onInvoke?: (frame: NodeInvokeRequestPayload) => void;
afterInvoke?: (frame: NodeInvokeRequestPayload, host: PairedNodeWorkerHost) => Promise<void>;
};
@@ -290,7 +292,7 @@ export type PairedNodeWorkerHost = {
readonly bundleInstaller: NodeWorkerBundleInstaller;
readonly workspace: NodeWorkerWorkspaceRuntime;
readonly client: GatewayClient | undefined;
connect(): Promise<void>;
connect(options?: { environmentSession?: boolean }): Promise<void>;
disconnect(): Promise<void>;
publishInventory(): Promise<void>;
waitForInvokes(): Promise<void>;
@@ -316,6 +318,7 @@ export async function createPairedNodeWorkerHost(
const workspace = new NodeWorkerWorkspaceRuntime({ root: nodeHostRoot, env: nodeEnv });
const bundleInstaller = new NodeWorkerBundleInstaller({ root: nodeHostRoot, env: nodeEnv });
let capacity = { total: options.capacity ?? 2, available: 0 };
let environmentSession = options.environmentSession ?? true;
let client: GatewayClient | undefined;
let closing = false;
const invokeTasks = new Set<Promise<void>>();
@@ -331,6 +334,9 @@ export async function createPairedNodeWorkerHost(
protocolFeatures: [NODE_WORKER_SUPERVISOR_PROTOCOL_FEATURE],
workerHost: {
enabled: true as const,
...(environmentSession
? { environmentSession: NODE_WORKER_ENVIRONMENT_SESSION_VERSION }
: {}),
capacity,
...(options.bundlePrewarm ? { bundlePrewarm: WORKER_BUNDLE_PREWARM_VERSION } : {}),
...(options.bundleRetention ? { bundleRetention: NODE_WORKER_BUNDLE_RETENTION_VERSION } : {}),
@@ -379,10 +385,11 @@ export async function createPairedNodeWorkerHost(
invokeTasks.add(task);
};
const connect = async () => {
const connect = async (connection?: { environmentSession?: boolean }) => {
if (closing) {
throw new Error("paired worker node is closing");
}
environmentSession = connection?.environmentSession ?? environmentSession;
const open = () =>
connectWireClient({
gateway: options.gateway,
@@ -444,9 +451,11 @@ export async function createPairedNodeWorkerHost(
const receipts = await Promise.all(
[...launchIds].map(async (launchId) => await supervisor.status(launchId)),
);
return receipts.every(
(receipt) => receipt !== undefined && !["pending", "running"].includes(receipt.state),
)
// Finished turns do not prove the physical worker or container has been removed.
return capacity.available === capacity.total &&
receipts.every(
(receipt) => receipt !== undefined && !["pending", "running"].includes(receipt.state),
)
? true
: undefined;
});