fix(swarm): keep collector results reliable through races and restarts (#112989)

* fix(swarm): harden collector lifecycle and dashboards

* fix(swarm): initialize collector completion state

* test(swarm): satisfy cross-environment type checks

* test(codex): allow direct request handler calls

* style(ui): avoid Swarm widget shadowing

* test(swarm): keep internal helpers private

* refactor(ui): own Swarm roster helpers in runtime
This commit is contained in:
Peter Steinberger
2026-07-23 06:26:31 -07:00
committed by GitHub
parent 09f5f2d305
commit ad505a7b55
40 changed files with 1638 additions and 233 deletions
+11 -5
View File
@@ -507,6 +507,11 @@ toward native `spawn_agent` for Codex-native subagent work, while
Message-tool-only source replies also stay direct, since that is a
turn-control contract.
Codex Code Mode projects generic OpenClaw dynamic-tool results as text. Parse a
JSON result before reading fields. Nested dynamic calls are serialized by the
Codex runtime, so `Promise.all` does not submit them concurrently; use a
bounded sequential launch loop when starting collector children.
Tools marked `catalogMode: "direct-only"`, including the OpenClaw `computer`
tool, are grouped under `openclaw_direct`. OpenClaw adds that namespace to
Codex's `code_mode.direct_only_tool_namespaces` list without replacing
@@ -534,15 +539,16 @@ first available timeout in this order:
converted to milliseconds, or the 60 second media default. For image
understanding, this applies to the request itself and is not reduced by
earlier preparation work.
- For the `message` tool, a fixed 120 second default.
- For the `message` tool, a fixed 600 second outer budget that covers Gateway delivery and bounded same-key reconciliation.
- The 90 second dynamic-tool default.
This watchdog is the outer dynamic `item/tool/call` budget. Provider-specific
request timeouts run inside that call and keep their own timeout semantics.
Dynamic tool budgets are capped at 600000 ms. On timeout, OpenClaw aborts the
tool signal where supported and returns a failed dynamic-tool response to
Codex so the turn can continue instead of leaving the session in
`processing`.
Dynamic tool budgets are capped at 600000 ms. `agents_wait` adds 30000 ms of
outer completion grace, and the app-server client allows 660000 ms so that
structured wait result can reach Codex. On timeout, OpenClaw aborts the tool
signal where supported and returns a failed dynamic-tool response to Codex so
the turn can continue instead of leaving the session in `processing`.
After Codex accepts a turn, and after OpenClaw responds to a turn-scoped
app-server request, the harness expects Codex to make current-turn progress
+7 -4
View File
@@ -149,10 +149,13 @@ hooks such as `SessionStart` and `UserPromptSubmit` remain Codex-level
controls; they are not exposed as OpenClaw plugin hooks in the v1 contract.
For OpenClaw dynamic tools, OpenClaw executes the tool after Codex asks for
the call, so plugin and middleware behavior runs in the harness adapter. For
Codex-native tools, Codex owns the canonical tool record; OpenClaw can mirror
selected events but cannot rewrite the native thread unless Codex exposes that
through app-server or native hook callbacks.
the call, so plugin and middleware behavior runs in the harness adapter. Codex
Code Mode receives generic dynamic results as text and serializes nested
dynamic calls; callers must parse JSON-looking results and cannot rely on
`Promise.all` for concurrent submission. For Codex-native tools, Codex owns the
canonical tool record; OpenClaw can mirror selected events but cannot rewrite
the native thread unless Codex exposes that through app-server or native hook
callbacks.
Codex app-server report-mode `PreToolUse` events defer plugin approval to the
matching app-server approval. If an OpenClaw `before_tool_call` hook returns
+12 -5
View File
@@ -789,11 +789,18 @@ Persistent effective search-policy changes rotate the bound Codex thread
before the next turn; transient per-turn restrictions use a temporary
restricted thread and preserve the existing binding for later resume.
`sessions_yield` and message-tool-only source replies stay direct because
those are turn-control contracts. `sessions_spawn` stays searchable so
Codex's native `spawn_agent` remains the primary Codex subagent surface,
while explicit OpenClaw or ACP delegation is still available through the
`openclaw` dynamic tool namespace. Heartbeat collaboration instructions
`sessions_yield`, `sessions_spawn`, and message-tool-only source replies stay
direct because they are turn-control or delegation contracts. Guidance still
prefers Codex's native `spawn_agent` as the primary Codex subagent surface,
while explicit OpenClaw or ACP delegation remains directly callable through
`sessions_spawn`. In Codex Code Mode, generic OpenClaw
dynamic-tool results are JSON text rather than JavaScript objects, so parse
JSON-looking results before reading fields. Codex also serializes nested
dynamic calls; submit several `sessions_spawn` calls in a bounded loop rather
than expecting `Promise.all` to launch them concurrently. Already-accepted
children can still overlap while later calls are submitted. See
[Swarm](/tools/swarm#use-swarm-from-other-harnesses) for a complete pattern.
Heartbeat collaboration instructions
tell Codex to search for `heartbeat_respond` before ending a heartbeat turn
when the tool is not already loaded.
+30 -12
View File
@@ -172,6 +172,12 @@ return await agents.run(
`maxConcurrent` children for the group and queues the rest in submission
order.
Code Mode separately bounds concurrent guest bridge calls with
`tools.codeMode.maxPendingToolCalls` (default `16`, maximum `128`). For very
large groups, launch bounded batches below that limit and leave headroom for
`phase()`, `log()`, and child wait transitions. `maxConcurrent` limits running
children; it does not raise the guest bridge-call limit.
### Loop on a decision gate
Use a bounded `while` loop when each pass decides whether another pass is
@@ -328,29 +334,39 @@ calls.
Codex Code Mode automatically exposes eligible dynamic OpenClaw tools under
`tools.*`. It does not use OpenClaw's QuickJS guest API or require
`tools.codeMode`, but `tools.swarm` must still be enabled. Codex harness
`agents_wait` calls support the full 600-second timeout. Use this pattern:
`agents_wait` calls support the full 600-second timeout.
With the currently supported Codex runtime, dynamic OpenClaw tool results reach
Code Mode as JSON text. Parse each result before reading fields. Codex also
serializes dynamic tool calls, so `Promise.all` does not submit several
`sessions_spawn` calls concurrently. Launch collectors in a bounded loop;
already-accepted children can still run while later launches are submitted.
```javascript
function parseToolResult(value) {
if (typeof value !== "string") return value;
return JSON.parse(value);
}
const tasks = [
"Check the authentication path.",
"Check the storage path.",
"Check the recovery path.",
];
const launches = [];
const launches = await Promise.all(
tasks.map((task, index) =>
tools.sessions_spawn({
for (const [index, task] of tasks.entries()) {
const launch = parseToolResult(
await tools.sessions_spawn({
task,
collect: true,
label: `review-${index + 1}`,
}),
),
);
for (const launch of launches) {
);
if (launch.status !== "accepted") {
throw new Error(launch.error ?? "Collector spawn was not accepted.");
}
launches.push(launch);
}
const pending = new Set(launches.map((launch) => launch.runId));
@@ -358,10 +374,12 @@ const completed = [];
while (pending.size > 0) {
const ids = [...pending].slice(0, 1000);
const batch = await tools.agents_wait({
ids,
timeoutSeconds: 30,
});
const batch = parseToolResult(
await tools.agents_wait({
ids,
timeoutSeconds: 30,
}),
);
// Rotate this bounded window behind ids that have not been checked yet.
for (const runId of ids) {
@@ -10,7 +10,7 @@ import { resetSharedCodexAppServerClientForTests } from "./shared-client.js";
import { createClientHarness } from "./test-support.js";
import { MAX_CODEX_APP_SERVER_VERSION, MIN_CODEX_APP_SERVER_VERSION } from "./version.js";
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 600_000;
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 660_000;
describe("CodexAppServerClient", () => {
const clients: CodexAppServerClient[] = [];
@@ -585,8 +585,10 @@ describe("CodexAppServerClient", () => {
const warn = vi.spyOn(embeddedAgentLog, "warn").mockImplementation(() => undefined);
const harness = createClientHarness();
clients.push(harness.client);
harness.client.addRequestHandler((request) => {
let requestSignal: AbortSignal | undefined;
harness.client.addRequestHandler((request, signal) => {
if (request.method === "item/tool/call") {
requestSignal = signal;
return new Promise<never>(() => {});
}
return undefined;
@@ -608,6 +610,7 @@ describe("CodexAppServerClient", () => {
],
},
});
expect(requestSignal?.aborted).toBe(true);
expect(warn).toHaveBeenCalledWith("codex app-server server request timed out", {
id: "srv-timeout",
method: "item/tool/call",
+13 -4
View File
@@ -34,7 +34,9 @@ import { MAX_CODEX_APP_SERVER_VERSION, MIN_CODEX_APP_SERVER_VERSION } from "./ve
const CODEX_APP_SERVER_PARSE_LOG_MAX = 500;
const CODEX_APP_SERVER_PARSE_BUFFER_MAX = 8 * 1024 * 1024;
const CODEX_APP_SERVER_PARSE_BUFFER_MAX_LINES = 1_000;
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 600_000;
// agents_wait can use a 600s inner budget plus 30s handler grace. Keep the
// app-server request guard outside that window so Codex receives the tool result.
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 660_000;
const CODEX_APP_SERVER_STDERR_TAIL_MAX = 2_000;
const CODEX_APP_SERVER_OVERLOADED_ERROR_CODE = -32_001;
const CODEX_APP_SERVER_OVERLOAD_MAX_RETRIES = 3;
@@ -204,6 +206,7 @@ export function isCodexAppServerConnectionClosedError(error: unknown): boolean {
type CodexServerRequestHandler = (
request: Required<Pick<RpcRequest, "id" | "method">> & { params?: JsonValue },
signal?: AbortSignal,
) => Promise<JsonValue | undefined> | JsonValue | undefined;
/** Notification handler registered on a Codex app-server client. */
@@ -870,15 +873,16 @@ export class CodexAppServerClient {
private async runServerRequestHandlers(
request: Required<Pick<RpcRequest, "id" | "method">> & { params?: JsonValue },
): Promise<JsonValue | undefined> {
const controller = new AbortController();
const timeoutResponse = timeoutServerRequestResponse(request);
if (!timeoutResponse) {
return await this.runServerRequestHandlersWithoutTimeout(request);
return await this.runServerRequestHandlersWithoutTimeout(request, controller.signal);
}
let timeout: ReturnType<typeof setTimeout> | undefined;
try {
return await Promise.race([
this.runServerRequestHandlersWithoutTimeout(request),
this.runServerRequestHandlersWithoutTimeout(request, controller.signal),
new Promise<JsonValue>((resolve) => {
timeout = setTimeout(() => {
embeddedAgentLog.warn("codex app-server server request timed out", {
@@ -886,6 +890,7 @@ export class CodexAppServerClient {
method: request.method,
timeoutMs: CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS,
});
controller.abort(new Error("codex app-server server request timed out"));
resolve(timeoutResponse);
}, CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS);
timeout.unref?.();
@@ -900,9 +905,13 @@ export class CodexAppServerClient {
private async runServerRequestHandlersWithoutTimeout(
request: Required<Pick<RpcRequest, "id" | "method">> & { params?: JsonValue },
signal: AbortSignal,
): Promise<JsonValue | undefined> {
for (const handler of this.requestHandlers) {
const result = await handler(request);
if (signal.aborted) {
return undefined;
}
const result = await handler(request, signal);
if (result !== undefined) {
return result;
}
@@ -19,6 +19,7 @@ const CODEX_DYNAMIC_TOOL_TIMEOUT_MS = 90_000;
const CODEX_DYNAMIC_TOOL_MAX_TIMEOUT_MS = 600_000;
const CODEX_DYNAMIC_IMAGE_TOOL_TIMEOUT_MS = 60_000;
const CODEX_DYNAMIC_MESSAGE_TOOL_TIMEOUT_MS = CODEX_DYNAMIC_TOOL_MAX_TIMEOUT_MS;
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 660_000;
describe("dynamic tool execution helpers", () => {
afterEach(() => {
@@ -138,9 +139,11 @@ describe("dynamic tool execution helpers", () => {
config: {
agents: {
defaults: {
imageGenerationModel: {
primary: "openai/gpt-image-1",
timeoutMs: 180_000,
mediaModels: {
image: {
primary: "openai/gpt-image-1",
timeoutMs: 180_000,
},
},
},
},
@@ -361,12 +364,12 @@ describe("dynamic tool execution helpers", () => {
config: undefined,
}),
).toBe(150_000);
expect(
resolveDynamicToolCallTimeoutMs({
call: { ...call, arguments: { ids: ["run-1"], timeoutSeconds: 600 } },
config: undefined,
}),
).toBe(630_000);
const fullWaitTimeoutMs = resolveDynamicToolCallTimeoutMs({
call: { ...call, arguments: { ids: ["run-1"], timeoutSeconds: 600 } },
config: undefined,
});
expect(fullWaitTimeoutMs).toBe(630_000);
expect(CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS).toBeGreaterThan(fullWaitTimeoutMs);
});
it("returns a failed dynamic tool response when an app-server tool call exceeds the deadline", async () => {
@@ -47,7 +47,7 @@ const CODEX_DYNAMIC_COMPUTER_COMPLETION_GRACE_MS = 30_000;
/** Timeout for image-understanding style dynamic tool calls. */
const CODEX_DYNAMIC_IMAGE_TOOL_TIMEOUT_MS = 60_000;
/** Timeout for message-delivery dynamic tool calls. */
const CODEX_DYNAMIC_MESSAGE_TOOL_TIMEOUT_MS = CODEX_DYNAMIC_TOOL_MAX_TIMEOUT_MS;
const CODEX_DYNAMIC_MESSAGE_TOOL_TIMEOUT_MS = 600_000;
/** Outer default for collector waits: full swarm budget plus completion grace. */
const CODEX_DYNAMIC_AGENTS_WAIT_TOOL_TIMEOUT_MS =
CODEX_DYNAMIC_TOOL_MAX_TIMEOUT_MS + CODEX_DYNAMIC_TOOL_TIMEOUT_SECONDS_GRACE_MS;
@@ -554,12 +554,12 @@ function readConfiguredDynamicToolTimeoutMs(
config: EmbeddedRunAttemptParams["config"],
): number | undefined {
if (toolName === "image_generate") {
const imageGenerationModel = config?.agents?.defaults?.imageGenerationModel;
if (!imageGenerationModel || typeof imageGenerationModel !== "object") {
const imageModel = config?.agents?.defaults?.mediaModels?.image;
if (!imageModel || typeof imageModel !== "object") {
return CODEX_DYNAMIC_IMAGE_GENERATION_TOOL_TIMEOUT_MS;
}
return (
readPositiveFiniteTimeoutMs(imageGenerationModel.timeoutMs) ??
readPositiveFiniteTimeoutMs(imageModel.timeoutMs) ??
CODEX_DYNAMIC_IMAGE_GENERATION_TOOL_TIMEOUT_MS
);
}
@@ -66,7 +66,9 @@ export function createCodexAttemptServerRequestController(
const handleServerRequest = async (
request: CodexAppServerServerRequest,
scope: CodexThreadRouteScope,
requestSignal: AbortSignal = new AbortController().signal,
) => {
const signal = AbortSignal.any([runAbortController.signal, requestSignal]);
const turnId = turnIdRef.current;
const projector = projectorRef.current;
let armCompletionWatchOnResponse = false;
@@ -96,7 +98,7 @@ export function createCodexAttemptServerRequestController(
...(computerUseConfig.enabled
? { computerUseMcpServerName: computerUseConfig.mcpServerName }
: {}),
signal: runAbortController.signal,
signal,
});
}
if (request.method === "item/tool/requestUserInput") {
@@ -123,7 +125,7 @@ export function createCodexAttemptServerRequestController(
turnId,
nativeHookRelay: resourceState.nativeHookRelay,
autoApprove: shouldAutoApproveCodexAppServerApprovals(appServer),
signal: runAbortController.signal,
signal,
onNativeToolFailureDisposition: (itemId, disposition) =>
projector?.recordNativeToolApprovalFailure(itemId, disposition),
});
@@ -210,7 +212,7 @@ export function createCodexAttemptServerRequestController(
handleDynamicToolCallWithTimeout({
call,
toolBridge,
signal: runAbortController.signal,
signal,
timeoutMs: dynamicToolTimeoutMs,
toolMeta,
toolCallOrdinal,
@@ -5,6 +5,8 @@ import type { JsonValue } from "./protocol.js";
import { createClientHarness } from "./test-support.js";
import { getCodexAppServerTurnRouter, type CodexAppServerServerRequest } from "./turn-router.js";
const CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS = 660_000;
type ClientHarness = ReturnType<typeof createClientHarness>;
type WireResponse = {
@@ -46,6 +48,31 @@ describe("CodexAppServerTurnRouter", () => {
expect(addCloseHandler).toHaveBeenCalledTimes(1);
});
it("does not dispatch a request that times out before route activation", async () => {
vi.useFakeTimers();
vi.spyOn(embeddedAgentLog, "warn").mockImplementation(() => undefined);
const harness = createHarness();
const requestHandler = vi.fn(() => ({ executed: true }));
const route = getCodexAppServerTurnRouter(harness.client).reserveThread({
threadId: "thread-late",
});
harness.send({
id: "request-late",
method: "item/tool/call",
params: { threadId: "thread-late", turnId: "turn-late", tool: "message" },
});
await vi.advanceTimersByTimeAsync(CODEX_DYNAMIC_TOOL_SERVER_REQUEST_TIMEOUT_MS);
expect(await waitForResponse(harness, "request-late")).toMatchObject({
id: "request-late",
result: { success: false },
});
await route.activate({ onRequest: requestHandler });
expect(requestHandler).not.toHaveBeenCalled();
});
it("routes concurrent traffic to the exact thread and turn", async () => {
const harness = createHarness();
const router = getCodexAppServerTurnRouter(harness.client);
+54 -16
View File
@@ -26,6 +26,7 @@ export type CodexThreadRouteScope = {
type CodexThreadRequestHandler = (
request: CodexAppServerServerRequest,
scope: CodexThreadRouteScope,
signal: AbortSignal,
) => Promise<JsonValue | undefined> | JsonValue | undefined;
type CodexThreadNotificationHandler = (
notification: CodexServerNotification,
@@ -127,7 +128,7 @@ class ClientTurnRouter implements CodexAppServerTurnRouter {
constructor(client: CodexAppServerClient) {
client.addNotificationHandler((notification) => this.routeNotification(notification));
client.addRequestHandler((request) => this.routeRequest(request));
client.addRequestHandler((request, signal) => this.routeRequest(request, signal));
client.addCloseHandler(() => this.dispose());
}
@@ -353,8 +354,11 @@ class ClientTurnRouter implements CodexAppServerTurnRouter {
return route.notificationTail;
}
private async routeRequest(request: CodexAppServerServerRequest): Promise<JsonValue | undefined> {
if (this.disposed) {
private async routeRequest(
request: CodexAppServerServerRequest,
signal: AbortSignal = new AbortController().signal,
): Promise<JsonValue | undefined> {
if (this.disposed || signal.aborted) {
return undefined;
}
const scope = readScope(request.params);
@@ -365,10 +369,10 @@ class ClientTurnRouter implements CodexAppServerTurnRouter {
if (!route || route.released) {
return undefined;
}
if (!route.handlers) {
await route.activated.promise;
if (!route.handlers && !(await waitForPromiseOrAbort(route.activated.promise, signal))) {
return undefined;
}
if (route.released || !route.handlers) {
if (signal.aborted || route.released || !route.handlers) {
return undefined;
}
const handler = route.handlers.onRequest;
@@ -378,8 +382,11 @@ class ClientTurnRouter implements CodexAppServerTurnRouter {
// Open routes service a resumed native turn. Arming starts the handoff to a
// new OpenClaw turn, whose requests must wait for its accepted turn id.
while (route.gate === "armed") {
await route.binding?.promise;
if (route.released) {
const binding = route.binding?.promise;
if (!binding || !(await waitForPromiseOrAbort(binding, signal))) {
return undefined;
}
if (signal.aborted || route.released) {
return undefined;
}
}
@@ -391,18 +398,24 @@ class ClientTurnRouter implements CodexAppServerTurnRouter {
return undefined;
}
}
await this.waitForNotifications(route);
if (route.released) {
if (!(await waitForPromiseOrAbort(this.waitForNotifications(route), signal))) {
return undefined;
}
if (signal.aborted || route.released) {
return undefined;
}
try {
const result = await handler(request, {
threadId: scope.threadId,
...(scope.turnId ? { turnId: scope.turnId } : {}),
});
return route.released ? undefined : result;
const result = await handler(
request,
{
threadId: scope.threadId,
...(scope.turnId ? { turnId: scope.turnId } : {}),
},
signal,
);
return signal.aborted || route.released ? undefined : result;
} catch (error) {
if (route.released) {
if (signal.aborted || route.released) {
return undefined;
}
throw error;
@@ -591,6 +604,31 @@ function isCodexTerminalTurnNotification(notification: CodexServerNotification):
);
}
async function waitForPromiseOrAbort(
promise: Promise<unknown>,
signal: AbortSignal,
): Promise<boolean> {
if (signal.aborted) {
return false;
}
let removeAbort: (() => void) | undefined;
try {
return await Promise.race([
promise.then(() => true),
new Promise<boolean>((resolve) => {
const onAbort = () => resolve(false);
signal.addEventListener("abort", onAbort, { once: true });
removeAbort = () => signal.removeEventListener("abort", onAbort);
if (signal.aborted) {
onAbort();
}
}),
]);
} finally {
removeAbort?.();
}
}
function deferred(): Deferred {
let resolve!: () => void;
const promise = new Promise<void>((resolvePromise) => {
@@ -4,12 +4,16 @@ import { promises as fs } from "node:fs";
import { afterEach, describe, expect, it, vi } from "vitest";
import { defaultRuntime } from "../runtime.js";
import {
backfillCollectorArchiveAtMs,
capFrozenResultText,
logAnnounceGiveUp,
reconcileOrphanedRestoredRuns,
reconcileOrphanedRun,
resolveSubagentArchiveAtMs,
safeRemoveAttachmentsDir,
} from "./subagent-registry-helpers.js";
import type { SubagentRunRecord } from "./subagent-registry.types.js";
import { updateSwarmCollectorCompletion } from "./swarm-collector.js";
function createRunEntry(overrides: Partial<SubagentRunRecord> = {}): SubagentRunRecord {
return {
@@ -36,6 +40,124 @@ describe("capFrozenResultText", () => {
});
});
describe("resolveSubagentArchiveAtMs", () => {
const cfg = { agents: { defaults: { subagents: { archiveAfterMinutes: 5 } } } };
it("defers collector retention until terminal completion", () => {
for (const cleanup of ["keep", "delete"] as const) {
expect(
resolveSubagentArchiveAtMs({
cfg,
now: 1_000,
spawnMode: "run",
cleanup,
collect: true,
}),
).toBeUndefined();
}
});
it("starts collector retention when terminal completion is frozen", () => {
const entry = createRunEntry({
collect: true,
endedAt: 2_000,
outcome: { status: "ok" },
completion: { required: false, resultText: "done", capturedAt: 2_000 },
});
expect(updateSwarmCollectorCompletion(entry, cfg)).toBe(true);
expect(entry.collectorCompletion).toEqual({ status: "done" });
expect(entry.archiveAtMs).toBe(302_000);
});
it("starts retention when a delayed result first becomes waitable", () => {
vi.useFakeTimers();
vi.setSystemTime(10_000);
const entry = createRunEntry({
collect: true,
endedAt: 2_000,
outcome: { status: "ok" },
completion: { required: false, resultText: "done" },
});
expect(updateSwarmCollectorCompletion(entry, cfg)).toBe(true);
expect(entry.completion?.capturedAt).toBe(10_000);
expect(entry.archiveAtMs).toBe(310_000);
vi.useRealTimers();
});
it("backfills legacy collectors from their terminal time", () => {
const entry = createRunEntry({
collect: true,
endedAt: 2_000,
archiveAtMs: 10_000,
});
expect(backfillCollectorArchiveAtMs(entry, cfg)).toBe(true);
expect(entry.archiveAtMs).toBe(302_000);
expect(backfillCollectorArchiveAtMs(entry, cfg)).toBe(false);
});
it("clears stale deadlines from active, persistent, or retention-disabled collectors", () => {
const active = createRunEntry({ collect: true, archiveAtMs: 10_000 });
expect(backfillCollectorArchiveAtMs(active, cfg)).toBe(true);
expect(active.archiveAtMs).toBeUndefined();
const persistent = createRunEntry({
collect: true,
spawnMode: "session",
endedAt: 2_000,
archiveAtMs: 10_000,
});
expect(backfillCollectorArchiveAtMs(persistent, cfg)).toBe(true);
expect(persistent.archiveAtMs).toBeUndefined();
const completed = createRunEntry({ collect: true, endedAt: 2_000, archiveAtMs: 10_000 });
expect(
backfillCollectorArchiveAtMs(completed, {
agents: { defaults: { subagents: { archiveAfterMinutes: 0 } } },
}),
).toBe(true);
expect(completed.archiveAtMs).toBeUndefined();
});
it("preserves ordinary keep and persistent session semantics", () => {
expect(
resolveSubagentArchiveAtMs({
cfg,
now: 1_000,
spawnMode: "run",
cleanup: "keep",
}),
).toBeUndefined();
expect(
resolveSubagentArchiveAtMs({
cfg,
now: 1_000,
spawnMode: "session",
cleanup: "delete",
collect: true,
}),
).toBeUndefined();
});
});
describe("reconcileOrphanedRestoredRuns", () => {
it("keeps waitable collector tombstones after delete-mode sessions disappear", () => {
const entry = createRunEntry({
collect: true,
cleanup: "delete",
endedAt: 2_000,
completion: { required: false, resultText: "done", capturedAt: 2_000 },
collectorCompletion: { status: "done" },
});
const runs = new Map([[entry.runId, entry]]);
expect(reconcileOrphanedRestoredRuns({ runs, resumedRuns: new Set() })).toBe(false);
expect(runs.get(entry.runId)).toBe(entry);
});
});
describe("safeRemoveAttachmentsDir", () => {
it("reports non-ENOENT realpath failures instead of treating cleanup as complete", async () => {
const realpathSpy = vi
+53 -1
View File
@@ -316,6 +316,10 @@ export function reconcileOrphanedRestoredRuns(params: {
const now = Date.now();
let changed = false;
for (const [runId, entry] of params.runs.entries()) {
if (entry.collect && entry.collectorCompletion) {
// Waitable collector tombstones intentionally outlive delete-mode sessions.
continue;
}
if (entry.requesterSettleWake) {
// Requester-settle outbox rows can intentionally outlive delete-mode
// child sessions. Restore replays the obligation before retiring them.
@@ -351,7 +355,7 @@ export function reconcileOrphanedRestoredRuns(params: {
}
/** Resolves the completed subagent archive delay from config. */
export function resolveArchiveAfterMs(cfg?: OpenClawConfig) {
function resolveArchiveAfterMs(cfg?: OpenClawConfig) {
const config = cfg ?? getRuntimeConfig();
const minutes =
config.agents?.defaults?.subagents?.archiveAfterMinutes ??
@@ -364,3 +368,51 @@ export function resolveArchiveAfterMs(cfg?: OpenClawConfig) {
}
return Math.max(1, Math.floor(minutes)) * 60_000;
}
/** Resolves the archive deadline for one newly registered run. */
export function resolveSubagentArchiveAtMs(params: {
cfg?: OpenClawConfig;
now: number;
spawnMode: "run" | "session";
cleanup: "keep" | "delete";
collect?: boolean;
}): number | undefined {
if (params.spawnMode === "session" || params.collect || params.cleanup === "keep") {
return undefined;
}
const archiveAfterMs = resolveArchiveAfterMs(params.cfg);
return archiveAfterMs ? params.now + archiveAfterMs : undefined;
}
/** Backfills the retention deadline added after collector groups first shipped. */
export function backfillCollectorArchiveAtMs(
entry: SubagentRunRecord,
cfg?: OpenClawConfig,
): boolean {
if (!entry.collect) {
return false;
}
const endedAt =
typeof entry.endedAt === "number" && Number.isFinite(entry.endedAt) ? entry.endedAt : undefined;
const capturedAt =
endedAt === undefined && !entry.collectorCompletion
? undefined
: typeof entry.completion?.capturedAt === "number" &&
Number.isFinite(entry.completion.capturedAt)
? entry.completion.capturedAt
: endedAt;
const archiveAfterMs = entry.spawnMode === "session" ? undefined : resolveArchiveAfterMs(cfg);
const expectedArchiveAt =
capturedAt !== undefined && archiveAfterMs !== undefined
? capturedAt + archiveAfterMs
: undefined;
if (entry.archiveAtMs === expectedArchiveAt) {
return false;
}
if (expectedArchiveAt === undefined) {
delete entry.archiveAtMs;
} else {
entry.archiveAtMs = expectedArchiveAt;
}
return true;
}
@@ -23,6 +23,7 @@ import {
import { createSubagentRegistryLifecycleController } from "./subagent-registry-lifecycle.js";
import { markSubagentRunPausedAfterYield } from "./subagent-registry-run-manager.js";
import type { SubagentRunRecord } from "./subagent-registry.types.js";
import { createStructuredOutputTool } from "./tools/structured-output-tool.js";
type LifecycleControllerParams = Parameters<typeof createSubagentRegistryLifecycleController>[0];
@@ -107,6 +108,7 @@ vi.mock("./subagent-registry-helpers.js", () => ({
MAX_ANNOUNCE_RETRY_COUNT: 3,
MIN_ANNOUNCE_RETRY_DELAY_MS: 1_000,
PROVISIONAL_KILL_RECONCILIATION_MS: 5 * 60_000,
backfillCollectorArchiveAtMs: () => false,
capFrozenResultText: (text: string) => text.trim(),
logAnnounceGiveUp: helperMocks.logAnnounceGiveUp,
persistSubagentSessionTiming: helperMocks.persistSubagentSessionTiming,
@@ -195,6 +197,7 @@ function createLifecycleController({
runs,
resumedRuns: new Set(),
subagentAnnounceTimeoutMs: 1_000,
getRuntimeConfig: () => ({}),
persist: vi.fn(),
persistOrThrow: vi.fn(),
clearPendingLifecycleError: vi.fn(),
@@ -2579,6 +2582,63 @@ describe("subagent registry lifecycle hardening", () => {
expect(entry.collectorCompletion).toEqual({ status: "done" });
});
it("treats accepted structured output as success for a tool-only collector turn", async () => {
const structured = { answer: "yes" };
const entry = createRunEntry({
expectsCompletionMessage: false,
collect: true,
outputSchema: { type: "object" },
});
const structuredOutput = createStructuredOutputTool({
runId: entry.runId,
schema: { type: "object" },
});
await structuredOutput.execute("tool-call", { result: structured });
const controller = createLifecycleController({
entry,
captureSubagentCompletionReply: vi.fn(async () => ""),
});
await controller.completeSubagentRun({
runId: entry.runId,
endedAt: 4_000,
outcome: { status: "error", error: "completed" },
reason: SUBAGENT_ENDED_REASON_ERROR,
triggerCleanup: true,
});
await waitForLifecycleState(() => expect(entry.cleanupCompletedAt).toBeTypeOf("number"));
expect(entry.collectorCompletion).toEqual({ status: "done", structured });
expect(entry.outcome).toMatchObject({ status: "ok" });
expect(entry.execution).toMatchObject({
status: "terminal",
outcome: expect.objectContaining({ status: "ok" }),
});
expect(entry.endedReason).toBe(SUBAGENT_ENDED_REASON_COMPLETE);
});
it("preserves a real failure after structured output was accepted", async () => {
const structured = { answer: "yes" };
const entry = createRunEntry({
expectsCompletionMessage: false,
collect: true,
outputSchema: { type: "object" },
structuredOutput: { structured, invalidAttempts: 0 },
});
const controller = createLifecycleController({ entry });
await controller.completeSubagentRun({
runId: entry.runId,
endedAt: 4_000,
outcome: { status: "error", error: "provider failed after tool output" },
reason: SUBAGENT_ENDED_REASON_ERROR,
triggerCleanup: true,
});
await waitForLifecycleState(() => expect(entry.cleanupCompletedAt).toBeTypeOf("number"));
expect(entry.collectorCompletion).toEqual({ status: "failed", structured });
});
it("marks a successful collector with invalid structured output failed", async () => {
const entry = createRunEntry({
expectsCompletionMessage: false,
+23 -1
View File
@@ -7,6 +7,7 @@ import { uniqueStrings } from "@openclaw/normalization-core/string-normalization
import { isSilentReplyText, SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
import type { cleanupBrowserSessionsForLifecycleEnd } from "../browser-lifecycle-cleanup.js";
import { formatSqliteSessionFileMarker } from "../config/sessions/sqlite-marker.js";
import type { OpenClawConfig } from "../config/types.openclaw.js";
import type { callGateway as defaultCallGateway } from "../gateway/call.js";
import { formatErrorMessage, readErrorName } from "../infra/errors.js";
import {
@@ -84,6 +85,7 @@ import {
import { deleteSubagentSessionForCleanup } from "./subagent-session-cleanup.js";
import { updateSwarmCollectorCompletion } from "./swarm-collector.js";
import { releaseSwarmRun } from "./swarm-scheduler.js";
import { peekSwarmStructuredOutput } from "./tools/structured-output-tool.js";
type CaptureSubagentCompletionReply =
(typeof import("./subagent-announce.js"))["captureSubagentCompletionReply"];
@@ -174,6 +176,7 @@ export function createSubagentRegistryLifecycleController(params: {
runs: Map<string, SubagentRunRecord>;
resumedRuns: Set<string>;
subagentAnnounceTimeoutMs: number;
getRuntimeConfig(): OpenClawConfig;
persist(): void;
persistOrThrow(): void;
clearPendingLifecycleError(runId: string): void;
@@ -1906,6 +1909,25 @@ export function createSubagentRegistryLifecycleController(params: {
let endedAt = requestedEndedAt;
let completionOutcome =
shouldDrainExistingTerminal && entry.outcome ? entry.outcome : completeParams.outcome;
const liveStructuredOutput = entry.collect
? (entry.structuredOutput ??
peekSwarmStructuredOutput(entry.runId) ??
(entry.swarmRunId ? peekSwarmStructuredOutput(entry.swarmRunId) : undefined))
: undefined;
if (!entry.structuredOutput && liveStructuredOutput) {
entry.structuredOutput = liveStructuredOutput;
mutated = true;
}
if (
liveStructuredOutput?.structured !== undefined &&
completionOutcome.status === "error" &&
completionOutcome.error === "completed"
) {
// Tool-only collector turns use this runner sentinel after the result is
// durably recorded. Normalize before every task/session/hook projection.
completionOutcome = { status: "ok" };
completionReason = SUBAGENT_ENDED_REASON_COMPLETE;
}
const observedStartedAt =
!shouldDrainExistingTerminal &&
typeof completeParams.startedAt === "number" &&
@@ -2085,7 +2107,7 @@ export function createSubagentRegistryLifecycleController(params: {
mutated = true;
}
}
if (updateSwarmCollectorCompletion(entry)) {
if (updateSwarmCollectorCompletion(entry, params.getRuntimeConfig())) {
mutated = true;
}
if (provisionalKillSnapshot) {
+78 -27
View File
@@ -43,7 +43,7 @@ import {
} from "./subagent-registry-completion.js";
import {
persistSubagentSessionTiming,
resolveArchiveAfterMs,
resolveSubagentArchiveAtMs,
safeRemoveAttachmentsDir,
} from "./subagent-registry-helpers.js";
import type {
@@ -659,14 +659,14 @@ export function createSubagentRunManager(params: {
source.childSessionKey,
);
const cfg = params.getRuntimeConfig();
const archiveAfterMs = resolveArchiveAfterMs(cfg);
const spawnMode = source.spawnMode === "session" ? "session" : "run";
const archiveAtMs =
spawnMode === "session" || source.cleanup === "keep"
? undefined
: archiveAfterMs
? now + archiveAfterMs
: undefined;
const archiveAtMs = resolveSubagentArchiveAtMs({
cfg,
now,
spawnMode,
cleanup: source.cleanup,
collect: source.collect,
});
const runTimeoutSeconds = replaceParams.runTimeoutSeconds ?? source.runTimeoutSeconds ?? 0;
const waitTimeoutMs = params.resolveSubagentWaitTimeoutMs(cfg, runTimeoutSeconds);
const preserveFrozenResultFallback = replaceParams.preserveFrozenResultFallback === true;
@@ -790,14 +790,14 @@ export function createSubagentRunManager(params: {
const now = Date.now();
const generation = nextSubagentRunGeneration(params.runs.values(), childSessionKey);
const cfg = params.getRuntimeConfig();
const archiveAfterMs = resolveArchiveAfterMs(cfg);
const spawnMode = registerParams.spawnMode === "session" ? "session" : "run";
const archiveAtMs =
spawnMode === "session" || registerParams.cleanup === "keep"
? undefined
: archiveAfterMs
? now + archiveAfterMs
: undefined;
const archiveAtMs = resolveSubagentArchiveAtMs({
cfg,
now,
spawnMode,
cleanup: registerParams.cleanup,
collect: registerParams.collect,
});
const runTimeoutSeconds = registerParams.runTimeoutSeconds ?? 0;
const waitTimeoutMs = params.resolveSubagentWaitTimeoutMs(cfg, runTimeoutSeconds);
const requesterOrigin = normalizeDeliveryContext(registerParams.requesterOrigin);
@@ -922,11 +922,26 @@ export function createSubagentRunManager(params: {
const entry =
params.runs.get(key) ??
[...params.runs.values()].find((candidate) => candidate.swarmRunId === key);
const lifecycleStarted =
entry?.execution?.status === "running" &&
typeof entry.execution.startedAt === "number" &&
entry.swarmLaunchPending === true;
const provisionalTerminalBeforeAcceptance =
entry?.swarmLaunchPending === true &&
typeof entry.endedAt === "number" &&
entry.collectorCompletion === undefined;
if (provisionalTerminalBeforeAcceptance) {
// Cancellation won before Gateway acceptance. The caller must abort the
// newly accepted run before freezing completion or releasing the FIFO slot.
return false;
}
// Completion clears swarmLaunchPending, but queuedLaunch remains until the
// delayed acceptance response remaps the durable terminal row.
const terminalBeforeAcceptance =
entry?.collectorCompletion !== undefined && entry.queuedLaunch !== undefined;
if (
!entry ||
entry.execution?.status !== "queued" ||
typeof entry.endedAt === "number" ||
entry.collectorCompletion
(!terminalBeforeAcceptance && entry.execution?.status !== "queued" && !lifecycleStarted)
) {
return false;
}
@@ -935,7 +950,7 @@ export function createSubagentRunManager(params: {
if (conflicting && conflicting !== entry) {
throw new Error(`collector gateway run id already exists: ${nextRunId}`);
}
const startedAt = Date.now();
const acceptedAt = Date.now();
const previousRunId = entry.runId;
const previousStartedAt = entry.startedAt;
const previousSessionStartedAt = entry.sessionStartedAt;
@@ -951,9 +966,45 @@ export function createSubagentRunManager(params: {
entry.runId = nextRunId;
params.runs.set(nextRunId, entry);
}
entry.startedAt = startedAt;
entry.sessionStartedAt ??= startedAt;
entry.execution = { ...entry.execution, status: "running", startedAt };
if (terminalBeforeAcceptance) {
entry.swarmLaunchPending = false;
entry.queuedLaunch = undefined;
try {
params.persistOrThrow();
return true;
} catch (error) {
if (previousRunId !== nextRunId) {
params.runs.delete(nextRunId);
entry.runId = previousRunId;
params.runs.set(previousRunId, entry);
}
entry.queuedLaunch = previousQueuedLaunch;
entry.swarmRunId = previousSwarmRunId;
entry.schedulerSlotId = previousSchedulerSlotId;
entry.swarmLaunchPending = previousSwarmLaunchPending;
throw error;
}
}
// Gateway acceptance only proves admission. Preserve a lifecycle start that
// raced ahead of this response; otherwise leave the run clock unset until
// preparation and lane dequeue emit the canonical start event.
const lifecycleStartedAt =
entry.execution?.status === "running" ? entry.execution.startedAt : undefined;
if (typeof lifecycleStartedAt === "number") {
entry.startedAt = lifecycleStartedAt;
entry.sessionStartedAt ??= lifecycleStartedAt;
entry.execution = {
...entry.execution,
status: "running",
acceptedAt,
startedAt: lifecycleStartedAt,
};
} else {
delete entry.startedAt;
delete entry.sessionStartedAt;
entry.execution = { ...entry.execution, status: "running", acceptedAt };
delete entry.execution.startedAt;
}
entry.swarmLaunchPending = false;
entry.queuedLaunch = undefined;
let persistedRunning = false;
@@ -964,8 +1015,8 @@ export function createSubagentRunManager(params: {
runId: entry.taskRunId ?? entry.runId,
runtime: "subagent",
sessionKey: entry.childSessionKey,
startedAt,
lastEventAt: startedAt,
startedAt: acceptedAt,
lastEventAt: acceptedAt,
});
} catch (error) {
if (previousRunId !== nextRunId) {
@@ -1019,7 +1070,7 @@ export function createSubagentRunManager(params: {
entry.queuedLaunch = undefined;
entry.collectorLaunchCleanupPending = true;
entry.completion = { required: false, resultText: error, capturedAt: endedAt };
updateSwarmCollectorCompletion(entry);
updateSwarmCollectorCompletion(entry, params.getRuntimeConfig());
try {
params.persistOrThrow();
} catch (persistError) {
@@ -1080,7 +1131,7 @@ export function createSubagentRunManager(params: {
resultText: entry.outcome?.status === "error" ? (entry.outcome.error ?? error) : error,
capturedAt: entry.endedAt,
};
updateSwarmCollectorCompletion(entry);
updateSwarmCollectorCompletion(entry, params.getRuntimeConfig());
try {
params.persistOrThrow();
} catch (persistError) {
@@ -1238,7 +1289,7 @@ export function createSubagentRunManager(params: {
supersededAt: existingKillReconciliation?.supersededAt,
};
if (wasQueuedCollector && !collectorLaunchInFlight) {
updateSwarmCollectorCompletion(entry);
updateSwarmCollectorCompletion(entry, params.getRuntimeConfig());
}
pendingTaskFinalizations.push({ entry, endedAt: taskEndedAt });
if (!entriesByChildSessionKey.has(entry.childSessionKey)) {
+122 -3
View File
@@ -17,6 +17,7 @@ import {
getActiveGatewayRootWorkCount,
markGatewayRestartDraining,
resetGatewayWorkAdmission,
tryBeginGatewaySuspendAdmission,
} from "../process/gateway-work-admission.js";
import { SUBAGENT_KILL_TASK_ERROR } from "../tasks/detached-task-runtime-contract.js";
import {
@@ -449,7 +450,7 @@ describe("subagent registry seam flow", () => {
runId: `run-collector-${suffix}`,
childSessionKey: `agent:main:subagent:collector-${suffix}`,
task: "retain lifetime group count",
cleanup: "delete",
cleanup: suffix === "one" ? "keep" : "delete",
createdAt: now - 10_000,
endedAt: now - 5_000,
cleanupCompletedAt: now - 4_000,
@@ -720,6 +721,8 @@ describe("subagent registry seam flow", () => {
expect(mod.markSubagentRunTerminated({ runId, reason: "manual kill" })).toBe(1);
expect(mod.getSubagentRunByRunId(runId)?.collectorCompletion).toBeUndefined();
expect(mod.startQueuedSubagentRun(runId, "gateway-launch-kill")).toBe(false);
expect(mod.getSubagentRunByRunId("gateway-launch-kill")).toBeUndefined();
expect(mod.settleFailedQueuedSubagentLaunch(runId, "launch response lost")).toBe(true);
expect(mod.getSubagentRunByRunId(runId)?.collectorCompletion).toMatchObject({
@@ -874,7 +877,7 @@ describe("subagent registry seam flow", () => {
);
});
it("rehydrates persisted collector FIFO queues after registry restore", async () => {
it("rehydrates persisted collector FIFO queues after admission reopens", async () => {
const now = Date.now();
mocks.getRuntimeConfig.mockReturnValue({
tools: { swarm: { enabled: true, maxConcurrent: 1 } },
@@ -921,7 +924,15 @@ describe("subagent registry seam flow", () => {
return request.method === "agent.wait" ? { status: "pending" } : {};
});
const suspension = tryBeginGatewaySuspendAdmission(() => {});
expect(suspension?.commit()).toBe(true);
mod.initSubagentRegistry();
await Promise.resolve();
expect(mocks.callGateway.mock.calls.filter(([request]) => request.method === "agent")).toEqual(
[],
);
suspension?.release();
await waitForFast(() => {
const agentCalls = mocks.callGateway.mock.calls.filter(
([request]) => request.method === "agent",
@@ -937,14 +948,122 @@ describe("subagent registry seam flow", () => {
});
});
expect(mod.getSubagentRunByRunId("run-queued-one")?.execution?.status).toBe("running");
expect(mod.getSubagentRunByRunId("gateway-run-one")).toMatchObject({
const acceptedRun = mod.getSubagentRunByRunId("gateway-run-one");
expect(acceptedRun).toMatchObject({
runId: "gateway-run-one",
swarmRunId: "run-queued-one",
schedulerSlotId: "run-queued-one",
execution: { status: "running" },
});
expect(acceptedRun).not.toHaveProperty("startedAt");
expect(acceptedRun).not.toHaveProperty("sessionStartedAt");
expect(acceptedRun?.execution).not.toHaveProperty("startedAt");
expect(mod.getSubagentRunByRunId("run-queued-two")?.execution?.status).toBe("queued");
});
it("preserves a lifecycle start that arrives before collector acceptance returns", async () => {
const startedAt = 12_345;
mod.registerSubagentRun({
runId: "run-start-race",
childSessionKey: "agent:main:subagent:start-race",
requesterSessionKey: "agent:main:main",
requesterDisplayKey: "main",
task: "start before acceptance",
cleanup: "keep",
collect: true,
groupId: "start-race",
queued: true,
expectsCompletionMessage: false,
});
const lastOnAgentEventCall = mocks.onAgentEvent.mock.calls.at(-1) as unknown as
| [(event: AgentEventPayload) => void]
| undefined;
const lifecycleHandler = lastOnAgentEventCall?.[0];
expect(lifecycleHandler).toBeTypeOf("function");
lifecycleHandler?.({
runId: "run-start-race",
seq: 1,
stream: "lifecycle",
ts: startedAt,
data: { phase: "start", startedAt },
});
await waitForFast(() =>
expect(mod.getSubagentRunByRunId("run-start-race")?.startedAt).toBe(startedAt),
);
expect(mod.startQueuedSubagentRun("run-start-race", "gateway-start-race")).toBe(true);
expect(mod.getSubagentRunByRunId("gateway-start-race")).toMatchObject({
startedAt,
sessionStartedAt: startedAt,
execution: { status: "running", acceptedAt: expect.any(Number), startedAt },
});
});
it("remaps a collector that completed before its acceptance response", () => {
mod.addSubagentRunForTests({
runId: "run-terminal-race",
childSessionKey: "agent:main:subagent:terminal-race",
requesterSessionKey: "agent:main:main",
requesterDisplayKey: "main",
task: "finish before acceptance",
cleanup: "keep",
collect: true,
swarmRunId: "run-terminal-race",
schedulerSlotId: "run-terminal-race",
swarmLaunchPending: false,
queuedLaunch: {
request: { sessionKey: "agent:main:subagent:terminal-race" },
timeoutMs: 1_000,
schedulerGroupKey: "terminal-race",
maxConcurrent: 1,
},
groupId: "terminal-race",
createdAt: 1_000,
endedAt: 2_000,
execution: { status: "terminal", endedAt: 2_000 },
completion: { required: false, resultText: "done", capturedAt: 2_000 },
collectorCompletion: { status: "done" },
});
expect(mod.startQueuedSubagentRun("run-terminal-race", "gateway-terminal-race")).toBe(true);
const remapped = mod.getSubagentRunByRunId("gateway-terminal-race");
expect(mod.getSubagentRunByRunId("run-terminal-race")).toBe(remapped);
expect(remapped).toMatchObject({
runId: "gateway-terminal-race",
swarmRunId: "run-terminal-race",
collectorCompletion: { status: "done" },
swarmLaunchPending: false,
});
});
it("refuses to remap an unrelated terminal collector without a pending launch", () => {
mod.addSubagentRunForTests({
runId: "run-terminal-stale",
childSessionKey: "agent:main:subagent:terminal-stale",
requesterSessionKey: "agent:main:main",
requesterDisplayKey: "main",
task: "stale acceptance callback",
cleanup: "keep",
collect: true,
swarmRunId: "run-terminal-stale",
schedulerSlotId: "run-terminal-stale",
groupId: "terminal-stale",
createdAt: 1_000,
endedAt: 2_000,
execution: { status: "terminal", endedAt: 2_000 },
completion: { required: false, resultText: "done", capturedAt: 2_000 },
collectorCompletion: { status: "done" },
});
expect(mod.startQueuedSubagentRun("run-terminal-stale", "gateway-terminal-stale")).toBe(false);
expect(mod.getSubagentRunByRunId("run-terminal-stale")).toMatchObject({
runId: "run-terminal-stale",
collectorCompletion: { status: "done" },
});
expect(mod.getSubagentRunByRunId("gateway-terminal-stale")).toBeUndefined();
});
it("holds a restored FIFO slot until an accepted collector is confirmed stopped", async () => {
vi.useRealTimers();
const now = Date.now();
+44 -33
View File
@@ -16,6 +16,7 @@ import { getAgentRunContext, onAgentEvent } from "../infra/agent-events.js";
import { isFastTestRuntimeEnv } from "../infra/env.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import {
GatewayDrainingError,
isGatewayRestartDraining,
runWithGatewayIndependentRootWorkAdmission,
} from "../process/gateway-work-admission.js";
@@ -67,6 +68,7 @@ import {
} from "./subagent-registry-completion.js";
import {
ANNOUNCE_EXPIRY_MS,
backfillCollectorArchiveAtMs,
MAX_ANNOUNCE_RETRY_COUNT,
PROVISIONAL_KILL_RECONCILIATION_MS,
reconcileOrphanedRestoredRuns,
@@ -865,6 +867,7 @@ const subagentLifecycleController = createSubagentRegistryLifecycleController({
runs: subagentRuns,
resumedRuns,
subagentAnnounceTimeoutMs: SUBAGENT_ANNOUNCE_TIMEOUT_MS,
getRuntimeConfig: () => subagentRegistryDeps.getRuntimeConfig(),
persist: persistSubagentRuns,
persistOrThrow: persistSubagentRunsOrThrow,
clearPendingLifecycleError,
@@ -1071,12 +1074,17 @@ function restoreSubagentRunsOnce() {
if (restoredCount === 0) {
return;
}
if (
reconcileOrphanedRestoredRuns({
runs: subagentRuns,
resumedRuns,
})
) {
const cfg = subagentRegistryDeps.getRuntimeConfig();
let restoredStateChanged = reconcileOrphanedRestoredRuns({
runs: subagentRuns,
resumedRuns,
});
for (const entry of subagentRuns.values()) {
if (backfillCollectorArchiveAtMs(entry, cfg)) {
restoredStateChanged = true;
}
}
if (restoredStateChanged) {
persistSubagentRuns();
}
const requesterTurns = new Map<string, Map<string, SubagentRunRecord[]>>();
@@ -1144,32 +1152,37 @@ function restoreSubagentRunsOnce() {
.filter((candidate) => candidate.execution?.status === "running")
.map((candidate) => candidate.schedulerSlotId ?? candidate.runId),
start: async () => {
const response = await subagentRegistryDeps.callGateway({
method: "agent",
params: applySubagentLaunchAuthorization(launch.request, launch.authorization),
// Restart replay must restore the trusted launch capability; otherwise
// the queued child silently falls back to its session/default route.
...(launch.authorization ? { scopes: [ADMIN_SCOPE] } : {}),
timeoutMs: launch.timeoutMs,
});
const gatewayRunId = readGatewayRunId(response) ?? runId;
try {
if (!startQueuedSubagentRun(runId, gatewayRunId)) {
throw new Error(
"collector registry row could not transition from queued to running",
);
}
} catch (error) {
await terminateAcceptedRestoredCollectorRun({
entry,
gatewayRunId,
await runWithGatewayIndependentRootWorkAdmission(async () => {
const response = await subagentRegistryDeps.callGateway({
method: "agent",
params: applySubagentLaunchAuthorization(launch.request, launch.authorization),
// Restart replay must restore the trusted launch capability; otherwise
// the queued child silently falls back to its session/default route.
...(launch.authorization ? { scopes: [ADMIN_SCOPE] } : {}),
timeoutMs: launch.timeoutMs,
});
launchTerminationConfirmed = true;
throw error;
}
const gatewayRunId = readGatewayRunId(response) ?? runId;
try {
if (!startQueuedSubagentRun(runId, gatewayRunId)) {
throw new Error(
"collector registry row could not transition from queued to running",
);
}
} catch (error) {
await terminateAcceptedRestoredCollectorRun({
entry,
gatewayRunId,
timeoutMs: launch.timeoutMs,
});
launchTerminationConfirmed = true;
throw error;
}
});
},
onStartFailure: (error) => {
if (error instanceof GatewayDrainingError) {
return false;
}
return failAndCleanupRestoredQueuedRun(
runId,
entry,
@@ -1827,12 +1840,9 @@ async function sweepSubagentRuns() {
continue;
}
let deleteFailed = false;
// Lifecycle cleanup already attempted each delete-mode session. Retry
// here only so a transient cleanup failure cannot survive group archive.
// Group retention owns the final session archive for both keep- and
// delete-mode collectors. Retry every member so the batch is idempotent.
for (const [candidateRunId, candidate] of groupEntries) {
if (candidate.cleanup !== "delete") {
continue;
}
try {
await subagentRegistryDeps.callGateway({
method: "sessions.delete",
@@ -2027,6 +2037,7 @@ function ensureListener() {
if (typeof entry.sessionStartedAt !== "number") {
entry.sessionStartedAt = startedAt;
}
entry.execution = { ...entry.execution, status: "running", startedAt };
persistSubagentRuns();
}
return;
+1
View File
@@ -39,6 +39,7 @@ export type PendingFinalDeliveryPayload = {
export type SubagentExecutionState = {
status: "queued" | "running" | "interrupted" | "terminal";
acceptedAt?: number;
startedAt?: number;
endedAt?: number;
outcome?: SubagentRunOutcome;
+27
View File
@@ -22,6 +22,33 @@ describe("subagent run timeout helpers", () => {
).toBe(2_592_001_000);
});
it("waits for the collector lifecycle start before setting its deadline", () => {
expect(
resolveSubagentRunDeadlineMs({
collect: true,
createdAt: 1_000,
runTimeoutSeconds: 60,
}),
).toBeUndefined();
expect(
resolveSubagentRunDeadlineMs({
collect: true,
createdAt: 1_000,
runTimeoutSeconds: 60,
}),
).toBeUndefined();
expect(
resolveSubagentRunDeadlineMs(
{
collect: true,
createdAt: 1_000,
runTimeoutSeconds: 60,
},
5_000,
),
).toBe(65_000);
});
it("caps actual timer delays without shortening semantic durations", () => {
// Long-lived subagent runs retain their requested deadline even though the
// watchdog timer must be scheduled in bounded chunks.
+5 -3
View File
@@ -29,7 +29,7 @@ export function resolveSubagentRunDurationMs(timeoutSeconds: unknown): number |
/** Resolve the absolute timeout deadline for a subagent run. */
export function resolveSubagentRunDeadlineMs(
entry: Pick<SubagentRunRecord, "createdAt" | "startedAt" | "runTimeoutSeconds">,
entry: Pick<SubagentRunRecord, "collect" | "createdAt" | "startedAt" | "runTimeoutSeconds">,
observedStartedAt?: number,
): number | undefined {
const durationMs = resolveSubagentRunDurationMs(entry.runTimeoutSeconds);
@@ -41,7 +41,9 @@ export function resolveSubagentRunDeadlineMs(
? observedStartedAt
: typeof entry.startedAt === "number" && Number.isFinite(entry.startedAt)
? entry.startedAt
: entry.createdAt;
: entry.collect
? undefined
: entry.createdAt;
const safeStartedAt = asDateTimestampMs(startedAt);
if (safeStartedAt === undefined) {
return undefined;
@@ -54,7 +56,7 @@ export function resolveSubagentRunDeadlineMs(
/** Clamp a reported terminal time to the run's explicit timeout deadline. */
export function resolveSubagentRunEffectiveEndedAt(
entry: Pick<SubagentRunRecord, "createdAt" | "startedAt" | "runTimeoutSeconds">,
entry: Pick<SubagentRunRecord, "collect" | "createdAt" | "startedAt" | "runTimeoutSeconds">,
endedAt: number,
observedStartedAt?: number,
): number {
@@ -18,8 +18,14 @@ import {
type dispatchGatewayMethodInProcess,
} from "../gateway/server-plugins.js";
import { withPluginRuntimeGatewayRequestScope } from "../plugins/runtime/gateway-request-scope.js";
import {
isGatewaySubordinateWorkAdmissionClosed,
resetGatewayWorkAdmission,
tryBeginGatewayRootWorkAdmission,
} from "../process/gateway-work-admission.js";
import { captureEnv, setTestEnvValue } from "../test-utils/env.js";
import { subagentRuns } from "./subagent-registry-memory.js";
import { markSubagentRunTerminated } from "./subagent-registry.js";
import {
resetSubagentRegistryForTests,
testing as subagentRegistryTesting,
@@ -90,6 +96,7 @@ async function waitForAssertion(assertion: () => void, timeoutMs = 2_000): Promi
describe("spawnSubagentDirect in-process Gateway collector launch", () => {
beforeEach(async () => {
resetGatewayWorkAdmission();
swarmSchedulerTesting.reset();
resetSubagentRegistryForTests({ persist: false });
clearFallbackGatewayContext();
@@ -109,7 +116,7 @@ describe("spawnSubagentDirect in-process Gateway collector launch", () => {
path.join(stateDir, "openclaw.json"),
`${JSON.stringify({
session: { mainKey: "main", scope: "per-sender" },
tools: { swarm: true },
tools: { swarm: { enabled: true, maxConcurrent: 1 } },
agents: {
defaults: { workspace: stateDir },
entries: { main: { workspace: stateDir } },
@@ -121,6 +128,7 @@ describe("spawnSubagentDirect in-process Gateway collector launch", () => {
afterEach(async () => {
clearFallbackGatewayContext();
resetGatewayWorkAdmission();
swarmSchedulerTesting.reset();
resetSubagentRegistryForTests({ persist: false });
subagentRegistryTesting.setDepsForTest();
@@ -134,6 +142,178 @@ describe("spawnSubagentDirect in-process Gateway collector launch", () => {
}
});
it("launches queued collectors after the parent admission lease is released", async () => {
const gatewayContext = makeGatewayContext();
let releaseFirstLaunch!: () => void;
const firstLaunchGate = new Promise<void>((resolve) => {
releaseFirstLaunch = resolve;
});
const subordinateAdmissionStates: boolean[] = [];
let launchCount = 0;
subagentSpawnTesting.setDepsForTest({
dispatchGatewayMethodInProcess: async <T>(
_method: string,
params: Record<string, unknown>,
) => {
subordinateAdmissionStates.push(isGatewaySubordinateWorkAdmissionClosed());
launchCount += 1;
if (launchCount === 1) {
await firstLaunchGate;
}
return {
runId: params.idempotencyKey as string,
status: "accepted",
} as T;
},
});
const parentAdmission = tryBeginGatewayRootWorkAdmission();
expect(parentAdmission).not.toBeNull();
const results = await parentAdmission!.run(() =>
withPluginRuntimeGatewayRequestScope(
{
context: gatewayContext,
client: externalCliClient(),
isWebchatConnect: () => false,
},
() =>
Promise.all([
spawnSubagentDirect(
{
task: "first collector",
collect: true,
context: "isolated",
lightContext: true,
groupId: "swarm-queued-launch",
swarmLaunchReplayKey: "code-mode:agentSpawn:1",
},
{
agentSessionKey: "agent:main:main",
requesterRunId: "parent-run",
},
),
spawnSubagentDirect(
{
task: "second collector",
collect: true,
context: "isolated",
lightContext: true,
groupId: "swarm-queued-launch",
swarmLaunchReplayKey: "code-mode:agentSpawn:2",
},
{
agentSessionKey: "agent:main:main",
requesterRunId: "parent-run",
},
),
]),
),
);
parentAdmission!.release();
expect(results.map((result) => result.status)).toEqual(["accepted", "accepted"]);
await waitForAssertion(() => {
expect(launchCount).toBe(1);
});
releaseFirstLaunch();
await waitForAssertion(() => {
expect(launchCount).toBe(2);
for (const result of results) {
expect(subagentRuns.get(result.runId!)).toMatchObject({
collect: true,
swarmLaunchPending: false,
});
}
});
expect(subordinateAdmissionStates).toEqual([false, false]);
});
it("aborts a collector cancelled while Gateway acceptance is in flight", async () => {
const gatewayContext = makeGatewayContext();
let releaseFirstLaunch!: () => void;
const firstLaunchGate = new Promise<void>((resolve) => {
releaseFirstLaunch = resolve;
});
const requests: Array<{ method: string; params: Record<string, unknown> }> = [];
let launchCount = 0;
subagentSpawnTesting.setDepsForTest({
dispatchGatewayMethodInProcess: async <T>(
method: string,
params: Record<string, unknown>,
) => {
requests.push({ method, params });
if (method === "agent") {
launchCount += 1;
if (launchCount === 1) {
await firstLaunchGate;
}
return { runId: `gateway-run-${launchCount}`, status: "accepted" } as T;
}
return {} as T;
},
});
const parentAdmission = tryBeginGatewayRootWorkAdmission();
expect(parentAdmission).not.toBeNull();
const results = await parentAdmission!.run(() =>
withPluginRuntimeGatewayRequestScope(
{
context: gatewayContext,
client: externalCliClient(),
isWebchatConnect: () => false,
},
() =>
Promise.all([
spawnSubagentDirect(
{
task: "cancelled collector",
collect: true,
context: "isolated",
lightContext: true,
groupId: "swarm-cancel-launch",
swarmLaunchReplayKey: "code-mode:agentSpawn:cancelled",
},
{ agentSessionKey: "agent:main:main", requesterRunId: "parent-run" },
),
spawnSubagentDirect(
{
task: "next collector",
collect: true,
context: "isolated",
lightContext: true,
groupId: "swarm-cancel-launch",
swarmLaunchReplayKey: "code-mode:agentSpawn:next",
},
{ agentSessionKey: "agent:main:main", requesterRunId: "parent-run" },
),
]),
),
);
parentAdmission!.release();
const firstRunId = results[0]?.runId;
expect(firstRunId).toBeTruthy();
await waitForAssertion(() => expect(launchCount).toBe(1));
expect(markSubagentRunTerminated({ runId: firstRunId, reason: "manual kill" })).toBe(1);
releaseFirstLaunch();
await waitForAssertion(() => {
expect(
requests.some(
(request) => request.method === "chat.abort" && request.params.runId === "gateway-run-1",
),
).toBe(true);
expect(launchCount).toBe(2);
expect(subagentRuns.get(firstRunId!)).toMatchObject({
collectorCompletion: { status: "killed" },
});
expect(subagentRuns.get("gateway-run-2")).toMatchObject({
swarmRunId: results[1]!.runId,
swarmLaunchPending: false,
});
});
});
it("hands a registered collector launch to Gateway as the host", async () => {
const gatewayContext = makeGatewayContext();
const dispatchOptions: Array<{ method: string; forceSyntheticClient?: boolean }> = [];
+22 -11
View File
@@ -26,6 +26,10 @@ import { isFastTestRuntimeEnv } from "../infra/env.js";
import { stringifyRouteThreadId } from "../plugin-sdk/channel-route.js";
import { listRegisteredPluginAgentPromptGuidance } from "../plugins/command-registry-state.js";
import type { SubagentLifecycleHookRunner } from "../plugins/hooks.js";
import {
GatewayDrainingError,
runWithGatewayIndependentRootWorkContinuation,
} from "../process/gateway-work-admission.js";
import { isValidAgentId, normalizeAgentId, parseAgentSessionKey } from "../routing/session-key.js";
import { recordSessionCreated, recordSubagentSpawned } from "../sessions/session-state-events.js";
import type { FastMode } from "../shared/fast-mode.js";
@@ -1846,20 +1850,27 @@ export async function spawnSubagentDirect(
groupId: swarmSchedulerGroupKey,
runId: childRunId,
start: async () => {
const response = await launchChildRun();
const gatewayRunId = readGatewayRunId(response) ?? childRunId;
try {
if (!startQueuedSubagentRun(childRunId, gatewayRunId)) {
throw new Error("collector registry row could not transition from queued to running");
await runWithGatewayIndependentRootWorkContinuation(async () => {
const response = await launchChildRun();
const gatewayRunId = readGatewayRunId(response) ?? childRunId;
try {
if (!startQueuedSubagentRun(childRunId, gatewayRunId)) {
throw new Error(
"collector registry row could not transition from queued to running",
);
}
} catch (error) {
await terminateAcceptedCollectorRun({ childSessionKey, gatewayRunId });
launchTerminationConfirmed = true;
throw error;
}
} catch (error) {
await terminateAcceptedCollectorRun({ childSessionKey, gatewayRunId });
launchTerminationConfirmed = true;
throw error;
}
await emitSpawnLifecycleHooks(gatewayRunId);
await emitSpawnLifecycleHooks(gatewayRunId);
});
},
onStartFailure: async (error) => {
if (error instanceof GatewayDrainingError) {
return false;
}
const launchError = summarizeError(error);
const [contextRollback, sessionCleanup] = await Promise.allSettled([
rollbackPreparedContextEngine(pipelineResult.state.contextEnginePreparation),
+23 -5
View File
@@ -1,27 +1,45 @@
import type { OpenClawConfig } from "../config/types.openclaw.js";
import { ensureCompletionState } from "./subagent-delivery-state.js";
import { SUBAGENT_ENDED_REASON_KILLED } from "./subagent-lifecycle-events.js";
import { backfillCollectorArchiveAtMs } from "./subagent-registry-helpers.js";
import type { SubagentRunRecord, SwarmCollectorStatus } from "./subagent-registry.types.js";
import { loadSubagentSessionEntry } from "./subagent-session-reconciliation.js";
import { consumeSwarmStructuredOutput } from "./tools/structured-output-tool.js";
function resolveStatus(entry: SubagentRunRecord): SwarmCollectorStatus {
function resolveStatus(
entry: SubagentRunRecord,
hasStructuredResult: boolean,
): SwarmCollectorStatus {
if (entry.endedReason === SUBAGENT_ENDED_REASON_KILLED) {
return "killed";
}
if (entry.outcome?.status === "timeout") {
return "timeout";
}
return entry.outcome?.status === "ok" ? "done" : "failed";
if (entry.outcome?.status === "ok") {
return "done";
}
// Tool-only structured turns can surface the runner's synthetic completion
// marker as an error despite having fulfilled the collector contract.
return hasStructuredResult && entry.outcome?.error === "completed" ? "done" : "failed";
}
/** Freeze the waitable collector record after raw completion capture. */
export function updateSwarmCollectorCompletion(entry: SubagentRunRecord): boolean {
export function updateSwarmCollectorCompletion(
entry: SubagentRunRecord,
cfg: OpenClawConfig,
): boolean {
if (!entry.collect) {
return false;
}
const clearedPendingLaunch = entry.swarmLaunchPending === true;
entry.swarmLaunchPending = false;
const completion = ensureCompletionState(entry);
const capturedAtAdded = completion.capturedAt === undefined;
completion.capturedAt ??= Date.now();
const archiveDeadlineAdded = backfillCollectorArchiveAtMs(entry, cfg);
if (entry.collectorCompletion) {
return clearedPendingLaunch;
return clearedPendingLaunch || capturedAtAdded || archiveDeadlineAdded;
}
const executionCaptured = consumeSwarmStructuredOutput(entry.runId);
const publicCaptured =
@@ -42,7 +60,7 @@ export function updateSwarmCollectorCompletion(entry: SubagentRunRecord): boolea
outputTokens: session.outputTokens ?? 0,
}
: undefined;
const resolvedStatus = resolveStatus(entry);
const resolvedStatus = resolveStatus(entry, captured?.structured !== undefined);
const next = {
status: schemaError && resolvedStatus === "done" ? ("failed" as const) : resolvedStatus,
...(captured?.structured !== undefined ? { structured: captured.structured } : {}),
@@ -21,6 +21,27 @@ describe("structured_output", () => {
expect(testing.readSwarmStructuredOutput("run-1")?.structured).toEqual({ answer: "yes" });
});
it("publishes a provider-valid schema while accepting any JSON result", () => {
const tool = createStructuredOutputTool({
runId: "run-json-value",
schema: {},
});
expect(tool.parameters).toEqual({
type: "object",
required: ["result"],
properties: {
result: {
type: ["object", "array", "string", "number", "boolean", "null"],
},
},
additionalProperties: false,
});
for (const result of [{ answer: "yes" }, ["yes"], "yes", 1, true, null]) {
expect(Value.Check(tool.parameters, { result })).toBe(true);
}
expect(Value.Check(tool.parameters, { result: undefined })).toBe(false);
});
it("nudges once then freezes schemaError", async () => {
const tool = createStructuredOutputTool({
runId: "run-2",
+13 -5
View File
@@ -14,7 +14,7 @@ function formatSchemaError(errors: Array<{ text: string }>): string {
.join("; ");
}
function readSwarmStructuredOutput(runId: string): SwarmStructuredOutputState | undefined {
export function peekSwarmStructuredOutput(runId: string): SwarmStructuredOutputState | undefined {
const state = states.get(runId);
return state ? structuredClone(state) : undefined;
}
@@ -22,7 +22,7 @@ function readSwarmStructuredOutput(runId: string): SwarmStructuredOutputState |
export function consumeSwarmStructuredOutput(
runId: string,
): SwarmStructuredOutputState | undefined {
const state = readSwarmStructuredOutput(runId);
const state = peekSwarmStructuredOutput(runId);
states.delete(runId);
return state;
}
@@ -59,8 +59,16 @@ export function createStructuredOutputTool(params: {
displaySummary: "Record the collector result.",
description: `Call exactly once as {"result": ...}, where result matches this JSON Schema: ${requestedSchema}`,
// Runtime argument validation must reach execute so invalid attempts consume
// the durable one-retry budget. The requested schema remains model-visible above.
parameters: Type.Object({ result: Type.Unknown() }, { additionalProperties: false }),
// the durable one-retry budget. Providers still require every tool property to
// declare a JSON type before they will send the request.
parameters: Type.Object(
{
result: Type.Unsafe({
type: ["object", "array", "string", "number", "boolean", "null"],
}),
},
{ additionalProperties: false },
),
execute: async (_toolCallId, args) => {
const prior = states.get(params.runId);
if (prior?.structured !== undefined) {
@@ -99,7 +107,7 @@ export function createStructuredOutputTool(params: {
}
const testing = {
readSwarmStructuredOutput,
readSwarmStructuredOutput: peekSwarmStructuredOutput,
reset() {
states.clear();
},
+3 -3
View File
@@ -9,8 +9,8 @@ type AdmissionCloseReason = "restart-signal fence" | "restart drain" | "suspend
type AdmissionReopenReason = "restart-signal fence" | "suspend phase";
export class GatewayDrainingError extends Error {
constructor() {
super("Gateway is draining; new tasks are not accepted");
constructor(message = "Gateway is draining; new tasks are not accepted") {
super(message);
this.name = "GatewayDrainingError";
}
}
@@ -302,7 +302,7 @@ export async function runWithGatewayIndependentRootWorkAdmission<T>(
): Promise<T> {
while (true) {
if (GATEWAY_WORK_ADMISSION_STATE.restartDraining) {
throw new Error("gateway is draining for restart");
throw new GatewayDrainingError("gateway is draining for restart");
}
const admission = tryBeginGatewayIndependentRootWorkAdmission();
if (admission) {
@@ -1,50 +1,9 @@
import type { GatewayBrowserClient } from "../api/gateway.ts";
import type { GatewaySessionRow } from "../api/types.ts";
import type { SessionCapability } from "../lib/sessions/index.ts";
export { fetchChildSessionRows } from "../lib/sessions/child-session-data.ts";
const MAX_SESSION_LINEAGE_DEPTH = 16;
export async function fetchChildSessionRows(params: {
sessions: SessionCapability;
parentKey: string;
isCurrent: () => boolean;
}): Promise<GatewaySessionRow[] | null> {
const rows: GatewaySessionRow[] = [];
const seenOffsets = new Set<number>();
let offset = 0;
while (!seenOffsets.has(offset)) {
seenOffsets.add(offset);
const result = await params.sessions.list({
spawnedBy: params.parentKey,
...(offset > 0 ? { offset } : {}),
limit: 20,
includeGlobal: false,
includeUnknown: false,
configuredAgentsOnly: true,
});
if (!params.isCurrent()) {
return null;
}
if (!result) {
throw new Error("child session list returned no result");
}
const runtimeSampledAt = Date.now();
for (const row of result.sessions) {
if (!rows.some((candidate) => candidate.key === row.key)) {
rows.push({ ...row, runtimeSampledAt });
}
}
const hasMore =
result.hasMore ?? (typeof result.totalCount === "number" && rows.length < result.totalCount);
const nextOffset = result.nextOffset ?? rows.length;
if (!hasMore || nextOffset <= offset) {
break;
}
offset = nextOffset;
}
return rows;
}
export function collectKnownSessionRows(
rootRows: readonly GatewaySessionRow[],
childRowsByParent: Readonly<Record<string, readonly GatewaySessionRow[]>>,
+6 -2
View File
@@ -1,7 +1,9 @@
import type { SessionObserverDigest } from "../../../../packages/gateway-protocol/src/schema/sessions.js";
import type { GatewaySessionRow } from "../../api/types.ts";
import { withObserverWidget } from "./observer-dashboard.ts";
import { withSwarmWidget } from "./swarm-dashboard.ts";
import { isSwarmEnabledInConfig, SwarmRosterHydrator, withSwarmWidget } from "./swarm-dashboard.ts";
export { isSwarmEnabledInConfig, SwarmRosterHydrator };
import type { BoardSnapshot } from "./types.ts";
import type { BoardViewSnapshot } from "./view-types.ts";
@@ -9,6 +11,8 @@ export function withBuiltinDashboardWidgets(
snapshot: BoardSnapshot,
sessions: readonly GatewaySessionRow[],
observerDigests: readonly SessionObserverDigest[],
swarmEnabled = true,
): BoardViewSnapshot {
return withObserverWidget(withSwarmWidget(snapshot, sessions), observerDigests);
const withSwarm = swarmEnabled ? withSwarmWidget(snapshot, sessions) : snapshot;
return withObserverWidget(withSwarm, observerDigests);
}
@@ -0,0 +1,40 @@
import type { GatewaySessionRow } from "../../api/types.ts";
import { fetchChildSessionRows } from "../sessions/child-session-data.ts";
import type { SessionCapability } from "../sessions/index.ts";
const SWARM_SESSION_PAGE_SIZE = 10_000;
function isNewerSessionRow(candidate: GatewaySessionRow, current: GatewaySessionRow): boolean {
// Callers pass hydrated rows first and the current lifecycle-decorated page
// second, so equal persisted timestamps intentionally prefer the latter.
return (candidate.updatedAt ?? 0) >= (current.updatedAt ?? 0);
}
export function mergeSwarmSessionRows(
childRows: readonly GatewaySessionRow[],
currentRows: readonly GatewaySessionRow[],
): GatewaySessionRow[] {
const merged = new Map<string, GatewaySessionRow>();
for (const row of [...childRows, ...currentRows]) {
const current = merged.get(row.key);
if (!current || isNewerSessionRow(row, current)) {
merged.set(row.key, row);
}
}
return [...merged.values()];
}
export async function hydrateSwarmSessionRows(params: {
sessions: SessionCapability;
parentKey: string;
currentRows: readonly GatewaySessionRow[];
isCurrent: () => boolean;
}): Promise<GatewaySessionRow[] | null> {
const childRows = await fetchChildSessionRows({
sessions: params.sessions,
parentKey: params.parentKey,
isCurrent: params.isCurrent,
pageSize: SWARM_SESSION_PAGE_SIZE,
});
return childRows ? mergeSwarmSessionRows(params.currentRows, childRows) : null;
}
+267
View File
@@ -0,0 +1,267 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import type { GatewaySessionRow, SessionsListResult } from "../../api/types.ts";
import type { SessionCapability, SessionListOptions } from "../sessions/index.ts";
import { hydrateSwarmSessionRows, mergeSwarmSessionRows } from "./swarm-dashboard-roster.ts";
import { isSwarmEnabledInConfig, SwarmRosterHydrator } from "./swarm-dashboard.ts";
function row(index: number): GatewaySessionRow {
return {
key: `agent:worker:subagent:${index}`,
kind: "other",
updatedAt: index,
spawnedBy: "agent:main:parent",
swarmGroupId: "swarm:agent:main:parent:run-1",
} as unknown as GatewaySessionRow;
}
function result(rows: GatewaySessionRow[], offset: number, totalCount: number): SessionsListResult {
const nextOffset = offset + rows.length;
return {
ts: Date.now(),
path: "state/openclaw.sqlite",
count: rows.length,
totalCount,
limitApplied: 10_000,
offset,
nextOffset: nextOffset < totalCount ? nextOffset : null,
hasMore: nextOffset < totalCount,
defaults: {} as SessionsListResult["defaults"],
sessions: rows,
};
}
afterEach(() => {
vi.useRealTimers();
});
describe("isSwarmEnabledInConfig", () => {
it("accepts both the boolean and object configuration forms", () => {
expect(isSwarmEnabledInConfig({ tools: { swarm: true } })).toBe(true);
expect(isSwarmEnabledInConfig({ tools: { swarm: { enabled: true } } })).toBe(true);
expect(isSwarmEnabledInConfig({ tools: { swarm: false } })).toBe(false);
expect(isSwarmEnabledInConfig({ tools: { swarm: { enabled: false } } })).toBe(false);
expect(
isSwarmEnabledInConfig(
{
tools: { swarm: false },
agents: { entries: { worker: { tools: { swarm: true } } } },
},
"worker",
),
).toBe(true);
expect(
isSwarmEnabledInConfig(
{
tools: { swarm: true },
agents: { entries: { worker: { tools: { swarm: false } } } },
},
"worker",
),
).toBe(false);
expect(
isSwarmEnabledInConfig(
{
tools: { swarm: false },
agents: { list: [{ id: "worker", tools: { swarm: true } }] },
},
"worker",
),
).toBe(true);
});
});
describe("SwarmRosterHydrator", () => {
it("clears rows when the gateway source epoch changes", () => {
vi.useFakeTimers();
const onRows = vi.fn();
const hydrator = new SwarmRosterHydrator();
const sessions = {
canonicalListRevision: 0,
list: vi.fn(async () => result([row(0)], 0, 1)),
} as unknown as SessionCapability;
hydrator.update({
sessions,
parentKey: "agent:main:parent",
sourceEpoch: 1,
currentRows: () => [row(0)],
onRows,
});
expect(hydrator.rows).toHaveLength(1);
hydrator.update({
sessions,
parentKey: "agent:main:parent",
sourceEpoch: 2,
currentRows: () => [],
onRows,
});
expect(hydrator.rows).toEqual([]);
expect(onRows).toHaveBeenLastCalledWith([]);
hydrator.dispose();
});
it("keeps a freshly fetched tie winner over an unchanged current page", async () => {
vi.useFakeTimers();
const running = { ...row(0), status: "running" as const, updatedAt: 5 };
const done = { ...row(0), status: "done" as const, updatedAt: 5 };
const hydrator = new SwarmRosterHydrator();
const sessions = {
canonicalListRevision: 0,
list: vi.fn(async () => result([done], 0, 1)),
} as unknown as SessionCapability;
hydrator.update({
sessions,
parentKey: "agent:main:parent",
sourceEpoch: 1,
currentRows: () => [running],
onRows: () => undefined,
});
await vi.runAllTimersAsync();
expect(hydrator.rows).toEqual([expect.objectContaining({ status: "done" })]);
hydrator.dispose();
});
it("keeps retrying at a bounded cadence after three transient failures", async () => {
vi.useFakeTimers();
const onRows = vi.fn();
const list = vi
.fn()
.mockRejectedValueOnce(new Error("offline"))
.mockRejectedValueOnce(new Error("offline"))
.mockRejectedValueOnce(new Error("offline"))
.mockResolvedValue(result([row(0)], 0, 1));
const hydrator = new SwarmRosterHydrator();
const sessions = { canonicalListRevision: 0, list } as unknown as SessionCapability;
hydrator.update({
sessions,
parentKey: "agent:main:parent",
sourceEpoch: 1,
currentRows: () => [],
onRows,
});
await vi.runAllTimersAsync();
expect(list).toHaveBeenCalledTimes(4);
expect(hydrator.rows).toEqual([expect.objectContaining({ key: row(0).key })]);
hydrator.dispose();
});
});
describe("hydrateSwarmSessionRows", () => {
it("hydrates paginated cross-agent children outside the normal session page", async () => {
const children = Array.from({ length: 10_055 }, (_, index) => row(index));
const list = vi.fn(async (options: SessionListOptions) => {
const offset = options.offset ?? 0;
return result(children.slice(offset, offset + 10_000), offset, children.length);
});
const currentChild = {
...row(0),
status: "running" as const,
updatedAt: 2_000,
} satisfies GatewaySessionRow;
const currentRows: GatewaySessionRow[] = [
{
key: "agent:main:parent",
kind: "main",
updatedAt: 2_000,
} as unknown as GatewaySessionRow,
currentChild,
];
const rows = await hydrateSwarmSessionRows({
sessions: { list } as unknown as SessionCapability,
parentKey: "agent:main:parent",
currentRows,
isCurrent: () => true,
});
expect(rows).toHaveLength(10_056);
expect(rows?.find((candidate) => candidate.key === currentChild.key)?.status).toBe("running");
expect(list).toHaveBeenCalledTimes(2);
expect(list).toHaveBeenNthCalledWith(
1,
expect.objectContaining({
spawnedBy: "agent:main:parent",
limit: 10_000,
includeGlobal: false,
configuredAgentsOnly: true,
}),
);
expect(list).toHaveBeenNthCalledWith(2, expect.objectContaining({ offset: 10_000 }));
});
it("prefers the post-request server row when persisted timestamps tie", async () => {
const current = { ...row(0), status: "running" as const, updatedAt: 5 };
const fetched = { ...row(0), status: "done" as const, updatedAt: 5 };
const rows = await hydrateSwarmSessionRows({
sessions: {
list: vi.fn(async () => result([fetched], 0, 1)),
} as unknown as SessionCapability,
parentKey: "agent:main:parent",
currentRows: [current],
isCurrent: () => true,
});
expect(rows).toEqual([expect.objectContaining({ key: fetched.key, status: "done" })]);
});
it("restarts pagination when updated rows move across offset boundaries", async () => {
const running = { ...row(1), status: "running" as const };
const done = { ...row(1), status: "done" as const, updatedAt: 10 };
const pages = [
[row(0), running],
[running, row(2)],
[row(3), row(0)],
[done, row(2)],
];
let callIndex = 0;
const list = vi.fn(async (options: SessionListOptions) => {
const rows = pages[callIndex] ?? [];
callIndex += 1;
return result(rows, options.offset ?? 0, 4);
});
const rows = await hydrateSwarmSessionRows({
sessions: { list } as unknown as SessionCapability,
parentKey: "agent:main:parent",
currentRows: [],
isCurrent: () => true,
});
expect(rows?.map((candidate) => candidate.key).toSorted()).toEqual(
[row(0).key, row(1).key, row(2).key, row(3).key].toSorted(),
);
expect(rows?.find((candidate) => candidate.key === done.key)?.status).toBe("done");
expect(list).toHaveBeenCalledTimes(4);
});
it("keeps the freshest row when hydration overlaps a current-page snapshot", () => {
const stale = { ...row(0), status: "running" as const, updatedAt: 5, runtimeSampledAt: 10 };
const fresh = { ...row(0), status: "done" as const, updatedAt: 6, runtimeSampledAt: 20 };
expect(mergeSwarmSessionRows([fresh], [stale])).toEqual([fresh]);
expect(mergeSwarmSessionRows([stale], [fresh])).toEqual([fresh]);
const decorated = { ...stale, status: "done" as const };
expect(mergeSwarmSessionRows([stale], [decorated])).toEqual([decorated]);
});
it("drops stale hydration results", async () => {
const rows = await hydrateSwarmSessionRows({
sessions: {
list: vi.fn(async () => result([row(0)], 0, 1)),
} as unknown as SessionCapability,
parentKey: "agent:main:parent",
currentRows: [],
isCurrent: () => false,
});
expect(rows).toBeNull();
});
});
+138
View File
@@ -1,12 +1,150 @@
import { asNullableRecord } from "@openclaw/normalization-core/record-coerce";
import type { GatewaySessionRow } from "../../api/types.ts";
import { t } from "../../i18n/index.ts";
import type { SessionCapability } from "../sessions/index.ts";
import { areUiSessionKeysEquivalent } from "../sessions/session-key.ts";
import { hydrateSwarmSessionRows, mergeSwarmSessionRows } from "./swarm-dashboard-roster.ts";
import type { BoardSnapshot } from "./types.ts";
import type { BoardViewSnapshot } from "./view-types.ts";
const SWARM_TAB_ID = "builtin-swarm";
const SWARM_WIDGET_NAME = "builtin:swarm";
function readSwarmEnabled(value: unknown): boolean | undefined {
if (typeof value === "boolean") {
return value;
}
const enabled = asNullableRecord(value)?.enabled;
return typeof enabled === "boolean" ? enabled : undefined;
}
export function isSwarmEnabledInConfig(config: unknown, agentId?: string): boolean {
const root = asNullableRecord(config);
const globalEnabled = readSwarmEnabled(asNullableRecord(root?.tools)?.swarm);
const agents = asNullableRecord(root?.agents);
const entries = asNullableRecord(agents?.entries);
const listedEntries = Array.isArray(agents?.list)
? agents.list
: Array.isArray(agents?.entries)
? agents.entries
: [];
const listedAgent = agentId
? listedEntries.map((entry) => asNullableRecord(entry)).find((entry) => entry?.id === agentId)
: undefined;
const agent = agentId ? (asNullableRecord(entries?.[agentId]) ?? listedAgent) : null;
const agentEnabled = readSwarmEnabled(asNullableRecord(agent?.tools)?.swarm);
return agentEnabled ?? globalEnabled ?? false;
}
type SwarmHydrationParams = {
sessions: SessionCapability;
parentKey: string;
sourceEpoch: number;
currentRows: () => readonly GatewaySessionRow[];
onRows: (rows: GatewaySessionRow[]) => void;
};
export class SwarmRosterHydrator {
rows: GatewaySessionRow[] = [];
private key = "";
private revision = -1;
private generation = 0;
private attemptRevision = -1;
private attempts = 0;
private timer: ReturnType<typeof setTimeout> | null = null;
update(params: SwarmHydrationParams): void {
const key = `${params.sourceEpoch}:${params.parentKey}`;
if (this.key !== key) {
this.reset(key);
}
this.rows = mergeSwarmSessionRows(this.rows, params.currentRows());
params.onRows(this.rows);
const revision = params.sessions.canonicalListRevision;
if (this.attemptRevision !== revision) {
this.attemptRevision = revision;
this.attempts = 0;
}
if (this.revision === revision || this.timer !== null) {
return;
}
this.timer = setTimeout(() => this.hydrate(params), 250);
}
dispose(): void {
this.reset("");
}
private hydrate(params: SwarmHydrationParams): void {
const generation = this.generation;
const revision = params.sessions.canonicalListRevision;
const key = `${params.sourceEpoch}:${params.parentKey}`;
const isCurrent = () => generation === this.generation && this.key === key;
const currentRowsAtStart = params.currentRows();
const currentRowsAtStartByKey = new Map(
currentRowsAtStart.map((row) => [row.key, JSON.stringify(row)]),
);
let hydrated = false;
let retrying = false;
this.attempts += 1;
void hydrateSwarmSessionRows({
sessions: params.sessions,
parentKey: params.parentKey,
currentRows: currentRowsAtStart,
isCurrent,
})
.then((rows) => {
if (!rows || !isCurrent()) {
return;
}
hydrated = true;
this.revision = revision;
const changedCurrentRows = params
.currentRows()
.filter((row) => currentRowsAtStartByKey.get(row.key) !== JSON.stringify(row));
this.rows = mergeSwarmSessionRows(rows, changedCurrentRows);
params.onRows(this.rows);
})
.catch(() => {
if (!isCurrent()) {
return;
}
retrying = true;
const retryDelayMs = Math.min(30_000, 1_000 * 2 ** Math.min(this.attempts - 1, 5));
this.timer = setTimeout(() => {
this.timer = null;
if (isCurrent()) {
this.update(params);
}
}, retryDelayMs);
})
.finally(() => {
if (!isCurrent()) {
return;
}
if (!retrying) {
this.timer = null;
}
if (hydrated && this.revision !== params.sessions.canonicalListRevision) {
this.update(params);
}
});
}
private reset(key: string): void {
if (this.timer !== null) {
clearTimeout(this.timer);
}
this.rows = [];
this.key = key;
this.revision = -1;
this.generation += 1;
this.attemptRevision = -1;
this.attempts = 0;
this.timer = null;
}
}
function hasSwarmRowsForSession(
sessions: readonly GatewaySessionRow[],
sessionKey: string,
+38
View File
@@ -64,6 +64,44 @@ describe("swarm board widget", () => {
]);
});
it("renders every child beyond the ordinary 50-row session page", () => {
const container = document.createElement("div");
document.body.append(container);
render(
renderSwarmWidget({
sessionKey: parentSessionKey,
sessions: Array.from({ length: 55 }, (_, index) =>
session({ key: `child-${index}`, status: "running" }),
),
}),
container,
);
expect(container.querySelectorAll(".swarm-widget__dot")).toHaveLength(55);
});
it("caps historical dots while keeping active workers visible", () => {
const container = document.createElement("div");
document.body.append(container);
render(
renderSwarmWidget({
sessionKey: parentSessionKey,
sessions: [
...Array.from({ length: 300 }, (_, index) =>
session({ key: `done-${index}`, status: "done" }),
),
session({ key: "running", status: "running" }),
],
}),
container,
);
expect(container.querySelectorAll(".swarm-widget__dot")).toHaveLength(256);
expect(container.querySelector(".swarm-widget__dot--running")).not.toBeNull();
expect(container.querySelector(".swarm-widget__more")?.textContent?.trim()).toBe("+45");
expect(container.textContent?.replace(/\s+/g, " ")).toContain("1 Running · 300 Done");
});
it("renders dot tooltips and keeps an empty state when no group is active", () => {
const container = document.createElement("div");
document.body.append(container);
+24 -1
View File
@@ -5,15 +5,20 @@ import { areUiSessionKeysEquivalent } from "../../sessions/session-key.ts";
type SwarmDotStatus = "queued" | "running" | "done" | "failed";
const SWARM_DOT_STATUS_RANK = { running: 0, queued: 1, failed: 2, done: 3 } as const;
type SwarmDot = {
key: string;
label: string;
status: SwarmDotStatus;
};
const MAX_RENDERED_DOTS_PER_PHASE = 256;
type SwarmPhase = {
title?: string;
dots: SwarmDot[];
hidden: number;
};
type SwarmGroup = {
@@ -140,7 +145,20 @@ function collectActiveSwarmGroups(
narrator: entries.map((entry) => entry.log).find(Boolean),
phases: [...phases.entries()]
.toSorted((left, right) => left[1].rank - right[1].rank)
.map(([title, bucket]) => ({ title, dots: bucket.dots })),
.map(([title, bucket]) => {
const visibleFirst =
bucket.dots.length > MAX_RENDERED_DOTS_PER_PHASE
? bucket.dots.toSorted(
(left, right) =>
SWARM_DOT_STATUS_RANK[left.status] - SWARM_DOT_STATUS_RANK[right.status],
)
: bucket.dots;
return {
title,
dots: visibleFirst.slice(0, MAX_RENDERED_DOTS_PER_PHASE),
hidden: Math.max(0, visibleFirst.length - MAX_RENDERED_DOTS_PER_PHASE),
};
}),
} satisfies SwarmGroup;
})
.filter((group) =>
@@ -196,6 +214,11 @@ export function renderSwarmWidget({
></span>
`,
)}
${phase.hidden > 0
? html`<span class="swarm-widget__more" role="listitem"
>+${phase.hidden}</span
>`
: nothing}
</div>
</div>
`,
+60
View File
@@ -0,0 +1,60 @@
import type { GatewaySessionRow } from "../../api/types.ts";
import type { SessionCapability } from "./index.ts";
const MAX_CHILD_SESSION_LIST_PASSES = 4;
export async function fetchChildSessionRows(params: {
sessions: SessionCapability;
parentKey: string;
isCurrent: () => boolean;
pageSize?: number;
}): Promise<GatewaySessionRow[] | null> {
const rowsByKey = new Map<string, GatewaySessionRow>();
const pageSize = params.pageSize ?? 20;
for (let pass = 0; pass < MAX_CHILD_SESSION_LIST_PASSES; pass += 1) {
const seenOffsets = new Set<number>();
const rowsBeforePass = rowsByKey.size;
let expectedTotal: number | undefined;
let offset = 0;
while (!seenOffsets.has(offset)) {
seenOffsets.add(offset);
const result = await params.sessions.list({
spawnedBy: params.parentKey,
...(offset > 0 ? { offset } : {}),
limit: pageSize,
includeGlobal: false,
includeUnknown: false,
configuredAgentsOnly: true,
});
if (!params.isCurrent()) {
return null;
}
if (!result) {
throw new Error("child session list returned no result");
}
expectedTotal = result.totalCount;
const runtimeSampledAt = Date.now();
for (const row of result.sessions) {
// A later pass is a fresher server observation even when the row moved
// across an updatedAt-sorted offset boundary.
rowsByKey.set(row.key, { ...row, runtimeSampledAt });
}
const hasMore =
result.hasMore ??
(typeof result.totalCount === "number" &&
offset + result.sessions.length < result.totalCount);
const nextOffset = result.nextOffset ?? offset + result.sessions.length;
if (!hasMore || nextOffset <= offset) {
break;
}
offset = nextOffset;
}
const addedThisPass = rowsByKey.size - rowsBeforePass;
if (addedThisPass === 0 || expectedTotal === undefined || rowsByKey.size >= expectedTotal) {
break;
}
// updatedAt ordering can move a child across an offset boundary while paging.
// Repeat from zero until the deduplicated roster reaches the latest total.
}
return [...rowsByKey.values()];
}
+2 -3
View File
@@ -189,7 +189,7 @@ export type SessionMessageSubscription = {
export type SessionCapability = {
readonly state: SessionState;
/** Advances only when a canonical sessions.list response is published. */
/** Advances only when a canonical sessions.list result is published. */
readonly canonicalListRevision: number;
list: (options?: SessionListOptions) => Promise<SessionsListResult | null>;
setCreatorFilter: (creatorId: string | null) => Promise<void>;
@@ -869,7 +869,7 @@ export function createSessionCapability(gateway: SessionGateway): SessionCapabil
return null;
}
const result = await requestSessionList(scope.client, options);
return isCurrentConnection(scope) ? (result ?? null) : null;
return isCurrentConnection(scope) ? swarmActivity.decorate(result ?? null) : null;
};
const publish = (next: SessionState) => {
@@ -1419,7 +1419,6 @@ export function createSessionCapability(gateway: SessionGateway): SessionCapabil
payload: unknown,
options?: SessionReconcileOptions,
): SessionChangedResult => {
swarmActivity.observe(payload);
const base = reconcileSessionChanged(state.result, payload, options);
const result = swarmActivity.decorate(base.result);
const reconciled =
@@ -96,8 +96,13 @@ describe("session swarm activity", () => {
emitEvent({ type: "event", event: "sessions.changed", payload });
await sessions.refresh({ force: true });
const revisionBeforePhase = sessions.canonicalListRevision;
emitChanged(note("phase", "Plan"));
expect(sessions.canonicalListRevision).toBe(revisionBeforePhase);
await waitForFast(() => expect(request).toHaveBeenCalledTimes(2));
await waitForFast(() =>
expect(sessions.canonicalListRevision).toBeGreaterThan(revisionBeforePhase),
);
rows = [
...rows,
{
+11 -9
View File
@@ -2,8 +2,10 @@ import { asNullableRecord } from "@openclaw/normalization-core/record-coerce";
import type { GatewaySessionRow, SessionsListResult } from "../../api/types.ts";
// Lifecycle notes are transient UI state, so bound them for long-lived board tabs.
const MAX_TRACKED_SWARM_GROUPS = 128;
const MAX_TRACKED_SWARM_CHILDREN = 2_048;
const MAX_TRACKED_SWARM_GROUPS = 10_000;
// Completed members stay visible while any group child is active, so retain the
// supported lifetime membership ceiling rather than only the live-child cap.
const MAX_TRACKED_SWARM_CHILDREN = 100_000;
type SwarmDisplayCarrier = {
swarmPhaseRank?: number;
@@ -45,17 +47,16 @@ export class SwarmActivityTracker {
this.phaseByChild.clear();
}
observe(payload: unknown): void {
observe(payload: unknown): boolean {
const event = asNullableRecord(payload);
if (!event) {
return;
return false;
}
const source = asNullableRecord(event.session) ?? event;
const groupId = normalizedString(event.swarmGroupId) ?? normalizedString(source.swarmGroupId);
if (!groupId) {
return;
return false;
}
const kind = normalizedString(event.kind);
const text = normalizedString(event.text);
if ((kind === "phase" || kind === "log") && text) {
@@ -77,17 +78,17 @@ export class SwarmActivityTracker {
text,
MAX_TRACKED_SWARM_GROUPS,
);
return;
return true;
}
const childKey = normalizedString(source.key) ?? normalizedString(event.sessionKey);
if (!childKey) {
return;
return true;
}
const explicitPhase = normalizedString(source.swarmPhase) ?? normalizedString(event.swarmPhase);
if (explicitPhase) {
setBounded(this.phaseByChild, childKey, explicitPhase, MAX_TRACKED_SWARM_CHILDREN);
return;
return true;
}
// Implicit phase assignment is a creation-time fact: only a child ADMITTED
// after phase('X') belongs to X. Status/completion updates for a child that
@@ -98,6 +99,7 @@ export class SwarmActivityTracker {
setBounded(this.phaseByChild, childKey, currentPhase, MAX_TRACKED_SWARM_CHILDREN);
}
}
return true;
}
decorate(result: SessionsListResult | null): SessionsListResult | null {
+62 -19
View File
@@ -91,6 +91,7 @@ import {
type BoardFace,
type BoardSessionView,
} from "../../lib/board/settings.ts";
import type { SwarmRosterHydrator } from "../../lib/board/swarm-dashboard.ts";
import type { BoardSnapshot, BoardTab } from "../../lib/board/types.ts";
import type { BoardViewSnapshot } from "../../lib/board/view-types.ts";
import {
@@ -492,7 +493,7 @@ class ChatPane extends OpenClawLightDomElement {
private readonly observerDigestHistory = new ObserverDigestHistory();
private builtinBoardSnapshot: BoardViewSnapshot | null = null;
private builtinBoardSnapshotBase: BoardSnapshot | null = null;
private builtinBoardSnapshotRequest = 0;
private swarmHydrator: SwarmRosterHydrator | null = null;
private readonly sessionDiscussionStates = new Map<string, SessionDiscussionState>();
private readonly sessionDiscussionOpenUrls = new Map<string, string | null>();
private readonly sessionDiscussionProbes = new Set<string>();
@@ -623,7 +624,11 @@ class ChatPane extends OpenClawLightDomElement {
)
.watch(
() => this.context?.runtimeConfig,
(runtimeConfig, notify) => runtimeConfig.subscribe(notify),
(runtimeConfig, notify) =>
runtimeConfig.subscribe(() => {
this.refreshBuiltinBoardSnapshot();
notify();
}),
)
.watch(
() => this.resolveBoardProvider(),
@@ -1825,22 +1830,54 @@ class ChatPane extends OpenClawLightDomElement {
if (!state) {
return;
}
const request = ++this.builtinBoardSnapshotRequest;
const sessions = state.sessionsResult?.sessions ?? [];
void import("../../lib/board/builtin-dashboard.ts").then(({ withBuiltinDashboardWidgets }) => {
if (request !== this.builtinBoardSnapshotRequest) {
return;
}
const currentBase = this.resolveBoardProvider().snapshot$.value;
const sessionKey = this.resolveBoardSessionKey(currentBase.sessionKey);
this.builtinBoardSnapshotBase = currentBase;
this.builtinBoardSnapshot = withBuiltinDashboardWidgets(
currentBase,
sessions,
this.observerDigestHistory.get(sessionKey),
);
this.requestUpdate();
});
const parentKey = this.resolveBoardSessionKey();
const sourceEpoch = state.connectionEpoch;
void import("../../lib/board/builtin-dashboard.ts").then(
({ isSwarmEnabledInConfig, SwarmRosterHydrator, withBuiltinDashboardWidgets }) => {
if (
!this.state ||
this.state.connectionEpoch !== sourceEpoch ||
parentKey !== this.resolveBoardSessionKey()
) {
return;
}
const swarmEnabled =
this.state.connected &&
isSwarmEnabledInConfig(
this.context.runtimeConfig?.state.configSnapshot?.config,
resolveAgentIdFromSessionKey(parentKey),
);
const applyRows = (rows: readonly GatewaySessionRow[], includeSwarm: boolean) => {
const base = this.resolveBoardProvider().snapshot$.value;
const sessionKey = this.resolveBoardSessionKey(base.sessionKey);
this.builtinBoardSnapshotBase = base;
this.builtinBoardSnapshot = withBuiltinDashboardWidgets(
base,
rows,
this.observerDigestHistory.get(sessionKey),
includeSwarm,
);
this.requestUpdate();
};
if (!swarmEnabled) {
this.swarmHydrator?.dispose();
this.swarmHydrator = null;
applyRows(this.state.sessionsResult?.sessions ?? [], false);
return;
}
this.swarmHydrator ??= new SwarmRosterHydrator();
this.swarmHydrator.update({
sessions: this.context.sessions,
parentKey,
sourceEpoch,
currentRows: () =>
this.state?.connectionEpoch === sourceEpoch
? (this.state.sessionsResult?.sessions ?? [])
: [],
onRows: (rows) => applyRows(rows, true),
});
},
);
}
private recordObserverDigest(digest: SessionObserverDigest): void {
@@ -2614,6 +2651,8 @@ class ChatPane extends OpenClawLightDomElement {
window.clearTimeout(this.headerCopiedTimer);
this.headerCopiedTimer = null;
}
this.swarmHydrator?.dispose();
this.swarmHydrator = null;
this.headerWorktreePaths.clear();
this.headerBranches.clear();
this.presencePayload = undefined;
@@ -2748,6 +2787,10 @@ class ChatPane extends OpenClawLightDomElement {
// A reconnect can retain the browser client. Keep async ownership tied
// to the logical connection, not only the transport object identity.
this.connectionGeneration += 1;
this.swarmHydrator?.dispose();
this.swarmHydrator = null;
this.builtinBoardSnapshot = null;
this.builtinBoardSnapshotBase = null;
this.taskSuggestionsRequestVersion += 1;
this.taskSuggestions = [];
this.taskSuggestionBusyIds.clear();
@@ -3907,7 +3950,7 @@ class ChatPane extends OpenClawLightDomElement {
board.hasBoard && board.face === "dashboard"
? renderBoardSessionSurface({
snapshot: board.snapshot,
sessions: state.sessionsResult?.sessions ?? [],
sessions: this.swarmHydrator?.rows ?? state.sessionsResult?.sessions ?? [],
observer: {
activeRunId: observerRunId,
digests: this.observerDigestHistory.get(
+6
View File
@@ -388,6 +388,12 @@ openclaw-board-widget-cell {
width: 9px;
}
.swarm-widget__more {
color: var(--muted, #8a919e);
font-size: 10px;
line-height: 9px;
}
.swarm-widget__dot--running {
animation: swarm-widget-pulse 1.25s ease-in-out infinite;
background: var(--accent, #ff5c5c);