Files
openclaw/test/e2e/qa-lab/runtime/managed-worktrees-workboard-lifecycle-product-proof.e2e.test.ts
Peter Steinberger 73bdb4b924 feat(agents): record run-end worktree cleanup outcome; prove Workboard dirty retention (#120434)
* feat(agents): record run-end worktree cleanup outcome

Persist removed, retained, and failed run-end cleanup outcomes on managed worktree records. Operators and QA can inspect the durable fact through worktrees.list and openclaw worktrees list --json.

Release note: Managed worktree run-end cleanup now records why a checkout was removed or retained in worktree list JSON.

* test(qa): prove dirty worktree retention outcome

* chore(protocol): regenerate swift gateway models

* fix(agents): harden worktree cleanup recovery

Register run_end_cleanup_json as a lazy compatible column so same-version v6 index repair and read-only doctor migration can recover databases created before the column existed.

Type removal contention at the registry boundary; unexpected claim failures now best-effort record a bounded failed outcome and rethrow the original error.

* fix(ci): clear repo-wide lint debt blocking merge gates

The red-main landing rule requires this PR to repair repository-wide merge-gate debt instead of bypassing it. Apply the current lint contracts mechanically and split turn-transition coverage into a concept-named sibling with per-file-safe test state.

Exact line delta: +676/-574 (net +102) across 44 test/support files.

* fix(ci): preserve cached health refresh proof

Require the public refresh call to exist before accepting that sensitive fields were omitted, so the boundary proof cannot pass on a missing call.

* fix(ci): correct test typing left by the lint sweep

Literal-widened totalTokensVersion fixtures, a WebSocket RawData overload
mismatch, and the protocol schema document cast broke check-test-types
after the repo-wide lint repair. Aligns the fixtures with SessionEntry,
narrows Buffer handling per RawData, and keeps the JSON-shaped undefined
omission under structuredClone.

* test(agents): reuse upstream resource-loader test support

The session-loop split and #120463's helper extraction landed the same
createResourceLoader/createCompactionHandlers twice; the rebase kept both,
orphaning main's agent-session-loop-resource-loader.test-support.ts and
failing the dead-code gate. Import the upstream helpers and delete the
duplicates.

* fix(agents): reject finalized rows at the worktree removal claim

Address the accepted ClawSweeper late-claim finding by rereading and rejecting missing or finalized worktree rows inside the synchronous removal-claim transaction.

Preserve the authoritative cleanup invariant: finalized contenders record nothing, while retained-busy is written only while the row remains live.

* refactor(agents): reuse registry update for busy outcomes

Keep the live-row conditional write in the canonical registry update path so the finalized-claim repair stays below the registry max-lines ratchet without weakening the authoritative-outcome invariant.

* test(agents): drop session test duplicates after rebase

Keep current main as the canonical owner of next-turn lifecycle coverage and correctness test support after replaying the older lint-debt split.

* fix(agents): guard post-abort cleanup outcomes against finalization

After abortWorktreeRemoval releases a stale remover's claim, its retained or
failed write raced a finalizing remover and could overwrite the authoritative
removed-lossless fact. Route every retained/failed write through the live-row
condition; only the finalizing remover's own removed-lossless write stays
unconditional.

* fix(agents): persist the removal outcome atomically with finalization

A delayed removed-lossless write after remove() finalized could race a
restore plus newer cleanup and overwrite the newer operator-visible fact.
The run-end outcome now rides remove()'s finalization update; every other
cleanup write stays live-row conditional, so no post-finalize write path
remains.

* test(qa): restore strict cached-health contract assertions

The lint sweep's Boolean() coercions let truthy non-booleans satisfy the
wire-typed cached-meta contract. Assert the literal boolean for unknown-typed
fields and use nullish-coalesced strict equivalents for boolean chains.

* fix(agents): clear the stale cleanup outcome when restoring a worktree

A restored checkout begins a new lifecycle; leaving the removed-lossless
fact on the live row showed operators a stale result until the next
cleanup. Restore clears the recorded outcome and the regression asserts
the cleared state before the next cleanup records fresh truth.

* fix(agents): scope stale cleanup outcomes to their observed lifecycle

A stale remover's retained/failed write raced a concurrent remove-plus-
restore: the revived row is live again, so the live-row condition alone
could stamp a prior-lifecycle outcome. Condition those writes on the
activity stamp the remover observed; restore bumps lastActiveAt, making
any prior-lifecycle write a no-op.

* fix(agents): advance the restore activity stamp within one millisecond

Stale cleanup writes fence on the activity stamp they observed; a restore
completing in the same millisecond could revive the row with an identical
stamp and let the fence match. Restore now always advances past the
stored value, and the ABA regression pins the clock to prove the
same-millisecond case.
2026-08-08 20:32:11 -07:00

327 lines
11 KiB
TypeScript

// QA Lab product proof for the Workboard-owned managed-worktree lifecycle.
import { execFile } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { setTimeout as sleep } from "node:timers/promises";
import { promisify } from "node:util";
import { afterEach, describe, expect, it } from "vitest";
import { startQaLiveLaneGateway } from "../../../../extensions/qa-lab/runtime-api.js";
import type { ManagedWorktreeRecord } from "../../../../src/agents/worktrees/types.js";
import { useAutoCleanupTempDirTracker } from "../../../helpers/temp-dir.js";
const execFileAsync = promisify(execFile);
type WorkboardWorkspace = {
kind: "worktree";
path: string;
branch?: string;
sourcePath?: string;
sourceBranch?: string;
};
type WorkboardCard = {
id: string;
runId?: string;
status?: string;
metadata?: { automation?: { workspace?: WorkboardWorkspace } };
};
type WorkboardCreateResult = { card: WorkboardCard };
type WorkboardCompleteResult = { card: WorkboardCard };
type WorkboardListResult = { cards: WorkboardCard[] };
type WorkboardDispatchResult = {
started: Array<{ cardId: string; runId: string; sessionKey: string; title: string }>;
startFailures: Array<{ cardId: string; error: string; title: string }>;
};
type WorktreeListResult = { worktrees: ManagedWorktreeRecord[] };
type GatewayRunResult = { status?: unknown };
let harness: Awaited<ReturnType<typeof startQaLiveLaneGateway>> | undefined;
afterEach(async () => {
await harness?.stop().catch(() => undefined);
harness = undefined;
});
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
async function git(cwd: string, ...args: string[]): Promise<string> {
const { stdout } = await execFileAsync("git", ["-C", cwd, ...args], { encoding: "utf8" });
return stdout.trimEnd();
}
async function initializeRepository(root: string): Promise<string> {
const repo = path.join(root, "source");
const remote = path.join(root, "origin.git");
await fs.mkdir(repo, { recursive: true });
await git(repo, "init", "-b", "main");
await git(repo, "config", "user.name", "OpenClaw Test");
await git(repo, "config", "user.email", "openclaw-test@example.invalid");
await fs.writeFile(path.join(repo, "README.md"), "base\n");
await git(repo, "add", "README.md");
await git(repo, "commit", "-m", "initialize Workboard worktree fixture");
await git(root, "init", "--bare", remote);
await git(repo, "remote", "add", "origin", remote);
await git(repo, "push", "-u", "origin", "main");
await git(remote, "symbolic-ref", "HEAD", "refs/heads/main");
return await fs.realpath(repo);
}
async function startHarness() {
harness = await startQaLiveLaneGateway({
repoRoot: process.cwd(),
providerMode: "mock-openai",
primaryModel: "mock-openai/gpt-5.6-luna",
alternateModel: "mock-openai/gpt-5.6-luna",
transport: {
requiredPluginIds: [],
createGatewayConfig: () => ({}),
},
transportBaseUrl: "http://127.0.0.1",
controlUiEnabled: false,
mutateConfig: (config) => ({
...config,
plugins: {
...config.plugins,
allow: [...new Set([...(config.plugins?.allow ?? []), "workboard"])],
entries: {
...config.plugins?.entries,
workboard: { enabled: true },
},
},
}),
});
return harness;
}
function managedWorktreeName(cardId: string): string {
const suffix = cardId
.toLowerCase()
.replace(/[^a-z0-9-]/g, "-")
.replace(/-+/g, "-");
return `wb-${suffix}`.slice(0, 64).replace(/-$/, "");
}
async function createCard(params: {
boardId: string;
repo: string;
title: string;
}): Promise<WorkboardCard> {
if (!harness) {
throw new Error("QA gateway harness is not running");
}
const created = (await harness.gateway.call("workboard.cards.create", {
title: params.title,
status: "ready",
agentId: "qa",
boardId: params.boardId,
workspace: { kind: "worktree", path: params.repo, branch: "main" },
})) as WorkboardCreateResult;
return created.card;
}
async function listWorktrees(): Promise<WorktreeListResult> {
if (!harness) {
throw new Error("QA gateway harness is not running");
}
return (await harness.gateway.call("worktrees.list", {})) as WorktreeListResult;
}
async function waitForMaterializedWorktree(params: {
name: string;
stateDir: string;
timeoutMs?: number;
}): Promise<string> {
const worktreesRoot = path.join(params.stateDir, "worktrees");
const deadline = Date.now() + (params.timeoutMs ?? 15_000);
while (Date.now() < deadline) {
const fingerprints = await fs.readdir(worktreesRoot, { withFileTypes: true }).catch(() => []);
for (const fingerprint of fingerprints) {
if (!fingerprint.isDirectory()) {
continue;
}
const candidate = path.join(worktreesRoot, fingerprint.name, params.name);
try {
return await fs.realpath(candidate);
} catch {
// The dispatcher has not materialized this checkout yet.
}
}
await sleep(20);
}
throw new Error(`timed out waiting for managed worktree ${params.name}`);
}
async function dispatchCardAndWaitForWorktree(params: {
boardId: string;
cardId: string;
name: string;
stateDir: string;
}): Promise<{ materializedPath: string; started: WorkboardDispatchResult["started"][number] }> {
if (!harness) {
throw new Error("QA gateway harness is not running");
}
const dispatchPromise = harness.gateway.call("workboard.cards.dispatch", {
boardId: params.boardId,
});
const materializedPromise = waitForMaterializedWorktree({
name: params.name,
stateDir: params.stateDir,
});
const dispatch = (await dispatchPromise) as WorkboardDispatchResult;
expect(dispatch.startFailures).toEqual([]);
expect(dispatch.started).toEqual([
expect.objectContaining({ cardId: params.cardId, runId: expect.any(String) }),
]);
return {
materializedPath: await materializedPromise,
started: dispatch.started[0]!,
};
}
async function waitForWorktreeState(params: {
id: string;
predicate: (record: ManagedWorktreeRecord | undefined) => boolean;
timeoutMs?: number;
}): Promise<ManagedWorktreeRecord | undefined> {
const deadline = Date.now() + (params.timeoutMs ?? 10_000);
let record: ManagedWorktreeRecord | undefined;
while (Date.now() < deadline) {
record = (await listWorktrees()).worktrees.find((entry) => entry.id === params.id);
if (params.predicate(record)) {
return record;
}
await sleep(50);
}
throw new Error(`timed out waiting for managed worktree state ${params.id}`);
}
describe("managed worktrees Workboard-owner product proof", () => {
it(
"removes clean card worktrees and records dirty run-end retention",
{ timeout: 240_000 },
async () => {
const canonicalTmp = await fs.realpath(os.tmpdir());
const fixtureRoot = tempDirs.make("openclaw-managed-worktree-workboard-", canonicalTmp);
const repo = await initializeRepository(fixtureRoot);
const activeHarness = await startHarness();
const stateDir = path.join(await fs.realpath(activeHarness.gateway.tempRoot), "state");
const boardId = "qa-worktree-clean";
const card = await createCard({ boardId, repo, title: "Clean worktree lifecycle" });
const name = managedWorktreeName(card.id);
const { materializedPath, started } = await dispatchCardAndWaitForWorktree({
boardId,
cardId: card.id,
name,
stateDir,
});
const cards = (await activeHarness.gateway.call("workboard.cards.list", {
boardId,
})) as WorkboardListResult;
const dispatchedCard = cards.cards.find((entry) => entry.id === card.id);
const dispatchedWorkspace = dispatchedCard?.metadata?.automation?.workspace;
expect(dispatchedWorkspace).toMatchObject({
kind: "worktree",
branch: `openclaw/${name}`,
sourcePath: repo,
sourceBranch: "main",
});
expect(await fs.realpath(dispatchedWorkspace?.path ?? "")).toBe(materializedPath);
expect(dispatchedCard?.runId).toBe(started.runId);
const activeRecord = (await listWorktrees()).worktrees.find(
(record) => record.ownerKind === "workboard" && record.ownerId === card.id,
);
expect(activeRecord).toMatchObject({
name,
branch: `openclaw/${name}`,
repoRoot: repo,
ownerKind: "workboard",
ownerId: card.id,
});
expect(await fs.realpath(activeRecord?.path ?? "")).toBe(materializedPath);
const terminal = (await activeHarness.gateway.call(
"agent.wait",
{ runId: started.runId, timeoutMs: 30_000 },
{ timeoutMs: 35_000 },
)) as GatewayRunResult;
expect(terminal.status).toBe("ok");
const removed = await waitForWorktreeState({
id: activeRecord?.id ?? "",
predicate: (record) =>
record?.removedAt !== undefined && record.runEndCleanup?.outcome === "removed-lossless",
timeoutMs: 30_000,
});
expect(removed).toMatchObject({
id: activeRecord?.id,
snapshotRef: `refs/openclaw/snapshots/${activeRecord?.id}`,
removedAt: expect.any(Number),
runEndCleanup: {
outcome: "removed-lossless",
at: expect.any(Number),
},
});
await expect(fs.access(materializedPath)).rejects.toMatchObject({ code: "ENOENT" });
// The mock provider reaches terminal without calling the Workboard worker protocol.
// Close the first card at the operator boundary so its agent-global owner slot is free.
const completed = (await activeHarness.gateway.call("workboard.cards.complete", {
id: card.id,
summary: "Clean worktree lifecycle completed.",
})) as WorkboardCompleteResult;
expect(completed.card).toMatchObject({ id: card.id, status: "done" });
const dirtyBoardId = "qa-worktree-dirty";
const dirtyCard = await createCard({
boardId: dirtyBoardId,
repo,
title: "Dirty worktree retention",
});
const dirtyName = managedWorktreeName(dirtyCard.id);
const { materializedPath: dirtyPath, started: dirtyStarted } =
await dispatchCardAndWaitForWorktree({
boardId: dirtyBoardId,
cardId: dirtyCard.id,
name: dirtyName,
stateDir,
});
const dirtyFile = path.join(dirtyPath, "untracked-note.txt");
await fs.writeFile(dirtyFile, "retain this worktree\n");
const dirtyRecord = (await listWorktrees()).worktrees.find(
(record) => record.ownerKind === "workboard" && record.ownerId === dirtyCard.id,
);
expect(dirtyRecord).toMatchObject({
name: dirtyName,
path: dirtyPath,
ownerKind: "workboard",
ownerId: dirtyCard.id,
});
const dirtyTerminal = (await activeHarness.gateway.call(
"agent.wait",
{ runId: dirtyStarted.runId, timeoutMs: 30_000 },
{ timeoutMs: 35_000 },
)) as GatewayRunResult;
expect(dirtyTerminal.status).toBe("ok");
const retained = await waitForWorktreeState({
id: dirtyRecord?.id ?? "",
predicate: (record) => record?.runEndCleanup?.outcome === "retained-dirty",
timeoutMs: 30_000,
});
expect(retained).toMatchObject({
id: dirtyRecord?.id,
runEndCleanup: {
outcome: "retained-dirty",
at: expect.any(Number),
},
});
expect(retained?.removedAt).toBeUndefined();
await expect(fs.readFile(dirtyFile, "utf8")).resolves.toBe("retain this worktree\n");
},
);
});