Audit exact-bound owner-native lifecycle receipts (#126082)

* feat(audit): add owner-native lifecycle receipts

* fix(audit): store lifecycle bindings additively

* test(cron): isolate execution binding fixtures

* fix(audit): reject stale lifecycle owners

* fix(acp): gate parent flow binding on retained task

* fix(audit): reject pruned owner cursors
This commit is contained in:
Josh Avant
2026-08-24 17:49:25 -07:00
committed by GitHub
parent 05a2a593f7
commit 2e9c8f8d3f
58 changed files with 3802 additions and 318 deletions
+8
View File
@@ -559,6 +559,14 @@ Model override note:
Gateway can expose HTTP webhook endpoints for external triggers. Enable in config:
A configured mapping `id` is retained only as bounded ingress-source
attribution when that mapping reaches agent admission. It is not an
authenticated service principal or invoker. Direct `/hooks/agent` and requests
authenticated only by the shared hook token stay unattributed unless another
authoritative principal source exists. If a transform returns `null`, the
request keeps its visible HTTP 204 outcome and stops before creating a run,
task, execution identity, or audit receipt.
```json5
{
hooks: {
+10 -2
View File
@@ -405,8 +405,16 @@ A sweeper runs every **60 seconds** (first pass about 5 seconds after gateway st
A task may reference a `childSessionKey` (where work runs) and a `requesterSessionKey` (who started it). Its `agentId` identifies the agent executing the work, while the requester and owner fields preserve launch and control context. Sessions are conversation context; tasks are activity tracking on top of that.
</Accordion>
<Accordion title="Tasks and agent runs">
A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually.
</Accordion>
A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually.
When execution identity collection is enabled, OpenClaw also binds the exact
admitted `contextId` and `executionId` to Gateway CLI, ACP, and automation task
rows and their mirrored flow rows. This is inspection provenance only: `runId`
remains correlation, task/flow status remains authoritative, and a missing or
mismatched binding never changes execution or settlement. `openclaw audit
--execution <id> --explain` adapts the existing rows without copying task or
flow content into the generic decision-fact table.
</Accordion>
</AccordionGroup>
## Related
+13 -2
View File
@@ -115,7 +115,8 @@ view renders these sections:
3. **Lineage**: parent context or an explicit absent, unknown, or unsupported
state.
4. **Decisions**: bounded run-admission and authoritative action-decision
receipts, including terminal operator approvals.
receipts, including terminal operator approvals and exact-bound cron, task,
and task-flow lifecycle rows.
5. **Missing evidence** and **Next steps**.
Every field includes `present`, `absent`, `unknown`, or `unsupported`; the CLI
@@ -125,6 +126,15 @@ ingress, an absent invoker, and
`unattributed` coverage. Its admission receipt says `not-applicable` because no
identity-aware policy or grant evaluation was proven.
Lifecycle rows from `cron_run_receipts`, `task_runs`, and `flow_runs` appear as
owner-native, attribution-only receipts when their keyed lifecycle metadata
carries the exact inspected context and execution ids. They contain status and
bounded record references, not prompts, task goals, hook payloads, paths, or raw
errors. Their decision is `not-applicable` because lifecycle attribution does
not prove authorization.
Treat every decision cursor as opaque: numeric and `a:`, `m:`, and `g:` values
remain compatible, while cron/task/flow pages may return `c:`, `t:`, or `f:`.
For Gateway runs, a resolved authenticated profile can make the invoker
`present` and coverage `attribution-only`. Paired devices and shared credentials
do not establish a person: without a durable profile the invoker stays absent,
@@ -340,7 +350,8 @@ and no identity context; its required `decisionDisplays` array is empty until
the caller selects an execution id.
For one selected context, receipt paging starts with admission, then reads
owner-native terminal approvals, merges outbound progress and terminal records,
and finally reads generic facts for boundaries without a native durable record.
then reads generic facts and the cron, task, and flow lifecycle owners. The
complete order is admission, approval, message, generic, cron, task, then flow.
The merge is deterministic across restart and rejects a cursor whose exact
owner row has expired. Approval and message selectors use the opaque
`approval-decision:` and `message-decision:` namespaces minted from the same
+25
View File
@@ -43,6 +43,16 @@ platform-send start use a lazy progress companion, while terminal message rows
remain in the activity ledger. Run inspection merges both sources directly;
neither is copied into the generic decision-fact table.
Scheduled runs, background tasks, and task flows are owner-native sources too.
After exact run admission, a lazy lifecycle metadata table binds the admitted
context and execution ids to the canonical `cron_run_receipts`, `task_runs`, or
`flow_runs` row. Inspection joins that metadata to the owner row directly and
preserves its status, including skipped, failed, timed-out, cancelled, blocked,
and lost outcomes. A `runId` alone never joins one of these rows to an
execution. Legacy, missing, deleted, corrupt, or mismatched bindings remain
unknown or absent; they never change task behavior and are never copied into
`execution_decision_facts`.
## Run identity inspection
Execution identity recording is off by default, including on fresh installs
@@ -160,6 +170,15 @@ this boundary. A run becomes
`attribution-only` only when an authoritative ingress supplies an invoker fact.
Neither state means that identity affected an allow or deny decision.
Configured webhook mapping ids identify only the matched ingress source. They
do not authenticate a person, service, or invoker. Shared hook authentication
and direct `/hooks/agent` requests therefore remain unattributed unless another
authoritative principal producer exists. A mapping transform that suppresses a
request before admission returns its normal HTTP response but creates no run,
execution identity, task, or decision receipt. Restart recovery records system
attribution only after the current durable recovery owner admits the exact
attempt.
Authenticated Gateway attach records immutable audit facts once. Session
creation separately reads the live canonical durable profile id so a profile
link performed after attach cannot orphan session ownership. Ordinary session
@@ -200,6 +219,12 @@ message-policy, or turn-capability denial that changed the result is
`enforced`. Portable actions and early suppressions without a durable owner
record use the generic fact owner on the same audit-writer FIFO.
Cron, task, and flow lifecycle receipts are `attribution-only` and have a
`not-applicable` decision outcome. They report what the authoritative lifecycle
owner retained; they do not claim an authorization decision. Their cursors are
opaque and source-specific. Existing numeric cursors and `a:`, `m:`, and `g:`
cursors remain accepted; newer owner stages use `c:`, `t:`, and `f:`.
When the same `runId` has a retained terminal row in `operator_approvals`, the
inspector also reads its owner-local `operator_approval_execution_identities`
binding. Only an exact context, execution, and run tuple projects the approval
@@ -116,6 +116,14 @@ describe("audit run inspection protocol", () => {
provenance: { state: "verified", producer: "receipt-owner" },
}),
).toBe(false);
for (const producer of ["cron-lifecycle", "task-lifecycle", "flow-lifecycle"]) {
expect(
validate.Check({
...display,
provenance: { state: "verified", producer },
}),
).toBe(true);
}
});
it("exports selector and discovery-pagination invariants for generated clients", () => {
@@ -209,6 +209,9 @@ const DecisionReceiptDisplayProvenanceV1Schema = Type.Union([
Type.Literal("run-admission"),
Type.Literal("operator-approval"),
Type.Literal("message-delivery"),
Type.Literal("cron-lifecycle"),
Type.Literal("task-lifecycle"),
Type.Literal("flow-lifecycle"),
]),
}),
closedObject({ state: Type.Literal("unverified") }),
@@ -0,0 +1,36 @@
title: Autonomous task lifecycle receipts
scenario:
id: autonomous-task-lifecycle-receipts
surface: gateway
coverage:
primary:
- gateway.identity-and-presence-apis
- automation.openclaw-tasks-flow
objective: Verify exact-bound cron, task, and flow owner lifecycle rows are explained without generic decision-fact copies.
successCriteria:
- A mapped hook transform suppression returns HTTP 204 before admission and creates no execution identity.
- A real forced scheduled agent run binds one exact context/execution to its cron receipt and task row.
- A real Gateway agent action binds one exact context/execution to its CLI task row; focused owner tests cover bound flow rows.
- JSON and human audit inspection expose only bounded attribution-only lifecycle displays with closed producers, accept the cron cursor prefix, and omit task/prompt content.
- A replacement Gateway returns byte-equivalent JSON and the generic decision-fact table remains empty.
docsRefs:
- docs/gateway/audit.md
- docs/cli/audit.md
- docs/automation/cron-jobs.md
- docs/automation/tasks.md
- docs/concepts/qa-e2e-automation.md
codeRefs:
- src/audit/execution-decision-receipts.ts
- src/cron/store/run-receipt-store.ts
- src/tasks/task-registry.store.sqlite.ts
- src/tasks/task-flow-registry.store.sqlite.ts
- src/gateway/hooks-mapping.ts
- test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts
execution:
kind: script
path: test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts
summary: Starts an ephemeral Gateway and mock provider, proves pre-admission suppression, runs cron and CLI task lifecycles, replaces the Gateway, and compares inspection output.
args:
- --artifact-base
- ${outputDir}
@@ -1,9 +1,25 @@
/** Regression coverage for ACP background-task summary truncation boundaries. */
import { describe, expect, it } from "vitest";
import { afterEach, describe, expect, it } from "vitest";
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
import type { OpenClawConfig } from "../../config/types.openclaw.js";
import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js";
import {
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../../state/openclaw-state-db.js";
import { findTaskByRunId } from "../../tasks/task-registry.js";
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
import {
resetTaskRegistryForTests,
resetTaskFlowRegistryForTests,
} from "../../tasks/task-runtime.test-helpers.js";
import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js";
import { AcpRuntimeError } from "../runtime/errors.js";
import {
appendBackgroundTaskProgressSummary,
bindBackgroundTaskExecution,
createBackgroundTaskRecord,
resolveBackgroundTaskContext,
resolveBackgroundTaskFailureStatus,
} from "./manager.background-task.js";
@@ -13,6 +29,12 @@ import type { AcpSessionManagerDeps } from "./manager.types.js";
// U+1F99E (🦞) is a surrogate pair in UTF-16; a raw .slice() boundary can split it.
const LOBSTER = "🦞";
afterEach(() => {
closeOpenClawStateDatabaseForTest();
resetTaskRegistryForTests({ persist: false });
resetTaskFlowRegistryForTests({ persist: false });
});
const HIGH_SURROGATE_WITHOUT_LOW = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])/;
function fakeDeps(): AcpSessionManagerDeps {
@@ -85,3 +107,128 @@ describe("resolveBackgroundTaskFailureStatus", () => {
).toBe("failed");
});
});
describe("ACP background task execution binding", () => {
it("binds the exact admitted execution only at prompt submission", async () => {
await withOpenClawTestState(
{ layout: "state-only", prefix: "openclaw-acp-execution-binding-" },
async () => {
resetTaskRegistryForTests();
resetTaskFlowRegistryForTests();
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", {
contextId: "context-acp",
executionId: "execution-acp",
}),
};
const record = createBackgroundTaskRecord(
{
requesterSessionKey: "agent:main:main",
childSessionKey: "agent:qa:child",
runId: "run-acp",
task: "private",
},
100,
);
const task = findTaskByRunId("run-acp");
if (!record || !task?.parentFlowId) {
throw new Error("expected ACP task and owner flow");
}
const db = openOpenClawStateDatabase().db;
expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false);
bindBackgroundTaskExecution(record, admitted);
expect(
db
.prepare(
`SELECT owner_kind, owner_id, context_id, execution_id
FROM execution_owner_lifecycle_bindings
ORDER BY owner_kind`,
)
.all(),
).toEqual([
{
owner_kind: "flow",
owner_id: task.parentFlowId,
context_id: "context-acp",
execution_id: "execution-acp",
},
{
owner_kind: "task",
owner_id: task.taskId,
context_id: "context-acp",
execution_id: "execution-acp",
},
]);
},
);
});
it.each(["missing", "mismatched"] as const)(
"does not bind a live parent flow when the task owner is %s",
async (taskOwnerState) => {
await withOpenClawTestState(
{ layout: "state-only", prefix: `openclaw-acp-${taskOwnerState}-task-binding-` },
async () => {
resetTaskRegistryForTests();
resetTaskFlowRegistryForTests();
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", {
contextId: "context-acp",
executionId: "execution-acp",
}),
};
const record = createBackgroundTaskRecord(
{
requesterSessionKey: "agent:main:main",
childSessionKey: "agent:qa:child",
runId: "run-acp",
task: "private",
},
100,
);
const task = findTaskByRunId("run-acp");
if (!record || !task?.parentFlowId) {
throw new Error("expected ACP task and owner flow");
}
const db = openOpenClawStateDatabase().db;
if (taskOwnerState === "missing") {
db.prepare("DELETE FROM task_runs WHERE task_id = ?").run(task.taskId);
} else {
expect(
bindTaskRunExecution({
taskId: task.taskId,
admitted: {
operationalRunInstance: { instanceId: "instance-other", runId: "run-other" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-other", {
contextId: "context-other",
executionId: "execution-other",
}),
},
}),
).toBe("bound");
}
bindBackgroundTaskExecution(record, admitted);
if (taskOwnerState === "missing") {
expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false);
} else {
expect(
db
.prepare(
`SELECT context_id, execution_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'flow' AND owner_id = ?`,
)
.get(task.parentFlowId),
).toBeUndefined();
}
},
);
},
);
});
@@ -1,5 +1,7 @@
/** Mirrors child ACP turns into detached-task status for requester-facing progress. */
import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice";
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import { isRetainedExecutionOwnerBinding } from "../../audit/execution-owner-binding.js";
import type { OpenClawConfig } from "../../config/types.openclaw.js";
import { logVerbose } from "../../globals.js";
import {
@@ -9,6 +11,8 @@ import {
startTaskRunByRunId,
} from "../../tasks/detached-task-runtime.js";
import { resolveRequiredCompletionTerminalResult } from "../../tasks/task-completion-contract.js";
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
import {
deliveryContextFromSession,
type DeliveryContext,
@@ -31,6 +35,11 @@ type BackgroundTaskContext = {
task: string;
};
type BackgroundTaskRecord = {
taskId: string;
parentFlowId?: string;
};
/** Produces the bounded task label shown for a child ACP background run. */
function summarizeBackgroundTaskText(text: string): string {
const normalized = normalizeText(text) ?? "ACP background task";
@@ -133,7 +142,7 @@ export function resolveBackgroundTaskContext(params: {
export function createBackgroundTaskRecord(
context: BackgroundTaskContext,
startedAt: number,
): void {
): BackgroundTaskRecord | undefined {
try {
const task = createRunningTaskRun({
runtime: "acp",
@@ -151,11 +160,37 @@ export function createBackgroundTaskRecord(
logVerbose(
`acp-manager: failed creating background task for ${context.runId}: persist_failed`,
);
return undefined;
}
return {
taskId: task.taskId,
...(task.parentFlowId ? { parentFlowId: task.parentFlowId } : {}),
};
} catch (error) {
logVerbose(
`acp-manager: failed creating background task for ${context.runId}: ${String(error)}`,
);
return undefined;
}
}
/** Links ACP owner rows only when the runtime reaches its prompt-submitted boundary. */
export function bindBackgroundTaskExecution(
record: BackgroundTaskRecord,
admitted: AdmittedRunContext,
): void {
try {
const taskResult = bindTaskRunExecution({ admitted, taskId: record.taskId });
const flowResult = record.parentFlowId
? isRetainedExecutionOwnerBinding(taskResult)
? bindTaskFlowExecution({ admitted, flowId: record.parentFlowId })
: taskResult
: undefined;
if ([taskResult, flowResult].some((result) => result === "mismatch" || result === "missing")) {
logVerbose("acp-manager: exact task execution binding was not retained");
}
} catch (error) {
logVerbose(`acp-manager: failed binding background task execution: ${String(error)}`);
}
}
+9 -3
View File
@@ -16,6 +16,7 @@ import {
} from "./manager.backend-failover.js";
import {
appendBackgroundTaskProgressSummary,
bindBackgroundTaskExecution,
createBackgroundTaskRecord,
markBackgroundTaskRunning,
markBackgroundTaskTerminal,
@@ -87,9 +88,10 @@ export async function runManagerTurn(params: {
text: input.text,
})
: null;
if (taskContext) {
createBackgroundTaskRecord(taskContext, turnStartedAt);
}
const taskRecord = taskContext
? createBackgroundTaskRecord(taskContext, turnStartedAt)
: undefined;
let taskExecutionBound = false;
let taskProgressSummary = "";
const initialResolution = params.resolveSession({
cfg: input.cfg,
@@ -267,6 +269,10 @@ export async function runManagerTurn(params: {
onBeforePrompt: input.onBeforePrompt,
onPromptStarted: async ({ authoritative }) => {
promptStarted = authoritative;
if (authoritative && taskRecord && !taskExecutionBound) {
taskExecutionBound = true;
bindBackgroundTaskExecution(taskRecord, input.admittedRunContext);
}
try {
await input.onLifecycle?.({
type: "prompt_submitted",
@@ -0,0 +1,32 @@
import { describe, expect, it } from "vitest";
import {
attachAgentCommandRecoveryAdmissionFacts,
getAgentCommandAdmissionFacts,
} from "./agent-command-admission-facts.js";
describe("agent command admission facts", () => {
it("records restart recovery as owner-bound system attribution", () => {
const runContext = {};
attachAgentCommandRecoveryAdmissionFacts(runContext);
expect(getAgentCommandAdmissionFacts(runContext)).toEqual({
ingress: {
kind: "recovery",
boundary: "gateway.main-session-recovery",
state: "present",
},
invoker: {
state: "present",
kind: "system",
rawPrincipalRef: "openclaw.main-session-recovery",
},
assurance: [
{
kind: "runtime-binding",
rawEvidenceRef: "gateway.main-session-recovery-owner",
strength: "boundary-verified",
},
],
});
expect(getAgentCommandAdmissionFacts({})).toBeUndefined();
});
});
@@ -18,3 +18,26 @@ export function getAgentCommandAdmissionFacts(
): AgentCommandAdmissionFacts | undefined {
return factsByIngress.get(ingress);
}
/** Records the exact system attribution only after the recovery owner admits the attempt. */
export function attachAgentCommandRecoveryAdmissionFacts(ingress: object): void {
attachAgentCommandAdmissionFacts(ingress, {
ingress: {
kind: "recovery",
boundary: "gateway.main-session-recovery",
state: "present",
},
invoker: {
state: "present",
kind: "system",
rawPrincipalRef: "openclaw.main-session-recovery",
},
assurance: [
{
kind: "runtime-binding",
rawEvidenceRef: "gateway.main-session-recovery-owner",
strength: "boundary-verified",
},
],
});
}
@@ -43,6 +43,36 @@ describe("sanitizePublicAgentCommandIngressOpts", () => {
});
describe("Gateway agent command execution identity", () => {
it("runs owner binding only after the awaited admission callback settles", async () => {
const events: string[] = [];
const prepared = prepareAgentCommandExecutionIdentity({
opts: {
message: "bind after admission",
onAdmittedRunContext: async () => {
await Promise.resolve();
events.push("admitted");
},
onPostAdmittedRunContext: () => {
events.push("owner-bound");
},
},
prepared: {
cfg: { logging: { audit: { enabled: true, executionIdentity: true } } },
runId: "run-post-admission",
sessionAgentId: "main",
sessionId: "session-post-admission",
},
ingress: { kind: "api", boundary: "agent-command.from-ingress", state: "unknown" },
lifecycleGeneration: "generation-1",
});
const admitted = await prepared.admit("embedded");
await prepared.admit("embedded");
expect(admitted.executionIdentityToken).toBeDefined();
expect(events).toEqual(["admitted", "owner-bound"]);
});
it("preserves trusted spawn facts across internal option preparation", () => {
const facts = {
ingress: {
@@ -3,6 +3,7 @@ import type {
ExecutionIdentityAdmissionToken,
} from "../audit/execution-identity-admission.js";
import { executionIdentitySpawnAdmission } from "../audit/execution-identity-spawn-admission.js";
import { withPostAdmissionExecutionOwnerBinding } from "../audit/execution-owner-binding.js";
import type { OpenClawConfig } from "../config/types.openclaw.js";
import { formatErrorMessage } from "../infra/errors.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
@@ -170,9 +171,12 @@ export function prepareAgentCommandExecutionIdentity(params: {
},
};
const spawnFacts = readAgentCommandExecutionIdentitySpawnFacts(opts);
return spawnFacts
const preparedAdmission = spawnFacts
? prepareAgentCommandRunAdmissionWithSpawnFacts(admissionParams, spawnFacts)
: executionIdentity.prepare(admissionParams);
return opts.onPostAdmittedRunContext
? withPostAdmissionExecutionOwnerBinding(preparedAdmission, opts.onPostAdmittedRunContext)
: preparedAdmission;
}
export function sanitizePublicAgentCommandIngressOpts(
@@ -187,6 +191,7 @@ export function sanitizePublicAgentCommandIngressOpts(
operationalRunInstance: undefined,
cronCreatorAuthorityCapability: undefined,
onAdmittedRunContext: undefined,
onPostAdmittedRunContext: undefined,
});
}
+6
View File
@@ -209,6 +209,10 @@ export type AgentCommandOpts = {
onAdmittedRunContext?: (
context: import("../admitted-run-context.js").AdmittedRunContext,
) => void | Promise<void>;
/** Private owner binding hook invoked only after exact admission has resolved. */
onPostAdmittedRunContext?: (
context: import("../admitted-run-context.js").AdmittedRunContext,
) => void;
/** Called when the actual run model is selected, including fallback retries. */
onActiveModelSelected?: (ctx: { provider: string; model: string }) => void | Promise<void>;
/** Called when every candidate in the run's model fallback chain failed. */
@@ -241,6 +245,7 @@ export type AgentCommandIngressOpts = Omit<
| "operationalRunInstance"
| "cronCreatorAuthorityCapability"
| "onAdmittedRunContext"
| "onPostAdmittedRunContext"
> & {
/** Trusted sender identity bit for command/channel-action auth; defaults false for ingress. */
senderIsOwner?: boolean;
@@ -259,4 +264,5 @@ export type AgentCommandGatewayIngressOpts = AgentCommandIngressOpts &
| "operationalRunInstance"
| "cronCreatorAuthorityCapability"
| "onAdmittedRunContext"
| "onPostAdmittedRunContext"
>;
@@ -27,6 +27,7 @@ import {
type ExecutionIdentityAdmissionEnvelope,
} from "./execution-identity-admission.js";
import { processExecutionIdentityAdmissionWork } from "./execution-identity-context.js";
import { bindExecutionOwnerLifecycleMetadata } from "./execution-owner-lifecycle-binding-store.js";
import {
configureMessageActionDecisionSink,
recordMessageActionDecision,
@@ -256,6 +257,79 @@ describe("execution decision facts", () => {
expect(receiptSearch).not.toContain(encodeURIComponent(messageReceipt?.receiptId ?? ""));
});
it("projects exact-bound cron, task, and flow owner rows without generic facts", () => {
const database = databaseOptions();
const context = seedExecutionContext(database);
const db = openOpenClawStateDatabase(database).db;
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
"cron-receipt-1",
"default",
"job-1",
"revision-1",
"main",
context.runId,
"ok",
1,
60,
70,
);
db.prepare(
`INSERT INTO task_runs (
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
notify_policy, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
"task-1",
"cron",
"owner-1",
"session",
"private task text",
"succeeded",
"not-requested",
"never",
61,
);
db.prepare(
`INSERT INTO flow_runs (
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
).run("flow-1", "owner-1", "succeeded", "never", "private flow goal", 62, 70);
for (const [ownerKind, ownerId] of [
["cron", "cron-receipt-1"],
["task", "task-1"],
["flow", "flow-1"],
] as const) {
expect(
bindExecutionOwnerLifecycleMetadata({
db,
ownerKind,
ownerId,
binding: { contextId: context.contextId, executionId: context.executionId },
}),
).toBe("bound");
}
const result = presentExecutionDecisionReceipts({
context,
decisionLimit: 10,
options: { ...database, now: 100 },
});
expect(result.decisions.map((item) => item.source.owner)).toEqual([
"agent-command",
"cron_run_receipts",
"task_runs",
"flow_runs",
]);
expect(JSON.stringify(result.decisions)).not.toContain("private task text");
expect(JSON.stringify(result.decisions)).not.toContain("private flow goal");
expect(tableExists(db, "execution_decision_facts")).toBe(false);
});
it("does not assign run-only delivery evidence to either exact execution sharing a run id", () => {
const database = databaseOptions();
const first = seedExecutionContext(database, {
+219 -135
View File
@@ -15,6 +15,12 @@ import {
pageExecutionDecisionFactsForContext,
summarizeExecutionDecisionFactsForContext,
} from "./execution-decision-facts.js";
import {
pageOwnerLifecycleReceipts,
summarizeOwnerLifecycleReceipts,
type OwnerLifecycleCursor,
type OwnerLifecycleStage,
} from "./execution-owner-lifecycle-receipts.js";
import {
pageMessageDeliveryReceiptsForRun,
summarizeMessageDeliveryReceiptsForRun,
@@ -34,9 +40,10 @@ type ProvenancedDecisionReceipt = {
const MAX_AGGREGATE_MISSING_EVIDENCE = 16;
const MISSING_EVIDENCE_TRUNCATED = "decision.missing_evidence_truncated";
type DecisionStage = "approval" | "message" | "generic" | OwnerLifecycleStage;
type DecisionCursor =
| {
stage: "approval" | "message" | "generic";
stage: DecisionStage;
after?: { occurredAt: number; rowId: number };
}
| {
@@ -58,7 +65,7 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi
if (offset !== null && offset !== undefined) {
return { offset };
}
const match = /^([amg]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value);
const match = /^([amgctf]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value);
if (!match) {
return null;
}
@@ -67,8 +74,20 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi
if (!Number.isSafeInteger(occurredAt) || !Number.isSafeInteger(rowId)) {
return null;
}
const stage =
match[1] === "a"
? "approval"
: match[1] === "m"
? "message"
: match[1] === "g"
? "generic"
: match[1] === "c"
? "cron"
: match[1] === "t"
? "task"
: "flow";
return {
stage: match[1] === "a" ? "approval" : match[1] === "m" ? "message" : "generic",
stage,
...(occurredAt === 0 && rowId === 0 ? {} : { after: { occurredAt, rowId } }),
};
}
@@ -78,10 +97,17 @@ export function isExecutionDecisionCursor(value: string): boolean {
}
function formatDecisionCursor(
stage: "approval" | "message" | "generic",
stage: DecisionStage,
cursor?: { occurredAt: number; rowId: number },
): string {
const prefix = stage === "approval" ? "a" : stage === "message" ? "m" : "g";
const prefix = {
approval: "a",
message: "m",
generic: "g",
cron: "c",
task: "t",
flow: "f",
}[stage];
return `${prefix}:${cursor?.occurredAt ?? 0}:${cursor?.rowId ?? 0}`;
}
@@ -200,10 +226,8 @@ export function presentExecutionDecisionReceipts(params: {
if (cursor === null) {
throw new ExecutionDecisionCursorError();
}
const limit = params.decisionLimit ?? 50;
const decisionLimit = params.decisionLimit ?? 50;
const now = params.options.now ?? Date.now();
// Numeric cursors are the shipped aggregate offset. Resolve its owner span
// once, then let the canonical bounded owner pagers emit opaque successors.
const opaqueCursor = cursor && "stage" in cursor ? cursor : undefined;
const legacyOffset = cursor && "offset" in cursor ? cursor.offset - 1 : undefined;
const approvalSummary = summarizeOperatorApprovalReceiptsForRun({
@@ -225,11 +249,132 @@ export function presentExecutionDecisionReceipts(params: {
context: params.context,
options: { ...params.options, now },
});
const cronSummary = summarizeOwnerLifecycleReceipts({
stage: "cron",
context: params.context,
options: params.options,
});
const taskSummary = summarizeOwnerLifecycleReceipts({
stage: "task",
context: params.context,
options: params.options,
});
const flowSummary = summarizeOwnerLifecycleReceipts({
stage: "flow",
context: params.context,
options: params.options,
});
const stages: Array<{
stage: DecisionStage;
count: number;
page: (params: { after?: OwnerLifecycleCursor; offset?: number; limit: number }) => {
entries: ProvenancedDecisionReceipt[];
nextCursor?: OwnerLifecycleCursor;
};
}> = [
{
stage: "approval",
count: approvalSummary.count,
page: ({ after, offset, limit }) => {
const page = pageOperatorApprovalReceiptsForRun({
context: {
contextId: params.context.contextId,
executionId: params.context.executionId,
runId: params.context.runId,
},
after,
offset,
limit,
nowMs: now,
databaseOptions: params.options,
});
return {
entries: page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "verified", producer: "operator-approval" },
selectorId: entry.selectorId,
})),
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
};
},
},
{
stage: "message",
count: messageSummary.count,
page: ({ after, offset, limit }) => {
const page = pageMessageDeliveryReceiptsForRun({
context: params.context,
after,
offset,
limit,
options: { ...params.options, now },
});
return {
entries: page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "verified", producer: "message-delivery" },
selectorId: entry.selectorId,
})),
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
};
},
},
{
stage: "generic",
count: genericSummary.count,
page: ({ after, offset, limit }) => {
const page = pageExecutionDecisionFactsForContext({
context: params.context,
after,
offset,
limit,
now,
database: params.options,
});
return {
entries: page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "unverified" },
selectorId: entry.selectorId,
})),
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
};
},
},
...(["cron", "task", "flow"] as const).map((stage) => ({
stage,
count: { cron: cronSummary, task: taskSummary, flow: flowSummary }[stage].count,
page: ({
after,
offset,
limit,
}: {
after?: OwnerLifecycleCursor;
offset?: number;
limit: number;
}) => {
const page = pageOwnerLifecycleReceipts({
stage,
context: params.context,
after,
offset,
limit,
options: params.options,
});
return {
entries: page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "verified" as const, producer: entry.displayProducer },
selectorId: entry.selectorId,
})),
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
};
},
})),
];
const decisions: ProvenancedDecisionReceipt[] = [];
let remainingLimit = limit;
let remainingLimit = decisionLimit;
let nextDecisionCursor: string | undefined;
const approvalOffset =
legacyOffset !== undefined && legacyOffset < approvalSummary.count ? legacyOffset : undefined;
if (cursor === undefined && remainingLimit > 0) {
decisions.push({
@@ -238,32 +383,44 @@ export function presentExecutionDecisionReceipts(params: {
selectorId: `${params.context.contextId}:admission`,
});
remainingLimit -= 1;
if (
remainingLimit === 0 &&
(approvalSummary.count > 0 || messageSummary.count > 0 || genericSummary.count > 0)
) {
if (remainingLimit === 0 && stages.some((stage) => stage.count > 0)) {
// The shipped first successor remains the approval cursor even when that owner is empty.
nextDecisionCursor = formatDecisionCursor("approval");
}
}
if (
remainingLimit > 0 &&
opaqueCursor?.stage !== "message" &&
opaqueCursor?.stage !== "generic" &&
(legacyOffset === undefined || approvalOffset !== undefined)
) {
let startStage = 0;
let firstStageOffset: number | undefined;
if (opaqueCursor) {
startStage = stages.findIndex((stage) => stage.stage === opaqueCursor.stage);
} else if (legacyOffset !== undefined) {
let preceding = 0;
startStage = stages.findIndex((stage) => {
if (legacyOffset < preceding + stage.count) {
firstStageOffset = legacyOffset - preceding;
return true;
}
preceding += stage.count;
return false;
});
if (startStage < 0) {
startStage = stages.length;
}
}
for (let index = startStage; index < stages.length && remainingLimit > 0; index += 1) {
const stage = stages[index];
if (!stage) {
continue;
}
let page;
try {
page = pageOperatorApprovalReceiptsForRun({
context: {
contextId: params.context.contextId,
executionId: params.context.executionId,
runId: params.context.runId,
},
after: opaqueCursor?.stage === "approval" ? opaqueCursor.after : undefined,
offset: approvalOffset,
page = stage.page({
...(index === startStage && opaqueCursor?.stage === stage.stage
? { after: opaqueCursor.after }
: {}),
...(index === startStage && firstStageOffset !== undefined
? { offset: firstStageOffset }
: {}),
limit: remainingLimit,
nowMs: now,
databaseOptions: params.options,
});
} catch (error) {
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
@@ -273,125 +430,52 @@ export function presentExecutionDecisionReceipts(params: {
}
throw error;
}
decisions.push(
...page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "verified" as const, producer: "operator-approval" as const },
selectorId: entry.selectorId,
})),
);
decisions.push(...page.entries);
remainingLimit -= page.entries.length;
if (page.nextCursor) {
nextDecisionCursor = formatDecisionCursor("approval", page.nextCursor);
} else if (remainingLimit === 0 && messageSummary.count > 0) {
nextDecisionCursor = formatDecisionCursor("message");
} else if (remainingLimit === 0 && genericSummary.count > 0) {
nextDecisionCursor = formatDecisionCursor("generic");
nextDecisionCursor = formatDecisionCursor(stage.stage, page.nextCursor);
break;
}
if (remainingLimit === 0) {
const successor = stages.slice(index + 1).find((candidate) => candidate.count > 0);
nextDecisionCursor = successor ? formatDecisionCursor(successor.stage) : undefined;
}
}
const messageOffset =
legacyOffset === undefined
? undefined
: legacyOffset >= approvalSummary.count &&
legacyOffset < approvalSummary.count + messageSummary.count
? legacyOffset - approvalSummary.count
: undefined;
if (
remainingLimit > 0 &&
nextDecisionCursor?.startsWith("a:") !== true &&
opaqueCursor?.stage !== "generic" &&
(legacyOffset === undefined || messageOffset !== undefined || approvalOffset !== undefined)
) {
let page;
try {
page = pageMessageDeliveryReceiptsForRun({
context: params.context,
after: opaqueCursor?.stage === "message" ? opaqueCursor.after : undefined,
offset: messageOffset,
limit: remainingLimit,
options: { ...params.options, now },
});
} catch (error) {
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
throw new ExecutionDecisionCursorError(
"decision cursor is no longer retained; restart inspection without --cursor",
);
}
throw error;
}
decisions.push(
...page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "verified" as const, producer: "message-delivery" as const },
selectorId: entry.selectorId,
})),
);
remainingLimit -= page.entries.length;
if (page.nextCursor) {
nextDecisionCursor = formatDecisionCursor("message", page.nextCursor);
} else if (remainingLimit === 0 && genericSummary.count > 0) {
nextDecisionCursor = formatDecisionCursor("generic");
}
}
const genericOffset =
legacyOffset === undefined
? undefined
: Math.max(0, legacyOffset - approvalSummary.count - messageSummary.count);
if (
remainingLimit > 0 &&
nextDecisionCursor?.startsWith("a:") !== true &&
nextDecisionCursor?.startsWith("m:") !== true
) {
let page;
try {
page = pageExecutionDecisionFactsForContext({
context: params.context,
after: opaqueCursor?.stage === "generic" ? opaqueCursor.after : undefined,
offset: genericOffset,
limit: remainingLimit,
now,
database: params.options,
});
} catch (error) {
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
throw new ExecutionDecisionCursorError(
"decision cursor is no longer retained; restart inspection without --cursor",
);
}
throw error;
}
decisions.push(
...page.entries.map((entry) => ({
receipt: entry.receipt,
provenance: { state: "unverified" as const },
selectorId: entry.selectorId,
})),
);
if (page.nextCursor) {
nextDecisionCursor = formatDecisionCursor("generic", page.nextCursor);
} else {
nextDecisionCursor = undefined;
}
}
const ownerCoverage = new Set([approvalSummary.coverageState, messageSummary.coverageState]);
const ownerCoverage = new Set<DecisionReceiptV1["enforcement"]["coverageState"]>(
[
approvalSummary.coverageState,
messageSummary.coverageState,
cronSummary.coverageState,
taskSummary.coverageState,
flowSummary.coverageState,
].filter(
(coverageState): coverageState is NonNullable<typeof coverageState> =>
coverageState !== undefined,
),
);
const hasUnverifiedGenericDecisions = genericSummary.count > 0;
const boundedEvidence = boundMissingEvidence([
...params.context.missingEvidence,
...approvalSummary.missingEvidence,
...messageSummary.missingEvidence,
...cronSummary.missingEvidence,
...taskSummary.missingEvidence,
...flowSummary.missingEvidence,
...(hasUnverifiedGenericDecisions ? ["decision.display_provenance"] : []),
]);
const coverageState = boundedEvidence.truncated
? "unknown"
: hasUnverifiedGenericDecisions
? "unknown"
: ownerCoverage.has("unknown")
? "unknown"
: ownerCoverage.has("enforced")
? "enforced"
: ownerCoverage.has("attribution-only")
? "attribution-only"
: params.context.coverageState;
: ownerCoverage.has("unsupported")
? "unsupported"
: ownerCoverage.has("unknown")
? "unknown"
: ownerCoverage.has("enforced")
? "enforced"
: ownerCoverage.has("attribution-only")
? "attribution-only"
: params.context.coverageState;
return {
schemaVersion: 1,
run: {
+96
View File
@@ -0,0 +1,96 @@
import type {
AdmittedRunContext,
PreparedAgentRunAdmission,
} from "../agents/admitted-run-context.js";
import { parseExecutionIdentityAdmissionToken } from "./execution-identity-admission.js";
export type ExecutionOwnerBindingResult =
| "disabled"
| "bound"
| "already-bound"
| "mismatch"
| "missing";
export function isRetainedExecutionOwnerBinding(
result: ExecutionOwnerBindingResult | undefined,
): result is "bound" | "already-bound" {
return result === "bound" || result === "already-bound";
}
type ExecutionOwnerBinding = Readonly<{
contextId: string;
executionId: string;
}>;
/** Extracts only an admitted exact identity; operational run correlation cannot bind owner rows. */
export function executionOwnerBindingFromAdmission(
admitted: AdmittedRunContext,
): ExecutionOwnerBinding | undefined {
if (!admitted.executionIdentityToken) {
return undefined;
}
const token = parseExecutionIdentityAdmissionToken(admitted.executionIdentityToken);
if (token.runId !== admitted.operationalRunInstance.runId) {
throw new Error("owner execution binding disagrees with the admitted run");
}
return { contextId: token.contextId, executionId: token.executionId };
}
export function classifyExecutionOwnerBinding(
current: { contextId: string | null; executionId: string | null },
binding: ExecutionOwnerBinding,
): Exclude<ExecutionOwnerBindingResult, "disabled" | "bound" | "missing"> | "unbound" {
if (current.contextId === null && current.executionId === null) {
return "unbound";
}
return current.contextId === binding.contextId && current.executionId === binding.executionId
? "already-bound"
: "mismatch";
}
/** Adds one exact owner write after admission resolves, never inside the admission callback. */
export function withPostAdmissionExecutionOwnerBinding(
prepared: PreparedAgentRunAdmission,
bind: (context: AdmittedRunContext) => void,
): PreparedAgentRunAdmission {
let bound = false;
return Object.freeze({
operationalRunInstance: prepared.operationalRunInstance,
admit: async (runtimeKind, runtimeInstanceId) => {
const admitted = await prepared.admit(runtimeKind, runtimeInstanceId);
if (!bound) {
bound = true;
bind(admitted);
}
return admitted;
},
close: prepared.close,
});
}
/** Requires both exact admission and actual execution start, in either runtime order. */
export function createExecutionStartedOwnerBinding(bind: (context: AdmittedRunContext) => void): {
onPostAdmission: (context: AdmittedRunContext) => void;
onExecutionStarted: () => void;
} {
let admitted: AdmittedRunContext | undefined;
let executionStarted = false;
let bound = false;
const bindIfReady = () => {
if (bound || !admitted || !executionStarted) {
return;
}
bound = true;
bind(admitted);
};
return {
onPostAdmission: (context) => {
admitted = context;
bindIfReady();
},
onExecutionStarted: () => {
executionStarted = true;
bindIfReady();
},
};
}
@@ -0,0 +1,155 @@
import type { DatabaseSync } from "node:sqlite";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../infra/kysely-sync.js";
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
import {
classifyExecutionOwnerBinding,
type ExecutionOwnerBindingResult,
} from "./execution-owner-binding.js";
export const EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE =
"execution_owner_lifecycle_bindings" as const;
type ExecutionOwnerLifecycleKind = "cron" | "task" | "flow";
type ExecutionOwnerLifecycleDatabase = Pick<
OpenClawStateDatabase,
"cron_run_receipts" | "execution_owner_lifecycle_bindings" | "flow_runs" | "task_runs"
>;
const SCHEMA_START = `CREATE TABLE IF NOT EXISTS ${EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE} (`;
const SCHEMA_END = ") STRICT;";
function lifecycleDb(db: DatabaseSync) {
return getNodeSqliteKysely<ExecutionOwnerLifecycleDatabase>(db);
}
/** Creates only the canonical additive metadata table at first admitted owner use. */
function ensureExecutionOwnerLifecycleBindingSchema(db: DatabaseSync): void {
if (tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) {
return;
}
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_START);
const end = start < 0 ? -1 : OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_END, start);
if (start < 0 || end < start) {
throw new Error("OpenClaw execution owner lifecycle binding schema marker is missing.");
}
// sqlite-allow-raw -- Canonical feature-local additive DDL only; metadata rows use Kysely.
db.exec(OPENCLAW_STATE_SCHEMA_SQL.slice(start, end + SCHEMA_END.length));
}
function classifyRetainedBinding(
current: { context_id: string; execution_id: string },
binding: { contextId: string; executionId: string },
): "already-bound" | "mismatch" {
const state = classifyExecutionOwnerBinding(
{ contextId: current.context_id, executionId: current.execution_id },
binding,
);
return state === "unbound" ? "mismatch" : state;
}
/** Stores one exact admission identity after its canonical owner row has been revalidated. */
export function bindExecutionOwnerLifecycleMetadata(params: {
db: DatabaseSync;
ownerKind: ExecutionOwnerLifecycleKind;
ownerId: string;
binding: { contextId: string; executionId: string };
}): Exclude<ExecutionOwnerBindingResult, "disabled" | "missing"> {
ensureExecutionOwnerLifecycleBindingSchema(params.db);
const database = lifecycleDb(params.db);
const current = executeSqliteQueryTakeFirstSync(
params.db,
database
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.select(["context_id", "execution_id"])
.where("owner_kind", "=", params.ownerKind)
.where("owner_id", "=", params.ownerId),
);
if (current) {
return classifyRetainedBinding(current, params.binding);
}
const inserted = executeSqliteQuerySync(
params.db,
database
.insertInto(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.values({
owner_kind: params.ownerKind,
owner_id: params.ownerId,
context_id: params.binding.contextId,
execution_id: params.binding.executionId,
})
.onConflict((conflict) => conflict.columns(["owner_kind", "owner_id"]).doNothing()),
);
if (Number(inserted.numAffectedRows ?? 0n) === 1) {
return "bound";
}
const raced = executeSqliteQueryTakeFirstSync(
params.db,
database
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.select(["context_id", "execution_id"])
.where("owner_kind", "=", params.ownerKind)
.where("owner_id", "=", params.ownerId),
);
return raced ? classifyRetainedBinding(raced, params.binding) : "mismatch";
}
/** Removes exact owner metadata without allocating the opt-in table. */
export function deleteExecutionOwnerLifecycleMetadata(params: {
db: DatabaseSync;
ownerKind: ExecutionOwnerLifecycleKind;
ownerIds: readonly string[];
}): void {
if (
params.ownerIds.length === 0 ||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
) {
return;
}
executeSqliteQuerySync(
params.db,
lifecycleDb(params.db)
.deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.where("owner_kind", "=", params.ownerKind)
.where("owner_id", "in", params.ownerIds),
);
}
/** Removes bindings whose canonical owner row was pruned in the same transaction. */
export function pruneOrphanedExecutionOwnerLifecycleMetadata(
db: DatabaseSync,
ownerKind: ExecutionOwnerLifecycleKind,
): void {
if (!tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) {
return;
}
const database = lifecycleDb(db);
const bindings = database
.deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.where("owner_kind", "=", ownerKind);
if (ownerKind === "cron") {
executeSqliteQuerySync(
db,
bindings.where(
"owner_id",
"not in",
database.selectFrom("cron_run_receipts").select("receipt_id"),
),
);
} else if (ownerKind === "task") {
executeSqliteQuerySync(
db,
bindings.where("owner_id", "not in", database.selectFrom("task_runs").select("task_id")),
);
} else {
executeSqliteQuerySync(
db,
bindings.where("owner_id", "not in", database.selectFrom("flow_runs").select("flow_id")),
);
}
}
@@ -0,0 +1,611 @@
import { rmSync } from "node:fs";
import path from "node:path";
import { DatabaseSync } from "node:sqlite";
import { afterEach, describe, expect, it } from "vitest";
import type { ExecutionIdentityContextV1 } from "../../packages/gateway-protocol/src/index.js";
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
import { bindCronRunReceiptExecution } from "../cron/store/run-receipt-store.js";
import { tableHasColumn, tableExists } from "../state/openclaw-state-db-schema-helpers.js";
import {
OPENCLAW_STATE_SCHEMA_VERSION,
closeOpenClawStateDatabaseForTest,
openOpenClawStateDatabase,
} from "../state/openclaw-state-db.js";
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
import { bindTaskFlowExecution } from "../tasks/task-flow-registry.store.sqlite.js";
import { bindTaskRunExecution } from "../tasks/task-registry.store.sqlite.js";
import { presentExecutionDecisionReceipts } from "./execution-decision-receipts.js";
import { createExecutionIdentityAdmissionToken } from "./execution-identity-admission.js";
import { pageOwnerLifecycleReceipts } from "./execution-owner-lifecycle-receipts.js";
afterEach(() => closeOpenClawStateDatabaseForTest());
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
function oldSchemaSql(): string {
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(
"CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (",
);
const endMarker = ") STRICT;";
const end = OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start);
if (start < 0 || end < start) {
throw new Error("owner lifecycle binding schema marker is missing");
}
return `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`;
}
function createOldOwnerDatabase() {
const pathname = path.join(tempDirs.make("owner-lifecycle-"), "openclaw.sqlite");
const oldReader = new DatabaseSync(pathname);
oldReader.exec(oldSchemaSql());
oldReader.exec(`PRAGMA user_version = ${OPENCLAW_STATE_SCHEMA_VERSION}`);
oldReader
.prepare(
`INSERT INTO schema_meta (
meta_key, role, schema_version, created_at, updated_at
) VALUES ('primary', 'global', ?, 1, 1)`,
)
.run(OPENCLAW_STATE_SCHEMA_VERSION);
oldReader
.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null);
oldReader
.prepare(
`INSERT INTO task_runs (
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
notify_policy, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run(
"task-1",
"cron",
"owner-1",
"system",
"private",
"running",
"not_applicable",
"silent",
61,
);
oldReader
.prepare(
`INSERT INTO flow_runs (
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
)
.run("flow-1", "owner-1", "running", "silent", "private", 62, 62);
oldReader.close();
return { path: pathname };
}
function admitted(contextId = "context-1", executionId = "execution-1"): AdmittedRunContext {
return {
operationalRunInstance: { instanceId: "instance-1", runId: "run-1" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-1", {
contextId,
executionId,
now: 50,
}),
};
}
function executionContext(contextId = "context-1"): ExecutionIdentityContextV1 {
return {
schemaVersion: 1,
contextId,
executionId: "execution-1",
runId: "run-1",
createdAt: 50,
trustDomain: { kind: "gateway-cell", domainRef: "host-1", state: "present" },
invoker: { state: "unknown" },
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
agentPrincipal: { kind: "agent", domainRef: "host-1", principalRef: "main" },
agentDefinition: { definitionRef: "main", state: "present" },
runtimeInstance: { runtimeRef: "runtime-1", kind: "embedded", state: "present" },
applicableGrants: [],
assurance: [],
coverageState: "unattributed",
missingEvidence: [],
};
}
const receiptHandle = {
receiptId: "cron-1",
storeKey: "default",
jobId: "job-1",
configRevision: "revision-1",
agentId: "main",
ownerPid: 1,
ownerStartTime: null,
startedAtMs: 60,
};
describe("owner-native execution lifecycle receipts", () => {
it("lazily binds exact owner rows while disabled collection allocates nothing", () => {
const options = createOldOwnerDatabase();
const current = openOpenClawStateDatabase(options).db;
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false);
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
expect(tableHasColumn(current, table, "context_id")).toBe(false);
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
}
const disabled: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-disabled", runId: "run-1" },
};
expect(
bindCronRunReceiptExecution({ admitted: disabled, handle: receiptHandle, options }),
).toBe("disabled");
expect(bindTaskRunExecution({ admitted: disabled, taskId: "task-1", options })).toBe(
"disabled",
);
expect(bindTaskFlowExecution({ admitted: disabled, flowId: "flow-1", options })).toBe(
"disabled",
);
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false);
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
expect(tableHasColumn(current, table, "context_id")).toBe(false);
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
}
expect(
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
).toBe("bound");
expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe("bound");
expect(bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options })).toBe(
"bound",
);
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(true);
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
expect(tableHasColumn(current, table, "context_id")).toBe(false);
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
}
expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe(
"already-bound",
);
expect(
bindTaskRunExecution({
admitted: admitted("context-1", "execution-other"),
taskId: "task-1",
options,
}),
).toBe("mismatch");
current.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run();
current.prepare("UPDATE task_runs SET status = 'succeeded'").run();
current.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run();
closeOpenClawStateDatabaseForTest();
const oldReader = new DatabaseSync(options.path);
expect(
oldReader.prepare("SELECT status FROM task_runs WHERE task_id = ?").get("task-1"),
).toEqual({
status: "succeeded",
});
oldReader.prepare("UPDATE task_runs SET status = ? WHERE task_id = ?").run("failed", "task-1");
oldReader.close();
const reopened = openOpenClawStateDatabase(options).db;
expect(reopened.prepare("PRAGMA user_version").get()).toEqual({
user_version: OPENCLAW_STATE_SCHEMA_VERSION,
});
expect(
reopened
.prepare(
`SELECT binding.context_id, binding.execution_id, task.status
FROM task_runs AS task
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
WHERE task.task_id = ?`,
)
.get("task-1"),
).toEqual({ context_id: "context-1", execution_id: "execution-1", status: "failed" });
expect(
reopened
.prepare(
`SELECT owner_kind, owner_id, context_id, execution_id
FROM execution_owner_lifecycle_bindings
ORDER BY owner_kind`,
)
.all(),
).toEqual([
{
owner_kind: "cron",
owner_id: "cron-1",
context_id: "context-1",
execution_id: "execution-1",
},
{
owner_kind: "flow",
owner_id: "flow-1",
context_id: "context-1",
execution_id: "execution-1",
},
{
owner_kind: "task",
owner_id: "task-1",
context_id: "context-1",
execution_id: "execution-1",
},
]);
expect(tableExists(reopened, "execution_decision_facts")).toBe(true);
expect(
reopened.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get(),
).toEqual({ count: 0 });
});
it("uses stable c/t/f cursors and rejects a mismatched exact execution", () => {
const options = createOldOwnerDatabase();
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options });
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options });
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options });
const db = openOpenClawStateDatabase(options).db;
db.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run();
db.prepare("UPDATE task_runs SET status = 'succeeded'").run();
db.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run();
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "skipped", 1, 60, 70);
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES (?, ?, ?, ?)`,
).run("cron", "cron-2", "context-1", "execution-other");
const context = executionContext();
const first = presentExecutionDecisionReceipts({
context,
decisionLimit: 1,
options,
});
expect(first.nextDecisionCursor).toBe("a:0:0");
const cronPage = presentExecutionDecisionReceipts({
context,
decisionCursor: first.nextDecisionCursor,
decisionLimit: 1,
options,
});
expect(cronPage.decisions[0]?.source.owner).toBe("cron_run_receipts");
expect(cronPage.decisionDisplays[0]).toMatchObject({
decision: { outcome: "not-applicable", reasonCode: "cron_run_ok" },
provenance: { state: "verified", producer: "cron-lifecycle" },
});
expect(cronPage.nextDecisionCursor).toMatch(/^c:/);
const mismatchPage = presentExecutionDecisionReceipts({
context,
decisionCursor: cronPage.nextDecisionCursor,
decisionLimit: 1,
options,
});
expect(mismatchPage.decisions[0]).toMatchObject({
decision: { outcome: "unknown", reasonCode: "cron_run_execution_link_mismatch" },
missingEvidence: ["decision.execution_link"],
});
expect(mismatchPage.nextDecisionCursor).toBe("t:0:0");
const taskPage = presentExecutionDecisionReceipts({
context,
decisionCursor: mismatchPage.nextDecisionCursor,
decisionLimit: 1,
options,
});
expect(taskPage.decisions[0]?.source.owner).toBe("task_runs");
expect(taskPage.decisionDisplays[0]).toMatchObject({
decision: { outcome: "not-applicable", reasonCode: "task_run_succeeded" },
provenance: { state: "verified", producer: "task-lifecycle" },
});
expect(taskPage.nextDecisionCursor).toBe("f:0:0");
const flowPage = presentExecutionDecisionReceipts({
context,
decisionCursor: taskPage.nextDecisionCursor,
decisionLimit: 1,
options,
});
expect(flowPage.decisions[0]?.source.owner).toBe("flow_runs");
expect(flowPage.decisionDisplays[0]).toMatchObject({
decision: { outcome: "not-applicable", reasonCode: "flow_run_succeeded" },
provenance: { state: "verified", producer: "flow-lifecycle" },
});
expect(
JSON.stringify([
cronPage.decisionDisplays,
taskPage.decisionDisplays,
flowPage.decisionDisplays,
]),
).not.toMatch(/cron-1|task-1|flow-1|private|cron_run_receipts|task_runs|flow_runs/);
expect(flowPage.nextDecisionCursor).toBeUndefined();
expect(
presentExecutionDecisionReceipts({
context,
decisionCursor: "1",
decisionLimit: 1,
options,
}).decisions[0]?.source.owner,
).toBe("cron_run_receipts");
});
const deletedAnchorCases = [
{
stage: "cron",
cursor: "c:0:0",
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
addSuccessor: (db: DatabaseSync) => {
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null);
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`,
).run();
},
deleteAnchor: (db: DatabaseSync) => {
db.prepare(
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'",
).run();
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
},
},
{
stage: "task",
cursor: "t:0:0",
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options }),
addSuccessor: (db: DatabaseSync) => {
db.prepare(
`INSERT INTO task_runs (
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
notify_policy, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
"task-2",
"cron",
"owner-2",
"system",
"private",
"running",
"not_applicable",
"silent",
63,
);
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES ('task', 'task-2', 'context-1', 'execution-1')`,
).run();
},
deleteAnchor: (db: DatabaseSync) => {
db.prepare(
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'task' AND owner_id = 'task-1'",
).run();
db.prepare("DELETE FROM task_runs WHERE task_id = 'task-1'").run();
},
},
{
stage: "flow",
cursor: "f:0:0",
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options }),
addSuccessor: (db: DatabaseSync) => {
db.prepare(
`INSERT INTO flow_runs (
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
).run("flow-2", "owner-2", "running", "silent", "private", 63, 63);
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES ('flow', 'flow-2', 'context-1', 'execution-1')`,
).run();
},
deleteAnchor: (db: DatabaseSync) => {
db.prepare(
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'flow' AND owner_id = 'flow-1'",
).run();
db.prepare("DELETE FROM flow_runs WHERE flow_id = 'flow-1'").run();
},
},
] as const;
it.each(deletedAnchorCases)(
"rejects a nonzero $stage cursor after its exact owner anchor is deleted",
({ cursor, bindFirst, addSuccessor, deleteAnchor }) => {
const options = createOldOwnerDatabase();
expect(bindFirst(options)).toBe("bound");
const db = openOpenClawStateDatabase(options).db;
addSuccessor(db);
const firstPage = presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: cursor,
decisionLimit: 1,
options,
});
expect(firstPage.nextDecisionCursor).toMatch(/^[ctf]:[1-9]\d*:[1-9]\d*$/);
deleteAnchor(db);
expect(() =>
presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: firstPage.nextDecisionCursor,
decisionLimit: 1,
options,
}),
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
},
);
it("rejects a nonzero owner cursor when its retained anchor belongs to another context", () => {
const options = createOldOwnerDatabase();
expect(
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
).toBe("bound");
const db = openOpenClawStateDatabase(options).db;
deletedAnchorCases[0].addSuccessor(db);
const firstPage = presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: "c:0:0",
decisionLimit: 1,
options,
});
expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/);
expect(() =>
presentExecutionDecisionReceipts({
context: executionContext("context-other"),
decisionCursor: firstPage.nextDecisionCursor,
decisionLimit: 1,
options,
}),
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
});
it("rejects a nonzero owner cursor after the state database is removed", () => {
const options = createOldOwnerDatabase();
expect(
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
).toBe("bound");
const db = openOpenClawStateDatabase(options).db;
deletedAnchorCases[0].addSuccessor(db);
const firstPage = presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: "c:0:0",
decisionLimit: 1,
options,
});
expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/);
closeOpenClawStateDatabaseForTest();
rmSync(options.path);
expect(() =>
presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: firstPage.nextDecisionCursor,
decisionLimit: 1,
options,
}),
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
});
it("rejects a reused owner rowid whose binding belongs to another execution", () => {
const options = createOldOwnerDatabase();
const db = openOpenClawStateDatabase(options).db;
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null);
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null);
expect(
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
).toBe("bound");
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`,
).run();
const firstPage = presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: "c:0:0",
decisionLimit: 1,
options,
});
expect(firstPage.nextDecisionCursor).toMatch(/^c:60:[1-9]\d*$/);
db.prepare(
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'",
).run();
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
db.prepare(
`INSERT INTO cron_run_receipts (
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
status, owner_pid, started_at_ms, finished_at_ms
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
"cron-replacement",
"default",
"job-replacement",
"revision-1",
"main",
"run-2",
"running",
1,
60,
null,
);
db.prepare(
`INSERT INTO execution_owner_lifecycle_bindings (
owner_kind, owner_id, context_id, execution_id
) VALUES ('cron', 'cron-replacement', 'context-1', 'execution-other')`,
).run();
expect(() =>
presentExecutionDecisionReceipts({
context: executionContext(),
decisionCursor: firstPage.nextDecisionCursor,
decisionLimit: 1,
options,
}),
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
});
it("projects every owner terminal state without rederiving lifecycle precedence", () => {
const options = createOldOwnerDatabase();
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options });
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options });
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options });
const context = executionContext();
const db = openOpenClawStateDatabase(options).db;
for (const status of ["ok", "error", "skipped", "interrupted", "superseded"]) {
db.prepare(
"UPDATE cron_run_receipts SET status = ?, finished_at_ms = 70 WHERE receipt_id = ?",
).run(status, "cron-1");
expect(
pageOwnerLifecycleReceipts({ stage: "cron", context, limit: 1, options }).entries[0]
?.receipt.decision,
).toEqual({ outcome: "not-applicable", reasonCode: `cron_run_${status}` });
}
for (const status of ["succeeded", "failed", "timed_out", "cancelled", "lost"]) {
db.prepare("UPDATE task_runs SET status = ?, terminal_outcome = NULL WHERE task_id = ?").run(
status,
"task-1",
);
expect(
pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0]
?.receipt.decision.reasonCode,
).toBe(`task_run_${status}`);
}
db.prepare("UPDATE task_runs SET terminal_outcome = 'blocked' WHERE task_id = ?").run("task-1");
expect(
pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0]?.receipt
.decision.reasonCode,
).toBe("task_run_blocked");
for (const status of ["blocked", "succeeded", "failed", "cancelled", "lost"]) {
db.prepare("UPDATE flow_runs SET status = ? WHERE flow_id = ?").run(status, "flow-1");
expect(
pageOwnerLifecycleReceipts({ stage: "flow", context, limit: 1, options }).entries[0]
?.receipt.decision.reasonCode,
).toBe(`flow_run_${status}`);
}
});
});
@@ -0,0 +1,445 @@
/** Static owner-native cron/task/flow lifecycle projection for run inspection. */
import type { DatabaseSync } from "node:sqlite";
import type {
DecisionReceiptV1,
ExecutionIdentityContextV1,
} from "../../packages/gateway-protocol/src/index.js";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../infra/kysely-sync.js";
import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js";
import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js";
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
import type { OpenClawStateDatabaseOptions } from "../state/openclaw-state-db.js";
import { EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE } from "./execution-owner-lifecycle-binding-store.js";
type WithSqliteRowId<Row> = Row & { rowid: number };
type OwnerLifecycleDatabase = {
cron_run_receipts: WithSqliteRowId<OpenClawStateDatabase["cron_run_receipts"]>;
execution_owner_lifecycle_bindings: OpenClawStateDatabase["execution_owner_lifecycle_bindings"];
flow_runs: WithSqliteRowId<OpenClawStateDatabase["flow_runs"]>;
task_runs: WithSqliteRowId<OpenClawStateDatabase["task_runs"]>;
};
export type OwnerLifecycleStage = "cron" | "task" | "flow";
export type OwnerLifecycleCursor = { occurredAt: number; rowId: number };
type OwnerLifecycleDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle";
type OwnerLifecycleReceiptEntry = {
receipt: DecisionReceiptV1;
selectorId: string;
displayProducer: OwnerLifecycleDisplayProducer;
};
type OwnerLifecycleRow = {
executionId: string | null;
occurredAt: number;
owner: "cron_run_receipts" | "task_runs" | "flow_runs";
recordId: string;
rowId: number;
status: string;
};
const KNOWN_STATUSES: Record<OwnerLifecycleStage, ReadonlySet<string>> = {
cron: new Set(["running", "ok", "error", "skipped", "interrupted", "superseded"]),
task: new Set([
"queued",
"running",
"succeeded",
"failed",
"timed_out",
"cancelled",
"lost",
"blocked",
]),
flow: new Set([
"queued",
"running",
"waiting",
"blocked",
"succeeded",
"failed",
"cancelled",
"lost",
]),
};
const OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR = "owner lifecycle cursor is no longer retained";
function ownerName(stage: OwnerLifecycleStage): OwnerLifecycleRow["owner"] {
return stage === "cron" ? "cron_run_receipts" : stage === "task" ? "task_runs" : "flow_runs";
}
function displayProducer(stage: OwnerLifecycleStage): OwnerLifecycleDisplayProducer {
return stage === "cron"
? "cron-lifecycle"
: stage === "task"
? "task-lifecycle"
: "flow-lifecycle";
}
function assertRetainedCursor(params: {
db: DatabaseSync;
stage: OwnerLifecycleStage;
contextId: string;
executionId: string;
after?: OwnerLifecycleCursor;
}): void {
if (!params.after) {
return;
}
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
const ownerQuery =
params.stage === "cron"
? kysely
.selectFrom("cron_run_receipts")
.select("receipt_id as ownerId")
.where("rowid", "=", params.after.rowId)
.where("started_at_ms", "=", params.after.occurredAt)
: params.stage === "task"
? kysely
.selectFrom("task_runs")
.select("task_id as ownerId")
.where("rowid", "=", params.after.rowId)
.where("created_at", "=", params.after.occurredAt)
: kysely
.selectFrom("flow_runs")
.select("flow_id as ownerId")
.where("rowid", "=", params.after.rowId)
.where("created_at", "=", params.after.occurredAt);
const owner = executeSqliteQueryTakeFirstSync(params.db, ownerQuery);
// Admission binds at most one owner per lifecycle kind to an execution.
// The exact execution match therefore rejects a different owner reusing this rowid.
const binding = owner
? executeSqliteQueryTakeFirstSync(
params.db,
kysely
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
.select("owner_id")
.where("owner_kind", "=", params.stage)
.where("owner_id", "=", owner.ownerId)
.where("context_id", "=", params.contextId)
.where("execution_id", "=", params.executionId),
)
: undefined;
if (!binding) {
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
}
}
function readRows(params: {
db: DatabaseSync;
stage: OwnerLifecycleStage;
contextId: string;
executionId: string;
after?: OwnerLifecycleCursor;
offset?: number;
limit: number;
}): OwnerLifecycleRow[] {
const owner = ownerName(params.stage);
if (
!tableExists(params.db, owner) ||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
) {
if (params.after) {
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
}
return [];
}
assertRetainedCursor(params);
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
if (params.stage === "cron") {
let query = kysely
.selectFrom("cron_run_receipts")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "cron_run_receipts.receipt_id")
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"),
)
.select([
"cron_run_receipts.receipt_id as recordId",
"execution_owner_lifecycle_bindings.execution_id as executionId",
"cron_run_receipts.started_at_ms as occurredAt",
"cron_run_receipts.status",
"cron_run_receipts.rowid as rowId",
])
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.orderBy("cron_run_receipts.started_at_ms", "asc")
.orderBy("cron_run_receipts.rowid", "asc")
.limit(params.limit);
if (params.after) {
query = query.where((eb) =>
eb.or([
eb("cron_run_receipts.started_at_ms", ">", params.after!.occurredAt),
eb.and([
eb("cron_run_receipts.started_at_ms", "=", params.after!.occurredAt),
eb("cron_run_receipts.rowid", ">", params.after!.rowId),
]),
]),
);
} else if (params.offset) {
query = query.offset(params.offset);
}
return executeSqliteQuerySync(params.db, query).rows.map((row) =>
Object.assign(row, { owner }),
);
}
if (params.stage === "task") {
let query = kysely
.selectFrom("task_runs")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id")
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"),
)
.select([
"task_runs.task_id as recordId",
"execution_owner_lifecycle_bindings.execution_id as executionId",
"task_runs.created_at as occurredAt",
"task_runs.status",
"task_runs.terminal_outcome as terminalOutcome",
"task_runs.rowid as rowId",
])
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.orderBy("task_runs.created_at", "asc")
.orderBy("task_runs.rowid", "asc")
.limit(params.limit);
if (params.after) {
query = query.where((eb) =>
eb.or([
eb("task_runs.created_at", ">", params.after!.occurredAt),
eb.and([
eb("task_runs.created_at", "=", params.after!.occurredAt),
eb("task_runs.rowid", ">", params.after!.rowId),
]),
]),
);
} else if (params.offset) {
query = query.offset(params.offset);
}
return executeSqliteQuerySync(params.db, query).rows.map((row) =>
Object.assign(row, {
owner,
status: row.terminalOutcome === "blocked" ? "blocked" : row.status,
}),
);
}
let query = kysely
.selectFrom("flow_runs")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id")
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"),
)
.select([
"flow_runs.flow_id as recordId",
"execution_owner_lifecycle_bindings.execution_id as executionId",
"flow_runs.created_at as occurredAt",
"flow_runs.status",
"flow_runs.rowid as rowId",
])
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.orderBy("flow_runs.created_at", "asc")
.orderBy("flow_runs.rowid", "asc")
.limit(params.limit);
if (params.after) {
query = query.where((eb) =>
eb.or([
eb("flow_runs.created_at", ">", params.after!.occurredAt),
eb.and([
eb("flow_runs.created_at", "=", params.after!.occurredAt),
eb("flow_runs.rowid", ">", params.after!.rowId),
]),
]),
);
} else if (params.offset) {
query = query.offset(params.offset);
}
return executeSqliteQuerySync(params.db, query).rows.map((row) => Object.assign(row, { owner }));
}
function countRows(params: {
db: DatabaseSync;
stage: OwnerLifecycleStage;
contextId: string;
executionId?: string;
}): number {
const owner = ownerName(params.stage);
if (
!tableExists(params.db, owner) ||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
) {
return 0;
}
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
const query =
params.stage === "cron"
? kysely
.selectFrom("cron_run_receipts")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef(
"execution_owner_lifecycle_bindings.owner_id",
"=",
"cron_run_receipts.receipt_id",
)
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"),
)
.select((eb) => eb.fn.countAll<number>().as("count"))
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.$if(params.executionId !== undefined, (qb) =>
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
)
: params.stage === "task"
? kysely
.selectFrom("task_runs")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id")
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"),
)
.select((eb) => eb.fn.countAll<number>().as("count"))
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.$if(params.executionId !== undefined, (qb) =>
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
)
: kysely
.selectFrom("flow_runs")
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
join
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id")
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"),
)
.select((eb) => eb.fn.countAll<number>().as("count"))
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
.$if(params.executionId !== undefined, (qb) =>
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
);
return executeSqliteQueryTakeFirstSync(params.db, query)?.count ?? 0;
}
function projectReceipt(
stage: OwnerLifecycleStage,
row: OwnerLifecycleRow,
context: ExecutionIdentityContextV1,
): DecisionReceiptV1 {
const exact = row.executionId === context.executionId;
const known = KNOWN_STATUSES[stage].has(row.status);
const valid = exact && known;
const missingEvidence = valid
? []
: [exact ? `decision.${stage}_owner_status` : "decision.execution_link"];
return {
schemaVersion: 1,
receiptId: `${stage}:${row.recordId}`,
contextId: context.contextId,
executionId: context.executionId,
runId: context.runId,
actionId: row.recordId,
occurredAt: row.occurredAt,
action: {
family: stage === "cron" ? "scheduled-run" : stage === "task" ? "task" : "flow",
operation: "lifecycle",
summary: valid
? `${stage === "cron" ? "Scheduled run" : stage === "task" ? "Task" : "Flow"} lifecycle: ${row.status.replaceAll("_", "-")}.`
: "Owner lifecycle evidence could not be matched exactly.",
},
decision: {
outcome: valid ? "not-applicable" : "unknown",
reasonCode: valid
? `${stage}_run_${row.status}`
: exact
? `${stage}_run_status_unknown`
: `${stage}_run_execution_link_mismatch`,
},
enforcement: {
coverageState: valid ? "attribution-only" : "unknown",
evaluatorRef: `${stage}-lifecycle-owner`,
policyRefs: [],
grantRefs: [],
contextFieldsUsed: ["contextId", "executionId"],
},
source: {
owner: row.owner,
recordRef: row.recordId,
decisionBoundary: `${stage}.run.lifecycle`,
},
missingEvidence,
remediation: valid
? []
: [
{
code: "inspect_owner_execution_binding",
text: "Inspect the owner row and its exact admission binding before drawing a lifecycle conclusion.",
},
],
};
}
export function summarizeOwnerLifecycleReceipts(params: {
stage: OwnerLifecycleStage;
context: ExecutionIdentityContextV1;
options: OpenClawStateDatabaseOptions;
}): { count: number; coverageState?: "attribution-only" | "unknown"; missingEvidence: string[] } {
return (
withExistingOpenClawStateDatabaseReadOnly(({ db }) => {
const count = countRows({ db, stage: params.stage, contextId: params.context.contextId });
const exactCount = countRows({
db,
stage: params.stage,
contextId: params.context.contextId,
executionId: params.context.executionId,
});
const mismatch = count !== exactCount;
return {
count,
...(count > 0
? { coverageState: mismatch ? ("unknown" as const) : ("attribution-only" as const) }
: {}),
missingEvidence: mismatch ? ["decision.execution_link"] : [],
};
}, params.options) ?? { count: 0, missingEvidence: [] }
);
}
export function pageOwnerLifecycleReceipts(params: {
stage: OwnerLifecycleStage;
context: ExecutionIdentityContextV1;
after?: OwnerLifecycleCursor;
offset?: number;
limit: number;
options: OpenClawStateDatabaseOptions;
}): { entries: OwnerLifecycleReceiptEntry[]; nextCursor?: OwnerLifecycleCursor } {
const retainedRows = withExistingOpenClawStateDatabaseReadOnly(
({ db }) =>
runSqliteDeferredTransactionSync(
db,
() =>
readRows({
db,
stage: params.stage,
contextId: params.context.contextId,
executionId: params.context.executionId,
after: params.after,
offset: params.offset,
limit: params.limit + 1,
}),
{ operationLabel: "owner lifecycle receipt page" },
),
params.options,
);
if (!retainedRows && params.after) {
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
}
const rows = retainedRows ?? [];
const hasMore = rows.length > params.limit;
const page = hasMore ? rows.slice(0, params.limit) : rows;
const last = page.at(-1);
return {
entries: page.map((row) => ({
receipt: projectReceipt(params.stage, row, params.context),
selectorId: `${params.stage}-lifecycle:${row.occurredAt}:${row.rowId}`,
displayProducer: displayProducer(params.stage),
})),
...(hasMore && last ? { nextCursor: { occurredAt: last.occurredAt, rowId: last.rowId } } : {}),
};
}
+22 -3
View File
@@ -24,6 +24,7 @@ import { resolveScheduledToolPolicyContext } from "../../agents/scheduled-tool-p
import { withLocalSessionPlacementTurnAdmission } from "../../agents/session-placement-admission.js";
import { resolveSessionRuntimeOverrideForProvider } from "../../agents/session-runtime-compat.js";
import { hasResolvedThinkingCatalogEntry } from "../../agents/thinking-runtime.js";
import { withPostAdmissionExecutionOwnerBinding } from "../../audit/execution-owner-binding.js";
import type { ThinkLevel, VerboseLevel } from "../../auto-reply/thinking.js";
import type { CliSessionBinding } from "../../config/sessions.js";
import type { AgentDefaultsConfig } from "../../config/types.agent-defaults.js";
@@ -288,6 +289,7 @@ function createCronPromptExecutor(params: {
Partial<Omit<CronAgentExecutionPhaseUpdate, "jobId" | "phase">>,
) => void;
onLaneWait?: (info?: { waiting?: boolean }) => void;
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
}) {
const sessionFile = params.runSessionKey;
const cronFallbacksOverride =
@@ -396,15 +398,30 @@ function createCronPromptExecutor(params: {
});
let acceptedContextEngineTurnCandidate: ContextEngineTurnAttemptFacts | undefined;
const runId = params.cronSession.sessionEntry.sessionId;
const preparedRunAdmission = prepareAgentRunAdmission({
const basePreparedRunAdmission = prepareAgentRunAdmission({
operationalRunInstance: createOperationalRunInstanceRef(runId),
cfg: params.cfgWithAgentDefaults,
facts: {
runId,
agentId: params.agentId,
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
ingress: params.executionIdentity?.ingress ?? {
kind: "schedule",
boundary: "cron.isolated-agent",
state: "present",
},
...(params.executionIdentity?.invoker ? { invoker: params.executionIdentity.invoker } : {}),
},
});
const preparedRunAdmission = params.executionIdentity?.onPostAdmission
? withPostAdmissionExecutionOwnerBinding(
basePreparedRunAdmission,
params.executionIdentity.onPostAdmission,
)
: basePreparedRunAdmission;
const onExecutionStarted = (info?: CronRunnerStartedInfo) => {
params.onExecutionStarted?.(info);
params.executionIdentity?.onExecutionStarted?.();
};
const fallbackResult = await runWithModelFallback({
cfg: params.cfgWithAgentDefaults,
provider: params.liveSelection.provider,
@@ -454,7 +471,7 @@ function createCronPromptExecutor(params: {
const isFallback = candidateStarted;
candidateStarted = true;
const notifyExecutionStarted = (info?: { lifecycleGeneration?: string }) =>
params.onExecutionStarted?.({
onExecutionStarted({
...info,
...(isFallback ? { isFallback: true } : {}),
provider: providerOverride,
@@ -867,6 +884,7 @@ export async function executeCronRun(params: {
Partial<Omit<CronAgentExecutionPhaseUpdate, "jobId" | "phase">>,
) => void;
onLaneWait?: (info?: { waiting?: boolean }) => void;
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
immutableThinkLevel: ThinkLevel | undefined;
thinkingCatalog?: ModelCatalogEntry[];
loadThinkingCatalog: (provider: string, model: string) => Promise<ModelCatalogEntry[]>;
@@ -923,6 +941,7 @@ export async function executeCronRun(params: {
onExecutionStarted: params.onExecutionStarted,
onExecutionPhase: params.onExecutionPhase,
onLaneWait: params.onLaneWait,
executionIdentity: params.executionIdentity,
});
const runStartedAt = params.runStartedAt ?? Date.now();
+2
View File
@@ -93,6 +93,7 @@ export async function runCronIsolatedAgentTurn(params: {
sessionKey: string;
agentId?: string;
lane?: string;
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
}): Promise<RunCronAgentTurnResult> {
const admittedLifecycleGeneration = getAgentEventLifecycleGeneration();
const upstreamAbortSignal = params.abortSignal ?? params.signal;
@@ -264,6 +265,7 @@ export async function runCronIsolatedAgentTurn(params: {
runTimeoutOverrideMs: prepared.context.runTimeoutOverrideMs,
suppressExecNotifyOnExit: prepared.context.suppressExecNotifyOnExit,
pluginRegistry: prepared.context.pluginRegistry,
executionIdentity: params.executionIdentity,
};
const execution = await prepared.context.sessionWorkAdmission.run(() =>
withAgentRunLifecycleGeneration(runLifecycleGeneration, () =>
+7 -2
View File
@@ -38,7 +38,7 @@ import type {
} from "./state.js";
import { cronFailureNotificationEventContext, emit, isImmediateCronRunMode } from "./state.js";
import { ensureLoaded, runPostPersistCronNotifications, warnIfDisabled } from "./store.js";
import { tryCreateCronTaskRun, tryFinishCronTaskRun } from "./task-runs.js";
import { tryCreateCronTaskRunHandle, tryFinishCronTaskRun } from "./task-runs.js";
import { applyJobResult, armTimer, type CronTriggerEvalOutcome } from "./timer.js";
type PreparedManualRun =
@@ -70,6 +70,8 @@ type PreparedManualRun =
export type ActivatedManualRun = Extract<PreparedManualRun, { ran: true }> & {
startedAt: number;
taskRunId?: string;
taskId?: string;
flowId?: string;
activeJobMarker?: CronActiveJobMarker;
admittedJob: CronJob;
executionJob: CronJob;
@@ -503,12 +505,13 @@ export async function activatePreparedManualRun(
job: activatedJob,
runAtMs: startedAt,
});
const taskRunId = tryCreateCronTaskRun({
const taskRun = tryCreateCronTaskRunHandle({
state,
job: activatedJob,
startedAt,
publicRunId: prepared.runId ?? activation.runReceipt.receiptId,
});
const taskRunId = taskRun?.runId;
const activeJobMarker = markManualCronJobActive(state, job);
// Execute against a snapshot so later reload/merge can preserve delivery
// target writeback from disk without mutating the running object.
@@ -527,6 +530,8 @@ export async function activatePreparedManualRun(
startedAt,
runId: prepared.runId ?? taskRunId,
taskRunId,
taskId: taskRun?.taskId,
flowId: taskRun?.flowId,
activeJobMarker,
admittedJob,
executionJob,
+10 -1
View File
@@ -43,7 +43,10 @@ import type {
} from "./state.js";
import { emit, isImmediateCronRunMode } from "./state.js";
import { ensureLoaded, publishCronRuntimeRows, runPostPersistCronNotifications } from "./store.js";
import { tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
import {
createCronOwnerExecutionIdentityAdmission,
tryFinishCronTaskRunWithoutHistory,
} from "./task-runs.js";
import { recordCronOutcomeForJob } from "./timer-outcome-events.js";
import {
resolveCronRunScheduleOwnership,
@@ -161,6 +164,12 @@ async function finishPreparedManualRun(
streamScheduleKey: prepared.streamScheduleKey,
streamSourceIdentity: prepared.streamSourceIdentity,
runReceipt: prepared.runReceipt,
executionIdentity: createCronOwnerExecutionIdentityAdmission({
state,
runReceipt: prepared.runReceipt,
taskId: prepared.taskId,
flowId: prepared.flowId,
}),
});
} catch (err) {
if (err instanceof CronRunReceiptRevisionError && err.reason === "owner-unavailable") {
@@ -0,0 +1,232 @@
// Focused proof that manual cron admission binds exact owner-native rows.
import { describe, expect, it, vi } from "vitest";
import {
createDueIsolatedJob,
noopLogger,
setupCronRegressionFixtures,
} from "../../../test/helpers/cron/service-regression-fixtures.js";
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js";
import {
openOpenClawStateDatabase,
runOpenClawStateWriteTransaction,
} from "../../state/openclaw-state-db.js";
import { createManagedTaskFlow } from "../../tasks/task-flow-registry.js";
import {
resetTaskFlowRegistryForTests,
resetTaskRegistryForTests,
} from "../../tasks/task-runtime.test-helpers.js";
import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js";
import { saveCronStore } from "../store.js";
import {
claimCronRunReceiptInDatabase,
finishCronRunReceipt,
prepareCronRunReceiptClaim,
} from "../store/run-receipt-store.js";
import { run } from "./ops-run.js";
import { createCronServiceState } from "./state.js";
import {
createCronOwnerExecutionIdentityAdmission,
tryCreateCronTaskRunHandle,
} from "./task-runs.js";
const fixtures = setupCronRegressionFixtures({
prefix: "cron-service-execution-binding-",
});
describe("cron run execution binding", () => {
it("binds the exact admitted execution to the cron receipt and task rows", async () => {
await withOpenClawTestState(
{ layout: "state-only", prefix: "openclaw-cron-execution-binding-" },
async () => {
resetTaskRegistryForTests();
resetTaskFlowRegistryForTests();
const store = fixtures.makeStorePath();
const dueAt = Date.parse("2026-02-06T10:05:06.525Z");
const job = createDueIsolatedJob({
id: "exact-owner-binding",
nowMs: dueAt,
nextRunAtMs: dueAt,
});
await saveCronStore(store.storePath, { version: 1, jobs: [job] });
const runIsolatedAgentJob = vi.fn(
async (params: {
executionIdentity?: {
ingress: { kind: string };
onPostAdmission?: (context: AdmittedRunContext) => void;
onExecutionStarted?: () => void;
};
}) => {
const admitted = {
operationalRunInstance: { instanceId: "instance-exact", runId: "run-exact" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-exact", {
contextId: "context-exact",
executionId: "execution-exact",
now: dueAt,
}),
} satisfies AdmittedRunContext;
const beforeAdmissionSettles = openOpenClawStateDatabase().db;
expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe(
false,
);
params.executionIdentity?.onPostAdmission?.(admitted);
expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe(
false,
);
params.executionIdentity?.onExecutionStarted?.();
return { status: "ok" as const };
},
);
const state = createCronServiceState({
cronEnabled: true,
storePath: store.storePath,
log: noopLogger,
nowMs: () => dueAt,
enqueueSystemEvent: vi.fn(),
requestHeartbeat: vi.fn(),
runIsolatedAgentJob,
});
await expect(run(state, job.id, "force")).resolves.toEqual({ ok: true, ran: true });
expect(runIsolatedAgentJob).toHaveBeenCalledWith(
expect.objectContaining({
executionIdentity: expect.objectContaining({
ingress: expect.objectContaining({ kind: "schedule" }),
}),
}),
);
const db = openOpenClawStateDatabase().db;
expect(
db
.prepare(
`SELECT binding.context_id, binding.execution_id, receipt.status, receipt.error_text
FROM cron_run_receipts AS receipt
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
WHERE receipt.job_id = ?`,
)
.get(job.id),
).toEqual({
context_id: "context-exact",
execution_id: "execution-exact",
status: "ok",
error_text: null,
});
expect(
db
.prepare(
`SELECT binding.context_id, binding.execution_id, task.status
FROM task_runs AS task
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
WHERE task.source_id = ? AND task.runtime = 'cron'`,
)
.get(job.id),
).toEqual({
context_id: "context-exact",
execution_id: "execution-exact",
status: "succeeded",
});
},
);
});
it("does not partially bind task or flow rows after the cron owner is replaced", async () => {
await withOpenClawTestState(
{ layout: "state-only", prefix: "openclaw-cron-stale-execution-binding-" },
async () => {
resetTaskRegistryForTests();
resetTaskFlowRegistryForTests();
const store = fixtures.makeStorePath();
const dueAt = Date.parse("2026-02-06T10:06:06.525Z");
const job = {
...createDueIsolatedJob({
id: "stale-owner-binding",
nowMs: dueAt,
nextRunAtMs: dueAt,
}),
agentId: "main",
};
await saveCronStore(store.storePath, { version: 1, jobs: [job] });
const state = createCronServiceState({
cronEnabled: true,
storePath: store.storePath,
log: noopLogger,
nowMs: () => dueAt,
enqueueSystemEvent: vi.fn(),
requestHeartbeat: vi.fn(),
runIsolatedAgentJob: vi.fn(),
});
const prepared = prepareCronRunReceiptClaim({
storePath: store.storePath,
job,
agentId: job.agentId!,
startedAtMs: dueAt,
});
const initial = runOpenClawStateWriteTransaction(({ db }) =>
claimCronRunReceiptInDatabase({
database: db,
prepared,
resolveAgentId: (current) => current.agentId!,
}),
);
const task = tryCreateCronTaskRunHandle({ state, job, startedAt: dueAt });
expect(task).toBeDefined();
const flow = createManagedTaskFlow({
ownerKey: "agent:main:main",
controllerId: "tests/stale-owner-binding",
goal: "Reject partial stale-owner provenance",
status: "running",
});
expect(flow).not.toBeNull();
if (!task || !flow) {
throw new Error("expected live task and flow owners");
}
const executionIdentity = createCronOwnerExecutionIdentityAdmission({
state,
runReceipt: initial,
taskId: task.taskId,
flowId: flow.flowId,
});
const admitted = {
operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", {
contextId: "context-stale",
executionId: "execution-stale",
now: dueAt,
}),
} satisfies AdmittedRunContext;
executionIdentity.onPostAdmission?.(admitted);
const db = openOpenClawStateDatabase().db;
db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?").run(
2_147_483_647,
initial.receiptId,
);
const replacementPrepared = prepareCronRunReceiptClaim({
storePath: store.storePath,
job,
agentId: job.agentId!,
startedAtMs: dueAt + 1,
});
const replacement = runOpenClawStateWriteTransaction(({ db: transactionDb }) =>
claimCronRunReceiptInDatabase({
database: transactionDb,
prepared: replacementPrepared,
resolveAgentId: (current) => current.agentId!,
}),
);
executionIdentity.onExecutionStarted?.();
expect(
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
).toBe(false);
finishCronRunReceipt({
handle: replacement,
status: "skipped",
finishedAtMs: dueAt + 2,
});
},
);
});
});
+9 -5
View File
@@ -1109,7 +1109,7 @@ describe("cron service ops seam coverage", () => {
});
const taskRunId =
reservationOffsetMs === undefined
? taskRuns.tryCreateCronTaskRun({ state, job, startedAt })
? taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId
: taskExecutor.createRunningTaskRunCore({
runtime: "cron",
sourceId: job.id,
@@ -1227,7 +1227,7 @@ describe("cron service ops seam coverage", () => {
requestHeartbeat: vi.fn(),
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
});
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
if (!taskRunId) {
throw new Error("expected invalid finalized cron task run");
}
@@ -1296,7 +1296,7 @@ describe("cron service ops seam coverage", () => {
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
onEvent: (event) => events.push(structuredClone(event)),
});
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
if (!taskRunId) {
throw new Error("expected cron task run");
}
@@ -1363,7 +1363,7 @@ describe("cron service ops seam coverage", () => {
requestHeartbeat: vi.fn(),
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
});
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
if (!taskRunId) {
throw new Error("expected cron task run");
}
@@ -1437,7 +1437,11 @@ describe("cron service ops seam coverage", () => {
requestHeartbeat: vi.fn(),
runIsolatedAgentJob,
});
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job: original, startedAt });
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({
state,
job: original,
startedAt,
})?.runId;
if (!taskRunId) {
throw new Error("expected cron task run");
}
+104
View File
@@ -0,0 +1,104 @@
import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js";
import type { CronServiceState } from "./state.js";
export function resolveRunConcurrency(): number {
return DEFAULT_CRON_MAX_CONCURRENT_RUNS;
}
function acquireCronRunSlot(state: CronServiceState): () => void {
state.runAdmission.active += 1;
let released = false;
return () => {
if (released) {
return;
}
released = true;
state.runAdmission.active -= 1;
dispatchWaiters(state);
};
}
function dispatchWaiters(state: CronServiceState): void {
const admission = state.runAdmission;
if (state.stopped) {
cancelCronRunAdmissionWaiters(state);
return;
}
const maxConcurrentRuns = resolveRunConcurrency();
while (admission.active < maxConcurrentRuns) {
const waiter = admission.waiters.shift();
if (!waiter) {
break;
}
waiter(acquireCronRunSlot(state));
}
if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) {
const listener = admission.capacityListener;
admission.capacityListener = null;
if (listener) {
queueMicrotask(listener);
}
}
}
/**
* Acquire only the slots currently available to scheduled work. Unlike the
* waiter-based path used by direct runs, this never retains a timer batch while
* the pool is saturated.
*/
export function tryAcquireCronRunSlots(
state: CronServiceState,
requested: number,
): Array<() => void> {
if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) {
return [];
}
const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active);
return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state));
}
/** Keep the first wake-up until capacity release consumes or cancellation clears it. */
export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void {
state.runAdmission.capacityListener ??= listener;
}
async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> {
const admission = state.runAdmission;
if (state.stopped) {
return null;
}
if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) {
return acquireCronRunSlot(state);
}
return await new Promise<(() => void) | null>((resolve) => {
admission.waiters.push(resolve);
});
}
/** Wake queued work on stop so each caller can release its durable reservation. */
export function cancelCronRunAdmissionWaiters(state: CronServiceState): void {
state.runAdmission.capacityListener = null;
const waiters = state.runAdmission.waiters.splice(0);
for (const waiter of waiters) {
waiter(null);
}
}
/** Apply one service-level cap to every cron execution source. Queue waiters
* keep their job reservation, then recheck scheduler state before execution.
*/
export async function runWithCronAdmission<T>(
state: CronServiceState,
execute: () => Promise<T>,
acquiredRelease?: () => void,
): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> {
const release = acquiredRelease ?? (await acquireCronRunAdmission(state));
if (!release) {
return { kind: "stopped" };
}
try {
return { kind: "admitted", value: await execute() };
} finally {
release();
}
}
+20 -104
View File
@@ -1,4 +1,3 @@
import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js";
import { markCronJobActive } from "../active-jobs.js";
import { resolveCronJobConfigRevision } from "../config-revision.js";
import { createCronRunDiagnosticsFromError } from "../run-diagnostics.js";
@@ -17,6 +16,7 @@ import { normalizeCronRunErrorText } from "./execution-errors.js";
import { enrollForeignReceipt } from "./foreign-receipt-monitor.js";
import { recomputeJobNextRunAtMs } from "./jobs-scheduling.js";
import { locked } from "./locked.js";
import { runWithCronAdmission } from "./run-admission-capacity.js";
import {
activateServiceCronRunReceiptInDatabase,
claimServiceCronRunReceiptInDatabase,
@@ -27,7 +27,10 @@ import {
import { applyCronRuntimeRowsToState, commitCronRuntimeRows } from "./runtime-store.js";
import { type CronServiceState, type DeferredCronNotifications, emit } from "./state.js";
import { ensureLoaded, runPostPersistCronNotifications } from "./store.js";
import { tryCreateCronTaskRun } from "./task-runs.js";
import {
createCronOwnerExecutionIdentityAdmission,
tryCreateCronTaskRunHandle,
} from "./task-runs.js";
import {
runsDetachedFromMainSession,
type TimedCronRunOutcome,
@@ -35,88 +38,13 @@ import {
import { authorCronRunCompletion, executeJobCoreWithTimeout } from "./timer-job-runner.js";
import { isRunnableJob } from "./timer-runnable.js";
export function resolveRunConcurrency(): number {
return DEFAULT_CRON_MAX_CONCURRENT_RUNS;
}
function acquireCronRunSlot(state: CronServiceState): () => void {
state.runAdmission.active += 1;
let released = false;
return () => {
if (released) {
return;
}
released = true;
state.runAdmission.active -= 1;
dispatchWaiters(state);
};
}
function dispatchWaiters(state: CronServiceState): void {
const admission = state.runAdmission;
if (state.stopped) {
cancelCronRunAdmissionWaiters(state);
return;
}
const maxConcurrentRuns = resolveRunConcurrency();
while (admission.active < maxConcurrentRuns) {
const waiter = admission.waiters.shift();
if (!waiter) {
break;
}
waiter(acquireCronRunSlot(state));
}
if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) {
const listener = admission.capacityListener;
admission.capacityListener = null;
if (listener) {
queueMicrotask(listener);
}
}
}
/**
* Acquire only the slots currently available to scheduled work. Unlike the
* waiter-based path used by direct runs, this never retains a timer batch while
* the pool is saturated.
*/
export function tryAcquireCronRunSlots(
state: CronServiceState,
requested: number,
): Array<() => void> {
if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) {
return [];
}
const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active);
return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state));
}
/** Keep the first wake-up until capacity release consumes or cancellation clears it. */
export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void {
state.runAdmission.capacityListener ??= listener;
}
async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> {
const admission = state.runAdmission;
if (state.stopped) {
return null;
}
if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) {
return acquireCronRunSlot(state);
}
return await new Promise<(() => void) | null>((resolve) => {
admission.waiters.push(resolve);
});
}
/** Wake queued work on stop so each caller can release its durable reservation. */
export function cancelCronRunAdmissionWaiters(state: CronServiceState): void {
state.runAdmission.capacityListener = null;
const waiters = state.runAdmission.waiters.splice(0);
for (const waiter of waiters) {
waiter(null);
}
}
export {
cancelCronRunAdmissionWaiters,
resolveRunConcurrency,
runWithCronAdmission,
setCronRunCapacityListener,
tryAcquireCronRunSlots,
} from "./run-admission-capacity.js";
/** Track a persisted marker through shared admission and payload execution. */
export function reserveQueuedCronRun(
@@ -543,25 +471,6 @@ export async function activateQueuedCronRun(params: {
return { kind: "unavailable", reason: "stopped" };
}
/** Apply one service-level cap to every cron execution source. Queue waiters
* keep their job reservation, then recheck scheduler state before execution.
*/
export async function runWithCronAdmission<T>(
state: CronServiceState,
execute: () => Promise<T>,
acquiredRelease?: () => void,
): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> {
const release = acquiredRelease ?? (await acquireCronRunAdmission(state));
if (!release) {
return { kind: "stopped" };
}
try {
return { kind: "admitted", value: await execute() };
} finally {
release();
}
}
export async function executeQueuedCronRun(params: {
state: CronServiceState;
jobId: string;
@@ -663,12 +572,13 @@ export async function executeQueuedCronRun(params: {
const executionJob = structuredClone(started.job);
executionJob.state.runningAtMs = started.startedAt;
executionJob.state.lastError = undefined;
const taskRunId = tryCreateCronTaskRun({
const taskRun = tryCreateCronTaskRunHandle({
state,
job: executionJob,
startedAt: started.startedAt,
publicRunId: started.runReceipt.receiptId,
});
const taskRunId = taskRun?.runId;
const activeJobMarker = markCronJobActive(executionJob.id, {
preserveAcrossGenerationAdvance: !runsDetachedFromMainSession(executionJob),
});
@@ -694,6 +604,12 @@ export async function executeQueuedCronRun(params: {
runId: taskRunId,
activeJobMarker,
runReceipt: started.runReceipt,
executionIdentity: createCronOwnerExecutionIdentityAdmission({
state,
runReceipt: started.runReceipt,
taskId: taskRun?.taskId,
flowId: taskRun?.flowId,
}),
});
const result = state.deps.runSchedulerOwned
? await state.deps.runSchedulerOwned(execute)
+7 -1
View File
@@ -15,7 +15,13 @@ import type { CronJob } from "../types.js";
import { proposeCronRunRecovery, recoverCronRunProposal } from "./run-recovery.js";
import { createCronServiceState } from "./state.js";
import { runPostPersistCronNotifications } from "./store.js";
import { tryCreateCronTaskRun, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
import { tryCreateCronTaskRunHandle, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
function tryCreateCronTaskRun(
params: Parameters<typeof tryCreateCronTaskRunHandle>[0],
): string | undefined {
return tryCreateCronTaskRunHandle(params)?.runId;
}
const { logger, makeStorePath } = setupCronServiceSuite({ prefix: "cron-run-recovery-" });
+10
View File
@@ -1,5 +1,7 @@
/** Cron service dependency, event, state, and public result types. */
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import type { ExecutionIdentityAdmissionFacts } from "../../audit/execution-identity-admission.js";
import type { ReplyPayload } from "../../auto-reply/reply-payload.js";
import type { CronConfig } from "../../config/types.cron.js";
import type { HeartbeatRunResult, HeartbeatWakeRequest } from "../../infra/heartbeat-wake.js";
@@ -188,6 +190,7 @@ export type CronServiceDeps = {
onExecutionStarted?: (info?: CronAgentExecutionStarted) => void;
onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void;
onLaneWait?: (info?: { waiting?: boolean }) => void;
executionIdentity?: CronExecutionIdentityAdmission;
}) => Promise<
{
summary?: string;
@@ -267,6 +270,13 @@ export type CronServiceDeps = {
onEvent?: (evt: CronEvent, context?: CronEventContext) => void;
};
export type CronExecutionIdentityAdmission = {
ingress: ExecutionIdentityAdmissionFacts["ingress"];
invoker?: ExecutionIdentityAdmissionFacts["invoker"];
onPostAdmission?: (context: AdmittedRunContext) => void;
onExecutionStarted?: () => void;
};
/** Cron deps after optional defaults have been made concrete. */
type CronServiceDepsInternal = Omit<CronServiceDeps, "nowMs"> & {
nowMs: () => number;
+7 -1
View File
@@ -20,10 +20,16 @@ import { createCronServiceState as createCronServiceStateBase } from "./state.js
import {
getActiveCronTaskRunId,
findCronTaskRunRecoveryInDatabase,
tryCreateCronTaskRun,
tryCreateCronTaskRunHandle,
tryFinishCronTaskRun,
tryFinishCronTaskRunWithoutHistory,
} from "./task-runs.js";
function tryCreateCronTaskRun(
params: Parameters<typeof tryCreateCronTaskRunHandle>[0],
): string | undefined {
return tryCreateCronTaskRunHandle(params)?.runId;
}
import { executeJobCoreWithTimeout } from "./timer-job-runner.js";
function createCronServiceState(
+77 -12
View File
@@ -2,6 +2,10 @@
import { AsyncLocalStorage } from "node:async_hooks";
import { randomUUID } from "node:crypto";
import type { DatabaseSync } from "node:sqlite";
import {
createExecutionStartedOwnerBinding,
isRetainedExecutionOwnerBinding,
} from "../../audit/execution-owner-binding.js";
import { normalizeAgentId, resolveAgentIdFromSessionKey } from "../../routing/session-key.js";
import { CRON_TASK_KIND } from "../../tasks/cron-task-contract.js";
import {
@@ -11,7 +15,11 @@ import {
findTaskByRunId,
recordTaskRunProgressByRunIdCore,
} from "../../tasks/task-executor.js";
import { listTaskRecordsByRuntimeSourceIdInDatabase } from "../../tasks/task-registry.store.sqlite.js";
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
import {
bindTaskRunExecution,
listTaskRecordsByRuntimeSourceIdInDatabase,
} from "../../tasks/task-registry.store.sqlite.js";
import type { JsonValue, TaskRecord, TaskStatus } from "../../tasks/task-registry.types.js";
import {
CRON_AGENT_SELECTION_REQUIRED_MESSAGE,
@@ -20,6 +28,10 @@ import {
import { createCronExecutionId } from "../run-id.js";
import type { CronRunLogEntry } from "../run-log-types.js";
import { cronStoreKey } from "../store/key.js";
import {
bindCronRunReceiptExecution,
type CronRunReceiptHandle,
} from "../store/run-receipt-store.js";
import {
cronRunLogEntryToTaskDetail,
cronRunStatusToTaskStatus,
@@ -38,7 +50,7 @@ import type {
CronRunStatus,
} from "../types.js";
import { normalizeCronRunErrorText } from "./execution-errors.js";
import type { CronEvent, CronServiceState } from "./state.js";
import type { CronEvent, CronExecutionIdentityAdmission, CronServiceState } from "./state.js";
import { CRON_TASK_RUNNING_PROGRESS_SUMMARY } from "./task-ledger.js";
function requireCronAgentId(agentId: string | undefined): string {
@@ -64,6 +76,55 @@ export function getActiveCronTaskRunId(): string | undefined {
return activeCronTaskRunId.getStore();
}
/** Carries exact admission into the first post-admission owner lifecycle phase. */
export function createCronOwnerExecutionIdentityAdmission(params: {
state: CronServiceState;
runReceipt: CronRunReceiptHandle;
taskId?: string;
flowId?: string;
}): CronExecutionIdentityAdmission {
const ownerBinding = createExecutionStartedOwnerBinding((admitted) => {
try {
const receiptResult = bindCronRunReceiptExecution({
admitted,
handle: params.runReceipt,
});
const taskResult = params.taskId
? isRetainedExecutionOwnerBinding(receiptResult)
? bindTaskRunExecution({ admitted, taskId: params.taskId })
: receiptResult
: undefined;
const flowParentResult = params.taskId ? taskResult : receiptResult;
const flowResult = params.flowId
? isRetainedExecutionOwnerBinding(receiptResult) &&
isRetainedExecutionOwnerBinding(flowParentResult)
? bindTaskFlowExecution({ admitted, flowId: params.flowId })
: flowParentResult
: undefined;
if (
[receiptResult, taskResult, flowResult].some(
(result) => result === "mismatch" || result === "missing",
)
) {
params.state.deps.log.warn(
{ receiptResult, taskResult, flowResult },
"cron: exact execution identity binding was not retained",
);
}
} catch (error) {
params.state.deps.log.warn(
{ error },
"cron: failed to retain exact execution identity binding",
);
}
});
return {
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
onPostAdmission: ownerBinding.onPostAdmission,
onExecutionStarted: ownerBinding.onExecutionStarted,
};
}
/** Updates an active cron task with the exact transcript identity reported by its runner. */
export function tryUpdateCronTaskRunSession(
state: CronServiceState,
@@ -88,13 +149,12 @@ export function tryUpdateCronTaskRunSession(
}
}
/** Creates a best-effort detached task row keyed to the persisted execution start. */
export function tryCreateCronTaskRun(params: {
export function tryCreateCronTaskRunHandle(params: {
state: CronServiceState;
job: CronJob;
startedAt: number;
publicRunId?: string;
}): string | undefined {
}): { runId: string; taskId: string; flowId?: string } | undefined {
const runId = createCronTaskRunId(params.job.id, params.startedAt, params.publicRunId);
return tryCreateCronTaskRunRecord({
state: params.state,
@@ -214,7 +274,7 @@ function tryCreateCronTaskRunRecord(params: {
startedAt: number;
runId: string;
childSessionKey?: string;
}): string | undefined {
}): { runId: string; taskId: string; flowId?: string } | undefined {
try {
const childSessionKey = params.childSessionKey;
const effectiveJobAgentId = params.job
@@ -252,7 +312,11 @@ function tryCreateCronTaskRunRecord(params: {
);
return undefined;
}
return params.runId;
return {
runId: params.runId,
taskId: task.taskId,
...(task.parentFlowId ? { flowId: task.parentFlowId } : {}),
};
} catch (error) {
params.state.deps.log.warn(
{ jobId: params.jobId, error },
@@ -341,9 +405,9 @@ export function tryFinishCronTaskRun(
result.taskRunId ?? createCronTaskRunId(entry.jobId, startedAt, entry.runId);
try {
const existingCandidate = findTaskByRunId(candidateRunId);
const taskRunId =
const created =
existingCandidate?.runtime === "cron"
? candidateRunId
? undefined
: tryCreateCronTaskRunRecord({
state,
job: result.job ?? result.event.job,
@@ -352,6 +416,7 @@ export function tryFinishCronTaskRun(
runId: candidateRunId,
childSessionKey: entry.sessionKey,
});
const taskRunId = existingCandidate?.runtime === "cron" ? candidateRunId : created?.runId;
if (!taskRunId) {
return;
}
@@ -414,7 +479,7 @@ export function tryFinishCronTaskRun(
updated = finalize(taskRunId);
} else {
// A terminal event still owns one durable row if its active mirror vanished.
const recreatedRunId = tryCreateCronTaskRunRecord({
const recreated = tryCreateCronTaskRunRecord({
state,
job: result.job ?? result.event.job,
jobId: entry.jobId,
@@ -422,8 +487,8 @@ export function tryFinishCronTaskRun(
runId: taskRunId,
childSessionKey: entry.sessionKey,
});
if (recreatedRunId) {
updated = finalize(recreatedRunId);
if (recreated) {
updated = finalize(recreated.runId);
}
}
}
@@ -125,6 +125,7 @@ export type ExecuteJobCoreOptions = {
onExecutionStarted?: (info?: CronAgentExecutionStarted) => void;
onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void;
onLaneWait?: (info?: { waiting?: boolean }) => void;
executionIdentity?: import("./state.js").CronExecutionIdentityAdmission;
/** Revalidates the durable run fence after awaited planning and before effects. */
assertRunCurrent?: () => void;
streamBatch?: string;
+1
View File
@@ -459,6 +459,7 @@ async function executeDetachedCronJob(
onExecutionStarted: options?.onExecutionStarted,
onExecutionPhase: options?.onExecutionPhase,
onLaneWait: options?.onLaneWait,
executionIdentity: options?.executionIdentity,
});
if (abortSignal?.aborted) {
+3
View File
@@ -51,6 +51,7 @@ type CronCoreRunOptions = {
streamScheduleKey?: string;
streamSourceIdentity?: string;
runReceipt?: import("../store/run-receipt-store.js").CronRunReceiptHandle;
executionIdentity?: import("./state.js").CronExecutionIdentityAdmission;
};
async function deliverPrimaryWebhook(
@@ -274,6 +275,7 @@ async function executeJobCoreWithTimeoutUnfinalized(
onExecutionStarted: noteExecutionStarted,
onExecutionPhase: accumulateExecution,
assertRunCurrent,
executionIdentity: opts?.executionIdentity,
};
const corePromise = withCronTaskRunId(opts?.runId, () =>
executeJobCore(state, job, runAbortController.signal, coreOptions),
@@ -361,6 +363,7 @@ async function executeJobCoreWithTimeoutUnfinalized(
onExecutionPhase: deferTimeoutUntilExecutionStart ? watchdog.notePhase : undefined,
onLaneWait: deferTimeoutUntilExecutionStart ? noteLaneState : undefined,
assertRunCurrent,
executionIdentity: opts?.executionIdentity,
};
const corePromise = withCronTaskRunId(opts?.runId, () =>
executeJobCore(state, job, runAbortController.signal, coreOptions),
+61
View File
@@ -1,4 +1,6 @@
import { describe, expect, it } from "vitest";
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
import {
openOpenClawStateDatabase,
runOpenClawStateWriteTransaction,
@@ -9,6 +11,7 @@ import type { CronJob } from "../types.js";
import { cronStoreKey } from "./key.js";
import {
assertCronRunReceiptCurrent,
bindCronRunReceiptExecution,
claimCronRunReceiptInDatabase,
CronRunReceiptConflictError,
CronRunReceiptRevisionError,
@@ -127,14 +130,56 @@ describe("cron run receipt store", () => {
]);
});
it("rejects a delayed binding after a successor replaces its exact owner", async () => {
const { storePath } = await makeStorePath();
const job = makeJob("stale-binding");
await saveCronStore(storePath, { version: 1, jobs: [job] });
const abandoned = claim(storePath, job, 230);
openOpenClawStateDatabase()
.db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?")
.run(2_147_483_647, abandoned.receiptId);
const replacement = claim(storePath, job, 240);
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", {
contextId: "context-stale",
executionId: "execution-stale",
}),
};
expect(bindCronRunReceiptExecution({ admitted, handle: abandoned })).toBe("missing");
expect(bindCronRunReceiptExecution({ admitted, handle: replacement })).toBe("bound");
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT owner_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'cron'`,
)
.all(),
).toEqual([{ owner_id: replacement.receiptId }]);
finishCronRunReceipt({ handle: replacement, status: "ok", finishedAtMs: 250 });
});
it("prunes old terminal receipts while preserving the active and 64 newest rows", async () => {
const { storePath } = await makeStorePath();
const job = makeJob("retention");
await saveCronStore(storePath, { version: 1, jobs: [job] });
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-retention", runId: "run-retention" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-retention", {
contextId: "context-retention",
executionId: "execution-retention",
}),
};
const finishedReceiptIds: string[] = [];
for (let index = 0; index < 70; index += 1) {
const handle = claim(storePath, job, 1_000 + index * 2);
finishedReceiptIds.push(handle.receiptId);
if (index === 0 || index === 69) {
expect(bindCronRunReceiptExecution({ admitted, handle })).toBe("bound");
}
finishCronRunReceipt({
handle,
status: "ok",
@@ -142,6 +187,7 @@ describe("cron run receipt store", () => {
});
}
const active = claim(storePath, job, 2_000);
expect(bindCronRunReceiptExecution({ admitted, handle: active })).toBe("bound");
const retained = receipts(storePath, job.id);
const retainedIds = new Set(retained.map((receipt) => receipt.receiptId));
@@ -158,6 +204,21 @@ describe("cron run receipt store", () => {
for (const receiptId of finishedReceiptIds.slice(-64)) {
expect(retainedIds.has(receiptId)).toBe(true);
}
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT binding.owner_id
FROM execution_owner_lifecycle_bindings AS binding
JOIN cron_run_receipts AS receipt ON receipt.receipt_id = binding.owner_id
WHERE binding.owner_kind = 'cron' AND receipt.job_id = ?
ORDER BY owner_id`,
)
.all(job.id),
).toEqual(
[active.receiptId, finishedReceiptIds.at(-1)]
.toSorted((left, right) => (left ?? "").localeCompare(right ?? ""))
.map((owner_id) => ({ owner_id })),
);
finishCronRunReceipt({ handle: active, status: "skipped", finishedAtMs: 2_001 });
});
+46
View File
@@ -1,6 +1,15 @@
import crypto from "node:crypto";
import type { DatabaseSync } from "node:sqlite";
import type { Selectable } from "kysely";
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
import {
executionOwnerBindingFromAdmission,
type ExecutionOwnerBindingResult,
} from "../../audit/execution-owner-binding.js";
import {
bindExecutionOwnerLifecycleMetadata,
deleteExecutionOwnerLifecycleMetadata,
} from "../../audit/execution-owner-lifecycle-binding-store.js";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
@@ -176,6 +185,38 @@ function withReceiptWrite<T>(
return result;
}
/** Binds the exact admitted execution to its authoritative receipt without changing lifecycle. */
export function bindCronRunReceiptExecution(params: {
admitted: AdmittedRunContext;
handle: CronRunReceiptHandle;
options?: OpenClawStateDatabaseOptions;
}): ExecutionOwnerBindingResult {
const binding = executionOwnerBindingFromAdmission(params.admitted);
if (!binding) {
return "disabled";
}
return withReceiptWrite(
"cron.run-receipt.execution-binding",
params.options ?? {},
(database) => {
try {
assertCronRunReceiptOwnedInDatabase({ database, handle: params.handle });
} catch (error) {
if (!(error instanceof CronRunReceiptRevisionError)) {
throw error;
}
return "missing";
}
return bindExecutionOwnerLifecycleMetadata({
db: database,
ownerKind: "cron",
ownerId: params.handle.receiptId,
binding,
});
},
);
}
function isReceiptStatus(value: string): value is CronRunReceiptStatus {
return (
value === "running" ||
@@ -329,6 +370,11 @@ function pruneTerminalReceipts(database: DatabaseSync, storeKey: string, jobId:
const receiptIds = terminalIds
.slice(index, index + CRON_RUN_RECEIPT_DELETE_BATCH_SIZE)
.map((row) => row.receipt_id);
deleteExecutionOwnerLifecycleMetadata({
db: database,
ownerKind: "cron",
ownerIds: receiptIds,
});
executeSqliteQuerySync(
database,
query(database)
+51 -5
View File
@@ -6,6 +6,7 @@ import {
type AgentRunTerminalOutcome,
} from "../../agents/agent-run-terminal-outcome.js";
import type { PreparedAgentCommandRuntimeContext } from "../../agents/command/prepare.js";
import type { AgentCommandOpts } from "../../agents/command/types.js";
import {
createCronCreatorAuthorityCapability,
runWithCronCreatorAuthorityCapability,
@@ -15,6 +16,10 @@ import type { MainSessionRecoveryPendingTarget } from "../../agents/main-session
import { isAgentRunRestartAbortReason } from "../../agents/run-termination.js";
import { normalizeAgentRunTimeoutPhase } from "../../agents/run-timeout-attribution.js";
import { runWithCanonicalSkillWorkspace } from "../../agents/skill-workshop-workspace-context.js";
import {
createExecutionStartedOwnerBinding,
isRetainedExecutionOwnerBinding,
} from "../../audit/execution-owner-binding.js";
import { readAgentRunTerminalOutcome } from "../../channels/turn/agent-run-terminal-outcome.js";
import { agentCommandFromGatewayIngress } from "../../commands/agent.js";
import { isAbortError } from "../../infra/abort-signal.js";
@@ -22,7 +27,10 @@ import { clearAgentRunContext } from "../../infra/agent-run-registry.js";
import { formatErrorMessageWithCode, readErrorName } from "../../infra/errors.js";
import { defaultRuntime } from "../../runtime.js";
import { createRunningTaskRun } from "../../tasks/detached-task-runtime.js";
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
import { mapAgentRunTerminalOutcomeToTaskStatus } from "../../tasks/task-registry-common.js";
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
import type { TaskRecord } from "../../tasks/task-registry.types.js";
import { normalizeDeliveryContext } from "../../utils/delivery-context.shared.js";
import type { ChatAbortControllerEntry } from "../chat-abort.js";
import {
@@ -136,9 +144,10 @@ export function dispatchAgentRunFromGateway(params: {
}) {
const shouldTrackTask = params.taskTrackingMode === "cli";
let taskTracked = false;
let trackedTask: TaskRecord | undefined;
if (shouldTrackTask) {
try {
taskTracked = Boolean(
trackedTask =
createRunningTaskRun({
runtime: "cli",
sourceId: params.runId,
@@ -155,8 +164,8 @@ export function dispatchAgentRunFromGateway(params: {
task: params.ingressOpts.message,
deliveryStatus: "not_applicable",
startedAt: Date.now(),
}),
);
}) ?? undefined;
taskTracked = Boolean(trackedTask);
} catch (err) {
// Best-effort only: background task tracking must not block agent runs.
// Still surface the swallowed error so non-transient tracking failures stay observable.
@@ -188,12 +197,49 @@ export function dispatchAgentRunFromGateway(params: {
params.ingressOpts,
readAgentRunDispatchExecutionIdentity(params),
);
const trackedTaskBinding = trackedTask
? createExecutionStartedOwnerBinding(
(admitted: Parameters<NonNullable<AgentCommandOpts["onPostAdmittedRunContext"]>>[0]) => {
try {
const taskResult = bindTaskRunExecution({ admitted, taskId: trackedTask.taskId });
const flowResult = trackedTask.parentFlowId
? isRetainedExecutionOwnerBinding(taskResult)
? bindTaskFlowExecution({ admitted, flowId: trackedTask.parentFlowId })
: taskResult
: undefined;
if (
[taskResult, flowResult].some(
(result) => result === "mismatch" || result === "missing",
)
) {
params.context.logGateway.warn(
`exact tracked-task execution binding was not retained for ${params.runId}`,
);
}
} catch (error) {
params.context.logGateway.warn(
`failed to retain tracked-task execution binding ${params.runId}: ${formatForLog(error)}`,
);
}
},
)
: undefined;
const ingressOptsWithTaskBinding = trackedTask
? {
...ingressOptsWithSpawnFacts,
onPostAdmittedRunContext: trackedTaskBinding?.onPostAdmission,
onExecutionStarted: () => {
ingressOptsWithSpawnFacts.onExecutionStarted?.();
trackedTaskBinding?.onExecutionStarted();
},
}
: ingressOptsWithSpawnFacts;
const runAgent = () =>
runWithCanonicalSkillWorkspace(params.canonicalSkillWorkspaceDir, () =>
agentCommandFromGatewayIngress(
cronCreatorAuthorityCapability
? { ...ingressOptsWithSpawnFacts, cronCreatorAuthorityCapability }
: ingressOptsWithSpawnFacts,
? { ...ingressOptsWithTaskBinding, cronCreatorAuthorityCapability }
: ingressOptsWithTaskBinding,
defaultRuntime,
params.context.deps,
{
@@ -1,6 +1,9 @@
import { ErrorCodes, errorShape } from "../../../packages/gateway-protocol/src/index.js";
import { getAdmittedRunDelegatedAuthority } from "../../agents/admitted-run-context.js";
import { attachAgentCommandAdmissionFacts } from "../../agents/agent-command-admission-facts.js";
import {
attachAgentCommandAdmissionFacts,
attachAgentCommandRecoveryAdmissionFacts,
} from "../../agents/agent-command-admission-facts.js";
import type { AgentRunTerminalOutcome } from "../../agents/agent-run-terminal-outcome.js";
import { prepareGitCoauthorAttribution } from "../../agents/git-coauthor-attribution.js";
import { repairMainSessionRecoveryMutation } from "../../agents/main-session-recovery/main-session-recovery-lifecycle.js";
@@ -265,7 +268,9 @@ export function startAgentRunExecution(params: {
);
const localUserIngress = getGatewayLocalUserIngress(params.client);
if (localUserIngress) {
if (params.isRestartRecoveryResumeRun) {
attachAgentCommandRecoveryAdmissionFacts(runContext);
} else if (localUserIngress) {
attachAgentCommandAdmissionFacts(runContext, localUserIngress.facts);
}
finalizePreparedAgentRunUserTurn(prepared.userTurn);
@@ -0,0 +1,28 @@
// Hook mapping identity coverage keeps mapping IDs as ingress attribution only.
import { describe, expect, it } from "vitest";
import { applyHookMappings, resolveHookMappings } from "./hooks-mapping.js";
describe("hook mapping execution identity", () => {
it("carries the configured mapping id on the resolved agent action", async () => {
const mappings = resolveHookMappings({
mappings: [
{
id: "source-gmail",
match: { path: "gmail" },
action: "agent",
messageTemplate: "{{messages[0].subject}}",
},
],
});
const result = await applyHookMappings(mappings, {
payload: { messages: [{ subject: "Hello" }] },
headers: {},
url: new URL("http://127.0.0.1:18789/hooks/gmail"),
path: "gmail",
});
expect(result).toMatchObject({
ok: true,
action: { kind: "agent", mappingId: "source-gmail", message: "Hello" },
});
});
});
+6
View File
@@ -49,6 +49,7 @@ type HookMappingContext = {
type HookAction =
| {
kind: "wake";
mappingId: string;
text: string;
mode: "now" | "next-heartbeat";
agentId?: string;
@@ -57,6 +58,7 @@ type HookAction =
}
| {
kind: "agent";
mappingId: string;
message: string;
name?: string;
agentId?: string;
@@ -272,6 +274,7 @@ function buildActionFromMapping(
ok: true,
action: {
kind: "wake",
mappingId: mapping.id,
text,
mode: mapping.wakeMode ?? "now",
agentId: mapping.agentId,
@@ -285,6 +288,7 @@ function buildActionFromMapping(
ok: true,
action: {
kind: "agent",
mappingId: mapping.id,
message,
name: renderOptional(mapping.name, ctx),
agentId: mapping.agentId,
@@ -318,6 +322,7 @@ function mergeAction(
const mode = override.mode === "next-heartbeat" ? "next-heartbeat" : (baseWake?.mode ?? "now");
return validateAction({
kind: "wake",
mappingId: base.mappingId,
text,
mode,
agentId: override.agentId ?? baseWake?.agentId,
@@ -332,6 +337,7 @@ function mergeAction(
override.wakeMode === "next-heartbeat" ? "next-heartbeat" : (baseAgent?.wakeMode ?? "now");
return validateAction({
kind: "agent",
mappingId: base.mappingId,
message,
wakeMode,
name: override.name ?? baseAgent?.name,
+2
View File
@@ -306,6 +306,8 @@ export type HookAgentDispatchPayload = Omit<HookAgentPayload, "sessionKey"> & {
sourcePath: string;
allowUnsafeExternalContent?: boolean;
externalContentSource?: HookExternalContentSource;
/** Configured ingress source attribution; never an authenticated principal. */
mappingId?: string;
};
const listHookChannelValues = () => ["last", ...listChannelPlugins().map((plugin) => plugin.id)];
+2
View File
@@ -850,6 +850,7 @@ export function buildGatewayCronService(params: {
onExecutionStarted,
onExecutionPhase,
onLaneWait,
executionIdentity,
}) => {
const { agentId, cfg: runtimeConfig } = resolveCronAgent(job.agentId);
const sessionKey = resolveCronSessionTargetSessionKey(job.sessionTarget) ?? `cron:${job.id}`;
@@ -862,6 +863,7 @@ export function buildGatewayCronService(params: {
onExecutionStarted,
onExecutionPhase,
onLaneWait,
executionIdentity,
agentId,
sessionKey,
lane: "cron",
+39 -28
View File
@@ -314,34 +314,6 @@ describe("audit gateway methods", () => {
decisionLimit: 20,
});
await runAuditHandler("audit.run.inspect", {
runId: "run-1",
executionCursor: "1",
decisionCursor: "1",
decisionLimit: 25,
});
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
runId: "run-1",
executionOffset: 1,
executionLimit: 50,
decisionCursor: "1",
decisionLimit: 25,
});
await runAuditHandler("audit.run.inspect", {
runId: "run-1",
executionCursor: "001",
decisionCursor: "001",
decisionLimit: 25,
});
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
runId: "run-1",
executionOffset: 1,
executionLimit: 50,
decisionCursor: "001",
decisionLimit: 25,
});
await runAuditHandler("audit.run.inspect", {
executionId: "execution-1",
decisionCursor: "1",
@@ -503,6 +475,45 @@ describe("audit gateway methods", () => {
expect(JSON.stringify(result)).not.toContain('"decisions"');
});
it.each(["1", "001"])(
"preserves mirrored numeric cursor %s for execution paging",
async (decisionCursor) => {
await runAuditHandler("audit.run.inspect", {
runId: "run-1",
executionCursor: decisionCursor,
decisionCursor,
decisionLimit: 25,
});
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
runId: "run-1",
executionOffset: 1,
executionLimit: 50,
decisionCursor,
decisionLimit: 25,
});
},
);
it.each(["a:2000:42", "m:2000:42", "g:2000:42", "c:2000:42", "t:2000:42", "f:2000:42"])(
"treats mirrored owner decision cursor %s as decision-only",
async (decisionCursor) => {
await runAuditHandler("audit.run.inspect", {
runId: "run-1",
executionCursor: decisionCursor,
decisionCursor,
decisionLimit: 25,
});
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
runId: "run-1",
executionLimit: 50,
decisionCursor,
decisionLimit: 25,
});
},
);
it("rejects malformed run inspection before storage access", async () => {
expect(
await runAuditHandler("audit.run.inspect", { runId: "", extra: true }),
+5 -3
View File
@@ -49,6 +49,10 @@ function serializeAuditRunInspectResult(
return result;
}
function isOwnerDecisionCursor(value: string): boolean {
return parsePositiveAuditCursor(value) === null && isExecutionDecisionCursor(value);
}
/** Preserve the shipped audit.list result shape for run/tool-only clients. */
function mapLegacyAuditEvent(
event: AgentRunAuditEventRecord | ToolActionAuditEventRecord,
@@ -198,9 +202,7 @@ export const auditHandlers: GatewayRequestHandlers = {
typeof params.runId !== "string" ||
(params.executionCursor === decisionCursor &&
decisionCursor !== undefined &&
(decisionCursor.startsWith("a:") ||
decisionCursor.startsWith("m:") ||
decisionCursor.startsWith("g:")))
isOwnerDecisionCursor(decisionCursor))
? undefined
: parsePositiveAuditCursor(params.executionCursor);
if (
+92 -3
View File
@@ -1,10 +1,12 @@
/** Focused HTTP coverage for hook admission feedback and pending replay behavior. */
import fs from "node:fs/promises";
import { Agent, request as httpRequest } from "node:http";
import path from "node:path";
import { afterEach, describe, expect, test, vi } from "vitest";
import { createDeferred } from "../../test/helpers/promise.js";
import { resolveMainSessionKeyFromConfig } from "../config/sessions.js";
import { DEFAULT_WEBHOOK_MAX_BODY_BYTES } from "../infra/http-body.js";
import { drainSystemEvents } from "../infra/system-events.js";
import { drainSystemEvents, peekSystemEventEntries } from "../infra/system-events.js";
import {
cronIsolatedRun,
installGatewayTestHooks,
@@ -25,11 +27,11 @@ afterEach(() => {
async function postHook(
port: number,
path: string,
hookPath: string,
body: Record<string, unknown>,
idempotencyKey: string,
): Promise<Response> {
return await fetch(`http://127.0.0.1:${port}${path}`, {
return await fetch(`http://127.0.0.1:${port}${hookPath}`, {
method: "POST",
headers: {
Authorization: `Bearer ${HOOK_TOKEN}`,
@@ -106,6 +108,24 @@ async function postOversizedChunkedHook(port: number): Promise<{
}
}
async function writeHookTransformModule(moduleName: string, source: string): Promise<void> {
const configPath = process.env.OPENCLAW_CONFIG_PATH;
if (!configPath) {
throw new Error("expected OPENCLAW_CONFIG_PATH");
}
const transformsDir = path.join(path.dirname(configPath), "hooks", "transforms");
await fs.mkdir(transformsDir, { recursive: true });
await fs.writeFile(path.join(transformsDir, moduleName), source, "utf8");
}
function readExecutionIdentityCall(index: number): unknown {
const call = cronIsolatedRun.mock.calls[index]?.[0];
if (!call || typeof call !== "object" || !("executionIdentity" in call)) {
return undefined;
}
return call.executionIdentity;
}
describe("gateway hook admission", () => {
test("flushes an oversized chunked hook response before closing the socket", async () => {
testState.hooksConfig = { enabled: true, token: HOOK_TOKEN };
@@ -144,6 +164,75 @@ describe("gateway hook admission", () => {
});
});
test("keeps direct hooks unattributed and mapped IDs as ingress attribution", async () => {
testState.hooksConfig = {
enabled: true,
token: HOOK_TOKEN,
mappings: [
{
id: "gmail-source",
match: { path: "gmail" },
action: "agent",
messageTemplate: "New email from {{messages[0].from}}",
},
],
};
await withGatewayServer(async ({ port }) => {
cronIsolatedRun.mockClear();
cronIsolatedRun.mockResolvedValue({ status: "ok", summary: "done" });
expect(
(await postHook(port, "/hooks/agent", { message: "Direct" }, "direct-source")).status,
).toBe(200);
expect(
(await postHook(port, "/hooks/gmail", { messages: [{ from: "Ada" }] }, "mapped-source"))
.status,
).toBe(200);
expect(readExecutionIdentityCall(0)).toEqual({
ingress: { kind: "webhook", boundary: "gateway.hooks.agent", state: "present" },
});
expect(readExecutionIdentityCall(1)).toEqual({
ingress: {
kind: "webhook",
boundary: "gateway.hooks.agent",
state: "present",
rawSourceRef: "gmail-source",
},
});
});
});
test("returns visible suppression without admitting a hook run", async () => {
await writeHookTransformModule(
"suppress.mjs",
"export default function suppress() { return null; }",
);
testState.hooksConfig = {
enabled: true,
token: HOOK_TOKEN,
mappings: [
{
id: "suppressed-source",
match: { path: "suppressed" },
action: "agent",
messageTemplate: "private {{payload.subject}}",
transform: { module: "suppress.mjs" },
},
],
};
await withGatewayServer(async ({ port }) => {
cronIsolatedRun.mockClear();
const response = await postHook(
port,
"/hooks/suppressed",
{ subject: "secret" },
"suppressed-source",
);
expect(response.status).toBe(204);
expect(cronIsolatedRun).not.toHaveBeenCalled();
expect(peekSystemEventEntries(resolveMainSessionKeyFromConfig())).toEqual([]);
});
});
test("shares one pending persistent dispatch without losing its session target", async () => {
testState.hooksConfig = {
enabled: true,
@@ -598,6 +598,7 @@ export function createHooksRequestHandler(
model: action.model,
thinking: action.thinking,
timeoutSeconds: action.timeoutSeconds,
mappingId: action.mappingId,
allowUnsafeExternalContent: action.allowUnsafeExternalContent,
externalContentSource: resolveMappedHookExternalContentSource({
subPath,
+8
View File
@@ -484,6 +484,14 @@ export function createGatewayHooksRequestHandler(params: {
// cannot starve them. Aggregate capacity stays bounded by the lane
// group that owns both lanes.
lane: CommandLane.HookDispatch,
executionIdentity: {
ingress: {
kind: "webhook",
boundary: "gateway.hooks.agent",
state: "present",
...(acceptedValue.mappingId ? { rawSourceRef: acceptedValue.mappingId } : {}),
},
},
abortSignal: startupAbortController.signal,
onExecutionStarted: () => {
// Existing runner-entry callbacks are the final owner-boundary fence:
@@ -103,6 +103,43 @@ describe("OpenClaw database maintenance schema validation", () => {
}
});
it("keeps lifecycle bindings additive and keyed only by canonical owner identity", () => {
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(
"CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (",
);
const endMarker = ") STRICT;";
const end = start >= 0 ? OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start) : -1;
expect(start).toBeGreaterThanOrEqual(0);
expect(end).toBeGreaterThan(start);
const previousSchema = `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`;
const database = createGlobalDatabase();
try {
expect(() =>
assertSqliteSchemaContains(database, "previous global schema", previousSchema),
).not.toThrow();
expect(
database.prepare("PRAGMA table_info(execution_owner_lifecycle_bindings)").all(),
).toEqual([
{ cid: 0, name: "owner_kind", type: "TEXT", notnull: 1, dflt_value: null, pk: 1 },
{ cid: 1, name: "owner_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 2 },
{ cid: 2, name: "context_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 },
{ cid: 3, name: "execution_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 },
]);
expect(
database
.prepare(
`SELECT COUNT(*) AS count
FROM sqlite_schema
WHERE type = 'index' AND tbl_name = 'execution_owner_lifecycle_bindings'
AND sql IS NOT NULL`,
)
.get(),
).toEqual({ count: 0 });
} finally {
database.close();
}
});
it("accepts compatible future columns in shared-state and agent databases", () => {
const globalDatabase = createGlobalDatabase();
const agentDatabase = createAgentDatabase();
+1
View File
@@ -18,6 +18,7 @@ export const FIRST_USE_STATE_TABLES = [
"node_worker_launches",
"operator_approval_execution_identities",
"execution_decision_facts",
"execution_owner_lifecycle_bindings",
"outbound_message_execution_bindings",
"outbound_message_progress",
] as const;
+8
View File
@@ -700,6 +700,13 @@ export interface ExecutionIdentityContexts {
run_id: string;
}
export interface ExecutionOwnerLifecycleBindings {
context_id: string;
execution_id: string;
owner_id: string;
owner_kind: string;
}
export interface FleetCells {
container_name: string;
created_at_ms: number;
@@ -1871,6 +1878,7 @@ export interface DB {
exec_approvals_config: ExecApprovalsConfig;
execution_decision_facts: ExecutionDecisionFacts;
execution_identity_contexts: ExecutionIdentityContexts;
execution_owner_lifecycle_bindings: ExecutionOwnerLifecycleBindings;
fleet_cells: FleetCells;
flow_runs: FlowRuns;
gateway_boot_lifecycle: GatewayBootLifecycle;
+10
View File
@@ -304,6 +304,16 @@ CREATE INDEX IF NOT EXISTS execution_decision_facts_context_occurred_idx
CREATE INDEX IF NOT EXISTS execution_decision_facts_run_occurred_idx
ON execution_decision_facts (run_id, occurred_at, receipt_id);
-- Exact admission identity stays separate from owner-native lifecycle rows so
-- older readers retain byte-compatible cron/task/flow table definitions.
CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (
owner_kind TEXT NOT NULL,
owner_id TEXT NOT NULL,
context_id TEXT NOT NULL,
execution_id TEXT NOT NULL,
PRIMARY KEY (owner_kind, owner_id)
) STRICT;
CREATE TABLE IF NOT EXISTS session_state_events (
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
dedupe_key TEXT UNIQUE,
+81 -2
View File
@@ -1,7 +1,21 @@
// Persists managed task-flow records through the OpenClaw SQLite state database.
import type { DatabaseSync } from "node:sqlite";
import type { Insertable, Selectable } from "kysely";
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
import {
executionOwnerBindingFromAdmission,
type ExecutionOwnerBindingResult,
} from "../audit/execution-owner-binding.js";
import {
bindExecutionOwnerLifecycleMetadata,
deleteExecutionOwnerLifecycleMetadata,
pruneOrphanedExecutionOwnerLifecycleMetadata,
} from "../audit/execution-owner-lifecycle-binding-store.js";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../infra/kysely-sync.js";
import { normalizeSqliteNumber } from "../infra/sqlite-number.js";
import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js";
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
@@ -9,6 +23,7 @@ import {
closeOpenClawStateDatabase,
openOpenClawStateDatabase,
runOpenClawStateWriteTransaction,
type OpenClawStateDatabaseOptions,
} from "../state/openclaw-state-db.js";
import type { TaskFlowRegistryStoreSnapshot } from "./task-flow-registry.store.types.js";
import {
@@ -42,7 +57,10 @@ function serializeJson(value: unknown): string | null {
return value === undefined ? null : JSON.stringify(value);
}
function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
function resolveFlowSyncMode(row: {
sync_mode: string | null;
shape: string | null;
}): TaskFlowSyncMode {
// Older single_task rows did not persist sync_mode; preserve their mirrored semantics.
const syncMode = parseOptionalTaskFlowSyncMode(row.sync_mode);
if (syncMode) {
@@ -51,6 +69,29 @@ function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
return row.shape === "single_task" ? "task_mirrored" : "managed";
}
function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
return resolveFlowSyncMode(row);
}
function isFlowExecutionOwnerActive(row: {
sync_mode: string | null;
shape: string | null;
status: string;
cancel_requested_at: number | null;
ended_at: number | null;
}): boolean {
const syncMode = resolveFlowSyncMode(row);
const status = parseTaskFlowStatus(row.status);
if (row.cancel_requested_at !== null || row.ended_at !== null) {
return false;
}
// Mirrored `blocked` is derived from a terminal task; managed `blocked`
// remains live while its controller waits for the blocking task.
return syncMode === "task_mirrored"
? status === "queued" || status === "running"
: status === "queued" || status === "running" || status === "waiting" || status === "blocked";
}
function rowToFlowRecord(row: FlowRegistryRow): TaskFlowRecord {
const endedAt = normalizeSqliteNumber(row.ended_at);
const cancelRequestedAt = normalizeSqliteNumber(row.cancel_requested_at);
@@ -233,12 +274,14 @@ export function saveTaskFlowRegistryStateToSqlite(snapshot: TaskFlowRegistryStor
const flowIds = [...snapshot.flows.keys()];
if (flowIds.length === 0) {
executeSqliteQuerySync(db, kysely.deleteFrom("flow_runs"));
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow");
return;
}
pruneFlowsNotInSnapshot({ db, ids: flowIds });
for (const flow of snapshot.flows.values()) {
upsertFlowRow(db, bindFlowRecord(flow));
}
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow");
});
}
@@ -248,12 +291,48 @@ export function upsertTaskFlowRegistryRecordToSqlite(flow: TaskFlowRecord) {
});
}
/** Binds only the exact flow selected before admission; lifecycle settlement stays owner-native. */
export function bindTaskFlowExecution(params: {
admitted: AdmittedRunContext;
flowId: string;
options?: OpenClawStateDatabaseOptions;
}): ExecutionOwnerBindingResult {
const binding = executionOwnerBindingFromAdmission(params.admitted);
if (!binding) {
return "disabled";
}
return runOpenClawStateWriteTransaction(
({ db }) => {
const kysely = getFlowRegistryKysely(db);
const current = executeSqliteQueryTakeFirstSync(
db,
kysely
.selectFrom("flow_runs")
.select(["flow_id", "sync_mode", "shape", "status", "cancel_requested_at", "ended_at"])
.where("flow_id", "=", params.flowId),
);
if (!current || !isFlowExecutionOwnerActive(current)) {
return "missing";
}
return bindExecutionOwnerLifecycleMetadata({
db,
ownerKind: "flow",
ownerId: current.flow_id,
binding,
});
},
params.options,
{ operationLabel: "task.flow.execution-binding" },
);
}
export function deleteTaskFlowRegistryRecordFromSqlite(flowId: string) {
withWriteTransaction(({ db }) => {
executeSqliteQuerySync(
db,
getFlowRegistryKysely(db).deleteFrom("flow_runs").where("flow_id", "=", flowId),
);
deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "flow", ownerIds: [flowId] });
});
}
+114
View File
@@ -2,7 +2,10 @@
import { statSync } from "node:fs";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js";
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
import { openOpenClawStateDatabase } from "../state/openclaw-state-db.js";
import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js";
@@ -14,6 +17,7 @@ import {
setFlowWaiting,
} from "./task-flow-registry.js";
import {
bindTaskFlowExecution,
loadTaskFlowRegistryStateFromSqlite,
loadTaskFlowRegistryStateFromSqliteReadOnly,
saveTaskFlowRegistryStateToSqlite,
@@ -307,13 +311,25 @@ describe("task-flow-registry store runtime", () => {
...createStoredFlow(),
flowId: `flow-large-${index}`,
controllerId: `tests/large-flow-${index}`,
status: "running",
createdAt: index,
updatedAt: index,
cancelRequestedAt: undefined,
endedAt: undefined,
};
flows.set(flow.flowId, flow);
}
saveTaskFlowRegistryStateToSqlite({ flows });
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-flow-prune", runId: "run-flow-prune" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-prune", {
contextId: "context-flow-prune",
executionId: "execution-flow-prune",
}),
};
expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-0" })).toBe("bound");
expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-1199" })).toBe("bound");
const retainedFlows = new Map([...flows].slice(100));
saveTaskFlowRegistryStateToSqlite({ flows: retainedFlows });
@@ -321,6 +337,104 @@ describe("task-flow-registry store runtime", () => {
expect(restored.flows.size).toBe(1_100);
expect(restored.flows.has("flow-large-0")).toBe(false);
expect(restored.flows.has("flow-large-1199")).toBe(true);
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT owner_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'flow'
ORDER BY owner_id`,
)
.all(),
).toEqual([{ owner_id: "flow-large-1199" }]);
});
});
it("binds only source-live flow owners across managed and mirrored lifecycles", async () => {
await withFlowRegistryTempDir(async () => {
const managed: TaskFlowRecord = {
...createStoredFlow(),
flowId: "flow-binding-managed",
status: "blocked",
endedAt: undefined,
cancelRequestedAt: undefined,
};
const mirrored: TaskFlowRecord = {
...createStoredFlow(),
flowId: "flow-binding-mirrored",
syncMode: "task_mirrored",
controllerId: undefined,
status: "running",
endedAt: undefined,
cancelRequestedAt: undefined,
};
const managedTerminal: TaskFlowRecord = {
...managed,
flowId: "flow-binding-managed-terminal",
status: "succeeded",
endedAt: 200,
};
const mirroredTerminal: TaskFlowRecord = {
...mirrored,
flowId: "flow-binding-mirrored-terminal",
status: "blocked",
endedAt: 201,
};
const managedCancelling: TaskFlowRecord = {
...managed,
flowId: "flow-binding-managed-cancelling",
status: "running",
cancelRequestedAt: 199,
};
saveTaskFlowRegistryStateToSqlite({
flows: new Map(
[managed, mirrored, managedTerminal, mirroredTerminal, managedCancelling].map((flow) => [
flow.flowId,
flow,
]),
),
});
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-flow-owner", runId: "run-flow-owner" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-owner", {
contextId: "context-flow-owner",
executionId: "execution-flow-owner",
}),
};
expect(
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
).toBe(false);
expect(bindTaskFlowExecution({ admitted, flowId: managedTerminal.flowId })).toBe("missing");
expect(bindTaskFlowExecution({ admitted, flowId: mirroredTerminal.flowId })).toBe("missing");
expect(bindTaskFlowExecution({ admitted, flowId: managedCancelling.flowId })).toBe("missing");
expect(
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
).toBe(false);
expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("bound");
expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("bound");
saveTaskFlowRegistryStateToSqlite({
flows: new Map([
[managed.flowId, { ...managed, status: "succeeded", endedAt: 210 }],
[mirrored.flowId, { ...mirrored, status: "blocked", endedAt: 211 }],
[managedTerminal.flowId, managedTerminal],
[mirroredTerminal.flowId, mirroredTerminal],
[managedCancelling.flowId, managedCancelling],
]),
});
expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("missing");
expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("missing");
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT owner_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'flow'
ORDER BY owner_id`,
)
.all(),
).toEqual([{ owner_id: managed.flowId }, { owner_id: mirrored.flowId }]);
});
});
+55 -1
View File
@@ -1,7 +1,21 @@
// Persists task registry records and events through the OpenClaw SQLite state database.
import type { DatabaseSync } from "node:sqlite";
import type { Insertable, Selectable } from "kysely";
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
import {
executionOwnerBindingFromAdmission,
type ExecutionOwnerBindingResult,
} from "../audit/execution-owner-binding.js";
import {
bindExecutionOwnerLifecycleMetadata,
deleteExecutionOwnerLifecycleMetadata,
pruneOrphanedExecutionOwnerLifecycleMetadata,
} from "../audit/execution-owner-lifecycle-binding-store.js";
import {
executeSqliteQuerySync,
executeSqliteQueryTakeFirstSync,
getNodeSqliteKysely,
} from "../infra/kysely-sync.js";
import { assertSqliteTableIntegrity } from "../infra/sqlite-integrity.js";
import { normalizeSqliteNumber } from "../infra/sqlite-number.js";
import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js";
@@ -13,6 +27,7 @@ import {
openOpenClawStateDatabase,
runOpenClawStateWriteTransaction,
type OpenClawStateDatabase,
type OpenClawStateDatabaseOptions,
} from "../state/openclaw-state-db.js";
import { parseDeliveryContextJson, parseSqliteJsonValue } from "./task-registry.sqlite.shared.js";
import type { TaskRegistryStoreSnapshot } from "./task-registry.store.types.js";
@@ -329,6 +344,7 @@ function deleteTaskRowsWithDeliveryState(db: DatabaseSync, taskId: string): void
kysely.deleteFrom("task_delivery_state").where("task_id", "=", taskId),
);
executeSqliteQuerySync(db, kysely.deleteFrom("task_runs").where("task_id", "=", taskId));
deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "task", ownerIds: [taskId] });
}
function openTaskRegistryDatabase(): TaskRegistryDatabase {
@@ -435,6 +451,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho
if (taskIds.length === 0) {
executeSqliteQuerySync(db, kysely.deleteFrom("task_delivery_state"));
executeSqliteQuerySync(db, kysely.deleteFrom("task_runs"));
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task");
return;
}
pruneRowsNotInSnapshot({
@@ -462,6 +479,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho
for (const state of snapshot.deliveryStates.values()) {
replaceTaskDeliveryStateRow(db, bindTaskDeliveryState(state));
}
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task");
});
}
@@ -471,6 +489,42 @@ export function upsertTaskRegistryRecordToSqlite(task: TaskRecord) {
});
}
/** Binds only the exact task row selected before admission; runId is never a join key. */
export function bindTaskRunExecution(params: {
admitted: AdmittedRunContext;
taskId: string;
options?: OpenClawStateDatabaseOptions;
}): ExecutionOwnerBindingResult {
const binding = executionOwnerBindingFromAdmission(params.admitted);
if (!binding) {
return "disabled";
}
return runOpenClawStateWriteTransaction(
({ db }) => {
const kysely = getTaskRegistryKysely(db);
const current = executeSqliteQueryTakeFirstSync(
db,
kysely
.selectFrom("task_runs")
.select(["task_id", "status", "ended_at"])
.where("task_id", "=", params.taskId),
);
const status = current ? parseTaskStatus(current.status) : undefined;
if (!current || (status !== "queued" && status !== "running") || current.ended_at !== null) {
return "missing";
}
return bindExecutionOwnerLifecycleMetadata({
db,
ownerKind: "task",
ownerId: current.task_id,
binding,
});
},
params.options,
{ operationLabel: "task.run.execution-binding" },
);
}
export function upsertTaskWithDeliveryStateToSqlite(params: {
task: TaskRecord;
deliveryState?: TaskDeliveryState;
+88
View File
@@ -3,6 +3,8 @@ import { statSync } from "node:fs";
import path from "node:path";
import type { DatabaseSync } from "node:sqlite";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js";
import { emitAgentEvent } from "../infra/agent-events.js";
import {
executeSqliteQuerySync,
@@ -14,6 +16,7 @@ import { readSqliteNumberPragma } from "../infra/sqlite-pragma.test-support.js";
import { resetLogger, setLoggerOverride } from "../logging/logger.js";
import { loggingState } from "../logging/state.js";
import { createWarnLogCapture } from "../logging/test-helpers/warn-log-capture.js";
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
import {
closeOpenClawStateDatabase,
@@ -42,6 +45,7 @@ import {
type TaskRegistryObserverEvent,
} from "./task-registry.store.js";
import {
bindTaskRunExecution,
loadTaskRegistryStateFromSqlite,
loadTaskRegistryStateFromSqliteReadOnly,
loadTaskRegistryStateFromSqliteReadOnlyResult,
@@ -1249,6 +1253,15 @@ describe("task-registry store runtime", () => {
}
saveTaskRegistryStateToSqlite({ tasks, deliveryStates });
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-task-prune", runId: "run-task-prune" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-prune", {
contextId: "context-task-prune",
executionId: "execution-task-prune",
}),
};
expect(bindTaskRunExecution({ admitted, taskId: "task-large-0" })).toBe("bound");
expect(bindTaskRunExecution({ admitted, taskId: "task-large-1199" })).toBe("bound");
const retainedTasks = new Map([...tasks].slice(100));
const retainedDeliveryStates = new Map([...deliveryStates].slice(100));
saveTaskRegistryStateToSqlite({
@@ -1261,6 +1274,81 @@ describe("task-registry store runtime", () => {
expect(restored.deliveryStates.size).toBe(1_100);
expect(restored.tasks.has("task-large-0")).toBe(false);
expect(restored.tasks.has("task-large-1199")).toBe(true);
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT owner_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'task'
ORDER BY owner_id`,
)
.all(),
).toEqual([{ owner_id: "task-large-1199" }]);
},
);
});
it("binds only live task owners and retains their metadata after terminalization", async () => {
await withOpenClawTestState(
{ layout: "state-only", prefix: "openclaw-task-binding-owner-" },
async () => {
const active = { ...createStoredTask(), taskId: "task-binding-active" };
const terminal: TaskRecord = {
...createStoredTask(),
taskId: "task-binding-terminal",
status: "succeeded",
endedAt: 200,
};
const stale: TaskRecord = {
...createStoredTask(),
taskId: "task-binding-stale",
endedAt: 199,
};
saveTaskRegistryStateToSqlite({
tasks: new Map([
[active.taskId, active],
[terminal.taskId, terminal],
[stale.taskId, stale],
]),
deliveryStates: new Map(),
});
const admitted: AdmittedRunContext = {
operationalRunInstance: { instanceId: "instance-task-owner", runId: "run-task-owner" },
executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-owner", {
contextId: "context-task-owner",
executionId: "execution-task-owner",
}),
};
expect(
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
).toBe(false);
expect(bindTaskRunExecution({ admitted, taskId: terminal.taskId })).toBe("missing");
expect(bindTaskRunExecution({ admitted, taskId: stale.taskId })).toBe("missing");
expect(
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
).toBe(false);
expect(bindTaskRunExecution({ admitted, taskId: active.taskId })).toBe("bound");
const finished = { ...active, status: "succeeded" as const, endedAt: 210 };
saveTaskRegistryStateToSqlite({
tasks: new Map([
[finished.taskId, finished],
[terminal.taskId, terminal],
[stale.taskId, stale],
]),
deliveryStates: new Map(),
});
expect(bindTaskRunExecution({ admitted, taskId: finished.taskId })).toBe("missing");
expect(
openOpenClawStateDatabase()
.db.prepare(
`SELECT owner_id
FROM execution_owner_lifecycle_bindings
WHERE owner_kind = 'task'`,
)
.all(),
).toEqual([{ owner_id: active.taskId }]);
},
);
});
@@ -0,0 +1,555 @@
// QA Lab producer proves cron/task owner receipts through a real mock Gateway.
import { createHash, randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { DatabaseSync } from "node:sqlite";
import { setTimeout as delay } from "node:timers/promises";
import { pathToFileURL } from "node:url";
import {
QA_EVIDENCE_FILENAME,
type QaEvidenceSummaryJson,
} from "../../../../extensions/qa-lab/src/evidence-summary.js";
import { startQaGatewayChild } from "../../../../extensions/qa-lab/src/gateway-child.js";
import { startQaMockOpenAiServer } from "../../../../extensions/qa-lab/src/providers/mock-openai/server.js";
import type { AuditRunInspectResult } from "../../../../packages/gateway-protocol/src/index.js";
import { formatErrorMessage } from "../../../../src/infra/errors.js";
import { createQaScriptEvidenceWriter, type QaScriptEvidenceStatus } from "./script-evidence.js";
const SCENARIO_ID = "autonomous-task-lifecycle-receipts";
const SNAPSHOT_FILE = `${SCENARIO_ID}-summary.json`;
const HOOK_TOKEN = "qa-autonomous-hook-token";
type ProducerOptions = { artifactBase: string; repoRoot: string };
type ProofResult = {
artifacts?: Array<{ filePath: string; kind: string }>;
details?: string;
durationMs: number;
status: QaScriptEvidenceStatus;
};
type ExactOwnerRow = {
context_id: string;
execution_id: string;
run_id: string;
status: string;
};
type OwnerDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle";
function hasSqliteColumns(db: DatabaseSync, table: string, columns: readonly string[]): boolean {
const exists = db
.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?")
.get(table);
if (!exists) {
return false;
}
const present = new Set(
(db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>).map(
(row) => row.name,
),
);
return columns.every((column) => present.has(column));
}
function parseOptions(argv: readonly string[]): ProducerOptions {
const readValue = (name: string) => {
const index = argv.indexOf(name);
return index >= 0 ? argv[index + 1] : undefined;
};
const artifactBase = readValue("--artifact-base");
if (!artifactBase) {
throw new Error("--artifact-base is required");
}
return {
artifactBase: path.resolve(artifactBase),
repoRoot: path.resolve(readValue("--repo-root") ?? process.cwd()),
};
}
function parseJson<T>(raw: string, label: string): T {
try {
return JSON.parse(raw) as T;
} catch (error) {
throw new Error(`${label} was not JSON: ${formatErrorMessage(error)}`);
}
}
function sha256(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function stateDatabasePath(gateway: Awaited<ReturnType<typeof startQaGatewayChild>>): string {
const stateDir = gateway.runtimeEnv.OPENCLAW_STATE_DIR;
if (!stateDir) {
throw new Error("QA Gateway did not expose its isolated state directory");
}
return path.join(stateDir, "state", "openclaw.sqlite");
}
function countExecutionContexts(gateway: Awaited<ReturnType<typeof startQaGatewayChild>>): number {
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
try {
if (!hasSqliteColumns(db, "execution_identity_contexts", ["context_id"])) {
return 0;
}
const row = db.prepare("SELECT COUNT(*) AS count FROM execution_identity_contexts").get() as {
count: number;
};
return row.count;
} finally {
db.close();
}
}
function readCronOwnerRows(
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
jobId: string,
): { cron: ExactOwnerRow; task: ExactOwnerRow } | undefined {
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
try {
if (
!hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) ||
!hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [
"owner_kind",
"owner_id",
"context_id",
"execution_id",
]) ||
!hasSqliteColumns(db, "task_runs", ["task_id"])
) {
return undefined;
}
const cron = db
.prepare(
`SELECT binding.context_id, binding.execution_id, context.run_id, receipt.status
FROM cron_run_receipts AS receipt
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
JOIN execution_identity_contexts AS context
ON context.context_id = binding.context_id
AND context.execution_id = binding.execution_id
WHERE receipt.job_id = ? AND receipt.status != 'running'
ORDER BY receipt.started_at_ms DESC LIMIT 1`,
)
.get(jobId) as ExactOwnerRow | undefined;
const task = db
.prepare(
`SELECT binding.context_id, binding.execution_id, context.run_id, task.status
FROM task_runs AS task
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
JOIN execution_identity_contexts AS context
ON context.context_id = binding.context_id
AND context.execution_id = binding.execution_id
WHERE task.runtime = 'cron' AND task.source_id = ? AND task.ended_at IS NOT NULL
ORDER BY task.created_at DESC LIMIT 1`,
)
.get(jobId) as ExactOwnerRow | undefined;
return cron && task ? { cron, task } : undefined;
} finally {
db.close();
}
}
function readCronOwnerBindingDiagnostic(
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
jobId: string,
): unknown {
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
try {
return {
cron: db
.prepare(
`SELECT binding.context_id, binding.execution_id, receipt.status
FROM cron_run_receipts AS receipt
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
WHERE receipt.job_id = ?
ORDER BY receipt.started_at_ms DESC LIMIT 1`,
)
.get(jobId),
task: db
.prepare(
`SELECT binding.context_id, binding.execution_id, task.status
FROM task_runs AS task
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
WHERE task.runtime = 'cron' AND task.source_id = ?
ORDER BY task.created_at DESC LIMIT 1`,
)
.get(jobId),
};
} finally {
db.close();
}
}
function readCliOwnerRows(
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
runId: string,
): { task: ExactOwnerRow } | undefined {
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
try {
if (
!hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) ||
!hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [
"owner_kind",
"owner_id",
"context_id",
"execution_id",
]) ||
!hasSqliteColumns(db, "task_runs", ["task_id"])
) {
return undefined;
}
const task = db
.prepare(
`SELECT binding.context_id, binding.execution_id, context.run_id, task.status
FROM task_runs AS task
JOIN execution_owner_lifecycle_bindings AS binding
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
JOIN execution_identity_contexts AS context
ON context.context_id = binding.context_id
AND context.execution_id = binding.execution_id
WHERE task.runtime = 'cli' AND task.run_id = ? AND task.ended_at IS NOT NULL
LIMIT 1`,
)
.get(runId) as ExactOwnerRow | undefined;
return task ? { task } : undefined;
} finally {
db.close();
}
}
async function waitFor<T>(label: string, read: () => T | undefined): Promise<T> {
const deadline = Date.now() + 30_000;
while (Date.now() < deadline) {
const value = read();
if (value !== undefined) {
return value;
}
await delay(50);
}
throw new Error(`timed out waiting for ${label}`);
}
function requireOwnerDisplay(result: AuditRunInspectResult, producer: OwnerDisplayProducer) {
const receipt = result.decisionDisplays.find(
(candidate) =>
candidate.provenance.state === "verified" && candidate.provenance.producer === producer,
);
if (
!receipt ||
receipt.enforcement.coverageState !== "attribution-only" ||
receipt.decision.outcome !== "not-applicable"
) {
throw new Error(`inspection omitted exact attribution-only ${producer} display`);
}
return receipt;
}
async function inspectExecution(params: {
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>;
executionId: string;
producers: OwnerDisplayProducer[];
privateSentinels: string[];
}) {
const jsonRaw = await params.gateway.runCli([
"audit",
"--execution",
params.executionId,
"--explain",
"--json",
]);
const json = parseJson<AuditRunInspectResult>(jsonRaw, "owner lifecycle inspection");
for (const producer of params.producers) {
requireOwnerDisplay(json, producer);
}
for (const sentinel of params.privateSentinels) {
if (jsonRaw.includes(sentinel)) {
throw new Error(`owner receipt leaked private sentinel ${sentinel}`);
}
}
const human = await params.gateway.runCli([
"audit",
"--execution",
params.executionId,
"--explain",
]);
for (const producer of params.producers) {
if (!human.includes(`Display producer: ${producer}`)) {
throw new Error(`human inspection omitted ${producer}`);
}
}
return { json, jsonRaw, human };
}
async function runProof(options: ProducerOptions): Promise<string> {
const mock = await startQaMockOpenAiServer();
let gateway: Awaited<ReturnType<typeof startQaGatewayChild>> | undefined;
try {
gateway = await startQaGatewayChild({
repoRoot: options.repoRoot,
useRepoCli: true,
providerBaseUrl: `${mock.baseUrl}/v1`,
providerMode: "mock-openai",
transportBaseUrl: "http://127.0.0.1",
controlUiEnabled: false,
mutateConfig: (config) => ({
...config,
logging: {
...config.logging,
audit: { ...config.logging?.audit, enabled: true, executionIdentity: true },
},
hooks: {
enabled: true,
token: HOOK_TOKEN,
mappings: [
{
id: "qa-suppressed-source",
match: { path: "suppressed" },
action: "agent",
messageTemplate: "PRIVATE-SUPPRESSED-{{payload.value}}",
transform: { module: "suppress.mjs" },
},
],
},
}),
});
const transformDir = path.join(path.dirname(gateway.configPath), "hooks", "transforms");
await fs.mkdir(transformDir, { recursive: true });
await fs.writeFile(
path.join(transformDir, "suppress.mjs"),
"export default function suppress() { return null; }\n",
"utf8",
);
const contextsBeforeSuppression = countExecutionContexts(gateway);
const suppression = await fetch(`${gateway.baseUrl}/hooks/suppressed`, {
method: "POST",
headers: {
Authorization: `Bearer ${HOOK_TOKEN}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ value: randomUUID() }),
});
if (suppression.status !== 204) {
throw new Error(`mapped suppression returned HTTP ${suppression.status}`);
}
await delay(100);
if (countExecutionContexts(gateway) !== contextsBeforeSuppression) {
throw new Error("pre-admission mapping suppression allocated execution identity");
}
const cronSentinel = `PRIVATE-CRON-${randomUUID()}`;
const cronJob = (await gateway.call("cron.add", {
name: "QA autonomous receipt",
enabled: true,
schedule: { kind: "at", at: new Date(Date.now() + 3_600_000).toISOString() },
sessionTarget: "isolated",
wakeMode: "next-heartbeat",
payload: { kind: "agentTurn", message: `${cronSentinel}: reply CRON-DONE` },
delivery: { mode: "none" },
})) as { id: string };
await gateway.call("cron.run", { id: cronJob.id, mode: "force" });
let cronRows: { cron: ExactOwnerRow; task: ExactOwnerRow };
try {
cronRows = await waitFor("terminal cron/task exact bindings", () =>
readCronOwnerRows(gateway!, cronJob.id),
);
} catch (error) {
throw new Error(
`${formatErrorMessage(error)}; owner rows=${JSON.stringify(readCronOwnerBindingDiagnostic(gateway, cronJob.id))}; gateway logs=${gateway.logs()}`,
{ cause: error },
);
}
if (
cronRows.cron.context_id !== cronRows.task.context_id ||
cronRows.cron.execution_id !== cronRows.task.execution_id
) {
throw new Error("cron receipt/task rows did not retain one exact admitted execution");
}
const cronInspection = await inspectExecution({
gateway,
executionId: cronRows.cron.execution_id,
producers: ["cron-lifecycle", "task-lifecycle"],
privateSentinels: [cronSentinel],
});
const cronCursorPage = parseJson<AuditRunInspectResult>(
await gateway.runCli([
"audit",
"--execution",
cronRows.cron.execution_id,
"--explain",
"--cursor",
"c:0:0",
"--limit",
"1",
"--json",
]),
"cron cursor page",
);
requireOwnerDisplay(cronCursorPage, "cron-lifecycle");
const taskSentinel = `PRIVATE-TASK-${randomUUID()}`;
const cliRunId = `qa-cli-${randomUUID()}`;
const accepted = (await gateway.call(
"agent",
{
sessionKey: "agent:qa:main",
message: `${taskSentinel}: reply TASK-DONE`,
deliver: false,
idempotencyKey: cliRunId,
},
{ expectFinal: false },
)) as { runId: string; status: string };
await gateway.call(
"agent.wait",
{ runId: accepted.runId, timeoutMs: 30_000 },
{ timeoutMs: 35_000 },
);
const cliRows = await waitFor("terminal CLI task exact binding", () =>
readCliOwnerRows(gateway!, accepted.runId),
);
const taskInspection = await inspectExecution({
gateway,
executionId: cliRows.task.execution_id,
producers: ["task-lifecycle"],
privateSentinels: [taskSentinel],
});
const beforeRestart = JSON.stringify({
cron: cronInspection.json,
task: taskInspection.json,
});
await gateway.restartAfterStateMutation(async () => {});
const cronAfter = await inspectExecution({
gateway,
executionId: cronRows.cron.execution_id,
producers: ["cron-lifecycle", "task-lifecycle"],
privateSentinels: [cronSentinel],
});
const taskAfter = await inspectExecution({
gateway,
executionId: cliRows.task.execution_id,
producers: ["task-lifecycle"],
privateSentinels: [taskSentinel],
});
const afterRestart = JSON.stringify({ cron: cronAfter.json, task: taskAfter.json });
if (afterRestart !== beforeRestart) {
throw new Error("owner lifecycle JSON changed across Gateway replacement");
}
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
let genericCount = 0;
try {
if (hasSqliteColumns(db, "execution_decision_facts", ["receipt_id"])) {
genericCount = (
db.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get() as {
count: number;
}
).count;
}
} finally {
db.close();
}
if (genericCount !== 0) {
throw new Error("owner lifecycle rows were duplicated into execution_decision_facts");
}
const snapshotPath = path.join(options.artifactBase, SNAPSHOT_FILE);
await fs.mkdir(options.artifactBase, { recursive: true });
await fs.writeFile(
snapshotPath,
`${JSON.stringify(
{
suppression: { httpStatus: 204, identityAllocation: 0 },
cron: {
contextId: cronRows.cron.context_id,
executionId: cronRows.cron.execution_id,
statuses: [cronRows.cron.status, cronRows.task.status],
displayProducers: ["cron-lifecycle", "task-lifecycle"],
},
task: {
contextId: cliRows.task.context_id,
executionId: cliRows.task.execution_id,
statuses: [cliRows.task.status],
displayProducers: ["task-lifecycle"],
},
cursorCompatibility: { cronPrefixAccepted: true },
genericDuplicateAbsent: true,
byteEquivalentAfterRestart: true,
privacy: { cronPromptAbsent: true, taskPromptAbsent: true },
resultSha256: sha256(afterRestart),
},
null,
2,
)}\n`,
"utf8",
);
return `cron=${cronRows.cron.execution_id}; task=${cliRows.task.execution_id}; suppression=204; restart sha256=${sha256(afterRestart)}`;
} finally {
await gateway?.stop().catch(() => undefined);
await mock.stop();
}
}
async function produceProof(options: ProducerOptions): Promise<ProofResult> {
const startedAt = Date.now();
try {
return {
artifacts: [{ filePath: SNAPSHOT_FILE, kind: "summary" }],
details: await runProof(options),
durationMs: Math.max(1, Date.now() - startedAt),
status: "pass",
};
} catch (error) {
return {
details: formatErrorMessage(error),
durationMs: Math.max(1, Date.now() - startedAt),
status: "fail",
};
}
}
async function runProducer(options: ProducerOptions): Promise<QaEvidenceSummaryJson> {
const writer = createQaScriptEvidenceWriter({
artifactBase: options.artifactBase,
logFileName: `${SCENARIO_ID}.log`,
primaryModel: "mock-openai/gpt-5.6-luna",
providerMode: "mock-openai",
repoRoot: options.repoRoot,
target: {
id: SCENARIO_ID,
title: "Autonomous task lifecycle receipts",
sourcePath: `qa/scenarios/runtime/${SCENARIO_ID}.yaml`,
docsRefs: ["docs/gateway/audit.md", "docs/automation/tasks.md"],
codeRefs: [
"src/audit/execution-decision-receipts.ts",
"src/cron/store/run-receipt-store.ts",
"src/tasks/task-registry.store.sqlite.ts",
"src/tasks/task-flow-registry.store.sqlite.ts",
],
},
});
const result = await produceProof(options);
writer.appendLog(`${result.status}: ${result.details ?? "no details"}\n`);
return await writer.write(result);
}
async function main(argv: readonly string[]) {
const evidence = await runProducer(parseOptions(argv));
const status = evidence.entries[0]?.result.status;
console.log(`Autonomous task lifecycle evidence: ${QA_EVIDENCE_FILENAME}`);
console.log(`Autonomous task lifecycle status: ${status}`);
return status === "pass" ? 0 : 1;
}
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
main(process.argv.slice(2))
.then((exitCode) => {
process.exitCode = exitCode;
})
.catch((error) => {
console.error(formatErrorMessage(error));
process.exitCode = 1;
});
}