diff --git a/docs/automation/cron-jobs.md b/docs/automation/cron-jobs.md index e58cae384cf1..e73a919c03b3 100644 --- a/docs/automation/cron-jobs.md +++ b/docs/automation/cron-jobs.md @@ -559,6 +559,14 @@ Model override note: Gateway can expose HTTP webhook endpoints for external triggers. Enable in config: +A configured mapping `id` is retained only as bounded ingress-source +attribution when that mapping reaches agent admission. It is not an +authenticated service principal or invoker. Direct `/hooks/agent` and requests +authenticated only by the shared hook token stay unattributed unless another +authoritative principal source exists. If a transform returns `null`, the +request keeps its visible HTTP 204 outcome and stops before creating a run, +task, execution identity, or audit receipt. + ```json5 { hooks: { diff --git a/docs/automation/tasks.md b/docs/automation/tasks.md index 81b134e83148..5e8cc896079f 100644 --- a/docs/automation/tasks.md +++ b/docs/automation/tasks.md @@ -405,8 +405,16 @@ A sweeper runs every **60 seconds** (first pass about 5 seconds after gateway st A task may reference a `childSessionKey` (where work runs) and a `requesterSessionKey` (who started it). Its `agentId` identifies the agent executing the work, while the requester and owner fields preserve launch and control context. Sessions are conversation context; tasks are activity tracking on top of that. - A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually. - +A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually. + +When execution identity collection is enabled, OpenClaw also binds the exact +admitted `contextId` and `executionId` to Gateway CLI, ACP, and automation task +rows and their mirrored flow rows. This is inspection provenance only: `runId` +remains correlation, task/flow status remains authoritative, and a missing or +mismatched binding never changes execution or settlement. `openclaw audit +--execution --explain` adapts the existing rows without copying task or +flow content into the generic decision-fact table. + ## Related diff --git a/docs/cli/audit.md b/docs/cli/audit.md index b98173872a60..e4e034ea0b3e 100644 --- a/docs/cli/audit.md +++ b/docs/cli/audit.md @@ -115,7 +115,8 @@ view renders these sections: 3. **Lineage**: parent context or an explicit absent, unknown, or unsupported state. 4. **Decisions**: bounded run-admission and authoritative action-decision - receipts, including terminal operator approvals. + receipts, including terminal operator approvals and exact-bound cron, task, + and task-flow lifecycle rows. 5. **Missing evidence** and **Next steps**. Every field includes `present`, `absent`, `unknown`, or `unsupported`; the CLI @@ -125,6 +126,15 @@ ingress, an absent invoker, and `unattributed` coverage. Its admission receipt says `not-applicable` because no identity-aware policy or grant evaluation was proven. +Lifecycle rows from `cron_run_receipts`, `task_runs`, and `flow_runs` appear as +owner-native, attribution-only receipts when their keyed lifecycle metadata +carries the exact inspected context and execution ids. They contain status and +bounded record references, not prompts, task goals, hook payloads, paths, or raw +errors. Their decision is `not-applicable` because lifecycle attribution does +not prove authorization. +Treat every decision cursor as opaque: numeric and `a:`, `m:`, and `g:` values +remain compatible, while cron/task/flow pages may return `c:`, `t:`, or `f:`. + For Gateway runs, a resolved authenticated profile can make the invoker `present` and coverage `attribution-only`. Paired devices and shared credentials do not establish a person: without a durable profile the invoker stays absent, @@ -340,7 +350,8 @@ and no identity context; its required `decisionDisplays` array is empty until the caller selects an execution id. For one selected context, receipt paging starts with admission, then reads owner-native terminal approvals, merges outbound progress and terminal records, -and finally reads generic facts for boundaries without a native durable record. +then reads generic facts and the cron, task, and flow lifecycle owners. The +complete order is admission, approval, message, generic, cron, task, then flow. The merge is deterministic across restart and rejects a cursor whose exact owner row has expired. Approval and message selectors use the opaque `approval-decision:` and `message-decision:` namespaces minted from the same diff --git a/docs/gateway/audit.md b/docs/gateway/audit.md index b623421932ae..ec323fc83b8e 100644 --- a/docs/gateway/audit.md +++ b/docs/gateway/audit.md @@ -43,6 +43,16 @@ platform-send start use a lazy progress companion, while terminal message rows remain in the activity ledger. Run inspection merges both sources directly; neither is copied into the generic decision-fact table. +Scheduled runs, background tasks, and task flows are owner-native sources too. +After exact run admission, a lazy lifecycle metadata table binds the admitted +context and execution ids to the canonical `cron_run_receipts`, `task_runs`, or +`flow_runs` row. Inspection joins that metadata to the owner row directly and +preserves its status, including skipped, failed, timed-out, cancelled, blocked, +and lost outcomes. A `runId` alone never joins one of these rows to an +execution. Legacy, missing, deleted, corrupt, or mismatched bindings remain +unknown or absent; they never change task behavior and are never copied into +`execution_decision_facts`. + ## Run identity inspection Execution identity recording is off by default, including on fresh installs @@ -160,6 +170,15 @@ this boundary. A run becomes `attribution-only` only when an authoritative ingress supplies an invoker fact. Neither state means that identity affected an allow or deny decision. +Configured webhook mapping ids identify only the matched ingress source. They +do not authenticate a person, service, or invoker. Shared hook authentication +and direct `/hooks/agent` requests therefore remain unattributed unless another +authoritative principal producer exists. A mapping transform that suppresses a +request before admission returns its normal HTTP response but creates no run, +execution identity, task, or decision receipt. Restart recovery records system +attribution only after the current durable recovery owner admits the exact +attempt. + Authenticated Gateway attach records immutable audit facts once. Session creation separately reads the live canonical durable profile id so a profile link performed after attach cannot orphan session ownership. Ordinary session @@ -200,6 +219,12 @@ message-policy, or turn-capability denial that changed the result is `enforced`. Portable actions and early suppressions without a durable owner record use the generic fact owner on the same audit-writer FIFO. +Cron, task, and flow lifecycle receipts are `attribution-only` and have a +`not-applicable` decision outcome. They report what the authoritative lifecycle +owner retained; they do not claim an authorization decision. Their cursors are +opaque and source-specific. Existing numeric cursors and `a:`, `m:`, and `g:` +cursors remain accepted; newer owner stages use `c:`, `t:`, and `f:`. + When the same `runId` has a retained terminal row in `operator_approvals`, the inspector also reads its owner-local `operator_approval_execution_identities` binding. Only an exact context, execution, and run tuple projects the approval diff --git a/packages/gateway-protocol/src/schema/audit-run.test.ts b/packages/gateway-protocol/src/schema/audit-run.test.ts index ac91e8b89264..eeefca0ab151 100644 --- a/packages/gateway-protocol/src/schema/audit-run.test.ts +++ b/packages/gateway-protocol/src/schema/audit-run.test.ts @@ -116,6 +116,14 @@ describe("audit run inspection protocol", () => { provenance: { state: "verified", producer: "receipt-owner" }, }), ).toBe(false); + for (const producer of ["cron-lifecycle", "task-lifecycle", "flow-lifecycle"]) { + expect( + validate.Check({ + ...display, + provenance: { state: "verified", producer }, + }), + ).toBe(true); + } }); it("exports selector and discovery-pagination invariants for generated clients", () => { diff --git a/packages/gateway-protocol/src/schema/audit-run.ts b/packages/gateway-protocol/src/schema/audit-run.ts index fdef097116e8..d69a81c387a5 100644 --- a/packages/gateway-protocol/src/schema/audit-run.ts +++ b/packages/gateway-protocol/src/schema/audit-run.ts @@ -209,6 +209,9 @@ const DecisionReceiptDisplayProvenanceV1Schema = Type.Union([ Type.Literal("run-admission"), Type.Literal("operator-approval"), Type.Literal("message-delivery"), + Type.Literal("cron-lifecycle"), + Type.Literal("task-lifecycle"), + Type.Literal("flow-lifecycle"), ]), }), closedObject({ state: Type.Literal("unverified") }), diff --git a/qa/scenarios/runtime/autonomous-task-lifecycle-receipts.yaml b/qa/scenarios/runtime/autonomous-task-lifecycle-receipts.yaml new file mode 100644 index 000000000000..10309a1952e6 --- /dev/null +++ b/qa/scenarios/runtime/autonomous-task-lifecycle-receipts.yaml @@ -0,0 +1,36 @@ +title: Autonomous task lifecycle receipts + +scenario: + id: autonomous-task-lifecycle-receipts + surface: gateway + coverage: + primary: + - gateway.identity-and-presence-apis + - automation.openclaw-tasks-flow + objective: Verify exact-bound cron, task, and flow owner lifecycle rows are explained without generic decision-fact copies. + successCriteria: + - A mapped hook transform suppression returns HTTP 204 before admission and creates no execution identity. + - A real forced scheduled agent run binds one exact context/execution to its cron receipt and task row. + - A real Gateway agent action binds one exact context/execution to its CLI task row; focused owner tests cover bound flow rows. + - JSON and human audit inspection expose only bounded attribution-only lifecycle displays with closed producers, accept the cron cursor prefix, and omit task/prompt content. + - A replacement Gateway returns byte-equivalent JSON and the generic decision-fact table remains empty. + docsRefs: + - docs/gateway/audit.md + - docs/cli/audit.md + - docs/automation/cron-jobs.md + - docs/automation/tasks.md + - docs/concepts/qa-e2e-automation.md + codeRefs: + - src/audit/execution-decision-receipts.ts + - src/cron/store/run-receipt-store.ts + - src/tasks/task-registry.store.sqlite.ts + - src/tasks/task-flow-registry.store.sqlite.ts + - src/gateway/hooks-mapping.ts + - test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts + execution: + kind: script + path: test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts + summary: Starts an ephemeral Gateway and mock provider, proves pre-admission suppression, runs cron and CLI task lifecycles, replaces the Gateway, and compares inspection output. + args: + - --artifact-base + - ${outputDir} diff --git a/src/acp/control-plane/manager.background-task.test.ts b/src/acp/control-plane/manager.background-task.test.ts index 0f1c5e8ca9c7..c7c827becb45 100644 --- a/src/acp/control-plane/manager.background-task.test.ts +++ b/src/acp/control-plane/manager.background-task.test.ts @@ -1,9 +1,25 @@ /** Regression coverage for ACP background-task summary truncation boundaries. */ -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it } from "vitest"; +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js"; import type { OpenClawConfig } from "../../config/types.openclaw.js"; +import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js"; +import { + closeOpenClawStateDatabaseForTest, + openOpenClawStateDatabase, +} from "../../state/openclaw-state-db.js"; +import { findTaskByRunId } from "../../tasks/task-registry.js"; +import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js"; +import { + resetTaskRegistryForTests, + resetTaskFlowRegistryForTests, +} from "../../tasks/task-runtime.test-helpers.js"; +import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js"; import { AcpRuntimeError } from "../runtime/errors.js"; import { appendBackgroundTaskProgressSummary, + bindBackgroundTaskExecution, + createBackgroundTaskRecord, resolveBackgroundTaskContext, resolveBackgroundTaskFailureStatus, } from "./manager.background-task.js"; @@ -13,6 +29,12 @@ import type { AcpSessionManagerDeps } from "./manager.types.js"; // U+1F99E (🦞) is a surrogate pair in UTF-16; a raw .slice() boundary can split it. const LOBSTER = "🦞"; +afterEach(() => { + closeOpenClawStateDatabaseForTest(); + resetTaskRegistryForTests({ persist: false }); + resetTaskFlowRegistryForTests({ persist: false }); +}); + const HIGH_SURROGATE_WITHOUT_LOW = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])/; function fakeDeps(): AcpSessionManagerDeps { @@ -85,3 +107,128 @@ describe("resolveBackgroundTaskFailureStatus", () => { ).toBe("failed"); }); }); + +describe("ACP background task execution binding", () => { + it("binds the exact admitted execution only at prompt submission", async () => { + await withOpenClawTestState( + { layout: "state-only", prefix: "openclaw-acp-execution-binding-" }, + async () => { + resetTaskRegistryForTests(); + resetTaskFlowRegistryForTests(); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", { + contextId: "context-acp", + executionId: "execution-acp", + }), + }; + const record = createBackgroundTaskRecord( + { + requesterSessionKey: "agent:main:main", + childSessionKey: "agent:qa:child", + runId: "run-acp", + task: "private", + }, + 100, + ); + const task = findTaskByRunId("run-acp"); + if (!record || !task?.parentFlowId) { + throw new Error("expected ACP task and owner flow"); + } + const db = openOpenClawStateDatabase().db; + expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false); + + bindBackgroundTaskExecution(record, admitted); + + expect( + db + .prepare( + `SELECT owner_kind, owner_id, context_id, execution_id + FROM execution_owner_lifecycle_bindings + ORDER BY owner_kind`, + ) + .all(), + ).toEqual([ + { + owner_kind: "flow", + owner_id: task.parentFlowId, + context_id: "context-acp", + execution_id: "execution-acp", + }, + { + owner_kind: "task", + owner_id: task.taskId, + context_id: "context-acp", + execution_id: "execution-acp", + }, + ]); + }, + ); + }); + + it.each(["missing", "mismatched"] as const)( + "does not bind a live parent flow when the task owner is %s", + async (taskOwnerState) => { + await withOpenClawTestState( + { layout: "state-only", prefix: `openclaw-acp-${taskOwnerState}-task-binding-` }, + async () => { + resetTaskRegistryForTests(); + resetTaskFlowRegistryForTests(); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", { + contextId: "context-acp", + executionId: "execution-acp", + }), + }; + const record = createBackgroundTaskRecord( + { + requesterSessionKey: "agent:main:main", + childSessionKey: "agent:qa:child", + runId: "run-acp", + task: "private", + }, + 100, + ); + const task = findTaskByRunId("run-acp"); + if (!record || !task?.parentFlowId) { + throw new Error("expected ACP task and owner flow"); + } + const db = openOpenClawStateDatabase().db; + if (taskOwnerState === "missing") { + db.prepare("DELETE FROM task_runs WHERE task_id = ?").run(task.taskId); + } else { + expect( + bindTaskRunExecution({ + taskId: task.taskId, + admitted: { + operationalRunInstance: { instanceId: "instance-other", runId: "run-other" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-other", { + contextId: "context-other", + executionId: "execution-other", + }), + }, + }), + ).toBe("bound"); + } + + bindBackgroundTaskExecution(record, admitted); + + if (taskOwnerState === "missing") { + expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false); + } else { + expect( + db + .prepare( + `SELECT context_id, execution_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'flow' AND owner_id = ?`, + ) + .get(task.parentFlowId), + ).toBeUndefined(); + } + }, + ); + }, + ); +}); diff --git a/src/acp/control-plane/manager.background-task.ts b/src/acp/control-plane/manager.background-task.ts index 4781de7df3dd..3b3d0999350b 100644 --- a/src/acp/control-plane/manager.background-task.ts +++ b/src/acp/control-plane/manager.background-task.ts @@ -1,5 +1,7 @@ /** Mirrors child ACP turns into detached-task status for requester-facing progress. */ import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice"; +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import { isRetainedExecutionOwnerBinding } from "../../audit/execution-owner-binding.js"; import type { OpenClawConfig } from "../../config/types.openclaw.js"; import { logVerbose } from "../../globals.js"; import { @@ -9,6 +11,8 @@ import { startTaskRunByRunId, } from "../../tasks/detached-task-runtime.js"; import { resolveRequiredCompletionTerminalResult } from "../../tasks/task-completion-contract.js"; +import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js"; +import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js"; import { deliveryContextFromSession, type DeliveryContext, @@ -31,6 +35,11 @@ type BackgroundTaskContext = { task: string; }; +type BackgroundTaskRecord = { + taskId: string; + parentFlowId?: string; +}; + /** Produces the bounded task label shown for a child ACP background run. */ function summarizeBackgroundTaskText(text: string): string { const normalized = normalizeText(text) ?? "ACP background task"; @@ -133,7 +142,7 @@ export function resolveBackgroundTaskContext(params: { export function createBackgroundTaskRecord( context: BackgroundTaskContext, startedAt: number, -): void { +): BackgroundTaskRecord | undefined { try { const task = createRunningTaskRun({ runtime: "acp", @@ -151,11 +160,37 @@ export function createBackgroundTaskRecord( logVerbose( `acp-manager: failed creating background task for ${context.runId}: persist_failed`, ); + return undefined; } + return { + taskId: task.taskId, + ...(task.parentFlowId ? { parentFlowId: task.parentFlowId } : {}), + }; } catch (error) { logVerbose( `acp-manager: failed creating background task for ${context.runId}: ${String(error)}`, ); + return undefined; + } +} + +/** Links ACP owner rows only when the runtime reaches its prompt-submitted boundary. */ +export function bindBackgroundTaskExecution( + record: BackgroundTaskRecord, + admitted: AdmittedRunContext, +): void { + try { + const taskResult = bindTaskRunExecution({ admitted, taskId: record.taskId }); + const flowResult = record.parentFlowId + ? isRetainedExecutionOwnerBinding(taskResult) + ? bindTaskFlowExecution({ admitted, flowId: record.parentFlowId }) + : taskResult + : undefined; + if ([taskResult, flowResult].some((result) => result === "mismatch" || result === "missing")) { + logVerbose("acp-manager: exact task execution binding was not retained"); + } + } catch (error) { + logVerbose(`acp-manager: failed binding background task execution: ${String(error)}`); } } diff --git a/src/acp/control-plane/manager.turn-runner.ts b/src/acp/control-plane/manager.turn-runner.ts index 8fa8e95cae1d..5f5330d8c2e1 100644 --- a/src/acp/control-plane/manager.turn-runner.ts +++ b/src/acp/control-plane/manager.turn-runner.ts @@ -16,6 +16,7 @@ import { } from "./manager.backend-failover.js"; import { appendBackgroundTaskProgressSummary, + bindBackgroundTaskExecution, createBackgroundTaskRecord, markBackgroundTaskRunning, markBackgroundTaskTerminal, @@ -87,9 +88,10 @@ export async function runManagerTurn(params: { text: input.text, }) : null; - if (taskContext) { - createBackgroundTaskRecord(taskContext, turnStartedAt); - } + const taskRecord = taskContext + ? createBackgroundTaskRecord(taskContext, turnStartedAt) + : undefined; + let taskExecutionBound = false; let taskProgressSummary = ""; const initialResolution = params.resolveSession({ cfg: input.cfg, @@ -267,6 +269,10 @@ export async function runManagerTurn(params: { onBeforePrompt: input.onBeforePrompt, onPromptStarted: async ({ authoritative }) => { promptStarted = authoritative; + if (authoritative && taskRecord && !taskExecutionBound) { + taskExecutionBound = true; + bindBackgroundTaskExecution(taskRecord, input.admittedRunContext); + } try { await input.onLifecycle?.({ type: "prompt_submitted", diff --git a/src/agents/agent-command-admission-facts.test.ts b/src/agents/agent-command-admission-facts.test.ts new file mode 100644 index 000000000000..b5c4f3662a5b --- /dev/null +++ b/src/agents/agent-command-admission-facts.test.ts @@ -0,0 +1,32 @@ +import { describe, expect, it } from "vitest"; +import { + attachAgentCommandRecoveryAdmissionFacts, + getAgentCommandAdmissionFacts, +} from "./agent-command-admission-facts.js"; + +describe("agent command admission facts", () => { + it("records restart recovery as owner-bound system attribution", () => { + const runContext = {}; + attachAgentCommandRecoveryAdmissionFacts(runContext); + expect(getAgentCommandAdmissionFacts(runContext)).toEqual({ + ingress: { + kind: "recovery", + boundary: "gateway.main-session-recovery", + state: "present", + }, + invoker: { + state: "present", + kind: "system", + rawPrincipalRef: "openclaw.main-session-recovery", + }, + assurance: [ + { + kind: "runtime-binding", + rawEvidenceRef: "gateway.main-session-recovery-owner", + strength: "boundary-verified", + }, + ], + }); + expect(getAgentCommandAdmissionFacts({})).toBeUndefined(); + }); +}); diff --git a/src/agents/agent-command-admission-facts.ts b/src/agents/agent-command-admission-facts.ts index 0da29dec541e..eb295917fcdb 100644 --- a/src/agents/agent-command-admission-facts.ts +++ b/src/agents/agent-command-admission-facts.ts @@ -18,3 +18,26 @@ export function getAgentCommandAdmissionFacts( ): AgentCommandAdmissionFacts | undefined { return factsByIngress.get(ingress); } + +/** Records the exact system attribution only after the recovery owner admits the attempt. */ +export function attachAgentCommandRecoveryAdmissionFacts(ingress: object): void { + attachAgentCommandAdmissionFacts(ingress, { + ingress: { + kind: "recovery", + boundary: "gateway.main-session-recovery", + state: "present", + }, + invoker: { + state: "present", + kind: "system", + rawPrincipalRef: "openclaw.main-session-recovery", + }, + assurance: [ + { + kind: "runtime-binding", + rawEvidenceRef: "gateway.main-session-recovery-owner", + strength: "boundary-verified", + }, + ], + }); +} diff --git a/src/agents/agent-command-execution-identity.test.ts b/src/agents/agent-command-execution-identity.test.ts index aeea1f6caf0d..dd040ea53196 100644 --- a/src/agents/agent-command-execution-identity.test.ts +++ b/src/agents/agent-command-execution-identity.test.ts @@ -43,6 +43,36 @@ describe("sanitizePublicAgentCommandIngressOpts", () => { }); describe("Gateway agent command execution identity", () => { + it("runs owner binding only after the awaited admission callback settles", async () => { + const events: string[] = []; + const prepared = prepareAgentCommandExecutionIdentity({ + opts: { + message: "bind after admission", + onAdmittedRunContext: async () => { + await Promise.resolve(); + events.push("admitted"); + }, + onPostAdmittedRunContext: () => { + events.push("owner-bound"); + }, + }, + prepared: { + cfg: { logging: { audit: { enabled: true, executionIdentity: true } } }, + runId: "run-post-admission", + sessionAgentId: "main", + sessionId: "session-post-admission", + }, + ingress: { kind: "api", boundary: "agent-command.from-ingress", state: "unknown" }, + lifecycleGeneration: "generation-1", + }); + + const admitted = await prepared.admit("embedded"); + await prepared.admit("embedded"); + + expect(admitted.executionIdentityToken).toBeDefined(); + expect(events).toEqual(["admitted", "owner-bound"]); + }); + it("preserves trusted spawn facts across internal option preparation", () => { const facts = { ingress: { diff --git a/src/agents/agent-command-execution-identity.ts b/src/agents/agent-command-execution-identity.ts index 9e2e33de4f10..d9cdcee499d2 100644 --- a/src/agents/agent-command-execution-identity.ts +++ b/src/agents/agent-command-execution-identity.ts @@ -3,6 +3,7 @@ import type { ExecutionIdentityAdmissionToken, } from "../audit/execution-identity-admission.js"; import { executionIdentitySpawnAdmission } from "../audit/execution-identity-spawn-admission.js"; +import { withPostAdmissionExecutionOwnerBinding } from "../audit/execution-owner-binding.js"; import type { OpenClawConfig } from "../config/types.openclaw.js"; import { formatErrorMessage } from "../infra/errors.js"; import { createSubsystemLogger } from "../logging/subsystem.js"; @@ -170,9 +171,12 @@ export function prepareAgentCommandExecutionIdentity(params: { }, }; const spawnFacts = readAgentCommandExecutionIdentitySpawnFacts(opts); - return spawnFacts + const preparedAdmission = spawnFacts ? prepareAgentCommandRunAdmissionWithSpawnFacts(admissionParams, spawnFacts) : executionIdentity.prepare(admissionParams); + return opts.onPostAdmittedRunContext + ? withPostAdmissionExecutionOwnerBinding(preparedAdmission, opts.onPostAdmittedRunContext) + : preparedAdmission; } export function sanitizePublicAgentCommandIngressOpts( @@ -187,6 +191,7 @@ export function sanitizePublicAgentCommandIngressOpts( operationalRunInstance: undefined, cronCreatorAuthorityCapability: undefined, onAdmittedRunContext: undefined, + onPostAdmittedRunContext: undefined, }); } diff --git a/src/agents/command/types.ts b/src/agents/command/types.ts index 282605b68ffe..ef918462e4d2 100644 --- a/src/agents/command/types.ts +++ b/src/agents/command/types.ts @@ -209,6 +209,10 @@ export type AgentCommandOpts = { onAdmittedRunContext?: ( context: import("../admitted-run-context.js").AdmittedRunContext, ) => void | Promise; + /** Private owner binding hook invoked only after exact admission has resolved. */ + onPostAdmittedRunContext?: ( + context: import("../admitted-run-context.js").AdmittedRunContext, + ) => void; /** Called when the actual run model is selected, including fallback retries. */ onActiveModelSelected?: (ctx: { provider: string; model: string }) => void | Promise; /** Called when every candidate in the run's model fallback chain failed. */ @@ -241,6 +245,7 @@ export type AgentCommandIngressOpts = Omit< | "operationalRunInstance" | "cronCreatorAuthorityCapability" | "onAdmittedRunContext" + | "onPostAdmittedRunContext" > & { /** Trusted sender identity bit for command/channel-action auth; defaults false for ingress. */ senderIsOwner?: boolean; @@ -259,4 +264,5 @@ export type AgentCommandGatewayIngressOpts = AgentCommandIngressOpts & | "operationalRunInstance" | "cronCreatorAuthorityCapability" | "onAdmittedRunContext" + | "onPostAdmittedRunContext" >; diff --git a/src/audit/execution-decision-facts.test.ts b/src/audit/execution-decision-facts.test.ts index b255e409d532..c8cd40e37a59 100644 --- a/src/audit/execution-decision-facts.test.ts +++ b/src/audit/execution-decision-facts.test.ts @@ -27,6 +27,7 @@ import { type ExecutionIdentityAdmissionEnvelope, } from "./execution-identity-admission.js"; import { processExecutionIdentityAdmissionWork } from "./execution-identity-context.js"; +import { bindExecutionOwnerLifecycleMetadata } from "./execution-owner-lifecycle-binding-store.js"; import { configureMessageActionDecisionSink, recordMessageActionDecision, @@ -256,6 +257,79 @@ describe("execution decision facts", () => { expect(receiptSearch).not.toContain(encodeURIComponent(messageReceipt?.receiptId ?? "")); }); + it("projects exact-bound cron, task, and flow owner rows without generic facts", () => { + const database = databaseOptions(); + const context = seedExecutionContext(database); + const db = openOpenClawStateDatabase(database).db; + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run( + "cron-receipt-1", + "default", + "job-1", + "revision-1", + "main", + context.runId, + "ok", + 1, + 60, + 70, + ); + db.prepare( + `INSERT INTO task_runs ( + task_id, runtime, owner_key, scope_kind, task, status, delivery_status, + notify_policy, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run( + "task-1", + "cron", + "owner-1", + "session", + "private task text", + "succeeded", + "not-requested", + "never", + 61, + ); + db.prepare( + `INSERT INTO flow_runs ( + flow_id, owner_key, status, notify_policy, goal, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?)`, + ).run("flow-1", "owner-1", "succeeded", "never", "private flow goal", 62, 70); + for (const [ownerKind, ownerId] of [ + ["cron", "cron-receipt-1"], + ["task", "task-1"], + ["flow", "flow-1"], + ] as const) { + expect( + bindExecutionOwnerLifecycleMetadata({ + db, + ownerKind, + ownerId, + binding: { contextId: context.contextId, executionId: context.executionId }, + }), + ).toBe("bound"); + } + + const result = presentExecutionDecisionReceipts({ + context, + decisionLimit: 10, + options: { ...database, now: 100 }, + }); + expect(result.decisions.map((item) => item.source.owner)).toEqual([ + "agent-command", + "cron_run_receipts", + "task_runs", + "flow_runs", + ]); + expect(JSON.stringify(result.decisions)).not.toContain("private task text"); + expect(JSON.stringify(result.decisions)).not.toContain("private flow goal"); + expect(tableExists(db, "execution_decision_facts")).toBe(false); + }); + it("does not assign run-only delivery evidence to either exact execution sharing a run id", () => { const database = databaseOptions(); const first = seedExecutionContext(database, { diff --git a/src/audit/execution-decision-receipts.ts b/src/audit/execution-decision-receipts.ts index ec65124afda2..c3495ae3209c 100644 --- a/src/audit/execution-decision-receipts.ts +++ b/src/audit/execution-decision-receipts.ts @@ -15,6 +15,12 @@ import { pageExecutionDecisionFactsForContext, summarizeExecutionDecisionFactsForContext, } from "./execution-decision-facts.js"; +import { + pageOwnerLifecycleReceipts, + summarizeOwnerLifecycleReceipts, + type OwnerLifecycleCursor, + type OwnerLifecycleStage, +} from "./execution-owner-lifecycle-receipts.js"; import { pageMessageDeliveryReceiptsForRun, summarizeMessageDeliveryReceiptsForRun, @@ -34,9 +40,10 @@ type ProvenancedDecisionReceipt = { const MAX_AGGREGATE_MISSING_EVIDENCE = 16; const MISSING_EVIDENCE_TRUNCATED = "decision.missing_evidence_truncated"; +type DecisionStage = "approval" | "message" | "generic" | OwnerLifecycleStage; type DecisionCursor = | { - stage: "approval" | "message" | "generic"; + stage: DecisionStage; after?: { occurredAt: number; rowId: number }; } | { @@ -58,7 +65,7 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi if (offset !== null && offset !== undefined) { return { offset }; } - const match = /^([amg]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value); + const match = /^([amgctf]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value); if (!match) { return null; } @@ -67,8 +74,20 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi if (!Number.isSafeInteger(occurredAt) || !Number.isSafeInteger(rowId)) { return null; } + const stage = + match[1] === "a" + ? "approval" + : match[1] === "m" + ? "message" + : match[1] === "g" + ? "generic" + : match[1] === "c" + ? "cron" + : match[1] === "t" + ? "task" + : "flow"; return { - stage: match[1] === "a" ? "approval" : match[1] === "m" ? "message" : "generic", + stage, ...(occurredAt === 0 && rowId === 0 ? {} : { after: { occurredAt, rowId } }), }; } @@ -78,10 +97,17 @@ export function isExecutionDecisionCursor(value: string): boolean { } function formatDecisionCursor( - stage: "approval" | "message" | "generic", + stage: DecisionStage, cursor?: { occurredAt: number; rowId: number }, ): string { - const prefix = stage === "approval" ? "a" : stage === "message" ? "m" : "g"; + const prefix = { + approval: "a", + message: "m", + generic: "g", + cron: "c", + task: "t", + flow: "f", + }[stage]; return `${prefix}:${cursor?.occurredAt ?? 0}:${cursor?.rowId ?? 0}`; } @@ -200,10 +226,8 @@ export function presentExecutionDecisionReceipts(params: { if (cursor === null) { throw new ExecutionDecisionCursorError(); } - const limit = params.decisionLimit ?? 50; + const decisionLimit = params.decisionLimit ?? 50; const now = params.options.now ?? Date.now(); - // Numeric cursors are the shipped aggregate offset. Resolve its owner span - // once, then let the canonical bounded owner pagers emit opaque successors. const opaqueCursor = cursor && "stage" in cursor ? cursor : undefined; const legacyOffset = cursor && "offset" in cursor ? cursor.offset - 1 : undefined; const approvalSummary = summarizeOperatorApprovalReceiptsForRun({ @@ -225,11 +249,132 @@ export function presentExecutionDecisionReceipts(params: { context: params.context, options: { ...params.options, now }, }); + const cronSummary = summarizeOwnerLifecycleReceipts({ + stage: "cron", + context: params.context, + options: params.options, + }); + const taskSummary = summarizeOwnerLifecycleReceipts({ + stage: "task", + context: params.context, + options: params.options, + }); + const flowSummary = summarizeOwnerLifecycleReceipts({ + stage: "flow", + context: params.context, + options: params.options, + }); + const stages: Array<{ + stage: DecisionStage; + count: number; + page: (params: { after?: OwnerLifecycleCursor; offset?: number; limit: number }) => { + entries: ProvenancedDecisionReceipt[]; + nextCursor?: OwnerLifecycleCursor; + }; + }> = [ + { + stage: "approval", + count: approvalSummary.count, + page: ({ after, offset, limit }) => { + const page = pageOperatorApprovalReceiptsForRun({ + context: { + contextId: params.context.contextId, + executionId: params.context.executionId, + runId: params.context.runId, + }, + after, + offset, + limit, + nowMs: now, + databaseOptions: params.options, + }); + return { + entries: page.entries.map((entry) => ({ + receipt: entry.receipt, + provenance: { state: "verified", producer: "operator-approval" }, + selectorId: entry.selectorId, + })), + ...(page.nextCursor ? { nextCursor: page.nextCursor } : {}), + }; + }, + }, + { + stage: "message", + count: messageSummary.count, + page: ({ after, offset, limit }) => { + const page = pageMessageDeliveryReceiptsForRun({ + context: params.context, + after, + offset, + limit, + options: { ...params.options, now }, + }); + return { + entries: page.entries.map((entry) => ({ + receipt: entry.receipt, + provenance: { state: "verified", producer: "message-delivery" }, + selectorId: entry.selectorId, + })), + ...(page.nextCursor ? { nextCursor: page.nextCursor } : {}), + }; + }, + }, + { + stage: "generic", + count: genericSummary.count, + page: ({ after, offset, limit }) => { + const page = pageExecutionDecisionFactsForContext({ + context: params.context, + after, + offset, + limit, + now, + database: params.options, + }); + return { + entries: page.entries.map((entry) => ({ + receipt: entry.receipt, + provenance: { state: "unverified" }, + selectorId: entry.selectorId, + })), + ...(page.nextCursor ? { nextCursor: page.nextCursor } : {}), + }; + }, + }, + ...(["cron", "task", "flow"] as const).map((stage) => ({ + stage, + count: { cron: cronSummary, task: taskSummary, flow: flowSummary }[stage].count, + page: ({ + after, + offset, + limit, + }: { + after?: OwnerLifecycleCursor; + offset?: number; + limit: number; + }) => { + const page = pageOwnerLifecycleReceipts({ + stage, + context: params.context, + after, + offset, + limit, + options: params.options, + }); + return { + entries: page.entries.map((entry) => ({ + receipt: entry.receipt, + provenance: { state: "verified" as const, producer: entry.displayProducer }, + selectorId: entry.selectorId, + })), + ...(page.nextCursor ? { nextCursor: page.nextCursor } : {}), + }; + }, + })), + ]; const decisions: ProvenancedDecisionReceipt[] = []; - let remainingLimit = limit; + let remainingLimit = decisionLimit; let nextDecisionCursor: string | undefined; - const approvalOffset = - legacyOffset !== undefined && legacyOffset < approvalSummary.count ? legacyOffset : undefined; if (cursor === undefined && remainingLimit > 0) { decisions.push({ @@ -238,32 +383,44 @@ export function presentExecutionDecisionReceipts(params: { selectorId: `${params.context.contextId}:admission`, }); remainingLimit -= 1; - if ( - remainingLimit === 0 && - (approvalSummary.count > 0 || messageSummary.count > 0 || genericSummary.count > 0) - ) { + if (remainingLimit === 0 && stages.some((stage) => stage.count > 0)) { + // The shipped first successor remains the approval cursor even when that owner is empty. nextDecisionCursor = formatDecisionCursor("approval"); } } - if ( - remainingLimit > 0 && - opaqueCursor?.stage !== "message" && - opaqueCursor?.stage !== "generic" && - (legacyOffset === undefined || approvalOffset !== undefined) - ) { + let startStage = 0; + let firstStageOffset: number | undefined; + if (opaqueCursor) { + startStage = stages.findIndex((stage) => stage.stage === opaqueCursor.stage); + } else if (legacyOffset !== undefined) { + let preceding = 0; + startStage = stages.findIndex((stage) => { + if (legacyOffset < preceding + stage.count) { + firstStageOffset = legacyOffset - preceding; + return true; + } + preceding += stage.count; + return false; + }); + if (startStage < 0) { + startStage = stages.length; + } + } + for (let index = startStage; index < stages.length && remainingLimit > 0; index += 1) { + const stage = stages[index]; + if (!stage) { + continue; + } let page; try { - page = pageOperatorApprovalReceiptsForRun({ - context: { - contextId: params.context.contextId, - executionId: params.context.executionId, - runId: params.context.runId, - }, - after: opaqueCursor?.stage === "approval" ? opaqueCursor.after : undefined, - offset: approvalOffset, + page = stage.page({ + ...(index === startStage && opaqueCursor?.stage === stage.stage + ? { after: opaqueCursor.after } + : {}), + ...(index === startStage && firstStageOffset !== undefined + ? { offset: firstStageOffset } + : {}), limit: remainingLimit, - nowMs: now, - databaseOptions: params.options, }); } catch (error) { if (error instanceof Error && error.message.includes("cursor is no longer retained")) { @@ -273,125 +430,52 @@ export function presentExecutionDecisionReceipts(params: { } throw error; } - decisions.push( - ...page.entries.map((entry) => ({ - receipt: entry.receipt, - provenance: { state: "verified" as const, producer: "operator-approval" as const }, - selectorId: entry.selectorId, - })), - ); + decisions.push(...page.entries); remainingLimit -= page.entries.length; if (page.nextCursor) { - nextDecisionCursor = formatDecisionCursor("approval", page.nextCursor); - } else if (remainingLimit === 0 && messageSummary.count > 0) { - nextDecisionCursor = formatDecisionCursor("message"); - } else if (remainingLimit === 0 && genericSummary.count > 0) { - nextDecisionCursor = formatDecisionCursor("generic"); + nextDecisionCursor = formatDecisionCursor(stage.stage, page.nextCursor); + break; + } + if (remainingLimit === 0) { + const successor = stages.slice(index + 1).find((candidate) => candidate.count > 0); + nextDecisionCursor = successor ? formatDecisionCursor(successor.stage) : undefined; } } - const messageOffset = - legacyOffset === undefined - ? undefined - : legacyOffset >= approvalSummary.count && - legacyOffset < approvalSummary.count + messageSummary.count - ? legacyOffset - approvalSummary.count - : undefined; - if ( - remainingLimit > 0 && - nextDecisionCursor?.startsWith("a:") !== true && - opaqueCursor?.stage !== "generic" && - (legacyOffset === undefined || messageOffset !== undefined || approvalOffset !== undefined) - ) { - let page; - try { - page = pageMessageDeliveryReceiptsForRun({ - context: params.context, - after: opaqueCursor?.stage === "message" ? opaqueCursor.after : undefined, - offset: messageOffset, - limit: remainingLimit, - options: { ...params.options, now }, - }); - } catch (error) { - if (error instanceof Error && error.message.includes("cursor is no longer retained")) { - throw new ExecutionDecisionCursorError( - "decision cursor is no longer retained; restart inspection without --cursor", - ); - } - throw error; - } - decisions.push( - ...page.entries.map((entry) => ({ - receipt: entry.receipt, - provenance: { state: "verified" as const, producer: "message-delivery" as const }, - selectorId: entry.selectorId, - })), - ); - remainingLimit -= page.entries.length; - if (page.nextCursor) { - nextDecisionCursor = formatDecisionCursor("message", page.nextCursor); - } else if (remainingLimit === 0 && genericSummary.count > 0) { - nextDecisionCursor = formatDecisionCursor("generic"); - } - } - const genericOffset = - legacyOffset === undefined - ? undefined - : Math.max(0, legacyOffset - approvalSummary.count - messageSummary.count); - if ( - remainingLimit > 0 && - nextDecisionCursor?.startsWith("a:") !== true && - nextDecisionCursor?.startsWith("m:") !== true - ) { - let page; - try { - page = pageExecutionDecisionFactsForContext({ - context: params.context, - after: opaqueCursor?.stage === "generic" ? opaqueCursor.after : undefined, - offset: genericOffset, - limit: remainingLimit, - now, - database: params.options, - }); - } catch (error) { - if (error instanceof Error && error.message.includes("cursor is no longer retained")) { - throw new ExecutionDecisionCursorError( - "decision cursor is no longer retained; restart inspection without --cursor", - ); - } - throw error; - } - decisions.push( - ...page.entries.map((entry) => ({ - receipt: entry.receipt, - provenance: { state: "unverified" as const }, - selectorId: entry.selectorId, - })), - ); - if (page.nextCursor) { - nextDecisionCursor = formatDecisionCursor("generic", page.nextCursor); - } else { - nextDecisionCursor = undefined; - } - } - const ownerCoverage = new Set([approvalSummary.coverageState, messageSummary.coverageState]); + const ownerCoverage = new Set( + [ + approvalSummary.coverageState, + messageSummary.coverageState, + cronSummary.coverageState, + taskSummary.coverageState, + flowSummary.coverageState, + ].filter( + (coverageState): coverageState is NonNullable => + coverageState !== undefined, + ), + ); const hasUnverifiedGenericDecisions = genericSummary.count > 0; const boundedEvidence = boundMissingEvidence([ ...params.context.missingEvidence, ...approvalSummary.missingEvidence, ...messageSummary.missingEvidence, + ...cronSummary.missingEvidence, + ...taskSummary.missingEvidence, + ...flowSummary.missingEvidence, ...(hasUnverifiedGenericDecisions ? ["decision.display_provenance"] : []), ]); const coverageState = boundedEvidence.truncated ? "unknown" : hasUnverifiedGenericDecisions ? "unknown" - : ownerCoverage.has("unknown") - ? "unknown" - : ownerCoverage.has("enforced") - ? "enforced" - : ownerCoverage.has("attribution-only") - ? "attribution-only" - : params.context.coverageState; + : ownerCoverage.has("unsupported") + ? "unsupported" + : ownerCoverage.has("unknown") + ? "unknown" + : ownerCoverage.has("enforced") + ? "enforced" + : ownerCoverage.has("attribution-only") + ? "attribution-only" + : params.context.coverageState; return { schemaVersion: 1, run: { diff --git a/src/audit/execution-owner-binding.ts b/src/audit/execution-owner-binding.ts new file mode 100644 index 000000000000..9af5d1764730 --- /dev/null +++ b/src/audit/execution-owner-binding.ts @@ -0,0 +1,96 @@ +import type { + AdmittedRunContext, + PreparedAgentRunAdmission, +} from "../agents/admitted-run-context.js"; +import { parseExecutionIdentityAdmissionToken } from "./execution-identity-admission.js"; + +export type ExecutionOwnerBindingResult = + | "disabled" + | "bound" + | "already-bound" + | "mismatch" + | "missing"; + +export function isRetainedExecutionOwnerBinding( + result: ExecutionOwnerBindingResult | undefined, +): result is "bound" | "already-bound" { + return result === "bound" || result === "already-bound"; +} + +type ExecutionOwnerBinding = Readonly<{ + contextId: string; + executionId: string; +}>; + +/** Extracts only an admitted exact identity; operational run correlation cannot bind owner rows. */ +export function executionOwnerBindingFromAdmission( + admitted: AdmittedRunContext, +): ExecutionOwnerBinding | undefined { + if (!admitted.executionIdentityToken) { + return undefined; + } + const token = parseExecutionIdentityAdmissionToken(admitted.executionIdentityToken); + if (token.runId !== admitted.operationalRunInstance.runId) { + throw new Error("owner execution binding disagrees with the admitted run"); + } + return { contextId: token.contextId, executionId: token.executionId }; +} + +export function classifyExecutionOwnerBinding( + current: { contextId: string | null; executionId: string | null }, + binding: ExecutionOwnerBinding, +): Exclude | "unbound" { + if (current.contextId === null && current.executionId === null) { + return "unbound"; + } + return current.contextId === binding.contextId && current.executionId === binding.executionId + ? "already-bound" + : "mismatch"; +} + +/** Adds one exact owner write after admission resolves, never inside the admission callback. */ +export function withPostAdmissionExecutionOwnerBinding( + prepared: PreparedAgentRunAdmission, + bind: (context: AdmittedRunContext) => void, +): PreparedAgentRunAdmission { + let bound = false; + return Object.freeze({ + operationalRunInstance: prepared.operationalRunInstance, + admit: async (runtimeKind, runtimeInstanceId) => { + const admitted = await prepared.admit(runtimeKind, runtimeInstanceId); + if (!bound) { + bound = true; + bind(admitted); + } + return admitted; + }, + close: prepared.close, + }); +} + +/** Requires both exact admission and actual execution start, in either runtime order. */ +export function createExecutionStartedOwnerBinding(bind: (context: AdmittedRunContext) => void): { + onPostAdmission: (context: AdmittedRunContext) => void; + onExecutionStarted: () => void; +} { + let admitted: AdmittedRunContext | undefined; + let executionStarted = false; + let bound = false; + const bindIfReady = () => { + if (bound || !admitted || !executionStarted) { + return; + } + bound = true; + bind(admitted); + }; + return { + onPostAdmission: (context) => { + admitted = context; + bindIfReady(); + }, + onExecutionStarted: () => { + executionStarted = true; + bindIfReady(); + }, + }; +} diff --git a/src/audit/execution-owner-lifecycle-binding-store.ts b/src/audit/execution-owner-lifecycle-binding-store.ts new file mode 100644 index 000000000000..afdb0e5ebe05 --- /dev/null +++ b/src/audit/execution-owner-lifecycle-binding-store.ts @@ -0,0 +1,155 @@ +import type { DatabaseSync } from "node:sqlite"; +import { + executeSqliteQuerySync, + executeSqliteQueryTakeFirstSync, + getNodeSqliteKysely, +} from "../infra/kysely-sync.js"; +import { tableExists } from "../state/openclaw-state-db-schema-helpers.js"; +import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js"; +import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js"; +import { + classifyExecutionOwnerBinding, + type ExecutionOwnerBindingResult, +} from "./execution-owner-binding.js"; + +export const EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE = + "execution_owner_lifecycle_bindings" as const; +type ExecutionOwnerLifecycleKind = "cron" | "task" | "flow"; + +type ExecutionOwnerLifecycleDatabase = Pick< + OpenClawStateDatabase, + "cron_run_receipts" | "execution_owner_lifecycle_bindings" | "flow_runs" | "task_runs" +>; + +const SCHEMA_START = `CREATE TABLE IF NOT EXISTS ${EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE} (`; +const SCHEMA_END = ") STRICT;"; + +function lifecycleDb(db: DatabaseSync) { + return getNodeSqliteKysely(db); +} + +/** Creates only the canonical additive metadata table at first admitted owner use. */ +function ensureExecutionOwnerLifecycleBindingSchema(db: DatabaseSync): void { + if (tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) { + return; + } + const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_START); + const end = start < 0 ? -1 : OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_END, start); + if (start < 0 || end < start) { + throw new Error("OpenClaw execution owner lifecycle binding schema marker is missing."); + } + // sqlite-allow-raw -- Canonical feature-local additive DDL only; metadata rows use Kysely. + db.exec(OPENCLAW_STATE_SCHEMA_SQL.slice(start, end + SCHEMA_END.length)); +} + +function classifyRetainedBinding( + current: { context_id: string; execution_id: string }, + binding: { contextId: string; executionId: string }, +): "already-bound" | "mismatch" { + const state = classifyExecutionOwnerBinding( + { contextId: current.context_id, executionId: current.execution_id }, + binding, + ); + return state === "unbound" ? "mismatch" : state; +} + +/** Stores one exact admission identity after its canonical owner row has been revalidated. */ +export function bindExecutionOwnerLifecycleMetadata(params: { + db: DatabaseSync; + ownerKind: ExecutionOwnerLifecycleKind; + ownerId: string; + binding: { contextId: string; executionId: string }; +}): Exclude { + ensureExecutionOwnerLifecycleBindingSchema(params.db); + const database = lifecycleDb(params.db); + const current = executeSqliteQueryTakeFirstSync( + params.db, + database + .selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .select(["context_id", "execution_id"]) + .where("owner_kind", "=", params.ownerKind) + .where("owner_id", "=", params.ownerId), + ); + if (current) { + return classifyRetainedBinding(current, params.binding); + } + const inserted = executeSqliteQuerySync( + params.db, + database + .insertInto(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .values({ + owner_kind: params.ownerKind, + owner_id: params.ownerId, + context_id: params.binding.contextId, + execution_id: params.binding.executionId, + }) + .onConflict((conflict) => conflict.columns(["owner_kind", "owner_id"]).doNothing()), + ); + if (Number(inserted.numAffectedRows ?? 0n) === 1) { + return "bound"; + } + const raced = executeSqliteQueryTakeFirstSync( + params.db, + database + .selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .select(["context_id", "execution_id"]) + .where("owner_kind", "=", params.ownerKind) + .where("owner_id", "=", params.ownerId), + ); + return raced ? classifyRetainedBinding(raced, params.binding) : "mismatch"; +} + +/** Removes exact owner metadata without allocating the opt-in table. */ +export function deleteExecutionOwnerLifecycleMetadata(params: { + db: DatabaseSync; + ownerKind: ExecutionOwnerLifecycleKind; + ownerIds: readonly string[]; +}): void { + if ( + params.ownerIds.length === 0 || + !tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + ) { + return; + } + executeSqliteQuerySync( + params.db, + lifecycleDb(params.db) + .deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .where("owner_kind", "=", params.ownerKind) + .where("owner_id", "in", params.ownerIds), + ); +} + +/** Removes bindings whose canonical owner row was pruned in the same transaction. */ +export function pruneOrphanedExecutionOwnerLifecycleMetadata( + db: DatabaseSync, + ownerKind: ExecutionOwnerLifecycleKind, +): void { + if (!tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) { + return; + } + const database = lifecycleDb(db); + const bindings = database + .deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .where("owner_kind", "=", ownerKind); + if (ownerKind === "cron") { + executeSqliteQuerySync( + db, + bindings.where( + "owner_id", + "not in", + database.selectFrom("cron_run_receipts").select("receipt_id"), + ), + ); + } else if (ownerKind === "task") { + executeSqliteQuerySync( + db, + bindings.where("owner_id", "not in", database.selectFrom("task_runs").select("task_id")), + ); + } else { + executeSqliteQuerySync( + db, + bindings.where("owner_id", "not in", database.selectFrom("flow_runs").select("flow_id")), + ); + } +} diff --git a/src/audit/execution-owner-lifecycle-receipts.test.ts b/src/audit/execution-owner-lifecycle-receipts.test.ts new file mode 100644 index 000000000000..6b19cc659047 --- /dev/null +++ b/src/audit/execution-owner-lifecycle-receipts.test.ts @@ -0,0 +1,611 @@ +import { rmSync } from "node:fs"; +import path from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { afterEach, describe, expect, it } from "vitest"; +import type { ExecutionIdentityContextV1 } from "../../packages/gateway-protocol/src/index.js"; +import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js"; +import type { AdmittedRunContext } from "../agents/admitted-run-context.js"; +import { bindCronRunReceiptExecution } from "../cron/store/run-receipt-store.js"; +import { tableHasColumn, tableExists } from "../state/openclaw-state-db-schema-helpers.js"; +import { + OPENCLAW_STATE_SCHEMA_VERSION, + closeOpenClawStateDatabaseForTest, + openOpenClawStateDatabase, +} from "../state/openclaw-state-db.js"; +import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js"; +import { bindTaskFlowExecution } from "../tasks/task-flow-registry.store.sqlite.js"; +import { bindTaskRunExecution } from "../tasks/task-registry.store.sqlite.js"; +import { presentExecutionDecisionReceipts } from "./execution-decision-receipts.js"; +import { createExecutionIdentityAdmissionToken } from "./execution-identity-admission.js"; +import { pageOwnerLifecycleReceipts } from "./execution-owner-lifecycle-receipts.js"; + +afterEach(() => closeOpenClawStateDatabaseForTest()); +const tempDirs = useAutoCleanupTempDirTracker(afterEach); + +function oldSchemaSql(): string { + const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf( + "CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (", + ); + const endMarker = ") STRICT;"; + const end = OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start); + if (start < 0 || end < start) { + throw new Error("owner lifecycle binding schema marker is missing"); + } + return `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`; +} + +function createOldOwnerDatabase() { + const pathname = path.join(tempDirs.make("owner-lifecycle-"), "openclaw.sqlite"); + const oldReader = new DatabaseSync(pathname); + oldReader.exec(oldSchemaSql()); + oldReader.exec(`PRAGMA user_version = ${OPENCLAW_STATE_SCHEMA_VERSION}`); + oldReader + .prepare( + `INSERT INTO schema_meta ( + meta_key, role, schema_version, created_at, updated_at + ) VALUES ('primary', 'global', ?, 1, 1)`, + ) + .run(OPENCLAW_STATE_SCHEMA_VERSION); + oldReader + .prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null); + oldReader + .prepare( + `INSERT INTO task_runs ( + task_id, runtime, owner_key, scope_kind, task, status, delivery_status, + notify_policy, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + "task-1", + "cron", + "owner-1", + "system", + "private", + "running", + "not_applicable", + "silent", + 61, + ); + oldReader + .prepare( + `INSERT INTO flow_runs ( + flow_id, owner_key, status, notify_policy, goal, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?)`, + ) + .run("flow-1", "owner-1", "running", "silent", "private", 62, 62); + oldReader.close(); + return { path: pathname }; +} + +function admitted(contextId = "context-1", executionId = "execution-1"): AdmittedRunContext { + return { + operationalRunInstance: { instanceId: "instance-1", runId: "run-1" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-1", { + contextId, + executionId, + now: 50, + }), + }; +} + +function executionContext(contextId = "context-1"): ExecutionIdentityContextV1 { + return { + schemaVersion: 1, + contextId, + executionId: "execution-1", + runId: "run-1", + createdAt: 50, + trustDomain: { kind: "gateway-cell", domainRef: "host-1", state: "present" }, + invoker: { state: "unknown" }, + ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" }, + agentPrincipal: { kind: "agent", domainRef: "host-1", principalRef: "main" }, + agentDefinition: { definitionRef: "main", state: "present" }, + runtimeInstance: { runtimeRef: "runtime-1", kind: "embedded", state: "present" }, + applicableGrants: [], + assurance: [], + coverageState: "unattributed", + missingEvidence: [], + }; +} + +const receiptHandle = { + receiptId: "cron-1", + storeKey: "default", + jobId: "job-1", + configRevision: "revision-1", + agentId: "main", + ownerPid: 1, + ownerStartTime: null, + startedAtMs: 60, +}; + +describe("owner-native execution lifecycle receipts", () => { + it("lazily binds exact owner rows while disabled collection allocates nothing", () => { + const options = createOldOwnerDatabase(); + const current = openOpenClawStateDatabase(options).db; + expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false); + for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) { + expect(tableHasColumn(current, table, "context_id")).toBe(false); + expect(tableHasColumn(current, table, "execution_id")).toBe(false); + } + + const disabled: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-disabled", runId: "run-1" }, + }; + expect( + bindCronRunReceiptExecution({ admitted: disabled, handle: receiptHandle, options }), + ).toBe("disabled"); + expect(bindTaskRunExecution({ admitted: disabled, taskId: "task-1", options })).toBe( + "disabled", + ); + expect(bindTaskFlowExecution({ admitted: disabled, flowId: "flow-1", options })).toBe( + "disabled", + ); + expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false); + for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) { + expect(tableHasColumn(current, table, "context_id")).toBe(false); + expect(tableHasColumn(current, table, "execution_id")).toBe(false); + } + + expect( + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }), + ).toBe("bound"); + expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe("bound"); + expect(bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options })).toBe( + "bound", + ); + expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(true); + for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) { + expect(tableHasColumn(current, table, "context_id")).toBe(false); + expect(tableHasColumn(current, table, "execution_id")).toBe(false); + } + expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe( + "already-bound", + ); + expect( + bindTaskRunExecution({ + admitted: admitted("context-1", "execution-other"), + taskId: "task-1", + options, + }), + ).toBe("mismatch"); + current.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run(); + current.prepare("UPDATE task_runs SET status = 'succeeded'").run(); + current.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run(); + + closeOpenClawStateDatabaseForTest(); + const oldReader = new DatabaseSync(options.path); + expect( + oldReader.prepare("SELECT status FROM task_runs WHERE task_id = ?").get("task-1"), + ).toEqual({ + status: "succeeded", + }); + oldReader.prepare("UPDATE task_runs SET status = ? WHERE task_id = ?").run("failed", "task-1"); + oldReader.close(); + + const reopened = openOpenClawStateDatabase(options).db; + expect(reopened.prepare("PRAGMA user_version").get()).toEqual({ + user_version: OPENCLAW_STATE_SCHEMA_VERSION, + }); + expect( + reopened + .prepare( + `SELECT binding.context_id, binding.execution_id, task.status + FROM task_runs AS task + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id + WHERE task.task_id = ?`, + ) + .get("task-1"), + ).toEqual({ context_id: "context-1", execution_id: "execution-1", status: "failed" }); + expect( + reopened + .prepare( + `SELECT owner_kind, owner_id, context_id, execution_id + FROM execution_owner_lifecycle_bindings + ORDER BY owner_kind`, + ) + .all(), + ).toEqual([ + { + owner_kind: "cron", + owner_id: "cron-1", + context_id: "context-1", + execution_id: "execution-1", + }, + { + owner_kind: "flow", + owner_id: "flow-1", + context_id: "context-1", + execution_id: "execution-1", + }, + { + owner_kind: "task", + owner_id: "task-1", + context_id: "context-1", + execution_id: "execution-1", + }, + ]); + expect(tableExists(reopened, "execution_decision_facts")).toBe(true); + expect( + reopened.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get(), + ).toEqual({ count: 0 }); + }); + + it("uses stable c/t/f cursors and rejects a mismatched exact execution", () => { + const options = createOldOwnerDatabase(); + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }); + bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options }); + bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options }); + const db = openOpenClawStateDatabase(options).db; + db.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run(); + db.prepare("UPDATE task_runs SET status = 'succeeded'").run(); + db.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run(); + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "skipped", 1, 60, 70); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES (?, ?, ?, ?)`, + ).run("cron", "cron-2", "context-1", "execution-other"); + const context = executionContext(); + + const first = presentExecutionDecisionReceipts({ + context, + decisionLimit: 1, + options, + }); + expect(first.nextDecisionCursor).toBe("a:0:0"); + const cronPage = presentExecutionDecisionReceipts({ + context, + decisionCursor: first.nextDecisionCursor, + decisionLimit: 1, + options, + }); + expect(cronPage.decisions[0]?.source.owner).toBe("cron_run_receipts"); + expect(cronPage.decisionDisplays[0]).toMatchObject({ + decision: { outcome: "not-applicable", reasonCode: "cron_run_ok" }, + provenance: { state: "verified", producer: "cron-lifecycle" }, + }); + expect(cronPage.nextDecisionCursor).toMatch(/^c:/); + const mismatchPage = presentExecutionDecisionReceipts({ + context, + decisionCursor: cronPage.nextDecisionCursor, + decisionLimit: 1, + options, + }); + expect(mismatchPage.decisions[0]).toMatchObject({ + decision: { outcome: "unknown", reasonCode: "cron_run_execution_link_mismatch" }, + missingEvidence: ["decision.execution_link"], + }); + expect(mismatchPage.nextDecisionCursor).toBe("t:0:0"); + const taskPage = presentExecutionDecisionReceipts({ + context, + decisionCursor: mismatchPage.nextDecisionCursor, + decisionLimit: 1, + options, + }); + expect(taskPage.decisions[0]?.source.owner).toBe("task_runs"); + expect(taskPage.decisionDisplays[0]).toMatchObject({ + decision: { outcome: "not-applicable", reasonCode: "task_run_succeeded" }, + provenance: { state: "verified", producer: "task-lifecycle" }, + }); + expect(taskPage.nextDecisionCursor).toBe("f:0:0"); + const flowPage = presentExecutionDecisionReceipts({ + context, + decisionCursor: taskPage.nextDecisionCursor, + decisionLimit: 1, + options, + }); + expect(flowPage.decisions[0]?.source.owner).toBe("flow_runs"); + expect(flowPage.decisionDisplays[0]).toMatchObject({ + decision: { outcome: "not-applicable", reasonCode: "flow_run_succeeded" }, + provenance: { state: "verified", producer: "flow-lifecycle" }, + }); + expect( + JSON.stringify([ + cronPage.decisionDisplays, + taskPage.decisionDisplays, + flowPage.decisionDisplays, + ]), + ).not.toMatch(/cron-1|task-1|flow-1|private|cron_run_receipts|task_runs|flow_runs/); + expect(flowPage.nextDecisionCursor).toBeUndefined(); + + expect( + presentExecutionDecisionReceipts({ + context, + decisionCursor: "1", + decisionLimit: 1, + options, + }).decisions[0]?.source.owner, + ).toBe("cron_run_receipts"); + }); + + const deletedAnchorCases = [ + { + stage: "cron", + cursor: "c:0:0", + bindFirst: (options: ReturnType) => + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }), + addSuccessor: (db: DatabaseSync) => { + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`, + ).run(); + }, + deleteAnchor: (db: DatabaseSync) => { + db.prepare( + "DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'", + ).run(); + db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run(); + }, + }, + { + stage: "task", + cursor: "t:0:0", + bindFirst: (options: ReturnType) => + bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options }), + addSuccessor: (db: DatabaseSync) => { + db.prepare( + `INSERT INTO task_runs ( + task_id, runtime, owner_key, scope_kind, task, status, delivery_status, + notify_policy, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run( + "task-2", + "cron", + "owner-2", + "system", + "private", + "running", + "not_applicable", + "silent", + 63, + ); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES ('task', 'task-2', 'context-1', 'execution-1')`, + ).run(); + }, + deleteAnchor: (db: DatabaseSync) => { + db.prepare( + "DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'task' AND owner_id = 'task-1'", + ).run(); + db.prepare("DELETE FROM task_runs WHERE task_id = 'task-1'").run(); + }, + }, + { + stage: "flow", + cursor: "f:0:0", + bindFirst: (options: ReturnType) => + bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options }), + addSuccessor: (db: DatabaseSync) => { + db.prepare( + `INSERT INTO flow_runs ( + flow_id, owner_key, status, notify_policy, goal, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?)`, + ).run("flow-2", "owner-2", "running", "silent", "private", 63, 63); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES ('flow', 'flow-2', 'context-1', 'execution-1')`, + ).run(); + }, + deleteAnchor: (db: DatabaseSync) => { + db.prepare( + "DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'flow' AND owner_id = 'flow-1'", + ).run(); + db.prepare("DELETE FROM flow_runs WHERE flow_id = 'flow-1'").run(); + }, + }, + ] as const; + + it.each(deletedAnchorCases)( + "rejects a nonzero $stage cursor after its exact owner anchor is deleted", + ({ cursor, bindFirst, addSuccessor, deleteAnchor }) => { + const options = createOldOwnerDatabase(); + expect(bindFirst(options)).toBe("bound"); + const db = openOpenClawStateDatabase(options).db; + addSuccessor(db); + const firstPage = presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: cursor, + decisionLimit: 1, + options, + }); + expect(firstPage.nextDecisionCursor).toMatch(/^[ctf]:[1-9]\d*:[1-9]\d*$/); + + deleteAnchor(db); + + expect(() => + presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: firstPage.nextDecisionCursor, + decisionLimit: 1, + options, + }), + ).toThrow("decision cursor is no longer retained; restart inspection without --cursor"); + }, + ); + + it("rejects a nonzero owner cursor when its retained anchor belongs to another context", () => { + const options = createOldOwnerDatabase(); + expect( + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }), + ).toBe("bound"); + const db = openOpenClawStateDatabase(options).db; + deletedAnchorCases[0].addSuccessor(db); + const firstPage = presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: "c:0:0", + decisionLimit: 1, + options, + }); + expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/); + + expect(() => + presentExecutionDecisionReceipts({ + context: executionContext("context-other"), + decisionCursor: firstPage.nextDecisionCursor, + decisionLimit: 1, + options, + }), + ).toThrow("decision cursor is no longer retained; restart inspection without --cursor"); + }); + + it("rejects a nonzero owner cursor after the state database is removed", () => { + const options = createOldOwnerDatabase(); + expect( + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }), + ).toBe("bound"); + const db = openOpenClawStateDatabase(options).db; + deletedAnchorCases[0].addSuccessor(db); + const firstPage = presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: "c:0:0", + decisionLimit: 1, + options, + }); + expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/); + + closeOpenClawStateDatabaseForTest(); + rmSync(options.path); + + expect(() => + presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: firstPage.nextDecisionCursor, + decisionLimit: 1, + options, + }), + ).toThrow("decision cursor is no longer retained; restart inspection without --cursor"); + }); + + it("rejects a reused owner rowid whose binding belongs to another execution", () => { + const options = createOldOwnerDatabase(); + const db = openOpenClawStateDatabase(options).db; + db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run(); + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null); + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null); + expect( + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }), + ).toBe("bound"); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`, + ).run(); + const firstPage = presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: "c:0:0", + decisionLimit: 1, + options, + }); + expect(firstPage.nextDecisionCursor).toMatch(/^c:60:[1-9]\d*$/); + + db.prepare( + "DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'", + ).run(); + db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run(); + db.prepare( + `INSERT INTO cron_run_receipts ( + receipt_id, store_key, job_id, config_revision, agent_id, request_run_id, + status, owner_pid, started_at_ms, finished_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ).run( + "cron-replacement", + "default", + "job-replacement", + "revision-1", + "main", + "run-2", + "running", + 1, + 60, + null, + ); + db.prepare( + `INSERT INTO execution_owner_lifecycle_bindings ( + owner_kind, owner_id, context_id, execution_id + ) VALUES ('cron', 'cron-replacement', 'context-1', 'execution-other')`, + ).run(); + + expect(() => + presentExecutionDecisionReceipts({ + context: executionContext(), + decisionCursor: firstPage.nextDecisionCursor, + decisionLimit: 1, + options, + }), + ).toThrow("decision cursor is no longer retained; restart inspection without --cursor"); + }); + + it("projects every owner terminal state without rederiving lifecycle precedence", () => { + const options = createOldOwnerDatabase(); + bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }); + bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options }); + bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options }); + const context = executionContext(); + const db = openOpenClawStateDatabase(options).db; + + for (const status of ["ok", "error", "skipped", "interrupted", "superseded"]) { + db.prepare( + "UPDATE cron_run_receipts SET status = ?, finished_at_ms = 70 WHERE receipt_id = ?", + ).run(status, "cron-1"); + expect( + pageOwnerLifecycleReceipts({ stage: "cron", context, limit: 1, options }).entries[0] + ?.receipt.decision, + ).toEqual({ outcome: "not-applicable", reasonCode: `cron_run_${status}` }); + } + for (const status of ["succeeded", "failed", "timed_out", "cancelled", "lost"]) { + db.prepare("UPDATE task_runs SET status = ?, terminal_outcome = NULL WHERE task_id = ?").run( + status, + "task-1", + ); + expect( + pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0] + ?.receipt.decision.reasonCode, + ).toBe(`task_run_${status}`); + } + db.prepare("UPDATE task_runs SET terminal_outcome = 'blocked' WHERE task_id = ?").run("task-1"); + expect( + pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0]?.receipt + .decision.reasonCode, + ).toBe("task_run_blocked"); + for (const status of ["blocked", "succeeded", "failed", "cancelled", "lost"]) { + db.prepare("UPDATE flow_runs SET status = ? WHERE flow_id = ?").run(status, "flow-1"); + expect( + pageOwnerLifecycleReceipts({ stage: "flow", context, limit: 1, options }).entries[0] + ?.receipt.decision.reasonCode, + ).toBe(`flow_run_${status}`); + } + }); +}); diff --git a/src/audit/execution-owner-lifecycle-receipts.ts b/src/audit/execution-owner-lifecycle-receipts.ts new file mode 100644 index 000000000000..442598a068a8 --- /dev/null +++ b/src/audit/execution-owner-lifecycle-receipts.ts @@ -0,0 +1,445 @@ +/** Static owner-native cron/task/flow lifecycle projection for run inspection. */ +import type { DatabaseSync } from "node:sqlite"; +import type { + DecisionReceiptV1, + ExecutionIdentityContextV1, +} from "../../packages/gateway-protocol/src/index.js"; +import { + executeSqliteQuerySync, + executeSqliteQueryTakeFirstSync, + getNodeSqliteKysely, +} from "../infra/kysely-sync.js"; +import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js"; +import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js"; +import { tableExists } from "../state/openclaw-state-db-schema-helpers.js"; +import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js"; +import type { OpenClawStateDatabaseOptions } from "../state/openclaw-state-db.js"; +import { EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE } from "./execution-owner-lifecycle-binding-store.js"; + +type WithSqliteRowId = Row & { rowid: number }; +type OwnerLifecycleDatabase = { + cron_run_receipts: WithSqliteRowId; + execution_owner_lifecycle_bindings: OpenClawStateDatabase["execution_owner_lifecycle_bindings"]; + flow_runs: WithSqliteRowId; + task_runs: WithSqliteRowId; +}; +export type OwnerLifecycleStage = "cron" | "task" | "flow"; +export type OwnerLifecycleCursor = { occurredAt: number; rowId: number }; +type OwnerLifecycleDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle"; +type OwnerLifecycleReceiptEntry = { + receipt: DecisionReceiptV1; + selectorId: string; + displayProducer: OwnerLifecycleDisplayProducer; +}; +type OwnerLifecycleRow = { + executionId: string | null; + occurredAt: number; + owner: "cron_run_receipts" | "task_runs" | "flow_runs"; + recordId: string; + rowId: number; + status: string; +}; + +const KNOWN_STATUSES: Record> = { + cron: new Set(["running", "ok", "error", "skipped", "interrupted", "superseded"]), + task: new Set([ + "queued", + "running", + "succeeded", + "failed", + "timed_out", + "cancelled", + "lost", + "blocked", + ]), + flow: new Set([ + "queued", + "running", + "waiting", + "blocked", + "succeeded", + "failed", + "cancelled", + "lost", + ]), +}; +const OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR = "owner lifecycle cursor is no longer retained"; + +function ownerName(stage: OwnerLifecycleStage): OwnerLifecycleRow["owner"] { + return stage === "cron" ? "cron_run_receipts" : stage === "task" ? "task_runs" : "flow_runs"; +} + +function displayProducer(stage: OwnerLifecycleStage): OwnerLifecycleDisplayProducer { + return stage === "cron" + ? "cron-lifecycle" + : stage === "task" + ? "task-lifecycle" + : "flow-lifecycle"; +} + +function assertRetainedCursor(params: { + db: DatabaseSync; + stage: OwnerLifecycleStage; + contextId: string; + executionId: string; + after?: OwnerLifecycleCursor; +}): void { + if (!params.after) { + return; + } + const kysely = getNodeSqliteKysely(params.db); + const ownerQuery = + params.stage === "cron" + ? kysely + .selectFrom("cron_run_receipts") + .select("receipt_id as ownerId") + .where("rowid", "=", params.after.rowId) + .where("started_at_ms", "=", params.after.occurredAt) + : params.stage === "task" + ? kysely + .selectFrom("task_runs") + .select("task_id as ownerId") + .where("rowid", "=", params.after.rowId) + .where("created_at", "=", params.after.occurredAt) + : kysely + .selectFrom("flow_runs") + .select("flow_id as ownerId") + .where("rowid", "=", params.after.rowId) + .where("created_at", "=", params.after.occurredAt); + const owner = executeSqliteQueryTakeFirstSync(params.db, ownerQuery); + // Admission binds at most one owner per lifecycle kind to an execution. + // The exact execution match therefore rejects a different owner reusing this rowid. + const binding = owner + ? executeSqliteQueryTakeFirstSync( + params.db, + kysely + .selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + .select("owner_id") + .where("owner_kind", "=", params.stage) + .where("owner_id", "=", owner.ownerId) + .where("context_id", "=", params.contextId) + .where("execution_id", "=", params.executionId), + ) + : undefined; + if (!binding) { + throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR); + } +} + +function readRows(params: { + db: DatabaseSync; + stage: OwnerLifecycleStage; + contextId: string; + executionId: string; + after?: OwnerLifecycleCursor; + offset?: number; + limit: number; +}): OwnerLifecycleRow[] { + const owner = ownerName(params.stage); + if ( + !tableExists(params.db, owner) || + !tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + ) { + if (params.after) { + throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR); + } + return []; + } + assertRetainedCursor(params); + const kysely = getNodeSqliteKysely(params.db); + if (params.stage === "cron") { + let query = kysely + .selectFrom("cron_run_receipts") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef("execution_owner_lifecycle_bindings.owner_id", "=", "cron_run_receipts.receipt_id") + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"), + ) + .select([ + "cron_run_receipts.receipt_id as recordId", + "execution_owner_lifecycle_bindings.execution_id as executionId", + "cron_run_receipts.started_at_ms as occurredAt", + "cron_run_receipts.status", + "cron_run_receipts.rowid as rowId", + ]) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .orderBy("cron_run_receipts.started_at_ms", "asc") + .orderBy("cron_run_receipts.rowid", "asc") + .limit(params.limit); + if (params.after) { + query = query.where((eb) => + eb.or([ + eb("cron_run_receipts.started_at_ms", ">", params.after!.occurredAt), + eb.and([ + eb("cron_run_receipts.started_at_ms", "=", params.after!.occurredAt), + eb("cron_run_receipts.rowid", ">", params.after!.rowId), + ]), + ]), + ); + } else if (params.offset) { + query = query.offset(params.offset); + } + return executeSqliteQuerySync(params.db, query).rows.map((row) => + Object.assign(row, { owner }), + ); + } + if (params.stage === "task") { + let query = kysely + .selectFrom("task_runs") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id") + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"), + ) + .select([ + "task_runs.task_id as recordId", + "execution_owner_lifecycle_bindings.execution_id as executionId", + "task_runs.created_at as occurredAt", + "task_runs.status", + "task_runs.terminal_outcome as terminalOutcome", + "task_runs.rowid as rowId", + ]) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .orderBy("task_runs.created_at", "asc") + .orderBy("task_runs.rowid", "asc") + .limit(params.limit); + if (params.after) { + query = query.where((eb) => + eb.or([ + eb("task_runs.created_at", ">", params.after!.occurredAt), + eb.and([ + eb("task_runs.created_at", "=", params.after!.occurredAt), + eb("task_runs.rowid", ">", params.after!.rowId), + ]), + ]), + ); + } else if (params.offset) { + query = query.offset(params.offset); + } + return executeSqliteQuerySync(params.db, query).rows.map((row) => + Object.assign(row, { + owner, + status: row.terminalOutcome === "blocked" ? "blocked" : row.status, + }), + ); + } + let query = kysely + .selectFrom("flow_runs") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id") + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"), + ) + .select([ + "flow_runs.flow_id as recordId", + "execution_owner_lifecycle_bindings.execution_id as executionId", + "flow_runs.created_at as occurredAt", + "flow_runs.status", + "flow_runs.rowid as rowId", + ]) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .orderBy("flow_runs.created_at", "asc") + .orderBy("flow_runs.rowid", "asc") + .limit(params.limit); + if (params.after) { + query = query.where((eb) => + eb.or([ + eb("flow_runs.created_at", ">", params.after!.occurredAt), + eb.and([ + eb("flow_runs.created_at", "=", params.after!.occurredAt), + eb("flow_runs.rowid", ">", params.after!.rowId), + ]), + ]), + ); + } else if (params.offset) { + query = query.offset(params.offset); + } + return executeSqliteQuerySync(params.db, query).rows.map((row) => Object.assign(row, { owner })); +} + +function countRows(params: { + db: DatabaseSync; + stage: OwnerLifecycleStage; + contextId: string; + executionId?: string; +}): number { + const owner = ownerName(params.stage); + if ( + !tableExists(params.db, owner) || + !tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE) + ) { + return 0; + } + const kysely = getNodeSqliteKysely(params.db); + const query = + params.stage === "cron" + ? kysely + .selectFrom("cron_run_receipts") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef( + "execution_owner_lifecycle_bindings.owner_id", + "=", + "cron_run_receipts.receipt_id", + ) + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"), + ) + .select((eb) => eb.fn.countAll().as("count")) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .$if(params.executionId !== undefined, (qb) => + qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!), + ) + : params.stage === "task" + ? kysely + .selectFrom("task_runs") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id") + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"), + ) + .select((eb) => eb.fn.countAll().as("count")) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .$if(params.executionId !== undefined, (qb) => + qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!), + ) + : kysely + .selectFrom("flow_runs") + .innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) => + join + .onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id") + .on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"), + ) + .select((eb) => eb.fn.countAll().as("count")) + .where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId) + .$if(params.executionId !== undefined, (qb) => + qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!), + ); + return executeSqliteQueryTakeFirstSync(params.db, query)?.count ?? 0; +} + +function projectReceipt( + stage: OwnerLifecycleStage, + row: OwnerLifecycleRow, + context: ExecutionIdentityContextV1, +): DecisionReceiptV1 { + const exact = row.executionId === context.executionId; + const known = KNOWN_STATUSES[stage].has(row.status); + const valid = exact && known; + const missingEvidence = valid + ? [] + : [exact ? `decision.${stage}_owner_status` : "decision.execution_link"]; + return { + schemaVersion: 1, + receiptId: `${stage}:${row.recordId}`, + contextId: context.contextId, + executionId: context.executionId, + runId: context.runId, + actionId: row.recordId, + occurredAt: row.occurredAt, + action: { + family: stage === "cron" ? "scheduled-run" : stage === "task" ? "task" : "flow", + operation: "lifecycle", + summary: valid + ? `${stage === "cron" ? "Scheduled run" : stage === "task" ? "Task" : "Flow"} lifecycle: ${row.status.replaceAll("_", "-")}.` + : "Owner lifecycle evidence could not be matched exactly.", + }, + decision: { + outcome: valid ? "not-applicable" : "unknown", + reasonCode: valid + ? `${stage}_run_${row.status}` + : exact + ? `${stage}_run_status_unknown` + : `${stage}_run_execution_link_mismatch`, + }, + enforcement: { + coverageState: valid ? "attribution-only" : "unknown", + evaluatorRef: `${stage}-lifecycle-owner`, + policyRefs: [], + grantRefs: [], + contextFieldsUsed: ["contextId", "executionId"], + }, + source: { + owner: row.owner, + recordRef: row.recordId, + decisionBoundary: `${stage}.run.lifecycle`, + }, + missingEvidence, + remediation: valid + ? [] + : [ + { + code: "inspect_owner_execution_binding", + text: "Inspect the owner row and its exact admission binding before drawing a lifecycle conclusion.", + }, + ], + }; +} + +export function summarizeOwnerLifecycleReceipts(params: { + stage: OwnerLifecycleStage; + context: ExecutionIdentityContextV1; + options: OpenClawStateDatabaseOptions; +}): { count: number; coverageState?: "attribution-only" | "unknown"; missingEvidence: string[] } { + return ( + withExistingOpenClawStateDatabaseReadOnly(({ db }) => { + const count = countRows({ db, stage: params.stage, contextId: params.context.contextId }); + const exactCount = countRows({ + db, + stage: params.stage, + contextId: params.context.contextId, + executionId: params.context.executionId, + }); + const mismatch = count !== exactCount; + return { + count, + ...(count > 0 + ? { coverageState: mismatch ? ("unknown" as const) : ("attribution-only" as const) } + : {}), + missingEvidence: mismatch ? ["decision.execution_link"] : [], + }; + }, params.options) ?? { count: 0, missingEvidence: [] } + ); +} + +export function pageOwnerLifecycleReceipts(params: { + stage: OwnerLifecycleStage; + context: ExecutionIdentityContextV1; + after?: OwnerLifecycleCursor; + offset?: number; + limit: number; + options: OpenClawStateDatabaseOptions; +}): { entries: OwnerLifecycleReceiptEntry[]; nextCursor?: OwnerLifecycleCursor } { + const retainedRows = withExistingOpenClawStateDatabaseReadOnly( + ({ db }) => + runSqliteDeferredTransactionSync( + db, + () => + readRows({ + db, + stage: params.stage, + contextId: params.context.contextId, + executionId: params.context.executionId, + after: params.after, + offset: params.offset, + limit: params.limit + 1, + }), + { operationLabel: "owner lifecycle receipt page" }, + ), + params.options, + ); + if (!retainedRows && params.after) { + throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR); + } + const rows = retainedRows ?? []; + const hasMore = rows.length > params.limit; + const page = hasMore ? rows.slice(0, params.limit) : rows; + const last = page.at(-1); + return { + entries: page.map((row) => ({ + receipt: projectReceipt(params.stage, row, params.context), + selectorId: `${params.stage}-lifecycle:${row.occurredAt}:${row.rowId}`, + displayProducer: displayProducer(params.stage), + })), + ...(hasMore && last ? { nextCursor: { occurredAt: last.occurredAt, rowId: last.rowId } } : {}), + }; +} diff --git a/src/cron/isolated-agent/run-executor.ts b/src/cron/isolated-agent/run-executor.ts index 54ac837cacec..0a104cffd609 100644 --- a/src/cron/isolated-agent/run-executor.ts +++ b/src/cron/isolated-agent/run-executor.ts @@ -24,6 +24,7 @@ import { resolveScheduledToolPolicyContext } from "../../agents/scheduled-tool-p import { withLocalSessionPlacementTurnAdmission } from "../../agents/session-placement-admission.js"; import { resolveSessionRuntimeOverrideForProvider } from "../../agents/session-runtime-compat.js"; import { hasResolvedThinkingCatalogEntry } from "../../agents/thinking-runtime.js"; +import { withPostAdmissionExecutionOwnerBinding } from "../../audit/execution-owner-binding.js"; import type { ThinkLevel, VerboseLevel } from "../../auto-reply/thinking.js"; import type { CliSessionBinding } from "../../config/sessions.js"; import type { AgentDefaultsConfig } from "../../config/types.agent-defaults.js"; @@ -288,6 +289,7 @@ function createCronPromptExecutor(params: { Partial>, ) => void; onLaneWait?: (info?: { waiting?: boolean }) => void; + executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission; }) { const sessionFile = params.runSessionKey; const cronFallbacksOverride = @@ -396,15 +398,30 @@ function createCronPromptExecutor(params: { }); let acceptedContextEngineTurnCandidate: ContextEngineTurnAttemptFacts | undefined; const runId = params.cronSession.sessionEntry.sessionId; - const preparedRunAdmission = prepareAgentRunAdmission({ + const basePreparedRunAdmission = prepareAgentRunAdmission({ operationalRunInstance: createOperationalRunInstanceRef(runId), cfg: params.cfgWithAgentDefaults, facts: { runId, agentId: params.agentId, - ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" }, + ingress: params.executionIdentity?.ingress ?? { + kind: "schedule", + boundary: "cron.isolated-agent", + state: "present", + }, + ...(params.executionIdentity?.invoker ? { invoker: params.executionIdentity.invoker } : {}), }, }); + const preparedRunAdmission = params.executionIdentity?.onPostAdmission + ? withPostAdmissionExecutionOwnerBinding( + basePreparedRunAdmission, + params.executionIdentity.onPostAdmission, + ) + : basePreparedRunAdmission; + const onExecutionStarted = (info?: CronRunnerStartedInfo) => { + params.onExecutionStarted?.(info); + params.executionIdentity?.onExecutionStarted?.(); + }; const fallbackResult = await runWithModelFallback({ cfg: params.cfgWithAgentDefaults, provider: params.liveSelection.provider, @@ -454,7 +471,7 @@ function createCronPromptExecutor(params: { const isFallback = candidateStarted; candidateStarted = true; const notifyExecutionStarted = (info?: { lifecycleGeneration?: string }) => - params.onExecutionStarted?.({ + onExecutionStarted({ ...info, ...(isFallback ? { isFallback: true } : {}), provider: providerOverride, @@ -867,6 +884,7 @@ export async function executeCronRun(params: { Partial>, ) => void; onLaneWait?: (info?: { waiting?: boolean }) => void; + executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission; immutableThinkLevel: ThinkLevel | undefined; thinkingCatalog?: ModelCatalogEntry[]; loadThinkingCatalog: (provider: string, model: string) => Promise; @@ -923,6 +941,7 @@ export async function executeCronRun(params: { onExecutionStarted: params.onExecutionStarted, onExecutionPhase: params.onExecutionPhase, onLaneWait: params.onLaneWait, + executionIdentity: params.executionIdentity, }); const runStartedAt = params.runStartedAt ?? Date.now(); diff --git a/src/cron/isolated-agent/run.ts b/src/cron/isolated-agent/run.ts index 110516925a17..be55f115c8fa 100644 --- a/src/cron/isolated-agent/run.ts +++ b/src/cron/isolated-agent/run.ts @@ -93,6 +93,7 @@ export async function runCronIsolatedAgentTurn(params: { sessionKey: string; agentId?: string; lane?: string; + executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission; }): Promise { const admittedLifecycleGeneration = getAgentEventLifecycleGeneration(); const upstreamAbortSignal = params.abortSignal ?? params.signal; @@ -264,6 +265,7 @@ export async function runCronIsolatedAgentTurn(params: { runTimeoutOverrideMs: prepared.context.runTimeoutOverrideMs, suppressExecNotifyOnExit: prepared.context.suppressExecNotifyOnExit, pluginRegistry: prepared.context.pluginRegistry, + executionIdentity: params.executionIdentity, }; const execution = await prepared.context.sessionWorkAdmission.run(() => withAgentRunLifecycleGeneration(runLifecycleGeneration, () => diff --git a/src/cron/service/ops-run-preparation.ts b/src/cron/service/ops-run-preparation.ts index 522895566263..2d8ec99db942 100644 --- a/src/cron/service/ops-run-preparation.ts +++ b/src/cron/service/ops-run-preparation.ts @@ -38,7 +38,7 @@ import type { } from "./state.js"; import { cronFailureNotificationEventContext, emit, isImmediateCronRunMode } from "./state.js"; import { ensureLoaded, runPostPersistCronNotifications, warnIfDisabled } from "./store.js"; -import { tryCreateCronTaskRun, tryFinishCronTaskRun } from "./task-runs.js"; +import { tryCreateCronTaskRunHandle, tryFinishCronTaskRun } from "./task-runs.js"; import { applyJobResult, armTimer, type CronTriggerEvalOutcome } from "./timer.js"; type PreparedManualRun = @@ -70,6 +70,8 @@ type PreparedManualRun = export type ActivatedManualRun = Extract & { startedAt: number; taskRunId?: string; + taskId?: string; + flowId?: string; activeJobMarker?: CronActiveJobMarker; admittedJob: CronJob; executionJob: CronJob; @@ -503,12 +505,13 @@ export async function activatePreparedManualRun( job: activatedJob, runAtMs: startedAt, }); - const taskRunId = tryCreateCronTaskRun({ + const taskRun = tryCreateCronTaskRunHandle({ state, job: activatedJob, startedAt, publicRunId: prepared.runId ?? activation.runReceipt.receiptId, }); + const taskRunId = taskRun?.runId; const activeJobMarker = markManualCronJobActive(state, job); // Execute against a snapshot so later reload/merge can preserve delivery // target writeback from disk without mutating the running object. @@ -527,6 +530,8 @@ export async function activatePreparedManualRun( startedAt, runId: prepared.runId ?? taskRunId, taskRunId, + taskId: taskRun?.taskId, + flowId: taskRun?.flowId, activeJobMarker, admittedJob, executionJob, diff --git a/src/cron/service/ops-run.ts b/src/cron/service/ops-run.ts index cb1c37b2268f..05b42c6308c2 100644 --- a/src/cron/service/ops-run.ts +++ b/src/cron/service/ops-run.ts @@ -43,7 +43,10 @@ import type { } from "./state.js"; import { emit, isImmediateCronRunMode } from "./state.js"; import { ensureLoaded, publishCronRuntimeRows, runPostPersistCronNotifications } from "./store.js"; -import { tryFinishCronTaskRunWithoutHistory } from "./task-runs.js"; +import { + createCronOwnerExecutionIdentityAdmission, + tryFinishCronTaskRunWithoutHistory, +} from "./task-runs.js"; import { recordCronOutcomeForJob } from "./timer-outcome-events.js"; import { resolveCronRunScheduleOwnership, @@ -161,6 +164,12 @@ async function finishPreparedManualRun( streamScheduleKey: prepared.streamScheduleKey, streamSourceIdentity: prepared.streamSourceIdentity, runReceipt: prepared.runReceipt, + executionIdentity: createCronOwnerExecutionIdentityAdmission({ + state, + runReceipt: prepared.runReceipt, + taskId: prepared.taskId, + flowId: prepared.flowId, + }), }); } catch (err) { if (err instanceof CronRunReceiptRevisionError && err.reason === "owner-unavailable") { diff --git a/src/cron/service/ops.run-execution-binding.test.ts b/src/cron/service/ops.run-execution-binding.test.ts new file mode 100644 index 000000000000..11f7a9e807ef --- /dev/null +++ b/src/cron/service/ops.run-execution-binding.test.ts @@ -0,0 +1,232 @@ +// Focused proof that manual cron admission binds exact owner-native rows. +import { describe, expect, it, vi } from "vitest"; +import { + createDueIsolatedJob, + noopLogger, + setupCronRegressionFixtures, +} from "../../../test/helpers/cron/service-regression-fixtures.js"; +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js"; +import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js"; +import { + openOpenClawStateDatabase, + runOpenClawStateWriteTransaction, +} from "../../state/openclaw-state-db.js"; +import { createManagedTaskFlow } from "../../tasks/task-flow-registry.js"; +import { + resetTaskFlowRegistryForTests, + resetTaskRegistryForTests, +} from "../../tasks/task-runtime.test-helpers.js"; +import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js"; +import { saveCronStore } from "../store.js"; +import { + claimCronRunReceiptInDatabase, + finishCronRunReceipt, + prepareCronRunReceiptClaim, +} from "../store/run-receipt-store.js"; +import { run } from "./ops-run.js"; +import { createCronServiceState } from "./state.js"; +import { + createCronOwnerExecutionIdentityAdmission, + tryCreateCronTaskRunHandle, +} from "./task-runs.js"; + +const fixtures = setupCronRegressionFixtures({ + prefix: "cron-service-execution-binding-", +}); + +describe("cron run execution binding", () => { + it("binds the exact admitted execution to the cron receipt and task rows", async () => { + await withOpenClawTestState( + { layout: "state-only", prefix: "openclaw-cron-execution-binding-" }, + async () => { + resetTaskRegistryForTests(); + resetTaskFlowRegistryForTests(); + const store = fixtures.makeStorePath(); + const dueAt = Date.parse("2026-02-06T10:05:06.525Z"); + const job = createDueIsolatedJob({ + id: "exact-owner-binding", + nowMs: dueAt, + nextRunAtMs: dueAt, + }); + await saveCronStore(store.storePath, { version: 1, jobs: [job] }); + const runIsolatedAgentJob = vi.fn( + async (params: { + executionIdentity?: { + ingress: { kind: string }; + onPostAdmission?: (context: AdmittedRunContext) => void; + onExecutionStarted?: () => void; + }; + }) => { + const admitted = { + operationalRunInstance: { instanceId: "instance-exact", runId: "run-exact" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-exact", { + contextId: "context-exact", + executionId: "execution-exact", + now: dueAt, + }), + } satisfies AdmittedRunContext; + const beforeAdmissionSettles = openOpenClawStateDatabase().db; + expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe( + false, + ); + params.executionIdentity?.onPostAdmission?.(admitted); + expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe( + false, + ); + params.executionIdentity?.onExecutionStarted?.(); + return { status: "ok" as const }; + }, + ); + const state = createCronServiceState({ + cronEnabled: true, + storePath: store.storePath, + log: noopLogger, + nowMs: () => dueAt, + enqueueSystemEvent: vi.fn(), + requestHeartbeat: vi.fn(), + runIsolatedAgentJob, + }); + + await expect(run(state, job.id, "force")).resolves.toEqual({ ok: true, ran: true }); + expect(runIsolatedAgentJob).toHaveBeenCalledWith( + expect.objectContaining({ + executionIdentity: expect.objectContaining({ + ingress: expect.objectContaining({ kind: "schedule" }), + }), + }), + ); + const db = openOpenClawStateDatabase().db; + expect( + db + .prepare( + `SELECT binding.context_id, binding.execution_id, receipt.status, receipt.error_text + FROM cron_run_receipts AS receipt + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id + WHERE receipt.job_id = ?`, + ) + .get(job.id), + ).toEqual({ + context_id: "context-exact", + execution_id: "execution-exact", + status: "ok", + error_text: null, + }); + expect( + db + .prepare( + `SELECT binding.context_id, binding.execution_id, task.status + FROM task_runs AS task + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id + WHERE task.source_id = ? AND task.runtime = 'cron'`, + ) + .get(job.id), + ).toEqual({ + context_id: "context-exact", + execution_id: "execution-exact", + status: "succeeded", + }); + }, + ); + }); + + it("does not partially bind task or flow rows after the cron owner is replaced", async () => { + await withOpenClawTestState( + { layout: "state-only", prefix: "openclaw-cron-stale-execution-binding-" }, + async () => { + resetTaskRegistryForTests(); + resetTaskFlowRegistryForTests(); + const store = fixtures.makeStorePath(); + const dueAt = Date.parse("2026-02-06T10:06:06.525Z"); + const job = { + ...createDueIsolatedJob({ + id: "stale-owner-binding", + nowMs: dueAt, + nextRunAtMs: dueAt, + }), + agentId: "main", + }; + await saveCronStore(store.storePath, { version: 1, jobs: [job] }); + const state = createCronServiceState({ + cronEnabled: true, + storePath: store.storePath, + log: noopLogger, + nowMs: () => dueAt, + enqueueSystemEvent: vi.fn(), + requestHeartbeat: vi.fn(), + runIsolatedAgentJob: vi.fn(), + }); + const prepared = prepareCronRunReceiptClaim({ + storePath: store.storePath, + job, + agentId: job.agentId!, + startedAtMs: dueAt, + }); + const initial = runOpenClawStateWriteTransaction(({ db }) => + claimCronRunReceiptInDatabase({ + database: db, + prepared, + resolveAgentId: (current) => current.agentId!, + }), + ); + const task = tryCreateCronTaskRunHandle({ state, job, startedAt: dueAt }); + expect(task).toBeDefined(); + const flow = createManagedTaskFlow({ + ownerKey: "agent:main:main", + controllerId: "tests/stale-owner-binding", + goal: "Reject partial stale-owner provenance", + status: "running", + }); + expect(flow).not.toBeNull(); + if (!task || !flow) { + throw new Error("expected live task and flow owners"); + } + const executionIdentity = createCronOwnerExecutionIdentityAdmission({ + state, + runReceipt: initial, + taskId: task.taskId, + flowId: flow.flowId, + }); + const admitted = { + operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", { + contextId: "context-stale", + executionId: "execution-stale", + now: dueAt, + }), + } satisfies AdmittedRunContext; + executionIdentity.onPostAdmission?.(admitted); + const db = openOpenClawStateDatabase().db; + db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?").run( + 2_147_483_647, + initial.receiptId, + ); + const replacementPrepared = prepareCronRunReceiptClaim({ + storePath: store.storePath, + job, + agentId: job.agentId!, + startedAtMs: dueAt + 1, + }); + const replacement = runOpenClawStateWriteTransaction(({ db: transactionDb }) => + claimCronRunReceiptInDatabase({ + database: transactionDb, + prepared: replacementPrepared, + resolveAgentId: (current) => current.agentId!, + }), + ); + + executionIdentity.onExecutionStarted?.(); + expect( + tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"), + ).toBe(false); + finishCronRunReceipt({ + handle: replacement, + status: "skipped", + finishedAtMs: dueAt + 2, + }); + }, + ); + }); +}); diff --git a/src/cron/service/ops.test.ts b/src/cron/service/ops.test.ts index 9510df5403c3..492a473afa9e 100644 --- a/src/cron/service/ops.test.ts +++ b/src/cron/service/ops.test.ts @@ -1109,7 +1109,7 @@ describe("cron service ops seam coverage", () => { }); const taskRunId = reservationOffsetMs === undefined - ? taskRuns.tryCreateCronTaskRun({ state, job, startedAt }) + ? taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId : taskExecutor.createRunningTaskRunCore({ runtime: "cron", sourceId: job.id, @@ -1227,7 +1227,7 @@ describe("cron service ops seam coverage", () => { requestHeartbeat: vi.fn(), runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })), }); - const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt }); + const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId; if (!taskRunId) { throw new Error("expected invalid finalized cron task run"); } @@ -1296,7 +1296,7 @@ describe("cron service ops seam coverage", () => { runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })), onEvent: (event) => events.push(structuredClone(event)), }); - const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt }); + const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId; if (!taskRunId) { throw new Error("expected cron task run"); } @@ -1363,7 +1363,7 @@ describe("cron service ops seam coverage", () => { requestHeartbeat: vi.fn(), runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })), }); - const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt }); + const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId; if (!taskRunId) { throw new Error("expected cron task run"); } @@ -1437,7 +1437,11 @@ describe("cron service ops seam coverage", () => { requestHeartbeat: vi.fn(), runIsolatedAgentJob, }); - const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job: original, startedAt }); + const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ + state, + job: original, + startedAt, + })?.runId; if (!taskRunId) { throw new Error("expected cron task run"); } diff --git a/src/cron/service/run-admission-capacity.ts b/src/cron/service/run-admission-capacity.ts new file mode 100644 index 000000000000..50a79624ee80 --- /dev/null +++ b/src/cron/service/run-admission-capacity.ts @@ -0,0 +1,104 @@ +import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js"; +import type { CronServiceState } from "./state.js"; + +export function resolveRunConcurrency(): number { + return DEFAULT_CRON_MAX_CONCURRENT_RUNS; +} + +function acquireCronRunSlot(state: CronServiceState): () => void { + state.runAdmission.active += 1; + let released = false; + return () => { + if (released) { + return; + } + released = true; + state.runAdmission.active -= 1; + dispatchWaiters(state); + }; +} + +function dispatchWaiters(state: CronServiceState): void { + const admission = state.runAdmission; + if (state.stopped) { + cancelCronRunAdmissionWaiters(state); + return; + } + const maxConcurrentRuns = resolveRunConcurrency(); + while (admission.active < maxConcurrentRuns) { + const waiter = admission.waiters.shift(); + if (!waiter) { + break; + } + waiter(acquireCronRunSlot(state)); + } + if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) { + const listener = admission.capacityListener; + admission.capacityListener = null; + if (listener) { + queueMicrotask(listener); + } + } +} + +/** + * Acquire only the slots currently available to scheduled work. Unlike the + * waiter-based path used by direct runs, this never retains a timer batch while + * the pool is saturated. + */ +export function tryAcquireCronRunSlots( + state: CronServiceState, + requested: number, +): Array<() => void> { + if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) { + return []; + } + const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active); + return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state)); +} + +/** Keep the first wake-up until capacity release consumes or cancellation clears it. */ +export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void { + state.runAdmission.capacityListener ??= listener; +} + +async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> { + const admission = state.runAdmission; + if (state.stopped) { + return null; + } + if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) { + return acquireCronRunSlot(state); + } + return await new Promise<(() => void) | null>((resolve) => { + admission.waiters.push(resolve); + }); +} + +/** Wake queued work on stop so each caller can release its durable reservation. */ +export function cancelCronRunAdmissionWaiters(state: CronServiceState): void { + state.runAdmission.capacityListener = null; + const waiters = state.runAdmission.waiters.splice(0); + for (const waiter of waiters) { + waiter(null); + } +} + +/** Apply one service-level cap to every cron execution source. Queue waiters + * keep their job reservation, then recheck scheduler state before execution. + */ +export async function runWithCronAdmission( + state: CronServiceState, + execute: () => Promise, + acquiredRelease?: () => void, +): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> { + const release = acquiredRelease ?? (await acquireCronRunAdmission(state)); + if (!release) { + return { kind: "stopped" }; + } + try { + return { kind: "admitted", value: await execute() }; + } finally { + release(); + } +} diff --git a/src/cron/service/run-admission.ts b/src/cron/service/run-admission.ts index 14b4a73a47c6..bc47c4a7a47b 100644 --- a/src/cron/service/run-admission.ts +++ b/src/cron/service/run-admission.ts @@ -1,4 +1,3 @@ -import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js"; import { markCronJobActive } from "../active-jobs.js"; import { resolveCronJobConfigRevision } from "../config-revision.js"; import { createCronRunDiagnosticsFromError } from "../run-diagnostics.js"; @@ -17,6 +16,7 @@ import { normalizeCronRunErrorText } from "./execution-errors.js"; import { enrollForeignReceipt } from "./foreign-receipt-monitor.js"; import { recomputeJobNextRunAtMs } from "./jobs-scheduling.js"; import { locked } from "./locked.js"; +import { runWithCronAdmission } from "./run-admission-capacity.js"; import { activateServiceCronRunReceiptInDatabase, claimServiceCronRunReceiptInDatabase, @@ -27,7 +27,10 @@ import { import { applyCronRuntimeRowsToState, commitCronRuntimeRows } from "./runtime-store.js"; import { type CronServiceState, type DeferredCronNotifications, emit } from "./state.js"; import { ensureLoaded, runPostPersistCronNotifications } from "./store.js"; -import { tryCreateCronTaskRun } from "./task-runs.js"; +import { + createCronOwnerExecutionIdentityAdmission, + tryCreateCronTaskRunHandle, +} from "./task-runs.js"; import { runsDetachedFromMainSession, type TimedCronRunOutcome, @@ -35,88 +38,13 @@ import { import { authorCronRunCompletion, executeJobCoreWithTimeout } from "./timer-job-runner.js"; import { isRunnableJob } from "./timer-runnable.js"; -export function resolveRunConcurrency(): number { - return DEFAULT_CRON_MAX_CONCURRENT_RUNS; -} - -function acquireCronRunSlot(state: CronServiceState): () => void { - state.runAdmission.active += 1; - let released = false; - return () => { - if (released) { - return; - } - released = true; - state.runAdmission.active -= 1; - dispatchWaiters(state); - }; -} - -function dispatchWaiters(state: CronServiceState): void { - const admission = state.runAdmission; - if (state.stopped) { - cancelCronRunAdmissionWaiters(state); - return; - } - const maxConcurrentRuns = resolveRunConcurrency(); - while (admission.active < maxConcurrentRuns) { - const waiter = admission.waiters.shift(); - if (!waiter) { - break; - } - waiter(acquireCronRunSlot(state)); - } - if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) { - const listener = admission.capacityListener; - admission.capacityListener = null; - if (listener) { - queueMicrotask(listener); - } - } -} - -/** - * Acquire only the slots currently available to scheduled work. Unlike the - * waiter-based path used by direct runs, this never retains a timer batch while - * the pool is saturated. - */ -export function tryAcquireCronRunSlots( - state: CronServiceState, - requested: number, -): Array<() => void> { - if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) { - return []; - } - const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active); - return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state)); -} - -/** Keep the first wake-up until capacity release consumes or cancellation clears it. */ -export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void { - state.runAdmission.capacityListener ??= listener; -} - -async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> { - const admission = state.runAdmission; - if (state.stopped) { - return null; - } - if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) { - return acquireCronRunSlot(state); - } - return await new Promise<(() => void) | null>((resolve) => { - admission.waiters.push(resolve); - }); -} - -/** Wake queued work on stop so each caller can release its durable reservation. */ -export function cancelCronRunAdmissionWaiters(state: CronServiceState): void { - state.runAdmission.capacityListener = null; - const waiters = state.runAdmission.waiters.splice(0); - for (const waiter of waiters) { - waiter(null); - } -} +export { + cancelCronRunAdmissionWaiters, + resolveRunConcurrency, + runWithCronAdmission, + setCronRunCapacityListener, + tryAcquireCronRunSlots, +} from "./run-admission-capacity.js"; /** Track a persisted marker through shared admission and payload execution. */ export function reserveQueuedCronRun( @@ -543,25 +471,6 @@ export async function activateQueuedCronRun(params: { return { kind: "unavailable", reason: "stopped" }; } -/** Apply one service-level cap to every cron execution source. Queue waiters - * keep their job reservation, then recheck scheduler state before execution. - */ -export async function runWithCronAdmission( - state: CronServiceState, - execute: () => Promise, - acquiredRelease?: () => void, -): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> { - const release = acquiredRelease ?? (await acquireCronRunAdmission(state)); - if (!release) { - return { kind: "stopped" }; - } - try { - return { kind: "admitted", value: await execute() }; - } finally { - release(); - } -} - export async function executeQueuedCronRun(params: { state: CronServiceState; jobId: string; @@ -663,12 +572,13 @@ export async function executeQueuedCronRun(params: { const executionJob = structuredClone(started.job); executionJob.state.runningAtMs = started.startedAt; executionJob.state.lastError = undefined; - const taskRunId = tryCreateCronTaskRun({ + const taskRun = tryCreateCronTaskRunHandle({ state, job: executionJob, startedAt: started.startedAt, publicRunId: started.runReceipt.receiptId, }); + const taskRunId = taskRun?.runId; const activeJobMarker = markCronJobActive(executionJob.id, { preserveAcrossGenerationAdvance: !runsDetachedFromMainSession(executionJob), }); @@ -694,6 +604,12 @@ export async function executeQueuedCronRun(params: { runId: taskRunId, activeJobMarker, runReceipt: started.runReceipt, + executionIdentity: createCronOwnerExecutionIdentityAdmission({ + state, + runReceipt: started.runReceipt, + taskId: taskRun?.taskId, + flowId: taskRun?.flowId, + }), }); const result = state.deps.runSchedulerOwned ? await state.deps.runSchedulerOwned(execute) diff --git a/src/cron/service/run-recovery.test.ts b/src/cron/service/run-recovery.test.ts index b59614bc516c..63305976150e 100644 --- a/src/cron/service/run-recovery.test.ts +++ b/src/cron/service/run-recovery.test.ts @@ -15,7 +15,13 @@ import type { CronJob } from "../types.js"; import { proposeCronRunRecovery, recoverCronRunProposal } from "./run-recovery.js"; import { createCronServiceState } from "./state.js"; import { runPostPersistCronNotifications } from "./store.js"; -import { tryCreateCronTaskRun, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js"; +import { tryCreateCronTaskRunHandle, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js"; + +function tryCreateCronTaskRun( + params: Parameters[0], +): string | undefined { + return tryCreateCronTaskRunHandle(params)?.runId; +} const { logger, makeStorePath } = setupCronServiceSuite({ prefix: "cron-run-recovery-" }); diff --git a/src/cron/service/state.ts b/src/cron/service/state.ts index 34b6e4631996..4a5b2347d959 100644 --- a/src/cron/service/state.ts +++ b/src/cron/service/state.ts @@ -1,5 +1,7 @@ /** Cron service dependency, event, state, and public result types. */ +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import type { ExecutionIdentityAdmissionFacts } from "../../audit/execution-identity-admission.js"; import type { ReplyPayload } from "../../auto-reply/reply-payload.js"; import type { CronConfig } from "../../config/types.cron.js"; import type { HeartbeatRunResult, HeartbeatWakeRequest } from "../../infra/heartbeat-wake.js"; @@ -188,6 +190,7 @@ export type CronServiceDeps = { onExecutionStarted?: (info?: CronAgentExecutionStarted) => void; onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void; onLaneWait?: (info?: { waiting?: boolean }) => void; + executionIdentity?: CronExecutionIdentityAdmission; }) => Promise< { summary?: string; @@ -267,6 +270,13 @@ export type CronServiceDeps = { onEvent?: (evt: CronEvent, context?: CronEventContext) => void; }; +export type CronExecutionIdentityAdmission = { + ingress: ExecutionIdentityAdmissionFacts["ingress"]; + invoker?: ExecutionIdentityAdmissionFacts["invoker"]; + onPostAdmission?: (context: AdmittedRunContext) => void; + onExecutionStarted?: () => void; +}; + /** Cron deps after optional defaults have been made concrete. */ type CronServiceDepsInternal = Omit & { nowMs: () => number; diff --git a/src/cron/service/task-runs.test.ts b/src/cron/service/task-runs.test.ts index 0527fc5fae57..15cfdbb15f0c 100644 --- a/src/cron/service/task-runs.test.ts +++ b/src/cron/service/task-runs.test.ts @@ -20,10 +20,16 @@ import { createCronServiceState as createCronServiceStateBase } from "./state.js import { getActiveCronTaskRunId, findCronTaskRunRecoveryInDatabase, - tryCreateCronTaskRun, + tryCreateCronTaskRunHandle, tryFinishCronTaskRun, tryFinishCronTaskRunWithoutHistory, } from "./task-runs.js"; + +function tryCreateCronTaskRun( + params: Parameters[0], +): string | undefined { + return tryCreateCronTaskRunHandle(params)?.runId; +} import { executeJobCoreWithTimeout } from "./timer-job-runner.js"; function createCronServiceState( diff --git a/src/cron/service/task-runs.ts b/src/cron/service/task-runs.ts index 9ff0a996c60f..ddb5f8da7b81 100644 --- a/src/cron/service/task-runs.ts +++ b/src/cron/service/task-runs.ts @@ -2,6 +2,10 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { randomUUID } from "node:crypto"; import type { DatabaseSync } from "node:sqlite"; +import { + createExecutionStartedOwnerBinding, + isRetainedExecutionOwnerBinding, +} from "../../audit/execution-owner-binding.js"; import { normalizeAgentId, resolveAgentIdFromSessionKey } from "../../routing/session-key.js"; import { CRON_TASK_KIND } from "../../tasks/cron-task-contract.js"; import { @@ -11,7 +15,11 @@ import { findTaskByRunId, recordTaskRunProgressByRunIdCore, } from "../../tasks/task-executor.js"; -import { listTaskRecordsByRuntimeSourceIdInDatabase } from "../../tasks/task-registry.store.sqlite.js"; +import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js"; +import { + bindTaskRunExecution, + listTaskRecordsByRuntimeSourceIdInDatabase, +} from "../../tasks/task-registry.store.sqlite.js"; import type { JsonValue, TaskRecord, TaskStatus } from "../../tasks/task-registry.types.js"; import { CRON_AGENT_SELECTION_REQUIRED_MESSAGE, @@ -20,6 +28,10 @@ import { import { createCronExecutionId } from "../run-id.js"; import type { CronRunLogEntry } from "../run-log-types.js"; import { cronStoreKey } from "../store/key.js"; +import { + bindCronRunReceiptExecution, + type CronRunReceiptHandle, +} from "../store/run-receipt-store.js"; import { cronRunLogEntryToTaskDetail, cronRunStatusToTaskStatus, @@ -38,7 +50,7 @@ import type { CronRunStatus, } from "../types.js"; import { normalizeCronRunErrorText } from "./execution-errors.js"; -import type { CronEvent, CronServiceState } from "./state.js"; +import type { CronEvent, CronExecutionIdentityAdmission, CronServiceState } from "./state.js"; import { CRON_TASK_RUNNING_PROGRESS_SUMMARY } from "./task-ledger.js"; function requireCronAgentId(agentId: string | undefined): string { @@ -64,6 +76,55 @@ export function getActiveCronTaskRunId(): string | undefined { return activeCronTaskRunId.getStore(); } +/** Carries exact admission into the first post-admission owner lifecycle phase. */ +export function createCronOwnerExecutionIdentityAdmission(params: { + state: CronServiceState; + runReceipt: CronRunReceiptHandle; + taskId?: string; + flowId?: string; +}): CronExecutionIdentityAdmission { + const ownerBinding = createExecutionStartedOwnerBinding((admitted) => { + try { + const receiptResult = bindCronRunReceiptExecution({ + admitted, + handle: params.runReceipt, + }); + const taskResult = params.taskId + ? isRetainedExecutionOwnerBinding(receiptResult) + ? bindTaskRunExecution({ admitted, taskId: params.taskId }) + : receiptResult + : undefined; + const flowParentResult = params.taskId ? taskResult : receiptResult; + const flowResult = params.flowId + ? isRetainedExecutionOwnerBinding(receiptResult) && + isRetainedExecutionOwnerBinding(flowParentResult) + ? bindTaskFlowExecution({ admitted, flowId: params.flowId }) + : flowParentResult + : undefined; + if ( + [receiptResult, taskResult, flowResult].some( + (result) => result === "mismatch" || result === "missing", + ) + ) { + params.state.deps.log.warn( + { receiptResult, taskResult, flowResult }, + "cron: exact execution identity binding was not retained", + ); + } + } catch (error) { + params.state.deps.log.warn( + { error }, + "cron: failed to retain exact execution identity binding", + ); + } + }); + return { + ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" }, + onPostAdmission: ownerBinding.onPostAdmission, + onExecutionStarted: ownerBinding.onExecutionStarted, + }; +} + /** Updates an active cron task with the exact transcript identity reported by its runner. */ export function tryUpdateCronTaskRunSession( state: CronServiceState, @@ -88,13 +149,12 @@ export function tryUpdateCronTaskRunSession( } } -/** Creates a best-effort detached task row keyed to the persisted execution start. */ -export function tryCreateCronTaskRun(params: { +export function tryCreateCronTaskRunHandle(params: { state: CronServiceState; job: CronJob; startedAt: number; publicRunId?: string; -}): string | undefined { +}): { runId: string; taskId: string; flowId?: string } | undefined { const runId = createCronTaskRunId(params.job.id, params.startedAt, params.publicRunId); return tryCreateCronTaskRunRecord({ state: params.state, @@ -214,7 +274,7 @@ function tryCreateCronTaskRunRecord(params: { startedAt: number; runId: string; childSessionKey?: string; -}): string | undefined { +}): { runId: string; taskId: string; flowId?: string } | undefined { try { const childSessionKey = params.childSessionKey; const effectiveJobAgentId = params.job @@ -252,7 +312,11 @@ function tryCreateCronTaskRunRecord(params: { ); return undefined; } - return params.runId; + return { + runId: params.runId, + taskId: task.taskId, + ...(task.parentFlowId ? { flowId: task.parentFlowId } : {}), + }; } catch (error) { params.state.deps.log.warn( { jobId: params.jobId, error }, @@ -341,9 +405,9 @@ export function tryFinishCronTaskRun( result.taskRunId ?? createCronTaskRunId(entry.jobId, startedAt, entry.runId); try { const existingCandidate = findTaskByRunId(candidateRunId); - const taskRunId = + const created = existingCandidate?.runtime === "cron" - ? candidateRunId + ? undefined : tryCreateCronTaskRunRecord({ state, job: result.job ?? result.event.job, @@ -352,6 +416,7 @@ export function tryFinishCronTaskRun( runId: candidateRunId, childSessionKey: entry.sessionKey, }); + const taskRunId = existingCandidate?.runtime === "cron" ? candidateRunId : created?.runId; if (!taskRunId) { return; } @@ -414,7 +479,7 @@ export function tryFinishCronTaskRun( updated = finalize(taskRunId); } else { // A terminal event still owns one durable row if its active mirror vanished. - const recreatedRunId = tryCreateCronTaskRunRecord({ + const recreated = tryCreateCronTaskRunRecord({ state, job: result.job ?? result.event.job, jobId: entry.jobId, @@ -422,8 +487,8 @@ export function tryFinishCronTaskRun( runId: taskRunId, childSessionKey: entry.sessionKey, }); - if (recreatedRunId) { - updated = finalize(recreatedRunId); + if (recreated) { + updated = finalize(recreated.runId); } } } diff --git a/src/cron/service/timer-execution-timeout.ts b/src/cron/service/timer-execution-timeout.ts index cea870104a01..b5a0e804e483 100644 --- a/src/cron/service/timer-execution-timeout.ts +++ b/src/cron/service/timer-execution-timeout.ts @@ -125,6 +125,7 @@ export type ExecuteJobCoreOptions = { onExecutionStarted?: (info?: CronAgentExecutionStarted) => void; onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void; onLaneWait?: (info?: { waiting?: boolean }) => void; + executionIdentity?: import("./state.js").CronExecutionIdentityAdmission; /** Revalidates the durable run fence after awaited planning and before effects. */ assertRunCurrent?: () => void; streamBatch?: string; diff --git a/src/cron/service/timer-execution.ts b/src/cron/service/timer-execution.ts index cbf1d7c793ec..b41e88f89ae9 100644 --- a/src/cron/service/timer-execution.ts +++ b/src/cron/service/timer-execution.ts @@ -459,6 +459,7 @@ async function executeDetachedCronJob( onExecutionStarted: options?.onExecutionStarted, onExecutionPhase: options?.onExecutionPhase, onLaneWait: options?.onLaneWait, + executionIdentity: options?.executionIdentity, }); if (abortSignal?.aborted) { diff --git a/src/cron/service/timer-job-runner.ts b/src/cron/service/timer-job-runner.ts index f16d4081cc45..d6fddb281fe0 100644 --- a/src/cron/service/timer-job-runner.ts +++ b/src/cron/service/timer-job-runner.ts @@ -51,6 +51,7 @@ type CronCoreRunOptions = { streamScheduleKey?: string; streamSourceIdentity?: string; runReceipt?: import("../store/run-receipt-store.js").CronRunReceiptHandle; + executionIdentity?: import("./state.js").CronExecutionIdentityAdmission; }; async function deliverPrimaryWebhook( @@ -274,6 +275,7 @@ async function executeJobCoreWithTimeoutUnfinalized( onExecutionStarted: noteExecutionStarted, onExecutionPhase: accumulateExecution, assertRunCurrent, + executionIdentity: opts?.executionIdentity, }; const corePromise = withCronTaskRunId(opts?.runId, () => executeJobCore(state, job, runAbortController.signal, coreOptions), @@ -361,6 +363,7 @@ async function executeJobCoreWithTimeoutUnfinalized( onExecutionPhase: deferTimeoutUntilExecutionStart ? watchdog.notePhase : undefined, onLaneWait: deferTimeoutUntilExecutionStart ? noteLaneState : undefined, assertRunCurrent, + executionIdentity: opts?.executionIdentity, }; const corePromise = withCronTaskRunId(opts?.runId, () => executeJobCore(state, job, runAbortController.signal, coreOptions), diff --git a/src/cron/store/run-receipt-store.test.ts b/src/cron/store/run-receipt-store.test.ts index 0a96e197797d..62e91ef4e781 100644 --- a/src/cron/store/run-receipt-store.test.ts +++ b/src/cron/store/run-receipt-store.test.ts @@ -1,4 +1,6 @@ import { describe, expect, it } from "vitest"; +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js"; import { openOpenClawStateDatabase, runOpenClawStateWriteTransaction, @@ -9,6 +11,7 @@ import type { CronJob } from "../types.js"; import { cronStoreKey } from "./key.js"; import { assertCronRunReceiptCurrent, + bindCronRunReceiptExecution, claimCronRunReceiptInDatabase, CronRunReceiptConflictError, CronRunReceiptRevisionError, @@ -127,14 +130,56 @@ describe("cron run receipt store", () => { ]); }); + it("rejects a delayed binding after a successor replaces its exact owner", async () => { + const { storePath } = await makeStorePath(); + const job = makeJob("stale-binding"); + await saveCronStore(storePath, { version: 1, jobs: [job] }); + const abandoned = claim(storePath, job, 230); + openOpenClawStateDatabase() + .db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?") + .run(2_147_483_647, abandoned.receiptId); + const replacement = claim(storePath, job, 240); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", { + contextId: "context-stale", + executionId: "execution-stale", + }), + }; + + expect(bindCronRunReceiptExecution({ admitted, handle: abandoned })).toBe("missing"); + expect(bindCronRunReceiptExecution({ admitted, handle: replacement })).toBe("bound"); + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT owner_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'cron'`, + ) + .all(), + ).toEqual([{ owner_id: replacement.receiptId }]); + + finishCronRunReceipt({ handle: replacement, status: "ok", finishedAtMs: 250 }); + }); + it("prunes old terminal receipts while preserving the active and 64 newest rows", async () => { const { storePath } = await makeStorePath(); const job = makeJob("retention"); await saveCronStore(storePath, { version: 1, jobs: [job] }); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-retention", runId: "run-retention" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-retention", { + contextId: "context-retention", + executionId: "execution-retention", + }), + }; const finishedReceiptIds: string[] = []; for (let index = 0; index < 70; index += 1) { const handle = claim(storePath, job, 1_000 + index * 2); finishedReceiptIds.push(handle.receiptId); + if (index === 0 || index === 69) { + expect(bindCronRunReceiptExecution({ admitted, handle })).toBe("bound"); + } finishCronRunReceipt({ handle, status: "ok", @@ -142,6 +187,7 @@ describe("cron run receipt store", () => { }); } const active = claim(storePath, job, 2_000); + expect(bindCronRunReceiptExecution({ admitted, handle: active })).toBe("bound"); const retained = receipts(storePath, job.id); const retainedIds = new Set(retained.map((receipt) => receipt.receiptId)); @@ -158,6 +204,21 @@ describe("cron run receipt store", () => { for (const receiptId of finishedReceiptIds.slice(-64)) { expect(retainedIds.has(receiptId)).toBe(true); } + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT binding.owner_id + FROM execution_owner_lifecycle_bindings AS binding + JOIN cron_run_receipts AS receipt ON receipt.receipt_id = binding.owner_id + WHERE binding.owner_kind = 'cron' AND receipt.job_id = ? + ORDER BY owner_id`, + ) + .all(job.id), + ).toEqual( + [active.receiptId, finishedReceiptIds.at(-1)] + .toSorted((left, right) => (left ?? "").localeCompare(right ?? "")) + .map((owner_id) => ({ owner_id })), + ); finishCronRunReceipt({ handle: active, status: "skipped", finishedAtMs: 2_001 }); }); diff --git a/src/cron/store/run-receipt-store.ts b/src/cron/store/run-receipt-store.ts index 77a5b5e37583..331dea3e2445 100644 --- a/src/cron/store/run-receipt-store.ts +++ b/src/cron/store/run-receipt-store.ts @@ -1,6 +1,15 @@ import crypto from "node:crypto"; import type { DatabaseSync } from "node:sqlite"; import type { Selectable } from "kysely"; +import type { AdmittedRunContext } from "../../agents/admitted-run-context.js"; +import { + executionOwnerBindingFromAdmission, + type ExecutionOwnerBindingResult, +} from "../../audit/execution-owner-binding.js"; +import { + bindExecutionOwnerLifecycleMetadata, + deleteExecutionOwnerLifecycleMetadata, +} from "../../audit/execution-owner-lifecycle-binding-store.js"; import { executeSqliteQuerySync, executeSqliteQueryTakeFirstSync, @@ -176,6 +185,38 @@ function withReceiptWrite( return result; } +/** Binds the exact admitted execution to its authoritative receipt without changing lifecycle. */ +export function bindCronRunReceiptExecution(params: { + admitted: AdmittedRunContext; + handle: CronRunReceiptHandle; + options?: OpenClawStateDatabaseOptions; +}): ExecutionOwnerBindingResult { + const binding = executionOwnerBindingFromAdmission(params.admitted); + if (!binding) { + return "disabled"; + } + return withReceiptWrite( + "cron.run-receipt.execution-binding", + params.options ?? {}, + (database) => { + try { + assertCronRunReceiptOwnedInDatabase({ database, handle: params.handle }); + } catch (error) { + if (!(error instanceof CronRunReceiptRevisionError)) { + throw error; + } + return "missing"; + } + return bindExecutionOwnerLifecycleMetadata({ + db: database, + ownerKind: "cron", + ownerId: params.handle.receiptId, + binding, + }); + }, + ); +} + function isReceiptStatus(value: string): value is CronRunReceiptStatus { return ( value === "running" || @@ -329,6 +370,11 @@ function pruneTerminalReceipts(database: DatabaseSync, storeKey: string, jobId: const receiptIds = terminalIds .slice(index, index + CRON_RUN_RECEIPT_DELETE_BATCH_SIZE) .map((row) => row.receipt_id); + deleteExecutionOwnerLifecycleMetadata({ + db: database, + ownerKind: "cron", + ownerIds: receiptIds, + }); executeSqliteQuerySync( database, query(database) diff --git a/src/gateway/agent-turn/agent-run-dispatch.ts b/src/gateway/agent-turn/agent-run-dispatch.ts index fc6dd16fa861..530db63a2f5c 100644 --- a/src/gateway/agent-turn/agent-run-dispatch.ts +++ b/src/gateway/agent-turn/agent-run-dispatch.ts @@ -6,6 +6,7 @@ import { type AgentRunTerminalOutcome, } from "../../agents/agent-run-terminal-outcome.js"; import type { PreparedAgentCommandRuntimeContext } from "../../agents/command/prepare.js"; +import type { AgentCommandOpts } from "../../agents/command/types.js"; import { createCronCreatorAuthorityCapability, runWithCronCreatorAuthorityCapability, @@ -15,6 +16,10 @@ import type { MainSessionRecoveryPendingTarget } from "../../agents/main-session import { isAgentRunRestartAbortReason } from "../../agents/run-termination.js"; import { normalizeAgentRunTimeoutPhase } from "../../agents/run-timeout-attribution.js"; import { runWithCanonicalSkillWorkspace } from "../../agents/skill-workshop-workspace-context.js"; +import { + createExecutionStartedOwnerBinding, + isRetainedExecutionOwnerBinding, +} from "../../audit/execution-owner-binding.js"; import { readAgentRunTerminalOutcome } from "../../channels/turn/agent-run-terminal-outcome.js"; import { agentCommandFromGatewayIngress } from "../../commands/agent.js"; import { isAbortError } from "../../infra/abort-signal.js"; @@ -22,7 +27,10 @@ import { clearAgentRunContext } from "../../infra/agent-run-registry.js"; import { formatErrorMessageWithCode, readErrorName } from "../../infra/errors.js"; import { defaultRuntime } from "../../runtime.js"; import { createRunningTaskRun } from "../../tasks/detached-task-runtime.js"; +import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js"; import { mapAgentRunTerminalOutcomeToTaskStatus } from "../../tasks/task-registry-common.js"; +import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js"; +import type { TaskRecord } from "../../tasks/task-registry.types.js"; import { normalizeDeliveryContext } from "../../utils/delivery-context.shared.js"; import type { ChatAbortControllerEntry } from "../chat-abort.js"; import { @@ -136,9 +144,10 @@ export function dispatchAgentRunFromGateway(params: { }) { const shouldTrackTask = params.taskTrackingMode === "cli"; let taskTracked = false; + let trackedTask: TaskRecord | undefined; if (shouldTrackTask) { try { - taskTracked = Boolean( + trackedTask = createRunningTaskRun({ runtime: "cli", sourceId: params.runId, @@ -155,8 +164,8 @@ export function dispatchAgentRunFromGateway(params: { task: params.ingressOpts.message, deliveryStatus: "not_applicable", startedAt: Date.now(), - }), - ); + }) ?? undefined; + taskTracked = Boolean(trackedTask); } catch (err) { // Best-effort only: background task tracking must not block agent runs. // Still surface the swallowed error so non-transient tracking failures stay observable. @@ -188,12 +197,49 @@ export function dispatchAgentRunFromGateway(params: { params.ingressOpts, readAgentRunDispatchExecutionIdentity(params), ); + const trackedTaskBinding = trackedTask + ? createExecutionStartedOwnerBinding( + (admitted: Parameters>[0]) => { + try { + const taskResult = bindTaskRunExecution({ admitted, taskId: trackedTask.taskId }); + const flowResult = trackedTask.parentFlowId + ? isRetainedExecutionOwnerBinding(taskResult) + ? bindTaskFlowExecution({ admitted, flowId: trackedTask.parentFlowId }) + : taskResult + : undefined; + if ( + [taskResult, flowResult].some( + (result) => result === "mismatch" || result === "missing", + ) + ) { + params.context.logGateway.warn( + `exact tracked-task execution binding was not retained for ${params.runId}`, + ); + } + } catch (error) { + params.context.logGateway.warn( + `failed to retain tracked-task execution binding ${params.runId}: ${formatForLog(error)}`, + ); + } + }, + ) + : undefined; + const ingressOptsWithTaskBinding = trackedTask + ? { + ...ingressOptsWithSpawnFacts, + onPostAdmittedRunContext: trackedTaskBinding?.onPostAdmission, + onExecutionStarted: () => { + ingressOptsWithSpawnFacts.onExecutionStarted?.(); + trackedTaskBinding?.onExecutionStarted(); + }, + } + : ingressOptsWithSpawnFacts; const runAgent = () => runWithCanonicalSkillWorkspace(params.canonicalSkillWorkspaceDir, () => agentCommandFromGatewayIngress( cronCreatorAuthorityCapability - ? { ...ingressOptsWithSpawnFacts, cronCreatorAuthorityCapability } - : ingressOptsWithSpawnFacts, + ? { ...ingressOptsWithTaskBinding, cronCreatorAuthorityCapability } + : ingressOptsWithTaskBinding, defaultRuntime, params.context.deps, { diff --git a/src/gateway/agent-turn/agent-run-execution-phase.ts b/src/gateway/agent-turn/agent-run-execution-phase.ts index 357109262bbe..6e65bceba8a1 100644 --- a/src/gateway/agent-turn/agent-run-execution-phase.ts +++ b/src/gateway/agent-turn/agent-run-execution-phase.ts @@ -1,6 +1,9 @@ import { ErrorCodes, errorShape } from "../../../packages/gateway-protocol/src/index.js"; import { getAdmittedRunDelegatedAuthority } from "../../agents/admitted-run-context.js"; -import { attachAgentCommandAdmissionFacts } from "../../agents/agent-command-admission-facts.js"; +import { + attachAgentCommandAdmissionFacts, + attachAgentCommandRecoveryAdmissionFacts, +} from "../../agents/agent-command-admission-facts.js"; import type { AgentRunTerminalOutcome } from "../../agents/agent-run-terminal-outcome.js"; import { prepareGitCoauthorAttribution } from "../../agents/git-coauthor-attribution.js"; import { repairMainSessionRecoveryMutation } from "../../agents/main-session-recovery/main-session-recovery-lifecycle.js"; @@ -265,7 +268,9 @@ export function startAgentRunExecution(params: { ); const localUserIngress = getGatewayLocalUserIngress(params.client); - if (localUserIngress) { + if (params.isRestartRecoveryResumeRun) { + attachAgentCommandRecoveryAdmissionFacts(runContext); + } else if (localUserIngress) { attachAgentCommandAdmissionFacts(runContext, localUserIngress.facts); } finalizePreparedAgentRunUserTurn(prepared.userTurn); diff --git a/src/gateway/hooks-mapping.execution-identity.test.ts b/src/gateway/hooks-mapping.execution-identity.test.ts new file mode 100644 index 000000000000..8fae2871c911 --- /dev/null +++ b/src/gateway/hooks-mapping.execution-identity.test.ts @@ -0,0 +1,28 @@ +// Hook mapping identity coverage keeps mapping IDs as ingress attribution only. +import { describe, expect, it } from "vitest"; +import { applyHookMappings, resolveHookMappings } from "./hooks-mapping.js"; + +describe("hook mapping execution identity", () => { + it("carries the configured mapping id on the resolved agent action", async () => { + const mappings = resolveHookMappings({ + mappings: [ + { + id: "source-gmail", + match: { path: "gmail" }, + action: "agent", + messageTemplate: "{{messages[0].subject}}", + }, + ], + }); + const result = await applyHookMappings(mappings, { + payload: { messages: [{ subject: "Hello" }] }, + headers: {}, + url: new URL("http://127.0.0.1:18789/hooks/gmail"), + path: "gmail", + }); + expect(result).toMatchObject({ + ok: true, + action: { kind: "agent", mappingId: "source-gmail", message: "Hello" }, + }); + }); +}); diff --git a/src/gateway/hooks-mapping.ts b/src/gateway/hooks-mapping.ts index c973c067e756..28348ef473ad 100644 --- a/src/gateway/hooks-mapping.ts +++ b/src/gateway/hooks-mapping.ts @@ -49,6 +49,7 @@ type HookMappingContext = { type HookAction = | { kind: "wake"; + mappingId: string; text: string; mode: "now" | "next-heartbeat"; agentId?: string; @@ -57,6 +58,7 @@ type HookAction = } | { kind: "agent"; + mappingId: string; message: string; name?: string; agentId?: string; @@ -272,6 +274,7 @@ function buildActionFromMapping( ok: true, action: { kind: "wake", + mappingId: mapping.id, text, mode: mapping.wakeMode ?? "now", agentId: mapping.agentId, @@ -285,6 +288,7 @@ function buildActionFromMapping( ok: true, action: { kind: "agent", + mappingId: mapping.id, message, name: renderOptional(mapping.name, ctx), agentId: mapping.agentId, @@ -318,6 +322,7 @@ function mergeAction( const mode = override.mode === "next-heartbeat" ? "next-heartbeat" : (baseWake?.mode ?? "now"); return validateAction({ kind: "wake", + mappingId: base.mappingId, text, mode, agentId: override.agentId ?? baseWake?.agentId, @@ -332,6 +337,7 @@ function mergeAction( override.wakeMode === "next-heartbeat" ? "next-heartbeat" : (baseAgent?.wakeMode ?? "now"); return validateAction({ kind: "agent", + mappingId: base.mappingId, message, wakeMode, name: override.name ?? baseAgent?.name, diff --git a/src/gateway/hooks.ts b/src/gateway/hooks.ts index ec4155d3419c..9475a21e746a 100644 --- a/src/gateway/hooks.ts +++ b/src/gateway/hooks.ts @@ -306,6 +306,8 @@ export type HookAgentDispatchPayload = Omit & { sourcePath: string; allowUnsafeExternalContent?: boolean; externalContentSource?: HookExternalContentSource; + /** Configured ingress source attribution; never an authenticated principal. */ + mappingId?: string; }; const listHookChannelValues = () => ["last", ...listChannelPlugins().map((plugin) => plugin.id)]; diff --git a/src/gateway/server-cron.ts b/src/gateway/server-cron.ts index 6bba3afdc372..3d1dedeeb7f4 100644 --- a/src/gateway/server-cron.ts +++ b/src/gateway/server-cron.ts @@ -850,6 +850,7 @@ export function buildGatewayCronService(params: { onExecutionStarted, onExecutionPhase, onLaneWait, + executionIdentity, }) => { const { agentId, cfg: runtimeConfig } = resolveCronAgent(job.agentId); const sessionKey = resolveCronSessionTargetSessionKey(job.sessionTarget) ?? `cron:${job.id}`; @@ -862,6 +863,7 @@ export function buildGatewayCronService(params: { onExecutionStarted, onExecutionPhase, onLaneWait, + executionIdentity, agentId, sessionKey, lane: "cron", diff --git a/src/gateway/server-methods/audit.test.ts b/src/gateway/server-methods/audit.test.ts index 81eed6370df5..d01153f46aad 100644 --- a/src/gateway/server-methods/audit.test.ts +++ b/src/gateway/server-methods/audit.test.ts @@ -314,34 +314,6 @@ describe("audit gateway methods", () => { decisionLimit: 20, }); - await runAuditHandler("audit.run.inspect", { - runId: "run-1", - executionCursor: "1", - decisionCursor: "1", - decisionLimit: 25, - }); - expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({ - runId: "run-1", - executionOffset: 1, - executionLimit: 50, - decisionCursor: "1", - decisionLimit: 25, - }); - - await runAuditHandler("audit.run.inspect", { - runId: "run-1", - executionCursor: "001", - decisionCursor: "001", - decisionLimit: 25, - }); - expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({ - runId: "run-1", - executionOffset: 1, - executionLimit: 50, - decisionCursor: "001", - decisionLimit: 25, - }); - await runAuditHandler("audit.run.inspect", { executionId: "execution-1", decisionCursor: "1", @@ -503,6 +475,45 @@ describe("audit gateway methods", () => { expect(JSON.stringify(result)).not.toContain('"decisions"'); }); + it.each(["1", "001"])( + "preserves mirrored numeric cursor %s for execution paging", + async (decisionCursor) => { + await runAuditHandler("audit.run.inspect", { + runId: "run-1", + executionCursor: decisionCursor, + decisionCursor, + decisionLimit: 25, + }); + + expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({ + runId: "run-1", + executionOffset: 1, + executionLimit: 50, + decisionCursor, + decisionLimit: 25, + }); + }, + ); + + it.each(["a:2000:42", "m:2000:42", "g:2000:42", "c:2000:42", "t:2000:42", "f:2000:42"])( + "treats mirrored owner decision cursor %s as decision-only", + async (decisionCursor) => { + await runAuditHandler("audit.run.inspect", { + runId: "run-1", + executionCursor: decisionCursor, + decisionCursor, + decisionLimit: 25, + }); + + expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({ + runId: "run-1", + executionLimit: 50, + decisionCursor, + decisionLimit: 25, + }); + }, + ); + it("rejects malformed run inspection before storage access", async () => { expect( await runAuditHandler("audit.run.inspect", { runId: "", extra: true }), diff --git a/src/gateway/server-methods/audit.ts b/src/gateway/server-methods/audit.ts index 36065cd56cad..7cb991fc0965 100644 --- a/src/gateway/server-methods/audit.ts +++ b/src/gateway/server-methods/audit.ts @@ -49,6 +49,10 @@ function serializeAuditRunInspectResult( return result; } +function isOwnerDecisionCursor(value: string): boolean { + return parsePositiveAuditCursor(value) === null && isExecutionDecisionCursor(value); +} + /** Preserve the shipped audit.list result shape for run/tool-only clients. */ function mapLegacyAuditEvent( event: AgentRunAuditEventRecord | ToolActionAuditEventRecord, @@ -198,9 +202,7 @@ export const auditHandlers: GatewayRequestHandlers = { typeof params.runId !== "string" || (params.executionCursor === decisionCursor && decisionCursor !== undefined && - (decisionCursor.startsWith("a:") || - decisionCursor.startsWith("m:") || - decisionCursor.startsWith("g:"))) + isOwnerDecisionCursor(decisionCursor)) ? undefined : parsePositiveAuditCursor(params.executionCursor); if ( diff --git a/src/gateway/server.hooks-admission.test.ts b/src/gateway/server.hooks-admission.test.ts index e5249104f13b..ef2076d43350 100644 --- a/src/gateway/server.hooks-admission.test.ts +++ b/src/gateway/server.hooks-admission.test.ts @@ -1,10 +1,12 @@ /** Focused HTTP coverage for hook admission feedback and pending replay behavior. */ +import fs from "node:fs/promises"; import { Agent, request as httpRequest } from "node:http"; +import path from "node:path"; import { afterEach, describe, expect, test, vi } from "vitest"; import { createDeferred } from "../../test/helpers/promise.js"; import { resolveMainSessionKeyFromConfig } from "../config/sessions.js"; import { DEFAULT_WEBHOOK_MAX_BODY_BYTES } from "../infra/http-body.js"; -import { drainSystemEvents } from "../infra/system-events.js"; +import { drainSystemEvents, peekSystemEventEntries } from "../infra/system-events.js"; import { cronIsolatedRun, installGatewayTestHooks, @@ -25,11 +27,11 @@ afterEach(() => { async function postHook( port: number, - path: string, + hookPath: string, body: Record, idempotencyKey: string, ): Promise { - return await fetch(`http://127.0.0.1:${port}${path}`, { + return await fetch(`http://127.0.0.1:${port}${hookPath}`, { method: "POST", headers: { Authorization: `Bearer ${HOOK_TOKEN}`, @@ -106,6 +108,24 @@ async function postOversizedChunkedHook(port: number): Promise<{ } } +async function writeHookTransformModule(moduleName: string, source: string): Promise { + const configPath = process.env.OPENCLAW_CONFIG_PATH; + if (!configPath) { + throw new Error("expected OPENCLAW_CONFIG_PATH"); + } + const transformsDir = path.join(path.dirname(configPath), "hooks", "transforms"); + await fs.mkdir(transformsDir, { recursive: true }); + await fs.writeFile(path.join(transformsDir, moduleName), source, "utf8"); +} + +function readExecutionIdentityCall(index: number): unknown { + const call = cronIsolatedRun.mock.calls[index]?.[0]; + if (!call || typeof call !== "object" || !("executionIdentity" in call)) { + return undefined; + } + return call.executionIdentity; +} + describe("gateway hook admission", () => { test("flushes an oversized chunked hook response before closing the socket", async () => { testState.hooksConfig = { enabled: true, token: HOOK_TOKEN }; @@ -144,6 +164,75 @@ describe("gateway hook admission", () => { }); }); + test("keeps direct hooks unattributed and mapped IDs as ingress attribution", async () => { + testState.hooksConfig = { + enabled: true, + token: HOOK_TOKEN, + mappings: [ + { + id: "gmail-source", + match: { path: "gmail" }, + action: "agent", + messageTemplate: "New email from {{messages[0].from}}", + }, + ], + }; + await withGatewayServer(async ({ port }) => { + cronIsolatedRun.mockClear(); + cronIsolatedRun.mockResolvedValue({ status: "ok", summary: "done" }); + expect( + (await postHook(port, "/hooks/agent", { message: "Direct" }, "direct-source")).status, + ).toBe(200); + expect( + (await postHook(port, "/hooks/gmail", { messages: [{ from: "Ada" }] }, "mapped-source")) + .status, + ).toBe(200); + expect(readExecutionIdentityCall(0)).toEqual({ + ingress: { kind: "webhook", boundary: "gateway.hooks.agent", state: "present" }, + }); + expect(readExecutionIdentityCall(1)).toEqual({ + ingress: { + kind: "webhook", + boundary: "gateway.hooks.agent", + state: "present", + rawSourceRef: "gmail-source", + }, + }); + }); + }); + + test("returns visible suppression without admitting a hook run", async () => { + await writeHookTransformModule( + "suppress.mjs", + "export default function suppress() { return null; }", + ); + testState.hooksConfig = { + enabled: true, + token: HOOK_TOKEN, + mappings: [ + { + id: "suppressed-source", + match: { path: "suppressed" }, + action: "agent", + messageTemplate: "private {{payload.subject}}", + transform: { module: "suppress.mjs" }, + }, + ], + }; + await withGatewayServer(async ({ port }) => { + cronIsolatedRun.mockClear(); + const response = await postHook( + port, + "/hooks/suppressed", + { subject: "secret" }, + "suppressed-source", + ); + expect(response.status).toBe(204); + expect(cronIsolatedRun).not.toHaveBeenCalled(); + expect(peekSystemEventEntries(resolveMainSessionKeyFromConfig())).toEqual([]); + }); + }); + test("shares one pending persistent dispatch without losing its session target", async () => { testState.hooksConfig = { enabled: true, diff --git a/src/gateway/server/hooks-request-handler.ts b/src/gateway/server/hooks-request-handler.ts index cd53bc8dd75f..8eeefc460b77 100644 --- a/src/gateway/server/hooks-request-handler.ts +++ b/src/gateway/server/hooks-request-handler.ts @@ -598,6 +598,7 @@ export function createHooksRequestHandler( model: action.model, thinking: action.thinking, timeoutSeconds: action.timeoutSeconds, + mappingId: action.mappingId, allowUnsafeExternalContent: action.allowUnsafeExternalContent, externalContentSource: resolveMappedHookExternalContentSource({ subPath, diff --git a/src/gateway/server/hooks.ts b/src/gateway/server/hooks.ts index c62578eb6839..e45906acd732 100644 --- a/src/gateway/server/hooks.ts +++ b/src/gateway/server/hooks.ts @@ -484,6 +484,14 @@ export function createGatewayHooksRequestHandler(params: { // cannot starve them. Aggregate capacity stays bounded by the lane // group that owns both lanes. lane: CommandLane.HookDispatch, + executionIdentity: { + ingress: { + kind: "webhook", + boundary: "gateway.hooks.agent", + state: "present", + ...(acceptedValue.mappingId ? { rawSourceRef: acceptedValue.mappingId } : {}), + }, + }, abortSignal: startupAbortController.signal, onExecutionStarted: () => { // Existing runner-entry callbacks are the final owner-boundary fence: diff --git a/src/state/openclaw-database-maintenance.test.ts b/src/state/openclaw-database-maintenance.test.ts index e1ae8dd9e1c8..27662b6b8969 100644 --- a/src/state/openclaw-database-maintenance.test.ts +++ b/src/state/openclaw-database-maintenance.test.ts @@ -103,6 +103,43 @@ describe("OpenClaw database maintenance schema validation", () => { } }); + it("keeps lifecycle bindings additive and keyed only by canonical owner identity", () => { + const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf( + "CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (", + ); + const endMarker = ") STRICT;"; + const end = start >= 0 ? OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start) : -1; + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + const previousSchema = `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`; + const database = createGlobalDatabase(); + try { + expect(() => + assertSqliteSchemaContains(database, "previous global schema", previousSchema), + ).not.toThrow(); + expect( + database.prepare("PRAGMA table_info(execution_owner_lifecycle_bindings)").all(), + ).toEqual([ + { cid: 0, name: "owner_kind", type: "TEXT", notnull: 1, dflt_value: null, pk: 1 }, + { cid: 1, name: "owner_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 2 }, + { cid: 2, name: "context_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 }, + { cid: 3, name: "execution_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 }, + ]); + expect( + database + .prepare( + `SELECT COUNT(*) AS count + FROM sqlite_schema + WHERE type = 'index' AND tbl_name = 'execution_owner_lifecycle_bindings' + AND sql IS NOT NULL`, + ) + .get(), + ).toEqual({ count: 0 }); + } finally { + database.close(); + } + }); + it("accepts compatible future columns in shared-state and agent databases", () => { const globalDatabase = createGlobalDatabase(); const agentDatabase = createAgentDatabase(); diff --git a/src/state/openclaw-state-db-contract.ts b/src/state/openclaw-state-db-contract.ts index 5a294b0443f3..98791cef09cc 100644 --- a/src/state/openclaw-state-db-contract.ts +++ b/src/state/openclaw-state-db-contract.ts @@ -18,6 +18,7 @@ export const FIRST_USE_STATE_TABLES = [ "node_worker_launches", "operator_approval_execution_identities", "execution_decision_facts", + "execution_owner_lifecycle_bindings", "outbound_message_execution_bindings", "outbound_message_progress", ] as const; diff --git a/src/state/openclaw-state-db.generated.d.ts b/src/state/openclaw-state-db.generated.d.ts index 7eb6903a5afe..392f30254ebd 100644 --- a/src/state/openclaw-state-db.generated.d.ts +++ b/src/state/openclaw-state-db.generated.d.ts @@ -700,6 +700,13 @@ export interface ExecutionIdentityContexts { run_id: string; } +export interface ExecutionOwnerLifecycleBindings { + context_id: string; + execution_id: string; + owner_id: string; + owner_kind: string; +} + export interface FleetCells { container_name: string; created_at_ms: number; @@ -1871,6 +1878,7 @@ export interface DB { exec_approvals_config: ExecApprovalsConfig; execution_decision_facts: ExecutionDecisionFacts; execution_identity_contexts: ExecutionIdentityContexts; + execution_owner_lifecycle_bindings: ExecutionOwnerLifecycleBindings; fleet_cells: FleetCells; flow_runs: FlowRuns; gateway_boot_lifecycle: GatewayBootLifecycle; diff --git a/src/state/openclaw-state-schema.sql b/src/state/openclaw-state-schema.sql index 0a511e1e6d85..53c0fa5e6ee5 100644 --- a/src/state/openclaw-state-schema.sql +++ b/src/state/openclaw-state-schema.sql @@ -304,6 +304,16 @@ CREATE INDEX IF NOT EXISTS execution_decision_facts_context_occurred_idx CREATE INDEX IF NOT EXISTS execution_decision_facts_run_occurred_idx ON execution_decision_facts (run_id, occurred_at, receipt_id); +-- Exact admission identity stays separate from owner-native lifecycle rows so +-- older readers retain byte-compatible cron/task/flow table definitions. +CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings ( + owner_kind TEXT NOT NULL, + owner_id TEXT NOT NULL, + context_id TEXT NOT NULL, + execution_id TEXT NOT NULL, + PRIMARY KEY (owner_kind, owner_id) +) STRICT; + CREATE TABLE IF NOT EXISTS session_state_events ( sequence INTEGER PRIMARY KEY AUTOINCREMENT, dedupe_key TEXT UNIQUE, diff --git a/src/tasks/task-flow-registry.store.sqlite.ts b/src/tasks/task-flow-registry.store.sqlite.ts index 378f1369036c..7ff12af22cec 100644 --- a/src/tasks/task-flow-registry.store.sqlite.ts +++ b/src/tasks/task-flow-registry.store.sqlite.ts @@ -1,7 +1,21 @@ // Persists managed task-flow records through the OpenClaw SQLite state database. import type { DatabaseSync } from "node:sqlite"; import type { Insertable, Selectable } from "kysely"; -import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js"; +import type { AdmittedRunContext } from "../agents/admitted-run-context.js"; +import { + executionOwnerBindingFromAdmission, + type ExecutionOwnerBindingResult, +} from "../audit/execution-owner-binding.js"; +import { + bindExecutionOwnerLifecycleMetadata, + deleteExecutionOwnerLifecycleMetadata, + pruneOrphanedExecutionOwnerLifecycleMetadata, +} from "../audit/execution-owner-lifecycle-binding-store.js"; +import { + executeSqliteQuerySync, + executeSqliteQueryTakeFirstSync, + getNodeSqliteKysely, +} from "../infra/kysely-sync.js"; import { normalizeSqliteNumber } from "../infra/sqlite-number.js"; import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js"; import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js"; @@ -9,6 +23,7 @@ import { closeOpenClawStateDatabase, openOpenClawStateDatabase, runOpenClawStateWriteTransaction, + type OpenClawStateDatabaseOptions, } from "../state/openclaw-state-db.js"; import type { TaskFlowRegistryStoreSnapshot } from "./task-flow-registry.store.types.js"; import { @@ -42,7 +57,10 @@ function serializeJson(value: unknown): string | null { return value === undefined ? null : JSON.stringify(value); } -function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode { +function resolveFlowSyncMode(row: { + sync_mode: string | null; + shape: string | null; +}): TaskFlowSyncMode { // Older single_task rows did not persist sync_mode; preserve their mirrored semantics. const syncMode = parseOptionalTaskFlowSyncMode(row.sync_mode); if (syncMode) { @@ -51,6 +69,29 @@ function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode { return row.shape === "single_task" ? "task_mirrored" : "managed"; } +function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode { + return resolveFlowSyncMode(row); +} + +function isFlowExecutionOwnerActive(row: { + sync_mode: string | null; + shape: string | null; + status: string; + cancel_requested_at: number | null; + ended_at: number | null; +}): boolean { + const syncMode = resolveFlowSyncMode(row); + const status = parseTaskFlowStatus(row.status); + if (row.cancel_requested_at !== null || row.ended_at !== null) { + return false; + } + // Mirrored `blocked` is derived from a terminal task; managed `blocked` + // remains live while its controller waits for the blocking task. + return syncMode === "task_mirrored" + ? status === "queued" || status === "running" + : status === "queued" || status === "running" || status === "waiting" || status === "blocked"; +} + function rowToFlowRecord(row: FlowRegistryRow): TaskFlowRecord { const endedAt = normalizeSqliteNumber(row.ended_at); const cancelRequestedAt = normalizeSqliteNumber(row.cancel_requested_at); @@ -233,12 +274,14 @@ export function saveTaskFlowRegistryStateToSqlite(snapshot: TaskFlowRegistryStor const flowIds = [...snapshot.flows.keys()]; if (flowIds.length === 0) { executeSqliteQuerySync(db, kysely.deleteFrom("flow_runs")); + pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow"); return; } pruneFlowsNotInSnapshot({ db, ids: flowIds }); for (const flow of snapshot.flows.values()) { upsertFlowRow(db, bindFlowRecord(flow)); } + pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow"); }); } @@ -248,12 +291,48 @@ export function upsertTaskFlowRegistryRecordToSqlite(flow: TaskFlowRecord) { }); } +/** Binds only the exact flow selected before admission; lifecycle settlement stays owner-native. */ +export function bindTaskFlowExecution(params: { + admitted: AdmittedRunContext; + flowId: string; + options?: OpenClawStateDatabaseOptions; +}): ExecutionOwnerBindingResult { + const binding = executionOwnerBindingFromAdmission(params.admitted); + if (!binding) { + return "disabled"; + } + return runOpenClawStateWriteTransaction( + ({ db }) => { + const kysely = getFlowRegistryKysely(db); + const current = executeSqliteQueryTakeFirstSync( + db, + kysely + .selectFrom("flow_runs") + .select(["flow_id", "sync_mode", "shape", "status", "cancel_requested_at", "ended_at"]) + .where("flow_id", "=", params.flowId), + ); + if (!current || !isFlowExecutionOwnerActive(current)) { + return "missing"; + } + return bindExecutionOwnerLifecycleMetadata({ + db, + ownerKind: "flow", + ownerId: current.flow_id, + binding, + }); + }, + params.options, + { operationLabel: "task.flow.execution-binding" }, + ); +} + export function deleteTaskFlowRegistryRecordFromSqlite(flowId: string) { withWriteTransaction(({ db }) => { executeSqliteQuerySync( db, getFlowRegistryKysely(db).deleteFrom("flow_runs").where("flow_id", "=", flowId), ); + deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "flow", ownerIds: [flowId] }); }); } diff --git a/src/tasks/task-flow-registry.store.test.ts b/src/tasks/task-flow-registry.store.test.ts index 745634c4a85e..d14f8ad7ddb8 100644 --- a/src/tasks/task-flow-registry.store.test.ts +++ b/src/tasks/task-flow-registry.store.test.ts @@ -2,7 +2,10 @@ import { statSync } from "node:fs"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { AdmittedRunContext } from "../agents/admitted-run-context.js"; +import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js"; import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js"; +import { tableExists } from "../state/openclaw-state-db-schema-helpers.js"; import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js"; import { openOpenClawStateDatabase } from "../state/openclaw-state-db.js"; import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js"; @@ -14,6 +17,7 @@ import { setFlowWaiting, } from "./task-flow-registry.js"; import { + bindTaskFlowExecution, loadTaskFlowRegistryStateFromSqlite, loadTaskFlowRegistryStateFromSqliteReadOnly, saveTaskFlowRegistryStateToSqlite, @@ -307,13 +311,25 @@ describe("task-flow-registry store runtime", () => { ...createStoredFlow(), flowId: `flow-large-${index}`, controllerId: `tests/large-flow-${index}`, + status: "running", createdAt: index, updatedAt: index, + cancelRequestedAt: undefined, + endedAt: undefined, }; flows.set(flow.flowId, flow); } saveTaskFlowRegistryStateToSqlite({ flows }); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-flow-prune", runId: "run-flow-prune" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-prune", { + contextId: "context-flow-prune", + executionId: "execution-flow-prune", + }), + }; + expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-0" })).toBe("bound"); + expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-1199" })).toBe("bound"); const retainedFlows = new Map([...flows].slice(100)); saveTaskFlowRegistryStateToSqlite({ flows: retainedFlows }); @@ -321,6 +337,104 @@ describe("task-flow-registry store runtime", () => { expect(restored.flows.size).toBe(1_100); expect(restored.flows.has("flow-large-0")).toBe(false); expect(restored.flows.has("flow-large-1199")).toBe(true); + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT owner_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'flow' + ORDER BY owner_id`, + ) + .all(), + ).toEqual([{ owner_id: "flow-large-1199" }]); + }); + }); + + it("binds only source-live flow owners across managed and mirrored lifecycles", async () => { + await withFlowRegistryTempDir(async () => { + const managed: TaskFlowRecord = { + ...createStoredFlow(), + flowId: "flow-binding-managed", + status: "blocked", + endedAt: undefined, + cancelRequestedAt: undefined, + }; + const mirrored: TaskFlowRecord = { + ...createStoredFlow(), + flowId: "flow-binding-mirrored", + syncMode: "task_mirrored", + controllerId: undefined, + status: "running", + endedAt: undefined, + cancelRequestedAt: undefined, + }; + const managedTerminal: TaskFlowRecord = { + ...managed, + flowId: "flow-binding-managed-terminal", + status: "succeeded", + endedAt: 200, + }; + const mirroredTerminal: TaskFlowRecord = { + ...mirrored, + flowId: "flow-binding-mirrored-terminal", + status: "blocked", + endedAt: 201, + }; + const managedCancelling: TaskFlowRecord = { + ...managed, + flowId: "flow-binding-managed-cancelling", + status: "running", + cancelRequestedAt: 199, + }; + saveTaskFlowRegistryStateToSqlite({ + flows: new Map( + [managed, mirrored, managedTerminal, mirroredTerminal, managedCancelling].map((flow) => [ + flow.flowId, + flow, + ]), + ), + }); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-flow-owner", runId: "run-flow-owner" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-owner", { + contextId: "context-flow-owner", + executionId: "execution-flow-owner", + }), + }; + + expect( + tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"), + ).toBe(false); + expect(bindTaskFlowExecution({ admitted, flowId: managedTerminal.flowId })).toBe("missing"); + expect(bindTaskFlowExecution({ admitted, flowId: mirroredTerminal.flowId })).toBe("missing"); + expect(bindTaskFlowExecution({ admitted, flowId: managedCancelling.flowId })).toBe("missing"); + expect( + tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"), + ).toBe(false); + expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("bound"); + expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("bound"); + + saveTaskFlowRegistryStateToSqlite({ + flows: new Map([ + [managed.flowId, { ...managed, status: "succeeded", endedAt: 210 }], + [mirrored.flowId, { ...mirrored, status: "blocked", endedAt: 211 }], + [managedTerminal.flowId, managedTerminal], + [mirroredTerminal.flowId, mirroredTerminal], + [managedCancelling.flowId, managedCancelling], + ]), + }); + expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("missing"); + expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("missing"); + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT owner_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'flow' + ORDER BY owner_id`, + ) + .all(), + ).toEqual([{ owner_id: managed.flowId }, { owner_id: mirrored.flowId }]); }); }); diff --git a/src/tasks/task-registry.store.sqlite.ts b/src/tasks/task-registry.store.sqlite.ts index 02d7a0989861..aceaa62a01dc 100644 --- a/src/tasks/task-registry.store.sqlite.ts +++ b/src/tasks/task-registry.store.sqlite.ts @@ -1,7 +1,21 @@ // Persists task registry records and events through the OpenClaw SQLite state database. import type { DatabaseSync } from "node:sqlite"; import type { Insertable, Selectable } from "kysely"; -import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js"; +import type { AdmittedRunContext } from "../agents/admitted-run-context.js"; +import { + executionOwnerBindingFromAdmission, + type ExecutionOwnerBindingResult, +} from "../audit/execution-owner-binding.js"; +import { + bindExecutionOwnerLifecycleMetadata, + deleteExecutionOwnerLifecycleMetadata, + pruneOrphanedExecutionOwnerLifecycleMetadata, +} from "../audit/execution-owner-lifecycle-binding-store.js"; +import { + executeSqliteQuerySync, + executeSqliteQueryTakeFirstSync, + getNodeSqliteKysely, +} from "../infra/kysely-sync.js"; import { assertSqliteTableIntegrity } from "../infra/sqlite-integrity.js"; import { normalizeSqliteNumber } from "../infra/sqlite-number.js"; import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js"; @@ -13,6 +27,7 @@ import { openOpenClawStateDatabase, runOpenClawStateWriteTransaction, type OpenClawStateDatabase, + type OpenClawStateDatabaseOptions, } from "../state/openclaw-state-db.js"; import { parseDeliveryContextJson, parseSqliteJsonValue } from "./task-registry.sqlite.shared.js"; import type { TaskRegistryStoreSnapshot } from "./task-registry.store.types.js"; @@ -329,6 +344,7 @@ function deleteTaskRowsWithDeliveryState(db: DatabaseSync, taskId: string): void kysely.deleteFrom("task_delivery_state").where("task_id", "=", taskId), ); executeSqliteQuerySync(db, kysely.deleteFrom("task_runs").where("task_id", "=", taskId)); + deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "task", ownerIds: [taskId] }); } function openTaskRegistryDatabase(): TaskRegistryDatabase { @@ -435,6 +451,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho if (taskIds.length === 0) { executeSqliteQuerySync(db, kysely.deleteFrom("task_delivery_state")); executeSqliteQuerySync(db, kysely.deleteFrom("task_runs")); + pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task"); return; } pruneRowsNotInSnapshot({ @@ -462,6 +479,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho for (const state of snapshot.deliveryStates.values()) { replaceTaskDeliveryStateRow(db, bindTaskDeliveryState(state)); } + pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task"); }); } @@ -471,6 +489,42 @@ export function upsertTaskRegistryRecordToSqlite(task: TaskRecord) { }); } +/** Binds only the exact task row selected before admission; runId is never a join key. */ +export function bindTaskRunExecution(params: { + admitted: AdmittedRunContext; + taskId: string; + options?: OpenClawStateDatabaseOptions; +}): ExecutionOwnerBindingResult { + const binding = executionOwnerBindingFromAdmission(params.admitted); + if (!binding) { + return "disabled"; + } + return runOpenClawStateWriteTransaction( + ({ db }) => { + const kysely = getTaskRegistryKysely(db); + const current = executeSqliteQueryTakeFirstSync( + db, + kysely + .selectFrom("task_runs") + .select(["task_id", "status", "ended_at"]) + .where("task_id", "=", params.taskId), + ); + const status = current ? parseTaskStatus(current.status) : undefined; + if (!current || (status !== "queued" && status !== "running") || current.ended_at !== null) { + return "missing"; + } + return bindExecutionOwnerLifecycleMetadata({ + db, + ownerKind: "task", + ownerId: current.task_id, + binding, + }); + }, + params.options, + { operationLabel: "task.run.execution-binding" }, + ); +} + export function upsertTaskWithDeliveryStateToSqlite(params: { task: TaskRecord; deliveryState?: TaskDeliveryState; diff --git a/src/tasks/task-registry.store.test.ts b/src/tasks/task-registry.store.test.ts index 275b250539e2..3d5453c732d3 100644 --- a/src/tasks/task-registry.store.test.ts +++ b/src/tasks/task-registry.store.test.ts @@ -3,6 +3,8 @@ import { statSync } from "node:fs"; import path from "node:path"; import type { DatabaseSync } from "node:sqlite"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import type { AdmittedRunContext } from "../agents/admitted-run-context.js"; +import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js"; import { emitAgentEvent } from "../infra/agent-events.js"; import { executeSqliteQuerySync, @@ -14,6 +16,7 @@ import { readSqliteNumberPragma } from "../infra/sqlite-pragma.test-support.js"; import { resetLogger, setLoggerOverride } from "../logging/logger.js"; import { loggingState } from "../logging/state.js"; import { createWarnLogCapture } from "../logging/test-helpers/warn-log-capture.js"; +import { tableExists } from "../state/openclaw-state-db-schema-helpers.js"; import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js"; import { closeOpenClawStateDatabase, @@ -42,6 +45,7 @@ import { type TaskRegistryObserverEvent, } from "./task-registry.store.js"; import { + bindTaskRunExecution, loadTaskRegistryStateFromSqlite, loadTaskRegistryStateFromSqliteReadOnly, loadTaskRegistryStateFromSqliteReadOnlyResult, @@ -1249,6 +1253,15 @@ describe("task-registry store runtime", () => { } saveTaskRegistryStateToSqlite({ tasks, deliveryStates }); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-task-prune", runId: "run-task-prune" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-prune", { + contextId: "context-task-prune", + executionId: "execution-task-prune", + }), + }; + expect(bindTaskRunExecution({ admitted, taskId: "task-large-0" })).toBe("bound"); + expect(bindTaskRunExecution({ admitted, taskId: "task-large-1199" })).toBe("bound"); const retainedTasks = new Map([...tasks].slice(100)); const retainedDeliveryStates = new Map([...deliveryStates].slice(100)); saveTaskRegistryStateToSqlite({ @@ -1261,6 +1274,81 @@ describe("task-registry store runtime", () => { expect(restored.deliveryStates.size).toBe(1_100); expect(restored.tasks.has("task-large-0")).toBe(false); expect(restored.tasks.has("task-large-1199")).toBe(true); + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT owner_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'task' + ORDER BY owner_id`, + ) + .all(), + ).toEqual([{ owner_id: "task-large-1199" }]); + }, + ); + }); + + it("binds only live task owners and retains their metadata after terminalization", async () => { + await withOpenClawTestState( + { layout: "state-only", prefix: "openclaw-task-binding-owner-" }, + async () => { + const active = { ...createStoredTask(), taskId: "task-binding-active" }; + const terminal: TaskRecord = { + ...createStoredTask(), + taskId: "task-binding-terminal", + status: "succeeded", + endedAt: 200, + }; + const stale: TaskRecord = { + ...createStoredTask(), + taskId: "task-binding-stale", + endedAt: 199, + }; + saveTaskRegistryStateToSqlite({ + tasks: new Map([ + [active.taskId, active], + [terminal.taskId, terminal], + [stale.taskId, stale], + ]), + deliveryStates: new Map(), + }); + const admitted: AdmittedRunContext = { + operationalRunInstance: { instanceId: "instance-task-owner", runId: "run-task-owner" }, + executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-owner", { + contextId: "context-task-owner", + executionId: "execution-task-owner", + }), + }; + + expect( + tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"), + ).toBe(false); + expect(bindTaskRunExecution({ admitted, taskId: terminal.taskId })).toBe("missing"); + expect(bindTaskRunExecution({ admitted, taskId: stale.taskId })).toBe("missing"); + expect( + tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"), + ).toBe(false); + expect(bindTaskRunExecution({ admitted, taskId: active.taskId })).toBe("bound"); + + const finished = { ...active, status: "succeeded" as const, endedAt: 210 }; + saveTaskRegistryStateToSqlite({ + tasks: new Map([ + [finished.taskId, finished], + [terminal.taskId, terminal], + [stale.taskId, stale], + ]), + deliveryStates: new Map(), + }); + expect(bindTaskRunExecution({ admitted, taskId: finished.taskId })).toBe("missing"); + expect( + openOpenClawStateDatabase() + .db.prepare( + `SELECT owner_id + FROM execution_owner_lifecycle_bindings + WHERE owner_kind = 'task'`, + ) + .all(), + ).toEqual([{ owner_id: active.taskId }]); }, ); }); diff --git a/test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts b/test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts new file mode 100644 index 000000000000..e3417c64366d --- /dev/null +++ b/test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts @@ -0,0 +1,555 @@ +// QA Lab producer proves cron/task owner receipts through a real mock Gateway. +import { createHash, randomUUID } from "node:crypto"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { setTimeout as delay } from "node:timers/promises"; +import { pathToFileURL } from "node:url"; +import { + QA_EVIDENCE_FILENAME, + type QaEvidenceSummaryJson, +} from "../../../../extensions/qa-lab/src/evidence-summary.js"; +import { startQaGatewayChild } from "../../../../extensions/qa-lab/src/gateway-child.js"; +import { startQaMockOpenAiServer } from "../../../../extensions/qa-lab/src/providers/mock-openai/server.js"; +import type { AuditRunInspectResult } from "../../../../packages/gateway-protocol/src/index.js"; +import { formatErrorMessage } from "../../../../src/infra/errors.js"; +import { createQaScriptEvidenceWriter, type QaScriptEvidenceStatus } from "./script-evidence.js"; + +const SCENARIO_ID = "autonomous-task-lifecycle-receipts"; +const SNAPSHOT_FILE = `${SCENARIO_ID}-summary.json`; +const HOOK_TOKEN = "qa-autonomous-hook-token"; + +type ProducerOptions = { artifactBase: string; repoRoot: string }; +type ProofResult = { + artifacts?: Array<{ filePath: string; kind: string }>; + details?: string; + durationMs: number; + status: QaScriptEvidenceStatus; +}; +type ExactOwnerRow = { + context_id: string; + execution_id: string; + run_id: string; + status: string; +}; +type OwnerDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle"; + +function hasSqliteColumns(db: DatabaseSync, table: string, columns: readonly string[]): boolean { + const exists = db + .prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?") + .get(table); + if (!exists) { + return false; + } + const present = new Set( + (db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>).map( + (row) => row.name, + ), + ); + return columns.every((column) => present.has(column)); +} + +function parseOptions(argv: readonly string[]): ProducerOptions { + const readValue = (name: string) => { + const index = argv.indexOf(name); + return index >= 0 ? argv[index + 1] : undefined; + }; + const artifactBase = readValue("--artifact-base"); + if (!artifactBase) { + throw new Error("--artifact-base is required"); + } + return { + artifactBase: path.resolve(artifactBase), + repoRoot: path.resolve(readValue("--repo-root") ?? process.cwd()), + }; +} + +function parseJson(raw: string, label: string): T { + try { + return JSON.parse(raw) as T; + } catch (error) { + throw new Error(`${label} was not JSON: ${formatErrorMessage(error)}`); + } +} + +function sha256(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +function stateDatabasePath(gateway: Awaited>): string { + const stateDir = gateway.runtimeEnv.OPENCLAW_STATE_DIR; + if (!stateDir) { + throw new Error("QA Gateway did not expose its isolated state directory"); + } + return path.join(stateDir, "state", "openclaw.sqlite"); +} + +function countExecutionContexts(gateway: Awaited>): number { + const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true }); + try { + if (!hasSqliteColumns(db, "execution_identity_contexts", ["context_id"])) { + return 0; + } + const row = db.prepare("SELECT COUNT(*) AS count FROM execution_identity_contexts").get() as { + count: number; + }; + return row.count; + } finally { + db.close(); + } +} + +function readCronOwnerRows( + gateway: Awaited>, + jobId: string, +): { cron: ExactOwnerRow; task: ExactOwnerRow } | undefined { + const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true }); + try { + if ( + !hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) || + !hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [ + "owner_kind", + "owner_id", + "context_id", + "execution_id", + ]) || + !hasSqliteColumns(db, "task_runs", ["task_id"]) + ) { + return undefined; + } + const cron = db + .prepare( + `SELECT binding.context_id, binding.execution_id, context.run_id, receipt.status + FROM cron_run_receipts AS receipt + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id + JOIN execution_identity_contexts AS context + ON context.context_id = binding.context_id + AND context.execution_id = binding.execution_id + WHERE receipt.job_id = ? AND receipt.status != 'running' + ORDER BY receipt.started_at_ms DESC LIMIT 1`, + ) + .get(jobId) as ExactOwnerRow | undefined; + const task = db + .prepare( + `SELECT binding.context_id, binding.execution_id, context.run_id, task.status + FROM task_runs AS task + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id + JOIN execution_identity_contexts AS context + ON context.context_id = binding.context_id + AND context.execution_id = binding.execution_id + WHERE task.runtime = 'cron' AND task.source_id = ? AND task.ended_at IS NOT NULL + ORDER BY task.created_at DESC LIMIT 1`, + ) + .get(jobId) as ExactOwnerRow | undefined; + return cron && task ? { cron, task } : undefined; + } finally { + db.close(); + } +} + +function readCronOwnerBindingDiagnostic( + gateway: Awaited>, + jobId: string, +): unknown { + const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true }); + try { + return { + cron: db + .prepare( + `SELECT binding.context_id, binding.execution_id, receipt.status + FROM cron_run_receipts AS receipt + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id + WHERE receipt.job_id = ? + ORDER BY receipt.started_at_ms DESC LIMIT 1`, + ) + .get(jobId), + task: db + .prepare( + `SELECT binding.context_id, binding.execution_id, task.status + FROM task_runs AS task + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id + WHERE task.runtime = 'cron' AND task.source_id = ? + ORDER BY task.created_at DESC LIMIT 1`, + ) + .get(jobId), + }; + } finally { + db.close(); + } +} + +function readCliOwnerRows( + gateway: Awaited>, + runId: string, +): { task: ExactOwnerRow } | undefined { + const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true }); + try { + if ( + !hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) || + !hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [ + "owner_kind", + "owner_id", + "context_id", + "execution_id", + ]) || + !hasSqliteColumns(db, "task_runs", ["task_id"]) + ) { + return undefined; + } + const task = db + .prepare( + `SELECT binding.context_id, binding.execution_id, context.run_id, task.status + FROM task_runs AS task + JOIN execution_owner_lifecycle_bindings AS binding + ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id + JOIN execution_identity_contexts AS context + ON context.context_id = binding.context_id + AND context.execution_id = binding.execution_id + WHERE task.runtime = 'cli' AND task.run_id = ? AND task.ended_at IS NOT NULL + LIMIT 1`, + ) + .get(runId) as ExactOwnerRow | undefined; + return task ? { task } : undefined; + } finally { + db.close(); + } +} + +async function waitFor(label: string, read: () => T | undefined): Promise { + const deadline = Date.now() + 30_000; + while (Date.now() < deadline) { + const value = read(); + if (value !== undefined) { + return value; + } + await delay(50); + } + throw new Error(`timed out waiting for ${label}`); +} + +function requireOwnerDisplay(result: AuditRunInspectResult, producer: OwnerDisplayProducer) { + const receipt = result.decisionDisplays.find( + (candidate) => + candidate.provenance.state === "verified" && candidate.provenance.producer === producer, + ); + if ( + !receipt || + receipt.enforcement.coverageState !== "attribution-only" || + receipt.decision.outcome !== "not-applicable" + ) { + throw new Error(`inspection omitted exact attribution-only ${producer} display`); + } + return receipt; +} + +async function inspectExecution(params: { + gateway: Awaited>; + executionId: string; + producers: OwnerDisplayProducer[]; + privateSentinels: string[]; +}) { + const jsonRaw = await params.gateway.runCli([ + "audit", + "--execution", + params.executionId, + "--explain", + "--json", + ]); + const json = parseJson(jsonRaw, "owner lifecycle inspection"); + for (const producer of params.producers) { + requireOwnerDisplay(json, producer); + } + for (const sentinel of params.privateSentinels) { + if (jsonRaw.includes(sentinel)) { + throw new Error(`owner receipt leaked private sentinel ${sentinel}`); + } + } + const human = await params.gateway.runCli([ + "audit", + "--execution", + params.executionId, + "--explain", + ]); + for (const producer of params.producers) { + if (!human.includes(`Display producer: ${producer}`)) { + throw new Error(`human inspection omitted ${producer}`); + } + } + return { json, jsonRaw, human }; +} + +async function runProof(options: ProducerOptions): Promise { + const mock = await startQaMockOpenAiServer(); + let gateway: Awaited> | undefined; + try { + gateway = await startQaGatewayChild({ + repoRoot: options.repoRoot, + useRepoCli: true, + providerBaseUrl: `${mock.baseUrl}/v1`, + providerMode: "mock-openai", + transportBaseUrl: "http://127.0.0.1", + controlUiEnabled: false, + mutateConfig: (config) => ({ + ...config, + logging: { + ...config.logging, + audit: { ...config.logging?.audit, enabled: true, executionIdentity: true }, + }, + hooks: { + enabled: true, + token: HOOK_TOKEN, + mappings: [ + { + id: "qa-suppressed-source", + match: { path: "suppressed" }, + action: "agent", + messageTemplate: "PRIVATE-SUPPRESSED-{{payload.value}}", + transform: { module: "suppress.mjs" }, + }, + ], + }, + }), + }); + const transformDir = path.join(path.dirname(gateway.configPath), "hooks", "transforms"); + await fs.mkdir(transformDir, { recursive: true }); + await fs.writeFile( + path.join(transformDir, "suppress.mjs"), + "export default function suppress() { return null; }\n", + "utf8", + ); + + const contextsBeforeSuppression = countExecutionContexts(gateway); + const suppression = await fetch(`${gateway.baseUrl}/hooks/suppressed`, { + method: "POST", + headers: { + Authorization: `Bearer ${HOOK_TOKEN}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ value: randomUUID() }), + }); + if (suppression.status !== 204) { + throw new Error(`mapped suppression returned HTTP ${suppression.status}`); + } + await delay(100); + if (countExecutionContexts(gateway) !== contextsBeforeSuppression) { + throw new Error("pre-admission mapping suppression allocated execution identity"); + } + + const cronSentinel = `PRIVATE-CRON-${randomUUID()}`; + const cronJob = (await gateway.call("cron.add", { + name: "QA autonomous receipt", + enabled: true, + schedule: { kind: "at", at: new Date(Date.now() + 3_600_000).toISOString() }, + sessionTarget: "isolated", + wakeMode: "next-heartbeat", + payload: { kind: "agentTurn", message: `${cronSentinel}: reply CRON-DONE` }, + delivery: { mode: "none" }, + })) as { id: string }; + await gateway.call("cron.run", { id: cronJob.id, mode: "force" }); + let cronRows: { cron: ExactOwnerRow; task: ExactOwnerRow }; + try { + cronRows = await waitFor("terminal cron/task exact bindings", () => + readCronOwnerRows(gateway!, cronJob.id), + ); + } catch (error) { + throw new Error( + `${formatErrorMessage(error)}; owner rows=${JSON.stringify(readCronOwnerBindingDiagnostic(gateway, cronJob.id))}; gateway logs=${gateway.logs()}`, + { cause: error }, + ); + } + if ( + cronRows.cron.context_id !== cronRows.task.context_id || + cronRows.cron.execution_id !== cronRows.task.execution_id + ) { + throw new Error("cron receipt/task rows did not retain one exact admitted execution"); + } + const cronInspection = await inspectExecution({ + gateway, + executionId: cronRows.cron.execution_id, + producers: ["cron-lifecycle", "task-lifecycle"], + privateSentinels: [cronSentinel], + }); + const cronCursorPage = parseJson( + await gateway.runCli([ + "audit", + "--execution", + cronRows.cron.execution_id, + "--explain", + "--cursor", + "c:0:0", + "--limit", + "1", + "--json", + ]), + "cron cursor page", + ); + requireOwnerDisplay(cronCursorPage, "cron-lifecycle"); + + const taskSentinel = `PRIVATE-TASK-${randomUUID()}`; + const cliRunId = `qa-cli-${randomUUID()}`; + const accepted = (await gateway.call( + "agent", + { + sessionKey: "agent:qa:main", + message: `${taskSentinel}: reply TASK-DONE`, + deliver: false, + idempotencyKey: cliRunId, + }, + { expectFinal: false }, + )) as { runId: string; status: string }; + await gateway.call( + "agent.wait", + { runId: accepted.runId, timeoutMs: 30_000 }, + { timeoutMs: 35_000 }, + ); + const cliRows = await waitFor("terminal CLI task exact binding", () => + readCliOwnerRows(gateway!, accepted.runId), + ); + const taskInspection = await inspectExecution({ + gateway, + executionId: cliRows.task.execution_id, + producers: ["task-lifecycle"], + privateSentinels: [taskSentinel], + }); + + const beforeRestart = JSON.stringify({ + cron: cronInspection.json, + task: taskInspection.json, + }); + await gateway.restartAfterStateMutation(async () => {}); + const cronAfter = await inspectExecution({ + gateway, + executionId: cronRows.cron.execution_id, + producers: ["cron-lifecycle", "task-lifecycle"], + privateSentinels: [cronSentinel], + }); + const taskAfter = await inspectExecution({ + gateway, + executionId: cliRows.task.execution_id, + producers: ["task-lifecycle"], + privateSentinels: [taskSentinel], + }); + const afterRestart = JSON.stringify({ cron: cronAfter.json, task: taskAfter.json }); + if (afterRestart !== beforeRestart) { + throw new Error("owner lifecycle JSON changed across Gateway replacement"); + } + + const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true }); + let genericCount = 0; + try { + if (hasSqliteColumns(db, "execution_decision_facts", ["receipt_id"])) { + genericCount = ( + db.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get() as { + count: number; + } + ).count; + } + } finally { + db.close(); + } + if (genericCount !== 0) { + throw new Error("owner lifecycle rows were duplicated into execution_decision_facts"); + } + + const snapshotPath = path.join(options.artifactBase, SNAPSHOT_FILE); + await fs.mkdir(options.artifactBase, { recursive: true }); + await fs.writeFile( + snapshotPath, + `${JSON.stringify( + { + suppression: { httpStatus: 204, identityAllocation: 0 }, + cron: { + contextId: cronRows.cron.context_id, + executionId: cronRows.cron.execution_id, + statuses: [cronRows.cron.status, cronRows.task.status], + displayProducers: ["cron-lifecycle", "task-lifecycle"], + }, + task: { + contextId: cliRows.task.context_id, + executionId: cliRows.task.execution_id, + statuses: [cliRows.task.status], + displayProducers: ["task-lifecycle"], + }, + cursorCompatibility: { cronPrefixAccepted: true }, + genericDuplicateAbsent: true, + byteEquivalentAfterRestart: true, + privacy: { cronPromptAbsent: true, taskPromptAbsent: true }, + resultSha256: sha256(afterRestart), + }, + null, + 2, + )}\n`, + "utf8", + ); + return `cron=${cronRows.cron.execution_id}; task=${cliRows.task.execution_id}; suppression=204; restart sha256=${sha256(afterRestart)}`; + } finally { + await gateway?.stop().catch(() => undefined); + await mock.stop(); + } +} + +async function produceProof(options: ProducerOptions): Promise { + const startedAt = Date.now(); + try { + return { + artifacts: [{ filePath: SNAPSHOT_FILE, kind: "summary" }], + details: await runProof(options), + durationMs: Math.max(1, Date.now() - startedAt), + status: "pass", + }; + } catch (error) { + return { + details: formatErrorMessage(error), + durationMs: Math.max(1, Date.now() - startedAt), + status: "fail", + }; + } +} + +async function runProducer(options: ProducerOptions): Promise { + const writer = createQaScriptEvidenceWriter({ + artifactBase: options.artifactBase, + logFileName: `${SCENARIO_ID}.log`, + primaryModel: "mock-openai/gpt-5.6-luna", + providerMode: "mock-openai", + repoRoot: options.repoRoot, + target: { + id: SCENARIO_ID, + title: "Autonomous task lifecycle receipts", + sourcePath: `qa/scenarios/runtime/${SCENARIO_ID}.yaml`, + docsRefs: ["docs/gateway/audit.md", "docs/automation/tasks.md"], + codeRefs: [ + "src/audit/execution-decision-receipts.ts", + "src/cron/store/run-receipt-store.ts", + "src/tasks/task-registry.store.sqlite.ts", + "src/tasks/task-flow-registry.store.sqlite.ts", + ], + }, + }); + const result = await produceProof(options); + writer.appendLog(`${result.status}: ${result.details ?? "no details"}\n`); + return await writer.write(result); +} + +async function main(argv: readonly string[]) { + const evidence = await runProducer(parseOptions(argv)); + const status = evidence.entries[0]?.result.status; + console.log(`Autonomous task lifecycle evidence: ${QA_EVIDENCE_FILENAME}`); + console.log(`Autonomous task lifecycle status: ${status}`); + return status === "pass" ? 0 : 1; +} + +if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) { + main(process.argv.slice(2)) + .then((exitCode) => { + process.exitCode = exitCode; + }) + .catch((error) => { + console.error(formatErrorMessage(error)); + process.exitCode = 1; + }); +}