mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-25 03:45:46 -06:00
Audit exact-bound owner-native lifecycle receipts (#126082)
* feat(audit): add owner-native lifecycle receipts * fix(audit): store lifecycle bindings additively * test(cron): isolate execution binding fixtures * fix(audit): reject stale lifecycle owners * fix(acp): gate parent flow binding on retained task * fix(audit): reject pruned owner cursors
This commit is contained in:
@@ -559,6 +559,14 @@ Model override note:
|
||||
|
||||
Gateway can expose HTTP webhook endpoints for external triggers. Enable in config:
|
||||
|
||||
A configured mapping `id` is retained only as bounded ingress-source
|
||||
attribution when that mapping reaches agent admission. It is not an
|
||||
authenticated service principal or invoker. Direct `/hooks/agent` and requests
|
||||
authenticated only by the shared hook token stay unattributed unless another
|
||||
authoritative principal source exists. If a transform returns `null`, the
|
||||
request keeps its visible HTTP 204 outcome and stops before creating a run,
|
||||
task, execution identity, or audit receipt.
|
||||
|
||||
```json5
|
||||
{
|
||||
hooks: {
|
||||
|
||||
@@ -405,8 +405,16 @@ A sweeper runs every **60 seconds** (first pass about 5 seconds after gateway st
|
||||
A task may reference a `childSessionKey` (where work runs) and a `requesterSessionKey` (who started it). Its `agentId` identifies the agent executing the work, while the requester and owner fields preserve launch and control context. Sessions are conversation context; tasks are activity tracking on top of that.
|
||||
</Accordion>
|
||||
<Accordion title="Tasks and agent runs">
|
||||
A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually.
|
||||
</Accordion>
|
||||
A task's `runId` links to the agent run doing the work. Agent lifecycle events (start, end, error) automatically update the task status - you do not need to manage the lifecycle manually.
|
||||
|
||||
When execution identity collection is enabled, OpenClaw also binds the exact
|
||||
admitted `contextId` and `executionId` to Gateway CLI, ACP, and automation task
|
||||
rows and their mirrored flow rows. This is inspection provenance only: `runId`
|
||||
remains correlation, task/flow status remains authoritative, and a missing or
|
||||
mismatched binding never changes execution or settlement. `openclaw audit
|
||||
--execution <id> --explain` adapts the existing rows without copying task or
|
||||
flow content into the generic decision-fact table.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Related
|
||||
|
||||
+13
-2
@@ -115,7 +115,8 @@ view renders these sections:
|
||||
3. **Lineage**: parent context or an explicit absent, unknown, or unsupported
|
||||
state.
|
||||
4. **Decisions**: bounded run-admission and authoritative action-decision
|
||||
receipts, including terminal operator approvals.
|
||||
receipts, including terminal operator approvals and exact-bound cron, task,
|
||||
and task-flow lifecycle rows.
|
||||
5. **Missing evidence** and **Next steps**.
|
||||
|
||||
Every field includes `present`, `absent`, `unknown`, or `unsupported`; the CLI
|
||||
@@ -125,6 +126,15 @@ ingress, an absent invoker, and
|
||||
`unattributed` coverage. Its admission receipt says `not-applicable` because no
|
||||
identity-aware policy or grant evaluation was proven.
|
||||
|
||||
Lifecycle rows from `cron_run_receipts`, `task_runs`, and `flow_runs` appear as
|
||||
owner-native, attribution-only receipts when their keyed lifecycle metadata
|
||||
carries the exact inspected context and execution ids. They contain status and
|
||||
bounded record references, not prompts, task goals, hook payloads, paths, or raw
|
||||
errors. Their decision is `not-applicable` because lifecycle attribution does
|
||||
not prove authorization.
|
||||
Treat every decision cursor as opaque: numeric and `a:`, `m:`, and `g:` values
|
||||
remain compatible, while cron/task/flow pages may return `c:`, `t:`, or `f:`.
|
||||
|
||||
For Gateway runs, a resolved authenticated profile can make the invoker
|
||||
`present` and coverage `attribution-only`. Paired devices and shared credentials
|
||||
do not establish a person: without a durable profile the invoker stays absent,
|
||||
@@ -340,7 +350,8 @@ and no identity context; its required `decisionDisplays` array is empty until
|
||||
the caller selects an execution id.
|
||||
For one selected context, receipt paging starts with admission, then reads
|
||||
owner-native terminal approvals, merges outbound progress and terminal records,
|
||||
and finally reads generic facts for boundaries without a native durable record.
|
||||
then reads generic facts and the cron, task, and flow lifecycle owners. The
|
||||
complete order is admission, approval, message, generic, cron, task, then flow.
|
||||
The merge is deterministic across restart and rejects a cursor whose exact
|
||||
owner row has expired. Approval and message selectors use the opaque
|
||||
`approval-decision:` and `message-decision:` namespaces minted from the same
|
||||
|
||||
@@ -43,6 +43,16 @@ platform-send start use a lazy progress companion, while terminal message rows
|
||||
remain in the activity ledger. Run inspection merges both sources directly;
|
||||
neither is copied into the generic decision-fact table.
|
||||
|
||||
Scheduled runs, background tasks, and task flows are owner-native sources too.
|
||||
After exact run admission, a lazy lifecycle metadata table binds the admitted
|
||||
context and execution ids to the canonical `cron_run_receipts`, `task_runs`, or
|
||||
`flow_runs` row. Inspection joins that metadata to the owner row directly and
|
||||
preserves its status, including skipped, failed, timed-out, cancelled, blocked,
|
||||
and lost outcomes. A `runId` alone never joins one of these rows to an
|
||||
execution. Legacy, missing, deleted, corrupt, or mismatched bindings remain
|
||||
unknown or absent; they never change task behavior and are never copied into
|
||||
`execution_decision_facts`.
|
||||
|
||||
## Run identity inspection
|
||||
|
||||
Execution identity recording is off by default, including on fresh installs
|
||||
@@ -160,6 +170,15 @@ this boundary. A run becomes
|
||||
`attribution-only` only when an authoritative ingress supplies an invoker fact.
|
||||
Neither state means that identity affected an allow or deny decision.
|
||||
|
||||
Configured webhook mapping ids identify only the matched ingress source. They
|
||||
do not authenticate a person, service, or invoker. Shared hook authentication
|
||||
and direct `/hooks/agent` requests therefore remain unattributed unless another
|
||||
authoritative principal producer exists. A mapping transform that suppresses a
|
||||
request before admission returns its normal HTTP response but creates no run,
|
||||
execution identity, task, or decision receipt. Restart recovery records system
|
||||
attribution only after the current durable recovery owner admits the exact
|
||||
attempt.
|
||||
|
||||
Authenticated Gateway attach records immutable audit facts once. Session
|
||||
creation separately reads the live canonical durable profile id so a profile
|
||||
link performed after attach cannot orphan session ownership. Ordinary session
|
||||
@@ -200,6 +219,12 @@ message-policy, or turn-capability denial that changed the result is
|
||||
`enforced`. Portable actions and early suppressions without a durable owner
|
||||
record use the generic fact owner on the same audit-writer FIFO.
|
||||
|
||||
Cron, task, and flow lifecycle receipts are `attribution-only` and have a
|
||||
`not-applicable` decision outcome. They report what the authoritative lifecycle
|
||||
owner retained; they do not claim an authorization decision. Their cursors are
|
||||
opaque and source-specific. Existing numeric cursors and `a:`, `m:`, and `g:`
|
||||
cursors remain accepted; newer owner stages use `c:`, `t:`, and `f:`.
|
||||
|
||||
When the same `runId` has a retained terminal row in `operator_approvals`, the
|
||||
inspector also reads its owner-local `operator_approval_execution_identities`
|
||||
binding. Only an exact context, execution, and run tuple projects the approval
|
||||
|
||||
@@ -116,6 +116,14 @@ describe("audit run inspection protocol", () => {
|
||||
provenance: { state: "verified", producer: "receipt-owner" },
|
||||
}),
|
||||
).toBe(false);
|
||||
for (const producer of ["cron-lifecycle", "task-lifecycle", "flow-lifecycle"]) {
|
||||
expect(
|
||||
validate.Check({
|
||||
...display,
|
||||
provenance: { state: "verified", producer },
|
||||
}),
|
||||
).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("exports selector and discovery-pagination invariants for generated clients", () => {
|
||||
|
||||
@@ -209,6 +209,9 @@ const DecisionReceiptDisplayProvenanceV1Schema = Type.Union([
|
||||
Type.Literal("run-admission"),
|
||||
Type.Literal("operator-approval"),
|
||||
Type.Literal("message-delivery"),
|
||||
Type.Literal("cron-lifecycle"),
|
||||
Type.Literal("task-lifecycle"),
|
||||
Type.Literal("flow-lifecycle"),
|
||||
]),
|
||||
}),
|
||||
closedObject({ state: Type.Literal("unverified") }),
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
title: Autonomous task lifecycle receipts
|
||||
|
||||
scenario:
|
||||
id: autonomous-task-lifecycle-receipts
|
||||
surface: gateway
|
||||
coverage:
|
||||
primary:
|
||||
- gateway.identity-and-presence-apis
|
||||
- automation.openclaw-tasks-flow
|
||||
objective: Verify exact-bound cron, task, and flow owner lifecycle rows are explained without generic decision-fact copies.
|
||||
successCriteria:
|
||||
- A mapped hook transform suppression returns HTTP 204 before admission and creates no execution identity.
|
||||
- A real forced scheduled agent run binds one exact context/execution to its cron receipt and task row.
|
||||
- A real Gateway agent action binds one exact context/execution to its CLI task row; focused owner tests cover bound flow rows.
|
||||
- JSON and human audit inspection expose only bounded attribution-only lifecycle displays with closed producers, accept the cron cursor prefix, and omit task/prompt content.
|
||||
- A replacement Gateway returns byte-equivalent JSON and the generic decision-fact table remains empty.
|
||||
docsRefs:
|
||||
- docs/gateway/audit.md
|
||||
- docs/cli/audit.md
|
||||
- docs/automation/cron-jobs.md
|
||||
- docs/automation/tasks.md
|
||||
- docs/concepts/qa-e2e-automation.md
|
||||
codeRefs:
|
||||
- src/audit/execution-decision-receipts.ts
|
||||
- src/cron/store/run-receipt-store.ts
|
||||
- src/tasks/task-registry.store.sqlite.ts
|
||||
- src/tasks/task-flow-registry.store.sqlite.ts
|
||||
- src/gateway/hooks-mapping.ts
|
||||
- test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts
|
||||
execution:
|
||||
kind: script
|
||||
path: test/e2e/qa-lab/runtime/autonomous-task-lifecycle-receipts.ts
|
||||
summary: Starts an ephemeral Gateway and mock provider, proves pre-admission suppression, runs cron and CLI task lifecycles, replaces the Gateway, and compares inspection output.
|
||||
args:
|
||||
- --artifact-base
|
||||
- ${outputDir}
|
||||
@@ -1,9 +1,25 @@
|
||||
/** Regression coverage for ACP background-task summary truncation boundaries. */
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
|
||||
import type { OpenClawConfig } from "../../config/types.openclaw.js";
|
||||
import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js";
|
||||
import {
|
||||
closeOpenClawStateDatabaseForTest,
|
||||
openOpenClawStateDatabase,
|
||||
} from "../../state/openclaw-state-db.js";
|
||||
import { findTaskByRunId } from "../../tasks/task-registry.js";
|
||||
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
|
||||
import {
|
||||
resetTaskRegistryForTests,
|
||||
resetTaskFlowRegistryForTests,
|
||||
} from "../../tasks/task-runtime.test-helpers.js";
|
||||
import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js";
|
||||
import { AcpRuntimeError } from "../runtime/errors.js";
|
||||
import {
|
||||
appendBackgroundTaskProgressSummary,
|
||||
bindBackgroundTaskExecution,
|
||||
createBackgroundTaskRecord,
|
||||
resolveBackgroundTaskContext,
|
||||
resolveBackgroundTaskFailureStatus,
|
||||
} from "./manager.background-task.js";
|
||||
@@ -13,6 +29,12 @@ import type { AcpSessionManagerDeps } from "./manager.types.js";
|
||||
// U+1F99E (🦞) is a surrogate pair in UTF-16; a raw .slice() boundary can split it.
|
||||
const LOBSTER = "🦞";
|
||||
|
||||
afterEach(() => {
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
resetTaskRegistryForTests({ persist: false });
|
||||
resetTaskFlowRegistryForTests({ persist: false });
|
||||
});
|
||||
|
||||
const HIGH_SURROGATE_WITHOUT_LOW = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])/;
|
||||
|
||||
function fakeDeps(): AcpSessionManagerDeps {
|
||||
@@ -85,3 +107,128 @@ describe("resolveBackgroundTaskFailureStatus", () => {
|
||||
).toBe("failed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ACP background task execution binding", () => {
|
||||
it("binds the exact admitted execution only at prompt submission", async () => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "state-only", prefix: "openclaw-acp-execution-binding-" },
|
||||
async () => {
|
||||
resetTaskRegistryForTests();
|
||||
resetTaskFlowRegistryForTests();
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", {
|
||||
contextId: "context-acp",
|
||||
executionId: "execution-acp",
|
||||
}),
|
||||
};
|
||||
const record = createBackgroundTaskRecord(
|
||||
{
|
||||
requesterSessionKey: "agent:main:main",
|
||||
childSessionKey: "agent:qa:child",
|
||||
runId: "run-acp",
|
||||
task: "private",
|
||||
},
|
||||
100,
|
||||
);
|
||||
const task = findTaskByRunId("run-acp");
|
||||
if (!record || !task?.parentFlowId) {
|
||||
throw new Error("expected ACP task and owner flow");
|
||||
}
|
||||
const db = openOpenClawStateDatabase().db;
|
||||
expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false);
|
||||
|
||||
bindBackgroundTaskExecution(record, admitted);
|
||||
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
`SELECT owner_kind, owner_id, context_id, execution_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
ORDER BY owner_kind`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([
|
||||
{
|
||||
owner_kind: "flow",
|
||||
owner_id: task.parentFlowId,
|
||||
context_id: "context-acp",
|
||||
execution_id: "execution-acp",
|
||||
},
|
||||
{
|
||||
owner_kind: "task",
|
||||
owner_id: task.taskId,
|
||||
context_id: "context-acp",
|
||||
execution_id: "execution-acp",
|
||||
},
|
||||
]);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["missing", "mismatched"] as const)(
|
||||
"does not bind a live parent flow when the task owner is %s",
|
||||
async (taskOwnerState) => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "state-only", prefix: `openclaw-acp-${taskOwnerState}-task-binding-` },
|
||||
async () => {
|
||||
resetTaskRegistryForTests();
|
||||
resetTaskFlowRegistryForTests();
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-acp", runId: "run-acp" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-acp", {
|
||||
contextId: "context-acp",
|
||||
executionId: "execution-acp",
|
||||
}),
|
||||
};
|
||||
const record = createBackgroundTaskRecord(
|
||||
{
|
||||
requesterSessionKey: "agent:main:main",
|
||||
childSessionKey: "agent:qa:child",
|
||||
runId: "run-acp",
|
||||
task: "private",
|
||||
},
|
||||
100,
|
||||
);
|
||||
const task = findTaskByRunId("run-acp");
|
||||
if (!record || !task?.parentFlowId) {
|
||||
throw new Error("expected ACP task and owner flow");
|
||||
}
|
||||
const db = openOpenClawStateDatabase().db;
|
||||
if (taskOwnerState === "missing") {
|
||||
db.prepare("DELETE FROM task_runs WHERE task_id = ?").run(task.taskId);
|
||||
} else {
|
||||
expect(
|
||||
bindTaskRunExecution({
|
||||
taskId: task.taskId,
|
||||
admitted: {
|
||||
operationalRunInstance: { instanceId: "instance-other", runId: "run-other" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-other", {
|
||||
contextId: "context-other",
|
||||
executionId: "execution-other",
|
||||
}),
|
||||
},
|
||||
}),
|
||||
).toBe("bound");
|
||||
}
|
||||
|
||||
bindBackgroundTaskExecution(record, admitted);
|
||||
|
||||
if (taskOwnerState === "missing") {
|
||||
expect(tableExists(db, "execution_owner_lifecycle_bindings")).toBe(false);
|
||||
} else {
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
`SELECT context_id, execution_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'flow' AND owner_id = ?`,
|
||||
)
|
||||
.get(task.parentFlowId),
|
||||
).toBeUndefined();
|
||||
}
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
/** Mirrors child ACP turns into detached-task status for requester-facing progress. */
|
||||
import { truncateUtf16Safe } from "@openclaw/normalization-core/utf16-slice";
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import { isRetainedExecutionOwnerBinding } from "../../audit/execution-owner-binding.js";
|
||||
import type { OpenClawConfig } from "../../config/types.openclaw.js";
|
||||
import { logVerbose } from "../../globals.js";
|
||||
import {
|
||||
@@ -9,6 +11,8 @@ import {
|
||||
startTaskRunByRunId,
|
||||
} from "../../tasks/detached-task-runtime.js";
|
||||
import { resolveRequiredCompletionTerminalResult } from "../../tasks/task-completion-contract.js";
|
||||
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
|
||||
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
|
||||
import {
|
||||
deliveryContextFromSession,
|
||||
type DeliveryContext,
|
||||
@@ -31,6 +35,11 @@ type BackgroundTaskContext = {
|
||||
task: string;
|
||||
};
|
||||
|
||||
type BackgroundTaskRecord = {
|
||||
taskId: string;
|
||||
parentFlowId?: string;
|
||||
};
|
||||
|
||||
/** Produces the bounded task label shown for a child ACP background run. */
|
||||
function summarizeBackgroundTaskText(text: string): string {
|
||||
const normalized = normalizeText(text) ?? "ACP background task";
|
||||
@@ -133,7 +142,7 @@ export function resolveBackgroundTaskContext(params: {
|
||||
export function createBackgroundTaskRecord(
|
||||
context: BackgroundTaskContext,
|
||||
startedAt: number,
|
||||
): void {
|
||||
): BackgroundTaskRecord | undefined {
|
||||
try {
|
||||
const task = createRunningTaskRun({
|
||||
runtime: "acp",
|
||||
@@ -151,11 +160,37 @@ export function createBackgroundTaskRecord(
|
||||
logVerbose(
|
||||
`acp-manager: failed creating background task for ${context.runId}: persist_failed`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
taskId: task.taskId,
|
||||
...(task.parentFlowId ? { parentFlowId: task.parentFlowId } : {}),
|
||||
};
|
||||
} catch (error) {
|
||||
logVerbose(
|
||||
`acp-manager: failed creating background task for ${context.runId}: ${String(error)}`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** Links ACP owner rows only when the runtime reaches its prompt-submitted boundary. */
|
||||
export function bindBackgroundTaskExecution(
|
||||
record: BackgroundTaskRecord,
|
||||
admitted: AdmittedRunContext,
|
||||
): void {
|
||||
try {
|
||||
const taskResult = bindTaskRunExecution({ admitted, taskId: record.taskId });
|
||||
const flowResult = record.parentFlowId
|
||||
? isRetainedExecutionOwnerBinding(taskResult)
|
||||
? bindTaskFlowExecution({ admitted, flowId: record.parentFlowId })
|
||||
: taskResult
|
||||
: undefined;
|
||||
if ([taskResult, flowResult].some((result) => result === "mismatch" || result === "missing")) {
|
||||
logVerbose("acp-manager: exact task execution binding was not retained");
|
||||
}
|
||||
} catch (error) {
|
||||
logVerbose(`acp-manager: failed binding background task execution: ${String(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
} from "./manager.backend-failover.js";
|
||||
import {
|
||||
appendBackgroundTaskProgressSummary,
|
||||
bindBackgroundTaskExecution,
|
||||
createBackgroundTaskRecord,
|
||||
markBackgroundTaskRunning,
|
||||
markBackgroundTaskTerminal,
|
||||
@@ -87,9 +88,10 @@ export async function runManagerTurn(params: {
|
||||
text: input.text,
|
||||
})
|
||||
: null;
|
||||
if (taskContext) {
|
||||
createBackgroundTaskRecord(taskContext, turnStartedAt);
|
||||
}
|
||||
const taskRecord = taskContext
|
||||
? createBackgroundTaskRecord(taskContext, turnStartedAt)
|
||||
: undefined;
|
||||
let taskExecutionBound = false;
|
||||
let taskProgressSummary = "";
|
||||
const initialResolution = params.resolveSession({
|
||||
cfg: input.cfg,
|
||||
@@ -267,6 +269,10 @@ export async function runManagerTurn(params: {
|
||||
onBeforePrompt: input.onBeforePrompt,
|
||||
onPromptStarted: async ({ authoritative }) => {
|
||||
promptStarted = authoritative;
|
||||
if (authoritative && taskRecord && !taskExecutionBound) {
|
||||
taskExecutionBound = true;
|
||||
bindBackgroundTaskExecution(taskRecord, input.admittedRunContext);
|
||||
}
|
||||
try {
|
||||
await input.onLifecycle?.({
|
||||
type: "prompt_submitted",
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
attachAgentCommandRecoveryAdmissionFacts,
|
||||
getAgentCommandAdmissionFacts,
|
||||
} from "./agent-command-admission-facts.js";
|
||||
|
||||
describe("agent command admission facts", () => {
|
||||
it("records restart recovery as owner-bound system attribution", () => {
|
||||
const runContext = {};
|
||||
attachAgentCommandRecoveryAdmissionFacts(runContext);
|
||||
expect(getAgentCommandAdmissionFacts(runContext)).toEqual({
|
||||
ingress: {
|
||||
kind: "recovery",
|
||||
boundary: "gateway.main-session-recovery",
|
||||
state: "present",
|
||||
},
|
||||
invoker: {
|
||||
state: "present",
|
||||
kind: "system",
|
||||
rawPrincipalRef: "openclaw.main-session-recovery",
|
||||
},
|
||||
assurance: [
|
||||
{
|
||||
kind: "runtime-binding",
|
||||
rawEvidenceRef: "gateway.main-session-recovery-owner",
|
||||
strength: "boundary-verified",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(getAgentCommandAdmissionFacts({})).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -18,3 +18,26 @@ export function getAgentCommandAdmissionFacts(
|
||||
): AgentCommandAdmissionFacts | undefined {
|
||||
return factsByIngress.get(ingress);
|
||||
}
|
||||
|
||||
/** Records the exact system attribution only after the recovery owner admits the attempt. */
|
||||
export function attachAgentCommandRecoveryAdmissionFacts(ingress: object): void {
|
||||
attachAgentCommandAdmissionFacts(ingress, {
|
||||
ingress: {
|
||||
kind: "recovery",
|
||||
boundary: "gateway.main-session-recovery",
|
||||
state: "present",
|
||||
},
|
||||
invoker: {
|
||||
state: "present",
|
||||
kind: "system",
|
||||
rawPrincipalRef: "openclaw.main-session-recovery",
|
||||
},
|
||||
assurance: [
|
||||
{
|
||||
kind: "runtime-binding",
|
||||
rawEvidenceRef: "gateway.main-session-recovery-owner",
|
||||
strength: "boundary-verified",
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
|
||||
@@ -43,6 +43,36 @@ describe("sanitizePublicAgentCommandIngressOpts", () => {
|
||||
});
|
||||
|
||||
describe("Gateway agent command execution identity", () => {
|
||||
it("runs owner binding only after the awaited admission callback settles", async () => {
|
||||
const events: string[] = [];
|
||||
const prepared = prepareAgentCommandExecutionIdentity({
|
||||
opts: {
|
||||
message: "bind after admission",
|
||||
onAdmittedRunContext: async () => {
|
||||
await Promise.resolve();
|
||||
events.push("admitted");
|
||||
},
|
||||
onPostAdmittedRunContext: () => {
|
||||
events.push("owner-bound");
|
||||
},
|
||||
},
|
||||
prepared: {
|
||||
cfg: { logging: { audit: { enabled: true, executionIdentity: true } } },
|
||||
runId: "run-post-admission",
|
||||
sessionAgentId: "main",
|
||||
sessionId: "session-post-admission",
|
||||
},
|
||||
ingress: { kind: "api", boundary: "agent-command.from-ingress", state: "unknown" },
|
||||
lifecycleGeneration: "generation-1",
|
||||
});
|
||||
|
||||
const admitted = await prepared.admit("embedded");
|
||||
await prepared.admit("embedded");
|
||||
|
||||
expect(admitted.executionIdentityToken).toBeDefined();
|
||||
expect(events).toEqual(["admitted", "owner-bound"]);
|
||||
});
|
||||
|
||||
it("preserves trusted spawn facts across internal option preparation", () => {
|
||||
const facts = {
|
||||
ingress: {
|
||||
|
||||
@@ -3,6 +3,7 @@ import type {
|
||||
ExecutionIdentityAdmissionToken,
|
||||
} from "../audit/execution-identity-admission.js";
|
||||
import { executionIdentitySpawnAdmission } from "../audit/execution-identity-spawn-admission.js";
|
||||
import { withPostAdmissionExecutionOwnerBinding } from "../audit/execution-owner-binding.js";
|
||||
import type { OpenClawConfig } from "../config/types.openclaw.js";
|
||||
import { formatErrorMessage } from "../infra/errors.js";
|
||||
import { createSubsystemLogger } from "../logging/subsystem.js";
|
||||
@@ -170,9 +171,12 @@ export function prepareAgentCommandExecutionIdentity(params: {
|
||||
},
|
||||
};
|
||||
const spawnFacts = readAgentCommandExecutionIdentitySpawnFacts(opts);
|
||||
return spawnFacts
|
||||
const preparedAdmission = spawnFacts
|
||||
? prepareAgentCommandRunAdmissionWithSpawnFacts(admissionParams, spawnFacts)
|
||||
: executionIdentity.prepare(admissionParams);
|
||||
return opts.onPostAdmittedRunContext
|
||||
? withPostAdmissionExecutionOwnerBinding(preparedAdmission, opts.onPostAdmittedRunContext)
|
||||
: preparedAdmission;
|
||||
}
|
||||
|
||||
export function sanitizePublicAgentCommandIngressOpts(
|
||||
@@ -187,6 +191,7 @@ export function sanitizePublicAgentCommandIngressOpts(
|
||||
operationalRunInstance: undefined,
|
||||
cronCreatorAuthorityCapability: undefined,
|
||||
onAdmittedRunContext: undefined,
|
||||
onPostAdmittedRunContext: undefined,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -209,6 +209,10 @@ export type AgentCommandOpts = {
|
||||
onAdmittedRunContext?: (
|
||||
context: import("../admitted-run-context.js").AdmittedRunContext,
|
||||
) => void | Promise<void>;
|
||||
/** Private owner binding hook invoked only after exact admission has resolved. */
|
||||
onPostAdmittedRunContext?: (
|
||||
context: import("../admitted-run-context.js").AdmittedRunContext,
|
||||
) => void;
|
||||
/** Called when the actual run model is selected, including fallback retries. */
|
||||
onActiveModelSelected?: (ctx: { provider: string; model: string }) => void | Promise<void>;
|
||||
/** Called when every candidate in the run's model fallback chain failed. */
|
||||
@@ -241,6 +245,7 @@ export type AgentCommandIngressOpts = Omit<
|
||||
| "operationalRunInstance"
|
||||
| "cronCreatorAuthorityCapability"
|
||||
| "onAdmittedRunContext"
|
||||
| "onPostAdmittedRunContext"
|
||||
> & {
|
||||
/** Trusted sender identity bit for command/channel-action auth; defaults false for ingress. */
|
||||
senderIsOwner?: boolean;
|
||||
@@ -259,4 +264,5 @@ export type AgentCommandGatewayIngressOpts = AgentCommandIngressOpts &
|
||||
| "operationalRunInstance"
|
||||
| "cronCreatorAuthorityCapability"
|
||||
| "onAdmittedRunContext"
|
||||
| "onPostAdmittedRunContext"
|
||||
>;
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
type ExecutionIdentityAdmissionEnvelope,
|
||||
} from "./execution-identity-admission.js";
|
||||
import { processExecutionIdentityAdmissionWork } from "./execution-identity-context.js";
|
||||
import { bindExecutionOwnerLifecycleMetadata } from "./execution-owner-lifecycle-binding-store.js";
|
||||
import {
|
||||
configureMessageActionDecisionSink,
|
||||
recordMessageActionDecision,
|
||||
@@ -256,6 +257,79 @@ describe("execution decision facts", () => {
|
||||
expect(receiptSearch).not.toContain(encodeURIComponent(messageReceipt?.receiptId ?? ""));
|
||||
});
|
||||
|
||||
it("projects exact-bound cron, task, and flow owner rows without generic facts", () => {
|
||||
const database = databaseOptions();
|
||||
const context = seedExecutionContext(database);
|
||||
const db = openOpenClawStateDatabase(database).db;
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
"cron-receipt-1",
|
||||
"default",
|
||||
"job-1",
|
||||
"revision-1",
|
||||
"main",
|
||||
context.runId,
|
||||
"ok",
|
||||
1,
|
||||
60,
|
||||
70,
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO task_runs (
|
||||
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
|
||||
notify_policy, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
"task-1",
|
||||
"cron",
|
||||
"owner-1",
|
||||
"session",
|
||||
"private task text",
|
||||
"succeeded",
|
||||
"not-requested",
|
||||
"never",
|
||||
61,
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO flow_runs (
|
||||
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("flow-1", "owner-1", "succeeded", "never", "private flow goal", 62, 70);
|
||||
for (const [ownerKind, ownerId] of [
|
||||
["cron", "cron-receipt-1"],
|
||||
["task", "task-1"],
|
||||
["flow", "flow-1"],
|
||||
] as const) {
|
||||
expect(
|
||||
bindExecutionOwnerLifecycleMetadata({
|
||||
db,
|
||||
ownerKind,
|
||||
ownerId,
|
||||
binding: { contextId: context.contextId, executionId: context.executionId },
|
||||
}),
|
||||
).toBe("bound");
|
||||
}
|
||||
|
||||
const result = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionLimit: 10,
|
||||
options: { ...database, now: 100 },
|
||||
});
|
||||
expect(result.decisions.map((item) => item.source.owner)).toEqual([
|
||||
"agent-command",
|
||||
"cron_run_receipts",
|
||||
"task_runs",
|
||||
"flow_runs",
|
||||
]);
|
||||
expect(JSON.stringify(result.decisions)).not.toContain("private task text");
|
||||
expect(JSON.stringify(result.decisions)).not.toContain("private flow goal");
|
||||
expect(tableExists(db, "execution_decision_facts")).toBe(false);
|
||||
});
|
||||
|
||||
it("does not assign run-only delivery evidence to either exact execution sharing a run id", () => {
|
||||
const database = databaseOptions();
|
||||
const first = seedExecutionContext(database, {
|
||||
|
||||
@@ -15,6 +15,12 @@ import {
|
||||
pageExecutionDecisionFactsForContext,
|
||||
summarizeExecutionDecisionFactsForContext,
|
||||
} from "./execution-decision-facts.js";
|
||||
import {
|
||||
pageOwnerLifecycleReceipts,
|
||||
summarizeOwnerLifecycleReceipts,
|
||||
type OwnerLifecycleCursor,
|
||||
type OwnerLifecycleStage,
|
||||
} from "./execution-owner-lifecycle-receipts.js";
|
||||
import {
|
||||
pageMessageDeliveryReceiptsForRun,
|
||||
summarizeMessageDeliveryReceiptsForRun,
|
||||
@@ -34,9 +40,10 @@ type ProvenancedDecisionReceipt = {
|
||||
|
||||
const MAX_AGGREGATE_MISSING_EVIDENCE = 16;
|
||||
const MISSING_EVIDENCE_TRUNCATED = "decision.missing_evidence_truncated";
|
||||
type DecisionStage = "approval" | "message" | "generic" | OwnerLifecycleStage;
|
||||
type DecisionCursor =
|
||||
| {
|
||||
stage: "approval" | "message" | "generic";
|
||||
stage: DecisionStage;
|
||||
after?: { occurredAt: number; rowId: number };
|
||||
}
|
||||
| {
|
||||
@@ -58,7 +65,7 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi
|
||||
if (offset !== null && offset !== undefined) {
|
||||
return { offset };
|
||||
}
|
||||
const match = /^([amg]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value);
|
||||
const match = /^([amgctf]):(0|[1-9]\d*):(0|[1-9]\d*)$/.exec(value);
|
||||
if (!match) {
|
||||
return null;
|
||||
}
|
||||
@@ -67,8 +74,20 @@ function parseDecisionCursor(value: string | undefined): DecisionCursor | undefi
|
||||
if (!Number.isSafeInteger(occurredAt) || !Number.isSafeInteger(rowId)) {
|
||||
return null;
|
||||
}
|
||||
const stage =
|
||||
match[1] === "a"
|
||||
? "approval"
|
||||
: match[1] === "m"
|
||||
? "message"
|
||||
: match[1] === "g"
|
||||
? "generic"
|
||||
: match[1] === "c"
|
||||
? "cron"
|
||||
: match[1] === "t"
|
||||
? "task"
|
||||
: "flow";
|
||||
return {
|
||||
stage: match[1] === "a" ? "approval" : match[1] === "m" ? "message" : "generic",
|
||||
stage,
|
||||
...(occurredAt === 0 && rowId === 0 ? {} : { after: { occurredAt, rowId } }),
|
||||
};
|
||||
}
|
||||
@@ -78,10 +97,17 @@ export function isExecutionDecisionCursor(value: string): boolean {
|
||||
}
|
||||
|
||||
function formatDecisionCursor(
|
||||
stage: "approval" | "message" | "generic",
|
||||
stage: DecisionStage,
|
||||
cursor?: { occurredAt: number; rowId: number },
|
||||
): string {
|
||||
const prefix = stage === "approval" ? "a" : stage === "message" ? "m" : "g";
|
||||
const prefix = {
|
||||
approval: "a",
|
||||
message: "m",
|
||||
generic: "g",
|
||||
cron: "c",
|
||||
task: "t",
|
||||
flow: "f",
|
||||
}[stage];
|
||||
return `${prefix}:${cursor?.occurredAt ?? 0}:${cursor?.rowId ?? 0}`;
|
||||
}
|
||||
|
||||
@@ -200,10 +226,8 @@ export function presentExecutionDecisionReceipts(params: {
|
||||
if (cursor === null) {
|
||||
throw new ExecutionDecisionCursorError();
|
||||
}
|
||||
const limit = params.decisionLimit ?? 50;
|
||||
const decisionLimit = params.decisionLimit ?? 50;
|
||||
const now = params.options.now ?? Date.now();
|
||||
// Numeric cursors are the shipped aggregate offset. Resolve its owner span
|
||||
// once, then let the canonical bounded owner pagers emit opaque successors.
|
||||
const opaqueCursor = cursor && "stage" in cursor ? cursor : undefined;
|
||||
const legacyOffset = cursor && "offset" in cursor ? cursor.offset - 1 : undefined;
|
||||
const approvalSummary = summarizeOperatorApprovalReceiptsForRun({
|
||||
@@ -225,11 +249,132 @@ export function presentExecutionDecisionReceipts(params: {
|
||||
context: params.context,
|
||||
options: { ...params.options, now },
|
||||
});
|
||||
const cronSummary = summarizeOwnerLifecycleReceipts({
|
||||
stage: "cron",
|
||||
context: params.context,
|
||||
options: params.options,
|
||||
});
|
||||
const taskSummary = summarizeOwnerLifecycleReceipts({
|
||||
stage: "task",
|
||||
context: params.context,
|
||||
options: params.options,
|
||||
});
|
||||
const flowSummary = summarizeOwnerLifecycleReceipts({
|
||||
stage: "flow",
|
||||
context: params.context,
|
||||
options: params.options,
|
||||
});
|
||||
const stages: Array<{
|
||||
stage: DecisionStage;
|
||||
count: number;
|
||||
page: (params: { after?: OwnerLifecycleCursor; offset?: number; limit: number }) => {
|
||||
entries: ProvenancedDecisionReceipt[];
|
||||
nextCursor?: OwnerLifecycleCursor;
|
||||
};
|
||||
}> = [
|
||||
{
|
||||
stage: "approval",
|
||||
count: approvalSummary.count,
|
||||
page: ({ after, offset, limit }) => {
|
||||
const page = pageOperatorApprovalReceiptsForRun({
|
||||
context: {
|
||||
contextId: params.context.contextId,
|
||||
executionId: params.context.executionId,
|
||||
runId: params.context.runId,
|
||||
},
|
||||
after,
|
||||
offset,
|
||||
limit,
|
||||
nowMs: now,
|
||||
databaseOptions: params.options,
|
||||
});
|
||||
return {
|
||||
entries: page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "verified", producer: "operator-approval" },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
|
||||
};
|
||||
},
|
||||
},
|
||||
{
|
||||
stage: "message",
|
||||
count: messageSummary.count,
|
||||
page: ({ after, offset, limit }) => {
|
||||
const page = pageMessageDeliveryReceiptsForRun({
|
||||
context: params.context,
|
||||
after,
|
||||
offset,
|
||||
limit,
|
||||
options: { ...params.options, now },
|
||||
});
|
||||
return {
|
||||
entries: page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "verified", producer: "message-delivery" },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
|
||||
};
|
||||
},
|
||||
},
|
||||
{
|
||||
stage: "generic",
|
||||
count: genericSummary.count,
|
||||
page: ({ after, offset, limit }) => {
|
||||
const page = pageExecutionDecisionFactsForContext({
|
||||
context: params.context,
|
||||
after,
|
||||
offset,
|
||||
limit,
|
||||
now,
|
||||
database: params.options,
|
||||
});
|
||||
return {
|
||||
entries: page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "unverified" },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
|
||||
};
|
||||
},
|
||||
},
|
||||
...(["cron", "task", "flow"] as const).map((stage) => ({
|
||||
stage,
|
||||
count: { cron: cronSummary, task: taskSummary, flow: flowSummary }[stage].count,
|
||||
page: ({
|
||||
after,
|
||||
offset,
|
||||
limit,
|
||||
}: {
|
||||
after?: OwnerLifecycleCursor;
|
||||
offset?: number;
|
||||
limit: number;
|
||||
}) => {
|
||||
const page = pageOwnerLifecycleReceipts({
|
||||
stage,
|
||||
context: params.context,
|
||||
after,
|
||||
offset,
|
||||
limit,
|
||||
options: params.options,
|
||||
});
|
||||
return {
|
||||
entries: page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "verified" as const, producer: entry.displayProducer },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
...(page.nextCursor ? { nextCursor: page.nextCursor } : {}),
|
||||
};
|
||||
},
|
||||
})),
|
||||
];
|
||||
const decisions: ProvenancedDecisionReceipt[] = [];
|
||||
let remainingLimit = limit;
|
||||
let remainingLimit = decisionLimit;
|
||||
let nextDecisionCursor: string | undefined;
|
||||
const approvalOffset =
|
||||
legacyOffset !== undefined && legacyOffset < approvalSummary.count ? legacyOffset : undefined;
|
||||
|
||||
if (cursor === undefined && remainingLimit > 0) {
|
||||
decisions.push({
|
||||
@@ -238,32 +383,44 @@ export function presentExecutionDecisionReceipts(params: {
|
||||
selectorId: `${params.context.contextId}:admission`,
|
||||
});
|
||||
remainingLimit -= 1;
|
||||
if (
|
||||
remainingLimit === 0 &&
|
||||
(approvalSummary.count > 0 || messageSummary.count > 0 || genericSummary.count > 0)
|
||||
) {
|
||||
if (remainingLimit === 0 && stages.some((stage) => stage.count > 0)) {
|
||||
// The shipped first successor remains the approval cursor even when that owner is empty.
|
||||
nextDecisionCursor = formatDecisionCursor("approval");
|
||||
}
|
||||
}
|
||||
if (
|
||||
remainingLimit > 0 &&
|
||||
opaqueCursor?.stage !== "message" &&
|
||||
opaqueCursor?.stage !== "generic" &&
|
||||
(legacyOffset === undefined || approvalOffset !== undefined)
|
||||
) {
|
||||
let startStage = 0;
|
||||
let firstStageOffset: number | undefined;
|
||||
if (opaqueCursor) {
|
||||
startStage = stages.findIndex((stage) => stage.stage === opaqueCursor.stage);
|
||||
} else if (legacyOffset !== undefined) {
|
||||
let preceding = 0;
|
||||
startStage = stages.findIndex((stage) => {
|
||||
if (legacyOffset < preceding + stage.count) {
|
||||
firstStageOffset = legacyOffset - preceding;
|
||||
return true;
|
||||
}
|
||||
preceding += stage.count;
|
||||
return false;
|
||||
});
|
||||
if (startStage < 0) {
|
||||
startStage = stages.length;
|
||||
}
|
||||
}
|
||||
for (let index = startStage; index < stages.length && remainingLimit > 0; index += 1) {
|
||||
const stage = stages[index];
|
||||
if (!stage) {
|
||||
continue;
|
||||
}
|
||||
let page;
|
||||
try {
|
||||
page = pageOperatorApprovalReceiptsForRun({
|
||||
context: {
|
||||
contextId: params.context.contextId,
|
||||
executionId: params.context.executionId,
|
||||
runId: params.context.runId,
|
||||
},
|
||||
after: opaqueCursor?.stage === "approval" ? opaqueCursor.after : undefined,
|
||||
offset: approvalOffset,
|
||||
page = stage.page({
|
||||
...(index === startStage && opaqueCursor?.stage === stage.stage
|
||||
? { after: opaqueCursor.after }
|
||||
: {}),
|
||||
...(index === startStage && firstStageOffset !== undefined
|
||||
? { offset: firstStageOffset }
|
||||
: {}),
|
||||
limit: remainingLimit,
|
||||
nowMs: now,
|
||||
databaseOptions: params.options,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
|
||||
@@ -273,125 +430,52 @@ export function presentExecutionDecisionReceipts(params: {
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
decisions.push(
|
||||
...page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "verified" as const, producer: "operator-approval" as const },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
);
|
||||
decisions.push(...page.entries);
|
||||
remainingLimit -= page.entries.length;
|
||||
if (page.nextCursor) {
|
||||
nextDecisionCursor = formatDecisionCursor("approval", page.nextCursor);
|
||||
} else if (remainingLimit === 0 && messageSummary.count > 0) {
|
||||
nextDecisionCursor = formatDecisionCursor("message");
|
||||
} else if (remainingLimit === 0 && genericSummary.count > 0) {
|
||||
nextDecisionCursor = formatDecisionCursor("generic");
|
||||
nextDecisionCursor = formatDecisionCursor(stage.stage, page.nextCursor);
|
||||
break;
|
||||
}
|
||||
if (remainingLimit === 0) {
|
||||
const successor = stages.slice(index + 1).find((candidate) => candidate.count > 0);
|
||||
nextDecisionCursor = successor ? formatDecisionCursor(successor.stage) : undefined;
|
||||
}
|
||||
}
|
||||
const messageOffset =
|
||||
legacyOffset === undefined
|
||||
? undefined
|
||||
: legacyOffset >= approvalSummary.count &&
|
||||
legacyOffset < approvalSummary.count + messageSummary.count
|
||||
? legacyOffset - approvalSummary.count
|
||||
: undefined;
|
||||
if (
|
||||
remainingLimit > 0 &&
|
||||
nextDecisionCursor?.startsWith("a:") !== true &&
|
||||
opaqueCursor?.stage !== "generic" &&
|
||||
(legacyOffset === undefined || messageOffset !== undefined || approvalOffset !== undefined)
|
||||
) {
|
||||
let page;
|
||||
try {
|
||||
page = pageMessageDeliveryReceiptsForRun({
|
||||
context: params.context,
|
||||
after: opaqueCursor?.stage === "message" ? opaqueCursor.after : undefined,
|
||||
offset: messageOffset,
|
||||
limit: remainingLimit,
|
||||
options: { ...params.options, now },
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
|
||||
throw new ExecutionDecisionCursorError(
|
||||
"decision cursor is no longer retained; restart inspection without --cursor",
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
decisions.push(
|
||||
...page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "verified" as const, producer: "message-delivery" as const },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
);
|
||||
remainingLimit -= page.entries.length;
|
||||
if (page.nextCursor) {
|
||||
nextDecisionCursor = formatDecisionCursor("message", page.nextCursor);
|
||||
} else if (remainingLimit === 0 && genericSummary.count > 0) {
|
||||
nextDecisionCursor = formatDecisionCursor("generic");
|
||||
}
|
||||
}
|
||||
const genericOffset =
|
||||
legacyOffset === undefined
|
||||
? undefined
|
||||
: Math.max(0, legacyOffset - approvalSummary.count - messageSummary.count);
|
||||
if (
|
||||
remainingLimit > 0 &&
|
||||
nextDecisionCursor?.startsWith("a:") !== true &&
|
||||
nextDecisionCursor?.startsWith("m:") !== true
|
||||
) {
|
||||
let page;
|
||||
try {
|
||||
page = pageExecutionDecisionFactsForContext({
|
||||
context: params.context,
|
||||
after: opaqueCursor?.stage === "generic" ? opaqueCursor.after : undefined,
|
||||
offset: genericOffset,
|
||||
limit: remainingLimit,
|
||||
now,
|
||||
database: params.options,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.includes("cursor is no longer retained")) {
|
||||
throw new ExecutionDecisionCursorError(
|
||||
"decision cursor is no longer retained; restart inspection without --cursor",
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
decisions.push(
|
||||
...page.entries.map((entry) => ({
|
||||
receipt: entry.receipt,
|
||||
provenance: { state: "unverified" as const },
|
||||
selectorId: entry.selectorId,
|
||||
})),
|
||||
);
|
||||
if (page.nextCursor) {
|
||||
nextDecisionCursor = formatDecisionCursor("generic", page.nextCursor);
|
||||
} else {
|
||||
nextDecisionCursor = undefined;
|
||||
}
|
||||
}
|
||||
const ownerCoverage = new Set([approvalSummary.coverageState, messageSummary.coverageState]);
|
||||
const ownerCoverage = new Set<DecisionReceiptV1["enforcement"]["coverageState"]>(
|
||||
[
|
||||
approvalSummary.coverageState,
|
||||
messageSummary.coverageState,
|
||||
cronSummary.coverageState,
|
||||
taskSummary.coverageState,
|
||||
flowSummary.coverageState,
|
||||
].filter(
|
||||
(coverageState): coverageState is NonNullable<typeof coverageState> =>
|
||||
coverageState !== undefined,
|
||||
),
|
||||
);
|
||||
const hasUnverifiedGenericDecisions = genericSummary.count > 0;
|
||||
const boundedEvidence = boundMissingEvidence([
|
||||
...params.context.missingEvidence,
|
||||
...approvalSummary.missingEvidence,
|
||||
...messageSummary.missingEvidence,
|
||||
...cronSummary.missingEvidence,
|
||||
...taskSummary.missingEvidence,
|
||||
...flowSummary.missingEvidence,
|
||||
...(hasUnverifiedGenericDecisions ? ["decision.display_provenance"] : []),
|
||||
]);
|
||||
const coverageState = boundedEvidence.truncated
|
||||
? "unknown"
|
||||
: hasUnverifiedGenericDecisions
|
||||
? "unknown"
|
||||
: ownerCoverage.has("unknown")
|
||||
? "unknown"
|
||||
: ownerCoverage.has("enforced")
|
||||
? "enforced"
|
||||
: ownerCoverage.has("attribution-only")
|
||||
? "attribution-only"
|
||||
: params.context.coverageState;
|
||||
: ownerCoverage.has("unsupported")
|
||||
? "unsupported"
|
||||
: ownerCoverage.has("unknown")
|
||||
? "unknown"
|
||||
: ownerCoverage.has("enforced")
|
||||
? "enforced"
|
||||
: ownerCoverage.has("attribution-only")
|
||||
? "attribution-only"
|
||||
: params.context.coverageState;
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
run: {
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
import type {
|
||||
AdmittedRunContext,
|
||||
PreparedAgentRunAdmission,
|
||||
} from "../agents/admitted-run-context.js";
|
||||
import { parseExecutionIdentityAdmissionToken } from "./execution-identity-admission.js";
|
||||
|
||||
export type ExecutionOwnerBindingResult =
|
||||
| "disabled"
|
||||
| "bound"
|
||||
| "already-bound"
|
||||
| "mismatch"
|
||||
| "missing";
|
||||
|
||||
export function isRetainedExecutionOwnerBinding(
|
||||
result: ExecutionOwnerBindingResult | undefined,
|
||||
): result is "bound" | "already-bound" {
|
||||
return result === "bound" || result === "already-bound";
|
||||
}
|
||||
|
||||
type ExecutionOwnerBinding = Readonly<{
|
||||
contextId: string;
|
||||
executionId: string;
|
||||
}>;
|
||||
|
||||
/** Extracts only an admitted exact identity; operational run correlation cannot bind owner rows. */
|
||||
export function executionOwnerBindingFromAdmission(
|
||||
admitted: AdmittedRunContext,
|
||||
): ExecutionOwnerBinding | undefined {
|
||||
if (!admitted.executionIdentityToken) {
|
||||
return undefined;
|
||||
}
|
||||
const token = parseExecutionIdentityAdmissionToken(admitted.executionIdentityToken);
|
||||
if (token.runId !== admitted.operationalRunInstance.runId) {
|
||||
throw new Error("owner execution binding disagrees with the admitted run");
|
||||
}
|
||||
return { contextId: token.contextId, executionId: token.executionId };
|
||||
}
|
||||
|
||||
export function classifyExecutionOwnerBinding(
|
||||
current: { contextId: string | null; executionId: string | null },
|
||||
binding: ExecutionOwnerBinding,
|
||||
): Exclude<ExecutionOwnerBindingResult, "disabled" | "bound" | "missing"> | "unbound" {
|
||||
if (current.contextId === null && current.executionId === null) {
|
||||
return "unbound";
|
||||
}
|
||||
return current.contextId === binding.contextId && current.executionId === binding.executionId
|
||||
? "already-bound"
|
||||
: "mismatch";
|
||||
}
|
||||
|
||||
/** Adds one exact owner write after admission resolves, never inside the admission callback. */
|
||||
export function withPostAdmissionExecutionOwnerBinding(
|
||||
prepared: PreparedAgentRunAdmission,
|
||||
bind: (context: AdmittedRunContext) => void,
|
||||
): PreparedAgentRunAdmission {
|
||||
let bound = false;
|
||||
return Object.freeze({
|
||||
operationalRunInstance: prepared.operationalRunInstance,
|
||||
admit: async (runtimeKind, runtimeInstanceId) => {
|
||||
const admitted = await prepared.admit(runtimeKind, runtimeInstanceId);
|
||||
if (!bound) {
|
||||
bound = true;
|
||||
bind(admitted);
|
||||
}
|
||||
return admitted;
|
||||
},
|
||||
close: prepared.close,
|
||||
});
|
||||
}
|
||||
|
||||
/** Requires both exact admission and actual execution start, in either runtime order. */
|
||||
export function createExecutionStartedOwnerBinding(bind: (context: AdmittedRunContext) => void): {
|
||||
onPostAdmission: (context: AdmittedRunContext) => void;
|
||||
onExecutionStarted: () => void;
|
||||
} {
|
||||
let admitted: AdmittedRunContext | undefined;
|
||||
let executionStarted = false;
|
||||
let bound = false;
|
||||
const bindIfReady = () => {
|
||||
if (bound || !admitted || !executionStarted) {
|
||||
return;
|
||||
}
|
||||
bound = true;
|
||||
bind(admitted);
|
||||
};
|
||||
return {
|
||||
onPostAdmission: (context) => {
|
||||
admitted = context;
|
||||
bindIfReady();
|
||||
},
|
||||
onExecutionStarted: () => {
|
||||
executionStarted = true;
|
||||
bindIfReady();
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
executeSqliteQueryTakeFirstSync,
|
||||
getNodeSqliteKysely,
|
||||
} from "../infra/kysely-sync.js";
|
||||
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
|
||||
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
|
||||
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
|
||||
import {
|
||||
classifyExecutionOwnerBinding,
|
||||
type ExecutionOwnerBindingResult,
|
||||
} from "./execution-owner-binding.js";
|
||||
|
||||
export const EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE =
|
||||
"execution_owner_lifecycle_bindings" as const;
|
||||
type ExecutionOwnerLifecycleKind = "cron" | "task" | "flow";
|
||||
|
||||
type ExecutionOwnerLifecycleDatabase = Pick<
|
||||
OpenClawStateDatabase,
|
||||
"cron_run_receipts" | "execution_owner_lifecycle_bindings" | "flow_runs" | "task_runs"
|
||||
>;
|
||||
|
||||
const SCHEMA_START = `CREATE TABLE IF NOT EXISTS ${EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE} (`;
|
||||
const SCHEMA_END = ") STRICT;";
|
||||
|
||||
function lifecycleDb(db: DatabaseSync) {
|
||||
return getNodeSqliteKysely<ExecutionOwnerLifecycleDatabase>(db);
|
||||
}
|
||||
|
||||
/** Creates only the canonical additive metadata table at first admitted owner use. */
|
||||
function ensureExecutionOwnerLifecycleBindingSchema(db: DatabaseSync): void {
|
||||
if (tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) {
|
||||
return;
|
||||
}
|
||||
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_START);
|
||||
const end = start < 0 ? -1 : OPENCLAW_STATE_SCHEMA_SQL.indexOf(SCHEMA_END, start);
|
||||
if (start < 0 || end < start) {
|
||||
throw new Error("OpenClaw execution owner lifecycle binding schema marker is missing.");
|
||||
}
|
||||
// sqlite-allow-raw -- Canonical feature-local additive DDL only; metadata rows use Kysely.
|
||||
db.exec(OPENCLAW_STATE_SCHEMA_SQL.slice(start, end + SCHEMA_END.length));
|
||||
}
|
||||
|
||||
function classifyRetainedBinding(
|
||||
current: { context_id: string; execution_id: string },
|
||||
binding: { contextId: string; executionId: string },
|
||||
): "already-bound" | "mismatch" {
|
||||
const state = classifyExecutionOwnerBinding(
|
||||
{ contextId: current.context_id, executionId: current.execution_id },
|
||||
binding,
|
||||
);
|
||||
return state === "unbound" ? "mismatch" : state;
|
||||
}
|
||||
|
||||
/** Stores one exact admission identity after its canonical owner row has been revalidated. */
|
||||
export function bindExecutionOwnerLifecycleMetadata(params: {
|
||||
db: DatabaseSync;
|
||||
ownerKind: ExecutionOwnerLifecycleKind;
|
||||
ownerId: string;
|
||||
binding: { contextId: string; executionId: string };
|
||||
}): Exclude<ExecutionOwnerBindingResult, "disabled" | "missing"> {
|
||||
ensureExecutionOwnerLifecycleBindingSchema(params.db);
|
||||
const database = lifecycleDb(params.db);
|
||||
const current = executeSqliteQueryTakeFirstSync(
|
||||
params.db,
|
||||
database
|
||||
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.select(["context_id", "execution_id"])
|
||||
.where("owner_kind", "=", params.ownerKind)
|
||||
.where("owner_id", "=", params.ownerId),
|
||||
);
|
||||
if (current) {
|
||||
return classifyRetainedBinding(current, params.binding);
|
||||
}
|
||||
const inserted = executeSqliteQuerySync(
|
||||
params.db,
|
||||
database
|
||||
.insertInto(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.values({
|
||||
owner_kind: params.ownerKind,
|
||||
owner_id: params.ownerId,
|
||||
context_id: params.binding.contextId,
|
||||
execution_id: params.binding.executionId,
|
||||
})
|
||||
.onConflict((conflict) => conflict.columns(["owner_kind", "owner_id"]).doNothing()),
|
||||
);
|
||||
if (Number(inserted.numAffectedRows ?? 0n) === 1) {
|
||||
return "bound";
|
||||
}
|
||||
const raced = executeSqliteQueryTakeFirstSync(
|
||||
params.db,
|
||||
database
|
||||
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.select(["context_id", "execution_id"])
|
||||
.where("owner_kind", "=", params.ownerKind)
|
||||
.where("owner_id", "=", params.ownerId),
|
||||
);
|
||||
return raced ? classifyRetainedBinding(raced, params.binding) : "mismatch";
|
||||
}
|
||||
|
||||
/** Removes exact owner metadata without allocating the opt-in table. */
|
||||
export function deleteExecutionOwnerLifecycleMetadata(params: {
|
||||
db: DatabaseSync;
|
||||
ownerKind: ExecutionOwnerLifecycleKind;
|
||||
ownerIds: readonly string[];
|
||||
}): void {
|
||||
if (
|
||||
params.ownerIds.length === 0 ||
|
||||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
executeSqliteQuerySync(
|
||||
params.db,
|
||||
lifecycleDb(params.db)
|
||||
.deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.where("owner_kind", "=", params.ownerKind)
|
||||
.where("owner_id", "in", params.ownerIds),
|
||||
);
|
||||
}
|
||||
|
||||
/** Removes bindings whose canonical owner row was pruned in the same transaction. */
|
||||
export function pruneOrphanedExecutionOwnerLifecycleMetadata(
|
||||
db: DatabaseSync,
|
||||
ownerKind: ExecutionOwnerLifecycleKind,
|
||||
): void {
|
||||
if (!tableExists(db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)) {
|
||||
return;
|
||||
}
|
||||
const database = lifecycleDb(db);
|
||||
const bindings = database
|
||||
.deleteFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.where("owner_kind", "=", ownerKind);
|
||||
if (ownerKind === "cron") {
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
bindings.where(
|
||||
"owner_id",
|
||||
"not in",
|
||||
database.selectFrom("cron_run_receipts").select("receipt_id"),
|
||||
),
|
||||
);
|
||||
} else if (ownerKind === "task") {
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
bindings.where("owner_id", "not in", database.selectFrom("task_runs").select("task_id")),
|
||||
);
|
||||
} else {
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
bindings.where("owner_id", "not in", database.selectFrom("flow_runs").select("flow_id")),
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,611 @@
|
||||
import { rmSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import type { ExecutionIdentityContextV1 } from "../../packages/gateway-protocol/src/index.js";
|
||||
import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js";
|
||||
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
|
||||
import { bindCronRunReceiptExecution } from "../cron/store/run-receipt-store.js";
|
||||
import { tableHasColumn, tableExists } from "../state/openclaw-state-db-schema-helpers.js";
|
||||
import {
|
||||
OPENCLAW_STATE_SCHEMA_VERSION,
|
||||
closeOpenClawStateDatabaseForTest,
|
||||
openOpenClawStateDatabase,
|
||||
} from "../state/openclaw-state-db.js";
|
||||
import { OPENCLAW_STATE_SCHEMA_SQL } from "../state/openclaw-state-schema.js";
|
||||
import { bindTaskFlowExecution } from "../tasks/task-flow-registry.store.sqlite.js";
|
||||
import { bindTaskRunExecution } from "../tasks/task-registry.store.sqlite.js";
|
||||
import { presentExecutionDecisionReceipts } from "./execution-decision-receipts.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "./execution-identity-admission.js";
|
||||
import { pageOwnerLifecycleReceipts } from "./execution-owner-lifecycle-receipts.js";
|
||||
|
||||
afterEach(() => closeOpenClawStateDatabaseForTest());
|
||||
const tempDirs = useAutoCleanupTempDirTracker(afterEach);
|
||||
|
||||
function oldSchemaSql(): string {
|
||||
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(
|
||||
"CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (",
|
||||
);
|
||||
const endMarker = ") STRICT;";
|
||||
const end = OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start);
|
||||
if (start < 0 || end < start) {
|
||||
throw new Error("owner lifecycle binding schema marker is missing");
|
||||
}
|
||||
return `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`;
|
||||
}
|
||||
|
||||
function createOldOwnerDatabase() {
|
||||
const pathname = path.join(tempDirs.make("owner-lifecycle-"), "openclaw.sqlite");
|
||||
const oldReader = new DatabaseSync(pathname);
|
||||
oldReader.exec(oldSchemaSql());
|
||||
oldReader.exec(`PRAGMA user_version = ${OPENCLAW_STATE_SCHEMA_VERSION}`);
|
||||
oldReader
|
||||
.prepare(
|
||||
`INSERT INTO schema_meta (
|
||||
meta_key, role, schema_version, created_at, updated_at
|
||||
) VALUES ('primary', 'global', ?, 1, 1)`,
|
||||
)
|
||||
.run(OPENCLAW_STATE_SCHEMA_VERSION);
|
||||
oldReader
|
||||
.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null);
|
||||
oldReader
|
||||
.prepare(
|
||||
`INSERT INTO task_runs (
|
||||
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
|
||||
notify_policy, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
"task-1",
|
||||
"cron",
|
||||
"owner-1",
|
||||
"system",
|
||||
"private",
|
||||
"running",
|
||||
"not_applicable",
|
||||
"silent",
|
||||
61,
|
||||
);
|
||||
oldReader
|
||||
.prepare(
|
||||
`INSERT INTO flow_runs (
|
||||
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run("flow-1", "owner-1", "running", "silent", "private", 62, 62);
|
||||
oldReader.close();
|
||||
return { path: pathname };
|
||||
}
|
||||
|
||||
function admitted(contextId = "context-1", executionId = "execution-1"): AdmittedRunContext {
|
||||
return {
|
||||
operationalRunInstance: { instanceId: "instance-1", runId: "run-1" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-1", {
|
||||
contextId,
|
||||
executionId,
|
||||
now: 50,
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function executionContext(contextId = "context-1"): ExecutionIdentityContextV1 {
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
contextId,
|
||||
executionId: "execution-1",
|
||||
runId: "run-1",
|
||||
createdAt: 50,
|
||||
trustDomain: { kind: "gateway-cell", domainRef: "host-1", state: "present" },
|
||||
invoker: { state: "unknown" },
|
||||
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
|
||||
agentPrincipal: { kind: "agent", domainRef: "host-1", principalRef: "main" },
|
||||
agentDefinition: { definitionRef: "main", state: "present" },
|
||||
runtimeInstance: { runtimeRef: "runtime-1", kind: "embedded", state: "present" },
|
||||
applicableGrants: [],
|
||||
assurance: [],
|
||||
coverageState: "unattributed",
|
||||
missingEvidence: [],
|
||||
};
|
||||
}
|
||||
|
||||
const receiptHandle = {
|
||||
receiptId: "cron-1",
|
||||
storeKey: "default",
|
||||
jobId: "job-1",
|
||||
configRevision: "revision-1",
|
||||
agentId: "main",
|
||||
ownerPid: 1,
|
||||
ownerStartTime: null,
|
||||
startedAtMs: 60,
|
||||
};
|
||||
|
||||
describe("owner-native execution lifecycle receipts", () => {
|
||||
it("lazily binds exact owner rows while disabled collection allocates nothing", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
const current = openOpenClawStateDatabase(options).db;
|
||||
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false);
|
||||
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
|
||||
expect(tableHasColumn(current, table, "context_id")).toBe(false);
|
||||
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
|
||||
}
|
||||
|
||||
const disabled: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-disabled", runId: "run-1" },
|
||||
};
|
||||
expect(
|
||||
bindCronRunReceiptExecution({ admitted: disabled, handle: receiptHandle, options }),
|
||||
).toBe("disabled");
|
||||
expect(bindTaskRunExecution({ admitted: disabled, taskId: "task-1", options })).toBe(
|
||||
"disabled",
|
||||
);
|
||||
expect(bindTaskFlowExecution({ admitted: disabled, flowId: "flow-1", options })).toBe(
|
||||
"disabled",
|
||||
);
|
||||
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(false);
|
||||
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
|
||||
expect(tableHasColumn(current, table, "context_id")).toBe(false);
|
||||
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
|
||||
}
|
||||
|
||||
expect(
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
|
||||
).toBe("bound");
|
||||
expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe("bound");
|
||||
expect(bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options })).toBe(
|
||||
"bound",
|
||||
);
|
||||
expect(tableExists(current, "execution_owner_lifecycle_bindings")).toBe(true);
|
||||
for (const table of ["cron_run_receipts", "task_runs", "flow_runs"]) {
|
||||
expect(tableHasColumn(current, table, "context_id")).toBe(false);
|
||||
expect(tableHasColumn(current, table, "execution_id")).toBe(false);
|
||||
}
|
||||
expect(bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options })).toBe(
|
||||
"already-bound",
|
||||
);
|
||||
expect(
|
||||
bindTaskRunExecution({
|
||||
admitted: admitted("context-1", "execution-other"),
|
||||
taskId: "task-1",
|
||||
options,
|
||||
}),
|
||||
).toBe("mismatch");
|
||||
current.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run();
|
||||
current.prepare("UPDATE task_runs SET status = 'succeeded'").run();
|
||||
current.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run();
|
||||
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
const oldReader = new DatabaseSync(options.path);
|
||||
expect(
|
||||
oldReader.prepare("SELECT status FROM task_runs WHERE task_id = ?").get("task-1"),
|
||||
).toEqual({
|
||||
status: "succeeded",
|
||||
});
|
||||
oldReader.prepare("UPDATE task_runs SET status = ? WHERE task_id = ?").run("failed", "task-1");
|
||||
oldReader.close();
|
||||
|
||||
const reopened = openOpenClawStateDatabase(options).db;
|
||||
expect(reopened.prepare("PRAGMA user_version").get()).toEqual({
|
||||
user_version: OPENCLAW_STATE_SCHEMA_VERSION,
|
||||
});
|
||||
expect(
|
||||
reopened
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, task.status
|
||||
FROM task_runs AS task
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
|
||||
WHERE task.task_id = ?`,
|
||||
)
|
||||
.get("task-1"),
|
||||
).toEqual({ context_id: "context-1", execution_id: "execution-1", status: "failed" });
|
||||
expect(
|
||||
reopened
|
||||
.prepare(
|
||||
`SELECT owner_kind, owner_id, context_id, execution_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
ORDER BY owner_kind`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([
|
||||
{
|
||||
owner_kind: "cron",
|
||||
owner_id: "cron-1",
|
||||
context_id: "context-1",
|
||||
execution_id: "execution-1",
|
||||
},
|
||||
{
|
||||
owner_kind: "flow",
|
||||
owner_id: "flow-1",
|
||||
context_id: "context-1",
|
||||
execution_id: "execution-1",
|
||||
},
|
||||
{
|
||||
owner_kind: "task",
|
||||
owner_id: "task-1",
|
||||
context_id: "context-1",
|
||||
execution_id: "execution-1",
|
||||
},
|
||||
]);
|
||||
expect(tableExists(reopened, "execution_decision_facts")).toBe(true);
|
||||
expect(
|
||||
reopened.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get(),
|
||||
).toEqual({ count: 0 });
|
||||
});
|
||||
|
||||
it("uses stable c/t/f cursors and rejects a mismatched exact execution", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options });
|
||||
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options });
|
||||
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options });
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
db.prepare("UPDATE cron_run_receipts SET status = 'ok', finished_at_ms = 70").run();
|
||||
db.prepare("UPDATE task_runs SET status = 'succeeded'").run();
|
||||
db.prepare("UPDATE flow_runs SET status = 'succeeded', ended_at = 70").run();
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "skipped", 1, 60, 70);
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES (?, ?, ?, ?)`,
|
||||
).run("cron", "cron-2", "context-1", "execution-other");
|
||||
const context = executionContext();
|
||||
|
||||
const first = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(first.nextDecisionCursor).toBe("a:0:0");
|
||||
const cronPage = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionCursor: first.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(cronPage.decisions[0]?.source.owner).toBe("cron_run_receipts");
|
||||
expect(cronPage.decisionDisplays[0]).toMatchObject({
|
||||
decision: { outcome: "not-applicable", reasonCode: "cron_run_ok" },
|
||||
provenance: { state: "verified", producer: "cron-lifecycle" },
|
||||
});
|
||||
expect(cronPage.nextDecisionCursor).toMatch(/^c:/);
|
||||
const mismatchPage = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionCursor: cronPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(mismatchPage.decisions[0]).toMatchObject({
|
||||
decision: { outcome: "unknown", reasonCode: "cron_run_execution_link_mismatch" },
|
||||
missingEvidence: ["decision.execution_link"],
|
||||
});
|
||||
expect(mismatchPage.nextDecisionCursor).toBe("t:0:0");
|
||||
const taskPage = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionCursor: mismatchPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(taskPage.decisions[0]?.source.owner).toBe("task_runs");
|
||||
expect(taskPage.decisionDisplays[0]).toMatchObject({
|
||||
decision: { outcome: "not-applicable", reasonCode: "task_run_succeeded" },
|
||||
provenance: { state: "verified", producer: "task-lifecycle" },
|
||||
});
|
||||
expect(taskPage.nextDecisionCursor).toBe("f:0:0");
|
||||
const flowPage = presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionCursor: taskPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(flowPage.decisions[0]?.source.owner).toBe("flow_runs");
|
||||
expect(flowPage.decisionDisplays[0]).toMatchObject({
|
||||
decision: { outcome: "not-applicable", reasonCode: "flow_run_succeeded" },
|
||||
provenance: { state: "verified", producer: "flow-lifecycle" },
|
||||
});
|
||||
expect(
|
||||
JSON.stringify([
|
||||
cronPage.decisionDisplays,
|
||||
taskPage.decisionDisplays,
|
||||
flowPage.decisionDisplays,
|
||||
]),
|
||||
).not.toMatch(/cron-1|task-1|flow-1|private|cron_run_receipts|task_runs|flow_runs/);
|
||||
expect(flowPage.nextDecisionCursor).toBeUndefined();
|
||||
|
||||
expect(
|
||||
presentExecutionDecisionReceipts({
|
||||
context,
|
||||
decisionCursor: "1",
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
}).decisions[0]?.source.owner,
|
||||
).toBe("cron_run_receipts");
|
||||
});
|
||||
|
||||
const deletedAnchorCases = [
|
||||
{
|
||||
stage: "cron",
|
||||
cursor: "c:0:0",
|
||||
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
|
||||
addSuccessor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null);
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`,
|
||||
).run();
|
||||
},
|
||||
deleteAnchor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'",
|
||||
).run();
|
||||
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
|
||||
},
|
||||
},
|
||||
{
|
||||
stage: "task",
|
||||
cursor: "t:0:0",
|
||||
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
|
||||
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options }),
|
||||
addSuccessor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
`INSERT INTO task_runs (
|
||||
task_id, runtime, owner_key, scope_kind, task, status, delivery_status,
|
||||
notify_policy, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
"task-2",
|
||||
"cron",
|
||||
"owner-2",
|
||||
"system",
|
||||
"private",
|
||||
"running",
|
||||
"not_applicable",
|
||||
"silent",
|
||||
63,
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES ('task', 'task-2', 'context-1', 'execution-1')`,
|
||||
).run();
|
||||
},
|
||||
deleteAnchor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'task' AND owner_id = 'task-1'",
|
||||
).run();
|
||||
db.prepare("DELETE FROM task_runs WHERE task_id = 'task-1'").run();
|
||||
},
|
||||
},
|
||||
{
|
||||
stage: "flow",
|
||||
cursor: "f:0:0",
|
||||
bindFirst: (options: ReturnType<typeof createOldOwnerDatabase>) =>
|
||||
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options }),
|
||||
addSuccessor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
`INSERT INTO flow_runs (
|
||||
flow_id, owner_key, status, notify_policy, goal, created_at, updated_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("flow-2", "owner-2", "running", "silent", "private", 63, 63);
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES ('flow', 'flow-2', 'context-1', 'execution-1')`,
|
||||
).run();
|
||||
},
|
||||
deleteAnchor: (db: DatabaseSync) => {
|
||||
db.prepare(
|
||||
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'flow' AND owner_id = 'flow-1'",
|
||||
).run();
|
||||
db.prepare("DELETE FROM flow_runs WHERE flow_id = 'flow-1'").run();
|
||||
},
|
||||
},
|
||||
] as const;
|
||||
|
||||
it.each(deletedAnchorCases)(
|
||||
"rejects a nonzero $stage cursor after its exact owner anchor is deleted",
|
||||
({ cursor, bindFirst, addSuccessor, deleteAnchor }) => {
|
||||
const options = createOldOwnerDatabase();
|
||||
expect(bindFirst(options)).toBe("bound");
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
addSuccessor(db);
|
||||
const firstPage = presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: cursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(firstPage.nextDecisionCursor).toMatch(/^[ctf]:[1-9]\d*:[1-9]\d*$/);
|
||||
|
||||
deleteAnchor(db);
|
||||
|
||||
expect(() =>
|
||||
presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: firstPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
}),
|
||||
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects a nonzero owner cursor when its retained anchor belongs to another context", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
expect(
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
|
||||
).toBe("bound");
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
deletedAnchorCases[0].addSuccessor(db);
|
||||
const firstPage = presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: "c:0:0",
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/);
|
||||
|
||||
expect(() =>
|
||||
presentExecutionDecisionReceipts({
|
||||
context: executionContext("context-other"),
|
||||
decisionCursor: firstPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
}),
|
||||
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
|
||||
});
|
||||
|
||||
it("rejects a nonzero owner cursor after the state database is removed", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
expect(
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
|
||||
).toBe("bound");
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
deletedAnchorCases[0].addSuccessor(db);
|
||||
const firstPage = presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: "c:0:0",
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(firstPage.nextDecisionCursor).toMatch(/^c:[1-9]\d*:[1-9]\d*$/);
|
||||
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
rmSync(options.path);
|
||||
|
||||
expect(() =>
|
||||
presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: firstPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
}),
|
||||
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
|
||||
});
|
||||
|
||||
it("rejects a reused owner rowid whose binding belongs to another execution", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("cron-2", "default", "job-2", "revision-1", "main", "run-1", "running", 1, 63, null);
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run("cron-1", "default", "job-1", "revision-1", "main", "run-1", "running", 1, 60, null);
|
||||
expect(
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options }),
|
||||
).toBe("bound");
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES ('cron', 'cron-2', 'context-1', 'execution-1')`,
|
||||
).run();
|
||||
const firstPage = presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: "c:0:0",
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
});
|
||||
expect(firstPage.nextDecisionCursor).toMatch(/^c:60:[1-9]\d*$/);
|
||||
|
||||
db.prepare(
|
||||
"DELETE FROM execution_owner_lifecycle_bindings WHERE owner_kind = 'cron' AND owner_id = 'cron-1'",
|
||||
).run();
|
||||
db.prepare("DELETE FROM cron_run_receipts WHERE receipt_id = 'cron-1'").run();
|
||||
db.prepare(
|
||||
`INSERT INTO cron_run_receipts (
|
||||
receipt_id, store_key, job_id, config_revision, agent_id, request_run_id,
|
||||
status, owner_pid, started_at_ms, finished_at_ms
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
"cron-replacement",
|
||||
"default",
|
||||
"job-replacement",
|
||||
"revision-1",
|
||||
"main",
|
||||
"run-2",
|
||||
"running",
|
||||
1,
|
||||
60,
|
||||
null,
|
||||
);
|
||||
db.prepare(
|
||||
`INSERT INTO execution_owner_lifecycle_bindings (
|
||||
owner_kind, owner_id, context_id, execution_id
|
||||
) VALUES ('cron', 'cron-replacement', 'context-1', 'execution-other')`,
|
||||
).run();
|
||||
|
||||
expect(() =>
|
||||
presentExecutionDecisionReceipts({
|
||||
context: executionContext(),
|
||||
decisionCursor: firstPage.nextDecisionCursor,
|
||||
decisionLimit: 1,
|
||||
options,
|
||||
}),
|
||||
).toThrow("decision cursor is no longer retained; restart inspection without --cursor");
|
||||
});
|
||||
|
||||
it("projects every owner terminal state without rederiving lifecycle precedence", () => {
|
||||
const options = createOldOwnerDatabase();
|
||||
bindCronRunReceiptExecution({ admitted: admitted(), handle: receiptHandle, options });
|
||||
bindTaskRunExecution({ admitted: admitted(), taskId: "task-1", options });
|
||||
bindTaskFlowExecution({ admitted: admitted(), flowId: "flow-1", options });
|
||||
const context = executionContext();
|
||||
const db = openOpenClawStateDatabase(options).db;
|
||||
|
||||
for (const status of ["ok", "error", "skipped", "interrupted", "superseded"]) {
|
||||
db.prepare(
|
||||
"UPDATE cron_run_receipts SET status = ?, finished_at_ms = 70 WHERE receipt_id = ?",
|
||||
).run(status, "cron-1");
|
||||
expect(
|
||||
pageOwnerLifecycleReceipts({ stage: "cron", context, limit: 1, options }).entries[0]
|
||||
?.receipt.decision,
|
||||
).toEqual({ outcome: "not-applicable", reasonCode: `cron_run_${status}` });
|
||||
}
|
||||
for (const status of ["succeeded", "failed", "timed_out", "cancelled", "lost"]) {
|
||||
db.prepare("UPDATE task_runs SET status = ?, terminal_outcome = NULL WHERE task_id = ?").run(
|
||||
status,
|
||||
"task-1",
|
||||
);
|
||||
expect(
|
||||
pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0]
|
||||
?.receipt.decision.reasonCode,
|
||||
).toBe(`task_run_${status}`);
|
||||
}
|
||||
db.prepare("UPDATE task_runs SET terminal_outcome = 'blocked' WHERE task_id = ?").run("task-1");
|
||||
expect(
|
||||
pageOwnerLifecycleReceipts({ stage: "task", context, limit: 1, options }).entries[0]?.receipt
|
||||
.decision.reasonCode,
|
||||
).toBe("task_run_blocked");
|
||||
for (const status of ["blocked", "succeeded", "failed", "cancelled", "lost"]) {
|
||||
db.prepare("UPDATE flow_runs SET status = ? WHERE flow_id = ?").run(status, "flow-1");
|
||||
expect(
|
||||
pageOwnerLifecycleReceipts({ stage: "flow", context, limit: 1, options }).entries[0]
|
||||
?.receipt.decision.reasonCode,
|
||||
).toBe(`flow_run_${status}`);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,445 @@
|
||||
/** Static owner-native cron/task/flow lifecycle projection for run inspection. */
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import type {
|
||||
DecisionReceiptV1,
|
||||
ExecutionIdentityContextV1,
|
||||
} from "../../packages/gateway-protocol/src/index.js";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
executeSqliteQueryTakeFirstSync,
|
||||
getNodeSqliteKysely,
|
||||
} from "../infra/kysely-sync.js";
|
||||
import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js";
|
||||
import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js";
|
||||
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
|
||||
import type { DB as OpenClawStateDatabase } from "../state/openclaw-state-db.generated.js";
|
||||
import type { OpenClawStateDatabaseOptions } from "../state/openclaw-state-db.js";
|
||||
import { EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE } from "./execution-owner-lifecycle-binding-store.js";
|
||||
|
||||
type WithSqliteRowId<Row> = Row & { rowid: number };
|
||||
type OwnerLifecycleDatabase = {
|
||||
cron_run_receipts: WithSqliteRowId<OpenClawStateDatabase["cron_run_receipts"]>;
|
||||
execution_owner_lifecycle_bindings: OpenClawStateDatabase["execution_owner_lifecycle_bindings"];
|
||||
flow_runs: WithSqliteRowId<OpenClawStateDatabase["flow_runs"]>;
|
||||
task_runs: WithSqliteRowId<OpenClawStateDatabase["task_runs"]>;
|
||||
};
|
||||
export type OwnerLifecycleStage = "cron" | "task" | "flow";
|
||||
export type OwnerLifecycleCursor = { occurredAt: number; rowId: number };
|
||||
type OwnerLifecycleDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle";
|
||||
type OwnerLifecycleReceiptEntry = {
|
||||
receipt: DecisionReceiptV1;
|
||||
selectorId: string;
|
||||
displayProducer: OwnerLifecycleDisplayProducer;
|
||||
};
|
||||
type OwnerLifecycleRow = {
|
||||
executionId: string | null;
|
||||
occurredAt: number;
|
||||
owner: "cron_run_receipts" | "task_runs" | "flow_runs";
|
||||
recordId: string;
|
||||
rowId: number;
|
||||
status: string;
|
||||
};
|
||||
|
||||
const KNOWN_STATUSES: Record<OwnerLifecycleStage, ReadonlySet<string>> = {
|
||||
cron: new Set(["running", "ok", "error", "skipped", "interrupted", "superseded"]),
|
||||
task: new Set([
|
||||
"queued",
|
||||
"running",
|
||||
"succeeded",
|
||||
"failed",
|
||||
"timed_out",
|
||||
"cancelled",
|
||||
"lost",
|
||||
"blocked",
|
||||
]),
|
||||
flow: new Set([
|
||||
"queued",
|
||||
"running",
|
||||
"waiting",
|
||||
"blocked",
|
||||
"succeeded",
|
||||
"failed",
|
||||
"cancelled",
|
||||
"lost",
|
||||
]),
|
||||
};
|
||||
const OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR = "owner lifecycle cursor is no longer retained";
|
||||
|
||||
function ownerName(stage: OwnerLifecycleStage): OwnerLifecycleRow["owner"] {
|
||||
return stage === "cron" ? "cron_run_receipts" : stage === "task" ? "task_runs" : "flow_runs";
|
||||
}
|
||||
|
||||
function displayProducer(stage: OwnerLifecycleStage): OwnerLifecycleDisplayProducer {
|
||||
return stage === "cron"
|
||||
? "cron-lifecycle"
|
||||
: stage === "task"
|
||||
? "task-lifecycle"
|
||||
: "flow-lifecycle";
|
||||
}
|
||||
|
||||
function assertRetainedCursor(params: {
|
||||
db: DatabaseSync;
|
||||
stage: OwnerLifecycleStage;
|
||||
contextId: string;
|
||||
executionId: string;
|
||||
after?: OwnerLifecycleCursor;
|
||||
}): void {
|
||||
if (!params.after) {
|
||||
return;
|
||||
}
|
||||
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
|
||||
const ownerQuery =
|
||||
params.stage === "cron"
|
||||
? kysely
|
||||
.selectFrom("cron_run_receipts")
|
||||
.select("receipt_id as ownerId")
|
||||
.where("rowid", "=", params.after.rowId)
|
||||
.where("started_at_ms", "=", params.after.occurredAt)
|
||||
: params.stage === "task"
|
||||
? kysely
|
||||
.selectFrom("task_runs")
|
||||
.select("task_id as ownerId")
|
||||
.where("rowid", "=", params.after.rowId)
|
||||
.where("created_at", "=", params.after.occurredAt)
|
||||
: kysely
|
||||
.selectFrom("flow_runs")
|
||||
.select("flow_id as ownerId")
|
||||
.where("rowid", "=", params.after.rowId)
|
||||
.where("created_at", "=", params.after.occurredAt);
|
||||
const owner = executeSqliteQueryTakeFirstSync(params.db, ownerQuery);
|
||||
// Admission binds at most one owner per lifecycle kind to an execution.
|
||||
// The exact execution match therefore rejects a different owner reusing this rowid.
|
||||
const binding = owner
|
||||
? executeSqliteQueryTakeFirstSync(
|
||||
params.db,
|
||||
kysely
|
||||
.selectFrom(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
.select("owner_id")
|
||||
.where("owner_kind", "=", params.stage)
|
||||
.where("owner_id", "=", owner.ownerId)
|
||||
.where("context_id", "=", params.contextId)
|
||||
.where("execution_id", "=", params.executionId),
|
||||
)
|
||||
: undefined;
|
||||
if (!binding) {
|
||||
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
|
||||
}
|
||||
}
|
||||
|
||||
function readRows(params: {
|
||||
db: DatabaseSync;
|
||||
stage: OwnerLifecycleStage;
|
||||
contextId: string;
|
||||
executionId: string;
|
||||
after?: OwnerLifecycleCursor;
|
||||
offset?: number;
|
||||
limit: number;
|
||||
}): OwnerLifecycleRow[] {
|
||||
const owner = ownerName(params.stage);
|
||||
if (
|
||||
!tableExists(params.db, owner) ||
|
||||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
) {
|
||||
if (params.after) {
|
||||
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
|
||||
}
|
||||
return [];
|
||||
}
|
||||
assertRetainedCursor(params);
|
||||
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
|
||||
if (params.stage === "cron") {
|
||||
let query = kysely
|
||||
.selectFrom("cron_run_receipts")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "cron_run_receipts.receipt_id")
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"),
|
||||
)
|
||||
.select([
|
||||
"cron_run_receipts.receipt_id as recordId",
|
||||
"execution_owner_lifecycle_bindings.execution_id as executionId",
|
||||
"cron_run_receipts.started_at_ms as occurredAt",
|
||||
"cron_run_receipts.status",
|
||||
"cron_run_receipts.rowid as rowId",
|
||||
])
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.orderBy("cron_run_receipts.started_at_ms", "asc")
|
||||
.orderBy("cron_run_receipts.rowid", "asc")
|
||||
.limit(params.limit);
|
||||
if (params.after) {
|
||||
query = query.where((eb) =>
|
||||
eb.or([
|
||||
eb("cron_run_receipts.started_at_ms", ">", params.after!.occurredAt),
|
||||
eb.and([
|
||||
eb("cron_run_receipts.started_at_ms", "=", params.after!.occurredAt),
|
||||
eb("cron_run_receipts.rowid", ">", params.after!.rowId),
|
||||
]),
|
||||
]),
|
||||
);
|
||||
} else if (params.offset) {
|
||||
query = query.offset(params.offset);
|
||||
}
|
||||
return executeSqliteQuerySync(params.db, query).rows.map((row) =>
|
||||
Object.assign(row, { owner }),
|
||||
);
|
||||
}
|
||||
if (params.stage === "task") {
|
||||
let query = kysely
|
||||
.selectFrom("task_runs")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id")
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"),
|
||||
)
|
||||
.select([
|
||||
"task_runs.task_id as recordId",
|
||||
"execution_owner_lifecycle_bindings.execution_id as executionId",
|
||||
"task_runs.created_at as occurredAt",
|
||||
"task_runs.status",
|
||||
"task_runs.terminal_outcome as terminalOutcome",
|
||||
"task_runs.rowid as rowId",
|
||||
])
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.orderBy("task_runs.created_at", "asc")
|
||||
.orderBy("task_runs.rowid", "asc")
|
||||
.limit(params.limit);
|
||||
if (params.after) {
|
||||
query = query.where((eb) =>
|
||||
eb.or([
|
||||
eb("task_runs.created_at", ">", params.after!.occurredAt),
|
||||
eb.and([
|
||||
eb("task_runs.created_at", "=", params.after!.occurredAt),
|
||||
eb("task_runs.rowid", ">", params.after!.rowId),
|
||||
]),
|
||||
]),
|
||||
);
|
||||
} else if (params.offset) {
|
||||
query = query.offset(params.offset);
|
||||
}
|
||||
return executeSqliteQuerySync(params.db, query).rows.map((row) =>
|
||||
Object.assign(row, {
|
||||
owner,
|
||||
status: row.terminalOutcome === "blocked" ? "blocked" : row.status,
|
||||
}),
|
||||
);
|
||||
}
|
||||
let query = kysely
|
||||
.selectFrom("flow_runs")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id")
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"),
|
||||
)
|
||||
.select([
|
||||
"flow_runs.flow_id as recordId",
|
||||
"execution_owner_lifecycle_bindings.execution_id as executionId",
|
||||
"flow_runs.created_at as occurredAt",
|
||||
"flow_runs.status",
|
||||
"flow_runs.rowid as rowId",
|
||||
])
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.orderBy("flow_runs.created_at", "asc")
|
||||
.orderBy("flow_runs.rowid", "asc")
|
||||
.limit(params.limit);
|
||||
if (params.after) {
|
||||
query = query.where((eb) =>
|
||||
eb.or([
|
||||
eb("flow_runs.created_at", ">", params.after!.occurredAt),
|
||||
eb.and([
|
||||
eb("flow_runs.created_at", "=", params.after!.occurredAt),
|
||||
eb("flow_runs.rowid", ">", params.after!.rowId),
|
||||
]),
|
||||
]),
|
||||
);
|
||||
} else if (params.offset) {
|
||||
query = query.offset(params.offset);
|
||||
}
|
||||
return executeSqliteQuerySync(params.db, query).rows.map((row) => Object.assign(row, { owner }));
|
||||
}
|
||||
|
||||
function countRows(params: {
|
||||
db: DatabaseSync;
|
||||
stage: OwnerLifecycleStage;
|
||||
contextId: string;
|
||||
executionId?: string;
|
||||
}): number {
|
||||
const owner = ownerName(params.stage);
|
||||
if (
|
||||
!tableExists(params.db, owner) ||
|
||||
!tableExists(params.db, EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE)
|
||||
) {
|
||||
return 0;
|
||||
}
|
||||
const kysely = getNodeSqliteKysely<OwnerLifecycleDatabase>(params.db);
|
||||
const query =
|
||||
params.stage === "cron"
|
||||
? kysely
|
||||
.selectFrom("cron_run_receipts")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef(
|
||||
"execution_owner_lifecycle_bindings.owner_id",
|
||||
"=",
|
||||
"cron_run_receipts.receipt_id",
|
||||
)
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "cron"),
|
||||
)
|
||||
.select((eb) => eb.fn.countAll<number>().as("count"))
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.$if(params.executionId !== undefined, (qb) =>
|
||||
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
|
||||
)
|
||||
: params.stage === "task"
|
||||
? kysely
|
||||
.selectFrom("task_runs")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "task_runs.task_id")
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "task"),
|
||||
)
|
||||
.select((eb) => eb.fn.countAll<number>().as("count"))
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.$if(params.executionId !== undefined, (qb) =>
|
||||
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
|
||||
)
|
||||
: kysely
|
||||
.selectFrom("flow_runs")
|
||||
.innerJoin(EXECUTION_OWNER_LIFECYCLE_BINDING_TABLE, (join) =>
|
||||
join
|
||||
.onRef("execution_owner_lifecycle_bindings.owner_id", "=", "flow_runs.flow_id")
|
||||
.on("execution_owner_lifecycle_bindings.owner_kind", "=", "flow"),
|
||||
)
|
||||
.select((eb) => eb.fn.countAll<number>().as("count"))
|
||||
.where("execution_owner_lifecycle_bindings.context_id", "=", params.contextId)
|
||||
.$if(params.executionId !== undefined, (qb) =>
|
||||
qb.where("execution_owner_lifecycle_bindings.execution_id", "=", params.executionId!),
|
||||
);
|
||||
return executeSqliteQueryTakeFirstSync(params.db, query)?.count ?? 0;
|
||||
}
|
||||
|
||||
function projectReceipt(
|
||||
stage: OwnerLifecycleStage,
|
||||
row: OwnerLifecycleRow,
|
||||
context: ExecutionIdentityContextV1,
|
||||
): DecisionReceiptV1 {
|
||||
const exact = row.executionId === context.executionId;
|
||||
const known = KNOWN_STATUSES[stage].has(row.status);
|
||||
const valid = exact && known;
|
||||
const missingEvidence = valid
|
||||
? []
|
||||
: [exact ? `decision.${stage}_owner_status` : "decision.execution_link"];
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
receiptId: `${stage}:${row.recordId}`,
|
||||
contextId: context.contextId,
|
||||
executionId: context.executionId,
|
||||
runId: context.runId,
|
||||
actionId: row.recordId,
|
||||
occurredAt: row.occurredAt,
|
||||
action: {
|
||||
family: stage === "cron" ? "scheduled-run" : stage === "task" ? "task" : "flow",
|
||||
operation: "lifecycle",
|
||||
summary: valid
|
||||
? `${stage === "cron" ? "Scheduled run" : stage === "task" ? "Task" : "Flow"} lifecycle: ${row.status.replaceAll("_", "-")}.`
|
||||
: "Owner lifecycle evidence could not be matched exactly.",
|
||||
},
|
||||
decision: {
|
||||
outcome: valid ? "not-applicable" : "unknown",
|
||||
reasonCode: valid
|
||||
? `${stage}_run_${row.status}`
|
||||
: exact
|
||||
? `${stage}_run_status_unknown`
|
||||
: `${stage}_run_execution_link_mismatch`,
|
||||
},
|
||||
enforcement: {
|
||||
coverageState: valid ? "attribution-only" : "unknown",
|
||||
evaluatorRef: `${stage}-lifecycle-owner`,
|
||||
policyRefs: [],
|
||||
grantRefs: [],
|
||||
contextFieldsUsed: ["contextId", "executionId"],
|
||||
},
|
||||
source: {
|
||||
owner: row.owner,
|
||||
recordRef: row.recordId,
|
||||
decisionBoundary: `${stage}.run.lifecycle`,
|
||||
},
|
||||
missingEvidence,
|
||||
remediation: valid
|
||||
? []
|
||||
: [
|
||||
{
|
||||
code: "inspect_owner_execution_binding",
|
||||
text: "Inspect the owner row and its exact admission binding before drawing a lifecycle conclusion.",
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
export function summarizeOwnerLifecycleReceipts(params: {
|
||||
stage: OwnerLifecycleStage;
|
||||
context: ExecutionIdentityContextV1;
|
||||
options: OpenClawStateDatabaseOptions;
|
||||
}): { count: number; coverageState?: "attribution-only" | "unknown"; missingEvidence: string[] } {
|
||||
return (
|
||||
withExistingOpenClawStateDatabaseReadOnly(({ db }) => {
|
||||
const count = countRows({ db, stage: params.stage, contextId: params.context.contextId });
|
||||
const exactCount = countRows({
|
||||
db,
|
||||
stage: params.stage,
|
||||
contextId: params.context.contextId,
|
||||
executionId: params.context.executionId,
|
||||
});
|
||||
const mismatch = count !== exactCount;
|
||||
return {
|
||||
count,
|
||||
...(count > 0
|
||||
? { coverageState: mismatch ? ("unknown" as const) : ("attribution-only" as const) }
|
||||
: {}),
|
||||
missingEvidence: mismatch ? ["decision.execution_link"] : [],
|
||||
};
|
||||
}, params.options) ?? { count: 0, missingEvidence: [] }
|
||||
);
|
||||
}
|
||||
|
||||
export function pageOwnerLifecycleReceipts(params: {
|
||||
stage: OwnerLifecycleStage;
|
||||
context: ExecutionIdentityContextV1;
|
||||
after?: OwnerLifecycleCursor;
|
||||
offset?: number;
|
||||
limit: number;
|
||||
options: OpenClawStateDatabaseOptions;
|
||||
}): { entries: OwnerLifecycleReceiptEntry[]; nextCursor?: OwnerLifecycleCursor } {
|
||||
const retainedRows = withExistingOpenClawStateDatabaseReadOnly(
|
||||
({ db }) =>
|
||||
runSqliteDeferredTransactionSync(
|
||||
db,
|
||||
() =>
|
||||
readRows({
|
||||
db,
|
||||
stage: params.stage,
|
||||
contextId: params.context.contextId,
|
||||
executionId: params.context.executionId,
|
||||
after: params.after,
|
||||
offset: params.offset,
|
||||
limit: params.limit + 1,
|
||||
}),
|
||||
{ operationLabel: "owner lifecycle receipt page" },
|
||||
),
|
||||
params.options,
|
||||
);
|
||||
if (!retainedRows && params.after) {
|
||||
throw new Error(OWNER_LIFECYCLE_CURSOR_RETAINED_ERROR);
|
||||
}
|
||||
const rows = retainedRows ?? [];
|
||||
const hasMore = rows.length > params.limit;
|
||||
const page = hasMore ? rows.slice(0, params.limit) : rows;
|
||||
const last = page.at(-1);
|
||||
return {
|
||||
entries: page.map((row) => ({
|
||||
receipt: projectReceipt(params.stage, row, params.context),
|
||||
selectorId: `${params.stage}-lifecycle:${row.occurredAt}:${row.rowId}`,
|
||||
displayProducer: displayProducer(params.stage),
|
||||
})),
|
||||
...(hasMore && last ? { nextCursor: { occurredAt: last.occurredAt, rowId: last.rowId } } : {}),
|
||||
};
|
||||
}
|
||||
@@ -24,6 +24,7 @@ import { resolveScheduledToolPolicyContext } from "../../agents/scheduled-tool-p
|
||||
import { withLocalSessionPlacementTurnAdmission } from "../../agents/session-placement-admission.js";
|
||||
import { resolveSessionRuntimeOverrideForProvider } from "../../agents/session-runtime-compat.js";
|
||||
import { hasResolvedThinkingCatalogEntry } from "../../agents/thinking-runtime.js";
|
||||
import { withPostAdmissionExecutionOwnerBinding } from "../../audit/execution-owner-binding.js";
|
||||
import type { ThinkLevel, VerboseLevel } from "../../auto-reply/thinking.js";
|
||||
import type { CliSessionBinding } from "../../config/sessions.js";
|
||||
import type { AgentDefaultsConfig } from "../../config/types.agent-defaults.js";
|
||||
@@ -288,6 +289,7 @@ function createCronPromptExecutor(params: {
|
||||
Partial<Omit<CronAgentExecutionPhaseUpdate, "jobId" | "phase">>,
|
||||
) => void;
|
||||
onLaneWait?: (info?: { waiting?: boolean }) => void;
|
||||
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
|
||||
}) {
|
||||
const sessionFile = params.runSessionKey;
|
||||
const cronFallbacksOverride =
|
||||
@@ -396,15 +398,30 @@ function createCronPromptExecutor(params: {
|
||||
});
|
||||
let acceptedContextEngineTurnCandidate: ContextEngineTurnAttemptFacts | undefined;
|
||||
const runId = params.cronSession.sessionEntry.sessionId;
|
||||
const preparedRunAdmission = prepareAgentRunAdmission({
|
||||
const basePreparedRunAdmission = prepareAgentRunAdmission({
|
||||
operationalRunInstance: createOperationalRunInstanceRef(runId),
|
||||
cfg: params.cfgWithAgentDefaults,
|
||||
facts: {
|
||||
runId,
|
||||
agentId: params.agentId,
|
||||
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
|
||||
ingress: params.executionIdentity?.ingress ?? {
|
||||
kind: "schedule",
|
||||
boundary: "cron.isolated-agent",
|
||||
state: "present",
|
||||
},
|
||||
...(params.executionIdentity?.invoker ? { invoker: params.executionIdentity.invoker } : {}),
|
||||
},
|
||||
});
|
||||
const preparedRunAdmission = params.executionIdentity?.onPostAdmission
|
||||
? withPostAdmissionExecutionOwnerBinding(
|
||||
basePreparedRunAdmission,
|
||||
params.executionIdentity.onPostAdmission,
|
||||
)
|
||||
: basePreparedRunAdmission;
|
||||
const onExecutionStarted = (info?: CronRunnerStartedInfo) => {
|
||||
params.onExecutionStarted?.(info);
|
||||
params.executionIdentity?.onExecutionStarted?.();
|
||||
};
|
||||
const fallbackResult = await runWithModelFallback({
|
||||
cfg: params.cfgWithAgentDefaults,
|
||||
provider: params.liveSelection.provider,
|
||||
@@ -454,7 +471,7 @@ function createCronPromptExecutor(params: {
|
||||
const isFallback = candidateStarted;
|
||||
candidateStarted = true;
|
||||
const notifyExecutionStarted = (info?: { lifecycleGeneration?: string }) =>
|
||||
params.onExecutionStarted?.({
|
||||
onExecutionStarted({
|
||||
...info,
|
||||
...(isFallback ? { isFallback: true } : {}),
|
||||
provider: providerOverride,
|
||||
@@ -867,6 +884,7 @@ export async function executeCronRun(params: {
|
||||
Partial<Omit<CronAgentExecutionPhaseUpdate, "jobId" | "phase">>,
|
||||
) => void;
|
||||
onLaneWait?: (info?: { waiting?: boolean }) => void;
|
||||
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
|
||||
immutableThinkLevel: ThinkLevel | undefined;
|
||||
thinkingCatalog?: ModelCatalogEntry[];
|
||||
loadThinkingCatalog: (provider: string, model: string) => Promise<ModelCatalogEntry[]>;
|
||||
@@ -923,6 +941,7 @@ export async function executeCronRun(params: {
|
||||
onExecutionStarted: params.onExecutionStarted,
|
||||
onExecutionPhase: params.onExecutionPhase,
|
||||
onLaneWait: params.onLaneWait,
|
||||
executionIdentity: params.executionIdentity,
|
||||
});
|
||||
|
||||
const runStartedAt = params.runStartedAt ?? Date.now();
|
||||
|
||||
@@ -93,6 +93,7 @@ export async function runCronIsolatedAgentTurn(params: {
|
||||
sessionKey: string;
|
||||
agentId?: string;
|
||||
lane?: string;
|
||||
executionIdentity?: import("../service/state.js").CronExecutionIdentityAdmission;
|
||||
}): Promise<RunCronAgentTurnResult> {
|
||||
const admittedLifecycleGeneration = getAgentEventLifecycleGeneration();
|
||||
const upstreamAbortSignal = params.abortSignal ?? params.signal;
|
||||
@@ -264,6 +265,7 @@ export async function runCronIsolatedAgentTurn(params: {
|
||||
runTimeoutOverrideMs: prepared.context.runTimeoutOverrideMs,
|
||||
suppressExecNotifyOnExit: prepared.context.suppressExecNotifyOnExit,
|
||||
pluginRegistry: prepared.context.pluginRegistry,
|
||||
executionIdentity: params.executionIdentity,
|
||||
};
|
||||
const execution = await prepared.context.sessionWorkAdmission.run(() =>
|
||||
withAgentRunLifecycleGeneration(runLifecycleGeneration, () =>
|
||||
|
||||
@@ -38,7 +38,7 @@ import type {
|
||||
} from "./state.js";
|
||||
import { cronFailureNotificationEventContext, emit, isImmediateCronRunMode } from "./state.js";
|
||||
import { ensureLoaded, runPostPersistCronNotifications, warnIfDisabled } from "./store.js";
|
||||
import { tryCreateCronTaskRun, tryFinishCronTaskRun } from "./task-runs.js";
|
||||
import { tryCreateCronTaskRunHandle, tryFinishCronTaskRun } from "./task-runs.js";
|
||||
import { applyJobResult, armTimer, type CronTriggerEvalOutcome } from "./timer.js";
|
||||
|
||||
type PreparedManualRun =
|
||||
@@ -70,6 +70,8 @@ type PreparedManualRun =
|
||||
export type ActivatedManualRun = Extract<PreparedManualRun, { ran: true }> & {
|
||||
startedAt: number;
|
||||
taskRunId?: string;
|
||||
taskId?: string;
|
||||
flowId?: string;
|
||||
activeJobMarker?: CronActiveJobMarker;
|
||||
admittedJob: CronJob;
|
||||
executionJob: CronJob;
|
||||
@@ -503,12 +505,13 @@ export async function activatePreparedManualRun(
|
||||
job: activatedJob,
|
||||
runAtMs: startedAt,
|
||||
});
|
||||
const taskRunId = tryCreateCronTaskRun({
|
||||
const taskRun = tryCreateCronTaskRunHandle({
|
||||
state,
|
||||
job: activatedJob,
|
||||
startedAt,
|
||||
publicRunId: prepared.runId ?? activation.runReceipt.receiptId,
|
||||
});
|
||||
const taskRunId = taskRun?.runId;
|
||||
const activeJobMarker = markManualCronJobActive(state, job);
|
||||
// Execute against a snapshot so later reload/merge can preserve delivery
|
||||
// target writeback from disk without mutating the running object.
|
||||
@@ -527,6 +530,8 @@ export async function activatePreparedManualRun(
|
||||
startedAt,
|
||||
runId: prepared.runId ?? taskRunId,
|
||||
taskRunId,
|
||||
taskId: taskRun?.taskId,
|
||||
flowId: taskRun?.flowId,
|
||||
activeJobMarker,
|
||||
admittedJob,
|
||||
executionJob,
|
||||
|
||||
@@ -43,7 +43,10 @@ import type {
|
||||
} from "./state.js";
|
||||
import { emit, isImmediateCronRunMode } from "./state.js";
|
||||
import { ensureLoaded, publishCronRuntimeRows, runPostPersistCronNotifications } from "./store.js";
|
||||
import { tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
|
||||
import {
|
||||
createCronOwnerExecutionIdentityAdmission,
|
||||
tryFinishCronTaskRunWithoutHistory,
|
||||
} from "./task-runs.js";
|
||||
import { recordCronOutcomeForJob } from "./timer-outcome-events.js";
|
||||
import {
|
||||
resolveCronRunScheduleOwnership,
|
||||
@@ -161,6 +164,12 @@ async function finishPreparedManualRun(
|
||||
streamScheduleKey: prepared.streamScheduleKey,
|
||||
streamSourceIdentity: prepared.streamSourceIdentity,
|
||||
runReceipt: prepared.runReceipt,
|
||||
executionIdentity: createCronOwnerExecutionIdentityAdmission({
|
||||
state,
|
||||
runReceipt: prepared.runReceipt,
|
||||
taskId: prepared.taskId,
|
||||
flowId: prepared.flowId,
|
||||
}),
|
||||
});
|
||||
} catch (err) {
|
||||
if (err instanceof CronRunReceiptRevisionError && err.reason === "owner-unavailable") {
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
// Focused proof that manual cron admission binds exact owner-native rows.
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
createDueIsolatedJob,
|
||||
noopLogger,
|
||||
setupCronRegressionFixtures,
|
||||
} from "../../../test/helpers/cron/service-regression-fixtures.js";
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
|
||||
import { tableExists } from "../../state/openclaw-state-db-schema-helpers.js";
|
||||
import {
|
||||
openOpenClawStateDatabase,
|
||||
runOpenClawStateWriteTransaction,
|
||||
} from "../../state/openclaw-state-db.js";
|
||||
import { createManagedTaskFlow } from "../../tasks/task-flow-registry.js";
|
||||
import {
|
||||
resetTaskFlowRegistryForTests,
|
||||
resetTaskRegistryForTests,
|
||||
} from "../../tasks/task-runtime.test-helpers.js";
|
||||
import { withOpenClawTestState } from "../../test-utils/openclaw-test-state.js";
|
||||
import { saveCronStore } from "../store.js";
|
||||
import {
|
||||
claimCronRunReceiptInDatabase,
|
||||
finishCronRunReceipt,
|
||||
prepareCronRunReceiptClaim,
|
||||
} from "../store/run-receipt-store.js";
|
||||
import { run } from "./ops-run.js";
|
||||
import { createCronServiceState } from "./state.js";
|
||||
import {
|
||||
createCronOwnerExecutionIdentityAdmission,
|
||||
tryCreateCronTaskRunHandle,
|
||||
} from "./task-runs.js";
|
||||
|
||||
const fixtures = setupCronRegressionFixtures({
|
||||
prefix: "cron-service-execution-binding-",
|
||||
});
|
||||
|
||||
describe("cron run execution binding", () => {
|
||||
it("binds the exact admitted execution to the cron receipt and task rows", async () => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "state-only", prefix: "openclaw-cron-execution-binding-" },
|
||||
async () => {
|
||||
resetTaskRegistryForTests();
|
||||
resetTaskFlowRegistryForTests();
|
||||
const store = fixtures.makeStorePath();
|
||||
const dueAt = Date.parse("2026-02-06T10:05:06.525Z");
|
||||
const job = createDueIsolatedJob({
|
||||
id: "exact-owner-binding",
|
||||
nowMs: dueAt,
|
||||
nextRunAtMs: dueAt,
|
||||
});
|
||||
await saveCronStore(store.storePath, { version: 1, jobs: [job] });
|
||||
const runIsolatedAgentJob = vi.fn(
|
||||
async (params: {
|
||||
executionIdentity?: {
|
||||
ingress: { kind: string };
|
||||
onPostAdmission?: (context: AdmittedRunContext) => void;
|
||||
onExecutionStarted?: () => void;
|
||||
};
|
||||
}) => {
|
||||
const admitted = {
|
||||
operationalRunInstance: { instanceId: "instance-exact", runId: "run-exact" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-exact", {
|
||||
contextId: "context-exact",
|
||||
executionId: "execution-exact",
|
||||
now: dueAt,
|
||||
}),
|
||||
} satisfies AdmittedRunContext;
|
||||
const beforeAdmissionSettles = openOpenClawStateDatabase().db;
|
||||
expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe(
|
||||
false,
|
||||
);
|
||||
params.executionIdentity?.onPostAdmission?.(admitted);
|
||||
expect(tableExists(beforeAdmissionSettles, "execution_owner_lifecycle_bindings")).toBe(
|
||||
false,
|
||||
);
|
||||
params.executionIdentity?.onExecutionStarted?.();
|
||||
return { status: "ok" as const };
|
||||
},
|
||||
);
|
||||
const state = createCronServiceState({
|
||||
cronEnabled: true,
|
||||
storePath: store.storePath,
|
||||
log: noopLogger,
|
||||
nowMs: () => dueAt,
|
||||
enqueueSystemEvent: vi.fn(),
|
||||
requestHeartbeat: vi.fn(),
|
||||
runIsolatedAgentJob,
|
||||
});
|
||||
|
||||
await expect(run(state, job.id, "force")).resolves.toEqual({ ok: true, ran: true });
|
||||
expect(runIsolatedAgentJob).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
executionIdentity: expect.objectContaining({
|
||||
ingress: expect.objectContaining({ kind: "schedule" }),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
const db = openOpenClawStateDatabase().db;
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, receipt.status, receipt.error_text
|
||||
FROM cron_run_receipts AS receipt
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
|
||||
WHERE receipt.job_id = ?`,
|
||||
)
|
||||
.get(job.id),
|
||||
).toEqual({
|
||||
context_id: "context-exact",
|
||||
execution_id: "execution-exact",
|
||||
status: "ok",
|
||||
error_text: null,
|
||||
});
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, task.status
|
||||
FROM task_runs AS task
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
|
||||
WHERE task.source_id = ? AND task.runtime = 'cron'`,
|
||||
)
|
||||
.get(job.id),
|
||||
).toEqual({
|
||||
context_id: "context-exact",
|
||||
execution_id: "execution-exact",
|
||||
status: "succeeded",
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("does not partially bind task or flow rows after the cron owner is replaced", async () => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "state-only", prefix: "openclaw-cron-stale-execution-binding-" },
|
||||
async () => {
|
||||
resetTaskRegistryForTests();
|
||||
resetTaskFlowRegistryForTests();
|
||||
const store = fixtures.makeStorePath();
|
||||
const dueAt = Date.parse("2026-02-06T10:06:06.525Z");
|
||||
const job = {
|
||||
...createDueIsolatedJob({
|
||||
id: "stale-owner-binding",
|
||||
nowMs: dueAt,
|
||||
nextRunAtMs: dueAt,
|
||||
}),
|
||||
agentId: "main",
|
||||
};
|
||||
await saveCronStore(store.storePath, { version: 1, jobs: [job] });
|
||||
const state = createCronServiceState({
|
||||
cronEnabled: true,
|
||||
storePath: store.storePath,
|
||||
log: noopLogger,
|
||||
nowMs: () => dueAt,
|
||||
enqueueSystemEvent: vi.fn(),
|
||||
requestHeartbeat: vi.fn(),
|
||||
runIsolatedAgentJob: vi.fn(),
|
||||
});
|
||||
const prepared = prepareCronRunReceiptClaim({
|
||||
storePath: store.storePath,
|
||||
job,
|
||||
agentId: job.agentId!,
|
||||
startedAtMs: dueAt,
|
||||
});
|
||||
const initial = runOpenClawStateWriteTransaction(({ db }) =>
|
||||
claimCronRunReceiptInDatabase({
|
||||
database: db,
|
||||
prepared,
|
||||
resolveAgentId: (current) => current.agentId!,
|
||||
}),
|
||||
);
|
||||
const task = tryCreateCronTaskRunHandle({ state, job, startedAt: dueAt });
|
||||
expect(task).toBeDefined();
|
||||
const flow = createManagedTaskFlow({
|
||||
ownerKey: "agent:main:main",
|
||||
controllerId: "tests/stale-owner-binding",
|
||||
goal: "Reject partial stale-owner provenance",
|
||||
status: "running",
|
||||
});
|
||||
expect(flow).not.toBeNull();
|
||||
if (!task || !flow) {
|
||||
throw new Error("expected live task and flow owners");
|
||||
}
|
||||
const executionIdentity = createCronOwnerExecutionIdentityAdmission({
|
||||
state,
|
||||
runReceipt: initial,
|
||||
taskId: task.taskId,
|
||||
flowId: flow.flowId,
|
||||
});
|
||||
const admitted = {
|
||||
operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", {
|
||||
contextId: "context-stale",
|
||||
executionId: "execution-stale",
|
||||
now: dueAt,
|
||||
}),
|
||||
} satisfies AdmittedRunContext;
|
||||
executionIdentity.onPostAdmission?.(admitted);
|
||||
const db = openOpenClawStateDatabase().db;
|
||||
db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?").run(
|
||||
2_147_483_647,
|
||||
initial.receiptId,
|
||||
);
|
||||
const replacementPrepared = prepareCronRunReceiptClaim({
|
||||
storePath: store.storePath,
|
||||
job,
|
||||
agentId: job.agentId!,
|
||||
startedAtMs: dueAt + 1,
|
||||
});
|
||||
const replacement = runOpenClawStateWriteTransaction(({ db: transactionDb }) =>
|
||||
claimCronRunReceiptInDatabase({
|
||||
database: transactionDb,
|
||||
prepared: replacementPrepared,
|
||||
resolveAgentId: (current) => current.agentId!,
|
||||
}),
|
||||
);
|
||||
|
||||
executionIdentity.onExecutionStarted?.();
|
||||
expect(
|
||||
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
|
||||
).toBe(false);
|
||||
finishCronRunReceipt({
|
||||
handle: replacement,
|
||||
status: "skipped",
|
||||
finishedAtMs: dueAt + 2,
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -1109,7 +1109,7 @@ describe("cron service ops seam coverage", () => {
|
||||
});
|
||||
const taskRunId =
|
||||
reservationOffsetMs === undefined
|
||||
? taskRuns.tryCreateCronTaskRun({ state, job, startedAt })
|
||||
? taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId
|
||||
: taskExecutor.createRunningTaskRunCore({
|
||||
runtime: "cron",
|
||||
sourceId: job.id,
|
||||
@@ -1227,7 +1227,7 @@ describe("cron service ops seam coverage", () => {
|
||||
requestHeartbeat: vi.fn(),
|
||||
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
|
||||
});
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
|
||||
if (!taskRunId) {
|
||||
throw new Error("expected invalid finalized cron task run");
|
||||
}
|
||||
@@ -1296,7 +1296,7 @@ describe("cron service ops seam coverage", () => {
|
||||
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
|
||||
onEvent: (event) => events.push(structuredClone(event)),
|
||||
});
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
|
||||
if (!taskRunId) {
|
||||
throw new Error("expected cron task run");
|
||||
}
|
||||
@@ -1363,7 +1363,7 @@ describe("cron service ops seam coverage", () => {
|
||||
requestHeartbeat: vi.fn(),
|
||||
runIsolatedAgentJob: vi.fn(async () => ({ status: "ok" as const })),
|
||||
});
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job, startedAt });
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({ state, job, startedAt })?.runId;
|
||||
if (!taskRunId) {
|
||||
throw new Error("expected cron task run");
|
||||
}
|
||||
@@ -1437,7 +1437,11 @@ describe("cron service ops seam coverage", () => {
|
||||
requestHeartbeat: vi.fn(),
|
||||
runIsolatedAgentJob,
|
||||
});
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRun({ state, job: original, startedAt });
|
||||
const taskRunId = taskRuns.tryCreateCronTaskRunHandle({
|
||||
state,
|
||||
job: original,
|
||||
startedAt,
|
||||
})?.runId;
|
||||
if (!taskRunId) {
|
||||
throw new Error("expected cron task run");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js";
|
||||
import type { CronServiceState } from "./state.js";
|
||||
|
||||
export function resolveRunConcurrency(): number {
|
||||
return DEFAULT_CRON_MAX_CONCURRENT_RUNS;
|
||||
}
|
||||
|
||||
function acquireCronRunSlot(state: CronServiceState): () => void {
|
||||
state.runAdmission.active += 1;
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) {
|
||||
return;
|
||||
}
|
||||
released = true;
|
||||
state.runAdmission.active -= 1;
|
||||
dispatchWaiters(state);
|
||||
};
|
||||
}
|
||||
|
||||
function dispatchWaiters(state: CronServiceState): void {
|
||||
const admission = state.runAdmission;
|
||||
if (state.stopped) {
|
||||
cancelCronRunAdmissionWaiters(state);
|
||||
return;
|
||||
}
|
||||
const maxConcurrentRuns = resolveRunConcurrency();
|
||||
while (admission.active < maxConcurrentRuns) {
|
||||
const waiter = admission.waiters.shift();
|
||||
if (!waiter) {
|
||||
break;
|
||||
}
|
||||
waiter(acquireCronRunSlot(state));
|
||||
}
|
||||
if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) {
|
||||
const listener = admission.capacityListener;
|
||||
admission.capacityListener = null;
|
||||
if (listener) {
|
||||
queueMicrotask(listener);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire only the slots currently available to scheduled work. Unlike the
|
||||
* waiter-based path used by direct runs, this never retains a timer batch while
|
||||
* the pool is saturated.
|
||||
*/
|
||||
export function tryAcquireCronRunSlots(
|
||||
state: CronServiceState,
|
||||
requested: number,
|
||||
): Array<() => void> {
|
||||
if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) {
|
||||
return [];
|
||||
}
|
||||
const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active);
|
||||
return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state));
|
||||
}
|
||||
|
||||
/** Keep the first wake-up until capacity release consumes or cancellation clears it. */
|
||||
export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void {
|
||||
state.runAdmission.capacityListener ??= listener;
|
||||
}
|
||||
|
||||
async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> {
|
||||
const admission = state.runAdmission;
|
||||
if (state.stopped) {
|
||||
return null;
|
||||
}
|
||||
if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) {
|
||||
return acquireCronRunSlot(state);
|
||||
}
|
||||
return await new Promise<(() => void) | null>((resolve) => {
|
||||
admission.waiters.push(resolve);
|
||||
});
|
||||
}
|
||||
|
||||
/** Wake queued work on stop so each caller can release its durable reservation. */
|
||||
export function cancelCronRunAdmissionWaiters(state: CronServiceState): void {
|
||||
state.runAdmission.capacityListener = null;
|
||||
const waiters = state.runAdmission.waiters.splice(0);
|
||||
for (const waiter of waiters) {
|
||||
waiter(null);
|
||||
}
|
||||
}
|
||||
|
||||
/** Apply one service-level cap to every cron execution source. Queue waiters
|
||||
* keep their job reservation, then recheck scheduler state before execution.
|
||||
*/
|
||||
export async function runWithCronAdmission<T>(
|
||||
state: CronServiceState,
|
||||
execute: () => Promise<T>,
|
||||
acquiredRelease?: () => void,
|
||||
): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> {
|
||||
const release = acquiredRelease ?? (await acquireCronRunAdmission(state));
|
||||
if (!release) {
|
||||
return { kind: "stopped" };
|
||||
}
|
||||
try {
|
||||
return { kind: "admitted", value: await execute() };
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,3 @@
|
||||
import { DEFAULT_CRON_MAX_CONCURRENT_RUNS } from "../../config/cron-limits.js";
|
||||
import { markCronJobActive } from "../active-jobs.js";
|
||||
import { resolveCronJobConfigRevision } from "../config-revision.js";
|
||||
import { createCronRunDiagnosticsFromError } from "../run-diagnostics.js";
|
||||
@@ -17,6 +16,7 @@ import { normalizeCronRunErrorText } from "./execution-errors.js";
|
||||
import { enrollForeignReceipt } from "./foreign-receipt-monitor.js";
|
||||
import { recomputeJobNextRunAtMs } from "./jobs-scheduling.js";
|
||||
import { locked } from "./locked.js";
|
||||
import { runWithCronAdmission } from "./run-admission-capacity.js";
|
||||
import {
|
||||
activateServiceCronRunReceiptInDatabase,
|
||||
claimServiceCronRunReceiptInDatabase,
|
||||
@@ -27,7 +27,10 @@ import {
|
||||
import { applyCronRuntimeRowsToState, commitCronRuntimeRows } from "./runtime-store.js";
|
||||
import { type CronServiceState, type DeferredCronNotifications, emit } from "./state.js";
|
||||
import { ensureLoaded, runPostPersistCronNotifications } from "./store.js";
|
||||
import { tryCreateCronTaskRun } from "./task-runs.js";
|
||||
import {
|
||||
createCronOwnerExecutionIdentityAdmission,
|
||||
tryCreateCronTaskRunHandle,
|
||||
} from "./task-runs.js";
|
||||
import {
|
||||
runsDetachedFromMainSession,
|
||||
type TimedCronRunOutcome,
|
||||
@@ -35,88 +38,13 @@ import {
|
||||
import { authorCronRunCompletion, executeJobCoreWithTimeout } from "./timer-job-runner.js";
|
||||
import { isRunnableJob } from "./timer-runnable.js";
|
||||
|
||||
export function resolveRunConcurrency(): number {
|
||||
return DEFAULT_CRON_MAX_CONCURRENT_RUNS;
|
||||
}
|
||||
|
||||
function acquireCronRunSlot(state: CronServiceState): () => void {
|
||||
state.runAdmission.active += 1;
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) {
|
||||
return;
|
||||
}
|
||||
released = true;
|
||||
state.runAdmission.active -= 1;
|
||||
dispatchWaiters(state);
|
||||
};
|
||||
}
|
||||
|
||||
function dispatchWaiters(state: CronServiceState): void {
|
||||
const admission = state.runAdmission;
|
||||
if (state.stopped) {
|
||||
cancelCronRunAdmissionWaiters(state);
|
||||
return;
|
||||
}
|
||||
const maxConcurrentRuns = resolveRunConcurrency();
|
||||
while (admission.active < maxConcurrentRuns) {
|
||||
const waiter = admission.waiters.shift();
|
||||
if (!waiter) {
|
||||
break;
|
||||
}
|
||||
waiter(acquireCronRunSlot(state));
|
||||
}
|
||||
if (admission.active < maxConcurrentRuns && admission.waiters.length === 0) {
|
||||
const listener = admission.capacityListener;
|
||||
admission.capacityListener = null;
|
||||
if (listener) {
|
||||
queueMicrotask(listener);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire only the slots currently available to scheduled work. Unlike the
|
||||
* waiter-based path used by direct runs, this never retains a timer batch while
|
||||
* the pool is saturated.
|
||||
*/
|
||||
export function tryAcquireCronRunSlots(
|
||||
state: CronServiceState,
|
||||
requested: number,
|
||||
): Array<() => void> {
|
||||
if (state.stopped || requested <= 0 || state.runAdmission.waiters.length > 0) {
|
||||
return [];
|
||||
}
|
||||
const available = Math.max(0, resolveRunConcurrency() - state.runAdmission.active);
|
||||
return Array.from({ length: Math.min(requested, available) }, () => acquireCronRunSlot(state));
|
||||
}
|
||||
|
||||
/** Keep the first wake-up until capacity release consumes or cancellation clears it. */
|
||||
export function setCronRunCapacityListener(state: CronServiceState, listener: () => void): void {
|
||||
state.runAdmission.capacityListener ??= listener;
|
||||
}
|
||||
|
||||
async function acquireCronRunAdmission(state: CronServiceState): Promise<(() => void) | null> {
|
||||
const admission = state.runAdmission;
|
||||
if (state.stopped) {
|
||||
return null;
|
||||
}
|
||||
if (admission.waiters.length === 0 && admission.active < resolveRunConcurrency()) {
|
||||
return acquireCronRunSlot(state);
|
||||
}
|
||||
return await new Promise<(() => void) | null>((resolve) => {
|
||||
admission.waiters.push(resolve);
|
||||
});
|
||||
}
|
||||
|
||||
/** Wake queued work on stop so each caller can release its durable reservation. */
|
||||
export function cancelCronRunAdmissionWaiters(state: CronServiceState): void {
|
||||
state.runAdmission.capacityListener = null;
|
||||
const waiters = state.runAdmission.waiters.splice(0);
|
||||
for (const waiter of waiters) {
|
||||
waiter(null);
|
||||
}
|
||||
}
|
||||
export {
|
||||
cancelCronRunAdmissionWaiters,
|
||||
resolveRunConcurrency,
|
||||
runWithCronAdmission,
|
||||
setCronRunCapacityListener,
|
||||
tryAcquireCronRunSlots,
|
||||
} from "./run-admission-capacity.js";
|
||||
|
||||
/** Track a persisted marker through shared admission and payload execution. */
|
||||
export function reserveQueuedCronRun(
|
||||
@@ -543,25 +471,6 @@ export async function activateQueuedCronRun(params: {
|
||||
return { kind: "unavailable", reason: "stopped" };
|
||||
}
|
||||
|
||||
/** Apply one service-level cap to every cron execution source. Queue waiters
|
||||
* keep their job reservation, then recheck scheduler state before execution.
|
||||
*/
|
||||
export async function runWithCronAdmission<T>(
|
||||
state: CronServiceState,
|
||||
execute: () => Promise<T>,
|
||||
acquiredRelease?: () => void,
|
||||
): Promise<{ kind: "admitted"; value: T } | { kind: "stopped" }> {
|
||||
const release = acquiredRelease ?? (await acquireCronRunAdmission(state));
|
||||
if (!release) {
|
||||
return { kind: "stopped" };
|
||||
}
|
||||
try {
|
||||
return { kind: "admitted", value: await execute() };
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function executeQueuedCronRun(params: {
|
||||
state: CronServiceState;
|
||||
jobId: string;
|
||||
@@ -663,12 +572,13 @@ export async function executeQueuedCronRun(params: {
|
||||
const executionJob = structuredClone(started.job);
|
||||
executionJob.state.runningAtMs = started.startedAt;
|
||||
executionJob.state.lastError = undefined;
|
||||
const taskRunId = tryCreateCronTaskRun({
|
||||
const taskRun = tryCreateCronTaskRunHandle({
|
||||
state,
|
||||
job: executionJob,
|
||||
startedAt: started.startedAt,
|
||||
publicRunId: started.runReceipt.receiptId,
|
||||
});
|
||||
const taskRunId = taskRun?.runId;
|
||||
const activeJobMarker = markCronJobActive(executionJob.id, {
|
||||
preserveAcrossGenerationAdvance: !runsDetachedFromMainSession(executionJob),
|
||||
});
|
||||
@@ -694,6 +604,12 @@ export async function executeQueuedCronRun(params: {
|
||||
runId: taskRunId,
|
||||
activeJobMarker,
|
||||
runReceipt: started.runReceipt,
|
||||
executionIdentity: createCronOwnerExecutionIdentityAdmission({
|
||||
state,
|
||||
runReceipt: started.runReceipt,
|
||||
taskId: taskRun?.taskId,
|
||||
flowId: taskRun?.flowId,
|
||||
}),
|
||||
});
|
||||
const result = state.deps.runSchedulerOwned
|
||||
? await state.deps.runSchedulerOwned(execute)
|
||||
|
||||
@@ -15,7 +15,13 @@ import type { CronJob } from "../types.js";
|
||||
import { proposeCronRunRecovery, recoverCronRunProposal } from "./run-recovery.js";
|
||||
import { createCronServiceState } from "./state.js";
|
||||
import { runPostPersistCronNotifications } from "./store.js";
|
||||
import { tryCreateCronTaskRun, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
|
||||
import { tryCreateCronTaskRunHandle, tryFinishCronTaskRunWithoutHistory } from "./task-runs.js";
|
||||
|
||||
function tryCreateCronTaskRun(
|
||||
params: Parameters<typeof tryCreateCronTaskRunHandle>[0],
|
||||
): string | undefined {
|
||||
return tryCreateCronTaskRunHandle(params)?.runId;
|
||||
}
|
||||
|
||||
const { logger, makeStorePath } = setupCronServiceSuite({ prefix: "cron-run-recovery-" });
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
/** Cron service dependency, event, state, and public result types. */
|
||||
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import type { ExecutionIdentityAdmissionFacts } from "../../audit/execution-identity-admission.js";
|
||||
import type { ReplyPayload } from "../../auto-reply/reply-payload.js";
|
||||
import type { CronConfig } from "../../config/types.cron.js";
|
||||
import type { HeartbeatRunResult, HeartbeatWakeRequest } from "../../infra/heartbeat-wake.js";
|
||||
@@ -188,6 +190,7 @@ export type CronServiceDeps = {
|
||||
onExecutionStarted?: (info?: CronAgentExecutionStarted) => void;
|
||||
onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void;
|
||||
onLaneWait?: (info?: { waiting?: boolean }) => void;
|
||||
executionIdentity?: CronExecutionIdentityAdmission;
|
||||
}) => Promise<
|
||||
{
|
||||
summary?: string;
|
||||
@@ -267,6 +270,13 @@ export type CronServiceDeps = {
|
||||
onEvent?: (evt: CronEvent, context?: CronEventContext) => void;
|
||||
};
|
||||
|
||||
export type CronExecutionIdentityAdmission = {
|
||||
ingress: ExecutionIdentityAdmissionFacts["ingress"];
|
||||
invoker?: ExecutionIdentityAdmissionFacts["invoker"];
|
||||
onPostAdmission?: (context: AdmittedRunContext) => void;
|
||||
onExecutionStarted?: () => void;
|
||||
};
|
||||
|
||||
/** Cron deps after optional defaults have been made concrete. */
|
||||
type CronServiceDepsInternal = Omit<CronServiceDeps, "nowMs"> & {
|
||||
nowMs: () => number;
|
||||
|
||||
@@ -20,10 +20,16 @@ import { createCronServiceState as createCronServiceStateBase } from "./state.js
|
||||
import {
|
||||
getActiveCronTaskRunId,
|
||||
findCronTaskRunRecoveryInDatabase,
|
||||
tryCreateCronTaskRun,
|
||||
tryCreateCronTaskRunHandle,
|
||||
tryFinishCronTaskRun,
|
||||
tryFinishCronTaskRunWithoutHistory,
|
||||
} from "./task-runs.js";
|
||||
|
||||
function tryCreateCronTaskRun(
|
||||
params: Parameters<typeof tryCreateCronTaskRunHandle>[0],
|
||||
): string | undefined {
|
||||
return tryCreateCronTaskRunHandle(params)?.runId;
|
||||
}
|
||||
import { executeJobCoreWithTimeout } from "./timer-job-runner.js";
|
||||
|
||||
function createCronServiceState(
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
import { AsyncLocalStorage } from "node:async_hooks";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import {
|
||||
createExecutionStartedOwnerBinding,
|
||||
isRetainedExecutionOwnerBinding,
|
||||
} from "../../audit/execution-owner-binding.js";
|
||||
import { normalizeAgentId, resolveAgentIdFromSessionKey } from "../../routing/session-key.js";
|
||||
import { CRON_TASK_KIND } from "../../tasks/cron-task-contract.js";
|
||||
import {
|
||||
@@ -11,7 +15,11 @@ import {
|
||||
findTaskByRunId,
|
||||
recordTaskRunProgressByRunIdCore,
|
||||
} from "../../tasks/task-executor.js";
|
||||
import { listTaskRecordsByRuntimeSourceIdInDatabase } from "../../tasks/task-registry.store.sqlite.js";
|
||||
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
|
||||
import {
|
||||
bindTaskRunExecution,
|
||||
listTaskRecordsByRuntimeSourceIdInDatabase,
|
||||
} from "../../tasks/task-registry.store.sqlite.js";
|
||||
import type { JsonValue, TaskRecord, TaskStatus } from "../../tasks/task-registry.types.js";
|
||||
import {
|
||||
CRON_AGENT_SELECTION_REQUIRED_MESSAGE,
|
||||
@@ -20,6 +28,10 @@ import {
|
||||
import { createCronExecutionId } from "../run-id.js";
|
||||
import type { CronRunLogEntry } from "../run-log-types.js";
|
||||
import { cronStoreKey } from "../store/key.js";
|
||||
import {
|
||||
bindCronRunReceiptExecution,
|
||||
type CronRunReceiptHandle,
|
||||
} from "../store/run-receipt-store.js";
|
||||
import {
|
||||
cronRunLogEntryToTaskDetail,
|
||||
cronRunStatusToTaskStatus,
|
||||
@@ -38,7 +50,7 @@ import type {
|
||||
CronRunStatus,
|
||||
} from "../types.js";
|
||||
import { normalizeCronRunErrorText } from "./execution-errors.js";
|
||||
import type { CronEvent, CronServiceState } from "./state.js";
|
||||
import type { CronEvent, CronExecutionIdentityAdmission, CronServiceState } from "./state.js";
|
||||
import { CRON_TASK_RUNNING_PROGRESS_SUMMARY } from "./task-ledger.js";
|
||||
|
||||
function requireCronAgentId(agentId: string | undefined): string {
|
||||
@@ -64,6 +76,55 @@ export function getActiveCronTaskRunId(): string | undefined {
|
||||
return activeCronTaskRunId.getStore();
|
||||
}
|
||||
|
||||
/** Carries exact admission into the first post-admission owner lifecycle phase. */
|
||||
export function createCronOwnerExecutionIdentityAdmission(params: {
|
||||
state: CronServiceState;
|
||||
runReceipt: CronRunReceiptHandle;
|
||||
taskId?: string;
|
||||
flowId?: string;
|
||||
}): CronExecutionIdentityAdmission {
|
||||
const ownerBinding = createExecutionStartedOwnerBinding((admitted) => {
|
||||
try {
|
||||
const receiptResult = bindCronRunReceiptExecution({
|
||||
admitted,
|
||||
handle: params.runReceipt,
|
||||
});
|
||||
const taskResult = params.taskId
|
||||
? isRetainedExecutionOwnerBinding(receiptResult)
|
||||
? bindTaskRunExecution({ admitted, taskId: params.taskId })
|
||||
: receiptResult
|
||||
: undefined;
|
||||
const flowParentResult = params.taskId ? taskResult : receiptResult;
|
||||
const flowResult = params.flowId
|
||||
? isRetainedExecutionOwnerBinding(receiptResult) &&
|
||||
isRetainedExecutionOwnerBinding(flowParentResult)
|
||||
? bindTaskFlowExecution({ admitted, flowId: params.flowId })
|
||||
: flowParentResult
|
||||
: undefined;
|
||||
if (
|
||||
[receiptResult, taskResult, flowResult].some(
|
||||
(result) => result === "mismatch" || result === "missing",
|
||||
)
|
||||
) {
|
||||
params.state.deps.log.warn(
|
||||
{ receiptResult, taskResult, flowResult },
|
||||
"cron: exact execution identity binding was not retained",
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
params.state.deps.log.warn(
|
||||
{ error },
|
||||
"cron: failed to retain exact execution identity binding",
|
||||
);
|
||||
}
|
||||
});
|
||||
return {
|
||||
ingress: { kind: "schedule", boundary: "cron.isolated-agent", state: "present" },
|
||||
onPostAdmission: ownerBinding.onPostAdmission,
|
||||
onExecutionStarted: ownerBinding.onExecutionStarted,
|
||||
};
|
||||
}
|
||||
|
||||
/** Updates an active cron task with the exact transcript identity reported by its runner. */
|
||||
export function tryUpdateCronTaskRunSession(
|
||||
state: CronServiceState,
|
||||
@@ -88,13 +149,12 @@ export function tryUpdateCronTaskRunSession(
|
||||
}
|
||||
}
|
||||
|
||||
/** Creates a best-effort detached task row keyed to the persisted execution start. */
|
||||
export function tryCreateCronTaskRun(params: {
|
||||
export function tryCreateCronTaskRunHandle(params: {
|
||||
state: CronServiceState;
|
||||
job: CronJob;
|
||||
startedAt: number;
|
||||
publicRunId?: string;
|
||||
}): string | undefined {
|
||||
}): { runId: string; taskId: string; flowId?: string } | undefined {
|
||||
const runId = createCronTaskRunId(params.job.id, params.startedAt, params.publicRunId);
|
||||
return tryCreateCronTaskRunRecord({
|
||||
state: params.state,
|
||||
@@ -214,7 +274,7 @@ function tryCreateCronTaskRunRecord(params: {
|
||||
startedAt: number;
|
||||
runId: string;
|
||||
childSessionKey?: string;
|
||||
}): string | undefined {
|
||||
}): { runId: string; taskId: string; flowId?: string } | undefined {
|
||||
try {
|
||||
const childSessionKey = params.childSessionKey;
|
||||
const effectiveJobAgentId = params.job
|
||||
@@ -252,7 +312,11 @@ function tryCreateCronTaskRunRecord(params: {
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return params.runId;
|
||||
return {
|
||||
runId: params.runId,
|
||||
taskId: task.taskId,
|
||||
...(task.parentFlowId ? { flowId: task.parentFlowId } : {}),
|
||||
};
|
||||
} catch (error) {
|
||||
params.state.deps.log.warn(
|
||||
{ jobId: params.jobId, error },
|
||||
@@ -341,9 +405,9 @@ export function tryFinishCronTaskRun(
|
||||
result.taskRunId ?? createCronTaskRunId(entry.jobId, startedAt, entry.runId);
|
||||
try {
|
||||
const existingCandidate = findTaskByRunId(candidateRunId);
|
||||
const taskRunId =
|
||||
const created =
|
||||
existingCandidate?.runtime === "cron"
|
||||
? candidateRunId
|
||||
? undefined
|
||||
: tryCreateCronTaskRunRecord({
|
||||
state,
|
||||
job: result.job ?? result.event.job,
|
||||
@@ -352,6 +416,7 @@ export function tryFinishCronTaskRun(
|
||||
runId: candidateRunId,
|
||||
childSessionKey: entry.sessionKey,
|
||||
});
|
||||
const taskRunId = existingCandidate?.runtime === "cron" ? candidateRunId : created?.runId;
|
||||
if (!taskRunId) {
|
||||
return;
|
||||
}
|
||||
@@ -414,7 +479,7 @@ export function tryFinishCronTaskRun(
|
||||
updated = finalize(taskRunId);
|
||||
} else {
|
||||
// A terminal event still owns one durable row if its active mirror vanished.
|
||||
const recreatedRunId = tryCreateCronTaskRunRecord({
|
||||
const recreated = tryCreateCronTaskRunRecord({
|
||||
state,
|
||||
job: result.job ?? result.event.job,
|
||||
jobId: entry.jobId,
|
||||
@@ -422,8 +487,8 @@ export function tryFinishCronTaskRun(
|
||||
runId: taskRunId,
|
||||
childSessionKey: entry.sessionKey,
|
||||
});
|
||||
if (recreatedRunId) {
|
||||
updated = finalize(recreatedRunId);
|
||||
if (recreated) {
|
||||
updated = finalize(recreated.runId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -125,6 +125,7 @@ export type ExecuteJobCoreOptions = {
|
||||
onExecutionStarted?: (info?: CronAgentExecutionStarted) => void;
|
||||
onExecutionPhase?: (info: CronAgentExecutionPhaseUpdate) => void;
|
||||
onLaneWait?: (info?: { waiting?: boolean }) => void;
|
||||
executionIdentity?: import("./state.js").CronExecutionIdentityAdmission;
|
||||
/** Revalidates the durable run fence after awaited planning and before effects. */
|
||||
assertRunCurrent?: () => void;
|
||||
streamBatch?: string;
|
||||
|
||||
@@ -459,6 +459,7 @@ async function executeDetachedCronJob(
|
||||
onExecutionStarted: options?.onExecutionStarted,
|
||||
onExecutionPhase: options?.onExecutionPhase,
|
||||
onLaneWait: options?.onLaneWait,
|
||||
executionIdentity: options?.executionIdentity,
|
||||
});
|
||||
|
||||
if (abortSignal?.aborted) {
|
||||
|
||||
@@ -51,6 +51,7 @@ type CronCoreRunOptions = {
|
||||
streamScheduleKey?: string;
|
||||
streamSourceIdentity?: string;
|
||||
runReceipt?: import("../store/run-receipt-store.js").CronRunReceiptHandle;
|
||||
executionIdentity?: import("./state.js").CronExecutionIdentityAdmission;
|
||||
};
|
||||
|
||||
async function deliverPrimaryWebhook(
|
||||
@@ -274,6 +275,7 @@ async function executeJobCoreWithTimeoutUnfinalized(
|
||||
onExecutionStarted: noteExecutionStarted,
|
||||
onExecutionPhase: accumulateExecution,
|
||||
assertRunCurrent,
|
||||
executionIdentity: opts?.executionIdentity,
|
||||
};
|
||||
const corePromise = withCronTaskRunId(opts?.runId, () =>
|
||||
executeJobCore(state, job, runAbortController.signal, coreOptions),
|
||||
@@ -361,6 +363,7 @@ async function executeJobCoreWithTimeoutUnfinalized(
|
||||
onExecutionPhase: deferTimeoutUntilExecutionStart ? watchdog.notePhase : undefined,
|
||||
onLaneWait: deferTimeoutUntilExecutionStart ? noteLaneState : undefined,
|
||||
assertRunCurrent,
|
||||
executionIdentity: opts?.executionIdentity,
|
||||
};
|
||||
const corePromise = withCronTaskRunId(opts?.runId, () =>
|
||||
executeJobCore(state, job, runAbortController.signal, coreOptions),
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "../../audit/execution-identity-admission.js";
|
||||
import {
|
||||
openOpenClawStateDatabase,
|
||||
runOpenClawStateWriteTransaction,
|
||||
@@ -9,6 +11,7 @@ import type { CronJob } from "../types.js";
|
||||
import { cronStoreKey } from "./key.js";
|
||||
import {
|
||||
assertCronRunReceiptCurrent,
|
||||
bindCronRunReceiptExecution,
|
||||
claimCronRunReceiptInDatabase,
|
||||
CronRunReceiptConflictError,
|
||||
CronRunReceiptRevisionError,
|
||||
@@ -127,14 +130,56 @@ describe("cron run receipt store", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects a delayed binding after a successor replaces its exact owner", async () => {
|
||||
const { storePath } = await makeStorePath();
|
||||
const job = makeJob("stale-binding");
|
||||
await saveCronStore(storePath, { version: 1, jobs: [job] });
|
||||
const abandoned = claim(storePath, job, 230);
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare("UPDATE cron_run_receipts SET owner_pid = ? WHERE receipt_id = ?")
|
||||
.run(2_147_483_647, abandoned.receiptId);
|
||||
const replacement = claim(storePath, job, 240);
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-stale", runId: "run-stale" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-stale", {
|
||||
contextId: "context-stale",
|
||||
executionId: "execution-stale",
|
||||
}),
|
||||
};
|
||||
|
||||
expect(bindCronRunReceiptExecution({ admitted, handle: abandoned })).toBe("missing");
|
||||
expect(bindCronRunReceiptExecution({ admitted, handle: replacement })).toBe("bound");
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT owner_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'cron'`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([{ owner_id: replacement.receiptId }]);
|
||||
|
||||
finishCronRunReceipt({ handle: replacement, status: "ok", finishedAtMs: 250 });
|
||||
});
|
||||
|
||||
it("prunes old terminal receipts while preserving the active and 64 newest rows", async () => {
|
||||
const { storePath } = await makeStorePath();
|
||||
const job = makeJob("retention");
|
||||
await saveCronStore(storePath, { version: 1, jobs: [job] });
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-retention", runId: "run-retention" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-retention", {
|
||||
contextId: "context-retention",
|
||||
executionId: "execution-retention",
|
||||
}),
|
||||
};
|
||||
const finishedReceiptIds: string[] = [];
|
||||
for (let index = 0; index < 70; index += 1) {
|
||||
const handle = claim(storePath, job, 1_000 + index * 2);
|
||||
finishedReceiptIds.push(handle.receiptId);
|
||||
if (index === 0 || index === 69) {
|
||||
expect(bindCronRunReceiptExecution({ admitted, handle })).toBe("bound");
|
||||
}
|
||||
finishCronRunReceipt({
|
||||
handle,
|
||||
status: "ok",
|
||||
@@ -142,6 +187,7 @@ describe("cron run receipt store", () => {
|
||||
});
|
||||
}
|
||||
const active = claim(storePath, job, 2_000);
|
||||
expect(bindCronRunReceiptExecution({ admitted, handle: active })).toBe("bound");
|
||||
|
||||
const retained = receipts(storePath, job.id);
|
||||
const retainedIds = new Set(retained.map((receipt) => receipt.receiptId));
|
||||
@@ -158,6 +204,21 @@ describe("cron run receipt store", () => {
|
||||
for (const receiptId of finishedReceiptIds.slice(-64)) {
|
||||
expect(retainedIds.has(receiptId)).toBe(true);
|
||||
}
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT binding.owner_id
|
||||
FROM execution_owner_lifecycle_bindings AS binding
|
||||
JOIN cron_run_receipts AS receipt ON receipt.receipt_id = binding.owner_id
|
||||
WHERE binding.owner_kind = 'cron' AND receipt.job_id = ?
|
||||
ORDER BY owner_id`,
|
||||
)
|
||||
.all(job.id),
|
||||
).toEqual(
|
||||
[active.receiptId, finishedReceiptIds.at(-1)]
|
||||
.toSorted((left, right) => (left ?? "").localeCompare(right ?? ""))
|
||||
.map((owner_id) => ({ owner_id })),
|
||||
);
|
||||
|
||||
finishCronRunReceipt({ handle: active, status: "skipped", finishedAtMs: 2_001 });
|
||||
});
|
||||
|
||||
@@ -1,6 +1,15 @@
|
||||
import crypto from "node:crypto";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import type { Selectable } from "kysely";
|
||||
import type { AdmittedRunContext } from "../../agents/admitted-run-context.js";
|
||||
import {
|
||||
executionOwnerBindingFromAdmission,
|
||||
type ExecutionOwnerBindingResult,
|
||||
} from "../../audit/execution-owner-binding.js";
|
||||
import {
|
||||
bindExecutionOwnerLifecycleMetadata,
|
||||
deleteExecutionOwnerLifecycleMetadata,
|
||||
} from "../../audit/execution-owner-lifecycle-binding-store.js";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
executeSqliteQueryTakeFirstSync,
|
||||
@@ -176,6 +185,38 @@ function withReceiptWrite<T>(
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Binds the exact admitted execution to its authoritative receipt without changing lifecycle. */
|
||||
export function bindCronRunReceiptExecution(params: {
|
||||
admitted: AdmittedRunContext;
|
||||
handle: CronRunReceiptHandle;
|
||||
options?: OpenClawStateDatabaseOptions;
|
||||
}): ExecutionOwnerBindingResult {
|
||||
const binding = executionOwnerBindingFromAdmission(params.admitted);
|
||||
if (!binding) {
|
||||
return "disabled";
|
||||
}
|
||||
return withReceiptWrite(
|
||||
"cron.run-receipt.execution-binding",
|
||||
params.options ?? {},
|
||||
(database) => {
|
||||
try {
|
||||
assertCronRunReceiptOwnedInDatabase({ database, handle: params.handle });
|
||||
} catch (error) {
|
||||
if (!(error instanceof CronRunReceiptRevisionError)) {
|
||||
throw error;
|
||||
}
|
||||
return "missing";
|
||||
}
|
||||
return bindExecutionOwnerLifecycleMetadata({
|
||||
db: database,
|
||||
ownerKind: "cron",
|
||||
ownerId: params.handle.receiptId,
|
||||
binding,
|
||||
});
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
function isReceiptStatus(value: string): value is CronRunReceiptStatus {
|
||||
return (
|
||||
value === "running" ||
|
||||
@@ -329,6 +370,11 @@ function pruneTerminalReceipts(database: DatabaseSync, storeKey: string, jobId:
|
||||
const receiptIds = terminalIds
|
||||
.slice(index, index + CRON_RUN_RECEIPT_DELETE_BATCH_SIZE)
|
||||
.map((row) => row.receipt_id);
|
||||
deleteExecutionOwnerLifecycleMetadata({
|
||||
db: database,
|
||||
ownerKind: "cron",
|
||||
ownerIds: receiptIds,
|
||||
});
|
||||
executeSqliteQuerySync(
|
||||
database,
|
||||
query(database)
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
type AgentRunTerminalOutcome,
|
||||
} from "../../agents/agent-run-terminal-outcome.js";
|
||||
import type { PreparedAgentCommandRuntimeContext } from "../../agents/command/prepare.js";
|
||||
import type { AgentCommandOpts } from "../../agents/command/types.js";
|
||||
import {
|
||||
createCronCreatorAuthorityCapability,
|
||||
runWithCronCreatorAuthorityCapability,
|
||||
@@ -15,6 +16,10 @@ import type { MainSessionRecoveryPendingTarget } from "../../agents/main-session
|
||||
import { isAgentRunRestartAbortReason } from "../../agents/run-termination.js";
|
||||
import { normalizeAgentRunTimeoutPhase } from "../../agents/run-timeout-attribution.js";
|
||||
import { runWithCanonicalSkillWorkspace } from "../../agents/skill-workshop-workspace-context.js";
|
||||
import {
|
||||
createExecutionStartedOwnerBinding,
|
||||
isRetainedExecutionOwnerBinding,
|
||||
} from "../../audit/execution-owner-binding.js";
|
||||
import { readAgentRunTerminalOutcome } from "../../channels/turn/agent-run-terminal-outcome.js";
|
||||
import { agentCommandFromGatewayIngress } from "../../commands/agent.js";
|
||||
import { isAbortError } from "../../infra/abort-signal.js";
|
||||
@@ -22,7 +27,10 @@ import { clearAgentRunContext } from "../../infra/agent-run-registry.js";
|
||||
import { formatErrorMessageWithCode, readErrorName } from "../../infra/errors.js";
|
||||
import { defaultRuntime } from "../../runtime.js";
|
||||
import { createRunningTaskRun } from "../../tasks/detached-task-runtime.js";
|
||||
import { bindTaskFlowExecution } from "../../tasks/task-flow-registry.store.sqlite.js";
|
||||
import { mapAgentRunTerminalOutcomeToTaskStatus } from "../../tasks/task-registry-common.js";
|
||||
import { bindTaskRunExecution } from "../../tasks/task-registry.store.sqlite.js";
|
||||
import type { TaskRecord } from "../../tasks/task-registry.types.js";
|
||||
import { normalizeDeliveryContext } from "../../utils/delivery-context.shared.js";
|
||||
import type { ChatAbortControllerEntry } from "../chat-abort.js";
|
||||
import {
|
||||
@@ -136,9 +144,10 @@ export function dispatchAgentRunFromGateway(params: {
|
||||
}) {
|
||||
const shouldTrackTask = params.taskTrackingMode === "cli";
|
||||
let taskTracked = false;
|
||||
let trackedTask: TaskRecord | undefined;
|
||||
if (shouldTrackTask) {
|
||||
try {
|
||||
taskTracked = Boolean(
|
||||
trackedTask =
|
||||
createRunningTaskRun({
|
||||
runtime: "cli",
|
||||
sourceId: params.runId,
|
||||
@@ -155,8 +164,8 @@ export function dispatchAgentRunFromGateway(params: {
|
||||
task: params.ingressOpts.message,
|
||||
deliveryStatus: "not_applicable",
|
||||
startedAt: Date.now(),
|
||||
}),
|
||||
);
|
||||
}) ?? undefined;
|
||||
taskTracked = Boolean(trackedTask);
|
||||
} catch (err) {
|
||||
// Best-effort only: background task tracking must not block agent runs.
|
||||
// Still surface the swallowed error so non-transient tracking failures stay observable.
|
||||
@@ -188,12 +197,49 @@ export function dispatchAgentRunFromGateway(params: {
|
||||
params.ingressOpts,
|
||||
readAgentRunDispatchExecutionIdentity(params),
|
||||
);
|
||||
const trackedTaskBinding = trackedTask
|
||||
? createExecutionStartedOwnerBinding(
|
||||
(admitted: Parameters<NonNullable<AgentCommandOpts["onPostAdmittedRunContext"]>>[0]) => {
|
||||
try {
|
||||
const taskResult = bindTaskRunExecution({ admitted, taskId: trackedTask.taskId });
|
||||
const flowResult = trackedTask.parentFlowId
|
||||
? isRetainedExecutionOwnerBinding(taskResult)
|
||||
? bindTaskFlowExecution({ admitted, flowId: trackedTask.parentFlowId })
|
||||
: taskResult
|
||||
: undefined;
|
||||
if (
|
||||
[taskResult, flowResult].some(
|
||||
(result) => result === "mismatch" || result === "missing",
|
||||
)
|
||||
) {
|
||||
params.context.logGateway.warn(
|
||||
`exact tracked-task execution binding was not retained for ${params.runId}`,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
params.context.logGateway.warn(
|
||||
`failed to retain tracked-task execution binding ${params.runId}: ${formatForLog(error)}`,
|
||||
);
|
||||
}
|
||||
},
|
||||
)
|
||||
: undefined;
|
||||
const ingressOptsWithTaskBinding = trackedTask
|
||||
? {
|
||||
...ingressOptsWithSpawnFacts,
|
||||
onPostAdmittedRunContext: trackedTaskBinding?.onPostAdmission,
|
||||
onExecutionStarted: () => {
|
||||
ingressOptsWithSpawnFacts.onExecutionStarted?.();
|
||||
trackedTaskBinding?.onExecutionStarted();
|
||||
},
|
||||
}
|
||||
: ingressOptsWithSpawnFacts;
|
||||
const runAgent = () =>
|
||||
runWithCanonicalSkillWorkspace(params.canonicalSkillWorkspaceDir, () =>
|
||||
agentCommandFromGatewayIngress(
|
||||
cronCreatorAuthorityCapability
|
||||
? { ...ingressOptsWithSpawnFacts, cronCreatorAuthorityCapability }
|
||||
: ingressOptsWithSpawnFacts,
|
||||
? { ...ingressOptsWithTaskBinding, cronCreatorAuthorityCapability }
|
||||
: ingressOptsWithTaskBinding,
|
||||
defaultRuntime,
|
||||
params.context.deps,
|
||||
{
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import { ErrorCodes, errorShape } from "../../../packages/gateway-protocol/src/index.js";
|
||||
import { getAdmittedRunDelegatedAuthority } from "../../agents/admitted-run-context.js";
|
||||
import { attachAgentCommandAdmissionFacts } from "../../agents/agent-command-admission-facts.js";
|
||||
import {
|
||||
attachAgentCommandAdmissionFacts,
|
||||
attachAgentCommandRecoveryAdmissionFacts,
|
||||
} from "../../agents/agent-command-admission-facts.js";
|
||||
import type { AgentRunTerminalOutcome } from "../../agents/agent-run-terminal-outcome.js";
|
||||
import { prepareGitCoauthorAttribution } from "../../agents/git-coauthor-attribution.js";
|
||||
import { repairMainSessionRecoveryMutation } from "../../agents/main-session-recovery/main-session-recovery-lifecycle.js";
|
||||
@@ -265,7 +268,9 @@ export function startAgentRunExecution(params: {
|
||||
);
|
||||
|
||||
const localUserIngress = getGatewayLocalUserIngress(params.client);
|
||||
if (localUserIngress) {
|
||||
if (params.isRestartRecoveryResumeRun) {
|
||||
attachAgentCommandRecoveryAdmissionFacts(runContext);
|
||||
} else if (localUserIngress) {
|
||||
attachAgentCommandAdmissionFacts(runContext, localUserIngress.facts);
|
||||
}
|
||||
finalizePreparedAgentRunUserTurn(prepared.userTurn);
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
// Hook mapping identity coverage keeps mapping IDs as ingress attribution only.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { applyHookMappings, resolveHookMappings } from "./hooks-mapping.js";
|
||||
|
||||
describe("hook mapping execution identity", () => {
|
||||
it("carries the configured mapping id on the resolved agent action", async () => {
|
||||
const mappings = resolveHookMappings({
|
||||
mappings: [
|
||||
{
|
||||
id: "source-gmail",
|
||||
match: { path: "gmail" },
|
||||
action: "agent",
|
||||
messageTemplate: "{{messages[0].subject}}",
|
||||
},
|
||||
],
|
||||
});
|
||||
const result = await applyHookMappings(mappings, {
|
||||
payload: { messages: [{ subject: "Hello" }] },
|
||||
headers: {},
|
||||
url: new URL("http://127.0.0.1:18789/hooks/gmail"),
|
||||
path: "gmail",
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
action: { kind: "agent", mappingId: "source-gmail", message: "Hello" },
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -49,6 +49,7 @@ type HookMappingContext = {
|
||||
type HookAction =
|
||||
| {
|
||||
kind: "wake";
|
||||
mappingId: string;
|
||||
text: string;
|
||||
mode: "now" | "next-heartbeat";
|
||||
agentId?: string;
|
||||
@@ -57,6 +58,7 @@ type HookAction =
|
||||
}
|
||||
| {
|
||||
kind: "agent";
|
||||
mappingId: string;
|
||||
message: string;
|
||||
name?: string;
|
||||
agentId?: string;
|
||||
@@ -272,6 +274,7 @@ function buildActionFromMapping(
|
||||
ok: true,
|
||||
action: {
|
||||
kind: "wake",
|
||||
mappingId: mapping.id,
|
||||
text,
|
||||
mode: mapping.wakeMode ?? "now",
|
||||
agentId: mapping.agentId,
|
||||
@@ -285,6 +288,7 @@ function buildActionFromMapping(
|
||||
ok: true,
|
||||
action: {
|
||||
kind: "agent",
|
||||
mappingId: mapping.id,
|
||||
message,
|
||||
name: renderOptional(mapping.name, ctx),
|
||||
agentId: mapping.agentId,
|
||||
@@ -318,6 +322,7 @@ function mergeAction(
|
||||
const mode = override.mode === "next-heartbeat" ? "next-heartbeat" : (baseWake?.mode ?? "now");
|
||||
return validateAction({
|
||||
kind: "wake",
|
||||
mappingId: base.mappingId,
|
||||
text,
|
||||
mode,
|
||||
agentId: override.agentId ?? baseWake?.agentId,
|
||||
@@ -332,6 +337,7 @@ function mergeAction(
|
||||
override.wakeMode === "next-heartbeat" ? "next-heartbeat" : (baseAgent?.wakeMode ?? "now");
|
||||
return validateAction({
|
||||
kind: "agent",
|
||||
mappingId: base.mappingId,
|
||||
message,
|
||||
wakeMode,
|
||||
name: override.name ?? baseAgent?.name,
|
||||
|
||||
@@ -306,6 +306,8 @@ export type HookAgentDispatchPayload = Omit<HookAgentPayload, "sessionKey"> & {
|
||||
sourcePath: string;
|
||||
allowUnsafeExternalContent?: boolean;
|
||||
externalContentSource?: HookExternalContentSource;
|
||||
/** Configured ingress source attribution; never an authenticated principal. */
|
||||
mappingId?: string;
|
||||
};
|
||||
|
||||
const listHookChannelValues = () => ["last", ...listChannelPlugins().map((plugin) => plugin.id)];
|
||||
|
||||
@@ -850,6 +850,7 @@ export function buildGatewayCronService(params: {
|
||||
onExecutionStarted,
|
||||
onExecutionPhase,
|
||||
onLaneWait,
|
||||
executionIdentity,
|
||||
}) => {
|
||||
const { agentId, cfg: runtimeConfig } = resolveCronAgent(job.agentId);
|
||||
const sessionKey = resolveCronSessionTargetSessionKey(job.sessionTarget) ?? `cron:${job.id}`;
|
||||
@@ -862,6 +863,7 @@ export function buildGatewayCronService(params: {
|
||||
onExecutionStarted,
|
||||
onExecutionPhase,
|
||||
onLaneWait,
|
||||
executionIdentity,
|
||||
agentId,
|
||||
sessionKey,
|
||||
lane: "cron",
|
||||
|
||||
@@ -314,34 +314,6 @@ describe("audit gateway methods", () => {
|
||||
decisionLimit: 20,
|
||||
});
|
||||
|
||||
await runAuditHandler("audit.run.inspect", {
|
||||
runId: "run-1",
|
||||
executionCursor: "1",
|
||||
decisionCursor: "1",
|
||||
decisionLimit: 25,
|
||||
});
|
||||
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
|
||||
runId: "run-1",
|
||||
executionOffset: 1,
|
||||
executionLimit: 50,
|
||||
decisionCursor: "1",
|
||||
decisionLimit: 25,
|
||||
});
|
||||
|
||||
await runAuditHandler("audit.run.inspect", {
|
||||
runId: "run-1",
|
||||
executionCursor: "001",
|
||||
decisionCursor: "001",
|
||||
decisionLimit: 25,
|
||||
});
|
||||
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
|
||||
runId: "run-1",
|
||||
executionOffset: 1,
|
||||
executionLimit: 50,
|
||||
decisionCursor: "001",
|
||||
decisionLimit: 25,
|
||||
});
|
||||
|
||||
await runAuditHandler("audit.run.inspect", {
|
||||
executionId: "execution-1",
|
||||
decisionCursor: "1",
|
||||
@@ -503,6 +475,45 @@ describe("audit gateway methods", () => {
|
||||
expect(JSON.stringify(result)).not.toContain('"decisions"');
|
||||
});
|
||||
|
||||
it.each(["1", "001"])(
|
||||
"preserves mirrored numeric cursor %s for execution paging",
|
||||
async (decisionCursor) => {
|
||||
await runAuditHandler("audit.run.inspect", {
|
||||
runId: "run-1",
|
||||
executionCursor: decisionCursor,
|
||||
decisionCursor,
|
||||
decisionLimit: 25,
|
||||
});
|
||||
|
||||
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
|
||||
runId: "run-1",
|
||||
executionOffset: 1,
|
||||
executionLimit: 50,
|
||||
decisionCursor,
|
||||
decisionLimit: 25,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["a:2000:42", "m:2000:42", "g:2000:42", "c:2000:42", "t:2000:42", "f:2000:42"])(
|
||||
"treats mirrored owner decision cursor %s as decision-only",
|
||||
async (decisionCursor) => {
|
||||
await runAuditHandler("audit.run.inspect", {
|
||||
runId: "run-1",
|
||||
executionCursor: decisionCursor,
|
||||
decisionCursor,
|
||||
decisionLimit: 25,
|
||||
});
|
||||
|
||||
expect(inspectExecutionIdentityRun).toHaveBeenLastCalledWith({
|
||||
runId: "run-1",
|
||||
executionLimit: 50,
|
||||
decisionCursor,
|
||||
decisionLimit: 25,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects malformed run inspection before storage access", async () => {
|
||||
expect(
|
||||
await runAuditHandler("audit.run.inspect", { runId: "", extra: true }),
|
||||
|
||||
@@ -49,6 +49,10 @@ function serializeAuditRunInspectResult(
|
||||
return result;
|
||||
}
|
||||
|
||||
function isOwnerDecisionCursor(value: string): boolean {
|
||||
return parsePositiveAuditCursor(value) === null && isExecutionDecisionCursor(value);
|
||||
}
|
||||
|
||||
/** Preserve the shipped audit.list result shape for run/tool-only clients. */
|
||||
function mapLegacyAuditEvent(
|
||||
event: AgentRunAuditEventRecord | ToolActionAuditEventRecord,
|
||||
@@ -198,9 +202,7 @@ export const auditHandlers: GatewayRequestHandlers = {
|
||||
typeof params.runId !== "string" ||
|
||||
(params.executionCursor === decisionCursor &&
|
||||
decisionCursor !== undefined &&
|
||||
(decisionCursor.startsWith("a:") ||
|
||||
decisionCursor.startsWith("m:") ||
|
||||
decisionCursor.startsWith("g:")))
|
||||
isOwnerDecisionCursor(decisionCursor))
|
||||
? undefined
|
||||
: parsePositiveAuditCursor(params.executionCursor);
|
||||
if (
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
/** Focused HTTP coverage for hook admission feedback and pending replay behavior. */
|
||||
import fs from "node:fs/promises";
|
||||
import { Agent, request as httpRequest } from "node:http";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, test, vi } from "vitest";
|
||||
import { createDeferred } from "../../test/helpers/promise.js";
|
||||
import { resolveMainSessionKeyFromConfig } from "../config/sessions.js";
|
||||
import { DEFAULT_WEBHOOK_MAX_BODY_BYTES } from "../infra/http-body.js";
|
||||
import { drainSystemEvents } from "../infra/system-events.js";
|
||||
import { drainSystemEvents, peekSystemEventEntries } from "../infra/system-events.js";
|
||||
import {
|
||||
cronIsolatedRun,
|
||||
installGatewayTestHooks,
|
||||
@@ -25,11 +27,11 @@ afterEach(() => {
|
||||
|
||||
async function postHook(
|
||||
port: number,
|
||||
path: string,
|
||||
hookPath: string,
|
||||
body: Record<string, unknown>,
|
||||
idempotencyKey: string,
|
||||
): Promise<Response> {
|
||||
return await fetch(`http://127.0.0.1:${port}${path}`, {
|
||||
return await fetch(`http://127.0.0.1:${port}${hookPath}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${HOOK_TOKEN}`,
|
||||
@@ -106,6 +108,24 @@ async function postOversizedChunkedHook(port: number): Promise<{
|
||||
}
|
||||
}
|
||||
|
||||
async function writeHookTransformModule(moduleName: string, source: string): Promise<void> {
|
||||
const configPath = process.env.OPENCLAW_CONFIG_PATH;
|
||||
if (!configPath) {
|
||||
throw new Error("expected OPENCLAW_CONFIG_PATH");
|
||||
}
|
||||
const transformsDir = path.join(path.dirname(configPath), "hooks", "transforms");
|
||||
await fs.mkdir(transformsDir, { recursive: true });
|
||||
await fs.writeFile(path.join(transformsDir, moduleName), source, "utf8");
|
||||
}
|
||||
|
||||
function readExecutionIdentityCall(index: number): unknown {
|
||||
const call = cronIsolatedRun.mock.calls[index]?.[0];
|
||||
if (!call || typeof call !== "object" || !("executionIdentity" in call)) {
|
||||
return undefined;
|
||||
}
|
||||
return call.executionIdentity;
|
||||
}
|
||||
|
||||
describe("gateway hook admission", () => {
|
||||
test("flushes an oversized chunked hook response before closing the socket", async () => {
|
||||
testState.hooksConfig = { enabled: true, token: HOOK_TOKEN };
|
||||
@@ -144,6 +164,75 @@ describe("gateway hook admission", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("keeps direct hooks unattributed and mapped IDs as ingress attribution", async () => {
|
||||
testState.hooksConfig = {
|
||||
enabled: true,
|
||||
token: HOOK_TOKEN,
|
||||
mappings: [
|
||||
{
|
||||
id: "gmail-source",
|
||||
match: { path: "gmail" },
|
||||
action: "agent",
|
||||
messageTemplate: "New email from {{messages[0].from}}",
|
||||
},
|
||||
],
|
||||
};
|
||||
await withGatewayServer(async ({ port }) => {
|
||||
cronIsolatedRun.mockClear();
|
||||
cronIsolatedRun.mockResolvedValue({ status: "ok", summary: "done" });
|
||||
expect(
|
||||
(await postHook(port, "/hooks/agent", { message: "Direct" }, "direct-source")).status,
|
||||
).toBe(200);
|
||||
expect(
|
||||
(await postHook(port, "/hooks/gmail", { messages: [{ from: "Ada" }] }, "mapped-source"))
|
||||
.status,
|
||||
).toBe(200);
|
||||
expect(readExecutionIdentityCall(0)).toEqual({
|
||||
ingress: { kind: "webhook", boundary: "gateway.hooks.agent", state: "present" },
|
||||
});
|
||||
expect(readExecutionIdentityCall(1)).toEqual({
|
||||
ingress: {
|
||||
kind: "webhook",
|
||||
boundary: "gateway.hooks.agent",
|
||||
state: "present",
|
||||
rawSourceRef: "gmail-source",
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
test("returns visible suppression without admitting a hook run", async () => {
|
||||
await writeHookTransformModule(
|
||||
"suppress.mjs",
|
||||
"export default function suppress() { return null; }",
|
||||
);
|
||||
testState.hooksConfig = {
|
||||
enabled: true,
|
||||
token: HOOK_TOKEN,
|
||||
mappings: [
|
||||
{
|
||||
id: "suppressed-source",
|
||||
match: { path: "suppressed" },
|
||||
action: "agent",
|
||||
messageTemplate: "private {{payload.subject}}",
|
||||
transform: { module: "suppress.mjs" },
|
||||
},
|
||||
],
|
||||
};
|
||||
await withGatewayServer(async ({ port }) => {
|
||||
cronIsolatedRun.mockClear();
|
||||
const response = await postHook(
|
||||
port,
|
||||
"/hooks/suppressed",
|
||||
{ subject: "secret" },
|
||||
"suppressed-source",
|
||||
);
|
||||
expect(response.status).toBe(204);
|
||||
expect(cronIsolatedRun).not.toHaveBeenCalled();
|
||||
expect(peekSystemEventEntries(resolveMainSessionKeyFromConfig())).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
test("shares one pending persistent dispatch without losing its session target", async () => {
|
||||
testState.hooksConfig = {
|
||||
enabled: true,
|
||||
|
||||
@@ -598,6 +598,7 @@ export function createHooksRequestHandler(
|
||||
model: action.model,
|
||||
thinking: action.thinking,
|
||||
timeoutSeconds: action.timeoutSeconds,
|
||||
mappingId: action.mappingId,
|
||||
allowUnsafeExternalContent: action.allowUnsafeExternalContent,
|
||||
externalContentSource: resolveMappedHookExternalContentSource({
|
||||
subPath,
|
||||
|
||||
@@ -484,6 +484,14 @@ export function createGatewayHooksRequestHandler(params: {
|
||||
// cannot starve them. Aggregate capacity stays bounded by the lane
|
||||
// group that owns both lanes.
|
||||
lane: CommandLane.HookDispatch,
|
||||
executionIdentity: {
|
||||
ingress: {
|
||||
kind: "webhook",
|
||||
boundary: "gateway.hooks.agent",
|
||||
state: "present",
|
||||
...(acceptedValue.mappingId ? { rawSourceRef: acceptedValue.mappingId } : {}),
|
||||
},
|
||||
},
|
||||
abortSignal: startupAbortController.signal,
|
||||
onExecutionStarted: () => {
|
||||
// Existing runner-entry callbacks are the final owner-boundary fence:
|
||||
|
||||
@@ -103,6 +103,43 @@ describe("OpenClaw database maintenance schema validation", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps lifecycle bindings additive and keyed only by canonical owner identity", () => {
|
||||
const start = OPENCLAW_STATE_SCHEMA_SQL.indexOf(
|
||||
"CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (",
|
||||
);
|
||||
const endMarker = ") STRICT;";
|
||||
const end = start >= 0 ? OPENCLAW_STATE_SCHEMA_SQL.indexOf(endMarker, start) : -1;
|
||||
expect(start).toBeGreaterThanOrEqual(0);
|
||||
expect(end).toBeGreaterThan(start);
|
||||
const previousSchema = `${OPENCLAW_STATE_SCHEMA_SQL.slice(0, start)}${OPENCLAW_STATE_SCHEMA_SQL.slice(end + endMarker.length)}`;
|
||||
const database = createGlobalDatabase();
|
||||
try {
|
||||
expect(() =>
|
||||
assertSqliteSchemaContains(database, "previous global schema", previousSchema),
|
||||
).not.toThrow();
|
||||
expect(
|
||||
database.prepare("PRAGMA table_info(execution_owner_lifecycle_bindings)").all(),
|
||||
).toEqual([
|
||||
{ cid: 0, name: "owner_kind", type: "TEXT", notnull: 1, dflt_value: null, pk: 1 },
|
||||
{ cid: 1, name: "owner_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 2 },
|
||||
{ cid: 2, name: "context_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 },
|
||||
{ cid: 3, name: "execution_id", type: "TEXT", notnull: 1, dflt_value: null, pk: 0 },
|
||||
]);
|
||||
expect(
|
||||
database
|
||||
.prepare(
|
||||
`SELECT COUNT(*) AS count
|
||||
FROM sqlite_schema
|
||||
WHERE type = 'index' AND tbl_name = 'execution_owner_lifecycle_bindings'
|
||||
AND sql IS NOT NULL`,
|
||||
)
|
||||
.get(),
|
||||
).toEqual({ count: 0 });
|
||||
} finally {
|
||||
database.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("accepts compatible future columns in shared-state and agent databases", () => {
|
||||
const globalDatabase = createGlobalDatabase();
|
||||
const agentDatabase = createAgentDatabase();
|
||||
|
||||
@@ -18,6 +18,7 @@ export const FIRST_USE_STATE_TABLES = [
|
||||
"node_worker_launches",
|
||||
"operator_approval_execution_identities",
|
||||
"execution_decision_facts",
|
||||
"execution_owner_lifecycle_bindings",
|
||||
"outbound_message_execution_bindings",
|
||||
"outbound_message_progress",
|
||||
] as const;
|
||||
|
||||
+8
@@ -700,6 +700,13 @@ export interface ExecutionIdentityContexts {
|
||||
run_id: string;
|
||||
}
|
||||
|
||||
export interface ExecutionOwnerLifecycleBindings {
|
||||
context_id: string;
|
||||
execution_id: string;
|
||||
owner_id: string;
|
||||
owner_kind: string;
|
||||
}
|
||||
|
||||
export interface FleetCells {
|
||||
container_name: string;
|
||||
created_at_ms: number;
|
||||
@@ -1871,6 +1878,7 @@ export interface DB {
|
||||
exec_approvals_config: ExecApprovalsConfig;
|
||||
execution_decision_facts: ExecutionDecisionFacts;
|
||||
execution_identity_contexts: ExecutionIdentityContexts;
|
||||
execution_owner_lifecycle_bindings: ExecutionOwnerLifecycleBindings;
|
||||
fleet_cells: FleetCells;
|
||||
flow_runs: FlowRuns;
|
||||
gateway_boot_lifecycle: GatewayBootLifecycle;
|
||||
|
||||
@@ -304,6 +304,16 @@ CREATE INDEX IF NOT EXISTS execution_decision_facts_context_occurred_idx
|
||||
CREATE INDEX IF NOT EXISTS execution_decision_facts_run_occurred_idx
|
||||
ON execution_decision_facts (run_id, occurred_at, receipt_id);
|
||||
|
||||
-- Exact admission identity stays separate from owner-native lifecycle rows so
|
||||
-- older readers retain byte-compatible cron/task/flow table definitions.
|
||||
CREATE TABLE IF NOT EXISTS execution_owner_lifecycle_bindings (
|
||||
owner_kind TEXT NOT NULL,
|
||||
owner_id TEXT NOT NULL,
|
||||
context_id TEXT NOT NULL,
|
||||
execution_id TEXT NOT NULL,
|
||||
PRIMARY KEY (owner_kind, owner_id)
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS session_state_events (
|
||||
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dedupe_key TEXT UNIQUE,
|
||||
|
||||
@@ -1,7 +1,21 @@
|
||||
// Persists managed task-flow records through the OpenClaw SQLite state database.
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import type { Insertable, Selectable } from "kysely";
|
||||
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
|
||||
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
|
||||
import {
|
||||
executionOwnerBindingFromAdmission,
|
||||
type ExecutionOwnerBindingResult,
|
||||
} from "../audit/execution-owner-binding.js";
|
||||
import {
|
||||
bindExecutionOwnerLifecycleMetadata,
|
||||
deleteExecutionOwnerLifecycleMetadata,
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata,
|
||||
} from "../audit/execution-owner-lifecycle-binding-store.js";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
executeSqliteQueryTakeFirstSync,
|
||||
getNodeSqliteKysely,
|
||||
} from "../infra/kysely-sync.js";
|
||||
import { normalizeSqliteNumber } from "../infra/sqlite-number.js";
|
||||
import { withExistingOpenClawStateDatabaseReadOnly } from "../state/openclaw-state-db-readonly.js";
|
||||
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
|
||||
@@ -9,6 +23,7 @@ import {
|
||||
closeOpenClawStateDatabase,
|
||||
openOpenClawStateDatabase,
|
||||
runOpenClawStateWriteTransaction,
|
||||
type OpenClawStateDatabaseOptions,
|
||||
} from "../state/openclaw-state-db.js";
|
||||
import type { TaskFlowRegistryStoreSnapshot } from "./task-flow-registry.store.types.js";
|
||||
import {
|
||||
@@ -42,7 +57,10 @@ function serializeJson(value: unknown): string | null {
|
||||
return value === undefined ? null : JSON.stringify(value);
|
||||
}
|
||||
|
||||
function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
|
||||
function resolveFlowSyncMode(row: {
|
||||
sync_mode: string | null;
|
||||
shape: string | null;
|
||||
}): TaskFlowSyncMode {
|
||||
// Older single_task rows did not persist sync_mode; preserve their mirrored semantics.
|
||||
const syncMode = parseOptionalTaskFlowSyncMode(row.sync_mode);
|
||||
if (syncMode) {
|
||||
@@ -51,6 +69,29 @@ function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
|
||||
return row.shape === "single_task" ? "task_mirrored" : "managed";
|
||||
}
|
||||
|
||||
function rowToSyncMode(row: FlowRegistryRow): TaskFlowSyncMode {
|
||||
return resolveFlowSyncMode(row);
|
||||
}
|
||||
|
||||
function isFlowExecutionOwnerActive(row: {
|
||||
sync_mode: string | null;
|
||||
shape: string | null;
|
||||
status: string;
|
||||
cancel_requested_at: number | null;
|
||||
ended_at: number | null;
|
||||
}): boolean {
|
||||
const syncMode = resolveFlowSyncMode(row);
|
||||
const status = parseTaskFlowStatus(row.status);
|
||||
if (row.cancel_requested_at !== null || row.ended_at !== null) {
|
||||
return false;
|
||||
}
|
||||
// Mirrored `blocked` is derived from a terminal task; managed `blocked`
|
||||
// remains live while its controller waits for the blocking task.
|
||||
return syncMode === "task_mirrored"
|
||||
? status === "queued" || status === "running"
|
||||
: status === "queued" || status === "running" || status === "waiting" || status === "blocked";
|
||||
}
|
||||
|
||||
function rowToFlowRecord(row: FlowRegistryRow): TaskFlowRecord {
|
||||
const endedAt = normalizeSqliteNumber(row.ended_at);
|
||||
const cancelRequestedAt = normalizeSqliteNumber(row.cancel_requested_at);
|
||||
@@ -233,12 +274,14 @@ export function saveTaskFlowRegistryStateToSqlite(snapshot: TaskFlowRegistryStor
|
||||
const flowIds = [...snapshot.flows.keys()];
|
||||
if (flowIds.length === 0) {
|
||||
executeSqliteQuerySync(db, kysely.deleteFrom("flow_runs"));
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow");
|
||||
return;
|
||||
}
|
||||
pruneFlowsNotInSnapshot({ db, ids: flowIds });
|
||||
for (const flow of snapshot.flows.values()) {
|
||||
upsertFlowRow(db, bindFlowRecord(flow));
|
||||
}
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "flow");
|
||||
});
|
||||
}
|
||||
|
||||
@@ -248,12 +291,48 @@ export function upsertTaskFlowRegistryRecordToSqlite(flow: TaskFlowRecord) {
|
||||
});
|
||||
}
|
||||
|
||||
/** Binds only the exact flow selected before admission; lifecycle settlement stays owner-native. */
|
||||
export function bindTaskFlowExecution(params: {
|
||||
admitted: AdmittedRunContext;
|
||||
flowId: string;
|
||||
options?: OpenClawStateDatabaseOptions;
|
||||
}): ExecutionOwnerBindingResult {
|
||||
const binding = executionOwnerBindingFromAdmission(params.admitted);
|
||||
if (!binding) {
|
||||
return "disabled";
|
||||
}
|
||||
return runOpenClawStateWriteTransaction(
|
||||
({ db }) => {
|
||||
const kysely = getFlowRegistryKysely(db);
|
||||
const current = executeSqliteQueryTakeFirstSync(
|
||||
db,
|
||||
kysely
|
||||
.selectFrom("flow_runs")
|
||||
.select(["flow_id", "sync_mode", "shape", "status", "cancel_requested_at", "ended_at"])
|
||||
.where("flow_id", "=", params.flowId),
|
||||
);
|
||||
if (!current || !isFlowExecutionOwnerActive(current)) {
|
||||
return "missing";
|
||||
}
|
||||
return bindExecutionOwnerLifecycleMetadata({
|
||||
db,
|
||||
ownerKind: "flow",
|
||||
ownerId: current.flow_id,
|
||||
binding,
|
||||
});
|
||||
},
|
||||
params.options,
|
||||
{ operationLabel: "task.flow.execution-binding" },
|
||||
);
|
||||
}
|
||||
|
||||
export function deleteTaskFlowRegistryRecordFromSqlite(flowId: string) {
|
||||
withWriteTransaction(({ db }) => {
|
||||
executeSqliteQuerySync(
|
||||
db,
|
||||
getFlowRegistryKysely(db).deleteFrom("flow_runs").where("flow_id", "=", flowId),
|
||||
);
|
||||
deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "flow", ownerIds: [flowId] });
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -2,7 +2,10 @@
|
||||
import { statSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js";
|
||||
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
|
||||
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
|
||||
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
|
||||
import { openOpenClawStateDatabase } from "../state/openclaw-state-db.js";
|
||||
import { resolveOpenClawStateSqlitePath } from "../state/openclaw-state-db.paths.js";
|
||||
@@ -14,6 +17,7 @@ import {
|
||||
setFlowWaiting,
|
||||
} from "./task-flow-registry.js";
|
||||
import {
|
||||
bindTaskFlowExecution,
|
||||
loadTaskFlowRegistryStateFromSqlite,
|
||||
loadTaskFlowRegistryStateFromSqliteReadOnly,
|
||||
saveTaskFlowRegistryStateToSqlite,
|
||||
@@ -307,13 +311,25 @@ describe("task-flow-registry store runtime", () => {
|
||||
...createStoredFlow(),
|
||||
flowId: `flow-large-${index}`,
|
||||
controllerId: `tests/large-flow-${index}`,
|
||||
status: "running",
|
||||
createdAt: index,
|
||||
updatedAt: index,
|
||||
cancelRequestedAt: undefined,
|
||||
endedAt: undefined,
|
||||
};
|
||||
flows.set(flow.flowId, flow);
|
||||
}
|
||||
|
||||
saveTaskFlowRegistryStateToSqlite({ flows });
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-flow-prune", runId: "run-flow-prune" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-prune", {
|
||||
contextId: "context-flow-prune",
|
||||
executionId: "execution-flow-prune",
|
||||
}),
|
||||
};
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-0" })).toBe("bound");
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: "flow-large-1199" })).toBe("bound");
|
||||
const retainedFlows = new Map([...flows].slice(100));
|
||||
saveTaskFlowRegistryStateToSqlite({ flows: retainedFlows });
|
||||
|
||||
@@ -321,6 +337,104 @@ describe("task-flow-registry store runtime", () => {
|
||||
expect(restored.flows.size).toBe(1_100);
|
||||
expect(restored.flows.has("flow-large-0")).toBe(false);
|
||||
expect(restored.flows.has("flow-large-1199")).toBe(true);
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT owner_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'flow'
|
||||
ORDER BY owner_id`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([{ owner_id: "flow-large-1199" }]);
|
||||
});
|
||||
});
|
||||
|
||||
it("binds only source-live flow owners across managed and mirrored lifecycles", async () => {
|
||||
await withFlowRegistryTempDir(async () => {
|
||||
const managed: TaskFlowRecord = {
|
||||
...createStoredFlow(),
|
||||
flowId: "flow-binding-managed",
|
||||
status: "blocked",
|
||||
endedAt: undefined,
|
||||
cancelRequestedAt: undefined,
|
||||
};
|
||||
const mirrored: TaskFlowRecord = {
|
||||
...createStoredFlow(),
|
||||
flowId: "flow-binding-mirrored",
|
||||
syncMode: "task_mirrored",
|
||||
controllerId: undefined,
|
||||
status: "running",
|
||||
endedAt: undefined,
|
||||
cancelRequestedAt: undefined,
|
||||
};
|
||||
const managedTerminal: TaskFlowRecord = {
|
||||
...managed,
|
||||
flowId: "flow-binding-managed-terminal",
|
||||
status: "succeeded",
|
||||
endedAt: 200,
|
||||
};
|
||||
const mirroredTerminal: TaskFlowRecord = {
|
||||
...mirrored,
|
||||
flowId: "flow-binding-mirrored-terminal",
|
||||
status: "blocked",
|
||||
endedAt: 201,
|
||||
};
|
||||
const managedCancelling: TaskFlowRecord = {
|
||||
...managed,
|
||||
flowId: "flow-binding-managed-cancelling",
|
||||
status: "running",
|
||||
cancelRequestedAt: 199,
|
||||
};
|
||||
saveTaskFlowRegistryStateToSqlite({
|
||||
flows: new Map(
|
||||
[managed, mirrored, managedTerminal, mirroredTerminal, managedCancelling].map((flow) => [
|
||||
flow.flowId,
|
||||
flow,
|
||||
]),
|
||||
),
|
||||
});
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-flow-owner", runId: "run-flow-owner" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-flow-owner", {
|
||||
contextId: "context-flow-owner",
|
||||
executionId: "execution-flow-owner",
|
||||
}),
|
||||
};
|
||||
|
||||
expect(
|
||||
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
|
||||
).toBe(false);
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: managedTerminal.flowId })).toBe("missing");
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: mirroredTerminal.flowId })).toBe("missing");
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: managedCancelling.flowId })).toBe("missing");
|
||||
expect(
|
||||
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
|
||||
).toBe(false);
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("bound");
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("bound");
|
||||
|
||||
saveTaskFlowRegistryStateToSqlite({
|
||||
flows: new Map([
|
||||
[managed.flowId, { ...managed, status: "succeeded", endedAt: 210 }],
|
||||
[mirrored.flowId, { ...mirrored, status: "blocked", endedAt: 211 }],
|
||||
[managedTerminal.flowId, managedTerminal],
|
||||
[mirroredTerminal.flowId, mirroredTerminal],
|
||||
[managedCancelling.flowId, managedCancelling],
|
||||
]),
|
||||
});
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: managed.flowId })).toBe("missing");
|
||||
expect(bindTaskFlowExecution({ admitted, flowId: mirrored.flowId })).toBe("missing");
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT owner_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'flow'
|
||||
ORDER BY owner_id`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([{ owner_id: managed.flowId }, { owner_id: mirrored.flowId }]);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -1,7 +1,21 @@
|
||||
// Persists task registry records and events through the OpenClaw SQLite state database.
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import type { Insertable, Selectable } from "kysely";
|
||||
import { executeSqliteQuerySync, getNodeSqliteKysely } from "../infra/kysely-sync.js";
|
||||
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
|
||||
import {
|
||||
executionOwnerBindingFromAdmission,
|
||||
type ExecutionOwnerBindingResult,
|
||||
} from "../audit/execution-owner-binding.js";
|
||||
import {
|
||||
bindExecutionOwnerLifecycleMetadata,
|
||||
deleteExecutionOwnerLifecycleMetadata,
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata,
|
||||
} from "../audit/execution-owner-lifecycle-binding-store.js";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
executeSqliteQueryTakeFirstSync,
|
||||
getNodeSqliteKysely,
|
||||
} from "../infra/kysely-sync.js";
|
||||
import { assertSqliteTableIntegrity } from "../infra/sqlite-integrity.js";
|
||||
import { normalizeSqliteNumber } from "../infra/sqlite-number.js";
|
||||
import { runSqliteDeferredTransactionSync } from "../infra/sqlite-transaction.js";
|
||||
@@ -13,6 +27,7 @@ import {
|
||||
openOpenClawStateDatabase,
|
||||
runOpenClawStateWriteTransaction,
|
||||
type OpenClawStateDatabase,
|
||||
type OpenClawStateDatabaseOptions,
|
||||
} from "../state/openclaw-state-db.js";
|
||||
import { parseDeliveryContextJson, parseSqliteJsonValue } from "./task-registry.sqlite.shared.js";
|
||||
import type { TaskRegistryStoreSnapshot } from "./task-registry.store.types.js";
|
||||
@@ -329,6 +344,7 @@ function deleteTaskRowsWithDeliveryState(db: DatabaseSync, taskId: string): void
|
||||
kysely.deleteFrom("task_delivery_state").where("task_id", "=", taskId),
|
||||
);
|
||||
executeSqliteQuerySync(db, kysely.deleteFrom("task_runs").where("task_id", "=", taskId));
|
||||
deleteExecutionOwnerLifecycleMetadata({ db, ownerKind: "task", ownerIds: [taskId] });
|
||||
}
|
||||
|
||||
function openTaskRegistryDatabase(): TaskRegistryDatabase {
|
||||
@@ -435,6 +451,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho
|
||||
if (taskIds.length === 0) {
|
||||
executeSqliteQuerySync(db, kysely.deleteFrom("task_delivery_state"));
|
||||
executeSqliteQuerySync(db, kysely.deleteFrom("task_runs"));
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task");
|
||||
return;
|
||||
}
|
||||
pruneRowsNotInSnapshot({
|
||||
@@ -462,6 +479,7 @@ export function saveTaskRegistryStateToSqlite(snapshot: TaskRegistryStoreSnapsho
|
||||
for (const state of snapshot.deliveryStates.values()) {
|
||||
replaceTaskDeliveryStateRow(db, bindTaskDeliveryState(state));
|
||||
}
|
||||
pruneOrphanedExecutionOwnerLifecycleMetadata(db, "task");
|
||||
});
|
||||
}
|
||||
|
||||
@@ -471,6 +489,42 @@ export function upsertTaskRegistryRecordToSqlite(task: TaskRecord) {
|
||||
});
|
||||
}
|
||||
|
||||
/** Binds only the exact task row selected before admission; runId is never a join key. */
|
||||
export function bindTaskRunExecution(params: {
|
||||
admitted: AdmittedRunContext;
|
||||
taskId: string;
|
||||
options?: OpenClawStateDatabaseOptions;
|
||||
}): ExecutionOwnerBindingResult {
|
||||
const binding = executionOwnerBindingFromAdmission(params.admitted);
|
||||
if (!binding) {
|
||||
return "disabled";
|
||||
}
|
||||
return runOpenClawStateWriteTransaction(
|
||||
({ db }) => {
|
||||
const kysely = getTaskRegistryKysely(db);
|
||||
const current = executeSqliteQueryTakeFirstSync(
|
||||
db,
|
||||
kysely
|
||||
.selectFrom("task_runs")
|
||||
.select(["task_id", "status", "ended_at"])
|
||||
.where("task_id", "=", params.taskId),
|
||||
);
|
||||
const status = current ? parseTaskStatus(current.status) : undefined;
|
||||
if (!current || (status !== "queued" && status !== "running") || current.ended_at !== null) {
|
||||
return "missing";
|
||||
}
|
||||
return bindExecutionOwnerLifecycleMetadata({
|
||||
db,
|
||||
ownerKind: "task",
|
||||
ownerId: current.task_id,
|
||||
binding,
|
||||
});
|
||||
},
|
||||
params.options,
|
||||
{ operationLabel: "task.run.execution-binding" },
|
||||
);
|
||||
}
|
||||
|
||||
export function upsertTaskWithDeliveryStateToSqlite(params: {
|
||||
task: TaskRecord;
|
||||
deliveryState?: TaskDeliveryState;
|
||||
|
||||
@@ -3,6 +3,8 @@ import { statSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import type { AdmittedRunContext } from "../agents/admitted-run-context.js";
|
||||
import { createExecutionIdentityAdmissionToken } from "../audit/execution-identity-admission.js";
|
||||
import { emitAgentEvent } from "../infra/agent-events.js";
|
||||
import {
|
||||
executeSqliteQuerySync,
|
||||
@@ -14,6 +16,7 @@ import { readSqliteNumberPragma } from "../infra/sqlite-pragma.test-support.js";
|
||||
import { resetLogger, setLoggerOverride } from "../logging/logger.js";
|
||||
import { loggingState } from "../logging/state.js";
|
||||
import { createWarnLogCapture } from "../logging/test-helpers/warn-log-capture.js";
|
||||
import { tableExists } from "../state/openclaw-state-db-schema-helpers.js";
|
||||
import type { DB as OpenClawStateKyselyDatabase } from "../state/openclaw-state-db.generated.js";
|
||||
import {
|
||||
closeOpenClawStateDatabase,
|
||||
@@ -42,6 +45,7 @@ import {
|
||||
type TaskRegistryObserverEvent,
|
||||
} from "./task-registry.store.js";
|
||||
import {
|
||||
bindTaskRunExecution,
|
||||
loadTaskRegistryStateFromSqlite,
|
||||
loadTaskRegistryStateFromSqliteReadOnly,
|
||||
loadTaskRegistryStateFromSqliteReadOnlyResult,
|
||||
@@ -1249,6 +1253,15 @@ describe("task-registry store runtime", () => {
|
||||
}
|
||||
|
||||
saveTaskRegistryStateToSqlite({ tasks, deliveryStates });
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-task-prune", runId: "run-task-prune" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-prune", {
|
||||
contextId: "context-task-prune",
|
||||
executionId: "execution-task-prune",
|
||||
}),
|
||||
};
|
||||
expect(bindTaskRunExecution({ admitted, taskId: "task-large-0" })).toBe("bound");
|
||||
expect(bindTaskRunExecution({ admitted, taskId: "task-large-1199" })).toBe("bound");
|
||||
const retainedTasks = new Map([...tasks].slice(100));
|
||||
const retainedDeliveryStates = new Map([...deliveryStates].slice(100));
|
||||
saveTaskRegistryStateToSqlite({
|
||||
@@ -1261,6 +1274,81 @@ describe("task-registry store runtime", () => {
|
||||
expect(restored.deliveryStates.size).toBe(1_100);
|
||||
expect(restored.tasks.has("task-large-0")).toBe(false);
|
||||
expect(restored.tasks.has("task-large-1199")).toBe(true);
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT owner_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'task'
|
||||
ORDER BY owner_id`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([{ owner_id: "task-large-1199" }]);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("binds only live task owners and retains their metadata after terminalization", async () => {
|
||||
await withOpenClawTestState(
|
||||
{ layout: "state-only", prefix: "openclaw-task-binding-owner-" },
|
||||
async () => {
|
||||
const active = { ...createStoredTask(), taskId: "task-binding-active" };
|
||||
const terminal: TaskRecord = {
|
||||
...createStoredTask(),
|
||||
taskId: "task-binding-terminal",
|
||||
status: "succeeded",
|
||||
endedAt: 200,
|
||||
};
|
||||
const stale: TaskRecord = {
|
||||
...createStoredTask(),
|
||||
taskId: "task-binding-stale",
|
||||
endedAt: 199,
|
||||
};
|
||||
saveTaskRegistryStateToSqlite({
|
||||
tasks: new Map([
|
||||
[active.taskId, active],
|
||||
[terminal.taskId, terminal],
|
||||
[stale.taskId, stale],
|
||||
]),
|
||||
deliveryStates: new Map(),
|
||||
});
|
||||
const admitted: AdmittedRunContext = {
|
||||
operationalRunInstance: { instanceId: "instance-task-owner", runId: "run-task-owner" },
|
||||
executionIdentityToken: createExecutionIdentityAdmissionToken("run-task-owner", {
|
||||
contextId: "context-task-owner",
|
||||
executionId: "execution-task-owner",
|
||||
}),
|
||||
};
|
||||
|
||||
expect(
|
||||
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
|
||||
).toBe(false);
|
||||
expect(bindTaskRunExecution({ admitted, taskId: terminal.taskId })).toBe("missing");
|
||||
expect(bindTaskRunExecution({ admitted, taskId: stale.taskId })).toBe("missing");
|
||||
expect(
|
||||
tableExists(openOpenClawStateDatabase().db, "execution_owner_lifecycle_bindings"),
|
||||
).toBe(false);
|
||||
expect(bindTaskRunExecution({ admitted, taskId: active.taskId })).toBe("bound");
|
||||
|
||||
const finished = { ...active, status: "succeeded" as const, endedAt: 210 };
|
||||
saveTaskRegistryStateToSqlite({
|
||||
tasks: new Map([
|
||||
[finished.taskId, finished],
|
||||
[terminal.taskId, terminal],
|
||||
[stale.taskId, stale],
|
||||
]),
|
||||
deliveryStates: new Map(),
|
||||
});
|
||||
expect(bindTaskRunExecution({ admitted, taskId: finished.taskId })).toBe("missing");
|
||||
expect(
|
||||
openOpenClawStateDatabase()
|
||||
.db.prepare(
|
||||
`SELECT owner_id
|
||||
FROM execution_owner_lifecycle_bindings
|
||||
WHERE owner_kind = 'task'`,
|
||||
)
|
||||
.all(),
|
||||
).toEqual([{ owner_id: active.taskId }]);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,555 @@
|
||||
// QA Lab producer proves cron/task owner receipts through a real mock Gateway.
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
QA_EVIDENCE_FILENAME,
|
||||
type QaEvidenceSummaryJson,
|
||||
} from "../../../../extensions/qa-lab/src/evidence-summary.js";
|
||||
import { startQaGatewayChild } from "../../../../extensions/qa-lab/src/gateway-child.js";
|
||||
import { startQaMockOpenAiServer } from "../../../../extensions/qa-lab/src/providers/mock-openai/server.js";
|
||||
import type { AuditRunInspectResult } from "../../../../packages/gateway-protocol/src/index.js";
|
||||
import { formatErrorMessage } from "../../../../src/infra/errors.js";
|
||||
import { createQaScriptEvidenceWriter, type QaScriptEvidenceStatus } from "./script-evidence.js";
|
||||
|
||||
const SCENARIO_ID = "autonomous-task-lifecycle-receipts";
|
||||
const SNAPSHOT_FILE = `${SCENARIO_ID}-summary.json`;
|
||||
const HOOK_TOKEN = "qa-autonomous-hook-token";
|
||||
|
||||
type ProducerOptions = { artifactBase: string; repoRoot: string };
|
||||
type ProofResult = {
|
||||
artifacts?: Array<{ filePath: string; kind: string }>;
|
||||
details?: string;
|
||||
durationMs: number;
|
||||
status: QaScriptEvidenceStatus;
|
||||
};
|
||||
type ExactOwnerRow = {
|
||||
context_id: string;
|
||||
execution_id: string;
|
||||
run_id: string;
|
||||
status: string;
|
||||
};
|
||||
type OwnerDisplayProducer = "cron-lifecycle" | "task-lifecycle" | "flow-lifecycle";
|
||||
|
||||
function hasSqliteColumns(db: DatabaseSync, table: string, columns: readonly string[]): boolean {
|
||||
const exists = db
|
||||
.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?")
|
||||
.get(table);
|
||||
if (!exists) {
|
||||
return false;
|
||||
}
|
||||
const present = new Set(
|
||||
(db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>).map(
|
||||
(row) => row.name,
|
||||
),
|
||||
);
|
||||
return columns.every((column) => present.has(column));
|
||||
}
|
||||
|
||||
function parseOptions(argv: readonly string[]): ProducerOptions {
|
||||
const readValue = (name: string) => {
|
||||
const index = argv.indexOf(name);
|
||||
return index >= 0 ? argv[index + 1] : undefined;
|
||||
};
|
||||
const artifactBase = readValue("--artifact-base");
|
||||
if (!artifactBase) {
|
||||
throw new Error("--artifact-base is required");
|
||||
}
|
||||
return {
|
||||
artifactBase: path.resolve(artifactBase),
|
||||
repoRoot: path.resolve(readValue("--repo-root") ?? process.cwd()),
|
||||
};
|
||||
}
|
||||
|
||||
function parseJson<T>(raw: string, label: string): T {
|
||||
try {
|
||||
return JSON.parse(raw) as T;
|
||||
} catch (error) {
|
||||
throw new Error(`${label} was not JSON: ${formatErrorMessage(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
function sha256(value: string): string {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
function stateDatabasePath(gateway: Awaited<ReturnType<typeof startQaGatewayChild>>): string {
|
||||
const stateDir = gateway.runtimeEnv.OPENCLAW_STATE_DIR;
|
||||
if (!stateDir) {
|
||||
throw new Error("QA Gateway did not expose its isolated state directory");
|
||||
}
|
||||
return path.join(stateDir, "state", "openclaw.sqlite");
|
||||
}
|
||||
|
||||
function countExecutionContexts(gateway: Awaited<ReturnType<typeof startQaGatewayChild>>): number {
|
||||
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
|
||||
try {
|
||||
if (!hasSqliteColumns(db, "execution_identity_contexts", ["context_id"])) {
|
||||
return 0;
|
||||
}
|
||||
const row = db.prepare("SELECT COUNT(*) AS count FROM execution_identity_contexts").get() as {
|
||||
count: number;
|
||||
};
|
||||
return row.count;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function readCronOwnerRows(
|
||||
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
|
||||
jobId: string,
|
||||
): { cron: ExactOwnerRow; task: ExactOwnerRow } | undefined {
|
||||
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
|
||||
try {
|
||||
if (
|
||||
!hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) ||
|
||||
!hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [
|
||||
"owner_kind",
|
||||
"owner_id",
|
||||
"context_id",
|
||||
"execution_id",
|
||||
]) ||
|
||||
!hasSqliteColumns(db, "task_runs", ["task_id"])
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const cron = db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, context.run_id, receipt.status
|
||||
FROM cron_run_receipts AS receipt
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
|
||||
JOIN execution_identity_contexts AS context
|
||||
ON context.context_id = binding.context_id
|
||||
AND context.execution_id = binding.execution_id
|
||||
WHERE receipt.job_id = ? AND receipt.status != 'running'
|
||||
ORDER BY receipt.started_at_ms DESC LIMIT 1`,
|
||||
)
|
||||
.get(jobId) as ExactOwnerRow | undefined;
|
||||
const task = db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, context.run_id, task.status
|
||||
FROM task_runs AS task
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
|
||||
JOIN execution_identity_contexts AS context
|
||||
ON context.context_id = binding.context_id
|
||||
AND context.execution_id = binding.execution_id
|
||||
WHERE task.runtime = 'cron' AND task.source_id = ? AND task.ended_at IS NOT NULL
|
||||
ORDER BY task.created_at DESC LIMIT 1`,
|
||||
)
|
||||
.get(jobId) as ExactOwnerRow | undefined;
|
||||
return cron && task ? { cron, task } : undefined;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function readCronOwnerBindingDiagnostic(
|
||||
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
|
||||
jobId: string,
|
||||
): unknown {
|
||||
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
|
||||
try {
|
||||
return {
|
||||
cron: db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, receipt.status
|
||||
FROM cron_run_receipts AS receipt
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'cron' AND binding.owner_id = receipt.receipt_id
|
||||
WHERE receipt.job_id = ?
|
||||
ORDER BY receipt.started_at_ms DESC LIMIT 1`,
|
||||
)
|
||||
.get(jobId),
|
||||
task: db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, task.status
|
||||
FROM task_runs AS task
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
|
||||
WHERE task.runtime = 'cron' AND task.source_id = ?
|
||||
ORDER BY task.created_at DESC LIMIT 1`,
|
||||
)
|
||||
.get(jobId),
|
||||
};
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
function readCliOwnerRows(
|
||||
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>,
|
||||
runId: string,
|
||||
): { task: ExactOwnerRow } | undefined {
|
||||
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
|
||||
try {
|
||||
if (
|
||||
!hasSqliteColumns(db, "execution_identity_contexts", ["context_id", "execution_id"]) ||
|
||||
!hasSqliteColumns(db, "execution_owner_lifecycle_bindings", [
|
||||
"owner_kind",
|
||||
"owner_id",
|
||||
"context_id",
|
||||
"execution_id",
|
||||
]) ||
|
||||
!hasSqliteColumns(db, "task_runs", ["task_id"])
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const task = db
|
||||
.prepare(
|
||||
`SELECT binding.context_id, binding.execution_id, context.run_id, task.status
|
||||
FROM task_runs AS task
|
||||
JOIN execution_owner_lifecycle_bindings AS binding
|
||||
ON binding.owner_kind = 'task' AND binding.owner_id = task.task_id
|
||||
JOIN execution_identity_contexts AS context
|
||||
ON context.context_id = binding.context_id
|
||||
AND context.execution_id = binding.execution_id
|
||||
WHERE task.runtime = 'cli' AND task.run_id = ? AND task.ended_at IS NOT NULL
|
||||
LIMIT 1`,
|
||||
)
|
||||
.get(runId) as ExactOwnerRow | undefined;
|
||||
return task ? { task } : undefined;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function waitFor<T>(label: string, read: () => T | undefined): Promise<T> {
|
||||
const deadline = Date.now() + 30_000;
|
||||
while (Date.now() < deadline) {
|
||||
const value = read();
|
||||
if (value !== undefined) {
|
||||
return value;
|
||||
}
|
||||
await delay(50);
|
||||
}
|
||||
throw new Error(`timed out waiting for ${label}`);
|
||||
}
|
||||
|
||||
function requireOwnerDisplay(result: AuditRunInspectResult, producer: OwnerDisplayProducer) {
|
||||
const receipt = result.decisionDisplays.find(
|
||||
(candidate) =>
|
||||
candidate.provenance.state === "verified" && candidate.provenance.producer === producer,
|
||||
);
|
||||
if (
|
||||
!receipt ||
|
||||
receipt.enforcement.coverageState !== "attribution-only" ||
|
||||
receipt.decision.outcome !== "not-applicable"
|
||||
) {
|
||||
throw new Error(`inspection omitted exact attribution-only ${producer} display`);
|
||||
}
|
||||
return receipt;
|
||||
}
|
||||
|
||||
async function inspectExecution(params: {
|
||||
gateway: Awaited<ReturnType<typeof startQaGatewayChild>>;
|
||||
executionId: string;
|
||||
producers: OwnerDisplayProducer[];
|
||||
privateSentinels: string[];
|
||||
}) {
|
||||
const jsonRaw = await params.gateway.runCli([
|
||||
"audit",
|
||||
"--execution",
|
||||
params.executionId,
|
||||
"--explain",
|
||||
"--json",
|
||||
]);
|
||||
const json = parseJson<AuditRunInspectResult>(jsonRaw, "owner lifecycle inspection");
|
||||
for (const producer of params.producers) {
|
||||
requireOwnerDisplay(json, producer);
|
||||
}
|
||||
for (const sentinel of params.privateSentinels) {
|
||||
if (jsonRaw.includes(sentinel)) {
|
||||
throw new Error(`owner receipt leaked private sentinel ${sentinel}`);
|
||||
}
|
||||
}
|
||||
const human = await params.gateway.runCli([
|
||||
"audit",
|
||||
"--execution",
|
||||
params.executionId,
|
||||
"--explain",
|
||||
]);
|
||||
for (const producer of params.producers) {
|
||||
if (!human.includes(`Display producer: ${producer}`)) {
|
||||
throw new Error(`human inspection omitted ${producer}`);
|
||||
}
|
||||
}
|
||||
return { json, jsonRaw, human };
|
||||
}
|
||||
|
||||
async function runProof(options: ProducerOptions): Promise<string> {
|
||||
const mock = await startQaMockOpenAiServer();
|
||||
let gateway: Awaited<ReturnType<typeof startQaGatewayChild>> | undefined;
|
||||
try {
|
||||
gateway = await startQaGatewayChild({
|
||||
repoRoot: options.repoRoot,
|
||||
useRepoCli: true,
|
||||
providerBaseUrl: `${mock.baseUrl}/v1`,
|
||||
providerMode: "mock-openai",
|
||||
transportBaseUrl: "http://127.0.0.1",
|
||||
controlUiEnabled: false,
|
||||
mutateConfig: (config) => ({
|
||||
...config,
|
||||
logging: {
|
||||
...config.logging,
|
||||
audit: { ...config.logging?.audit, enabled: true, executionIdentity: true },
|
||||
},
|
||||
hooks: {
|
||||
enabled: true,
|
||||
token: HOOK_TOKEN,
|
||||
mappings: [
|
||||
{
|
||||
id: "qa-suppressed-source",
|
||||
match: { path: "suppressed" },
|
||||
action: "agent",
|
||||
messageTemplate: "PRIVATE-SUPPRESSED-{{payload.value}}",
|
||||
transform: { module: "suppress.mjs" },
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
});
|
||||
const transformDir = path.join(path.dirname(gateway.configPath), "hooks", "transforms");
|
||||
await fs.mkdir(transformDir, { recursive: true });
|
||||
await fs.writeFile(
|
||||
path.join(transformDir, "suppress.mjs"),
|
||||
"export default function suppress() { return null; }\n",
|
||||
"utf8",
|
||||
);
|
||||
|
||||
const contextsBeforeSuppression = countExecutionContexts(gateway);
|
||||
const suppression = await fetch(`${gateway.baseUrl}/hooks/suppressed`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${HOOK_TOKEN}`,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({ value: randomUUID() }),
|
||||
});
|
||||
if (suppression.status !== 204) {
|
||||
throw new Error(`mapped suppression returned HTTP ${suppression.status}`);
|
||||
}
|
||||
await delay(100);
|
||||
if (countExecutionContexts(gateway) !== contextsBeforeSuppression) {
|
||||
throw new Error("pre-admission mapping suppression allocated execution identity");
|
||||
}
|
||||
|
||||
const cronSentinel = `PRIVATE-CRON-${randomUUID()}`;
|
||||
const cronJob = (await gateway.call("cron.add", {
|
||||
name: "QA autonomous receipt",
|
||||
enabled: true,
|
||||
schedule: { kind: "at", at: new Date(Date.now() + 3_600_000).toISOString() },
|
||||
sessionTarget: "isolated",
|
||||
wakeMode: "next-heartbeat",
|
||||
payload: { kind: "agentTurn", message: `${cronSentinel}: reply CRON-DONE` },
|
||||
delivery: { mode: "none" },
|
||||
})) as { id: string };
|
||||
await gateway.call("cron.run", { id: cronJob.id, mode: "force" });
|
||||
let cronRows: { cron: ExactOwnerRow; task: ExactOwnerRow };
|
||||
try {
|
||||
cronRows = await waitFor("terminal cron/task exact bindings", () =>
|
||||
readCronOwnerRows(gateway!, cronJob.id),
|
||||
);
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`${formatErrorMessage(error)}; owner rows=${JSON.stringify(readCronOwnerBindingDiagnostic(gateway, cronJob.id))}; gateway logs=${gateway.logs()}`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
if (
|
||||
cronRows.cron.context_id !== cronRows.task.context_id ||
|
||||
cronRows.cron.execution_id !== cronRows.task.execution_id
|
||||
) {
|
||||
throw new Error("cron receipt/task rows did not retain one exact admitted execution");
|
||||
}
|
||||
const cronInspection = await inspectExecution({
|
||||
gateway,
|
||||
executionId: cronRows.cron.execution_id,
|
||||
producers: ["cron-lifecycle", "task-lifecycle"],
|
||||
privateSentinels: [cronSentinel],
|
||||
});
|
||||
const cronCursorPage = parseJson<AuditRunInspectResult>(
|
||||
await gateway.runCli([
|
||||
"audit",
|
||||
"--execution",
|
||||
cronRows.cron.execution_id,
|
||||
"--explain",
|
||||
"--cursor",
|
||||
"c:0:0",
|
||||
"--limit",
|
||||
"1",
|
||||
"--json",
|
||||
]),
|
||||
"cron cursor page",
|
||||
);
|
||||
requireOwnerDisplay(cronCursorPage, "cron-lifecycle");
|
||||
|
||||
const taskSentinel = `PRIVATE-TASK-${randomUUID()}`;
|
||||
const cliRunId = `qa-cli-${randomUUID()}`;
|
||||
const accepted = (await gateway.call(
|
||||
"agent",
|
||||
{
|
||||
sessionKey: "agent:qa:main",
|
||||
message: `${taskSentinel}: reply TASK-DONE`,
|
||||
deliver: false,
|
||||
idempotencyKey: cliRunId,
|
||||
},
|
||||
{ expectFinal: false },
|
||||
)) as { runId: string; status: string };
|
||||
await gateway.call(
|
||||
"agent.wait",
|
||||
{ runId: accepted.runId, timeoutMs: 30_000 },
|
||||
{ timeoutMs: 35_000 },
|
||||
);
|
||||
const cliRows = await waitFor("terminal CLI task exact binding", () =>
|
||||
readCliOwnerRows(gateway!, accepted.runId),
|
||||
);
|
||||
const taskInspection = await inspectExecution({
|
||||
gateway,
|
||||
executionId: cliRows.task.execution_id,
|
||||
producers: ["task-lifecycle"],
|
||||
privateSentinels: [taskSentinel],
|
||||
});
|
||||
|
||||
const beforeRestart = JSON.stringify({
|
||||
cron: cronInspection.json,
|
||||
task: taskInspection.json,
|
||||
});
|
||||
await gateway.restartAfterStateMutation(async () => {});
|
||||
const cronAfter = await inspectExecution({
|
||||
gateway,
|
||||
executionId: cronRows.cron.execution_id,
|
||||
producers: ["cron-lifecycle", "task-lifecycle"],
|
||||
privateSentinels: [cronSentinel],
|
||||
});
|
||||
const taskAfter = await inspectExecution({
|
||||
gateway,
|
||||
executionId: cliRows.task.execution_id,
|
||||
producers: ["task-lifecycle"],
|
||||
privateSentinels: [taskSentinel],
|
||||
});
|
||||
const afterRestart = JSON.stringify({ cron: cronAfter.json, task: taskAfter.json });
|
||||
if (afterRestart !== beforeRestart) {
|
||||
throw new Error("owner lifecycle JSON changed across Gateway replacement");
|
||||
}
|
||||
|
||||
const db = new DatabaseSync(stateDatabasePath(gateway), { readOnly: true });
|
||||
let genericCount = 0;
|
||||
try {
|
||||
if (hasSqliteColumns(db, "execution_decision_facts", ["receipt_id"])) {
|
||||
genericCount = (
|
||||
db.prepare("SELECT COUNT(*) AS count FROM execution_decision_facts").get() as {
|
||||
count: number;
|
||||
}
|
||||
).count;
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
if (genericCount !== 0) {
|
||||
throw new Error("owner lifecycle rows were duplicated into execution_decision_facts");
|
||||
}
|
||||
|
||||
const snapshotPath = path.join(options.artifactBase, SNAPSHOT_FILE);
|
||||
await fs.mkdir(options.artifactBase, { recursive: true });
|
||||
await fs.writeFile(
|
||||
snapshotPath,
|
||||
`${JSON.stringify(
|
||||
{
|
||||
suppression: { httpStatus: 204, identityAllocation: 0 },
|
||||
cron: {
|
||||
contextId: cronRows.cron.context_id,
|
||||
executionId: cronRows.cron.execution_id,
|
||||
statuses: [cronRows.cron.status, cronRows.task.status],
|
||||
displayProducers: ["cron-lifecycle", "task-lifecycle"],
|
||||
},
|
||||
task: {
|
||||
contextId: cliRows.task.context_id,
|
||||
executionId: cliRows.task.execution_id,
|
||||
statuses: [cliRows.task.status],
|
||||
displayProducers: ["task-lifecycle"],
|
||||
},
|
||||
cursorCompatibility: { cronPrefixAccepted: true },
|
||||
genericDuplicateAbsent: true,
|
||||
byteEquivalentAfterRestart: true,
|
||||
privacy: { cronPromptAbsent: true, taskPromptAbsent: true },
|
||||
resultSha256: sha256(afterRestart),
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
return `cron=${cronRows.cron.execution_id}; task=${cliRows.task.execution_id}; suppression=204; restart sha256=${sha256(afterRestart)}`;
|
||||
} finally {
|
||||
await gateway?.stop().catch(() => undefined);
|
||||
await mock.stop();
|
||||
}
|
||||
}
|
||||
|
||||
async function produceProof(options: ProducerOptions): Promise<ProofResult> {
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
return {
|
||||
artifacts: [{ filePath: SNAPSHOT_FILE, kind: "summary" }],
|
||||
details: await runProof(options),
|
||||
durationMs: Math.max(1, Date.now() - startedAt),
|
||||
status: "pass",
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
details: formatErrorMessage(error),
|
||||
durationMs: Math.max(1, Date.now() - startedAt),
|
||||
status: "fail",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function runProducer(options: ProducerOptions): Promise<QaEvidenceSummaryJson> {
|
||||
const writer = createQaScriptEvidenceWriter({
|
||||
artifactBase: options.artifactBase,
|
||||
logFileName: `${SCENARIO_ID}.log`,
|
||||
primaryModel: "mock-openai/gpt-5.6-luna",
|
||||
providerMode: "mock-openai",
|
||||
repoRoot: options.repoRoot,
|
||||
target: {
|
||||
id: SCENARIO_ID,
|
||||
title: "Autonomous task lifecycle receipts",
|
||||
sourcePath: `qa/scenarios/runtime/${SCENARIO_ID}.yaml`,
|
||||
docsRefs: ["docs/gateway/audit.md", "docs/automation/tasks.md"],
|
||||
codeRefs: [
|
||||
"src/audit/execution-decision-receipts.ts",
|
||||
"src/cron/store/run-receipt-store.ts",
|
||||
"src/tasks/task-registry.store.sqlite.ts",
|
||||
"src/tasks/task-flow-registry.store.sqlite.ts",
|
||||
],
|
||||
},
|
||||
});
|
||||
const result = await produceProof(options);
|
||||
writer.appendLog(`${result.status}: ${result.details ?? "no details"}\n`);
|
||||
return await writer.write(result);
|
||||
}
|
||||
|
||||
async function main(argv: readonly string[]) {
|
||||
const evidence = await runProducer(parseOptions(argv));
|
||||
const status = evidence.entries[0]?.result.status;
|
||||
console.log(`Autonomous task lifecycle evidence: ${QA_EVIDENCE_FILENAME}`);
|
||||
console.log(`Autonomous task lifecycle status: ${status}`);
|
||||
return status === "pass" ? 0 : 1;
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
main(process.argv.slice(2))
|
||||
.then((exitCode) => {
|
||||
process.exitCode = exitCode;
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error(formatErrorMessage(error));
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
Reference in New Issue
Block a user