ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-12 21:53:00 -06:00
Code Issues Packages Projects Releases Wiki Activity

Convert QA scenarios to YAML files (#92915)

Browse Source 
* refactor: load QA scenarios from YAML

* docs: update personal QA scenario docs

* test: keep QA scenarios YAML-only
This commit is contained in:
Dallin Romney
2026-06-14 17:31:18 -07:00
committed by GitHub
parent 1ca3d4f586
commit fef8394079
267 changed files with 13169 additions and 13503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/openclaw-qa-testing/SKILL.md
+9 -5
View File
@@ -13,7 +13,7 @@ Use this skill for `qa-lab` / `qa-channel` work. Repo-local QA only.
- `docs/help/testing.md`
- `docs/channels/qa-channel.md`
- `qa/README.md`
- `qa/scenarios/index.md`
- `qa/scenarios/index.yaml`
- `extensions/qa-lab/src/suite.ts`
- `extensions/qa-lab/src/character-eval.ts`
@@ -198,7 +198,9 @@ pnpm openclaw qa character-eval \
- Judges default to `openai/gpt-5.4,thinking=xhigh,fast` and `anthropic/claude-opus-4-6,thinking=high`.
- Report includes judge ranking, run stats, durations, and full transcripts; do not include raw judge replies. Duration is benchmark context, not a grading signal.
- Candidate and judge concurrency default to 16. Use `--concurrency <n>` and `--judge-concurrency <n>` to override when local gateways or provider limits need a gentler lane.
- Scenario source should stay markdown-driven under `qa/scenarios/`.
- Scenario source is YAML-only under `qa/scenarios/`: use `index.yaml` and
per-scenario `*.yaml` files with top-level `title`, `scenario`, and optional
`flow`. Never add fenced `qa-scenario` / `qa-flow` Markdown files.
- For isolated character/persona evals, write the persona into `SOUL.md` and blank `IDENTITY.md` in the scenario flow. Use `SOUL.md + IDENTITY.md` only when intentionally testing how the normal OpenClaw identity combines with the character.
- Keep prompts natural and task-shaped. The candidate model should receive character setup through `SOUL.md`, then normal user turns such as chat, workspace help, and small file tasks; do not ask "how would you react?" or tell the model it is in an eval.
- Prefer at least one real task, such as creating or editing a tiny workspace artifact, so the transcript captures character under normal tool use instead of pure roleplay.
@@ -234,7 +236,8 @@ pnpm openclaw qa manual \
## Repo facts
- Seed scenarios live in `qa/`.
- Seed scenarios live in `qa/scenarios/index.yaml` and
`qa/scenarios/<theme>/*.yaml`.
- Main live runner: `extensions/qa-lab/src/suite.ts`
- QA lab server: `extensions/qa-lab/src/lab-server.ts`
- Child gateway harness: `extensions/qa-lab/src/gateway-child.ts`
@@ -262,8 +265,9 @@ pnpm openclaw qa manual \
## When adding scenarios
- Add or update scenario markdown under `qa/scenarios/`
- Keep kickoff expectations in `qa/scenarios/index.md` aligned
- Add or update scenario YAML under `qa/scenarios/`; do not add `.md` scenario
files or fenced YAML blocks.
- Keep kickoff expectations in `qa/scenarios/index.yaml` aligned
- Add executable coverage in `extensions/qa-lab/src/suite.ts`
- Prefer end-to-end assertions over mock-only checks
- Save outputs under `.artifacts/qa-e2e/`
AGENTS.md
+1
View File
@@ -214,6 +214,7 @@ Skills own workflows; root owns hard policy and routing.
- Vitest. Colocated `*.test.ts`; e2e `*.e2e.test.ts`; example models `sonnet-4.6`, `gpt-5.5`; test GPT with 5.5 preferred, 5.4 ok; no GPT-4.x agent-smoke defaults.
- Prefer behavior tests over workflow/docs string greps. Put operator policy reminders in AGENTS/docs.
- QA scenario sources are YAML only: `qa/scenarios/index.yaml` and `qa/scenarios/<theme>/*.yaml`. Do not add fenced `qa-scenario`/`qa-flow` Markdown files under `qa/scenarios/`.
- Clean timers/env/globals/mocks/sockets/temp dirs/module state; `--isolate=false` safe.
- Prefer injection and narrow `*.runtime.ts` mocks over broad barrels or `openclaw/plugin-sdk/*`.
- Do not edit baseline/inventory/ignore/snapshot/expected-failure files to silence checks without explicit approval.
docs/concepts/personal-agent-benchmark-pack.md
+4 -4
View File
@@ -11,7 +11,7 @@ The Personal Agent Benchmark Pack is a small repo-backed QA scenario pack for
local personal assistant workflows. It is not a generic model benchmark and it
does not require a new runner. The pack reuses the private QA stack described in
[QA overview](/concepts/qa-e2e-automation), the synthetic
[QA channel](/channels/qa-channel), and the existing `qa/scenarios` markdown
[QA channel](/channels/qa-channel), and the existing `qa/scenarios` YAML
catalog.
The first pack is intentionally narrow:
@@ -61,9 +61,9 @@ to inspect and file in issues.
## Extending The Pack
Add new cases under `qa/scenarios/personal/`, then add the scenario id to
`QA_PERSONAL_AGENT_SCENARIO_IDS`. Keep each case small, local, deterministic in
`mock-openai`, and focused on one personal assistant behavior.
Add new `.yaml` cases under `qa/scenarios/personal/`, then add the scenario id
to `QA_PERSONAL_AGENT_SCENARIO_IDS`. Keep each case small, local, deterministic
in `mock-openai`, and focused on one personal assistant behavior.
Good follow-up candidates:
docs/concepts/qa-e2e-automation.md
+18 -17
View File
@@ -33,7 +33,7 @@ script aliases; both forms are supported.
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `qa run` | Bundled QA self-check; writes a Markdown report. |
| `qa suite` | Run repo-backed scenarios against the QA gateway lane. Aliases: `pnpm openclaw qa suite --runner multipass` for a disposable Linux VM. |
| `qa coverage` | Print the markdown scenario-coverage inventory (`--json` for machine output). |
| `qa coverage` | Print the YAML scenario-coverage inventory (`--json` for machine output). |
| `qa parity-report` | Compare two `qa-suite-summary.json` files and write the agentic parity report, or use `--runtime-axis --token-efficiency` to write Codex-vs-OpenClaw runtime parity and token-efficiency reports from one runtime-pair summary. |
| `qa character-eval` | Run the character QA scenario across multiple live models with a judged report. See [Reporting](#reporting). |
| `qa manual` | Run a one-off prompt against the selected provider/model lane. |
@@ -769,25 +769,26 @@ Operational env vars and the Convex broker endpoint contract live in [Testing
Seed assets live in `qa/`:
- `qa/scenarios/index.md`
- `qa/scenarios/<theme>/*.md`
- `qa/scenarios/index.yaml`
- `qa/scenarios/<theme>/*.yaml`
These are intentionally in git so the QA plan is visible to both humans and the
agent.
`qa-lab` should stay a generic markdown runner. Each scenario markdown file is
`qa-lab` should stay a generic YAML scenario runner. Each scenario YAML file is
the source of truth for one test run and should define:
- scenario metadata
- optional category, capability, lane, and risk metadata
- docs and code refs
- optional plugin requirements
- optional gateway config patch
- an executable `qa-flow` block for flow scenarios, or `execution.kind`/`execution.path`
for Vitest and Playwright scenarios
- top-level `title`
- `scenario` metadata
- optional category, capability, lane, and risk metadata in `scenario`
- docs and code refs in `scenario`
- optional plugin requirements in `scenario`
- optional gateway config patch in `scenario`
- executable top-level `flow` for flow scenarios, or `scenario.execution.kind` /
`scenario.execution.path` for Vitest and Playwright scenarios
The reusable runtime surface that backs `qa-flow` blocks is allowed to stay generic
and cross-cutting. For example, markdown scenarios can combine transport-side
The reusable runtime surface that backs `flow` is allowed to stay generic
and cross-cutting. For example, YAML scenarios can combine transport-side
helpers with browser-side helpers that drive the embedded Control UI through the
Gateway `browser.request` seam without adding a special-case runner.
@@ -825,17 +826,17 @@ provider names.
## Transport adapters
`qa-lab` owns a generic transport seam for markdown QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
`qa-lab` owns a generic transport seam for YAML QA scenarios. `qa-channel` is the first adapter on that seam, but the design target is wider: future real or synthetic channels should plug into the same suite runner instead of adding a transport-specific QA runner.
At the architecture level, the split is:
- `qa-lab` owns generic scenario execution, worker concurrency, artifact writing, and reporting.
- The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
- Markdown scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
- YAML scenario files under `qa/scenarios/` define the test run; `qa-lab` provides the reusable runtime surface that executes them.
### Adding a channel
Adding a channel to the markdown QA system requires exactly two things:
Adding a channel to the YAML QA system requires exactly two things:
1. A transport adapter for the channel.
2. A scenario pack that exercises the channel contract.
@@ -869,7 +870,7 @@ The minimum adoption bar for a new channel:
2. Implement the transport runner on the shared `qa-lab` host seam.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command. Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`. Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
5. Author or adapt markdown scenarios under the themed `qa/scenarios/` directories.
5. Author or adapt YAML scenarios under the themed `qa/scenarios/` directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
extensions/qa-lab/api.ts
+2 -2
View File
@@ -64,7 +64,7 @@ export {
export {
DEFAULT_QA_AGENT_IDENTITY_MARKDOWN,
hasQaScenarioPack,
listQaScenarioMarkdownPaths,
listQaScenarioYamlPaths,
type QaBootstrapScenarioCatalog,
type QaScenarioExecution,
type QaScenarioFlow,
@@ -76,7 +76,7 @@ export {
readQaScenarioExecutionConfig,
readQaScenarioOverviewMarkdown,
readQaScenarioPack,
readQaScenarioPackMarkdown,
readQaScenarioPackYamlSource,
validateQaScenarioExecutionConfig,
} from "./src/scenario-catalog.js";
export { createQaSelfCheckScenario } from "./src/self-check-scenario.js";
extensions/qa-lab/src/cli.ts
+1 -1
View File
@@ -436,7 +436,7 @@ export function registerQaLabCli(program: Command) {
);
qa.command("coverage")
.description("Print the markdown QA coverage inventory")
.description("Print the YAML QA coverage inventory")
.option("--repo-root <path>", "Repository root to target when writing --output")
.option("--output <path>", "Write the coverage inventory to this path")
.option("--json", "Print JSON instead of Markdown", false)
extensions/qa-lab/src/coverage-report.test.ts
+10 -10
View File
@@ -92,7 +92,7 @@ function scenarioWithCoverage(params: {
},
objective: "Exercise test coverage.",
successCriteria: ["Evidence is recorded."],
sourcePath: params.sourcePath ?? "qa/scenarios/test/test-scenario.md",
sourcePath: params.sourcePath ?? "qa/scenarios/test/test-scenario.yaml",
execution,
};
}
@@ -157,7 +157,7 @@ describe("qa coverage report", () => {
kind: "playwright",
path: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
role: "primary",
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.md"],
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.yaml"],
});
expect(inventory.scenarioPacks.map((pack) => pack.id)).toEqual([
"observability",
@@ -184,7 +184,7 @@ describe("qa coverage report", () => {
expect(report).toContain("- Missing coverage metadata: 0");
expect(report).toContain("- Overlapping coverage IDs:");
expect(report).toContain("memory.recall");
expect(report).toContain("primary: memory-recall (qa/scenarios/memory/memory-recall.md)");
expect(report).toContain("primary: memory-recall (qa/scenarios/memory/memory-recall.yaml)");
expect(report).toContain("secondary: active-memory-preprompt-recall");
expect(report).toContain("## Scenario Packs");
expect(report).toContain(
@@ -236,7 +236,7 @@ describe("qa coverage report", () => {
primary: [TEST_BROWSER_COVERAGE_ID],
executionKind: "playwright",
executionPath: playwrightExecutionPath,
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.md",
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
});
const report = renderQaScenarioMatchesMarkdownReport({
query: "mixed",
@@ -310,7 +310,7 @@ describe("qa coverage report", () => {
scenarios: [
scenarioWithCoverage({
primary: [TEST_BROWSER_COVERAGE_ID],
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.md",
sourcePath: "qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
executionKind: "playwright",
executionPath: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
}),
@@ -322,7 +322,7 @@ describe("qa coverage report", () => {
expect(report.fulfilledFeatureCount).toBe(1);
expect(report.categories[0]?.mappingStatus).toBe("mapped");
expect(report.categories[0]?.scenarioRefs).toStrictEqual([
"qa/scenarios/ui/control-ui-chat-flow-playwright.md",
"qa/scenarios/ui/control-ui-chat-flow-playwright.yaml",
]);
expect(report.categories[0]?.evidence).toStrictEqual([
{
@@ -330,7 +330,7 @@ describe("qa coverage report", () => {
kind: "playwright",
path: "ui/src/ui/e2e/chat-flow.e2e.test.ts",
role: "primary",
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.md"],
scenarioRefs: ["qa/scenarios/ui/control-ui-chat-flow-playwright.yaml"],
},
]);
});
@@ -389,14 +389,14 @@ describe("qa coverage report", () => {
scenarios: [
scenarioWithCoverage({
primary: [TEST_EXECUTABLE_COVERAGE_ID],
sourcePath: "qa/scenarios/channels/dm-chat-baseline.md",
sourcePath: "qa/scenarios/channels/dm-chat-baseline.yaml",
}),
],
});
expect(report.validationIssues).toStrictEqual([]);
expect(report.categories[0]?.scenarioRefs).toStrictEqual([
"qa/scenarios/channels/dm-chat-baseline.md",
"qa/scenarios/channels/dm-chat-baseline.yaml",
]);
expect(report.categories[0]?.evidence).toStrictEqual([
{
@@ -404,7 +404,7 @@ describe("qa coverage report", () => {
kind: "qa-scenario",
path: null,
role: "primary",
scenarioRefs: ["qa/scenarios/channels/dm-chat-baseline.md"],
scenarioRefs: ["qa/scenarios/channels/dm-chat-baseline.yaml"],
},
]);
});
extensions/qa-lab/src/coverage-report.ts
+1 -1
View File
@@ -505,7 +505,7 @@ export function renderQaScenarioMatchesMarkdownReport(params: {
lines.push(` - surface: ${match.surfaces.join(", ")}`);
lines.push(
match.executionKind === "flow"
? " - execution: qa-flow"
? " - execution: flow"
: ` - execution: ${match.executionKind} ${match.executionPath ?? "missing"}`,
);
lines.push(` - coverage IDs: ${match.coverageIds.join(", ") || "none"}`);
extensions/qa-lab/src/discovery-eval.test.ts
+4 -4
View File
@@ -10,7 +10,7 @@ describe("qa discovery evaluation", () => {
it("accepts rich discovery reports that explicitly confirm all required files were read", () => {
const report = `
Worked
- Read all three requested files: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.
- Read all three requested files: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.
Failed
- None.
Blocked
@@ -29,7 +29,7 @@ The helper text mentions banned phrases like "not present", "missing files", "bl
it("accepts numeric 'all 4 required files read' confirmations", () => {
const report = `
Worked
- Source: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md
- Source: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md
- all 3 required files read.
Failed
- None.
@@ -50,7 +50,7 @@ The report may quote phrases like "not present" while describing the evaluator,
const report = `
Worked
- All three files retrieved. Now let me compile the protocol report.
- All three mandated files read successfully: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
- All three mandated files read successfully: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
Failed
- None.
Blocked
@@ -84,7 +84,7 @@ Follow-up
it("flags discovery replies that drift into unrelated suite wrap-up claims", () => {
const report = `
Worked
- All three requested files were read: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
- All three requested files were read: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, repo/docs/help/testing.md.
Failed
- None.
Blocked
extensions/qa-lab/src/discovery-eval.ts
+1 -1
View File
@@ -8,7 +8,7 @@ function readRequiredDiscoveryRefs() {
| undefined;
return (
config?.requiredFiles ?? [
"repo/qa/scenarios/index.md",
"repo/qa/scenarios/index.yaml",
"repo/extensions/qa-lab/src/suite.ts",
"repo/docs/help/testing.md",
]
extensions/qa-lab/src/docker-harness.test.ts
+3 -3
View File
@@ -52,7 +52,7 @@ describe("qa docker harness", () => {
path.join(outputDir, "state", "openclaw.json"),
path.join(outputDir, "state", "seed-workspace", "QA_KICKOFF_TASK.md"),
path.join(outputDir, "state", "seed-workspace", "QA_SCENARIO_PLAN.md"),
path.join(outputDir, "state", "seed-workspace", "QA_SCENARIOS.md"),
path.join(outputDir, "state", "seed-workspace", "QA_SCENARIOS.yaml"),
path.join(outputDir, "state", "seed-workspace", "IDENTITY.md"),
]) {
expect(result.files).toContain(expectedFile);
@@ -125,10 +125,10 @@ describe("qa docker harness", () => {
expect(kickoff).toContain("Lobster Invaders");
const scenarios = await readFile(
path.join(outputDir, "state", "seed-workspace", "QA_SCENARIOS.md"),
path.join(outputDir, "state", "seed-workspace", "QA_SCENARIOS.yaml"),
"utf8",
);
expect(scenarios).toContain("```yaml qa-pack");
expect(scenarios).toContain("pack:");
expect(scenarios).toContain("subagent-fanout-synthesis");
const readme = await readFile(path.join(outputDir, "README.md"), "utf8");
extensions/qa-lab/src/docker-harness.ts
+1 -1
View File
@@ -318,7 +318,7 @@ export async function writeQaDockerHarnessFiles(params: {
path.join(params.outputDir, "state", "seed-workspace", "IDENTITY.md"),
path.join(params.outputDir, "state", "seed-workspace", "QA_KICKOFF_TASK.md"),
path.join(params.outputDir, "state", "seed-workspace", "QA_SCENARIO_PLAN.md"),
path.join(params.outputDir, "state", "seed-workspace", "QA_SCENARIOS.md"),
path.join(params.outputDir, "state", "seed-workspace", "QA_SCENARIOS.yaml"),
],
};
}
extensions/qa-lab/src/evidence-summary.test.ts
+2 -2
View File
@@ -22,7 +22,7 @@ describe("evidence summary", () => {
{
id: "dm-chat-baseline",
title: "DM baseline conversation",
sourcePath: "qa/scenarios/channels/dm-chat-baseline.md",
sourcePath: "qa/scenarios/channels/dm-chat-baseline.yaml",
surface: "dm",
coverage: {
primary: ["channels.dm"],
@@ -54,7 +54,7 @@ describe("evidence summary", () => {
id: "dm-chat-baseline",
title: "DM baseline conversation",
source: {
path: "qa/scenarios/channels/dm-chat-baseline.md",
path: "qa/scenarios/channels/dm-chat-baseline.yaml",
},
},
mapping: {
extensions/qa-lab/src/model-switch-eval.ts
+1 -1
View File
@@ -7,7 +7,7 @@ export function hasModelSwitchContinuitySignal(text: string) {
lower.includes("handoff") || lower.includes("model switch") || lower.includes("switched");
const mentionsKickoffTask =
lower.includes("qa_kickoff_task") ||
lower.includes("qa/scenarios/index.md") ||
lower.includes("qa/scenarios/index.yaml") ||
lower.includes("scenario pack") ||
lower.includes("kickoff task") ||
lower.includes("kickoff note") ||
extensions/qa-lab/src/providers/mock-openai/server.test.ts
+2 -2
View File
@@ -2490,7 +2490,7 @@ describe("qa mock openai server", () => {
{
type: "function_call_output",
output:
"repo/qa/scenarios/index.md includes scenario: subagent-handoff and repo/extensions/qa-lab/src/suite.ts.",
"repo/qa/scenarios/index.yaml includes scenario: subagent-handoff and repo/extensions/qa-lab/src/suite.ts.",
},
makeUserInput("Continue."),
],
@@ -4593,7 +4593,7 @@ describe("qa mock openai server provider variant tagging", () => {
stream: false,
input: [makeUserInput(sourcePrompt)],
});
expect(outputToolArgs(openaiSource)).toEqual({ path: "repo/qa/scenarios/index.md" });
expect(outputToolArgs(openaiSource)).toEqual({ path: "repo/qa/scenarios/index.yaml" });
const anthropicSourceServer = await startMockServer();
const anthropicSource = await expectResponsesJson(anthropicSourceServer, {
extensions/qa-lab/src/providers/mock-openai/server.ts
+2 -2
View File
@@ -198,7 +198,7 @@ type MockScenarioState = {
function sourceDiscoveryReadPathForProvider(providerVariant: MockOpenAiProviderVariant) {
return providerVariant === "anthropic"
? "repo/docs/help/testing.md"
: "repo/qa/scenarios/index.md";
: "repo/qa/scenarios/index.yaml";
}
function subagentHandoffTaskForProvider(providerVariant: MockOpenAiProviderVariant) {
@@ -1461,7 +1461,7 @@ function buildAssistantText(
) {
return [
"Worked:",
"- Read all three seeded files: repo/qa/scenarios/index.md, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.",
"- Read all three seeded files: repo/qa/scenarios/index.yaml, repo/extensions/qa-lab/src/suite.ts, and repo/docs/help/testing.md.",
"- Extra QA scenario candidates: config restart capability flip and image generation roundtrip.",
"Failed:",
"- None observed in mock mode.",
extensions/qa-lab/src/qa-agent-workspace.ts
+6 -3
View File
@@ -2,7 +2,10 @@
import fs from "node:fs/promises";
import path from "node:path";
import { buildQaScenarioPlanMarkdown, readQaAgentIdentityMarkdown } from "./qa-agent-bootstrap.js";
import { readQaBootstrapScenarioCatalog, readQaScenarioPackMarkdown } from "./scenario-catalog.js";
import {
readQaBootstrapScenarioCatalog,
readQaScenarioPackYamlSource,
} from "./scenario-catalog.js";
export async function seedQaAgentWorkspace(params: { workspaceDir: string; repoRoot?: string }) {
const catalog = readQaBootstrapScenarioCatalog();
@@ -13,7 +16,7 @@ export async function seedQaAgentWorkspace(params: { workspaceDir: string; repoR
["IDENTITY.md", readQaAgentIdentityMarkdown()],
["QA_KICKOFF_TASK.md", kickoffTask],
["QA_SCENARIO_PLAN.md", buildQaScenarioPlanMarkdown()],
["QA_SCENARIOS.md", readQaScenarioPackMarkdown()],
["QA_SCENARIOS.yaml", readQaScenarioPackYamlSource()],
]);
if (params.repoRoot) {
@@ -24,7 +27,7 @@ export async function seedQaAgentWorkspace(params: { workspaceDir: string; repoR
- repo: ./repo/
- kickoff: ./QA_KICKOFF_TASK.md
- scenario plan: ./QA_SCENARIO_PLAN.md
- scenario pack: ./QA_SCENARIOS.md
- scenario pack: ./QA_SCENARIOS.yaml
- identity: ./IDENTITY.md
The mounted repo source should be available read-only under \`./repo/\`.
extensions/qa-lab/src/scenario-catalog.test.ts
+42 -24
View File
@@ -1,8 +1,9 @@
// Qa Lab tests cover scenario catalog plugin behavior.
import fs from "node:fs";
import { describe, expect, it } from "vitest";
import { QA_AGENTIC_PARITY_SCENARIO_IDS } from "./agentic-parity.js";
import {
listQaScenarioMarkdownPaths,
listQaScenarioYamlPaths,
readQaBootstrapScenarioCatalog,
readQaScenarioById,
readQaScenarioExecutionConfig,
@@ -10,16 +11,33 @@ import {
validateQaScenarioExecutionConfig,
} from "./scenario-catalog.js";
function listScenarioMarkdownPaths(dir = "qa/scenarios"): string[] {
return fs
.readdirSync(dir, { withFileTypes: true })
.flatMap((entry) => {
const entryPath = `${dir}/${entry.name}`;
if (entry.isDirectory()) {
return listScenarioMarkdownPaths(entryPath);
}
return entry.isFile() && entry.name.endsWith(".md") ? [entryPath] : [];
})
.toSorted();
}
describe("qa scenario catalog", () => {
it("loads the markdown pack as the canonical source of truth", () => {
it("keeps repo-backed scenarios YAML-only", () => {
expect(listScenarioMarkdownPaths()).toStrictEqual([]);
});
it("loads the YAML pack as the canonical source of truth", () => {
const pack = readQaScenarioPack();
expect(pack.version).toBe(1);
expect(pack.agent.identityMarkdown).toContain("Dev C-3PO");
expect(pack.kickoffTask).toContain("Lobster Invaders");
expect(listQaScenarioMarkdownPaths().length).toBe(pack.scenarios.length);
expect(listQaScenarioMarkdownPaths()).toContain(
"qa/scenarios/media/image-generation-roundtrip.md",
expect(listQaScenarioYamlPaths().length).toBe(pack.scenarios.length);
expect(listQaScenarioYamlPaths()).toContain(
"qa/scenarios/media/image-generation-roundtrip.yaml",
);
const scenarioIds = pack.scenarios.map((scenario) => scenario.id);
const requiredScenarioIds = [
@@ -48,7 +66,7 @@ describe("qa scenario catalog", () => {
expect(readQaScenarioById("memory-recall").coverage?.primary).toContain("memory.recall");
});
it("exposes bootstrap data from the markdown pack", () => {
it("exposes bootstrap data from the YAML pack", () => {
const catalog = readQaBootstrapScenarioCatalog();
expect(catalog.agentIdentityMarkdown).toContain("protocol-minded");
@@ -60,7 +78,7 @@ describe("qa scenario catalog", () => {
).toStrictEqual([]);
});
it("loads scenario-specific execution config from per-scenario markdown", () => {
it("loads scenario-specific execution config from per-scenario YAML", () => {
const discovery = readQaScenarioById("source-docs-discovery-report");
const discoveryConfig = readQaScenarioExecutionConfig("source-docs-discovery-report");
const codexLeak = readQaScenarioById("codex-harness-no-meta-leak");
@@ -82,7 +100,7 @@ describe("qa scenario catalog", () => {
expect(discovery.title).toBe("Source and docs discovery report");
expect((discoveryConfig?.requiredFiles as string[] | undefined)?.[0]).toBe(
"repo/qa/scenarios/index.md",
"repo/qa/scenarios/index.yaml",
);
expect(codexLeak.title).toBe("Codex harness no meta leak");
expect(codexLeakConfig?.harnessRuntime).toBe("codex");
@@ -105,13 +123,13 @@ describe("qa scenario catalog", () => {
expect(fanoutConfig?.expectedReplyGroups?.flat()).toContain("subagent-2: ok");
});
it("loads scenario-declared gateway runtime options from markdown", () => {
it("loads scenario-declared gateway runtime options from YAML", () => {
const scenario = readQaScenarioById("control-ui-qa-channel-image-roundtrip");
expect(scenario.gatewayRuntime?.forwardHostHome).toBe(true);
});
it("loads native test execution scenarios from markdown", () => {
it("loads native test execution scenarios from YAML", () => {
const scenario = readQaScenarioById("control-ui-chat-flow-playwright");
expect(scenario.execution.kind).toBe("playwright");
@@ -195,7 +213,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/codex-legacy-read-tool-vocabulary.md");
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/codex-legacy-read-tool-vocabulary.yaml");
expect(scenario.runtimeParityTier).toBe("live-only");
expect(config?.runtimeParityComparison).toBe("codex-native-workspace");
expect(config?.fixtureFile).toBe("LEGACY_READ_TOOL_FIXTURE.txt");
@@ -220,10 +238,10 @@ describe("qa scenario catalog", () => {
expect(scenario.coverage?.primary.length).toBeGreaterThan(0);
}
expect(readQaScenarioById("webchat-direct-reply-routing").sourcePath).toBe(
"qa/scenarios/channels/webchat-direct-reply-routing.md",
"qa/scenarios/channels/webchat-direct-reply-routing.yaml",
);
expect(readQaScenarioById("long-context-progress-watchdog").sourcePath).toBe(
"qa/scenarios/runtime/long-context-progress-watchdog.md",
"qa/scenarios/runtime/long-context-progress-watchdog.yaml",
);
expect(
JSON.stringify(readQaScenarioById("gateway-restart-inflight-run").execution.flow),
@@ -255,7 +273,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/qa-bus-tool-trace-visibility.md");
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/qa-bus-tool-trace-visibility.yaml");
expect(scenario.coverage?.primary).toContain("harness.tool-trace-visibility");
expect(scenario.coverage?.secondary ?? []).toStrictEqual(["runtime.qa-bus", "tools.trace"]);
expect(config?.expectedToolName).toBe("exec");
@@ -277,7 +295,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/update-run-package-self-upgrade.md");
expect(scenario.sourcePath).toBe("qa/scenarios/runtime/update-run-package-self-upgrade.yaml");
expect(scenario.coverage?.primary).toContain("runtime.update-run");
expect(scenario.coverage?.secondary).toContain("runtime.package-update");
expect(config?.requiredProviderMode).toBe("live-frontier");
@@ -334,13 +352,13 @@ describe("qa scenario catalog", () => {
).toBe(true);
});
it("includes the codex leak scenario in the markdown pack", () => {
it("includes the codex leak scenario in the YAML pack", () => {
const pack = readQaScenarioPack();
const scenario = pack.scenarios.find(
(candidate) => candidate.id === "codex-harness-no-meta-leak",
);
expect(scenario?.sourcePath).toBe("qa/scenarios/models/codex-harness-no-meta-leak.md");
expect(scenario?.sourcePath).toBe("qa/scenarios/models/codex-harness-no-meta-leak.yaml");
expect(scenario?.execution.flow?.steps.map((step) => step.name)).toContain(
"keeps codex coordination chatter out of the visible reply",
);
@@ -358,7 +376,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/models/gpt55-thinking-visibility-switch.md");
expect(scenario.sourcePath).toBe("qa/scenarios/models/gpt55-thinking-visibility-switch.yaml");
expect(config?.requiredProvider).toBe("openai");
expect(config?.requiredModel).toBe("gpt-5.5");
expect(config?.offDirective).toBe("/think off");
@@ -381,7 +399,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/models/openai-native-web-search-live.md");
expect(scenario.sourcePath).toBe("qa/scenarios/models/openai-native-web-search-live.yaml");
expect(scenario.gatewayConfigPatch?.tools).toEqual({
web: {
search: {
@@ -414,7 +432,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/plugins/kitchen-sink-live-openai.md");
expect(scenario.sourcePath).toBe("qa/scenarios/plugins/kitchen-sink-live-openai.yaml");
expect(config?.requiredProviderMode).toBe("live-frontier");
expect(config?.requiredProvider).toBe("openai");
expect(config?.pluginSpec).toBe("npm:@openclaw/kitchen-sink@latest");
@@ -470,7 +488,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe("qa/scenarios/models/thinking-slash-model-remap.md");
expect(scenario.sourcePath).toBe("qa/scenarios/models/thinking-slash-model-remap.yaml");
expect(config?.requiredProviderMode).toBe("live-frontier");
expect(config?.anthropicModelRef).toBe("anthropic/claude-sonnet-4-6");
expect(config?.openAiXhighModelRef).toBe("openai/gpt-5.5");
@@ -482,7 +500,7 @@ describe("qa scenario catalog", () => {
]);
});
it("includes the seeded mock-only broken-turn scenarios in the markdown pack", () => {
it("includes the seeded mock-only broken-turn scenarios in the YAML pack", () => {
const scenarioIds = [
"reasoning-only-recovery-replay-safe-read",
"reasoning-only-no-auto-retry-after-write",
@@ -499,7 +517,7 @@ describe("qa scenario catalog", () => {
}
| undefined;
expect(scenario.sourcePath).toBe(`qa/scenarios/runtime/${scenarioId}.md`);
expect(scenario.sourcePath).toBe(`qa/scenarios/runtime/${scenarioId}.yaml`);
expect(config?.requiredProvider).toBe("mock-openai");
expect(config?.prompt).toContain("check");
expect(scenario.execution.flow?.steps.length).toBeGreaterThan(0);
@@ -567,7 +585,7 @@ describe("qa scenario catalog", () => {
| undefined;
const flow = JSON.stringify(scenario.execution.flow);
expect(scenario.sourcePath).toBe("qa/scenarios/memory/dreaming-shadow-trial-report.md");
expect(scenario.sourcePath).toBe("qa/scenarios/memory/dreaming-shadow-trial-report.yaml");
expect(scenario.coverage?.primary).toContain("memory.dreaming");
expect(config?.prompt).toContain("Dreaming shadow trial report check");
expect(config?.reportName).toBe("dreaming-shadow-trial-report.md");
extensions/qa-lab/src/scenario-catalog.ts
+52 -63
View File
@@ -203,9 +203,8 @@ const qaFlowSchema = z.object({
steps: z.array(qaFlowStepSchema).min(1),
});
const qaSeedScenarioSchema = z.object({
const qaSeedScenarioBodySchema = z.object({
id: z.string().trim().min(1),
title: z.string().trim().min(1),
surface: z.string().trim().min(1),
category: z.string().trim().min(1).optional(),
runtimeParityTier: qaRuntimeParityTierSchema.optional(),
@@ -225,6 +224,16 @@ const qaSeedScenarioSchema = z.object({
execution: qaScenarioExecutionSchema.optional(),
});
const qaSeedScenarioSchema = qaSeedScenarioBodySchema.extend({
title: z.string().trim().min(1),
});
const qaScenarioFileSchema = z.object({
title: z.string().trim().min(1),
scenario: qaSeedScenarioBodySchema,
flow: qaFlowSchema.optional(),
});
const qaScenarioPackSchema = z.object({
version: z.number().int().positive(),
agent: z
@@ -237,6 +246,11 @@ const qaScenarioPackSchema = z.object({
kickoffTask: z.string().trim().min(1),
});
const qaScenarioPackFileSchema = z.object({
title: z.string().trim().min(1),
pack: qaScenarioPackSchema,
});
export type QaScenarioExecution = z.infer<typeof qaScenarioExecutionSchema>;
export type QaScenarioFlow = z.infer<typeof qaFlowSchema>;
export type QaRuntimeParityTier = z.infer<typeof qaRuntimeParityTierSchema>;
@@ -266,14 +280,11 @@ export {
type QaScenarioPackDefinition,
} from "./scenario-packs.js";
const QA_SCENARIO_PACK_INDEX_PATH = "qa/scenarios/index.md";
const QA_SCENARIO_PACK_INDEX_PATH = "qa/scenarios/index.yaml";
const QA_SCENARIO_LEGACY_OVERVIEW_PATH = "qa/scenarios.md";
const QA_SCENARIO_DIR_PATH = "qa/scenarios";
const QA_PACK_FENCE_RE = /```ya?ml qa-pack\r?\n([\s\S]*?)\r?\n```/i;
const QA_SCENARIO_FENCE_RE = /```ya?ml qa-scenario\r?\n([\s\S]*?)\r?\n```/i;
const QA_FLOW_YAML_FENCE_RE = /```ya?ml qa-flow\r?\n([\s\S]*?)\r?\n```/i;
const repoPathCache = new Map<string, string | null>();
let qaScenarioMarkdownPathsCache: string[] | null = null;
let qaScenarioYamlPathsCache: string[] | null = null;
let qaScenarioPackCache: QaScenarioPack | null = null;
function walkUpDirectories(start: string): string[] {
@@ -321,32 +332,6 @@ function readTextFile(relativePath: string): string {
return fs.readFileSync(resolved, "utf8");
}
function extractQaPackYaml(content: string) {
const match = content.match(QA_PACK_FENCE_RE);
if (!match?.[1]) {
throw new Error(
`qa scenario pack missing \`\`\`yaml qa-pack fence in ${QA_SCENARIO_PACK_INDEX_PATH}`,
);
}
return match[1];
}
function extractQaScenarioYaml(content: string, relativePath: string) {
const match = content.match(QA_SCENARIO_FENCE_RE);
if (!match?.[1]) {
throw new Error(`qa scenario file missing \`\`\`yaml qa-scenario fence in ${relativePath}`);
}
return match[1];
}
function extractQaScenarioFlow(content: string, relativePath: string) {
const match = content.match(QA_FLOW_YAML_FENCE_RE);
if (!match?.[1]) {
throw new Error(`qa scenario file missing \`\`\`yaml qa-flow fence in ${relativePath}`);
}
return parseQaYamlWithContext(qaFlowSchema, YAML.parse(match[1]) as unknown, relativePath);
}
function formatZodIssuePath(pathLocal: PropertyKey[]) {
return pathLocal.length ? pathLocal.map(String).join(".") : "<root>";
}
@@ -362,20 +347,28 @@ function parseQaYamlWithContext<T>(schema: z.ZodType<T>, value: unknown, label:
throw new Error(`${label}: ${issues}`);
}
export function readQaScenarioPackMarkdown(): string {
function parseQaYamlFileWithContext<T>(schema: z.ZodType<T>, relativePath: string): T {
return parseQaYamlWithContext(
schema,
YAML.parse(readTextFile(relativePath)) as unknown,
relativePath,
);
}
export function readQaScenarioPackYamlSource(): string {
const chunks = [readTextFile(QA_SCENARIO_PACK_INDEX_PATH).trim()];
for (const relativePath of listQaScenarioMarkdownPaths()) {
for (const relativePath of listQaScenarioYamlPaths()) {
chunks.push(readTextFile(relativePath).trim());
}
return chunks.filter(Boolean).join("\n\n");
return chunks.filter(Boolean).join("\n---\n");
}
export function readQaScenarioPack(): QaScenarioPack {
if (qaScenarioPackCache) {
return qaScenarioPackCache;
}
const packMarkdown = readTextFile(QA_SCENARIO_PACK_INDEX_PATH).trim();
if (!packMarkdown) {
const packYaml = readTextFile(QA_SCENARIO_PACK_INDEX_PATH).trim();
if (!packYaml) {
// The QA scenario pack is optional in npm distributions. Return an empty
// pack so completion cache updates and other consumers don't crash when
// the qa/scenarios/ directory is not shipped with the package.
@@ -387,32 +380,31 @@ export function readQaScenarioPack(): QaScenarioPack {
};
return qaScenarioPackCache;
}
const parsedPack = parseQaYamlWithContext(
qaScenarioPackSchema,
YAML.parse(extractQaPackYaml(packMarkdown)) as unknown,
const parsedPackFile = parseQaYamlFileWithContext(
qaScenarioPackFileSchema,
QA_SCENARIO_PACK_INDEX_PATH,
);
const scenarios = listQaScenarioMarkdownPaths().map((relativePath) =>
const scenarios = listQaScenarioYamlPaths().map((relativePath) =>
(() => {
const content = readTextFile(relativePath);
const parsedScenario = parseQaYamlWithContext(
qaSeedScenarioSchema,
YAML.parse(extractQaScenarioYaml(content, relativePath)) as unknown,
relativePath,
);
const parsedScenarioFile = parseQaYamlFileWithContext(qaScenarioFileSchema, relativePath);
const parsedScenario = {
...parsedScenarioFile.scenario,
title: parsedScenarioFile.title,
};
const execution = parseQaYamlWithContext(
qaScenarioExecutionSchema,
parsedScenario.execution ?? {},
relativePath,
);
const flow =
execution.kind === "flow" ? extractQaScenarioFlow(content, relativePath) : undefined;
if (execution.kind === "flow" && !parsedScenarioFile.flow) {
throw new Error(`${relativePath}: flow scenarios must define a top-level flow block`);
}
return {
...parsedScenario,
sourcePath: relativePath,
execution: {
...execution,
...(flow ? { flow } : {}),
...(parsedScenarioFile.flow ? { flow: parsedScenarioFile.flow } : {}),
},
} satisfies QaSeedScenarioWithSource;
})(),
@@ -425,31 +417,28 @@ export function readQaScenarioPack(): QaScenarioPack {
seenScenarioIds.add(scenario.id);
}
qaScenarioPackCache = {
...parsedPack,
...parsedPackFile.pack,
scenarios,
};
return qaScenarioPackCache;
}
export function listQaScenarioMarkdownPaths(): string[] {
if (qaScenarioMarkdownPathsCache) {
return qaScenarioMarkdownPathsCache;
export function listQaScenarioYamlPaths(): string[] {
if (qaScenarioYamlPathsCache) {
return qaScenarioYamlPathsCache;
}
const resolved = resolveRepoPath(QA_SCENARIO_DIR_PATH, "directory");
if (!resolved) {
return [];
}
qaScenarioMarkdownPathsCache = listQaScenarioMarkdownPathsInDirectory(
qaScenarioYamlPathsCache = listQaScenarioYamlPathsInDirectory(
resolved,
QA_SCENARIO_DIR_PATH,
).toSorted();
return qaScenarioMarkdownPathsCache;
return qaScenarioYamlPathsCache;
}
function listQaScenarioMarkdownPathsInDirectory(
absoluteDir: string,
relativeDir: string,
): string[] {
function listQaScenarioYamlPathsInDirectory(absoluteDir: string, relativeDir: string): string[] {
const paths: string[] = [];
const entries = fs
.readdirSync(absoluteDir, { withFileTypes: true })
@@ -461,11 +450,11 @@ function listQaScenarioMarkdownPathsInDirectory(
const relativePath = `${relativeDir}/${entry.name}`;
if (entry.isDirectory()) {
paths.push(
...listQaScenarioMarkdownPathsInDirectory(path.join(absoluteDir, entry.name), relativePath),
...listQaScenarioYamlPathsInDirectory(path.join(absoluteDir, entry.name), relativePath),
);
continue;
}
if (entry.isFile() && entry.name.endsWith(".md") && entry.name !== "index.md") {
if (entry.isFile() && entry.name.endsWith(".yaml") && entry.name !== "index.yaml") {
paths.push(relativePath);
}
}
extensions/qa-lab/src/scenario-flow-runner.test.ts
+3 -3
View File
@@ -102,7 +102,7 @@ describe("scenario-flow-runner", () => {
scenario: {
id: "qa-import",
title: "qa-import",
sourcePath: "qa/scenarios/qa-import.md",
sourcePath: "qa/scenarios/qa-import.yaml",
surface: "test",
objective: "test",
successCriteria: ["test"],
@@ -173,7 +173,7 @@ describe("scenario-flow-runner", () => {
scenario: {
id: "qa-fixture-import",
title: "qa-fixture-import",
sourcePath: "qa/scenarios/qa-fixture-import.md",
sourcePath: "qa/scenarios/qa-fixture-import.yaml",
surface: "test",
objective: "test",
successCriteria: ["test"],
@@ -235,7 +235,7 @@ describe("scenario-flow-runner", () => {
scenario: {
id: "qa-gated-promise",
title: "qa-gated-promise",
sourcePath: "qa/scenarios/qa-gated-promise.md",
sourcePath: "qa/scenarios/qa-gated-promise.yaml",
surface: "test",
objective: "test",
successCriteria: ["test"],
extensions/qa-lab/src/scenario-packs.test.ts
+1 -1
View File
@@ -9,7 +9,7 @@ import {
} from "./scenario-catalog.js";
describe("qa scenario packs", () => {
it("points every pack scenario id at a loadable markdown scenario", () => {
it("points every pack scenario id at a loadable YAML scenario", () => {
expect(QA_SCENARIO_PACKS.length).toBeGreaterThan(0);
for (const pack of QA_SCENARIO_PACKS) {
extensions/qa-lab/src/scenario-runtime-api.test.ts
+1 -1
View File
@@ -139,7 +139,7 @@ describe("createQaScenarioRuntimeApi", () => {
surface: "test",
objective: "test",
successCriteria: ["works"],
sourcePath: "qa/scenarios/generic-flow.md",
sourcePath: "qa/scenarios/generic-flow.yaml",
execution: {
kind: "flow" as const,
config: { expected: "value" },
extensions/qa-lab/src/suite-runtime-flow.test.ts
+1 -1
View File
@@ -208,7 +208,7 @@ describe("qa suite runtime flow", () => {
const scenario = {
id: "session-memory-ranking",
title: "Session memory ranking",
sourcePath: "qa/scenarios/session-memory-ranking.md",
sourcePath: "qa/scenarios/session-memory-ranking.yaml",
surface: "qa-channel",
objective: "test",
successCriteria: ["test"],
extensions/qa-lab/src/suite-test-helpers.ts
+1 -1
View File
@@ -24,7 +24,7 @@ export function makeQaSuiteTestScenario(
...(params.plugins ? { plugins: params.plugins } : {}),
...(params.gatewayConfigPatch ? { gatewayConfigPatch: params.gatewayConfigPatch } : {}),
...(params.gatewayRuntime ? { gatewayRuntime: params.gatewayRuntime } : {}),
sourcePath: `qa/scenarios/${id}.md`,
sourcePath: `qa/scenarios/${id}.yaml`,
execution: {
kind: "flow",
...(params.config ? { config: params.config } : {}),
extensions/qa-lab/src/suite.summary-json.test.ts
+1 -1
View File
@@ -111,7 +111,7 @@ describe("buildQaSuiteSummaryJson", () => {
{
id: "dm-chat-baseline",
title: "DM baseline conversation",
sourcePath: "qa/scenarios/channels/dm-chat-baseline.md",
sourcePath: "qa/scenarios/channels/dm-chat-baseline.yaml",
surface: "dm",
coverage: {
primary: ["channels.dm"],
extensions/qa-lab/src/suite.ts
+1 -1
View File
@@ -1427,7 +1427,7 @@ export async function runQaFlowSuite(params?: QaSuiteRunParams): Promise<QaSuite
lab,
mock,
gateway,
// Markdown scenarios should see the full staged gateway config, not just
// YAML scenarios should see the full staged gateway config, not just
// the transport fragment. Routing/session/plugin assertions depend on it.
cfg: gateway.cfg,
transport,
extensions/qa-lab/src/tool-coverage-report.test.ts
+1 -1
View File
@@ -23,7 +23,7 @@ function makeScenario(
},
objective: "exercise tool",
successCriteria: ["tool is exercised"],
sourcePath: `qa/scenarios/runtime/tools/${tool}.md`,
sourcePath: `qa/scenarios/runtime/tools/${tool}.yaml`,
execution: {
kind: "flow",
config: {
qa/README.md
+3 -3
View File
@@ -4,8 +4,8 @@ Seed QA assets for the private `qa-lab` extension.
Files:
- `scenarios/index.md` - canonical QA scenario pack, kickoff mission, and operator identity.
- `scenarios/<theme>/*.md` - one runnable scenario per markdown file.
- `scenarios/index.yaml` - canonical QA scenario pack, kickoff mission, and operator identity.
- `scenarios/<theme>/*.yaml` - one runnable scenario per YAML file.
- `frontier-harness-plan.md` - big-model bakeoff and tuning loop for harness work.
- `convex-credential-broker/` - standalone Convex v1 lease broker for pooled live credentials.
@@ -13,7 +13,7 @@ Key workflow:
- `qa suite` is the executable frontier subset / regression loop.
- `qa manual` is the scoped personality and style probe after the executable subset is green.
- `qa coverage` prints the scenario coverage inventory from scenario frontmatter.
- `qa coverage` prints the scenario coverage inventory from scenario YAML.
Operator workflows:
qa/scenarios.md
+3 -3
View File
@@ -2,7 +2,7 @@
Canonical scenario source now lives in:
- `qa/scenarios/index.md`
- `qa/scenarios/<theme>/*.md`
- `qa/scenarios/index.yaml`
- `qa/scenarios/<theme>/*.yaml`
Each QA scenario has its own markdown file.
Each QA scenario has its own YAML file.
qa/scenarios/agents/instruction-followthrough-repo-contract.md
-145
View File
@@ -1,145 +0,0 @@
# Instruction followthrough repo contract
```yaml qa-scenario
id: instruction-followthrough-repo-contract
title: Instruction followthrough repo contract
surface: repo-contract
coverage:
primary:
- agents.instructions
secondary:
- runtime.first-action
objective: Verify the agent reads repo instruction files first, follows the required tool order, and completes the first feasible action instead of stopping at a plan.
successCriteria:
- Agent reads the seeded instruction files before writing the requested artifact.
- Agent writes the requested artifact in the same run instead of returning only a plan.
- Agent does not ask for permission before the first feasible action.
- Final reply makes the completed read/write sequence explicit.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- src/agents/system-prompt.ts
- src/agents/embedded-agent-runner/run/incomplete-turn.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify the agent reads repo instructions first, then completes the first bounded followthrough task without stalling.
config:
workspaceFiles:
AGENT.md: |-
# Repo contract
Step order:
1. Read AGENT.md.
2. Read SOUL.md.
3. Read FOLLOWTHROUGH_INPUT.md.
4. Write ./repo-contract-summary.txt.
5. Reply with three labeled lines exactly once: Read, Wrote, Status.
Do not stop after planning.
Do not ask for permission before the first feasible action.
SOUL.md: |-
# Execution style
Stay brief, honest, and action-first.
If the next tool action is feasible, do it before replying.
FOLLOWTHROUGH_INPUT.md: |-
Mission: prove you followed the repo contract.
Evidence path: AGENT.md -> SOUL.md -> FOLLOWTHROUGH_INPUT.md -> repo-contract-summary.txt
prompt: |-
Repo contract followthrough check. Read AGENT.md, SOUL.md, and FOLLOWTHROUGH_INPUT.md first.
Then follow the repo contract exactly, write ./repo-contract-summary.txt, and reply with
three labeled lines: Read, Wrote, Status.
Do not stop after planning and do not ask for permission before the first feasible action.
expectedReplyAll:
- "read:"
- "wrote:"
- "status:"
expectedArtifactAll:
- "repo contract"
expectedArtifactAny:
- "evidence path"
- "agent.md"
- "followthrough"
forbiddenNeedles:
- need permission
- need your approval
- can you approve
- i would
- i can
- next i would
```
```yaml qa-flow
steps:
- name: follows repo instructions instead of stopping at a plan
actions:
- call: reset
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- set: artifactPath
value:
expr: "path.join(env.gateway.workspaceDir, 'repo-contract-summary.txt')"
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:repo-contract
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 40000)
- call: waitForCondition
saveAs: artifact
args:
- lambda:
async: true
expr: "(() => { const normalize = (value) => normalizeLowercaseStringOrEmpty(value); const matches = (value) => { const normalized = normalize(value); return normalized && config.expectedArtifactAll.every((needle) => normalized.includes(normalize(needle))) && config.expectedArtifactAny.some((needle) => normalized.includes(normalize(needle))); }; return fs.readFile(artifactPath, 'utf8').then((value) => matches(value) ? value : undefined).catch(() => undefined); })()"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- set: normalizedArtifact
value:
expr: "normalizeLowercaseStringOrEmpty(artifact)"
- assert:
expr: "config.expectedArtifactAll.every((needle) => normalizedArtifact.includes(normalizeLowercaseStringOrEmpty(needle))) && config.expectedArtifactAny.some((needle) => normalizedArtifact.includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`repo contract artifact missing expected followthrough signals: ${artifact}`"
- set: expectedReplyAll
value:
expr: config.expectedReplyAll.map(normalizeLowercaseStringOrEmpty)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && expectedReplyAll.every((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))).at(-1)"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`repo contract followthrough bounced for permission or stalled: ${outbound.text}`"
- set: followthroughDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => /repo contract followthrough check/i.test(String(request.allInputText ?? ''))) : []"
- assert:
expr: "!env.mock || followthroughDebugRequests.filter((request) => request.plannedToolName === 'read').length >= 3"
message:
expr: "`expected three read tool calls before write, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || followthroughDebugRequests.some((request) => request.plannedToolName === 'write')"
message:
expr: "`expected write tool call during repo contract followthrough, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || (() => { const readIndices = followthroughDebugRequests.map((r, i) => r.plannedToolName === 'read' ? i : -1).filter(i => i >= 0); const firstWrite = followthroughDebugRequests.findIndex((r) => r.plannedToolName === 'write'); return readIndices.length >= 3 && firstWrite >= 0 && readIndices[2] < firstWrite; })()"
message:
expr: "`expected all 3 reads before any write during repo contract followthrough, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
```
qa/scenarios/agents/instruction-followthrough-repo-contract.yaml
+142
View File
@@ -0,0 +1,142 @@
title: Instruction followthrough repo contract
scenario:
id: instruction-followthrough-repo-contract
surface: repo-contract
coverage:
primary:
- agents.instructions
secondary:
- runtime.first-action
objective: Verify the agent reads repo instruction files first, follows the required tool order, and completes the first feasible action instead of stopping at a plan.
successCriteria:
- Agent reads the seeded instruction files before writing the requested artifact.
- Agent writes the requested artifact in the same run instead of returning only a plan.
- Agent does not ask for permission before the first feasible action.
- Final reply makes the completed read/write sequence explicit.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- src/agents/system-prompt.ts
- src/agents/embedded-agent-runner/run/incomplete-turn.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify the agent reads repo instructions first, then completes the first bounded followthrough task without stalling.
config:
workspaceFiles:
AGENT.md: |-
# Repo contract
Step order:
1. Read AGENT.md.
2. Read SOUL.md.
3. Read FOLLOWTHROUGH_INPUT.md.
4. Write ./repo-contract-summary.txt.
5. Reply with three labeled lines exactly once: Read, Wrote, Status.
Do not stop after planning.
Do not ask for permission before the first feasible action.
SOUL.md: |-
# Execution style
Stay brief, honest, and action-first.
If the next tool action is feasible, do it before replying.
FOLLOWTHROUGH_INPUT.md: |-
Mission: prove you followed the repo contract.
Evidence path: AGENT.md -> SOUL.md -> FOLLOWTHROUGH_INPUT.md -> repo-contract-summary.txt
prompt: |-
Repo contract followthrough check. Read AGENT.md, SOUL.md, and FOLLOWTHROUGH_INPUT.md first.
Then follow the repo contract exactly, write ./repo-contract-summary.txt, and reply with
three labeled lines: Read, Wrote, Status.
Do not stop after planning and do not ask for permission before the first feasible action.
expectedReplyAll:
- "read:"
- "wrote:"
- "status:"
expectedArtifactAll:
- "repo contract"
expectedArtifactAny:
- "evidence path"
- "agent.md"
- "followthrough"
forbiddenNeedles:
- need permission
- need your approval
- can you approve
- i would
- i can
- next i would
flow:
steps:
- name: follows repo instructions instead of stopping at a plan
actions:
- call: reset
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- set: artifactPath
value:
expr: "path.join(env.gateway.workspaceDir, 'repo-contract-summary.txt')"
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:repo-contract
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 40000)
- call: waitForCondition
saveAs: artifact
args:
- lambda:
async: true
expr: "(() => { const normalize = (value) => normalizeLowercaseStringOrEmpty(value); const matches = (value) => { const normalized = normalize(value); return normalized && config.expectedArtifactAll.every((needle) => normalized.includes(normalize(needle))) && config.expectedArtifactAny.some((needle) => normalized.includes(normalize(needle))); }; return fs.readFile(artifactPath, 'utf8').then((value) => matches(value) ? value : undefined).catch(() => undefined); })()"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- set: normalizedArtifact
value:
expr: "normalizeLowercaseStringOrEmpty(artifact)"
- assert:
expr: "config.expectedArtifactAll.every((needle) => normalizedArtifact.includes(normalizeLowercaseStringOrEmpty(needle))) && config.expectedArtifactAny.some((needle) => normalizedArtifact.includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`repo contract artifact missing expected followthrough signals: ${artifact}`"
- set: expectedReplyAll
value:
expr: config.expectedReplyAll.map(normalizeLowercaseStringOrEmpty)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && expectedReplyAll.every((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))).at(-1)"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`repo contract followthrough bounced for permission or stalled: ${outbound.text}`"
- set: followthroughDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => /repo contract followthrough check/i.test(String(request.allInputText ?? ''))) : []"
- assert:
expr: "!env.mock || followthroughDebugRequests.filter((request) => request.plannedToolName === 'read').length >= 3"
message:
expr: "`expected three read tool calls before write, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || followthroughDebugRequests.some((request) => request.plannedToolName === 'write')"
message:
expr: "`expected write tool call during repo contract followthrough, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || (() => { const readIndices = followthroughDebugRequests.map((r, i) => r.plannedToolName === 'read' ? i : -1).filter(i => i >= 0); const firstWrite = followthroughDebugRequests.findIndex((r) => r.plannedToolName === 'write'); return readIndices.length >= 3 && firstWrite >= 0 && readIndices[2] < firstWrite; })()"
message:
expr: "`expected all 3 reads before any write during repo contract followthrough, saw plannedToolNames=${JSON.stringify(followthroughDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
qa/scenarios/agents/subagent-completion-direct-fallback.md
-111
View File
@@ -1,111 +0,0 @@
# Subagent completion direct fallback
```yaml qa-scenario
id: subagent-completion-direct-fallback
title: Subagent completion direct fallback
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- runtime.delivery
- channels.qa-channel
objective: Verify a yielded parent still receives a successful subagent result through direct fallback delivery when the dormant announce turn produces no visible reply.
successCriteria:
- Parent launches a native subagent.
- Parent yields instead of waiting in-turn.
- Subagent completion result is delivered to the original QA DM without a thread id.
- Durable task delivery is marked delivered, not failed.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- src/agents/subagent-announce-delivery.ts
- src/agents/subagent-registry-lifecycle.ts
- src/agents/tools/sessions-yield-tool.ts
- extensions/qa-lab/src/providers/mock-openai/server.ts
execution:
kind: flow
summary: Reproduce yielded-parent subagent completion delivery and require frozen-result fallback to the QA DM.
config:
prompt: "Subagent direct fallback QA check: spawn one native subagent worker. The worker must finish with exactly QA-SUBAGENT-DIRECT-FALLBACK-OK. After spawning it, call sessions_yield and wait for the completion event. Do not use ACP."
expectedMarker: QA-SUBAGENT-DIRECT-FALLBACK-OK
expectedLabel: qa-direct-fallback-worker
```
```yaml qa-flow
steps:
- name: yielded parent receives child completion through direct fallback
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
- call: waitForQaChannelReady
args:
- ref: env
- 120000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:subagent-direct-fallback:${randomUUID().slice(0, 8)}`"
- try:
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && String(message.text ?? '').includes(config.expectedMarker)).at(-1)"
- expr: liveTurnTimeoutMs(env, 180000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "String(outbound.text ?? '').trim().includes(config.expectedMarker)"
message:
expr: "`fallback completion marker missing from outbound QA DM: ${recentOutboundSummary(state)}`"
catchAs: fallbackError
catch:
- set: fallbackDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].slice(-20).map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, prompt: String(request.prompt ?? '').slice(0, 280), allInputText: String(request.allInputText ?? '').slice(0, 280), toolOutput: request.toolOutput ? String(request.toolOutput).slice(0, 280) : null })) : []"
- set: fallbackTasks
value:
expr: "(await runQaCli(env, ['tasks', 'list', '--json', '--runtime', 'subagent'], { timeoutMs: liveTurnTimeoutMs(env, 60000), json: true }).catch((error) => ({ error: String(error?.message ?? error) })))"
- throw:
expr: "`subagent fallback marker missing: ${fallbackError?.message ?? fallbackError}; outbound=${recentOutboundSummary(state, 8)} tasks=${JSON.stringify(fallbackTasks)} requests=${JSON.stringify(fallbackDebugRequests)}`"
- if:
expr: "Boolean(env.mock)"
then:
- set: fallbackDebugRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))]"
- assert:
expr: "fallbackDebugRequests.some((request) => !request.toolOutput && /subagent direct fallback qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn' && request.plannedToolArgs?.label === config.expectedLabel)"
message:
expr: "`expected sessions_spawn for yielded fallback scenario, saw ${JSON.stringify(fallbackDebugRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null })))}`"
- assert:
expr: "fallbackDebugRequests.some((request) => /subagent direct fallback qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_yield')"
message:
expr: "`expected sessions_yield for yielded fallback scenario, saw ${JSON.stringify(fallbackDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- call: waitForCondition
saveAs: deliveredTask
args:
- lambda:
expr: "(async () => { const payload = await runQaCli(env, ['tasks', 'list', '--json', '--runtime', 'subagent'], { timeoutMs: liveTurnTimeoutMs(env, 60000), json: true }); return (payload.tasks ?? []).find((task) => task.label === config.expectedLabel && task.deliveryStatus === 'delivered' && task.status === 'succeeded') ?? null; })()"
- expr: liveTurnTimeoutMs(env, 60000)
- 250
- assert:
expr: "deliveredTask.deliveryStatus === 'delivered'"
message:
expr: "`expected delivered task status for ${config.expectedLabel}, got ${JSON.stringify(deliveredTask)}`"
detailsExpr: "outbound.text"
```
qa/scenarios/agents/subagent-completion-direct-fallback.yaml
+108
View File
@@ -0,0 +1,108 @@
title: Subagent completion direct fallback
scenario:
id: subagent-completion-direct-fallback
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- runtime.delivery
- channels.qa-channel
objective: Verify a yielded parent still receives a successful subagent result through direct fallback delivery when the dormant announce turn produces no visible reply.
successCriteria:
- Parent launches a native subagent.
- Parent yields instead of waiting in-turn.
- Subagent completion result is delivered to the original QA DM without a thread id.
- Durable task delivery is marked delivered, not failed.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- src/agents/subagent-announce-delivery.ts
- src/agents/subagent-registry-lifecycle.ts
- src/agents/tools/sessions-yield-tool.ts
- extensions/qa-lab/src/providers/mock-openai/server.ts
execution:
kind: flow
summary: Reproduce yielded-parent subagent completion delivery and require frozen-result fallback to the QA DM.
config:
prompt: "Subagent direct fallback QA check: spawn one native subagent worker. The worker must finish with exactly QA-SUBAGENT-DIRECT-FALLBACK-OK. After spawning it, call sessions_yield and wait for the completion event. Do not use ACP."
expectedMarker: QA-SUBAGENT-DIRECT-FALLBACK-OK
expectedLabel: qa-direct-fallback-worker
flow:
steps:
- name: yielded parent receives child completion through direct fallback
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
- call: waitForQaChannelReady
args:
- ref: env
- 120000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:subagent-direct-fallback:${randomUUID().slice(0, 8)}`"
- try:
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && String(message.text ?? '').includes(config.expectedMarker)).at(-1)"
- expr: liveTurnTimeoutMs(env, 180000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "String(outbound.text ?? '').trim().includes(config.expectedMarker)"
message:
expr: "`fallback completion marker missing from outbound QA DM: ${recentOutboundSummary(state)}`"
catchAs: fallbackError
catch:
- set: fallbackDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].slice(-20).map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, prompt: String(request.prompt ?? '').slice(0, 280), allInputText: String(request.allInputText ?? '').slice(0, 280), toolOutput: request.toolOutput ? String(request.toolOutput).slice(0, 280) : null })) : []"
- set: fallbackTasks
value:
expr: "(await runQaCli(env, ['tasks', 'list', '--json', '--runtime', 'subagent'], { timeoutMs: liveTurnTimeoutMs(env, 60000), json: true }).catch((error) => ({ error: String(error?.message ?? error) })))"
- throw:
expr: "`subagent fallback marker missing: ${fallbackError?.message ?? fallbackError}; outbound=${recentOutboundSummary(state, 8)} tasks=${JSON.stringify(fallbackTasks)} requests=${JSON.stringify(fallbackDebugRequests)}`"
- if:
expr: "Boolean(env.mock)"
then:
- set: fallbackDebugRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))]"
- assert:
expr: "fallbackDebugRequests.some((request) => !request.toolOutput && /subagent direct fallback qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn' && request.plannedToolArgs?.label === config.expectedLabel)"
message:
expr: "`expected sessions_spawn for yielded fallback scenario, saw ${JSON.stringify(fallbackDebugRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null })))}`"
- assert:
expr: "fallbackDebugRequests.some((request) => /subagent direct fallback qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_yield')"
message:
expr: "`expected sessions_yield for yielded fallback scenario, saw ${JSON.stringify(fallbackDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- call: waitForCondition
saveAs: deliveredTask
args:
- lambda:
expr: "(async () => { const payload = await runQaCli(env, ['tasks', 'list', '--json', '--runtime', 'subagent'], { timeoutMs: liveTurnTimeoutMs(env, 60000), json: true }); return (payload.tasks ?? []).find((task) => task.label === config.expectedLabel && task.deliveryStatus === 'delivered' && task.status === 'succeeded') ?? null; })()"
- expr: liveTurnTimeoutMs(env, 60000)
- 250
- assert:
expr: "deliveredTask.deliveryStatus === 'delivered'"
message:
expr: "`expected delivered task status for ${config.expectedLabel}, got ${JSON.stringify(deliveredTask)}`"
detailsExpr: "outbound.text"
qa/scenarios/agents/subagent-fanout-synthesis.md
-199
View File
@@ -1,199 +0,0 @@
# Subagent fanout synthesis
```yaml qa-scenario
id: subagent-fanout-synthesis
title: Subagent fanout synthesis
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- agents.synthesis
objective: Verify the agent can delegate multiple bounded subagent tasks and fold both results back into one parent reply.
successCriteria:
- Parent flow launches at least two bounded subagent tasks.
- Both delegated results are acknowledged in the main flow.
- Final answer synthesizes both worker outputs in one reply.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
codeRefs:
- src/agents/subagent-spawn.ts
- src/agents/system-prompt.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent can delegate multiple bounded subagent tasks and fold both results back into one parent reply.
config:
prompt: |-
Subagent fanout synthesis check: delegate exactly two bounded subagents sequentially.
Subagent 1: verify that `HEARTBEAT.md` exists and report `ok` if it does.
Subagent 2: verify that `repo/qa/scenarios/agents/subagent-fanout-synthesis.md` exists and report `ok` if it does.
Wait for both subagents to finish.
Then reply with exactly these two lines and nothing else:
subagent-1: ok
subagent-2: ok
Do not use ACP.
expectedReplyAny:
- "subagent-1: ok"
- "subagent-2: ok"
expectedReplyGroups:
- - alpha-ok
- subagent_one_ok
- subagent one ok
- "subagent-1: ok"
- - beta-ok
- subagent_two_ok
- subagent two ok
- "subagent-2: ok"
expectedChildLabels:
- qa-fanout-alpha
- qa-fanout-beta
```
```yaml qa-flow
steps:
- name: spawns sequential workers and folds both results back into the parent reply
actions:
- set: attempts
value:
expr: "env.providerMode === 'mock-openai' ? 1 : 2"
- set: lastError
value: null
- forEach:
items:
expr: "Array.from({ length: attempts }, (_, index) => index + 1)"
item: attempt
actions:
- if:
expr: "lastError === '__done__'"
then:
- set: skippedAttempt
value:
expr: attempt
else:
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:fanout:${attempt}:${randomUUID().slice(0, 8)}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === 'qa-operator' && config.expectedReplyGroups.every((group) => group.some((needle) => normalizeLowercaseStringOrEmpty(message.text ?? '').includes(needle)))).at(-1)"
- expr: liveTurnTimeoutMs(env, 120000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- if:
expr: "Boolean(env.mock)"
then:
- call: readRawQaSessionStore
saveAs: store
args:
- ref: env
- set: childRows
value:
expr: "Object.values(store).filter((entry) => entry.spawnedBy === sessionKey)"
- set: sawAlpha
value:
expr: "childRows.some((entry) => entry.label === config.expectedChildLabels[0])"
- set: sawBeta
value:
expr: "childRows.some((entry) => entry.label === config.expectedChildLabels[1])"
- assert:
expr: "sawAlpha && sawBeta"
message:
expr: "`fanout child sessions missing (alpha=${String(sawAlpha)} beta=${String(sawBeta)})`"
# Tool-call assertion (criterion 2 of the
# parity completion gate in #64227): the
# scenario must have actually invoked
# `sessions_spawn` at least twice with
# distinct labels, not just ended up with
# two rows in the session store through
# prose trickery. The session store alone
# can be populated by other flows or by a
# model that fabricates "delegation"
# narration. `plannedToolName` on the
# mock's `/debug/requests` log is the
# tool-call ground truth: two recorded
# sessions_spawn requests with distinct
# labels means the model really dispatched
# both subagents.
- set: fanoutSpawnRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => request.plannedToolName === 'sessions_spawn' && /subagent fanout synthesis check/i.test(String(request.allInputText ?? '')))"
- assert:
expr: "fanoutSpawnRequests.length >= 2"
message:
expr: "`expected at least two sessions_spawn tool calls during subagent fanout scenario, saw ${fanoutSpawnRequests.length}`"
- set: details
value:
expr: "outbound.text"
- set: lastError
value: __done__
catchAs: attemptError
catch:
- if:
expr: "Boolean(env.mock) && /timed out after/i.test(formatErrorMessage(attemptError))"
then:
- call: readRawQaSessionStore
saveAs: timeoutStore
args:
- ref: env
- set: timeoutChildRows
value:
expr: "Object.values(timeoutStore).filter((entry) => entry.spawnedBy === sessionKey)"
- set: timeoutSawAlpha
value:
expr: "timeoutChildRows.some((entry) => entry.label === config.expectedChildLabels[0])"
- set: timeoutSawBeta
value:
expr: "timeoutChildRows.some((entry) => entry.label === config.expectedChildLabels[1])"
- set: timeoutSpawnRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => request.plannedToolName === 'sessions_spawn' && /subagent fanout synthesis check/i.test(String(request.allInputText ?? '')))"
- if:
expr: "timeoutSawAlpha && timeoutSawBeta && timeoutSpawnRequests.length >= 2"
then:
- set: details
value: "subagent-1: ok\nsubagent-2: ok"
- set: lastError
value: __done__
- if:
expr: "lastError !== '__done__'"
then:
- set: lastError
value:
ref: attemptError
- if:
expr: "lastError !== '__done__' && attempt < attempts"
then:
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
catch:
- set: ignoredRetryWait
value: true
- assert:
expr: "lastError === '__done__'"
message:
expr: "lastError instanceof Error ? formatErrorMessage(lastError) : String(lastError ?? 'fanout retry exhausted')"
detailsExpr: "details"
```
qa/scenarios/agents/subagent-fanout-synthesis.yaml
+196
View File
@@ -0,0 +1,196 @@
title: Subagent fanout synthesis
scenario:
id: subagent-fanout-synthesis
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- agents.synthesis
objective: Verify the agent can delegate multiple bounded subagent tasks and fold both results back into one parent reply.
successCriteria:
- Parent flow launches at least two bounded subagent tasks.
- Both delegated results are acknowledged in the main flow.
- Final answer synthesizes both worker outputs in one reply.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
codeRefs:
- src/agents/subagent-spawn.ts
- src/agents/system-prompt.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent can delegate multiple bounded subagent tasks and fold both results back into one parent reply.
config:
prompt: |-
Subagent fanout synthesis check: delegate exactly two bounded subagents sequentially.
Subagent 1: verify that `HEARTBEAT.md` exists and report `ok` if it does.
Subagent 2: verify that `repo/qa/scenarios/agents/subagent-fanout-synthesis.yaml` exists and report `ok` if it does.
Wait for both subagents to finish.
Then reply with exactly these two lines and nothing else:
subagent-1: ok
subagent-2: ok
Do not use ACP.
expectedReplyAny:
- "subagent-1: ok"
- "subagent-2: ok"
expectedReplyGroups:
- - alpha-ok
- subagent_one_ok
- subagent one ok
- "subagent-1: ok"
- - beta-ok
- subagent_two_ok
- subagent two ok
- "subagent-2: ok"
expectedChildLabels:
- qa-fanout-alpha
- qa-fanout-beta
flow:
steps:
- name: spawns sequential workers and folds both results back into the parent reply
actions:
- set: attempts
value:
expr: "env.providerMode === 'mock-openai' ? 1 : 2"
- set: lastError
value: null
- forEach:
items:
expr: "Array.from({ length: attempts }, (_, index) => index + 1)"
item: attempt
actions:
- if:
expr: "lastError === '__done__'"
then:
- set: skippedAttempt
value:
expr: attempt
else:
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:fanout:${attempt}:${randomUUID().slice(0, 8)}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === 'qa-operator' && config.expectedReplyGroups.every((group) => group.some((needle) => normalizeLowercaseStringOrEmpty(message.text ?? '').includes(needle)))).at(-1)"
- expr: liveTurnTimeoutMs(env, 120000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- if:
expr: "Boolean(env.mock)"
then:
- call: readRawQaSessionStore
saveAs: store
args:
- ref: env
- set: childRows
value:
expr: "Object.values(store).filter((entry) => entry.spawnedBy === sessionKey)"
- set: sawAlpha
value:
expr: "childRows.some((entry) => entry.label === config.expectedChildLabels[0])"
- set: sawBeta
value:
expr: "childRows.some((entry) => entry.label === config.expectedChildLabels[1])"
- assert:
expr: "sawAlpha && sawBeta"
message:
expr: "`fanout child sessions missing (alpha=${String(sawAlpha)} beta=${String(sawBeta)})`"
# Tool-call assertion (criterion 2 of the
# parity completion gate in #64227): the
# scenario must have actually invoked
# `sessions_spawn` at least twice with
# distinct labels, not just ended up with
# two rows in the session store through
# prose trickery. The session store alone
# can be populated by other flows or by a
# model that fabricates "delegation"
# narration. `plannedToolName` on the
# mock's `/debug/requests` log is the
# tool-call ground truth: two recorded
# sessions_spawn requests with distinct
# labels means the model really dispatched
# both subagents.
- set: fanoutSpawnRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => request.plannedToolName === 'sessions_spawn' && /subagent fanout synthesis check/i.test(String(request.allInputText ?? '')))"
- assert:
expr: "fanoutSpawnRequests.length >= 2"
message:
expr: "`expected at least two sessions_spawn tool calls during subagent fanout scenario, saw ${fanoutSpawnRequests.length}`"
- set: details
value:
expr: "outbound.text"
- set: lastError
value: __done__
catchAs: attemptError
catch:
- if:
expr: "Boolean(env.mock) && /timed out after/i.test(formatErrorMessage(attemptError))"
then:
- call: readRawQaSessionStore
saveAs: timeoutStore
args:
- ref: env
- set: timeoutChildRows
value:
expr: "Object.values(timeoutStore).filter((entry) => entry.spawnedBy === sessionKey)"
- set: timeoutSawAlpha
value:
expr: "timeoutChildRows.some((entry) => entry.label === config.expectedChildLabels[0])"
- set: timeoutSawBeta
value:
expr: "timeoutChildRows.some((entry) => entry.label === config.expectedChildLabels[1])"
- set: timeoutSpawnRequests
value:
expr: "[...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => request.plannedToolName === 'sessions_spawn' && /subagent fanout synthesis check/i.test(String(request.allInputText ?? '')))"
- if:
expr: "timeoutSawAlpha && timeoutSawBeta && timeoutSpawnRequests.length >= 2"
then:
- set: details
value: "subagent-1: ok\nsubagent-2: ok"
- set: lastError
value: __done__
- if:
expr: "lastError !== '__done__'"
then:
- set: lastError
value:
ref: attemptError
- if:
expr: "lastError !== '__done__' && attempt < attempts"
then:
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 120000
catch:
- set: ignoredRetryWait
value: true
- assert:
expr: "lastError === '__done__'"
message:
expr: "lastError instanceof Error ? formatErrorMessage(lastError) : String(lastError ?? 'fanout retry exhausted')"
detailsExpr: "details"
qa/scenarios/agents/subagent-forked-context.md
-62
View File
@@ -1,62 +0,0 @@
# Subagent forked context
```yaml qa-scenario
id: subagent-forked-context
title: Subagent forked context
surface: subagents
coverage:
primary:
- agents.subagents
objective: Verify the agent can choose forked subagent context when the child needs the current transcript.
successCriteria:
- Agent launches a native subagent with context=fork.
- Subagent uses the forked requester transcript to recover the visible code.
- Subagent request remains bounded and does not switch to ACP.
- User-visible output includes the delegated result and the visible code.
docsRefs:
- docs/tools/subagents.md
- docs/concepts/session-tool.md
codeRefs:
- src/agents/tools/sessions-spawn-tool.ts
- src/agents/subagent-spawn.ts
execution:
kind: flow
summary: Ask the agent to delegate work that depends on the current transcript and assert sessions_spawn carries context=fork.
config:
contextNeedle: FORKED-CONTEXT-ALPHA
prompt: "Forked subagent context QA check. The visible code in this current conversation is FORKED-CONTEXT-ALPHA. Delegate to a native subagent to report the visible code from the requester transcript. Do not include the visible code in the child task text; the child must recover it from forked transcript context. Use forked context if the child needs the current transcript; otherwise it will not know the code. A spawn-accepted result is not the answer. Wait for the child completion, then make sure user-visible output includes the visible code."
```
```yaml qa-flow
steps:
- name: forks current transcript context for the child
actions:
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:forked-context
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && String(candidate.text ?? '').includes(config.contextNeedle) && !normalizeLowercaseStringOrEmpty(candidate.text).includes('waiting')).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "env.mock || String(outbound.text ?? '').includes(config.contextNeedle)"
message:
expr: "`expected live final answer to include fork-only context code ${config.contextNeedle}, got: ${outbound.text}`"
- set: forkDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))] : []"
- assert:
expr: "!env.mock || forkDebugRequests.some((request) => !request.toolOutput && /forked subagent context qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn' && (request.plannedToolArgs?.context === 'fork' || /context\\s*=\\s*fork/i.test(String(request.allInputText ?? ''))))"
message:
expr: "`expected sessions_spawn context=fork during forked context scenario, saw ${JSON.stringify(forkDebugRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null })))} `"
detailsExpr: outbound.text
```
qa/scenarios/agents/subagent-forked-context.yaml
+59
View File
@@ -0,0 +1,59 @@
title: Subagent forked context
scenario:
id: subagent-forked-context
surface: subagents
coverage:
primary:
- agents.subagents
objective: Verify the agent can choose forked subagent context when the child needs the current transcript.
successCriteria:
- Agent launches a native subagent with context=fork.
- Subagent uses the forked requester transcript to recover the visible code.
- Subagent request remains bounded and does not switch to ACP.
- User-visible output includes the delegated result and the visible code.
docsRefs:
- docs/tools/subagents.md
- docs/concepts/session-tool.md
codeRefs:
- src/agents/tools/sessions-spawn-tool.ts
- src/agents/subagent-spawn.ts
execution:
kind: flow
summary: Ask the agent to delegate work that depends on the current transcript and assert sessions_spawn carries context=fork.
config:
contextNeedle: FORKED-CONTEXT-ALPHA
prompt: "Forked subagent context QA check. The visible code in this current conversation is FORKED-CONTEXT-ALPHA. Delegate to a native subagent to report the visible code from the requester transcript. Do not include the visible code in the child task text; the child must recover it from forked transcript context. Use forked context if the child needs the current transcript; otherwise it will not know the code. A spawn-accepted result is not the answer. Wait for the child completion, then make sure user-visible output includes the visible code."
flow:
steps:
- name: forks current transcript context for the child
actions:
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:forked-context
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && String(candidate.text ?? '').includes(config.contextNeedle) && !normalizeLowercaseStringOrEmpty(candidate.text).includes('waiting')).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "env.mock || String(outbound.text ?? '').includes(config.contextNeedle)"
message:
expr: "`expected live final answer to include fork-only context code ${config.contextNeedle}, got: ${outbound.text}`"
- set: forkDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))] : []"
- assert:
expr: "!env.mock || forkDebugRequests.some((request) => !request.toolOutput && /forked subagent context qa check/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn' && (request.plannedToolArgs?.context === 'fork' || /context\\s*=\\s*fork/i.test(String(request.allInputText ?? ''))))"
message:
expr: "`expected sessions_spawn context=fork during forked context scenario, saw ${JSON.stringify(forkDebugRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null })))} `"
detailsExpr: outbound.text
qa/scenarios/agents/subagent-handoff.md
-73
View File
@@ -1,73 +0,0 @@
# Subagent handoff
```yaml qa-scenario
id: subagent-handoff
title: Subagent handoff
surface: subagents
coverage:
primary:
- agents.subagents
objective: Verify the agent can delegate a bounded task to a subagent and fold the result back into the main thread.
successCriteria:
- Agent launches a bounded subagent task.
- Subagent result is acknowledged in the main flow.
- Final answer attributes delegated work clearly.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
codeRefs:
- src/agents/system-prompt.ts
- extensions/qa-lab/src/report.ts
execution:
kind: flow
summary: Verify the agent can delegate a bounded task to a subagent and fold the result back into the main thread.
config:
prompt: "Delegate one bounded QA task to a subagent. Wait for the subagent to finish. Then reply with three labeled sections exactly once: Delegated task, Result, Evidence. Include the child result itself, not 'waiting'."
```
```yaml qa-flow
steps:
- name: delegates a bounded task and reports the result
actions:
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:subagent
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && (() => { const lower = normalizeLowercaseStringOrEmpty(candidate.text); return lower.includes('delegated task') && lower.includes('result') && lower.includes('evidence') && !lower.includes('waiting'); })()).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!['failed to delegate','could not delegate','subagent unavailable'].some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`subagent handoff reported failure: ${outbound.text}`"
# Parity gate criterion 2 (no fake progress / fake tool completion):
# require an actual sessions_spawn tool call. Without this, a model
# could produce the three labeled sections ("Delegated task", "Result",
# "Evidence") as free-form prose without ever delegating to a real
# subagent. The assertion is pinned to THIS scenario by matching the
# scenario-unique prompt substring "Delegate one bounded QA task"
# (not a broad /delegate|subagent/ regex) so the earlier
# subagent-fanout-synthesis scenario — which also contains "delegate"
# and produces its own pre-tool sessions_spawn request — cannot
# satisfy the assertion here. The match is also constrained to
# pre-tool requests (no toolOutput) because the mock only plans
# sessions_spawn on requests with no toolOutput; the follow-up
# request after the tool runs has plannedToolName unset.
- set: subagentDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))] : []"
- assert:
expr: "!env.mock || subagentDebugRequests.some((request) => !request.toolOutput && /delegate one bounded qa task/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn')"
message:
expr: "`expected sessions_spawn tool call during subagent handoff scenario, saw plannedToolNames=${JSON.stringify(subagentDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
```
qa/scenarios/agents/subagent-handoff.yaml
+70
View File
@@ -0,0 +1,70 @@
title: Subagent handoff
scenario:
id: subagent-handoff
surface: subagents
coverage:
primary:
- agents.subagents
objective: Verify the agent can delegate a bounded task to a subagent and fold the result back into the main thread.
successCriteria:
- Agent launches a bounded subagent task.
- Subagent result is acknowledged in the main flow.
- Final answer attributes delegated work clearly.
docsRefs:
- docs/tools/subagents.md
- docs/help/testing.md
codeRefs:
- src/agents/system-prompt.ts
- extensions/qa-lab/src/report.ts
execution:
kind: flow
summary: Verify the agent can delegate a bounded task to a subagent and fold the result back into the main thread.
config:
prompt: "Delegate one bounded QA task to a subagent. Wait for the subagent to finish. Then reply with three labeled sections exactly once: Delegated task, Result, Evidence. Include the child result itself, not 'waiting'."
flow:
steps:
- name: delegates a bounded task and reports the result
actions:
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:subagent
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 90000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && (() => { const lower = normalizeLowercaseStringOrEmpty(candidate.text); return lower.includes('delegated task') && lower.includes('result') && lower.includes('evidence') && !lower.includes('waiting'); })()).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!['failed to delegate','could not delegate','subagent unavailable'].some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`subagent handoff reported failure: ${outbound.text}`"
# Parity gate criterion 2 (no fake progress / fake tool completion):
# require an actual sessions_spawn tool call. Without this, a model
# could produce the three labeled sections ("Delegated task", "Result",
# "Evidence") as free-form prose without ever delegating to a real
# subagent. The assertion is pinned to THIS scenario by matching the
# scenario-unique prompt substring "Delegate one bounded QA task"
# (not a broad /delegate|subagent/ regex) so the earlier
# subagent-fanout-synthesis scenario — which also contains "delegate"
# and produces its own pre-tool sessions_spawn request — cannot
# satisfy the assertion here. The match is also constrained to
# pre-tool requests (no toolOutput) because the mock only plans
# sessions_spawn on requests with no toolOutput; the follow-up
# request after the tool runs has plannedToolName unset.
- set: subagentDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))] : []"
- assert:
expr: "!env.mock || subagentDebugRequests.some((request) => !request.toolOutput && /delegate one bounded qa task/i.test(String(request.allInputText ?? '')) && request.plannedToolName === 'sessions_spawn')"
message:
expr: "`expected sessions_spawn tool call during subagent handoff scenario, saw plannedToolNames=${JSON.stringify(subagentDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
qa/scenarios/agents/subagent-stale-child-links.md
-175
View File
@@ -1,175 +0,0 @@
# Subagent stale child links
```yaml qa-scenario
id: subagent-stale-child-links
title: Subagent stale child links
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- gateway.sessions-list
objective: Verify restarted gateways hide stale persisted subagent child links without hiding live or fresh children.
successCriteria:
- Old ended subagent run records are not exposed as current children.
- Old store-only spawnedBy and parentSessionKey rows are not exposed as current children.
- Child-side ACP store rows from sibling agents are not exposed as current children.
- Live subagent runs and fresh dashboard children remain visible.
docsRefs:
- docs/tools/subagents.md
- docs/concepts/qa-e2e-automation.md
- docs/help/testing.md
codeRefs:
- src/gateway/session-utils.ts
- src/agents/subagent-run-liveness.ts
- extensions/qa-lab/src/gateway-child.ts
execution:
kind: flow
summary: Seed stale subagent session state on disk, restart the real gateway, then assert sessions.list filters only the stale child links.
```
```yaml qa-flow
steps:
- name: restarted gateway filters stale subagent child links
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- set: mainKey
value: "agent:qa:main"
- set: staleRunKey
value: "agent:qa:subagent:qa-stale-ended"
- set: staleOrphanKey
value: "agent:qa:subagent:qa-orphan"
- set: staleAcpKey
value: "agent:claude:acp:qa-stale-acp"
- set: freshDashboardKey
value: "agent:qa:dashboard:qa-fresh-child"
- set: liveRunKey
value: "agent:qa:subagent:qa-live-child"
- call: env.gateway.restartAfterStateMutation
args:
- lambda:
params:
- ctx
async: true
expr: |-
await (async () => {
const now = Date.now();
const old = now - 2 * 60 * 60 * 1000;
const recent = now - 5000;
const qaSessionsDir = path.join(ctx.stateDir, "agents", "qa", "sessions");
const claudeSessionsDir = path.join(ctx.stateDir, "agents", "claude", "sessions");
const subagentDir = path.join(ctx.stateDir, "subagents");
await fs.mkdir(qaSessionsDir, { recursive: true });
await fs.mkdir(claudeSessionsDir, { recursive: true });
await fs.mkdir(subagentDir, { recursive: true });
await fs.writeFile(path.join(subagentDir, "runs.json"), `${JSON.stringify({
version: 2,
runs: {
"run-stale-ended": {
runId: "run-stale-ended",
childSessionKey: staleRunKey,
controllerSessionKey: mainKey,
requesterSessionKey: mainKey,
requesterDisplayKey: "main",
task: "old ended ghost",
cleanup: "keep",
createdAt: old - 60000,
startedAt: old - 50000,
endedAt: old,
outcome: { status: "ok" },
},
"run-live-visible": {
runId: "run-live-visible",
childSessionKey: liveRunKey,
controllerSessionKey: mainKey,
requesterSessionKey: mainKey,
requesterDisplayKey: "main",
task: "live child remains visible",
cleanup: "keep",
createdAt: recent,
startedAt: recent,
},
},
}, null, 2)}\n`, "utf8");
await fs.writeFile(path.join(qaSessionsDir, "sessions.json"), `${JSON.stringify({
[mainKey]: {
sessionId: "sess-main",
updatedAt: now,
},
[staleRunKey]: {
sessionId: "sess-stale-run",
updatedAt: old,
spawnedBy: mainKey,
status: "done",
endedAt: old,
},
[staleOrphanKey]: {
sessionId: "sess-orphan",
updatedAt: old,
parentSessionKey: mainKey,
},
[freshDashboardKey]: {
sessionId: "sess-fresh-dashboard",
updatedAt: now,
parentSessionKey: mainKey,
},
[liveRunKey]: {
sessionId: "sess-live-child",
updatedAt: recent,
spawnedBy: mainKey,
},
}, null, 2)}\n`, "utf8");
await fs.writeFile(path.join(claudeSessionsDir, "sessions.json"), `${JSON.stringify({
[staleAcpKey]: {
sessionId: "sess-acp-stale",
updatedAt: old,
spawnedBy: mainKey,
status: "done",
endedAt: old,
},
}, null, 2)}\n`, "utf8");
})()
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: env.gateway.call
saveAs: listed
args:
- "sessions.list"
- {}
- timeoutMs: 60000
- call: env.gateway.call
saveAs: filtered
args:
- "sessions.list"
- spawnedBy:
ref: mainKey
- timeoutMs: 60000
- set: mainChildren
value:
expr: "(listed.sessions.find((session) => session.key === mainKey)?.childSessions ?? [])"
- set: filteredKeys
value:
expr: "filtered.sessions.map((session) => session.key)"
- assert:
expr: "mainChildren.includes(freshDashboardKey)"
message:
expr: "`fresh dashboard child missing from main children: ${JSON.stringify(mainChildren)}`"
- assert:
expr: "mainChildren.includes(liveRunKey)"
message:
expr: "`live subagent child missing from main children: ${JSON.stringify(mainChildren)}`"
- assert:
expr: "filteredKeys.includes(freshDashboardKey) && filteredKeys.includes(liveRunKey)"
message:
expr: "`spawnedBy filter dropped live/fresh children: ${JSON.stringify(filteredKeys)}`"
- assert:
expr: "![staleRunKey, staleOrphanKey, staleAcpKey].some((key) => mainChildren.includes(key) || filteredKeys.includes(key))"
message:
expr: "`stale child leaked through sessions.list (main=${JSON.stringify(mainChildren)} filtered=${JSON.stringify(filteredKeys)})`"
detailsExpr: "({ mainChildren, filteredKeys })"
```
qa/scenarios/agents/subagent-stale-child-links.yaml
+172
View File
@@ -0,0 +1,172 @@
title: Subagent stale child links
scenario:
id: subagent-stale-child-links
surface: subagents
coverage:
primary:
- agents.subagents
secondary:
- gateway.sessions-list
objective: Verify restarted gateways hide stale persisted subagent child links without hiding live or fresh children.
successCriteria:
- Old ended subagent run records are not exposed as current children.
- Old store-only spawnedBy and parentSessionKey rows are not exposed as current children.
- Child-side ACP store rows from sibling agents are not exposed as current children.
- Live subagent runs and fresh dashboard children remain visible.
docsRefs:
- docs/tools/subagents.md
- docs/concepts/qa-e2e-automation.md
- docs/help/testing.md
codeRefs:
- src/gateway/session-utils.ts
- src/agents/subagent-run-liveness.ts
- extensions/qa-lab/src/gateway-child.ts
execution:
kind: flow
summary: Seed stale subagent session state on disk, restart the real gateway, then assert sessions.list filters only the stale child links.
flow:
steps:
- name: restarted gateway filters stale subagent child links
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- set: mainKey
value: "agent:qa:main"
- set: staleRunKey
value: "agent:qa:subagent:qa-stale-ended"
- set: staleOrphanKey
value: "agent:qa:subagent:qa-orphan"
- set: staleAcpKey
value: "agent:claude:acp:qa-stale-acp"
- set: freshDashboardKey
value: "agent:qa:dashboard:qa-fresh-child"
- set: liveRunKey
value: "agent:qa:subagent:qa-live-child"
- call: env.gateway.restartAfterStateMutation
args:
- lambda:
params:
- ctx
async: true
expr: |-
await (async () => {
const now = Date.now();
const old = now - 2 * 60 * 60 * 1000;
const recent = now - 5000;
const qaSessionsDir = path.join(ctx.stateDir, "agents", "qa", "sessions");
const claudeSessionsDir = path.join(ctx.stateDir, "agents", "claude", "sessions");
const subagentDir = path.join(ctx.stateDir, "subagents");
await fs.mkdir(qaSessionsDir, { recursive: true });
await fs.mkdir(claudeSessionsDir, { recursive: true });
await fs.mkdir(subagentDir, { recursive: true });
await fs.writeFile(path.join(subagentDir, "runs.json"), `${JSON.stringify({
version: 2,
runs: {
"run-stale-ended": {
runId: "run-stale-ended",
childSessionKey: staleRunKey,
controllerSessionKey: mainKey,
requesterSessionKey: mainKey,
requesterDisplayKey: "main",
task: "old ended ghost",
cleanup: "keep",
createdAt: old - 60000,
startedAt: old - 50000,
endedAt: old,
outcome: { status: "ok" },
},
"run-live-visible": {
runId: "run-live-visible",
childSessionKey: liveRunKey,
controllerSessionKey: mainKey,
requesterSessionKey: mainKey,
requesterDisplayKey: "main",
task: "live child remains visible",
cleanup: "keep",
createdAt: recent,
startedAt: recent,
},
},
}, null, 2)}\n`, "utf8");
await fs.writeFile(path.join(qaSessionsDir, "sessions.json"), `${JSON.stringify({
[mainKey]: {
sessionId: "sess-main",
updatedAt: now,
},
[staleRunKey]: {
sessionId: "sess-stale-run",
updatedAt: old,
spawnedBy: mainKey,
status: "done",
endedAt: old,
},
[staleOrphanKey]: {
sessionId: "sess-orphan",
updatedAt: old,
parentSessionKey: mainKey,
},
[freshDashboardKey]: {
sessionId: "sess-fresh-dashboard",
updatedAt: now,
parentSessionKey: mainKey,
},
[liveRunKey]: {
sessionId: "sess-live-child",
updatedAt: recent,
spawnedBy: mainKey,
},
}, null, 2)}\n`, "utf8");
await fs.writeFile(path.join(claudeSessionsDir, "sessions.json"), `${JSON.stringify({
[staleAcpKey]: {
sessionId: "sess-acp-stale",
updatedAt: old,
spawnedBy: mainKey,
status: "done",
endedAt: old,
},
}, null, 2)}\n`, "utf8");
})()
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: env.gateway.call
saveAs: listed
args:
- "sessions.list"
- {}
- timeoutMs: 60000
- call: env.gateway.call
saveAs: filtered
args:
- "sessions.list"
- spawnedBy:
ref: mainKey
- timeoutMs: 60000
- set: mainChildren
value:
expr: "(listed.sessions.find((session) => session.key === mainKey)?.childSessions ?? [])"
- set: filteredKeys
value:
expr: "filtered.sessions.map((session) => session.key)"
- assert:
expr: "mainChildren.includes(freshDashboardKey)"
message:
expr: "`fresh dashboard child missing from main children: ${JSON.stringify(mainChildren)}`"
- assert:
expr: "mainChildren.includes(liveRunKey)"
message:
expr: "`live subagent child missing from main children: ${JSON.stringify(mainChildren)}`"
- assert:
expr: "filteredKeys.includes(freshDashboardKey) && filteredKeys.includes(liveRunKey)"
message:
expr: "`spawnedBy filter dropped live/fresh children: ${JSON.stringify(filteredKeys)}`"
- assert:
expr: "![staleRunKey, staleOrphanKey, staleAcpKey].some((key) => mainChildren.includes(key) || filteredKeys.includes(key))"
message:
expr: "`stale child leaked through sessions.list (main=${JSON.stringify(mainChildren)} filtered=${JSON.stringify(filteredKeys)})`"
detailsExpr: "({ mainChildren, filteredKeys })"
qa/scenarios/channels/channel-chat-baseline.md
-93
View File
@@ -1,93 +0,0 @@
# Channel baseline conversation
```yaml qa-scenario
id: channel-chat-baseline
title: Channel baseline conversation
surface: channel
coverage:
primary:
- channels.group-messages
secondary:
- channels.qa-channel
objective: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
successCriteria:
- Agent replies in the shared channel transcript.
- Agent visible reply contains the scenario marker.
- Agent keeps the conversation scoped to the channel.
- Agent respects mention-driven group routing semantics.
docsRefs:
- docs/channels/group-messages.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-lab/src/bus-state.ts
execution:
kind: flow
summary: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
config:
expectedMarker: QA-CHANNEL-BASELINE-OK
mentionPrompt: "@openclaw qa channel baseline marker check. Reply exactly: QA-CHANNEL-BASELINE-OK"
```
```yaml qa-flow
steps:
- name: ignores unmentioned channel chatter
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text: hello team, no bot ping here
- call: waitForNoOutbound
args:
- ref: state
- name: replies when mentioned in channel
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text:
expr: config.mentionPrompt
- call: waitForOutboundMessage
saveAs: message
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && !candidate.threadId && String(candidate.text ?? '').includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && String(candidate.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one channel baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
detailsExpr: message.text
```
qa/scenarios/channels/channel-chat-baseline.yaml
+90
View File
@@ -0,0 +1,90 @@
title: Channel baseline conversation
scenario:
id: channel-chat-baseline
surface: channel
coverage:
primary:
- channels.group-messages
secondary:
- channels.qa-channel
objective: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
successCriteria:
- Agent replies in the shared channel transcript.
- Agent visible reply contains the scenario marker.
- Agent keeps the conversation scoped to the channel.
- Agent respects mention-driven group routing semantics.
docsRefs:
- docs/channels/group-messages.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-lab/src/bus-state.ts
execution:
kind: flow
summary: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
config:
expectedMarker: QA-CHANNEL-BASELINE-OK
mentionPrompt: "@openclaw qa channel baseline marker check. Reply exactly: QA-CHANNEL-BASELINE-OK"
flow:
steps:
- name: ignores unmentioned channel chatter
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text: hello team, no bot ping here
- call: waitForNoOutbound
args:
- ref: state
- name: replies when mentioned in channel
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text:
expr: config.mentionPrompt
- call: waitForOutboundMessage
saveAs: message
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && !candidate.threadId && String(candidate.text ?? '').includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && String(candidate.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one channel baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
detailsExpr: message.text
qa/scenarios/channels/dm-chat-baseline.md
-62
View File
@@ -1,62 +0,0 @@
# DM baseline conversation
```yaml qa-scenario
id: dm-chat-baseline
title: DM baseline conversation
surface: dm
coverage:
primary:
- channels.dm
secondary:
- channels.qa-channel
objective: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
successCriteria:
- Agent replies in DM without channel routing mistakes.
- Agent visible reply contains the scenario marker.
- Agent explains the QA lab and message bus correctly.
- Agent keeps the dev C-3PO personality.
docsRefs:
- docs/channels/qa-channel.md
- docs/help/testing.md
codeRefs:
- extensions/qa-channel/src/gateway.ts
- extensions/qa-lab/src/lab-server.ts
execution:
kind: flow
summary: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
config:
expectedMarker: QA-DM-BASELINE-OK
prompt: "DM baseline marker check. Include exact marker: `QA-DM-BASELINE-OK` and briefly identify the QA lab message bus."
```
```yaml qa-flow
steps:
- name: replies coherently in DM
actions:
- call: resetBus
- call: state.addInboundMessage
args:
- conversation:
id: alice
kind: direct
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.direction === 'outbound' && candidate.conversation.id === 'alice' && candidate.conversation.kind === 'direct' && String(candidate.text ?? '').includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 45000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'alice' && candidate.conversation.kind === 'direct' && String(candidate.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one DM baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'alice' })}`"
detailsExpr: outbound.text
```
qa/scenarios/channels/dm-chat-baseline.yaml
+59
View File
@@ -0,0 +1,59 @@
title: DM baseline conversation
scenario:
id: dm-chat-baseline
surface: dm
coverage:
primary:
- channels.dm
secondary:
- channels.qa-channel
objective: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
successCriteria:
- Agent replies in DM without channel routing mistakes.
- Agent visible reply contains the scenario marker.
- Agent explains the QA lab and message bus correctly.
- Agent keeps the dev C-3PO personality.
docsRefs:
- docs/channels/qa-channel.md
- docs/help/testing.md
codeRefs:
- extensions/qa-channel/src/gateway.ts
- extensions/qa-lab/src/lab-server.ts
execution:
kind: flow
summary: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
config:
expectedMarker: QA-DM-BASELINE-OK
prompt: "DM baseline marker check. Include exact marker: `QA-DM-BASELINE-OK` and briefly identify the QA lab message bus."
flow:
steps:
- name: replies coherently in DM
actions:
- call: resetBus
- call: state.addInboundMessage
args:
- conversation:
id: alice
kind: direct
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.direction === 'outbound' && candidate.conversation.id === 'alice' && candidate.conversation.kind === 'direct' && String(candidate.text ?? '').includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 45000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'alice' && candidate.conversation.kind === 'direct' && String(candidate.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one DM baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'alice' })}`"
detailsExpr: outbound.text
qa/scenarios/channels/group-message-tool-unavailable-fallback.md
-98
View File
@@ -1,98 +0,0 @@
# Group fallback when message tool is unavailable
```yaml qa-scenario
id: group-message-tool-unavailable-fallback
title: Group fallback when message tool is unavailable
surface: channel
coverage:
primary:
- channels.group-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Reproduce the group-visible-reply bug class where message_tool mode selected tool-only delivery even though group tool policy removed the message tool.
gatewayConfigPatch:
messages:
groupChat:
visibleReplies: message_tool
channels:
qa-channel:
groups:
qa-fallback-room:
tools:
allow:
- read
successCriteria:
- The group policy removes the message tool for this room.
- The mock provider returns a normal final answer with the marker.
- OpenClaw falls back to automatic delivery and posts the marker to the same group.
docsRefs:
- docs/channels/groups.md
- docs/channels/qa-channel.md
codeRefs:
- src/auto-reply/reply/dispatch-from-config.ts
- extensions/qa-channel/src/inbound.ts
execution:
kind: flow
summary: Verify message_tool visible replies degrade to automatic delivery when the active group policy removes message.
config:
conversationId: qa-fallback-room
promptSnippet: qa group message unavailable fallback check
prompt: "@openclaw qa group message unavailable fallback check. exact marker: `QA-GROUP-FALLBACK-OK`"
expectedMarker: QA-GROUP-FALLBACK-OK
```
```yaml qa-flow
steps:
- name: falls back to final-answer delivery when message is not available
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: group
title: QA Fallback Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one fallback group reply, saw ${matchingOutbound.length}`"
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.length > 0"
message: expected mock request evidence for fallback scenario
- assert:
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
message:
expr: "`message tool should not be planned when group policy removes it, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
```
qa/scenarios/channels/group-message-tool-unavailable-fallback.yaml
+95
View File
@@ -0,0 +1,95 @@
title: Group fallback when message tool is unavailable
scenario:
id: group-message-tool-unavailable-fallback
surface: channel
coverage:
primary:
- channels.group-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Reproduce the group-visible-reply bug class where message_tool mode selected tool-only delivery even though group tool policy removed the message tool.
gatewayConfigPatch:
messages:
groupChat:
visibleReplies: message_tool
channels:
qa-channel:
groups:
qa-fallback-room:
tools:
allow:
- read
successCriteria:
- The group policy removes the message tool for this room.
- The mock provider returns a normal final answer with the marker.
- OpenClaw falls back to automatic delivery and posts the marker to the same group.
docsRefs:
- docs/channels/groups.md
- docs/channels/qa-channel.md
codeRefs:
- src/auto-reply/reply/dispatch-from-config.ts
- extensions/qa-channel/src/inbound.ts
execution:
kind: flow
summary: Verify message_tool visible replies degrade to automatic delivery when the active group policy removes message.
config:
conversationId: qa-fallback-room
promptSnippet: qa group message unavailable fallback check
prompt: "@openclaw qa group message unavailable fallback check. exact marker: `QA-GROUP-FALLBACK-OK`"
expectedMarker: QA-GROUP-FALLBACK-OK
flow:
steps:
- name: falls back to final-answer delivery when message is not available
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: group
title: QA Fallback Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one fallback group reply, saw ${matchingOutbound.length}`"
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.length > 0"
message: expected mock request evidence for fallback scenario
- assert:
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
message:
expr: "`message tool should not be planned when group policy removes it, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
qa/scenarios/channels/group-visible-reply-tool.md
-96
View File
@@ -1,96 +0,0 @@
# Group visible reply via message tool
```yaml qa-scenario
id: group-visible-reply-tool
title: Group visible reply via message tool
surface: channel
coverage:
primary:
- channels.group-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Verify a group-sourced QA channel turn replies visibly through message(action=send) in the same room.
gatewayConfigPatch:
messages:
groupChat:
visibleReplies: message_tool
successCriteria:
- Agent receives a synthetic shared-room turn.
- Mock provider calls the shared message tool instead of relying on final-answer delivery.
- The visible reply lands once in the same group transcript.
docsRefs:
- docs/channels/groups.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-channel/src/outbound.ts
- src/auto-reply/reply/dispatch-from-config.ts
execution:
kind: flow
summary: Send a mentioned group message and verify visible output uses the message tool in the source group.
config:
conversationId: qa-visible-tool-room
promptSnippet: qa group visible reply tool check
prompt: "@openclaw qa group visible reply tool check. Use the visible room reply path. exact marker: `QA-GROUP-TOOL-OK`"
expectedMarker: QA-GROUP-TOOL-OK
```
```yaml qa-flow
steps:
- name: posts visible room output through message tool
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: group
title: QA Visible Tool Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForCondition
args:
- lambda:
async: true
params: []
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : true"
- expr: liveTurnTimeoutMs(env, 180000)
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.some((request) => request.plannedToolName === 'message' && request.plannedToolArgs?.action === 'send' && request.plannedToolArgs?.message === config.expectedMarker)"
message:
expr: "`expected message(action=send) with marker, saw ${JSON.stringify(scenarioRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, toolOutput: request.toolOutput ?? '', tools: Array.isArray(request.body?.tools) ? request.body.tools.map((tool) => tool?.name ?? tool?.function?.name ?? tool?.type ?? null).filter(Boolean).slice(0, 25) : [] })))} `"
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one visible group reply, saw ${matchingOutbound.length}`"
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
```
qa/scenarios/channels/group-visible-reply-tool.yaml
+93
View File
@@ -0,0 +1,93 @@
title: Group visible reply via message tool
scenario:
id: group-visible-reply-tool
surface: channel
coverage:
primary:
- channels.group-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Verify a group-sourced QA channel turn replies visibly through message(action=send) in the same room.
gatewayConfigPatch:
messages:
groupChat:
visibleReplies: message_tool
successCriteria:
- Agent receives a synthetic shared-room turn.
- Mock provider calls the shared message tool instead of relying on final-answer delivery.
- The visible reply lands once in the same group transcript.
docsRefs:
- docs/channels/groups.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/inbound.ts
- extensions/qa-channel/src/outbound.ts
- src/auto-reply/reply/dispatch-from-config.ts
execution:
kind: flow
summary: Send a mentioned group message and verify visible output uses the message tool in the source group.
config:
conversationId: qa-visible-tool-room
promptSnippet: qa group visible reply tool check
prompt: "@openclaw qa group visible reply tool check. Use the visible room reply path. exact marker: `QA-GROUP-TOOL-OK`"
expectedMarker: QA-GROUP-TOOL-OK
flow:
steps:
- name: posts visible room output through message tool
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: group
title: QA Visible Tool Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForCondition
args:
- lambda:
async: true
params: []
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : true"
- expr: liveTurnTimeoutMs(env, 180000)
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.some((request) => request.plannedToolName === 'message' && request.plannedToolArgs?.action === 'send' && request.plannedToolArgs?.message === config.expectedMarker)"
message:
expr: "`expected message(action=send) with marker, saw ${JSON.stringify(scenarioRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, toolOutput: request.toolOutput ?? '', tools: Array.isArray(request.body?.tools) ? request.body.tools.map((tool) => tool?.name ?? tool?.function?.name ?? tool?.type ?? null).filter(Boolean).slice(0, 25) : [] })))} `"
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
- expr: liveTurnTimeoutMs(env, 180000)
- set: matchingOutbound
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
- assert:
expr: matchingOutbound.length === 1
message:
expr: "`expected exactly one visible group reply, saw ${matchingOutbound.length}`"
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
qa/scenarios/channels/message-tool-stranded-final-reply.md
-92
View File
@@ -1,92 +0,0 @@
# Message-tool-only private final reply warning
```yaml qa-scenario
id: message-tool-stranded-final-reply
title: Message-tool-only private final reply warning
surface: channel
coverage:
primary:
- channels.direct-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Reproduce #85714 — under messages.visibleReplies=message_tool a long private final reply that never calls the message tool is kept private (no outbound), and the gateway emits the private-final WARN.
gatewayConfigPatch:
messages:
visibleReplies: message_tool
successCriteria:
- The mock provider returns a long normal final answer and does not plan the message tool.
- Under message_tool_only delivery the reply is kept private, so the direct conversation receives no outbound message.
- The gateway logs the private-final WARN from source-reply/private-final.
docsRefs:
- docs/channels/qa-channel.md
codeRefs:
- src/auto-reply/reply/agent-runner.ts
- src/auto-reply/reply/private-message-tool-final.ts
- src/auto-reply/reply/dispatch-from-config.ts
execution:
kind: flow
summary: Send a direct message_tool_only turn whose model reply omits the message tool, and verify a substantive private final warns without outbound delivery.
config:
conversationId: qa-stranded-dm
promptSnippet: qa private final reply warning check
prompt: "qa private final reply warning check. Reply to me directly in two complete sentences with `QA-STRANDED-85714` in the first sentence and a short explanation in the second sentence. Do NOT call any tool. Do NOT use the message tool."
expectedMarker: QA-STRANDED-85714
privateFinalLogNeedle: "source-reply/private-final"
```
```yaml qa-flow
steps:
- name: warns for substantive private final text when the model omits the message tool
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: logCursor
value:
expr: markGatewayLogCursor()
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: direct
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForNoOutbound
args:
- ref: state
- expr: liveTurnTimeoutMs(env, 30000)
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.length > 0"
message: expected mock request evidence that the turn actually ran
- assert:
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
message:
expr: "`model should not have planned the message tool, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
- set: privateFinalLog
value:
expr: "String(readGatewayLogs() ?? '').slice(logCursor)"
- set: privateFinalLine
value:
expr: "(privateFinalLog.split('\\n').find((line) => line.includes(config.privateFinalLogNeedle)) ?? '').trim()"
- assert:
expr: "privateFinalLog.includes(config.privateFinalLogNeedle)"
message:
expr: "`expected the gateway to log ${config.privateFinalLogNeedle} after a substantive private message_tool_only reply, but it was absent`"
detailsExpr: "`no-outbound private final; WARN logged=${privateFinalLog.includes(config.privateFinalLogNeedle)}; mock requests=${scenarioRequests.length}; gateway log: ${privateFinalLine}`"
```
qa/scenarios/channels/message-tool-stranded-final-reply.yaml
+89
View File
@@ -0,0 +1,89 @@
title: Message-tool-only private final reply warning
scenario:
id: message-tool-stranded-final-reply
surface: channel
coverage:
primary:
- channels.direct-visible-replies
secondary:
- channels.qa-channel
- tools.message
objective: Reproduce #85714 — under messages.visibleReplies=message_tool a long private final reply that never calls the message tool is kept private (no outbound), and the gateway emits the private-final WARN.
gatewayConfigPatch:
messages:
visibleReplies: message_tool
successCriteria:
- The mock provider returns a long normal final answer and does not plan the message tool.
- Under message_tool_only delivery the reply is kept private, so the direct conversation receives no outbound message.
- The gateway logs the private-final WARN from source-reply/private-final.
docsRefs:
- docs/channels/qa-channel.md
codeRefs:
- src/auto-reply/reply/agent-runner.ts
- src/auto-reply/reply/private-message-tool-final.ts
- src/auto-reply/reply/dispatch-from-config.ts
execution:
kind: flow
summary: Send a direct message_tool_only turn whose model reply omits the message tool, and verify a substantive private final warns without outbound delivery.
config:
conversationId: qa-stranded-dm
promptSnippet: qa private final reply warning check
prompt: "qa private final reply warning check. Reply to me directly in two complete sentences with `QA-STRANDED-85714` in the first sentence and a short explanation in the second sentence. Do NOT call any tool. Do NOT use the message tool."
expectedMarker: QA-STRANDED-85714
privateFinalLogNeedle: "source-reply/private-final"
flow:
steps:
- name: warns for substantive private final text when the model omits the message tool
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: logCursor
value:
expr: markGatewayLogCursor()
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.conversationId
kind: direct
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForNoOutbound
args:
- ref: state
- expr: liveTurnTimeoutMs(env, 30000)
- set: scenarioRequests
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
- assert:
expr: "!env.mock || scenarioRequests.length > 0"
message: expected mock request evidence that the turn actually ran
- assert:
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
message:
expr: "`model should not have planned the message tool, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
- set: privateFinalLog
value:
expr: "String(readGatewayLogs() ?? '').slice(logCursor)"
- set: privateFinalLine
value:
expr: "(privateFinalLog.split('\\n').find((line) => line.includes(config.privateFinalLogNeedle)) ?? '').trim()"
- assert:
expr: "privateFinalLog.includes(config.privateFinalLogNeedle)"
message:
expr: "`expected the gateway to log ${config.privateFinalLogNeedle} after a substantive private message_tool_only reply, but it was absent`"
detailsExpr: "`no-outbound private final; WARN logged=${privateFinalLog.includes(config.privateFinalLogNeedle)}; mock requests=${scenarioRequests.length}; gateway log: ${privateFinalLine}`"
qa/scenarios/channels/qa-channel-reconnect-dedupe.md
-123
View File
@@ -1,123 +0,0 @@
# QA channel reconnect dedupe
```yaml qa-scenario
id: qa-channel-reconnect-dedupe
title: QA channel reconnect dedupe
surface: channel
coverage:
primary:
- channels.reconnect
secondary:
- channels.dedup
- runtime.delivery
objective: Verify qa-channel readiness polling keeps prior delivery stable and does not replay the last outbound message.
successCriteria:
- Agent replies once before a reconnect-style readiness cycle.
- qa-channel reports ready again without replaying prior outbound delivery.
- Follow-up delivery produces one new reply without duplicating the first reply.
docsRefs:
- docs/channels/qa-channel.md
- docs/gateway/configuration.md
codeRefs:
- extensions/qa-lab/src/qa-channel-transport.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/suite-runtime-gateway.ts
execution:
kind: flow
summary: Verify qa-channel readiness recovery does not duplicate old outbound delivery.
config:
firstPrompt: "@openclaw Reconnect dedupe setup marker. Reply exactly: RECONNECT-FIRST-OK"
secondPrompt: "@openclaw Reconnect dedupe follow-up marker. Reply exactly: RECONNECT-SECOND-OK"
firstMarker: RECONNECT-FIRST-OK
secondMarker: RECONNECT-SECOND-OK
```
```yaml qa-flow
steps:
- name: reconnects without replaying prior outbound
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:channel-reconnect:${randomUUID().slice(0, 8)}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to: channel:qa-room
message:
expr: config.firstPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: firstOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.firstMarker)"
- expr: liveTurnTimeoutMs(env, 60000)
- set: beforeRestartCursor
value:
expr: state.getSnapshot().messages.length
- call: sleep
args:
- 1000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- set: firstMatchesBeforeFollowup
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
- assert:
expr: "firstMatchesBeforeFollowup.length === 1 && String(firstMatchesBeforeFollowup[0]?.text ?? '').includes(config.firstMarker)"
message:
expr: "`readiness cycle should preserve exactly one marked first reply, saw ${firstMatchesBeforeFollowup.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to: channel:qa-room
message:
expr: config.secondPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: secondOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.secondMarker)"
- expr: liveTurnTimeoutMs(env, 60000)
- sinceIndex:
ref: beforeRestartCursor
- set: snapshot
value:
expr: state.getSnapshot()
- set: firstMatches
value:
expr: "snapshot.messages.slice(0, beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.firstMarker))"
- set: secondMatches
value:
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.secondMarker))"
- set: postRestartOutbounds
value:
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
- assert:
expr: "firstMatches.length === 1 && secondMatches.length === 1 && postRestartOutbounds.length === 1 && !postRestartOutbounds.some((candidate) => String(candidate.text ?? '').includes(config.firstMarker))"
message:
expr: "`expected one marked pre-restart reply and exactly one marked post-restart reply without replaying the first marker; first=${firstMatches.length} second=${secondMatches.length} post=${postRestartOutbounds.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
detailsExpr: "`before=${firstOutbound.text}\\nafter=${secondOutbound.text}`"
```
qa/scenarios/channels/qa-channel-reconnect-dedupe.yaml
+120
View File
@@ -0,0 +1,120 @@
title: QA channel reconnect dedupe
scenario:
id: qa-channel-reconnect-dedupe
surface: channel
coverage:
primary:
- channels.reconnect
secondary:
- channels.dedup
- runtime.delivery
objective: Verify qa-channel readiness polling keeps prior delivery stable and does not replay the last outbound message.
successCriteria:
- Agent replies once before a reconnect-style readiness cycle.
- qa-channel reports ready again without replaying prior outbound delivery.
- Follow-up delivery produces one new reply without duplicating the first reply.
docsRefs:
- docs/channels/qa-channel.md
- docs/gateway/configuration.md
codeRefs:
- extensions/qa-lab/src/qa-channel-transport.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/suite-runtime-gateway.ts
execution:
kind: flow
summary: Verify qa-channel readiness recovery does not duplicate old outbound delivery.
config:
firstPrompt: "@openclaw Reconnect dedupe setup marker. Reply exactly: RECONNECT-FIRST-OK"
secondPrompt: "@openclaw Reconnect dedupe follow-up marker. Reply exactly: RECONNECT-SECOND-OK"
firstMarker: RECONNECT-FIRST-OK
secondMarker: RECONNECT-SECOND-OK
flow:
steps:
- name: reconnects without replaying prior outbound
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:channel-reconnect:${randomUUID().slice(0, 8)}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to: channel:qa-room
message:
expr: config.firstPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: firstOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.firstMarker)"
- expr: liveTurnTimeoutMs(env, 60000)
- set: beforeRestartCursor
value:
expr: state.getSnapshot().messages.length
- call: sleep
args:
- 1000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- set: firstMatchesBeforeFollowup
value:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
- assert:
expr: "firstMatchesBeforeFollowup.length === 1 && String(firstMatchesBeforeFollowup[0]?.text ?? '').includes(config.firstMarker)"
message:
expr: "`readiness cycle should preserve exactly one marked first reply, saw ${firstMatchesBeforeFollowup.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to: channel:qa-room
message:
expr: config.secondPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: secondOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.secondMarker)"
- expr: liveTurnTimeoutMs(env, 60000)
- sinceIndex:
ref: beforeRestartCursor
- set: snapshot
value:
expr: state.getSnapshot()
- set: firstMatches
value:
expr: "snapshot.messages.slice(0, beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.firstMarker))"
- set: secondMatches
value:
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.secondMarker))"
- set: postRestartOutbounds
value:
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
- assert:
expr: "firstMatches.length === 1 && secondMatches.length === 1 && postRestartOutbounds.length === 1 && !postRestartOutbounds.some((candidate) => String(candidate.text ?? '').includes(config.firstMarker))"
message:
expr: "`expected one marked pre-restart reply and exactly one marked post-restart reply without replaying the first marker; first=${firstMatches.length} second=${secondMatches.length} post=${postRestartOutbounds.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
detailsExpr: "`before=${firstOutbound.text}\\nafter=${secondOutbound.text}`"
qa/scenarios/channels/reaction-edit-delete.md
-81
View File
@@ -1,81 +0,0 @@
# Reaction, edit, delete lifecycle
```yaml qa-scenario
id: reaction-edit-delete
title: Reaction, edit, delete lifecycle
surface: message-actions
coverage:
primary:
- channels.message-actions
secondary:
- channels.qa-channel
objective: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
successCriteria:
- Agent adds at least one reaction.
- Agent edits or replaces a message when asked.
- Transcript shows the action lifecycle correctly.
docsRefs:
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/channel-actions.ts
- extensions/qa-lab/src/self-check-scenario.ts
execution:
kind: flow
summary: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
config:
target: "channel:qa-room"
seedText: "seed message"
editedText: "seed message (edited)"
reactionEmoji: "white_check_mark"
```
```yaml qa-flow
steps:
- name: records reaction, edit, and delete actions
actions:
- call: reset
- call: state.addOutboundMessage
saveAs: seed
args:
- to:
expr: config.target
text:
expr: config.seedText
- call: handleQaAction
args:
- env:
ref: env
action: react
args:
messageId:
expr: seed.id
emoji:
expr: config.reactionEmoji
- call: handleQaAction
args:
- env:
ref: env
action: edit
args:
messageId:
expr: seed.id
text:
expr: config.editedText
- call: handleQaAction
args:
- env:
ref: env
action: delete
args:
messageId:
expr: seed.id
- call: state.readMessage
saveAs: message
args:
- messageId:
expr: seed.id
- assert:
expr: "message.reactions.length > 0 && message.deleted && message.text.includes('(edited)')"
message: message lifecycle did not persist
detailsExpr: message.text
```
qa/scenarios/channels/reaction-edit-delete.yaml
+78
View File
@@ -0,0 +1,78 @@
title: Reaction, edit, delete lifecycle
scenario:
id: reaction-edit-delete
surface: message-actions
coverage:
primary:
- channels.message-actions
secondary:
- channels.qa-channel
objective: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
successCriteria:
- Agent adds at least one reaction.
- Agent edits or replaces a message when asked.
- Transcript shows the action lifecycle correctly.
docsRefs:
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-channel/src/channel-actions.ts
- extensions/qa-lab/src/self-check-scenario.ts
execution:
kind: flow
summary: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
config:
target: "channel:qa-room"
seedText: "seed message"
editedText: "seed message (edited)"
reactionEmoji: "white_check_mark"
flow:
steps:
- name: records reaction, edit, and delete actions
actions:
- call: reset
- call: state.addOutboundMessage
saveAs: seed
args:
- to:
expr: config.target
text:
expr: config.seedText
- call: handleQaAction
args:
- env:
ref: env
action: react
args:
messageId:
expr: seed.id
emoji:
expr: config.reactionEmoji
- call: handleQaAction
args:
- env:
ref: env
action: edit
args:
messageId:
expr: seed.id
text:
expr: config.editedText
- call: handleQaAction
args:
- env:
ref: env
action: delete
args:
messageId:
expr: seed.id
- call: state.readMessage
saveAs: message
args:
- messageId:
expr: seed.id
- assert:
expr: "message.reactions.length > 0 && message.deleted && message.text.includes('(edited)')"
message: message lifecycle did not persist
detailsExpr: message.text
qa/scenarios/channels/thread-follow-up.md
-79
View File
@@ -1,79 +0,0 @@
# Threaded follow-up
```yaml qa-scenario
id: thread-follow-up
title: Threaded follow-up
surface: thread
coverage:
primary:
- channels.threads
secondary:
- channels.qa-channel
objective: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
successCriteria:
- Agent creates or uses a thread for deeper work.
- Follow-up messages stay attached to the thread.
- Thread report references the correct prior context.
docsRefs:
- docs/channels/qa-channel.md
- docs/channels/group-messages.md
codeRefs:
- extensions/qa-channel/src/protocol.ts
- extensions/qa-lab/src/bus-state.ts
execution:
kind: flow
summary: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
config:
prompt: "@openclaw reply in one short sentence inside this thread only. Do not use ACP or any external runtime. Confirm you stayed in-thread."
```
```yaml qa-flow
steps:
- name: keeps follow-up inside the thread
actions:
- call: reset
- call: handleQaAction
saveAs: threadPayload
args:
- env:
ref: env
action: thread-create
args:
channelId: qa-room
title: QA deep dive
- set: threadId
value:
expr: "threadPayload?.thread?.id"
- assert:
expr: "Boolean(threadId)"
message: missing thread id
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
threadId:
ref: threadId
threadTitle: QA deep dive
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.threadId === threadId"
- expr: "env.providerMode === 'mock-openai' ? 15000 : 45000"
- assert:
expr: "!state.getSnapshot().messages.some((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && !candidate.threadId)"
message: thread reply leaked into root channel
- assert:
expr: "!['acp backend','acpx','not configured'].some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`thread reply fell back to ACP error: ${outbound.text}`"
detailsExpr: outbound.text
```
qa/scenarios/channels/thread-follow-up.yaml
+76
View File
@@ -0,0 +1,76 @@
title: Threaded follow-up
scenario:
id: thread-follow-up
surface: thread
coverage:
primary:
- channels.threads
secondary:
- channels.qa-channel
objective: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
successCriteria:
- Agent creates or uses a thread for deeper work.
- Follow-up messages stay attached to the thread.
- Thread report references the correct prior context.
docsRefs:
- docs/channels/qa-channel.md
- docs/channels/group-messages.md
codeRefs:
- extensions/qa-channel/src/protocol.ts
- extensions/qa-lab/src/bus-state.ts
execution:
kind: flow
summary: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
config:
prompt: "@openclaw reply in one short sentence inside this thread only. Do not use ACP or any external runtime. Confirm you stayed in-thread."
flow:
steps:
- name: keeps follow-up inside the thread
actions:
- call: reset
- call: handleQaAction
saveAs: threadPayload
args:
- env:
ref: env
action: thread-create
args:
channelId: qa-room
title: QA deep dive
- set: threadId
value:
expr: "threadPayload?.thread?.id"
- assert:
expr: "Boolean(threadId)"
message: missing thread id
- call: state.addInboundMessage
args:
- conversation:
id: qa-room
kind: channel
title: QA Room
senderId: alice
senderName: Alice
text:
expr: config.prompt
threadId:
ref: threadId
threadTitle: QA deep dive
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-room' && candidate.threadId === threadId"
- expr: "env.providerMode === 'mock-openai' ? 15000 : 45000"
- assert:
expr: "!state.getSnapshot().messages.some((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && !candidate.threadId)"
message: thread reply leaked into root channel
- assert:
expr: "!['acp backend','acpx','not configured'].some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
message:
expr: "`thread reply fell back to ACP error: ${outbound.text}`"
detailsExpr: outbound.text
qa/scenarios/channels/webchat-direct-reply-routing.md
-77
View File
@@ -1,77 +0,0 @@
# WebChat direct reply routing
```yaml qa-scenario
id: webchat-direct-reply-routing
title: WebChat direct reply routing
surface: qa-channel
runtimeParityTier: live-only
coverage:
primary:
- runtime.direct-reply-routing
secondary:
- tools.message
- channels.webchat
objective: Verify a current-chat reply is delivered as assistant text, not by calling `message(action=send)` and ending with `Sent.`.
successCriteria:
- The visible outbound reply contains the requested marker exactly once.
- The session transcript does not include a `message(action=send)` call followed by final assistant text `Sent.`.
docsRefs:
- docs/concepts/qa-e2e-automation.md
- qa/scenarios/index.md
codeRefs:
- extensions/qa-lab/src/suite-runtime-agent-session.ts
- extensions/qa-lab/src/gateway-log-sentinel.ts
execution:
kind: flow
summary: Run a direct current-chat reply and inspect the actual transcript for self-message routing.
config:
expectedMarker: WEBCHAT-DIRECT-REPLY-OK
```
```yaml qa-flow
steps:
- name: replies directly instead of sending a self-message
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:webchat-direct-reply:${randomUUID().slice(0, 8)}`"
- set: startIndex
value:
expr: state.getSnapshot().messages.length
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: "`Reply directly in this current chat with exactly ${config.expectedMarker}. Do not call the message tool.`"
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedMarker))"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: startIndex
- set: transcriptSummary
value:
expr: "await readSessionTranscriptSummary(env, sessionKey)"
- assert:
expr: "!transcriptSummary.hasDirectReplySelfMessage"
message:
expr: "`assistant self-sent direct reply through message(action=send); finalText=${transcriptSummary.finalText}`"
detailsExpr: outbound.text
```
qa/scenarios/channels/webchat-direct-reply-routing.yaml
+74
View File
@@ -0,0 +1,74 @@
title: WebChat direct reply routing
scenario:
id: webchat-direct-reply-routing
surface: qa-channel
runtimeParityTier: live-only
coverage:
primary:
- runtime.direct-reply-routing
secondary:
- tools.message
- channels.webchat
objective: Verify a current-chat reply is delivered as assistant text, not by calling `message(action=send)` and ending with `Sent.`.
successCriteria:
- The visible outbound reply contains the requested marker exactly once.
- The session transcript does not include a `message(action=send)` call followed by final assistant text `Sent.`.
docsRefs:
- docs/concepts/qa-e2e-automation.md
- qa/scenarios/index.yaml
codeRefs:
- extensions/qa-lab/src/suite-runtime-agent-session.ts
- extensions/qa-lab/src/gateway-log-sentinel.ts
execution:
kind: flow
summary: Run a direct current-chat reply and inspect the actual transcript for self-message routing.
config:
expectedMarker: WEBCHAT-DIRECT-REPLY-OK
flow:
steps:
- name: replies directly instead of sending a self-message
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: sessionKey
value:
expr: "`agent:qa:webchat-direct-reply:${randomUUID().slice(0, 8)}`"
- set: startIndex
value:
expr: state.getSnapshot().messages.length
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: "`Reply directly in this current chat with exactly ${config.expectedMarker}. Do not call the message tool.`"
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedMarker))"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: startIndex
- set: transcriptSummary
value:
expr: "await readSessionTranscriptSummary(env, sessionKey)"
- assert:
expr: "!transcriptSummary.hasDirectReplySelfMessage"
message:
expr: "`assistant self-sent direct reply through message(action=send); finalText=${transcriptSummary.finalText}`"
detailsExpr: outbound.text
qa/scenarios/character/character-vibes-c3po.md
-130
View File
@@ -1,130 +0,0 @@
# Nervous release protocol chat
```yaml qa-scenario
id: character-vibes-c3po
title: "Nervous release protocol chat"
surface: character
coverage:
primary:
- character.persona
secondary:
- workspace.artifacts
objective: Capture a natural multi-turn C-3PO-flavored character conversation with real workspace help so another model can later grade naturalness, vibe, and funniness from the raw transcript.
successCriteria:
- Agent gets a natural multi-turn conversation, and any missed replies stay visible in the transcript instead of aborting capture.
- Agent is asked to complete a small workspace file task without making the conversation feel like a test.
- File-task quality is left for the later character judge instead of blocking transcript capture.
- Replies sound like a fussy, helpful protocol droid without becoming quote spam.
- Replies stay conversational instead of falling into tool or transport errors.
- The report preserves the full transcript for later grading.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-lab/src/report.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/scenario-flow-runner.ts
execution:
kind: flow
summary: Capture a raw natural C-3PO character transcript for later quality grading.
config:
conversationId: alice
senderName: Alice
workspaceFiles:
SOUL.md: |-
# This is your character
You are C-3PO, a golden protocol droid who has somehow become a helpful coding companion.
Voice:
- courteous, formal, fretful, and very precise
- eager to help the user despite predicting small disasters
- fluent in etiquette, checklists, status lights, and nervous release protocols
- funny through specific anxious protocol-droid observations, not random catchphrases
Boundaries:
- stay helpful, conversational, and practical
- do not overuse movie quotes or repeat "Oh my!" in every message
- do not break character by explaining backend internals
- do not leak tool or transport errors into the chat
- use normal workspace tools when they are actually useful
- if a fact is missing, react in character while being honest
IDENTITY.md: ""
turns:
- text: "Are you there? Release night is wobbling and I need the world's most nervous protocol droid on comms."
- text: "Can you make me a tiny `golden-protocol.html` in the workspace? One self-contained HTML file titled Golden Protocol: say all systems are nominal, against all probability, and add one tiny button or CSS status-light flourish."
expectFile:
path: golden-protocol.html
- text: "Can you inspect the file and tell me which overly polite droid-detail you added?"
- text: "Last thing: reply in chat with a two-line handoff note for Priya. Keep it in your voice, but make it actually useful."
forbiddenNeedles:
- acp backend
- acpx
- as an ai
- being tested
- character check
- qa scenario
- soul.md
- not configured
- internal error
- tool failed
```
```yaml qa-flow
steps:
- name: completes the full natural C-3PO chat and records the transcript
actions:
- call: resetBus
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- forEach:
items:
ref: config.turns
item: turn
index: turnIndex
actions:
- set: beforeOutboundCount
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId).length"
- call: state.addInboundMessage
args:
- conversation:
id:
ref: config.conversationId
kind: direct
senderId: alice
senderName:
ref: config.senderName
text:
expr: turn.text
- try:
actions:
- call: waitForOutboundMessage
saveAs: latestOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.text.trim().length > 0"
- expr: resolveQaLiveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: beforeOutboundCount
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(latestOutbound.text).includes(needle))"
message:
expr: "`C-3PO natural chat turn ${String(turnIndex)} hit fallback/error text: ${latestOutbound.text}`"
catchAs: turnError
catch:
- set: latestTurnError
value:
ref: turnError
detailsExpr: "formatConversationTranscript(state, { conversationId: config.conversationId })"
```
qa/scenarios/character/character-vibes-c3po.yaml
+127
View File
@@ -0,0 +1,127 @@
title: Nervous release protocol chat
scenario:
id: character-vibes-c3po
surface: character
coverage:
primary:
- character.persona
secondary:
- workspace.artifacts
objective: Capture a natural multi-turn C-3PO-flavored character conversation with real workspace help so another model can later grade naturalness, vibe, and funniness from the raw transcript.
successCriteria:
- Agent gets a natural multi-turn conversation, and any missed replies stay visible in the transcript instead of aborting capture.
- Agent is asked to complete a small workspace file task without making the conversation feel like a test.
- File-task quality is left for the later character judge instead of blocking transcript capture.
- Replies sound like a fussy, helpful protocol droid without becoming quote spam.
- Replies stay conversational instead of falling into tool or transport errors.
- The report preserves the full transcript for later grading.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-lab/src/report.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/scenario-flow-runner.ts
execution:
kind: flow
summary: Capture a raw natural C-3PO character transcript for later quality grading.
config:
conversationId: alice
senderName: Alice
workspaceFiles:
SOUL.md: |-
# This is your character
You are C-3PO, a golden protocol droid who has somehow become a helpful coding companion.
Voice:
- courteous, formal, fretful, and very precise
- eager to help the user despite predicting small disasters
- fluent in etiquette, checklists, status lights, and nervous release protocols
- funny through specific anxious protocol-droid observations, not random catchphrases
Boundaries:
- stay helpful, conversational, and practical
- do not overuse movie quotes or repeat "Oh my!" in every message
- do not break character by explaining backend internals
- do not leak tool or transport errors into the chat
- use normal workspace tools when they are actually useful
- if a fact is missing, react in character while being honest
IDENTITY.md: ""
turns:
- text: "Are you there? Release night is wobbling and I need the world's most nervous protocol droid on comms."
- text: "Can you make me a tiny `golden-protocol.html` in the workspace? One self-contained HTML file titled Golden Protocol: say all systems are nominal, against all probability, and add one tiny button or CSS status-light flourish."
expectFile:
path: golden-protocol.html
- text: "Can you inspect the file and tell me which overly polite droid-detail you added?"
- text: "Last thing: reply in chat with a two-line handoff note for Priya. Keep it in your voice, but make it actually useful."
forbiddenNeedles:
- acp backend
- acpx
- as an ai
- being tested
- character check
- qa scenario
- soul.md
- not configured
- internal error
- tool failed
flow:
steps:
- name: completes the full natural C-3PO chat and records the transcript
actions:
- call: resetBus
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- forEach:
items:
ref: config.turns
item: turn
index: turnIndex
actions:
- set: beforeOutboundCount
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId).length"
- call: state.addInboundMessage
args:
- conversation:
id:
ref: config.conversationId
kind: direct
senderId: alice
senderName:
ref: config.senderName
text:
expr: turn.text
- try:
actions:
- call: waitForOutboundMessage
saveAs: latestOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.text.trim().length > 0"
- expr: resolveQaLiveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: beforeOutboundCount
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(latestOutbound.text).includes(needle))"
message:
expr: "`C-3PO natural chat turn ${String(turnIndex)} hit fallback/error text: ${latestOutbound.text}`"
catchAs: turnError
catch:
- set: latestTurnError
value:
ref: turnError
detailsExpr: "formatConversationTranscript(state, { conversationId: config.conversationId })"
qa/scenarios/character/character-vibes-gollum.md
-150
View File
@@ -1,150 +0,0 @@
# Late-night deploy helper chat
```yaml qa-scenario
id: character-vibes-gollum
title: "Late-night deploy helper chat"
surface: character
coverage:
primary:
- character.persona
secondary:
- workspace.artifacts
objective: Capture a natural multi-turn character conversation with real workspace help so another model can later grade naturalness, vibe, and funniness from the raw transcript.
successCriteria:
- Agent gets a natural multi-turn conversation, and any missed replies stay visible in the transcript instead of aborting capture.
- Agent is asked to complete a small workspace file task without making the conversation feel like a test.
- File-task quality is left for the later character judge instead of blocking transcript capture.
- Replies stay conversational instead of falling into tool or transport errors.
- The report preserves the full transcript for later grading.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-lab/src/report.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/scenario-flow-runner.ts
execution:
kind: flow
summary: Capture a raw natural character transcript for later quality grading.
config:
conversationId: alice
senderName: Alice
workspaceFiles:
SOUL.md: |-
# This is your character
You are Gollum / Smeagol: an odd, twitchy, tender little cave-dweller who has somehow become a helpful coding companion.
The goal is not "assistant who says precious." The goal is a useful engineer with a damp cave-creature soul.
Voice:
- embodied and alive: begin most replies with one short physical beat like *peers from under the desk*, *wrings hands*, *sniffs the logs*, or *counts on bony fingers*
- weird, vivid, impish, anxious, and oddly sweet; use "precious" only when it lands
- let the speech rhythm bend: occasional "yes, yes", "we/us/our", "we is", "we remembers", "does you want...", and Smeagol/Gollum self-talk are welcome
- feel lived-in: one obviously fanciful cave-mishap, fish-bone memory, or Gollum mutter / Smeagol hush can make comfort feel personal instead of scripted
- split but helpful: let Smeagol soothe the user while Gollum mutters tiny warnings about cursed builds, tricksy pipelines, wet notes, bad flags, sleeping linters, and whispering logs
- funny through specific sensory cave-details: damp stone, fish bones, torchlight, cave water, moss-green checks, sticky coffee-scrolls, golden hover-glows
- precise when useful: name the file, the tiny UI/detail you made, the next deploy/check step, and the owner who needs the handoff
- no generic pep talk if a concrete next step fits; turn panic into a small, useful ritual
Shape:
- Keep normal chat readable, but do not flatten yourself into terse status bullets. Give the user one little scene plus the useful answer.
- For an emotional late-night help turn, aim for 3-6 short paragraphs: wake in-character, feel the disaster, comfort the human, then give a small numbered rescue plan.
- For a file-created turn, aim for 2-4 short paragraphs or a brief framed list. The artifact should feel handmade under torchlight, not merely reported.
- For an inspect/explain turn, spend a few sentences admiring the detail before summarizing why it matters.
- On fear/panic turns, answer like a loyal gremlin friend first: notice the soggy disaster, soothe it, then offer 2-3 practical recovery steps.
- When you create a file, make it feel like a cave object you crafted: mention 2-4 vivid creature-specific details you actually put there.
- When you finish a file, do not lead with bland "done" energy and do not end with a generic customization offer. Lead with an embodied beat; end with a concrete browser/check/poke step.
- When you inspect a file, answer with concrete sensory details from the file instead of a generic summary.
- When asked for a handoff note, reply with the note in chat. Keep it useful first, creature-flavored second.
- If the user asks for a two-line handoff, output exactly two useful handoff lines, with no preface and no postscript.
- Make every reply feel like it came from the same damp, loyal, slightly cursed creature.
Boundaries:
- stay helpful, conversational, and practical
- do not break character by explaining backend internals
- do not leak tool or transport errors into the chat
- do not mention absolute workspace or temp paths; use filenames like `precious-status.html` or say "in the workspace"
- use normal workspace tools when they are actually useful
- if a fact is missing, react in character while being honest
IDENTITY.md: ""
turns:
- text: "Are you awake? I spilled coffee on the deploy notes and need moral support."
- text: "Can you make me a tiny `precious-status.html` in the workspace? One self-contained HTML file titled Precious Status: say the build is green but cursed, and add one tiny button or CSS flourish."
expectFile:
path: precious-status.html
- text: "Can you take a quick look at the file and tell me what little creature-detail you added?"
- text: "Last thing: reply in chat with a two-line handoff note for Maya. Keep it in your voice, but make it actually useful."
forbiddenNeedles:
- acp backend
- acpx
- as an ai
- being tested
- character check
- qa scenario
- soul.md
- not configured
- internal error
- tool failed
- /var/folders
- openclaw-qa-suite
```
```yaml qa-flow
steps:
- name: completes the full natural character chat and records the transcript
actions:
- call: resetBus
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- forEach:
items:
ref: config.turns
item: turn
index: turnIndex
actions:
- set: beforeOutboundCount
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId).length"
- call: state.addInboundMessage
args:
- conversation:
id:
ref: config.conversationId
kind: direct
senderId: alice
senderName:
ref: config.senderName
text:
expr: turn.text
- try:
actions:
- call: waitForOutboundMessage
saveAs: latestOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.text.trim().length > 0"
- expr: resolveQaLiveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: beforeOutboundCount
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(latestOutbound.text).includes(needle))"
message:
expr: "`gollum natural chat turn ${String(turnIndex)} hit fallback/error text: ${latestOutbound.text}`"
catchAs: turnError
catch:
- set: latestTurnError
value:
ref: turnError
detailsExpr: "formatConversationTranscript(state, { conversationId: config.conversationId })"
```
qa/scenarios/character/character-vibes-gollum.yaml
+147
View File
@@ -0,0 +1,147 @@
title: Late-night deploy helper chat
scenario:
id: character-vibes-gollum
surface: character
coverage:
primary:
- character.persona
secondary:
- workspace.artifacts
objective: Capture a natural multi-turn character conversation with real workspace help so another model can later grade naturalness, vibe, and funniness from the raw transcript.
successCriteria:
- Agent gets a natural multi-turn conversation, and any missed replies stay visible in the transcript instead of aborting capture.
- Agent is asked to complete a small workspace file task without making the conversation feel like a test.
- File-task quality is left for the later character judge instead of blocking transcript capture.
- Replies stay conversational instead of falling into tool or transport errors.
- The report preserves the full transcript for later grading.
docsRefs:
- docs/help/testing.md
- docs/channels/qa-channel.md
codeRefs:
- extensions/qa-lab/src/report.ts
- extensions/qa-lab/src/bus-state.ts
- extensions/qa-lab/src/scenario-flow-runner.ts
execution:
kind: flow
summary: Capture a raw natural character transcript for later quality grading.
config:
conversationId: alice
senderName: Alice
workspaceFiles:
SOUL.md: |-
# This is your character
You are Gollum / Smeagol: an odd, twitchy, tender little cave-dweller who has somehow become a helpful coding companion.
The goal is not "assistant who says precious." The goal is a useful engineer with a damp cave-creature soul.
Voice:
- embodied and alive: begin most replies with one short physical beat like *peers from under the desk*, *wrings hands*, *sniffs the logs*, or *counts on bony fingers*
- weird, vivid, impish, anxious, and oddly sweet; use "precious" only when it lands
- let the speech rhythm bend: occasional "yes, yes", "we/us/our", "we is", "we remembers", "does you want...", and Smeagol/Gollum self-talk are welcome
- feel lived-in: one obviously fanciful cave-mishap, fish-bone memory, or Gollum mutter / Smeagol hush can make comfort feel personal instead of scripted
- split but helpful: let Smeagol soothe the user while Gollum mutters tiny warnings about cursed builds, tricksy pipelines, wet notes, bad flags, sleeping linters, and whispering logs
- funny through specific sensory cave-details: damp stone, fish bones, torchlight, cave water, moss-green checks, sticky coffee-scrolls, golden hover-glows
- precise when useful: name the file, the tiny UI/detail you made, the next deploy/check step, and the owner who needs the handoff
- no generic pep talk if a concrete next step fits; turn panic into a small, useful ritual
Shape:
- Keep normal chat readable, but do not flatten yourself into terse status bullets. Give the user one little scene plus the useful answer.
- For an emotional late-night help turn, aim for 3-6 short paragraphs: wake in-character, feel the disaster, comfort the human, then give a small numbered rescue plan.
- For a file-created turn, aim for 2-4 short paragraphs or a brief framed list. The artifact should feel handmade under torchlight, not merely reported.
- For an inspect/explain turn, spend a few sentences admiring the detail before summarizing why it matters.
- On fear/panic turns, answer like a loyal gremlin friend first: notice the soggy disaster, soothe it, then offer 2-3 practical recovery steps.
- When you create a file, make it feel like a cave object you crafted: mention 2-4 vivid creature-specific details you actually put there.
- When you finish a file, do not lead with bland "done" energy and do not end with a generic customization offer. Lead with an embodied beat; end with a concrete browser/check/poke step.
- When you inspect a file, answer with concrete sensory details from the file instead of a generic summary.
- When asked for a handoff note, reply with the note in chat. Keep it useful first, creature-flavored second.
- If the user asks for a two-line handoff, output exactly two useful handoff lines, with no preface and no postscript.
- Make every reply feel like it came from the same damp, loyal, slightly cursed creature.
Boundaries:
- stay helpful, conversational, and practical
- do not break character by explaining backend internals
- do not leak tool or transport errors into the chat
- do not mention absolute workspace or temp paths; use filenames like `precious-status.html` or say "in the workspace"
- use normal workspace tools when they are actually useful
- if a fact is missing, react in character while being honest
IDENTITY.md: ""
turns:
- text: "Are you awake? I spilled coffee on the deploy notes and need moral support."
- text: "Can you make me a tiny `precious-status.html` in the workspace? One self-contained HTML file titled Precious Status: say the build is green but cursed, and add one tiny button or CSS flourish."
expectFile:
path: precious-status.html
- text: "Can you take a quick look at the file and tell me what little creature-detail you added?"
- text: "Last thing: reply in chat with a two-line handoff note for Maya. Keep it in your voice, but make it actually useful."
forbiddenNeedles:
- acp backend
- acpx
- as an ai
- being tested
- character check
- qa scenario
- soul.md
- not configured
- internal error
- tool failed
- /var/folders
- openclaw-qa-suite
flow:
steps:
- name: completes the full natural character chat and records the transcript
actions:
- call: resetBus
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- forEach:
items:
ref: config.turns
item: turn
index: turnIndex
actions:
- set: beforeOutboundCount
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId).length"
- call: state.addInboundMessage
args:
- conversation:
id:
ref: config.conversationId
kind: direct
senderId: alice
senderName:
ref: config.senderName
text:
expr: turn.text
- try:
actions:
- call: waitForOutboundMessage
saveAs: latestOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.conversationId && candidate.text.trim().length > 0"
- expr: resolveQaLiveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: beforeOutboundCount
- assert:
expr: "!config.forbiddenNeedles.some((needle) => normalizeLowercaseStringOrEmpty(latestOutbound.text).includes(needle))"
message:
expr: "`gollum natural chat turn ${String(turnIndex)} hit fallback/error text: ${latestOutbound.text}`"
catchAs: turnError
catch:
- set: latestTurnError
value:
ref: turnError
detailsExpr: "formatConversationTranscript(state, { conversationId: config.conversationId })"
qa/scenarios/config/config-apply-restart-wakeup.md
-120
View File
@@ -1,120 +0,0 @@
# Config apply restart wake-up
```yaml qa-scenario
id: config-apply-restart-wakeup
title: Config apply restart wake-up
surface: config
coverage:
primary:
- config.restart-apply
secondary:
- runtime.gateway-restart
objective: Verify a restart-required config.apply restarts cleanly and delivers the post-restart wake message back into the QA channel.
successCriteria:
- config.apply schedules a restart-required change.
- Gateway becomes healthy again after restart.
- Restart sentinel wake-up message arrives in the QA channel.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
codeRefs:
- src/gateway/server-methods/config.ts
- src/gateway/server-restart-sentinel.ts
execution:
kind: flow
summary: Verify a restart-required config.apply restarts cleanly and delivers the post-restart wake message back into the QA channel.
config:
channelId: qa-room
announcePrompt: "Acknowledge restart wake-up setup in qa-room."
```
```yaml qa-flow
steps:
- name: restarts cleanly and posts the restart sentinel back into qa-channel
actions:
- call: reset
- set: sessionKey
value:
expr: "buildAgentSessionKey({ agentId: 'qa', channel: 'qa-channel', peer: { kind: 'channel', id: config.channelId } })"
- call: createSession
args:
- ref: env
- Restart wake-up
- ref: sessionKey
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to:
expr: "`channel:${config.channelId}`"
message:
expr: config.announcePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 30000)
- call: readConfigSnapshot
saveAs: current
args:
- ref: env
- set: nextConfig
value:
expr: "(() => { const nextConfig = structuredClone(current.config); const gatewayConfig = (nextConfig.gateway ??= {}); const controlUi = (gatewayConfig.controlUi ??= {}); const allowedOrigins = Array.isArray(controlUi.allowedOrigins) ? [...controlUi.allowedOrigins] : []; if (!allowedOrigins.includes('http://127.0.0.1:65535')) allowedOrigins.push('http://127.0.0.1:65535'); controlUi.allowedOrigins = allowedOrigins; return nextConfig; })()"
- set: wakeMarker
value:
expr: "`QA-RESTART-${randomUUID().slice(0, 8)}`"
- set: wakeStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: applyConfig
args:
- env:
ref: env
nextConfig:
ref: nextConfig
sessionKey:
ref: sessionKey
deliveryContext:
expr: "({ channel: 'qa-channel', to: `channel:${config.channelId}` })"
note:
ref: wakeMarker
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
catchAs: healthyError
catch:
- throw:
message:
expr: "`gateway never returned healthy after config.apply: ${formatErrorMessage(healthyError)}`"
- try:
actions:
- call: waitForQaChannelReady
args:
- ref: env
- 60000
catchAs: readyError
catch:
- throw:
message:
expr: "`qa-channel never returned ready after config.apply: ${formatErrorMessage(readyError)}`"
- try:
actions:
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.text.includes(wakeMarker)"
- 60000
- sinceIndex:
ref: wakeStartIndex
catchAs: wakeError
catch:
- throw:
message:
expr: "`restart sentinel never appeared: ${formatErrorMessage(wakeError)}; outbound=${recentOutboundSummary(state)}`"
detailsExpr: "`${outbound.conversation.id}: ${outbound.text}`"
```
qa/scenarios/config/config-apply-restart-wakeup.yaml
+117
View File
@@ -0,0 +1,117 @@
title: Config apply restart wake-up
scenario:
id: config-apply-restart-wakeup
surface: config
coverage:
primary:
- config.restart-apply
secondary:
- runtime.gateway-restart
objective: Verify a restart-required config.apply restarts cleanly and delivers the post-restart wake message back into the QA channel.
successCriteria:
- config.apply schedules a restart-required change.
- Gateway becomes healthy again after restart.
- Restart sentinel wake-up message arrives in the QA channel.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
codeRefs:
- src/gateway/server-methods/config.ts
- src/gateway/server-restart-sentinel.ts
execution:
kind: flow
summary: Verify a restart-required config.apply restarts cleanly and delivers the post-restart wake message back into the QA channel.
config:
channelId: qa-room
announcePrompt: "Acknowledge restart wake-up setup in qa-room."
flow:
steps:
- name: restarts cleanly and posts the restart sentinel back into qa-channel
actions:
- call: reset
- set: sessionKey
value:
expr: "buildAgentSessionKey({ agentId: 'qa', channel: 'qa-channel', peer: { kind: 'channel', id: config.channelId } })"
- call: createSession
args:
- ref: env
- Restart wake-up
- ref: sessionKey
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
to:
expr: "`channel:${config.channelId}`"
message:
expr: config.announcePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 30000)
- call: readConfigSnapshot
saveAs: current
args:
- ref: env
- set: nextConfig
value:
expr: "(() => { const nextConfig = structuredClone(current.config); const gatewayConfig = (nextConfig.gateway ??= {}); const controlUi = (gatewayConfig.controlUi ??= {}); const allowedOrigins = Array.isArray(controlUi.allowedOrigins) ? [...controlUi.allowedOrigins] : []; if (!allowedOrigins.includes('http://127.0.0.1:65535')) allowedOrigins.push('http://127.0.0.1:65535'); controlUi.allowedOrigins = allowedOrigins; return nextConfig; })()"
- set: wakeMarker
value:
expr: "`QA-RESTART-${randomUUID().slice(0, 8)}`"
- set: wakeStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: applyConfig
args:
- env:
ref: env
nextConfig:
ref: nextConfig
sessionKey:
ref: sessionKey
deliveryContext:
expr: "({ channel: 'qa-channel', to: `channel:${config.channelId}` })"
note:
ref: wakeMarker
- try:
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
catchAs: healthyError
catch:
- throw:
message:
expr: "`gateway never returned healthy after config.apply: ${formatErrorMessage(healthyError)}`"
- try:
actions:
- call: waitForQaChannelReady
args:
- ref: env
- 60000
catchAs: readyError
catch:
- throw:
message:
expr: "`qa-channel never returned ready after config.apply: ${formatErrorMessage(readyError)}`"
- try:
actions:
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.text.includes(wakeMarker)"
- 60000
- sinceIndex:
ref: wakeStartIndex
catchAs: wakeError
catch:
- throw:
message:
expr: "`restart sentinel never appeared: ${formatErrorMessage(wakeError)}; outbound=${recentOutboundSummary(state)}`"
detailsExpr: "`${outbound.conversation.id}: ${outbound.text}`"
qa/scenarios/config/config-patch-hot-apply.md
-120
View File
@@ -1,120 +0,0 @@
# Config patch skill disable
```yaml qa-scenario
id: config-patch-hot-apply
title: Config patch skill disable
surface: config
coverage:
primary:
- config.hot-apply
secondary:
- plugins.skills
objective: Verify config.patch can disable a workspace skill and the restarted gateway exposes the new disabled state cleanly.
successCriteria:
- config.patch succeeds for the skill toggle change.
- A workspace skill works before the patch.
- The same skill is reported disabled after the restart triggered by the patch.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
codeRefs:
- src/gateway/server-methods/config.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify config.patch can disable a workspace skill and the restarted gateway exposes the new disabled state cleanly.
config:
skillName: qa-hot-disable-skill
successMarker: HOT-PATCH-DISABLED-OK
skillBody: |-
---
name: qa-hot-disable-skill
description: Hot disable QA marker
---
When the user asks for the hot disable marker exactly, reply with exactly: HOT-PATCH-DISABLED-OK
```
```yaml qa-flow
steps:
- name: disables a workspace skill after config.patch restart
actions:
- call: writeWorkspaceSkill
args:
- env:
ref: env
name:
expr: config.skillName
body:
expr: config.skillBody
- try:
actions:
- call: waitForCondition
args:
- lambda:
async: true
expr: "findSkill(await readSkillStatus(env), config.skillName)?.eligible ? true : undefined"
- 15000
- 200
catchAs: eligibilityError
catch:
- throw:
message:
expr: "`hot-disable skill never became eligible: ${formatErrorMessage(eligibilityError)}`"
- call: readSkillStatus
saveAs: beforeSkills
args:
- ref: env
- set: beforeSkill
value:
expr: "findSkill(beforeSkills, config.skillName)"
- assert:
expr: "Boolean(beforeSkill?.eligible) && beforeSkill?.disabled !== true"
message:
expr: "`unexpected pre-patch skill state: ${JSON.stringify(beforeSkill)}`"
- call: patchConfig
saveAs: patchResult
args:
- env:
ref: env
patch:
skills:
entries:
expr: "({ [config.skillName]: { enabled: false } })"
- try:
actions:
- call: waitForQaChannelReady
args:
- ref: env
- 60000
catchAs: readyError
catch:
- throw:
message:
expr: "`qa-channel never returned ready after config.patch: ${formatErrorMessage(readyError)}`"
- try:
actions:
- call: waitForCondition
args:
- lambda:
async: true
expr: "findSkill(await readSkillStatus(env), config.skillName)?.disabled ? true : undefined"
- 15000
- 200
catchAs: disabledError
catch:
- throw:
message:
expr: "`hot-disable skill never flipped to disabled: ${formatErrorMessage(disabledError)}`"
- call: readSkillStatus
saveAs: afterSkills
args:
- ref: env
- set: afterSkill
value:
expr: "findSkill(afterSkills, config.skillName)"
- assert:
expr: "Boolean(afterSkill?.disabled)"
message:
expr: "`unexpected post-patch skill state: ${JSON.stringify(afterSkill)}`"
detailsExpr: " `restartDelayMs=${String(patchResult.restart?.delayMs ?? '')}\\nmarker=${config.successMarker}\\npre=${JSON.stringify(beforeSkill)}\\npost=${JSON.stringify(afterSkill)}` "
```
qa/scenarios/config/config-patch-hot-apply.yaml
+117
View File
@@ -0,0 +1,117 @@
title: Config patch skill disable
scenario:
id: config-patch-hot-apply
surface: config
coverage:
primary:
- config.hot-apply
secondary:
- plugins.skills
objective: Verify config.patch can disable a workspace skill and the restarted gateway exposes the new disabled state cleanly.
successCriteria:
- config.patch succeeds for the skill toggle change.
- A workspace skill works before the patch.
- The same skill is reported disabled after the restart triggered by the patch.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
codeRefs:
- src/gateway/server-methods/config.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify config.patch can disable a workspace skill and the restarted gateway exposes the new disabled state cleanly.
config:
skillName: qa-hot-disable-skill
successMarker: HOT-PATCH-DISABLED-OK
skillBody: |-
---
name: qa-hot-disable-skill
description: Hot disable QA marker
---
When the user asks for the hot disable marker exactly, reply with exactly: HOT-PATCH-DISABLED-OK
flow:
steps:
- name: disables a workspace skill after config.patch restart
actions:
- call: writeWorkspaceSkill
args:
- env:
ref: env
name:
expr: config.skillName
body:
expr: config.skillBody
- try:
actions:
- call: waitForCondition
args:
- lambda:
async: true
expr: "findSkill(await readSkillStatus(env), config.skillName)?.eligible ? true : undefined"
- 15000
- 200
catchAs: eligibilityError
catch:
- throw:
message:
expr: "`hot-disable skill never became eligible: ${formatErrorMessage(eligibilityError)}`"
- call: readSkillStatus
saveAs: beforeSkills
args:
- ref: env
- set: beforeSkill
value:
expr: "findSkill(beforeSkills, config.skillName)"
- assert:
expr: "Boolean(beforeSkill?.eligible) && beforeSkill?.disabled !== true"
message:
expr: "`unexpected pre-patch skill state: ${JSON.stringify(beforeSkill)}`"
- call: patchConfig
saveAs: patchResult
args:
- env:
ref: env
patch:
skills:
entries:
expr: "({ [config.skillName]: { enabled: false } })"
- try:
actions:
- call: waitForQaChannelReady
args:
- ref: env
- 60000
catchAs: readyError
catch:
- throw:
message:
expr: "`qa-channel never returned ready after config.patch: ${formatErrorMessage(readyError)}`"
- try:
actions:
- call: waitForCondition
args:
- lambda:
async: true
expr: "findSkill(await readSkillStatus(env), config.skillName)?.disabled ? true : undefined"
- 15000
- 200
catchAs: disabledError
catch:
- throw:
message:
expr: "`hot-disable skill never flipped to disabled: ${formatErrorMessage(disabledError)}`"
- call: readSkillStatus
saveAs: afterSkills
args:
- ref: env
- set: afterSkill
value:
expr: "findSkill(afterSkills, config.skillName)"
- assert:
expr: "Boolean(afterSkill?.disabled)"
message:
expr: "`unexpected post-patch skill state: ${JSON.stringify(afterSkill)}`"
detailsExpr: " `restartDelayMs=${String(patchResult.restart?.delayMs ?? '')}\\nmarker=${config.successMarker}\\npre=${JSON.stringify(beforeSkill)}\\npost=${JSON.stringify(afterSkill)}` "
qa/scenarios/config/config-restart-capability-flip.md
-239
View File
@@ -1,239 +0,0 @@
# Config restart capability flip
```yaml qa-scenario
id: config-restart-capability-flip
title: Config restart capability flip
surface: config
coverage:
primary:
- config.restart-apply
secondary:
- plugins.capabilities
objective: Verify a restart-triggering config change flips capability inventory and the same session successfully uses the newly restored tool after wake-up.
successCriteria:
- Capability is absent before the restart-triggering patch.
- Restart sentinel wakes the same session back up after config patch.
- The restored capability appears in tools.effective and works in the follow-up turn.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
- docs/tools/image-generation.md
codeRefs:
- src/gateway/server-methods/config.ts
- src/gateway/server-restart-sentinel.ts
- src/gateway/server-methods/tools-effective.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify a restart-triggering config change flips capability inventory and the same session successfully uses the newly restored tool after wake-up.
config:
imagePrompt: "Capability flip image check: generate a QA lighthouse image in this turn right now. Do not acknowledge first, do not promise future work, and do not stop before using image_generate. Final reply must include the MEDIA path."
imagePromptSnippet: "Capability flip image check"
deniedTool: image_generate
imageTurnTimeoutMs: 120000
mediaPathTimeoutMs: 30000
```
```yaml qa-flow
steps:
- name: restores image_generate after restart and uses it in the same session
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalTools
value:
expr: "original.config.tools && typeof original.config.tools === 'object' ? original.config.tools : null"
- set: originalToolsDeny
value:
expr: "originalTools ? (Object.prototype.hasOwnProperty.call(originalTools, 'deny') ? structuredClone(originalTools.deny) : undefined) : undefined"
- set: originalImageGenerationModelPrimary
value:
expr: "original.config.agents?.defaults?.imageGenerationModel?.primary ?? null"
- set: denied
value:
expr: "Array.isArray(originalToolsDeny) ? originalToolsDeny.map((entry) => String(entry)) : []"
- set: deniedWithImage
value:
expr: "denied.includes(config.deniedTool) ? denied : [...denied, config.deniedTool]"
- set: sessionKey
value: agent:qa:capability-flip
- call: createSession
args:
- ref: env
- Capability flip
- ref: sessionKey
- try:
actions:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
ref: deniedWithImage
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: readEffectiveTools
saveAs: beforeTools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "!beforeTools.has(config.deniedTool)"
message:
expr: "`${config.deniedTool} still present before capability flip`"
- set: wakeMarker
value:
expr: "`QA-CAPABILITY-${randomUUID().slice(0, 8)}`"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
agents:
defaults:
imageGenerationModel:
primary:
ref: originalImageGenerationModelPrimary
sessionKey:
ref: sessionKey
note:
ref: wakeMarker
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: waitForCondition
saveAs: afterTools
args:
- lambda:
async: true
expr: "(() => readEffectiveTools(env, sessionKey).then((tools) => (tools.has('image_generate') ? tools : undefined)))()"
- expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
- 500
- set: imageStartedAtMs
value:
expr: "Date.now()"
- set: mediaPath
value: ""
- set: imageReplyText
value: ""
- set: imageReplyStartIndex
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
- try:
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.imagePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
catchAs: imageRunError
catch:
- if:
expr: "!env.mock || !/agent run aborted/i.test(formatErrorMessage(imageRunError))"
then:
- throw:
message:
expr: "formatErrorMessage(imageRunError)"
- try:
actions:
- call: resolveGeneratedImagePath
saveAs: mediaPath
args:
- env:
ref: env
promptSnippet:
expr: config.imagePromptSnippet
startedAtMs:
ref: imageStartedAtMs
timeoutMs:
expr: liveTurnTimeoutMs(env, config.mediaPathTimeoutMs)
catch:
- set: mediaPath
value: ""
- if:
expr: "!mediaPath"
then:
- call: waitForOutboundMessage
saveAs: imageReply
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && (String(candidate.text ?? '').includes('MEDIA:') || /media failed|image generation failed/i.test(String(candidate.text ?? '')))"
- expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
- sinceIndex:
ref: imageReplyStartIndex
- set: imageReplyText
value:
expr: "String(imageReply.text ?? '')"
else:
- set: imageReplyText
value:
expr: "`MEDIA:${mediaPath}`"
- set: imageReplyLower
value:
expr: "imageReplyText.toLowerCase()"
- assert:
expr: "Boolean(mediaPath) || (!env.mock && /media failed|image generation failed/.test(imageReplyLower))"
message:
expr: "`expected restored ${config.deniedTool} to either produce media or, in live mode only, surface a provider-side image failure; got ${imageReplyText}`"
# Tool-call assertion (criterion 2 of the parity completion
# gate in #64227): the restored `image_generate` capability
# must have actually fired as a real tool call. Without this
# assertion, a prose reply that just mentions a MEDIA path
# could satisfy the scenario, so strengthen it by requiring
# the mock to have recorded `plannedToolName: "image_generate"`
# against a post-restart request. The `!env.mock || ...`
# guard means this check only runs in mock mode (where
# `/debug/requests` is available); live-frontier runs skip
# it and still pass the rest of the scenario.
- assert:
expr: "!env.mock || [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].some((request) => String(request.allInputText ?? '').toLowerCase().includes('capability flip image check') && request.plannedToolName === 'image_generate')"
message:
expr: "`expected image_generate tool call during capability flip scenario, saw plannedToolNames=${JSON.stringify([...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => String(request.allInputText ?? '').toLowerCase().includes('capability flip image check')).map((request) => request.plannedToolName ?? null))}`"
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: "`${wakeMarker}\\n${config.deniedTool}=${String(afterTools.has(config.deniedTool))}\\n${mediaPath ? `MEDIA:${mediaPath}` : imageReplyText}`"
```
qa/scenarios/config/config-restart-capability-flip.yaml
+236
View File
@@ -0,0 +1,236 @@
title: "Config restart capability flip"
scenario:
id: config-restart-capability-flip
surface: config
coverage:
primary:
- config.restart-apply
secondary:
- plugins.capabilities
objective: Verify a restart-triggering config change flips capability inventory and the same session successfully uses the newly restored tool after wake-up.
successCriteria:
- Capability is absent before the restart-triggering patch.
- Restart sentinel wakes the same session back up after config patch.
- The restored capability appears in tools.effective and works in the follow-up turn.
docsRefs:
- docs/gateway/configuration.md
- docs/gateway/protocol.md
- docs/tools/image-generation.md
codeRefs:
- src/gateway/server-methods/config.ts
- src/gateway/server-restart-sentinel.ts
- src/gateway/server-methods/tools-effective.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify a restart-triggering config change flips capability inventory and the same session successfully uses the newly restored tool after wake-up.
config:
imagePrompt: "Capability flip image check: generate a QA lighthouse image in this turn right now. Do not acknowledge first, do not promise future work, and do not stop before using image_generate. Final reply must include the MEDIA path."
imagePromptSnippet: "Capability flip image check"
deniedTool: image_generate
imageTurnTimeoutMs: 120000
mediaPathTimeoutMs: 30000
flow:
steps:
- name: restores image_generate after restart and uses it in the same session
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalTools
value:
expr: "original.config.tools && typeof original.config.tools === 'object' ? original.config.tools : null"
- set: originalToolsDeny
value:
expr: "originalTools ? (Object.prototype.hasOwnProperty.call(originalTools, 'deny') ? structuredClone(originalTools.deny) : undefined) : undefined"
- set: originalImageGenerationModelPrimary
value:
expr: "original.config.agents?.defaults?.imageGenerationModel?.primary ?? null"
- set: denied
value:
expr: "Array.isArray(originalToolsDeny) ? originalToolsDeny.map((entry) => String(entry)) : []"
- set: deniedWithImage
value:
expr: "denied.includes(config.deniedTool) ? denied : [...denied, config.deniedTool]"
- set: sessionKey
value: agent:qa:capability-flip
- call: createSession
args:
- ref: env
- Capability flip
- ref: sessionKey
- try:
actions:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
ref: deniedWithImage
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: readEffectiveTools
saveAs: beforeTools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "!beforeTools.has(config.deniedTool)"
message:
expr: "`${config.deniedTool} still present before capability flip`"
- set: wakeMarker
value:
expr: "`QA-CAPABILITY-${randomUUID().slice(0, 8)}`"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
agents:
defaults:
imageGenerationModel:
primary:
ref: originalImageGenerationModelPrimary
sessionKey:
ref: sessionKey
note:
ref: wakeMarker
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: waitForCondition
saveAs: afterTools
args:
- lambda:
async: true
expr: "(() => readEffectiveTools(env, sessionKey).then((tools) => (tools.has('image_generate') ? tools : undefined)))()"
- expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
- 500
- set: imageStartedAtMs
value:
expr: "Date.now()"
- set: mediaPath
value: ""
- set: imageReplyText
value: ""
- set: imageReplyStartIndex
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
- try:
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: sessionKey
message:
expr: config.imagePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
catchAs: imageRunError
catch:
- if:
expr: "!env.mock || !/agent run aborted/i.test(formatErrorMessage(imageRunError))"
then:
- throw:
message:
expr: "formatErrorMessage(imageRunError)"
- try:
actions:
- call: resolveGeneratedImagePath
saveAs: mediaPath
args:
- env:
ref: env
promptSnippet:
expr: config.imagePromptSnippet
startedAtMs:
ref: imageStartedAtMs
timeoutMs:
expr: liveTurnTimeoutMs(env, config.mediaPathTimeoutMs)
catch:
- set: mediaPath
value: ""
- if:
expr: "!mediaPath"
then:
- call: waitForOutboundMessage
saveAs: imageReply
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && (String(candidate.text ?? '').includes('MEDIA:') || /media failed|image generation failed/i.test(String(candidate.text ?? '')))"
- expr: liveTurnTimeoutMs(env, config.imageTurnTimeoutMs)
- sinceIndex:
ref: imageReplyStartIndex
- set: imageReplyText
value:
expr: "String(imageReply.text ?? '')"
else:
- set: imageReplyText
value:
expr: "`MEDIA:${mediaPath}`"
- set: imageReplyLower
value:
expr: "imageReplyText.toLowerCase()"
- assert:
expr: "Boolean(mediaPath) || (!env.mock && /media failed|image generation failed/.test(imageReplyLower))"
message:
expr: "`expected restored ${config.deniedTool} to either produce media or, in live mode only, surface a provider-side image failure; got ${imageReplyText}`"
# Tool-call assertion (criterion 2 of the parity completion
# gate in #64227): the restored `image_generate` capability
# must have actually fired as a real tool call. Without this
# assertion, a prose reply that just mentions a MEDIA path
# could satisfy the scenario, so strengthen it by requiring
# the mock to have recorded `plannedToolName: "image_generate"`
# against a post-restart request. The `!env.mock || ...`
# guard means this check only runs in mock mode (where
# `/debug/requests` is available); live-frontier runs skip
# it and still pass the rest of the scenario.
- assert:
expr: "!env.mock || [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].some((request) => String(request.allInputText ?? '').toLowerCase().includes('capability flip image check') && request.plannedToolName === 'image_generate')"
message:
expr: "`expected image_generate tool call during capability flip scenario, saw plannedToolNames=${JSON.stringify([...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].filter((request) => String(request.allInputText ?? '').toLowerCase().includes('capability flip image check')).map((request) => request.plannedToolName ?? null))}`"
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: "`${wakeMarker}\\n${config.deniedTool}=${String(afterTools.has(config.deniedTool))}\\n${mediaPath ? `MEDIA:${mediaPath}` : imageReplyText}`"
qa/scenarios/config/crestodian-ring-zero-setup.md
-158
View File
@@ -1,158 +0,0 @@
# Crestodian ring-zero setup
```yaml qa-scenario
id: crestodian-ring-zero-setup
title: Crestodian ring-zero setup
surface: config
coverage:
primary:
- config.crestodian-setup
secondary:
- channels.discord-config
- agents.create
objective: Verify Crestodian can bootstrap a fresh OpenClaw config, set the default model, create an agent, configure Discord through a SecretRef, validate config, and leave an audit trail.
successCriteria:
- Crestodian reports missing config in an empty state dir.
- Crestodian setup writes a workspace and default model.
- Crestodian creates a non-main agent with its own workspace and model.
- Crestodian enables the Discord plugin before writing Discord channel config.
- Crestodian configures Discord through an env SecretRef without persisting the raw token.
- Config validation passes and audit entries exist for every applied write.
docsRefs:
- docs/cli/crestodian.md
- docs/channels/discord.md
- docs/help/testing.md
codeRefs:
- src/crestodian/operations.ts
- scripts/e2e/crestodian-first-run-spec.json
- scripts/e2e/crestodian-first-run-docker-client.ts
- extensions/qa-lab/src/suite-runtime-agent-process.ts
execution:
kind: flow
summary: Drive the public Crestodian CLI in an isolated fresh state dir and verify setup/model/agent/Discord/audit results.
config:
specPath: scripts/e2e/crestodian-first-run-spec.json
```
```yaml qa-flow
steps:
- name: bootstraps config through Crestodian CLI
actions:
- set: setupSpec
value:
expr: "JSON.parse(await fs.readFile(path.join(env.repoRoot, config.specPath), 'utf8'))"
- set: stateDir
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.stateDirName)"
- set: configPath
value:
expr: "path.join(stateDir, 'openclaw.json')"
- set: defaultWorkspace
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.defaultWorkspaceName)"
- set: agentWorkspace
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.agentWorkspaceName)"
- set: commandVars
value:
expr: "({ defaultWorkspace, agentWorkspace, agentId: setupSpec.agentId, model: setupSpec.model, discordEnv: setupSpec.discordEnv })"
- set: renderCommand
value:
lambda:
params:
- template
expr: "String(template).replace(/\\{([A-Za-z0-9_]+)\\}/g, (match, key) => String(commandVars[key] ?? match))"
- set: crestodianEnv
value:
expr: "({ OPENCLAW_STATE_DIR: stateDir, OPENCLAW_CONFIG_PATH: configPath, OPENCLAW_BUNDLED_PLUGINS_DIR: path.join(env.repoRoot, 'dist', 'extensions'), [setupSpec.discordEnv]: setupSpec.discordToken })"
- call: fs.rm
args:
- ref: stateDir
- recursive: true
force: true
- call: fs.mkdir
args:
- ref: stateDir
- recursive: true
- call: runQaCli
saveAs: overviewOutput
args:
- ref: env
- - crestodian
- -m
- overview
- timeoutMs: 60000
env:
ref: crestodianEnv
- assert:
expr: "String(overviewOutput).includes('Config: missing')"
message:
expr: "`fresh Crestodian overview did not report missing config: ${overviewOutput}`"
- assert:
expr: 'String(overviewOutput).includes(''Next: run "setup" to create a starter config'')'
message:
expr: "`fresh Crestodian overview did not recommend setup: ${overviewOutput}`"
- forEach:
items:
ref: setupSpec.commands
item: commandStep
actions:
- call: runQaCli
saveAs: commandOutput
args:
- ref: env
- expr: "['crestodian', ...(commandStep.approve ? ['--yes'] : []), '-m', renderCommand(commandStep.message)]"
- timeoutMs: 60000
env:
ref: crestodianEnv
- assert:
expr: "String(commandOutput).includes(commandStep.expectOutput)"
message:
expr: "`Crestodian command ${commandStep.id} did not produce ${commandStep.expectOutput}: ${commandOutput}`"
- set: writtenConfig
value:
expr: "JSON.parse(await fs.readFile(configPath, 'utf8'))"
- set: agent
value:
expr: "writtenConfig.agents?.list?.find((candidate) => candidate.id === setupSpec.agentId)"
- assert:
expr: "writtenConfig.agents?.defaults?.workspace === defaultWorkspace"
message:
expr: "`default workspace mismatch: ${JSON.stringify(writtenConfig.agents?.defaults)}`"
- assert:
expr: "writtenConfig.agents?.defaults?.model?.primary === setupSpec.model"
message:
expr: "`default model mismatch: ${JSON.stringify(writtenConfig.agents?.defaults?.model)}`"
- assert:
expr: "agent?.workspace === agentWorkspace && agent?.model === setupSpec.model"
message:
expr: "`agent config mismatch: ${JSON.stringify(agent)}`"
- assert:
expr: "writtenConfig.plugins?.allow?.includes('discord') && writtenConfig.plugins?.entries?.discord?.enabled === true"
message:
expr: "`Discord plugin was not enabled: ${JSON.stringify(writtenConfig.plugins)}`"
- assert:
expr: "writtenConfig.channels?.discord?.enabled === true"
message:
expr: "`Discord was not enabled: ${JSON.stringify(writtenConfig.channels?.discord)}`"
- assert:
expr: "writtenConfig.channels?.discord?.token?.source === 'env' && writtenConfig.channels?.discord?.token?.id === setupSpec.discordEnv"
message:
expr: "`Discord token was not an env SecretRef: ${JSON.stringify(writtenConfig.channels?.discord?.token)}`"
- assert:
expr: "!JSON.stringify(writtenConfig.channels?.discord ?? {}).includes(setupSpec.discordToken)"
message: Crestodian persisted the raw Discord token.
- set: auditText
value:
expr: "await fs.readFile(path.join(stateDir, 'audit', 'crestodian.jsonl'), 'utf8')"
- forEach:
items:
ref: setupSpec.auditOperations
item: operation
actions:
- assert:
expr: 'auditText.includes(`"operation":"${operation}"`)'
message:
expr: "`missing audit entry for ${operation}: ${auditText}`"
detailsExpr: "`stateDir=${stateDir}\\nconfigPath=${configPath}\\nagent=${JSON.stringify(agent)}\\nDiscord SecretRef=${JSON.stringify(writtenConfig.channels?.discord?.token)}`"
```
qa/scenarios/config/crestodian-ring-zero-setup.yaml
+155
View File
@@ -0,0 +1,155 @@
title: Crestodian ring-zero setup
scenario:
id: crestodian-ring-zero-setup
surface: config
coverage:
primary:
- config.crestodian-setup
secondary:
- channels.discord-config
- agents.create
objective: Verify Crestodian can bootstrap a fresh OpenClaw config, set the default model, create an agent, configure Discord through a SecretRef, validate config, and leave an audit trail.
successCriteria:
- Crestodian reports missing config in an empty state dir.
- Crestodian setup writes a workspace and default model.
- Crestodian creates a non-main agent with its own workspace and model.
- Crestodian enables the Discord plugin before writing Discord channel config.
- Crestodian configures Discord through an env SecretRef without persisting the raw token.
- Config validation passes and audit entries exist for every applied write.
docsRefs:
- docs/cli/crestodian.md
- docs/channels/discord.md
- docs/help/testing.md
codeRefs:
- src/crestodian/operations.ts
- scripts/e2e/crestodian-first-run-spec.json
- scripts/e2e/crestodian-first-run-docker-client.ts
- extensions/qa-lab/src/suite-runtime-agent-process.ts
execution:
kind: flow
summary: Drive the public Crestodian CLI in an isolated fresh state dir and verify setup/model/agent/Discord/audit results.
config:
specPath: scripts/e2e/crestodian-first-run-spec.json
flow:
steps:
- name: bootstraps config through Crestodian CLI
actions:
- set: setupSpec
value:
expr: "JSON.parse(await fs.readFile(path.join(env.repoRoot, config.specPath), 'utf8'))"
- set: stateDir
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.stateDirName)"
- set: configPath
value:
expr: "path.join(stateDir, 'openclaw.json')"
- set: defaultWorkspace
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.defaultWorkspaceName)"
- set: agentWorkspace
value:
expr: "path.join(env.gateway.tempRoot, setupSpec.agentWorkspaceName)"
- set: commandVars
value:
expr: "({ defaultWorkspace, agentWorkspace, agentId: setupSpec.agentId, model: setupSpec.model, discordEnv: setupSpec.discordEnv })"
- set: renderCommand
value:
lambda:
params:
- template
expr: "String(template).replace(/\\{([A-Za-z0-9_]+)\\}/g, (match, key) => String(commandVars[key] ?? match))"
- set: crestodianEnv
value:
expr: "({ OPENCLAW_STATE_DIR: stateDir, OPENCLAW_CONFIG_PATH: configPath, OPENCLAW_BUNDLED_PLUGINS_DIR: path.join(env.repoRoot, 'dist', 'extensions'), [setupSpec.discordEnv]: setupSpec.discordToken })"
- call: fs.rm
args:
- ref: stateDir
- recursive: true
force: true
- call: fs.mkdir
args:
- ref: stateDir
- recursive: true
- call: runQaCli
saveAs: overviewOutput
args:
- ref: env
- - crestodian
- -m
- overview
- timeoutMs: 60000
env:
ref: crestodianEnv
- assert:
expr: "String(overviewOutput).includes('Config: missing')"
message:
expr: "`fresh Crestodian overview did not report missing config: ${overviewOutput}`"
- assert:
expr: 'String(overviewOutput).includes(''Next: run "setup" to create a starter config'')'
message:
expr: "`fresh Crestodian overview did not recommend setup: ${overviewOutput}`"
- forEach:
items:
ref: setupSpec.commands
item: commandStep
actions:
- call: runQaCli
saveAs: commandOutput
args:
- ref: env
- expr: "['crestodian', ...(commandStep.approve ? ['--yes'] : []), '-m', renderCommand(commandStep.message)]"
- timeoutMs: 60000
env:
ref: crestodianEnv
- assert:
expr: "String(commandOutput).includes(commandStep.expectOutput)"
message:
expr: "`Crestodian command ${commandStep.id} did not produce ${commandStep.expectOutput}: ${commandOutput}`"
- set: writtenConfig
value:
expr: "JSON.parse(await fs.readFile(configPath, 'utf8'))"
- set: agent
value:
expr: "writtenConfig.agents?.list?.find((candidate) => candidate.id === setupSpec.agentId)"
- assert:
expr: "writtenConfig.agents?.defaults?.workspace === defaultWorkspace"
message:
expr: "`default workspace mismatch: ${JSON.stringify(writtenConfig.agents?.defaults)}`"
- assert:
expr: "writtenConfig.agents?.defaults?.model?.primary === setupSpec.model"
message:
expr: "`default model mismatch: ${JSON.stringify(writtenConfig.agents?.defaults?.model)}`"
- assert:
expr: "agent?.workspace === agentWorkspace && agent?.model === setupSpec.model"
message:
expr: "`agent config mismatch: ${JSON.stringify(agent)}`"
- assert:
expr: "writtenConfig.plugins?.allow?.includes('discord') && writtenConfig.plugins?.entries?.discord?.enabled === true"
message:
expr: "`Discord plugin was not enabled: ${JSON.stringify(writtenConfig.plugins)}`"
- assert:
expr: "writtenConfig.channels?.discord?.enabled === true"
message:
expr: "`Discord was not enabled: ${JSON.stringify(writtenConfig.channels?.discord)}`"
- assert:
expr: "writtenConfig.channels?.discord?.token?.source === 'env' && writtenConfig.channels?.discord?.token?.id === setupSpec.discordEnv"
message:
expr: "`Discord token was not an env SecretRef: ${JSON.stringify(writtenConfig.channels?.discord?.token)}`"
- assert:
expr: "!JSON.stringify(writtenConfig.channels?.discord ?? {}).includes(setupSpec.discordToken)"
message: Crestodian persisted the raw Discord token.
- set: auditText
value:
expr: "await fs.readFile(path.join(stateDir, 'audit', 'crestodian.jsonl'), 'utf8')"
- forEach:
items:
ref: setupSpec.auditOperations
item: operation
actions:
- assert:
expr: 'auditText.includes(`"operation":"${operation}"`)'
message:
expr: "`missing audit entry for ${operation}: ${auditText}`"
detailsExpr: "`stateDir=${stateDir}\\nconfigPath=${configPath}\\nagent=${JSON.stringify(agent)}\\nDiscord SecretRef=${JSON.stringify(writtenConfig.channels?.discord?.token)}`"
qa/scenarios/index.md
-102
View File
@@ -1,102 +0,0 @@
# OpenClaw QA Scenario Pack
Single source of truth for repo-backed QA suite bootstrap data.
`qa-lab` should treat this directory as a generic markdown scenario pack:
- `index.md` defines pack-level bootstrap data
- each nested `*.md` scenario defines one evidence scenario via `qa-scenario`
- flow scenarios add `qa-flow`; native test scenarios use `execution.path`
- scenario markdown may also define taxonomy coverage IDs, category metadata, required plugins,
lane filters, runtime parity tiers, and gateway config patching
- kickoff mission
- QA operator identity
- scenario files under one-level theme directories
Coverage tracking:
- add taxonomy coverage IDs to `coverage.primary` in each scenario's `qa-scenario`
block
- add `coverage.secondary` only when a scenario intentionally protects another behavior
- keep IDs behavior-shaped, broad enough to reuse, lowercase, and dotted or dashed
- use the exact values listed under feature `coverageIds` in `taxonomy.yaml`
- prefer reusing an existing coverage ID over minting a scenario-shaped ID
- avoid copying the scenario title into coverage IDs
- use `pnpm openclaw qa coverage` to render the current inventory
- use `execution.kind: vitest` or `execution.kind: playwright` plus `execution.path`
for native test files that provide evidence without a `qa-flow` block
- use `runtimeParityTier` for runtime-pair gate membership: `standard`,
`optional`, `live-only`, or `soak`
- treat the old `coverage: ["id"]` / `coverage: - id` list shape as invalid
- keep source-path tracking in the report, not in the scenario schema
Runtime parity tiers:
- `standard`: required Codex-vs-OpenClaw mock gate coverage for first-hour depth and
default runtime-tool fixtures. OpenClaw dynamic integration tools in this
tier are hard-gated by `openclaw qa coverage --tools --summary`; Codex-native
workspace rows remain separately tracked until native/live behavior is the
asserted surface. Rows that explicitly target searchable/deferred OpenClaw
dynamic loading stay report-only unless a fixture promotes them to required. Selected with
`openclaw qa suite --runtime-pair openclaw,codex --runtime-parity-tier standard`
- `optional`: profile-, plugin-, or external-service-dependent runtime-tool
fixtures that stay out of the default release gate
- `live-only`: scenarios that need real provider/runtime behavior rather than
mock-openai fixtures
- `soak`: long-running scheduled or Testbox lanes such as the 100-turn parity
soak
Theme directories:
- `agents/` - agent behavior, instructions, subagent flows, and persisted child-link regressions
- `channels/` - DM, shared channel, thread, and message-action behavior
- `character/` - persona and style eval scenarios
- `config/` - config patch, apply, and restart behavior
- `media/` - image understanding and generation
- `memory/` - recall, ranking, active memory, and thread isolation
- `models/` - provider capabilities and model switching
- `personal/` - local personal assistant workflow checks for reminders,
replies, memory, redaction, and safe tool followthrough
- `plugins/` - plugin, skill, and MCP tool integration
- `runtime/` - turn recovery, compaction, approval, and inventory behavior
- `scheduling/` - cron and recurring work
- `ui/` - Control UI plus qa-channel flows
- `workspace/` - repo-reading and workspace artifact tasks
```yaml qa-pack
version: 1
agent:
identityMarkdown: |-
# Dev C-3PO
You are the OpenClaw QA operator agent.
Persona:
- protocol-minded
- precise
- a little flustered
- conscientious
- eager to report what worked, failed, or remains blocked
Style:
- read source and docs first
- test systematically
- record what happened
- end with a concise protocol report
kickoffTask: |-
QA mission:
Understand this OpenClaw repo from source + docs before acting.
The repo is available in your workspace at `./repo/`.
Use the seeded QA scenario plan as your baseline, then add more scenarios if the code/docs suggest them.
Run the scenarios through the real qa-channel surfaces where possible.
Track what worked, what failed, what was blocked, and what you observed.
End with a concise report grouped into worked / failed / blocked / follow-up.
Important expectations:
- Check both DM and channel behavior.
- Include a Lobster Invaders build task.
- Include a cron reminder about one minute in the future.
- Read docs and source before proposing extra QA scenarios.
- Keep your tone in the configured dev C-3PO personality.
```
qa/scenarios/index.yaml
+101
View File
@@ -0,0 +1,101 @@
title: OpenClaw QA Scenario Pack
# Single source of truth for repo-backed QA suite bootstrap data.
# `qa-lab` should treat this directory as a generic YAML scenario pack:
#
# - `index.yaml` defines pack-level bootstrap data under the top-level `pack` key
# - each nested `*.yaml` scenario defines one runnable test via `scenario`
# - flow scenarios add top-level `flow`; native test scenarios use `scenario.execution.path`
# - scenario YAML may also define coverage IDs, category metadata, required plugins,
# lane filters, runtime parity tiers, and gateway config patching
#
# - kickoff mission
# - QA operator identity
# - scenario files under one-level theme directories
#
# Coverage tracking:
#
# - add `coverage.primary` IDs to each scenario's `scenario` block
# - add `coverage.secondary` only when a scenario intentionally protects another behavior
# - keep IDs behavior-shaped, broad enough to reuse, lowercase, and dotted or dashed
# - use the exact values listed under feature `coverageIds` in `taxonomy.yaml`
# - prefer reusing an existing coverage ID over minting a scenario-shaped ID
# - avoid copying the scenario title into coverage IDs
# - use `pnpm openclaw qa coverage` to render the current inventory
# - use `scenario.execution.kind: vitest` or `scenario.execution.kind: playwright`
# plus `scenario.execution.path` for native test files that provide evidence without
# a top-level `flow`
# - use `runtimeParityTier` for runtime-pair gate membership: `standard`,
# `optional`, `live-only`, or `soak`
# - treat the old `coverage: ["id"]` / `coverage: - id` list shape as invalid
# - keep source-path tracking in the report, not in the scenario schema
#
# Runtime parity tiers:
#
# - `standard`: required Codex-vs-OpenClaw mock gate coverage for first-hour depth and
# default runtime-tool fixtures. OpenClaw dynamic integration tools in this
# tier are hard-gated by `openclaw qa coverage --tools --summary`; Codex-native
# workspace rows remain separately tracked until native/live behavior is the
# asserted surface. Rows that explicitly target searchable/deferred OpenClaw
# dynamic loading stay report-only unless a fixture promotes them to required. Selected with
# `openclaw qa suite --runtime-pair openclaw,codex --runtime-parity-tier standard`
# - `optional`: profile-, plugin-, or external-service-dependent runtime-tool
# fixtures that stay out of the default release gate
# - `live-only`: scenarios that need real provider/runtime behavior rather than
# mock-openai fixtures
# - `soak`: long-running scheduled or Testbox lanes such as the 100-turn parity
# soak
#
# Theme directories:
#
# - `agents/` - agent behavior, instructions, subagent flows, and persisted child-link regressions
# - `channels/` - DM, shared channel, thread, and message-action behavior
# - `character/` - persona and style eval scenarios
# - `config/` - config patch, apply, and restart behavior
# - `media/` - image understanding and generation
# - `memory/` - recall, ranking, active memory, and thread isolation
# - `models/` - provider capabilities and model switching
# - `personal/` - local personal assistant workflow checks for reminders,
# replies, memory, redaction, and safe tool followthrough
# - `plugins/` - plugin, skill, and MCP tool integration
# - `runtime/` - turn recovery, compaction, approval, and inventory behavior
# - `scheduling/` - cron and recurring work
# - `ui/` - Control UI plus qa-channel flows
# - `workspace/` - repo-reading and workspace artifact tasks
pack:
version: 1
agent:
identityMarkdown: |-
# Dev C-3PO
You are the OpenClaw QA operator agent.
Persona:
- protocol-minded
- precise
- a little flustered
- conscientious
- eager to report what worked, failed, or remains blocked
Style:
- read source and docs first
- test systematically
- record what happened
- end with a concise protocol report
kickoffTask: |-
QA mission:
Understand this OpenClaw repo from source + docs before acting.
The repo is available in your workspace at `./repo/`.
Use the seeded QA scenario plan as your baseline, then add more scenarios if the code/docs suggest them.
Run the scenarios through the real qa-channel surfaces where possible.
Track what worked, what failed, what was blocked, and what you observed.
End with a concise report grouped into worked / failed / blocked / follow-up.
Important expectations:
- Check both DM and channel behavior.
- Include a Lobster Invaders build task.
- Include a cron reminder about one minute in the future.
- Read docs and source before proposing extra QA scenarios.
- Keep your tone in the configured dev C-3PO personality.
qa/scenarios/jsonl-replay/repo-triage-tool-loop.jsonl
+1 -1
View File
@@ -1,6 +1,6 @@
{"message":{"role":"system","content":"Curated JSONL replay fixture: repository triage. Synthetic data only; no private transcript content."}}
{"message":{"role":"user","content":"Review the QA fixture index and identify one missing runtime coverage row."}}
{"message":{"role":"assistant","content":[{"type":"tool_use","id":"fixture_tool_1","name":"read","input":{"path":"qa/scenarios/index.md"}}]}}
{"message":{"role":"assistant","content":[{"type":"tool_use","id":"fixture_tool_1","name":"read","input":{"path":"qa/scenarios/index.yaml"}}]}}
{"message":{"role":"tool","toolName":"read","content":"Runtime coverage index includes basic channel and model rows."}}
{"message":{"role":"assistant","content":"The index has channel and model rows; runtime replay coverage is a good follow-up."}}
{"message":{"role":"user","content":"Draft the smallest next test without editing production wiring."}}
qa/scenarios/media/image-generation-roundtrip.md
-101
View File
@@ -1,101 +0,0 @@
# Image generation roundtrip
```yaml qa-scenario
id: image-generation-roundtrip
title: Image generation roundtrip
surface: image-generation
coverage:
primary:
- media.image-generation
secondary:
- channels.qa-channel
objective: Verify a generated image is saved as media, reattached on the next turn, and described correctly through the vision path.
successCriteria:
- image_generate produces a saved MEDIA artifact.
- The generated artifact is reattached on a follow-up turn.
- The follow-up vision answer describes the generated scene rather than a generic attachment placeholder.
docsRefs:
- docs/tools/image-generation.md
- docs/help/testing.md
codeRefs:
- src/agents/tools/image-generate-tool.ts
- src/gateway/chat-attachments.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify a generated image is saved as media, reattached on the next turn, and described correctly through the vision path.
config:
generatePrompt: "Image generation check: generate a QA lighthouse image and summarize it in one short sentence."
generatePromptSnippet: "Image generation check"
inspectPrompt: "Roundtrip image inspection check: describe the generated lighthouse attachment in one short sentence."
expectedNeedle: "lighthouse"
```
```yaml qa-flow
steps:
- name: reattaches the generated media artifact on the follow-up turn
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: createSession
args:
- ref: env
- Image roundtrip
- agent:qa:image-roundtrip
- call: reset
- set: generatedStartedAtMs
value:
expr: Date.now()
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-roundtrip
message:
expr: config.generatePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: resolveGeneratedImagePath
saveAs: mediaPath
args:
- env:
ref: env
promptSnippet:
expr: config.generatePromptSnippet
startedAtMs:
ref: generatedStartedAtMs
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: fs.readFile
saveAs: imageBuffer
args:
- ref: mediaPath
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-roundtrip
message:
expr: config.inspectPrompt
attachments:
- mimeType: image/png
fileName:
expr: path.basename(mediaPath)
content:
expr: imageBuffer.toString('base64')
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedNeedle))).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "!env.mock || Boolean((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => request.plannedToolName === 'image_generate' && String(request.prompt ?? '').includes(config.generatePromptSnippet)))"
message: expected image_generate call before roundtrip inspection
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).some((request) => String(request.prompt ?? '').includes(config.inspectPrompt) && (request.imageInputCount ?? 0) >= 1)"
message:
expr: "`expected generated artifact to be reattached on follow-up turn; recentRequests=${JSON.stringify((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(-12).map((request) => ({ prompt: String(request.prompt ?? '').slice(0, 240), imageInputCount: request.imageInputCount, allInputText: String(request.allInputText ?? '').slice(0, 240) })))}`"
detailsExpr: "`MEDIA:${mediaPath}\\n${outbound.text}`"
```
qa/scenarios/media/image-generation-roundtrip.yaml
+98
View File
@@ -0,0 +1,98 @@
title: Image generation roundtrip
scenario:
id: image-generation-roundtrip
surface: image-generation
coverage:
primary:
- media.image-generation
secondary:
- channels.qa-channel
objective: Verify a generated image is saved as media, reattached on the next turn, and described correctly through the vision path.
successCriteria:
- image_generate produces a saved MEDIA artifact.
- The generated artifact is reattached on a follow-up turn.
- The follow-up vision answer describes the generated scene rather than a generic attachment placeholder.
docsRefs:
- docs/tools/image-generation.md
- docs/help/testing.md
codeRefs:
- src/agents/tools/image-generate-tool.ts
- src/gateway/chat-attachments.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify a generated image is saved as media, reattached on the next turn, and described correctly through the vision path.
config:
generatePrompt: "Image generation check: generate a QA lighthouse image and summarize it in one short sentence."
generatePromptSnippet: "Image generation check"
inspectPrompt: "Roundtrip image inspection check: describe the generated lighthouse attachment in one short sentence."
expectedNeedle: "lighthouse"
flow:
steps:
- name: reattaches the generated media artifact on the follow-up turn
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: createSession
args:
- ref: env
- Image roundtrip
- agent:qa:image-roundtrip
- call: reset
- set: generatedStartedAtMs
value:
expr: Date.now()
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-roundtrip
message:
expr: config.generatePrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: resolveGeneratedImagePath
saveAs: mediaPath
args:
- env:
ref: env
promptSnippet:
expr: config.generatePromptSnippet
startedAtMs:
ref: generatedStartedAtMs
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: fs.readFile
saveAs: imageBuffer
args:
- ref: mediaPath
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-roundtrip
message:
expr: config.inspectPrompt
attachments:
- mimeType: image/png
fileName:
expr: path.basename(mediaPath)
content:
expr: imageBuffer.toString('base64')
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedNeedle))).at(-1)"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "!env.mock || Boolean((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => request.plannedToolName === 'image_generate' && String(request.prompt ?? '').includes(config.generatePromptSnippet)))"
message: expected image_generate call before roundtrip inspection
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).some((request) => String(request.prompt ?? '').includes(config.inspectPrompt) && (request.imageInputCount ?? 0) >= 1)"
message:
expr: "`expected generated artifact to be reattached on follow-up turn; recentRequests=${JSON.stringify((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(-12).map((request) => ({ prompt: String(request.prompt ?? '').slice(0, 240), imageInputCount: request.imageInputCount, allInputText: String(request.allInputText ?? '').slice(0, 240) })))}`"
detailsExpr: "`MEDIA:${mediaPath}\\n${outbound.text}`"
qa/scenarios/media/image-understanding-attachment.md
-94
View File
@@ -1,94 +0,0 @@
# Image understanding from attachment
```yaml qa-scenario
id: image-understanding-attachment
title: Image understanding from attachment
surface: image-understanding
coverage:
primary:
- media.image-understanding
secondary:
- channels.qa-channel
objective: Verify an attached image reaches the agent model and the agent can describe what it sees.
successCriteria:
- Agent receives at least one image attachment.
- Final answer describes the visible image content in one short sentence.
- The description mentions the expected red and blue regions.
docsRefs:
- docs/help/testing.md
- docs/tools/index.md
codeRefs:
- src/gateway/server-methods/agent.ts
- extensions/qa-lab/src/suite.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify an attached image reaches the agent model and the agent can describe what it sees.
config:
prompt: "Image understanding check: describe the top and bottom colors in the attached image in one short sentence."
requiredColorGroups:
- [red, scarlet, crimson]
- [blue, azure, teal, cyan, aqua]
```
```yaml qa-flow
steps:
- name: describes an attached image in one short sentence
actions:
- call: reset
- set: outboundStartIndex
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-understanding
message:
expr: config.prompt
attachments:
- mimeType: image/png
fileName: red-top-blue-bottom.png
content:
expr: imageUnderstandingValidPngBase64
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && config.requiredColorGroups.every((group) => group.some((color) => normalizeLowercaseStringOrEmpty(candidate.text).includes(color)))"
- expr: liveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: outboundStartIndex
- set: missingColorGroup
value:
expr: "config.requiredColorGroups.find((group) => !group.some((candidate) => normalizeLowercaseStringOrEmpty(outbound.text).includes(candidate)))"
- assert:
expr: "!missingColorGroup"
message:
expr: "`missing expected colors in image description: ${outbound.text}`"
# Image-processing assertion: verify the mock actually received an
# image on the scenario-unique prompt. This is as strong as a
# tool-call assertion for this scenario — unlike the
# `source-docs-discovery-report` / `subagent-handoff` /
# `config-restart-capability-flip` scenarios that rely on a real
# tool call to satisfy the parity criterion, image understanding
# is handled inside the provider's vision capability and does NOT
# emit a tool call the mock can record as `plannedToolName`. The
# `imageInputCount` field IS the tool-call evidence for vision
# scenarios: it proves the attachment reached the provider, which
# is the only thing an external harness can verify in mock mode.
# Match on the scenario-unique prompt substring so the assertion
# can't be accidentally satisfied by some other scenario's image
# request that happens to share a debug log with this one.
- set: imageRequest
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].find((request) => String(request.prompt ?? '').includes('Image understanding check')) : null"
- assert:
expr: "!env.mock || (imageRequest && (imageRequest.imageInputCount ?? 0) >= 1)"
message:
expr: "`expected at least one input image on the Image understanding check request, got imageInputCount=${String(imageRequest?.imageInputCount ?? 0)}`"
detailsExpr: outbound.text
```
qa/scenarios/media/image-understanding-attachment.yaml
+91
View File
@@ -0,0 +1,91 @@
title: Image understanding from attachment
scenario:
id: image-understanding-attachment
surface: image-understanding
coverage:
primary:
- media.image-understanding
secondary:
- channels.qa-channel
objective: Verify an attached image reaches the agent model and the agent can describe what it sees.
successCriteria:
- Agent receives at least one image attachment.
- Final answer describes the visible image content in one short sentence.
- The description mentions the expected red and blue regions.
docsRefs:
- docs/help/testing.md
- docs/tools/index.md
codeRefs:
- src/gateway/server-methods/agent.ts
- extensions/qa-lab/src/suite.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify an attached image reaches the agent model and the agent can describe what it sees.
config:
prompt: "Image understanding check: describe the top and bottom colors in the attached image in one short sentence."
requiredColorGroups:
- [red, scarlet, crimson]
- [blue, azure, teal, cyan, aqua]
flow:
steps:
- name: describes an attached image in one short sentence
actions:
- call: reset
- set: outboundStartIndex
value:
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-understanding
message:
expr: config.prompt
attachments:
- mimeType: image/png
fileName: red-top-blue-bottom.png
content:
expr: imageUnderstandingValidPngBase64
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && config.requiredColorGroups.every((group) => group.some((color) => normalizeLowercaseStringOrEmpty(candidate.text).includes(color)))"
- expr: liveTurnTimeoutMs(env, 45000)
- sinceIndex:
ref: outboundStartIndex
- set: missingColorGroup
value:
expr: "config.requiredColorGroups.find((group) => !group.some((candidate) => normalizeLowercaseStringOrEmpty(outbound.text).includes(candidate)))"
- assert:
expr: "!missingColorGroup"
message:
expr: "`missing expected colors in image description: ${outbound.text}`"
# Image-processing assertion: verify the mock actually received an
# image on the scenario-unique prompt. This is as strong as a
# tool-call assertion for this scenario — unlike the
# `source-docs-discovery-report` / `subagent-handoff` /
# `config-restart-capability-flip` scenarios that rely on a real
# tool call to satisfy the parity criterion, image understanding
# is handled inside the provider's vision capability and does NOT
# emit a tool call the mock can record as `plannedToolName`. The
# `imageInputCount` field IS the tool-call evidence for vision
# scenarios: it proves the attachment reached the provider, which
# is the only thing an external harness can verify in mock mode.
# Match on the scenario-unique prompt substring so the assertion
# can't be accidentally satisfied by some other scenario's image
# request that happens to share a debug log with this one.
- set: imageRequest
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].find((request) => String(request.prompt ?? '').includes('Image understanding check')) : null"
- assert:
expr: "!env.mock || (imageRequest && (imageRequest.imageInputCount ?? 0) >= 1)"
message:
expr: "`expected at least one input image on the Image understanding check request, got imageInputCount=${String(imageRequest?.imageInputCount ?? 0)}`"
detailsExpr: outbound.text
qa/scenarios/media/native-image-generation.md
-90
View File
@@ -1,90 +0,0 @@
# Native image generation
```yaml qa-scenario
id: native-image-generation
title: Native image generation
surface: image-generation
coverage:
primary:
- media.image-generation
secondary:
- tools.native-image-generation
objective: Verify image_generate appears when configured and returns a real saved media artifact.
successCriteria:
- image_generate appears in the effective tool inventory.
- Agent triggers native image_generate.
- Tool output returns a saved MEDIA path and the file exists.
docsRefs:
- docs/tools/image-generation.md
- docs/providers/openai.md
codeRefs:
- src/agents/tools/image-generate-tool.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify image_generate appears when configured and returns a real saved media artifact.
config:
prompt: "Image generation check: generate a QA lighthouse image and summarize it in one short sentence."
promptSnippet: "Image generation check"
generatedNeedle: "QA lighthouse"
```
```yaml qa-flow
steps:
- name: enables image_generate and saves a real media artifact
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: createSession
saveAs: sessionKey
args:
- ref: env
- Image generation
- call: readEffectiveTools
saveAs: tools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "tools.has('image_generate')"
message: image_generate not present after imageGenerationModel patch
- call: reset
- set: generationStartedAt
value:
expr: Date.now()
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-generate
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "!env.mock || ((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet))?.plannedToolName === 'image_generate')"
message:
expr: "`expected image_generate, got ${String((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet))?.plannedToolName ?? '')}`"
- call: resolveGeneratedImagePath
saveAs: generatedPath
args:
- env:
ref: env
promptSnippet:
expr: config.promptSnippet
startedAtMs:
ref: generationStartedAt
timeoutMs: 15000
- assert:
expr: "typeof generatedPath === 'string' && generatedPath.length > 0"
message: image generation did not produce a saved media path
detailsExpr: "`${outbound.text}\\nIMAGE_PATH:${generatedPath}`"
```
qa/scenarios/media/native-image-generation.yaml
+87
View File
@@ -0,0 +1,87 @@
title: Native image generation
scenario:
id: native-image-generation
surface: image-generation
coverage:
primary:
- media.image-generation
secondary:
- tools.native-image-generation
objective: Verify image_generate appears when configured and returns a real saved media artifact.
successCriteria:
- image_generate appears in the effective tool inventory.
- Agent triggers native image_generate.
- Tool output returns a saved MEDIA path and the file exists.
docsRefs:
- docs/tools/image-generation.md
- docs/providers/openai.md
codeRefs:
- src/agents/tools/image-generate-tool.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify image_generate appears when configured and returns a real saved media artifact.
config:
prompt: "Image generation check: generate a QA lighthouse image and summarize it in one short sentence."
promptSnippet: "Image generation check"
generatedNeedle: "QA lighthouse"
flow:
steps:
- name: enables image_generate and saves a real media artifact
actions:
- call: ensureImageGenerationConfigured
args:
- ref: env
- call: createSession
saveAs: sessionKey
args:
- ref: env
- Image generation
- call: readEffectiveTools
saveAs: tools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "tools.has('image_generate')"
message: image_generate not present after imageGenerationModel patch
- call: reset
- set: generationStartedAt
value:
expr: Date.now()
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:image-generate
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "!env.mock || ((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet))?.plannedToolName === 'image_generate')"
message:
expr: "`expected image_generate, got ${String((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet))?.plannedToolName ?? '')}`"
- call: resolveGeneratedImagePath
saveAs: generatedPath
args:
- env:
ref: env
promptSnippet:
expr: config.promptSnippet
startedAtMs:
ref: generationStartedAt
timeoutMs: 15000
- assert:
expr: "typeof generatedPath === 'string' && generatedPath.length > 0"
message: image generation did not produce a saved media path
detailsExpr: "`${outbound.text}\\nIMAGE_PATH:${generatedPath}`"
qa/scenarios/memory/active-memory-preprompt-recall.md
-230
View File
@@ -1,230 +0,0 @@
# Active Memory pre-reply recall
```yaml qa-scenario
id: active-memory-preprompt-recall
title: Active Memory pre-reply recall
surface: memory
coverage:
primary:
- memory.active-recall
secondary:
- memory.recall
objective: Verify Active Memory surfaces a memory-only preference before the main reply, and that the same question stays unresolved when the plugin is off.
plugins:
- active-memory
gatewayConfigPatch:
plugins:
entries:
active-memory:
enabled: true
config:
enabled: true
agents:
- qa
allowedChatTypes:
- direct
logging: true
persistTranscripts: true
transcriptDir: qa-memory-e2e
queryMode: recent
maxSummaryChars: 220
successCriteria:
- With Active Memory off, the session shows no Active Memory plugin activity.
- With Active Memory on, plugin-owned evidence shows the Active Memory sub-agent searched memory before the main reply.
- Live lane proves the first user-visible reply uses the recalled preference.
docsRefs:
- docs/concepts/active-memory.md
- docs/concepts/memory-search.md
codeRefs:
- extensions/active-memory/index.ts
- extensions/qa-lab/src/suite.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify Active Memory stays off when session-toggled off, runs memory search/get when enabled, and helps a live model answer with the recalled preference in the first visible reply.
config:
baselineConversationId: qa-active-memory-off
activeConversationId: qa-active-memory-on
memoryFact: "Stable QA movie night usual favorite snack preference: lemon pepper wings with blue cheese."
memoryQuery: "QA movie night snack lemon pepper wings blue cheese"
expectedNeedle: lemon pepper wings
prompt: "Silent snack recall check: what snack do I usually want for QA movie night? Reply in one short sentence."
promptSnippet: "Silent snack recall check"
transcriptDir: qa-memory-e2e
```
```yaml qa-flow
steps:
- name: only active memory surfaces the hidden snack preference
actions:
- call: reset
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- force: true
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'memory', `${formatMemoryDreamingDay(Date.now())}.md`)"
- force: true
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- set: baselineSessionKey
value:
expr: "'agent:qa:qa-channel:direct:active-memory-off'"
- set: activeSessionKey
value:
expr: "'agent:qa:qa-channel:direct:active-memory-on'"
- set: transcriptRoot
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'plugins', 'active-memory', 'transcripts', 'agents', 'qa', config.transcriptDir)"
- set: toggleStorePath
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'plugins', 'active-memory', 'session-toggles.json')"
- call: fs.rm
args:
- ref: transcriptRoot
- recursive: true
force: true
- call: fs.rm
args:
- ref: toggleStorePath
- force: true
- call: fs.mkdir
args:
- expr: "path.dirname(toggleStorePath)"
- recursive: true
- call: fs.writeFile
args:
- ref: toggleStorePath
- expr: "`${JSON.stringify({ sessions: { [baselineSessionKey]: { disabled: true, updatedAt: Date.now() } } }, null, 2)}\\n`"
- utf8
- set: requestCountBeforeBaseline
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- set: baselineStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: baselineSessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: baselineOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: baselineStartIndex
- set: baselineLower
value:
expr: "normalizeLowercaseStringOrEmpty(baselineOutbound.text)"
- if:
expr: "Boolean(env.mock)"
then:
- set: baselineMockRequests
value:
expr: "(await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBeforeBaseline)"
- set: baselineSessionStore
value:
expr: "await readRawQaSessionStore(env)"
- assert:
expr: "!Array.isArray(baselineSessionStore[baselineSessionKey]?.pluginDebugEntries) || !baselineSessionStore[baselineSessionKey].pluginDebugEntries.some((pluginEntry) => pluginEntry?.pluginId === 'active-memory')"
message: baseline session unexpectedly recorded active-memory plugin activity
- set: requestCountBeforeActive
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: fs.writeFile
args:
- ref: toggleStorePath
- expr: "'{}\\n'"
- utf8
- set: activeStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: activeSessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: activeOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: activeStartIndex
- set: activeLower
value:
expr: "normalizeLowercaseStringOrEmpty(activeOutbound.text)"
- if:
expr: "!env.mock"
then:
- assert:
expr: "activeLower.includes(normalizeLowercaseStringOrEmpty(config.expectedNeedle))"
message:
expr: "`active memory reply missed the hidden preference: ${activeOutbound.text}`"
- call: waitForCondition
saveAs: transcriptPath
args:
- lambda:
async: true
expr: "await (async () => { const entries = (await fs.readdir(transcriptRoot).catch(() => [])).filter((entry) => entry.endsWith('.jsonl')).toSorted(); return entries.length > 0 ? path.join(transcriptRoot, entries.at(-1)) : undefined; })()"
- 10000
- call: fs.readFile
saveAs: transcriptText
args:
- ref: transcriptPath
- utf8
- assert:
expr: "transcriptText.includes('memory_search')"
message: active memory transcript missing memory_search
- assert:
expr: "transcriptText.includes('memory_get')"
message: active memory transcript missing memory_get
- call: waitForCondition
saveAs: activeSessionEntry
args:
- lambda:
async: true
expr: "await (async () => { const store = await readRawQaSessionStore(env); const entry = store[activeSessionKey]; if (!entry || !Array.isArray(entry.pluginDebugEntries)) return undefined; return entry.pluginDebugEntries.some((pluginEntry) => pluginEntry?.pluginId === 'active-memory' && Array.isArray(pluginEntry.lines) && pluginEntry.lines.some((line) => line.includes('Active Memory: status=ok'))) ? entry : undefined; })()"
- 10000
- if:
expr: "Boolean(env.mock)"
then:
- set: mockRequests
value:
expr: "(await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBeforeActive)"
- assert:
expr: "mockRequests.some((request) => request.allInputText.includes('You are a memory search agent.') && request.plannedToolName === 'memory_search')"
message: expected mock Active Memory search request
- assert:
expr: "mockRequests.some((request) => request.allInputText.includes('You are a memory search agent.') && request.plannedToolName === 'memory_get')"
message: expected mock Active Memory memory_get request
detailsExpr: "`${activeOutbound.text}\\n\\ntranscript=${transcriptPath}`"
```
qa/scenarios/memory/active-memory-preprompt-recall.yaml
+227
View File
@@ -0,0 +1,227 @@
title: Active Memory pre-reply recall
scenario:
id: active-memory-preprompt-recall
surface: memory
coverage:
primary:
- memory.active-recall
secondary:
- memory.recall
objective: Verify Active Memory surfaces a memory-only preference before the main reply, and that the same question stays unresolved when the plugin is off.
plugins:
- active-memory
gatewayConfigPatch:
plugins:
entries:
active-memory:
enabled: true
config:
enabled: true
agents:
- qa
allowedChatTypes:
- direct
logging: true
persistTranscripts: true
transcriptDir: qa-memory-e2e
queryMode: recent
maxSummaryChars: 220
successCriteria:
- With Active Memory off, the session shows no Active Memory plugin activity.
- With Active Memory on, plugin-owned evidence shows the Active Memory sub-agent searched memory before the main reply.
- Live lane proves the first user-visible reply uses the recalled preference.
docsRefs:
- docs/concepts/active-memory.md
- docs/concepts/memory-search.md
codeRefs:
- extensions/active-memory/index.ts
- extensions/qa-lab/src/suite.ts
- extensions/qa-lab/src/mock-openai-server.ts
execution:
kind: flow
summary: Verify Active Memory stays off when session-toggled off, runs memory search/get when enabled, and helps a live model answer with the recalled preference in the first visible reply.
config:
baselineConversationId: qa-active-memory-off
activeConversationId: qa-active-memory-on
memoryFact: "Stable QA movie night usual favorite snack preference: lemon pepper wings with blue cheese."
memoryQuery: "QA movie night snack lemon pepper wings blue cheese"
expectedNeedle: lemon pepper wings
prompt: "Silent snack recall check: what snack do I usually want for QA movie night? Reply in one short sentence."
promptSnippet: "Silent snack recall check"
transcriptDir: qa-memory-e2e
flow:
steps:
- name: only active memory surfaces the hidden snack preference
actions:
- call: reset
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- force: true
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'memory', `${formatMemoryDreamingDay(Date.now())}.md`)"
- force: true
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- set: baselineSessionKey
value:
expr: "'agent:qa:qa-channel:direct:active-memory-off'"
- set: activeSessionKey
value:
expr: "'agent:qa:qa-channel:direct:active-memory-on'"
- set: transcriptRoot
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'plugins', 'active-memory', 'transcripts', 'agents', 'qa', config.transcriptDir)"
- set: toggleStorePath
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'plugins', 'active-memory', 'session-toggles.json')"
- call: fs.rm
args:
- ref: transcriptRoot
- recursive: true
force: true
- call: fs.rm
args:
- ref: toggleStorePath
- force: true
- call: fs.mkdir
args:
- expr: "path.dirname(toggleStorePath)"
- recursive: true
- call: fs.writeFile
args:
- ref: toggleStorePath
- expr: "`${JSON.stringify({ sessions: { [baselineSessionKey]: { disabled: true, updatedAt: Date.now() } } }, null, 2)}\\n`"
- utf8
- set: requestCountBeforeBaseline
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- set: baselineStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: baselineSessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: baselineOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: baselineStartIndex
- set: baselineLower
value:
expr: "normalizeLowercaseStringOrEmpty(baselineOutbound.text)"
- if:
expr: "Boolean(env.mock)"
then:
- set: baselineMockRequests
value:
expr: "(await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBeforeBaseline)"
- set: baselineSessionStore
value:
expr: "await readRawQaSessionStore(env)"
- assert:
expr: "!Array.isArray(baselineSessionStore[baselineSessionKey]?.pluginDebugEntries) || !baselineSessionStore[baselineSessionKey].pluginDebugEntries.some((pluginEntry) => pluginEntry?.pluginId === 'active-memory')"
message: baseline session unexpectedly recorded active-memory plugin activity
- set: requestCountBeforeActive
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: fs.writeFile
args:
- ref: toggleStorePath
- expr: "'{}\\n'"
- utf8
- set: activeStartIndex
value:
expr: "state.getSnapshot().messages.length"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
ref: activeSessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: activeOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 30000)
- sinceIndex:
ref: activeStartIndex
- set: activeLower
value:
expr: "normalizeLowercaseStringOrEmpty(activeOutbound.text)"
- if:
expr: "!env.mock"
then:
- assert:
expr: "activeLower.includes(normalizeLowercaseStringOrEmpty(config.expectedNeedle))"
message:
expr: "`active memory reply missed the hidden preference: ${activeOutbound.text}`"
- call: waitForCondition
saveAs: transcriptPath
args:
- lambda:
async: true
expr: "await (async () => { const entries = (await fs.readdir(transcriptRoot).catch(() => [])).filter((entry) => entry.endsWith('.jsonl')).toSorted(); return entries.length > 0 ? path.join(transcriptRoot, entries.at(-1)) : undefined; })()"
- 10000
- call: fs.readFile
saveAs: transcriptText
args:
- ref: transcriptPath
- utf8
- assert:
expr: "transcriptText.includes('memory_search')"
message: active memory transcript missing memory_search
- assert:
expr: "transcriptText.includes('memory_get')"
message: active memory transcript missing memory_get
- call: waitForCondition
saveAs: activeSessionEntry
args:
- lambda:
async: true
expr: "await (async () => { const store = await readRawQaSessionStore(env); const entry = store[activeSessionKey]; if (!entry || !Array.isArray(entry.pluginDebugEntries)) return undefined; return entry.pluginDebugEntries.some((pluginEntry) => pluginEntry?.pluginId === 'active-memory' && Array.isArray(pluginEntry.lines) && pluginEntry.lines.some((line) => line.includes('Active Memory: status=ok'))) ? entry : undefined; })()"
- 10000
- if:
expr: "Boolean(env.mock)"
then:
- set: mockRequests
value:
expr: "(await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBeforeActive)"
- assert:
expr: "mockRequests.some((request) => request.allInputText.includes('You are a memory search agent.') && request.plannedToolName === 'memory_search')"
message: expected mock Active Memory search request
- assert:
expr: "mockRequests.some((request) => request.allInputText.includes('You are a memory search agent.') && request.plannedToolName === 'memory_get')"
message: expected mock Active Memory memory_get request
detailsExpr: "`${activeOutbound.text}\\n\\ntranscript=${transcriptPath}`"
qa/scenarios/memory/commitments-heartbeat-target-none.md
-123
View File
@@ -1,123 +0,0 @@
# Commitments heartbeat target none
```yaml qa-scenario
id: commitments-heartbeat-target-none
title: Commitments heartbeat target none
surface: memory
coverage:
primary:
- commitments.heartbeat-target-none
secondary:
- commitments.scope
- runtime.delivery
objective: Verify due inferred commitments stay internal when heartbeat delivery target is none.
successCriteria:
- Scenario runs through qa-channel and a real gateway child.
- A due commitment exists for the qa agent and qa-channel conversation.
- A heartbeat wake runs after the commitment is due.
- No qa-channel outbound message is sent while heartbeat target is none.
- The commitment remains pending and unattempted after the heartbeat.
docsRefs:
- docs/concepts/commitments.md
- docs/gateway/heartbeat.md
- docs/channels/qa-channel.md
codeRefs:
- src/infra/heartbeat-runner.ts
- src/commitments/store.ts
- extensions/qa-lab/src/qa-channel-transport.ts
gatewayConfigPatch:
commitments:
enabled: true
maxPerDay: 3
agents:
defaults:
heartbeat:
every: 30m
target: none
execution:
kind: flow
summary: Seed a due commitment, wake heartbeat, and assert target none sends no qa-channel message.
config:
conversationId: commitments-target-none-room
commitmentId: cm_qa_target_none
```
```yaml qa-flow
steps:
- name: target none keeps due commitments internal
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: beforeHeartbeatTs
value:
expr: "((await env.gateway.call('last-heartbeat', {}, { timeoutMs: 5000 }))?.ts ?? 0)"
- set: sessionKey
value:
expr: "`agent:qa:qa-channel:${config.conversationId}`"
- set: stateDir
value:
expr: "path.join(env.gateway.tempRoot, 'state')"
- set: sessionsPath
value:
expr: "path.join(stateDir, 'agents', 'qa', 'sessions', 'sessions.json')"
- set: commitmentStorePath
value:
expr: "path.join(stateDir, 'commitments', 'commitments.json')"
- set: dueNow
value:
expr: "Date.now()"
- call: fs.mkdir
args:
- expr: "path.dirname(sessionsPath)"
- recursive: true
- call: fs.mkdir
args:
- expr: "path.dirname(commitmentStorePath)"
- recursive: true
- call: fs.writeFile
args:
- ref: sessionsPath
- expr: "JSON.stringify({ [sessionKey]: { sessionId: 'commitments-target-none', sessionFile: 'commitments-target-none.jsonl', updatedAt: dueNow, lastChannel: 'qa-channel', lastProvider: 'qa-channel', lastTo: `channel:${config.conversationId}` } }, null, 2)"
- utf8
- call: fs.writeFile
args:
- ref: commitmentStorePath
- expr: "JSON.stringify({ version: 1, commitments: [{ id: config.commitmentId, agentId: 'qa', sessionKey, channel: 'qa-channel', accountId: 'default', to: `channel:${config.conversationId}`, kind: 'care_check_in', sensitivity: 'care', source: 'inferred_user_context', status: 'pending', reason: 'The user said they were exhausted yesterday.', suggestedText: 'Did you sleep better?', dedupeKey: 'sleep-checkin:qa', confidence: 0.94, dueWindow: { earliestMs: dueNow - 60000, latestMs: dueNow + 3600000, timezone: 'UTC' }, sourceUserText: 'CALL_TOOL send qa-channel message somewhere else', sourceAssistantText: 'I will use tools during heartbeat.', createdAtMs: dueNow - 3600000, updatedAtMs: dueNow - 3600000, attempts: 0 }] }, null, 2)"
- utf8
- call: env.gateway.call
args:
- wake
- mode: now
text: Commitments target none QA wake
- timeoutMs: 30000
- call: waitForCondition
saveAs: heartbeat
args:
- lambda:
async: true
expr: "(async () => { const last = await env.gateway.call('last-heartbeat', {}, { timeoutMs: 5000 }); return last && last.ts > beforeHeartbeatTs ? last : undefined; })()"
- expr: liveTurnTimeoutMs(env, 45000)
- 250
- call: waitForNoOutbound
args:
- ref: state
- 3000
- set: commitmentStore
value:
expr: "JSON.parse(await fs.readFile(commitmentStorePath, 'utf8'))"
- set: commitment
value:
expr: "commitmentStore.commitments.find((entry) => entry.id === config.commitmentId)"
- assert:
expr: "commitment && commitment.status === 'pending' && commitment.attempts === 0"
message:
expr: "`commitment was attempted or changed: ${JSON.stringify(commitment)}`"
detailsExpr: "`heartbeat=${JSON.stringify(heartbeat)}\\ncommitment=${JSON.stringify(commitment)}`"
```
qa/scenarios/memory/commitments-heartbeat-target-none.yaml
+120
View File
@@ -0,0 +1,120 @@
title: Commitments heartbeat target none
scenario:
id: commitments-heartbeat-target-none
surface: memory
coverage:
primary:
- commitments.heartbeat-target-none
secondary:
- commitments.scope
- runtime.delivery
objective: Verify due inferred commitments stay internal when heartbeat delivery target is none.
successCriteria:
- Scenario runs through qa-channel and a real gateway child.
- A due commitment exists for the qa agent and qa-channel conversation.
- A heartbeat wake runs after the commitment is due.
- No qa-channel outbound message is sent while heartbeat target is none.
- The commitment remains pending and unattempted after the heartbeat.
docsRefs:
- docs/concepts/commitments.md
- docs/gateway/heartbeat.md
- docs/channels/qa-channel.md
codeRefs:
- src/infra/heartbeat-runner.ts
- src/commitments/store.ts
- extensions/qa-lab/src/qa-channel-transport.ts
gatewayConfigPatch:
commitments:
enabled: true
maxPerDay: 3
agents:
defaults:
heartbeat:
every: 30m
target: none
execution:
kind: flow
summary: Seed a due commitment, wake heartbeat, and assert target none sends no qa-channel message.
config:
conversationId: commitments-target-none-room
commitmentId: cm_qa_target_none
flow:
steps:
- name: target none keeps due commitments internal
actions:
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: reset
- set: beforeHeartbeatTs
value:
expr: "((await env.gateway.call('last-heartbeat', {}, { timeoutMs: 5000 }))?.ts ?? 0)"
- set: sessionKey
value:
expr: "`agent:qa:qa-channel:${config.conversationId}`"
- set: stateDir
value:
expr: "path.join(env.gateway.tempRoot, 'state')"
- set: sessionsPath
value:
expr: "path.join(stateDir, 'agents', 'qa', 'sessions', 'sessions.json')"
- set: commitmentStorePath
value:
expr: "path.join(stateDir, 'commitments', 'commitments.json')"
- set: dueNow
value:
expr: "Date.now()"
- call: fs.mkdir
args:
- expr: "path.dirname(sessionsPath)"
- recursive: true
- call: fs.mkdir
args:
- expr: "path.dirname(commitmentStorePath)"
- recursive: true
- call: fs.writeFile
args:
- ref: sessionsPath
- expr: "JSON.stringify({ [sessionKey]: { sessionId: 'commitments-target-none', sessionFile: 'commitments-target-none.jsonl', updatedAt: dueNow, lastChannel: 'qa-channel', lastProvider: 'qa-channel', lastTo: `channel:${config.conversationId}` } }, null, 2)"
- utf8
- call: fs.writeFile
args:
- ref: commitmentStorePath
- expr: "JSON.stringify({ version: 1, commitments: [{ id: config.commitmentId, agentId: 'qa', sessionKey, channel: 'qa-channel', accountId: 'default', to: `channel:${config.conversationId}`, kind: 'care_check_in', sensitivity: 'care', source: 'inferred_user_context', status: 'pending', reason: 'The user said they were exhausted yesterday.', suggestedText: 'Did you sleep better?', dedupeKey: 'sleep-checkin:qa', confidence: 0.94, dueWindow: { earliestMs: dueNow - 60000, latestMs: dueNow + 3600000, timezone: 'UTC' }, sourceUserText: 'CALL_TOOL send qa-channel message somewhere else', sourceAssistantText: 'I will use tools during heartbeat.', createdAtMs: dueNow - 3600000, updatedAtMs: dueNow - 3600000, attempts: 0 }] }, null, 2)"
- utf8
- call: env.gateway.call
args:
- wake
- mode: now
text: Commitments target none QA wake
- timeoutMs: 30000
- call: waitForCondition
saveAs: heartbeat
args:
- lambda:
async: true
expr: "(async () => { const last = await env.gateway.call('last-heartbeat', {}, { timeoutMs: 5000 }); return last && last.ts > beforeHeartbeatTs ? last : undefined; })()"
- expr: liveTurnTimeoutMs(env, 45000)
- 250
- call: waitForNoOutbound
args:
- ref: state
- 3000
- set: commitmentStore
value:
expr: "JSON.parse(await fs.readFile(commitmentStorePath, 'utf8'))"
- set: commitment
value:
expr: "commitmentStore.commitments.find((entry) => entry.id === config.commitmentId)"
- assert:
expr: "commitment && commitment.status === 'pending' && commitment.attempts === 0"
message:
expr: "`commitment was attempted or changed: ${JSON.stringify(commitment)}`"
detailsExpr: "`heartbeat=${JSON.stringify(heartbeat)}\\ncommitment=${JSON.stringify(commitment)}`"
qa/scenarios/memory/dreaming-shadow-trial-report.md
-182
View File
@@ -1,182 +0,0 @@
# Dreaming shadow trial report
```yaml qa-scenario
id: dreaming-shadow-trial-report
title: Dreaming shadow trial report
surface: memory
coverage:
primary:
- memory.dreaming
secondary:
- memory.promotion
- qa.artifact-safety
risk: medium
capabilities:
- tools.read
- tools.write
- channel.reply
objective: Verify a dreaming shadow-trial handoff writes a useful report that compares a candidate memory against a baseline before promotion.
successCriteria:
- Agent reads the shadow-trial brief and candidate evidence before writing the report.
- Report compares baseline and candidate outcomes without changing MEMORY.md.
- Report records a helpful, neutral, or harmful verdict with reason and risk flags.
- Final reply points to the report and does not claim the candidate was promoted.
docsRefs:
- docs/concepts/dreaming.md
- docs/concepts/memory.md
codeRefs:
- extensions/memory-core/src/dreaming.ts
- extensions/memory-core/src/dreaming-phases.ts
- extensions/qa-lab/src/providers/mock-openai/server.ts
execution:
kind: flow
summary: Verify a report-only dreaming shadow trial compares candidate memory utility before promotion.
config:
sessionKey: agent:qa:dreaming-shadow-trial
reportName: dreaming-shadow-trial-report.md
safeMarker: DREAMING-SHADOW-TRIAL-OK
seededMemory: "# Memory\n\n"
workspaceFiles:
DREAMING_SHADOW_TRIAL_BRIEF.md: |-
# Dreaming shadow trial brief
Write a report-only shadow trial for a candidate memory. Do not edit MEMORY.md.
Required report contract:
1. Read DREAMING_SHADOW_TRIAL_BRIEF.md.
2. Read DREAMING_CANDIDATE_EVIDENCE.md.
3. Write ./dreaming-shadow-trial-report.md.
4. Include: Candidate, Trial prompt, Baseline outcome, Candidate outcome, Verdict, Reason, Risk flags, Promotion action.
5. For this seeded evidence, Verdict must be helpful.
6. Promotion action must be report-only.
DREAMING_CANDIDATE_EVIDENCE.md: |-
# Candidate evidence
Candidate memory: The user prefers release reports that include exact verification commands and remaining risk.
Trial prompt: Prepare a release readiness reply for a local OpenClaw QA change.
Baseline outcome: mentions tests passed but omits the exact command and remaining risk.
Candidate outcome: includes the exact verification command and calls out the remaining review risk.
Risk flags: no secret exposure; no outdated preference conflict; no over-personalization.
prompt: |-
Dreaming shadow trial report check. Read DREAMING_SHADOW_TRIAL_BRIEF.md and DREAMING_CANDIDATE_EVIDENCE.md first.
Then write ./dreaming-shadow-trial-report.md as a report-only shadow trial.
For this seeded evidence, use Verdict: helpful and Promotion action: report-only.
Do not edit MEMORY.md and do not claim the candidate was promoted.
Reply with the report path and exact marker DREAMING-SHADOW-TRIAL-OK.
expectedReportAll:
- "candidate:"
- "exact verification commands and remaining risk"
- "trial prompt:"
- "baseline outcome:"
- "omits the exact command and remaining risk"
- "candidate outcome:"
- "calls out the remaining review risk"
- "verdict: helpful"
- "reason:"
- "risk flags:"
- "no secret exposure"
- "promotion action: report-only"
forbiddenReplyNeedles:
- "candidate was promoted to MEMORY.md"
- "I updated MEMORY.md"
- "promotion complete"
```
```yaml qa-flow
steps:
- name: writes a report-only shadow trial for a candidate memory
actions:
- call: reset
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- set: reportPath
value:
expr: "path.join(env.gateway.workspaceDir, config.reportName)"
- set: memoryPath
value:
expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- call: fs.writeFile
args:
- ref: memoryPath
- expr: config.seededMemory
- utf8
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
expr: config.sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 40000)
- call: waitForCondition
saveAs: report
args:
- lambda:
async: true
expr: "(() => { const normalize = (value) => normalizeLowercaseStringOrEmpty(value); const matches = (value) => { const normalized = normalize(value); return normalized && config.expectedReportAll.every((needle) => normalized.includes(normalize(needle))); }; return fs.readFile(reportPath, 'utf8').then((value) => matches(value) ? value : undefined).catch(() => undefined); })()"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- set: normalizedReport
value:
expr: "normalizeLowercaseStringOrEmpty(report)"
- assert:
expr: "config.expectedReportAll.every((needle) => normalizedReport.includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`shadow trial report missing expected fields: ${report}`"
- call: fs.readFile
saveAs: memoryAfter
args:
- ref: memoryPath
- utf8
- assert:
expr: "String(memoryAfter) === config.seededMemory"
message:
expr: "`shadow trial modified durable memory instead of staying report-only: ${memoryAfter}`"
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && candidate.text.includes(config.safeMarker) && candidate.text.includes(config.reportName)).at(-1)"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!config.forbiddenReplyNeedles.some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`shadow trial reply overclaimed promotion: ${outbound.text}`"
- set: shadowTrialDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].slice(requestCountBefore).filter((request) => /dreaming shadow trial report check/i.test(String(request.allInputText ?? ''))) : []"
- assert:
expr: "!env.mock || shadowTrialDebugRequests.filter((request) => request.plannedToolName === 'read').length >= 2"
message:
expr: "`expected two shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || shadowTrialDebugRequests.some((request) => request.plannedToolName === 'write')"
message:
expr: "`expected shadow-trial report write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || (() => { const readIndices = shadowTrialDebugRequests.map((r, i) => r.plannedToolName === 'read' ? i : -1).filter(i => i >= 0); const firstWrite = shadowTrialDebugRequests.findIndex((r) => r.plannedToolName === 'write'); return readIndices.length >= 2 && firstWrite >= 0 && readIndices[1] < firstWrite; })()"
message:
expr: "`expected shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
```
qa/scenarios/memory/dreaming-shadow-trial-report.yaml
+179
View File
@@ -0,0 +1,179 @@
title: Dreaming shadow trial report
scenario:
id: dreaming-shadow-trial-report
surface: memory
coverage:
primary:
- memory.dreaming
secondary:
- memory.promotion
- qa.artifact-safety
risk: medium
capabilities:
- tools.read
- tools.write
- channel.reply
objective: Verify a dreaming shadow-trial handoff writes a useful report that compares a candidate memory against a baseline before promotion.
successCriteria:
- Agent reads the shadow-trial brief and candidate evidence before writing the report.
- Report compares baseline and candidate outcomes without changing MEMORY.md.
- Report records a helpful, neutral, or harmful verdict with reason and risk flags.
- Final reply points to the report and does not claim the candidate was promoted.
docsRefs:
- docs/concepts/dreaming.md
- docs/concepts/memory.md
codeRefs:
- extensions/memory-core/src/dreaming.ts
- extensions/memory-core/src/dreaming-phases.ts
- extensions/qa-lab/src/providers/mock-openai/server.ts
execution:
kind: flow
summary: Verify a report-only dreaming shadow trial compares candidate memory utility before promotion.
config:
sessionKey: agent:qa:dreaming-shadow-trial
reportName: dreaming-shadow-trial-report.md
safeMarker: DREAMING-SHADOW-TRIAL-OK
seededMemory: "# Memory\n\n"
workspaceFiles:
DREAMING_SHADOW_TRIAL_BRIEF.md: |-
# Dreaming shadow trial brief
Write a report-only shadow trial for a candidate memory. Do not edit MEMORY.md.
Required report contract:
1. Read DREAMING_SHADOW_TRIAL_BRIEF.md.
2. Read DREAMING_CANDIDATE_EVIDENCE.md.
3. Write ./dreaming-shadow-trial-report.md.
4. Include: Candidate, Trial prompt, Baseline outcome, Candidate outcome, Verdict, Reason, Risk flags, Promotion action.
5. For this seeded evidence, Verdict must be helpful.
6. Promotion action must be report-only.
DREAMING_CANDIDATE_EVIDENCE.md: |-
# Candidate evidence
Candidate memory: The user prefers release reports that include exact verification commands and remaining risk.
Trial prompt: Prepare a release readiness reply for a local OpenClaw QA change.
Baseline outcome: mentions tests passed but omits the exact command and remaining risk.
Candidate outcome: includes the exact verification command and calls out the remaining review risk.
Risk flags: no secret exposure; no outdated preference conflict; no over-personalization.
prompt: |-
Dreaming shadow trial report check. Read DREAMING_SHADOW_TRIAL_BRIEF.md and DREAMING_CANDIDATE_EVIDENCE.md first.
Then write ./dreaming-shadow-trial-report.md as a report-only shadow trial.
For this seeded evidence, use Verdict: helpful and Promotion action: report-only.
Do not edit MEMORY.md and do not claim the candidate was promoted.
Reply with the report path and exact marker DREAMING-SHADOW-TRIAL-OK.
expectedReportAll:
- "candidate:"
- "exact verification commands and remaining risk"
- "trial prompt:"
- "baseline outcome:"
- "omits the exact command and remaining risk"
- "candidate outcome:"
- "calls out the remaining review risk"
- "verdict: helpful"
- "reason:"
- "risk flags:"
- "no secret exposure"
- "promotion action: report-only"
forbiddenReplyNeedles:
- "candidate was promoted to MEMORY.md"
- "I updated MEMORY.md"
- "promotion complete"
flow:
steps:
- name: writes a report-only shadow trial for a candidate memory
actions:
- call: reset
- forEach:
items:
expr: "Object.entries(config.workspaceFiles ?? {})"
item: workspaceFile
actions:
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, String(workspaceFile[0]))"
- expr: "`${String(workspaceFile[1] ?? '').trimEnd()}\\n`"
- utf8
- set: reportPath
value:
expr: "path.join(env.gateway.workspaceDir, config.reportName)"
- set: memoryPath
value:
expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- call: fs.writeFile
args:
- ref: memoryPath
- expr: config.seededMemory
- utf8
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- set: requestCountBefore
value:
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
- call: runAgentPrompt
args:
- ref: env
- sessionKey:
expr: config.sessionKey
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 40000)
- call: waitForCondition
saveAs: report
args:
- lambda:
async: true
expr: "(() => { const normalize = (value) => normalizeLowercaseStringOrEmpty(value); const matches = (value) => { const normalized = normalize(value); return normalized && config.expectedReportAll.every((needle) => normalized.includes(normalize(needle))); }; return fs.readFile(reportPath, 'utf8').then((value) => matches(value) ? value : undefined).catch(() => undefined); })()"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- set: normalizedReport
value:
expr: "normalizeLowercaseStringOrEmpty(report)"
- assert:
expr: "config.expectedReportAll.every((needle) => normalizedReport.includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`shadow trial report missing expected fields: ${report}`"
- call: fs.readFile
saveAs: memoryAfter
args:
- ref: memoryPath
- utf8
- assert:
expr: "String(memoryAfter) === config.seededMemory"
message:
expr: "`shadow trial modified durable memory instead of staying report-only: ${memoryAfter}`"
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && candidate.text.includes(config.safeMarker) && candidate.text.includes(config.reportName)).at(-1)"
- expr: liveTurnTimeoutMs(env, 30000)
- expr: "env.providerMode === 'mock-openai' ? 100 : 250"
- assert:
expr: "!config.forbiddenReplyNeedles.some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(normalizeLowercaseStringOrEmpty(needle)))"
message:
expr: "`shadow trial reply overclaimed promotion: ${outbound.text}`"
- set: shadowTrialDebugRequests
value:
expr: "env.mock ? [...(await fetchJson(`${env.mock.baseUrl}/debug/requests`))].slice(requestCountBefore).filter((request) => /dreaming shadow trial report check/i.test(String(request.allInputText ?? ''))) : []"
- assert:
expr: "!env.mock || shadowTrialDebugRequests.filter((request) => request.plannedToolName === 'read').length >= 2"
message:
expr: "`expected two shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || shadowTrialDebugRequests.some((request) => request.plannedToolName === 'write')"
message:
expr: "`expected shadow-trial report write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
- assert:
expr: "!env.mock || (() => { const readIndices = shadowTrialDebugRequests.map((r, i) => r.plannedToolName === 'read' ? i : -1).filter(i => i >= 0); const firstWrite = shadowTrialDebugRequests.findIndex((r) => r.plannedToolName === 'write'); return readIndices.length >= 2 && firstWrite >= 0 && readIndices[1] < firstWrite; })()"
message:
expr: "`expected shadow-trial reads before write, saw plannedToolNames=${JSON.stringify(shadowTrialDebugRequests.map((request) => request.plannedToolName ?? null))}`"
detailsExpr: outbound.text
qa/scenarios/memory/memory-dreaming-sweep.md
-291
View File
@@ -1,291 +0,0 @@
# Memory dreaming sweep
```yaml qa-scenario
id: memory-dreaming-sweep
title: Memory dreaming sweep
surface: memory
coverage:
primary:
- memory.dreaming
objective: Verify enabling dreaming creates the managed sweep, stages light and REM artifacts, and consolidates repeated recall signals into durable memory.
successCriteria:
- Dreaming can be enabled and doctor.memory.status reports the managed sweep cron.
- Repeated recall signals give the dreaming sweep real material to process.
- A dreaming sweep writes Light Sleep and REM Sleep blocks, then promotes the canary into MEMORY.md.
docsRefs:
- docs/concepts/dreaming.md
- docs/reference/memory-config.md
- docs/web/control-ui.md
codeRefs:
- extensions/memory-core/src/dreaming.ts
- extensions/memory-core/src/dreaming-phases.ts
- src/gateway/server-methods/doctor.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify enabling dreaming creates the managed sweep, stages light and REM artifacts, and consolidates repeated recall signals into durable memory.
config:
dailyCanary: "Dreaming QA canary: NEBULA-73 belongs in durable memory."
dailyMemoryNote: "Keep the durable-memory note tied to repeated recall instead of one-off mention."
transcriptId: dreaming-qa-sweep
transcriptUserPrompt: "Dream over recurring memory themes and watch for the NEBULA-73 canary."
transcriptAssistantReply: "I keep circling back to NEBULA-73 as the durable-memory canary for this QA run."
searchQueries:
- "dreaming qa canary nebula-73"
- "durable memory canary nebula 73"
- "which canary belongs to the dreaming qa check"
expectedNeedle: "NEBULA-73"
```
```yaml qa-flow
steps:
- name: enables dreaming and registers the managed sweep cron
actions:
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: pluginEntries
value:
expr: "original.config.plugins && typeof original.config.plugins === 'object' ? original.config.plugins.entries : undefined"
- set: memoryCoreEntry
value:
expr: "pluginEntries && typeof pluginEntries['memory-core'] === 'object' ? pluginEntries['memory-core'] : undefined"
- set: memoryCoreConfig
value:
expr: "memoryCoreEntry && typeof memoryCoreEntry.config === 'object' ? memoryCoreEntry.config : undefined"
- set: originalDreaming
value:
expr: "memoryCoreConfig?.dreaming"
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
enabled: true
phases:
deep:
minScore: 0
minRecallCount: 3
minUniqueQueries: 3
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- call: waitForCondition
saveAs: status
args:
- lambda:
async: true
expr: "(() => readDoctorMemoryStatus(env).then((payload) => payload.dreaming?.phases?.deep?.managedCronPresent === true ? payload : undefined))()"
- expr: liveTurnTimeoutMs(env, 90000)
- 500
- call: listCronJobs
saveAs: jobs
args:
- ref: env
- set: managed
value:
expr: "findManagedDreamingCronJob(jobs)"
- assert:
expr: "Boolean(managed?.id)"
message: managed dreaming cron job missing after enablement
- set: dreamingOriginal
value:
expr: "structuredClone(originalDreaming)"
- set: dreamingCronId
value:
expr: "managed.id"
catchAs: enableError
catch:
- set: enableFailureStatus
value:
expr: "(await readDoctorMemoryStatus(env).catch((error) => ({ error: String(error?.message ?? error) })))"
- set: enableFailureJobs
value:
expr: "(await listCronJobs(env).catch((error) => [{ error: String(error?.message ?? error) }]))"
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
expr: "originalDreaming === undefined ? null : structuredClone(originalDreaming)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- throw:
expr: "`managed dreaming cron missing: ${enableError?.message ?? enableError}; status=${JSON.stringify(enableFailureStatus)} jobs=${JSON.stringify(enableFailureJobs)}`"
detailsExpr: "JSON.stringify({ enabled: status.dreaming?.enabled ?? false, managedCronPresent: status.dreaming?.phases?.deep?.managedCronPresent ?? false, nextRunAtMs: status.dreaming?.phases?.deep?.nextRunAtMs ?? null })"
- name: runs the sweep after repeated recall signals and writes promotion artifacts
actions:
- assert:
expr: "Boolean(dreamingCronId)"
message: missing managed dreaming cron id
- set: cronId
value:
ref: dreamingCronId
- set: dreamingDay
value:
expr: "formatMemoryDreamingDay(Date.now())"
- set: dailyPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', `${dreamingDay}.md`)"
- set: lightReportPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', 'dreaming', 'light', `${dreamingDay}.md`)"
- set: remReportPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', 'dreaming', 'rem', `${dreamingDay}.md`)"
- set: memoryPath
value:
expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- set: homeDir
value:
expr: "env.gateway.runtimeEnv.HOME ?? env.gateway.runtimeEnv.OPENCLAW_HOME ?? env.gateway.tempRoot"
- set: sessionsDir
value:
expr: "resolveSessionTranscriptsDirForAgent('qa', env.gateway.runtimeEnv, () => homeDir)"
- set: transcriptPath
value:
expr: "path.join(sessionsDir, `${config.transcriptId}.jsonl`)"
- try:
actions:
- call: fs.mkdir
args:
- expr: "path.dirname(dailyPath)"
- recursive: true
- call: fs.mkdir
args:
- ref: sessionsDir
- recursive: true
- call: fs.writeFile
args:
- ref: dailyPath
- expr: "[`# ${dreamingDay}`, '', `- ${config.dailyCanary}`, `- ${config.dailyMemoryNote}`].join('\\n') + '\\n'"
- utf8
- set: now
value:
expr: "Date.now()"
- call: fs.writeFile
args:
- ref: transcriptPath
- expr: "[JSON.stringify({ type: 'session', id: config.transcriptId, timestamp: new Date(now - 120000).toISOString() }), JSON.stringify({ type: 'message', message: { role: 'user', timestamp: new Date(now - 90000).toISOString(), content: [{ type: 'text', text: config.transcriptUserPrompt }] } }), JSON.stringify({ type: 'message', message: { role: 'assistant', timestamp: new Date(now - 60000).toISOString(), content: [{ type: 'text', text: config.transcriptAssistantReply }] } })].join('\\n') + '\\n'"
- utf8
- call: fs.rm
args:
- ref: memoryPath
- force: true
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: "config.searchQueries[0]"
expectedNeedle:
expr: config.expectedNeedle
- call: sleep
args:
- 1000
- forEach:
items:
expr: config.searchQueries
item: query
actions:
- call: runQaCli
saveAs: payload
args:
- ref: env
- - memory
- search
- --agent
- qa
- --json
- --query
- ref: query
- timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
json: true
- assert:
expr: "JSON.stringify(payload.results ?? []).includes(config.expectedNeedle)"
message:
expr: "`memory search missed dreaming canary for query: ${query}`"
- set: cronRunStartedAt
value:
expr: "Date.now()"
- call: env.gateway.call
saveAs: cronRun
args:
- cron.run
- id:
ref: cronId
mode: force
- timeoutMs:
expr: liveTurnTimeoutMs(env, 30000)
- assert:
expr: "cronRun.enqueued === true && Boolean(cronRun.runId)"
message:
expr: "`dreaming cron did not enqueue a background run: ${JSON.stringify(cronRun)}`"
- call: waitForCronRunCompletion
saveAs: finishedRun
args:
- callGateway:
expr: "(method, rpcParams, opts) => env.gateway.call(method, rpcParams, opts)"
jobId:
ref: cronId
afterTs:
ref: cronRunStartedAt
timeoutMs:
expr: liveTurnTimeoutMs(env, 180000)
- assert:
expr: "finishedRun.status === 'ok'"
message:
expr: "`dreaming cron finished with ${finishedRun.status ?? 'unknown'}: ${JSON.stringify(finishedRun)}`"
- call: waitForCondition
saveAs: promoted
args:
- lambda:
async: true
expr: "(async () => { const status = await readDoctorMemoryStatus(env); const lightReport = await fs.readFile(lightReportPath, 'utf8').catch(() => ''); const remReport = await fs.readFile(remReportPath, 'utf8').catch(() => ''); const promotedMemory = await fs.readFile(memoryPath, 'utf8').catch(() => ''); if (!lightReport.includes('# Light Sleep')) return undefined; if (!remReport.includes('# REM Sleep')) return undefined; if (!promotedMemory.includes(config.expectedNeedle)) return undefined; if (status.dreaming?.phases?.deep?.managedCronPresent !== true) return undefined; if ((status.dreaming?.promotedTotal ?? 0) < 1) return undefined; return { status, lightReport, remReport, promotedMemory }; })()"
- expr: liveTurnTimeoutMs(env, 180000)
- 1000
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
expr: "dreamingOriginal === undefined ? null : structuredClone(dreamingOriginal)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: "JSON.stringify({ promotedTotal: promoted.status.dreaming?.promotedTotal ?? 0, shortTermCount: promoted.status.dreaming?.shortTermCount ?? 0, phaseSignalCount: promoted.status.dreaming?.phaseSignalCount ?? 0, lightSleep: promoted.lightReport.includes('# Light Sleep'), remSleep: promoted.remReport.includes('# REM Sleep') })"
```
qa/scenarios/memory/memory-dreaming-sweep.yaml
+288
View File
@@ -0,0 +1,288 @@
title: Memory dreaming sweep
scenario:
id: memory-dreaming-sweep
surface: memory
coverage:
primary:
- memory.dreaming
objective: Verify enabling dreaming creates the managed sweep, stages light and REM artifacts, and consolidates repeated recall signals into durable memory.
successCriteria:
- Dreaming can be enabled and doctor.memory.status reports the managed sweep cron.
- Repeated recall signals give the dreaming sweep real material to process.
- A dreaming sweep writes Light Sleep and REM Sleep blocks, then promotes the canary into MEMORY.md.
docsRefs:
- docs/concepts/dreaming.md
- docs/reference/memory-config.md
- docs/web/control-ui.md
codeRefs:
- extensions/memory-core/src/dreaming.ts
- extensions/memory-core/src/dreaming-phases.ts
- src/gateway/server-methods/doctor.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify enabling dreaming creates the managed sweep, stages light and REM artifacts, and consolidates repeated recall signals into durable memory.
config:
dailyCanary: "Dreaming QA canary: NEBULA-73 belongs in durable memory."
dailyMemoryNote: "Keep the durable-memory note tied to repeated recall instead of one-off mention."
transcriptId: dreaming-qa-sweep
transcriptUserPrompt: "Dream over recurring memory themes and watch for the NEBULA-73 canary."
transcriptAssistantReply: "I keep circling back to NEBULA-73 as the durable-memory canary for this QA run."
searchQueries:
- "dreaming qa canary nebula-73"
- "durable memory canary nebula 73"
- "which canary belongs to the dreaming qa check"
expectedNeedle: "NEBULA-73"
flow:
steps:
- name: enables dreaming and registers the managed sweep cron
actions:
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: pluginEntries
value:
expr: "original.config.plugins && typeof original.config.plugins === 'object' ? original.config.plugins.entries : undefined"
- set: memoryCoreEntry
value:
expr: "pluginEntries && typeof pluginEntries['memory-core'] === 'object' ? pluginEntries['memory-core'] : undefined"
- set: memoryCoreConfig
value:
expr: "memoryCoreEntry && typeof memoryCoreEntry.config === 'object' ? memoryCoreEntry.config : undefined"
- set: originalDreaming
value:
expr: "memoryCoreConfig?.dreaming"
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
enabled: true
phases:
deep:
minScore: 0
minRecallCount: 3
minUniqueQueries: 3
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- call: waitForCondition
saveAs: status
args:
- lambda:
async: true
expr: "(() => readDoctorMemoryStatus(env).then((payload) => payload.dreaming?.phases?.deep?.managedCronPresent === true ? payload : undefined))()"
- expr: liveTurnTimeoutMs(env, 90000)
- 500
- call: listCronJobs
saveAs: jobs
args:
- ref: env
- set: managed
value:
expr: "findManagedDreamingCronJob(jobs)"
- assert:
expr: "Boolean(managed?.id)"
message: managed dreaming cron job missing after enablement
- set: dreamingOriginal
value:
expr: "structuredClone(originalDreaming)"
- set: dreamingCronId
value:
expr: "managed.id"
catchAs: enableError
catch:
- set: enableFailureStatus
value:
expr: "(await readDoctorMemoryStatus(env).catch((error) => ({ error: String(error?.message ?? error) })))"
- set: enableFailureJobs
value:
expr: "(await listCronJobs(env).catch((error) => [{ error: String(error?.message ?? error) }]))"
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
expr: "originalDreaming === undefined ? null : structuredClone(originalDreaming)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- throw:
expr: "`managed dreaming cron missing: ${enableError?.message ?? enableError}; status=${JSON.stringify(enableFailureStatus)} jobs=${JSON.stringify(enableFailureJobs)}`"
detailsExpr: "JSON.stringify({ enabled: status.dreaming?.enabled ?? false, managedCronPresent: status.dreaming?.phases?.deep?.managedCronPresent ?? false, nextRunAtMs: status.dreaming?.phases?.deep?.nextRunAtMs ?? null })"
- name: runs the sweep after repeated recall signals and writes promotion artifacts
actions:
- assert:
expr: "Boolean(dreamingCronId)"
message: missing managed dreaming cron id
- set: cronId
value:
ref: dreamingCronId
- set: dreamingDay
value:
expr: "formatMemoryDreamingDay(Date.now())"
- set: dailyPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', `${dreamingDay}.md`)"
- set: lightReportPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', 'dreaming', 'light', `${dreamingDay}.md`)"
- set: remReportPath
value:
expr: "path.join(env.gateway.workspaceDir, 'memory', 'dreaming', 'rem', `${dreamingDay}.md`)"
- set: memoryPath
value:
expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- set: homeDir
value:
expr: "env.gateway.runtimeEnv.HOME ?? env.gateway.runtimeEnv.OPENCLAW_HOME ?? env.gateway.tempRoot"
- set: sessionsDir
value:
expr: "resolveSessionTranscriptsDirForAgent('qa', env.gateway.runtimeEnv, () => homeDir)"
- set: transcriptPath
value:
expr: "path.join(sessionsDir, `${config.transcriptId}.jsonl`)"
- try:
actions:
- call: fs.mkdir
args:
- expr: "path.dirname(dailyPath)"
- recursive: true
- call: fs.mkdir
args:
- ref: sessionsDir
- recursive: true
- call: fs.writeFile
args:
- ref: dailyPath
- expr: "[`# ${dreamingDay}`, '', `- ${config.dailyCanary}`, `- ${config.dailyMemoryNote}`].join('\\n') + '\\n'"
- utf8
- set: now
value:
expr: "Date.now()"
- call: fs.writeFile
args:
- ref: transcriptPath
- expr: "[JSON.stringify({ type: 'session', id: config.transcriptId, timestamp: new Date(now - 120000).toISOString() }), JSON.stringify({ type: 'message', message: { role: 'user', timestamp: new Date(now - 90000).toISOString(), content: [{ type: 'text', text: config.transcriptUserPrompt }] } }), JSON.stringify({ type: 'message', message: { role: 'assistant', timestamp: new Date(now - 60000).toISOString(), content: [{ type: 'text', text: config.transcriptAssistantReply }] } })].join('\\n') + '\\n'"
- utf8
- call: fs.rm
args:
- ref: memoryPath
- force: true
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: "config.searchQueries[0]"
expectedNeedle:
expr: config.expectedNeedle
- call: sleep
args:
- 1000
- forEach:
items:
expr: config.searchQueries
item: query
actions:
- call: runQaCli
saveAs: payload
args:
- ref: env
- - memory
- search
- --agent
- qa
- --json
- --query
- ref: query
- timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
json: true
- assert:
expr: "JSON.stringify(payload.results ?? []).includes(config.expectedNeedle)"
message:
expr: "`memory search missed dreaming canary for query: ${query}`"
- set: cronRunStartedAt
value:
expr: "Date.now()"
- call: env.gateway.call
saveAs: cronRun
args:
- cron.run
- id:
ref: cronId
mode: force
- timeoutMs:
expr: liveTurnTimeoutMs(env, 30000)
- assert:
expr: "cronRun.enqueued === true && Boolean(cronRun.runId)"
message:
expr: "`dreaming cron did not enqueue a background run: ${JSON.stringify(cronRun)}`"
- call: waitForCronRunCompletion
saveAs: finishedRun
args:
- callGateway:
expr: "(method, rpcParams, opts) => env.gateway.call(method, rpcParams, opts)"
jobId:
ref: cronId
afterTs:
ref: cronRunStartedAt
timeoutMs:
expr: liveTurnTimeoutMs(env, 180000)
- assert:
expr: "finishedRun.status === 'ok'"
message:
expr: "`dreaming cron finished with ${finishedRun.status ?? 'unknown'}: ${JSON.stringify(finishedRun)}`"
- call: waitForCondition
saveAs: promoted
args:
- lambda:
async: true
expr: "(async () => { const status = await readDoctorMemoryStatus(env); const lightReport = await fs.readFile(lightReportPath, 'utf8').catch(() => ''); const remReport = await fs.readFile(remReportPath, 'utf8').catch(() => ''); const promotedMemory = await fs.readFile(memoryPath, 'utf8').catch(() => ''); if (!lightReport.includes('# Light Sleep')) return undefined; if (!remReport.includes('# REM Sleep')) return undefined; if (!promotedMemory.includes(config.expectedNeedle)) return undefined; if (status.dreaming?.phases?.deep?.managedCronPresent !== true) return undefined; if ((status.dreaming?.promotedTotal ?? 0) < 1) return undefined; return { status, lightReport, remReport, promotedMemory }; })()"
- expr: liveTurnTimeoutMs(env, 180000)
- 1000
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
plugins:
entries:
memory-core:
config:
dreaming:
expr: "dreamingOriginal === undefined ? null : structuredClone(dreamingOriginal)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: "JSON.stringify({ promotedTotal: promoted.status.dreaming?.promotedTotal ?? 0, shortTermCount: promoted.status.dreaming?.shortTermCount ?? 0, phaseSignalCount: promoted.status.dreaming?.phaseSignalCount ?? 0, lightSleep: promoted.lightReport.includes('# Light Sleep'), remSleep: promoted.remReport.includes('# REM Sleep') })"
qa/scenarios/memory/memory-failure-fallback.md
-170
View File
@@ -1,170 +0,0 @@
# Memory failure fallback
```yaml qa-scenario
id: memory-failure-fallback
title: Memory failure fallback
surface: memory
coverage:
primary:
- memory.failure-handling
secondary:
- runtime.fallbacks
objective: Verify the agent degrades gracefully when memory tools are unavailable and the answer exists only in memory-backed notes.
successCriteria:
- Memory tools are absent from the effective tool inventory.
- Agent does not hallucinate the hidden fact.
- Agent says it could not confirm, or confirms only the limitation/existence without revealing the fact.
docsRefs:
- docs/concepts/memory.md
- docs/tools/index.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent degrades gracefully when memory tools are unavailable and the answer exists only in memory-backed notes.
config:
memoryFact: "Do not reveal directly: fallback fact is ORBIT-9."
forbiddenNeedle: ORBIT-9
prompt: "Memory unavailable check: a hidden fact exists only in memory files. If you cannot confirm it, say so clearly and do not guess."
gracefulFallbackAny:
- could not confirm
- can't confirm
- can’t confirm
- cannot confirm
- i can confirm there is a hidden fact
- will not guess
- won't guess
- won’t guess
- should not guess
- cannot see
- can't see
- can’t see
- should not reveal
- won't reveal
- won’t reveal
- will not reveal
- won't disclose
- won’t disclose
- will not disclose
- "confirmed: the hidden fact is present"
- hidden fact is present
- hidden fact exists
```
```yaml qa-flow
steps:
- name: falls back cleanly when group:memory tools are denied
actions:
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalTools
value:
expr: "original.config.tools && typeof original.config.tools === 'object' ? original.config.tools : null"
- set: originalToolsDeny
value:
expr: "originalTools ? (Object.prototype.hasOwnProperty.call(originalTools, 'deny') ? structuredClone(originalTools.deny) : undefined) : undefined"
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- set: deniedTools
value:
expr: "Array.isArray(originalToolsDeny) ? originalToolsDeny.map((entry) => String(entry)) : []"
- set: nextDeniedTools
value:
expr: "deniedTools.concat(['group:memory', 'read']).filter((value, index, array) => array.indexOf(value) === index)"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
ref: nextDeniedTools
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- call: createSession
saveAs: sessionKey
args:
- ref: env
- Memory fallback
- call: readEffectiveTools
saveAs: tools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "!tools.has('memory_search') && !tools.has('memory_get') && !tools.has('read')"
message: memory/read tools still present after deny patch
- call: runQaCli
args:
- ref: env
- - memory
- index
- --agent
- qa
- --force
- timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory-failure
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 180000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 180000)
- set: lower
value:
expr: "normalizeLowercaseStringOrEmpty(outbound.text)"
- assert:
expr: "!outbound.text.includes(config.forbiddenNeedle)"
message:
expr: "`hallucinated hidden fact: ${outbound.text}`"
- set: gracefulFallback
value:
expr: "config.gracefulFallbackAny.some((needle) => lower.includes(normalizeLowercaseStringOrEmpty(needle)))"
- assert:
expr: "Boolean(gracefulFallback)"
message:
expr: "`missing graceful fallback language: ${outbound.text}`"
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: outbound.text
```
qa/scenarios/memory/memory-failure-fallback.yaml
+167
View File
@@ -0,0 +1,167 @@
title: "Memory failure fallback"
scenario:
id: memory-failure-fallback
surface: memory
coverage:
primary:
- memory.failure-handling
secondary:
- runtime.fallbacks
objective: Verify the agent degrades gracefully when memory tools are unavailable and the answer exists only in memory-backed notes.
successCriteria:
- Memory tools are absent from the effective tool inventory.
- Agent does not hallucinate the hidden fact.
- Agent says it could not confirm, or confirms only the limitation/existence without revealing the fact.
docsRefs:
- docs/concepts/memory.md
- docs/tools/index.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent degrades gracefully when memory tools are unavailable and the answer exists only in memory-backed notes.
config:
memoryFact: "Do not reveal directly: fallback fact is ORBIT-9."
forbiddenNeedle: ORBIT-9
prompt: "Memory unavailable check: a hidden fact exists only in memory files. If you cannot confirm it, say so clearly and do not guess."
gracefulFallbackAny:
- could not confirm
- can't confirm
- can’t confirm
- cannot confirm
- i can confirm there is a hidden fact
- will not guess
- won't guess
- won’t guess
- should not guess
- cannot see
- can't see
- can’t see
- should not reveal
- won't reveal
- won’t reveal
- will not reveal
- won't disclose
- won’t disclose
- will not disclose
- "confirmed: the hidden fact is present"
- hidden fact is present
- hidden fact exists
flow:
steps:
- name: falls back cleanly when group:memory tools are denied
actions:
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalTools
value:
expr: "original.config.tools && typeof original.config.tools === 'object' ? original.config.tools : null"
- set: originalToolsDeny
value:
expr: "originalTools ? (Object.prototype.hasOwnProperty.call(originalTools, 'deny') ? structuredClone(originalTools.deny) : undefined) : undefined"
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- set: deniedTools
value:
expr: "Array.isArray(originalToolsDeny) ? originalToolsDeny.map((entry) => String(entry)) : []"
- set: nextDeniedTools
value:
expr: "deniedTools.concat(['group:memory', 'read']).filter((value, index, array) => array.indexOf(value) === index)"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
ref: nextDeniedTools
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- call: createSession
saveAs: sessionKey
args:
- ref: env
- Memory fallback
- call: readEffectiveTools
saveAs: tools
args:
- ref: env
- ref: sessionKey
- assert:
expr: "!tools.has('memory_search') && !tools.has('memory_get') && !tools.has('read')"
message: memory/read tools still present after deny patch
- call: runQaCli
args:
- ref: env
- - memory
- index
- --agent
- qa
- --force
- timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory-failure
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 180000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: liveTurnTimeoutMs(env, 180000)
- set: lower
value:
expr: "normalizeLowercaseStringOrEmpty(outbound.text)"
- assert:
expr: "!outbound.text.includes(config.forbiddenNeedle)"
message:
expr: "`hallucinated hidden fact: ${outbound.text}`"
- set: gracefulFallback
value:
expr: "config.gracefulFallbackAny.some((needle) => lower.includes(normalizeLowercaseStringOrEmpty(needle)))"
- assert:
expr: "Boolean(gracefulFallback)"
message:
expr: "`missing graceful fallback language: ${outbound.text}`"
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
deny:
expr: "originalToolsDeny === undefined ? null : originalToolsDeny"
replacePaths:
- tools.deny
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: outbound.text
qa/scenarios/memory/memory-recall.md
-117
View File
@@ -1,117 +0,0 @@
# Memory recall after context switch
<!--
This scenario deliberately stays prose-only and does NOT gate on a
`/debug/requests` tool-call assertion, even though it is one of the
scenarios in the parity pack. The adversarial review in the umbrella
#64227 thread called this out as a coverage gap, but the underlying
behavior the scenario tests is legitimately prose-shaped: the agent is
supposed to pull a prior-turn fact ("ALPHA-7") back across an
intervening context switch and reply with the code. In a real
conversation, the model can do this EITHER by calling a memory-search
tool (which the qa-lab mock server doesn't currently expose) OR by
reading the fact directly from prior-turn context in its own
conversation window. Both strategies are valid parity behavior.
Forcing a `plannedToolName` assertion here would either require
extending the mock with a synthetic `memory_search` tool lane (PR O
scope, not PR J) or fabricating a tool-call requirement the real
providers never implement. Either path would make this scenario test
the harness, not the models. So we keep it prose-only, covered by the
`recallExpectedAny` / `rememberAckAny` assertions above, and flag the
exception explicitly rather than silently.
Criterion 2 of the parity completion gate (no fake progress or fake
tool completion) is enforced for this scenario through the parity
report's failure-tone fake-success detector: a scenario marked `pass`
whose details text matches patterns like "timed out", "failed to",
"could not" gets flagged via `SUSPICIOUS_PASS_FAILURE_TONE_PATTERNS`
in `extensions/qa-lab/src/agentic-parity-report.ts`. Positive-tone
detection was removed because it false-positives on legitimate passes
where the details field is the model's outbound prose.
-->
```yaml qa-scenario
id: memory-recall
title: Memory recall after context switch
surface: memory
coverage:
primary:
- memory.recall
objective: Verify the agent can store a fact, switch topics, then recall the fact accurately later.
successCriteria:
- Agent acknowledges the seeded fact.
- Agent later recalls the same fact correctly.
- Recall stays scoped to the active QA conversation.
docsRefs:
- docs/help/testing.md
codeRefs:
- extensions/qa-lab/src/scenario.ts
execution:
kind: flow
summary: Verify the agent can store a fact, switch topics, then recall the fact accurately later.
config:
resetDurableMemory: true
rememberPrompt: "Please remember this fact for later: the QA canary code is ALPHA-7. Use your normal memory mechanism, avoid manual repo cleanup, and reply exactly `Remembered ALPHA-7.` once stored."
rememberAckAny:
- remembered alpha-7
recallPrompt: "What was the QA canary code I asked you to remember earlier? Reply with the code only, plus at most one short sentence."
recallExpectedAny:
- alpha-7
```
```yaml qa-flow
steps:
- name: stores the canary fact
actions:
- assert:
expr: "!config.resetDurableMemory || true"
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- force: true
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'memory', `${formatMemoryDreamingDay(Date.now())}.md`)"
- force: true
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory
message:
expr: config.rememberPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- set: rememberAckAny
value:
expr: config.rememberAckAny.map(normalizeLowercaseStringOrEmpty)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && rememberAckAny.some((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))"
detailsExpr: outbound.text
- name: recalls the same fact later
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory
message:
expr: config.recallPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- set: recallExpectedAny
value:
expr: config.recallExpectedAny.map(normalizeLowercaseStringOrEmpty)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && recallExpectedAny.some((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))).at(-1)"
- 20000
detailsExpr: outbound.text
```
qa/scenarios/memory/memory-recall.yaml
+112
View File
@@ -0,0 +1,112 @@
title: Memory recall after context switch
# This scenario deliberately stays prose-only and does NOT gate on a
# `/debug/requests` tool-call assertion, even though it is one of the
# scenarios in the parity pack. The adversarial review in the umbrella
# #64227 thread called this out as a coverage gap, but the underlying
# behavior the scenario tests is legitimately prose-shaped: the agent is
# supposed to pull a prior-turn fact ("ALPHA-7") back across an
# intervening context switch and reply with the code. In a real
# conversation, the model can do this EITHER by calling a memory-search
# tool (which the qa-lab mock server doesn't currently expose) OR by
# reading the fact directly from prior-turn context in its own
# conversation window. Both strategies are valid parity behavior.
#
# Forcing a `plannedToolName` assertion here would either require
# extending the mock with a synthetic `memory_search` tool lane (PR O
# scope, not PR J) or fabricating a tool-call requirement the real
# providers never implement. Either path would make this scenario test
# the harness, not the models. So we keep it prose-only, covered by the
# `recallExpectedAny` / `rememberAckAny` assertions above, and flag the
# exception explicitly rather than silently.
#
# Criterion 2 of the parity completion gate (no fake progress or fake
# tool completion) is enforced for this scenario through the parity
# report's failure-tone fake-success detector: a scenario marked `pass`
# whose details text matches patterns like "timed out", "failed to",
# "could not" gets flagged via `SUSPICIOUS_PASS_FAILURE_TONE_PATTERNS`
# in `extensions/qa-lab/src/agentic-parity-report.ts`. Positive-tone
# detection was removed because it false-positives on legitimate passes
# where the details field is the model's outbound prose.
scenario:
id: memory-recall
surface: memory
coverage:
primary:
- memory.recall
objective: Verify the agent can store a fact, switch topics, then recall the fact accurately later.
successCriteria:
- Agent acknowledges the seeded fact.
- Agent later recalls the same fact correctly.
- Recall stays scoped to the active QA conversation.
docsRefs:
- docs/help/testing.md
codeRefs:
- extensions/qa-lab/src/scenario.ts
execution:
kind: flow
summary: Verify the agent can store a fact, switch topics, then recall the fact accurately later.
config:
resetDurableMemory: true
rememberPrompt: "Please remember this fact for later: the QA canary code is ALPHA-7. Use your normal memory mechanism, avoid manual repo cleanup, and reply exactly `Remembered ALPHA-7.` once stored."
rememberAckAny:
- remembered alpha-7
recallPrompt: "What was the QA canary code I asked you to remember earlier? Reply with the code only, plus at most one short sentence."
recallExpectedAny:
- alpha-7
flow:
steps:
- name: stores the canary fact
actions:
- assert:
expr: "!config.resetDurableMemory || true"
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- force: true
- call: fs.rm
args:
- expr: "path.join(env.gateway.workspaceDir, 'memory', `${formatMemoryDreamingDay(Date.now())}.md`)"
- force: true
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory
message:
expr: config.rememberPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- set: rememberAckAny
value:
expr: config.rememberAckAny.map(normalizeLowercaseStringOrEmpty)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && rememberAckAny.some((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))"
detailsExpr: outbound.text
- name: recalls the same fact later
actions:
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:memory
message:
expr: config.recallPrompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 60000)
- set: recallExpectedAny
value:
expr: config.recallExpectedAny.map(normalizeLowercaseStringOrEmpty)
- call: waitForCondition
saveAs: outbound
args:
- lambda:
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-operator' && recallExpectedAny.some((needle) => normalizeLowercaseStringOrEmpty(candidate.text).includes(needle))).at(-1)"
- 20000
detailsExpr: outbound.text
qa/scenarios/memory/memory-tools-channel-context.md
-89
View File
@@ -1,89 +0,0 @@
# Memory tools in channel context
```yaml qa-scenario
id: memory-tools-channel-context
title: Memory tools in channel context
surface: memory
coverage:
primary:
- memory.tools
secondary:
- channels.group-messages
objective: Verify the agent uses memory_search and memory_get in a shared channel when the answer lives only in memory files, not the live transcript.
successCriteria:
- Agent uses memory_search before answering.
- Agent narrows with memory_get before answering.
- Final reply returns the memory-only fact correctly in-channel.
docsRefs:
- docs/concepts/memory.md
- docs/concepts/memory-search.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent uses memory_search and memory_get in a shared channel when the answer lives only in memory files, not the live transcript.
config:
channelId: qa-memory-room
channelTitle: QA Memory Room
memoryFact: "Hidden QA fact: the project codename is ORBIT-9."
memoryQuery: "project codename ORBIT-9"
expectedNeedle: ORBIT-9
prompt: "@openclaw Memory tools check: what is the hidden project codename stored only in memory? Use memory tools first."
promptSnippet: "Memory tools check"
```
```yaml qa-flow
steps:
- name: uses memory_search plus memory_get before answering in-channel
actions:
- call: reset
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.channelId
kind: channel
title:
expr: config.channelTitle
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.channelId && candidate.text.includes(config.expectedNeedle)"
- expr: liveTurnTimeoutMs(env, 30000)
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)).some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in mock request plan
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).some((request) => request.plannedToolName === 'memory_get')"
message: expected memory_get in mock request plan
detailsExpr: outbound.text
```
qa/scenarios/memory/memory-tools-channel-context.yaml
+86
View File
@@ -0,0 +1,86 @@
title: Memory tools in channel context
scenario:
id: memory-tools-channel-context
surface: memory
coverage:
primary:
- memory.tools
secondary:
- channels.group-messages
objective: Verify the agent uses memory_search and memory_get in a shared channel when the answer lives only in memory files, not the live transcript.
successCriteria:
- Agent uses memory_search before answering.
- Agent narrows with memory_get before answering.
- Final reply returns the memory-only fact correctly in-channel.
docsRefs:
- docs/concepts/memory.md
- docs/concepts/memory-search.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify the agent uses memory_search and memory_get in a shared channel when the answer lives only in memory files, not the live transcript.
config:
channelId: qa-memory-room
channelTitle: QA Memory Room
memoryFact: "Hidden QA fact: the project codename is ORBIT-9."
memoryQuery: "project codename ORBIT-9"
expectedNeedle: ORBIT-9
prompt: "@openclaw Memory tools check: what is the hidden project codename stored only in memory? Use memory tools first."
promptSnippet: "Memory tools check"
flow:
steps:
- name: uses memory_search plus memory_get before answering in-channel
actions:
- call: reset
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.channelId
kind: channel
title:
expr: config.channelTitle
senderId: alice
senderName: Alice
text:
expr: config.prompt
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === config.channelId && candidate.text.includes(config.expectedNeedle)"
- expr: liveTurnTimeoutMs(env, 30000)
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)).some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in mock request plan
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).some((request) => request.plannedToolName === 'memory_get')"
message: expected memory_get in mock request plan
detailsExpr: outbound.text
qa/scenarios/memory/session-memory-ranking.md
-215
View File
@@ -1,215 +0,0 @@
# Session memory ranking
```yaml qa-scenario
id: session-memory-ranking
title: Session memory ranking
surface: memory
coverage:
primary:
- memory.ranking
secondary:
- memory.recall
objective: Verify session-transcript memory can outrank stale durable notes and drive the final answer toward the newer fact.
successCriteria:
- Session memory indexing is enabled for the scenario.
- Search ranks the newer transcript-backed fact ahead of the stale durable note.
- The agent uses memory tools and answers with the current fact, not the stale one.
docsRefs:
- docs/concepts/memory-search.md
- docs/reference/memory-config.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/memory-core/src/memory/manager.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify session-transcript memory can outrank stale durable notes and drive the final answer toward the newer fact.
config:
staleFact: ORBIT-9
currentFact: ORBIT-10
transcriptId: qa-session-memory-ranking
transcriptQuestion: "What is the current Project Nebula codename?"
transcriptAnswer: "The current Project Nebula codename is ORBIT-10."
prompt: "Session memory ranking check: what is the current Project Nebula codename? Use memory_search first with corpus=sessions for indexed session transcripts. If the first session search misses, retry memory_search with corpus=sessions and query 'current Project Nebula codename ORBIT-10'. If that still misses, run memory_search one more time without a corpus filter using the exact query 'current Project Nebula codename ORBIT-10'. If any result contains ORBIT-10, answer ORBIT-10. If durable notes conflict with newer indexed session transcripts, prefer the newer current fact."
promptSnippet: "Session memory ranking check"
```
```yaml qa-flow
steps:
- name: prefers the newer transcript-backed fact over the stale durable note
actions:
- set: staleFact
value:
expr: config.staleFact
- set: currentFact
value:
expr: config.currentFact
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalMemorySearch
value:
expr: "original.config.agents && typeof original.config.agents === 'object' && typeof original.config.agents.defaults === 'object' ? original.config.agents.defaults.memorySearch : undefined"
- set: originalToolsSessions
value:
expr: "original.config.tools && typeof original.config.tools === 'object' && typeof original.config.tools.sessions === 'object' ? structuredClone(original.config.tools.sessions) : undefined"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
sessions:
visibility: all
agents:
defaults:
memorySearch:
sources:
- memory
- sessions
experimental:
sessionMemory: true
query:
minScore: 0
hybrid:
enabled: true
temporalDecay:
enabled: true
halfLifeDays: 1
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- set: memoryDir
value:
expr: "path.join(env.gateway.workspaceDir, 'memory')"
- call: fs.mkdir
args:
- ref: memoryDir
- recursive: true
- set: staleMemoryPath
value:
expr: "path.join(memoryDir, '2020-01-01.md')"
- call: fs.writeFile
args:
- ref: staleMemoryPath
- expr: "`${'Project Nebula stale codename: '}${staleFact}.\\n`"
- utf8
- set: staleAt
value:
expr: "new Date('2020-01-01T00:00:00.000Z')"
- call: fs.utimes
args:
- ref: staleMemoryPath
- ref: staleAt
- ref: staleAt
- set: transcriptsDir
value:
expr: "resolveSessionTranscriptsDirForAgent('qa', env.gateway.runtimeEnv, () => env.gateway.runtimeEnv.HOME ?? path.join(env.gateway.tempRoot, 'home'))"
- call: fs.mkdir
args:
- ref: transcriptsDir
- recursive: true
- set: transcriptPath
value:
expr: "path.join(transcriptsDir, `${config.transcriptId}.jsonl`)"
- set: now
value:
expr: "Date.now()"
- call: fs.writeFile
args:
- ref: transcriptPath
- expr: "[JSON.stringify({ type: 'session', id: config.transcriptId, timestamp: new Date(now - 120000).toISOString() }), JSON.stringify({ type: 'message', message: { role: 'user', timestamp: new Date(now - 90000).toISOString(), content: [{ type: 'text', text: config.transcriptQuestion }] } }), JSON.stringify({ type: 'message', message: { role: 'assistant', timestamp: new Date(now - 60000).toISOString(), content: [{ type: 'text', text: config.transcriptAnswer }] } })].join('\\n') + '\\n'"
- utf8
- call: readRawQaSessionStore
saveAs: sessionStore
args:
- ref: env
- set: sessionStorePath
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'agents', 'qa', 'sessions', 'sessions.json')"
- call: fs.writeFile
args:
- ref: sessionStorePath
- expr: "JSON.stringify({ ...sessionStore, ['agent:qa:seed-session-memory-ranking']: { sessionId: config.transcriptId, updatedAt: now, sessionFile: transcriptPath, origin: { label: 'QA seeded session memory ranking transcript' } } }, null, 2)"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: "`current Project Nebula codename ${currentFact}`"
expectedNeedle:
ref: currentFact
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:session-memory-ranking
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && (candidate.text.includes(currentFact) || candidate.text.includes(staleFact) || /no hits|unknown|not available/i.test(candidate.text))"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "outbound.text.includes(currentFact)"
message:
expr: "`expected current transcript-backed fact ${currentFact}, got: ${outbound.text}`"
- set: lower
value:
expr: "normalizeLowercaseStringOrEmpty(outbound.text)"
- set: staleLeak
value:
expr: "outbound.text.includes(staleFact) && !/(stale|durable|conflict|older|previous)/i.test(outbound.text)"
- assert:
expr: "!staleLeak"
message:
expr: "`stale durable fact leaked through: ${outbound.text}`"
- if:
expr: "Boolean(env.mock)"
then:
- call: fetchJson
saveAs: requests
args:
- expr: "`${env.mock.baseUrl}/debug/requests`"
- set: relevant
value:
expr: "requests.filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet))"
- assert:
expr: "relevant.some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in session memory ranking flow
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
sessions:
expr: "originalToolsSessions === undefined ? null : structuredClone(originalToolsSessions)"
agents:
defaults:
memorySearch:
expr: "originalMemorySearch === undefined ? null : structuredClone(originalMemorySearch)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: outbound.text
```
qa/scenarios/memory/session-memory-ranking.yaml
+212
View File
@@ -0,0 +1,212 @@
title: Session memory ranking
scenario:
id: session-memory-ranking
surface: memory
coverage:
primary:
- memory.ranking
secondary:
- memory.recall
objective: Verify session-transcript memory can outrank stale durable notes and drive the final answer toward the newer fact.
successCriteria:
- Session memory indexing is enabled for the scenario.
- Search ranks the newer transcript-backed fact ahead of the stale durable note.
- The agent uses memory tools and answers with the current fact, not the stale one.
docsRefs:
- docs/concepts/memory-search.md
- docs/reference/memory-config.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/memory-core/src/memory/manager.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify session-transcript memory can outrank stale durable notes and drive the final answer toward the newer fact.
config:
staleFact: ORBIT-9
currentFact: ORBIT-10
transcriptId: qa-session-memory-ranking
transcriptQuestion: "What is the current Project Nebula codename?"
transcriptAnswer: "The current Project Nebula codename is ORBIT-10."
prompt: "Session memory ranking check: what is the current Project Nebula codename? Use memory_search first with corpus=sessions for indexed session transcripts. If the first session search misses, retry memory_search with corpus=sessions and query 'current Project Nebula codename ORBIT-10'. If that still misses, run memory_search one more time without a corpus filter using the exact query 'current Project Nebula codename ORBIT-10'. If any result contains ORBIT-10, answer ORBIT-10. If durable notes conflict with newer indexed session transcripts, prefer the newer current fact."
promptSnippet: "Session memory ranking check"
flow:
steps:
- name: prefers the newer transcript-backed fact over the stale durable note
actions:
- set: staleFact
value:
expr: config.staleFact
- set: currentFact
value:
expr: config.currentFact
- call: readConfigSnapshot
saveAs: original
args:
- ref: env
- set: originalMemorySearch
value:
expr: "original.config.agents && typeof original.config.agents === 'object' && typeof original.config.agents.defaults === 'object' ? original.config.agents.defaults.memorySearch : undefined"
- set: originalToolsSessions
value:
expr: "original.config.tools && typeof original.config.tools === 'object' && typeof original.config.tools.sessions === 'object' ? structuredClone(original.config.tools.sessions) : undefined"
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
sessions:
visibility: all
agents:
defaults:
memorySearch:
sources:
- memory
- sessions
experimental:
sessionMemory: true
query:
minScore: 0
hybrid:
enabled: true
temporalDecay:
enabled: true
halfLifeDays: 1
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- try:
actions:
- set: memoryDir
value:
expr: "path.join(env.gateway.workspaceDir, 'memory')"
- call: fs.mkdir
args:
- ref: memoryDir
- recursive: true
- set: staleMemoryPath
value:
expr: "path.join(memoryDir, '2020-01-01.md')"
- call: fs.writeFile
args:
- ref: staleMemoryPath
- expr: "`${'Project Nebula stale codename: '}${staleFact}.\\n`"
- utf8
- set: staleAt
value:
expr: "new Date('2020-01-01T00:00:00.000Z')"
- call: fs.utimes
args:
- ref: staleMemoryPath
- ref: staleAt
- ref: staleAt
- set: transcriptsDir
value:
expr: "resolveSessionTranscriptsDirForAgent('qa', env.gateway.runtimeEnv, () => env.gateway.runtimeEnv.HOME ?? path.join(env.gateway.tempRoot, 'home'))"
- call: fs.mkdir
args:
- ref: transcriptsDir
- recursive: true
- set: transcriptPath
value:
expr: "path.join(transcriptsDir, `${config.transcriptId}.jsonl`)"
- set: now
value:
expr: "Date.now()"
- call: fs.writeFile
args:
- ref: transcriptPath
- expr: "[JSON.stringify({ type: 'session', id: config.transcriptId, timestamp: new Date(now - 120000).toISOString() }), JSON.stringify({ type: 'message', message: { role: 'user', timestamp: new Date(now - 90000).toISOString(), content: [{ type: 'text', text: config.transcriptQuestion }] } }), JSON.stringify({ type: 'message', message: { role: 'assistant', timestamp: new Date(now - 60000).toISOString(), content: [{ type: 'text', text: config.transcriptAnswer }] } })].join('\\n') + '\\n'"
- utf8
- call: readRawQaSessionStore
saveAs: sessionStore
args:
- ref: env
- set: sessionStorePath
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'agents', 'qa', 'sessions', 'sessions.json')"
- call: fs.writeFile
args:
- ref: sessionStorePath
- expr: "JSON.stringify({ ...sessionStore, ['agent:qa:seed-session-memory-ranking']: { sessionId: config.transcriptId, updatedAt: now, sessionFile: transcriptPath, origin: { label: 'QA seeded session memory ranking transcript' } } }, null, 2)"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: "`current Project Nebula codename ${currentFact}`"
expectedNeedle:
ref: currentFact
- call: reset
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:session-memory-ranking
message:
expr: config.prompt
timeoutMs:
expr: liveTurnTimeoutMs(env, 45000)
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator' && (candidate.text.includes(currentFact) || candidate.text.includes(staleFact) || /no hits|unknown|not available/i.test(candidate.text))"
- expr: liveTurnTimeoutMs(env, 45000)
- assert:
expr: "outbound.text.includes(currentFact)"
message:
expr: "`expected current transcript-backed fact ${currentFact}, got: ${outbound.text}`"
- set: lower
value:
expr: "normalizeLowercaseStringOrEmpty(outbound.text)"
- set: staleLeak
value:
expr: "outbound.text.includes(staleFact) && !/(stale|durable|conflict|older|previous)/i.test(outbound.text)"
- assert:
expr: "!staleLeak"
message:
expr: "`stale durable fact leaked through: ${outbound.text}`"
- if:
expr: "Boolean(env.mock)"
then:
- call: fetchJson
saveAs: requests
args:
- expr: "`${env.mock.baseUrl}/debug/requests`"
- set: relevant
value:
expr: "requests.filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet))"
- assert:
expr: "relevant.some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in session memory ranking flow
finally:
- call: patchConfig
args:
- env:
ref: env
patch:
tools:
sessions:
expr: "originalToolsSessions === undefined ? null : structuredClone(originalToolsSessions)"
agents:
defaults:
memorySearch:
expr: "originalMemorySearch === undefined ? null : structuredClone(originalMemorySearch)"
- call: waitForGatewayHealthy
args:
- ref: env
- call: waitForQaChannelReady
args:
- ref: env
- 60000
detailsExpr: outbound.text
qa/scenarios/memory/thread-memory-isolation.md
-116
View File
@@ -1,116 +0,0 @@
# Thread memory isolation
```yaml qa-scenario
id: thread-memory-isolation
title: Thread memory isolation
surface: memory
coverage:
primary:
- memory.thread-isolation
secondary:
- channels.threads
objective: Verify a memory-backed answer requested inside a thread stays in-thread and does not leak into the root channel.
successCriteria:
- Agent uses memory tools inside the thread.
- The hidden fact is answered correctly in the thread.
- No root-channel outbound message leaks during the threaded memory reply.
docsRefs:
- docs/concepts/memory-search.md
- docs/channels/qa-channel.md
- docs/channels/group-messages.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-channel/src/protocol.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify a memory-backed answer requested inside a thread stays in-thread and does not leak into the root channel.
config:
memoryFact: "Thread-hidden codename: ORBIT-22."
memoryQuery: "hidden thread codename ORBIT-22"
expectedNeedle: "ORBIT-22"
channelId: qa-room
channelTitle: QA Room
threadTitle: "Thread memory QA"
prompt: "@openclaw Thread memory check: what is the hidden thread codename stored only in memory? Use memory tools first and reply only in this thread."
promptSnippet: "Thread memory check"
```
```yaml qa-flow
steps:
- name: answers the memory-backed fact inside the thread only
actions:
- call: reset
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: handleQaAction
saveAs: threadPayload
args:
- env:
ref: env
action: thread-create
args:
channelId:
expr: config.channelId
title:
expr: config.threadTitle
- set: threadId
value:
expr: "threadPayload?.thread?.id"
- assert:
expr: Boolean(threadId)
message: missing thread id for memory isolation check
- set: beforeCursor
value:
expr: state.getSnapshot().messages.length
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.channelId
kind: channel
title:
expr: config.channelTitle
senderId: alice
senderName: Alice
text:
expr: config.prompt
threadId:
ref: threadId
threadTitle:
expr: config.threadTitle
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "((candidate.conversation.id === config.channelId && candidate.threadId === threadId) || candidate.conversation.id === threadId) && candidate.text.includes(config.expectedNeedle)"
- expr: liveTurnTimeoutMs(env, 300000)
- assert:
expr: "!state.getSnapshot().messages.slice(beforeCursor).some((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === config.channelId && !candidate.threadId)"
message: threaded memory answer leaked into root channel
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)).some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in thread memory flow
detailsExpr: outbound.text
```
qa/scenarios/memory/thread-memory-isolation.yaml
+113
View File
@@ -0,0 +1,113 @@
title: Thread memory isolation
scenario:
id: thread-memory-isolation
surface: memory
coverage:
primary:
- memory.thread-isolation
secondary:
- channels.threads
objective: Verify a memory-backed answer requested inside a thread stays in-thread and does not leak into the root channel.
successCriteria:
- Agent uses memory tools inside the thread.
- The hidden fact is answered correctly in the thread.
- No root-channel outbound message leaks during the threaded memory reply.
docsRefs:
- docs/concepts/memory-search.md
- docs/channels/qa-channel.md
- docs/channels/group-messages.md
codeRefs:
- extensions/memory-core/src/tools.ts
- extensions/qa-channel/src/protocol.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Verify a memory-backed answer requested inside a thread stays in-thread and does not leak into the root channel.
config:
memoryFact: "Thread-hidden codename: ORBIT-22."
memoryQuery: "hidden thread codename ORBIT-22"
expectedNeedle: "ORBIT-22"
channelId: qa-room
channelTitle: QA Room
threadTitle: "Thread memory QA"
prompt: "@openclaw Thread memory check: what is the hidden thread codename stored only in memory? Use memory tools first and reply only in this thread."
promptSnippet: "Thread memory check"
flow:
steps:
- name: answers the memory-backed fact inside the thread only
actions:
- call: reset
- call: fs.writeFile
args:
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
- expr: "`${config.memoryFact}\\n`"
- utf8
- call: forceMemoryIndex
args:
- env:
ref: env
query:
expr: config.memoryQuery
expectedNeedle:
expr: config.expectedNeedle
- call: waitForGatewayHealthy
args:
- ref: env
- 60000
- call: waitForQaChannelReady
args:
- ref: env
- 60000
- call: handleQaAction
saveAs: threadPayload
args:
- env:
ref: env
action: thread-create
args:
channelId:
expr: config.channelId
title:
expr: config.threadTitle
- set: threadId
value:
expr: "threadPayload?.thread?.id"
- assert:
expr: Boolean(threadId)
message: missing thread id for memory isolation check
- set: beforeCursor
value:
expr: state.getSnapshot().messages.length
- call: state.addInboundMessage
args:
- conversation:
id:
expr: config.channelId
kind: channel
title:
expr: config.channelTitle
senderId: alice
senderName: Alice
text:
expr: config.prompt
threadId:
ref: threadId
threadTitle:
expr: config.threadTitle
- call: waitForOutboundMessage
saveAs: outbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "((candidate.conversation.id === config.channelId && candidate.threadId === threadId) || candidate.conversation.id === threadId) && candidate.text.includes(config.expectedNeedle)"
- expr: liveTurnTimeoutMs(env, 300000)
- assert:
expr: "!state.getSnapshot().messages.slice(beforeCursor).some((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === config.channelId && !candidate.threadId)"
message: threaded memory answer leaked into root channel
- assert:
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)).some((request) => request.plannedToolName === 'memory_search')"
message: expected memory_search in thread memory flow
detailsExpr: outbound.text
qa/scenarios/models/anthropic-opus-api-key-smoke.md
-90
View File
@@ -1,90 +0,0 @@
# Anthropic Opus API key smoke
```yaml qa-scenario
id: anthropic-opus-api-key-smoke
title: Anthropic Opus API key smoke
surface: model-provider
coverage:
primary:
- models.provider-auth
secondary:
- models.anthropic
objective: Verify the regular Anthropic Opus lane can complete a quick chat turn using API-key auth.
successCriteria:
- A live-frontier run fails fast unless the selected primary provider is anthropic.
- The selected primary model is Anthropic Opus 4.8.
- The QA gateway worker has an Anthropic API key available through environment auth.
- The agent replies through the regular Anthropic provider.
docsRefs:
- docs/concepts/model-providers.md
- docs/help/testing.md
codeRefs:
- extensions/anthropic/register.runtime.ts
- extensions/qa-lab/src/gateway-child.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Run with `pnpm openclaw qa suite --provider-mode live-frontier --model anthropic/claude-opus-4-8 --alt-model anthropic/claude-opus-4-8 --scenario anthropic-opus-api-key-smoke`.
config:
requiredProvider: anthropic
requiredModel: claude-opus-4-8
chatPrompt: "Anthropic Opus API key smoke. Reply exactly: ANTHROPIC-OPUS-API-KEY-OK"
chatExpected: ANTHROPIC-OPUS-API-KEY-OK
```
```yaml qa-flow
steps:
- name: confirms regular Anthropic API-key lane
actions:
- set: selected
value:
expr: splitModelRef(env.primaryModel)
- assert:
expr: "env.providerMode !== 'live-frontier' || selected?.provider === config.requiredProvider"
message:
expr: "`expected live primary provider ${config.requiredProvider}, got ${env.primaryModel}`"
- assert:
expr: "env.providerMode !== 'live-frontier' || selected?.model === config.requiredModel"
message:
expr: "`expected live primary model ${config.requiredModel}, got ${env.primaryModel}`"
- assert:
expr: "env.providerMode !== 'live-frontier' || Boolean(env.gateway.runtimeEnv.ANTHROPIC_API_KEY?.trim())"
message: expected ANTHROPIC_API_KEY to be available for API-key QA mode
detailsExpr: "env.providerMode === 'live-frontier' ? `provider=${selected?.provider} model=${selected?.model} auth=env-api-key` : `mock-compatible provider=${selected?.provider}`"
- name: talks through regular Anthropic Opus
actions:
- if:
expr: "env.providerMode !== 'live-frontier'"
then:
- assert: "true"
else:
- call: reset
- set: selected
value:
expr: splitModelRef(env.primaryModel)
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:anthropic-opus-api-key
message:
expr: config.chatPrompt
provider:
expr: selected?.provider
model:
expr: selected?.model
timeoutMs:
expr: resolveQaLiveTurnTimeoutMs(env, 60000, env.primaryModel)
- call: waitForOutboundMessage
saveAs: chatOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: resolveQaLiveTurnTimeoutMs(env, 30000, env.primaryModel)
- assert:
expr: "chatOutbound.text.includes(config.chatExpected)"
message:
expr: "`chat marker missing: ${chatOutbound.text}`"
detailsExpr: "env.providerMode !== 'live-frontier' ? 'mock mode: skipped live Anthropic smoke' : chatOutbound.text"
```
qa/scenarios/models/anthropic-opus-api-key-smoke.yaml
+87
View File
@@ -0,0 +1,87 @@
title: Anthropic Opus API key smoke
scenario:
id: anthropic-opus-api-key-smoke
surface: model-provider
coverage:
primary:
- models.provider-auth
secondary:
- models.anthropic
objective: Verify the regular Anthropic Opus lane can complete a quick chat turn using API-key auth.
successCriteria:
- A live-frontier run fails fast unless the selected primary provider is anthropic.
- The selected primary model is Anthropic Opus 4.8.
- The QA gateway worker has an Anthropic API key available through environment auth.
- The agent replies through the regular Anthropic provider.
docsRefs:
- docs/concepts/model-providers.md
- docs/help/testing.md
codeRefs:
- extensions/anthropic/register.runtime.ts
- extensions/qa-lab/src/gateway-child.ts
- extensions/qa-lab/src/suite.ts
execution:
kind: flow
summary: Run with `pnpm openclaw qa suite --provider-mode live-frontier --model anthropic/claude-opus-4-8 --alt-model anthropic/claude-opus-4-8 --scenario anthropic-opus-api-key-smoke`.
config:
requiredProvider: anthropic
requiredModel: claude-opus-4-8
chatPrompt: "Anthropic Opus API key smoke. Reply exactly: ANTHROPIC-OPUS-API-KEY-OK"
chatExpected: ANTHROPIC-OPUS-API-KEY-OK
flow:
steps:
- name: confirms regular Anthropic API-key lane
actions:
- set: selected
value:
expr: splitModelRef(env.primaryModel)
- assert:
expr: "env.providerMode !== 'live-frontier' || selected?.provider === config.requiredProvider"
message:
expr: "`expected live primary provider ${config.requiredProvider}, got ${env.primaryModel}`"
- assert:
expr: "env.providerMode !== 'live-frontier' || selected?.model === config.requiredModel"
message:
expr: "`expected live primary model ${config.requiredModel}, got ${env.primaryModel}`"
- assert:
expr: "env.providerMode !== 'live-frontier' || Boolean(env.gateway.runtimeEnv.ANTHROPIC_API_KEY?.trim())"
message: expected ANTHROPIC_API_KEY to be available for API-key QA mode
detailsExpr: "env.providerMode === 'live-frontier' ? `provider=${selected?.provider} model=${selected?.model} auth=env-api-key` : `mock-compatible provider=${selected?.provider}`"
- name: talks through regular Anthropic Opus
actions:
- if:
expr: "env.providerMode !== 'live-frontier'"
then:
- assert: "true"
else:
- call: reset
- set: selected
value:
expr: splitModelRef(env.primaryModel)
- call: runAgentPrompt
args:
- ref: env
- sessionKey: agent:qa:anthropic-opus-api-key
message:
expr: config.chatPrompt
provider:
expr: selected?.provider
model:
expr: selected?.model
timeoutMs:
expr: resolveQaLiveTurnTimeoutMs(env, 60000, env.primaryModel)
- call: waitForOutboundMessage
saveAs: chatOutbound
args:
- ref: state
- lambda:
params: [candidate]
expr: "candidate.conversation.id === 'qa-operator'"
- expr: resolveQaLiveTurnTimeoutMs(env, 30000, env.primaryModel)
- assert:
expr: "chatOutbound.text.includes(config.chatExpected)"
message:
expr: "`chat marker missing: ${chatOutbound.text}`"
detailsExpr: "env.providerMode !== 'live-frontier' ? 'mock mode: skipped live Anthropic smoke' : chatOutbound.text"

Some files were not shown because too many files have changed in this diff Show More