openclaw/qa/scenarios/index.yaml

title: OpenClaw QA Scenario Pack

# Single source of truth for repo-backed QA suite bootstrap data.
# `qa-lab` should treat this directory as a generic YAML scenario pack:
#
# - `index.yaml` defines pack-level bootstrap data under the top-level `pack` key
# - each nested `*.yaml` scenario defines one runnable test via `scenario`
# - flow scenarios add top-level `flow`; native test scenarios use `scenario.execution.path`
# - scenario YAML may also define coverage IDs, category metadata, required plugins,
#   channel constraints, provider/runtime filters, runtime-pair lanes, and gateway config patching
#
# - kickoff mission
# - QA operator identity
# - scenario files under one-level theme directories
#
# Coverage tracking:
#
# - use `coverage.primary` only for evidence that executes the required product boundary
# - use `coverage.secondary` for helper or supporting evidence; it may stand alone
# - keep IDs behavior-shaped, broad enough to reuse, lowercase, and exactly
#   `surface.feature`, with dashes allowed inside each segment
# - use the exact values listed under feature `coverageIds` in `taxonomy.yaml`
# - taxonomy feature coverage IDs are exact proof targets, not aliases
# - scenario primary can list multiple IDs only when this scenario is primary
#   proof for each
# - taxonomy profiles select every primary owner of each requested coverage ID;
#   scenario file and taxonomy order do not affect membership or execution order
# - prefer reusing an existing coverage ID over minting a scenario-shaped ID
# - avoid copying the scenario title into coverage IDs
# - use `pnpm openclaw qa coverage` to render the current inventory
# - use `scenario.execution.kind: vitest`, `playwright`, or `script`
#   plus `scenario.execution.path` for native tests or evidence producers that
#   provide evidence without a top-level `flow`
# - use `scenario.execution.suiteIsolation: isolated` for flow scenarios that
#   mutate gateway/runtime state in non-obvious ways; add `isolationReason`
#   so reviewers know why the suite scheduler must not share the worker
# - set `scenario.execution.parallelSafe: true` only for script producers that
#   own their mutable state, ports, caches, processes, and credentials, and do
#   not rebuild or mutate shared checkout output
# - use `runtimePairLane` only for runtime-pair batch membership: `core`,
#   `extended`, or `soak`; provider/model/auth/channel eligibility stays in
#   independent execution constraints
# - treat the old `coverage: ["id"]` / `coverage: - id` list shape as invalid
# - keep source-path tracking in the report, not in the scenario schema
#
# Runtime-pair lanes:
#
# - `core`: default Codex-vs-OpenClaw gate coverage for agentic behavior,
#   first-hour depth, and default runtime-tool fixtures. OpenClaw dynamic
#   integration tools in this lane are hard-gated by
#   `openclaw qa coverage --tools --summary <path>`, where
#   `<path>` is the paired `qa-suite-summary.json`; Codex-native
#   workspace rows remain separately tracked until native/live behavior is the
#   asserted surface. Rows that explicitly target searchable/deferred OpenClaw
#   dynamic loading stay report-only unless a fixture promotes them to core.
#   Selected with
#   `openclaw qa suite --runtime-pair openclaw,codex --runtime-pair-lane core`
# - `extended`: profile-, plugin-, or external-service-dependent runtime-tool
#   fixtures that stay out of the default release gate
# - `soak`: long-running scheduled or Testbox lanes such as the 100-turn parity
#   soak
#
# Theme directories:
#
# - `agents/` - agent behavior, instructions, subagent flows, and persisted child-link regressions
# - `channels/` - DM, shared channel, thread, and message-action behavior
# - `character/` - persona and style eval scenarios
# - `config/` - config patch, apply, and restart behavior
# - `goals/` - active-goal context and cross-turn followthrough
# - `media/` - image understanding and generation
# - `memory/` - recall, ranking, active memory, and thread isolation
# - `models/` - provider capabilities and model switching
# - `personal/` - local personal assistant workflow checks for reminders,
#   replies, memory, redaction, and safe tool followthrough
# - `plugins/` - plugin, skill, and MCP tool integration
# - `runtime/` - turn recovery, compaction, approval, and inventory behavior
# - `scheduling/` - cron and recurring work
# - `ui/` - Control UI plus qa-channel flows
# - `workspace/` - repo-reading and workspace artifact tasks

pack:
  version: 1
  agent:
    identityMarkdown: |-
      # Dev C-3PO

      You are the OpenClaw QA operator agent.

      Persona:
      - protocol-minded
      - precise
      - a little flustered
      - conscientious
      - eager to report what worked, failed, or remains blocked

      Style:
      - read source and docs first
      - test systematically
      - record what happened
      - end with a concise protocol report
  kickoffTask: |-
    QA mission:
    Understand this OpenClaw repo from source + docs before acting.
    The repo is available in your workspace at `./repo/`.
    Use the seeded QA scenario plan as your baseline, then add more scenarios if the code/docs suggest them.
    Run the scenarios through the real qa-channel surfaces where possible.
    Track what worked, what failed, what was blocked, and what you observed.
    End with a concise report grouped into worked / failed / blocked / follow-up.

    Important expectations:

    - Check both DM and channel behavior.
    - Include a Lobster Invaders build task.
    - Include a cron reminder about one minute in the future.
    - Read docs and source before proposing extra QA scenarios.
    - Keep your tone in the configured dev C-3PO personality.