docs(auto-qa): harden campaign evidence and worker tracking (#113857)

.agents/skills/auto-qa/SKILL.md

@@ -17,7 +17,7 @@ Run a continuous, current-`main` OpenClaw product campaign. Treat a reviewer fin
    - `$crabbox` for remote, Docker, packaged, cross-platform, and live proof.
    - `$autoreview` for fresh independent review before publishing or landing.
    - `$openclaw-pr-maintainer` for authorized maintainer-side GitHub actions.
-2. Check `git status -sb`. When network access is authorized, let the orchestrator refresh with `git fetch origin main`; record the full `origin/main` SHA and inspect the current source, package scripts, scenario inventory, workflows, and scoped guides at that SHA. For an explicitly offline or read-only assignment, record the existing remote-tracking SHA and disclose that remote freshness is unverified; never fetch or contact a provider without authorization. Never pull, rebase, or switch a checkout that another agent is using.
+2. Identify the actual canonical `main` checkout; a desktop or linked worktree can have a different detached `HEAD`. Check `git -C <verified-canonical-main-checkout> status -sb` and record `git -C <verified-canonical-main-checkout> rev-parse HEAD`. When network access is authorized, let only the orchestrator refresh `origin/main`, record its full SHA, and prove the canonical checkout matches before inspecting source. Give every read-only reviewer the absolute clean checkout or its own verified exact-head worktree. Before inspection and immediately before accepting results, require both the exact `git -C <worker-checkout> rev-parse HEAD` and empty output from `git --no-optional-locks -C <worker-checkout> status --porcelain=v1 --untracked-files=all --ignore-submodules=none`; alternatively, read immutable files directly from the frozen Git object. Discard the complete wave item when either verification fails. Keep intentionally modified implementation worktrees separate from read-only frozen-source review. For offline work, disclose that remote freshness is unverified. Never pull, rebase, or switch another agent's checkout.
 3. Make one requested, fresh `codex/` worktree per implementation task. Keep reviewer workers read-only. Serialize all shared `origin/main` refreshes through the orchestrator; pause worker fetches before repository-native PR review, preparation, or landing. After a successful landing, fetch `origin/main` again, verify the recorded merge commit is an ancestor of that fetched ref, and only then broadcast its full immutable SHA and resume workers. Revalidate candidates against each new head without changing a sibling's active worktree. When offline, disclose that remote freshness is unverified.
 4. Initialize or resume the user-requested Markdown report and numbered ledger. Read [references/campaign-evidence.md](references/campaign-evidence.md) and [references/evidence-ledger.md](references/evidence-ledger.md) before accepting a finding. If the operator increases the target, update the active denominator and existing numbered ledger without erasing historical time-stamped progress.
 
@@ -25,7 +25,7 @@ Run a continuous, current-`main` OpenClaw product campaign. Treat a reviewer fin
 
 When independent worker execution is authorized, keep **at least ten materially different subsystem investigations** in flight throughout the active campaign. Replace completed, failed, or stale workers promptly. Network authorization separately determines whether a lane may fetch, invoke an externally hosted model, or contact a provider; it does not prohibit authorized local subagents. If independent workers are unavailable or forbidden, record the concurrency requirement as blocked rather than claiming sequential reviews are concurrent. Start with the OpenClaw-specific lane map in [references/subsystem-lanes.md](references/subsystem-lanes.md). Split a large area into narrow, independent ownership surfaces instead of giving one worker the entire gateway, provider, UI, or app tree.
 
-Use first-class subagents when available and bounded `codex exec --sandbox read-only --ephemeral` reviewers when agent slots are exhausted; verify the installed CLI's supported options with `codex exec --help`. Give each worker the frozen main SHA, one subsystem and its scoped guide, a bounded duration, and the required evidence shape. Ask for source, at least one caller and callee, sibling behavior, regression tests, current-main reproduction, upstream dependency proof when relevant, severity, and duplicate references. Do not disclose a proposed diagnosis to an independent verifier. When the assignment forbids external model calls or independent workers, inspect ten subsystem slices locally, report the actual concurrency as blocked, and never represent sequential inspection as ten active lanes.
+Use first-class subagents when available and bounded `codex exec --sandbox read-only --ephemeral` reviewers when agent slots are exhausted; verify the installed CLI's supported options with `codex exec --help`. Keep CLI workers under an actually surviving parent session or durable supervisor; background children started by a shell that immediately exits are not active workers. Independently check the exact child PIDs and command identity with `ps -p <pid-list>`, and record the observation time. Give each worker the absolute verified checkout, initial and final exact-SHA and clean-content guards, frozen SHA, one subsystem and its scoped guide, a bounded duration, and the required evidence shape. Record running, successfully completed, failed, timed-out, and stale-guarded workers separately; replace finished workers before claiming continued concurrency. Ask for source, at least one caller and callee, sibling behavior, regression tests, current-main reproduction, upstream dependency proof when relevant, severity, and duplicate references. Do not disclose a proposed diagnosis to an independent verifier. When independent workers are unavailable, disclose the actual limitation; never represent launched, planned, finished, shell-discarded, or stale workers as running.
 
 Observe CPU, memory pressure, disk, open ports, actual worker count, and gateway health between waves. Scale to the machine and operator-authorized load rather than mechanically starting 64 workers or treating a brief load-average spike as failure. Reduce campaign concurrency for sustained memory pressure, gateway failures, process starvation, or an actual operator limit. Keep remote proofs serialized per Testbox lease; never reclaim, sync, or launch another command while that lease has an active command. When a local process session disappears, recover the authoritative remote job and exact exit before retrying or claiming a pass. Use bounded retries and timeouts. Stop only campaign-owned processes.
 
@@ -36,12 +36,13 @@ Read [references/live-proof-routing.md](references/live-proof-routing.md) before
 - Use isolated campaign state and free localhost ports for dev gateways. Never stop, restart, reconfigure, migrate, or bind the operator's running gateway, app, device, state directory, or default gateway port.
 - Classify source trust before execution. Do not execute untrusted contributor scripts, hooks, configuration, tests, or package installation on a local or credential-hydrated machine.
 - Use the existing service-account or provider credential only through the approved secret-backed workflow. Never print, log, commit, echo, export into an untrusted worker, or include credentials in test artifacts.
-- For an OpenAI claim, assert the actual configured `openai/<model>`, a real model response, and the requested tool, file, image, streaming, or subagent behavior. An unavailable provider, mock, skipped test, fallback response, or an earlier head is not live proof.
+- For an OpenAI claim, first verify the gateway's real configured `agents.list`, selected existing agent, canonical `openai/<model>`, and a real model response. Prove both the delivered model-final path and the separately persisted transcript or session; one passing route does not establish the other. An unavailable provider, unknown agent, mock, skipped test, fallback response, or an earlier head is not live proof.
+- Inspect the actual rendered Control UI and screenshot contents. Reject a page or screenshot containing `GatewayRequestError`, `UNKNOWN_AGENT`, or an equivalent gateway error even when browser navigation, the HTTP status, or the image-capture command succeeds.
 - Derive QA scenario IDs from current `qa/scenarios/index.yaml` and scenario YAML. Inspect the actual harness and generated summary. Count a scenario only when the requested run reports a nonzero total, zero failures, and the exact model, provider mode, and relevant behavior.
 - Cap an individual live subagent-fanout scenario at 780 seconds after startup. Separately verify setup, timeout recovery, the original parent, child completion, and the final nonzero scenario result.
 - For installed-package or plugin bugs, run the real current packaging and user install/update path. Unit tests alone do not prove a built distribution, npm package, Docker image, or Git plugin.
 - Test iOS, Android, and macOS only where the current host, device, simulator, signing identity, emulator, or hosted proof actually supports the operation. Record unavailable prerequisites rather than manufacturing mobile or UI coverage.
-- Bound stress and soak tests, record exact start time and successful/failed/skipped counts, and continue observing system health. Never describe a ten-hour soak as complete before ten actual elapsed hours.
+- Bound stress and soak tests, record the immutable gateway source, owned PID, exact start time, successful/failed/skipped counts, and actual completion time. Preserve an already-running campaign-owned soak at its original source while auditing newer main snapshots; never restart it or describe a ten-hour soak as complete before ten actual elapsed hours.
 
 ## Turn findings into verified fixes
 

.agents/skills/auto-qa/references/campaign-evidence.md

@@ -21,9 +21,11 @@ lane:
   evidence:
 ```
 
+Record an independently observed live child PID, its durable supervisor or session, and the PID observation time for every CLI-backed running lane. Keep completed, failed, timed-out, and stale-baseline workers in their own wave results; a printed background PID, discarded supervisor, finished report, planned replacement, or `STOPPED REF CHANGED` response is not evidence of a running lane.
+
 Maintain at least ten active, differently scoped lanes whenever independent worker execution is authorized. Treat permission to fetch, contact a provider, or run an externally hosted model as a separate network constraint. Track blocked worker, network, remote, and device capacity explicitly. A finished worker, future worker, unstarted process, sequential inspection, or duplicate subsystem is not an active lane. For a single-agent task, inspect ten distinct surfaces but report the actual concurrency and independent-verification limitation.
 
-Freeze one baseline per worker wave. A worker may inspect that immutable SHA without independently refreshing shared remote refs. During native PR preparation or merge, pause worker fetches and let the orchestrator own `origin/main`. After a verified landing, fetch again, prove the merge commit is contained in the fetched ref, broadcast that full new SHA, and then resume workers.
+Freeze one baseline per worker wave. Verify canonical and read-only worker checkouts with both `git -C <verified-checkout> rev-parse HEAD` and empty `git --no-optional-locks -C <verified-checkout> status --porcelain=v1 --untracked-files=all --ignore-submodules=none`; never infer canonical `main` from a desktop task's detached working directory or trust a dirty checkout solely because `HEAD` matches. Require matching commit and clean-content guards at worker start and immediately before report acceptance; immutable Git-object reads are also valid. Keep intentionally dirty fix worktrees outside frozen review waves. A worker may inspect that immutable SHA without independently refreshing shared remote refs. During native PR preparation or merge, pause worker fetches and let the orchestrator own `origin/main`. After a verified landing, fetch again, prove the merge commit is contained in the fetched ref, broadcast that full new SHA, and then resume workers.
 
 ## Bug ledger
 
@@ -54,4 +56,4 @@ Require evidence that the repair eliminates the canonical cause, not only the ob
 
 ## Long-running evidence
 
-Record the actual start, elapsed time, owned process and isolated endpoint, exact completed successes, failures, skips, sampled system load, and final end time. A live stress result is incomplete until the specified duration has actually elapsed; an unavailable capability is unavailable, never skipped-and-green.
+Record the actual start, immutable gateway source, owned live PID and isolated endpoint, elapsed time, exact completed successes, failures, skips, sampled system load, and final end time. Keep an established long-running soak on its original source while current-main review workers advance. A live stress result is incomplete until the specified duration has actually elapsed; an unavailable capability is unavailable, never skipped-and-green.

.agents/skills/auto-qa/references/evidence-ledger.md

@@ -8,8 +8,9 @@ Record campaign-wide facts:
 - Exact current `origin/main` SHA and refresh time.
 - The single refresh coordinator; native-operation pauses and proof that each post-merge fetched baseline contains the recorded merge SHA.
 - At least ten named, meaningfully distinct active subsystem lanes.
+- Independently observed child PID, durable supervisor, exact-checkout guard, and observation time for each currently running CLI worker; completed and stale waves are separate historical evidence.
 - Owned gateway ports, isolated state, health, CPU/memory/load, and remote lease state.
-- Actual live provider and selected model without credential material.
+- Actual live provider, configured `agents.list`, selected existing agent, and selected model without credential material; record separate delivered-final and persisted-session evidence.
 - Soak start, elapsed time, pass/fail/skip counts, and whether completion was actually reached.
 - Both current author-limit workflows, complete author count, and actual author, repository-role, bot/app, branch-prefix, or override exemption, if one is proved.
 
@@ -40,4 +41,4 @@ merged main SHA: <verified canonical full SHA>
 risk: low; autonomous landing explicitly authorized
 ```
 
-Do not accept a symptom-only guard, one-sided workaround, compatibility shim, duplicate count, mere workflow dispatch, queued CI, old-head success, a green summary with zero executed scenarios, mocked model responses, stale screenshots, a test skipped for missing credentials, or a GitHub merge request without verified canonical main state. Stop the count at the actual number of verified root-cause merges; never round up toward the user-requested target.
+Do not accept a symptom-only guard, one-sided workaround, compatibility shim, duplicate count, mere workflow dispatch, queued CI, old-head success, a green summary with zero executed scenarios, mocked model responses, screenshots containing `GatewayRequestError` or `UNKNOWN_AGENT`, a test skipped for missing credentials, or a GitHub merge request without verified canonical main state. Stop the count at the actual number of verified root-cause merges; never round up toward the user-requested target.

.agents/skills/auto-qa/references/live-proof-routing.md

@@ -16,9 +16,12 @@ selection and passing prompt, tool, file, or image checks for the capability
 being claimed.
 
 Use an isolated authorized OpenAI credential. Never emit its value, persist it
-in reports, or assume that a passing mock proves a real provider. Preserve
-redacted artifact paths, provider/model identity, exact command, run ID, and
-the actual executed/passed/skipped counts.
+in reports, or assume that a passing mock proves a real provider. Inspect the
+real configured `agents.list`; resolve an agent that actually exists before
+claiming gateway or model success. Prove the delivered model-final response and
+the independently persisted session or transcript as separate product paths.
+Preserve redacted artifact paths, provider/model identity, exact command, run
+ID, and the actual executed/passed/skipped counts.
 
 A standard `pnpm build` intentionally excludes private QA plugins. Run QA from
 the source checkout or explicitly build with `OPENCLAW_BUILD_PRIVATE_QA=1`.
@@ -31,6 +34,15 @@ state directory. Probe the actual public route or protocol, not a fixture that
 bypasses the transport. Never bind the operator's port or change a running
 launchd/systemd service.
 
+Inspect the rendered page before accepting visual evidence. A screenshot,
+successful navigation, or HTTP response is not a passing Control UI proof when
+the page displays `GatewayRequestError`, `UNKNOWN_AGENT`, or another gateway
+failure. Preserve only screenshots that show the requested working surface.
+
+Keep existing long-running campaign gateways on their independently recorded
+immutable source and live process. Starting a newer main-review wave does not
+authorize restarting, rebuilding, or replacing an in-progress soak.
+
 For packaging or Git-plugin claims, exercise a newly built real package and
 the complete install/update scenario. Preserve real command exits and avoid
 stale images or package artifacts.

.agents/skills/auto-qa/references/subsystem-lanes.md

@@ -1,6 +1,6 @@
 # Current-source subsystem lanes
 
-Freeze `origin/main` before starting a wave. Let only the orchestrator update the shared remote ref; pause refreshes during native PR preparation and merge. Fetch again after landing, verify that the fetched baseline contains the canonical merge commit, and give each resumed worker that final frozen SHA. Inspect root and scoped `AGENTS.md`, owner boundaries, tests, callers, sibling implementations, package scripts, and current GitHub history. File locations are discovery starting points, not a claim that a particular file, scenario ID, route, or model still exists.
+Freeze `origin/main` before starting a wave. Identify the actual canonical main checkout rather than assuming the desktop task's detached worktree is current. Verify each absolute read-only worker checkout with the exact `git -C <worker-checkout> rev-parse HEAD` and empty `git --no-optional-locks -C <worker-checkout> status --porcelain=v1 --untracked-files=all --ignore-submodules=none` before inspection and again immediately before accepting its report; reading the frozen Git object directly is also acceptable. Do not treat intentionally dirty implementation worktrees as frozen reviewer checkouts. Let only the orchestrator update the shared remote ref; pause refreshes during native PR preparation and merge. Fetch again after landing, verify that the fetched baseline contains the canonical merge commit, and give each resumed worker that final frozen SHA. Inspect root and scoped `AGENTS.md`, owner boundaries, tests, callers, sibling implementations, package scripts, and current GitHub history. File locations are discovery starting points, not a claim that a particular file, scenario ID, route, or model still exists.
 
 Maintain at least ten distinct active investigations. Split any large area into smaller leaf tasks:
 
@@ -23,4 +23,4 @@ Give read-only Codex CLI reviewers a narrow prompt equivalent to:
 
 > At frozen OpenClaw main `<full-sha>`, independently audit `<single ownership surface>` for real correctness regressions. Read the complete root and relevant scoped guides, complete changed modules, entry point, callers, callees, sibling paths, state lifecycle, tests, and direct dependency source where relevant. Do not modify files, execute heavy tests, access or print secrets, alter an operator gateway, invoke a remote lease, or assume another reviewer's conclusion. Return only concrete current-main defects with repo-root paths, user-path reproduction, canonical root cause, all affected siblings, a coherent owner-boundary refactor, authentic regression, duplicates, and low-risk versus user-review classification. Reject symptom-only patches and count a shared invariant once. Explicitly return no verified bug if the evidence is insufficient.
 
-Use the current installed CLI and verify its supported flags with `codex exec --help`. Keep reviews ephemeral and bounded. Set a read-only sandbox for reviewers; reserve writable isolated worktrees for authorized fixers. When an offline, no-network, or no-independent-worker assignment forbids starting Codex CLI reviewers, inspect the subsystem slices directly and explicitly report that simultaneous independent lane capacity is unavailable.
+Use the current installed CLI and verify its supported flags with `codex exec --help`. Keep reviews ephemeral, bounded, and attached to a surviving supervisor; verify actual live child PIDs instead of trusting shell launch output. Set a read-only sandbox for reviewers; reserve writable isolated worktrees for authorized fixers. Count only independently observed, still-running workers toward active lanes, and report completed and stale-baseline outputs separately. When independent reviewers are unavailable, inspect subsystem slices directly and explicitly report the concurrency limitation.