mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-12 21:53:00 -06:00
test(buzz): add live QA Lab channel lane (#116298)
* feat(buzz): add live QA runner * ci(qa): add gated Buzz live lane * docs(qa): document Buzz live testing * fix(buzz): keep QA relay type private * test(buzz): type QA suite mock * fix(buzz): preserve racing QA reply relations * fix(qa): harden Buzz credential redaction * fix(buzz): ignore retained QA relay events * fix(buzz): clean up observer setup failures * test(buzz): return complete racing messages * fix(buzz): default QA credentials to file * fix(qa): validate pooled Buzz secrets * fix(qa): redact structured Buzz auth tags * fix(buzz): preserve QA credential env override * fix(qa): redact inspected Buzz auth tags * test(buzz): limit live QA to supported scenarios * ci(qa): run Buzz lane in release checks * fix(qa): require secure remote Buzz relays * fix(qa): satisfy Buzz QA validation gates * fix(qa): inject credential leasing into transport adapters
This commit is contained in:
Patrick Erichsen
@@ -124,7 +124,7 @@ it before dispatching. A narrower `rerun_group` skips this preflight.
|
||||
| Docker assets preflight | **Job:** `Verify Docker runtime image assets`<br />**Child workflow:** none<br />**Proves:** the `runtime-assets` Docker build target still succeeds before any other stage dispatches. Runs only for `rerun_group=all`.<br />**Rerun:** rerun the umbrella with `rerun_group=all`. |
|
||||
| Vitest and normal CI | **Job:** `Run normal full CI`<br />**Child workflow:** `CI`<br />**Proves:** manual full CI graph against the target ref, including Linux Node lanes, bundled plugin shards, plugin and channel contract shards, Node 22 compatibility, `check-*`, `check-additional-*`, built-artifact smoke checks, docs checks, Python skills, Windows, macOS, Control UI i18n, and Android via the umbrella.<br />**Rerun:** `rerun_group=ci`. |
|
||||
| Plugin prerelease | **Job:** `Run plugin prerelease validation`<br />**Child workflow:** `Plugin Prerelease`<br />**Proves:** release-only plugin static checks, agentic plugin coverage, full plugin batch shards, plugin prerelease Docker lanes, and a non-blocking `plugin-inspector-advisory` artifact for compatibility triage.<br />**Rerun:** `rerun_group=plugin-prerelease`. |
|
||||
| Release checks | **Job:** `Run release/live/Docker/QA validation`<br />**Child workflow:** `OpenClaw Release Checks`<br />**Proves:** install smoke, cross-OS package checks, Package Acceptance, QA Lab parity, live Matrix and Telegram, plus gated advisory Discord, WhatsApp, and Slack lanes. Stable and full profiles also run exhaustive live/E2E suites and Docker release-path chunks; beta can opt in with `run_release_soak=true`.<br />**Rerun:** `rerun_group=release-checks` or a narrower release-checks handle. |
|
||||
| Release checks | **Job:** `Run release/live/Docker/QA validation`<br />**Child workflow:** `OpenClaw Release Checks`<br />**Proves:** install smoke, cross-OS package checks, Package Acceptance, QA Lab parity, live Matrix, Buzz, and Telegram, plus gated advisory Discord, WhatsApp, and Slack lanes. Stable and full profiles also run exhaustive live/E2E suites and Docker release-path chunks; beta can opt in with `run_release_soak=true`.<br />**Rerun:** `rerun_group=release-checks` or a narrower release-checks handle. |
|
||||
| Package Telegram | **Job:** `Run package Telegram E2E`<br />**Child workflow:** `NPM Telegram Beta E2E`<br />**Proves:** a focused published-package Telegram E2E when `release_package_spec` or `npm_telegram_package_spec` is set. Full candidate validation uses the canonical Package Acceptance Telegram E2E instead.<br />**Rerun:** `rerun_group=npm-telegram` with `release_package_spec` or `npm_telegram_package_spec`. |
|
||||
| Product performance | **Job:** `Run product performance evidence`<br />**Child workflow:** `OpenClaw Performance`<br />**Proves:** release-profile performance run (`profile=release`, `repeat=3`, `fail_on_regression=true`, `publish_reports=false`) against the target SHA. Kova output stays in workflow artifacts and the child must prove its report publisher was skipped. Required (blocking) only for `rerun_group=all` or `rerun_group=performance`; not required for narrower rerun groups.<br />**Rerun:** `rerun_group=performance`. |
|
||||
| Umbrella verifier | **Job:** `Verify full validation`<br />**Child workflow:** none<br />**Proves:** re-checks recorded child run conclusions and appends slowest-job tables from child workflows.<br />**Rerun:** rerun only this job after rerunning a failed child to green. |
|
||||
@@ -175,6 +175,7 @@ artifact when package or Docker-facing stages need it.
|
||||
| QA runtime parity | **Job:** `Verify QA Lab runtime-pair lanes`<br />**Backing workflow:** direct job<br />**Tests:** the canonical core `openclaw`/`codex` lane (`pnpm openclaw qa suite --runtime-pair openclaw,codex --runtime-pair-lane core`) and, with `run_release_soak=true`, the soak lane. Advisory: individual lane jobs do not block the release-check verifier.<br />**Rerun:** `rerun_group=qa-parity` or `rerun_group=qa`. |
|
||||
| QA runtime tool coverage | **Job:** `Enforce QA Lab runtime tool coverage`<br />**Backing workflow:** direct job<br />**Tests:** dynamic tool drift between `openclaw` and `codex` in the canonical core runtime-pair lane (`pnpm openclaw qa coverage --tools`), using that lane's output. Blocking: this job is not advisory-overridable.<br />**Rerun:** `rerun_group=qa-parity` or `rerun_group=qa`. |
|
||||
| QA live Matrix | **Job:** `Run QA Live Matrix catalog`<br />**Backing workflow:** `QA-Lab - All Lanes` reusable workflow<br />**Tests:** catalog-derived YAML scenarios through the shared Matrix live adapter in the `qa-live-shared` environment, distributed across deterministic shards.<br />**Rerun:** `rerun_group=qa-live` or `rerun_group=qa`; use `live_suite_filter=qa-live-matrix` for a focused Matrix rerun. |
|
||||
| QA live Buzz | **Job:** `Run QA Lab live Buzz lane`<br />**Backing workflow:** `QA-Lab - All Lanes` reusable workflow<br />**Tests:** signed canary and mention-gating round trips through the real Buzz plugin using dedicated Convex-leased identities and a hosted relay room.<br />**Rerun:** `rerun_group=qa-live` or `rerun_group=qa`; use `live_suite_filter=qa-live-buzz` for a focused Buzz rerun. |
|
||||
| QA live Telegram | **Job:** `Run QA Lab live Telegram lane`<br />**Backing workflow:** trusted `OpenClaw Release Telegram QA` dispatch<br />**Tests:** live Telegram QA with Convex CI credential leases.<br />**Rerun:** `rerun_group=qa-live` or `rerun_group=qa`. |
|
||||
| QA live Discord | **Job:** `Run QA Lab live Discord lane`<br />**Backing workflow:** direct advisory job<br />**Tests:** live Discord QA with Convex CI credential leases when `OPENCLAW_RELEASE_QA_DISCORD_LIVE_CI_ENABLED` is enabled.<br />**Rerun:** `rerun_group=qa-live` with `live_suite_filter=qa-live-discord`. |
|
||||
| QA live WhatsApp | **Job:** `Run QA Lab live WhatsApp lane`<br />**Backing workflow:** direct advisory job<br />**Tests:** live WhatsApp QA with Convex CI credential leases when `OPENCLAW_RELEASE_QA_WHATSAPP_LIVE_CI_ENABLED` is enabled.<br />**Rerun:** `rerun_group=qa-live` with `live_suite_filter=qa-live-whatsapp`. |
|
||||
@@ -251,7 +252,7 @@ Use `rerun_group` to avoid repeating unrelated release boxes:
|
||||
| `package` | Package Acceptance. |
|
||||
| `qa` | QA parity plus QA live lanes. |
|
||||
| `qa-parity` | QA parity lanes and report only. |
|
||||
| `qa-live` | QA live Matrix/Telegram plus gated Discord, WhatsApp, and Slack lanes when enabled. |
|
||||
| `qa-live` | QA live Matrix, Buzz, and Telegram plus gated Discord, WhatsApp, and Slack lanes when enabled. |
|
||||
| `npm-telegram` | Published-package Telegram E2E; requires `release_package_spec` or `npm_telegram_package_spec`. |
|
||||
| `performance` | Product performance evidence only. |
|
||||
|
||||
@@ -264,8 +265,8 @@ Valid filter ids are defined in the reusable live/E2E workflow, including
|
||||
`live-codex-harness-docker`.
|
||||
|
||||
For a focused QA transport rerun, set `rerun_group=qa-live` and use the
|
||||
canonical selector `qa-live-matrix`, `qa-live-telegram`, `qa-live-discord`,
|
||||
`qa-live-whatsapp`, or `qa-live-slack`.
|
||||
canonical selector `qa-live-matrix`, `qa-live-buzz`, `qa-live-telegram`,
|
||||
`qa-live-discord`, `qa-live-whatsapp`, or `qa-live-slack`.
|
||||
|
||||
The `live-gateway-advisory-docker` handle is an aggregate rerun handle for its
|
||||
three provider shards, so it still fans out to all advisory Docker gateway jobs.
|
||||
@@ -310,8 +311,8 @@ Useful artifacts:
|
||||
- Docker release-path artifacts under `.artifacts/docker-tests/`
|
||||
- Package Acceptance `package-under-test` and Docker acceptance artifacts
|
||||
- Cross-OS release-check artifacts for each OS and suite
|
||||
- QA parity, runtime parity, and selected Matrix, Telegram, Discord, WhatsApp,
|
||||
or Slack artifacts
|
||||
- QA parity, runtime parity, and selected Matrix, Buzz, Telegram, Discord,
|
||||
WhatsApp, or Slack artifacts
|
||||
|
||||
## Workflow files
|
||||
|
||||
|
||||
Reference in New Issue
Block a user