From 3e2a82bc0b8a22292177af15ac577ced25eebdf7 Mon Sep 17 00:00:00 2001 From: Patrick Erichsen Date: Tue, 18 Aug 2026 13:48:11 -0700 Subject: [PATCH] feat(release): add interactive release validation skill (#125642) * feat(release): model interactive validation campaigns * feat(release): add validation skill workflow * fix(release): select validation scope before missions * fix(release): harden copied validation setup * fix(release): require manual validation skill invocation * fix(release): harden copied validation finalization * fix(release): narrow validation issue feedback * fix(release): surface upgrade results before testing * fix(release): select validation missions one at a time * refactor(release): simplify validation to markdown worksheet * feat(release): prioritize validation from release notes * refactor(release): tighten validation worksheet hierarchy * refactor(release): summarize validation priorities by theme * refactor(release): integrate priorities into subsystem notes * refactor(release): align subsystem note hierarchy * refactor(release): remove cherry-picked examples * refactor(release): derive validation from maturity scorecard * fix(release): publish absolute scorecard links * docs(release): preserve scorecard generation step * docs(release): clarify generated surface headings * refactor(release): present validation guidance in tables * docs(release): normalize surface table example * refactor(release): embed notes in surface tables * refactor(release): sequence validation campaign setup * fix(release): trust explicit validation issue * fix(release): preserve campaign worksheet * fix(release): clarify validation worksheet --- .../openclaw-release-validation/SKILL.md | 270 ++++++++++++++++++ .../assets/validation-worksheet.md | 49 ++++ .../release-openclaw-maintainer/SKILL.md | 32 ++- 3 files changed, 341 insertions(+), 10 deletions(-) create mode 100644 .agents/skills/openclaw-release-validation/SKILL.md create mode 100644 .agents/skills/openclaw-release-validation/assets/validation-worksheet.md diff --git a/.agents/skills/openclaw-release-validation/SKILL.md b/.agents/skills/openclaw-release-validation/SKILL.md new file mode 100644 index 000000000000..56a26162d8b0 --- /dev/null +++ b/.agents/skills/openclaw-release-validation/SKILL.md @@ -0,0 +1,270 @@ +--- +name: openclaw-release-validation +description: Safely copy an existing gateway, upgrade it to an OpenClaw beta, and guide human release testing with one Markdown worksheet. +user-invocable: true +disable-model-invocation: true +--- + +# OpenClaw Release Validation + +Help a human validate one beta against a copy of a real gateway. Automate only +fixture setup and reporting. Let the human drive OpenClaw and judge quality. + +Use one editable Markdown worksheet as the entire run record. Do not create +`run.json`, mission state, receipts, or other tracking files. + +## Workflows + +Choose the workflow from the request: + +- **Initialize campaign** is the asynchronous release-process path. Create or + reuse the canonical issue for the exact candidate, close older open campaign + issues, print the current issue URL, and stop. +- **Validate release** is the default human-testing path. Join the existing + candidate issue, copy and upgrade a gateway, then guide testing. This workflow + never creates or rewrites the canonical issue. + +Before the upgrade reaches a terminal ready or blocked result, keep tester-facing +output to the campaign issue, candidate identity, gateway choice, and upgrade +progress or errors. The worksheet, priority surfaces, testing instructions, and +`finish validation` phrase are disclosed only after that gate. + +## 1. Candidate and shared issue + +Use an explicit beta when supplied; otherwise resolve the newest published tag +matching `vYYYY.M.D-beta.N`. Record its version and commit. + +When the request supplies an issue URL or number, resolve it directly with +`gh issue view`. Accept it only when it is open and its body contains the exact +`` marker. This direct verification is +authoritative: do not run a subsequent search or let a search result override it. + +When no issue is supplied, enumerate open repository issues through `gh api` +and inspect their bodies locally for the exact marker. Ignore pull requests and +closed issues. Do not use GitHub full-text search for this lookup: hidden HTML +comments are not reliably indexed. Fail clearly if more than one open issue has +the marker. + +Whenever the workflow reaches its issue announcement, use this exact shape with +one raw URL and no commentary about discovery or campaign counts: + +```text +Issue: https://github.com/openclaw/openclaw/issues/ +``` + +In **Validate release**, fail with `Release validation has not been initialized +for .` when the issue is absent. When it exists, announce it once in the +format above, then read its body and use the worksheet between +`` and +``. Keep its release priorities and template +unchanged. Those exact bytes are the canonical campaign template for this run. + +In **Initialize campaign**, reuse the current issue unchanged when it already +exists. When it does not exist, generate it: + +1. Read the GitHub release notes for the exact tag. If they are empty or + incomplete, also read that tag's section of `CHANGELOG.md`. +2. Fetch the live scorecard Markdown from + `https://docs.openclaw.ai/maturity/scorecard.md`. From its **All surfaces** + table, extract each unique surface's display name, taxonomy link, M-level, + and maturity label. Also extract the score bands. Treat this live response as + the complete catalog; do not use a cached or hardcoded surface list. Resolve + relative taxonomy links against `https://docs.openclaw.ai` before publishing. + Stop before issue creation when the scorecard is unavailable or cannot be + parsed. +3. Read the complete release notes and group every user-visible or + upgrade-sensitive item under one or more live scorecard surfaces. Use linked + PR or commit metadata privately when it helps estimate change size, but never + publish cherry-picked examples. +4. Rank exactly five priority surfaces using all of: change count and breadth, + change size and complexity, upgrade sensitivity, scope of user impact, and + maturity expectations. A touched Stable or Clawesome surface carries more + regression risk than an equally changed early-stage surface because users + rely on its stronger quality promise. Keep the ranking qualitative; do not + expose a fake-precision score. +5. Generate one section for every live scorecard surface. Put the five selected + surfaces under **Priority surfaces to test** and all remaining surfaces under + **Other surfaces to test**. Format every section exactly like this: + + ```md + ### [surface](taxonomy-url) + + | **Maturity score** | | + | ----------------------- | --------------------- | + | **What changed** | | + | **Recommended testing** | | + | **Testing notes** | | + ``` + + Keep the **Testing notes** value cell truly empty: add no placeholder text or + hidden comment. + Use `No notable changes in this release.` and an em dash in the last two + table rows when no release item is relevant. Escape table pipes and keep each + cell concise. Every priority surface must have a real recommended exercise. + + Make every **Recommended testing** cell a bounded operator workflow: name the + exact action, the observable pass condition, and a runnable OCM-scoped command + or concrete URL when the surface has one. Use `
` inside a cell when a + command and pass condition need separation. For example, onboarding should + name `ocm @ -- onboard`, the TUI should name + `ocm @ -- tui`, and channel health should name + `ocm @ -- channels status --probe`. Avoid broad prompts that bundle + unrelated features or say only to "use," "exercise," or "verify" a surface. + + For each **What changed**, synthesize the dominant themes across the + surface's complete group instead of listing a few fixes. Do not include + issue, PR, commit, or workflow examples; a handful of links misrepresents the + full release surface. Each **Recommended testing** is one concise human-driven + exercise. + +6. Resolve the campaign creator's GitHub login with `gh api user`; ask for a + login only when authentication cannot identify it. Enumerate every PR authored + by that login whose merge commit is included between the previous release tag + and the candidate tag. Add the complete linked list under **Your changes in + this release**, or `- None in this release.` when empty. This explicit author + list is separate from surface summaries and may contain PR links. +7. Make a working copy of the worksheet asset and fill it with the exact + candidate identity, release-notes URL, live scorecard and taxonomy URLs, + score-band guidance, and generated surface sections. The issue callout must + say that its catalog and labels come from the live maturity taxonomy and that + priority reflects release change volume, size, impact, upgrade risk, and + maturity expectations. Remove the campaign-creator comment and ensure no + template placeholder remains. +8. Create the issue with the stable marker, a short participation note, and the + completed worksheet verbatim between the worksheet markers. Read it back and + require the marker contents to equal the rendered worksheet before treating + campaign initialization as complete. Re-query open issues for the marker + after creation and fail on duplicates. + +After the current issue exists, find open campaign issues whose marker names a +release published before the current candidate. Comment on each with the current +issue URL, then close it as completed. Never close the current issue or a campaign +for a later release. Re-query and require the current candidate to be the only +open campaign. Announce its URL once in the exact format above and end the +initializer workflow without waiting for testing. + +Only **Initialize campaign** performs release-note analysis or generates the +canonical template. Validation runs consume the issue body without rewriting +it, but replace **Your changes in this release** in their private worksheet with +the current tester's complete authored-PR list for the same tag range. The +bundled worksheet asset is initializer-only; a validation run never reads it. + +## 2. Choose and copy a real gateway + +Discover once with `ocm env list --json`, then add plain `~/.openclaw` when it +is not already represented. Keep this overview shallow: show each gateway's +name, known version, and running state without inspecting every gateway's +plugins or paths. Ask which one the tester wants to copy. Never silently select +or modify the personal gateway. + +After selection, inspect only that gateway and record its version and commit. +Import its `.openclaw` state with OCM so sessions and other real user state are +preserved in the fixture: + +```sh +ocm adopt import --name --json +``` + +Use the `stateDir` returned by `ocm env list --json` for an OCM environment and +`~/.openclaw` for the plain gateway. Let OCM create the stopped, disposable +environment and assign a non-conflicting port; do not make an additional staged +copy. Keep the source unchanged. Before activating copied channel credentials, +stop the current credential owner and restore it when validation ends. + +## 3. Upgrade and report errors + +Install the exact candidate runtime and use the runtime name returned by OCM: + +```sh +ocm runtime install --version --json +ocm runtime verify --json +ocm upgrade --runtime --dry-run --json +ocm upgrade --runtime --json +ocm start --runtime --json +``` + +Stop any current owner of copied channel credentials immediately before the +`ocm start` command. + +Verify `ocm service status `, `ocm @ -- --version`, and +`ocm logs --tail 100`. OCM's successful managed upgrade already +requires HTTP health and gateway reachability. + +Report every error to the tester immediately, including errors recovered by a +retry. Retain candidate OpenClaw behavior caused by the upgrade for **Upgrade +findings** after the worksheet is created; it is eligible for the GitHub +comment. Keep OCM, copying, local tooling, setup, and cleanup problems in the +conversation only; they never enter the worksheet or GitHub comment. + +Complete this step only when candidate readiness is either verified or blocked +with a concrete terminal finding. Do not continue to testing while the upgrade +or gateway readiness is unresolved. + +## 4. Create and reveal the worksheet + +Only after the upgrade gate above, copy the canonical worksheet between the +shared issue's markers byte-for-byte to +`.artifacts/openclaw-release-validation/-.md`. Fill in the +source, shared issue URL, terminal upgrade result, and eligible upgrade findings +without changing the campaign priorities. Refresh **Your changes in this +release** for the current tester. + +Preserve every other heading, table, callout, surface order, maturity score, +release theme, and recommended test exactly as copied. The only validation-run +edits are the source fields, **Your changes in this release**, **Upgrade +findings**, **Upgrade result**, non-empty **Testing notes** cells, and **Final +feedback**. Never regenerate, reformat, or substitute the campaign template. + +Resolve and print the worksheet's absolute path, followed by one exact +platform-appropriate command that opens it. Use this shape on macOS: + +```text +Testing worksheet: /absolute/path/to/worksheet.md +Open it: open '/absolute/path/to/worksheet.md' +``` + +Use `xdg-open ''` on Linux or `start "" ""` on +Windows. Shell-quote the actual path. Then briefly point out the five priority +surfaces and tell the tester: **Edit the worksheet directly or tell me what to +record. Reply exactly `finish validation` when you are done.** + +This worksheet is the only checklist and note store. If readiness is verified, +continue to human-driven testing. If readiness is blocked, state that testing +cannot begin and wait for final feedback or `finish validation`. + +## 5. Human-driven testing + +Ask: **What do you want to test first?** Recommend starting with a release +priority, but let the tester choose one surface at a time in any order. After +each item, add their notes to that surface's **Testing notes** table cell, then +ask what they want to test next. + +The tester drives interactive surfaces such as the TUI, Control UI, onboarding, +channels, pairing, and approvals. Provide the command or URL and explain what +to look for, then wait for their result. Take control only when explicitly +asked. Do not turn the checklist into an automated scenario runner. + +A surface counts as tested only when tester-authored text appears in its +**Testing notes** row. The **Maturity score**, **What changed**, and +**Recommended testing** rows are campaign guidance, never test evidence. An +empty Testing notes value means untouched. Escape table pipes and use `
` +between multiple notes. Add candidate problems found during surface testing to +that cell. + +## 6. Finish and publish + +When the tester says `finish validation`: + +1. Read the worksheet and ask only for a missing promotion vote or final + feedback. +2. Stop the copied gateway and restore any source gateway stopped for channel + ownership. Ask before destroying the disposable environment. +3. Synthesize one final release-analysis comment from candidate identity, source + version/commit, upgrade findings, tester feedback, the yes/no promotion vote, + and only the surfaces with non-empty Testing notes cells. Use those cells as + the source of observed results; do not report the other table rows as evidence. +4. Remove local paths, gateway names, secrets, user identifiers, raw logs, OCM + notes, setup details, and cleanup details from the comment. +5. Post the comment once with `gh` and show the tester its URL. + +The skill collects release feedback; it does not make the go/no-go decision. diff --git a/.agents/skills/openclaw-release-validation/assets/validation-worksheet.md b/.agents/skills/openclaw-release-validation/assets/validation-worksheet.md new file mode 100644 index 000000000000..83084116b57e --- /dev/null +++ b/.agents/skills/openclaw-release-validation/assets/validation-worksheet.md @@ -0,0 +1,49 @@ +# OpenClaw release validation + +## Candidate + +- Release: +- Commit: +- Release notes: {{RELEASE_NOTES_URL}} +- Source version: +- Source commit: +- Shared issue: +- Upgrade result: pending + +## Upgrade findings + +Automatically populated by the agent with candidate OpenClaw problems observed +while upgrading or starting the copied gateway. + +- None yet. + +## Your changes in this release + + + +## Priority surfaces to test + +> [!NOTE] +> Add findings to the empty **Testing notes** cell for each surface you test; +> leave untouched cells empty. Those cells are the source for the final +> release-analysis comment. +> +> This surface catalog and its maturity labels are derived from the live +> [OpenClaw maturity scorecard]({{SCORECARD_URL}}) and +> [maturity taxonomy]({{TAXONOMY_URL}}). Priority reflects this release's change +> volume, change size, impact scope, upgrade risk, and maturity expectations. +> +> **Score bands:** Experimental 0–50%; Alpha 50–70%; Beta 70–80%; Stable +> 80–95%; Clawesome 95–100%. Higher maturity means a stronger regression +> expectation. + + + +## Other surfaces to test + + + +## Final feedback + +- Overall feedback: +- Polished enough to promote: yes / no diff --git a/.agents/skills/release-openclaw-maintainer/SKILL.md b/.agents/skills/release-openclaw-maintainer/SKILL.md index 8713db0059cf..f69fc17ce6bd 100644 --- a/.agents/skills/release-openclaw-maintainer/SKILL.md +++ b/.agents/skills/release-openclaw-maintainer/SKILL.md @@ -20,6 +20,7 @@ current: - release version, tag, branch, cut SHA, Code SHA, Tooling SHA, and Release SHA - active Full Release Validation parent run id and attempt - npm preflight and publish parent run ids +- release-validation initializer task id and campaign issue URL when available - completed phases and immutable child artifacts - approved backports or main changes - current phase, next action, and one precise blocker if stopped @@ -1124,7 +1125,18 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts with the original child run IDs and an evidence output path before manually recreating the workflow's draft, dependency evidence asset, proof section, and publish step. -27. Run the post-published beta verification roster. Do not scan current `main` +27. After the GitHub release is published, launch a separate persistent Codex + task in the OpenClaw project and continue immediately without waiting for + it. Give the task this explicit prompt with the real tag substituted: + `Run $openclaw-release-validation in initialize campaign mode for .` + The initializer creates or reuses the current campaign issue and closes + older open campaign issues. Record the task id, then let publication and + postpublish verification continue in parallel. Issue generation is not a + publication blocker; confirm its resulting URL before inviting humans to + validate the release. If persistent task creation is unavailable, record + campaign initialization as an explicit release follow-up instead of running + it synchronously inside the release task. +28. Run the post-published beta verification roster. Do not scan current `main` for extra fixes unless the operator explicitly requests a backport audit. Apply only operator-selected backports, and increment to the next beta if a selected fix must change the already-published package. A failed confidence @@ -1144,10 +1156,10 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts all-group cycle. The operator's separate beta-attempt cap, normally 4, remains a ceiling for admitted product attempts, not an automatic retry budget. -28. Announce the beta/stable release on Discord best-effort using the configured secret workflow. -29. If the operator requested beta only, stop after beta verification and the +29. Announce the beta/stable release on Discord best-effort using the configured secret workflow. +30. If the operator requested beta only, stop after beta verification and the announcement. -30. If the stable release was published to `beta`, use the light stable +31. If the stable release was published to `beta`, use the light stable promotion roster when the matching beta already carried the full confidence pass: published npm postpublish verify, Docker install/update smoke, macOS-only Parallels install/update smoke, and required QA signal. @@ -1155,25 +1167,25 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts `openclaw/releases/.github/workflows/openclaw-npm-dist-tags.yml` workflow to promote that stable version from `beta` to `latest`, then verify `latest` now points at that version. -31. If the stable release was published directly to `latest` and `beta` should +32. If the stable release was published directly to `latest` and `beta` should follow it, start that same release-ops dist-tag workflow to point `beta` at the stable version, then verify both `latest` and `beta` point at that version. -32. For stable releases, start +33. For stable releases, start `openclaw/releases/.github/workflows/openclaw-macos-publish.yml` for the real publish with the successful release-ops mac `preflight_run_id` and wait for success. -33. Verify the successful real release-ops mac run uploaded the `.zip`, `.dmg`, +34. Verify the successful real release-ops mac run uploaded the `.zip`, `.dmg`, and `.dSYM.zip` artifacts to the existing GitHub release in `openclaw/openclaw`. -34. For stable releases, download `macos-appcast-` from the successful +35. For stable releases, download `macos-appcast-` from the successful release-ops mac run, update `appcast.xml` on `main`, verify the feed, then complete the **Close stable releases on main** gate. -35. For beta releases, publish the mac assets only when intentionally requested; +36. For beta releases, publish the mac assets only when intentionally requested; expect no shared production `appcast.xml` artifact and do not update the shared production feed unless a separate beta feed exists. -36. After stable main closeout, verify npm and the attached release artifacts. +37. After stable main closeout, verify npm and the attached release artifacts. ## GHSA advisory work