ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-25 11:55:47 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(release): add interactive release validation skill (#125642)

Browse Source 
* feat(release): model interactive validation campaigns

* feat(release): add validation skill workflow

* fix(release): select validation scope before missions

* fix(release): harden copied validation setup

* fix(release): require manual validation skill invocation

* fix(release): harden copied validation finalization

* fix(release): narrow validation issue feedback

* fix(release): surface upgrade results before testing

* fix(release): select validation missions one at a time

* refactor(release): simplify validation to markdown worksheet

* feat(release): prioritize validation from release notes

* refactor(release): tighten validation worksheet hierarchy

* refactor(release): summarize validation priorities by theme

* refactor(release): integrate priorities into subsystem notes

* refactor(release): align subsystem note hierarchy

* refactor(release): remove cherry-picked examples

* refactor(release): derive validation from maturity scorecard

* fix(release): publish absolute scorecard links

* docs(release): preserve scorecard generation step

* docs(release): clarify generated surface headings

* refactor(release): present validation guidance in tables

* docs(release): normalize surface table example

* refactor(release): embed notes in surface tables

* refactor(release): sequence validation campaign setup

* fix(release): trust explicit validation issue

* fix(release): preserve campaign worksheet

* fix(release): clarify validation worksheet
This commit is contained in:
Patrick Erichsen
2026-08-18 13:48:11 -07:00
committed by GitHub
parent 99acca309c
commit 3e2a82bc0b
3 changed files with 341 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/openclaw-release-validation/SKILL.md
+270
View File
@@ -0,0 +1,270 @@
---
name: openclaw-release-validation
description: Safely copy an existing gateway, upgrade it to an OpenClaw beta, and guide human release testing with one Markdown worksheet.
user-invocable: true
disable-model-invocation: true
---
# OpenClaw Release Validation
Help a human validate one beta against a copy of a real gateway. Automate only
fixture setup and reporting. Let the human drive OpenClaw and judge quality.
Use one editable Markdown worksheet as the entire run record. Do not create
`run.json`, mission state, receipts, or other tracking files.
## Workflows
Choose the workflow from the request:
- **Initialize campaign** is the asynchronous release-process path. Create or
reuse the canonical issue for the exact candidate, close older open campaign
issues, print the current issue URL, and stop.
- **Validate release** is the default human-testing path. Join the existing
candidate issue, copy and upgrade a gateway, then guide testing. This workflow
never creates or rewrites the canonical issue.
Before the upgrade reaches a terminal ready or blocked result, keep tester-facing
output to the campaign issue, candidate identity, gateway choice, and upgrade
progress or errors. The worksheet, priority surfaces, testing instructions, and
`finish validation` phrase are disclosed only after that gate.
## 1. Candidate and shared issue
Use an explicit beta when supplied; otherwise resolve the newest published tag
matching `vYYYY.M.D-beta.N`. Record its version and commit.
When the request supplies an issue URL or number, resolve it directly with
`gh issue view`. Accept it only when it is open and its body contains the exact
`<!-- openclaw-release-validation:<tag> -->` marker. This direct verification is
authoritative: do not run a subsequent search or let a search result override it.
When no issue is supplied, enumerate open repository issues through `gh api`
and inspect their bodies locally for the exact marker. Ignore pull requests and
closed issues. Do not use GitHub full-text search for this lookup: hidden HTML
comments are not reliably indexed. Fail clearly if more than one open issue has
the marker.
Whenever the workflow reaches its issue announcement, use this exact shape with
one raw URL and no commentary about discovery or campaign counts:
```text
Issue: https://github.com/openclaw/openclaw/issues/<number>
```
In **Validate release**, fail with `Release validation has not been initialized
for <tag>.` when the issue is absent. When it exists, announce it once in the
format above, then read its body and use the worksheet between
`<!-- validation-worksheet:start -->` and
`<!-- validation-worksheet:end -->`. Keep its release priorities and template
unchanged. Those exact bytes are the canonical campaign template for this run.
In **Initialize campaign**, reuse the current issue unchanged when it already
exists. When it does not exist, generate it:
1. Read the GitHub release notes for the exact tag. If they are empty or
incomplete, also read that tag's section of `CHANGELOG.md`.
2. Fetch the live scorecard Markdown from
`https://docs.openclaw.ai/maturity/scorecard.md`. From its **All surfaces**
table, extract each unique surface's display name, taxonomy link, M-level,
and maturity label. Also extract the score bands. Treat this live response as
the complete catalog; do not use a cached or hardcoded surface list. Resolve
relative taxonomy links against `https://docs.openclaw.ai` before publishing.
Stop before issue creation when the scorecard is unavailable or cannot be
parsed.
3. Read the complete release notes and group every user-visible or
upgrade-sensitive item under one or more live scorecard surfaces. Use linked
PR or commit metadata privately when it helps estimate change size, but never
publish cherry-picked examples.
4. Rank exactly five priority surfaces using all of: change count and breadth,
change size and complexity, upgrade sensitivity, scope of user impact, and
maturity expectations. A touched Stable or Clawesome surface carries more
regression risk than an equally changed early-stage surface because users
rely on its stronger quality promise. Keep the ranking qualitative; do not
expose a fake-precision score.
5. Generate one section for every live scorecard surface. Put the five selected
surfaces under **Priority surfaces to test** and all remaining surfaces under
**Other surfaces to test**. Format every section exactly like this:
```md
### [surface](taxonomy-url)
| **Maturity score** | <maturity-label> |
| ----------------------- | --------------------- |
| **What changed** | <release-theme> |
| **Recommended testing** | <exercise-or-em-dash> |
| **Testing notes** | |
```
Keep the **Testing notes** value cell truly empty: add no placeholder text or
hidden comment.
Use `No notable changes in this release.` and an em dash in the last two
table rows when no release item is relevant. Escape table pipes and keep each
cell concise. Every priority surface must have a real recommended exercise.
Make every **Recommended testing** cell a bounded operator workflow: name the
exact action, the observable pass condition, and a runnable OCM-scoped command
or concrete URL when the surface has one. Use `<br>` inside a cell when a
command and pass condition need separation. For example, onboarding should
name `ocm @<test-env> -- onboard`, the TUI should name
`ocm @<test-env> -- tui`, and channel health should name
`ocm @<test-env> -- channels status --probe`. Avoid broad prompts that bundle
unrelated features or say only to "use," "exercise," or "verify" a surface.
For each **What changed**, synthesize the dominant themes across the
surface's complete group instead of listing a few fixes. Do not include
issue, PR, commit, or workflow examples; a handful of links misrepresents the
full release surface. Each **Recommended testing** is one concise human-driven
exercise.
6. Resolve the campaign creator's GitHub login with `gh api user`; ask for a
login only when authentication cannot identify it. Enumerate every PR authored
by that login whose merge commit is included between the previous release tag
and the candidate tag. Add the complete linked list under **Your changes in
this release**, or `- None in this release.` when empty. This explicit author
list is separate from surface summaries and may contain PR links.
7. Make a working copy of the worksheet asset and fill it with the exact
candidate identity, release-notes URL, live scorecard and taxonomy URLs,
score-band guidance, and generated surface sections. The issue callout must
say that its catalog and labels come from the live maturity taxonomy and that
priority reflects release change volume, size, impact, upgrade risk, and
maturity expectations. Remove the campaign-creator comment and ensure no
template placeholder remains.
8. Create the issue with the stable marker, a short participation note, and the
completed worksheet verbatim between the worksheet markers. Read it back and
require the marker contents to equal the rendered worksheet before treating
campaign initialization as complete. Re-query open issues for the marker
after creation and fail on duplicates.
After the current issue exists, find open campaign issues whose marker names a
release published before the current candidate. Comment on each with the current
issue URL, then close it as completed. Never close the current issue or a campaign
for a later release. Re-query and require the current candidate to be the only
open campaign. Announce its URL once in the exact format above and end the
initializer workflow without waiting for testing.
Only **Initialize campaign** performs release-note analysis or generates the
canonical template. Validation runs consume the issue body without rewriting
it, but replace **Your changes in this release** in their private worksheet with
the current tester's complete authored-PR list for the same tag range. The
bundled worksheet asset is initializer-only; a validation run never reads it.
## 2. Choose and copy a real gateway
Discover once with `ocm env list --json`, then add plain `~/.openclaw` when it
is not already represented. Keep this overview shallow: show each gateway's
name, known version, and running state without inspecting every gateway's
plugins or paths. Ask which one the tester wants to copy. Never silently select
or modify the personal gateway.
After selection, inspect only that gateway and record its version and commit.
Import its `.openclaw` state with OCM so sessions and other real user state are
preserved in the fixture:
```sh
ocm adopt import --name <test-env> <selected-state-dir> --json
```
Use the `stateDir` returned by `ocm env list --json` for an OCM environment and
`~/.openclaw` for the plain gateway. Let OCM create the stopped, disposable
environment and assign a non-conflicting port; do not make an additional staged
copy. Keep the source unchanged. Before activating copied channel credentials,
stop the current credential owner and restore it when validation ends.
## 3. Upgrade and report errors
Install the exact candidate runtime and use the runtime name returned by OCM:
```sh
ocm runtime install --version <tag-without-v> --json
ocm runtime verify <runtime-name> --json
ocm upgrade <test-env> --runtime <runtime-name> --dry-run --json
ocm upgrade <test-env> --runtime <runtime-name> --json
ocm start <test-env> --runtime <runtime-name> --json
```
Stop any current owner of copied channel credentials immediately before the
`ocm start` command.
Verify `ocm service status <test-env>`, `ocm @<test-env> -- --version`, and
`ocm logs <test-env> --tail 100`. OCM's successful managed upgrade already
requires HTTP health and gateway reachability.
Report every error to the tester immediately, including errors recovered by a
retry. Retain candidate OpenClaw behavior caused by the upgrade for **Upgrade
findings** after the worksheet is created; it is eligible for the GitHub
comment. Keep OCM, copying, local tooling, setup, and cleanup problems in the
conversation only; they never enter the worksheet or GitHub comment.
Complete this step only when candidate readiness is either verified or blocked
with a concrete terminal finding. Do not continue to testing while the upgrade
or gateway readiness is unresolved.
## 4. Create and reveal the worksheet
Only after the upgrade gate above, copy the canonical worksheet between the
shared issue's markers byte-for-byte to
`.artifacts/openclaw-release-validation/<tag>-<timestamp>.md`. Fill in the
source, shared issue URL, terminal upgrade result, and eligible upgrade findings
without changing the campaign priorities. Refresh **Your changes in this
release** for the current tester.
Preserve every other heading, table, callout, surface order, maturity score,
release theme, and recommended test exactly as copied. The only validation-run
edits are the source fields, **Your changes in this release**, **Upgrade
findings**, **Upgrade result**, non-empty **Testing notes** cells, and **Final
feedback**. Never regenerate, reformat, or substitute the campaign template.
Resolve and print the worksheet's absolute path, followed by one exact
platform-appropriate command that opens it. Use this shape on macOS:
```text
Testing worksheet: /absolute/path/to/worksheet.md
Open it: open '/absolute/path/to/worksheet.md'
```
Use `xdg-open '<absolute-path>'` on Linux or `start "" "<absolute-path>"` on
Windows. Shell-quote the actual path. Then briefly point out the five priority
surfaces and tell the tester: **Edit the worksheet directly or tell me what to
record. Reply exactly `finish validation` when you are done.**
This worksheet is the only checklist and note store. If readiness is verified,
continue to human-driven testing. If readiness is blocked, state that testing
cannot begin and wait for final feedback or `finish validation`.
## 5. Human-driven testing
Ask: **What do you want to test first?** Recommend starting with a release
priority, but let the tester choose one surface at a time in any order. After
each item, add their notes to that surface's **Testing notes** table cell, then
ask what they want to test next.
The tester drives interactive surfaces such as the TUI, Control UI, onboarding,
channels, pairing, and approvals. Provide the command or URL and explain what
to look for, then wait for their result. Take control only when explicitly
asked. Do not turn the checklist into an automated scenario runner.
A surface counts as tested only when tester-authored text appears in its
**Testing notes** row. The **Maturity score**, **What changed**, and
**Recommended testing** rows are campaign guidance, never test evidence. An
empty Testing notes value means untouched. Escape table pipes and use `<br>`
between multiple notes. Add candidate problems found during surface testing to
that cell.
## 6. Finish and publish
When the tester says `finish validation`:
1. Read the worksheet and ask only for a missing promotion vote or final
feedback.
2. Stop the copied gateway and restore any source gateway stopped for channel
ownership. Ask before destroying the disposable environment.
3. Synthesize one final release-analysis comment from candidate identity, source
version/commit, upgrade findings, tester feedback, the yes/no promotion vote,
and only the surfaces with non-empty Testing notes cells. Use those cells as
the source of observed results; do not report the other table rows as evidence.
4. Remove local paths, gateway names, secrets, user identifiers, raw logs, OCM
notes, setup details, and cleanup details from the comment.
5. Post the comment once with `gh` and show the tester its URL.
The skill collects release feedback; it does not make the go/no-go decision.
.agents/skills/openclaw-release-validation/assets/validation-worksheet.md
+49
View File
@@ -0,0 +1,49 @@
# OpenClaw release validation
## Candidate
- Release:
- Commit:
- Release notes: {{RELEASE_NOTES_URL}}
- Source version:
- Source commit:
- Shared issue:
- Upgrade result: pending
## Upgrade findings
Automatically populated by the agent with candidate OpenClaw problems observed
while upgrading or starting the copied gateway.
- None yet.
## Your changes in this release
<!-- Campaign creator: enumerate every PR by the authenticated GitHub user included between the previous release tag and this candidate, then remove this comment. -->
## Priority surfaces to test
> [!NOTE]
> Add findings to the empty **Testing notes** cell for each surface you test;
> leave untouched cells empty. Those cells are the source for the final
> release-analysis comment.
>
> This surface catalog and its maturity labels are derived from the live
> [OpenClaw maturity scorecard]({{SCORECARD_URL}}) and
> [maturity taxonomy]({{TAXONOMY_URL}}). Priority reflects this release's change
> volume, change size, impact scope, upgrade risk, and maturity expectations.
>
> **Score bands:** Experimental 0–50%; Alpha 50–70%; Beta 70–80%; Stable
> 80–95%; Clawesome 95–100%. Higher maturity means a stronger regression
> expectation.
<!-- Campaign creator: generate exactly five priority surface tables with empty Testing notes cells, then remove this comment. -->
## Other surfaces to test
<!-- Campaign creator: generate every remaining live scorecard surface as a table with an empty Testing notes cell, then remove this comment. -->
## Final feedback
- Overall feedback:
- Polished enough to promote: yes / no
.agents/skills/release-openclaw-maintainer/SKILL.md
+22 -10
View File
@@ -20,6 +20,7 @@ current:
- release version, tag, branch, cut SHA, Code SHA, Tooling SHA, and Release SHA
- active Full Release Validation parent run id and attempt
- npm preflight and publish parent run ids
- release-validation initializer task id and campaign issue URL when available
- completed phases and immutable child artifacts
- approved backports or main changes
- current phase, next action, and one precise blocker if stopped
@@ -1124,7 +1125,18 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts <published-version>
with the original child run IDs and an evidence output path before manually
recreating the workflow's draft, dependency evidence asset, proof section,
and publish step.
27. Run the post-published beta verification roster. Do not scan current `main`
27. After the GitHub release is published, launch a separate persistent Codex
task in the OpenClaw project and continue immediately without waiting for
it. Give the task this explicit prompt with the real tag substituted:
`Run $openclaw-release-validation in initialize campaign mode for <tag>.`
The initializer creates or reuses the current campaign issue and closes
older open campaign issues. Record the task id, then let publication and
postpublish verification continue in parallel. Issue generation is not a
publication blocker; confirm its resulting URL before inviting humans to
validate the release. If persistent task creation is unavailable, record
campaign initialization as an explicit release follow-up instead of running
it synchronously inside the release task.
28. Run the post-published beta verification roster. Do not scan current `main`
for extra fixes unless the operator explicitly requests a backport audit.
Apply only operator-selected backports, and increment to the next beta if a
selected fix must change the already-published package. A failed confidence
@@ -1144,10 +1156,10 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts <published-version>
all-group cycle. The operator's separate beta-attempt cap, normally 4,
remains a ceiling for admitted product attempts, not an automatic retry
budget.
28. Announce the beta/stable release on Discord best-effort using the configured secret workflow.
29. If the operator requested beta only, stop after beta verification and the
29. Announce the beta/stable release on Discord best-effort using the configured secret workflow.
30. If the operator requested beta only, stop after beta verification and the
announcement.
30. If the stable release was published to `beta`, use the light stable
31. If the stable release was published to `beta`, use the light stable
promotion roster when the matching beta already carried the full confidence
pass: published npm postpublish verify, Docker install/update smoke,
macOS-only Parallels install/update smoke, and required QA signal.
@@ -1155,25 +1167,25 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts <published-version>
`openclaw/releases/.github/workflows/openclaw-npm-dist-tags.yml` workflow
to promote that stable version from `beta` to `latest`, then verify
`latest` now points at that version.
31. If the stable release was published directly to `latest` and `beta` should
32. If the stable release was published directly to `latest` and `beta` should
follow it, start that same release-ops dist-tag workflow to point `beta` at
the stable version, then verify both `latest` and `beta` point at that
version.
32. For stable releases, start
33. For stable releases, start
`openclaw/releases/.github/workflows/openclaw-macos-publish.yml` for the
real publish with the successful release-ops mac `preflight_run_id` and wait
for success.
33. Verify the successful real release-ops mac run uploaded the `.zip`, `.dmg`,
34. Verify the successful real release-ops mac run uploaded the `.zip`, `.dmg`,
and `.dSYM.zip` artifacts to the existing GitHub release in
`openclaw/openclaw`.
34. For stable releases, download `macos-appcast-<tag>` from the successful
35. For stable releases, download `macos-appcast-<tag>` from the successful
release-ops mac run, update `appcast.xml` on `main`, verify the feed, then
complete the **Close stable releases on main** gate.
35. For beta releases, publish the mac assets only when intentionally requested;
36. For beta releases, publish the mac assets only when intentionally requested;
expect no shared production
`appcast.xml` artifact and do not update the shared production feed unless a
separate beta feed exists.
36. After stable main closeout, verify npm and the attached release artifacts.
37. After stable main closeout, verify npm and the attached release artifacts.
## GHSA advisory work