From 50feb488ebf8566f1ff3f0d2ff3f80a9912cb47f Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Fri, 10 Jul 2026 01:29:06 -0700 Subject: [PATCH] fix(ci): isolate OpenWebUI release smoke Backport the missing scheduler, documentation, and focused regression proof from #103095 so stable/full release-path runs execute OpenWebUI exactly once on the dedicated lane while legacy aggregate aliases remain compatible. Partial backport of ff28c1f66639f00b8aab00ae81b9b73788c2ac06; the workflow and workflow-test portions were already present on this release branch, while the skill-only documentation remains main-only. --- docs/ci.md | 6 ++-- docs/reference/RELEASING.md | 4 +-- docs/reference/full-release-validation.md | 3 +- scripts/lib/docker-e2e-scenarios.mjs | 4 +-- test/scripts/docker-e2e-plan.test.ts | 36 ++++++++++++++++++++++- 5 files changed, 43 insertions(+), 10 deletions(-) diff --git a/docs/ci.md b/docs/ci.md index 007485dab163..cfa46943fea5 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -485,11 +485,11 @@ The reusable live/E2E workflow asks `scripts/test-docker-all.mjs --plan-json` wh Release Docker coverage runs smaller chunked jobs with `OPENCLAW_SKIP_DOCKER_BUILD=1` so each chunk pulls only the image kind it needs and executes multiple lanes through the same weighted scheduler: - `OPENCLAW_DOCKER_ALL_PROFILE=release-path` -- `OPENCLAW_DOCKER_ALL_CHUNK=core | package-update-openai | package-update-anthropic | package-update-core | plugins-runtime-plugins | plugins-runtime-services | plugins-runtime-install-a..h` +- `OPENCLAW_DOCKER_ALL_CHUNK=core | package-update-openai | package-update-anthropic | package-update-core | plugins-runtime-plugins | plugins-runtime-services | plugins-runtime-install-a..h | openwebui` -Current release Docker chunks are `core`, `package-update-openai`, `package-update-anthropic`, `package-update-core`, `plugins-runtime-plugins`, `plugins-runtime-services`, and `plugins-runtime-install-a` through `plugins-runtime-install-h`. `package-update-openai` includes the live Codex plugin package lane, which installs the candidate OpenClaw package, installs the Codex plugin from `codex_plugin_spec` or a same-ref tarball with explicit Codex CLI install approval, runs Codex CLI preflight, then runs multiple same-session OpenClaw agent turns against OpenAI. `plugins-runtime-core`, `plugins-runtime`, and `plugins-integrations` remain aggregate plugin/runtime aliases. The `install-e2e` lane alias remains the aggregate manual rerun alias for both provider installer lanes. +Current release Docker chunks are `core`, `package-update-openai`, `package-update-anthropic`, `package-update-core`, `plugins-runtime-plugins`, `plugins-runtime-services`, `plugins-runtime-install-a` through `plugins-runtime-install-h`, and `openwebui`. `package-update-openai` includes the live Codex plugin package lane, which installs the candidate OpenClaw package, installs the Codex plugin from `codex_plugin_spec` or a same-ref tarball with explicit Codex CLI install approval, runs Codex CLI preflight, then runs multiple same-session OpenClaw agent turns against OpenAI. `plugins-runtime-core`, `plugins-runtime`, and `plugins-integrations` remain aggregate plugin/runtime aliases. The `install-e2e` lane alias remains the aggregate manual rerun alias for both provider installer lanes. -OpenWebUI is folded into `plugins-runtime-services` when full release-path coverage requests it, and keeps a standalone `openwebui` chunk only for OpenWebUI-only dispatches. Bundled-channel update lanes retry once for transient npm network failures. +OpenWebUI runs as a standalone `openwebui` chunk on a dedicated large-disk Blacksmith runner whenever stable or full release-path coverage requests it, even when the reusable workflow routes supported jobs to GitHub-hosted runners. Keeping the external image pull separate prevents the large image from competing with the shared package and plugin images in `plugins-runtime-services`; legacy aggregate plugin/runtime chunks still include OpenWebUI for compatible manual reruns. Bundled-channel update lanes retry once for transient npm network failures. Each chunk uploads `.artifacts/docker-tests/` with lane logs, timings, `summary.json`, `failures.json`, phase timings, scheduler plan JSON, slow-lane tables, and per-lane rerun commands. The workflow `docker_lanes` input runs selected lanes against the prepared images instead of the chunk jobs, which keeps failed-lane debugging bounded to one targeted Docker job and prepares, downloads, or reuses the package artifact for that run; if a selected lane is a live Docker lane, the targeted job builds the live-test image locally for that rerun. Generated per-lane GitHub rerun commands include `package_artifact_run_id`, `package_artifact_name`, and prepared image inputs when those values exist, so a failed lane can reuse the exact package and images from the failed run. diff --git a/docs/reference/RELEASING.md b/docs/reference/RELEASING.md index 5b2f7a87064d..ff6ff949269c 100644 --- a/docs/reference/RELEASING.md +++ b/docs/reference/RELEASING.md @@ -375,8 +375,8 @@ Release Docker coverage includes: - full install smoke with the slow Bun global install smoke enabled - root Dockerfile smoke image preparation/reuse by target SHA, with QR, root/gateway, and installer/Bun smoke jobs running as separate install-smoke shards - repository E2E lanes -- release-path Docker chunks: `core`, `package-update-openai`, `package-update-anthropic`, `package-update-core`, `plugins-runtime-plugins`, `plugins-runtime-services`, `plugins-runtime-install-a` through `plugins-runtime-install-h` -- OpenWebUI coverage inside the `plugins-runtime-services` chunk when requested +- release-path Docker chunks: `core`, `package-update-openai`, `package-update-anthropic`, `package-update-core`, `plugins-runtime-plugins`, `plugins-runtime-services`, `plugins-runtime-install-a` through `plugins-runtime-install-h`, and `openwebui` +- OpenWebUI coverage on a dedicated large-disk runner when requested - split bundled plugin install/uninstall lanes `bundled-plugin-install-uninstall-0` through `bundled-plugin-install-uninstall-23` - live/E2E provider suites and Docker live model coverage when release checks include live suites diff --git a/docs/reference/full-release-validation.md b/docs/reference/full-release-validation.md index 456d7652661d..b9ae790a0424 100644 --- a/docs/reference/full-release-validation.md +++ b/docs/reference/full-release-validation.md @@ -114,8 +114,9 @@ empty: | `package-update-anthropic` | Anthropic package install and update behavior. | | `package-update-core` | Provider-neutral package and update behavior. | | `plugins-runtime-plugins` | Plugin runtime lanes that exercise plugin behavior. | -| `plugins-runtime-services` | Service-backed and live plugin runtime lanes; includes OpenWebUI when requested. | +| `plugins-runtime-services` | Service-backed and live plugin runtime lanes. | | `plugins-runtime-install-a` through `plugins-runtime-install-h` | Plugin install/runtime batches split for parallel release validation. | +| `openwebui` | OpenWebUI compatibility smoke isolated on a dedicated large-disk runner when requested. | Use targeted `docker_lanes=` on the reusable live/E2E workflow when only one Docker lane failed. The release artifacts include per-lane rerun diff --git a/scripts/lib/docker-e2e-scenarios.mjs b/scripts/lib/docker-e2e-scenarios.mjs index 7b5b9bfe4905..c0cac0954e71 100644 --- a/scripts/lib/docker-e2e-scenarios.mjs +++ b/scripts/lib/docker-e2e-scenarios.mjs @@ -921,8 +921,7 @@ export function releasePathChunkLanes(chunk, options = {}) { return options.includeOpenWebUI ? [openWebUILane()] : []; } if ( - (chunk !== "plugins-runtime-services" && - chunk !== "plugins-runtime-core" && + (chunk !== "plugins-runtime-core" && chunk !== "plugins-runtime" && chunk !== "plugins-integrations") || !options.includeOpenWebUI @@ -935,7 +934,6 @@ export function releasePathChunkLanes(chunk, options = {}) { export function allReleasePathLanes(options = {}) { const releaseProfile = normalizeReleaseProfile(options.releaseProfile); return Object.keys(primaryReleasePathChunks) - .filter((chunk) => chunk !== "openwebui") .flatMap((chunk) => releasePathChunkLanes(chunk, { includeOpenWebUI: options.includeOpenWebUI, diff --git a/test/scripts/docker-e2e-plan.test.ts b/test/scripts/docker-e2e-plan.test.ts index 8d725fdead3e..41855a7eb645 100644 --- a/test/scripts/docker-e2e-plan.test.ts +++ b/test/scripts/docker-e2e-plan.test.ts @@ -181,7 +181,7 @@ describe("scripts/lib/docker-e2e-plan", () => { }); expect(withoutOpenWebUI.lanes.map((lane) => lane.name)).not.toContain("openwebui"); - expect(withOpenWebUI.lanes.map((lane) => lane.name)).toContain("openwebui"); + expect(withOpenWebUI.lanes.filter((lane) => lane.name === "openwebui")).toHaveLength(1); }); it("keeps beta release-path coverage to install, provider, and update proof lanes", () => { @@ -264,6 +264,11 @@ describe("scripts/lib/docker-e2e-plan", () => { profile: RELEASE_PATH_PROFILE, releaseChunk: "plugins-runtime-services", }); + const openWebUI = planFor({ + includeOpenWebUI: true, + profile: RELEASE_PATH_PROFILE, + releaseChunk: "openwebui", + }); const pluginsRuntimeInstallA = planFor({ includeOpenWebUI: true, profile: RELEASE_PATH_PROFILE, @@ -457,6 +462,8 @@ describe("scripts/lib/docker-e2e-plan", () => { timeoutMs: 1_200_000, weight: 3, }, + ]); + expect(openWebUI.lanes.map(summarizeLane)).toEqual([ { command: "OPENCLAW_OPENWEBUI_MODEL=openai/gpt-5.4-mini OPENCLAW_OPENWEBUI_PROVIDER_TIMEOUT_SECONDS=300 OPENCLAW_SKIP_DOCKER_BUILD=1 pnpm test:docker:openwebui", @@ -592,6 +599,33 @@ describe("scripts/lib/docker-e2e-plan", () => { ]); }); + it("includes OpenWebUI exactly once in each legacy plugin aggregate", () => { + for (const releaseChunk of [ + "plugins-runtime-core", + "plugins-runtime", + "plugins-integrations", + ]) { + const withOpenWebUI = planFor({ + includeOpenWebUI: true, + profile: RELEASE_PATH_PROFILE, + releaseChunk, + }); + const withoutOpenWebUI = planFor({ + includeOpenWebUI: false, + profile: RELEASE_PATH_PROFILE, + releaseChunk, + }); + + expect( + withOpenWebUI.lanes.filter((lane) => lane.name === "openwebui"), + releaseChunk, + ).toHaveLength(1); + expect(withoutOpenWebUI.lanes.map((lane) => lane.name), releaseChunk).not.toContain( + "openwebui", + ); + } + }); + it("expands the published upgrade survivor lane across deduped baselines", () => { const plan = planFor({ selectedLaneNames: ["published-upgrade-survivor"],