`checks-windows-node-test` ran the whole 222-266s Windows body in a single
Blacksmith lane, so every run that scheduled it finished at 287-334s against a
~180s plateau. Measured across 45 canonical main runs it was the wall in all
seven runs that triggered it (324-519s vs a 210s median elsewhere).
Blacksmith's Windows class admits exactly 2 concurrent jobs (run 31865243804),
which is why the earlier 3-part split failed: its tail queued 211s behind a
finished part. Pin the split width to that measured capacity instead. The two
parts are balanced by measured per-project wall time -- part 1 keeps the old
list 1 plus list 2's non-infra projects (~108s), part 2 takes list 2's infra
project plus the old list 3 (~112s) -- so both land near 113s and the lane
stops setting the wall.
One canonical partition now serves every backend; the github-mode/dispatch
3-part variant and its separate file inventory are gone, along with the
single-lane `task: test` case and the per-row `matrix.runner` the collapsed
lane needed. The complete 65-file Windows inventory is preserved exactly, and
the guards pin the width on both sides (mutation-checked: restoring 3 parts
fails the matrix guard).
`checks-ui-e2e-real-gateway` folded `hybrid` into the `github` breaker clause
instead of taking the attempt-gated route its sibling `checks-ui-e2e` uses, so
it was the only Chromium lane pinned to hosted capacity on attempt 1. Measured
across 46 canonical main runs it averaged 205s (max 225s) against a 150-190s
plateau, making it the run's slowest job whenever the compact Node matrix
behaved.
Route it like `checks-ui-e2e`: Blacksmith 16-vCPU on attempt 1, hosted on rerun,
manual dispatch, fork, and same-repo PR retry. Cache inputs stay on the shared
pnpm store for both backends, matching the sibling lane.
The guard now derives both Chromium lanes' `runs-on` from one pinned template so
a hosted-only divergence cannot return unnoticed; mutation-checked by reverting
the workflow clause (2 failures).
Fork pull requests were pinned to GitHub-hosted runners while maintainer
PRs rode Blacksmith. That split arrived with ci.yml's first commit
(f4f990a) and was never a reasoned posture -- no rationale in docs,
commit bodies, or the ci-limits skill -- and this week's Blacksmith
tranches widened the gap: maintainer walls are ~3:36 while the last fork
PR I measured (#124633) took 13m0s. We already paid for it once in
#118530, which raised the cross-repo artifact budget to 35 minutes
because contributor PR #117992 timed out twice on hosted capacity.
Runner choice now follows author_association: OWNER, MEMBER, COLLABORATOR,
and CONTRIBUTOR get Blacksmith; FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE,
and MANNEQUIN stay on hosted runners, which are free for public repos, so
an unreviewed author cannot spend Blacksmith capacity. Earning the fast
path requires a landed commit, which requires a maintainer merge.
Note for anyone tempted to trim that list: maintainers report CONTRIBUTOR
here, not OWNER/MEMBER, because org membership is concealed -- steipete's
74 sampled PRs are all CONTRIBUTOR. Dropping CONTRIBUTOR would move
maintainer PRs to hosted.
Scope is deliberately runner-only: 27 runs-on clauses. The 34
dependency-cache/use-actions-cache conditions and 6 job ifs stay
fork-gated, because cache poisoning is a different risk from runner
choice -- a fork run still never writes an archive a trusted run restores.
Verified by evaluating all 25 configurable runs-on expressions: maintainer
PR 12/25 Blacksmith, returning-contributor fork 12/25 (identical), unknown
author 0/25, FIRST_TIME_CONTRIBUTOR 0/25, push to main 12/25 unchanged.
Guard tests gained trusted/untrusted fork cases; 119 pass.
Removing build-artifacts' unread job outputs left the step still writing
four `*-result` step outputs and carrying an `id` that nothing
references. The wave already reports failures through ::error
annotations and its exit code, so these were pure leftovers.
Re-simulated the step body with stubbed pnpm/node: normal run exits 0, a
failing verifier still exits 1.
Two pieces of dead workflow surface, found auditing where CI time goes:
- ci-timings-summary was hard-disabled (`if: ${{ false && ... }}`) with a
TODO to re-enable or delete it after the next timing-optimization
review. That review happened; the local `pnpm ci:timings` helper is what
we actually use, and docs already pointed there. The job carried a
25-entry needs list that had to be kept in sync to stay lintable.
- build-artifacts exported four `*-result` outputs that no job or workflow
reads.
Removing the job lets the gate guard assert the stronger invariant it
wanted all along: ci-gate needs *every* job in the file, so a new lane
cannot slip in ungated (28 jobs, 27 gated, zero exceptions).
No runtime behavior changes: the job could never run and the outputs had
no consumers. Also audited every `pnpm <script>` and `node scripts/...`
reference in ci.yml for rot -- all resolve.
build-artifacts is the wall's pole in 4 of the last 5 main runs (171-186s,
~15-20s ahead of the next lane), so its serial steps are the wall. The
Doctor plugin-index proof, singleton smoke, and startup-memory check ran
as their own 13s step even though they are independent dist readers that
the 47s artifact-check wave could absorb.
They now run inside that wave: on Blacksmith all seven start together, so
the verifiers cost the wave's max instead of 13s of serial time; hosted
runners still serialize the three through run_verifier so the RSS ceiling
measures an unloaded process. The step drops its selection gate because
the verifiers always run -- each artifact check already self-gates on its
own RUN_* flag, so a run with no checks selected still verifies.
Proof: extracted the step body and ran it with stubbed pnpm/node. Both
modes behave (Blacksmith 7 checks started, hosted-with-nothing-selected
still runs the 3 verifiers), and a failing verifier exits 1 with its
::error annotation in both -- the wave cannot swallow it.
* fix(ui): reset logs when source changes
Reload the active tail without a cursor when logs.tail switches files so the Control UI never mixes sources or skips a new file prefix.
* fix(ui): reset log source after reconnect
* test(ui): harden logs lifecycle proof
* test(ui): route Logs lifecycle through real Gateway lane
Pin OCM v0.2.32 so Performance validation uses upstream native .mts discovery and transitive workspace dependency handling instead of the local Bash/tsx trampoline.
The warmer replays the same test envelopes as the compact shard lanes
through scripts/ci-run-node-test-shard.mts, but without the lanes'
NODE_OPTIONS=--max-old-space-size=8192. Run 31874567859 lost a worker in
agents-embedded-agent-run under the default heap and the failure skipped
every cache-save post step, leaving the day's transform/compile caches
unrefreshed. Rerun 31918467031 passed, confirming load-dependence;
matching the lane environment removes the gap.
Measured on run 31865243804: the 3-part split on
blacksmith-8vcpu-windows-2025 got exactly 2 runners; part 2 queued 211s
and only started when part 1 finished, pushing the wall to 6:45 despite
133-163s job bodies (vs 342s hosted). The runner class admits ~2
concurrent jobs, so any profile that can land on Blacksmith now uses the
single lane; the 3-part split stays for guaranteed-hosted executions
(github mode and dispatches, whose runner expression forces hosted in
every mode). A hybrid retry reruns the single lane hosted - slower but
bounded, and Windows-scoped retries are rare.
Three measured cuts toward the sub-4-minute wall:
- checks-ui-e2e hosted-planner matrix 10 -> 12 rows (11 Vitest shards +
browser-extension). Shard 1 measured 230s with ~123s test body; Vitest
hash sharding leaves count as the only balance lever.
- checks-windows hybrid attempt 1 now rides blacksmith-8vcpu-windows-2025
(retries stay hosted windows-2025). The 3-part hosted split measured a
342s pole on full-graph runs.
- build-artifacts folds the Doctor plugin-index proof (12s), singleton
smoke, and startup-memory check (7s) into one step that overlaps them
on Blacksmith and stays serial on hosted 4-core so the RSS measurement
is unperturbed; any missing startup asset rebuild completes before the
fork so no verifier reads dist mid-write.
Run 31861511624 made checks-ui the UI-scoped wall pole (247s job, 177s body
on hosted 4-core even in hybrid). Attempt 1 now uses the 8 vCPU class like
the ui-e2e shards; retries stay hosted.
The manifest planner closure and the protocol coverage script import only
node builtins and relative files (verified importing the full closure with
an empty node_modules under native type stripping), so push/PR preflight
drops the pnpm store restore and install (~30s off the barrier every lane
waits behind). Manual dispatches keep the tsx path for frozen targets, and
the coverage script inlines the record guard under the documented
dependency-free exception.
The store archive accretes every prior lockfile generation through
prefix-key restores (measured 2.05 GiB, ~36s restore in every hosted job);
the warmup writer now prunes to the current lockfile closure before saving.
Sub-agent caching audit tranche, all measured on runs 31856622489 and
31857193681:
- QA Smoke parts rebuilt the runtime twice and packed a docker tgz that no
scenario in the smoke coverage set consumes (verified against the resolved
plan: zero docker-lane, zero Control UI scenarios). One private overlay
build replaces the public-build/ui:build/pack/private-rebuild chain
(~35-50s per part); the run step fails closed if a docker-lane scenario
ever returns, and the guard now pins never-pack-after-private-build.
- report-plugin-sdk-api-diff is pure reporting (nothing passes
--require-acknowledgement); it now runs on manual/release dispatches only.
- check-npm-lock resolved ~94 npm graphs against the registry on every push;
push/PR runs now use the reviewed --changed scoping (2.4s on dep-free
diffs) with a fail-open full sweep on base-fetch failure, and dispatches
keep --all for registry-drift coverage.
- Six restore-test-caches gates said 'github' only while the writer already
runs in hybrid; hybrid attempt-1 lanes now restore the transform seed.
- QA smoke lane reads the protected build compile cache; test-type stripes
widen to five.
Green tip run 31855637027 measured the remaining poles: ui-e2e shards up to
282s (imbalanced eight-way split), report-plugin-sdk-api-diff 246s on 4 vCPU,
and 220s test-type stripes (tsgo saturates a machine per graph, so stripe
count is the lever, not cores). Widen hosted-profile ui-e2e to ten rows,
test-type stripes to four, and lift the SDK diff to the 8 vCPU class.
Each graph is a serial single-project build, so tsgo gains little past four
cores; run 31852071103 measured ~34s/graph on 8 vCPU versus ~42s on hosted
4-core. Overlapping two fresh shard processes per CI stripe (--concurrency 2)
uses the idle cores and roughly halves the stripe body; local runs keep the
heap-bounded serial default.
First hybrid main cycles (runs 31850815388-31851455918) landed at 5:47-6:32
walls with every remaining pole a hosted lane: the three test-type stripes
(232-269s), check-additional-runtime-topology (274s), check-dependencies
(260s), and report-plugin-sdk-api-diff (209s). Route them to their Blacksmith
labels on hybrid attempt 1 (retries stay hosted) and lift runtime topology to
the 8 vCPU class.
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
* feat(secrets): add authenticated egress substitution proxy
* feat(secrets): bind egress substitution to hosts
* ci(codeql): classify egress proxy bypass tunnel in network boundary query
* refactor(proxy-capture): use the canonical IP parser instead of node:net
* fix(secrets): compare proxy tokens with a process-keyed MAC
* feat(docker): schedule image refreshes
* docs(docker): explain weekly image refreshes
* test(scripts): gate workflow-step execution on bash 4 mapfile support
Stock macOS bash 3.2 lacks mapfile; CI truth is Linux bash 5.
* test: cover docker-release suffix threading and sanctioned second caller
* test(codex): wire run-attempt-state into the attempt-extra project
#123345 added the file without a project owner; the full-suite coverage
guard fails for any PR that runs it.
* fix(ci): run build-artifacts PR validation on hosted runners
ci-build-artifacts-testbox.yml pinned PR runs to blacksmith-16vcpu and
ran Testbox lifecycle steps unconditionally, so the prepare-run landing
gate starved for every PR during a Blacksmith outage even with
OPENCLAW_CI_RUNNER_BACKEND=github. PR events now build on ubuntu-24.04
with dispatch-only Testbox steps, mirroring ci-check-testbox.yml.
* test(ci): align build-artifacts dispatch guard
* ci: codify manual-only Node 22 compatibility
* ci: split remaining hosted tail lanes
* fix(qa-lab): carry the validated part index into the plan id type
* test(ci): repair hosted tail lane guards
* ci: run push compacts without tooling lanes and stripe giants three-way
* fix(ci): keep CompactNodeTestPlanMode internal
knip --production flags the exported type as unused; no external module
imports it.
* ci: shard hosted lint and node tests
Split core oxlint across hosted runners and use backend-specific compact Node admission caps while preserving Blacksmith behavior.
* test(ci): guard plugin prerelease planner isolation
Peter Steinberger