Blacksmith's image tracks an older runner-images snapshot. Measured on a leased
box 2026-08-16, its toolcache holds Node 20.20.0, 22.22.0 and 24.13.0 while this
repo's engines floor is >=22.22.3 and >=24.15.0 -- short by three and two
patches. Every candidate is rejected, so all 306 of 306 sampled jobs fell
through to a nodejs.org download. GitHub-hosted runners carry 24.19.0 and
resolve from the toolcache in about a second, which is why only Blacksmith pays.
Normally that download is 2.6s (p99 3.3s), but ~46 jobs fetch the same 50 MB
simultaneously and the mirror throttles: three of 53 sampled runs had setup-node
medians of 44-93s with maxes to 139s, and because every job pays at once it
lands whole on the wall -- those runs went ~210s to 325s.
Keep the payload in the Actions cache, which Blacksmith serves from its
colocated backend. Measured on Blacksmith: cold 1605ms, warm 77ms.
Restores are prefix-keyed and the save carries the resolved patch. An exact key
would be worse than nothing: cache entries are immutable and an exact hit
suppresses the post-job save, so a floating `24.x` key would pin the first Node
it ever saw and, once the floor advanced past it, every job would restore the
rejected payload and re-download forever. Keying the save on the installed
version lets a newer resolve publish a new entry that later prefix restores
pick up.
A rejected payload is pruned before the replacement installs, because the entry
is saved wholesale and a leftover would ride along in every future save.
Windows keeps its existing path. Proven on Blacksmith across cold, warm, stale
and truncated-binary cases; both guards are mutation-checked.
This stays useful even if Blacksmith refreshes their image: the floor moves
independently of the snapshot, so the gap recurs. The image refresh is still
the better fix and is worth asking them for.
`checks-windows-node-test` ran the whole 222-266s Windows body in a single
Blacksmith lane, so every run that scheduled it finished at 287-334s against a
~180s plateau. Measured across 45 canonical main runs it was the wall in all
seven runs that triggered it (324-519s vs a 210s median elsewhere).
Blacksmith's Windows class admits exactly 2 concurrent jobs (run 31865243804),
which is why the earlier 3-part split failed: its tail queued 211s behind a
finished part. Pin the split width to that measured capacity instead. The two
parts are balanced by measured per-project wall time -- part 1 keeps the old
list 1 plus list 2's non-infra projects (~108s), part 2 takes list 2's infra
project plus the old list 3 (~112s) -- so both land near 113s and the lane
stops setting the wall.
One canonical partition now serves every backend; the github-mode/dispatch
3-part variant and its separate file inventory are gone, along with the
single-lane `task: test` case and the per-row `matrix.runner` the collapsed
lane needed. The complete 65-file Windows inventory is preserved exactly, and
the guards pin the width on both sides (mutation-checked: restoring 3 parts
fails the matrix guard).
`checks-ui-e2e-real-gateway` folded `hybrid` into the `github` breaker clause
instead of taking the attempt-gated route its sibling `checks-ui-e2e` uses, so
it was the only Chromium lane pinned to hosted capacity on attempt 1. Measured
across 46 canonical main runs it averaged 205s (max 225s) against a 150-190s
plateau, making it the run's slowest job whenever the compact Node matrix
behaved.
Route it like `checks-ui-e2e`: Blacksmith 16-vCPU on attempt 1, hosted on rerun,
manual dispatch, fork, and same-repo PR retry. Cache inputs stay on the shared
pnpm store for both backends, matching the sibling lane.
The guard now derives both Chromium lanes' `runs-on` from one pinned template so
a hosted-only divergence cannot return unnoticed; mutation-checked by reverting
the workflow clause (2 failures).
Fork pull requests were pinned to GitHub-hosted runners while maintainer
PRs rode Blacksmith. That split arrived with ci.yml's first commit
(f4f990a) and was never a reasoned posture -- no rationale in docs,
commit bodies, or the ci-limits skill -- and this week's Blacksmith
tranches widened the gap: maintainer walls are ~3:36 while the last fork
PR I measured (#124633) took 13m0s. We already paid for it once in
#118530, which raised the cross-repo artifact budget to 35 minutes
because contributor PR #117992 timed out twice on hosted capacity.
Runner choice now follows author_association: OWNER, MEMBER, COLLABORATOR,
and CONTRIBUTOR get Blacksmith; FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE,
and MANNEQUIN stay on hosted runners, which are free for public repos, so
an unreviewed author cannot spend Blacksmith capacity. Earning the fast
path requires a landed commit, which requires a maintainer merge.
Note for anyone tempted to trim that list: maintainers report CONTRIBUTOR
here, not OWNER/MEMBER, because org membership is concealed -- steipete's
74 sampled PRs are all CONTRIBUTOR. Dropping CONTRIBUTOR would move
maintainer PRs to hosted.
Scope is deliberately runner-only: 27 runs-on clauses. The 34
dependency-cache/use-actions-cache conditions and 6 job ifs stay
fork-gated, because cache poisoning is a different risk from runner
choice -- a fork run still never writes an archive a trusted run restores.
Verified by evaluating all 25 configurable runs-on expressions: maintainer
PR 12/25 Blacksmith, returning-contributor fork 12/25 (identical), unknown
author 0/25, FIRST_TIME_CONTRIBUTOR 0/25, push to main 12/25 unchanged.
Guard tests gained trusted/untrusted fork cases; 119 pass.
Two pieces of dead workflow surface, found auditing where CI time goes:
- ci-timings-summary was hard-disabled (`if: ${{ false && ... }}`) with a
TODO to re-enable or delete it after the next timing-optimization
review. That review happened; the local `pnpm ci:timings` helper is what
we actually use, and docs already pointed there. The job carried a
25-entry needs list that had to be kept in sync to stay lintable.
- build-artifacts exported four `*-result` outputs that no job or workflow
reads.
Removing the job lets the gate guard assert the stronger invariant it
wanted all along: ci-gate needs *every* job in the file, so a new lane
cannot slip in ungated (28 jobs, 27 gated, zero exceptions).
No runtime behavior changes: the job could never run and the outputs had
no consumers. Also audited every `pnpm <script>` and `node scripts/...`
reference in ci.yml for rot -- all resolve.
build-artifacts is the wall's pole in 4 of the last 5 main runs (171-186s,
~15-20s ahead of the next lane), so its serial steps are the wall. The
Doctor plugin-index proof, singleton smoke, and startup-memory check ran
as their own 13s step even though they are independent dist readers that
the 47s artifact-check wave could absorb.
They now run inside that wave: on Blacksmith all seven start together, so
the verifiers cost the wave's max instead of 13s of serial time; hosted
runners still serialize the three through run_verifier so the RSS ceiling
measures an unloaded process. The step drops its selection gate because
the verifiers always run -- each artifact check already self-gates on its
own RUN_* flag, so a run with no checks selected still verifies.
Proof: extracted the step body and ran it with stubbed pnpm/node. Both
modes behave (Blacksmith 7 checks started, hosted-with-nothing-selected
still runs the 3 verifiers), and a failing verifier exits 1 with its
::error annotation in both -- the wave cannot swallow it.
Measured on run 31865243804: the 3-part split on
blacksmith-8vcpu-windows-2025 got exactly 2 runners; part 2 queued 211s
and only started when part 1 finished, pushing the wall to 6:45 despite
133-163s job bodies (vs 342s hosted). The runner class admits ~2
concurrent jobs, so any profile that can land on Blacksmith now uses the
single lane; the 3-part split stays for guaranteed-hosted executions
(github mode and dispatches, whose runner expression forces hosted in
every mode). A hybrid retry reruns the single lane hosted - slower but
bounded, and Windows-scoped retries are rare.
Three measured cuts toward the sub-4-minute wall:
- checks-ui-e2e hosted-planner matrix 10 -> 12 rows (11 Vitest shards +
browser-extension). Shard 1 measured 230s with ~123s test body; Vitest
hash sharding leaves count as the only balance lever.
- checks-windows hybrid attempt 1 now rides blacksmith-8vcpu-windows-2025
(retries stay hosted windows-2025). The 3-part hosted split measured a
342s pole on full-graph runs.
- build-artifacts folds the Doctor plugin-index proof (12s), singleton
smoke, and startup-memory check (7s) into one step that overlaps them
on Blacksmith and stays serial on hosted 4-core so the RSS measurement
is unperturbed; any missing startup asset rebuild completes before the
fork so no verifier reads dist mid-write.
Sub-agent caching audit tranche, all measured on runs 31856622489 and
31857193681:
- QA Smoke parts rebuilt the runtime twice and packed a docker tgz that no
scenario in the smoke coverage set consumes (verified against the resolved
plan: zero docker-lane, zero Control UI scenarios). One private overlay
build replaces the public-build/ui:build/pack/private-rebuild chain
(~35-50s per part); the run step fails closed if a docker-lane scenario
ever returns, and the guard now pins never-pack-after-private-build.
- report-plugin-sdk-api-diff is pure reporting (nothing passes
--require-acknowledgement); it now runs on manual/release dispatches only.
- check-npm-lock resolved ~94 npm graphs against the registry on every push;
push/PR runs now use the reviewed --changed scoping (2.4s on dep-free
diffs) with a fail-open full sweep on base-fetch failure, and dispatches
keep --all for registry-drift coverage.
- Six restore-test-caches gates said 'github' only while the writer already
runs in hybrid; hybrid attempt-1 lanes now restore the transform seed.
- QA smoke lane reads the protected build compile cache; test-type stripes
widen to five.
Green tip run 31855637027 measured the remaining poles: ui-e2e shards up to
282s (imbalanced eight-way split), report-plugin-sdk-api-diff 246s on 4 vCPU,
and 220s test-type stripes (tsgo saturates a machine per graph, so stripe
count is the lever, not cores). Widen hosted-profile ui-e2e to ten rows,
test-type stripes to four, and lift the SDK diff to the 8 vCPU class.
First hybrid main cycles (runs 31850815388-31851455918) landed at 5:47-6:32
walls with every remaining pole a hosted lane: the three test-type stripes
(232-269s), check-additional-runtime-topology (274s), check-dependencies
(260s), and report-plugin-sdk-api-diff (209s). Route them to their Blacksmith
labels on hybrid attempt 1 (retries stay hosted) and lift runtime topology to
the 8 vCPU class.
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
* ci: codify manual-only Node 22 compatibility
* ci: split remaining hosted tail lanes
* fix(qa-lab): carry the validated part index into the plan id type
* test(ci): repair hosted tail lane guards
* ci: run push compacts without tooling lanes and stripe giants three-way
* fix(ci): keep CompactNodeTestPlanMode internal
knip --production flags the exported type as unused; no external module
imports it.
* ci: shard hosted lint and node tests
Split core oxlint across hosted runners and use backend-specific compact Node admission caps while preserving Blacksmith behavior.
* test(ci): guard plugin prerelease planner isolation
Measured canonical-main first-start delay: median 180s, average 209s; 10 of the last 30 runs waited more than 300s.
Approved tradeoff: up to ~+30 concurrent Blacksmith VMs during merge storms. Scope is canonical main pushes only; pull requests, manual runs, non-main refs, and forks keep their existing semantics.
Bump the canonical-main concurrency group from v7 to v8 so in-flight old-group runs cannot mix with the two-slot pipeline.
Dallin Romney