GitHub-hosted images carry Node 24.19.0 in /opt/hostedtoolcache, so hosted jobs
resolved from there and never populated the cached root. They still ran a
restore that could only miss and then a save whose path did not exist, logging
"Path Validation Error: Path(s) specified in the action for caching do(es) not
exist" on every hosted job. No entry was ever written, so nothing was poisoned,
but the warning is noise and the steps are pure waste on ~30 jobs per run.
Gate both steps on runner.environment != 'github-hosted', the signal this
workflow already uses for Blacksmith-only behavior, and gate the save on the
setup step reporting that a download actually populated the root. That second
guard also covers a future self-hosted image whose toolcache clears the floor.
Verified on Blacksmith: a satisfying image toolcache leaves the root absent so
the save is skipped, and a download populates it so the save runs.
Blacksmith's image tracks an older runner-images snapshot. Measured on a leased
box 2026-08-16, its toolcache holds Node 20.20.0, 22.22.0 and 24.13.0 while this
repo's engines floor is >=22.22.3 and >=24.15.0 -- short by three and two
patches. Every candidate is rejected, so all 306 of 306 sampled jobs fell
through to a nodejs.org download. GitHub-hosted runners carry 24.19.0 and
resolve from the toolcache in about a second, which is why only Blacksmith pays.
Normally that download is 2.6s (p99 3.3s), but ~46 jobs fetch the same 50 MB
simultaneously and the mirror throttles: three of 53 sampled runs had setup-node
medians of 44-93s with maxes to 139s, and because every job pays at once it
lands whole on the wall -- those runs went ~210s to 325s.
Keep the payload in the Actions cache, which Blacksmith serves from its
colocated backend. Measured on Blacksmith: cold 1605ms, warm 77ms.
Restores are prefix-keyed and the save carries the resolved patch. An exact key
would be worse than nothing: cache entries are immutable and an exact hit
suppresses the post-job save, so a floating `24.x` key would pin the first Node
it ever saw and, once the floor advanced past it, every job would restore the
rejected payload and re-download forever. Keying the save on the installed
version lets a newer resolve publish a new entry that later prefix restores
pick up.
A rejected payload is pruned before the replacement installs, because the entry
is saved wholesale and a leftover would ride along in every future save.
Windows keeps its existing path. Proven on Blacksmith across cold, warm, stale
and truncated-binary cases; both guards are mutation-checked.
This stays useful even if Blacksmith refreshes their image: the floor moves
independently of the snapshot, so the gap recurs. The image refresh is still
the better fix and is worth asking them for.
`checks-windows-node-test` ran the whole 222-266s Windows body in a single
Blacksmith lane, so every run that scheduled it finished at 287-334s against a
~180s plateau. Measured across 45 canonical main runs it was the wall in all
seven runs that triggered it (324-519s vs a 210s median elsewhere).
Blacksmith's Windows class admits exactly 2 concurrent jobs (run 31865243804),
which is why the earlier 3-part split failed: its tail queued 211s behind a
finished part. Pin the split width to that measured capacity instead. The two
parts are balanced by measured per-project wall time -- part 1 keeps the old
list 1 plus list 2's non-infra projects (~108s), part 2 takes list 2's infra
project plus the old list 3 (~112s) -- so both land near 113s and the lane
stops setting the wall.
One canonical partition now serves every backend; the github-mode/dispatch
3-part variant and its separate file inventory are gone, along with the
single-lane `task: test` case and the per-row `matrix.runner` the collapsed
lane needed. The complete 65-file Windows inventory is preserved exactly, and
the guards pin the width on both sides (mutation-checked: restoring 3 parts
fails the matrix guard).
`checks-ui-e2e-real-gateway` folded `hybrid` into the `github` breaker clause
instead of taking the attempt-gated route its sibling `checks-ui-e2e` uses, so
it was the only Chromium lane pinned to hosted capacity on attempt 1. Measured
across 46 canonical main runs it averaged 205s (max 225s) against a 150-190s
plateau, making it the run's slowest job whenever the compact Node matrix
behaved.
Route it like `checks-ui-e2e`: Blacksmith 16-vCPU on attempt 1, hosted on rerun,
manual dispatch, fork, and same-repo PR retry. Cache inputs stay on the shared
pnpm store for both backends, matching the sibling lane.
The guard now derives both Chromium lanes' `runs-on` from one pinned template so
a hosted-only divergence cannot return unnoticed; mutation-checked by reverting
the workflow clause (2 failures).
Fork pull requests were pinned to GitHub-hosted runners while maintainer
PRs rode Blacksmith. That split arrived with ci.yml's first commit
(f4f990a) and was never a reasoned posture -- no rationale in docs,
commit bodies, or the ci-limits skill -- and this week's Blacksmith
tranches widened the gap: maintainer walls are ~3:36 while the last fork
PR I measured (#124633) took 13m0s. We already paid for it once in
#118530, which raised the cross-repo artifact budget to 35 minutes
because contributor PR #117992 timed out twice on hosted capacity.
Runner choice now follows author_association: OWNER, MEMBER, COLLABORATOR,
and CONTRIBUTOR get Blacksmith; FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE,
and MANNEQUIN stay on hosted runners, which are free for public repos, so
an unreviewed author cannot spend Blacksmith capacity. Earning the fast
path requires a landed commit, which requires a maintainer merge.
Note for anyone tempted to trim that list: maintainers report CONTRIBUTOR
here, not OWNER/MEMBER, because org membership is concealed -- steipete's
74 sampled PRs are all CONTRIBUTOR. Dropping CONTRIBUTOR would move
maintainer PRs to hosted.
Scope is deliberately runner-only: 27 runs-on clauses. The 34
dependency-cache/use-actions-cache conditions and 6 job ifs stay
fork-gated, because cache poisoning is a different risk from runner
choice -- a fork run still never writes an archive a trusted run restores.
Verified by evaluating all 25 configurable runs-on expressions: maintainer
PR 12/25 Blacksmith, returning-contributor fork 12/25 (identical), unknown
author 0/25, FIRST_TIME_CONTRIBUTOR 0/25, push to main 12/25 unchanged.
Guard tests gained trusted/untrusted fork cases; 119 pass.
Removing build-artifacts' unread job outputs left the step still writing
four `*-result` step outputs and carrying an `id` that nothing
references. The wave already reports failures through ::error
annotations and its exit code, so these were pure leftovers.
Re-simulated the step body with stubbed pnpm/node: normal run exits 0, a
failing verifier still exits 1.
Two pieces of dead workflow surface, found auditing where CI time goes:
- ci-timings-summary was hard-disabled (`if: ${{ false && ... }}`) with a
TODO to re-enable or delete it after the next timing-optimization
review. That review happened; the local `pnpm ci:timings` helper is what
we actually use, and docs already pointed there. The job carried a
25-entry needs list that had to be kept in sync to stay lintable.
- build-artifacts exported four `*-result` outputs that no job or workflow
reads.
Removing the job lets the gate guard assert the stronger invariant it
wanted all along: ci-gate needs *every* job in the file, so a new lane
cannot slip in ungated (28 jobs, 27 gated, zero exceptions).
No runtime behavior changes: the job could never run and the outputs had
no consumers. Also audited every `pnpm <script>` and `node scripts/...`
reference in ci.yml for rot -- all resolve.
build-artifacts is the wall's pole in 4 of the last 5 main runs (171-186s,
~15-20s ahead of the next lane), so its serial steps are the wall. The
Doctor plugin-index proof, singleton smoke, and startup-memory check ran
as their own 13s step even though they are independent dist readers that
the 47s artifact-check wave could absorb.
They now run inside that wave: on Blacksmith all seven start together, so
the verifiers cost the wave's max instead of 13s of serial time; hosted
runners still serialize the three through run_verifier so the RSS ceiling
measures an unloaded process. The step drops its selection gate because
the verifiers always run -- each artifact check already self-gates on its
own RUN_* flag, so a run with no checks selected still verifies.
Proof: extracted the step body and ran it with stubbed pnpm/node. Both
modes behave (Blacksmith 7 checks started, hosted-with-nothing-selected
still runs the 3 verifiers), and a failing verifier exits 1 with its
::error annotation in both -- the wave cannot swallow it.
* fix(ui): reset logs when source changes
Reload the active tail without a cursor when logs.tail switches files so the Control UI never mixes sources or skips a new file prefix.
* fix(ui): reset log source after reconnect
* test(ui): harden logs lifecycle proof
* test(ui): route Logs lifecycle through real Gateway lane
Pin OCM v0.2.32 so Performance validation uses upstream native .mts discovery and transitive workspace dependency handling instead of the local Bash/tsx trampoline.
The warmer replays the same test envelopes as the compact shard lanes
through scripts/ci-run-node-test-shard.mts, but without the lanes'
NODE_OPTIONS=--max-old-space-size=8192. Run 31874567859 lost a worker in
agents-embedded-agent-run under the default heap and the failure skipped
every cache-save post step, leaving the day's transform/compile caches
unrefreshed. Rerun 31918467031 passed, confirming load-dependence;
matching the lane environment removes the gap.
Measured on run 31865243804: the 3-part split on
blacksmith-8vcpu-windows-2025 got exactly 2 runners; part 2 queued 211s
and only started when part 1 finished, pushing the wall to 6:45 despite
133-163s job bodies (vs 342s hosted). The runner class admits ~2
concurrent jobs, so any profile that can land on Blacksmith now uses the
single lane; the 3-part split stays for guaranteed-hosted executions
(github mode and dispatches, whose runner expression forces hosted in
every mode). A hybrid retry reruns the single lane hosted - slower but
bounded, and Windows-scoped retries are rare.
Three measured cuts toward the sub-4-minute wall:
- checks-ui-e2e hosted-planner matrix 10 -> 12 rows (11 Vitest shards +
browser-extension). Shard 1 measured 230s with ~123s test body; Vitest
hash sharding leaves count as the only balance lever.
- checks-windows hybrid attempt 1 now rides blacksmith-8vcpu-windows-2025
(retries stay hosted windows-2025). The 3-part hosted split measured a
342s pole on full-graph runs.
- build-artifacts folds the Doctor plugin-index proof (12s), singleton
smoke, and startup-memory check (7s) into one step that overlaps them
on Blacksmith and stays serial on hosted 4-core so the RSS measurement
is unperturbed; any missing startup asset rebuild completes before the
fork so no verifier reads dist mid-write.
Run 31861511624 made checks-ui the UI-scoped wall pole (247s job, 177s body
on hosted 4-core even in hybrid). Attempt 1 now uses the 8 vCPU class like
the ui-e2e shards; retries stay hosted.
The manifest planner closure and the protocol coverage script import only
node builtins and relative files (verified importing the full closure with
an empty node_modules under native type stripping), so push/PR preflight
drops the pnpm store restore and install (~30s off the barrier every lane
waits behind). Manual dispatches keep the tsx path for frozen targets, and
the coverage script inlines the record guard under the documented
dependency-free exception.
The store archive accretes every prior lockfile generation through
prefix-key restores (measured 2.05 GiB, ~36s restore in every hosted job);
the warmup writer now prunes to the current lockfile closure before saving.
Sub-agent caching audit tranche, all measured on runs 31856622489 and
31857193681:
- QA Smoke parts rebuilt the runtime twice and packed a docker tgz that no
scenario in the smoke coverage set consumes (verified against the resolved
plan: zero docker-lane, zero Control UI scenarios). One private overlay
build replaces the public-build/ui:build/pack/private-rebuild chain
(~35-50s per part); the run step fails closed if a docker-lane scenario
ever returns, and the guard now pins never-pack-after-private-build.
- report-plugin-sdk-api-diff is pure reporting (nothing passes
--require-acknowledgement); it now runs on manual/release dispatches only.
- check-npm-lock resolved ~94 npm graphs against the registry on every push;
push/PR runs now use the reviewed --changed scoping (2.4s on dep-free
diffs) with a fail-open full sweep on base-fetch failure, and dispatches
keep --all for registry-drift coverage.
- Six restore-test-caches gates said 'github' only while the writer already
runs in hybrid; hybrid attempt-1 lanes now restore the transform seed.
- QA smoke lane reads the protected build compile cache; test-type stripes
widen to five.
Green tip run 31855637027 measured the remaining poles: ui-e2e shards up to
282s (imbalanced eight-way split), report-plugin-sdk-api-diff 246s on 4 vCPU,
and 220s test-type stripes (tsgo saturates a machine per graph, so stripe
count is the lever, not cores). Widen hosted-profile ui-e2e to ten rows,
test-type stripes to four, and lift the SDK diff to the 8 vCPU class.
Each graph is a serial single-project build, so tsgo gains little past four
cores; run 31852071103 measured ~34s/graph on 8 vCPU versus ~42s on hosted
4-core. Overlapping two fresh shard processes per CI stripe (--concurrency 2)
uses the idle cores and roughly halves the stripe body; local runs keep the
heap-bounded serial default.
First hybrid main cycles (runs 31850815388-31851455918) landed at 5:47-6:32
walls with every remaining pole a hosted lane: the three test-type stripes
(232-269s), check-additional-runtime-topology (274s), check-dependencies
(260s), and report-plugin-sdk-api-diff (209s). Route them to their Blacksmith
labels on hybrid attempt 1 (retries stay hosted) and lift runtime topology to
the 8 vCPU class.
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
Peter Steinberger