* fix(scripts): size the tsdown heap from the build's own cgroup budget
The build heap probe only read the cgroup root (/sys/fs/cgroup/memory.max and
the v1 equivalent). Those files exist only when the process runs in a
namespaced container cgroup; under systemd the budget lives on the process's
own slice, and the v2 root carries no limit at all. So every systemd-managed
build found no limit, fell back to /proc/meminfo MemTotal, and took the full
12288 MB default heap regardless of its actual budget.
Observed on a 15.4 GiB host: openclaw-main-update.service ran tsdown with
NODE_OPTIONS=--max-old-space-size=12288 while its user@999.service slice was
bounded at 5 GiB, reaching 3.2 GB RSS and 6.25 GB peak before the host began
OOM-killing unrelated services.
Resolve the limit from /proc/self/cgroup and walk that chain instead, reading
memory.high alongside memory.max (memory.high throttles reclaim rather than
failing allocation, so a heap above it stalls the build instead of OOM-ing),
and take the tightest bound found. Root paths stay as the container fallback,
and an explicitly injected path list still disables detection.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve the build heap budget from the v1 memory controller too
The slice walk only accepted the unified 0:: record, so a legacy or hybrid
systemd host fell back to the root probe and kept taking host memory. One
resolver now walks both hierarchies leaf-to-root, which makes the static root
list its own depth-0 case and removes it.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): read cgroup controller mounts instead of assuming their paths
v1 controllers can be co-mounted at the cgroup root, where memory.limit_in_bytes
sits under the slice with no per-controller directory, so the hardcoded
/sys/fs/cgroup/memory probe missed the budget and the build took the full
12288MB default. Mount points now come from mountinfo.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): translate cgroup records through the mount root
mountinfo field 4 is the subtree a cgroupfs mount exposes. Under a container
mount the /proc/self/cgroup record stays host-absolute, so walking it verbatim
probed paths below the visible mount and the build fell back to host memory.
Records now translate through the mount root before the walk.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): skip cgroup mounts that cannot represent this process
Falling back to the mount root for a record outside the mount's subtree sized
the build from an unrelated cgroup: an inherited namespace clamped the heap to
the 2048MB floor from a foreign 1GiB limit. Non-representable mounts are now
skipped, and the blind root probe only runs when no memory record exists.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): keep every cgroup mount view, not just the last one seen
One hierarchy can be visible through several mounts and only some expose a
subtree containing this process. Retaining only the last view dropped the
budget whenever a non-representable bind view came later, sending the build
back to host MemTotal.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): decode octal-escaped mountinfo paths before matching cgroups
ClawSweeper P2 on 7e64ad61f7: the cgroup resolver compared mountinfo's mount
root and mount point verbatim. The kernel escapes space, tab, newline, and
backslash in those two fields, so any cgroup mounted under such a path never
matched, the bounded slice was missed, and heap sizing silently fell back to
host memory.
Decode both fields before matching. The decoder lives in scripts/lib beside the
other shared script helpers rather than inline, so the scripts program has one
copy rather than a new ad hoc one.
Regression test fails pre-fix: a v2 mount at "/sys/fs/cgroup\040dir" with a
5 GiB memory.high yields --max-old-space-size=12288 (host fallback) before the
fix and 4352 after.
Follow-up, deliberately not bundled here: src/infra/sqlite-wal.ts,
src/commands/doctor-state-integrity.ts, and src/plugins/bundled-source-overlays.ts
each carry their own private copy of this same decoder. Consolidating all four
into @openclaw/normalization-core is the right end state, but it touches a
shared package plus three core modules and belongs in its own reviewable change.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve cgroup-namespace-relative records to their mount
ClawSweeper P1 on d6fe49dd3f: inside a cgroup namespace /proc/self/cgroup
reports the namespace root ("0::/") while mountinfo field 4 stays the host
subtree the cgroupfs was mounted from ("/docker/<id>"). relativeCgroupPath then
found no prefix match and returned null; because a memory record had already
been seen, the root probe was skipped and the build fell back to host MemTotal.
A constrained container therefore missed its own budget entirely.
That namespace root is exactly what the mount exposes at its mount point, so it
resolves to "/" rather than failing closed.
Regression test fails pre-fix: a "0::/" record against a /docker/2f1a9c mount
root with a 5 GiB memory.max yields --max-old-space-size=12288 before the fix
and 4352 after.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): reject inherited cgroup mount views instead of guessing
ClawSweeper P1 on b4d200c5d2: the previous commit resolved a namespace-relative
record against any mount root, including the inherited views cgroup_namespaces(7)
documents, whose field-4 root reads "/..". Which cgroup such a view exposes is not
derivable from mountinfo, so probing it can size the build from an unrelated
cgroup's limit.
Reject non-canonical mount roots outright. An undecidable view now falls back to
host sizing, which is current main's behavior, rather than silently adopting the
wrong budget.
Regression test covers the "/.." inherited mount: it must yield host MemTotal
sizing, not the 5 GiB limit sitting behind that mount.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): fail closed on namespace-root records against non-root mounts
ClawSweeper P1 on 731d3bbc8e: a "0::/" record does not prove that a mount
rooted at some other subtree exposes this process's cgroup. Resolving that pair
could cap the build heap from an unrelated cgroup's limit.
Return no mapping for it. An undecidable pair now falls back to host sizing,
which is current main's behavior, so the failure mode is a missed optimisation
rather than a wrong budget. The "/.." inherited-mount rejection stays; this
covers the broader ambiguous mapping it did not.
The namespace-relative test is repointed accordingly: an unrelated mounted
subtree must yield host sizing, not that subtree's limit.
Net production change: none (4 lines swapped).
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(build): cap tsdown heap to the real budget and refuse hosts that cannot build
The 2048MB floor was applied on top of a discovered cgroup limit, so a small
container was handed a heap larger than it could honour. Measured in real
cgroups, that does not OOM-kill, it thrashes: a 1500MiB container sat pinned at
its ceiling for 10 minutes with oom_kill at 0, never finished the second of
eleven invocations, and starved every other process on the host.
Cap to the discovered budget, then refuse up front when that budget cannot hold
the build. The threshold is the whole-build peak, not a single pass: a full
eleven-invocation build peaks at 4730MiB, so a 5GiB slice completes while 4GiB
and 2816MiB slices are both killed partway through the third invocation.
The refusal runs before any output is cleaned, so a host that cannot rebuild
does not also lose the build it has.
* fix(build): harden tsdown heap admission
* fix(build): guard the default tsdown plan
* fix(build): preserve runtime-only Docker builds
* fix(build): admit only declaration cache misses
* fix(build): scope heap admission to real budgets
* fix(build): guard direct unified declarations
* fix(build): guard the canonical tsdown config
* fix(build): satisfy cache planning lint
* fix(gateway): release empty orphan leases
* fix(build): cap cgroup budget by host memory
* fix(build): serialize the canonical tsdown config
* test(build): freeze host memory fixtures
* fix(build): honor cgroup v1 soft limits
* fix(build): respect cgroup v1 hierarchy mode
* fix(build): admit unified runtime plans
* fix(build): admit every unified runtime path
* fix(build): collect repeated tsdown filters
* fix(build): ignore cgroup v1 soft limits
* fix(build): use explicit heap override as opt-in
* refactor(build): simplify memory admission
* fix(build): harden constrained build recovery
* fix(ci): prebuild runtime before real CLI shards
* fix(build): honor runtime-only runner environment
* fix(ci): satisfy tooling shard lint
Give maintainers immediate visibility when Mantis is requested. Bare mentions now react, link the active run, and keep one run-owned status comment through proof, short-circuit, or failure.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
`checks-windows-node-test` ran the whole 222-266s Windows body in a single
Blacksmith lane, so every run that scheduled it finished at 287-334s against a
~180s plateau. Measured across 45 canonical main runs it was the wall in all
seven runs that triggered it (324-519s vs a 210s median elsewhere).
Blacksmith's Windows class admits exactly 2 concurrent jobs (run 31865243804),
which is why the earlier 3-part split failed: its tail queued 211s behind a
finished part. Pin the split width to that measured capacity instead. The two
parts are balanced by measured per-project wall time -- part 1 keeps the old
list 1 plus list 2's non-infra projects (~108s), part 2 takes list 2's infra
project plus the old list 3 (~112s) -- so both land near 113s and the lane
stops setting the wall.
One canonical partition now serves every backend; the github-mode/dispatch
3-part variant and its separate file inventory are gone, along with the
single-lane `task: test` case and the per-row `matrix.runner` the collapsed
lane needed. The complete 65-file Windows inventory is preserved exactly, and
the guards pin the width on both sides (mutation-checked: restoring 3 parts
fails the matrix guard).
`checks-ui-e2e-real-gateway` folded `hybrid` into the `github` breaker clause
instead of taking the attempt-gated route its sibling `checks-ui-e2e` uses, so
it was the only Chromium lane pinned to hosted capacity on attempt 1. Measured
across 46 canonical main runs it averaged 205s (max 225s) against a 150-190s
plateau, making it the run's slowest job whenever the compact Node matrix
behaved.
Route it like `checks-ui-e2e`: Blacksmith 16-vCPU on attempt 1, hosted on rerun,
manual dispatch, fork, and same-repo PR retry. Cache inputs stay on the shared
pnpm store for both backends, matching the sibling lane.
The guard now derives both Chromium lanes' `runs-on` from one pinned template so
a hosted-only divergence cannot return unnoticed; mutation-checked by reverting
the workflow clause (2 failures).
Fork pull requests were pinned to GitHub-hosted runners while maintainer
PRs rode Blacksmith. That split arrived with ci.yml's first commit
(f4f990a) and was never a reasoned posture -- no rationale in docs,
commit bodies, or the ci-limits skill -- and this week's Blacksmith
tranches widened the gap: maintainer walls are ~3:36 while the last fork
PR I measured (#124633) took 13m0s. We already paid for it once in
#118530, which raised the cross-repo artifact budget to 35 minutes
because contributor PR #117992 timed out twice on hosted capacity.
Runner choice now follows author_association: OWNER, MEMBER, COLLABORATOR,
and CONTRIBUTOR get Blacksmith; FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE,
and MANNEQUIN stay on hosted runners, which are free for public repos, so
an unreviewed author cannot spend Blacksmith capacity. Earning the fast
path requires a landed commit, which requires a maintainer merge.
Note for anyone tempted to trim that list: maintainers report CONTRIBUTOR
here, not OWNER/MEMBER, because org membership is concealed -- steipete's
74 sampled PRs are all CONTRIBUTOR. Dropping CONTRIBUTOR would move
maintainer PRs to hosted.
Scope is deliberately runner-only: 27 runs-on clauses. The 34
dependency-cache/use-actions-cache conditions and 6 job ifs stay
fork-gated, because cache poisoning is a different risk from runner
choice -- a fork run still never writes an archive a trusted run restores.
Verified by evaluating all 25 configurable runs-on expressions: maintainer
PR 12/25 Blacksmith, returning-contributor fork 12/25 (identical), unknown
author 0/25, FIRST_TIME_CONTRIBUTOR 0/25, push to main 12/25 unchanged.
Guard tests gained trusted/untrusted fork cases; 119 pass.
Two pieces of dead workflow surface, found auditing where CI time goes:
- ci-timings-summary was hard-disabled (`if: ${{ false && ... }}`) with a
TODO to re-enable or delete it after the next timing-optimization
review. That review happened; the local `pnpm ci:timings` helper is what
we actually use, and docs already pointed there. The job carried a
25-entry needs list that had to be kept in sync to stay lintable.
- build-artifacts exported four `*-result` outputs that no job or workflow
reads.
Removing the job lets the gate guard assert the stronger invariant it
wanted all along: ci-gate needs *every* job in the file, so a new lane
cannot slip in ungated (28 jobs, 27 gated, zero exceptions).
No runtime behavior changes: the job could never run and the outputs had
no consumers. Also audited every `pnpm <script>` and `node scripts/...`
reference in ci.yml for rot -- all resolve.
build-artifacts is the wall's pole in 4 of the last 5 main runs (171-186s,
~15-20s ahead of the next lane), so its serial steps are the wall. The
Doctor plugin-index proof, singleton smoke, and startup-memory check ran
as their own 13s step even though they are independent dist readers that
the 47s artifact-check wave could absorb.
They now run inside that wave: on Blacksmith all seven start together, so
the verifiers cost the wave's max instead of 13s of serial time; hosted
runners still serialize the three through run_verifier so the RSS ceiling
measures an unloaded process. The step drops its selection gate because
the verifiers always run -- each artifact check already self-gates on its
own RUN_* flag, so a run with no checks selected still verifies.
Proof: extracted the step body and ran it with stubbed pnpm/node. Both
modes behave (Blacksmith 7 checks started, hosted-with-nothing-selected
still runs the 3 verifiers), and a failing verifier exits 1 with its
::error annotation in both -- the wave cannot swallow it.
* fix(ui): reset logs when source changes
Reload the active tail without a cursor when logs.tail switches files so the Control UI never mixes sources or skips a new file prefix.
* fix(ui): reset log source after reconnect
* test(ui): harden logs lifecycle proof
* test(ui): route Logs lifecycle through real Gateway lane
Measured on run 31865243804: the 3-part split on
blacksmith-8vcpu-windows-2025 got exactly 2 runners; part 2 queued 211s
and only started when part 1 finished, pushing the wall to 6:45 despite
133-163s job bodies (vs 342s hosted). The runner class admits ~2
concurrent jobs, so any profile that can land on Blacksmith now uses the
single lane; the 3-part split stays for guaranteed-hosted executions
(github mode and dispatches, whose runner expression forces hosted in
every mode). A hybrid retry reruns the single lane hosted - slower but
bounded, and Windows-scoped retries are rare.
Three measured cuts toward the sub-4-minute wall:
- checks-ui-e2e hosted-planner matrix 10 -> 12 rows (11 Vitest shards +
browser-extension). Shard 1 measured 230s with ~123s test body; Vitest
hash sharding leaves count as the only balance lever.
- checks-windows hybrid attempt 1 now rides blacksmith-8vcpu-windows-2025
(retries stay hosted windows-2025). The 3-part hosted split measured a
342s pole on full-graph runs.
- build-artifacts folds the Doctor plugin-index proof (12s), singleton
smoke, and startup-memory check (7s) into one step that overlaps them
on Blacksmith and stays serial on hosted 4-core so the RSS measurement
is unperturbed; any missing startup asset rebuild completes before the
fork so no verifier reads dist mid-write.
Run 31861511624 made checks-ui the UI-scoped wall pole (247s job, 177s body
on hosted 4-core even in hybrid). Attempt 1 now uses the 8 vCPU class like
the ui-e2e shards; retries stay hosted.
The manifest planner closure and the protocol coverage script import only
node builtins and relative files (verified importing the full closure with
an empty node_modules under native type stripping), so push/PR preflight
drops the pnpm store restore and install (~30s off the barrier every lane
waits behind). Manual dispatches keep the tsx path for frozen targets, and
the coverage script inlines the record guard under the documented
dependency-free exception.
The store archive accretes every prior lockfile generation through
prefix-key restores (measured 2.05 GiB, ~36s restore in every hosted job);
the warmup writer now prunes to the current lockfile closure before saving.
Sub-agent caching audit tranche, all measured on runs 31856622489 and
31857193681:
- QA Smoke parts rebuilt the runtime twice and packed a docker tgz that no
scenario in the smoke coverage set consumes (verified against the resolved
plan: zero docker-lane, zero Control UI scenarios). One private overlay
build replaces the public-build/ui:build/pack/private-rebuild chain
(~35-50s per part); the run step fails closed if a docker-lane scenario
ever returns, and the guard now pins never-pack-after-private-build.
- report-plugin-sdk-api-diff is pure reporting (nothing passes
--require-acknowledgement); it now runs on manual/release dispatches only.
- check-npm-lock resolved ~94 npm graphs against the registry on every push;
push/PR runs now use the reviewed --changed scoping (2.4s on dep-free
diffs) with a fail-open full sweep on base-fetch failure, and dispatches
keep --all for registry-drift coverage.
- Six restore-test-caches gates said 'github' only while the writer already
runs in hybrid; hybrid attempt-1 lanes now restore the transform seed.
- QA smoke lane reads the protected build compile cache; test-type stripes
widen to five.
Green tip run 31855637027 measured the remaining poles: ui-e2e shards up to
282s (imbalanced eight-way split), report-plugin-sdk-api-diff 246s on 4 vCPU,
and 220s test-type stripes (tsgo saturates a machine per graph, so stripe
count is the lever, not cores). Widen hosted-profile ui-e2e to ten rows,
test-type stripes to four, and lift the SDK diff to the 8 vCPU class.
First hybrid main cycles (runs 31850815388-31851455918) landed at 5:47-6:32
walls with every remaining pole a hosted lane: the three test-type stripes
(232-269s), check-additional-runtime-topology (274s), check-dependencies
(260s), and report-plugin-sdk-api-diff (209s). Route them to their Blacksmith
labels on hybrid attempt 1 (retries stay hosted) and lift runtime topology to
the 8 vCPU class.
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
Peter Steinberger