Both halves of the docs publish starvation bug (openclaw/docs b4b130bd8,
#131155) were discovered by noticing stale published docs, not by any alert:
nothing watched whether the openclaw/docs mirror actually tracked docs-touching
commits on main. A broken sync (dead token, MDX failure, disabled workflow,
GitHub incident) would again go unnoticed.
Add a scheduled Docs Mirror Freshness check (every 30m) that parses the watched
paths from docs-sync-publish.yml itself - so the staleness definition can never
drift from the sync trigger - finds the newest docs-touching main commit via
the commits API, and compares it against the mirror's .openclaw-sync/source.json
SHA. When the mirror trails by more than 60 minutes it dispatches one
docs-sync-publish recovery run (skipped when a sync is already queued or in
progress) and fails the run so the workflow actor is notified.
The check lives in openclaw/openclaw rather than openclaw/docs because the path
filters it must judge against live here, and same-repo workflow_dispatch heals
with the default GITHUB_TOKEN instead of a cross-repo PAT.
Live-proven: fresh path against the real repos; stale path (detection,
active-run suppression, failing exit) via a simulated stale mirror SHA;
recovery dispatch endpoint via a real manual sync run (33113089552).
Every docs-touching push to main cancelled the in-flight docs-sync-publish
run (per-ref concurrency group with cancel-in-progress: true), and cancelled
runs leave no successor. Under sustained merge velocity the openclaw/docs
mirror never advanced: observed live 2026-08-27 as four consecutive
'completed cancelled' runs while the mirror sat 6 commits behind main.
Queue instead of cancel: GitHub keeps only the newest pending run per
concurrency group, so bursts still collapse to one follow-up run, and the
existing skip_stale_source guard keeps back-to-back completions idempotent.
This is the source-to-mirror half of the docs publish starvation bug; the
mirror-to-R2 half was fixed in openclaw/docs b4b130bd8.
Keep the post-merge locale refresh within GitHub API limits by resolving the exact commit and canonical locale targets together.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* fix(tooling): avoid shared tsx cache startup stalls
Use the shared preloader before tsx initializes so maintained build, check, and test commands retain memory transforms without scanning other checkouts' disk caches. Preserve changed-cwd forks and copied tooling closures.
* test: align command contracts with tooling bootstrap
Allow source-only locale contributions to load intentional English bootstrap copy until post-merge generated memory exists. Derive refresh targets from canonical locale data while preserving strict validation.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* ci(macos): budget macos-swift by runner so fork PRs finish
runs-on falls back to hosted macos-26 for several cases: workflow_dispatch,
run_attempt > 1, fork pull requests, and (as of the runner-backend circuit
breaker landed on main the same day this PR was authored) breaker-routed
GitHub-hosted fallback. timeout-minutes did not cover the fork-PR case, so
fork PRs ran on the slow hosted runner with the Blacksmith-tuned 20-minute
budget.
Measured on PR #118989 (a fork PR): macos-swift was cancelled after 20m25s,
killed mid-compile at step 1365/1416. No test executed, and the log has no
swift compile error - only 'The operation was canceled.' ci-gate then fails
because it lists macos-swift as required, so the PR reads red for a reason
unrelated to its diff, and a contributor cannot rerun it.
Make the budget follow the runner instead of the trigger: every hosted path
gets 30 (folded into the same budget the circuit breaker's own hosted-fallback
timeout extension already established for this job), so this doesn't
reintroduce a second, competing hosted-timeout value. Blacksmith paths keep
20 unchanged. Replaces the single pinned-string guard with a table-driven
test covering runs-on and timeout-minutes together across every trigger
context that can route to a hosted runner.
* chore: refresh PR head (keep open for maintainer review)
* ci(macos): track main's author-association runner routing
Main now routes macos-swift by pull_request author_association rather than
fork-ness, so the timeout predicate and its guard scenarios follow it.
* ci(macos): tighten hosted runner budget coverage
Co-authored-by: harjoth <harjoth.khara@gmail.com>
* docs(ci): document hosted macOS budgets without merge conflicts
Co-authored-by: harjoth <harjoth.khara@gmail.com>
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix(release): preserve VCR mirror source digests
Transport only attestation-verified digests across secret-scanned job outputs, reconstruct immutable GHCR refs inside the VCR mirror, and add an approved mirror-only recovery path.\n\nCloses #129466
* fix(release): verify VCR recovery sources
Revalidate attestations and release-version labels before any VCR registry write so manual recovery preserves the immutable source boundary.
* test(release): keep VCR regression scoped
Leave global workflow-to-test routing cleanup for a follow-up; this PR directly changes and runs both VCR regression suites without forcing metadata-complete CI.
* fix(ci): preserve caches after warmer failures
Finish every selected cache-warm group, save content-keyed transform and compile caches, then fail visibly after the save steps. Ordinary CI remains fail-fast.
Queue Telegram-visible Mantis proofs at workflow level so only one shared-user run allocates a runner at a time.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* fix(scripts): size the tsdown heap from the build's own cgroup budget
The build heap probe only read the cgroup root (/sys/fs/cgroup/memory.max and
the v1 equivalent). Those files exist only when the process runs in a
namespaced container cgroup; under systemd the budget lives on the process's
own slice, and the v2 root carries no limit at all. So every systemd-managed
build found no limit, fell back to /proc/meminfo MemTotal, and took the full
12288 MB default heap regardless of its actual budget.
Observed on a 15.4 GiB host: openclaw-main-update.service ran tsdown with
NODE_OPTIONS=--max-old-space-size=12288 while its user@999.service slice was
bounded at 5 GiB, reaching 3.2 GB RSS and 6.25 GB peak before the host began
OOM-killing unrelated services.
Resolve the limit from /proc/self/cgroup and walk that chain instead, reading
memory.high alongside memory.max (memory.high throttles reclaim rather than
failing allocation, so a heap above it stalls the build instead of OOM-ing),
and take the tightest bound found. Root paths stay as the container fallback,
and an explicitly injected path list still disables detection.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve the build heap budget from the v1 memory controller too
The slice walk only accepted the unified 0:: record, so a legacy or hybrid
systemd host fell back to the root probe and kept taking host memory. One
resolver now walks both hierarchies leaf-to-root, which makes the static root
list its own depth-0 case and removes it.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): read cgroup controller mounts instead of assuming their paths
v1 controllers can be co-mounted at the cgroup root, where memory.limit_in_bytes
sits under the slice with no per-controller directory, so the hardcoded
/sys/fs/cgroup/memory probe missed the budget and the build took the full
12288MB default. Mount points now come from mountinfo.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): translate cgroup records through the mount root
mountinfo field 4 is the subtree a cgroupfs mount exposes. Under a container
mount the /proc/self/cgroup record stays host-absolute, so walking it verbatim
probed paths below the visible mount and the build fell back to host memory.
Records now translate through the mount root before the walk.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): skip cgroup mounts that cannot represent this process
Falling back to the mount root for a record outside the mount's subtree sized
the build from an unrelated cgroup: an inherited namespace clamped the heap to
the 2048MB floor from a foreign 1GiB limit. Non-representable mounts are now
skipped, and the blind root probe only runs when no memory record exists.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): keep every cgroup mount view, not just the last one seen
One hierarchy can be visible through several mounts and only some expose a
subtree containing this process. Retaining only the last view dropped the
budget whenever a non-representable bind view came later, sending the build
back to host MemTotal.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): decode octal-escaped mountinfo paths before matching cgroups
ClawSweeper P2 on 7e64ad61f7: the cgroup resolver compared mountinfo's mount
root and mount point verbatim. The kernel escapes space, tab, newline, and
backslash in those two fields, so any cgroup mounted under such a path never
matched, the bounded slice was missed, and heap sizing silently fell back to
host memory.
Decode both fields before matching. The decoder lives in scripts/lib beside the
other shared script helpers rather than inline, so the scripts program has one
copy rather than a new ad hoc one.
Regression test fails pre-fix: a v2 mount at "/sys/fs/cgroup\040dir" with a
5 GiB memory.high yields --max-old-space-size=12288 (host fallback) before the
fix and 4352 after.
Follow-up, deliberately not bundled here: src/infra/sqlite-wal.ts,
src/commands/doctor-state-integrity.ts, and src/plugins/bundled-source-overlays.ts
each carry their own private copy of this same decoder. Consolidating all four
into @openclaw/normalization-core is the right end state, but it touches a
shared package plus three core modules and belongs in its own reviewable change.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve cgroup-namespace-relative records to their mount
ClawSweeper P1 on d6fe49dd3f: inside a cgroup namespace /proc/self/cgroup
reports the namespace root ("0::/") while mountinfo field 4 stays the host
subtree the cgroupfs was mounted from ("/docker/<id>"). relativeCgroupPath then
found no prefix match and returned null; because a memory record had already
been seen, the root probe was skipped and the build fell back to host MemTotal.
A constrained container therefore missed its own budget entirely.
That namespace root is exactly what the mount exposes at its mount point, so it
resolves to "/" rather than failing closed.
Regression test fails pre-fix: a "0::/" record against a /docker/2f1a9c mount
root with a 5 GiB memory.max yields --max-old-space-size=12288 before the fix
and 4352 after.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): reject inherited cgroup mount views instead of guessing
ClawSweeper P1 on b4d200c5d2: the previous commit resolved a namespace-relative
record against any mount root, including the inherited views cgroup_namespaces(7)
documents, whose field-4 root reads "/..". Which cgroup such a view exposes is not
derivable from mountinfo, so probing it can size the build from an unrelated
cgroup's limit.
Reject non-canonical mount roots outright. An undecidable view now falls back to
host sizing, which is current main's behavior, rather than silently adopting the
wrong budget.
Regression test covers the "/.." inherited mount: it must yield host MemTotal
sizing, not the 5 GiB limit sitting behind that mount.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): fail closed on namespace-root records against non-root mounts
ClawSweeper P1 on 731d3bbc8e: a "0::/" record does not prove that a mount
rooted at some other subtree exposes this process's cgroup. Resolving that pair
could cap the build heap from an unrelated cgroup's limit.
Return no mapping for it. An undecidable pair now falls back to host sizing,
which is current main's behavior, so the failure mode is a missed optimisation
rather than a wrong budget. The "/.." inherited-mount rejection stays; this
covers the broader ambiguous mapping it did not.
The namespace-relative test is repointed accordingly: an unrelated mounted
subtree must yield host sizing, not that subtree's limit.
Net production change: none (4 lines swapped).
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(build): cap tsdown heap to the real budget and refuse hosts that cannot build
The 2048MB floor was applied on top of a discovered cgroup limit, so a small
container was handed a heap larger than it could honour. Measured in real
cgroups, that does not OOM-kill, it thrashes: a 1500MiB container sat pinned at
its ceiling for 10 minutes with oom_kill at 0, never finished the second of
eleven invocations, and starved every other process on the host.
Cap to the discovered budget, then refuse up front when that budget cannot hold
the build. The threshold is the whole-build peak, not a single pass: a full
eleven-invocation build peaks at 4730MiB, so a 5GiB slice completes while 4GiB
and 2816MiB slices are both killed partway through the third invocation.
The refusal runs before any output is cleaned, so a host that cannot rebuild
does not also lose the build it has.
* fix(build): harden tsdown heap admission
* fix(build): guard the default tsdown plan
* fix(build): preserve runtime-only Docker builds
* fix(build): admit only declaration cache misses
* fix(build): scope heap admission to real budgets
* fix(build): guard direct unified declarations
* fix(build): guard the canonical tsdown config
* fix(build): satisfy cache planning lint
* fix(gateway): release empty orphan leases
* fix(build): cap cgroup budget by host memory
* fix(build): serialize the canonical tsdown config
* test(build): freeze host memory fixtures
* fix(build): honor cgroup v1 soft limits
* fix(build): respect cgroup v1 hierarchy mode
* fix(build): admit unified runtime plans
* fix(build): admit every unified runtime path
* fix(build): collect repeated tsdown filters
* fix(build): ignore cgroup v1 soft limits
* fix(build): use explicit heap override as opt-in
* refactor(build): simplify memory admission
* fix(build): harden constrained build recovery
* fix(ci): prebuild runtime before real CLI shards
* fix(build): honor runtime-only runner environment
* fix(ci): satisfy tooling shard lint
* fix(mantis): enforce verdict-expectation coherence and publish agent analysis files
Mantis run 32619081130 on #127989 published an overall `pass` while its own
manifest recorded that the candidate expectation was not observed: per-lane
`status` was mechanical capture success and the agent's judgment lived only
in `expected` prose, so nothing reconciled the two before publication.
- `mantis-evidence.json` schemaVersion 2: each comparison lane carries a
required boolean `expectationMet`. The desktop agent sets it in the same
manifest edit as `expected`; mechanical producers (Telegram live, web UI,
Slack, Discord) derive it from lane status.
- `scripts/mantis/publish-pr-evidence.mjs` is the single enforcement owner:
it requires the booleans, recomputes `pass`/`outcome`, downgrades a
contradictory pass claim to `fail`, and renders a visible "verdict
downgraded" note. The desktop workflow invokes it with `--validate-only`
before upload or comment.
- Agent top-level `*.json`/`*.md` analysis files (assertions, comparisons,
recipe suggestion) now survive the quarantine rebuild and upload, so cited
evidence actually exists in the artifact.
* fix(mantis): derive expectations from trusted lane facts
Peter Steinberger