* fix(scripts): size the tsdown heap from the build's own cgroup budget
The build heap probe only read the cgroup root (/sys/fs/cgroup/memory.max and
the v1 equivalent). Those files exist only when the process runs in a
namespaced container cgroup; under systemd the budget lives on the process's
own slice, and the v2 root carries no limit at all. So every systemd-managed
build found no limit, fell back to /proc/meminfo MemTotal, and took the full
12288 MB default heap regardless of its actual budget.
Observed on a 15.4 GiB host: openclaw-main-update.service ran tsdown with
NODE_OPTIONS=--max-old-space-size=12288 while its user@999.service slice was
bounded at 5 GiB, reaching 3.2 GB RSS and 6.25 GB peak before the host began
OOM-killing unrelated services.
Resolve the limit from /proc/self/cgroup and walk that chain instead, reading
memory.high alongside memory.max (memory.high throttles reclaim rather than
failing allocation, so a heap above it stalls the build instead of OOM-ing),
and take the tightest bound found. Root paths stay as the container fallback,
and an explicitly injected path list still disables detection.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve the build heap budget from the v1 memory controller too
The slice walk only accepted the unified 0:: record, so a legacy or hybrid
systemd host fell back to the root probe and kept taking host memory. One
resolver now walks both hierarchies leaf-to-root, which makes the static root
list its own depth-0 case and removes it.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): read cgroup controller mounts instead of assuming their paths
v1 controllers can be co-mounted at the cgroup root, where memory.limit_in_bytes
sits under the slice with no per-controller directory, so the hardcoded
/sys/fs/cgroup/memory probe missed the budget and the build took the full
12288MB default. Mount points now come from mountinfo.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): translate cgroup records through the mount root
mountinfo field 4 is the subtree a cgroupfs mount exposes. Under a container
mount the /proc/self/cgroup record stays host-absolute, so walking it verbatim
probed paths below the visible mount and the build fell back to host memory.
Records now translate through the mount root before the walk.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): skip cgroup mounts that cannot represent this process
Falling back to the mount root for a record outside the mount's subtree sized
the build from an unrelated cgroup: an inherited namespace clamped the heap to
the 2048MB floor from a foreign 1GiB limit. Non-representable mounts are now
skipped, and the blind root probe only runs when no memory record exists.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): keep every cgroup mount view, not just the last one seen
One hierarchy can be visible through several mounts and only some expose a
subtree containing this process. Retaining only the last view dropped the
budget whenever a non-representable bind view came later, sending the build
back to host MemTotal.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): decode octal-escaped mountinfo paths before matching cgroups
ClawSweeper P2 on 7e64ad61f7: the cgroup resolver compared mountinfo's mount
root and mount point verbatim. The kernel escapes space, tab, newline, and
backslash in those two fields, so any cgroup mounted under such a path never
matched, the bounded slice was missed, and heap sizing silently fell back to
host memory.
Decode both fields before matching. The decoder lives in scripts/lib beside the
other shared script helpers rather than inline, so the scripts program has one
copy rather than a new ad hoc one.
Regression test fails pre-fix: a v2 mount at "/sys/fs/cgroup\040dir" with a
5 GiB memory.high yields --max-old-space-size=12288 (host fallback) before the
fix and 4352 after.
Follow-up, deliberately not bundled here: src/infra/sqlite-wal.ts,
src/commands/doctor-state-integrity.ts, and src/plugins/bundled-source-overlays.ts
each carry their own private copy of this same decoder. Consolidating all four
into @openclaw/normalization-core is the right end state, but it touches a
shared package plus three core modules and belongs in its own reviewable change.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve cgroup-namespace-relative records to their mount
ClawSweeper P1 on d6fe49dd3f: inside a cgroup namespace /proc/self/cgroup
reports the namespace root ("0::/") while mountinfo field 4 stays the host
subtree the cgroupfs was mounted from ("/docker/<id>"). relativeCgroupPath then
found no prefix match and returned null; because a memory record had already
been seen, the root probe was skipped and the build fell back to host MemTotal.
A constrained container therefore missed its own budget entirely.
That namespace root is exactly what the mount exposes at its mount point, so it
resolves to "/" rather than failing closed.
Regression test fails pre-fix: a "0::/" record against a /docker/2f1a9c mount
root with a 5 GiB memory.max yields --max-old-space-size=12288 before the fix
and 4352 after.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): reject inherited cgroup mount views instead of guessing
ClawSweeper P1 on b4d200c5d2: the previous commit resolved a namespace-relative
record against any mount root, including the inherited views cgroup_namespaces(7)
documents, whose field-4 root reads "/..". Which cgroup such a view exposes is not
derivable from mountinfo, so probing it can size the build from an unrelated
cgroup's limit.
Reject non-canonical mount roots outright. An undecidable view now falls back to
host sizing, which is current main's behavior, rather than silently adopting the
wrong budget.
Regression test covers the "/.." inherited mount: it must yield host MemTotal
sizing, not the 5 GiB limit sitting behind that mount.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): fail closed on namespace-root records against non-root mounts
ClawSweeper P1 on 731d3bbc8e: a "0::/" record does not prove that a mount
rooted at some other subtree exposes this process's cgroup. Resolving that pair
could cap the build heap from an unrelated cgroup's limit.
Return no mapping for it. An undecidable pair now falls back to host sizing,
which is current main's behavior, so the failure mode is a missed optimisation
rather than a wrong budget. The "/.." inherited-mount rejection stays; this
covers the broader ambiguous mapping it did not.
The namespace-relative test is repointed accordingly: an unrelated mounted
subtree must yield host sizing, not that subtree's limit.
Net production change: none (4 lines swapped).
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(build): cap tsdown heap to the real budget and refuse hosts that cannot build
The 2048MB floor was applied on top of a discovered cgroup limit, so a small
container was handed a heap larger than it could honour. Measured in real
cgroups, that does not OOM-kill, it thrashes: a 1500MiB container sat pinned at
its ceiling for 10 minutes with oom_kill at 0, never finished the second of
eleven invocations, and starved every other process on the host.
Cap to the discovered budget, then refuse up front when that budget cannot hold
the build. The threshold is the whole-build peak, not a single pass: a full
eleven-invocation build peaks at 4730MiB, so a 5GiB slice completes while 4GiB
and 2816MiB slices are both killed partway through the third invocation.
The refusal runs before any output is cleaned, so a host that cannot rebuild
does not also lose the build it has.
* fix(build): harden tsdown heap admission
* fix(build): guard the default tsdown plan
* fix(build): preserve runtime-only Docker builds
* fix(build): admit only declaration cache misses
* fix(build): scope heap admission to real budgets
* fix(build): guard direct unified declarations
* fix(build): guard the canonical tsdown config
* fix(build): satisfy cache planning lint
* fix(gateway): release empty orphan leases
* fix(build): cap cgroup budget by host memory
* fix(build): serialize the canonical tsdown config
* test(build): freeze host memory fixtures
* fix(build): honor cgroup v1 soft limits
* fix(build): respect cgroup v1 hierarchy mode
* fix(build): admit unified runtime plans
* fix(build): admit every unified runtime path
* fix(build): collect repeated tsdown filters
* fix(build): ignore cgroup v1 soft limits
* fix(build): use explicit heap override as opt-in
* refactor(build): simplify memory admission
* fix(build): harden constrained build recovery
* fix(ci): prebuild runtime before real CLI shards
* fix(build): honor runtime-only runner environment
* fix(ci): satisfy tooling shard lint
* fix(ci): stop codex lane cold-graph hangs
The side-question domain-policy test loaded the complete agent-harness tool graph inside a one-second readiness race, making the serial non-isolated Codex shard fail or stay silent under cold imports. Build the test's web_search marker and real web_fetch tool from the narrow implementation, then synchronize on turn startup before issuing the tool call. Cap each Codex test process at 12 files so CI gets bounded time-to-first-output as defense in depth.\n\nRefs #125839
* fix(test): keep codex web fetch fixture on sdk boundary
Load the real web_fetch factory on demand through the existing local-only plugin test runtime. This preserves the narrow cold-graph fix without letting a bundled plugin test reach into core internals.
* fix(agents): finalize guided creation safely
Run channel post-write hooks only after config publication, defer portable auth copying until agent creation succeeds without overwriting newer credentials, and provision existing workspaces before publishing updates.
Keep JSON-only guided creation interactive while routing wizard output to stderr so stdout remains one machine-readable summary.
* fix(terminal): preserve note call signature
* fix(agents): pass committed config to setup hooks
* ci: split heavy codex changed-test shards
Cap non-isolated Codex extension processes at 20 files so 4-vCPU changed-target jobs do not starve real-time watches or hit the no-output watchdog.
* test(ci): align codex shard cap fixture
* docs(cli): clarify agents add JSON mode
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
Five-file Telegram jobs finished the first file, then isolate re-imported the next graph in silence until the 300s watchdog killed the worker. Recycle the Vitest process after each file and keep five files per CI job.
The PR-changed test planner fails safe to the compact full-suite plan for any diff touching packages/**, but that compact plan excludes all extension test configs, so mixed package+extension PRs landed with zero extension test execution (escapes: PR #120534 breaking extensions/codex run-attempt.native-hook-relay.test.ts, PRs #122163/#121522 and cd7b7f639d breaking media-understanding-provider.test.ts and thread-lifecycle.test.ts on main full runs). The preflight now appends whole-config shards for the diff's touched extensions whenever the precise plan fails safe; whole configs (not precise targets) because the fail-safe cause leaves the non-extension diff's extension impact unbounded.
* refactor: use canonical re-export names
* fix(test): classify suite support as test source
* fix(agents): retarget gateway stub session-entry import
* test(gateway): retarget session-utils mock keys after alias removal
* test(gateway): boot a minimal test gateway in the gateway lane
A gateway startup stall (#120926, awaited chat-metadata refresh in minimal
mode) shipped with green CI and first surfaced by hanging every checks-ui-e2e
suite that boots a minimal test gateway: no gateway-lane test booted one with
bundled plugins enabled (gateway.test.ts disables them, which masked the
stall), and the changed-scope classifier only selects the ui-e2e lane for
ui-touching diffs.
Add a minimal-gateway boot smoke in src/gateway that mirrors the ui-e2e boot
environment (bundled plugins enabled, minimal skips, strict time budget).
Selection is automatic: the smoke imports the gateway server, so the changed
node test plan picks it up through the import graph for any diff that can
affect startup, and full-suite plans run it in the
agentic-control-plane-startup-core shard. Reintroducing the #120926 hunk makes
the smoke fail (watchdog kill) in the gateway lane; healthy code boots in ~10s.
A planner regression test keeps the smoke classified as a gateway-server test
file and import-graph-reachable from gateway startup sources, so a rename or
graph-invisible import shape cannot silently drop the coverage again.
* test(ui): resolve labs rows by registry id, not positional index
Unblocks landing onto red main: 8fdf7570a1 (#120727) added the Cloud Worker
Desktop labs entry with a hardcoded row index that collides with Message audit
metadata, so enabling it toggled the neighboring row and the full-suite lane
failed deterministically (labs-page.test.ts, exposed only on PRs that run the
compact full suite). Derive each table case's row index from LAB_FEATURES by
feature id — the idiom the rest of the suite already uses — so a new labs
entry can no longer silently retarget an existing case.
* fix(ci): address reviewer findings across the perf-lever batch
Seven follow-ups from Codex review feedback on the landed CI perf PRs:
docker packaging switches to the full build profile (the ciArtifacts
step env clobbered its canonical-dts requirement); the boundary sticky
restore gates on generative source tree OIDs in both lanes, making
restored artifacts valid by construction (kills the clock-skew mtime
false-skip and ghost declarations from deleted sources); the node-test
worker budget derives from nproc instead of the runner label (labels
over-promise under load) and keeps the 2-worker cap when plans can
overlap; the deadcode task falls back to serial Knip below 8 cores;
the lint-lane i18n verify also runs for any ui-touching diff (keys are
extracted from all ui sources, not just ui/src/i18n); the oxlint
concurrency diagnostic moves to stderr; and workspace packages join
the prompt-snapshot always-run surface.
* fix(ci): fingerprint every generative source the boundary artifacts consume
Autoreview flagged inputs outside the tree-OID gate; toolchain, config,
and script inputs were already covered by the sticky key hash, but the
old actions/cache key also fingerprinted src/plugins/types.ts and
src/video-generation sources the OID list missed. Add those plus
src/channels (the plugin-sdk declaration graph reaches its public
types) to the fingerprint in all three sites.
* fix(ci): fold runner toolchain identity into the boundary sticky fingerprint
Autoreview: the sticky key hashes config/scripts/package.json/lockfile,
but a node/pnpm toolchain bump (pinned in the workflow, not the key)
could leave the key unchanged and bless artifacts built with the prior
toolchain. Record node/pnpm versions alongside the source tree OIDs in
all three fingerprint sites so a toolchain-only change forces a cold
build.
check-prompt-snapshots costs 200-310s per PR regenerating real prompt
stacks, but snapshots only change when the generator's import graph or
its fixtures do. The manifest now computes that reach (same pattern as
QA smoke gating: always-run surface regex incl. the codex extension
whose test API loads via a dynamic module id, plus import-graph walk
from the snapshot helper; deletions and null diffs fail safe to
running). Unaffected PR diffs skip the lane with an explicit log line;
pushes and dispatches always run.
* perf(ci): diff-gate QA smoke by CLI import graph and isolate prompt snapshots lane
* fix(ci): run prompt snapshot lane unconditionally
* fix(ci): treat QA lane orchestration files as QA-impacting
Jesse Merhi