Queue Telegram-visible Mantis proofs at workflow level so only one shared-user run allocates a runner at a time.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* fix(scripts): size the tsdown heap from the build's own cgroup budget
The build heap probe only read the cgroup root (/sys/fs/cgroup/memory.max and
the v1 equivalent). Those files exist only when the process runs in a
namespaced container cgroup; under systemd the budget lives on the process's
own slice, and the v2 root carries no limit at all. So every systemd-managed
build found no limit, fell back to /proc/meminfo MemTotal, and took the full
12288 MB default heap regardless of its actual budget.
Observed on a 15.4 GiB host: openclaw-main-update.service ran tsdown with
NODE_OPTIONS=--max-old-space-size=12288 while its user@999.service slice was
bounded at 5 GiB, reaching 3.2 GB RSS and 6.25 GB peak before the host began
OOM-killing unrelated services.
Resolve the limit from /proc/self/cgroup and walk that chain instead, reading
memory.high alongside memory.max (memory.high throttles reclaim rather than
failing allocation, so a heap above it stalls the build instead of OOM-ing),
and take the tightest bound found. Root paths stay as the container fallback,
and an explicitly injected path list still disables detection.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve the build heap budget from the v1 memory controller too
The slice walk only accepted the unified 0:: record, so a legacy or hybrid
systemd host fell back to the root probe and kept taking host memory. One
resolver now walks both hierarchies leaf-to-root, which makes the static root
list its own depth-0 case and removes it.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): read cgroup controller mounts instead of assuming their paths
v1 controllers can be co-mounted at the cgroup root, where memory.limit_in_bytes
sits under the slice with no per-controller directory, so the hardcoded
/sys/fs/cgroup/memory probe missed the budget and the build took the full
12288MB default. Mount points now come from mountinfo.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): translate cgroup records through the mount root
mountinfo field 4 is the subtree a cgroupfs mount exposes. Under a container
mount the /proc/self/cgroup record stays host-absolute, so walking it verbatim
probed paths below the visible mount and the build fell back to host memory.
Records now translate through the mount root before the walk.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): skip cgroup mounts that cannot represent this process
Falling back to the mount root for a record outside the mount's subtree sized
the build from an unrelated cgroup: an inherited namespace clamped the heap to
the 2048MB floor from a foreign 1GiB limit. Non-representable mounts are now
skipped, and the blind root probe only runs when no memory record exists.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): keep every cgroup mount view, not just the last one seen
One hierarchy can be visible through several mounts and only some expose a
subtree containing this process. Retaining only the last view dropped the
budget whenever a non-representable bind view came later, sending the build
back to host MemTotal.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): decode octal-escaped mountinfo paths before matching cgroups
ClawSweeper P2 on 7e64ad61f7: the cgroup resolver compared mountinfo's mount
root and mount point verbatim. The kernel escapes space, tab, newline, and
backslash in those two fields, so any cgroup mounted under such a path never
matched, the bounded slice was missed, and heap sizing silently fell back to
host memory.
Decode both fields before matching. The decoder lives in scripts/lib beside the
other shared script helpers rather than inline, so the scripts program has one
copy rather than a new ad hoc one.
Regression test fails pre-fix: a v2 mount at "/sys/fs/cgroup\040dir" with a
5 GiB memory.high yields --max-old-space-size=12288 (host fallback) before the
fix and 4352 after.
Follow-up, deliberately not bundled here: src/infra/sqlite-wal.ts,
src/commands/doctor-state-integrity.ts, and src/plugins/bundled-source-overlays.ts
each carry their own private copy of this same decoder. Consolidating all four
into @openclaw/normalization-core is the right end state, but it touches a
shared package plus three core modules and belongs in its own reviewable change.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): resolve cgroup-namespace-relative records to their mount
ClawSweeper P1 on d6fe49dd3f: inside a cgroup namespace /proc/self/cgroup
reports the namespace root ("0::/") while mountinfo field 4 stays the host
subtree the cgroupfs was mounted from ("/docker/<id>"). relativeCgroupPath then
found no prefix match and returned null; because a memory record had already
been seen, the root probe was skipped and the build fell back to host MemTotal.
A constrained container therefore missed its own budget entirely.
That namespace root is exactly what the mount exposes at its mount point, so it
resolves to "/" rather than failing closed.
Regression test fails pre-fix: a "0::/" record against a /docker/2f1a9c mount
root with a 5 GiB memory.max yields --max-old-space-size=12288 before the fix
and 4352 after.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): reject inherited cgroup mount views instead of guessing
ClawSweeper P1 on b4d200c5d2: the previous commit resolved a namespace-relative
record against any mount root, including the inherited views cgroup_namespaces(7)
documents, whose field-4 root reads "/..". Which cgroup such a view exposes is not
derivable from mountinfo, so probing it can size the build from an unrelated
cgroup's limit.
Reject non-canonical mount roots outright. An undecidable view now falls back to
host sizing, which is current main's behavior, rather than silently adopting the
wrong budget.
Regression test covers the "/.." inherited mount: it must yield host MemTotal
sizing, not the 5 GiB limit sitting behind that mount.
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(scripts): fail closed on namespace-root records against non-root mounts
ClawSweeper P1 on 731d3bbc8e: a "0::/" record does not prove that a mount
rooted at some other subtree exposes this process's cgroup. Resolving that pair
could cap the build heap from an unrelated cgroup's limit.
Return no mapping for it. An undecidable pair now falls back to host sizing,
which is current main's behavior, so the failure mode is a missed optimisation
rather than a wrong budget. The "/.." inherited-mount rejection stays; this
covers the broader ambiguous mapping it did not.
The namespace-relative test is repointed accordingly: an unrelated mounted
subtree must yield host sizing, not that subtree's limit.
Net production change: none (4 lines swapped).
Co-authored-by: jesse-merhi <79823012+jesse-merhi@users.noreply.github.com>
* fix(build): cap tsdown heap to the real budget and refuse hosts that cannot build
The 2048MB floor was applied on top of a discovered cgroup limit, so a small
container was handed a heap larger than it could honour. Measured in real
cgroups, that does not OOM-kill, it thrashes: a 1500MiB container sat pinned at
its ceiling for 10 minutes with oom_kill at 0, never finished the second of
eleven invocations, and starved every other process on the host.
Cap to the discovered budget, then refuse up front when that budget cannot hold
the build. The threshold is the whole-build peak, not a single pass: a full
eleven-invocation build peaks at 4730MiB, so a 5GiB slice completes while 4GiB
and 2816MiB slices are both killed partway through the third invocation.
The refusal runs before any output is cleaned, so a host that cannot rebuild
does not also lose the build it has.
* fix(build): harden tsdown heap admission
* fix(build): guard the default tsdown plan
* fix(build): preserve runtime-only Docker builds
* fix(build): admit only declaration cache misses
* fix(build): scope heap admission to real budgets
* fix(build): guard direct unified declarations
* fix(build): guard the canonical tsdown config
* fix(build): satisfy cache planning lint
* fix(gateway): release empty orphan leases
* fix(build): cap cgroup budget by host memory
* fix(build): serialize the canonical tsdown config
* test(build): freeze host memory fixtures
* fix(build): honor cgroup v1 soft limits
* fix(build): respect cgroup v1 hierarchy mode
* fix(build): admit unified runtime plans
* fix(build): admit every unified runtime path
* fix(build): collect repeated tsdown filters
* fix(build): ignore cgroup v1 soft limits
* fix(build): use explicit heap override as opt-in
* refactor(build): simplify memory admission
* fix(build): harden constrained build recovery
* fix(ci): prebuild runtime before real CLI shards
* fix(build): honor runtime-only runner environment
* fix(ci): satisfy tooling shard lint
* fix(mantis): enforce verdict-expectation coherence and publish agent analysis files
Mantis run 32619081130 on #127989 published an overall `pass` while its own
manifest recorded that the candidate expectation was not observed: per-lane
`status` was mechanical capture success and the agent's judgment lived only
in `expected` prose, so nothing reconciled the two before publication.
- `mantis-evidence.json` schemaVersion 2: each comparison lane carries a
required boolean `expectationMet`. The desktop agent sets it in the same
manifest edit as `expected`; mechanical producers (Telegram live, web UI,
Slack, Discord) derive it from lane status.
- `scripts/mantis/publish-pr-evidence.mjs` is the single enforcement owner:
it requires the booleans, recomputes `pass`/`outcome`, downgrades a
contradictory pass claim to `fail`, and renders a visible "verdict
downgraded" note. The desktop workflow invokes it with `--validate-only`
before upload or comment.
- Agent top-level `*.json`/`*.md` analysis files (assertions, comparisons,
recipe suggestion) now survive the quarantine rebuild and upload, so cited
evidence actually exists in the artifact.
* fix(mantis): derive expectations from trusted lane facts
* feat(mantis): add exec and restart lane primitives
Give the proof agent a developer shell inside each SUT container and an
in-container gateway restart so it can design scenarios like a local
developer: patch openclaw.json and restart, stage plugins and fixtures,
run node/tsx against the read-only repo root, inspect SQLite state.
- container script: exec (docker exec as mantis-sut, bounded by timeout),
restart (request file + TERM), sut_command becomes a relaunch supervisor
- lane CLI: exec returns bounded stdout/stderr/exitCode and records a
redacted invocation; restart waits for a fresh [gateway] ready marker
- MAX_SENDS 12 -> 40 (shared-QA-bot flood safety, not a scenario bound)
- runtime root chown root:mantis-proof, mode 1770 so the agent can stage
files while root-owned attestation stays unreplaceable
* docs(mantis): let the proof agent design scenarios like a local developer
Lead with developer-shell parity, allow reading whatever code the scenario
needs (PR text still untrusted, PR code only inside SUT lanes), document
exec/restart shapes, and reserve block for hard impossibilities.
* fix(mantis): keep the SUT exec result type local
* fix(mantis): resume the agent when it ends without a manifest
Run 32615428295 (exec branch on #127950) hit Codex context compaction at
03:52:11 and the model answered with a confabulated "handoff" message instead
of continuing; codex exec exited 0 with no mantis-evidence.json and the
trusted-evidence step failed the run with no verdict.
The agent step now checks for the manifest after codex exits and, when it is
missing, resumes the same thread (`codex exec ... resume --last -`, verified
against codex-rs/exec/src/lib.rs at rust-v0.149.0: cwd-matched latest thread,
`-` reads the prompt from stdin) with a short correction prompt, bounded to
three resumes. The main prompt states that a handoff/summary is never an
acceptable final message.
The proof gateways execute runtime JS only; declarations forced a ~177s
unified rebuild per lane because the declaration cache key is an
aggregate source hash. OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 on profile
full now selects the runtime artifact surface via the uncached generic
tsdown graph, both Mantis lanes pass it through, and the baseline
archive moves to an isolated mantis-runtime-v1 namespace. Measured
candidate build: 177.1s -> 33.05s.
The Telegram Desktop proof previously built the candidate from the raw PR
head, so a head behind main failed for reasons main already fixed
(observed on #127770: 59 commits behind, hitting the Unknown-model defect
fixed by #127952). GitHub's cached PR test merge cannot be the candidate
either: it was observed 50 commits / 12 hours stale and never refreshed.
The workflow now resolves the live refs/heads/main tip and merge base at
dispatch, requires a main-targeting PR, and builds a deterministic local
merge (merge-tree --merge-base + commit-tree with pinned identity/date) as
the candidate; conflicts fail with a direct rebase message and there is no
fallback to the raw head. Lane labels and docs now say the candidate is
the PR merged onto main.
* refactor(mantis): reuse authorized desktop captures
* fix(mantis): budget desktop authorization failures
* chore(mantis): bound desktop proof retries
* fix(e2e): drop unused recorder failure fact type export
* fix(ci): route Mantis desktop teardown through the recorder wrapper
Cleanup invoked the internal recorder executable as mantis-sut, which is
deliberately kept out of the docker group and cannot read the
recorder-owned session file; teardown therefore failed and blocked
safe_to_release. The cleanup step already runs as the recorder user, so
call the public wrapper whose exec shim cds into the session root.
* fix(e2e): make recorder failure fact lane-readable; document v2 lifecycle
The Mantis workflow runs the recorder as the desktop user while the lane
reads the authorization-failure fact as mantis-sut; 0600 made that read
fail EACCES and silently disabled the two-attempt retry budget. Write the
fact 0644 — the 0770 attempt directory bounds visibility.
Update the mantis doc's recorder section for the v2 session lifecycle:
required --session handle with healthy-session reuse, capture-only stop,
and teardown owning authorization termination and lease release.
* perf(mantis): parallelize proof builds and warm caches
* fix(mantis): size build image for the copied pnpm store
* chore(ui): refresh startup JS gzip baseline after streamed-markdown perf work
Identical source measures 345034-345058 B across builds while the
committed baseline left only a 9 B margin under the 512 B ratchet
tolerance, so build-artifacts flips on gzip nondeterminism (green on
main run 32559609413, red on PR run 32559442295, red locally).
Regenerated with scripts/check-control-ui-performance.mts
--update-baseline; the 350 KiB hard ceiling still bounds creep.
## What Problem This Solves
Auto-triggered Mantis proof runs (label/`clawsweeper_label` and other non-comment request sources) end with no PR comment at all. The durable evidence publisher runs with `--create-missing false` and only edits an existing marker comment, but the inline status comment carrying that marker was only created when `request_source == 'issue_comment'`. Label-triggered runs therefore published nothing and logged the misleading "Skipped stale Mantis QA evidence comment because its status is no longer active" — observed on PR #127735. This is the silent-failure class: a Mantis run completes and the PR shows no visible outcome.
## Why This Change Was Made
- `.github/workflows/mantis-telegram-desktop-proof.yml`: the status ack comment (👀 + active-job link + run-scoped marker) is now created for every request source that resolves to a PR (`pr_number != ''`), not only `issue_comment`. The 👀 *reaction* stays `issue_comment`-only (it lives in `mantis-resolve-request.yml`, untouched — there is no triggering comment to react to on label runs).
- The start-failure fallback comment and the existing-artifact republish path now use the same run-scoped marker `<!-- mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} -->` as the status comment and the main publisher, so every publisher edits the single run-owned comment (ack → progress → final proof; no comment spam). The republish path gets an explicit `--create-missing false` to match. The design invariant that makes `false` safe: the fallback status-comment step is not `continue-on-error`, so a run in which no marker comment could be created fails `resolve_request` and never reaches publish.
- `scripts/mantis/publish-pr-evidence.mjs`: the two skip cases now log honestly — "no existing comment found" vs "could not update existing comment" — instead of one misleading stale-status message.
## User Impact
Operators triggering Mantis via labels (ClawSweeper flows) now get the same single evolving PR comment as comment-triggered runs: an immediate 👀 ack with the running job link, edited in place into the final proof evidence. No more runs that finish invisibly.
## Evidence
- Focused suite: `node scripts/run-vitest.mjs test/scripts/mantis-telegram-desktop-proof-workflow.test.ts` — 28/28, including new assertions that the status/failure comment gates use `pr_number != ''` (and not `request_source`) and that both publishers pass the run-scoped marker with `--create-missing false`.
- `node scripts/check-changed.mjs -- <touched files>` green; `git diff --check` clean.
- Marker alignment verified across all five sites in the workflow (status comment, prior-attempt cleanup regex, fallback comment, failure report, both publisher invocations): all use `mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`.
- Live-run proof of the label-triggered path requires a merged workflow (GitHub runs the workflow from the default branch for these triggers), so the first post-merge label-triggered Mantis run is the live verification; stated here as the known evidence gap.
Production LOC delta: −4 (workflow/tooling); tests +13.
Makes the Mantis Telegram Desktop proof agent programmable at its trust boundaries: declarative Bot API fault rules (drop/status per method), per-request scripted mock-provider responses, observe-until predicates (post-cursor events/text, cumulative provider count), and bounded proxy-side recording of outbound Bot API requests as trusted lane facts. Adds a reusable recipe library under .github/codex/prompts/mantis-recipes/ and raises the proof agent's reasoning effort to high.
Security: the SUT container shadows proxy-control with an inaccessible tmpfs so candidate PR code sharing the mantis-sut uid cannot read or rewrite the proxy's recorded evidence; unmount is blocked by cap-drop/no-new-privileges. Proof doctrine now treats proxy-recorded Bot API facts as trusted comparison evidence and provider request logs as diagnostics.
Follow-up named in PR: move the mock OpenAI server out of the SUT container so provider request facts also become candidate-tamper-proof.
* fix(release): preserve validation plan across reruns
* test(release): align rerun plan assertions
* refactor(release): use canonical plan cache action
* style(test): format release plan cache assertion
The workflow ran a Codex agent over the full test suite and pushed
`test: optimize slow tests` straight to `main` under `contents: write`, with no
pull request and no human review. Its gates were a path allowlist, a
no-add/delete/rename rule, a non-decreasing total test count, and
`pnpm check:changed` -- which covers changed lanes, not the full suite. Test
optimization is exactly the class of change where a plausible edit can weaken
coverage without moving the test count, so unattended landing is the wrong
trade. Autonomous commits to `main` are not something this repo wants.
It had also been inert since well before this. The daily-cadence gate excluded
prior runs with `select(.status != "cancelled")`, but a finished cancelled run
reports `status: "completed"` with `conclusion: "cancelled"` -- verified against
run 32506655531, which that filter counts as a prior run. Its `concurrency`
block sets `cancel-in-progress: false`, so main's push rate produced dozens of
cancelled runs per hour and every trigger skipped, reporting green after ~2
minutes of doing nothing. No `test: optimize slow tests` commit has ever landed
on `main`.
`pnpm test:perf:groups` and the rest of the performance tooling it drove stay;
they are useful by hand and documented in docs/reference/test.md.
Repository secret OPENCLAW_TEST_PERF_AGENT_OPENAI_API_KEY now has no consumer
and can be deleted.
Make long, free-form Telegram proof runs truthful and resilient. Keep the trusted mock harness current across historical SUTs, preserve intentional silence and blocked outcomes, remove fixed attempt/lifetime caps, and export cropped motion proof without the prior memory spike.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Remove the premature visibility classifier and let one proof agent configure and exercise the disposable Telegram gateway. Align mock response timing with the 15-minute lane budget while preserving credential isolation through the alias-token proxy.
Preserve honest blocked proof outcomes and publish visible stop-reports without marking them passed. Serialize burst runs through the authoritative Telegram-user lease while reserving time for proof and cleanup.
* fix(control-ui): stop config form save from corrupting 64-bit id strings
Saving the schema-driven config form coerced every numeric-looking string
to a JS number before submission. For union-typed fields such as
tools.elevated.allowFrom.* (anyOf: string | number), string entries
holding 64-bit ids (Discord/Telegram snowflakes) were rewritten through
Number(), which rounds past 2^53:
"1048113311314608148" -> 1048113311314608100. The corruption also hit
untouched fields, because serialization coerces the whole form, so merely
saving an unrelated setting silently broke elevated-approval allowlists
(fail-closed: the real user id no longer matched).
Two guards fix this:
- coerceFormValues keeps a string that already satisfies a string variant
of an anyOf/oneOf union instead of parsing it into another variant's
number.
- coerceConfigFormNumberString refuses lossy integer parses: plain
integer text beyond Number.MAX_SAFE_INTEGER that does not round-trip
through BigInt stays a string, so pure number/integer fields fail
validation loudly instead of storing a corrupted id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(control-ui): harden 64-bit config id preservation
* fix(control-ui): validate mixed-union scalar branches
* test(control-ui): prove real gateway id preservation
* test(control-ui): use communications route for config proof
* test(control-ui): grant config proof admin scope
* test(control-ui): reopen raw config for proof
* fix(control-ui): preserve explicit union input types
* test(control-ui): exercise union collection draft
* ci: retry flaky control ui e2e
* fix(control-ui): preserve mixed scalar branch types
* ci: retry service worker e2e
* fix(control-ui): preserve typeless string union branches
* fix(control-ui): reject lossy decimal coercion
* fix(control-ui): reject lossy pure numeric input
* fix(control-ui): preserve exact numeric branch semantics
* ci: retry checkout rate limit
* ci(control-ui): capture real gateway proof
* test(control-ui): frame config proof values
* ci: retry checkout download
* test(control-ui): prove Gateway-served production bundle
* fix(control-ui): preserve exact incremental union edits
* refactor(control-ui): isolate scalar edit session state
* fix(control-ui): keep scalar edit branch type internal
* fix(control-ui): avoid detached focus selector
* fix(control-ui): round-trip exact numeric branches
* refactor(control-ui): share exact scalar formatting
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Give maintainers immediate visibility when Mantis is requested. Bare mentions now react, link the active run, and keep one run-owned status comment through proof, short-circuit, or failure.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Add trusted ClawSweeper-label and maintainer-comment dispatch for Mantis Telegram proof. Short-circuit non-visible PRs before desktop setup while preserving exact-head, fork, credential, and comment-ownership boundaries.
Move Mantis Telegram Desktop proof from the remote AWS/Crabbox lane to a recorder-driven local Docker desktop. Keep proof scenarios agent-authored, cache trusted build outputs, and publish exact visible Telegram evidence without writing the QA bot token to artifacts.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Vincent Koc