* fix(scripts): bound tsgo runs with the managed-command watchdog
run-tsgo bypassed the repo's managed-command seam and called spawnSync
directly, so a wedged tsgo blocked its caller indefinitely: no timeout, no
process-group cleanup, and no SIGKILL escalation.
Observed in the wild as a tsgo holding 2.85 GB for 90+ minutes on 41s of
total CPU with RSS frozen to the byte, ignoring SIGTERM, with its wrapper
reparented to init. Because shouldReclaimLock() treats a live PID as a valid
lock owner, that orphan also held the heavy-check lock until every other
invocation hit the 10-minute lock timeout and threw.
Route the run through runManagedCommand, which already owns process-group
termination and SIGKILL escalation on timeout, and bound it with
OPENCLAW_TSGO_TIMEOUT_MS (default 45m) through the shared readPositiveEnvInt
helper, mirroring OPENCLAW_CLI_STARTUP_BUILD_TIMEOUT_MS in
ensure-cli-startup-build.mts.
* fix(scripts): saturate the tsgo watchdog at Node's timer ceiling
An OPENCLAW_TSGO_TIMEOUT_MS above 2147483647 reached setTimeout unchanged,
where Node collapses it to a 1ms delay, so raising the override killed
healthy typechecks immediately instead of loosening the bound.
* fix(scripts): make the tsgo watchdog opt-in and stop the harness leaking
ClawSweeper review on 6ba02c9d0a raised two findings.
[P1] The 45-minute default applied an unproven deadline to every tsgo
invocation. No supported duration contract covers every host and project, and
CI already bounds its own tsgo jobs at 15-20 minutes, so the default could only
ever fire outside CI where it was least validated. Drop it: an unset
OPENCLAW_TSGO_TIMEOUT_MS keeps the pre-existing unbounded wait, so no existing
run changes behavior, and operators opt in per host. Documented in
docs/help/testing.md beside the sibling Vitest watchdog.
[P2] The regression harness could leak its wedged child. The fake compiler
ignores SIGTERM by design, so a pre-fix or otherwise failing run left the tree
running after spawnSync gave up. Bound the fixture's loop as a backstop.
* fix(scripts): set the tsgo watchdog default from measured lane duration
ClawSweeper on c7a699ee82 reversed its earlier guidance: the opt-in default
adopted last iteration "deliberately preserves the indefinite tsgo hang that
this PR is meant to fix". Its objection was never that a default existed, only
that 45 minutes was unmeasured.
Measured instead of guessed: hosted tsgo lanes (check-test-types, and its core
stripes) complete in 1-2 minutes across recent successful main runs, against CI
job caps of 15-20 minutes. 30 minutes is 15-30x the observed duration, leaves
room for a far slower local host, and still bounds the 90-minute and multi-hour
wedges that motivated this PR. OPENCLAW_TSGO_TIMEOUT_MS remains the documented
override for hosts that need longer.
* test(scripts): reap the wedged fake tsgo tree on the harness outer timeout
ClawSweeper on 0d9f3604e8 flagged that the harness can still leave a detached
pre-fix process tree alive after its outer timeout. The bounded fixture loop
added earlier only capped the leak; it did not terminate the tree.
spawnSync's killSignal reaches the direct child only. runManagedCommand spawns
the compiler detached into its own process group, so the fake tsgo is a
grandchild that never receives that signal. The fixture now records its pid and
the harness reaps that group in a finally, with the bounded loop kept as a
last-resort backstop.
Verified: pid file written with the live pid, and killing that group terminates
the tree; focused suite 16/16 with no surviving fake-tsgo processes.
* fix(scripts): harden the tsgo watchdog after two-phase code review
Review fixes on top of the watchdog change, from one native pass and six cold
passes:
- A rejected OPENCLAW_TSGO_TIMEOUT_MS escaped main() as a raw module rejection.
It now reports one actionable line and exits 1. Strict validation was kept
rather than switching to coercion, so a typo cannot silently fall back to the
30-minute default.
- The rejection message named a numeric range while the parser enforces plain
decimal digits, so 1e5 and 007 were refused by a message saying they
qualified. It now names the real format and states that the watchdog cannot
be disabled.
- The timer ceiling is declared locally rather than imported from packages/.
A static import there resolves before the sparse-checkout guard runs, which
turned a clean sparse skip into ERR_MODULE_NOT_FOUND and flipped
check-changed's typecheck lane from exit 0 to exit 1.
- The wedge test asserted the kill message but not the outcome; it now captures
the wedged pid and asserts the process group is gone.
- Three near-duplicate "not killed" cases are table-driven.
- Doc bullet corrected: values ABOVE the ceiling saturate at it, and the
rejected-value list now includes non-decimal input.
Deferred follow-up, not fixed here: scripts/lib/tsx-cli-shim.mjs shares a
5000ms force-kill delay with managed-child-process, so Ctrl-C can still orphan
a wedged compiler about one run in three. Measured base 4/4 orphaned versus
4/10 here, so this change improves it; the fix is out of diff and shared with
four other wrappers.
* fix(scripts): close tsgo signal cleanup race
* fix(scripts): make tsgo watchdog opt-in
---------
Co-authored-by: ClawSweeper <steipete+clawsweeper@gmail.com>
* fix(mantis): enforce verdict-expectation coherence and publish agent analysis files
Mantis run 32619081130 on #127989 published an overall `pass` while its own
manifest recorded that the candidate expectation was not observed: per-lane
`status` was mechanical capture success and the agent's judgment lived only
in `expected` prose, so nothing reconciled the two before publication.
- `mantis-evidence.json` schemaVersion 2: each comparison lane carries a
required boolean `expectationMet`. The desktop agent sets it in the same
manifest edit as `expected`; mechanical producers (Telegram live, web UI,
Slack, Discord) derive it from lane status.
- `scripts/mantis/publish-pr-evidence.mjs` is the single enforcement owner:
it requires the booleans, recomputes `pass`/`outcome`, downgrades a
contradictory pass claim to `fail`, and renders a visible "verdict
downgraded" note. The desktop workflow invokes it with `--validate-only`
before upload or comment.
- Agent top-level `*.json`/`*.md` analysis files (assertions, comparisons,
recipe suggestion) now survive the quarantine rebuild and upload, so cited
evidence actually exists in the artifact.
* fix(mantis): derive expectations from trusted lane facts
* feat(mantis): add exec and restart lane primitives
Give the proof agent a developer shell inside each SUT container and an
in-container gateway restart so it can design scenarios like a local
developer: patch openclaw.json and restart, stage plugins and fixtures,
run node/tsx against the read-only repo root, inspect SQLite state.
- container script: exec (docker exec as mantis-sut, bounded by timeout),
restart (request file + TERM), sut_command becomes a relaunch supervisor
- lane CLI: exec returns bounded stdout/stderr/exitCode and records a
redacted invocation; restart waits for a fresh [gateway] ready marker
- MAX_SENDS 12 -> 40 (shared-QA-bot flood safety, not a scenario bound)
- runtime root chown root:mantis-proof, mode 1770 so the agent can stage
files while root-owned attestation stays unreplaceable
* docs(mantis): let the proof agent design scenarios like a local developer
Lead with developer-shell parity, allow reading whatever code the scenario
needs (PR text still untrusted, PR code only inside SUT lanes), document
exec/restart shapes, and reserve block for hard impossibilities.
* fix(mantis): keep the SUT exec result type local
* fix(mantis): resume the agent when it ends without a manifest
Run 32615428295 (exec branch on #127950) hit Codex context compaction at
03:52:11 and the model answered with a confabulated "handoff" message instead
of continuing; codex exec exited 0 with no mantis-evidence.json and the
trusted-evidence step failed the run with no verdict.
The agent step now checks for the manifest after codex exits and, when it is
missing, resumes the same thread (`codex exec ... resume --last -`, verified
against codex-rs/exec/src/lib.rs at rust-v0.149.0: cwd-matched latest thread,
`-` reads the prompt from stdin) with a short correction prompt, bounded to
three resumes. The main prompt states that a handoff/summary is never an
acceptable final message.
Pass verdicts rested on agent prose only. The evidence builder now digests
each lane's trusted mantis-lane-facts.json (sends, bot messages, edits,
deletes, provider requests, injected Bot API faults, observed seconds,
attempt, sanitized user inputs) into an additive per-lane digest field and
a comparison.differential line listing the counts that changed between
baseline and candidate; the PR comment renders both.
The proof gateways execute runtime JS only; declarations forced a ~177s
unified rebuild per lane because the declaration cache key is an
aggregate source hash. OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 on profile
full now selects the runtime artifact surface via the uncached generic
tsdown graph, both Mantis lanes pass it through, and the baseline
archive moves to an isolated mantis-runtime-v1 namespace. Measured
candidate build: 177.1s -> 33.05s.
The Telegram Desktop proof previously built the candidate from the raw PR
head, so a head behind main failed for reasons main already fixed
(observed on #127770: 59 commits behind, hitting the Unknown-model defect
fixed by #127952). GitHub's cached PR test merge cannot be the candidate
either: it was observed 50 commits / 12 hours stale and never refreshed.
The workflow now resolves the live refs/heads/main tip and merge base at
dispatch, requires a main-targeting PR, and builds a deterministic local
merge (merge-tree --merge-base + commit-tree with pinned identity/date) as
the candidate; conflicts fail with a direct rebase message and there is no
fallback to the raw head. Lane labels and docs now say the candidate is
the PR merged onto main.
Inside gateway runs the prepared model runtime is a static snapshot whose
ModelRegistry is empty on a fresh state dir; configured models are served
through the snapshot's prepared facts. The pdf tool resolved candidates via
the registry-only resolveModelFromRegistry helper, so every fallback
candidate failed with `Unknown model` while the main turn and image tool
(already on resolveModelAsync) worked.
Resolve each pdf candidate through resolveModelAsync with the prepared
stores, matching the image sibling, and delete the registry-only helper.
Adds a regression test that acquires a real static prepared runtime on a
fresh state dir with a config-inline model.
Observed live: Mantis run 32574012855 (both lanes died in the pdf bridge).
The pdf tool auto-resolves its model from plugin manifest defaults
(openai gpt-5.5), which the Mantis SUT catalog does not define, so every
pdf round trip died with "Unknown model: openai/gpt-5.5" before
native-vs-fallback dispatch (run 32572580656). Pin
agents.defaults.pdfModel to the only harness model and account for the
pdf tool's own provider request in the staged-media recipe wait/pins.
The lease fence backgrounds its child to watch the lost-lease marker, and
a backgrounded command's stdin defaults to /dev/null — so the workflow's
piped agent prompt vanished and every agentic proof run died instantly
with 'No prompt provided via stdin' (observed in run 32569181167). An
explicit <&0 keeps the caller's stdin; regression pipes input through the
fence and fails pre-fix.
* feat(e2e): record provider media content facts
* docs(mantis): list staged-media recipe in recipe index
* fix(mantis): isolate mock provider evidence
* fix(mantis): state provider facts as tamper-evident, not provenance-trusted
The mock sidecar makes provider request facts tamper-evident: candidate
code cannot alter or remove a recorded request. It cannot make them
provenance-authenticated — every process in the SUT container shares the
internal network, UID, and gateway config, so nothing distinguishes the
gateway flow from any other candidate-originated request. The proof
prompt and busy-queue recipe now state exactly that property instead of
calling the facts trusted.
* fix(mantis): redact unparseable provider request bodies
Media redaction walks the parsed JSON body, so a request that fails
JSON.parse fell back to logging the raw text — leaking base64 payloads
the redactor exists to strip. Unparseable bodies now log a bounded
byte-count marker instead; regression posts a malformed body carrying a
data URL and asserts the payload never reaches the record.
* fix(mantis): expose newest provider records through a seq-stamped tail
The lane's requests surface kept the first 100 provider records, so a
session longer than the window hid exactly the newest requests a proof
asserts on. The mock server now stamps each record with a producer-owned
absolute seq ordinal, and the lane reads a bounded 128-record tail —
mirroring the sibling botApiRequests window. Regression writes 130
records and asserts the tail keeps seq 3..130; it fails pre-fix.
* refactor(mantis): reuse authorized desktop captures
* fix(mantis): budget desktop authorization failures
* chore(mantis): bound desktop proof retries
* fix(e2e): drop unused recorder failure fact type export
* fix(ci): route Mantis desktop teardown through the recorder wrapper
Cleanup invoked the internal recorder executable as mantis-sut, which is
deliberately kept out of the docker group and cannot read the
recorder-owned session file; teardown therefore failed and blocked
safe_to_release. The cleanup step already runs as the recorder user, so
call the public wrapper whose exec shim cds into the session root.
* fix(e2e): make recorder failure fact lane-readable; document v2 lifecycle
The Mantis workflow runs the recorder as the desktop user while the lane
reads the authorization-failure fact as mantis-sut; 0600 made that read
fail EACCES and silently disabled the two-attempt retry budget. Write the
fact 0644 — the 0770 attempt directory bounds visibility.
Update the mantis doc's recorder section for the v2 session lifecycle:
required --session handle with healthy-session reuse, capture-only stop,
and teardown owning authorization termination and lease release.
Centralize untrusted audio transcript framing across channel plugins and keep Discord voice transcripts out of typed command provenance.
Co-authored-by: xialonglee <li.xialong@xydigit.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* perf(mantis): parallelize proof builds and warm caches
* fix(mantis): size build image for the copied pnpm store
* chore(ui): refresh startup JS gzip baseline after streamed-markdown perf work
Identical source measures 345034-345058 B across builds while the
committed baseline left only a 9 B margin under the 512 B ratchet
tolerance, so build-artifacts flips on gzip nondeterminism (green on
main run 32559609413, red on PR run 32559442295, red locally).
Regenerated with scripts/check-control-ui-performance.mts
--update-baseline; the 350 KiB hard ceiling still bounds creep.
## What Problem This Solves
Auto-triggered Mantis proof runs (label/`clawsweeper_label` and other non-comment request sources) end with no PR comment at all. The durable evidence publisher runs with `--create-missing false` and only edits an existing marker comment, but the inline status comment carrying that marker was only created when `request_source == 'issue_comment'`. Label-triggered runs therefore published nothing and logged the misleading "Skipped stale Mantis QA evidence comment because its status is no longer active" — observed on PR #127735. This is the silent-failure class: a Mantis run completes and the PR shows no visible outcome.
## Why This Change Was Made
- `.github/workflows/mantis-telegram-desktop-proof.yml`: the status ack comment (👀 + active-job link + run-scoped marker) is now created for every request source that resolves to a PR (`pr_number != ''`), not only `issue_comment`. The 👀 *reaction* stays `issue_comment`-only (it lives in `mantis-resolve-request.yml`, untouched — there is no triggering comment to react to on label runs).
- The start-failure fallback comment and the existing-artifact republish path now use the same run-scoped marker `<!-- mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} -->` as the status comment and the main publisher, so every publisher edits the single run-owned comment (ack → progress → final proof; no comment spam). The republish path gets an explicit `--create-missing false` to match. The design invariant that makes `false` safe: the fallback status-comment step is not `continue-on-error`, so a run in which no marker comment could be created fails `resolve_request` and never reaches publish.
- `scripts/mantis/publish-pr-evidence.mjs`: the two skip cases now log honestly — "no existing comment found" vs "could not update existing comment" — instead of one misleading stale-status message.
## User Impact
Operators triggering Mantis via labels (ClawSweeper flows) now get the same single evolving PR comment as comment-triggered runs: an immediate 👀 ack with the running job link, edited in place into the final proof evidence. No more runs that finish invisibly.
## Evidence
- Focused suite: `node scripts/run-vitest.mjs test/scripts/mantis-telegram-desktop-proof-workflow.test.ts` — 28/28, including new assertions that the status/failure comment gates use `pr_number != ''` (and not `request_source`) and that both publishers pass the run-scoped marker with `--create-missing false`.
- `node scripts/check-changed.mjs -- <touched files>` green; `git diff --check` clean.
- Marker alignment verified across all five sites in the workflow (status comment, prior-attempt cleanup regex, fallback comment, failure report, both publisher invocations): all use `mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`.
- Live-run proof of the label-triggered path requires a merged workflow (GitHub runs the workflow from the default branch for these triggers), so the first post-merge label-triggered Mantis run is the live verification; stated here as the known evidence gap.
Production LOC delta: −4 (workflow/tooling); tests +13.
Makes the Mantis Telegram Desktop proof agent programmable at its trust boundaries: declarative Bot API fault rules (drop/status per method), per-request scripted mock-provider responses, observe-until predicates (post-cursor events/text, cumulative provider count), and bounded proxy-side recording of outbound Bot API requests as trusted lane facts. Adds a reusable recipe library under .github/codex/prompts/mantis-recipes/ and raises the proof agent's reasoning effort to high.
Security: the SUT container shadows proxy-control with an inaccessible tmpfs so candidate PR code sharing the mantis-sut uid cannot read or rewrite the proxy's recorded evidence; unmount is blocked by cap-drop/no-new-privileges. Proof doctrine now treats proxy-recorded Bot API facts as trusted comparison evidence and provider request logs as diagnostics.
Follow-up named in PR: move the mock OpenAI server out of the SUT container so provider request facts also become candidate-tamper-proof.
Allow healthy current-version state databases to open without taking SQLite writer locks while preserving schema repair, legacy cron migration, and external ownership fencing.
Co-authored-by: Jason O'Neal <jason.allen.oneal@gmail.com>
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
The Mantis lane and the Telegram user driver reached for /proc to enforce
media containment and to prove a pid was still alive. Off Linux those paths
do not exist, so the containment check compared a literal "/proc/self/fd/N"
string against the output root and refused every file, while both liveness
probes silently reported "gone": the lane stole a held harness lock and the
driver deleted the observer pid file without ever signalling the process.
Containment now uses each runtime's strongest portable construct. Python
descends the media path one component at a time from an open directory
descriptor (O_DIRECTORY|O_NOFOLLOW, dir_fd), which is race-free and also
covers the intermediate directories the old leaf-only O_NOFOLLOW did not.
Node has no openat(2), so publicRelativePath re-walks the resolved
components after the descriptor is open and refuses any that became a
symlink. Liveness moves to process.kill(pid, 0) and to ps state/args, which
keeps the pid-reuse and zombie handling the /proc reads provided.
The compound-command approval binding test asserted an operand count that
only held where the host interpreters live in a root-owned prefix; a
Homebrew python3 is writable, so it correctly binds as a third executable
operand. Assert the script operands by path instead.
* fix(tui): own local shell process scopes
Keep local shell root results independent from authoritative process-scope
extinction, and make TUI shutdown cancel and join its ProcessSupervisor scope.
On Windows, create commands atomically inside a retained non-breakaway Job
Object and prove extinction from exact Job accounting.
Fixes#127488
* fix(process): preserve optional extinction waits
* feat(ui): add cpu/memory graphs to system busyness overlay
* refactor(ui): redesign busyness overlay vitals as sparkline stat tiles
Merge the duplicated sparkline row and numeric tile grid into three vital
tiles (CPU, memory, event-loop delay) with gradient area charts, a live
pulse dot, pointer scrubbing with value+age readout, and a danger tint
driven by the gateway's own event-loop degradation reasons. Memory
auto-ranges its baseline so RSS trends stay visible.
* feat(ui): add Person grouping mode for sessions sidebar and sessions page
Sessions can now be grouped by their durable owner identity: the sidebar
Group-by menu gains a capability-gated Person mode (self first, humans by
label, agent identities after; ownerless rows keep their smart zones), and
the sessions page gains the matching person mode. Person section headers
render the owner avatar and profile label; the mock dev server now
advertises the multi-identity policy and carries explicit row owners the
way the gateway projects createdActor fallbacks.
* feat(ui): gate sessions-page Person grouping on the identity capability
Mirrors the sidebar: the Person option hides without
hasMultipleSessionSharingIdentities and a stored Person preference
renders as None until the capability returns.
* fix(scripts): keep Windows lint runs from failing before any file is checked
`pnpm lint:extensions` aborts on Windows whenever the plugin SDK boundary
cache is cold: the boundary prep spawns `node_modules/.bin/tsgo` directly,
and Windows cannot execute the extensionless pnpm shim, so the run dies with
ENOENT before oxlint checks a single file. The stylelint runner reached by
`check-changed` fails the same way for the same reason.
Both now build their child process with `createManagedCommandInvocation`,
the launcher every other repo tool runner already uses, which routes the shim
through cmd.exe on Windows and returns the command unchanged everywhere else.
Off Windows the spawned command, arguments, and options are byte-identical to
before, so only the broken platform changes behavior.
* fix(scripts): stop the lint pipeline from spawning a tool shim directly
`pnpm lint` reaches stylelint through the same raw shim spawn that broke the
boundary prep, so the pipeline dies on Windows after oxlint succeeds. Route it
through the managed launcher like every other tool runner.
The remaining hazard is structural: the resolver hands out a path that only
some callers know to normalize, and the three sites that forgot were spread
across two spawn shapes. Add a static guard so a shim can only flow into a
launcher that understands the platform, and so the next occurrence fails on
Linux CI instead of on a contributor's Windows machine.
* test(scripts): trim Windows shim regression coverage
Punchcard-Session: cobalt-orchard-willow-2q
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
wahaha1223