Queue Telegram-visible Mantis proofs at workflow level so only one shared-user run allocates a runner at a time.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* fix(mantis): enforce verdict-expectation coherence and publish agent analysis files
Mantis run 32619081130 on #127989 published an overall `pass` while its own
manifest recorded that the candidate expectation was not observed: per-lane
`status` was mechanical capture success and the agent's judgment lived only
in `expected` prose, so nothing reconciled the two before publication.
- `mantis-evidence.json` schemaVersion 2: each comparison lane carries a
required boolean `expectationMet`. The desktop agent sets it in the same
manifest edit as `expected`; mechanical producers (Telegram live, web UI,
Slack, Discord) derive it from lane status.
- `scripts/mantis/publish-pr-evidence.mjs` is the single enforcement owner:
it requires the booleans, recomputes `pass`/`outcome`, downgrades a
contradictory pass claim to `fail`, and renders a visible "verdict
downgraded" note. The desktop workflow invokes it with `--validate-only`
before upload or comment.
- Agent top-level `*.json`/`*.md` analysis files (assertions, comparisons,
recipe suggestion) now survive the quarantine rebuild and upload, so cited
evidence actually exists in the artifact.
* fix(mantis): derive expectations from trusted lane facts
Capture the proven mechanics from proof run 32616859410 (PR #127950) as a
reusable recipe: dual 600s timeout config patch, Code Mode exec events file,
streamed chunkDelayMs hold, and the observe-loop pattern around the 60s cap.
Require agents to record recipe-suggestion.md after scenario-mechanics
iteration so future discoveries persist.
* feat(mantis): add exec and restart lane primitives
Give the proof agent a developer shell inside each SUT container and an
in-container gateway restart so it can design scenarios like a local
developer: patch openclaw.json and restart, stage plugins and fixtures,
run node/tsx against the read-only repo root, inspect SQLite state.
- container script: exec (docker exec as mantis-sut, bounded by timeout),
restart (request file + TERM), sut_command becomes a relaunch supervisor
- lane CLI: exec returns bounded stdout/stderr/exitCode and records a
redacted invocation; restart waits for a fresh [gateway] ready marker
- MAX_SENDS 12 -> 40 (shared-QA-bot flood safety, not a scenario bound)
- runtime root chown root:mantis-proof, mode 1770 so the agent can stage
files while root-owned attestation stays unreplaceable
* docs(mantis): let the proof agent design scenarios like a local developer
Lead with developer-shell parity, allow reading whatever code the scenario
needs (PR text still untrusted, PR code only inside SUT lanes), document
exec/restart shapes, and reserve block for hard impossibilities.
* fix(mantis): keep the SUT exec result type local
* fix(mantis): resume the agent when it ends without a manifest
Run 32615428295 (exec branch on #127950) hit Codex context compaction at
03:52:11 and the model answered with a confabulated "handoff" message instead
of continuing; codex exec exited 0 with no mantis-evidence.json and the
trusted-evidence step failed the run with no verdict.
The agent step now checks for the manifest after codex exits and, when it is
missing, resumes the same thread (`codex exec ... resume --last -`, verified
against codex-rs/exec/src/lib.rs at rust-v0.149.0: cwd-matched latest thread,
`-` reads the prompt from stdin) with a short correction prompt, bounded to
three resumes. The main prompt states that a handoff/summary is never an
acceptable final message.
The proof gateways execute runtime JS only; declarations forced a ~177s
unified rebuild per lane because the declaration cache key is an
aggregate source hash. OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 on profile
full now selects the runtime artifact surface via the uncached generic
tsdown graph, both Mantis lanes pass it through, and the baseline
archive moves to an isolated mantis-runtime-v1 namespace. Measured
candidate build: 177.1s -> 33.05s.
The Telegram Desktop proof previously built the candidate from the raw PR
head, so a head behind main failed for reasons main already fixed
(observed on #127770: 59 commits behind, hitting the Unknown-model defect
fixed by #127952). GitHub's cached PR test merge cannot be the candidate
either: it was observed 50 commits / 12 hours stale and never refreshed.
The workflow now resolves the live refs/heads/main tip and merge base at
dispatch, requires a main-targeting PR, and builds a deterministic local
merge (merge-tree --merge-base + commit-tree with pinned identity/date) as
the candidate; conflicts fail with a direct rebase message and there is no
fallback to the raw head. Lane labels and docs now say the candidate is
the PR merged onto main.
The pdf tool auto-resolves its model from plugin manifest defaults
(openai gpt-5.5), which the Mantis SUT catalog does not define, so every
pdf round trip died with "Unknown model: openai/gpt-5.5" before
native-vs-fallback dispatch (run 32572580656). Pin
agents.defaults.pdfModel to the only harness model and account for the
pdf tool's own provider request in the staged-media recipe wait/pins.
* docs(mantis): script catalog-tool turns through Code Mode exec
Run 32570733988 blocked both lanes with 'Tool pdf not found': the proof
agent scripted a top-level pdf function_call, but the SUT agent runs Code
Mode where catalog tools execute from JavaScript inside exec. Document the
canonical route in the proof prompt, add a complete exec(pdf) scripted-event
example plus the document-extract configPatch to the staged-media recipe,
and pin the new guidance in the workflow test.
* docs(mantis): wait for the exec round trip before finish
ClawSweeper P2: send releases the mock response once the outbound message
is visible, and finish tears the lane down, so a slow exec/pdf round trip
could lose its function_call_output follow-up. Observe until the cumulative
provider-request count covers the follow-up and assert the recorded
function_call_output before finish; pin the contract in the workflow test.
* feat(e2e): record provider media content facts
* docs(mantis): list staged-media recipe in recipe index
* fix(mantis): isolate mock provider evidence
* fix(mantis): state provider facts as tamper-evident, not provenance-trusted
The mock sidecar makes provider request facts tamper-evident: candidate
code cannot alter or remove a recorded request. It cannot make them
provenance-authenticated — every process in the SUT container shares the
internal network, UID, and gateway config, so nothing distinguishes the
gateway flow from any other candidate-originated request. The proof
prompt and busy-queue recipe now state exactly that property instead of
calling the facts trusted.
* fix(mantis): redact unparseable provider request bodies
Media redaction walks the parsed JSON body, so a request that fails
JSON.parse fell back to logging the raw text — leaking base64 payloads
the redactor exists to strip. Unparseable bodies now log a bounded
byte-count marker instead; regression posts a malformed body carrying a
data URL and asserts the payload never reaches the record.
* fix(mantis): expose newest provider records through a seq-stamped tail
The lane's requests surface kept the first 100 provider records, so a
session longer than the window hid exactly the newest requests a proof
asserts on. The mock server now stamps each record with a producer-owned
absolute seq ordinal, and the lane reads a bounded 128-record tail —
mirroring the sibling botApiRequests window. Regression writes 130
records and asserts the tail keeps seq 3..130; it fails pre-fix.
* refactor(mantis): reuse authorized desktop captures
* fix(mantis): budget desktop authorization failures
* chore(mantis): bound desktop proof retries
* fix(e2e): drop unused recorder failure fact type export
* fix(ci): route Mantis desktop teardown through the recorder wrapper
Cleanup invoked the internal recorder executable as mantis-sut, which is
deliberately kept out of the docker group and cannot read the
recorder-owned session file; teardown therefore failed and blocked
safe_to_release. The cleanup step already runs as the recorder user, so
call the public wrapper whose exec shim cds into the session root.
* fix(e2e): make recorder failure fact lane-readable; document v2 lifecycle
The Mantis workflow runs the recorder as the desktop user while the lane
reads the authorization-failure fact as mantis-sut; 0600 made that read
fail EACCES and silently disabled the two-attempt retry budget. Write the
fact 0644 — the 0770 attempt directory bounds visibility.
Update the mantis doc's recorder section for the v2 session lifecycle:
required --session handle with healthy-session reuse, capture-only stop,
and teardown owning authorization termination and lease release.
* perf(mantis): parallelize proof builds and warm caches
* fix(mantis): size build image for the copied pnpm store
* chore(ui): refresh startup JS gzip baseline after streamed-markdown perf work
Identical source measures 345034-345058 B across builds while the
committed baseline left only a 9 B margin under the 512 B ratchet
tolerance, so build-artifacts flips on gzip nondeterminism (green on
main run 32559609413, red on PR run 32559442295, red locally).
Regenerated with scripts/check-control-ui-performance.mts
--update-baseline; the 350 KiB hard ceiling still bounds creep.
## What Problem This Solves
Auto-triggered Mantis proof runs (label/`clawsweeper_label` and other non-comment request sources) end with no PR comment at all. The durable evidence publisher runs with `--create-missing false` and only edits an existing marker comment, but the inline status comment carrying that marker was only created when `request_source == 'issue_comment'`. Label-triggered runs therefore published nothing and logged the misleading "Skipped stale Mantis QA evidence comment because its status is no longer active" — observed on PR #127735. This is the silent-failure class: a Mantis run completes and the PR shows no visible outcome.
## Why This Change Was Made
- `.github/workflows/mantis-telegram-desktop-proof.yml`: the status ack comment (👀 + active-job link + run-scoped marker) is now created for every request source that resolves to a PR (`pr_number != ''`), not only `issue_comment`. The 👀 *reaction* stays `issue_comment`-only (it lives in `mantis-resolve-request.yml`, untouched — there is no triggering comment to react to on label runs).
- The start-failure fallback comment and the existing-artifact republish path now use the same run-scoped marker `<!-- mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} -->` as the status comment and the main publisher, so every publisher edits the single run-owned comment (ack → progress → final proof; no comment spam). The republish path gets an explicit `--create-missing false` to match. The design invariant that makes `false` safe: the fallback status-comment step is not `continue-on-error`, so a run in which no marker comment could be created fails `resolve_request` and never reaches publish.
- `scripts/mantis/publish-pr-evidence.mjs`: the two skip cases now log honestly — "no existing comment found" vs "could not update existing comment" — instead of one misleading stale-status message.
## User Impact
Operators triggering Mantis via labels (ClawSweeper flows) now get the same single evolving PR comment as comment-triggered runs: an immediate 👀 ack with the running job link, edited in place into the final proof evidence. No more runs that finish invisibly.
## Evidence
- Focused suite: `node scripts/run-vitest.mjs test/scripts/mantis-telegram-desktop-proof-workflow.test.ts` — 28/28, including new assertions that the status/failure comment gates use `pr_number != ''` (and not `request_source`) and that both publishers pass the run-scoped marker with `--create-missing false`.
- `node scripts/check-changed.mjs -- <touched files>` green; `git diff --check` clean.
- Marker alignment verified across all five sites in the workflow (status comment, prior-attempt cleanup regex, fallback comment, failure report, both publisher invocations): all use `mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`.
- Live-run proof of the label-triggered path requires a merged workflow (GitHub runs the workflow from the default branch for these triggers), so the first post-merge label-triggered Mantis run is the live verification; stated here as the known evidence gap.
Production LOC delta: −4 (workflow/tooling); tests +13.
Makes the Mantis Telegram Desktop proof agent programmable at its trust boundaries: declarative Bot API fault rules (drop/status per method), per-request scripted mock-provider responses, observe-until predicates (post-cursor events/text, cumulative provider count), and bounded proxy-side recording of outbound Bot API requests as trusted lane facts. Adds a reusable recipe library under .github/codex/prompts/mantis-recipes/ and raises the proof agent's reasoning effort to high.
Security: the SUT container shadows proxy-control with an inaccessible tmpfs so candidate PR code sharing the mantis-sut uid cannot read or rewrite the proxy's recorded evidence; unmount is blocked by cap-drop/no-new-privileges. Proof doctrine now treats proxy-recorded Bot API facts as trusted comparison evidence and provider request logs as diagnostics.
Follow-up named in PR: move the mock OpenAI server out of the SUT container so provider request facts also become candidate-tamper-proof.
The Mantis lane and the Telegram user driver reached for /proc to enforce
media containment and to prove a pid was still alive. Off Linux those paths
do not exist, so the containment check compared a literal "/proc/self/fd/N"
string against the output root and refused every file, while both liveness
probes silently reported "gone": the lane stole a held harness lock and the
driver deleted the observer pid file without ever signalling the process.
Containment now uses each runtime's strongest portable construct. Python
descends the media path one component at a time from an open directory
descriptor (O_DIRECTORY|O_NOFOLLOW, dir_fd), which is race-free and also
covers the intermediate directories the old leaf-only O_NOFOLLOW did not.
Node has no openat(2), so publicRelativePath re-walks the resolved
components after the descriptor is open and refuses any that became a
symlink. Liveness moves to process.kill(pid, 0) and to ps state/args, which
keeps the pid-reuse and zombie handling the /proc reads provided.
The compound-command approval binding test asserted an operand count that
only held where the host interpreters live in a root-owned prefix; a
Homebrew python3 is writable, so it correctly binds as a third executable
operand. Assert the script operands by path instead.
Make long, free-form Telegram proof runs truthful and resilient. Keep the trusted mock harness current across historical SUTs, preserve intentional silence and blocked outcomes, remove fixed attempt/lifetime caps, and export cropped motion proof without the prior memory spike.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Remove the premature visibility classifier and let one proof agent configure and exercise the disposable Telegram gateway. Align mock response timing with the 15-minute lane budget while preserving credential isolation through the alias-token proxy.
Preserve honest blocked proof outcomes and publish visible stop-reports without marking them passed. Serialize burst runs through the authoritative Telegram-user lease while reserving time for proof and cleanup.
Give maintainers immediate visibility when Mantis is requested. Bare mentions now react, link the active run, and keep one run-owned status comment through proof, short-circuit, or failure.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Add trusted ClawSweeper-label and maintainer-comment dispatch for Mantis Telegram proof. Short-circuit non-visible PRs before desktop setup while preserving exact-head, fork, credential, and comment-ownership boundaries.
Move Mantis Telegram Desktop proof from the remote AWS/Crabbox lane to a recorder-driven local Docker desktop. Keep proof scenarios agent-authored, cache trusted build outputs, and publish exact visible Telegram evidence without writing the QA bot token to artifacts.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Records native Telegram Desktop from a digest-pinned prebaked image: a ready desktop in ~12s with no per-lease apt installs or downloads, which also removes the dpkg-lock failures that killed recent runs.
The recorder only records; callers drive the turn and supply the TDLib driver used for QR authorization. Nothing in this repository invokes it yet - routing the Mantis Telegram Desktop Proof workflow through it is a follow-up.
* chore(types): add declaration files for scripts/lib and scripts/e2e modules
* chore(types): add declaration files for top-level script modules (a-m)
* chore(types): add declaration files for top-level script modules (n-z)
* test: use a non-secret-shaped gateway token fixture
* test: type ci workflow guard helpers for the root test lane
* chore(tooling): typecheck root test/** with a dedicated tsgo lane
- test/tsconfig/tsconfig.test.root.json: root-test program (strict unused checks,
fixtures excluded; two Docker E2E clients that import built dist/** stay out,
same rationale as the scripts/e2e exclusion in tsconfig.scripts.json)
- tsgo:test:root wired into tsgo:test, check:test-types, scripts/check.mjs, and
the ci.yml test-types shard, mirroring the tsgo:scripts lane (#104348)
- changed-lane routing: test/**/*.ts (excluding fixtures) and the lane tsconfig
now trigger 'typecheck test root' in check:changed; previously test/ paths ran
lint only, so harness type errors surfaced first in CI (#104287 envDir case)
- burn down all 1071 latent type errors in the program: precise param/local
types across test/scripts, test/vitest, test/e2e, and transitive scripts/e2e
program members; 205 sibling .d.mts declaration files for imported .mjs
modules (committed separately); zero any, zero ts-expect-error
- resolve the pre-existing testing star-export ambiguity in
scripts/e2e/parallels/common.ts with an explicit re-export
Closes#104388
* chore(types): correct declaration fidelity per structured review
- re-derive 51 .d.mts files from implementation data flow instead of
initializers: fix a wrong never return (runTestProjectsDelegation returns
the child), add encoding-sensitive exec/spawn overloads (plain-gh), restore
the full release profile union, make parsed paths string | null, add missing
parseArgs fields via help/non-help unions, add a missing sibling declaration
(budget-number-args), drop 15 unused lint directives
- precise install-record/tuple typing removes the type-aware oxlint
regressions the first declarations caused in scripts/e2e implementations
- route .mts declaration edits under test/ to the testRoot lane and reference
the test-root project from tsconfig.projects.json so tsgo:all covers it
(closes both review findings against the lane wiring)
* chore(scripts): keep telegram runner dist typing structural for the boundary guard
* chore(types): declare runtime pack and gateway readiness exports added on main
* test: pin the importTargetPlan form of the plugin-contract plan import
The guard expectation still referenced the raw await import( form that
7ae5996bb3 (#103975) replaced with the importTargetPlan fallback helper;
the assertion fails on current main.
Recreated from #85108 because the original branch could not be updated by maintainers.
Preserves current-main pnpm install hardening while switching workflow pnpm setup to packageManager, and adds exact version-scoped release-age exclusions for already-locked packages that pnpm 11.2.2 audits during install.
Co-authored-by: Altay <altay@hey.com>
Ayaan Zaidi