* fix(mantis): enforce verdict-expectation coherence and publish agent analysis files
Mantis run 32619081130 on #127989 published an overall `pass` while its own
manifest recorded that the candidate expectation was not observed: per-lane
`status` was mechanical capture success and the agent's judgment lived only
in `expected` prose, so nothing reconciled the two before publication.
- `mantis-evidence.json` schemaVersion 2: each comparison lane carries a
required boolean `expectationMet`. The desktop agent sets it in the same
manifest edit as `expected`; mechanical producers (Telegram live, web UI,
Slack, Discord) derive it from lane status.
- `scripts/mantis/publish-pr-evidence.mjs` is the single enforcement owner:
it requires the booleans, recomputes `pass`/`outcome`, downgrades a
contradictory pass claim to `fail`, and renders a visible "verdict
downgraded" note. The desktop workflow invokes it with `--validate-only`
before upload or comment.
- Agent top-level `*.json`/`*.md` analysis files (assertions, comparisons,
recipe suggestion) now survive the quarantine rebuild and upload, so cited
evidence actually exists in the artifact.
* fix(mantis): derive expectations from trusted lane facts
* feat(mantis): add exec and restart lane primitives
Give the proof agent a developer shell inside each SUT container and an
in-container gateway restart so it can design scenarios like a local
developer: patch openclaw.json and restart, stage plugins and fixtures,
run node/tsx against the read-only repo root, inspect SQLite state.
- container script: exec (docker exec as mantis-sut, bounded by timeout),
restart (request file + TERM), sut_command becomes a relaunch supervisor
- lane CLI: exec returns bounded stdout/stderr/exitCode and records a
redacted invocation; restart waits for a fresh [gateway] ready marker
- MAX_SENDS 12 -> 40 (shared-QA-bot flood safety, not a scenario bound)
- runtime root chown root:mantis-proof, mode 1770 so the agent can stage
files while root-owned attestation stays unreplaceable
* docs(mantis): let the proof agent design scenarios like a local developer
Lead with developer-shell parity, allow reading whatever code the scenario
needs (PR text still untrusted, PR code only inside SUT lanes), document
exec/restart shapes, and reserve block for hard impossibilities.
* fix(mantis): keep the SUT exec result type local
* fix(mantis): resume the agent when it ends without a manifest
Run 32615428295 (exec branch on #127950) hit Codex context compaction at
03:52:11 and the model answered with a confabulated "handoff" message instead
of continuing; codex exec exited 0 with no mantis-evidence.json and the
trusted-evidence step failed the run with no verdict.
The agent step now checks for the manifest after codex exits and, when it is
missing, resumes the same thread (`codex exec ... resume --last -`, verified
against codex-rs/exec/src/lib.rs at rust-v0.149.0: cwd-matched latest thread,
`-` reads the prompt from stdin) with a short correction prompt, bounded to
three resumes. The main prompt states that a handoff/summary is never an
acceptable final message.
Pass verdicts rested on agent prose only. The evidence builder now digests
each lane's trusted mantis-lane-facts.json (sends, bot messages, edits,
deletes, provider requests, injected Bot API faults, observed seconds,
attempt, sanitized user inputs) into an additive per-lane digest field and
a comparison.differential line listing the counts that changed between
baseline and candidate; the PR comment renders both.
The proof gateways execute runtime JS only; declarations forced a ~177s
unified rebuild per lane because the declaration cache key is an
aggregate source hash. OPENCLAW_RUN_NODE_SKIP_DTS_BUILD=1 on profile
full now selects the runtime artifact surface via the uncached generic
tsdown graph, both Mantis lanes pass it through, and the baseline
archive moves to an isolated mantis-runtime-v1 namespace. Measured
candidate build: 177.1s -> 33.05s.
The Telegram Desktop proof previously built the candidate from the raw PR
head, so a head behind main failed for reasons main already fixed
(observed on #127770: 59 commits behind, hitting the Unknown-model defect
fixed by #127952). GitHub's cached PR test merge cannot be the candidate
either: it was observed 50 commits / 12 hours stale and never refreshed.
The workflow now resolves the live refs/heads/main tip and merge base at
dispatch, requires a main-targeting PR, and builds a deterministic local
merge (merge-tree --merge-base + commit-tree with pinned identity/date) as
the candidate; conflicts fail with a direct rebase message and there is no
fallback to the raw head. Lane labels and docs now say the candidate is
the PR merged onto main.
The lease fence backgrounds its child to watch the lost-lease marker, and
a backgrounded command's stdin defaults to /dev/null — so the workflow's
piped agent prompt vanished and every agentic proof run died instantly
with 'No prompt provided via stdin' (observed in run 32569181167). An
explicit <&0 keeps the caller's stdin; regression pipes input through the
fence and fails pre-fix.
* feat(e2e): record provider media content facts
* docs(mantis): list staged-media recipe in recipe index
* fix(mantis): isolate mock provider evidence
* fix(mantis): state provider facts as tamper-evident, not provenance-trusted
The mock sidecar makes provider request facts tamper-evident: candidate
code cannot alter or remove a recorded request. It cannot make them
provenance-authenticated — every process in the SUT container shares the
internal network, UID, and gateway config, so nothing distinguishes the
gateway flow from any other candidate-originated request. The proof
prompt and busy-queue recipe now state exactly that property instead of
calling the facts trusted.
* fix(mantis): redact unparseable provider request bodies
Media redaction walks the parsed JSON body, so a request that fails
JSON.parse fell back to logging the raw text — leaking base64 payloads
the redactor exists to strip. Unparseable bodies now log a bounded
byte-count marker instead; regression posts a malformed body carrying a
data URL and asserts the payload never reaches the record.
* fix(mantis): expose newest provider records through a seq-stamped tail
The lane's requests surface kept the first 100 provider records, so a
session longer than the window hid exactly the newest requests a proof
asserts on. The mock server now stamps each record with a producer-owned
absolute seq ordinal, and the lane reads a bounded 128-record tail —
mirroring the sibling botApiRequests window. Regression writes 130
records and asserts the tail keeps seq 3..130; it fails pre-fix.
* perf(mantis): parallelize proof builds and warm caches
* fix(mantis): size build image for the copied pnpm store
* chore(ui): refresh startup JS gzip baseline after streamed-markdown perf work
Identical source measures 345034-345058 B across builds while the
committed baseline left only a 9 B margin under the 512 B ratchet
tolerance, so build-artifacts flips on gzip nondeterminism (green on
main run 32559609413, red on PR run 32559442295, red locally).
Regenerated with scripts/check-control-ui-performance.mts
--update-baseline; the 350 KiB hard ceiling still bounds creep.
## What Problem This Solves
Auto-triggered Mantis proof runs (label/`clawsweeper_label` and other non-comment request sources) end with no PR comment at all. The durable evidence publisher runs with `--create-missing false` and only edits an existing marker comment, but the inline status comment carrying that marker was only created when `request_source == 'issue_comment'`. Label-triggered runs therefore published nothing and logged the misleading "Skipped stale Mantis QA evidence comment because its status is no longer active" — observed on PR #127735. This is the silent-failure class: a Mantis run completes and the PR shows no visible outcome.
## Why This Change Was Made
- `.github/workflows/mantis-telegram-desktop-proof.yml`: the status ack comment (👀 + active-job link + run-scoped marker) is now created for every request source that resolves to a PR (`pr_number != ''`), not only `issue_comment`. The 👀 *reaction* stays `issue_comment`-only (it lives in `mantis-resolve-request.yml`, untouched — there is no triggering comment to react to on label runs).
- The start-failure fallback comment and the existing-artifact republish path now use the same run-scoped marker `<!-- mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} -->` as the status comment and the main publisher, so every publisher edits the single run-owned comment (ack → progress → final proof; no comment spam). The republish path gets an explicit `--create-missing false` to match. The design invariant that makes `false` safe: the fallback status-comment step is not `continue-on-error`, so a run in which no marker comment could be created fails `resolve_request` and never reaches publish.
- `scripts/mantis/publish-pr-evidence.mjs`: the two skip cases now log honestly — "no existing comment found" vs "could not update existing comment" — instead of one misleading stale-status message.
## User Impact
Operators triggering Mantis via labels (ClawSweeper flows) now get the same single evolving PR comment as comment-triggered runs: an immediate 👀 ack with the running job link, edited in place into the final proof evidence. No more runs that finish invisibly.
## Evidence
- Focused suite: `node scripts/run-vitest.mjs test/scripts/mantis-telegram-desktop-proof-workflow.test.ts` — 28/28, including new assertions that the status/failure comment gates use `pr_number != ''` (and not `request_source`) and that both publishers pass the run-scoped marker with `--create-missing false`.
- `node scripts/check-changed.mjs -- <touched files>` green; `git diff --check` clean.
- Marker alignment verified across all five sites in the workflow (status comment, prior-attempt cleanup regex, fallback comment, failure report, both publisher invocations): all use `mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`.
- Live-run proof of the label-triggered path requires a merged workflow (GitHub runs the workflow from the default branch for these triggers), so the first post-merge label-triggered Mantis run is the live verification; stated here as the known evidence gap.
Production LOC delta: −4 (workflow/tooling); tests +13.
Makes the Mantis Telegram Desktop proof agent programmable at its trust boundaries: declarative Bot API fault rules (drop/status per method), per-request scripted mock-provider responses, observe-until predicates (post-cursor events/text, cumulative provider count), and bounded proxy-side recording of outbound Bot API requests as trusted lane facts. Adds a reusable recipe library under .github/codex/prompts/mantis-recipes/ and raises the proof agent's reasoning effort to high.
Security: the SUT container shadows proxy-control with an inaccessible tmpfs so candidate PR code sharing the mantis-sut uid cannot read or rewrite the proxy's recorded evidence; unmount is blocked by cap-drop/no-new-privileges. Proof doctrine now treats proxy-recorded Bot API facts as trusted comparison evidence and provider request logs as diagnostics.
Follow-up named in PR: move the mock OpenAI server out of the SUT container so provider request facts also become candidate-tamper-proof.
Make long, free-form Telegram proof runs truthful and resilient. Keep the trusted mock harness current across historical SUTs, preserve intentional silence and blocked outcomes, remove fixed attempt/lifetime caps, and export cropped motion proof without the prior memory spike.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Preserve honest blocked proof outcomes and publish visible stop-reports without marking them passed. Serialize burst runs through the authoritative Telegram-user lease while reserving time for proof and cleanup.
Give maintainers immediate visibility when Mantis is requested. Bare mentions now react, link the active run, and keep one run-owned status comment through proof, short-circuit, or failure.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Move Mantis Telegram Desktop proof from the remote AWS/Crabbox lane to a recorder-driven local Docker desktop. Keep proof scenarios agent-authored, cache trusted build outputs, and publish exact visible Telegram evidence without writing the QA bot token to artifacts.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Records native Telegram Desktop from a digest-pinned prebaked image: a ready desktop in ~12s with no per-lease apt installs or downloads, which also removes the dpkg-lock failures that killed recent runs.
The recorder only records; callers drive the turn and supply the TDLib driver used for QR authorization. Nothing in this repository invokes it yet - routing the Mantis Telegram Desktop Proof workflow through it is a follow-up.
* fix(mantis): bound upload error response bodies
Replace unbounded response.text() on non-ok upload error paths with
readBoundedResponseText (64 KiB cap). Only swallow the size-exceeded
diagnostic; propagate signal aborts and other failures so timeouts
during body reading are not masked as generic upload failures.
- Import shared readBoundedResponseText from scripts/lib/bounded-response.mjs
- Add MANTIS_UPLOAD_ERROR_BODY_MAX_BYTES constant (64 KiB)
- Catch only size-exceeded errors; re-throw timeouts and stream errors
- Add test: oversized body bounded at 64 KiB
- Add test: signal abort during body read propagates correctly
- Add test: small error body within bound reads normally
* fix(mantis): contextualize bounded upload failures
Co-authored-by: 胡根深 0668000903 <hu.genshen@xydigit.com>
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* chore(types): add declaration files for scripts/lib and scripts/e2e modules
* chore(types): add declaration files for top-level script modules (a-m)
* chore(types): add declaration files for top-level script modules (n-z)
* test: use a non-secret-shaped gateway token fixture
* test: type ci workflow guard helpers for the root test lane
* chore(tooling): typecheck root test/** with a dedicated tsgo lane
- test/tsconfig/tsconfig.test.root.json: root-test program (strict unused checks,
fixtures excluded; two Docker E2E clients that import built dist/** stay out,
same rationale as the scripts/e2e exclusion in tsconfig.scripts.json)
- tsgo:test:root wired into tsgo:test, check:test-types, scripts/check.mjs, and
the ci.yml test-types shard, mirroring the tsgo:scripts lane (#104348)
- changed-lane routing: test/**/*.ts (excluding fixtures) and the lane tsconfig
now trigger 'typecheck test root' in check:changed; previously test/ paths ran
lint only, so harness type errors surfaced first in CI (#104287 envDir case)
- burn down all 1071 latent type errors in the program: precise param/local
types across test/scripts, test/vitest, test/e2e, and transitive scripts/e2e
program members; 205 sibling .d.mts declaration files for imported .mjs
modules (committed separately); zero any, zero ts-expect-error
- resolve the pre-existing testing star-export ambiguity in
scripts/e2e/parallels/common.ts with an explicit re-export
Closes#104388
* chore(types): correct declaration fidelity per structured review
- re-derive 51 .d.mts files from implementation data flow instead of
initializers: fix a wrong never return (runTestProjectsDelegation returns
the child), add encoding-sensitive exec/spawn overloads (plain-gh), restore
the full release profile union, make parsed paths string | null, add missing
parseArgs fields via help/non-help unions, add a missing sibling declaration
(budget-number-args), drop 15 unused lint directives
- precise install-record/tuple typing removes the type-aware oxlint
regressions the first declarations caused in scripts/e2e implementations
- route .mts declaration edits under test/ to the testRoot lane and reference
the test-root project from tsconfig.projects.json so tsgo:all covers it
(closes both review findings against the lane wiring)
* chore(scripts): keep telegram runner dist typing structural for the boundary guard
* chore(types): declare runtime pack and gateway readiness exports added on main
* test: pin the importTargetPlan form of the plugin-contract plan import
The guard expectation still referenced the raw await import( form that
7ae5996bb3 (#103975) replaced with the importTargetPlan fallback helper;
the assertion fails on current main.
* ci(mantis): add web ui chat proof lane
* ci(mantis): tighten web ui proof candidate parsing
* ci: tighten Mantis Web UI proof lane
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Peter Steinberger