feat(e2e): record provider media content facts for Mantis proofs (#127830)

* feat(e2e): record provider media content facts

* docs(mantis): list staged-media recipe in recipe index

* fix(mantis): isolate mock provider evidence

* fix(mantis): state provider facts as tamper-evident, not provenance-trusted

The mock sidecar makes provider request facts tamper-evident: candidate
code cannot alter or remove a recorded request. It cannot make them
provenance-authenticated — every process in the SUT container shares the
internal network, UID, and gateway config, so nothing distinguishes the
gateway flow from any other candidate-originated request. The proof
prompt and busy-queue recipe now state exactly that property instead of
calling the facts trusted.

* fix(mantis): redact unparseable provider request bodies

Media redaction walks the parsed JSON body, so a request that fails
JSON.parse fell back to logging the raw text — leaking base64 payloads
the redactor exists to strip. Unparseable bodies now log a bounded
byte-count marker instead; regression posts a malformed body carrying a
data URL and asserts the payload never reaches the record.

* fix(mantis): expose newest provider records through a seq-stamped tail

The lane's requests surface kept the first 100 provider records, so a
session longer than the window hid exactly the newest requests a proof
asserts on. The mock server now stamps each record with a producer-owned
absolute seq ordinal, and the lane reads a bounded 128-record tail —
mirroring the sibling botApiRequests window. Regression writes 130
records and asserts the tail keeps seq 3..130; it fails pre-fix.
This commit is contained in:
Ayaan Zaidi
2026-08-22 16:30:30 +05:30
committed by GitHub
parent da0cb592dd
commit d9d44981fb
14 changed files with 507 additions and 84 deletions
@@ -7,5 +7,6 @@ even when Telegram Desktop pixels match.
- `send-failure-injection.md`: outbound Bot API failures and retry behavior
- `busy-queue-scripted-provider.md`: ordered slow/fast multi-turn responses
- `callback-data-payload-proof.md`: byte-level callback payload differences
- `staged-media-provider-proof.md`: staged Telegram media proven through provider content facts
Return to `mantis-telegram-desktop-proof.md` for limits, cleanup, and publishing.
@@ -29,5 +29,5 @@ $lane finish --lane baseline
Repeat for `candidate`. Proof facts: session events and recorded Bot API
messages show the slow first and distinct second outcomes without a
control-file race. The provider request log (`scriptEntry` 0 then 1, turn
order in bodies) is diagnostic context, not the comparison evidence.
control-file race. The tamper-evident provider request facts (`scriptEntry` 0
then 1, turn order in bodies) independently prove provider arrival order.
@@ -0,0 +1,30 @@
# Staged media provider proof
Use when a change alters how an uploaded document or image reaches the provider.
```bash
lane="$OPENCLAW_TELEGRAM_MANTIS_LANE_CMD"
media="$MANTIS_OUTPUT_DIR/sample.pdf"
sent="$($lane send --lane baseline --media "$media" --text '@{sut} inspect this document')"
message_id="$(jq -er '.sent.messageId' <<<"$sent")"
$lane observe --lane baseline --seconds 60 --until-provider-requests 1
requests="$($lane requests --lane baseline)"
jq -e '[.requests[].contentFacts[]? | select(.type == "legacy_media")] | length > 0' \
<<<"$requests"
$lane finish --lane baseline --focus-message-id "$message_id"
```
For a reply-mention turn, first `send --media "$media"` without text, capture its
`.sent.messageId`, then `send --reply-to "$message_id" --text '@{sut} inspect this document'`.
A bare unmentioned upload stages the file but produces no provider turn.
Repeat for `candidate` with its returned message id, selecting `type == "input_file"`.
Assert the complete selected facts: `filename`, `mimeType`, and `byteLength` when present.
The structured facts are comparison evidence; never scrape `body` strings.
If baseline needs a tool round trip, take the tool argument from the recorded
`legacy_media.filename`. Build a complete response-events JSON array from
`toolCallEvents()` in `scripts/e2e/mock-openai-server.mjs`
(`response.output_item.added`, `response.function_call_arguments.delta`,
`response.output_item.done`, `response.completed`), then install it before the
next turn with `mock --lane baseline --response-events-file <public-json>`.
@@ -80,7 +80,8 @@ Use `$OPENCLAW_TELEGRAM_MANTIS_LANE_CMD` with `--lane baseline|candidate`:
[--until-provider-requests N]` (returns early when all supplied conditions hold;
event/text conditions count only events after the cursor, provider count is
cumulative for the lane)
- `requests` (redacted provider requests; zero is a valid recorded fact)
- `requests` (redacted provider requests; media/file items appear as structured
`contentFacts`; zero is a valid recorded fact)
- `press --message-id ID --button INDEX`
- `delete --message-id ID` (only user messages sent in this session)
- `desktop --actions-file <public-json> [--timeout-seconds N]` (run an
@@ -126,10 +127,13 @@ baseline lane that reproduces the defect is a successful capture. A PR-level
pass claim requires an observed, material baseline/candidate difference caused
by the changed behavior. That difference may be trusted Bot API payload/status
facts even when pixels are identical; screenshots remain comparison context.
Provider request logs are diagnostic and pacing signals, not standalone
comparison evidence. Identical pixels alone do not force `block` when the
trusted recorded facts differ materially. If neither pixels nor trusted facts prove a
difference, use `block`. When the expected result is silence, focus the
Provider request facts are tamper-evident comparison evidence: the provider
sidecar records them outside the candidate runtime, so candidate code cannot
alter or remove a recorded request after the fact. Requests still originate
inside the SUT, so the facts prove what the candidate runtime sent — the
behavior under proof — not who sent it. Identical pixels alone do not force `block`
when the recorded facts differ materially. If neither pixels nor recorded facts
prove a difference, use `block`. When the expected result is silence, focus the
session-owned user message that triggered the silent outcome.
Decide before finalizing each lane. If its setup did not exercise the intended
behavior, call `block`; do not call `finish` and describe the block only in prose.