mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-26 04:15:48 -06:00
feat(e2e): record provider media content facts for Mantis proofs (#127830)
* feat(e2e): record provider media content facts * docs(mantis): list staged-media recipe in recipe index * fix(mantis): isolate mock provider evidence * fix(mantis): state provider facts as tamper-evident, not provenance-trusted The mock sidecar makes provider request facts tamper-evident: candidate code cannot alter or remove a recorded request. It cannot make them provenance-authenticated — every process in the SUT container shares the internal network, UID, and gateway config, so nothing distinguishes the gateway flow from any other candidate-originated request. The proof prompt and busy-queue recipe now state exactly that property instead of calling the facts trusted. * fix(mantis): redact unparseable provider request bodies Media redaction walks the parsed JSON body, so a request that fails JSON.parse fell back to logging the raw text — leaking base64 payloads the redactor exists to strip. Unparseable bodies now log a bounded byte-count marker instead; regression posts a malformed body carrying a data URL and asserts the payload never reaches the record. * fix(mantis): expose newest provider records through a seq-stamped tail The lane's requests surface kept the first 100 provider records, so a session longer than the window hid exactly the newest requests a proof asserts on. The mock server now stamps each record with a producer-owned absolute seq ordinal, and the lane reads a bounded 128-record tail — mirroring the sibling botApiRequests window. Regression writes 130 records and asserts the tail keeps seq 3..130; it fails pre-fix.
This commit is contained in:
@@ -7,5 +7,6 @@ even when Telegram Desktop pixels match.
|
||||
- `send-failure-injection.md`: outbound Bot API failures and retry behavior
|
||||
- `busy-queue-scripted-provider.md`: ordered slow/fast multi-turn responses
|
||||
- `callback-data-payload-proof.md`: byte-level callback payload differences
|
||||
- `staged-media-provider-proof.md`: staged Telegram media proven through provider content facts
|
||||
|
||||
Return to `mantis-telegram-desktop-proof.md` for limits, cleanup, and publishing.
|
||||
|
||||
@@ -29,5 +29,5 @@ $lane finish --lane baseline
|
||||
|
||||
Repeat for `candidate`. Proof facts: session events and recorded Bot API
|
||||
messages show the slow first and distinct second outcomes without a
|
||||
control-file race. The provider request log (`scriptEntry` 0 then 1, turn
|
||||
order in bodies) is diagnostic context, not the comparison evidence.
|
||||
control-file race. The tamper-evident provider request facts (`scriptEntry` 0
|
||||
then 1, turn order in bodies) independently prove provider arrival order.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# Staged media provider proof
|
||||
|
||||
Use when a change alters how an uploaded document or image reaches the provider.
|
||||
|
||||
```bash
|
||||
lane="$OPENCLAW_TELEGRAM_MANTIS_LANE_CMD"
|
||||
media="$MANTIS_OUTPUT_DIR/sample.pdf"
|
||||
sent="$($lane send --lane baseline --media "$media" --text '@{sut} inspect this document')"
|
||||
message_id="$(jq -er '.sent.messageId' <<<"$sent")"
|
||||
$lane observe --lane baseline --seconds 60 --until-provider-requests 1
|
||||
requests="$($lane requests --lane baseline)"
|
||||
jq -e '[.requests[].contentFacts[]? | select(.type == "legacy_media")] | length > 0' \
|
||||
<<<"$requests"
|
||||
$lane finish --lane baseline --focus-message-id "$message_id"
|
||||
```
|
||||
|
||||
For a reply-mention turn, first `send --media "$media"` without text, capture its
|
||||
`.sent.messageId`, then `send --reply-to "$message_id" --text '@{sut} inspect this document'`.
|
||||
A bare unmentioned upload stages the file but produces no provider turn.
|
||||
|
||||
Repeat for `candidate` with its returned message id, selecting `type == "input_file"`.
|
||||
Assert the complete selected facts: `filename`, `mimeType`, and `byteLength` when present.
|
||||
The structured facts are comparison evidence; never scrape `body` strings.
|
||||
|
||||
If baseline needs a tool round trip, take the tool argument from the recorded
|
||||
`legacy_media.filename`. Build a complete response-events JSON array from
|
||||
`toolCallEvents()` in `scripts/e2e/mock-openai-server.mjs`
|
||||
(`response.output_item.added`, `response.function_call_arguments.delta`,
|
||||
`response.output_item.done`, `response.completed`), then install it before the
|
||||
next turn with `mock --lane baseline --response-events-file <public-json>`.
|
||||
@@ -80,7 +80,8 @@ Use `$OPENCLAW_TELEGRAM_MANTIS_LANE_CMD` with `--lane baseline|candidate`:
|
||||
[--until-provider-requests N]` (returns early when all supplied conditions hold;
|
||||
event/text conditions count only events after the cursor, provider count is
|
||||
cumulative for the lane)
|
||||
- `requests` (redacted provider requests; zero is a valid recorded fact)
|
||||
- `requests` (redacted provider requests; media/file items appear as structured
|
||||
`contentFacts`; zero is a valid recorded fact)
|
||||
- `press --message-id ID --button INDEX`
|
||||
- `delete --message-id ID` (only user messages sent in this session)
|
||||
- `desktop --actions-file <public-json> [--timeout-seconds N]` (run an
|
||||
@@ -126,10 +127,13 @@ baseline lane that reproduces the defect is a successful capture. A PR-level
|
||||
pass claim requires an observed, material baseline/candidate difference caused
|
||||
by the changed behavior. That difference may be trusted Bot API payload/status
|
||||
facts even when pixels are identical; screenshots remain comparison context.
|
||||
Provider request logs are diagnostic and pacing signals, not standalone
|
||||
comparison evidence. Identical pixels alone do not force `block` when the
|
||||
trusted recorded facts differ materially. If neither pixels nor trusted facts prove a
|
||||
difference, use `block`. When the expected result is silence, focus the
|
||||
Provider request facts are tamper-evident comparison evidence: the provider
|
||||
sidecar records them outside the candidate runtime, so candidate code cannot
|
||||
alter or remove a recorded request after the fact. Requests still originate
|
||||
inside the SUT, so the facts prove what the candidate runtime sent — the
|
||||
behavior under proof — not who sent it. Identical pixels alone do not force `block`
|
||||
when the recorded facts differ materially. If neither pixels nor recorded facts
|
||||
prove a difference, use `block`. When the expected result is silence, focus the
|
||||
session-owned user message that triggered the silent outcome.
|
||||
Decide before finalizing each lane. If its setup did not exercise the intended
|
||||
behavior, call `block`; do not call `finish` and describe the block only in prose.
|
||||
|
||||
Reference in New Issue
Block a user