mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-25 11:55:47 -06:00
docs(mantis): script catalog-tool turns through Code Mode exec (#127878)
* docs(mantis): script catalog-tool turns through Code Mode exec Run 32570733988 blocked both lanes with 'Tool pdf not found': the proof agent scripted a top-level pdf function_call, but the SUT agent runs Code Mode where catalog tools execute from JavaScript inside exec. Document the canonical route in the proof prompt, add a complete exec(pdf) scripted-event example plus the document-extract configPatch to the staged-media recipe, and pin the new guidance in the workflow test. * docs(mantis): wait for the exec round trip before finish ClawSweeper P2: send releases the mock response once the outbound message is visible, and finish tears the lane down, so a slow exec/pdf round trip could lose its function_call_output follow-up. Observe until the cumulative provider-request count covers the follow-up and assert the recorded function_call_output before finish; pin the contract in the workflow test.
This commit is contained in:
@@ -11,9 +11,11 @@ $lane observe --lane baseline --seconds 60 --until-provider-requests 1
|
||||
requests="$($lane requests --lane baseline)"
|
||||
jq -e '[.requests[].contentFacts[]? | select(.type == "legacy_media")] | length > 0' \
|
||||
<<<"$requests"
|
||||
$lane finish --lane baseline --focus-message-id "$message_id"
|
||||
```
|
||||
|
||||
With no tool round trip, finish now using `message_id`. Otherwise continue below;
|
||||
`finish` stops the lane.
|
||||
|
||||
For a reply-mention turn, first `send --media "$media"` without text, capture its
|
||||
`.sent.messageId`, then `send --reply-to "$message_id" --text '@{sut} inspect this document'`.
|
||||
A bare unmentioned upload stages the file but produces no provider turn.
|
||||
@@ -22,9 +24,93 @@ Repeat for `candidate` with its returned message id, selecting `type == "input_f
|
||||
Assert the complete selected facts: `filename`, `mimeType`, and `byteLength` when present.
|
||||
The structured facts are comparison evidence; never scrape `body` strings.
|
||||
|
||||
If baseline needs a tool round trip, take the tool argument from the recorded
|
||||
`legacy_media.filename`. Build a complete response-events JSON array from
|
||||
`toolCallEvents()` in `scripts/e2e/mock-openai-server.mjs`
|
||||
(`response.output_item.added`, `response.function_call_arguments.delta`,
|
||||
`response.output_item.done`, `response.completed`), then install it before the
|
||||
next turn with `mock --lane baseline --response-events-file <public-json>`.
|
||||
For a PDF tool round trip, start each lane with this patch. `pdf` is already in
|
||||
the Code Mode catalog; `document-extract` lets the mock OpenAI route execute it.
|
||||
|
||||
```json
|
||||
{ "configPatch": { "plugins": { "allow": ["telegram", "openai", "document-extract"] } } }
|
||||
```
|
||||
|
||||
Replace `<legacy_media.filename>` below with the recorded value and save the
|
||||
array as `pdf-exec-events.json` under `MANTIS_OUTPUT_DIR`:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"type": "response.output_item.added",
|
||||
"item": {
|
||||
"type": "function_call",
|
||||
"id": "fc_mantis_pdf_exec",
|
||||
"call_id": "call_mantis_pdf_exec",
|
||||
"name": "exec",
|
||||
"arguments": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "response.function_call_arguments.delta",
|
||||
"delta": "{\"language\":\"javascript\",\"code\":\"return await pdf({ pdf: \\\"<legacy_media.filename>\\\", prompt: \\\"Inspect this PDF.\\\" });\"}"
|
||||
},
|
||||
{
|
||||
"type": "response.output_item.done",
|
||||
"item": {
|
||||
"type": "function_call",
|
||||
"id": "fc_mantis_pdf_exec",
|
||||
"call_id": "call_mantis_pdf_exec",
|
||||
"name": "exec",
|
||||
"arguments": "{\"language\":\"javascript\",\"code\":\"return await pdf({ pdf: \\\"<legacy_media.filename>\\\", prompt: \\\"Inspect this PDF.\\\" });\"}"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "response.completed",
|
||||
"response": {
|
||||
"id": "resp_mantis_pdf_exec",
|
||||
"status": "completed",
|
||||
"output": [
|
||||
{
|
||||
"type": "function_call",
|
||||
"id": "fc_mantis_pdf_exec",
|
||||
"call_id": "call_mantis_pdf_exec",
|
||||
"name": "exec",
|
||||
"arguments": "{\"language\":\"javascript\",\"code\":\"return await pdf({ pdf: \\\"<legacy_media.filename>\\\", prompt: \\\"Inspect this PDF.\\\" });\"}"
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"input_tokens": 64,
|
||||
"output_tokens": 16,
|
||||
"total_tokens": 80,
|
||||
"input_tokens_details": { "cached_tokens": 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Save this beside it as `pdf-exec-script.json`, install the two-response script,
|
||||
then send the tool-driven turn:
|
||||
|
||||
```json
|
||||
{
|
||||
"responses": [
|
||||
{ "eventsFile": "pdf-exec-events.json" },
|
||||
{ "text": "PDF tool round trip complete." }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
script="$MANTIS_OUTPUT_DIR/pdf-exec-script.json"
|
||||
sha256="$(sha256sum "$script" | cut -d ' ' -f 1)"
|
||||
$lane mock --lane baseline --script "$script" "$sha256"
|
||||
tool_sent="$($lane send --lane baseline --text '@{sut} inspect the staged PDF with the pdf tool')"
|
||||
tool_message_id="$(jq -er '.sent.messageId' <<<"$tool_sent")"
|
||||
$lane observe --lane baseline --seconds 120 --until-provider-requests 3
|
||||
requests="$($lane requests --lane baseline)"
|
||||
jq -e '[.requests[] | .body.input[]? | select(.type == "function_call_output"
|
||||
and .call_id == "call_mantis_pdf_exec")] | length > 0' <<<"$requests"
|
||||
$lane finish --lane baseline --focus-message-id "$tool_message_id"
|
||||
```
|
||||
|
||||
`finish` tears the lane down, so wait for the cumulative provider-request count
|
||||
(staging turn, exec turn, follow-up) and assert the recorded
|
||||
`function_call_output` before finishing; its `output` carries the serialized
|
||||
exec result. Repeat the same script, turn, wait, and assertions for `candidate`.
|
||||
|
||||
@@ -106,6 +106,9 @@ from `responseEvents` in `scripts/e2e/mock-openai-server.mjs`, and use
|
||||
`packages/ai/src/transports/openai-responses-stream-parity.test.ts` for reasoning
|
||||
event examples. These harness sources are safe to read; prepared proof worktrees
|
||||
remain off limits.
|
||||
The SUT agent runs Code Mode. Script catalog-tool turns as an `exec` function
|
||||
call whose JavaScript invokes the catalog tool, such as `pdf(...)`. See
|
||||
`mantis-recipes/staged-media-provider-proof.md` for the complete event script.
|
||||
For normal group turns, address the current bot with `@{sut}`; the harness
|
||||
expands it to the live SUT username. Omit it only when an unmentioned message
|
||||
is intentionally part of the scenario.
|
||||
|
||||
Reference in New Issue
Block a user