mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-24 19:35:28 -06:00
fix(ci): settle Claude cache probe before generation baseline (#128507)
* test(gateway): settle Claude cache probe before baseline * docs(test): describe Claude cache settlement probe
This commit is contained in:
Vincent Koc
@@ -176,7 +176,7 @@ openclaw models list --json
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_IMAGE_ARG="--image"` to pass image file paths as CLI args instead of prompt injection.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_IMAGE_MODE="repeat"` (or `"list"`) to control how image args are passed when `IMAGE_ARG` is set.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_RESUME_PROBE=1` to send a second turn and validate resume flow.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_CACHE_PROBE=1` to run a fresh Claude CLI turn plus two native resumes and require at least 90% prompt-cache reuse on the second resume. This probe disables the image, MCP, and model-switch probes.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_CACHE_PROBE=1` to run a fresh Claude CLI turn, a tool-bearing warmup resume, and a no-tool settlement resume before requiring at least 90% prompt-cache reuse on the following dirty-workspace resume. It also verifies that a thinking-level change rotates the live-session generation and that the next steady resume restores at least 90% reuse. This probe disables the image, MCP, and model-switch probes.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_MODEL_SWITCH_PROBE=1` to opt into the Claude Sonnet -> Opus same-session continuity probe when the selected model supports a switch target. Off by default, including in Docker recipes.
|
||||
- `OPENCLAW_LIVE_CLI_BACKEND_MCP_PROBE=1` to opt into the MCP/tool loopback probe. Off by default in Docker recipes.
|
||||
|
||||
@@ -218,7 +218,7 @@ pnpm test:docker:live-cli-backend:gemini
|
||||
Notes:
|
||||
|
||||
- The Docker runner lives at `scripts/test-live-cli-backend-docker.sh`.
|
||||
- `pnpm test:docker:live-cli-backend:claude:cache` requires Anthropic API-key auth. It logs normalized cache usage for both resumes and fails when the second resume falls below 90% reuse.
|
||||
- `pnpm test:docker:live-cli-backend:claude:cache` requires Anthropic API-key auth. It logs normalized cache usage for every cache-probe resume and requires at least 90% reuse on both the post-settlement dirty-workspace resume and the steady resume after a thinking-level change.
|
||||
- It runs the live CLI-backend smoke inside the repo Docker image as the non-root `node` user.
|
||||
- It resolves CLI smoke metadata from the owning plugin, then installs the matching Linux CLI package (`@anthropic-ai/claude-code` or `@google/gemini-cli`) into a cached writable prefix at `OPENCLAW_DOCKER_CLI_TOOLS_DIR` (default: `~/.cache/openclaw/docker-cli-tools`).
|
||||
- `codex-cli` is no longer a bundled CLI backend; use `openai/*` with the Codex app-server runtime instead (see [Live: Codex app-server harness smoke](#live-codex-app-server-harness-smoke)).
|
||||
|
||||
@@ -209,7 +209,7 @@ OPENCLAW_LIVE_TEST=1 OPENCLAW_LIVE_CACHE_TEST=1 pnpm test:live:cache
|
||||
|
||||
The baseline file stores the most recently observed live numbers plus the provider-specific regression floors the test checks against. Each run uses fresh per-run session IDs and prompt namespaces so previous cache state does not pollute the current sample. Anthropic and OpenAI use different enforcement: an Anthropic floor miss is a hard regression (test fails), while an OpenAI floor miss is watch-only (recorded as a warning, does not fail the run). They do not share a single cross-provider threshold.
|
||||
|
||||
Claude CLI prompt reuse has a separate Docker lane because it exercises Claude Code's native session transport rather than the direct Anthropic API. It runs one fresh turn plus two native resumes, logs the normalized usage for both resumes, and requires at least 90% reuse on the second resume:
|
||||
Claude CLI prompt reuse has a separate Docker lane because it exercises Claude Code's native session transport rather than the direct Anthropic API. After a fresh turn and tool-bearing warmup resume, it allows a no-tool settlement resume to run hot or cold, dirties the workspace, and requires at least 90% reuse on the following resume without rotating the settled live-session generation. It also verifies that a thinking-level change rotates the generation and that the next steady resume restores at least 90% reuse:
|
||||
|
||||
```sh
|
||||
pnpm test:docker:live-cli-backend:claude:cache
|
||||
|
||||
@@ -158,7 +158,7 @@ Other behavior: the runner preflights Docker by default, cleans stale OpenClaw E
|
||||
| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `pnpm test:docker:browser-cdp-snapshot` | Chromium-backed source E2E container with raw CDP + isolated Gateway; `browser doctor --deep` CDP role snapshots include link URLs, cursor-promoted clickables, iframe refs, and frame metadata. |
|
||||
| `pnpm test:docker:skill-install` | Installs the packed tarball in a bare Docker runner with `skills.install.allowUploadedArchives: false`, resolves a current skill slug from live ClawHub search, installs via `openclaw skills install`, and verifies `SKILL.md`, `.clawhub/origin.json`, `.clawhub/lock.json`, and `skills info --json`. |
|
||||
| `pnpm test:docker:live-cli-backend:claude`, `:claude:resume`, `:claude:cache`, `:claude:mcp` | Focused CLI backend live probes; `:claude:cache` requires at least 90% prompt-cache reuse on the second native resume. Gemini has matching `:resume` and `:mcp` aliases. |
|
||||
| `pnpm test:docker:live-cli-backend:claude`, `:claude:resume`, `:claude:cache`, `:claude:mcp` | Focused CLI backend live probes; `:claude:cache` settles the no-tool prompt shape, then requires at least 90% prompt-cache reuse on the following dirty-workspace resume and on the steady resume after a thinking-level change. Gemini has matching `:resume` and `:mcp` aliases. |
|
||||
| `pnpm test:docker:openwebui` | Dockerized OpenClaw + Open WebUI: sign in, check `/api/models`, run a real proxied chat through `/api/chat/completions`. Requires a usable live model key and pulls an external image; not expected to be CI-stable like the unit/e2e suites. |
|
||||
| `pnpm test:docker:mcp-channels` | Seeded Gateway container plus a client container spawning `openclaw mcp serve`: routed conversation discovery, transcript reads, attachment metadata, live event queue behavior, outbound send routing, and Claude-style channel + permission notifications over the real stdio bridge (assertion reads raw stdio MCP frames directly). |
|
||||
| `pnpm test:docker:upgrade-survivor` | Installs the packed tarball over a dirty old-user fixture, runs package update plus non-interactive doctor without live provider/channel keys, starts a loopback Gateway, checks agents/channel config/plugin allowlists/workspace/session state/stale legacy plugin dependency state/startup/RPC status survive. |
|
||||
|
||||
@@ -803,10 +803,6 @@ describeLive("gateway live (cli backend)", () => {
|
||||
const resumeText = extractPayloadText(resumePayload?.result);
|
||||
if (CLI_CACHE_PROBE) {
|
||||
expect(resumeText).toContain(schemaProbePlugin?.resultToken);
|
||||
// The first turn advertises one bootstrap-only tool. Establish the process baseline
|
||||
// after that tool retires so steady-turn reuse is not confused with valid schema drift.
|
||||
cacheProbeSteadyGeneration = getClaudeGeneration(cacheProbeOwner!);
|
||||
expect(cacheProbeSteadyGeneration).toBeTruthy();
|
||||
} else if (providerId === "codex-cli") {
|
||||
expect(resumeText).toContain(`CLI-RESUME-${resumeNonce}`);
|
||||
} else if (resumeContinuityProbe) {
|
||||
@@ -851,6 +847,19 @@ describeLive("gateway live (cli backend)", () => {
|
||||
expect(extractPayloadText(probePayload.result)).toContain(marker);
|
||||
return logCliCacheUsage(turn, probePayload.result);
|
||||
};
|
||||
const settleNonce = randomBytes(3).toString("hex").toUpperCase();
|
||||
const settleHitRate = await requestCacheProbeTurn(
|
||||
"resume1-settle",
|
||||
`CLI-CACHE-SETTLE-${settleNonce}`,
|
||||
);
|
||||
if (settleHitRate === undefined) {
|
||||
return;
|
||||
}
|
||||
// The first turn advertises one bootstrap-only tool. Allow the no-tool settle turn to
|
||||
// run hot or cold, then capture the steady process after any valid schema rotation.
|
||||
cacheProbeSteadyGeneration = getClaudeGeneration(cacheProbeOwner!);
|
||||
expect(cacheProbeSteadyGeneration).toBeTruthy();
|
||||
|
||||
const cacheNonce = randomBytes(3).toString("hex").toUpperCase();
|
||||
// Dirty the workspace between captured turns while the compatible Claude flag keeps
|
||||
// its native Git-status section out of the stable prompt prefix.
|
||||
|
||||
Reference in New Issue
Block a user