diff --git a/.agents/skills/control-ui-e2e/SKILL.md b/.agents/skills/control-ui-e2e/SKILL.md index e6e0ab040528..b35d12fa980d 100644 --- a/.agents/skills/control-ui-e2e/SKILL.md +++ b/.agents/skills/control-ui-e2e/SKILL.md @@ -19,7 +19,7 @@ Use this for Control UI changes that need a real browser flow with deterministic - Target one E2E test in a Codex worktree: ```bash -node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/ui/e2e/chat-flow.e2e.test.ts +node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/e2e/chat-flow.messaging.e2e.test.ts ``` - Run the whole local lane in a normal checkout: diff --git a/.agents/skills/openclaw-test-heap-leaks/SKILL.md b/.agents/skills/openclaw-test-heap-leaks/SKILL.md index 9d83d7a889ce..b9cceb42c603 100644 --- a/.agents/skills/openclaw-test-heap-leaks/SKILL.md +++ b/.agents/skills/openclaw-test-heap-leaks/SKILL.md @@ -7,22 +7,30 @@ description: Investigate OpenClaw pnpm test memory growth, Vitest OOMs, RSS spik Use this skill for test-memory investigations. Do not guess from RSS alone when heap snapshots are available. Treat snapshot-name deltas as triage evidence, not proof, until retainers or dominators support the call. -For **runtime fixes** (e.g., closure leaks in long-running services like the gateway), see [Validating runtime fixes](#validating-runtime-fixes-not-test-memory) below — that uses a dedicated harness, not the test-parallel snapshot machinery. +Read `../openclaw-test-performance/SKILL.md` first for the current test-performance commands and proof routing. + +For **runtime fixes** (e.g., closure leaks in long-running services like the gateway), see [Validating runtime fixes](#validating-runtime-fixes-not-test-memory) below — that uses a dedicated harness rather than the unit-test profiling workflow. ## Workflow 1. Reproduce the failing shape first. - - Match the real entrypoint if possible. For Linux CI-style unit failures, start with: - - `pnpm canvas:a2ui:bundle && OPENCLAW_TEST_MEMORY_TRACE=1 OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap OPENCLAW_TEST_WORKERS=2 OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 pnpm test` - - Keep `OPENCLAW_TEST_MEMORY_TRACE=1` enabled so the wrapper prints per-file RSS summaries alongside the snapshots. - - If the report is about a specific shard or worker budget, preserve that shape. - - Before you analyze snapshots, identify the real lane names from `[test-parallel] start ...` lines or `pnpm test --plan`. Do not assume a single `unit-fast` lane; local plans often split into `unit-fast-batch-*`. + - Match the real entrypoint and worker budget. For a broad unit-fast baseline with per-config max RSS and top-file timing, start with: -2. Wait for repeated snapshots before concluding anything. - - Take at least two intervals from the same lane. - - Compare snapshots from the same PID inside the real lane directory such as `.tmp/heapsnap/unit-fast-batch-2/`. - - Use `.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs` to compare either two files directly or the earliest/latest pair per PID in one lane directory. - - If the helper suggests transformed-module retention, confirm the top entries in DevTools retainers/dominators before calling it solved. + ```bash + pnpm test:perf:groups \ + --config test/vitest/vitest.unit-fast.config.ts \ + --allow-failures \ + --output .artifacts/test-perf/unit-fast-memory.json + ``` + + - For a suspected file, rerun that file with one worker and collect wall/RSS evidence: `/usr/bin/time -l pnpm test --maxWorkers=1 --reporter=verbose`. + - Current `pnpm test` execution is planned by `scripts/test-projects.mjs`. Record the printed Vitest config or shard and preserve that shape when the report is configuration- or worker-budget-specific. + +2. Collect the strongest available heap evidence. + - Run `pnpm test:perf:profile:runner -- --output-dir .artifacts/test-perf/vitest-runner-profile -- ` for a CPU profile plus a sampling heap profile of the unit runner. Open the heap profile in DevTools and inspect the largest allocation families. + - Sampling `.heapprofile` output is not a `.heapsnapshot`; do not pass it to the snapshot delta helper. + - When a dedicated harness or another known producer emits repeated `.heapsnapshot` files, compare snapshots from the same PID with `.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs`. + - If no snapshot or heap profile is available, RSS growth is useful triage evidence, but keep the leak classification inconclusive. 3. Classify the growth before choosing a fix. - If growth is dominated by Vite/Vitest transformed source strings, `Module`, `system / Context`, bytecode, descriptor arrays, or property maps, treat it as likely retained module graph growth in long-lived workers. @@ -31,25 +39,23 @@ For **runtime fixes** (e.g., closure leaks in long-running services like the gat 4. Fix the right layer. - For likely retained transformed-module growth in shared workers: - - Prefer timing and hotspot-driven scheduling fixes first. Check whether the file is already represented in `test/fixtures/test-timings.unit.json` and whether `scripts/test-update-memory-hotspots.mjs` should refresh the measured hotspot manifest before hand-editing behavior overrides. - - Move hotspot files out of the real shared lane by updating `test/fixtures/test-parallel.behavior.json` only when timing-driven peeling is insufficient. - - Prefer `singletonIsolated` for files that are safe alone but inflate shared worker heaps. - - If the file should already have been peeled out by timings but is absent from `test/fixtures/test-timings.unit.json`, call that out explicitly. Missing timings are a scheduling blind spot. + - Inspect the owning Vitest config and the process chunking in `scripts/test-projects.test-support.mjs`. Fix process lifetime or project ownership there only when the same-shape evidence shows that shared-worker retention is the cause. + - `test/vitest/vitest.unit-fast-isolated.config.ts` is for audited stateful tests that need a fresh module graph. Do not use it as a generic memory-hotspot list. - For real leaks: - Patch the implicated test or runtime cleanup path. - Look for missing `afterEach`/`afterAll`, module-reset gaps, retained global state, unreleased DB handles, or listeners/timers that survive the file. 5. Verify with the most direct proof. - - Re-run the targeted lane or file with heap snapshots enabled if the suite still finishes in reasonable time. - - If snapshot overhead pushes tests over Vitest timeouts, fall back to the same lane without snapshots and confirm the RSS trend or OOM is reduced. - - For wrapper-only changes, at minimum verify the expected lanes start and the snapshot files are written. + - Re-run the same grouped or scoped command and confirm the max-RSS trend or OOM is reduced. + - When a dedicated producer emits heap profiles or snapshots, repeat the same producer and compare equivalent artifacts. + - For routing or config changes, verify the expected Vitest config or shard starts and the affected tests complete. ## Heuristics -- Do not call everything a leak. In this repo, large `unit-fast` or `unit-fast-batch-*` growth can be a worker-lifetime problem rather than an application object leak. -- `scripts/test-parallel.mjs` and `scripts/test-parallel-memory.mjs` are the primary control points for wrapper diagnostics. -- The lane names printed by `[test-parallel] start ...` and `[test-parallel][mem] summary ...` tell you where to focus. -- When one or two files account for most of the delta and they are missing from timings, reducing impact by isolating them is usually the first pragmatic fix. +- Do not call everything a leak. Growth in a non-isolated shared Vitest project can be a worker-lifetime problem rather than an application object leak. +- `scripts/test-projects.mjs`, `scripts/test-group-report.mjs`, and `scripts/run-vitest-profile.mjs` are the current execution, grouped-RSS, and profile entrypoints. +- The `[test] starting ...` lines identify the Vitest config or shard to reproduce. +- `.artifacts/vitest-shard-timings.json` stores config/shard durations for scheduling. It is not a file-level memory-hotspot or behavior manifest. - When the same retained object families grow across multiple intervals in the same worker PID, trust the snapshots over intuition, then confirm ambiguous calls with retainer evidence. ## Snapshot Comparison @@ -57,13 +63,13 @@ For **runtime fixes** (e.g., closure leaks in long-running services like the gat - Direct comparison: - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot` - Auto-select earliest/latest snapshots per PID within one lane: - - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast-batch-2` + - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir ` - Useful flags: - `--top 40` - `--min-kb 32` - `--pid 16133` -Read the top positive deltas first. Large positive growth in module-transform artifacts suggests lane isolation; large positive growth in runtime objects suggests a real leak. If the names alone do not settle it, open the same snapshot pair in DevTools and inspect retainers/dominators for the top rows before declaring root cause. +Read the top positive deltas first. Large positive growth in module-transform artifacts points to shared-process lifetime or project ownership; large positive growth in runtime objects suggests a real leak. If the names alone do not settle it, open the same snapshot pair in DevTools and inspect retainers/dominators for the top rows before declaring root cause. ## Validating runtime fixes (not test-memory) @@ -102,8 +108,8 @@ scope of the function where the leak lives, not be a generic abort-loop. When using this skill, report: - The exact reproduce command. -- Which lane and PID were compared. -- The dominant retained object families from the snapshot delta. +- Which Vitest config or target was reproduced, and which PID was compared when snapshots were available. +- The dominant retained object families from the heap profile or snapshot delta, when available. - Whether the issue is a likely real leak or likely shared-worker retained module growth, plus whether retainers/dominators confirmed it. - The concrete fix or impact-reduction patch. - What you verified, and what snapshot overhead prevented you from verifying. diff --git a/.agents/skills/openclaw-test-performance/SKILL.md b/.agents/skills/openclaw-test-performance/SKILL.md index cbd4a81e6eac..6c4ac6ef8d22 100644 --- a/.agents/skills/openclaw-test-performance/SKILL.md +++ b/.agents/skills/openclaw-test-performance/SKILL.md @@ -16,8 +16,9 @@ guesswork. - `src/channels/AGENTS.md` and `src/plugins/AGENTS.md` for plugin/channel laziness. - `src/gateway/AGENTS.md` for server lifecycle tests. - - `test/helpers/AGENTS.md` and `test/helpers/channels/AGENTS.md` for shared - contract helpers. + - `test/helpers/AGENTS.md` and + `src/channels/plugins/contracts/test-helpers/AGENTS.md` for shared contract + helpers. - `src/infra/outbound/AGENTS.md` for outbound/media/action tests. 2. Establish a baseline before changing code: - Prefer `pnpm test:perf:groups --full-suite --allow-failures --output ` @@ -113,7 +114,7 @@ same command. For Testbox comparisons, use the same `tbx_...` id when possible. | import duration | broad barrel/runtime loads | `OPENCLAW_VITEST_IMPORT_DURATIONS=1` | | max RSS | memory pressure and OOM risk | `/usr/bin/time -l`, `pnpm test:extensions:memory`, wrapper memory summaries | | CPU/user/sys | CPU-bound vs wait-bound split | `/usr/bin/time -l` locally, Testbox job timing when local CPU is noisy | -| heap snapshots | real leak vs retained module graph | `openclaw-test-heap-leaks` workflow | +| heap evidence | real leak vs retained module graph | `openclaw-test-heap-leaks` workflow | Local scoped command with CPU/RSS: @@ -137,12 +138,12 @@ pnpm test:extensions:memory -- --extension discord --extension telegram --skip-c Heap/RSS escalation: ```bash -OPENCLAW_TEST_MEMORY_TRACE=1 \ -OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 \ -OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap \ -OPENCLAW_TEST_WORKERS=2 \ -OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 \ -pnpm test +pnpm test:perf:groups \ + --config test/vitest/vitest.unit-fast.config.ts \ + --allow-failures \ + --output .artifacts/test-perf/unit-fast-memory.json +pnpm test:perf:profile:runner -- \ + --output-dir .artifacts/test-perf/vitest-runner-profile -- ``` Use `openclaw-test-heap-leaks` when RSS keeps growing across intervals, workers @@ -170,8 +171,6 @@ leak until snapshots or retainers support it. - Runtime/default model/auth selection paid by idle snapshots or fixtures. - Plugin-owned media/action discovery triggered before checking whether args contain plugin-owned fields. -- Timings missing from `test/fixtures/test-timings.unit.json`, causing hotspot - files to stay in shared workers. - Parallel Vitest runs sharing `node_modules/.experimental-vitest-cache` without distinct `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH` values. diff --git a/.agents/skills/release-openclaw-maintainer/SKILL.md b/.agents/skills/release-openclaw-maintainer/SKILL.md index 574d8a8f3f1d..72cc7d7ac40c 100644 --- a/.agents/skills/release-openclaw-maintainer/SKILL.md +++ b/.agents/skills/release-openclaw-maintainer/SKILL.md @@ -742,7 +742,7 @@ node --import tsx scripts/openclaw-npm-postpublish-verify.ts `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`. If either cannot be injected, stop before starting those local long lanes and report the missing key. - Live credentialed channel QA is the GitHub Actions workflow - `QA-Lab - All Lanes` (`.github/workflows/qa-live-telegram-convex.yml`), not a + `QA-Lab - All Lanes` (`.github/workflows/qa-live-transports-convex.yml`), not a local substitute. Dispatch it from Actions against the release tag and wait for it to pass before npm preflight/publish readiness. Use a SHA only when it satisfies the workflow's secret-bearing trust gate: main ancestor or open PR