Measured on run 31865243804: the 3-part split on
blacksmith-8vcpu-windows-2025 got exactly 2 runners; part 2 queued 211s
and only started when part 1 finished, pushing the wall to 6:45 despite
133-163s job bodies (vs 342s hosted). The runner class admits ~2
concurrent jobs, so any profile that can land on Blacksmith now uses the
single lane; the 3-part split stays for guaranteed-hosted executions
(github mode and dispatches, whose runner expression forces hosted in
every mode). A hybrid retry reruns the single lane hosted - slower but
bounded, and Windows-scoped retries are rare.
Three measured cuts toward the sub-4-minute wall:
- checks-ui-e2e hosted-planner matrix 10 -> 12 rows (11 Vitest shards +
browser-extension). Shard 1 measured 230s with ~123s test body; Vitest
hash sharding leaves count as the only balance lever.
- checks-windows hybrid attempt 1 now rides blacksmith-8vcpu-windows-2025
(retries stay hosted windows-2025). The 3-part hosted split measured a
342s pole on full-graph runs.
- build-artifacts folds the Doctor plugin-index proof (12s), singleton
smoke, and startup-memory check (7s) into one step that overlaps them
on Blacksmith and stays serial on hosted 4-core so the RSS measurement
is unperturbed; any missing startup asset rebuild completes before the
fork so no verifier reads dist mid-write.
Run 31861511624 made checks-ui the UI-scoped wall pole (247s job, 177s body
on hosted 4-core even in hybrid). Attempt 1 now uses the 8 vCPU class like
the ui-e2e shards; retries stay hosted.
The manifest planner closure and the protocol coverage script import only
node builtins and relative files (verified importing the full closure with
an empty node_modules under native type stripping), so push/PR preflight
drops the pnpm store restore and install (~30s off the barrier every lane
waits behind). Manual dispatches keep the tsx path for frozen targets, and
the coverage script inlines the record guard under the documented
dependency-free exception.
The store archive accretes every prior lockfile generation through
prefix-key restores (measured 2.05 GiB, ~36s restore in every hosted job);
the warmup writer now prunes to the current lockfile closure before saving.
Sub-agent caching audit tranche, all measured on runs 31856622489 and
31857193681:
- QA Smoke parts rebuilt the runtime twice and packed a docker tgz that no
scenario in the smoke coverage set consumes (verified against the resolved
plan: zero docker-lane, zero Control UI scenarios). One private overlay
build replaces the public-build/ui:build/pack/private-rebuild chain
(~35-50s per part); the run step fails closed if a docker-lane scenario
ever returns, and the guard now pins never-pack-after-private-build.
- report-plugin-sdk-api-diff is pure reporting (nothing passes
--require-acknowledgement); it now runs on manual/release dispatches only.
- check-npm-lock resolved ~94 npm graphs against the registry on every push;
push/PR runs now use the reviewed --changed scoping (2.4s on dep-free
diffs) with a fail-open full sweep on base-fetch failure, and dispatches
keep --all for registry-drift coverage.
- Six restore-test-caches gates said 'github' only while the writer already
runs in hybrid; hybrid attempt-1 lanes now restore the transform seed.
- QA smoke lane reads the protected build compile cache; test-type stripes
widen to five.
Green tip run 31855637027 measured the remaining poles: ui-e2e shards up to
282s (imbalanced eight-way split), report-plugin-sdk-api-diff 246s on 4 vCPU,
and 220s test-type stripes (tsgo saturates a machine per graph, so stripe
count is the lever, not cores). Widen hosted-profile ui-e2e to ten rows,
test-type stripes to four, and lift the SDK diff to the 8 vCPU class.
First hybrid main cycles (runs 31850815388-31851455918) landed at 5:47-6:32
walls with every remaining pole a hosted lane: the three test-type stripes
(232-269s), check-additional-runtime-topology (274s), check-dependencies
(260s), and report-plugin-sdk-api-diff (209s). Route them to their Blacksmith
labels on hybrid attempt 1 (retries stay hosted) and lift runtime topology to
the 8 vCPU class.
* feat(gateway): add remote-exec cloud placements
* feat(codex): run cloud turns through remote exec
* fix(sandbox): quote ssh_config path directives containing whitespace
Crabbox lease keys default to ~/Library/Application Support/... on macOS;
unquoted IdentityFile/UserKnownHostsFile/CertificateFile arguments tokenize
on the space and fail as 'extra arguments'. Found via live remote-exec
cloud-worker proof.
* test: consolidate gateway maintenance schedule coverage
* fix(ci): invalidate plugin sdk declarations on state changes
* perf: use blobless clone in git installer
* refactor(install): move clone-filter rationale above the call
Keep the file's comment convention (comments on their own lines) instead of a
150-column trailing comment, and record why blob:none is preferred over
--depth 1 plus why no fallback is needed.
Narrow the ordering test's source needle to the stable prefix so it stops
duplicating the behavioral flag assertion and no longer breaks whenever the
clone flags change.
install_build_tools_linux ran run_quiet_step and then returned 0
unconditionally from every package-manager branch, so a failed build-tool
install was printed as an error and immediately overwritten with a success
line. The installer emitted "Installing build tools failed" followed by
"Build tools installed", and the caller's "Continuing without auto-installing
build tools" warning was unreachable. The npm-failure recovery path likewise
reported "Build tools setup complete" and retried an install that could not
succeed. install_build_tools_macos already propagated its status; only the
Linux sibling swallowed it.
Return the package manager's exit status and collapse the five duplicated
is_root/sudo branches into one command list, which removes the bug class
rather than patching a single branch.
Verified against the live openclaw.ai script: a stubbed apt-get failure
reproduced the false success before the change and reports the honest warning
after it.
* perf(ci): cut hosted CI critical path toward five-minute walls
Stripe the serial core test-type graphs across two hosted jobs and drop the
duplicated tsgo:test:root pass; gate the six-part QA Smoke matrix off pull
requests unless a QA-owned surface changed; split the fat multi-config Node
shards (cli/cli-process, unit-fast isolated/fake-timers, infra
logging-process/runtime-config) and lower the hosted split ceiling to 150
predicted seconds so no compact lane owns a ~280s wall; expand tooling to
seven stripes.
* perf(ci): widen hosted test-type striping to three jobs
Run 31825922122 measured ~40s per core test-type graph on loaded hosted
runners (282s worst stripe body of the two-way split); three stripes keep
each lane near 150s body under load.
scripts/mac-elevation-host.sh ships as the portable installer added in
#123675, so it is deliberately self-contained and cannot source shared
constants. That forces it to duplicate ELEVATION_TEAM_ID and
ELEVATION_IDENTITY from scripts/codesign-mac-app.sh, with nothing keeping
the two copies in sync. Add a test asserting they agree, so rotation drift
fails at build time instead of on a target Mac.
Also document why the sign-time and install-time elevation verification
functions both exist: verify_elevation_app requires a notarized and stapled
bundle, so it cannot replace the sign-time check, and deleting that check
would defer every identity failure until after a notarization submission is
already spent.
No behavior change: no artifact bytes, filenames, modes, receipt fields,
signing, notarization, or install paths are touched.
The openclaw-mlx-tts voice helper pulls in the full mlx-swift Metal shader
stack, which some beta Xcode toolchains (e.g. Xcode 27 / macOS 27 SDK) cannot
compile: the metal compiler dies non-deterministically (a different .metal file
each run, 'Could not read serialized diagnostics file'). The main app builds
fine, so an unrelated dev/proof build should not be blocked by the helper.
Add OPENCLAW_SKIP_MLX_TTS=1 (matching the sibling SKIP_TSC/SKIP_UI_BUILD
toggles) to package the app without the voice helper, gating both the per-arch
build and the bundle copy. Refuse the flag for release builds, which must ship
the helper (notarization verifies it), so a skipped build can never become a
silently incomplete release.
* fix(snapshot): survive cold PowerShell starts in Windows staging gates
CI run 31775262530, checks-windows-node-test-1 attempt 1, showed the fail-closed ACL probe timing out during PowerShell first-use module preparation. Centralize encoded one-shot spawning, budget 60 seconds for cold starts, and preserve the underlying probe failure as the error cause.
* fix(snapshot): sanitize PowerShell failure causes in Windows staging gates
* fix(secrets): explain the sanitized plan-file failure cause suppression
check-lint-core-2 flagged preserve-caught-error at the private plan file
catch; retaining the raw error would re-leak the -EncodedCommand argv the
sanitization contract strips, so the suppression is intentional (same
idiom as setup-inference-activate.ts).
* test(lint): register the private-plan-file suppression in the inventory
* test(infra): give the LAN-host real PowerShell spawn a cold-start budget
checks-windows-node-test-2 (run 31804325922) hit the same cold-start flake
class this PR fixes: the codepage-proof test spawns real powershell.exe
bounded at 3s, which a cold runner cannot meet. Production keeps its
fail-open 3s route-hint probe; only the test's real-spawn verification
uses the shared cold-spawn budget.
Hosted CI runners restored the boundary-artifact cache and rebuilt it anyway: fresh checkouts re-stamp every input mtime, so mtime freshness never passed. Stamp files now record the input content digest and byte-identical inputs skip the rebuild (~60s saved per hosted lint/boundary job, 0.17s verify). Telegram CI shards pack ten files per job instead of five now that per-file import cost is back to seconds (#123607), halving the ~42-job fanout.
Five-file Telegram jobs finished the first file, then isolate re-imported the next graph in silence until the 300s watchdog killed the worker. Recycle the Vitest process after each file and keep five files per CI job.
Preserve externally scoped Telegram and Matrix test plans instead of expanding each CI shard back into the full extension suite. Keep broad runs bounded and retain external include ownership through directory run specs.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Bound Telegram extension tests to five files per Vitest process across explicit config, directory, and full-suite routes while preserving serial isolated execution.
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Peter Steinberger