mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-27 12:56:01 -06:00
perf(ci): serve node_modules snapshots from O(1) protected sticky disks (#109752)
* perf(ci): serve node_modules snapshots from O(1) protected sticky disks Every Blacksmith mount of the dependency sticky disk currently 429s: the v2 key minted one backing disk per PR and per manifest hash, which saturated Blacksmith's installation-wide 1000-sticky-disk budget (run 29559333389, job checks-node-compact-small-15: "Sticky disk limit exceeded"). Same-repo shards then fall back to a cold, storeless pnpm install (~40s) on every run - worse than the ~22s actions/cache path the sticky rollout replaced. Re-key the snapshot to one stable disk per node-version and move the install inputs into a runtime fingerprint marker: - Key is now `<repo>-node-deps-bind-v3-<node-version>`; dependency changes refresh the disk in place instead of allocating a new one. - The fingerprint (manifest hashFiles set + node version + lockfile mode) is evaluated on the bind step, before the mount lands, so the '**/package.json' glob cannot sweep snapshot-internal manifests. - Consumers mount read-only (commit: false); on fingerprint match they restore importer archives and skip pnpm install entirely, on mismatch they install against the clone's on-disk store (warm store, no actions/cache download) without capturing. - Writers are trusted non-PR jobs only: build-artifacts on canonical pushes plus the scheduled vitest-cache-warm run as a deadline writer (main pushes cancel each other under merge traffic and would starve the snapshot). commit: on-change keeps warm no-op runs from committing. - Roll the consumer flag out to every Linux Blacksmith lane that installs dependencies (build-artifacts, check-shard, check-additional-shard, check-docs, checks-ui, control-ui-i18n, native-i18n, qa-smoke-ci-profile, checks-fast-core, both contract shards) with the same fork/dispatch gates as the nondist shard. Guards now enumerate the consumer set, pin the O(1) key shape, enforce single-writer commit expressions, and cover the fingerprint marker in the importer capture/restore helper. Verified locally: importer archive capture 0.96s/228K and restore 0.07s against the real 2.0GB hoisted tree; a snapshot untarred into a fresh workspace resolves modules, runs bin shims, `pnpm exec`, and a Vitest suite (one-time pnpm reconcile only when the absolute path changes, which Blacksmith's fixed /home/runner/_work layout avoids). * fix(ci): commit dependency snapshots explicitly, not via on-change heuristic stickydisk's on-change mode compares allocated disk bytes with a 4KB threshold, so a fingerprint refresh whose reinstall keeps usage stable (metadata-only manifest edits, same-sized dependency swaps) could be silently discarded, stranding every consumer on a stale marker and a permanent reinstall path. Writers now commit explicitly, mirroring the Vitest transform disk's rationale; warm no-op writer runs re-commit an identical snapshot, which is cheap and safe. Also document that the non-PR commit gate binds cooperating code only - the enforced trust boundary stays the fork/dispatch runner gate, matching the protected node-compile disk's posture. * test(ci): guard sticky consumers against writerless node-version key splits Reviewer follow-up: the snapshot key is partitioned by node-version and both writers (build-artifacts, vitest-cache-warm) rely on the action default, so a consumer pinning any other version would split onto a key nobody seeds and silently regress to permanently cold installs. Pin the action default to 24.x and assert every sticky consumer resolves to that same key segment. * test: narrow sticky-consumer step.with before indexing The filter's runtime narrowing did not carry into the map callback, so step.with indexing failed check-test-types (TS18048). Collect a narrowed stepWith instead.
This commit is contained in:
Peter Steinberger
committed by
GitHub
GitHub
parent
d26951ae9a
commit
f69cb2f75e
@@ -1016,6 +1016,15 @@ jobs:
|
||||
install-bun: "false"
|
||||
node-compile-cache: "true"
|
||||
node-compile-cache-scope: "build"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
# Designated dependency-snapshot writer: exactly one trusted-push job
|
||||
# per node-version key refreshes the protected disk; every other
|
||||
# sticky consumer mounts read-only.
|
||||
save-sticky-disk: "true"
|
||||
runtime-cache-sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
save-node-compile-cache: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && 'true' || 'false' }}
|
||||
|
||||
@@ -1227,6 +1236,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Check native app i18n inventory
|
||||
run: pnpm native:i18n:check
|
||||
@@ -1254,6 +1268,11 @@ jobs:
|
||||
with:
|
||||
node-version: "24.x"
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
run: |
|
||||
@@ -1298,6 +1317,11 @@ jobs:
|
||||
with:
|
||||
node-version: "24.x"
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Check Control UI locale parity
|
||||
run: pnpm ui:i18n:check
|
||||
@@ -1371,6 +1395,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: ${{ matrix.task == 'bun-launcher' && 'true' || 'false' }}
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Run ${{ matrix.task }} (${{ matrix.runtime }})
|
||||
env:
|
||||
@@ -1500,6 +1529,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Set up Blacksmith Docker layer cache
|
||||
if: ${{ matrix.docker_cache == true && github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') }}
|
||||
@@ -1671,6 +1705,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Run plugin contract shard
|
||||
env:
|
||||
@@ -1709,6 +1748,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Run channel contract shard
|
||||
env:
|
||||
@@ -1925,6 +1969,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
# check-lint's shard runner rebuilds the same plugin-sdk boundary
|
||||
# artifacts the boundary lane snapshots (~72s cold); restore them from
|
||||
@@ -2143,6 +2192,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
# Same-repo runs carry boundary artifacts on a Blacksmith sticky disk:
|
||||
# the GitHub cache below is evicted so quickly under the repo quota that
|
||||
@@ -2382,6 +2436,11 @@ jobs:
|
||||
uses: ./.github/actions/setup-node-env
|
||||
with:
|
||||
install-bun: "false"
|
||||
# Blacksmith same-repo runs clone dependencies from a sticky disk.
|
||||
# Fork PRs must keep actions/cache: sticky snapshots are writable,
|
||||
# repository-global state and must never be produced by fork code.
|
||||
sticky-disk: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'true' || 'false' }}
|
||||
use-actions-cache: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'openclaw/openclaw' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == 'openclaw/openclaw') && 'false' || 'true' }}
|
||||
|
||||
- name: Check formatting
|
||||
if: needs.preflight.outputs.run_format_check == 'true'
|
||||
|
||||
@@ -30,6 +30,11 @@ jobs:
|
||||
node-compile-cache-scope: "test"
|
||||
runtime-cache-sticky-disk: "true"
|
||||
save-node-compile-cache: "true"
|
||||
# Daily deadline writer for the protected dependency snapshot:
|
||||
# canonical main pushes cancel each other under merge traffic, so
|
||||
# build-artifacts commits can starve; this scheduled run cannot be
|
||||
# superseded and keeps the snapshot within a day of main.
|
||||
save-sticky-disk: "true"
|
||||
save-vitest-fs-cache: "true"
|
||||
sticky-disk: "true"
|
||||
use-actions-cache: "false"
|
||||
|
||||
Reference in New Issue
Block a user