perf(ci): price agents-core-models from its measured hybrid wall

`agentic-agents-core-models` carried the 36s scaled estimate while measuring
56.3s (n=6, p90 58.6s) across 260 compact jobs on 2026-08-16. Packed beside
`agentic-agents-core-runtime-hosted-1` (60.4s) it built the only bin running
>=1.25x its prediction: compact-large-19 ran 122s of work priced at 88s, which
made it the most frequent slowest job in the workflow.

Add the measured value to the hybrid hint map, which already exists for exactly
these Blacksmith-specific observations. The packer then separates the two, and
the tallest actual compact bin drops 122s -> 109s with no bin above 1.25x and
no change to row counts (48 push / 56 pull-request), so this costs no extra
runner registrations.

Two things deliberately left alone. The `agentic-gateway-core-3` 140s pin looks
like a 2x over-prediction against its 68.5s median, but run 31938297538 really
did take 138.0s (109.03s + 28.98s across its two configs) on a run whose fleet
slowdown factor was 0.98 -- a genuine tail on a healthy run, so unpinning it
would rebuild a >200s bin whenever the tail lands. Every other bin already sits
within 1.24x of its prediction.

The replaced guard pinned one bin arrangement (`runtime-hosted-1` not sharing a
job with `agents-core-tools`) with no stated failure mode, so any honest refit
broke it. It now asserts the property that comment was reaching for -- both
weight sources survive rebalancing under the body ceiling -- plus the specific
regression this fixes. Mutation-checked: dropping the hint fails the new guard.

Measurement note for the next refit: sum a shard's per-config Duration lines
before taking a median. Pooling them reads as a large over-prediction that is
not there.
This commit is contained in:
Peter Steinberger
2026-08-16 16:34:04 -07:00
parent fecb377da6
commit 1d2d44c8ff
2 changed files with 28 additions and 6 deletions
+6
View File
@@ -531,7 +531,13 @@ const COMPACT_GITHUB_GROUP_SECONDS_HINTS = new Map<string, number>([
// Hybrid-specific Blacksmith observations from 31949756966, plus the
// gateway-core-3 139.5s spike in 31938297538 that must stay singleton.
// agents-core-models: 56.3s median (n=6, p90 58.6s) across 260 compact jobs on
// 2026-08-16 against a 36s scaled estimate. It was the dominant term in the
// only bin measuring >=1.25x its prediction (compact-large-19, 122s vs 88s).
// Sum a shard's per-config Duration lines before taking a median; pooling them
// reads as a large over-prediction that is not there.
const COMPACT_HYBRID_GROUP_SECONDS_HINTS = new Map<string, number>([
["agentic-agents-core-models", 56],
["agentic-commands-doctor", 64],
["agentic-gateway-core-3", 140],
["core-runtime-cron-service", 80],
+22 -6
View File
@@ -382,12 +382,28 @@ describe("scripts/lib/ci-node-test-plan.mts", () => {
"agentic-gateway-core-3",
]);
expect(hybridLargeTail?.predictedSeconds).toBe(140);
const hybridJobOf = (name: string) =>
hybridCompact.findIndex((shard) => shard.groups.some((group) => group.shard_name === name));
// Synthesized hosted stripes retain divided admission weights, while
// native hybrid groups use Blacksmith stripe hints during rebalance.
expect(hybridJobOf("agentic-agents-core-runtime-hosted-1")).not.toBe(
hybridJobOf("agentic-agents-core-tools"),
// Synthesized hosted stripes carry divided admission weights while native
// hybrid groups carry Blacksmith stripe hints, so both weight sources must
// survive rebalancing rather than one class monopolizing the tall bins.
// Assert that property, not one arrangement: bin membership legitimately
// moves whenever a hint is refit from fresh measurements.
const hybridStripeBins = hybridCompact.filter((shard) =>
shard.groups.some((group) => /-hosted-\d+$/u.test(group.shard_name)),
);
expect(hybridStripeBins.length).toBeGreaterThan(0);
expect(
hybridStripeBins.every((shard) => (shard.predictedSeconds ?? Infinity) <= 150),
"synthesized hosted stripes must respect the hybrid body ceiling",
).toBe(true);
// agents-core-models measured 56.3s (n=6) against a 36s scaled estimate, so
// packing it beside agents-core-runtime-hosted-1 (60.4s) built the only bin
// running >=1.25x its prediction: 122s of work priced at 88s. The measured
// hybrid hint separates them; without it they pack together again.
const modelsBin = hybridCompact.find((shard) =>
shard.groups.some((group) => group.shard_name === "agentic-agents-core-models"),
);
expect(modelsBin?.groups.map((group) => group.shard_name)).not.toContain(
"agentic-agents-core-runtime-hosted-1",
);
expect(compact.every((shard) => Array.isArray(shard.groups))).toBe(true);
expect(compact.every((shard) => shard.groups.length <= 10)).toBe(true);