Until now nothing in this harness could see the artefact the campaign
prevents. Every scenario counts .msg.user rows, and user rows never
travel on the SSE stream — a /send emits none, only /history replay
paints them — so a duplicated assistant bubble was invisible to all of
them. E8 counts a sentinel's occurrences in the transcript text instead,
which is structure-agnostic across duplicate bubbles and tool blocks.
The window it drives is the one readyState cannot see: the retry fires
with the transport OPEN, its /history is held, and inside that await the
transport drops and re-establishes. readyState reads OPEN afterwards
exactly as before. The redial is a real disconnect+connect rather than a
visibility change on purpose — a hide leaves evtSource null, which the
presence term already decides, so a hide-based control would pass for the
wrong reason. Control: stripping the generation term stamps dupes2.
E8's expectation needed correcting once: unlike E6, the stream is live at
flush time here, so the declined render's queued settle fires the
transport-free backstop and the pane converges in one settle. That is
correct behaviour, so the heal is asserted as a convergence leg — a "no
duplicates" verdict must not be earnable by rendering nothing.
E6 gains the non-vacuity it was missing: history_requests counts on
ARRIVAL, before the hold and before any status is chosen, so "the gate
declined a good payload" and "there was no good payload" stamped
identical observables. history_ok — incremented only when the production
route answers 200 — closes that, including the production-side-failure
hole an injected-fail budget cannot see.
Both hidden-window detectors widened for the additive jitter: sized on
the 2000 floor alone they would have closed before a top-of-range firing
and reported hidden0 for the wrong reason. delay_history(0) comments
corrected — it cannot release an in-flight hold.
Round-2 review. The render-time cursor-safety gate was point-in-time: a
transport that dropped AND finished re-establishing inside the /history
await reads back OPEN and is indistinguishable from one that never moved.
It is not — the redial re-presented the frozen cursor, the server answered
replay_ok, and the quiesce buffered that slice, so the render commits rows
the flush then repaints on top. Object identity cannot see it either,
since a native reconnect reuses the same EventSource; only a counter can.
_connectEpoch is bumped in onopen and nowhere else. Native auto-reconnect
calls neither connectSSE nor disconnectSSE, so those two are blind to the
exact case this exists for; connectSSE would also false-bump on its
document.hidden early return, which establishes no stream; and a closed
source can never fire a late open. Captured at dispatch, required
unchanged before a seedless render commits.
This is original-strata residual, not a regression this branch introduced:
before #900 the render was ungated entirely. The branch closed the
fire-time half and the still-down cases; these are the drop-and-recover
ones that were always open.
Two rulings written in at the gate, since neither is closed: a
fresh/truncated reconnect inside the await declines a render that would
have been safe (one wasted /history, self-healing via the flushed
synthetic state_change), and a refetch dispatched between onopen and the
replay slice arriving still renders past the frozen cursor — replay_ok
emits no end-of-replay marker, so no client-side signal exists (#903).
Coord's half of the same gate is #904; its exposure is a race rather than
this determinism, so it is not ported blind.
Also corrected: the claim that the idle-edge backstop's stream is live by
construction. It isn't — handleEvent also runs from the quiesce flush, so
a queued idle edge reaches the backstop with the transport down. The
render-time gate is what covers it. The clear_ui retry gains additive
jitter in BOTH clients from one shared constant: a declined render now
leaves the latch set, so a successful fetch can arm the retry, and the
decline trigger is herd-shaped. Kept small deliberately — the spread works
against #884's single-flight, which coalesces a lockstep herd.
test_coordinator_page.py anchored the fire guard on a literal `}, 2000);`
and on exact indentation; both would have ERRORED rather than failed once
the delay became an expression.
Every other interactive scenario mounts the Pane class directly, so the
factory closure that owns destroy() had no browser coverage at all — and
that is exactly where #900's largest hole lived. E7 mounts through
createInteractivePane (scenario-scoped: the factory owns its own
connect/recover-beat lifecycle, so switching the others would change what
they test), holds the first /history at the fault layer, destroys the
controller mid-flight, and lets the load resolve into the void.
Detector is a fault-layer non-occurrence: events_requests still 0, pane
detached, _visHandler null behind it. Without the bump it stamps
sse1-vis0 — the .finally reopens an EventSource on the detached pane and
re-registers the document-level visibilitychange listener destroy just
removed, which is the leak, now observed rather than traced.
An exhausted script queue still settles — into the error arm, which the
idle-edge backstop also consumes — so the heal leg would have passed for
a reason the scenario does not name. Queue the fourth turn like E4/E5 and
assert its sentinel.
E5's retry never fetches, so it cannot exercise the render-time check.
E6 reaches it the only way available: the retry fires on a live stream,
its /history is held at the fault layer, and a close-on-hide drops the
transport while the payload is in flight.
The detector is the stale-but-real PRE-rewind transcript surviving a
RESOLVED fetch — three user rows with the latch still set. The latch leg
is what makes it honest: replayHistory is the latch's only clear site, so
a held latch proves no render ran rather than inferring it from row
counts alone.
The interactive mirror of coord's G5. A close-on-hide inside the retry's
2s arm window is the reachable way to make it fire against a down
transport, and the detector is a NON-occurrence counted at the fault
layer: history_requests must be unchanged across the hidden window.
Remove the OPEN term and the hidden fetch lands, stamping hidden1.
The scenario also pins the two rulings the guard leans on: the latch
must still be SET after the skip (a skipped retry heals nothing), and
the show edge alone must not heal it — a replay_ok reconnect carries no
synthetic state_change, so the repair rides a plain send's organic
settle into the transport-free backstop, which is why exactly one new
SSE open spans show + heal.
Backport set from the #894 coordinator campaign, verified against
interactive.js source before fixing.
destroy() bumped no load token, which made it the WEAKER of the two
terminal paths (giveUp already bumped). Three escapes followed, all
reachable on the shell's onClose path: _loadHistoryThenConnect's
.finally reopened an EventSource on the detached pane and re-registered
the document-level visibilitychange listener destroy had just removed —
whose onerror then re-armed the host recover beat indefinitely, because
it gives up only on `dead`, which destroy never sets; a settling
_refetchHistory passed its supersession check and replayHistory'd into
detached DOM; and the clear_ui .then re-armed _staleRetryTimer after
destroy's own cancel. One bump at the terminal seam closes all three,
since the token is already the chokepoint every post-await consumer
reads. giveUp gains the matching timer cancel — inert is not dead.
The clear_ui retry could also fire against a DOWN transport:
disconnectSSE deliberately keeps it armed, so a hidden tab, a degraded
cooldown or a native redial holds the fire while _lastEventId is frozen.
A seedless refetch then paints rows the cursor still sits below and the
next connect's replay_ok paints them again (content and tool rows carry
no id dedup). The fire guard now requires an OPEN stream, and
_refetchHistory gains the render-time half at the chokepoint, covering
the await window a fire-time check cannot. Seeded loads are exempt by
construction — their caller disconnects first and readopts the cursor.
Deliberate trade, already ruled: the latch survives a skip, so
rewind/edit stay closed until the idle-edge backstop heals at the next
settle.
Two of the four filed findings are declined with the ruling written in
at the site, so an unprimed round re-derives rather than re-files them:
the same-token overlap is unreachable here (the replay quiesce
serializes what coord's refetchSeq stamp had to order, because coord has
no quiesce), and the joined-flight window is closed for both clients by
the shared make_history_handler's generation-keyed flight.
Review round 10 (1 minor bug; 2 major + 2 small quality — the majors
both pins-that-cannot-fail).
- The flight key's cold fallback was the literal 0, which collides
with a live session's generation 0: an eviction/close landing inside
a held flight's window let a post-truncation request rejoin a
generation-0 pre-truncation flight. Cold/detached workstreams now
key on None (rewinds need a live session, so two cold flights are
always mutually safe; a rehydrated session restarting at 0 can never
share the manager slot with its evicted predecessor — documented
at-site). The read is TYPED (live_session.session._history_generation)
so mypy carries the shape a getattr chain hid — and the typed access
immediately surfaced an unfaithful SimpleNamespace mock in the
reasoning-rehydration tests (no .session attr), now made faithful.
- Abort-Set producer pins: histCtrls.add exactly once and BEFORE the
await, delete exactly once and in the finally — without them the
destroy() consumer sweep was satisfiable by an always-empty Set.
- _make_session gains ws_id; the generation producer pin uses it.
- _coord_stick_latch: G2/G5's inline single-failure prologues RULED
deliberate at-site (their baselines/phase timings interleave into
the prologue; a per-divergence flag would obscure the choreography).
- Stray trailing whitespace stripped.
250 pins green; G2/G5/G7 re-run READY.
Review round 9 (4 minor bug, 4 quality, 1 perf nit; security zero).
- The r8 clearUiEpoch guard was UNREACHABLE (r9 bug find): clear_ui
always dispatches immediately after bumping, so a stale-epoch
dispatch is also a stale-seq dispatch and the currency gate discards
it before it can paint or clear — the client half of the joined-
flight fix was already carried by seq, and the server generation key
is the sole load-bearing layer. Machinery removed (decl, bump,
capture, conditional clear, section-9 pins); the latch-clear comment
now states the two-layer accounting.
- destroy()'s abort handle becomes a Set: a newest-wins single slot,
nulled by the newer dispatch's finally, left an OLDER overlapping
fetch unabortable — the destroyed closure pinned for the bound's
remainder. Pinned.
- _history_generation now bumps AFTER delete_messages_after: flights
rebuild from storage, so old-generation-reads-post-delete is the
harmless spuriously-fresh direction while new-generation-reads-
pre-delete would be wrongly joinable; the count/floor error paths
correctly leave it unbumped. Two-arm producer pin in
test_rewind_retry (persisted-rows bump on rewind AND retry;
in-memory-only error path must NOT bump) — the flight test's mock
can no longer mask a deleted bump.
- The harness load_calls increment takes a lock (to_thread workers
genuinely overlap under delay_load; a lost update false-fails G7).
- G7's viewer B is now a background authenticated GET (a raw request
enters load_messages identically; the second browser bought no
proof); stale two-tuple key comments and the coalescing matrix line
updated; the _send_in_page enumeration dropped for prose.
250 pins green; G1/G6/G7 re-run READY.
G7: two browsers on one ws; delay_load parks B's pre-rewind /history
flight open INSIDE load_messages — the flight layer. (A first cut
held via delay_history, which sleeps in the FAULT layer before the
route: flights never overlapped there and the 'negative control'
passed vacuously — a false detector, caught and rebuilt. The knob
also sleeps AFTER the load so a parked flight holds the rows it
actually read: its transaction point.) A rewinds mid-hold; the miss
proof is load_calls growing TWO (a joined request never enters
load_messages — the e2e twin of the unit test's proof) plus A
rendering the post-rewind single row.
The rebuilt detector immediately caught a real bug in the server fix:
mgr.get returns the Workstream WRAPPER, and the route's direct getattr
for _history_generation silently defaulted to 0 forever — joining
stayed enabled while the unit test's mock (attr on the wrong object)
masked the shape. The route now reads ws.session, and the mock pins
the nested shape so a wrong-object read can never pass again.
Negative control (flight key reverted to (ws_id, limit)): stamps
FAILED-loads1-rows3 — A joins the pre-rewind flight and paints three
stale rows as fresh truth. Fixed: READY-posts1-loads2-rows1.
The r8 joined-flight window, server half (Patrick-approved scope
expansion): the #884 single-flight key was (ws_id, limit), so a
/history dispatched AFTER a rewind/retry could join a flight whose
load_messages ran BEFORE the truncation committed — the joined
pre-rewind payload reads as fresh truth client-side (the client's
dispatch stamp is current; the staleness is the flight's transaction
point, visible only server-side) and reopened the over-rewind window
through the server seam. Reachable single-user (rewind clicked during
a truncated-resync fetch) and multi-viewer (any concurrent pane's
/history).
ChatSession gains _history_generation, bumped in _persist_truncation —
the shared rewind/retry chokepoint — BEFORE the storage write (the
in-memory tail is already trimmed by both callers; a spuriously fresh
flight is harmless, a wrongly-joined one is not). The flight key
becomes (ws_id, limit, generation): post-truncation dispatches can
never join pre-truncation flights, and the client-side clearUiEpoch
(prior commit) covers the converse (pre-rewind dispatches never CLEAR
a post-rewind latch). Cold workstreams key at generation 0 and the
first post-load truncation bumps, so cold flights cannot straddle a
rewind either.
Unit test mirrors the #884 coalescing determinism scheme: the owner
parks in load_messages under generation 0, the mid-flight bump
simulates the truncation commit, and the post-bump request must MISS
the held flight (load_calls -> 2, no coalesced record).
Negative-controlled: reverting the key to (ws_id, limit) fails the
test.
Review round 8 (2 major + 2 minor bug, 2 major + 3 small quality;
security/perf zero at five consecutive rounds).
- clearUiEpoch (r8 major): the #884 /history single-flight can hand a
joiner a payload whose load_messages ran BEFORE the rewind committed
(the flight key is (ws_id, limit); joining is invisible to the
client, and the client seq stamp cannot see server-side staleness) —
reachable single-user (rewind clicked during a truncated-resync
fetch joins that flight) and multi-viewer (any concurrent pane's
/history). The joined payload rendered as 'success' and CLEARED the
latch: the original over-rewind window, resurrected through the
server seam. Fix: the epoch bumps at clear_ui arrival, every
dispatch captures it pre-await, and only a dispatch that post-dates
the latest clear_ui may CLEAR the latch — a pre-rewind payload may
still paint (stale-but-real posture, gate holds), the surviving
latch arms the retry, and the retry's fresh dispatch starts a new
flight with post-rewind truth. Producer/consumer/placement pinned.
- destroy() aborts the in-flight bounded fetch (activeHistCtrl): the
r7 15s bound alone pinned a destroyed pane's closure until it fired
— the same dead-not-inert ruling destroy applies to staleRetryTimer.
Pinned.
- stop(hard=True) no longer sets force_exit: it skipped the ASGI
lifespan teardown and leaked the #885 daemon threads + sse_executor.
The 2s graceful-shutdown timeout already force-closes open SSE, and
the lifespan runs on both paths (docstring corrected; G6 re-verified
— the orphan still manifests).
- G6 pacing sized above the scenario's worst-case deadline sum (~200s
vs ~95s) so the in-process bash cannot resolve the orphan
mid-scenario and degrade the detector to a false READY.
- Quality: the r6 reachability comments rewritten to the r7 truth
(orphan REAL via hard crash; graceful-close-only synthesis); the
bound's WIRING pinned (signal reaches getJSON; getJSON forwards
init); _strip_comments deduped (4 inline copies); seq comment
re-paired with its asserts; retry_fire window tail-anchored.
Full harness (16/16 scenarios) + full suite (9724) green on the prior
commit; 136 pins green here.
Review round 7 (2 major + 1 minor bug, 4 minor quality; security/perf
zero). Both majors traced the r6 stratum:
- The closeStreamTransport drain of liveToolCalls rested on a false
re-announcement premise (verified: replay_ok yields only events past
the cursor; the coord fresh/truncated replay yields connected/status/
pending-cards/verdicts, never tool_pending/tool_info). An emptied
set fails OPEN — a mid-batch redial plus a slow seedless refetch
wiped the live batch. Retirement policy re-derived at the decl: an
id leaves on its RESULT, at the SETTLE edge, or with pane death;
transport death is NOT a retirement event; a stale id fails CLOSED
(skip, latch survives, settle heals). Site-anchored pins: the one
drain inside the idle/error block, the delete inside tool_result,
the adds inside tool_pending/tool_info, and closeStreamTransport's
comment-stripped code may not touch the set.
- G6's kill was not a kill: RecoveryServer.stop() gracefully closed
workstreams, and session.cancel()'s bash path persisted 'Cancelled by
user' BEFORE the reboot — the r6 'recovery synthesizes' ruling was
observing the cancel path. stop(hard=True) (skip the close sweep +
uvicorn force_exit: a crash does not drain SSE) leaves the orphan
genuinely unresulted — REACHABILITY FLIPS: the poisoned-pane state is
real, the live-set hardening is reachably load-bearing, and G6 is now
its behavioral detector: hard kill -> reload paints the orphan
(asserted PRESENT) -> the seedless rewind renders THROUGH the residue
(rewind-for-retry truth: the user message stays), negative-controlled
against the DOM-probe encoding (stamps orphan1, hist2). Discovered
and tracked separately: a hard-crashed reborn node answers stale-high
cursors with a silent fresh stream (no replay_truncated — the honest
truncation signal rides gracefully-persisted state).
- refetchHistory's await is now bounded (AbortController + 15s, the
coordSend shape): an accepted-never-answered /history pinned
refetchesInFlight and permanently disabled both heals. Pinned.
Quality: seq-producer position pinned earlier; the stale section-7
comment corrected; retry/backstop guard windows comment-stripped
(vacuous-by-comment-mention foreclosed); the shared G3/G4 double-fail
prologue extracted into _coord_stick_latch. G3/G4/G6 re-run READY.
Review round 6 (1 bug find + 5 quality; security/perf zero). The bug
finder out-traced r6-perf's dismissal: refetchHistory's own replay path
paints orphan batches (committed tool_calls, no persisted result) with
the same .conv-batch--running class the live path uses, and nothing
ever strips a dead orphan's class — so the r5 DOM-probed gate term
would let one orphan paint poison every seedless heal for the life of
the page (rewind/edit permanently dead; the seeded escape renders
through but REPAINTS the residue).
Reachability ruling (verified empirically): post-kill /history shows
the server synthesizes results for interrupted tool calls at recovery
('Cancelled by user. Outcome UNKNOWN'), so no persisted orphan exists
today and the poisoned state is unreachable — the client-side trace
was right, the server-side producer absent. Hardened regardless:
- liveToolCalls: an event-driven Set — fed ONLY by live tool_pending/
tool_info announces, retired by tool_result, drained at settle edges
and closeStreamTransport, and NEVER touched by any render (pinned:
refetchHistory's comment-stripped body may reference it exactly
once — the gate read). Liveness is read from the channel that
creates the hazard, never from DOM a render can forge.
- G6 coord-orphan-rewind: pins the SERVER invariant the client's
safety rests on — after a mid-bash node kill + reboot the batch must
render RESULTED (no --running residue) and the seedless rewind flow
must work end to end. Honestly scoped in its docstring: with
synthesis present a DOM-probe gate also passes, so the client
discipline is carried by the static pin set.
Quality batch: the seq stamp's producer position pinned (captured
before the await — the twin of the counter-bracket pin); two stale
G5 synthetic-idle comments corrected to the replay_ok-precise shape;
contract-test docstring item 7 restated to the enforced
universal-vs-seedless split; char-count pin windows replaced with
function-boundary slices (both test files); section 6 reuses _fn_slice.
Full coord family C + G1-G6 READY; 136 pins green.
Review round 5 step-back (fix-era critical): the r4 gate's busy term
conflated 'a turn is executing' with 'this DOM holds live turn state'.
_editAndResend flips busy BEFORE its POST and /rewind emits only
clear_ui (no state_change), so the busy term skipped the truncation
render the rewind exists to produce and appended the resent bubble onto
the PRE-rewind transcript; /retry's regenerated turn likewise raced its
own clear_ui refetch. The seam was re-derived once against the caller
x state matrix; the gate reads DOM-live signals only, split by scope:
- UNIVERSAL: dispatch seq (refetchSeq — overlapping fetches resolve
last-DISPATCH-wins; an older snapshot landing late can neither
double-render nor clear the latch over newer truth) and the content
refs (skipping always beats stranding a ref; seeded callers null
theirs before fetching, so it never blocks them).
- SEEDLESS-ONLY (keyed on the seedCursor arg): the
.conv-batch--running DOM marker for the tool phase (NOT activeBatch —
that is the pending-APPROVAL tracker, set only for opts.pending
batches; ruled at-site), coordSend's busySource === 'optimistic'
flavor (the one busy that marks un-committed DOM), and
stream-OPENness (CONNECTING keeps the handle with a frozen cursor
and a pending replay; handle-existence was not liveness — also
applied to the retry's fire guard). Seedless-only because the
SEEDED resync renders over these deliberately: after a node dies
mid-batch the --running class is dead residue no result will ever
strip, and the resync's render IS the recovery — a universal term
wedged the coord-restart scenario outright (family-run find; the
r5 finders missed the seeded-path interaction).
Backstop comment corrected (r5): fresh/truncated SSE replays DO carry
a synthetic state_change (replay_ok does not) — a latched pane pays one
refetch per reconnect, bounded by reconnect jitter/backoff and #884's
server single-flight; heal-caused triggers remain structurally
impossible. Caller fire-time ref guards demoted to the efficiency
layer at-site. Contract test re-pins the gate: term presence in
comment-stripped CODE, universal-vs-seedless placement, wipe between
failure guard and latch-clear, no plain busy. The edit-resend commit
gap under a second actor's clear_ui is accepted at-site (re-appears at
settle heal). G4's honesty note names the tool-phase branch it
behaviorally detects; G5 wording replay_ok-precise. Full coord family
(C + G1-G5) green; 136 pins green; 5 static mutants + the G4
behavioral control caught.
Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality;
bug-4≡q-2). Two correctness findings landed in one seam — the
refetch-vs-live-state chokepoint — so the seam was redesigned once
against its matrix (caller x stream-state-at-render x refs-at-render)
instead of patched per-finding:
- RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the
await is a real window (queued sends drain at exactly the idle edges
the backstop rides; another operator on a shared coordinator can send
any time; hide/suspend can land mid-fetch), and only the chokepoint
can see across it. Skip the wipe when a live turn exists (content
refs — a wipe strands the bubble and loses the rest of the turn
invisibly) or when a seedless render lost its idle/live-stream
precondition (busy covers the tool phase the ref check can't see;
a dead stream means rendering past the frozen cursor and
double-rendering on the show-edge replay). Both requirements key on
the seedCursor ARG — seeded callers own their reconnect flows and
legitimately rebuild mid-turn. Skips leave the latch set; heals
converge at the next organic settle.
- The retry's fire guard gains evtSource (close-on-hide keeps the timer
armed by design; a hidden firing must not fetch). The backstop needs
no term — it runs inside SSE dispatch.
- Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs
on both heal arms, the else-if exclusivity structure, the render-gate
order and terms, the evtSource guard tail. Seven mutants, all caught.
- Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its
clear_ui into a channel nobody joined and false-fails the scenario);
G4 coord-heal-midturn (a turn started under a held backstop fetch
survives its resolution; hist==2 is the discriminating bit — noted
honestly in the docstring); G5 coord-hidden-retry (hidden0
non-occurrence + organic-settle heal after show, per the accepted
liveness-lag ruling — a quiet reconnect delivers no state_change
edge). Negative controls: gate-stripped stamps hist1; guard-less
stamps hidden1. Docstring gains the G-family catalog.
Review round 3 (bug/security/perf zero; 1 quality minor): the contract
test pinned both CONSUMERS of refetchesInFlight (the backstop and
retry-fire yield guards) but not the PRODUCER ++/-- pair — dropping the
bracketing would leave the counter at 0 and both consumer pins
vacuously green. Count-pinned both sites; mutation-verified (the test
fails with the increment stripped).
Review round 2 (1 minor bug + 1 minor quality, security/perf zero):
- The retry's arm site was gated on historyStale alone, so a clear_ui
refetch in flight at destroy() that then FAILS re-arms the timer
AFTER destroy's clearTimeout — a no-op fire (the visHandler fire
guard holds) but the orphan pins the dead closure for its 2s delay,
contradicting destroy's dead-not-inert invariant. The arm gate is
now historyStale && visHandler, matching the fire guard; the seam
matrix (destroy/closeSession/live x arm-and-fire windows) closes
with that one term. The edit-resend in the same .then stays
deliberately ungated on teardown: the rewind committed server-side
and the workstream outlives the pane UI, so the committed edit
still delivers (comment at site).
- Pins: the arm gate (mutation-verified — the contract test fails
against a gate-stripped mutant), the fire guard's visHandler term
(sole coordCloseSession protection), and the re-arm clearTimeout.
G1/G2/G3 re-run READY; 136 static-pin tests green.
Review round 1 (0 correctness/security/perf; 1 minor + 1 nit) + the
suite's collateral:
- The latch-contract test now pins !refetchesInFlight on BOTH heal
paths (backstop arm + retry fire guard) — the yield guard is
load-bearing (same-snapshot double-render stomp without it) and was
previously deletable with every test green. Mutation-verified: the
backstop pin fails against a guard-stripped coordinator.js.
- test_app_js.py's clear_ui pin narrowed from all-clearTimeout to
clearTimeout(truncatedResyncTimer): the invariant it protects is that
clear_ui carries no path-local cancel of the TRUNCATED repair intent;
#894's staleRetryTimer re-arm cancel is the staleness latch's own
machinery, deliberately armed there.
- _send_in_page's caller enumeration gains G3.
G1/G2/G3 mirror interactive's E2/E3/E4 for the coordinator pane, adapted
to its structure: the pane object exposes no messagesEl and no
latch/quiesce fields (closure-private state), so every probe reads the
public #coord-messages container and the runners drive the verdicts off
the fault layer's authoritative counters — the in-flight edge is the
history_requests bump (counted on arrival, before the delay hold), the
closed phase is proven by the gated click's POST non-occurrence, and the
latch-cleared proof is the reopen POST rather than a field read.
- G1 coord-rewind-window: the busy||historyStale gate under a held-open
clear_ui refetch (delay_history); posts stays 1.
- G2 coord-rewind-failed-window: the failed-refetch aftermath — the
latch survives the failed exit, the bounded 2s retry heals (its fetch
held to defer the clear site), the healed render reopens the gate.
- G3 coord-stale-backstop: double failure (fail_history(2)) exhausts
clear_ui refetch + retry; a plain send's organic idle edge fires the
TRANSPORT-FREE backstop. Storm assertion: events_requests delta is 0
across the whole heal; history delta exactly 1.
Negative-control validated: pre-latch coordinator.js stamps
COORDREWINDWIN-posts2-rows0 and COORDREWINDFAIL-closed2-heal0; a
transport-touching backstop variant (loadHistoryThenReconnect) stamps
COORDSTALEBACKSTOP-...-sse1 — each detector has observed its bug.
The coord recovery page gains a scenario dispatch; the auto-send now
runs only for coord-restart (the rewind scenarios seed server-side),
verified against the existing coord-restart scenario.
From clear_ui arrival until the next SUCCESSFUL refetchHistory render the
visible transcript is the stale pre-rewind DOM with busy false, so a
second rewind/edit click counted it and POSTed an over-large turn count
against the already-restructured server conversation (the #890 sibling,
pre-existing since #888 accepted the stale-interactive window).
Port of interactive.js's converged #890 latch design, adapted to coord's
structure (no load token, no replay quiesce, no ref-resetting render):
- historyStale latch: set at clear_ui arrival, cleared ONLY by the
success-path render below the if-(!hist) failure guard — a flag would
reopen on the failed exit, which is exactly the over-rewind window.
- Gates: _rewindToMessage / _editAndResend / _startEdit now require
busy || historyStale; _rewindToTurns and _retryLast stay busy-only
(explicit-arg / no-DOM-count — rulings at-site).
- Heal A: one bounded turn-free retry armed in clear_ui's .then; fire
guards read the latch, refetchesInFlight (net-new await-window counter,
coord's quiesce-free yield discriminator — a COUNT because overlapping
fetches are reachable), busy, the streaming refs (load-bearing: coord's
refetch does not reset refs), and visHandler (teardown sentinel).
- Heal B: idle-edge backstop as the else-if behind the truncated-resync
consumer — TRANSPORT-FREE by ruling (plain seedless refetchHistory;
a reconnecting heal draws the synthetic state_change:idle back into
its own trigger = zero-backoff storm against a recovering node).
Carries ref guards the interactive template omits: this arm also
serves error edges where no stream_end nulled the refs.
- Teardown: destroy() cancels the retry timer (terminal-only);
closeStreamTransport deliberately does not (redials keep heal intent).
Static pins: the latch contract (set/clear/gate sites, transport-free
backstop, bounded arm, teardown split) + the widened guard-before-wipe
window; the contract pin fails against the pre-latch code.
PR #896 review follow-up, no behavior change:
- Pinned test EPOCH was 32-bit (token_hex(4)) with a matching comment, but
production widened to token_hex(8) in 2b3d0687 and the same file already
pins token_hex(8) at line 401. Widen EPOCH to 16 hex chars + fix the comment.
- Document why the fanout shutdown sentinel put stays inline on the loop: the
consumer is still alive and drains via non-blocking fan-out, so it returns
at once; the 1s timeout is a ceiling that never binds (off-loop is reserved
for the multi-second joins).
The suite's growth (~9.7k tests, coverage-instrumented, 3-version
matrix) started brushing the 20-minute hang cap on healthy runs; 30
keeps the hang-catching semantics with headroom.
Scenario F splits into F1 (manual ?last_event_id= transport) and F2, the
native-header sibling — the only behavioral coverage of two pure-browser
semantics no Tier-1 harness can express: the auto-reconnect header echo,
and id-less frames inheriting the connection's persisted lastEventId
(the mechanism behind app.js's node_snapshot-branch clear). F2's phase
C is the round-3 fix's discriminator: after the native heal, a forced
manual reconnect must go CURSORLESS with no second truncated round
(pre-fix: cursor1-trunc2), guarded by an idFrames precondition against
the aggregate tick.
Two harness seams earned by F2's first failures, both documented at
site: a failed EventSource reconnect attempt is TERMINAL per WHATWG, so
the restart must never expose a refused window — a SO_REUSEPORT
placeholder binds before the old node stops and hands its backlog to the
successor's uvicorn (make_listen_socket + RecoveryServer sock
injection); and an SSE stream still open at stop() parked uvicorn's
graceful drain indefinitely — timeout_graceful_shutdown=2 bounds it with
the #885 lifespan teardown intact.
Absence assertions tightened (round-4 review): ghost-gone now requires
absence from BOTH the model and the rail via _roster_absent_ws — the
negated AND-membership helper De Morganed into either-surface and could
false-pass a rail-render regression.
token_hex(4) left the epoch equality check — the only thing between a
prior-boot cursor and a silent replay_ok-empty alias — at 2^-32 per
same-node restart-pair; 64 bits puts a fleet-lifetime of restarts
engineered far below threshold (review round 4, classified
design-margin). Docstring rounds from the same pass: the resume
contract now notes reason=boot_epoch also covers the same-epoch
empty-ring fail-safe (not exclusively foreign epochs), and the collector
ruling says precisely that the staleness CHECK and envelope can never
fire there — the epoch-tagged ids are on the wire, just never read.
On a NATIVE reconnect into a boot_epoch truncation, the envelope and
node_snapshot frames are id-less, and an id-less frame's MessageEvent
inherits the connection's persisted pre-restart lastEventId — so the
pre-dispatch capture re-stored the dead cursor on the snapshot frame,
undoing the truncated branch's clear (a manual reconnect's fresh
EventSource starts with an empty string, which the guard blocks). A
manual reconnect racing in before the next id-bearing frame then
re-presented the dead cursor for a redundant, self-healing truncated
round. The snapshot branch now clears the cursor before the roster
rebuild — dead in every case that draws a snapshot (fresh has none,
truncated's is spent) — and the tripwire pins all three clear sites so a
simplify pass cannot drop one (round-3 review; verify classified the
mechanism redundancy-not-correctness: a cross-epoch cursor can only ever
redraw truncated+snapshot, never the silent ghost shape).
Round-1's lock-scope fix moved the snapshot build after listener
registration but left it unguarded: a raising _build_node_snapshot
(storage reads, per-ws locks) propagated before the generator — whose
finally owns de-registration — ever existed, stranding a dead 1000-slot
queue in the fan-out list forever (the fan-out thread never removes
listeners; pre-branch the append was the LAST locked statement precisely
so a raising build could not strand it). The whole post-registration
window (build, log, response construction) now runs under a guard that
de-registers on ANY exit and re-raises; _deregister is shared with the
generator's finally so the discipline has one owner. BaseException
because the window must stay guarded even if a future edit introduces an
await (today it is await-free, so a cancel cannot land inside it).
Tests (round-2 review): the leak path is pinned (raising build →
exception propagates AND the listener list is empty); the
registration-before-build + lock-released ordering is pinned by a probe
builder asserting both at build time; the caught-up-cursor test is
rebuilt around a sentinel live event so it asserts the no-envelope shape
positively instead of truncating the drain at the retry frame.
test_console.py covers the CONSUMER side of node_snapshot (hand-built
dicts fed to the collector), not _build_node_snapshot's production —
the helper docstring claimed otherwise. Point at the real end-to-end
coverage (the roster-restart scenario: membership + evict) and state
plainly that the producer's field projection has no direct unit test
(review round 1, quality finding).
_build_node_snapshot is an O(workstreams) walk taking each ws's _ws_lock;
under global_listeners_lock it serialized a restart herd's stale-cursor
reconnects against each other and against the fanout thread's per-event
stamping — stalling roster delivery to every listener exactly while the
reborn node emits its re-open events. Listener registration stays under
the lock (the ordering that guarantees no loss); the snapshot now builds
after release, keyed off replay_status so the build predicate and the
generator's emission branch stay one rule. A delta stamped during the
build is both reflected in the newer snapshot and queued behind it —
absorbed idempotently by the state-of-world consumers; the endpoint
docstring's atomicity claim is rewritten to this contract (review round
1, perf finding).
Scenario F drives the REAL node dashboard (/ + app.js) through a node
restart on the global stream — no custom page; transport instrumentation
is injected via CDP addScriptToEvaluateOnNewDocument, scoped to
/events/global URLs so per-ws streams can't pollute the counters.
Phase A is the negative control: live roster, live cursor, zero
replay_truncated. Phase B: hide, force the CLOSED state (a closed
EventSource never auto-retries, making the show edge's manual reconnect
the only reconnect), restart the node re-opening only one of two
workstreams, show. Asserted: cursor presented via ?last_event_id= and
replay_truncated observed at the transport, the not-reopened
workstream's ghost evicted from the roster model and rail (the dashboard
table's membership refreshes on interaction by design — documented at
_roster_has_ws), and the reborn node's global_events_requests counter
proves the reconnect hit the real endpoint. The native header
transport differs only in carriage and is pinned by the Tier-1
boot-epoch tests.
_global_fanout_thread, _aggregate_emitter_thread, and
_idle_cleanup_thread were daemon threads with no stop signal — shutdown
abandoned them mid-loop. The sleep-loop pair now waits on a shared
Event (wait doubles as the tick sleep, so a set wakes them immediately);
the fanout exits on an identity-checked queue sentinel, FIFO-draining
everything enqueued before it (sessions close earlier in the shutdown
tail, so their final events still fan out). Joins are bounded and
off-loop; daemon=True stays as the backstop for a join timeout, not the
mechanism. The recovery harness drops its thread-neutering workaround
(module docstring piece 4) — the global lane now runs REAL in harness
boots, which the #881 roster-restart scenario requires.
The global stream's manual reconnects were pinned cursorless because a
stale cursor on the reborn ring drew replay_ok-empty with no snapshot
(the ghost-roster shape). With epoch-tagged ids that shape is
unreachable — a stale cursor now draws replay_truncated + a fresh
node_snapshot — so app.js captures e.lastEventId (MessageEvent, house
guard form), presents it via ?last_event_id= on manual reconnects, and
clears it where the record dies: the replay_truncated handler and
onLogout. The cursor stays an opaque string end to end; the tripwire
that pinned cursorlessness now pins the capture, the guarded query-param
presentation, and the never-parse-numerically discipline instead.
The global ring's counter is process-local and reboots at 0, so after a
node restart a pre-restart cursor was first invisibly ahead of the reborn
ring (replay_ok with an empty slice) and then aliased into the new id
space as the counter re-grew — both silently skipping the restart
boundary (ghost rosters). Every global SSE id is now
"{boot_epoch}-{counter}" (per-process nonce); the browser echoes it
verbatim on native reconnect, so provenance rides every path with zero
client cooperation. A cursor from any other epoch — prior boot, another
node, a pre-epoch bare-int client, garbage — draws replay_truncated
(reason=boot_epoch, loss unknowable so the numeric fields are omitted)
plus the node_snapshot recovery floor; in-epoch ring misses keep honest
lost_count under reason=ring_evicted. Same-epoch cursors run the ring
logic unchanged. Chokepoint log line added; per-ws ids deliberately stay
bare ints (storage-seeded counter — asymmetry documented at both sites);
collector audit ruling recorded at its cursorless connect.
Addresses the Copilot review of #895 (docs/comments only, no behavior change):
- recovery_e2e.py / _sse_recovery_server.py: the mutating affordance gate
is `busy || _historyStale`, not the superseded `busy || _replayQueue`
quiesce gate the r3 latch replaced — corrected both docstrings (E2 now
matches E3).
- interactive.js cross-ws supersession: the branch drops the pending edit
and releases busy but does NOT clear `_historyStale` (its sole clear
site is replayHistory) — reworded so it no longer implies the latch is
released.
- interactive.js idle-edge backstop + bounded retry: documented that the
fire-and-forget `_refetchHistory` (no `.catch`) is deliberate — no
composer state to un-strand there, unlike the primary clear_ui caller,
so a render throw stays loud (peer of the load path's `.finally`).
Rider from the session queue: the comment cited ui/static/app.js
Pane.replayHistory, which moved to shared_static/interactive.js in the
L-shell step-5a lift — the old path no longer exists.
RecoveryServer grows an in-process fault layer (pure-ASGI wrapper; the
production app is untouched): fail_history(count) serves minimal 500s
for the next N GET /history requests, delay_history(ms) holds responses
to widen or hold open a refetch window, and per-route request counters
(history_requests, rewind_requests) let scenarios assert backend state
rather than scripted absence.
Five scenarios on that layer, all stamping RECOVERY-READY/FAILED
titles like their siblings:
- fail-refetch: hide mid-turn -> restart -> failed first resync ->
the stale transcript survives (no wipe, no empty-state) while the
truncation record stays armed -> the connect-chokepoint retry heals
(history_requests proves the re-fetch). The #890 acceptance
contract, browser-observed end to end.
- stale-ref-reload: mid-segment transport death -> turn completes
during the outage -> failed unarmed same-ws reload -> the next
turn renders in a FRESH bubble and the stale bubble's text is
unchanged (regression test for the resumability-gated ref reset).
- rewind-window: a second rewind clicked during a held clear_ui
refetch window never reaches the server (rewind_requests == 1) and
the transcript reflects one rewind (regression test for the
busy-or-latch affordance gate, in-window arm).
- rewind-failed-window: the failed-fetch AFTERMATH sibling — the
refetch 500s, the staleness latch keeps the gate closed over the
stale rows (rewind_requests stuck at 1, proven latch-not-quiesce
via a settle-poll), the bounded turn-free retry heals (3 -> 1 user
rows), and only then does the gate reopen (rewind_requests == 2).
Negative-control validated: with the interactive.js fixes reverted,
stale-ref-reload stamps fresh0-unchanged0 (the concatenation bug),
rewind-window stamps posts2-rows0 (the in-window over-rewind), and
rewind-failed-window stamps closed2-rows0 (the failed-exit
over-rewind) — every detector observes its bug, then stamps READY
again with the fixes restored.
Port the coordinator's #882 G3 guard-before-wipe: the wipe + streaming-
ref reset live in replayHistory, reached only on a successful fetch.
- clear_ui no longer pre-wipes the transcript; a failed refetch during
a rewind/retry/resume replay keeps stale-but-real content instead of
blanking the highest-traffic pane on a live stream (/history
failures cluster in exactly the restart windows that emit clear_ui).
- _refetchHistory's failure branch is a DOM/ref/repair-intent no-op:
no empty-state hint below stale content (the old resync-route wart),
no streaming-ref reset (which orphaned a mid-jitter turn's bubble on
the resync route); the truncation record stays armed for the
connect-chokepoint retry; only the quiesce releases.
- _loadHistoryThenConnect resets streaming refs on a ws SWITCH only --
the old ws's refs otherwise survive a failed fetch into the new ws's
stream; a same-ws reload keeps them so the reconnect resumes the
mid-jitter bubble instead of orphaning it.
- The factory connect() empty-state pre-seed is now the sole producer
of the failed-first-paint placeholder -- documented load-bearing.
The edit-and-resend dispatch, cross-ws supersession, and repair-intent
lifecycle are unchanged; a failed fetch keeps the resend firing (the
rewind already committed server-side), mirroring coord.
Pinned by test_interactive_refetch_failure_preserves_the_pane (the
mirror of coord's test_coordinator_refetch_failure_preserves_the_pane)
plus the re-pointed quiesce/agent-tracking pin.
After a node restart every open pane resyncs via REST /history inside
the same jitter window; client jitter spreads the peak but not the
total. Concurrent requests for the same (ws_id, limit) now share ONE
reconstruction (load_messages -> decoration -> projection) via a
single-flight task map in the handler closure.
Deliberately single-flight only, no TTL cache: the payload depends on
live-mutable inputs with no total cheap invalidation signal (the
surface_persisted_reasoning registry toggle emits no per-ws event;
cold workstreams have no event counter), so a cache could serve stale
reasoning/approval/cursor state for its whole TTL, while a joiner's
worst-case staleness equals the flight duration -- the window a lone
slow request already exposes.
All auth/tenant/kind/existence gates stay per-request ahead of the
join; only the caller-independent reconstruction is shared. A shared
draw that hit a transient load_messages failure is not fanned out:
joiners retry once, independently, so one storage blip cannot wipe
every coalesced pane (the 200-empty payload renders as an
authoritative empty pane in both clients, and the seedless clear_ui
path has no SSE redelivery to repair it). The flight is a detached
task (awaiters shield it) so an owner disconnect cannot strand
joiners, and each task pops its own key in a finally, so the map only
ever holds in-flight work. ws.history.load_failed rises to warning:
it now names the draw that triggers joiner retries and renders as a
pane wipe.
The drain comment and the architecture docs stated the zero-budget band
relative to the auto-compact threshold as if 0.8 were universal
("well below the auto-compact threshold"); with an operator-set
auto_compact_pct under the ~70% zero point the claim reads inverted.
State the geometry against the DEFAULT threshold and make explicit what
was always true of the mechanism: the trigger's predicate is the
exhausted budget itself, never a threshold, so with low thresholds the
owed path compacts first and the trigger is its bail/insufficient
backstop.
At an exhausted context budget the drain loop replaced every tool result
with a placeholder that read as a successful-but-trimmed call. For
structural results — spawn_workstream's ws_id, the tasks scratchpad —
the model lost the handle orchestration depends on and silently
stalled, while the UI (told the real summary before the drain) kept
showing success. Worse, the budget zeroes near 70% fullness when
max_tokens ≥ context_window/4, well below the 80% auto-compact
threshold, so a stalled coordinator could sit in that band indefinitely
with no compaction ever firing.
Three guarantees at the truncation seam, one renewal trigger at the
drain:
- structural-tool and error results get a guaranteed 2048-char
admission floor (head+tail beyond it) — never the zero-budget drop
- any result at or under the floor passes verbatim (denial notices,
spawn acks: never destroy what is smaller than the guarantee)
- bulky non-structural results get an explicit drop notice stating the
call RAN but its output could not be admitted — never a trim
impersonation the model cannot distinguish from success
- a zero truncation budget triggers one mid-turn compaction (no
threshold_pct — none was evaluated, same rule as the ctx-overflow
retry), closing the 70-80% band where the budget zeroed but
compaction was never owed
Background-bash spawn acks ride the small-result pass; a name-keyed
floor cannot distinguish them from foreground bash — see #891.
A mid-stream replay_truncated latches _pendingTruncatedResync; a
clear_ui rebuild (rewind / edit-and-resend) heals the gap but left the
latch — and any pending jittered _resyncTimer — armed, because clear_ui
keeps the stream live and only disconnectSSE cancelled the timer. The
next idle edge then fired a phantom _loadHistoryThenConnect against the
already-repaired gap: a false truncatedGaps bump and a needless
teardown, and on the phantom's failed-fetch leg the reconnect went
cursorless (_lastEventId nulled with no record armed) with nothing
left to re-cover the suspend window.
replayHistory now clears the gap record, the deferred latch, and the
pending timer together — the same one-site supersession the coordinator
port established in refetchHistory. The latch/timer clears are no-ops
on every _loadHistoryThenConnect flavor (each clears both before its
fetch); the clear_ui heal is the path they exist for. A failed fetch
still clears none (it never reaches replayHistory), keeping the connect
chokepoint's retry armed.
Found as a latent shared shape by the #882 review's round-4 pass and
confirmed against this file; pinned in the fresh-connect/churn-limit
test alongside a guard that clear_ui never grows a path-local cancel.
replay_truncated is now a dead-stream signal, mirroring the converged
interactive.js machinery:
- loadHistoryThenReconnect: tear the transport down first, drop the live
cursor, refetch /history with cursor adoption, reconnect in .finally.
The old in-place refetch discarded the /history cursor while /history
trims the trailing in-flight turn whenever it returns one — a mid-run
truncation wiped the executing turn with no redelivery and later tool
results orphaned into top-level bubbles. Both consumption sites
(immediate branch and idle-edge deferred consumer) route through it.
Dropping the cursor before the fetch is load-bearing, not just parity:
a post-restart heal on an idle ws gets no /history cursor, and
re-presenting the frozen pre-restart cursor against the reseeded empty
ring draws replay_truncated forever — an envelope→resync loop that
parks the pane in degraded cooldown cycles (caught by the new
browser-level scenario, invisible to source-pattern tests).
- truncatedFromCursor: the truncation-time cursor, recorded keep-oldest
at the envelope and cleared only by a successful full render; the
connect chokepoint presents it over the live cursor so every manual
reconnect re-draws the envelope and the repair survives any teardown
interleaving (hide/show, degraded cooldown, CLOSED retry, failed
fetch).
- churn ladder: truncated resyncs feed the same rolling window as
overflow closes via the extracted recordChurnAndMaybeTrip(); a trip
skips the resync (the degraded wake re-arms via the chokepoint).
- herd jitter: resyncs start behind a 0..TRUNCATED_RESYNC_JITTER_MS
spread; one pending resync at a time; the fire path nulls its handle
before loading; closeStreamTransport owns cancellation.
- sidebar refresh: while a truncation gap is on record the gap machinery
owns recovery outright — the envelope refreshes once per NEW gap, one
heal-time refresh covers the retry window, and onopen's no-cursor /
long-gap arm stands down — so a failed-resync retry loop cannot
stampede /children + /tasks un-jittered once per reconnect through
either path.
- a failed /history refetch no longer blanks the pane (wipe + tracking
resets sit below the !hist guard); a successful full render supersedes
ALL pending repair intent in one place (gap record, deferred latch,
pending resync timer) so a heal can never strand a phantom resync.
Behavioral coverage: scripts/recovery_e2e.py gains --scenario
coord-restart — the REAL coordinator pane (chrome, cookie auth,
EventSource, connect chokepoint, resync, churn limiter) mounted against
the interactive recovery node (/coord-static + /coord-recovery), driven
through hide → node restart → show over CDP, asserting the envelope is
drawn, the hidden-window turns heal, the stream re-opens, and the pane
converges. Revised the two tests that pinned the in-place shape, added
the coordinator mirror of interactive's fresh-connect/churn-limit pins
(keep-oldest record, chokepoint consult, clear-on-render, shared churn
step, trip-skip, jitter scheduler, cancellation site, cursor drop,
per-gap sidebar dedup).
The recovery e2e tests run a scripted provider — no LLM backend — so the
live co-mark was a lie told to keep the existing CI expression skipping
them. Both CI lanes now deselect explicitly via
-m "not live and not e2e_recovery", and the tests carry only their
honest marker. Select with -m e2e_recovery.
All three clients read lastEventId off the EventSource object, but per
WHATWG the property lives on the MessageEvent — EventSource exposes only
url/withCredentials/readyState. The object-form reads were dead
conditionals in every real browser: the cursor never tracked live
traffic, every MANUAL reconnect (close-on-hide show edge, degraded-
ladder retry, recover beat) opened cursorless as a fresh connect, and a
fresh connect does not refetch history — so turns committed while a tab
was hidden silently never painted. This is the cleanest mechanism behind
the 'turn disappeared, never healed' field reports, and it gated the
branch's recovery fixes: without a presented cursor, the empty-ring
truncated honesty could never fire for hidden-tab restarts and the
truncation record captured null. Native auto-reconnects were unaffected
(the browser sends its internal Last-Event-ID header), which is why the
bug stayed invisible: transient blips healed, deliberate closes lost.
Capture e.lastEventId in each onmessage instead, guarded != null and
!== "" — no-id frames carry the empty string and "0" is a valid id (the
error-surface snap_seq can be 0 on a brand-new workstream). The
coordinator's counter-reset detector, which compared against the same
dead property and so never fired, now works as documented.
Found by the recovery harness's first real-browser run: source-pattern
tests pin a wrong-object property read as happily as a right one, so a
tripwire test now forbids the object form by name across all three
clients, and Tier-2 scenario B is upgraded to hide MID-turn and require
the browser-observed replay_truncated envelope plus the healed gap
(RECOVERY-READY-RESTART-rows1-trunc1 demonstrated; was trunc0).
Tier 1 (tests/test_sse_recovery_e2e.py, opt-in e2e_recovery marker): six
scenarios against a real interactive server with a scripted provider and
ephemeral DBs — storm batching without loss, slow-consumer overflow with
lossless ring replay, mid-run truncation with cursor-adoption rebuild,
restart truncated-honesty (exact lost_count; no-loss variant replay_ok),
failed-resync retry via the truncation record, and sub-agent storm
attribution. BrowserlikeSSEClient (tests/_sse_recovery_helpers.py)
implements the browser cursor contract; RecoveryServer
(tests/_sse_recovery_server.py) boots the real app per test.
Tier 2 (scripts/recovery_e2e.py): the livepass idiom against a REAL node
— boots the real InteractivePane over real EventSource/authFetch, with a
dependency-free CDP runner driving the storm and hide-restart-show
scenarios; document.title stamps verdicts so a broken state cannot pass
silently.
Events are produced by the real session engine through the provider
boundary — no synthetic frames; teardown leaves no leaked threads; the
default suite keeps these deselected.
Line-chatty tools under the 4-wide pool emitted one SSE event per
stdout line — the event-storm source that overflowed listener queues
under parallel task agents — and each line's _enqueue force-flushed
the pending token batch, defeating token batching too.
Chunks now buffer per call_id in SessionUIBase and flush as one
concatenated event on the shared window/size cadence, bypassing
_enqueue entirely. Ordering rulings from the dataflow pass:
- The load-bearing ordering is chunk-vs-its-own tool_result (the
client removes the streaming pre at the result render), enforced by
a terminal flush+close in on_tool_result before the result enqueues.
- Chunk-vs-content interleaving is cosmetic (independent DOM
subtrees), so chunk traffic no longer touches the token batch.
- A chunk arriving after its call closed is a leaked drain thread
past the join timeout: discarded (the rendered result carries the
complete output), never mispainted or flushed unstamped.
- Teardown backstops (stream_end, the idle/error snapshot chokepoint,
turn commit, on_error) flush all pending batches; on_turn_start
discards stale-crash residue and resets the closed-call ledger.
The CLI is untouched by construction (TerminalUI implements the
SessionUI Protocol directly; its chunk hook is a no-op) and the
single-producer-per-call_id topology the batcher's ordering assumes
is pinned by a producer-surface test.
Two fixes for the field reports of permanently missing turns,
stuck-busy panes, and sub-agent tool calls escaping to the top level:
- Client: a replay_truncated envelope now runs the full fresh-connect
flow (_loadHistoryThenConnect — disconnect first, /history, adopt
the resume cursor, reconnect) on both the immediate and idle-edge
branches. The old in-place refetch discarded the cursor while
/history trims the trailing in-flight turn whenever it returns one,
so a mid-run truncation wiped the executing turn (task cards
included) with no redelivery; the orphan grace then escaped the
still-streaming children to top-level rows.
- Server: register_listener_with_replay reports truncated (not a
silent replay_ok) on an empty ring when the storage-seeded event
counter proves the client lost events — the rehydrate/node-restart
case that previously skipped the gap unsignalled. can_replay_from
deliberately stays False on an empty ring (docstrings record the
asymmetry ruling).
Truncated resyncs count into the same degraded catch-up window as
overflow closes, bounding the re-truncation loop under sustained
eviction; the limiter check runs before the resync starts so its
.finally reconnect cannot defeat a cooldown it just triggered.
Observability: _streamHealth.truncatedResyncs client-side and a
ws.events.replay_truncated log line at the envelope chokepoint.
Known-gap breadcrumbs: #881 (node-global stream), #882 (coordinator
pane parity).
The docstring claimed the non-string project_id coercion matched both
_coord_create_build_kwargs and the interactive create path, but
_interactive_create_build_kwargs passes body.get("project_id") through
rather than coercing. Restate it as the gate's own rule — only a
non-empty stripped string counts as an attached project — and reference
only the coordinator persistence that actually matches. Behavior
unchanged.
Wire create_gate_require_project=True on coord_endpoint_config: a
projectless coordinator create on the console is refused with the same
coded 400 as interactive creates. Operator tokens get no exemption; the
sessions a coordinator spawns remain exempt via the token_source branch
in require_project_denies_create (child spawns, a different seam).
The gate predicate now reads "no project" the way the create path
actually persists it — a non-string body value (int/bool/list/dict) is
coerced to absent, matching _coord_create_build_kwargs and the
interactive create — so a truthy non-string like project_id:123 cannot
stringify past the gate and mint a projectless session. Without this the
three sites disagreed: the old str(project_id or "") stringified a
number to a truthy value and waved it through while build_kwargs stored
None. Interactive was unaffected (its validator stringifies and 400s
first); the fix is at the shared predicate as defense-in-depth for both.
The console launcher's project picker mirrors the interactive strict
treatment when the flag is on — the seeded placeholder retitles to
"Select a project…" (or "No projects available") via
setOptionPlaceholder, computed before the + New project… sentinel is
appended; the server's coded 400 stays the enforcement. Settings label
and help text updated to say coordinators are covered and only
coordinator-SPAWNED sessions are exempt.
Real-mount wiring tests drive the mounted console endpoint end to end
(the synthetic-cfg tests can't catch a mis-wire on the actual mount),
including a non-string-project_id bypass regression, with an operator
token that carries admin.coordinator without the service scope.
Patrick Buckley