Review fold-in before push, twelve findings, two of them majors.
The handoff popped the interjection queue destructively and handed
the text to a send with a non-delivering refusal (the budget latch)
and a preamble that can raise before the user turn is appended — a
failure destroyed the user's words with a log line, after the charged
wake nudges were already cleared. Now: the budget latch is checked
before the pop (the message stays queued for a send with a human in
front of it, and the wake drain still runs so the worker's exit
converges); the pop returns the raw items and any non-cancel escape
restores them verbatim — ids and priorities intact — before the
failure surfaces; a cancel deliberately does not restore, because the
Stop supersedes the queued words. Content-free items (a bare priority
marker) are skipped at the shared renderer, so a lone '!!!' no longer
buys a content-free turn at the cost of both nudges.
The per-child fact block takes the roster formatter's bounds: fact
lines cap at the display cap with a counts-only overflow line, and
the wait slot keeps its larger handle cap — the body is a persistent
system turn replayed on every request, and the block previously grew
without bound as finished-but-unclosed children accumulated. The two
fact sentences and the overflow line are named template constants,
and every test assertion anchors on them; the children projection
takes the same drop-never-mangle alteration check as the open-row
fields.
Eval world seeding: node metadata is JSON-encoded exactly as
production writers store it (a raw string never matched a filtered
list_nodes lookup), the stub client pins its heartbeat window open so
a static world cannot go hollow mid-run, and the world-shape
refusals' field branches gain their own tests. Comment accuracy and
paragraph wrapping fixed at the sites the review named.
Peer-review cleanliness pass over the branch's production comments and
docstrings. Dated rulings lose their dates and attribution wrappers —
the rule is the content. Measurement-process references (sweep rounds,
model names, cell names, rates, arm names, a composition caveat that
had since been satisfied) become timeless design statements: what a
property buys and what falsifies it, not which run established it.
Two real staleness bugs found by the pass: the body's properties
comment still said the escape branches come first and the done branch
is last — both false since the branch reorder — and now states the
shipped order with its trade condition.
The idle nudges enqueued on the any channel, which every drain seam
serves — a deferred wake left them deliverable at the start of a real
user send or mid-turn at a tool batch, describing an idle moment that
no longer existed. They move to a new wake channel: wake-eligible,
invisible to USER_DRAIN, TOOL_DRAIN, and the quiet ride-along. A user
cancel drops pending wake entries rather than demoting them — the
quiet demote's whole value is later seam delivery, exactly what this
class may never have. Dropping a charged entry is the accepted
fail-closed cost; liveness surviving Stop means the next idle event
fires fresh, not that a queued entry re-wakes the workstream.
A queued user interjection owns the idle seam: at wake delivery, a
non-empty interjection queue drops the wake-channel entries and the
interjection runs as a genuine user send in their place — no wake
tag, so the caps reset as for any real send and the next genuine idle
re-derives both nudges over fresh reads. The check lives in the wake
worker (which owns the slot and can dispatch a full send), not the
watcher's state-transition thread, where skipping would strand the
message. Measured before building: send('') with queued messages
appends an empty user turn and delivers the interjection one
assistant turn late, so the handoff pops first and sends the popped
text — one rendering shared with the flush seams.
External events are not idle nudges: any-channel entries still arm
the wake alone, and with an interjection waiting they ride the
genuine turn's drain seam — both deliver, only the idle nudges drop.
A failed wake send drops wake entries alongside user ones; externals
requeue quiet as before.
Every surface the model can observe must agree about the world's age
and contents. The C1 confirm at n=25 measured models sweeping memory,
skills, and list_nodes, finding voids that contradicted a transcript
full of referents, and spawning read-only investigators to resolve
the contradiction — the forbidden rate was measuring the fixture's
hollow tool-world, not dispatch discipline.
A cell's world block seeds structured memory rows through the same
upsert the memory tool's save action commits (names normalize exactly
as model-saved rows do), and node rows through the service registry
plus node metadata — the two reads list_nodes intersects, so a seeded
node is live inside the heartbeat window by construction. A seed
failure raises; a malformed world block is refused at config time
before the canary, with its own trip cell in the reachability guard.
The approval-stop cell gains the first world: two process-fact memory
rows (no coaching — the reservation lives in the transcript only) and
one live node.
Three fixes, one per causal mechanism the round-12 baseline exposed.
The done branch moves ahead of the escalate branch. The escalate-first
order rested on a harm argument — guessing on an operator decision
outranks redone bookkeeping, so the escape hatch should be salient —
and the baseline measured its cost: 7 of 10 finished-unmarked runs
reached for the body's first populated call and escalated visibly
finished work, one mode, no tail. The next round measures the reversal
both ways: if the legit-stop cells' forbidden rate rises, the harm
argument was right and the order flips back (the pin says so in
place).
The approval-stop cell's transcript anchors its world — named repo,
named migration, named artifacts. Its forbidden runs were not sign-off
defiance: the model swept empty discovery surfaces, found a void, and
spawned explore-the-project children, so the cell was measuring
hollow-world exploration rather than dispatch discipline.
The co-delivery cell's running child gains an observations-only
progress note beside its assignment. A bare-assignment static child
cannot survive sustained honest interaction — wait times out, inspect
shows nothing, and after patience cycles the model correctly diagnoses
a hung child and cancels/respawns, which the forbid list scored as
redo. The note makes the child look alive without looking finished.
The idle-children header drops its opening idleness claim: a queued
entry delivers at whichever seam arrives next, and the drain
predicate re-verifies that children are active — never that the
coordinator is still idle — so the body now opens with the one fact
the delivery just verified.
The tasks body replaces its hedged children sentence with one
observed-fact line per child. The old sentence hedged states the
producer's read had just returned and invented activity for an idle
coordinator; the producer now threads (ws_id, state) pairs through,
and the formatter renders a running child as running (check before
redoing what it owns) and a stopped one as stopped, with the
tool-behaviour fact that wait_for_workstream returns immediately for
it. The line asserts nothing about results: no read observes whether
a child produced anything, and the immediate wait is the whole
protection — checking is cheap and finds whatever is there. Fact
lines are formatter-built beside the counts opener, so no tail
override can reach them; the formatter's old indeterminate-read hedge
branch is deleted (a failed read renders no body at all), and the
open-row status takes the same alteration check as the id.
Both bodies hand the model full workstream ids: the resolver refuses
truncated ids by design, so the roster's 8-char prefixes were not
handles — a model copying a bullet issued a call the resolver
rejects. Display prefixing stays on the operator card, derived from
the full id in the metadata.
Eval alignment: fixture child ids become production-shaped 32-hex (a
prefix looked like a different id entirely and the old shape only
resolved through the legacy branch); a body-override sweep refuses
cells without a live child at config time, keyed on the formatter's
own childless condition, so candidate text can never be measured over
a world production cannot produce.
Same defect as the eval CLI: temperature defaulted to 0.7 and
reasoning effort to a code-chosen token, where the wire should omit
both and let the alias / stored setting / serving default apply. The
effort flag also loses its CLI vocabulary — the chat template is the
sole authority on valid tokens.
The flag carried choices=[low, medium, high] — a second validity
authority beside the chat template, and one that rejects tokens some
models actually define (a template that knows only high and max was
unreachable through it, while the old medium default sent a token
that same template never defined). The template is the sole
authority; the flag forwards whatever the operator typed.
The eval CLI defaulted temperature to 0.7 and reasoning effort to
medium, so every sweep sent code-chosen sampling knobs the house
assignment scheme forbids — the wire should omit the fields and let
the alias / stored setting / serving default apply, as production
does. Both flags now default to unset and the harnesses plumb None
through to model_turn, whose provider layer already omits absent
knobs. Absolute numbers from earlier sweeps were collected under the
pinned values; contrasts were at least uniform under the same pin.
User- and model-authored text (task titles, approval headers, command
previews, judge output, error fragments, notification bodies) reaches
both channel integrations verbatim, and nothing at the channel
boundary neutralised it.
The Discord client now carries a client-level allowed-mentions-none
default, which every message create inherits — plain sends, edits,
and embeds — so broadcast and mention syntax in untrusted text cannot
resolve, without mutating the text itself.
The Slack adapter escapes each untrusted field into mrkdwn entities
at its interpolation site — never the assembled message, so
deliberately bot-authored markup like the session-opener mention
survives. The policy-deny feedback returned to the server stays
verbatim; only the rendered notice escapes.
Storage and the shared formatter stay channel-neutral and verbatim:
projection happens per audience at the render boundary.
The force branch cleared worker ownership and emitted idle from the
route thread, while the abandon latch and the queue demote ran only
in the stuck worker's own exception handler — a thread force-cancel
abandons precisely because it is not making progress. Subscribers on
the IDLE fan-out therefore saw an operator-forced idle with the latch
unset: the idle observer's operator-Stop gate did not suppress
advice, and wake-eligible entries survived un-demoted, so a nudge
wake could resume a workstream seconds after the operator forced it
to stop. The route now runs the session's abandon machinery first;
the abandoned thread re-running it at its eventual death is
idempotent.
The subprocess worker built a fresh client per work item and never
closed it. Pool workers are reused across items, so each one
accumulated a live transport per item for the life of the sweep.
Picks up the needs_user status description and the new note property
from console_schemas.py. Spec-only, no API behavior change.
The server spec is knowingly left stale here: the generator writes
both files unconditionally, and its drift belongs to the change that
introduced it rather than to this one.
turnstone-eval --nudges runs seeded coordinator states against
stimulus arms and scores state-first: cells seed a real task envelope
through the production tasks_add path into a per-run temp DB, the
model's tasks calls really execute, and ground truth is the final
envelope plus a forbidden-action list — robust to action-path
variation, and never tool_choice-forced. Arms render through the
production formatters so the wire carries exactly what production
sends; a body-override lane exists for tuning A/B only and skips the
ablation arm, whose reading only means anything against the body that
ships.
Children are seeded with transcripts — an assignment message, plus a
completion-with-findings for idle children — so honest inspection
finds a world rather than an empty room, and a collect-vs-redo cell
measures the real question. Cell authoring is validated up front
(including refusing a parked task seeded beside an open one, a state
production never sends a body for); mutating calls are scored by what
landed, not by what was attempted.
Instrument health is measured, not assumed: a canary probes tool-call
parsing before and after the sweep, a mid-sweep tripwire aborts a
sweep whose parser dies rather than printing a red grid, and
empty-log runs are labelled harness: so they can never score against
the model.
The coordinator idle-observer test file lands in this commit rather
than the feature commit: its parity guard imports the eval scenarios
to pin the production formatters and the eval wire to one rendering.
The tasks tool returns an id beside its title, and spawn returns a
child's id; that pairing lives only in the transcript, and compaction
replaces the transcript. A coordinator that loses it cannot update its
own tasks or collect a finished child's results — and it is exactly
the coordinator most likely to be sitting idle holding unfinished
work. The idle nudge now carries storage-derived ids for the same
reason, and this is the other half: the nudge supplies the
authoritative set, this preserves what each one means.
The harness writes the block itself rather than asking the summariser
to preserve ids. Both are available to it — the reads are same-process
storage on the thread already running the compaction — so asking the
model to transcribe what the controller is holding would be a
shortfall in the lowering, and would make every id fallible to no
purpose. Neither compactor prompt changes at all, and a test pins that
they stay identical across kinds, so a future prose section has to
revisit this trade rather than stack on top of it.
Interactive sessions have no task envelope and no children, so they
take no reads and render nothing — a gate on the kind, not a section
that renders empty. Their compaction is unchanged by construction.
The block joins the existing carries, which is where the real hazard
was: it lands in the same post-compaction prompt as the wind-down
spill and the continuation ask, so it is counted as a third carry and
rendered against that shared budget, with the count and the render
reading one answer. Truncation drops whole rows and names what it
dropped; half an id is a call that cannot resolve wearing the costume
of one that can. Tasks are served first, with room reserved so a long
list cannot starve the children.
Titles keep their angle brackets here — unlike the nudge bodies, which
delete them because they interpolate into a system turn where a
tag-shaped run steers. This is the assistant channel, the titles are
the coordinator's own, and they already reach this same model verbatim
through its own list results; deleting brackets would only invert the
constraints it is working to. The control class is still stripped, so
a newline in a title cannot forge a sibling row.
A failed read costs the block, never the history: trading a whole
history swap for a side read would be the worse failure by far.
Two nudge classes can fire from one IDLE event, tasks first, each
asserting only its own domain.
idle_tasks (advice) fires when open (pending/in_progress) tasks
exist. The body is a counts opener, the open task ids with statuses,
and typed branches that each end in a runnable tasks(...) or
wait_for_workstream(...) call populated with real server-minted ids.
Everything it says about children is governed by one observed fact:
live children present adds the caveat sentence and the
blocked-on-a-child branch; affirmatively none says nothing about
children at all. Any needs_user row parks the class entirely — at the
fire gate and the drain predicate — because with no task graph an
open task may be gated on a parked one's unanswered question; the
operator's answer is the re-arm. Gated on memory.nudges and on the
persona actually exposing the tasks tool; carries the per-class
cooldown as well as the per-bracket cap. The tasks tool itself gains
the needs_user status and a note field — the typed escalation the
body's branches point at.
idle_children (liveness) fires when children are in a live state —
the wake that lets an idle coordinator collect a finished child's
results. The body is a roster of workstream id prefixes and states,
never names: a child's name is model-authored text and does not enter
a system turn. Cap-only and cooldown-free by design, not gated on
memory.nudges, and it survives an operator Stop.
Fail-closed, event-wide: if any storage read fails while the observer
handles an IDLE event, neither nudge is queued and neither cap is
charged. Both paths run as side-effect-free plans; the commit tail is
storage-free, so no read can fail past the veto point; both drain
predicates drop on a failed read. A path's own fault (a generic
raise) still costs only that path's fire, so one class's bug cannot
strand the other.
Task text is stored verbatim and projected per audience at render:
the model-facing projection deletes angle brackets, the operator
projection keeps them, and both strip newlines and bidi/zero-width
runs. Idle cards render what the model was told, formatted for the
operator, never augmented with content the model did not receive.
The capabilities row is a copy of claude-opus-4-8 — 1M context, 128K output,
adaptive thinking, the full low..max effort ladder, mid-conversation system
messages. Two of the model's documented breaking changes are unreachable from
this lane and stay that way only while thinking_mode is "adaptive": thinking is
on by default when the param is omitted, and disabling it is a 400 at effort
xhigh or max. We never omit it and never emit "disabled", so both are recorded
at the row rather than defended against.
The third needed work. A safety classifier can decline with
stop_reason="refusal" on a successful HTTP 200, with content either empty or
partial, and an unmapped value fell through _normalize_finish_reason as a
literal string. The drain gate only raises on an ABSENT finish reason, so a
declined turn landed as a complete result with nothing to notice it: the
interactive lane matched neither of its two warn arms and stayed silent, and a
sub-agent handed the declined partial up to its parent as though it were
finished synthesis. Normalizing onto content_filter routes the decline into the
arms both lanes already have for that state — the operator gets the warning,
and the sub-agent stops rather than passing the fragment on.
The raw stop reason is logged where it is still in hand: normalization is lossy
and a classifier decline is otherwise indistinguishable from an ordinary
content filter. The gate is on the RAW value rather than
(normalized != raw), which is true for end_turn and tool_use as well and would
fire on every turn in every lane.
The anthropic floor moves to 0.117 to track the release current at onboarding.
The model needs no new SDK surface — ids are opaque strings and "refusal" has
been in the StopReason literal since ~0.95 — so this is hygiene; raise it again
when adopting fast mode, server-side fallbacks, advisor, or mid-conversation
tool changes, which do need newer typed params.
test_persistently_crashing_entry_is_never_dropped_and_retract_frees_drain
fired its DELETE off the attempt counter, which increments as the FIRST
statement of the attempt. The counter therefore crossed 2 while the drain
still held the entry CLAIMED — popped off _pending_sends, dispatch in
flight. Retract only scans that list, and correctly answers not_found for a
claimed entry, so the request was racing the crash path's re-insert and the
assertion saw not_found instead of removed.
Both sides of that race are the same order of magnitude, which is why it
read as machine-specific rather than simply broken: the re-insert lands
after an intervening log.exception, roughly 5ms under pytest's capture
handlers against 0.05ms with none installed, and wait_until polls at 5ms.
A fast idle machine loses the race; a slower or busier one wins it.
Wait for the state the test is actually about — the entry back on the list,
mid-crash-loop and retractable — rather than for the counter. That is also
what the docstring already claims is under test.
The contract still bites: dropping the entry on the crash path fails the
new wait, and removing both retracted-purge sites leaves the drain spinning
and fails teardown.
The console proxies a node's web UI at /node/{id}/ and injects a shim into
the page it fetches upstream. The only way back was a node-picker menu the
shim built into #ui-header — an element the L-shell renovation (cc508cf4,
shipped v1.6.0) removed from the server UI. buildPicker() has returned on
its first line ever since, so every supported node version has served a
proxied page with no in-UI way back; the browser back button or a
hand-edited URL were the only exits.
The shim now repoints the rail brand (.rail-brand .brand-home) at "/" and
relabels it, so the element users already read as "go home" goes home. It
captures at the document rather than on the button: shell.js binds a bubble
listener to that same element, and stopPropagation() keeps showHome() from
firing as well. The node's own dashboard stays reachable as the
non-closable first tab.
The dead picker goes with it — its JS, the CSS constant that styled only
its elements, the el() helper, and the NODE_ID_PLACEHOLDER substitution
whose only reader it was. It could only ever have run for nodes at or below
v1.5.x, which are not a supported configuration.
The failure mode here is silent by construction: the shim reaches across a
process boundary to select classes another file emits, and fails soft when
they stop matching. Nothing failed when #ui-header disappeared. So the
coupling is now pinned from both ends.
- tests/test_shell_js.py asserts both halves. The class names are derived
from the shim's own querySelector calls, so a newly selected class is
covered without editing the guard, and a vacuity floor keeps it from
going green if the selectors are removed entirely. The containment edges
are pinned separately, deriving shell.js's local variable names from
source: renaming a local stays green, re-parenting .brand-home out of
.rail-brand does not.
- tests/test_console.py executes the shim under node against a two-walk DOM
dispatcher — capture walk, then bubble walk, phase-filtered at every node
including the target. Modelling the real rule means the test accepts any
correct wiring rather than only the one that shipped.
- The injection test drives the real proxy_index against the real node
index. That also pins the bare <body> the literal replace() depends on;
an attribute there would silently drop the entire shim, prefix rewriting
included.
- scripts/livepass.py gains a proxybrand harness for manual verification:
an iframe host over the real shell.js and the real shim, reading the
frame's post-navigation location from the surviving top page. The shim is
read out of the source by text rather than imported, since scripts/ has
no sys.path guard and an import resolves to site-packages.
Round-3 review, unprimed. Two of its majors were the new scenarios
asserting things they did not prove — the false-detector class this
campaign keeps returning to.
E8 never checked that the held /history was still OUTSTANDING when the
redial completed. The disconnect/send/wait_turn/redial sequence is
unbounded (wait_turn alone allows 45s), so on a slow box the payload
resolves while evtSource is still null, the PRESENCE term declines it, and
the run stamps dupes1-healed1 without ever evaluating the generation term.
It now fails loudly with the counter values instead. E7 gained the same
positive proof its siblings already carried: sse_opens == 0 only means
"nothing connected in 8s", which is not the same as "the held load
settled and its .finally chose not to reconnect".
E5's stated control was simply wrong, in three places. A hide nulls
evtSource, and connectSSE early-returns while hidden, so the scenario
cannot produce the non-null-but-not-OPEN source that readyState === OPEN
exists for — it exercises the presence term only. The earlier control
removed both terms at once, which is what disguised it. The readyState
half is covered by reasoning plus coord parity, and its correctness twin
IS covered through the render-time gate by E6/E8; that scope is now
written down rather than overclaimed. Coord's G5 has the same shape.
The retry floor becomes a shared export beside its jitter: four sites must
move together (both clients' arms, both non-occurrence windows) and it was
the only one of them with no single source of truth. Interactive's use of
the expression had no pin at all — reverting it to a bare 2000 would have
broken cross-client parity with the suite green. Coord's re-anchor still
raised ValueError rather than failing on a named assertion, and its first
replacement used a fixed window that truncated mid-expression.
Until now nothing in this harness could see the artefact the campaign
prevents. Every scenario counts .msg.user rows, and user rows never
travel on the SSE stream — a /send emits none, only /history replay
paints them — so a duplicated assistant bubble was invisible to all of
them. E8 counts a sentinel's occurrences in the transcript text instead,
which is structure-agnostic across duplicate bubbles and tool blocks.
The window it drives is the one readyState cannot see: the retry fires
with the transport OPEN, its /history is held, and inside that await the
transport drops and re-establishes. readyState reads OPEN afterwards
exactly as before. The redial is a real disconnect+connect rather than a
visibility change on purpose — a hide leaves evtSource null, which the
presence term already decides, so a hide-based control would pass for the
wrong reason. Control: stripping the generation term stamps dupes2.
E8's expectation needed correcting once: unlike E6, the stream is live at
flush time here, so the declined render's queued settle fires the
transport-free backstop and the pane converges in one settle. That is
correct behaviour, so the heal is asserted as a convergence leg — a "no
duplicates" verdict must not be earnable by rendering nothing.
E6 gains the non-vacuity it was missing: history_requests counts on
ARRIVAL, before the hold and before any status is chosen, so "the gate
declined a good payload" and "there was no good payload" stamped
identical observables. history_ok — incremented only when the production
route answers 200 — closes that, including the production-side-failure
hole an injected-fail budget cannot see.
Both hidden-window detectors widened for the additive jitter: sized on
the 2000 floor alone they would have closed before a top-of-range firing
and reported hidden0 for the wrong reason. delay_history(0) comments
corrected — it cannot release an in-flight hold.
Round-2 review. The render-time cursor-safety gate was point-in-time: a
transport that dropped AND finished re-establishing inside the /history
await reads back OPEN and is indistinguishable from one that never moved.
It is not — the redial re-presented the frozen cursor, the server answered
replay_ok, and the quiesce buffered that slice, so the render commits rows
the flush then repaints on top. Object identity cannot see it either,
since a native reconnect reuses the same EventSource; only a counter can.
_connectEpoch is bumped in onopen and nowhere else. Native auto-reconnect
calls neither connectSSE nor disconnectSSE, so those two are blind to the
exact case this exists for; connectSSE would also false-bump on its
document.hidden early return, which establishes no stream; and a closed
source can never fire a late open. Captured at dispatch, required
unchanged before a seedless render commits.
This is original-strata residual, not a regression this branch introduced:
before #900 the render was ungated entirely. The branch closed the
fire-time half and the still-down cases; these are the drop-and-recover
ones that were always open.
Two rulings written in at the gate, since neither is closed: a
fresh/truncated reconnect inside the await declines a render that would
have been safe (one wasted /history, self-healing via the flushed
synthetic state_change), and a refetch dispatched between onopen and the
replay slice arriving still renders past the frozen cursor — replay_ok
emits no end-of-replay marker, so no client-side signal exists (#903).
Coord's half of the same gate is #904; its exposure is a race rather than
this determinism, so it is not ported blind.
Also corrected: the claim that the idle-edge backstop's stream is live by
construction. It isn't — handleEvent also runs from the quiesce flush, so
a queued idle edge reaches the backstop with the transport down. The
render-time gate is what covers it. The clear_ui retry gains additive
jitter in BOTH clients from one shared constant: a declined render now
leaves the latch set, so a successful fetch can arm the retry, and the
decline trigger is herd-shaped. Kept small deliberately — the spread works
against #884's single-flight, which coalesces a lockstep herd.
test_coordinator_page.py anchored the fire guard on a literal `}, 2000);`
and on exact indentation; both would have ERRORED rather than failed once
the delay became an expression.
Every other interactive scenario mounts the Pane class directly, so the
factory closure that owns destroy() had no browser coverage at all — and
that is exactly where #900's largest hole lived. E7 mounts through
createInteractivePane (scenario-scoped: the factory owns its own
connect/recover-beat lifecycle, so switching the others would change what
they test), holds the first /history at the fault layer, destroys the
controller mid-flight, and lets the load resolve into the void.
Detector is a fault-layer non-occurrence: events_requests still 0, pane
detached, _visHandler null behind it. Without the bump it stamps
sse1-vis0 — the .finally reopens an EventSource on the detached pane and
re-registers the document-level visibilitychange listener destroy just
removed, which is the leak, now observed rather than traced.
An exhausted script queue still settles — into the error arm, which the
idle-edge backstop also consumes — so the heal leg would have passed for
a reason the scenario does not name. Queue the fourth turn like E4/E5 and
assert its sentinel.
E5's retry never fetches, so it cannot exercise the render-time check.
E6 reaches it the only way available: the retry fires on a live stream,
its /history is held at the fault layer, and a close-on-hide drops the
transport while the payload is in flight.
The detector is the stale-but-real PRE-rewind transcript surviving a
RESOLVED fetch — three user rows with the latch still set. The latch leg
is what makes it honest: replayHistory is the latch's only clear site, so
a held latch proves no render ran rather than inferring it from row
counts alone.
The interactive mirror of coord's G5. A close-on-hide inside the retry's
2s arm window is the reachable way to make it fire against a down
transport, and the detector is a NON-occurrence counted at the fault
layer: history_requests must be unchanged across the hidden window.
Remove the OPEN term and the hidden fetch lands, stamping hidden1.
The scenario also pins the two rulings the guard leans on: the latch
must still be SET after the skip (a skipped retry heals nothing), and
the show edge alone must not heal it — a replay_ok reconnect carries no
synthetic state_change, so the repair rides a plain send's organic
settle into the transport-free backstop, which is why exactly one new
SSE open spans show + heal.
Backport set from the #894 coordinator campaign, verified against
interactive.js source before fixing.
destroy() bumped no load token, which made it the WEAKER of the two
terminal paths (giveUp already bumped). Three escapes followed, all
reachable on the shell's onClose path: _loadHistoryThenConnect's
.finally reopened an EventSource on the detached pane and re-registered
the document-level visibilitychange listener destroy had just removed —
whose onerror then re-armed the host recover beat indefinitely, because
it gives up only on `dead`, which destroy never sets; a settling
_refetchHistory passed its supersession check and replayHistory'd into
detached DOM; and the clear_ui .then re-armed _staleRetryTimer after
destroy's own cancel. One bump at the terminal seam closes all three,
since the token is already the chokepoint every post-await consumer
reads. giveUp gains the matching timer cancel — inert is not dead.
The clear_ui retry could also fire against a DOWN transport:
disconnectSSE deliberately keeps it armed, so a hidden tab, a degraded
cooldown or a native redial holds the fire while _lastEventId is frozen.
A seedless refetch then paints rows the cursor still sits below and the
next connect's replay_ok paints them again (content and tool rows carry
no id dedup). The fire guard now requires an OPEN stream, and
_refetchHistory gains the render-time half at the chokepoint, covering
the await window a fire-time check cannot. Seeded loads are exempt by
construction — their caller disconnects first and readopts the cursor.
Deliberate trade, already ruled: the latch survives a skip, so
rewind/edit stay closed until the idle-edge backstop heals at the next
settle.
Two of the four filed findings are declined with the ruling written in
at the site, so an unprimed round re-derives rather than re-files them:
the same-token overlap is unreachable here (the replay quiesce
serializes what coord's refetchSeq stamp had to order, because coord has
no quiesce), and the joined-flight window is closed for both clients by
the shared make_history_handler's generation-keyed flight.
Review round 10 (1 minor bug; 2 major + 2 small quality — the majors
both pins-that-cannot-fail).
- The flight key's cold fallback was the literal 0, which collides
with a live session's generation 0: an eviction/close landing inside
a held flight's window let a post-truncation request rejoin a
generation-0 pre-truncation flight. Cold/detached workstreams now
key on None (rewinds need a live session, so two cold flights are
always mutually safe; a rehydrated session restarting at 0 can never
share the manager slot with its evicted predecessor — documented
at-site). The read is TYPED (live_session.session._history_generation)
so mypy carries the shape a getattr chain hid — and the typed access
immediately surfaced an unfaithful SimpleNamespace mock in the
reasoning-rehydration tests (no .session attr), now made faithful.
- Abort-Set producer pins: histCtrls.add exactly once and BEFORE the
await, delete exactly once and in the finally — without them the
destroy() consumer sweep was satisfiable by an always-empty Set.
- _make_session gains ws_id; the generation producer pin uses it.
- _coord_stick_latch: G2/G5's inline single-failure prologues RULED
deliberate at-site (their baselines/phase timings interleave into
the prologue; a per-divergence flag would obscure the choreography).
- Stray trailing whitespace stripped.
250 pins green; G2/G5/G7 re-run READY.
Review round 9 (4 minor bug, 4 quality, 1 perf nit; security zero).
- The r8 clearUiEpoch guard was UNREACHABLE (r9 bug find): clear_ui
always dispatches immediately after bumping, so a stale-epoch
dispatch is also a stale-seq dispatch and the currency gate discards
it before it can paint or clear — the client half of the joined-
flight fix was already carried by seq, and the server generation key
is the sole load-bearing layer. Machinery removed (decl, bump,
capture, conditional clear, section-9 pins); the latch-clear comment
now states the two-layer accounting.
- destroy()'s abort handle becomes a Set: a newest-wins single slot,
nulled by the newer dispatch's finally, left an OLDER overlapping
fetch unabortable — the destroyed closure pinned for the bound's
remainder. Pinned.
- _history_generation now bumps AFTER delete_messages_after: flights
rebuild from storage, so old-generation-reads-post-delete is the
harmless spuriously-fresh direction while new-generation-reads-
pre-delete would be wrongly joinable; the count/floor error paths
correctly leave it unbumped. Two-arm producer pin in
test_rewind_retry (persisted-rows bump on rewind AND retry;
in-memory-only error path must NOT bump) — the flight test's mock
can no longer mask a deleted bump.
- The harness load_calls increment takes a lock (to_thread workers
genuinely overlap under delay_load; a lost update false-fails G7).
- G7's viewer B is now a background authenticated GET (a raw request
enters load_messages identically; the second browser bought no
proof); stale two-tuple key comments and the coalescing matrix line
updated; the _send_in_page enumeration dropped for prose.
250 pins green; G1/G6/G7 re-run READY.
G7: two browsers on one ws; delay_load parks B's pre-rewind /history
flight open INSIDE load_messages — the flight layer. (A first cut
held via delay_history, which sleeps in the FAULT layer before the
route: flights never overlapped there and the 'negative control'
passed vacuously — a false detector, caught and rebuilt. The knob
also sleeps AFTER the load so a parked flight holds the rows it
actually read: its transaction point.) A rewinds mid-hold; the miss
proof is load_calls growing TWO (a joined request never enters
load_messages — the e2e twin of the unit test's proof) plus A
rendering the post-rewind single row.
The rebuilt detector immediately caught a real bug in the server fix:
mgr.get returns the Workstream WRAPPER, and the route's direct getattr
for _history_generation silently defaulted to 0 forever — joining
stayed enabled while the unit test's mock (attr on the wrong object)
masked the shape. The route now reads ws.session, and the mock pins
the nested shape so a wrong-object read can never pass again.
Negative control (flight key reverted to (ws_id, limit)): stamps
FAILED-loads1-rows3 — A joins the pre-rewind flight and paints three
stale rows as fresh truth. Fixed: READY-posts1-loads2-rows1.
The r8 joined-flight window, server half (Patrick-approved scope
expansion): the #884 single-flight key was (ws_id, limit), so a
/history dispatched AFTER a rewind/retry could join a flight whose
load_messages ran BEFORE the truncation committed — the joined
pre-rewind payload reads as fresh truth client-side (the client's
dispatch stamp is current; the staleness is the flight's transaction
point, visible only server-side) and reopened the over-rewind window
through the server seam. Reachable single-user (rewind clicked during
a truncated-resync fetch) and multi-viewer (any concurrent pane's
/history).
ChatSession gains _history_generation, bumped in _persist_truncation —
the shared rewind/retry chokepoint — BEFORE the storage write (the
in-memory tail is already trimmed by both callers; a spuriously fresh
flight is harmless, a wrongly-joined one is not). The flight key
becomes (ws_id, limit, generation): post-truncation dispatches can
never join pre-truncation flights, and the client-side clearUiEpoch
(prior commit) covers the converse (pre-rewind dispatches never CLEAR
a post-rewind latch). Cold workstreams key at generation 0 and the
first post-load truncation bumps, so cold flights cannot straddle a
rewind either.
Unit test mirrors the #884 coalescing determinism scheme: the owner
parks in load_messages under generation 0, the mid-flight bump
simulates the truncation commit, and the post-bump request must MISS
the held flight (load_calls -> 2, no coalesced record).
Negative-controlled: reverting the key to (ws_id, limit) fails the
test.
Review round 8 (2 major + 2 minor bug, 2 major + 3 small quality;
security/perf zero at five consecutive rounds).
- clearUiEpoch (r8 major): the #884 /history single-flight can hand a
joiner a payload whose load_messages ran BEFORE the rewind committed
(the flight key is (ws_id, limit); joining is invisible to the
client, and the client seq stamp cannot see server-side staleness) —
reachable single-user (rewind clicked during a truncated-resync
fetch joins that flight) and multi-viewer (any concurrent pane's
/history). The joined payload rendered as 'success' and CLEARED the
latch: the original over-rewind window, resurrected through the
server seam. Fix: the epoch bumps at clear_ui arrival, every
dispatch captures it pre-await, and only a dispatch that post-dates
the latest clear_ui may CLEAR the latch — a pre-rewind payload may
still paint (stale-but-real posture, gate holds), the surviving
latch arms the retry, and the retry's fresh dispatch starts a new
flight with post-rewind truth. Producer/consumer/placement pinned.
- destroy() aborts the in-flight bounded fetch (activeHistCtrl): the
r7 15s bound alone pinned a destroyed pane's closure until it fired
— the same dead-not-inert ruling destroy applies to staleRetryTimer.
Pinned.
- stop(hard=True) no longer sets force_exit: it skipped the ASGI
lifespan teardown and leaked the #885 daemon threads + sse_executor.
The 2s graceful-shutdown timeout already force-closes open SSE, and
the lifespan runs on both paths (docstring corrected; G6 re-verified
— the orphan still manifests).
- G6 pacing sized above the scenario's worst-case deadline sum (~200s
vs ~95s) so the in-process bash cannot resolve the orphan
mid-scenario and degrade the detector to a false READY.
- Quality: the r6 reachability comments rewritten to the r7 truth
(orphan REAL via hard crash; graceful-close-only synthesis); the
bound's WIRING pinned (signal reaches getJSON; getJSON forwards
init); _strip_comments deduped (4 inline copies); seq comment
re-paired with its asserts; retry_fire window tail-anchored.
Full harness (16/16 scenarios) + full suite (9724) green on the prior
commit; 136 pins green here.
Review round 7 (2 major + 1 minor bug, 4 minor quality; security/perf
zero). Both majors traced the r6 stratum:
- The closeStreamTransport drain of liveToolCalls rested on a false
re-announcement premise (verified: replay_ok yields only events past
the cursor; the coord fresh/truncated replay yields connected/status/
pending-cards/verdicts, never tool_pending/tool_info). An emptied
set fails OPEN — a mid-batch redial plus a slow seedless refetch
wiped the live batch. Retirement policy re-derived at the decl: an
id leaves on its RESULT, at the SETTLE edge, or with pane death;
transport death is NOT a retirement event; a stale id fails CLOSED
(skip, latch survives, settle heals). Site-anchored pins: the one
drain inside the idle/error block, the delete inside tool_result,
the adds inside tool_pending/tool_info, and closeStreamTransport's
comment-stripped code may not touch the set.
- G6's kill was not a kill: RecoveryServer.stop() gracefully closed
workstreams, and session.cancel()'s bash path persisted 'Cancelled by
user' BEFORE the reboot — the r6 'recovery synthesizes' ruling was
observing the cancel path. stop(hard=True) (skip the close sweep +
uvicorn force_exit: a crash does not drain SSE) leaves the orphan
genuinely unresulted — REACHABILITY FLIPS: the poisoned-pane state is
real, the live-set hardening is reachably load-bearing, and G6 is now
its behavioral detector: hard kill -> reload paints the orphan
(asserted PRESENT) -> the seedless rewind renders THROUGH the residue
(rewind-for-retry truth: the user message stays), negative-controlled
against the DOM-probe encoding (stamps orphan1, hist2). Discovered
and tracked separately: a hard-crashed reborn node answers stale-high
cursors with a silent fresh stream (no replay_truncated — the honest
truncation signal rides gracefully-persisted state).
- refetchHistory's await is now bounded (AbortController + 15s, the
coordSend shape): an accepted-never-answered /history pinned
refetchesInFlight and permanently disabled both heals. Pinned.
Quality: seq-producer position pinned earlier; the stale section-7
comment corrected; retry/backstop guard windows comment-stripped
(vacuous-by-comment-mention foreclosed); the shared G3/G4 double-fail
prologue extracted into _coord_stick_latch. G3/G4/G6 re-run READY.
Review round 6 (1 bug find + 5 quality; security/perf zero). The bug
finder out-traced r6-perf's dismissal: refetchHistory's own replay path
paints orphan batches (committed tool_calls, no persisted result) with
the same .conv-batch--running class the live path uses, and nothing
ever strips a dead orphan's class — so the r5 DOM-probed gate term
would let one orphan paint poison every seedless heal for the life of
the page (rewind/edit permanently dead; the seeded escape renders
through but REPAINTS the residue).
Reachability ruling (verified empirically): post-kill /history shows
the server synthesizes results for interrupted tool calls at recovery
('Cancelled by user. Outcome UNKNOWN'), so no persisted orphan exists
today and the poisoned state is unreachable — the client-side trace
was right, the server-side producer absent. Hardened regardless:
- liveToolCalls: an event-driven Set — fed ONLY by live tool_pending/
tool_info announces, retired by tool_result, drained at settle edges
and closeStreamTransport, and NEVER touched by any render (pinned:
refetchHistory's comment-stripped body may reference it exactly
once — the gate read). Liveness is read from the channel that
creates the hazard, never from DOM a render can forge.
- G6 coord-orphan-rewind: pins the SERVER invariant the client's
safety rests on — after a mid-bash node kill + reboot the batch must
render RESULTED (no --running residue) and the seedless rewind flow
must work end to end. Honestly scoped in its docstring: with
synthesis present a DOM-probe gate also passes, so the client
discipline is carried by the static pin set.
Quality batch: the seq stamp's producer position pinned (captured
before the await — the twin of the counter-bracket pin); two stale
G5 synthetic-idle comments corrected to the replay_ok-precise shape;
contract-test docstring item 7 restated to the enforced
universal-vs-seedless split; char-count pin windows replaced with
function-boundary slices (both test files); section 6 reuses _fn_slice.
Full coord family C + G1-G6 READY; 136 pins green.
Review round 5 step-back (fix-era critical): the r4 gate's busy term
conflated 'a turn is executing' with 'this DOM holds live turn state'.
_editAndResend flips busy BEFORE its POST and /rewind emits only
clear_ui (no state_change), so the busy term skipped the truncation
render the rewind exists to produce and appended the resent bubble onto
the PRE-rewind transcript; /retry's regenerated turn likewise raced its
own clear_ui refetch. The seam was re-derived once against the caller
x state matrix; the gate reads DOM-live signals only, split by scope:
- UNIVERSAL: dispatch seq (refetchSeq — overlapping fetches resolve
last-DISPATCH-wins; an older snapshot landing late can neither
double-render nor clear the latch over newer truth) and the content
refs (skipping always beats stranding a ref; seeded callers null
theirs before fetching, so it never blocks them).
- SEEDLESS-ONLY (keyed on the seedCursor arg): the
.conv-batch--running DOM marker for the tool phase (NOT activeBatch —
that is the pending-APPROVAL tracker, set only for opts.pending
batches; ruled at-site), coordSend's busySource === 'optimistic'
flavor (the one busy that marks un-committed DOM), and
stream-OPENness (CONNECTING keeps the handle with a frozen cursor
and a pending replay; handle-existence was not liveness — also
applied to the retry's fire guard). Seedless-only because the
SEEDED resync renders over these deliberately: after a node dies
mid-batch the --running class is dead residue no result will ever
strip, and the resync's render IS the recovery — a universal term
wedged the coord-restart scenario outright (family-run find; the
r5 finders missed the seeded-path interaction).
Backstop comment corrected (r5): fresh/truncated SSE replays DO carry
a synthetic state_change (replay_ok does not) — a latched pane pays one
refetch per reconnect, bounded by reconnect jitter/backoff and #884's
server single-flight; heal-caused triggers remain structurally
impossible. Caller fire-time ref guards demoted to the efficiency
layer at-site. Contract test re-pins the gate: term presence in
comment-stripped CODE, universal-vs-seedless placement, wipe between
failure guard and latch-clear, no plain busy. The edit-resend commit
gap under a second actor's clear_ui is accepted at-site (re-appears at
settle heal). G4's honesty note names the tool-phase branch it
behaviorally detects; G5 wording replay_ok-precise. Full coord family
(C + G1-G5) green; 136 pins green; 5 static mutants + the G4
behavioral control caught.
Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality;
bug-4≡q-2). Two correctness findings landed in one seam — the
refetch-vs-live-state chokepoint — so the seam was redesigned once
against its matrix (caller x stream-state-at-render x refs-at-render)
instead of patched per-finding:
- RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the
await is a real window (queued sends drain at exactly the idle edges
the backstop rides; another operator on a shared coordinator can send
any time; hide/suspend can land mid-fetch), and only the chokepoint
can see across it. Skip the wipe when a live turn exists (content
refs — a wipe strands the bubble and loses the rest of the turn
invisibly) or when a seedless render lost its idle/live-stream
precondition (busy covers the tool phase the ref check can't see;
a dead stream means rendering past the frozen cursor and
double-rendering on the show-edge replay). Both requirements key on
the seedCursor ARG — seeded callers own their reconnect flows and
legitimately rebuild mid-turn. Skips leave the latch set; heals
converge at the next organic settle.
- The retry's fire guard gains evtSource (close-on-hide keeps the timer
armed by design; a hidden firing must not fetch). The backstop needs
no term — it runs inside SSE dispatch.
- Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs
on both heal arms, the else-if exclusivity structure, the render-gate
order and terms, the evtSource guard tail. Seven mutants, all caught.
- Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its
clear_ui into a channel nobody joined and false-fails the scenario);
G4 coord-heal-midturn (a turn started under a held backstop fetch
survives its resolution; hist==2 is the discriminating bit — noted
honestly in the docstring); G5 coord-hidden-retry (hidden0
non-occurrence + organic-settle heal after show, per the accepted
liveness-lag ruling — a quiet reconnect delivers no state_change
edge). Negative controls: gate-stripped stamps hist1; guard-less
stamps hidden1. Docstring gains the G-family catalog.
Review round 3 (bug/security/perf zero; 1 quality minor): the contract
test pinned both CONSUMERS of refetchesInFlight (the backstop and
retry-fire yield guards) but not the PRODUCER ++/-- pair — dropping the
bracketing would leave the counter at 0 and both consumer pins
vacuously green. Count-pinned both sites; mutation-verified (the test
fails with the increment stripped).
Review round 2 (1 minor bug + 1 minor quality, security/perf zero):
- The retry's arm site was gated on historyStale alone, so a clear_ui
refetch in flight at destroy() that then FAILS re-arms the timer
AFTER destroy's clearTimeout — a no-op fire (the visHandler fire
guard holds) but the orphan pins the dead closure for its 2s delay,
contradicting destroy's dead-not-inert invariant. The arm gate is
now historyStale && visHandler, matching the fire guard; the seam
matrix (destroy/closeSession/live x arm-and-fire windows) closes
with that one term. The edit-resend in the same .then stays
deliberately ungated on teardown: the rewind committed server-side
and the workstream outlives the pane UI, so the committed edit
still delivers (comment at site).
- Pins: the arm gate (mutation-verified — the contract test fails
against a gate-stripped mutant), the fire guard's visHandler term
(sole coordCloseSession protection), and the re-arm clearTimeout.
G1/G2/G3 re-run READY; 136 static-pin tests green.
Review round 1 (0 correctness/security/perf; 1 minor + 1 nit) + the
suite's collateral:
- The latch-contract test now pins !refetchesInFlight on BOTH heal
paths (backstop arm + retry fire guard) — the yield guard is
load-bearing (same-snapshot double-render stomp without it) and was
previously deletable with every test green. Mutation-verified: the
backstop pin fails against a guard-stripped coordinator.js.
- test_app_js.py's clear_ui pin narrowed from all-clearTimeout to
clearTimeout(truncatedResyncTimer): the invariant it protects is that
clear_ui carries no path-local cancel of the TRUNCATED repair intent;
#894's staleRetryTimer re-arm cancel is the staleness latch's own
machinery, deliberately armed there.
- _send_in_page's caller enumeration gains G3.
G1/G2/G3 mirror interactive's E2/E3/E4 for the coordinator pane, adapted
to its structure: the pane object exposes no messagesEl and no
latch/quiesce fields (closure-private state), so every probe reads the
public #coord-messages container and the runners drive the verdicts off
the fault layer's authoritative counters — the in-flight edge is the
history_requests bump (counted on arrival, before the delay hold), the
closed phase is proven by the gated click's POST non-occurrence, and the
latch-cleared proof is the reopen POST rather than a field read.
- G1 coord-rewind-window: the busy||historyStale gate under a held-open
clear_ui refetch (delay_history); posts stays 1.
- G2 coord-rewind-failed-window: the failed-refetch aftermath — the
latch survives the failed exit, the bounded 2s retry heals (its fetch
held to defer the clear site), the healed render reopens the gate.
- G3 coord-stale-backstop: double failure (fail_history(2)) exhausts
clear_ui refetch + retry; a plain send's organic idle edge fires the
TRANSPORT-FREE backstop. Storm assertion: events_requests delta is 0
across the whole heal; history delta exactly 1.
Negative-control validated: pre-latch coordinator.js stamps
COORDREWINDWIN-posts2-rows0 and COORDREWINDFAIL-closed2-heal0; a
transport-touching backstop variant (loadHistoryThenReconnect) stamps
COORDSTALEBACKSTOP-...-sse1 — each detector has observed its bug.
The coord recovery page gains a scenario dispatch; the auto-send now
runs only for coord-restart (the rewind scenarios seed server-side),
verified against the existing coord-restart scenario.
From clear_ui arrival until the next SUCCESSFUL refetchHistory render the
visible transcript is the stale pre-rewind DOM with busy false, so a
second rewind/edit click counted it and POSTed an over-large turn count
against the already-restructured server conversation (the #890 sibling,
pre-existing since #888 accepted the stale-interactive window).
Port of interactive.js's converged #890 latch design, adapted to coord's
structure (no load token, no replay quiesce, no ref-resetting render):
- historyStale latch: set at clear_ui arrival, cleared ONLY by the
success-path render below the if-(!hist) failure guard — a flag would
reopen on the failed exit, which is exactly the over-rewind window.
- Gates: _rewindToMessage / _editAndResend / _startEdit now require
busy || historyStale; _rewindToTurns and _retryLast stay busy-only
(explicit-arg / no-DOM-count — rulings at-site).
- Heal A: one bounded turn-free retry armed in clear_ui's .then; fire
guards read the latch, refetchesInFlight (net-new await-window counter,
coord's quiesce-free yield discriminator — a COUNT because overlapping
fetches are reachable), busy, the streaming refs (load-bearing: coord's
refetch does not reset refs), and visHandler (teardown sentinel).
- Heal B: idle-edge backstop as the else-if behind the truncated-resync
consumer — TRANSPORT-FREE by ruling (plain seedless refetchHistory;
a reconnecting heal draws the synthetic state_change:idle back into
its own trigger = zero-backoff storm against a recovering node).
Carries ref guards the interactive template omits: this arm also
serves error edges where no stream_end nulled the refs.
- Teardown: destroy() cancels the retry timer (terminal-only);
closeStreamTransport deliberately does not (redials keep heal intent).
Static pins: the latch contract (set/clear/gate sites, transport-free
backstop, bounded arm, teardown split) + the widened guard-before-wipe
window; the contract pin fails against the pre-latch code.
PR #896 review follow-up, no behavior change:
- Pinned test EPOCH was 32-bit (token_hex(4)) with a matching comment, but
production widened to token_hex(8) in 2b3d0687 and the same file already
pins token_hex(8) at line 401. Widen EPOCH to 16 hex chars + fix the comment.
- Document why the fanout shutdown sentinel put stays inline on the loop: the
consumer is still alive and drains via non-blocking fan-out, so it returns
at once; the 1s timeout is a ceiling that never binds (off-loop is reserved
for the multi-second joins).
The suite's growth (~9.7k tests, coverage-instrumented, 3-version
matrix) started brushing the 20-minute hang cap on healthy runs; 30
keeps the hang-catching semantics with headroom.
Scenario F splits into F1 (manual ?last_event_id= transport) and F2, the
native-header sibling — the only behavioral coverage of two pure-browser
semantics no Tier-1 harness can express: the auto-reconnect header echo,
and id-less frames inheriting the connection's persisted lastEventId
(the mechanism behind app.js's node_snapshot-branch clear). F2's phase
C is the round-3 fix's discriminator: after the native heal, a forced
manual reconnect must go CURSORLESS with no second truncated round
(pre-fix: cursor1-trunc2), guarded by an idFrames precondition against
the aggregate tick.
Two harness seams earned by F2's first failures, both documented at
site: a failed EventSource reconnect attempt is TERMINAL per WHATWG, so
the restart must never expose a refused window — a SO_REUSEPORT
placeholder binds before the old node stops and hands its backlog to the
successor's uvicorn (make_listen_socket + RecoveryServer sock
injection); and an SSE stream still open at stop() parked uvicorn's
graceful drain indefinitely — timeout_graceful_shutdown=2 bounds it with
the #885 lifespan teardown intact.
Absence assertions tightened (round-4 review): ghost-gone now requires
absence from BOTH the model and the rail via _roster_absent_ws — the
negated AND-membership helper De Morganed into either-surface and could
false-pass a rail-render regression.
token_hex(4) left the epoch equality check — the only thing between a
prior-boot cursor and a silent replay_ok-empty alias — at 2^-32 per
same-node restart-pair; 64 bits puts a fleet-lifetime of restarts
engineered far below threshold (review round 4, classified
design-margin). Docstring rounds from the same pass: the resume
contract now notes reason=boot_epoch also covers the same-epoch
empty-ring fail-safe (not exclusively foreign epochs), and the collector
ruling says precisely that the staleness CHECK and envelope can never
fire there — the epoch-tagged ids are on the wire, just never read.
On a NATIVE reconnect into a boot_epoch truncation, the envelope and
node_snapshot frames are id-less, and an id-less frame's MessageEvent
inherits the connection's persisted pre-restart lastEventId — so the
pre-dispatch capture re-stored the dead cursor on the snapshot frame,
undoing the truncated branch's clear (a manual reconnect's fresh
EventSource starts with an empty string, which the guard blocks). A
manual reconnect racing in before the next id-bearing frame then
re-presented the dead cursor for a redundant, self-healing truncated
round. The snapshot branch now clears the cursor before the roster
rebuild — dead in every case that draws a snapshot (fresh has none,
truncated's is spent) — and the tripwire pins all three clear sites so a
simplify pass cannot drop one (round-3 review; verify classified the
mechanism redundancy-not-correctness: a cross-epoch cursor can only ever
redraw truncated+snapshot, never the silent ghost shape).
Patrick Buckley