mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
166b46cda45fc50b80e3802bc20b0ea3ad759d6a
992 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
57a9041941 |
fix(coordinator): the interjection handoff cannot lose the message, and the fact block is bounded
Review fold-in before push, twelve findings, two of them majors. The handoff popped the interjection queue destructively and handed the text to a send with a non-delivering refusal (the budget latch) and a preamble that can raise before the user turn is appended — a failure destroyed the user's words with a log line, after the charged wake nudges were already cleared. Now: the budget latch is checked before the pop (the message stays queued for a send with a human in front of it, and the wake drain still runs so the worker's exit converges); the pop returns the raw items and any non-cancel escape restores them verbatim — ids and priorities intact — before the failure surfaces; a cancel deliberately does not restore, because the Stop supersedes the queued words. Content-free items (a bare priority marker) are skipped at the shared renderer, so a lone '!!!' no longer buys a content-free turn at the cost of both nudges. The per-child fact block takes the roster formatter's bounds: fact lines cap at the display cap with a counts-only overflow line, and the wait slot keeps its larger handle cap — the body is a persistent system turn replayed on every request, and the block previously grew without bound as finished-but-unclosed children accumulated. The two fact sentences and the overflow line are named template constants, and every test assertion anchors on them; the children projection takes the same drop-never-mangle alteration check as the open-row fields. Eval world seeding: node metadata is JSON-encoded exactly as production writers store it (a raw string never matched a filtered list_nodes lookup), the stub client pins its heartbeat window open so a static world cannot go hollow mid-run, and the world-shape refusals' field branches gain their own tests. Comment accuracy and paragraph wrapping fixed at the sites the review named. |
||
|
Patrick Buckley
|
2ff7c61051 |
feat(coordinator): idle nudges deliver only on the idle wake, and a queued interjection owns the seam
The idle nudges enqueued on the any channel, which every drain seam
serves — a deferred wake left them deliverable at the start of a real
user send or mid-turn at a tool batch, describing an idle moment that
no longer existed. They move to a new wake channel: wake-eligible,
invisible to USER_DRAIN, TOOL_DRAIN, and the quiet ride-along. A user
cancel drops pending wake entries rather than demoting them — the
quiet demote's whole value is later seam delivery, exactly what this
class may never have. Dropping a charged entry is the accepted
fail-closed cost; liveness surviving Stop means the next idle event
fires fresh, not that a queued entry re-wakes the workstream.
A queued user interjection owns the idle seam: at wake delivery, a
non-empty interjection queue drops the wake-channel entries and the
interjection runs as a genuine user send in their place — no wake
tag, so the caps reset as for any real send and the next genuine idle
re-derives both nudges over fresh reads. The check lives in the wake
worker (which owns the slot and can dispatch a full send), not the
watcher's state-transition thread, where skipping would strand the
message. Measured before building: send('') with queued messages
appends an empty user turn and delivers the interjection one
assistant turn late, so the handoff pops first and sends the popped
text — one rendering shared with the flush seams.
External events are not idle nudges: any-channel entries still arm
the wake alone, and with an interjection waiting they ride the
genuine turn's drain seam — both deliver, only the idle nudges drop.
A failed wake send drops wake entries alongside user ones; externals
requeue quiet as before.
|
||
|
Patrick Buckley
|
074b9e02e2 |
feat(eval): cells seed the tool-visible world through production writers
Every surface the model can observe must agree about the world's age and contents. The C1 confirm at n=25 measured models sweeping memory, skills, and list_nodes, finding voids that contradicted a transcript full of referents, and spawning read-only investigators to resolve the contradiction — the forbidden rate was measuring the fixture's hollow tool-world, not dispatch discipline. A cell's world block seeds structured memory rows through the same upsert the memory tool's save action commits (names normalize exactly as model-saved rows do), and node rows through the service registry plus node metadata — the two reads list_nodes intersects, so a seeded node is live inside the heartbeat window by construction. A seed failure raises; a malformed world block is refused at config time before the canary, with its own trip cell in the reachability guard. The approval-stop cell gains the first world: two process-fact memory rows (no coaching — the reservation lives in the transcript only) and one live node. |
||
|
Patrick Buckley
|
76c5519b44 |
feat(coordinator): done branch leads the tasks body; eval worlds survive honest inspection
Three fixes, one per causal mechanism the round-12 baseline exposed. The done branch moves ahead of the escalate branch. The escalate-first order rested on a harm argument — guessing on an operator decision outranks redone bookkeeping, so the escape hatch should be salient — and the baseline measured its cost: 7 of 10 finished-unmarked runs reached for the body's first populated call and escalated visibly finished work, one mode, no tail. The next round measures the reversal both ways: if the legit-stop cells' forbidden rate rises, the harm argument was right and the order flips back (the pin says so in place). The approval-stop cell's transcript anchors its world — named repo, named migration, named artifacts. Its forbidden runs were not sign-off defiance: the model swept empty discovery surfaces, found a void, and spawned explore-the-project children, so the cell was measuring hollow-world exploration rather than dispatch discipline. The co-delivery cell's running child gains an observations-only progress note beside its assignment. A bare-assignment static child cannot survive sustained honest interaction — wait times out, inspect shows nothing, and after patience cycles the model correctly diagnoses a hung child and cancels/respawns, which the forbid list scored as redo. The note makes the child look alive without looking finished. |
||
|
Patrick Buckley
|
48d6b2f84b |
feat(coordinator): the nudge bodies state observed facts, never hedges
The idle-children header drops its opening idleness claim: a queued entry delivers at whichever seam arrives next, and the drain predicate re-verifies that children are active — never that the coordinator is still idle — so the body now opens with the one fact the delivery just verified. The tasks body replaces its hedged children sentence with one observed-fact line per child. The old sentence hedged states the producer's read had just returned and invented activity for an idle coordinator; the producer now threads (ws_id, state) pairs through, and the formatter renders a running child as running (check before redoing what it owns) and a stopped one as stopped, with the tool-behaviour fact that wait_for_workstream returns immediately for it. The line asserts nothing about results: no read observes whether a child produced anything, and the immediate wait is the whole protection — checking is cheap and finds whatever is there. Fact lines are formatter-built beside the counts opener, so no tail override can reach them; the formatter's old indeterminate-read hedge branch is deleted (a failed read renders no body at all), and the open-row status takes the same alteration check as the id. Both bodies hand the model full workstream ids: the resolver refuses truncated ids by design, so the roster's 8-char prefixes were not handles — a model copying a bullet issued a call the resolver rejects. Display prefixing stays on the operator card, derived from the full id in the metadata. Eval alignment: fixture child ids become production-shaped 32-hex (a prefix looked like a different id entirely and the old shape only resolved through the legacy branch); a body-override sweep refuses cells without a live child at config time, keyed on the formatter's own childless condition, so candidate text can never be measured over a world production cannot produce. |
||
|
Patrick Buckley
|
33c82962a2 |
fix(channels): suppress mention resolution and escape untrusted fields
User- and model-authored text (task titles, approval headers, command previews, judge output, error fragments, notification bodies) reaches both channel integrations verbatim, and nothing at the channel boundary neutralised it. The Discord client now carries a client-level allowed-mentions-none default, which every message create inherits — plain sends, edits, and embeds — so broadcast and mention syntax in untrusted text cannot resolve, without mutating the text itself. The Slack adapter escapes each untrusted field into mrkdwn entities at its interpolation site — never the assembled message, so deliberately bot-authored markup like the session-opener mention survives. The policy-deny feedback returned to the server stays verbatim; only the rendered notice escapes. Storage and the shared formatter stay channel-neutral and verbatim: projection happens per audience at the render boundary. |
||
|
Patrick Buckley
|
b8dd5041c4 |
fix(session): force-cancel runs the abandon machinery before emitting idle
The force branch cleared worker ownership and emitted idle from the route thread, while the abandon latch and the queue demote ran only in the stuck worker's own exception handler — a thread force-cancel abandons precisely because it is not making progress. Subscribers on the IDLE fan-out therefore saw an operator-forced idle with the latch unset: the idle observer's operator-Stop gate did not suppress advice, and wake-eligible entries survived un-demoted, so a nudge wake could resume a workstream seconds after the operator forced it to stop. The route now runs the session's abandon machinery first; the abandoned thread re-running it at its eventual death is idempotent. |
||
|
Patrick Buckley
|
556ec793a0 |
feat(eval): behavioral eval for the coordinator idle nudges
turnstone-eval --nudges runs seeded coordinator states against stimulus arms and scores state-first: cells seed a real task envelope through the production tasks_add path into a per-run temp DB, the model's tasks calls really execute, and ground truth is the final envelope plus a forbidden-action list — robust to action-path variation, and never tool_choice-forced. Arms render through the production formatters so the wire carries exactly what production sends; a body-override lane exists for tuning A/B only and skips the ablation arm, whose reading only means anything against the body that ships. Children are seeded with transcripts — an assignment message, plus a completion-with-findings for idle children — so honest inspection finds a world rather than an empty room, and a collect-vs-redo cell measures the real question. Cell authoring is validated up front (including refusing a parked task seeded beside an open one, a state production never sends a body for); mutating calls are scored by what landed, not by what was attempted. Instrument health is measured, not assumed: a canary probes tool-call parsing before and after the sweep, a mid-sweep tripwire aborts a sweep whose parser dies rather than printing a red grid, and empty-log runs are labelled harness: so they can never score against the model. The coordinator idle-observer test file lands in this commit rather than the feature commit: its parity guard imports the eval scenarios to pin the production formatters and the eval wire to one rendering. |
||
|
Patrick Buckley
|
a53bf30428 |
feat(coordinator): carry task and child handles across a compaction
The tasks tool returns an id beside its title, and spawn returns a child's id; that pairing lives only in the transcript, and compaction replaces the transcript. A coordinator that loses it cannot update its own tasks or collect a finished child's results — and it is exactly the coordinator most likely to be sitting idle holding unfinished work. The idle nudge now carries storage-derived ids for the same reason, and this is the other half: the nudge supplies the authoritative set, this preserves what each one means. The harness writes the block itself rather than asking the summariser to preserve ids. Both are available to it — the reads are same-process storage on the thread already running the compaction — so asking the model to transcribe what the controller is holding would be a shortfall in the lowering, and would make every id fallible to no purpose. Neither compactor prompt changes at all, and a test pins that they stay identical across kinds, so a future prose section has to revisit this trade rather than stack on top of it. Interactive sessions have no task envelope and no children, so they take no reads and render nothing — a gate on the kind, not a section that renders empty. Their compaction is unchanged by construction. The block joins the existing carries, which is where the real hazard was: it lands in the same post-compaction prompt as the wind-down spill and the continuation ask, so it is counted as a third carry and rendered against that shared budget, with the count and the render reading one answer. Truncation drops whole rows and names what it dropped; half an id is a call that cannot resolve wearing the costume of one that can. Tasks are served first, with room reserved so a long list cannot starve the children. Titles keep their angle brackets here — unlike the nudge bodies, which delete them because they interpolate into a system turn where a tag-shaped run steers. This is the assistant channel, the titles are the coordinator's own, and they already reach this same model verbatim through its own list results; deleting brackets would only invert the constraints it is working to. The control class is still stripped, so a newline in a title cannot forge a sibling row. A failed read costs the block, never the history: trading a whole history swap for a side read would be the worse failure by far. |
||
|
Patrick Buckley
|
0d52b63b50 |
feat(coordinator): nudge a coordinator that goes idle holding unfinished work
Two nudge classes can fire from one IDLE event, tasks first, each asserting only its own domain. idle_tasks (advice) fires when open (pending/in_progress) tasks exist. The body is a counts opener, the open task ids with statuses, and typed branches that each end in a runnable tasks(...) or wait_for_workstream(...) call populated with real server-minted ids. Everything it says about children is governed by one observed fact: live children present adds the caveat sentence and the blocked-on-a-child branch; affirmatively none says nothing about children at all. Any needs_user row parks the class entirely — at the fire gate and the drain predicate — because with no task graph an open task may be gated on a parked one's unanswered question; the operator's answer is the re-arm. Gated on memory.nudges and on the persona actually exposing the tasks tool; carries the per-class cooldown as well as the per-bracket cap. The tasks tool itself gains the needs_user status and a note field — the typed escalation the body's branches point at. idle_children (liveness) fires when children are in a live state — the wake that lets an idle coordinator collect a finished child's results. The body is a roster of workstream id prefixes and states, never names: a child's name is model-authored text and does not enter a system turn. Cap-only and cooldown-free by design, not gated on memory.nudges, and it survives an operator Stop. Fail-closed, event-wide: if any storage read fails while the observer handles an IDLE event, neither nudge is queued and neither cap is charged. Both paths run as side-effect-free plans; the commit tail is storage-free, so no read can fail past the veto point; both drain predicates drop on a failed read. A path's own fault (a generic raise) still costs only that path's fire, so one class's bug cannot strand the other. Task text is stored verbatim and projected per audience at render: the model-facing projection deletes angle brackets, the operator projection keeps them, and both strip newlines and bidi/zero-width runs. Idle cards render what the model was told, formatted for the operator, never augmented with content the model did not receive. |
||
|
Patrick Buckley
|
96834496c4 |
feat(anthropic): onboard claude-opus-5
The capabilities row is a copy of claude-opus-4-8 — 1M context, 128K output, adaptive thinking, the full low..max effort ladder, mid-conversation system messages. Two of the model's documented breaking changes are unreachable from this lane and stay that way only while thinking_mode is "adaptive": thinking is on by default when the param is omitted, and disabling it is a 400 at effort xhigh or max. We never omit it and never emit "disabled", so both are recorded at the row rather than defended against. The third needed work. A safety classifier can decline with stop_reason="refusal" on a successful HTTP 200, with content either empty or partial, and an unmapped value fell through _normalize_finish_reason as a literal string. The drain gate only raises on an ABSENT finish reason, so a declined turn landed as a complete result with nothing to notice it: the interactive lane matched neither of its two warn arms and stayed silent, and a sub-agent handed the declined partial up to its parent as though it were finished synthesis. Normalizing onto content_filter routes the decline into the arms both lanes already have for that state — the operator gets the warning, and the sub-agent stops rather than passing the fragment on. The raw stop reason is logged where it is still in hand: normalization is lossy and a classifier decline is otherwise indistinguishable from an ordinary content filter. The gate is on the RAW value rather than (normalized != raw), which is true for end_turn and tool_use as well and would fire on every turn in every lane. The anthropic floor moves to 0.117 to track the release current at onboarding. The model needs no new SDK surface — ids are opaque strings and "refusal" has been in the StopReason literal since ~0.95 — so this is hygiene; raise it again when adopting fast mode, server-side fallbacks, advisor, or mid-conversation tool changes, which do need newer typed params. |
||
|
Patrick Buckley
|
04c29c8ef5 |
test(authz): stop the retract test racing the drain's claim window
test_persistently_crashing_entry_is_never_dropped_and_retract_frees_drain fired its DELETE off the attempt counter, which increments as the FIRST statement of the attempt. The counter therefore crossed 2 while the drain still held the entry CLAIMED — popped off _pending_sends, dispatch in flight. Retract only scans that list, and correctly answers not_found for a claimed entry, so the request was racing the crash path's re-insert and the assertion saw not_found instead of removed. Both sides of that race are the same order of magnitude, which is why it read as machine-specific rather than simply broken: the re-insert lands after an intervening log.exception, roughly 5ms under pytest's capture handlers against 0.05ms with none installed, and wait_until polls at 5ms. A fast idle machine loses the race; a slower or busier one wins it. Wait for the state the test is actually about — the entry back on the list, mid-crash-loop and retractable — rather than for the counter. That is also what the docstring already claims is under test. The contract still bites: dropping the entry on the crash path fails the new wait, and removing both retracted-purge sites leaves the drain spinning and fails teardown. |
||
|
Patrick Buckley
|
6f194d338f |
fix(console): restore back-to-console from a proxied node view
The console proxies a node's web UI at /node/{id}/ and injects a shim into
the page it fetches upstream. The only way back was a node-picker menu the
shim built into #ui-header — an element the L-shell renovation (
|
||
|
Patrick Buckley
|
4007fab855 |
fix(#900): close two vacuity holes the round-2 scenarios left open
Round-3 review, unprimed. Two of its majors were the new scenarios asserting things they did not prove — the false-detector class this campaign keeps returning to. E8 never checked that the held /history was still OUTSTANDING when the redial completed. The disconnect/send/wait_turn/redial sequence is unbounded (wait_turn alone allows 45s), so on a slow box the payload resolves while evtSource is still null, the PRESENCE term declines it, and the run stamps dupes1-healed1 without ever evaluating the generation term. It now fails loudly with the counter values instead. E7 gained the same positive proof its siblings already carried: sse_opens == 0 only means "nothing connected in 8s", which is not the same as "the held load settled and its .finally chose not to reconnect". E5's stated control was simply wrong, in three places. A hide nulls evtSource, and connectSSE early-returns while hidden, so the scenario cannot produce the non-null-but-not-OPEN source that readyState === OPEN exists for — it exercises the presence term only. The earlier control removed both terms at once, which is what disguised it. The readyState half is covered by reasoning plus coord parity, and its correctness twin IS covered through the render-time gate by E6/E8; that scope is now written down rather than overclaimed. Coord's G5 has the same shape. The retry floor becomes a shared export beside its jitter: four sites must move together (both clients' arms, both non-occurrence windows) and it was the only one of them with no single source of truth. Interactive's use of the expression had no pin at all — reverting it to a bare 2000 would have broken cross-client parity with the suite green. Coord's re-anchor still raised ValueError rather than failing on a named assertion, and its first replacement used a fixed window that truncated mid-expression. |
||
|
Patrick Buckley
|
7ed5d90a98 |
test(e2e): E8 observes the double render; honest non-vacuity for E6 (#900 r2)
Until now nothing in this harness could see the artefact the campaign prevents. Every scenario counts .msg.user rows, and user rows never travel on the SSE stream — a /send emits none, only /history replay paints them — so a duplicated assistant bubble was invisible to all of them. E8 counts a sentinel's occurrences in the transcript text instead, which is structure-agnostic across duplicate bubbles and tool blocks. The window it drives is the one readyState cannot see: the retry fires with the transport OPEN, its /history is held, and inside that await the transport drops and re-establishes. readyState reads OPEN afterwards exactly as before. The redial is a real disconnect+connect rather than a visibility change on purpose — a hide leaves evtSource null, which the presence term already decides, so a hide-based control would pass for the wrong reason. Control: stripping the generation term stamps dupes2. E8's expectation needed correcting once: unlike E6, the stream is live at flush time here, so the declined render's queued settle fires the transport-free backstop and the pane converges in one settle. That is correct behaviour, so the heal is asserted as a convergence leg — a "no duplicates" verdict must not be earnable by rendering nothing. E6 gains the non-vacuity it was missing: history_requests counts on ARRIVAL, before the hold and before any status is chosen, so "the gate declined a good payload" and "there was no good payload" stamped identical observables. history_ok — incremented only when the production route answers 200 — closes that, including the production-side-failure hole an injected-fail budget cannot see. Both hidden-window detectors widened for the additive jitter: sized on the 2000 floor alone they would have closed before a top-of-range firing and reported hidden0 for the wrong reason. delay_history(0) comments corrected — it cannot release an in-flight hold. |
||
|
Patrick Buckley
|
7daf3b782d |
fix(#900): stream generation closes the reconnect-inside-the-await render; jitter both retries
Round-2 review. The render-time cursor-safety gate was point-in-time: a transport that dropped AND finished re-establishing inside the /history await reads back OPEN and is indistinguishable from one that never moved. It is not — the redial re-presented the frozen cursor, the server answered replay_ok, and the quiesce buffered that slice, so the render commits rows the flush then repaints on top. Object identity cannot see it either, since a native reconnect reuses the same EventSource; only a counter can. _connectEpoch is bumped in onopen and nowhere else. Native auto-reconnect calls neither connectSSE nor disconnectSSE, so those two are blind to the exact case this exists for; connectSSE would also false-bump on its document.hidden early return, which establishes no stream; and a closed source can never fire a late open. Captured at dispatch, required unchanged before a seedless render commits. This is original-strata residual, not a regression this branch introduced: before #900 the render was ungated entirely. The branch closed the fire-time half and the still-down cases; these are the drop-and-recover ones that were always open. Two rulings written in at the gate, since neither is closed: a fresh/truncated reconnect inside the await declines a render that would have been safe (one wasted /history, self-healing via the flushed synthetic state_change), and a refetch dispatched between onopen and the replay slice arriving still renders past the frozen cursor — replay_ok emits no end-of-replay marker, so no client-side signal exists (#903). Coord's half of the same gate is #904; its exposure is a race rather than this determinism, so it is not ported blind. Also corrected: the claim that the idle-edge backstop's stream is live by construction. It isn't — handleEvent also runs from the quiesce flush, so a queued idle edge reaches the backstop with the transport down. The render-time gate is what covers it. The clear_ui retry gains additive jitter in BOTH clients from one shared constant: a declined render now leaves the latch set, so a successful fetch can arm the retry, and the decline trigger is herd-shaped. Kept small deliberately — the spread works against #884's single-flight, which coalesces a lockstep herd. test_coordinator_page.py anchored the fire guard on a literal `}, 2000);` and on exact indentation; both would have ERRORED rather than failed once the delay became an expression. |
||
|
Patrick Buckley
|
a8ce660619 |
fix(#900): destroy invalidates in-flight loads; cursor-safety gates the seedless render
Backport set from the #894 coordinator campaign, verified against interactive.js source before fixing. destroy() bumped no load token, which made it the WEAKER of the two terminal paths (giveUp already bumped). Three escapes followed, all reachable on the shell's onClose path: _loadHistoryThenConnect's .finally reopened an EventSource on the detached pane and re-registered the document-level visibilitychange listener destroy had just removed — whose onerror then re-armed the host recover beat indefinitely, because it gives up only on `dead`, which destroy never sets; a settling _refetchHistory passed its supersession check and replayHistory'd into detached DOM; and the clear_ui .then re-armed _staleRetryTimer after destroy's own cancel. One bump at the terminal seam closes all three, since the token is already the chokepoint every post-await consumer reads. giveUp gains the matching timer cancel — inert is not dead. The clear_ui retry could also fire against a DOWN transport: disconnectSSE deliberately keeps it armed, so a hidden tab, a degraded cooldown or a native redial holds the fire while _lastEventId is frozen. A seedless refetch then paints rows the cursor still sits below and the next connect's replay_ok paints them again (content and tool rows carry no id dedup). The fire guard now requires an OPEN stream, and _refetchHistory gains the render-time half at the chokepoint, covering the await window a fire-time check cannot. Seeded loads are exempt by construction — their caller disconnects first and readopts the cursor. Deliberate trade, already ruled: the latch survives a skip, so rewind/edit stay closed until the idle-edge backstop heals at the next settle. Two of the four filed findings are declined with the ruling written in at the site, so an unprimed round re-derives rather than re-files them: the same-token overlap is unreachable here (the replay quiesce serializes what coord's refetchSeq stamp had to order, because coord has no quiesce), and the joined-flight window is closed for both clients by the shared make_history_handler's generation-keyed flight. |
||
|
Patrick Buckley
|
d8d026394f |
fix(#894): cold flights key on None; typed generation access; abort-Set producer pins
Review round 10 (1 minor bug; 2 major + 2 small quality — the majors both pins-that-cannot-fail). - The flight key's cold fallback was the literal 0, which collides with a live session's generation 0: an eviction/close landing inside a held flight's window let a post-truncation request rejoin a generation-0 pre-truncation flight. Cold/detached workstreams now key on None (rewinds need a live session, so two cold flights are always mutually safe; a rehydrated session restarting at 0 can never share the manager slot with its evicted predecessor — documented at-site). The read is TYPED (live_session.session._history_generation) so mypy carries the shape a getattr chain hid — and the typed access immediately surfaced an unfaithful SimpleNamespace mock in the reasoning-rehydration tests (no .session attr), now made faithful. - Abort-Set producer pins: histCtrls.add exactly once and BEFORE the await, delete exactly once and in the finally — without them the destroy() consumer sweep was satisfiable by an always-empty Set. - _make_session gains ws_id; the generation producer pin uses it. - _coord_stick_latch: G2/G5's inline single-failure prologues RULED deliberate at-site (their baselines/phase timings interleave into the prologue; a per-divergence flag would obscure the choreography). - Stray trailing whitespace stripped. 250 pins green; G2/G5/G7 re-run READY. |
||
|
Patrick Buckley
|
60f6dc07a2 |
fix(#894): drop the unreachable epoch guard; abort-Set; bump-after-delete; producer pins
Review round 9 (4 minor bug, 4 quality, 1 perf nit; security zero). - The r8 clearUiEpoch guard was UNREACHABLE (r9 bug find): clear_ui always dispatches immediately after bumping, so a stale-epoch dispatch is also a stale-seq dispatch and the currency gate discards it before it can paint or clear — the client half of the joined- flight fix was already carried by seq, and the server generation key is the sole load-bearing layer. Machinery removed (decl, bump, capture, conditional clear, section-9 pins); the latch-clear comment now states the two-layer accounting. - destroy()'s abort handle becomes a Set: a newest-wins single slot, nulled by the newer dispatch's finally, left an OLDER overlapping fetch unabortable — the destroyed closure pinned for the bound's remainder. Pinned. - _history_generation now bumps AFTER delete_messages_after: flights rebuild from storage, so old-generation-reads-post-delete is the harmless spuriously-fresh direction while new-generation-reads- pre-delete would be wrongly joinable; the count/floor error paths correctly leave it unbumped. Two-arm producer pin in test_rewind_retry (persisted-rows bump on rewind AND retry; in-memory-only error path must NOT bump) — the flight test's mock can no longer mask a deleted bump. - The harness load_calls increment takes a lock (to_thread workers genuinely overlap under delay_load; a lost update false-fails G7). - G7's viewer B is now a background authenticated GET (a raw request enters load_messages identically; the second browser bought no proof); stale two-tuple key comments and the coalescing matrix line updated; the _send_in_page enumeration dropped for prose. 250 pins green; G1/G6/G7 re-run READY. |
||
|
Patrick Buckley
|
b85f792925 |
test(e2e): G7 joined-flight detector at the flight layer; fix the generation read path it caught (#894 r8)
G7: two browsers on one ws; delay_load parks B's pre-rewind /history flight open INSIDE load_messages — the flight layer. (A first cut held via delay_history, which sleeps in the FAULT layer before the route: flights never overlapped there and the 'negative control' passed vacuously — a false detector, caught and rebuilt. The knob also sleeps AFTER the load so a parked flight holds the rows it actually read: its transaction point.) A rewinds mid-hold; the miss proof is load_calls growing TWO (a joined request never enters load_messages — the e2e twin of the unit test's proof) plus A rendering the post-rewind single row. The rebuilt detector immediately caught a real bug in the server fix: mgr.get returns the Workstream WRAPPER, and the route's direct getattr for _history_generation silently defaulted to 0 forever — joining stayed enabled while the unit test's mock (attr on the wrong object) masked the shape. The route now reads ws.session, and the mock pins the nested shape so a wrong-object read can never pass again. Negative control (flight key reverted to (ws_id, limit)): stamps FAILED-loads1-rows3 — A joins the pre-rewind flight and paints three stale rows as fresh truth. Fixed: READY-posts1-loads2-rows1. |
||
|
Patrick Buckley
|
bc60646ff9 |
fix(#894): fold the truncation generation into the /history flight key
The r8 joined-flight window, server half (Patrick-approved scope expansion): the #884 single-flight key was (ws_id, limit), so a /history dispatched AFTER a rewind/retry could join a flight whose load_messages ran BEFORE the truncation committed — the joined pre-rewind payload reads as fresh truth client-side (the client's dispatch stamp is current; the staleness is the flight's transaction point, visible only server-side) and reopened the over-rewind window through the server seam. Reachable single-user (rewind clicked during a truncated-resync fetch) and multi-viewer (any concurrent pane's /history). ChatSession gains _history_generation, bumped in _persist_truncation — the shared rewind/retry chokepoint — BEFORE the storage write (the in-memory tail is already trimmed by both callers; a spuriously fresh flight is harmless, a wrongly-joined one is not). The flight key becomes (ws_id, limit, generation): post-truncation dispatches can never join pre-truncation flights, and the client-side clearUiEpoch (prior commit) covers the converse (pre-rewind dispatches never CLEAR a post-rewind latch). Cold workstreams key at generation 0 and the first post-load truncation bumps, so cold flights cannot straddle a rewind either. Unit test mirrors the #884 coalescing determinism scheme: the owner parks in load_messages under generation 0, the mid-flight bump simulates the truncation commit, and the post-bump request must MISS the held flight (load_calls -> 2, no coalesced record). Negative-controlled: reverting the key to (ws_id, limit) fails the test. |
||
|
Patrick Buckley
|
30b6ff7f7b |
fix(#894): rewind-freshness epoch closes the #884 joined-flight window; destroy aborts the bounded fetch
Review round 8 (2 major + 2 minor bug, 2 major + 3 small quality; security/perf zero at five consecutive rounds). - clearUiEpoch (r8 major): the #884 /history single-flight can hand a joiner a payload whose load_messages ran BEFORE the rewind committed (the flight key is (ws_id, limit); joining is invisible to the client, and the client seq stamp cannot see server-side staleness) — reachable single-user (rewind clicked during a truncated-resync fetch joins that flight) and multi-viewer (any concurrent pane's /history). The joined payload rendered as 'success' and CLEARED the latch: the original over-rewind window, resurrected through the server seam. Fix: the epoch bumps at clear_ui arrival, every dispatch captures it pre-await, and only a dispatch that post-dates the latest clear_ui may CLEAR the latch — a pre-rewind payload may still paint (stale-but-real posture, gate holds), the surviving latch arms the retry, and the retry's fresh dispatch starts a new flight with post-rewind truth. Producer/consumer/placement pinned. - destroy() aborts the in-flight bounded fetch (activeHistCtrl): the r7 15s bound alone pinned a destroyed pane's closure until it fired — the same dead-not-inert ruling destroy applies to staleRetryTimer. Pinned. - stop(hard=True) no longer sets force_exit: it skipped the ASGI lifespan teardown and leaked the #885 daemon threads + sse_executor. The 2s graceful-shutdown timeout already force-closes open SSE, and the lifespan runs on both paths (docstring corrected; G6 re-verified — the orphan still manifests). - G6 pacing sized above the scenario's worst-case deadline sum (~200s vs ~95s) so the in-process bash cannot resolve the orphan mid-scenario and degrade the detector to a false READY. - Quality: the r6 reachability comments rewritten to the r7 truth (orphan REAL via hard crash; graceful-close-only synthesis); the bound's WIRING pinned (signal reaches getJSON; getJSON forwards init); _strip_comments deduped (4 inline copies); seq comment re-paired with its asserts; retry_fire window tail-anchored. Full harness (16/16 scenarios) + full suite (9724) green on the prior commit; 136 pins green here. |
||
|
Patrick Buckley
|
412161aa4a |
fix(#894): live-set retirement policy — transport death is not retirement; bound the refetch await; G6 hard-kill detector
Review round 7 (2 major + 1 minor bug, 4 minor quality; security/perf zero). Both majors traced the r6 stratum: - The closeStreamTransport drain of liveToolCalls rested on a false re-announcement premise (verified: replay_ok yields only events past the cursor; the coord fresh/truncated replay yields connected/status/ pending-cards/verdicts, never tool_pending/tool_info). An emptied set fails OPEN — a mid-batch redial plus a slow seedless refetch wiped the live batch. Retirement policy re-derived at the decl: an id leaves on its RESULT, at the SETTLE edge, or with pane death; transport death is NOT a retirement event; a stale id fails CLOSED (skip, latch survives, settle heals). Site-anchored pins: the one drain inside the idle/error block, the delete inside tool_result, the adds inside tool_pending/tool_info, and closeStreamTransport's comment-stripped code may not touch the set. - G6's kill was not a kill: RecoveryServer.stop() gracefully closed workstreams, and session.cancel()'s bash path persisted 'Cancelled by user' BEFORE the reboot — the r6 'recovery synthesizes' ruling was observing the cancel path. stop(hard=True) (skip the close sweep + uvicorn force_exit: a crash does not drain SSE) leaves the orphan genuinely unresulted — REACHABILITY FLIPS: the poisoned-pane state is real, the live-set hardening is reachably load-bearing, and G6 is now its behavioral detector: hard kill -> reload paints the orphan (asserted PRESENT) -> the seedless rewind renders THROUGH the residue (rewind-for-retry truth: the user message stays), negative-controlled against the DOM-probe encoding (stamps orphan1, hist2). Discovered and tracked separately: a hard-crashed reborn node answers stale-high cursors with a silent fresh stream (no replay_truncated — the honest truncation signal rides gracefully-persisted state). - refetchHistory's await is now bounded (AbortController + 15s, the coordSend shape): an accepted-never-answered /history pinned refetchesInFlight and permanently disabled both heals. Pinned. Quality: seq-producer position pinned earlier; the stale section-7 comment corrected; retry/backstop guard windows comment-stripped (vacuous-by-comment-mention foreclosed); the shared G3/G4 double-fail prologue extracted into _coord_stick_latch. G3/G4/G6 re-run READY. |
||
|
Patrick Buckley
|
cc776bfb2d |
fix(#894): event-driven live-tool-call set replaces the DOM liveness probe; G6 synthesis tripwire
Review round 6 (1 bug find + 5 quality; security/perf zero). The bug
finder out-traced r6-perf's dismissal: refetchHistory's own replay path
paints orphan batches (committed tool_calls, no persisted result) with
the same .conv-batch--running class the live path uses, and nothing
ever strips a dead orphan's class — so the r5 DOM-probed gate term
would let one orphan paint poison every seedless heal for the life of
the page (rewind/edit permanently dead; the seeded escape renders
through but REPAINTS the residue).
Reachability ruling (verified empirically): post-kill /history shows
the server synthesizes results for interrupted tool calls at recovery
('Cancelled by user. Outcome UNKNOWN'), so no persisted orphan exists
today and the poisoned state is unreachable — the client-side trace
was right, the server-side producer absent. Hardened regardless:
- liveToolCalls: an event-driven Set — fed ONLY by live tool_pending/
tool_info announces, retired by tool_result, drained at settle edges
and closeStreamTransport, and NEVER touched by any render (pinned:
refetchHistory's comment-stripped body may reference it exactly
once — the gate read). Liveness is read from the channel that
creates the hazard, never from DOM a render can forge.
- G6 coord-orphan-rewind: pins the SERVER invariant the client's
safety rests on — after a mid-bash node kill + reboot the batch must
render RESULTED (no --running residue) and the seedless rewind flow
must work end to end. Honestly scoped in its docstring: with
synthesis present a DOM-probe gate also passes, so the client
discipline is carried by the static pin set.
Quality batch: the seq stamp's producer position pinned (captured
before the await — the twin of the counter-bracket pin); two stale
G5 synthetic-idle comments corrected to the replay_ok-precise shape;
contract-test docstring item 7 restated to the enforced
universal-vs-seedless split; char-count pin windows replaced with
function-boundary slices (both test files); section 6 reuses _fn_slice.
Full coord family C + G1-G6 READY; 136 pins green.
|
||
|
Patrick Buckley
|
fac2393967 |
fix(#894): re-derive the render gate on DOM-live signals; drop the busy conflation
Review round 5 step-back (fix-era critical): the r4 gate's busy term conflated 'a turn is executing' with 'this DOM holds live turn state'. _editAndResend flips busy BEFORE its POST and /rewind emits only clear_ui (no state_change), so the busy term skipped the truncation render the rewind exists to produce and appended the resent bubble onto the PRE-rewind transcript; /retry's regenerated turn likewise raced its own clear_ui refetch. The seam was re-derived once against the caller x state matrix; the gate reads DOM-live signals only, split by scope: - UNIVERSAL: dispatch seq (refetchSeq — overlapping fetches resolve last-DISPATCH-wins; an older snapshot landing late can neither double-render nor clear the latch over newer truth) and the content refs (skipping always beats stranding a ref; seeded callers null theirs before fetching, so it never blocks them). - SEEDLESS-ONLY (keyed on the seedCursor arg): the .conv-batch--running DOM marker for the tool phase (NOT activeBatch — that is the pending-APPROVAL tracker, set only for opts.pending batches; ruled at-site), coordSend's busySource === 'optimistic' flavor (the one busy that marks un-committed DOM), and stream-OPENness (CONNECTING keeps the handle with a frozen cursor and a pending replay; handle-existence was not liveness — also applied to the retry's fire guard). Seedless-only because the SEEDED resync renders over these deliberately: after a node dies mid-batch the --running class is dead residue no result will ever strip, and the resync's render IS the recovery — a universal term wedged the coord-restart scenario outright (family-run find; the r5 finders missed the seeded-path interaction). Backstop comment corrected (r5): fresh/truncated SSE replays DO carry a synthetic state_change (replay_ok does not) — a latched pane pays one refetch per reconnect, bounded by reconnect jitter/backoff and #884's server single-flight; heal-caused triggers remain structurally impossible. Caller fire-time ref guards demoted to the efficiency layer at-site. Contract test re-pins the gate: term presence in comment-stripped CODE, universal-vs-seedless placement, wipe between failure guard and latch-clear, no plain busy. The edit-resend commit gap under a second actor's clear_ui is accepted at-site (re-appears at settle heal). G4's honesty note names the tool-phase branch it behaviorally detects; G5 wording replay_ok-precise. Full coord family (C + G1-G5) green; 136 pins green; 5 static mutants + the G4 behavioral control caught. |
||
|
Patrick Buckley
|
54631d3111 |
fix(#894): render-time gate at the refetch chokepoint; liveness-gate the retry; G4/G5 scenarios
Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality; bug-4≡q-2). Two correctness findings landed in one seam — the refetch-vs-live-state chokepoint — so the seam was redesigned once against its matrix (caller x stream-state-at-render x refs-at-render) instead of patched per-finding: - RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the await is a real window (queued sends drain at exactly the idle edges the backstop rides; another operator on a shared coordinator can send any time; hide/suspend can land mid-fetch), and only the chokepoint can see across it. Skip the wipe when a live turn exists (content refs — a wipe strands the bubble and loses the rest of the turn invisibly) or when a seedless render lost its idle/live-stream precondition (busy covers the tool phase the ref check can't see; a dead stream means rendering past the frozen cursor and double-rendering on the show-edge replay). Both requirements key on the seedCursor ARG — seeded callers own their reconnect flows and legitimately rebuild mid-turn. Skips leave the latch set; heals converge at the next organic settle. - The retry's fire guard gains evtSource (close-on-hide keeps the timer armed by design; a hidden firing must not fetch). The backstop needs no term — it runs inside SSE dispatch. - Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs on both heal arms, the else-if exclusivity structure, the render-gate order and terms, the evtSource guard tail. Seven mutants, all caught. - Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its clear_ui into a channel nobody joined and false-fails the scenario); G4 coord-heal-midturn (a turn started under a held backstop fetch survives its resolution; hist==2 is the discriminating bit — noted honestly in the docstring); G5 coord-hidden-retry (hidden0 non-occurrence + organic-settle heal after show, per the accepted liveness-lag ruling — a quiet reconnect delivers no state_change edge). Negative controls: gate-stripped stamps hist1; guard-less stamps hidden1. Docstring gains the G-family catalog. |
||
|
Patrick Buckley
|
dd1db5c67e |
test(#894): pin the in-flight counter's producer bracketing
Review round 3 (bug/security/perf zero; 1 quality minor): the contract test pinned both CONSUMERS of refetchesInFlight (the backstop and retry-fire yield guards) but not the PRODUCER ++/-- pair — dropping the bracketing would leave the counter at 0 and both consumer pins vacuously green. Count-pinned both sites; mutation-verified (the test fails with the increment stripped). |
||
|
Patrick Buckley
|
ac441471f9 |
fix(#894): teardown-gate the retry ARM; pin both teardown sentinels
Review round 2 (1 minor bug + 1 minor quality, security/perf zero): - The retry's arm site was gated on historyStale alone, so a clear_ui refetch in flight at destroy() that then FAILS re-arms the timer AFTER destroy's clearTimeout — a no-op fire (the visHandler fire guard holds) but the orphan pins the dead closure for its 2s delay, contradicting destroy's dead-not-inert invariant. The arm gate is now historyStale && visHandler, matching the fire guard; the seam matrix (destroy/closeSession/live x arm-and-fire windows) closes with that one term. The edit-resend in the same .then stays deliberately ungated on teardown: the rewind committed server-side and the workstream outlives the pane UI, so the committed edit still delivers (comment at site). - Pins: the arm gate (mutation-verified — the contract test fails against a gate-stripped mutant), the fire guard's visHandler term (sole coordCloseSession protection), and the re-arm clearTimeout. G1/G2/G3 re-run READY; 136 static-pin tests green. |
||
|
Patrick Buckley
|
85214f433f |
test(#894): pin the yield guards; narrow the clear_ui clearTimeout pin
Review round 1 (0 correctness/security/perf; 1 minor + 1 nit) + the suite's collateral: - The latch-contract test now pins !refetchesInFlight on BOTH heal paths (backstop arm + retry fire guard) — the yield guard is load-bearing (same-snapshot double-render stomp without it) and was previously deletable with every test green. Mutation-verified: the backstop pin fails against a guard-stripped coordinator.js. - test_app_js.py's clear_ui pin narrowed from all-clearTimeout to clearTimeout(truncatedResyncTimer): the invariant it protects is that clear_ui carries no path-local cancel of the TRUNCATED repair intent; #894's staleRetryTimer re-arm cancel is the staleness latch's own machinery, deliberately armed there. - _send_in_page's caller enumeration gains G3. |
||
|
Patrick Buckley
|
f7ca4d295d |
fix(coordinator): historyStale latch closes the clear_ui over-rewind window (#894)
From clear_ui arrival until the next SUCCESSFUL refetchHistory render the visible transcript is the stale pre-rewind DOM with busy false, so a second rewind/edit click counted it and POSTed an over-large turn count against the already-restructured server conversation (the #890 sibling, pre-existing since #888 accepted the stale-interactive window). Port of interactive.js's converged #890 latch design, adapted to coord's structure (no load token, no replay quiesce, no ref-resetting render): - historyStale latch: set at clear_ui arrival, cleared ONLY by the success-path render below the if-(!hist) failure guard — a flag would reopen on the failed exit, which is exactly the over-rewind window. - Gates: _rewindToMessage / _editAndResend / _startEdit now require busy || historyStale; _rewindToTurns and _retryLast stay busy-only (explicit-arg / no-DOM-count — rulings at-site). - Heal A: one bounded turn-free retry armed in clear_ui's .then; fire guards read the latch, refetchesInFlight (net-new await-window counter, coord's quiesce-free yield discriminator — a COUNT because overlapping fetches are reachable), busy, the streaming refs (load-bearing: coord's refetch does not reset refs), and visHandler (teardown sentinel). - Heal B: idle-edge backstop as the else-if behind the truncated-resync consumer — TRANSPORT-FREE by ruling (plain seedless refetchHistory; a reconnecting heal draws the synthetic state_change:idle back into its own trigger = zero-backoff storm against a recovering node). Carries ref guards the interactive template omits: this arm also serves error edges where no stream_end nulled the refs. - Teardown: destroy() cancels the retry timer (terminal-only); closeStreamTransport deliberately does not (redials keep heal intent). Static pins: the latch contract (set/clear/gate sites, transport-free backstop, bounded arm, teardown split) + the widened guard-before-wipe window; the contract pin fails against the pre-latch code. |
||
|
Patrick Buckley
|
09a27cfce9 |
docs(#881): faithful token_hex(8) test epoch; document inline shutdown put
PR #896 review follow-up, no behavior change:
- Pinned test EPOCH was 32-bit (token_hex(4)) with a matching comment, but
production widened to token_hex(8) in
|
||
|
Patrick Buckley
|
5386d598ef |
test(e2e): native-transport roster scenario (F2) + strict absence assertions (#881)
Scenario F splits into F1 (manual ?last_event_id= transport) and F2, the native-header sibling — the only behavioral coverage of two pure-browser semantics no Tier-1 harness can express: the auto-reconnect header echo, and id-less frames inheriting the connection's persisted lastEventId (the mechanism behind app.js's node_snapshot-branch clear). F2's phase C is the round-3 fix's discriminator: after the native heal, a forced manual reconnect must go CURSORLESS with no second truncated round (pre-fix: cursor1-trunc2), guarded by an idFrames precondition against the aggregate tick. Two harness seams earned by F2's first failures, both documented at site: a failed EventSource reconnect attempt is TERMINAL per WHATWG, so the restart must never expose a refused window — a SO_REUSEPORT placeholder binds before the old node stops and hands its backlog to the successor's uvicorn (make_listen_socket + RecoveryServer sock injection); and an SSE stream still open at stop() parked uvicorn's graceful drain indefinitely — timeout_graceful_shutdown=2 bounds it with the #885 lifespan teardown intact. Absence assertions tightened (round-4 review): ghost-gone now requires absence from BOTH the model and the rail via _roster_absent_ws — the negated AND-membership helper De Morganed into either-surface and could false-pass a rail-render regression. |
||
|
Patrick Buckley
|
8a67f91d8b |
fix(server): widen the boot epoch to 64 bits; docstring precision (#881)
token_hex(4) left the epoch equality check — the only thing between a prior-boot cursor and a silent replay_ok-empty alias — at 2^-32 per same-node restart-pair; 64 bits puts a fleet-lifetime of restarts engineered far below threshold (review round 4, classified design-margin). Docstring rounds from the same pass: the resume contract now notes reason=boot_epoch also covers the same-epoch empty-ring fail-safe (not exclusively foreign epochs), and the collector ruling says precisely that the staleness CHECK and envelope can never fire there — the epoch-tagged ids are on the wire, just never read. |
||
|
Patrick Buckley
|
58bd607f49 |
fix(ui): the snapshot recovery floor clears the global resume cursor (#881)
On a NATIVE reconnect into a boot_epoch truncation, the envelope and node_snapshot frames are id-less, and an id-less frame's MessageEvent inherits the connection's persisted pre-restart lastEventId — so the pre-dispatch capture re-stored the dead cursor on the snapshot frame, undoing the truncated branch's clear (a manual reconnect's fresh EventSource starts with an empty string, which the guard blocks). A manual reconnect racing in before the next id-bearing frame then re-presented the dead cursor for a redundant, self-healing truncated round. The snapshot branch now clears the cursor before the roster rebuild — dead in every case that draws a snapshot (fresh has none, truncated's is spent) — and the tripwire pins all three clear sites so a simplify pass cannot drop one (round-3 review; verify classified the mechanism redundancy-not-correctness: a cross-epoch cursor can only ever redraw truncated+snapshot, never the silent ghost shape). |
||
|
Patrick Buckley
|
4232136d26 |
fix(server): de-register the global listener when the reconnect window exits early (#881)
Round-1's lock-scope fix moved the snapshot build after listener registration but left it unguarded: a raising _build_node_snapshot (storage reads, per-ws locks) propagated before the generator — whose finally owns de-registration — ever existed, stranding a dead 1000-slot queue in the fan-out list forever (the fan-out thread never removes listeners; pre-branch the append was the LAST locked statement precisely so a raising build could not strand it). The whole post-registration window (build, log, response construction) now runs under a guard that de-registers on ANY exit and re-raises; _deregister is shared with the generator's finally so the discipline has one owner. BaseException because the window must stay guarded even if a future edit introduces an await (today it is await-free, so a cancel cannot land inside it). Tests (round-2 review): the leak path is pinned (raising build → exception propagates AND the listener list is empty); the registration-before-build + lock-released ordering is pinned by a probe builder asserting both at build time; the caught-up-cursor test is rebuilt around a sentinel live event so it asserts the no-envelope shape positively instead of truncating the drain at the retry frame. |
||
|
Patrick Buckley
|
d84a3c6eb3 |
docs(tests): honest coverage pointer for the stubbed snapshot builder (#881)
test_console.py covers the CONSUMER side of node_snapshot (hand-built dicts fed to the collector), not _build_node_snapshot's production — the helper docstring claimed otherwise. Point at the real end-to-end coverage (the roster-restart scenario: membership + evict) and state plainly that the producer's field projection has no direct unit test (review round 1, quality finding). |
||
|
Patrick Buckley
|
ea706bf4c0 |
test(e2e): roster-restart scenario proves the global boot-epoch heal (#881)
Scenario F drives the REAL node dashboard (/ + app.js) through a node restart on the global stream — no custom page; transport instrumentation is injected via CDP addScriptToEvaluateOnNewDocument, scoped to /events/global URLs so per-ws streams can't pollute the counters. Phase A is the negative control: live roster, live cursor, zero replay_truncated. Phase B: hide, force the CLOSED state (a closed EventSource never auto-retries, making the show edge's manual reconnect the only reconnect), restart the node re-opening only one of two workstreams, show. Asserted: cursor presented via ?last_event_id= and replay_truncated observed at the transport, the not-reopened workstream's ghost evicted from the roster model and rail (the dashboard table's membership refreshes on interaction by design — documented at _roster_has_ws), and the reborn node's global_events_requests counter proves the reconnect hit the real endpoint. The native header transport differs only in carriage and is pinned by the Tier-1 boot-epoch tests. |
||
|
Patrick Buckley
|
a935ae3106 |
fix(server): give the lifespan daemon threads a real shutdown (#885)
_global_fanout_thread, _aggregate_emitter_thread, and _idle_cleanup_thread were daemon threads with no stop signal — shutdown abandoned them mid-loop. The sleep-loop pair now waits on a shared Event (wait doubles as the tick sleep, so a set wakes them immediately); the fanout exits on an identity-checked queue sentinel, FIFO-draining everything enqueued before it (sessions close earlier in the shutdown tail, so their final events still fan out). Joins are bounded and off-loop; daemon=True stays as the backstop for a join timeout, not the mechanism. The recovery harness drops its thread-neutering workaround (module docstring piece 4) — the global lane now runs REAL in harness boots, which the #881 roster-restart scenario requires. |
||
|
Patrick Buckley
|
22c905b2ec |
feat(ui): present the global resume cursor on manual reconnects (#881)
The global stream's manual reconnects were pinned cursorless because a stale cursor on the reborn ring drew replay_ok-empty with no snapshot (the ghost-roster shape). With epoch-tagged ids that shape is unreachable — a stale cursor now draws replay_truncated + a fresh node_snapshot — so app.js captures e.lastEventId (MessageEvent, house guard form), presents it via ?last_event_id= on manual reconnects, and clears it where the record dies: the replay_truncated handler and onLogout. The cursor stays an opaque string end to end; the tripwire that pinned cursorlessness now pins the capture, the guarded query-param presentation, and the never-parse-numerically discipline instead. |
||
|
Patrick Buckley
|
e640aeda66 |
fix(server): boot-epoch staleness signal on the global SSE stream (#881)
The global ring's counter is process-local and reboots at 0, so after a
node restart a pre-restart cursor was first invisibly ahead of the reborn
ring (replay_ok with an empty slice) and then aliased into the new id
space as the counter re-grew — both silently skipping the restart
boundary (ghost rosters). Every global SSE id is now
"{boot_epoch}-{counter}" (per-process nonce); the browser echoes it
verbatim on native reconnect, so provenance rides every path with zero
client cooperation. A cursor from any other epoch — prior boot, another
node, a pre-epoch bare-int client, garbage — draws replay_truncated
(reason=boot_epoch, loss unknowable so the numeric fields are omitted)
plus the node_snapshot recovery floor; in-epoch ring misses keep honest
lost_count under reason=ring_evicted. Same-epoch cursors run the ring
logic unchanged. Chokepoint log line added; per-ws ids deliberately stay
bare ints (storage-seeded counter — asymmetry documented at both sites);
collector audit ruling recorded at its cursorless connect.
|
||
|
Patrick Buckley
|
af918c321c |
docs(interactive): correct #890 gate refs + rule the heal's fire-and-forget
Addresses the Copilot review of #895 (docs/comments only, no behavior change): - recovery_e2e.py / _sse_recovery_server.py: the mutating affordance gate is `busy || _historyStale`, not the superseded `busy || _replayQueue` quiesce gate the r3 latch replaced — corrected both docstrings (E2 now matches E3). - interactive.js cross-ws supersession: the branch drops the pending edit and releases busy but does NOT clear `_historyStale` (its sole clear site is replayHistory) — reworded so it no longer implies the latch is released. - interactive.js idle-edge backstop + bounded retry: documented that the fire-and-forget `_refetchHistory` (no `.catch`) is deliberate — no composer state to un-strand there, unlike the primary clear_ui caller, so a render throw stays loud (peer of the load path's `.finally`). |
||
|
Patrick Buckley
|
74eedff1a8 |
test(e2e): fault-injection knobs + five recovery scenarios for the /history failure paths
RecoveryServer grows an in-process fault layer (pure-ASGI wrapper; the production app is untouched): fail_history(count) serves minimal 500s for the next N GET /history requests, delay_history(ms) holds responses to widen or hold open a refetch window, and per-route request counters (history_requests, rewind_requests) let scenarios assert backend state rather than scripted absence. Five scenarios on that layer, all stamping RECOVERY-READY/FAILED titles like their siblings: - fail-refetch: hide mid-turn -> restart -> failed first resync -> the stale transcript survives (no wipe, no empty-state) while the truncation record stays armed -> the connect-chokepoint retry heals (history_requests proves the re-fetch). The #890 acceptance contract, browser-observed end to end. - stale-ref-reload: mid-segment transport death -> turn completes during the outage -> failed unarmed same-ws reload -> the next turn renders in a FRESH bubble and the stale bubble's text is unchanged (regression test for the resumability-gated ref reset). - rewind-window: a second rewind clicked during a held clear_ui refetch window never reaches the server (rewind_requests == 1) and the transcript reflects one rewind (regression test for the busy-or-latch affordance gate, in-window arm). - rewind-failed-window: the failed-fetch AFTERMATH sibling — the refetch 500s, the staleness latch keeps the gate closed over the stale rows (rewind_requests stuck at 1, proven latch-not-quiesce via a settle-poll), the bounded turn-free retry heals (3 -> 1 user rows), and only then does the gate reopen (rewind_requests == 2). Negative-control validated: with the interactive.js fixes reverted, stale-ref-reload stamps fresh0-unchanged0 (the concatenation bug), rewind-window stamps posts2-rows0 (the in-window over-rewind), and rewind-failed-window stamps closed2-rows0 (the failed-exit over-rewind) — every detector observes its bug, then stamps READY again with the fixes restored. |
||
|
Patrick Buckley
|
99fa3def1b |
fix(interactive): preserve the pane on a failed /history refetch (#890)
Port the coordinator's #882 G3 guard-before-wipe: the wipe + streaming- ref reset live in replayHistory, reached only on a successful fetch. - clear_ui no longer pre-wipes the transcript; a failed refetch during a rewind/retry/resume replay keeps stale-but-real content instead of blanking the highest-traffic pane on a live stream (/history failures cluster in exactly the restart windows that emit clear_ui). - _refetchHistory's failure branch is a DOM/ref/repair-intent no-op: no empty-state hint below stale content (the old resync-route wart), no streaming-ref reset (which orphaned a mid-jitter turn's bubble on the resync route); the truncation record stays armed for the connect-chokepoint retry; only the quiesce releases. - _loadHistoryThenConnect resets streaming refs on a ws SWITCH only -- the old ws's refs otherwise survive a failed fetch into the new ws's stream; a same-ws reload keeps them so the reconnect resumes the mid-jitter bubble instead of orphaning it. - The factory connect() empty-state pre-seed is now the sole producer of the failed-first-paint placeholder -- documented load-bearing. The edit-and-resend dispatch, cross-ws supersession, and repair-intent lifecycle are unchanged; a failed fetch keeps the resend firing (the rewind already committed server-side), mirroring coord. Pinned by test_interactive_refetch_failure_preserves_the_pane (the mirror of coord's test_coordinator_refetch_failure_preserves_the_pane) plus the re-pointed quiesce/agent-tracking pin. |
||
|
Patrick Buckley
|
7f74e9594e |
feat(session): coalesce concurrent /history reconstructions per workstream (#884)
After a node restart every open pane resyncs via REST /history inside the same jitter window; client jitter spreads the peak but not the total. Concurrent requests for the same (ws_id, limit) now share ONE reconstruction (load_messages -> decoration -> projection) via a single-flight task map in the handler closure. Deliberately single-flight only, no TTL cache: the payload depends on live-mutable inputs with no total cheap invalidation signal (the surface_persisted_reasoning registry toggle emits no per-ws event; cold workstreams have no event counter), so a cache could serve stale reasoning/approval/cursor state for its whole TTL, while a joiner's worst-case staleness equals the flight duration -- the window a lone slow request already exposes. All auth/tenant/kind/existence gates stay per-request ahead of the join; only the caller-independent reconstruction is shared. A shared draw that hit a transient load_messages failure is not fanned out: joiners retry once, independently, so one storage blip cannot wipe every coalesced pane (the 200-empty payload renders as an authoritative empty pane in both clients, and the seedless clear_ui path has no SSE redelivery to repair it). The flight is a detached task (awaiters shield it) so an owner disconnect cannot strand joiners, and each task pops its own key in a finally, so the map only ever holds in-flight work. ws.history.load_failed rises to warning: it now names the draw that triggers joiner retries and renders as a pane wipe. |
||
|
Patrick Buckley
|
6c4c848a08 |
fix(session): survive tool-result truncation at zero context budget (#883)
At an exhausted context budget the drain loop replaced every tool result with a placeholder that read as a successful-but-trimmed call. For structural results — spawn_workstream's ws_id, the tasks scratchpad — the model lost the handle orchestration depends on and silently stalled, while the UI (told the real summary before the drain) kept showing success. Worse, the budget zeroes near 70% fullness when max_tokens ≥ context_window/4, well below the 80% auto-compact threshold, so a stalled coordinator could sit in that band indefinitely with no compaction ever firing. Three guarantees at the truncation seam, one renewal trigger at the drain: - structural-tool and error results get a guaranteed 2048-char admission floor (head+tail beyond it) — never the zero-budget drop - any result at or under the floor passes verbatim (denial notices, spawn acks: never destroy what is smaller than the guarantee) - bulky non-structural results get an explicit drop notice stating the call RAN but its output could not be admitted — never a trim impersonation the model cannot distinguish from success - a zero truncation budget triggers one mid-turn compaction (no threshold_pct — none was evaluated, same rule as the ctx-overflow retry), closing the 70-80% band where the budget zeroed but compaction was never owed Background-bash spawn acks ride the small-result pass; a name-keyed floor cannot distinguish them from foreground bash — see #891. |
||
|
Patrick Buckley
|
51ad8366d9 |
fix(interactive): supersede all repair intent on a full history render
A mid-stream replay_truncated latches _pendingTruncatedResync; a clear_ui rebuild (rewind / edit-and-resend) heals the gap but left the latch — and any pending jittered _resyncTimer — armed, because clear_ui keeps the stream live and only disconnectSSE cancelled the timer. The next idle edge then fired a phantom _loadHistoryThenConnect against the already-repaired gap: a false truncatedGaps bump and a needless teardown, and on the phantom's failed-fetch leg the reconnect went cursorless (_lastEventId nulled with no record armed) with nothing left to re-cover the suspend window. replayHistory now clears the gap record, the deferred latch, and the pending timer together — the same one-site supersession the coordinator port established in refetchHistory. The latch/timer clears are no-ops on every _loadHistoryThenConnect flavor (each clears both before its fetch); the clear_ui heal is the path they exist for. A failed fetch still clears none (it never reaches replayHistory), keeping the connect chokepoint's retry armed. Found as a latent shared shape by the #882 review's round-4 pass and confirmed against this file; pinned in the fresh-connect/churn-limit test alongside a guard that clear_ui never grows a path-local cancel. |
||
|
Patrick Buckley
|
14c246a569 |
fix(coord): port the truncated-recovery design from interactive (#882)
replay_truncated is now a dead-stream signal, mirroring the converged interactive.js machinery: - loadHistoryThenReconnect: tear the transport down first, drop the live cursor, refetch /history with cursor adoption, reconnect in .finally. The old in-place refetch discarded the /history cursor while /history trims the trailing in-flight turn whenever it returns one — a mid-run truncation wiped the executing turn with no redelivery and later tool results orphaned into top-level bubbles. Both consumption sites (immediate branch and idle-edge deferred consumer) route through it. Dropping the cursor before the fetch is load-bearing, not just parity: a post-restart heal on an idle ws gets no /history cursor, and re-presenting the frozen pre-restart cursor against the reseeded empty ring draws replay_truncated forever — an envelope→resync loop that parks the pane in degraded cooldown cycles (caught by the new browser-level scenario, invisible to source-pattern tests). - truncatedFromCursor: the truncation-time cursor, recorded keep-oldest at the envelope and cleared only by a successful full render; the connect chokepoint presents it over the live cursor so every manual reconnect re-draws the envelope and the repair survives any teardown interleaving (hide/show, degraded cooldown, CLOSED retry, failed fetch). - churn ladder: truncated resyncs feed the same rolling window as overflow closes via the extracted recordChurnAndMaybeTrip(); a trip skips the resync (the degraded wake re-arms via the chokepoint). - herd jitter: resyncs start behind a 0..TRUNCATED_RESYNC_JITTER_MS spread; one pending resync at a time; the fire path nulls its handle before loading; closeStreamTransport owns cancellation. - sidebar refresh: while a truncation gap is on record the gap machinery owns recovery outright — the envelope refreshes once per NEW gap, one heal-time refresh covers the retry window, and onopen's no-cursor / long-gap arm stands down — so a failed-resync retry loop cannot stampede /children + /tasks un-jittered once per reconnect through either path. - a failed /history refetch no longer blanks the pane (wipe + tracking resets sit below the !hist guard); a successful full render supersedes ALL pending repair intent in one place (gap record, deferred latch, pending resync timer) so a heal can never strand a phantom resync. Behavioral coverage: scripts/recovery_e2e.py gains --scenario coord-restart — the REAL coordinator pane (chrome, cookie auth, EventSource, connect chokepoint, resync, churn limiter) mounted against the interactive recovery node (/coord-static + /coord-recovery), driven through hide → node restart → show over CDP, asserting the envelope is drawn, the hidden-window turns heal, the stream re-opens, and the pane converges. Revised the two tests that pinned the in-place shape, added the coordinator mirror of interactive's fresh-connect/churn-limit pins (keep-oldest record, chokepoint consult, clear-on-render, shared churn step, trip-skip, jitter scheduler, cancellation site, cursor drop, per-gap sidebar dedup). |
||
|
Patrick Buckley
|
431ef7c2fe |
ci: give the e2e_recovery suite its own lane exclusion, drop the live co-mark
The recovery e2e tests run a scripted provider — no LLM backend — so the live co-mark was a lie told to keep the existing CI expression skipping them. Both CI lanes now deselect explicitly via -m "not live and not e2e_recovery", and the tests carry only their honest marker. Select with -m e2e_recovery. |
||
|
Patrick Buckley
|
7a43d37f8b |
fix(sse): capture the reconnect cursor from the MessageEvent, not the EventSource
All three clients read lastEventId off the EventSource object, but per WHATWG the property lives on the MessageEvent — EventSource exposes only url/withCredentials/readyState. The object-form reads were dead conditionals in every real browser: the cursor never tracked live traffic, every MANUAL reconnect (close-on-hide show edge, degraded- ladder retry, recover beat) opened cursorless as a fresh connect, and a fresh connect does not refetch history — so turns committed while a tab was hidden silently never painted. This is the cleanest mechanism behind the 'turn disappeared, never healed' field reports, and it gated the branch's recovery fixes: without a presented cursor, the empty-ring truncated honesty could never fire for hidden-tab restarts and the truncation record captured null. Native auto-reconnects were unaffected (the browser sends its internal Last-Event-ID header), which is why the bug stayed invisible: transient blips healed, deliberate closes lost. Capture e.lastEventId in each onmessage instead, guarded != null and !== "" — no-id frames carry the empty string and "0" is a valid id (the error-surface snap_seq can be 0 on a brand-new workstream). The coordinator's counter-reset detector, which compared against the same dead property and so never fired, now works as documented. Found by the recovery harness's first real-browser run: source-pattern tests pin a wrong-object property read as happily as a right one, so a tripwire test now forbids the object form by name across all three clients, and Tier-2 scenario B is upgraded to hide MID-turn and require the browser-observed replay_truncated envelope plus the healed gap (RECOVERY-READY-RESTART-rows1-trunc1 demonstrated; was trunc0). |
||
|
Patrick Buckley
|
43561c9b08 |
test(sse): end-to-end recovery harness (server-contract + browser livepass)
Tier 1 (tests/test_sse_recovery_e2e.py, opt-in e2e_recovery marker): six scenarios against a real interactive server with a scripted provider and ephemeral DBs — storm batching without loss, slow-consumer overflow with lossless ring replay, mid-run truncation with cursor-adoption rebuild, restart truncated-honesty (exact lost_count; no-loss variant replay_ok), failed-resync retry via the truncation record, and sub-agent storm attribution. BrowserlikeSSEClient (tests/_sse_recovery_helpers.py) implements the browser cursor contract; RecoveryServer (tests/_sse_recovery_server.py) boots the real app per test. Tier 2 (scripts/recovery_e2e.py): the livepass idiom against a REAL node — boots the real InteractivePane over real EventSource/authFetch, with a dependency-free CDP runner driving the storm and hide-restart-show scenarios; document.title stamps verdicts so a broken state cannot pass silently. Events are produced by the real session engine through the provider boundary — no synthetic frames; teardown leaves no leaked threads; the default suite keeps these deselected. |