From 54631d3111bf42db6a31d5746d3178c4bf91f92a Mon Sep 17 00:00:00 2001 From: Patrick Buckley Date: Fri, 24 Jul 2026 11:26:56 -0700 Subject: [PATCH] fix(#894): render-time gate at the refetch chokepoint; liveness-gate the retry; G4/G5 scenarios MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 4 (1 major + 3 minor bug, 1 major + 3 minor quality; bug-4≡q-2). Two correctness findings landed in one seam — the refetch-vs-live-state chokepoint — so the seam was redesigned once against its matrix (caller x stream-state-at-render x refs-at-render) instead of patched per-finding: - RENDER-TIME gate inside refetchHistory, post-await, pre-wipe: the await is a real window (queued sends drain at exactly the idle edges the backstop rides; another operator on a shared coordinator can send any time; hide/suspend can land mid-fetch), and only the chokepoint can see across it. Skip the wipe when a live turn exists (content refs — a wipe strands the bubble and loses the rest of the turn invisibly) or when a seedless render lost its idle/live-stream precondition (busy covers the tool phase the ref check can't see; a dead stream means rendering past the frozen cursor and double-rendering on the show-edge replay). Both requirements key on the seedCursor ARG — seeded callers own their reconnect flows and legitimately rebuild mid-turn. Skips leave the latch set; heals converge at the next organic settle. - The retry's fire guard gains evtSource (close-on-hide keeps the timer armed by design; a hidden firing must not fetch). The backstop needs no term — it runs inside SSE dispatch. - Pins: producer ORDER (inc < await < finally < dec), ref-guard pairs on both heal arms, the else-if exclusivity structure, the render-gate order and terms, the evtSource guard tail. Seven mutants, all caught. - Harness: __esOpens gate in G1-G3 (a pre-connect rewind drops its clear_ui into a channel nobody joined and false-fails the scenario); G4 coord-heal-midturn (a turn started under a held backstop fetch survives its resolution; hist==2 is the discriminating bit — noted honestly in the docstring); G5 coord-hidden-retry (hidden0 non-occurrence + organic-settle heal after show, per the accepted liveness-lag ruling — a quiet reconnect delivers no state_change edge). Negative controls: gate-stripped stamps hist1; guard-less stamps hidden1. Docstring gains the G-family catalog. --- scripts/recovery_e2e.py | 390 +++++++++++++++++- tests/test_app_js.py | 6 +- tests/test_coordinator_page.py | 64 ++- .../console/static/coordinator/coordinator.js | 47 ++- 4 files changed, 498 insertions(+), 9 deletions(-) diff --git a/scripts/recovery_e2e.py b/scripts/recovery_e2e.py index af4b4ad7..87563dc2 100644 --- a/scripts/recovery_e2e.py +++ b/scripts/recovery_e2e.py @@ -26,14 +26,16 @@ Usage:: python3 scripts/recovery_e2e.py --scenario coord-rewind-window # G1 (#894) python3 scripts/recovery_e2e.py --scenario coord-rewind-failed-window # G2 (#894) python3 scripts/recovery_e2e.py --scenario coord-stale-backstop # G3 (#894) + python3 scripts/recovery_e2e.py --scenario coord-heal-midturn # G4 (#894) + python3 scripts/recovery_e2e.py --scenario coord-hidden-retry # G5 (#894) python3 scripts/recovery_e2e.py --scenario roster-restart # F1 (#881) python3 scripts/recovery_e2e.py --scenario roster-restart-native # F2 (#881) python3 scripts/recovery_e2e.py --scenario both # A+B only (legacy) python3 scripts/recovery_e2e.py --keep-open 8971 # serve the storm page # for manual inspection -The #890 guard-before-wipe family (all against the interactive pane, no -coordinator): +The #890 guard-before-wipe family (against the interactive pane; the +coordinator ports are the G family below): Scenario D (fail-refetch): clones B, but arms one forced /history 500 (``RecoveryServer.fail_history``) just before the show edge so the first @@ -93,6 +95,52 @@ clear_ui). All polls are deadline-bounded so a regressed looping backstop stamps a clean FAILED, never a hang. Stamps ``RECOVERY-READY-STALEBACKSTOP-heal1-sse0``. +The #894 coordinator ports (G family — the coord pane's ``historyStale`` +latch; coord state is closure-private, so where E2-E4 read pane fields the +G runners read the fault layer's authoritative counters and prove gate +closure by POST NON-occurrence; the latch-cleared proof is the reopen POST): + +Scenario G1 (coord-rewind-window): E2's port. The clear_ui refetch is held +open; the in-flight edge is the ``history_requests`` bump (counted on +arrival, before the hold sleeps — the latch was set synchronously before +that fetch dispatched). A second rewind click mid-hold must be gated by +``busy || historyStale``. Stamps ``RECOVERY-READY-COORDREWINDWIN-posts1``. + +Scenario G2 (coord-rewind-failed-window): E3's port. The failed refetch +keeps the latch set (its only clear sits below the ``if (!hist) return`` +failure guard); the retry's held /history defers the heal so the gated +click provably lands in the aftermath window; the bounded 2s retry then +heals and reopens. Stamps ``RECOVERY-READY-COORDREWINDFAIL-posts2-heal1``. + +Scenario G3 (coord-stale-backstop): E4's port. Double failure exhausts +clear_ui refetch + retry; a plain send's ORGANIC settle fires the +TRANSPORT-FREE backstop (plain seedless ``refetchHistory``, never +``loadHistoryThenReconnect``). Same storm proof: ``events_requests`` +UNCHANGED, ``history_requests`` +1. Stamps +``RECOVERY-READY-COORDSTALEBACKSTOP-heal1-sse0``. + +Scenario G4 (coord-heal-midturn, #894 r4): the render-time gate. A turn +STARTED during a heal's held /history must survive the fetch resolution — +the gate skips the wipe (pre-gate code detached the live turn's DOM into +dangling refs and lost it), the latch stays set, and the turn's own settle +re-fires the backstop, which heals. Proofs: turn 5's sentinel paints into +the still-stale transcript (``mid1``), the heal lands with the rewound-away +seeds gone, ``events_requests`` UNCHANGED, ``history_requests`` +2 exactly +(the skipped fetch + the heal). Stamps +``RECOVERY-READY-COORDHEALMIDTURN-heal1-mid1-sse0``. + +Scenario G5 (coord-hidden-retry, #894 r4): the retry's stream-liveness +fire guard. A retry armed before close-on-hide must NOT fetch while the +transport is down (a seedless render past the frozen ``lastEventId`` +double-renders on the show-edge replay): ``history_requests`` UNCHANGED +across the hidden fire window (``hidden0``). A quiet reconnect delivers +no state_change edge, so post-show the latch stays closed (the accepted +liveness-lag residual) until the runner drives an ORGANIC settle with a +plain send; that idle edge fires the TRANSPORT-FREE backstop on the live +stream (exactly ONE new SSE open across show + heal — the user-driven +reconnect; the heal adds zero). Stamps +``RECOVERY-READY-COORDHIDDENRETRY-hidden0-heal1``. + Scenario A (storm): the page connects, POSTs ``/send`` on stream-open (so the listener is registered first), the node runs a 4-parallel-bash ``seq 1 500`` storm plus a task_agent whose sub-tools are chatty bashes; @@ -271,6 +319,14 @@ SECOND_SENTINEL = "SECOND-a7f3" # everything else on screen. BACKSTOP_SENTINEL = "BACKSTOP-b2e4" +# Scenario G4 (coord-heal-midturn) sentinels: turn 4's final text (the +# settle that fires the held backstop fetch) and turn 5's final text (the +# turn that STARTS during that held fetch and must survive its resolution). +# Same collision-proof discipline as the sentinels above; turn 5's bash +# command output is ``g4-N`` lines, which contain neither token. +MIDTURN_T4_SENTINEL = "MIDT4-c9d1" +MIDTURN_T5_SENTINEL = "MIDT5-f47a" + # --------------------------------------------------------------------------- # The recovery page — served same-origin by the node at /recovery. # --------------------------------------------------------------------------- @@ -902,6 +958,70 @@ COORD_PAGE_HTML = r""" userRows; }; + // G4 — the render-time gate (#894 r4): a turn that STARTS during a + // heal's in-flight /history must SURVIVE its resolution (the gate + // skips the wipe; pre-gate code detached the live turn into a + // dangling ref and lost it). midturnSurvived is the runner's + // observation that turn 5's sentinel painted while the stale + // transcript was still up — the not-wiped proof. + window.__verifyCoordHealMidturn = function ( + healed, + midturnSurvived, + sseDelta, + histDelta, + posts, + ) { + const userRows = _coordUserRows(); + const ok = + healed && + midturnSurvived && + sseDelta === 0 && + histDelta === 2 && + posts === 2; + document.title = ok + ? "RECOVERY-READY-COORDHEALMIDTURN-heal1-mid1-sse0" + : "RECOVERY-FAILED-COORDHEALMIDTURN-heal" + + (healed ? 1 : 0) + + "-mid" + + (midturnSurvived ? 1 : 0) + + "-sse" + + sseDelta + + "-hist" + + histDelta + + "-posts" + + posts + + "-rows" + + userRows; + }; + + // G5 — the retry's stream-liveness fire guard (#894 r4): a retry + // armed before a close-on-hide must NOT fetch while the transport + // is down (hiddenDelta 0 — a seedless render past the frozen + // cursor double-renders on the show-edge replay); the show edge's + // synthetic idle hands the heal to the backstop (one user-driven + // SSE open, rows healed, gate reopened). + window.__verifyCoordHiddenRetry = function ( + hiddenDelta, + healed, + showSse, + posts, + ) { + const userRows = _coordUserRows(); + const ok = hiddenDelta === 0 && healed && showSse === 1 && posts === 2; + document.title = ok + ? "RECOVERY-READY-COORDHIDDENRETRY-hidden0-heal1" + : "RECOVERY-FAILED-COORDHIDDENRETRY-hidden" + + hiddenDelta + + "-heal" + + (healed ? 1 : 0) + + "-sse" + + showSse + + "-posts" + + posts + + "-rows" + + userRows; + }; + // G3 — the TRANSPORT-FREE idle-edge backstop. Mirrors // __verifyStaleBackstop; the storm assertion is sseDelta === 0 (the // heal opened ZERO EventSource connections — a reconnecting backstop @@ -2372,6 +2492,14 @@ def run_coord_rewind_window(chrome: str) -> str: # Wait for the initial /history to paint all three user rows. if not _poll_until(lambda: cdp.evaluate(_COORD_ROWS_JS) == 3, 20, 0.2): raise AssertionError("coord-rewind-window: three user rows never rendered") + # Rows paint from init's ``await refetchHistory(true)`` and the pane + # dials SSE only AFTERWARDS — a rewind clicked in that gap emits + # clear_ui into a channel nobody joined (a cursorless connect takes + # the fresh no-replay branch), the latch never sets, and the + # scenario false-fails. Gate on the transport wrapper's counter, + # like the coord page's coord-restart send gate. + if not _poll_until(lambda: cdp.evaluate("window.__esOpens") >= 1, 10, 0.05): + raise AssertionError("coord-rewind-window: SSE stream never opened") # Relative baseline (never assume how many /history the boot ran). hist_baseline = node.history_requests # Hold every /history 3s so the clear_ui refetch keeps the latch's @@ -2437,6 +2565,10 @@ def run_coord_rewind_failed_window(chrome: str) -> str: # load MUST succeed, so arm the forced failure only AFTERWARDS. if not _poll_until(lambda: cdp.evaluate(_COORD_ROWS_JS) == 3, 20, 0.2): raise AssertionError("coord-rewind-failed-window: three user rows never rendered") + # SSE-open gate — see run_coord_rewind_window: a pre-connect rewind + # drops its clear_ui and false-fails the scenario. + if not _poll_until(lambda: cdp.evaluate("window.__esOpens") >= 1, 10, 0.05): + raise AssertionError("coord-rewind-failed-window: SSE stream never opened") hist_baseline = node.history_requests # Arm ONE forced /history 500: the NEXT /history — the rewind's # clear_ui refetch — fails. @@ -2560,6 +2692,10 @@ def run_coord_stale_backstop(chrome: str) -> str: # load MUST succeed, so arm the forced failures only AFTERWARDS. if not _poll_until(lambda: cdp.evaluate(_COORD_ROWS_JS) == 3, 20, 0.2): raise AssertionError("coord-stale-backstop: three user rows never rendered") + # SSE-open gate — see run_coord_rewind_window: a pre-connect rewind + # drops its clear_ui and false-fails the scenario. + if not _poll_until(lambda: cdp.evaluate("window.__esOpens") >= 1, 10, 0.05): + raise AssertionError("coord-stale-backstop: SSE stream never opened") # Arm TWO forced /history 500s: the rewind's clear_ui refetch AND its # one bounded 2s retry both fail, so ONLY the organic idle-edge # backstop can clear the latch. @@ -2652,6 +2788,246 @@ def run_coord_stale_backstop(chrome: str) -> str: node.stop() +def run_coord_heal_midturn(chrome: str) -> str: + """Scenario G4 — the render-time gate (#894 r4). A turn that STARTS + while a heal's /history is in flight must SURVIVE the fetch resolution: + pre-gate code ran ``replaceChildren`` under the live turn, detaching its + bubble/tool rows into dangling refs (every remaining token rendered + invisibly, the optimistic user row was destroyed, and nothing + re-rendered the lost turn). The gate skips the wipe instead — the + latch stays set and the turn's OWN settle re-fires the backstop. + + Choreography: G3's double-failure prologue leaves the latch stuck; + ``delay_history`` then holds the backstop fetch that turn 4's settle + fires; the moment the held fetch ARRIVES (history_requests bumps before + the hold sleeps) the runner sends turn 5 — a paced bash turn (~2.5s) + that is still mid-stream when the hold (1.5s) releases. The gate must + skip that resolution (turn 5's sentinel paints into the still-stale + transcript = midturnSurvived), and turn 5's settle re-fires the + backstop, which now heals: transcript = rewound turn + turns 4 and 5, + seeds two/three gone. Storm proof: zero SSE opens across the whole + episode; exactly TWO /history fetches (the skipped one + the heal). + + Detector honesty: ``history_delta == 2`` is the DISCRIMINATING bit. + Gate-stripped code renders the held fetch early, which CLEARS the + latch, kills the backstop refire, and stamps ``hist1`` (plus + downstream posts/rows drift). The ``mid1`` bit alone cannot + discriminate here: the early render repaints all three user rows from + the snapshot (the rewind pre-dates turn 4, so /history already + carries turns 4 and 5's user rows), and turn 5's tool phase has null + content refs — its per-token loss surface is the TOOL rows, which + this scenario does not fingerprint. On gated code ``mid1`` asserts + the stronger continuous-visibility claim (nothing wiped at any + point).""" + from tests._sse_recovery_server import final_text_script, parallel_bash_script + + paced5 = parallel_bash_script({"g4": "for i in $(seq 1 50); do echo g4-$i; sleep 0.05; done"}) + node, ws_id = _seed_three_completed_turns( + "browser-coord-heal-midturn", + extra_scripts=( + final_text_script(MIDTURN_T4_SENTINEL), + paced5, + final_text_script(MIDTURN_T5_SENTINEL), + ), + ) + profile = Path(_scratch()) / "chrome-coord-heal-midturn" + proc, cdp_port = _launch_chrome(chrome, profile) + cdp: CDP | None = None + try: + cdp = CDP(_page_ws_url(cdp_port)) + url = f"{node.base_url}/coord-recovery?ws_id={ws_id}&scenario=coord-heal-midturn" + _set_cookie_and_navigate(cdp, node.base_url, node.token, url) + if not _poll_until(lambda: cdp.evaluate(_COORD_ROWS_JS) == 3, 20, 0.2): + raise AssertionError("coord-heal-midturn: three user rows never rendered") + # SSE-open gate — see run_coord_rewind_window. + if not _poll_until(lambda: cdp.evaluate("window.__esOpens") >= 1, 10, 0.05): + raise AssertionError("coord-heal-midturn: SSE stream never opened") + # G3 prologue: latch stuck after clear_ui refetch + retry both 500. + node.fail_history(2) + if not cdp.evaluate("window.__clickCoordRewind(1)"): + raise AssertionError("coord-heal-midturn: second-row rewind button missing") + if not _poll_until(lambda: node.rewind_requests == 1, 5, 0.05): + raise AssertionError("coord-heal-midturn: first rewind never POSTed") + if not _poll_until(lambda: node.history_fail_remaining == 0, 20): + raise AssertionError("coord-heal-midturn: the two forced failures never both fired") + stale_rows = cdp.evaluate(_COORD_ROWS_JS) + if stale_rows != 3: + raise AssertionError( + f"coord-heal-midturn: failed fetches did not preserve the transcript " + f"(user rows={stale_rows}, expected 3)" + ) + events_baseline = node.events_requests + history_baseline = node.history_requests + # Hold every /history long enough for turn 5 to start under it, but + # shorter than turn 5's ~2.5s bash phase, so the held fetch resolves + # MID-turn — the exact window the gate exists for. + node.delay_history(1500) + # Turn 4: settles -> idle edge -> backstop fires -> its fetch is + # HELD. history_requests bumps on ARRIVAL (before the hold sleeps), + # which is the observable that the window is open. + _send_in_page(cdp, "fourth turn") + if not _poll_until(lambda: node.history_requests == history_baseline + 1, 20, 0.05): + raise AssertionError("coord-heal-midturn: backstop fetch never arrived") + # Turn 5, INSIDE the hold: paced bash keeps it mid-stream when the + # held fetch resolves. The gate must skip that render. + _send_in_page(cdp, "fifth turn") + # midturnSurvived: turn 5's final sentinel paints into the + # still-stale transcript (user rows unchanged at 3) — a wiped pane + # would have dropped to 1 row and swallowed the sentinel into a + # detached ref. Poll spans the hold release (+1.5s) and turn 5's + # full run. + midturn_survived = _poll_until( + lambda: cdp.evaluate( + "(document.getElementById('coord-messages').textContent||'')" + ".includes(" + json.dumps(MIDTURN_T5_SENTINEL) + ") && " + _COORD_ROWS_JS + " === 3" + ), + 20, + 0.2, + ) + # Release the hold so turn 5's settle-driven backstop refire heals + # promptly. + node.delay_history(0) + # HEAL: the refire renders the rewound truth — turn "first" + turns + # 4 and 5; the rewound-away seeds are GONE. Text discriminators, + # not row counts: the healed pane also has 3 user rows. + healed = _poll_until( + lambda: cdp.evaluate( + "(function(){var t=document.getElementById('coord-messages')" + ".textContent||'';return t.includes(" + + json.dumps(MIDTURN_T4_SENTINEL) + + ")&&t.includes(" + + json.dumps(MIDTURN_T5_SENTINEL) + + ")&&!t.includes('second');})()" + ), + 20, + 0.2, + ) + events_delta = node.events_requests - events_baseline + history_delta = node.history_requests - history_baseline + # REOPEN: the heal cleared the latch; a fresh rewind lands. + if healed: + if not cdp.evaluate("window.__clickCoordRewind(0)"): + raise AssertionError("coord-heal-midturn: healed-row rewind button missing") + _poll_until(lambda: node.rewind_requests == 2, 8, 0.05) + posts = node.rewind_requests + print( + f" coord-heal-midturn midturn_survived={midturn_survived} healed={healed} " + f"events_delta={events_delta} history_delta={history_delta} posts={posts}" + ) + cdp.evaluate( + "window.__verifyCoordHealMidturn(" + f"{'true' if healed else 'false'}, " + f"{'true' if midturn_survived else 'false'}, " + f"{events_delta}, {history_delta}, {posts})" + ) + return _poll_title(cdp, 15) + finally: + if cdp is not None: + cdp.close() + _kill(proc) + node.stop() + + +def run_coord_hidden_retry(chrome: str) -> str: + """Scenario G5 — the retry's stream-liveness fire guard (#894 r4). The + 2s retry deliberately survives close-on-hide (transport redials keep + heal intent), so it can FIRE while the tab is hidden and the transport + is down. A seedless fetch then would render the pane past the frozen + ``lastEventId``; the show-edge reconnect replays from that frozen + cursor and double-renders every turn the hidden render already painted. + The ``evtSource`` fire-guard term skips the hidden firing instead + (hiddenDelta 0); on show, the reconnect's synthetic idle re-fires the + TRANSPORT-FREE backstop, which heals on the live stream. + + A quiet reconnect delivers NO state_change edge, so the latch stays + closed after __show until the next ORGANIC settle — exactly the + accepted-residual ruling (heals ride organic edges; no timer may + shortcut the lag). The runner drives that settle with a plain send + (sends are never latch-gated), whose turn-settle idle edge fires the + TRANSPORT-FREE backstop on the live stream. + + Proofs: history_requests UNCHANGED across the hidden retry window (the + non-occurrence detector that regresses to hidden1 without the guard); + exactly ONE new SSE open across show + heal (the user-driven reconnect + — the heal itself adds zero); the healed render carries the rewound + turn + the sent turn (TWO user rows, sentinel present); the reopen + click lands.""" + from tests._sse_recovery_server import final_text_script + + node, ws_id = _seed_three_completed_turns( + "browser-coord-hidden-retry", + extra_scripts=(final_text_script(BACKSTOP_SENTINEL),), + ) + profile = Path(_scratch()) / "chrome-coord-hidden-retry" + proc, cdp_port = _launch_chrome(chrome, profile) + cdp: CDP | None = None + try: + cdp = CDP(_page_ws_url(cdp_port)) + url = f"{node.base_url}/coord-recovery?ws_id={ws_id}&scenario=coord-hidden-retry" + _set_cookie_and_navigate(cdp, node.base_url, node.token, url) + if not _poll_until(lambda: cdp.evaluate(_COORD_ROWS_JS) == 3, 20, 0.2): + raise AssertionError("coord-hidden-retry: three user rows never rendered") + # SSE-open gate — see run_coord_rewind_window. + if not _poll_until(lambda: cdp.evaluate("window.__esOpens") >= 1, 10, 0.05): + raise AssertionError("coord-hidden-retry: SSE stream never opened") + node.fail_history(1) + if not cdp.evaluate("window.__clickCoordRewind(1)"): + raise AssertionError("coord-hidden-retry: second-row rewind button missing") + if not _poll_until(lambda: node.rewind_requests == 1, 5, 0.05): + raise AssertionError("coord-hidden-retry: first rewind never POSTed") + if not _poll_until(lambda: node.history_fail_remaining == 0, 15): + raise AssertionError("coord-hidden-retry: forced /history failure never fired") + # Hide IMMEDIATELY — well inside the 2s arm window (the poll above + # settles ~0.3s after the failure). close-on-hide tears the + # transport down but deliberately leaves the retry timer armed. + cdp.evaluate("window.__hide && window.__hide()") + hidden_baseline = node.history_requests + # NON-occurrence window: the retry fires at ~2s post-failure; give + # it 3.5s. Without the evtSource guard this poll returns True + # (the hidden fetch lands) and hiddenDelta stamps 1. + _poll_until(lambda: node.history_requests != hidden_baseline, 3.5, 0.1) + hidden_delta = node.history_requests - hidden_baseline + # Show: the reconnect presents the frozen cursor. A quiet + # reconnect delivers NO state_change edge (the latch-closed lag is + # the accepted residual), so wait for the reconnect itself, then + # drive an ORGANIC settle with a plain send — its idle edge fires + # the TRANSPORT-FREE backstop on the live stream and the heal + # renders the rewound (ONE) + sent (a second) transcript. + events_before_show = node.events_requests + cdp.evaluate("window.__show && window.__show()") + if not _poll_until(lambda: node.events_requests == events_before_show + 1, 10, 0.05): + raise AssertionError("coord-hidden-retry: show-edge reconnect never arrived") + _send_in_page(cdp, "fourth turn") + healed = _poll_until( + lambda: cdp.evaluate( + _COORD_ROWS_JS + " === 2 && (document.getElementById('coord-messages')" + ".textContent||'').includes(" + json.dumps(BACKSTOP_SENTINEL) + ")" + ), + 20, + 0.2, + ) + show_sse = node.events_requests - events_before_show + if healed: + if not cdp.evaluate("window.__clickCoordRewind(0)"): + raise AssertionError("coord-hidden-retry: healed-row rewind button missing") + _poll_until(lambda: node.rewind_requests == 2, 8, 0.05) + posts = node.rewind_requests + print( + f" coord-hidden-retry hidden_delta={hidden_delta} healed={healed} " + f"show_sse={show_sse} posts={posts}" + ) + cdp.evaluate( + f"window.__verifyCoordHiddenRetry({hidden_delta}, " + f"{'true' if healed else 'false'}, {show_sse}, {posts})" + ) + return _poll_title(cdp, 15) + finally: + if cdp is not None: + cdp.close() + _kill(proc) + node.stop() + + def _wait_state(node: Any, ws_id: str, state: str, timeout: float) -> None: deadline = time.monotonic() + timeout while time.monotonic() < deadline: @@ -2700,6 +3076,8 @@ def main() -> None: "coord-rewind-window", "coord-rewind-failed-window", "coord-stale-backstop", + "coord-heal-midturn", + "coord-hidden-retry", "roster-restart", "roster-restart-native", "both", @@ -2765,6 +3143,14 @@ def main() -> None: verdict = run_coord_stale_backstop(chrome) print(f"scenario G3 (coord-stalebackstop): {verdict}") failures += 0 if verdict.startswith("RECOVERY-READY") else 1 + if args.scenario in ("coord-heal-midturn", "all"): + verdict = run_coord_heal_midturn(chrome) + print(f"scenario G4 (coord-healmidturn): {verdict}") + failures += 0 if verdict.startswith("RECOVERY-READY") else 1 + if args.scenario in ("coord-hidden-retry", "all"): + verdict = run_coord_hidden_retry(chrome) + print(f"scenario G5 (coord-hiddenretry): {verdict}") + failures += 0 if verdict.startswith("RECOVERY-READY") else 1 if args.scenario in ("roster-restart", "all"): verdict = run_roster_restart(chrome) print(f"scenario F1 (roster-manual): {verdict}") diff --git a/tests/test_app_js.py b/tests/test_app_js.py index 7aa656bc..6b0aeb6b 100644 --- a/tests/test_app_js.py +++ b/tests/test_app_js.py @@ -2093,9 +2093,11 @@ def test_coord_truncated_resync_is_full_fresh_connect_with_churn_limit() -> None body, ), "connectSSE must gate ?last_event_id= on connectCursor != null" # (2) cleared only by a successful full render — below the !hist guard, - # riding the wipe — and never by the teardown paths. + # riding the wipe — and never by the teardown paths. (Window sized + # past the #894 latch/render-gate comment blocks; the invariant is the + # ORDER, not the density.) start = body.index("async function refetchHistory(seedCursor = false)") - fn = body[start : start + 4500] + fn = body[start : start + 8000] guard = fn.index("if (!hist) return;") clear = fn.index("truncatedFromCursor = null;") assert guard < clear, "the record must only clear once a payload rendered" diff --git a/tests/test_coordinator_page.py b/tests/test_coordinator_page.py index d1c1a971..1c904afc 100644 --- a/tests/test_coordinator_page.py +++ b/tests/test_coordinator_page.py @@ -558,8 +558,9 @@ def test_coordinator_refetch_failure_preserves_the_pane(): body = coord_js.read_text(encoding="utf-8") start = body.index("async function refetchHistory(seedCursor = false)") # Window sized to reach the early-reset block past the #894 latch-clear - # comments; the pinned invariant is the ORDER below, not the density. - fn = body[start : start + 5000] + # and render-gate comments; the pinned invariant is the ORDER below, + # not the density. + fn = body[start : start + 8000] guard = fn.index("if (!hist) return;") wipe = fn.index("messagesEl.replaceChildren();") resets = fn.index("toolRows.clear();") @@ -651,9 +652,16 @@ def test_coordinator_history_stale_latch_contract(): ) # 4. Transport-free backstop: the arm after the truncated consumer - # refetches over REST and never reconnects. + # refetches over REST and never reconnects — and it must be the + # ELSE-IF of the truncated consumer (mutual exclusion: the + # truncated branch's own reload heals the latch too; two separate + # ifs would run both heals on one idle edge). trunc_arm = body.index("if (pendingTruncatedResync)") backstop = body.index("historyStale &&", trunc_arm) + assert "} else if (" in body[trunc_arm:backstop], ( + "the staleness backstop must be the else-if sibling of the " + "pendingTruncatedResync consumer, never an independent if." + ) idle_block_end = body.index('ev.state === "running"', backstop) backstop_arm = body[backstop:idle_block_end] assert "refetchHistory();" in backstop_arm, ( @@ -669,6 +677,12 @@ def test_coordinator_history_stale_latch_contract(): "guard it stomps a same-snapshot fetch with a double render " "(mirrors interactive's !_replayQueue pin)." ) + assert "!currentAssistantEl" in backstop_arm and "!currentReasoningEl" in backstop_arm, ( + "the backstop's ref guards are LOAD-BEARING (coord's " + "refetchHistory does not reset streaming refs, and this arm " + "serves error edges where no stream_end nulled them) — a " + "simplify-to-match-interactive edit must not delete them." + ) # 5. Bounded retry: exactly one arm site; the ARM is teardown-gated; # the fire guard yields to an in-flight fetch and carries the @@ -694,13 +708,28 @@ def test_coordinator_history_stale_latch_contract(): "the SOLE protection for the coordCloseSession path, which nulls " "visHandler but does not cancel the timer." ) + assert "!currentAssistantEl" in retry_fire and "!currentReasoningEl" in retry_fire, ( + "the retry's ref guards are LOAD-BEARING (coord's refetchHistory " + "does not reset streaming refs) — must not be deleted to match " + "interactive's quiesce-protected shape." + ) + assert "visHandler &&\n evtSource\n ) {" in body, ( + "the retry's fire guard must require a live stream (evtSource): " + "close-on-hide keeps this timer armed by design, and a seedless " + "heal on a dead stream renders past the frozen cursor (replay " + "double-render on the show edge). Exact-tail pin so a comment " + "mention cannot satisfy it; re-anchor if the guard reflows." + ) assert "if (staleRetryTimer) clearTimeout(staleRetryTimer);" in body, ( "re-arming on a newer clear_ui must cancel the pending timer " "first, or a double clear_ui leaks a timer." ) # The consumer pins above are only meaningful while the PRODUCER # brackets every fetch: without the ++/-- pair the counter is - # permanently 0 and both yield guards pass vacuously. + # permanently 0 and both yield guards pass vacuously. Count alone + # is not enough — the ORDER is the bracket (an increment moved below + # the await leaves the counter 0 during every fetch with both counts + # intact), so pin inc < await < finally < dec inside refetchHistory. assert body.count("refetchesInFlight++") == 1, ( "refetchHistory must increment the in-flight counter before its " "await — the yield guards read it." @@ -709,6 +738,33 @@ def test_coordinator_history_stale_latch_contract(): "the in-flight counter must decrement in exactly one place (the " "fetch finally) so every exit rebalances it." ) + inc = body.index("refetchesInFlight++", fetch_start) + awt = body.index("await getJSON(", fetch_start) + fin = body.index("} finally {", fetch_start) + dec = body.index("refetchesInFlight--", fetch_start) + assert inc < awt < fin < dec, ( + "the counter must bracket the await window: increment BEFORE the " + "fetch, decrement in its finally — any other order un-brackets " + "the very window the yield guards protect." + ) + + # 7. Render-time gate: the post-await re-checks must sit between the + # failure guard and the wipe. The await is a real window — a turn + # can START mid-fetch (refs re-check; a wipe then would detach the + # live bubble and lose the turn into the dangling ref), and a + # seedless render must not paint past a frozen cursor when the + # stream died mid-fetch (hide/suspend) or the show-edge replay + # double-renders. Caller-side guards cannot see across the await; + # only this chokepoint can. + hist_guard = body.index("if (!hist) return;", fetch_start) + ref_gate = body.index("if (currentAssistantEl || currentReasoningEl) return;", fetch_start) + live_gate = body.index("if (!seedCursor && (busy || !evtSource)) return;", fetch_start) + wipe = body.index("messagesEl.replaceChildren();", fetch_start) + assert hist_guard < ref_gate < wipe and hist_guard < live_gate < wipe, ( + "refetchHistory must re-check the live-turn refs AND seedless " + "stream-liveness AFTER the await and BEFORE the wipe — the " + "render-time correctness carrier for every caller." + ) # 6. Teardown: terminal cancel in destroy(); NOT in closeStreamTransport. destroy_slice = body[body.index("function destroy()") :] diff --git a/turnstone/console/static/coordinator/coordinator.js b/turnstone/console/static/coordinator/coordinator.js index 596e3a7e..d26084b2 100644 --- a/turnstone/console/static/coordinator/coordinator.js +++ b/turnstone/console/static/coordinator/coordinator.js @@ -3702,8 +3702,18 @@ function createCoordinatorPane(root, wsId, opts) { !busy && !currentAssistantEl && !currentReasoningEl && - visHandler + visHandler && + evtSource ) { + // evtSource: a seedless heal must not render past a + // frozen cursor (close-on-hide keeps this timer armed by + // design, so the fire can land with the transport down) + // — skip; the latch survives and the show-edge + // reconnect's synthetic idle hands the heal to the + // backstop. The backstop itself needs no such term: it + // runs inside SSE dispatch, so its stream is live by + // construction. refetchHistory's render-time gate + // re-checks both invariants across the await window. // Fire-and-forget, seedless (live stream — lastEventId // must not rewind); a render throw stays loud, as on the // backstop. @@ -5765,6 +5775,41 @@ function createCoordinatorPane(root, wsId, opts) { // refetch.) Success ordering is unchanged: the wipe always ran // after the await, never as immediate feedback. if (!hist) return; + // RENDER-TIME gate (#894 r4): the await above is a real window — pane + // state can change between a caller's fire-time checks and this + // render, and only THIS site can see across it (chokepoint, not + // per-caller guards). Two invariants must hold at the wipe itself: + // + // 1. No live turn mid-stream. A turn can START during the fetch (the + // server drains queued sends at exactly the idle edges the + // backstop rides; another operator on a shared coordinator can + // send any time; coordSend paints an optimistic user row). This + // render does not reset the streaming refs, so replaceChildren + // would detach the live bubble — every remaining token renders + // into the dangling ref (invisible), and the optimistic user row + // is destroyed with nothing to repaint it. Skip instead: if the + // staleness latch is set it stays set (clear is below), the gate + // stays closed, and the turn's own settle re-fires the backstop. + // 2. Seedless renders need an IDLE pane on a LIVE stream. busy: the + // ref check above only sees the CONTENT phase — a turn in its tool + // phase has null content refs but live tool rows, and the wipe + + // toolRows.clear() below would orphan them mid-stream identically. + // evtSource: a seedless render must not advance the DOM past a + // frozen lastEventId (hide/suspend can land mid-fetch) — the + // show-edge reconnect replays from the frozen cursor and every + // turn this render already painted would render twice. Seeded + // callers (init, loadHistoryThenReconnect) own their reconnect + // flow — they adopt hist.cursor below, and the truncated resync + // legitimately rebuilds MID-turn — so both requirements key on the + // seedCursor ARG, not caller identity. On skip the latch + // survives; the next organic settle (or the show-edge reconnect's + // synthetic idle) re-fires the backstop. + // + // The callers' fire-time ref/stream guards remain as the efficiency + // layer (skip the pointless fetch); THIS gate is the correctness + // carrier. + if (currentAssistantEl || currentReasoningEl) return; + if (!seedCursor && (busy || !evtSource)) return; messagesEl.replaceChildren(); // A full committed-history render repairs any recorded truncation gap — // whether this render came from the truncated resync itself or from an