fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds

Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed).
2026-08-27 14:24:47 -06:00 · 2026-07-07 07:49:14 -07:00
parent bbe92faca1
commit e60c19befd
23 changed files with 2977 additions and 198 deletions
@@ -2,7 +2,7 @@

 The shared worker dispatch is load-bearing for both the interactive
 ``/v1/api/workstreams/{ws_id}/send`` HTTP handler and the coordinator
-``CoordinatorAdapter.send`` path. Tests cover the four invariants the
+``CoordinatorAdapter.send`` path. Tests cover the five invariants the
 module must hold:

 * live worker → enqueue, no thread spawn
@@ -10,10 +10,14 @@ module must hold:
 * concurrent ``send`` calls produce exactly one worker thread
  (Stage 1 bug-1 — the racy ``Thread.is_alive()`` gate stays caught)
 * ``_worker_running`` cleared in ``finally`` even on uncaught exception
+* ownership-clear wake backstop: a worker exiting with USER_DRAIN
+  nudges queued on an IDLE workstream spawns the wake send that the
+  IDLE fan-out (which ran on this worker's own thread) had to drop

-Callers pass no-arg closures, so this module never touches
-``ws.session`` — keeps the contract narrow and lets watch-style
-dispatchers drive a session that isn't installed on ``ws``.
+Callers pass no-arg closures, so dispatch never touches ``ws.session``;
+the exit backstop only PEEKS it defensively (``getattr`` for
+``_nudge_queue``, bail on stubs) — watch-style dispatchers can still
+drive a session that isn't installed on ``ws``.
 """

 from __future__ import annotations
@@ -22,8 +26,10 @@ import queue
 import threading
 from typing import Any

+from tests._helpers import wait_until as _wait_until
 from turnstone.core import session_worker
-from turnstone.core.workstream import Workstream
+from turnstone.core.nudge_queue import USER_DRAIN, NudgeQueue
+from turnstone.core.workstream import Workstream, WorkstreamState


 class _SendSession:
@@ -140,6 +146,41 @@ def test_enqueue_unexpected_exception_returns_false_logged() -> None:
    assert ws._worker_running is True


+def test_closed_workstream_refused_no_spawn() -> None:
+    """Authoritative closed-check: ``close()`` sets ``_closed`` under
+    ``ws._lock``, so a wake (or send) racing it must be refused HERE —
+    the wake gate's lockless peek can go stale, and a spawn past this
+    point would run a full unattended turn (inference, tool calls,
+    storage writes) on a workstream whose ``ws_closed`` already fired.
+    """
+    session = _SendSession()
+    ws = _make_ws(session)
+    ws._closed = True
+
+    ok = _send_message(ws, session, "hello")
+
+    assert ok is False
+    assert session.send_calls == []
+    assert session.queue_calls == []
+    assert ws.worker_thread is None
+    assert ws._worker_running is False
+
+
+def test_closed_workstream_refused_on_reuse_path_too() -> None:
+    """The refusal precedes the enqueue branch: no interjection is queued
+    onto a session whose workstream is already closed."""
+    session = _SendSession()
+    ws = _make_ws(session)
+    ws._worker_running = True
+    ws._closed = True
+
+    ok = _send_message(ws, session, "hello")
+
+    assert ok is False
+    assert session.queue_calls == []
+    assert ws._worker_running is True  # untouched — not ours to clear
+
+
 # ---------------------------------------------------------------------------
 # _worker_running lifecycle
 # ---------------------------------------------------------------------------
@@ -301,6 +342,141 @@ def test_thread_name_explicit_override() -> None:
    ws.worker_thread.join(timeout=2.0)


+class _WakeCapableSession(_SendSession):
+    """Adds the ChatSession surface the exit backstop peeks at."""
+
+    def __init__(self, **kwargs: Any) -> None:
+        super().__init__(**kwargs)
+        self._nudge_queue = NudgeQueue()
+        self.deliver_calls = 0
+        self.deliver_thread_names: list[str] = []
+        self.delivered = threading.Event()
+
+    def deliver_wake_nudge_from_queue(self) -> None:
+        # Mirror the real contract: the wake drains its own queue, so
+        # the wake worker's OWN exit backstop sees nothing pending and
+        # the chain converges instead of spawning wakes forever.
+        self.deliver_calls += 1
+        self.deliver_thread_names.append(threading.current_thread().name)
+        self._nudge_queue.drain(USER_DRAIN)
+        self.delivered.set()
+
+
+class TestWorkerExitWakeBackstop:
+    """A worker exiting while its (idle) workstream has USER_DRAIN
+    nudges queued spawns the wake send the IDLE fan-out had to drop.
+
+    Production shape being modelled: ``set_state(IDLE)`` fires its
+    subscribers on the worker thread from inside ``run()`` —
+    ``CoordinatorIdleObserver`` enqueues ``idle_children``, then
+    ``IdleNudgeWatcher``'s wake dispatch lands on the reuse path
+    (this very worker still owns the flag) and no-ops.  The enqueue
+    inside ``run`` below stands in for that observer enqueue.
+    """
+
+    def test_worker_exit_delivers_pending_wake(self) -> None:
+        session = _WakeCapableSession()
+        ws = _make_ws(session)
+        assert ws.state is WorkstreamState.IDLE  # dataclass default
+
+        def run() -> None:
+            # What the IDLE fan-out's observer does, on this thread.
+            session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
+
+        ok = session_worker.send(ws, enqueue=lambda: None, run=run)
+        assert ok is True
+
+        # The wake is delivered on a fresh wake-named worker thread…
+        assert session.delivered.wait(timeout=2.0), (
+            "exit backstop did not deliver the pending nudge"
+        )
+        assert session.deliver_thread_names[0].startswith("wake-nudge-")
+        # …after which the wake worker's own exit backstop sees an empty
+        # queue and the chain converges: flag at rest, exactly one deliver.
+        _wait_until(lambda: ws._worker_running is False)
+        assert session.deliver_calls == 1
+        assert len(session._nudge_queue) == 0
+
+    def test_worker_exit_no_wake_when_queue_empty(self) -> None:
+        session = _WakeCapableSession()
+        ws = _make_ws(session)
+
+        ok = session_worker.send(ws, enqueue=lambda: None, run=lambda: None)
+        assert ok is True
+        original = ws.worker_thread
+        assert original is not None
+        original.join(timeout=2.0)
+
+        assert ws.worker_thread is original  # no wake spawned
+        assert session.deliver_calls == 0
+        assert ws._worker_running is False
+
+    def test_worker_exit_no_wake_for_stub_session_without_queue(self) -> None:
+        """The narrow-contract escape hatch: a session without a
+        ``_nudge_queue`` (watch-style stubs) is skipped by the shared
+        wake gate's own defensive peek — no AttributeError, no wake."""
+        session = _SendSession()
+        ws = _make_ws(session)
+
+        ok = _send_message(ws, session, "hello")
+        assert ok is True
+        original = ws.worker_thread
+        assert original is not None
+        original.join(timeout=2.0)
+
+        assert ws.worker_thread is original
+        assert ws._worker_running is False
+
+    def test_worker_exit_no_wake_when_state_not_idle(self) -> None:
+        """An ERROR exit stays parked for the operator — pending nudges
+        wait for the next real interaction rather than burning
+        unattended inference on a failed session."""
+        session = _WakeCapableSession()
+        ws = _make_ws(session)
+
+        def run() -> None:
+            session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
+            ws.state = WorkstreamState.ERROR
+
+        ok = session_worker.send(ws, enqueue=lambda: None, run=run)
+        assert ok is True
+        original = ws.worker_thread
+        assert original is not None
+        original.join(timeout=2.0)
+
+        assert ws.worker_thread is original
+        assert session.deliver_calls == 0
+        assert len(session._nudge_queue) == 1  # still queued for later seams
+
+    def test_abandoned_worker_does_not_run_wake_backstop(self) -> None:
+        """Only the owner retries: an abandoned worker (successor claimed
+        the flag) finishing late must not spawn a wake — the successor's
+        own exit runs the backstop."""
+        send_gate = threading.Event()
+        session = _WakeCapableSession(send_gate=send_gate)
+        ws = _make_ws(session)
+
+        ok = _send_message(ws, session, "hello")
+        assert ok is True
+        abandoned = ws.worker_thread
+        assert abandoned is not None
+
+        session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
+        sentinel = threading.Thread(target=lambda: None, name="successor")
+        with ws._lock:
+            ws.worker_thread = sentinel
+            ws._worker_running = True
+
+        send_gate.set()
+        abandoned.join(timeout=3.0)
+        assert not abandoned.is_alive()
+
+        # No wake spawned by the abandoned thread; ownership intact.
+        assert ws.worker_thread is sentinel
+        assert session.deliver_calls == 0
+        assert ws._worker_running is True
+
+
 def test_does_not_deadlock_when_run_briefly_grabs_ws_lock() -> None:
    """Sanity check: ``run`` is invoked OUTSIDE ``ws._lock``. A worker
    body that briefly takes the lock (e.g. to update worker state)