fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds

Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
  denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
  already-idle workstreams (no IDLE transition for the watcher to
  observe), wired as wake_fn at every set_watch_runner site via the
  shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
  — after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
  ownership clears: IDLE fans out on the worker thread, so
  transition-time wakes always landed on the reuse path and no-op'd
  (the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
  the wake worker's run() closure and only Exception is caught
  downstream.

Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
  redelivered on min(interval, 60s) without re-running the command,
  bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
  max_polls across cycles; the poll charge commits durably at hold
  time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
  cap, presence-only re-check under the lock, detection-only stall
  alerts (reclaiming a wedged admission would trade capped degradation
  for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
  genuinely-missing history (confirmed by a raising storage probe —
  the resume loader swallows read blips into []) deactivate the watch
  immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
  stashing/dispatching, cancel paths write the row BEFORE
  forget_terminal_dispatched, the HTTP cancel endpoint clears runner
  state, and a per-tick sweep bounds the residual stash-after-clear
  interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
  read but not write retries the row write instead of re-running the
  command every cycle; the fresh-fire unrestorable path stashes before
  its deactivation write for the same reason.

Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
  rebind now moves it: non-fork resume() and /new go through
  _follow_watch_registration (new key live before the old is removed,
  never stealing a registration another live session holds), removals
  are owner-checked so tearing down a watch-restore shell or a
  resumed-away session cannot unregister a live pane, the restore
  shell yields to a registration that appears mid-restore, CLI
  --resume registers after the successful resume, and both the open
  path and the detail-GET lazy rehydrate wire the registration.

Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
  teardown path (close, close_idle, evict, delete, discard) funnels
  through it — and session_worker.send re-checks under the same lock,
  so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
  message could not be delivered (queue_full / refused_closed) instead
  of reading as success; staged attachments survive for the retry;
  /send surfaces a closed workstream as 404 rather than queue_full.

Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.

Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).
This commit is contained in:
Patrick Buckley
2026-07-07 07:49:14 -07:00
parent bbe92faca1
commit e60c19befd
23 changed files with 2977 additions and 198 deletions
+1
View File
@@ -948,6 +948,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `name` | string | Auto-generated workstream name |
| `resumed` | bool | Whether a previous session was successfully resumed |
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
| `initial_message_status` | string | Present ONLY when the workstream was created but its `initial_message` could not be delivered: `"queue_full"` (a raced live worker's interjection queue was at capacity — resend via `/send`; any uploads stay staged) or `"refused_closed"` (the workstream was closed mid-create). Absent whenever the message was dispatched. |
**Error (limit reached):**
+3 -3
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Console API",
"version": "1.7.0a6",
"version": "1.7.0rc1",
"description": "Cluster-wide visibility and control across all turnstone nodes."
},
"paths": {
@@ -6688,7 +6688,7 @@
"tags": [
"Coordinator"
],
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). A workstream attached to a *private* project stays confidential to its members: a permitted caller who isn't its owner / creator / project member gets a 404 (same masking as an unknown id). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
"parameters": [
{
"name": "ws_id",
@@ -13361,7 +13361,7 @@
"type": "object"
},
"PendingApprovalItem": {
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"properties": {
"call_id": {
"default": "",
+15 -2
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "1.7.0a6",
"version": "1.7.0rc1",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -2564,6 +2564,19 @@
},
"title": "Attachment Ids",
"type": "array"
},
"initial_message_status": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Present ONLY when the workstream was created but its initial_message could not be delivered: 'queue_full' (a raced live worker's interjection queue was at capacity \u2014 resend via /send; any uploads stay staged) or 'refused_closed' (the workstream was closed mid-create). Absent whenever the message was dispatched.",
"title": "Initial Message Status"
}
},
"required": [
@@ -2747,7 +2760,7 @@
"type": "object"
},
"PendingApprovalItem": {
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
"properties": {
"call_id": {
"default": "",
+7
View File
@@ -164,6 +164,13 @@ export interface CreateWorkstreamResponse {
message_count?: number;
/** Ids of attachments saved by this request (multipart variant only). */
attachment_ids?: string[];
/**
* Present ONLY when the workstream was created but its initial_message
* could not be delivered: "queue_full" (raced live worker's interjection
* queue at capacity — resend via /send; uploads stay staged) or
* "refused_closed" (workstream closed mid-create).
*/
initial_message_status?: "queue_full" | "refused_closed";
}
export interface CloseWorkstreamRequest {
+29 -1
View File
@@ -3,9 +3,37 @@ not fixtures, and several test files want to import them directly."""
from __future__ import annotations
from typing import Any
import time
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock
if TYPE_CHECKING:
from collections.abc import Callable
def wait_until(cond: Callable[[], bool], timeout: float = 5.0) -> None:
"""Poll ``cond`` to True within ``timeout`` or fail the test.
The worker/wake tests can't join threads by identity:
``session_worker.send`` assigns ``ws.worker_thread`` under the lock
BEFORE ``t.start()``, so the instant a dispatching call returns, a
fast worker may already have run its exit backstop and installed the
(not-yet-started) wake thread joining whatever ``ws.worker_thread``
points at races ``RuntimeError: cannot join thread before it is
started``. Poll outcomes instead.
"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if cond():
return
time.sleep(0.005)
if cond():
# Final re-check: the condition can become true during the last
# sleep (or a CI descheduling stall past the deadline) — failing
# without re-looking makes the helper itself a flake source.
return
raise AssertionError("condition not met within timeout")
def make_chat_session(**overrides: Any) -> Any:
"""Build a minimal ``ChatSession`` with sane test defaults.
+18
View File
@@ -245,6 +245,24 @@ def test_cleanup_ui_tolerates_missing_session_and_ui() -> None:
ws.session = None
ws.ui = None
adapter.cleanup_ui(ws) # no crash
assert ws._closed is True # still marked dead
def test_cleanup_ui_marks_workstream_closed() -> None:
"""Every teardown path — close, close_idle, EVICTION, delete,
discard funnels through cleanup_ui, which marks the object dead
under ``ws._lock`` BEFORE the teardown body runs. The wake paths
that hold OBJECT references (the watch ``wake_fn``,
``session_worker``'s exit backstop) gate on ``_closed``, and
``session_worker.send`` re-checks it under the same lock without
this write here, a wake racing an eviction or delete (which never
set the flag) would spawn a full unattended turn on the torn-down
session."""
adapter, _ = _make_adapter()
ws = _make_ws()
assert ws._closed is False
adapter.cleanup_ui(ws)
assert ws._closed is True
# ---------------------------------------------------------------------------
+186 -1
View File
@@ -28,8 +28,10 @@ from unittest.mock import MagicMock, patch
import pytest
from tests._helpers import wait_until as _wait_until
from tests.test_session_manager import FakeStorage
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core import session_worker
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
from turnstone.core.session import ChatSession
from turnstone.core.session_manager import SessionManager
from turnstone.core.trajectory import dicts_from_turns, turn_from_dict
@@ -299,6 +301,72 @@ def test_idle_event_with_empty_queue_does_not_dispatch_wake(real_mgr, tmp_db):
watcher.shutdown()
def test_watch_fire_on_already_idle_session_drives_wake_send(real_mgr, tmp_db):
"""A watch firing on an ALREADY-idle workstream sees no IDLE
transition, so :class:`IdleNudgeWatcher` never re-checks the queue
the dispatch closure's ``wake_fn`` must drive the wake itself.
Boundary path under test (only the LLM stream is patched):
dispatch closure (real, built by ``set_watch_runner``)
NudgeQueue.enqueue (real)
wake_fn wake_workstream_if_pending (real)
session_worker.send (real) daemon thread
ChatSession.deliver_wake_nudge_from_queue (real)
ChatSession.send("") watch_triggered system turn in history
"""
mgr, _adapter = real_mgr
ws = mgr.create(user_id="u1", name="watch-wake-int", skill=None)
assert ws.session is not None
captured: dict[str, Any] = {}
class _StubRunner:
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
captured["fn"] = fn
# Production wiring shape (server.py): wake_fn closes over the
# Workstream OBJECT — not its id — so eviction+restore id drift
# can't strand the wake.
ws.session.set_watch_runner(
_StubRunner(), wake_fn=lambda: wake_workstream_if_pending(ws, trigger="watch-fire")
)
with (
patch.object(ws.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
ws.session,
"_stream_response",
return_value={"role": "assistant", "content": "ok"},
),
patch.object(ws.session, "_update_token_table"),
patch.object(ws.session, "_print_status_line"),
patch.object(ws.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
ws.session._title_generated = True
# Idle all along — no worker, and no state transition coming.
assert ws.state is WorkstreamState.IDLE
# Simulate the WatchRunner poll thread delivering a fire.
captured["fn"]({"type": "watch_triggered", "text": "deploy finished: OK"}, "watch-1")
_wait_for_worker_done(ws)
# Queue drained by the wake — not parked until the next user message.
assert len(ws.session._nudge_queue) == 0
msgs = dicts_from_turns(ws.session.messages)
user_msgs = [m for m in msgs if m.get("role") == "user"]
assert user_msgs, "expected a synthesized user message from the wake"
assert user_msgs[-1]["content"] == ""
assert user_msgs[-1].get("_source") == "system_nudge"
sys_turns = [m for m in msgs if m.get("role") == "system"]
assert any(
m.get("_source") == "watch_triggered" and "deploy finished: OK" in m.get("content", "")
for m in sys_turns
), f"expected a watch_triggered system turn, got {sys_turns!r}"
@pytest.fixture
def coord_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter, FakeStorage]:
"""Real coord-side SessionManager with the adapter's kind set to
@@ -411,3 +479,120 @@ def test_coord_idle_with_active_children_emits_envelope_via_real_managers(coord_
finally:
watcher.shutdown()
observer.shutdown()
def test_coord_idle_emitted_from_worker_thread_still_wakes(coord_mgr, tmp_db):
"""The production-shaped race the test above does NOT exercise: in
production, IDLE is emitted from INSIDE the worker (``set_state``
subscribers fire on the calling thread the coord's send emits IDLE
before its worker exits). The watcher's wake dispatch therefore
lands on ``session_worker.send``'s reuse path while the
transitioning worker still owns the flag, and no-ops. Without the
ownership-clear backstop the ``idle_children`` nudge strands until
the next user message a coord that forgot ``wait_for_workstream``
never revives.
Boundary path under test:
worker thread: mgr.set_state(IDLE)
observer enqueues (real) watcher wake no-ops (worker owns flag)
run() returns session_worker._runner finally clears the flag
_retry_pending_wake wake_workstream_if_pending (real)
wake daemon deliver_wake_nudge_from_queue send("")
idle_children system turn in history
"""
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.workstream import WorkstreamKind as _Kind
mgr, adapter, storage = coord_mgr
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
coord = mgr.create(user_id="u1", name="parent-coord-2", skill=None)
assert coord.session is not None
storage.register_workstream(
"child-x",
user_id="u1",
name="crawl-docs",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="running",
)
coord.session.messages.append(turn_from_dict({"role": "user", "content": "spawn 1"}))
coord.session.messages.append(turn_from_dict({"role": "assistant", "content": "ok"}))
with (
patch.object(coord.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
coord.session,
"_stream_response",
return_value={"role": "assistant", "content": "ack"},
),
patch.object(coord.session, "_full_messages", return_value=[]),
patch.object(coord.session, "_update_token_table"),
patch.object(coord.session, "_print_status_line"),
patch.object(coord.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
coord.session._title_generated = True
# Drive the IDLE transition from INSIDE a session_worker
# worker, as production does.
ok = session_worker.send(
coord,
enqueue=lambda: None,
run=lambda: mgr.set_state(coord.id, WorkstreamState.IDLE),
thread_name="coord-send-sim",
)
assert ok is True
# Without the backstop the queue never drains (the watcher's
# transition-time wake no-opped against the sim worker) and
# this poll times out. Queue-empty implies the wake worker's
# drain ran, so the follow-up flag poll waits for ITS exit.
_wait_until(lambda: len(coord.session._nudge_queue) == 0)
_wait_for_worker_done(coord)
# Queue drained by the wake, not waiting on the next user message.
assert len(coord.session._nudge_queue) == 0
msgs = dicts_from_turns(coord.session.messages)
user_msgs = [m for m in msgs if m.get("role") == "user"]
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
idle_turns = [
m for m in msgs if m.get("role") == "system" and m["_source"] == "idle_children"
]
assert len(idle_turns) == 1
assert "crawl-docs" in idle_turns[0]["content"]
assert "wait_for_workstream" in idle_turns[0]["content"]
finally:
watcher.shutdown()
observer.shutdown()
def test_wake_delivery_contains_generation_cancelled(tmp_db):
"""A close/force-cancel racing the wake turn raises
``GenerationCancelled`` (a BaseException) out of ``send("")`` the
wake method must contain it: it IS the wake worker's ``run()``
closure, and ``session_worker._runner`` catches only ``Exception``,
so an escape would land in ``threading.excepthook`` as stderr noise
on every close-vs-wake race."""
from tests._helpers import make_chat_session
from turnstone.core.session import GenerationCancelled
session = make_chat_session()
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
def _cancelled_send(*_a: Any, **_k: Any) -> None:
raise GenerationCancelled
session.send = _cancelled_send # type: ignore[method-assign]
session.deliver_wake_nudge_from_queue() # must not raise
assert session._wake_source_tag == ""
assert session._wake_drained_reminders is None
+117 -1
View File
@@ -9,13 +9,14 @@ module-level function to capture calls without spawning real threads.
from __future__ import annotations
import contextlib
import logging
import threading
from typing import Any
from unittest.mock import patch
import pytest
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamState
@@ -32,6 +33,7 @@ class _FakeSession:
class _FakeWorkstream:
def __init__(self, ws_id: str = "ws-test") -> None:
self.id = ws_id
self.state = WorkstreamState.IDLE
self.session: _FakeSession | None = _FakeSession()
self._lock = threading.Lock()
self._worker_running = False
@@ -163,3 +165,117 @@ class TestIdleNudgeWatcher:
watcher.start()
watcher.shutdown()
watcher.shutdown() # no error
class TestWakeWorkstreamIfPending:
"""Direct tests for the shared wake gate.
The IDLE-transition path (via the watcher) is covered above; these
pin the gates the watch dispatch closure relies on when it calls
the helper directly, with no state event involved.
"""
def test_wakes_idle_ws_with_pending_entry(self, fake_mgr_and_ws):
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
assert wake_workstream_if_pending(ws) is True
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
assert kwargs["enqueue"]() is None
kwargs["run"]()
assert ws.session.deliver_wake_nudge_from_queue_called == 1
assert kwargs["thread_name"].startswith("wake-nudge-")
def test_skips_session_none(self, fake_mgr_and_ws):
_mgr, ws = fake_mgr_and_ws
ws.session = None
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_closed_ws(self, fake_mgr_and_ws):
"""A workstream mid-``close()`` must not get a wake spawned on
its torn-down session, even while its ``state`` field still
reads IDLE (there is no CLOSED member close uses the
``_closed`` tombstone)."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
ws._closed = True
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_non_idle_states(self, fake_mgr_and_ws):
"""Busy states imply a live worker that drains at its own seams;
ERROR stays parked for the operator neither gets a wake."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with patch("turnstone.core.session_worker.send") as mock_send:
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
ws.state = state
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_skips_tool_only_entries(self, fake_mgr_and_ws):
"""Tool-channel entries belong to the next tool-result seam — a
synthetic empty user turn can't drain them, so no wake."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("tool_error", "check memories", "tool")
with patch("turnstone.core.session_worker.send") as mock_send:
assert wake_workstream_if_pending(ws) is False
assert mock_send.call_count == 0
def test_dispatched_path_logs_trigger(self, fake_mgr_and_ws, caplog):
"""A fresh spawn — ``send`` returns True without touching the
passed ``enqueue`` emits ``nudge_wake.dispatched`` tagged with
the trigger label (structlog renders the event name + ``%s``
placeholders into ``msg``; substring-match like the sibling
nudge_queue tests)."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
with (
patch("turnstone.core.session_worker.send", return_value=True) as mock_send,
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
):
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
assert mock_send.call_count == 1
dispatched = [r for r in caplog.records if "nudge_wake.dispatched" in r.getMessage()]
assert len(dispatched) == 1
assert dispatched[0].levelno == logging.INFO
assert "trigger=" in dispatched[0].getMessage()
# The reuse-path drop line must not appear on a fresh spawn.
assert not any("nudge_wake.deferred_worker_busy" in r.getMessage() for r in caplog.records)
def test_deferred_path_logs_worker_busy(self, fake_mgr_and_ws, caplog):
"""The reuse path — ``send`` invokes the passed ``enqueue`` and
returns True emits ``nudge_wake.deferred_worker_busy`` instead
of ``dispatched``. The entry stays owed to the owning worker's
exit backstop; the return value is still True."""
_mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
def _reuse_send(_ws: Any, *, enqueue: Any, run: Any, thread_name: Any) -> bool:
# Mimic a live worker owning the workstream: send routes the
# wake to the no-op enqueue rather than spawning a daemon.
enqueue()
return True
with (
patch("turnstone.core.session_worker.send", side_effect=_reuse_send) as mock_send,
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
):
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
assert mock_send.call_count == 1
deferred = [
r for r in caplog.records if "nudge_wake.deferred_worker_busy" in r.getMessage()
]
assert len(deferred) == 1
assert deferred[0].levelno == logging.INFO
assert "trigger=" in deferred[0].getMessage()
assert not any("nudge_wake.dispatched" in r.getMessage() for r in caplog.records)
@@ -459,6 +459,11 @@ class TestSendMessageAttachments:
session = MagicMock()
session._cancel_event = threading.Event()
session.queue_message = MagicMock()
# A bare Mock's auto-created ``_nudge_queue`` (truthy, has_pending
# truthy, no-op deliver) turns the worker-exit wake backstop into an
# endless respawn loop; declare this a stub session WITHOUT a queue
# so the wake gate's stub-guard bails.
session._nudge_queue = None
captured: dict = {}
def fake_send(message, attachments=None, send_id=None):
@@ -480,6 +485,7 @@ class TestSendMessageAttachments:
ws.session = session
ws.worker_thread = None
ws._worker_running = False
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return captured, session
@@ -658,6 +664,9 @@ class TestQueuedSendWithAttachments:
session = MagicMock()
session._cancel_event = threading.Event()
session.queue_message = fake_queue_message
# Stub session without a NudgeQueue — see _wire_ws for why a bare
# Mock queue would feed the exit backstop an endless wake loop.
session._nudge_queue = None
ui = MagicMock()
ui._ws_lock = threading.Lock()
@@ -675,6 +684,7 @@ class TestQueuedSendWithAttachments:
ws.session = session
ws.worker_thread = worker
ws._worker_running = True
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return captured
@@ -738,6 +748,7 @@ class TestBusyWorkerAttachments:
ws.ui = ui
ws.session = session
ws.worker_thread = worker
ws._closed = False # a bare Mock attr is truthy → send() would refuse
ws._lock = threading.RLock()
mgr.get.return_value = ws
return ws, session
+110
View File
@@ -434,6 +434,91 @@ class TestCreateMultipart:
# pane's rehydrate can't observe it as still-staged.
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is None
def test_create_raced_by_live_worker_keeps_attachments_staged(self, app_client, monkeypatch):
"""The enqueue branch (caller-supplied ws_id raced by a concurrent
/send claiming the worker first) can't deliver attachments through
the interjection seam they must REMAIN STAGED so the composer
still shows them and the user's next send delivers them, while the
message text itself rides the queue."""
from turnstone.core import session_worker
from turnstone.core.attachment_buffer import get_attachment_buffer
client, _sessions, _gq = app_client
queued: list[str] = []
def _record_queue(self, text, *a, **k):
queued.append(text)
return ("", "normal", "msg-x")
monkeypatch.setattr(_FakeSession, "queue_message", _record_queue)
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
enqueue() # a worker already owns the ws — reuse path
return True
monkeypatch.setattr(session_worker, "send", _live_worker_send)
meta = {"name": "raced", "initial_message": "look at this file"}
resp = client.post(
"/v1/api/workstreams/new",
data={"meta": json.dumps(meta)},
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
ws_id = resp.json()["ws_id"]
aid = resp.json()["attachment_ids"][0]
assert queued == ["look at this file"] # text preserved via the queue
# NOT drained: the upload stays staged, recoverable on the next send.
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is not None
# Delivered path → no dropped-message marker on the response.
assert "initial_message_status" not in resp.json()
def test_create_raced_queue_full_reports_dropped_message(self, app_client, monkeypatch):
"""``queue.Full`` on the raced enqueue path must not read as
success: it propagates out of ``_enqueue_init`` into
``session_worker.send``'s backpressure branch (→ ``False``), and
the create response carries ``initial_message_status:
"queue_full"`` instead of a bare 200 implying the first message
was delivered. Attachments stay staged for the retry."""
import queue as _queue
from turnstone.core import session_worker
from turnstone.core.attachment_buffer import get_attachment_buffer
client, _sessions, _gq = app_client
def _full_queue(self, *a, **k):
raise _queue.Full
monkeypatch.setattr(_FakeSession, "queue_message", _full_queue)
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
# Mirror the real send()'s reuse-path backpressure contract:
# queue.Full → False, never a raise to the caller.
try:
enqueue()
except _queue.Full:
return False
return True
monkeypatch.setattr(session_worker, "send", _live_worker_send)
meta = {"name": "raced-full", "initial_message": "look at this file"}
resp = client.post(
"/v1/api/workstreams/new",
data={"meta": json.dumps(meta)},
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["initial_message_status"] == "queue_full"
# Attachments untouched — the composer chips survive for the retry.
aid = body["attachment_ids"][0]
assert get_attachment_buffer().get(aid, ws_id=body["ws_id"], user_id="userA") is not None
def test_create_with_attachments_no_initial_message_keeps_staged(self, app_client):
import hashlib
@@ -527,3 +612,28 @@ class TestCreateJsonStillWorks:
assert data["ws_id"]
# New optional field, but always emitted (empty list when absent)
assert data["attachment_ids"] == []
def test_initial_message_routes_through_session_worker_send(self, app_client):
"""The initial-message worker is dispatched via
``session_worker.send`` (not an inlined ``threading.Thread``) so it
inherits the ownership-clear wake backstop. Patching the module
attribute captures the wiring without spawning a thread server.py
calls ``session_worker.send`` as a module attribute even from its
local import."""
from unittest.mock import patch
client, _sessions, _gq = app_client
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
resp = client.post(
"/v1/api/workstreams/new",
json={"name": "init-dispatch", "initial_message": "go"},
headers=_auth("userA"),
)
assert resp.status_code == 200, resp.text
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
assert kwargs["thread_name"].startswith("ws-init-")
# ``run`` is the init closure the shared dispatcher spawns; the
# dead-by-construction ``enqueue`` branch is still wired (loudly).
assert callable(kwargs["run"])
assert callable(kwargs["enqueue"])
+49
View File
@@ -5086,6 +5086,55 @@ class TestMetacognitiveBuffers:
assert sys_turn["_source"] == "tool_error"
assert sys_turn["content"] == "you hit an error; check memory"
def test_denial_nudge_queues_on_tool_channel(self, tmp_db):
"""A denial responds to the tool batch the user just rejected — the
producer must queue it on the TOOL channel so it drains through
``_collect_advisories`` alongside the denied results (the same seam
tool_error / repeat use), not sit on the user channel until the next
user-message seam by which point the model has already reacted to
the denial without the nudge.
Drives the REAL ``_execute_tools`` two-phase gate with real
``_nudges_enabled`` / ``should_nudge`` gating; only the prepare
step and the UI approval are stubbed."""
from turnstone.core.metacognition import format_nudge
session = _make_session()
# ``should_nudge`` skips the very first message — give the session
# the natural pre-batch shape (user turn + assistant tool-call turn).
session.messages.append(turn_from_dict({"role": "user", "content": "do the thing"}))
session.messages.append(turn_from_dict({"role": "assistant", "content": "calling"}))
item = {
"call_id": "call_1",
"func_name": "notify",
"needs_approval": True,
# Must NOT run — a denied tool never executes.
"execute": lambda p: (p["call_id"], "EXECUTED — must not happen"),
}
with (
patch.object(session, "_safe_prepare_tool", return_value=item),
patch.object(session.ui, "approve_tools", return_value=(False, "use /tmp instead")),
patch.object(session, "_visible_memory_count", return_value=0),
):
tool_calls = [
{
"id": "call_1",
"type": "function",
"function": {"name": "notify", "arguments": "{}"},
}
]
results, feedback = session._execute_tools(tool_calls)
# The denied item surfaced the operator's feedback as its result…
assert results == [("call_1", "Denied by user: use /tmp instead")]
assert feedback is None
# …and the denial nudge is queued on the TOOL channel, so the same
# batch's ``_collect_advisories`` drain delivers it; nothing defers
# to the next user turn.
assert session._nudge_queue.pending(channel="tool") == [("denial", format_nudge("denial"))]
assert session._nudge_queue.pending(channel="user") == []
def test_queued_message_appends_system_turn_after_tool_batch(self, tmp_db):
"""A queued message arriving during a tool batch becomes a
first-class ``{"role": "system", "_source": "user_interjection"}``
+181 -5
View File
@@ -2,7 +2,7 @@
The shared worker dispatch is load-bearing for both the interactive
``/v1/api/workstreams/{ws_id}/send`` HTTP handler and the coordinator
``CoordinatorAdapter.send`` path. Tests cover the four invariants the
``CoordinatorAdapter.send`` path. Tests cover the five invariants the
module must hold:
* live worker enqueue, no thread spawn
@@ -10,10 +10,14 @@ module must hold:
* concurrent ``send`` calls produce exactly one worker thread
(Stage 1 bug-1 the racy ``Thread.is_alive()`` gate stays caught)
* ``_worker_running`` cleared in ``finally`` even on uncaught exception
* ownership-clear wake backstop: a worker exiting with USER_DRAIN
nudges queued on an IDLE workstream spawns the wake send that the
IDLE fan-out (which ran on this worker's own thread) had to drop
Callers pass no-arg closures, so this module never touches
``ws.session`` keeps the contract narrow and lets watch-style
dispatchers drive a session that isn't installed on ``ws``.
Callers pass no-arg closures, so dispatch never touches ``ws.session``;
the exit backstop only PEEKS it defensively (``getattr`` for
``_nudge_queue``, bail on stubs) watch-style dispatchers can still
drive a session that isn't installed on ``ws``.
"""
from __future__ import annotations
@@ -22,8 +26,10 @@ import queue
import threading
from typing import Any
from tests._helpers import wait_until as _wait_until
from turnstone.core import session_worker
from turnstone.core.workstream import Workstream
from turnstone.core.nudge_queue import USER_DRAIN, NudgeQueue
from turnstone.core.workstream import Workstream, WorkstreamState
class _SendSession:
@@ -140,6 +146,41 @@ def test_enqueue_unexpected_exception_returns_false_logged() -> None:
assert ws._worker_running is True
def test_closed_workstream_refused_no_spawn() -> None:
"""Authoritative closed-check: ``close()`` sets ``_closed`` under
``ws._lock``, so a wake (or send) racing it must be refused HERE
the wake gate's lockless peek can go stale, and a spawn past this
point would run a full unattended turn (inference, tool calls,
storage writes) on a workstream whose ``ws_closed`` already fired.
"""
session = _SendSession()
ws = _make_ws(session)
ws._closed = True
ok = _send_message(ws, session, "hello")
assert ok is False
assert session.send_calls == []
assert session.queue_calls == []
assert ws.worker_thread is None
assert ws._worker_running is False
def test_closed_workstream_refused_on_reuse_path_too() -> None:
"""The refusal precedes the enqueue branch: no interjection is queued
onto a session whose workstream is already closed."""
session = _SendSession()
ws = _make_ws(session)
ws._worker_running = True
ws._closed = True
ok = _send_message(ws, session, "hello")
assert ok is False
assert session.queue_calls == []
assert ws._worker_running is True # untouched — not ours to clear
# ---------------------------------------------------------------------------
# _worker_running lifecycle
# ---------------------------------------------------------------------------
@@ -301,6 +342,141 @@ def test_thread_name_explicit_override() -> None:
ws.worker_thread.join(timeout=2.0)
class _WakeCapableSession(_SendSession):
"""Adds the ChatSession surface the exit backstop peeks at."""
def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
self._nudge_queue = NudgeQueue()
self.deliver_calls = 0
self.deliver_thread_names: list[str] = []
self.delivered = threading.Event()
def deliver_wake_nudge_from_queue(self) -> None:
# Mirror the real contract: the wake drains its own queue, so
# the wake worker's OWN exit backstop sees nothing pending and
# the chain converges instead of spawning wakes forever.
self.deliver_calls += 1
self.deliver_thread_names.append(threading.current_thread().name)
self._nudge_queue.drain(USER_DRAIN)
self.delivered.set()
class TestWorkerExitWakeBackstop:
"""A worker exiting while its (idle) workstream has USER_DRAIN
nudges queued spawns the wake send the IDLE fan-out had to drop.
Production shape being modelled: ``set_state(IDLE)`` fires its
subscribers on the worker thread from inside ``run()``
``CoordinatorIdleObserver`` enqueues ``idle_children``, then
``IdleNudgeWatcher``'s wake dispatch lands on the reuse path
(this very worker still owns the flag) and no-ops. The enqueue
inside ``run`` below stands in for that observer enqueue.
"""
def test_worker_exit_delivers_pending_wake(self) -> None:
session = _WakeCapableSession()
ws = _make_ws(session)
assert ws.state is WorkstreamState.IDLE # dataclass default
def run() -> None:
# What the IDLE fan-out's observer does, on this thread.
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
assert ok is True
# The wake is delivered on a fresh wake-named worker thread…
assert session.delivered.wait(timeout=2.0), (
"exit backstop did not deliver the pending nudge"
)
assert session.deliver_thread_names[0].startswith("wake-nudge-")
# …after which the wake worker's own exit backstop sees an empty
# queue and the chain converges: flag at rest, exactly one deliver.
_wait_until(lambda: ws._worker_running is False)
assert session.deliver_calls == 1
assert len(session._nudge_queue) == 0
def test_worker_exit_no_wake_when_queue_empty(self) -> None:
session = _WakeCapableSession()
ws = _make_ws(session)
ok = session_worker.send(ws, enqueue=lambda: None, run=lambda: None)
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original # no wake spawned
assert session.deliver_calls == 0
assert ws._worker_running is False
def test_worker_exit_no_wake_for_stub_session_without_queue(self) -> None:
"""The narrow-contract escape hatch: a session without a
``_nudge_queue`` (watch-style stubs) is skipped by the shared
wake gate's own defensive peek — no AttributeError, no wake."""
session = _SendSession()
ws = _make_ws(session)
ok = _send_message(ws, session, "hello")
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original
assert ws._worker_running is False
def test_worker_exit_no_wake_when_state_not_idle(self) -> None:
"""An ERROR exit stays parked for the operator — pending nudges
wait for the next real interaction rather than burning
unattended inference on a failed session."""
session = _WakeCapableSession()
ws = _make_ws(session)
def run() -> None:
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
ws.state = WorkstreamState.ERROR
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
assert ok is True
original = ws.worker_thread
assert original is not None
original.join(timeout=2.0)
assert ws.worker_thread is original
assert session.deliver_calls == 0
assert len(session._nudge_queue) == 1 # still queued for later seams
def test_abandoned_worker_does_not_run_wake_backstop(self) -> None:
"""Only the owner retries: an abandoned worker (successor claimed
the flag) finishing late must not spawn a wake the successor's
own exit runs the backstop."""
send_gate = threading.Event()
session = _WakeCapableSession(send_gate=send_gate)
ws = _make_ws(session)
ok = _send_message(ws, session, "hello")
assert ok is True
abandoned = ws.worker_thread
assert abandoned is not None
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
sentinel = threading.Thread(target=lambda: None, name="successor")
with ws._lock:
ws.worker_thread = sentinel
ws._worker_running = True
send_gate.set()
abandoned.join(timeout=3.0)
assert not abandoned.is_alive()
# No wake spawned by the abandoned thread; ownership intact.
assert ws.worker_thread is sentinel
assert session.deliver_calls == 0
assert ws._worker_running is True
def test_does_not_deadlock_when_run_briefly_grabs_ws_lock() -> None:
"""Sanity check: ``run`` is invoked OUTSIDE ``ws._lock``. A worker
body that briefly takes the lock (e.g. to update worker state)
+723 -2
View File
@@ -2,11 +2,15 @@
from __future__ import annotations
import threading
import time
from datetime import UTC, datetime
from typing import Any
from unittest.mock import MagicMock
import pytest
from tests._helpers import wait_until
from turnstone.core.watch import (
WatchRunner,
build_watch_reminder,
@@ -545,8 +549,12 @@ class TestWatchRunner:
assert runner.get_dispatch_fn("ws-1") is fn
# Unknown ws → None.
assert runner.get_dispatch_fn("ws-missing") is None
# After removal → None.
runner.remove_dispatch_fn("ws-1")
# Owner-checked removal: a non-owner's teardown must not remove a
# still-live registration (restore shell vs reopened pane).
runner.remove_dispatch_fn("ws-1", owner=MagicMock())
assert runner.get_dispatch_fn("ws-1") is fn
# The owner (or a blind removal) does remove it.
runner.remove_dispatch_fn("ws-1", owner=fn)
assert runner.get_dispatch_fn("ws-1") is None
def test_run_command_success(self):
@@ -566,3 +574,716 @@ class TestWatchRunner:
output, code = runner._run_command("sleep 30")
assert "timed out" in output.lower()
assert code == -1
def _watch_row(**over: Any) -> dict[str, Any]:
"""A firing watch row (condition matches ``echo hello``); override
fields per test."""
row: dict[str, Any] = {
"watch_id": "abc123",
"ws_id": "ws-1",
"name": "test-watch",
"command": "echo hello",
"stop_on": '"hello" in output',
"max_polls": 100,
"poll_count": 0,
"last_output": None,
"interval_secs": 60,
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
}
row.update(over)
return row
def _slow_restore(runner: WatchRunner, ws_id: str, calls: list[str], lock: threading.Lock) -> Any:
"""Restore_fn stand-in: record the call, register a dispatch fn (as the
real restore does via ``set_watch_runner``), and sleep briefly so a
concurrent second caller is guaranteed to be waiting on ``_restore_lock``
when we return.
"""
with lock:
calls.append(ws_id)
time.sleep(0.05)
fn = MagicMock()
runner.set_dispatch_fn(ws_id, fn)
return fn
class TestWatchRunnerDeliveryRetry:
"""Delivery failure HOLDS the built reminder and re-delivers it on a
later tick never re-running the command, so a transient stop_on match
isn't lost — bounded by ``MAX_DELIVERY_ATTEMPTS``. Until delivery lands
the row commits only ``next_poll`` plus the fire's durable poll charge:
no baseline advance and no deactivation of a fire the model never saw,
while a restart mid-hold (which re-runs the command) stays bounded by
``max_polls``."""
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
def test_delivery_failure_holds_reminder_and_defers_row(self):
storage = MagicMock()
storage.update_watch.return_value = True
# No dispatch fn registered, no restore_fn → delivery fails.
runner = self._make_runner(storage)
runner._poll_watch(_watch_row())
# Row commit is the retry cadence + this fire's durable poll charge
# — the fire stays fully retryable, and a restart mid-hold (which
# re-runs the command) stays bounded by max_polls.
storage.update_watch.assert_called_once()
args, kwargs = storage.update_watch.call_args
assert args[0] == "abc123"
assert set(kwargs) == {"next_poll", "poll_count"}
assert kwargs["poll_count"] == 1 # charged durably at hold time
assert kwargs["next_poll"] # advanced, not cleared
# The reminder is HELD for re-delivery; the row is NOT marked
# terminal-dispatched (the model never saw it).
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
assert runner._pending_delivery["abc123"]["attempts"] == 1
with runner._terminal_dispatched_lock:
assert "abc123" not in runner._terminal_dispatched
def test_redelivery_uses_held_reminder_without_rerunning_command(self):
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
# Poll 1: fails → holds. Capture the exact held reminder object.
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
held = runner._pending_delivery["abc123"]["reminder"]
# ws restored: register a fn, and make _run_command explode so the
# test proves re-delivery does NOT re-run the command.
dispatch_fn = MagicMock()
runner.set_dispatch_fn("ws-1", dispatch_fn)
runner._run_command = MagicMock( # type: ignore[method-assign]
side_effect=AssertionError("command must not re-run on re-delivery")
)
runner._poll_watch(_watch_row())
# Delivered the SAME held reminder; command untouched; committed
# terminal with the ORIGINAL fire's poll_count; hold cleared.
dispatch_fn.assert_called_once()
assert dispatch_fn.call_args[0][0] is held
runner._run_command.assert_not_called()
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False
assert kwargs["poll_count"] == 1 # retries consumed no ADDITIONAL budget
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_transient_exhaustion_keeps_watch_active(self):
# A purely transient cause (no fn, no restore → returns False) that
# outlasts the attempt budget must NOT silently deactivate the watch
# — it drops the held reminder, charges ONE poll to the max_polls
# budget, and leaves the watch active to re-fire on its next
# interval. The baseline (last_output) stays uncommitted so a
# delta-style stop_on re-fires on the change the model never saw.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # always fails, transiently
# Poll 1 fires + holds (attempts=1); polls 2..MAX bump attempts, the
# MAX-th hitting the exhaustion ceiling.
for _ in range(MAX_DELIVERY_ATTEMPTS):
runner._poll_watch(_watch_row())
# Hold dropped, but the watch was NEVER deactivated — no active=False
# commit anywhere; the final commit charges the poll and re-schedules
# (no last_output → the fire re-detects next cycle).
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
deactivations = [
c for c in storage.update_watch.call_args_list if c.kwargs.get("active") is False
]
assert deactivations == []
_a, kwargs = storage.update_watch.call_args # last commit
assert set(kwargs) == {"poll_count", "next_poll"}
assert kwargs["poll_count"] == 1 # one poll charged to the budget
def test_transient_exhaustion_with_budget_spent_deactivates(self):
# The keep-alive-on-transient behavior is bounded by the watch's own
# max_polls budget: once poll_count reaches it, exhaustion commits
# the held (deactivating) update instead of re-running the command
# every interval forever against an unreachable workstream.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # always fails, transiently
for _ in range(MAX_DELIVERY_ATTEMPTS):
runner._poll_watch(_watch_row(max_polls=1)) # budget spent on fire 1
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
_a, kwargs = storage.update_watch.call_args # last commit
assert kwargs["active"] is False # deactivated: budget spent
assert kwargs["poll_count"] == 1
def test_held_delivery_retries_on_capped_cadence_not_interval(self):
# Re-delivery is a cheap in-memory dispatch — a daily watch whose
# fire hit a busy restore slot must retry within
# DELIVERY_RETRY_CAP_SECS, not sit on the reminder for 24 h.
from turnstone.core.watch import DELIVERY_RETRY_CAP_SECS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row(interval_secs=86_400))
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
retry_at = datetime.strptime(kwargs["next_poll"], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=UTC)
delta = (retry_at - datetime.now(UTC)).total_seconds()
assert 0 < delta <= DELIVERY_RETRY_CAP_SECS + 5 # capped, not 86400
def test_permanent_unrestorable_deactivates_immediately(self):
# A permanent failure (restore raises WatchWorkstreamUnrestorable,
# e.g. corrupt persona stamp) deactivates the watch on the FIRST
# fire — no held reminder, no waiting out the attempt budget.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
runner = self._make_runner(storage, restore_fn=restore_fn)
runner._poll_watch(_watch_row())
restore_fn.assert_called_once_with("ws-1") # not retried 5×
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False # deactivated now
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery # nothing held
# The admission slot is released even on the raising path.
with runner._restore_lock:
assert "ws-1" not in runner._restoring
def test_pending_cleared_on_already_dispatched_retry(self):
# A re-delivery that succeeded but whose row-commit raised leaves
# the id in BOTH _terminal_dispatched and _pending_delivery. The
# next tick's already-dispatched branch must clear the hold too, or
# it leaks once the row deactivates.
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
with runner._terminal_dispatched_lock:
runner._terminal_dispatched.add("abc123")
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
runner._poll_watch(_watch_row()) # hits the already_dispatched branch
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
with runner._terminal_dispatched_lock:
assert "abc123" not in runner._terminal_dispatched
def test_restore_capacity_full_defers_without_restoring(self):
# When MAX_CONCURRENT_RESTORES restores are already in flight, a
# new evicted-ws poll must DEFER (return False, hold) rather than
# block a poll slot — and must not start a restore.
from turnstone.core.watch import MAX_CONCURRENT_RESTORES
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=MagicMock())
runner = self._make_runner(storage, restore_fn=restore_fn)
# Saturate the restore admission with other in-flight ws_ids.
with runner._restore_lock:
for i in range(MAX_CONCURRENT_RESTORES):
runner._restoring[f"other-{i}"] = time.monotonic()
result = runner._dispatch_result("ws-evicted", {"text": "x"}, "w1")
assert result is False # deferred
restore_fn.assert_not_called() # no restore admitted
def test_race_won_admission_delivers_outside_restore_lock(self):
# A dispatch fn registered between the fast-path miss and the
# admission check must be delivered WITHOUT holding _restore_lock:
# the closure can block (ws._lock, wake-thread spawn), and running
# it under the lock serialises every restore admission on the node
# behind one delivery.
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=None)
runner = self._make_runner(storage, restore_fn=restore_fn)
lock_free_during_dispatch: list[bool] = []
def probe(reminder: dict[str, Any], watch_id: str) -> None:
ok = runner._restore_lock.acquire(blocking=False)
lock_free_during_dispatch.append(ok)
if ok:
runner._restore_lock.release()
real_try = runner._try_dispatch_fn
calls = {"n": 0}
def fake_try(ws_id: str, reminder: dict[str, Any], watch_id: str) -> bool | None:
calls["n"] += 1
if calls["n"] == 1:
# Simulate a restore completing between the fast path and
# the admission check: the fn appears "while we waited".
runner.set_dispatch_fn("ws-1", probe)
return None
return real_try(ws_id, reminder, watch_id)
runner._try_dispatch_fn = fake_try # type: ignore[method-assign]
result = runner._dispatch_result("ws-1", {"text": "x"}, "w1")
assert result is True # race-won fn delivered
assert lock_free_during_dispatch == [True] # ...outside the lock
restore_fn.assert_not_called() # no restore admitted for a live fn
def test_restore_returning_none_holds(self):
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=None) # e.g. all slots active
runner = self._make_runner(storage, restore_fn=restore_fn)
runner._poll_watch(_watch_row())
restore_fn.assert_called_once_with("ws-1")
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
def test_live_fn_raise_holds_without_restoring(self):
# A registered fn that RAISES means the ws is live; we must NOT fall
# through to restore (that would spawn a duplicate session on a live
# conversation). The reminder is held for re-delivery instead.
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock()
runner = self._make_runner(storage, restore_fn=restore_fn)
runner.set_dispatch_fn("ws-1", MagicMock(side_effect=RuntimeError("stale closure")))
runner._poll_watch(_watch_row())
restore_fn.assert_not_called()
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
_a, kwargs = storage.update_watch.call_args
assert set(kwargs) == {"next_poll", "poll_count"}
def test_forget_terminal_dispatched_clears_held_reminder(self):
# User-cancel takes the row out of the due view; its held reminder
# must be dropped too or it would leak (never re-polled).
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
runner.forget_terminal_dispatched("abc123")
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_abandon_write_failure_keeps_hold_for_write_retry(self):
# Reads-succeed/writes-fail storage (e.g. disk-full SQLite): a
# failed abandon commit must keep the hold so the next tick retries
# the WRITE via the redeliver path — the clear-first order let the
# row re-list into a fresh COMMAND RUN every attempt-budget cycle,
# forever, with the poll budget never advancing.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row()) # fire → hold (write still OK here)
with runner._pending_delivery_lock:
pending = dict(runner._pending_delivery["abc123"])
runner._restore_fn = MagicMock( # type: ignore[assignment]
side_effect=WatchWorkstreamUnrestorable("ws-1")
)
storage.update_watch.side_effect = RuntimeError("disk full")
with pytest.raises(RuntimeError):
runner._redeliver_pending(_watch_row(), pending)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
def test_unrestorable_abandon_write_failure_keeps_hold(self):
# Fresh-fire permanent failure whose deactivation write fails must
# not strand the still-active row into a fresh command run every
# tick: the stash routes the next tick into the redeliver path,
# which retries the WRITE — never the command.
from turnstone.core.watch import WatchWorkstreamUnrestorable
storage = MagicMock()
storage.update_watch.side_effect = RuntimeError("disk full")
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
runner = self._make_runner(storage, restore_fn=restore_fn)
with pytest.raises(RuntimeError):
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
# Next tick: the write retries and lands; the command never re-runs.
runner._run_command = MagicMock( # type: ignore[method-assign]
side_effect=AssertionError("command must not re-run")
)
storage.update_watch.side_effect = None
storage.update_watch.return_value = True
runner._poll_watch(_watch_row())
runner._run_command.assert_not_called()
_a, kwargs = storage.update_watch.call_args
assert kwargs["active"] is False # deactivation landed on the retry
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_exhaustion_write_failure_keeps_hold_for_write_retry(self):
# Same pathology on the transient-exhaustion branch: the charge
# commit failing must keep the hold (write retried next tick), not
# drop it into a fresh command cycle with the budget never durable.
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage) # no fn, no restore → transient
runner._poll_watch(_watch_row()) # fire → hold
with runner._pending_delivery_lock:
runner._pending_delivery["abc123"]["attempts"] = MAX_DELIVERY_ATTEMPTS - 1
pending = dict(runner._pending_delivery["abc123"])
storage.update_watch.side_effect = RuntimeError("disk full")
with pytest.raises(RuntimeError):
runner._redeliver_pending(_watch_row(), pending)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery # hold survived
class TestWatchRunnerCancelRace:
"""User-cancel racing the poll pool: the delivery paths re-check the
row's active state so a cancelled watch can neither deliver nor leak a
held reminder, and the tick sweep mops up the one interleaving the
point checks can't reach (a stash landing after the cancel path's
``forget_terminal_dispatched`` already cleared)."""
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
def test_hold_dropped_when_watch_cancelled_mid_fire(self):
# Cancel lands while the fire's command is running: the hold path
# re-checks the row and DROPS instead of stashing — an inactive row
# never re-lists, so a stash here would leak for the process
# lifetime with nothing ever retrying or clearing it.
storage = MagicMock()
storage.update_watch.return_value = True
storage.is_watch_active.return_value = False
runner = self._make_runner(storage) # no fn, no restore → would hold
runner._poll_watch(_watch_row())
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
# No retry-cadence commit either — the row already left the view.
storage.update_watch.assert_not_called()
def test_redelivery_dropped_when_watch_cancelled(self):
# Cancel lands between the due listing and the redelivery dispatch:
# deliver nothing (the model must not act on — nor a restore be
# spawned for — a watch the user just cancelled) and drop the hold.
storage = MagicMock()
storage.update_watch.return_value = True
runner = self._make_runner(storage)
runner._poll_watch(_watch_row()) # fails → holds (row still active)
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
storage.is_watch_active.return_value = False # user cancels
dispatch_fn = MagicMock()
runner.set_dispatch_fn("ws-1", dispatch_fn) # ws even came back live
runner._poll_watch(_watch_row())
dispatch_fn.assert_not_called()
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
# The only row write remains the initial hold's cadence commit —
# no terminal commit lands over the cancel's row state.
assert storage.update_watch.call_count == 1
def test_tick_sweeps_cancelled_holds(self):
# The residual interleaving: a stash that landed AFTER the cancel's
# forget_terminal_dispatched cleared (its active re-check passed
# just before the cancel's row write). The sweep drops it within
# one tick.
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
storage.is_watch_active.return_value = False # row already cancelled
runner._tick()
with runner._pending_delivery_lock:
assert "abc123" not in runner._pending_delivery
def test_tick_sweep_keeps_active_holds(self):
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
storage.is_watch_active.return_value = True
runner._tick()
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
def test_active_checks_bias_toward_delivery_on_storage_error(self):
# is_watch_active RAISING must not drop a fire: the sweep keeps the
# hold and the delivery paths proceed (bounded by their own attempt
# and poll budgets) — a storage blip is not a cancellation.
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = []
storage.is_watch_active.side_effect = RuntimeError("storage down")
runner = self._make_runner(storage)
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
runner._tick() # sweep: biased active → kept
with runner._pending_delivery_lock:
assert "abc123" in runner._pending_delivery
class TestWatchRunnerRestoreSerialization:
"""Two watches on ONE evicted workstream, polled concurrently, must
trigger the restore path at most once otherwise each spawns a live
auto-approved session racing writes into one conversation history."""
def test_concurrent_same_ws_restores_once(self):
storage = MagicMock()
storage.update_watch.return_value = True
restore_calls: list[str] = []
calls_lock = threading.Lock()
runner = WatchRunner(
storage=storage,
node_id="n",
check_interval=0.1,
tool_timeout=5,
restore_fn=lambda ws_id: _slow_restore(runner, ws_id, restore_calls, calls_lock),
)
reminder = {"type": "watch_triggered", "text": "x"}
barrier = threading.Barrier(2)
results: list[bool] = []
results_lock = threading.Lock()
def call(wid: str) -> None:
barrier.wait(timeout=2.0)
ok = runner._dispatch_result("ws-shared", reminder, wid)
with results_lock:
results.append(ok)
threads = [threading.Thread(target=call, args=(f"w{i}",)) for i in range(2)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=3.0)
# Exactly one restore ran; the winner delivered (True) and the other
# DEFERRED (False, holds + re-delivers next tick) rather than blocking
# its poll slot on the in-flight restore or restoring a second time.
assert restore_calls == ["ws-shared"]
assert sorted(results) == [False, True]
# Admission slot released after the restore.
with runner._restore_lock:
assert "ws-shared" not in runner._restoring
def test_wedged_restore_admissions_defer_and_alert(self, caplog):
# Admission entries older than RESTORE_STALL_ALERT_SECS are alerted
# on but NEVER evicted: the wedged poll thread's pool slot is never
# released, so reclaiming its admission would just readmit a restore
# that can wedge another pool thread on the same cause — trading
# this capped degraded state (restores blocked, polling intact) for
# total poll-pool collapse. New restores keep deferring; the error
# log is the operator's restart signal.
from turnstone.core.watch import RESTORE_STALL_ALERT_SECS
storage = MagicMock()
storage.update_watch.return_value = True
restore_fn = MagicMock(return_value=MagicMock())
runner = WatchRunner(
storage=storage,
node_id="n",
check_interval=0.1,
tool_timeout=5,
restore_fn=restore_fn,
)
stalled_at = time.monotonic() - RESTORE_STALL_ALERT_SECS - 1
with runner._restore_lock:
runner._restoring["wedged-1"] = stalled_at
runner._restoring["wedged-2"] = stalled_at # both slots wedged
with caplog.at_level("ERROR"):
result = runner._dispatch_result("ws-new", {"text": "x"}, "w1")
assert result is False # wedged capacity stays consumed → defer
restore_fn.assert_not_called()
with runner._restore_lock:
assert "wedged-1" in runner._restoring
assert "wedged-2" in runner._restoring
assert any("watch_runner.restore_admission_wedged" in r.message for r in caplog.records)
class TestWatchRunnerConcurrency:
"""The tick thread only enumerates due rows; polls run on bounded
daemon threads. Pins: genuine concurrency, per-watch in-flight
dedup, saturation leaving rows due (not dropped), and ``stop``
draining in-flight polls."""
def _make_runner(self, rows: list[dict[str, Any]], **kwargs: Any) -> WatchRunner:
storage = MagicMock()
storage.update_watch.return_value = True
storage.list_due_watches.return_value = rows
return WatchRunner(
storage=storage,
node_id="test-node",
check_interval=0.1,
tool_timeout=5,
**kwargs,
)
@staticmethod
def _wait_in_flight_empty(runner: WatchRunner, timeout: float = 3.0) -> None:
def _drained() -> bool:
with runner._in_flight_lock:
return not runner._in_flight
wait_until(_drained, timeout=timeout)
def test_tick_polls_concurrently(self):
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(3)]
runner = self._make_runner(rows)
all_in = threading.Event()
release = threading.Event()
barrier = threading.Barrier(3)
def fake_poll(_row: dict[str, Any]) -> None:
# All three poll threads must be inside simultaneously for the
# barrier to trip — serial execution would deadlock here (and
# fail via the barrier timeout instead).
barrier.wait(timeout=2.0)
all_in.set()
release.wait(timeout=2.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert all_in.wait(timeout=2.0), "polls did not run concurrently"
release.set()
self._wait_in_flight_empty(runner)
def test_tick_skips_in_flight_watch(self):
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
runner = self._make_runner(rows)
polled: list[str] = []
runner._poll_watch = lambda row: polled.append(row["watch_id"]) # type: ignore[method-assign]
# Simulate a slow poll from a previous tick still running.
with runner._in_flight_lock:
runner._in_flight.add("w0")
runner._tick()
assert polled == []
# The foreign in-flight entry was not clobbered by the skip.
with runner._in_flight_lock:
assert "w0" in runner._in_flight
def test_tick_saturation_leaves_rows_due(self):
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(2)]
runner = self._make_runner(rows, max_concurrent_polls=1)
started = threading.Event()
release = threading.Event()
polled: list[str] = []
def fake_poll(row: dict[str, Any]) -> None:
polled.append(row["watch_id"])
started.set()
release.wait(timeout=2.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert started.wait(timeout=2.0)
# Only the first row got a slot this tick; the second stays due
# for the next tick rather than being dropped.
assert polled == ["w0"]
release.set()
self._wait_in_flight_empty(runner)
# Next tick (slot free again) picks up the remaining row.
runner._storage.list_due_watches.return_value = [rows[1]]
runner._tick()
self._wait_in_flight_empty(runner)
assert polled == ["w0", "w1"]
def test_stop_waits_for_in_flight_polls(self):
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
runner = self._make_runner(rows)
started = threading.Event()
release = threading.Event()
def fake_poll(_row: dict[str, Any]) -> None:
started.set()
release.wait(timeout=3.0)
runner._poll_watch = fake_poll # type: ignore[method-assign]
runner._tick()
assert started.wait(timeout=2.0)
stopper = threading.Thread(target=runner.stop, daemon=True)
stopper.start()
# stop() must be draining (poll still pinned), not returned.
time.sleep(0.15)
assert stopper.is_alive(), "stop() returned while a poll was in flight"
release.set()
stopper.join(timeout=3.0)
assert not stopper.is_alive()
with runner._in_flight_lock:
assert not runner._in_flight
+217 -2
View File
@@ -54,9 +54,11 @@ def _make_session_for_dispatch(**kwargs: Any) -> ChatSession:
return ChatSession(**defaults)
def _register_runner(session: ChatSession) -> tuple[Any, Any]:
def _register_runner(session: ChatSession, wake_fn: Any = None) -> tuple[Any, Any]:
"""Attach a minimal stub ``WatchRunner`` to *session* and return the
``(runner, dispatch_fn)`` pair captured by ``set_dispatch_fn``.
``wake_fn`` rides through to ``set_watch_runner`` (default ``None``
matches the pre-wake wiring most tests here exercise).
"""
captured: dict[str, Any] = {}
@@ -65,7 +67,7 @@ def _register_runner(session: ChatSession) -> tuple[Any, Any]:
captured["fn"] = fn
runner = _StubRunner()
session.set_watch_runner(runner)
session.set_watch_runner(runner, wake_fn=wake_fn)
return runner, captured["fn"]
@@ -400,3 +402,216 @@ class TestMetadataPropagation:
assert len(snapshot) == 1
_nt, _text, meta = snapshot[0]
assert meta is None
# ---------------------------------------------------------------------------
# Wake trigger
# ---------------------------------------------------------------------------
class TestWakeFn:
"""``set_watch_runner``'s optional ``wake_fn`` fires once per enqueued
dispatch AFTER the entry lands so a watch firing on an
already-idle workstream (no IDLE transition for the
``IdleNudgeWatcher`` to observe) can spawn the wake worker that
drains it. Failures are contained: the enqueue must survive a
raising ``wake_fn``, because a propagated raise would abort
``WatchRunner._poll_watch`` before the watch-row update commits and
re-fire the same reminder every subsequent tick.
"""
def test_wake_fn_called_after_enqueue(self, tmp_db):
session = _make_session_for_dispatch()
depth_at_wake: list[int] = []
_runner, dispatch = _register_runner(
session, wake_fn=lambda: depth_at_wake.append(len(session._nudge_queue))
)
dispatch(_reminder("watch fired body"), "watch-1")
# Fired exactly once, and the entry was already queued when it ran
# — the wake worker's drain must be able to see the fresh entry.
assert depth_at_wake == [1]
def test_wake_fn_not_called_when_payload_sanitizes_empty(self, tmp_db):
"""A fire whose payload strips to nothing enqueues nothing — and
must not wake anything either (a wake with an empty queue would
just spawn a worker that no-ops at the drain guard)."""
session = _make_session_for_dispatch()
wake = MagicMock()
_runner, dispatch = _register_runner(session, wake_fn=wake)
dispatch(_reminder("\x07\x0b\x7f"), "watch-1")
assert len(session._nudge_queue) == 0
wake.assert_not_called()
def test_wake_fn_exception_is_contained(self, tmp_db, caplog):
session = _make_session_for_dispatch()
wake = MagicMock(side_effect=RuntimeError("boom"))
_runner, dispatch = _register_runner(session, wake_fn=wake)
with caplog.at_level("WARNING"):
dispatch(_reminder("body"), "watch-1") # must not raise
# Entry survived; the failure surfaced as a warning, not a raise
# up into the poll loop.
assert len(session._nudge_queue) == 1
assert any("watch_dispatch.wake_failed" in r.message for r in caplog.records), (
"expected a watch_dispatch.wake_failed warning record"
)
class _RecordingRunner:
"""Stub WatchRunner recording registration/removal order. Mirrors the
production owner-checked removal semantics the resume tail passes
``owner`` and peeks ``get_dispatch_fn`` before re-registering."""
def __init__(self) -> None:
self.events: list[tuple[str, str]] = []
self.fns: dict[str, Any] = {}
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
self.events.append(("set", ws_id))
self.fns[ws_id] = fn
def get_dispatch_fn(self, ws_id: str) -> Any:
return self.fns.get(ws_id)
def remove_dispatch_fn(self, ws_id: str, owner: Any = None) -> None:
if owner is not None and self.fns.get(ws_id) is not owner:
return
self.events.append(("remove", ws_id))
self.fns.pop(ws_id, None)
class TestResumeReRegistration:
"""A non-fork ``resume()`` rebinds ``_ws_id``; the dispatch
registration must FOLLOW that identity otherwise watches stamped
with the adopted id never find the live session, and every fire
takes the restore path, spawning a duplicate auto-approved session
racing writes into the same conversation (CLI ``--resume`` and the
``/resume`` command both hit this)."""
def _saved_ws(self, ws_id: str) -> None:
from turnstone.core.memory import register_workstream, save_message
register_workstream(ws_id)
save_message(ws_id, "user", "hi")
def test_nonfork_resume_moves_registration_to_adopted_id(self, tmp_db):
self._saved_ws("resume-target")
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
assert session.resume("resume-target") is True
# New key live BEFORE the old key is removed — a fire during the
# transition can never observe an empty registry (which would
# divert it to the restore path).
assert runner.events == [
("set", old_id),
("set", "resume-target"),
("remove", old_id),
]
assert set(runner.fns) == {"resume-target"}
def test_fork_resume_keeps_registration(self, tmp_db):
self._saved_ws("fork-src")
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
assert session.resume("fork-src", fork=True) is True
# Fork keeps its own identity — registration untouched.
assert runner.events == [("set", old_id)]
def test_resume_without_runner_is_noop(self, tmp_db):
# CLI --resume / restore-fn shape: resume() runs BEFORE any
# set_watch_runner call — nothing to re-register, nothing raises.
self._saved_ws("resume-bare")
session = _make_session_for_dispatch()
assert session.resume("resume-bare") is True
assert session._watch_runner is None
def test_reregistered_closure_keeps_wake_fn(self, tmp_db):
# The stored wake_fn rides the re-registration: a watch firing on
# the ADOPTED id must still wake the workstream.
self._saved_ws("resume-wake")
session = _make_session_for_dispatch()
runner = _RecordingRunner()
wake = MagicMock()
session.set_watch_runner(runner, wake_fn=wake)
assert session.resume("resume-wake") is True
runner.fns["resume-wake"](_reminder("watch output"), "w1")
assert len(session._nudge_queue) == 1
wake.assert_called_once()
def test_resume_does_not_steal_another_live_registration(self, tmp_db):
# In-session /resume of a workstream that is OPEN IN ANOTHER PANE
# (a degenerate two-live-sessions state): the original owner keeps
# its watch fires — the adopter neither clobbers the target's
# registration nor (on a later resume-away or close) deletes it.
self._saved_ws("shared-A")
self._saved_ws("other-C")
runner = _RecordingRunner()
pane_a = _make_session_for_dispatch()
pane_a._ws_id = "shared-A" # pane A opened A and registered
pane_a.set_watch_runner(runner, wake_fn=None)
fn_a = runner.fns["shared-A"]
pane_b = _make_session_for_dispatch()
pane_b.set_watch_runner(runner, wake_fn=None)
assert pane_b.resume("shared-A") is True
# Pane A's registration survived the adoption…
assert runner.fns["shared-A"] is fn_a
assert pane_b.resume("other-C") is True
# …and the resume-away removed only pane B's own (absent) claim.
assert runner.fns["shared-A"] is fn_a
assert "other-C" in runner.fns
def test_new_command_moves_registration_to_fresh_id(self, tmp_db):
# /new is the other identity rebind: watches created AFTER it stamp
# the fresh id and must reach this session, while the old
# workstream's fires must stop landing in a conversation that no
# longer shows them (they divert to the restore path instead).
session = _make_session_for_dispatch()
old_id = session._ws_id
runner = _RecordingRunner()
session.set_watch_runner(runner, wake_fn=None)
# handle_command's return means "should exit" — /new never exits.
assert session.handle_command("/new") is False
assert session._ws_id != old_id
assert set(runner.fns) == {session._ws_id}
assert ("remove", old_id) in runner.events
def test_close_removes_only_own_registration(self, tmp_db):
# A watch-restore shell and a reopened pane can serve one ws_id in
# sequence; the shell's later teardown must not unregister the pane.
self._saved_ws("shared-W")
runner = _RecordingRunner()
shell = _make_session_for_dispatch()
shell._ws_id = "shared-W"
shell.set_watch_runner(runner, wake_fn=None)
pane = _make_session_for_dispatch()
pane._ws_id = "shared-W"
pane.set_watch_runner(runner, wake_fn=None) # pane re-registers (last writer)
pane_fn = runner.fns["shared-W"]
shell.close() # shell reaped (close_idle / eviction)
assert runner.fns.get("shared-W") is pane_fn # pane still registered
+2 -2
View File
@@ -340,8 +340,8 @@ def test_poll_watch_terminal_fire_survives_drain(
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# For the max_polls=1 case the first poll has prev_output=None and
# would not normally fire on output change; the max_polls branch
# at watch.py:412-414 still marks is_final=True so dispatch runs.
# would not normally fire on output change; _poll_watch's max_polls
# branch still marks is_final=True so dispatch runs.
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
+11
View File
@@ -224,6 +224,17 @@ class CreateWorkstreamResponse(BaseModel):
"/v1/api/workstreams/{ws_id}/send."
),
)
initial_message_status: str | None = Field(
default=None,
description=(
"Present ONLY when the workstream was created but its "
"initial_message could not be delivered: 'queue_full' (a raced "
"live worker's interjection queue was at capacity — resend via "
"/send; any uploads stay staged) or 'refused_closed' (the "
"workstream was closed mid-create). Absent whenever the message "
"was dispatched."
),
)
class CloseWorkstreamRequest(BaseModel):
+20 -4
View File
@@ -23,12 +23,28 @@ if TYPE_CHECKING:
def cleanup_session_ui(ws: Workstream) -> None:
"""Shared SessionKindAdapter cleanup_ui implementation.
Unblocks pending approval / plan / foreground events on the
workstream's UI, broadcasts ``ws_closed`` to per-UI listener
queues, then cancels + closes the session. The ``hasattr`` checks
guard stub UIs used in tests the real ``WebUI`` /
Marks the workstream object dead (``ws._closed``) FIRST, under
``ws._lock``, then unblocks pending approval / plan / foreground
events on the workstream's UI, broadcasts ``ws_closed`` to per-UI
listener queues, and cancels + closes the session. The ``hasattr``
checks guard stub UIs used in tests the real ``WebUI`` /
``ConsoleCoordinatorUI`` always have these attributes.
The flag write lives HERE because every teardown path funnels
through this function ``close``, ``close_idle``, EVICTION,
``delete``, ``discard`` and per-path writes are not enough: a
flag set by only some paths, or set after this teardown body,
leaves windows where the workstream is being torn down while
still reading as live. The wake paths that hold OBJECT
references (the watch ``wake_fn``, ``session_worker``'s exit
backstop) gate on this flag, and ``session_worker.send``
re-checks it under the same lock: once this write lands, no wake
can spawn a worker on the torn-down session including on
evicted or deleted workstreams, and including during the
remainder of this teardown.
"""
with ws._lock:
ws._closed = True
if ws.session is not None and hasattr(ws.session, "cancel"):
ws.session.cancel()
ui = ws.ui
+119 -25
View File
@@ -1,8 +1,18 @@
"""Idle wake-trigger for the metacog NudgeQueue pipeline.
"""Idle wake-triggers for the metacog NudgeQueue pipeline.
Hosts :class:`IdleNudgeWatcher` plus the
:func:`install_idle_nudge_watcher` / :func:`shutdown_idle_nudge_watchers`
lifespan helpers. Pulled out of :mod:`turnstone.core.metacognition`
Hosts the two wake entry points plus their lifespan helpers:
* :class:`IdleNudgeWatcher` event-driven: a workstream transitions
to IDLE while nudges are ALREADY queued.
* :func:`wake_workstream_if_pending` the shared wake gate, also
called directly by asynchronous producers that enqueue onto an
ALREADY-idle workstream (the watch dispatch closure, via
``ChatSession.set_watch_runner``'s ``wake_fn``). Such producers see
no IDLE transition the workstream has been idle all along so
the watcher alone would leave their entries queued until the next
user message.
Pulled out of :mod:`turnstone.core.metacognition`
because the watcher is subscriber-lifecycle / runtime-orchestration
code with different concerns from the static nudge-text templates and
detection heuristics that live in metacognition; mixing them grew the
@@ -23,25 +33,110 @@ if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.core.session_manager import SessionManager
from turnstone.core.workstream import Workstream
log = get_logger(__name__)
def wake_workstream_if_pending(ws: Workstream, *, trigger: str = "unspecified") -> bool:
"""Spawn a wake send for *ws* when it is idle with drainable nudges.
The shared gate behind both wake triggers:
* :class:`IdleNudgeWatcher` the workstream just transitioned to
IDLE with nudges already queued.
* the watch dispatch closure (``ChatSession.set_watch_runner``'s
``wake_fn``) a watch fired on a workstream that is ALREADY
idle, so no IDLE transition will ever re-check the queue.
*trigger* is a short label naming which path requested the wake
(``"idle-transition"``, ``"watch-fire"``, ``"worker-exit"``); it is
used only to tag the log lines below and never affects control flow.
Gates, in order:
* ``ws.session is None`` workstream tracked but session not
built or a bare stub session without a NudgeQueue (watch-style
dispatchers drive sessions that aren't installed on the
workstream).
* ``ws._closed`` ``close()`` already ran (or is racing us); its
storage row says ``closed`` and a wake send would drive a
torn-down session. Lockless FAST-PATH only: a stale ``False``
falls through to ``session_worker.send``, which re-checks
``_closed`` under ``ws._lock`` the same lock ``close()`` sets
it under and refuses, so a wake racing a close can never spawn
a worker on the torn-down session.
* ``ws.state is not IDLE`` a busy workstream's worker drains the
queue at its own seams (``ATTENTION``/``THINKING``/``RUNNING``
all imply a live worker), and ``ERROR`` stays parked for the
operator rather than burning inference unattended.
* nothing drainable under ``USER_DRAIN`` tool-only entries
belong to the next tool-result seam, not a synthetic empty user
turn (``deliver_wake_nudge_from_queue`` would no-op on them).
Past the gates, exactly one info line is emitted per call:
* ``nudge_wake.deferred_worker_busy`` the reuse-path drop: a
worker owned the workstream, so ``session_worker.send`` called
the no-op ``enqueue`` instead of spawning. The entry stays
queued; the owning worker's exit backstop (or its next drain
seam) delivers it.
* ``nudge_wake.dispatched`` a fresh wake daemon was spawned.
Returns ``True`` iff the wake was handed to
``session_worker.send`` which may still downgrade it to a no-op
enqueue when a worker owns the workstream (see the race-semantics
section on :class:`IdleNudgeWatcher`).
"""
session = ws.session
if session is None or ws._closed or ws.state is not WorkstreamState.IDLE:
return False
nudge_queue = getattr(session, "_nudge_queue", None)
if nudge_queue is None or not nudge_queue.has_pending(USER_DRAIN):
return False
deferred = False
def _noop_enqueue() -> None:
nonlocal deferred
deferred = True
ok = session_worker.send(
ws,
enqueue=_noop_enqueue,
run=session.deliver_wake_nudge_from_queue,
thread_name=f"wake-nudge-{ws.id[:8]}",
)
if deferred:
log.info("nudge_wake.deferred_worker_busy ws=%s trigger=%s", ws.id[:8], trigger)
elif ok:
log.info("nudge_wake.dispatched ws=%s trigger=%s", ws.id[:8], trigger)
return ok
class IdleNudgeWatcher:
"""Convert a workstream IDLE transition into a wake send when the
session has queued nudges.
Subscribes to :meth:`SessionManager.subscribe_to_state` and listens
for ``WorkstreamState.IDLE``. If the workstream's
for ``WorkstreamState.IDLE``, then defers to
:func:`wake_workstream_if_pending` (the shared gate see its
docstring for the full gate order). If the workstream's
:class:`NudgeQueue` has any drainable entry for the wake's drain
filter (``USER_DRAIN`` channels ``"user"`` or ``"any"``),
dispatches via ``session_worker.send`` with a no-op ``enqueue``
callback. Tool-only entries don't fire the wake — they belong to
the next tool-result seam, not a synthetic empty user turn
otherwise every IDLE event with a queued tool advisory would spawn
a wake daemon that immediately no-ops at
filter (``USER_DRAIN`` channels ``"user"`` or ``"any"``), the
gate dispatches via ``session_worker.send`` with a no-op
``enqueue`` callback. Tool-only entries don't fire the wake —
they belong to the next tool-result seam, not a synthetic empty
user turn otherwise every IDLE event with a queued tool advisory
would spawn a wake daemon that immediately no-ops at
``deliver_wake_nudge_from_queue``'s drain guard.
This watcher only covers nudges that are already queued when the
IDLE transition fires. Producers that enqueue asynchronously onto
an already-idle workstream (watch fires) call
:func:`wake_workstream_if_pending` themselves there is no state
transition for this watcher to observe in that case.
**Race semantics.** ``session_worker.send`` decides atomically
under ``ws._lock`` whether a worker thread already owns the
workstream. Three outcomes:
@@ -51,10 +146,17 @@ class IdleNudgeWatcher:
drains its own queue and runs the synthetic empty-user turn).
* Worker running call our ``enqueue`` lambda, which is a no-op.
The wake is silently dropped; the queued nudge stays in
``NudgeQueue`` and the in-flight worker picks it up at its next
user-message-attach or tool-result seam (whichever fires first
for the entry's channel). This is the load-bearing fallback —
we never spawn a competing worker.
``NudgeQueue``. We never spawn a competing worker. This branch
is the COMMON case for IDLE-transition wakes, not the exception:
``set_state`` subscribers fire on the calling thread, and IDLE
is emitted from inside ``run()`` at the end of a send so the
transitioning worker still owns the flag while this watcher
dispatches. Delivery is then owed to one of two follow-ups:
the in-flight worker's next drain seam (when IDLE fired
mid-turn), or for the end-of-send case, where no later seam
exists ``session_worker``'s ownership-clear backstop
(``_retry_pending_wake``), which re-runs
:func:`wake_workstream_if_pending` the moment the worker exits.
* Workstream gone (``ws is None``) or session not built
(``ws.session is None``) bail.
@@ -82,17 +184,9 @@ class IdleNudgeWatcher:
if state is not WorkstreamState.IDLE:
return
ws = self._manager.get(ws_id)
if ws is None or ws.session is None:
if ws is None:
return
session = ws.session
if not session._nudge_queue.has_pending(USER_DRAIN):
return
session_worker.send(
ws,
enqueue=lambda: None,
run=session.deliver_wake_nudge_from_queue,
thread_name=f"wake-nudge-{ws.id[:8]}",
)
wake_workstream_if_pending(ws, trigger="idle-transition")
self._callback = _on_state
self._manager.subscribe_to_state(_on_state)
+139 -27
View File
@@ -191,7 +191,7 @@ from turnstone.ui.colors import DIM, GRAY, GREEN, RED, RESET, YELLOW, bold, cyan
log = get_logger(__name__)
if TYPE_CHECKING:
from collections.abc import Iterable, Iterator
from collections.abc import Callable, Iterable, Iterator
from turnstone.core.config_store import ConfigStore
from turnstone.core.healthcheck import BackendHealthTracker, HealthTrackerRegistry
@@ -1485,6 +1485,16 @@ class ChatSession:
self._notify_count = 0
# Watch support: server-level runner injected via set_watch_runner()
self._watch_runner: Any = None # WatchRunner | None
# The wake_fn last passed to set_watch_runner, kept so a non-fork
# resume() can re-register the dispatch closure under the adopted
# ws_id with the same wake wiring (registration follows identity).
self._watch_wake_fn: Callable[[], object] | None = None
# The dispatch closure this session last registered — the OWNER
# token for registry removals: multiple live sessions can
# transiently serve one ws_id (watch-restore shell vs a reopened
# pane; in-session /resume of an id open in another pane), and a
# blind removal on teardown would unregister the OTHER session.
self._watch_dispatch_fn: Callable[[dict[str, Any], str], None] | None = None
# Metacognitive nudges: ephemeral prompts for proactive memory use.
# One ``NudgeQueue`` per session; producers tag each entry with a
# channel and consumers drain by filter, emitting each drained nudge
@@ -2823,7 +2833,7 @@ class ChatSession:
else:
self._tool_search = None
def set_watch_runner(self, runner: Any) -> None:
def set_watch_runner(self, runner: Any, wake_fn: Callable[[], object] | None = None) -> None:
"""Inject the server-level WatchRunner and register a dispatch fn
that routes watch results onto this session's NudgeQueue.
@@ -2842,14 +2852,37 @@ class ChatSession:
sourced from arbitrary shell output can't tamper with the
envelope at interpolation time.
``wake_fn`` runs once per enqueued fire, AFTER the entry lands
on the queue. The server wires it to
:func:`turnstone.core.idle_nudge_watcher.wake_workstream_if_pending`
closed over this session's Workstream: a watch firing on an
ALREADY-idle workstream sees no IDLE transition, so without an
explicit wake the entry would sit queued until the next user
message. Busy workstreams stay safe the wake gate defers to
``session_worker.send``'s atomic ownership check, which
downgrades the wake to a no-op while a worker owns the session
(the in-flight worker drains the ``"any"``-channel entry at its
next seam). Exceptions from ``wake_fn`` are logged and
swallowed: the enqueue already happened, and a raise here would
abort ``WatchRunner._poll_watch`` before the watch-row update
commits re-firing the same reminder every subsequent tick.
No ``valid_until`` predicate is wired: ``WatchRunner._poll_watch``
commits ``active=False`` for terminal fires right after dispatch
returns, and an ``is_watch_active`` predicate would race that
write at drain time and drop the fire the model was meant to see.
A user-cancelled watch's last splat is informative (the reminder
carries ``is_final=True``), not stale-noise to suppress.
The registration is keyed on ``self._ws_id`` AT CALL TIME. The
identity-rebind sites (non-fork :meth:`resume`, ``/new``) call
:meth:`_follow_watch_registration` to move it onto the new id
(re-invoking this method with the ``wake_fn`` stored below)
callers therefore don't need to order their own
``set_watch_runner``/``resume`` calls.
"""
self._watch_runner = runner
self._watch_wake_fn = wake_fn
nudge_queue = self._nudge_queue
ws_id = self._ws_id
@@ -2892,7 +2925,13 @@ class ChatSession:
"any",
metadata=metadata or None,
)
if wake_fn is not None:
try:
wake_fn()
except Exception:
log.warning("watch_dispatch.wake_failed ws=%s", ws_id, exc_info=True)
self._watch_dispatch_fn = _dispatch
runner.set_dispatch_fn(self._ws_id, _dispatch)
def close(self) -> None:
@@ -2929,8 +2968,12 @@ class ChatSession:
self._mcp_prompt_cb, user_id=self._mcp_listener_user_id
)
self._mcp_prompt_cb = None
if self._watch_runner:
self._watch_runner.remove_dispatch_fn(self._ws_id)
if self._watch_runner and self._watch_dispatch_fn is not None:
# Owner-checked: a watch-restore shell and a reopened pane can
# both have served this ws_id — tearing down one must not
# unregister the other (whose next fire would then restore a
# DUPLICATE auto-approved session onto the live conversation).
self._watch_runner.remove_dispatch_fn(self._ws_id, owner=self._watch_dispatch_fn)
if self._coord_client is not None and hasattr(self._coord_client, "close"):
try:
self._coord_client.close()
@@ -3369,6 +3412,9 @@ class ChatSession:
turns = load_message_turns(ws_id)
if not turns:
return False
# Pre-rebind identity, for moving the watch dispatch registration
# onto the adopted id at the end of a successful non-fork resume.
old_ws_id = self._ws_id
# Load persisted config and parse the persona stamp BEFORE touching
# session identity/history: a corrupt stamp must raise while this
# session is still intact — the web /command surface reports the
@@ -3585,9 +3631,42 @@ class ChatSession:
cooldown_secs=self._mem_cfg.nudge_cooldown,
):
self._queue_user_advisory("resume", format_nudge("resume"))
if not fork:
self._follow_watch_registration(old_ws_id)
self._init_system_messages()
return True
def _follow_watch_registration(self, old_ws_id: str) -> None:
"""Move the watch dispatch registration onto the current
``_ws_id`` after an identity rebind (non-fork :meth:`resume`,
``/new``).
The registry is keyed by ``_ws_id`` at registration time.
Without the move, watches stamped with the NEW id never find
this live session every fire takes the restore path and
spawns a DUPLICATE auto-approved session racing writes into the
same conversation while fires for the OLD id keep delivering
into a session that no longer displays that conversation. The
new key goes live BEFORE the old one is removed so no fire can
observe a window with no registration at all (which would
likewise divert to the restore path). If ANOTHER live session
already serves the new id (in-session /resume of a workstream
open in a second pane an inherently degenerate two-writers
state), its registration is NOT stolen: the original owner
keeps its watch fires. Removal of the old key is owner-checked
for the same reason.
"""
if self._watch_runner is None:
return
old_fn = self._watch_dispatch_fn
existing = self._watch_runner.get_dispatch_fn(self._ws_id)
if existing is None or existing is old_fn:
self.set_watch_runner(self._watch_runner, wake_fn=self._watch_wake_fn)
else:
log.warning("watch_registry.adopted_id_owned_elsewhere ws=%s", self._ws_id[:8])
if old_ws_id != self._ws_id:
self._watch_runner.remove_dispatch_fn(old_ws_id, owner=old_fn)
def _nudges_enabled(self, nudge_type: str) -> bool:
"""Config gate + persona lever 4 for metacognitive nudges.
@@ -6023,8 +6102,9 @@ class ChatSession:
# Operator context for this result: output-guard
# findings + queued user messages (Seam 1), plus
# tool-channel metacog nudges (tool_error / repeat) and
# any-channel nudges (watch_triggered / idle_children).
# tool-channel metacog nudges (tool_error / repeat /
# denial) and any-channel nudges (watch_triggered /
# idle_children).
# All of them are now emitted as first-class
# ``{"role": "system"}`` turns AFTER this clean tool
# message (uniform attach rule) — the tool message content
@@ -6239,10 +6319,11 @@ class ChatSession:
def _drain_pending_advisories(self) -> None:
"""Drop every pending nudge regardless of channel.
Tool-channel nudges (``tool_error``, ``repeat``) queued earlier
in this batch and user-channel nudges (``correction``,
``denial``, ) queued during ``_check_metacognitive_nudge`` but
not yet drained share the same per-session :class:`NudgeQueue`.
Tool-channel nudges (``tool_error``, ``repeat``, ``denial``)
queued earlier in this batch and user-channel nudges
(``correction``, ) queued during ``_check_metacognitive_nudge``
but not yet drained share the same per-session
:class:`NudgeQueue`.
When a generation is abandoned (cancel, KeyboardInterrupt,
unexpected exception) the entire queue drops so nothing bleeds
into the next send's tool loop or next user turn.
@@ -8653,10 +8734,11 @@ class ChatSession:
``priority`` so the UI can frame important interjections
distinctly. Cancel / exception / no-tool-call paths drain the
queue as a real user row instead (Seams 2 and 3).
- **Metacognitive tool-channel nudges** (``tool_error`` / ``repeat``)
and any-channel nudges (``watch_triggered`` / ``idle_children``)
drained on the last result. ``meta`` carries the producer's
optional fields (e.g. ``watch_triggered``'s ``watch_name``).
- **Metacognitive tool-channel nudges** (``tool_error`` /
``repeat`` / ``denial``) and any-channel nudges
(``watch_triggered`` / ``idle_children``) drained on the last
result. ``meta`` carries the producer's optional fields
(e.g. ``watch_triggered``'s ``watch_name``).
Empty when no advisories apply (common case). Guard findings
attach per-result; queued messages and metacognitive nudges drain
@@ -8867,7 +8949,13 @@ class ChatSession:
memory_count=self._visible_memory_count(),
cooldown_secs=self._mem_cfg.nudge_cooldown,
):
self._queue_user_advisory("denial", format_nudge("denial"))
# Tool channel, not user: the denial is a response to THIS
# batch, so the nudge rides ``_collect_advisories`` alongside
# the denied tool results (same seam as tool_error / repeat)
# instead of deferring to the next user-message seam — by
# which point the model has already reacted to the denial
# without it.
self._queue_tool_advisory("denial", format_nudge("denial"))
# Phase 3: execute (check cancellation before starting)
self._check_cancelled()
@@ -10599,8 +10687,10 @@ class ChatSession:
Drains in ``_emit_pending_user_nudges`` and is appended as a
first-class ``{"role": "system"}`` turn AFTER the user turn. Used
for nudges that respond to user behaviour: ``correction``,
``denial``, ``resume``, ``start``, ``completion``.
for nudges that respond to the user's message: ``correction``,
``resume``, ``start``, ``completion``. (``denial`` is user
behaviour too, but it responds to a specific TOOL BATCH it
rides the tool channel so it lands with the denied results.)
No-ops while the session is inside a wake-driven turn
(``_wake_source_tag`` set) so model behaviour during the wake
@@ -10624,8 +10714,8 @@ class ChatSession:
the system turns sit after the user turn they advise (uniform attach
rule). Each drained nudge becomes one ``{"role": "system",
"_source": <nudge_type>, ...}`` turn via :meth:`_append_system_turn`
the source is the nudge type (``correction`` / ``denial`` /
``resume`` / ``start`` / ``completion`` / ``idle_children`` /
the source is the nudge type (``correction`` / ``resume`` /
``start`` / ``completion`` / ``idle_children`` /
``watch_triggered``) and any optional metadata (e.g.
``watch_triggered``'s ``watch_name``) rides as sibling keys.
``_append_system_turn`` persists each row and fires the live
@@ -10662,8 +10752,9 @@ class ChatSession:
Drains in ``_collect_advisories`` alongside guard findings, then is
emitted as a first-class ``{"role": "system"}`` turn AFTER the tool
batch (see the per-result loop in ``_run_loop``). Used for nudges
that respond to model behaviour at a tool boundary: ``tool_error``,
``repeat``.
that respond to a tool batch: ``tool_error``, ``repeat`` (model
behaviour), and ``denial`` (the operator rejected the batch the
nudge belongs next to the denied results it explains).
No-ops while the session is inside a wake-driven turn (see
``_queue_user_advisory`` for the rationale).
@@ -10723,6 +10814,14 @@ class ChatSession:
self._wake_drained_reminders = wake_reminders
try:
self.send("", from_wake=True)
except GenerationCancelled:
# A close/force-cancel raced this unattended wake turn. This
# method IS the wake worker's run() closure, and
# ``session_worker._runner`` catches only ``Exception`` — a
# BaseException here would escape to ``threading.excepthook``
# as stderr noise. The teardown gates stop any respawn; the
# cancellation itself is the intended outcome.
log.info("wake_nudge.cancelled ws=%s", self._ws_id[:8])
finally:
self._wake_source_tag = ""
self._wake_drained_reminders = None
@@ -15882,15 +15981,22 @@ class ChatSession:
# view (already-inactive or just-cancelled with empty
# next_poll), so the runner's retry-deactivate branch will
# never reclaim a pending ``_terminal_dispatched`` entry.
# Clear it here to bound the lifetime of any leftover from
# a previous dispatch-then-failed-row-write.
if self._watch_runner is not None:
self._watch_runner.forget_terminal_dispatched(target["watch_id"])
# Clear it via ``forget_terminal_dispatched`` to bound the
# lifetime of any leftover from a previous
# dispatch-then-failed-row-write — AFTER the row write, per
# that method's ordering contract: the runner's delivery
# paths re-check the row before stashing, so clearing first
# would let a racing poll thread re-stash a held reminder
# behind the clear.
if not target["active"]:
if self._watch_runner is not None:
self._watch_runner.forget_terminal_dispatched(target["watch_id"])
msg = f'Watch "{target["name"]}" already completed (auto-cancelled).'
self._report_tool_result(call_id, "watch", msg)
return call_id, msg
storage.update_watch(target["watch_id"], active=False, next_poll="")
if self._watch_runner is not None:
self._watch_runner.forget_terminal_dispatched(target["watch_id"])
msg = f'Watch "{target["name"]}" cancelled.'
self._report_tool_result(call_id, "watch", msg)
return call_id, msg
@@ -16371,14 +16477,20 @@ class ChatSession:
self._last_usage = None
self._calibrated_msg_count = 0
self._msg_tokens = []
old_ws_id = self._ws_id
self._ws_id = uuid.uuid4().hex
# A brand-new ws_id is the same class of identity change as a
# non-fork resume(): the old workstream's participant state must
# not leak into this empty one, and its trust nonces must not
# carry over (see resume()'s matching reset + remint).
# not leak into this empty one, its trust nonces must not carry
# over (see resume()'s matching reset + remint), and the watch
# dispatch registration must follow the identity — otherwise
# watches created here (stamped with the new id) never reach
# this session, and the old workstream's fires land in a
# conversation that no longer shows them.
self._reset_shared_state()
self._envelope_nonce = fence.mint_nonce()
self._sender_label_nonce = fence.mint_nonce()
self._follow_watch_registration(old_ws_id)
self._title_generated = False
# The session keeps its persona across /new (the stamp is
# re-written by _save_config below) — carry the display slug
+42 -9
View File
@@ -2772,15 +2772,21 @@ def make_create_handler(
attachment_ids,
)
return JSONResponse(
{
"ws_id": ws.id,
"name": ws.name,
"resumed": bool(extra_response.get("resumed", False)),
"message_count": int(extra_response.get("message_count", 0)),
"attachment_ids": attachment_ids,
}
)
create_payload: dict[str, Any] = {
"ws_id": ws.id,
"name": ws.name,
"resumed": bool(extra_response.get("resumed", False)),
"message_count": int(extra_response.get("message_count", 0)),
"attachment_ids": attachment_ids,
}
if extra_response.get("initial_message_status"):
# Present only when the post-install hook could NOT deliver
# the initial message (raced live worker, interjection queue
# full) — the workstream exists, but a bare 200 would read as
# "first message accepted". Mirrors /send's in-body
# ``queue_full`` backpressure surface.
create_payload["initial_message_status"] = str(extra_response["initial_message_status"])
return JSONResponse(create_payload)
return create
@@ -3798,6 +3804,27 @@ def make_detail_handler(cfg: SessionEndpointConfig) -> Handler:
# mismatch, and tombstoned rows — all surface as 404.
return JSONResponse({"error": cfg.not_found_label}, status_code=404)
# A detail GET that lazily rehydrates IS an open — run the
# same kind-specific post-load the open handler runs.
# Skipping it leaves the now-live session with no watch
# dispatch registration (its next watch fire would take the
# restore path and spawn a duplicate auto-approved session
# racing writes into this live conversation) and never tells
# dashboards the workstream came live (``ws_created``).
if cfg.open_post_load is not None:
try:
# Off-loop: interactive's post_load does blocking
# storage I/O (display-name lookup).
await asyncio.to_thread(cfg.open_post_load, request, ws)
except Exception:
# Post-load is observational — never let a hook bug
# block the detail response. Log + continue.
log.debug(
"ws.detail.post_load_failed ws=%s",
ws.id[:8],
exc_info=True,
)
# Pending-approval snapshot — lets a freshly-loaded chat tab
# paint the inline approval gate from this single response
# instead of waiting for the SSE approve_request replay (which
@@ -4097,6 +4124,12 @@ def make_send_handler(cfg: SessionEndpointConfig) -> Handler:
thread_name=f"send-worker-{ws.id[:8]}",
)
if not ok:
if ws._closed:
# ``send`` refused because the workstream closed between our
# resolution and the dispatch — a ``queue_full`` here would
# tell the client to retry a workstream whose very next
# resolution 404s. Mirror the resolution miss instead.
return JSONResponse({"error": cfg.not_found_label}, status_code=404)
# queue.Full or session-disappeared race — surface as
# queue_full so clients retry rather than 500. ``attached_ids``
# is always empty on this path (the dispatch never took
+69 -6
View File
@@ -18,11 +18,22 @@ with no consumer. The flag transitions atomically inside the same lock
this module holds, so both coord and interactive callers inherit the
fix.
This module owns ONLY the dispatch decision and the
``_worker_running`` lifecycle. Per-kind concerns session resolution,
This module owns ONLY the dispatch decision, the ``_worker_running``
lifecycle, and the ownership-clear wake backstop
(:func:`_retry_pending_wake`). Per-kind concerns session resolution,
attachment resolution, error surfacing, UI callbacks,
``GenerationCancelled`` handling live in the caller's
``enqueue`` / ``run`` no-arg closures.
The wake backstop exists because IDLE state fans out from INSIDE
``run()`` (``set_state`` subscribers fire on the calling thread the
worker that did the transition). Any wake the IDLE fan-out dispatches
(``IdleNudgeWatcher``) therefore lands on the reuse path while this
worker still owns the flag and no-ops; with IDLE emitted at the END of
a send there is no later seam in this worker to drain the queue, so
the nudge would strand until the next user message. Re-running the
wake gate at the exact moment ownership clears is the only spot that
closes the window without ever racing a competing worker.
"""
from __future__ import annotations
@@ -41,6 +52,41 @@ if TYPE_CHECKING:
log = get_logger(__name__)
def _retry_pending_wake(ws: Workstream) -> None:
"""Deliver nudges that arrived while the exiting worker owned *ws*.
Runs in the worker's ``finally`` immediately after it cleared
``_worker_running`` (owner only abandoned threads skip it). The
canonical strand it closes: the coordinator's ``idle_children``
nudge, enqueued by ``CoordinatorIdleObserver`` during the IDLE
fan-out at the end of the coord's send — the fan-out runs on the
worker thread, so ``IdleNudgeWatcher``'s wake dispatch hits the
reuse path and no-ops, and nothing else ever re-checks the queue.
The same window covers a watch ``wake_fn`` firing while a worker
is mid-exit.
The wake gate
(:func:`~turnstone.core.idle_nudge_watcher.wake_workstream_if_pending`)
owns every defensive check session missing, bare stub without a
NudgeQueue (watch-style dispatchers drive sessions that aren't
installed on the workstream), closed, non-idle, nothing pending
and its ``session_worker.send`` dispatch is the same atomic spawn
as any other: a successor worker claimed between our flag-clear
and the retry just downgrades the wake to a no-op enqueue again,
and THAT worker's own exit re-runs this backstop. Convergence is
owned by the producers' gates (cooldown, hard caps, ``valid_until``
predicates): a wake worker whose drain empties the queue retries
once at its own exit, sees nothing pending, and stops.
"""
# Local import: idle_nudge_watcher imports this module at top level.
from turnstone.core.idle_nudge_watcher import wake_workstream_if_pending
try:
wake_workstream_if_pending(ws, trigger="worker-exit")
except Exception:
log.warning("session_worker.wake_retry_failed ws=%s", ws.id[:8], exc_info=True)
def send(
ws: Workstream,
*,
@@ -63,10 +109,11 @@ def send(
Returns:
``True`` on successful enqueue (existing worker accepted) or
thread spawn (no live worker).
``False`` when ``enqueue`` raises ``queue.Full`` (queue at
capacity caller surfaces 429) or any other exception
(logged). Falling through to spawn a second worker on a full
queue would corrupt ChatSession state.
``False`` when the workstream is already closed (see below), or
when ``enqueue`` raises ``queue.Full`` (queue at capacity
caller surfaces 429) or any other exception (logged). Falling
through to spawn a second worker on a full queue would corrupt
ChatSession state.
"""
name = thread_name or f"session-worker-{ws.id[:8]}"
@@ -85,6 +132,7 @@ def send(
# close style signals if the runtime ever delivers them).
log.exception("session_worker.uncaught ws=%s", ws.id[:8])
finally:
was_owner = False
with ws._lock:
# Only clear the flag if THIS thread is still the current
# worker. A force-cancel abandons the worker
@@ -96,8 +144,23 @@ def send(
# spawns a second concurrent worker on the same session.
if ws.worker_thread is threading.current_thread():
ws._worker_running = False
was_owner = True
# Outside the lock (the retry's wake dispatch re-acquires it).
# Owner only: an abandoned thread retrying would race the
# successor's own exit backstop for no benefit.
if was_owner:
_retry_pending_wake(ws)
with ws._lock:
if ws._closed:
# Authoritative closed-check: ``SessionManager.close`` sets
# ``_closed`` under this same lock, so unlike the wake gate's
# lockless peek this read cannot go stale. Without it, a
# wake (or send) racing ``close()`` spawns a worker that runs
# a full unattended turn — inference, tool calls, storage
# writes — on a workstream whose ``ws_closed`` already fired.
log.info("session_worker.closed_refused ws=%s", ws.id[:8])
return False
if ws._worker_running:
try:
enqueue()
+672 -54
View File
@@ -13,6 +13,7 @@ import json
import re
import subprocess
import threading
import time
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Any
@@ -33,6 +34,49 @@ MIN_INTERVAL = 10 # seconds
MAX_INTERVAL = 86_400 # 24 hours
DEFAULT_MAX_POLLS = 100
MAX_OUTPUT_SIZE = 65_536 # truncate stored/dispatched output at 64 KB
# Cap on delivery re-attempts for a fire whose workstream can't be
# reached (evicted + transiently unrestorable — all restore slots busy).
# On exhaustion the held reminder is dropped and one poll is charged to
# the watch's own ``max_polls`` budget: with budget left the watch stays
# ACTIVE and the fire cycle repeats on its normal cadence; with the
# budget spent it deactivates (loudly) — so a persistently unreachable
# workstream can never re-run its command forever. A PERMANENT failure
# (:class:`WatchWorkstreamUnrestorable`, e.g. corrupt persona stamp)
# deactivates immediately without waiting out either budget.
MAX_DELIVERY_ATTEMPTS = 5
# Cap on the held-reminder retry delay. Re-delivery is a cheap in-memory
# dispatch (never a command re-run), so it retries on a short cadence —
# ``min(interval_secs, this)`` — rather than the watch's own interval: a
# daily watch whose fire hit a busy restore slot must not sit on its
# reminder for 24 h when the cause clears in seconds.
DELIVERY_RETRY_CAP_SECS = 60
# Cap on concurrent restore paths in flight across the poll pool. Kept
# below ``max_concurrent_polls`` so a burst of evicted-workstream fires
# can never occupy every poll slot on slow restores (leaving normal polls
# unserved) and can't drain the shared DB connection pool. A poll that
# would exceed the cap DEFERS (holds its reminder, releases its slot)
# instead of blocking.
MAX_CONCURRENT_RESTORES = 2
# Age past which an in-flight restore's admission entry is presumed
# wedged inside ``restore_fn`` and ALERTED on (error log at every
# refused admission). Ten minutes exceeds every configured storage /
# MCP timeout by an order of magnitude, so a genuine restore never
# trips it. Deliberately detection-ONLY — the entry is never evicted:
# the wedged poll thread's pool slot is never released, so reclaiming
# its admission would just readmit a restore that can wedge ANOTHER
# pool thread on the same cause, converting this capped degraded state
# (restores blocked, normal polling intact) into total poll-pool
# collapse, one slot per threshold period. Recovery from a genuine
# wedge is a process restart; the loud log is what tells the operator.
RESTORE_STALL_ALERT_SECS = 600.0
# Cap on ``stop()``'s in-flight-poll drain. Shutdown runs under an
# external deadline (systemd's stop timeout defaults to 90 s), and the
# teardown steps AFTER the watch runner — state-writer drain, node
# deregistration — must still get their turn, so the drain waits
# ``min(tool_timeout, this) + 5`` rather than a full ``tool_timeout``
# (default 120 s). An abandoned poll is loud and safe: its row stays
# due and re-polls on the next boot.
STOP_DRAIN_CAP_SECS = 30.0
# Safe builtins exposed to condition expressions.
_SAFE_BUILTINS: dict[str, Any] = {
@@ -258,6 +302,11 @@ def build_watch_reminder(
}
def _iso_in(seconds: float) -> str:
"""``now + seconds`` in the storage layer's naive-UTC second format."""
return (datetime.now(UTC) + timedelta(seconds=seconds)).strftime("%Y-%m-%dT%H:%M:%S")
def format_interval(secs: float) -> str:
"""Human-readable duration (e.g. ``'5m'``, ``'1h30m'``)."""
if secs < 60:
@@ -276,11 +325,41 @@ def format_interval(secs: float) -> str:
# ---------------------------------------------------------------------------
class WatchWorkstreamUnrestorable(Exception): # noqa: N818
"""Raised by a ``restore_fn`` when a watch's workstream can NEVER be
restored (e.g. a corrupt persona stamp the operator must fix), as
opposed to a transient failure (all restore slots busy) which returns
``None``. Signals :class:`WatchRunner` to stop retrying delivery and
deactivate the watch immediately rather than burning the whole
attempt budget on a cause that can't clear on its own.
"""
class WatchRunner:
"""Polls the database for due watches and dispatches results.
Runs as a daemon thread in the server process, analogous to
``TaskScheduler`` in the console.
Poll execution is CONCURRENT and bounded: the tick thread only
enumerates due rows and hands each to a short-lived daemon thread
gated by a semaphore (``max_concurrent_polls``), so one hung
command itself bounded by ``tool_timeout`` delays at most one
slot instead of head-of-line blocking every watch on the node. A
per-``watch_id`` in-flight set prevents double-polling rows that
keep appearing in ``list_due_watches`` while their (slow) poll is
still running. When a fire can't reach its workstream, the built
reminder is HELD and re-delivered on a short retry cadence
(``min(interval_secs, DELIVERY_RETRY_CAP_SECS)``, never re-running
the command); transient failures retry up to
:data:`MAX_DELIVERY_ATTEMPTS`, after which one poll is charged to the
watch's ``max_polls`` budget and the fire cycle repeats on its normal
cadence or, with the budget spent, the watch deactivates loudly. A
permanent :class:`WatchWorkstreamUnrestorable` deactivates it at
once. Restore is admission-controlled (per-ws_id
dedup + a :data:`MAX_CONCURRENT_RESTORES` cap, both without blocking a
poll slot) so two watches on one evicted workstream can't each spawn a
live session and a restore burst can't starve the poll pool.
"""
def __init__(
@@ -290,6 +369,7 @@ class WatchRunner:
*,
check_interval: float = 15.0,
tool_timeout: float = 30.0,
max_concurrent_polls: int = 4,
restore_fn: Callable[[str], Callable[[dict[str, Any], str], None] | None] | None = None,
) -> None:
self._storage = storage
@@ -297,10 +377,48 @@ class WatchRunner:
self._check_interval = check_interval
self._tool_timeout = tool_timeout
self._restore_fn = restore_fn
# Bounded poll concurrency (see class docstring). The slot is
# acquired on the tick thread and released in ``_poll_one``'s
# ``finally``; the in-flight set is keyed by watch_id and holds
# entries for exactly the lifetime of their poll thread.
self._poll_slots = threading.BoundedSemaphore(max_concurrent_polls)
self._in_flight: set[str] = set()
self._in_flight_lock = threading.Lock()
self._dispatch_fns: dict[str, Callable[[dict[str, Any], str], None]] = {}
self._dispatch_lock = threading.Lock()
# Restore admission control. The restore path (``manager.create``
# + ``session.resume``) must not run twice for one ws_id, or two
# watches on the same evicted workstream — polled on separate pool
# threads — would each spawn a live auto-approved session racing
# writes into one conversation history. ``_restoring`` tracks the
# ws_ids with a restore in flight; ``_restore_lock`` guards it but
# is held only for the fast admit/reject check, NEVER across the
# slow restore (which would pin the caller's poll slot and starve
# the pool). A poll is admitted only when its ws_id isn't already
# restoring AND fewer than :data:`MAX_CONCURRENT_RESTORES` restores
# are in flight; otherwise it DEFERS (holds its reminder, releases
# its slot) and re-delivers on the capped retry cadence — by which
# point the winning restore has registered a dispatch fn. Values
# are ``time.monotonic()`` admission stamps, used ONLY to alert on
# wedged restores (:data:`RESTORE_STALL_ALERT_SECS`); entries are
# removed solely by their own restore's ``finally``.
self._restoring: dict[str, float] = {}
self._restore_lock = threading.Lock()
# Held reminders whose delivery failed, keyed by watch_id. Value:
# ``{"reminder", "update_fields", "attempts"}``. Held deliveries
# are always TERMINAL fires (``fired ⟹ is_final``, and reminders
# are only built for final polls), so committing ``update_fields``
# always deactivates the row. A later tick re-DELIVERS the
# reminder (never re-runs the command, so a transient stop_on
# match survives) up to :data:`MAX_DELIVERY_ATTEMPTS`. Access is
# guarded, though the per-watch_id in-flight gate already
# serialises pollers for a given watch_id.
self._pending_delivery: dict[str, dict[str, Any]] = {}
self._pending_delivery_lock = threading.Lock()
# Watch ids whose terminal reminder has already been dispatched
# but whose row write has not yet been confirmed. Populated
# between ``_dispatch_result`` and ``update_watch`` in
@@ -330,6 +448,31 @@ class WatchRunner:
if self._thread is not None:
self._thread.join(timeout=self._check_interval + 5)
self._thread = None
# Drain in-flight polls so their storage writes land before
# teardown. A poll's command can block up to ``tool_timeout``, so
# bound the wait on THAT (not the tick cadence): a poll that
# started just before ``stop()`` may legitimately still be running
# its command, and a ``check_interval``-based deadline would
# abandon it mid-run and skip its commit. The wait is capped at
# :data:`STOP_DRAIN_CAP_SECS` though — shutdown itself runs under
# an external deadline (systemd stop timeout), and the teardown
# steps queued after us must still run. On the deadline we
# abandon loudly; the daemon threads die with the process and the
# abandoned rows stay due (re-polled on next boot).
deadline = time.monotonic() + min(self._tool_timeout, STOP_DRAIN_CAP_SECS) + 5
while time.monotonic() < deadline:
with self._in_flight_lock:
if not self._in_flight:
break
time.sleep(0.05)
else:
with self._in_flight_lock:
leftover = len(self._in_flight)
# Only warn on a genuine abandonment — the last poll can drain
# in the same window the deadline is crossed, and a count=0
# warning would be a false alarm for log-based monitoring.
if leftover:
log.warning("watch_runner.stop_abandoned_polls count=%d", leftover)
log.info("watch_runner.stopped")
# -- Dispatch function registry ------------------------------------------
@@ -353,8 +496,21 @@ class WatchRunner:
with self._dispatch_lock:
self._dispatch_fns[ws_id] = fn
def remove_dispatch_fn(self, ws_id: str) -> None:
def remove_dispatch_fn(
self, ws_id: str, owner: Callable[[dict[str, Any], str], None] | None = None
) -> None:
"""Remove the registration for ``ws_id`` — with ``owner`` given,
ONLY if the registered fn IS that closure. Multiple live
sessions can transiently serve one ws_id (a watch-restore shell
vs a reopened pane; an in-session ``/resume`` of an id open in
another pane), and a blind removal from one session's teardown
would silently unregister the OTHER, still-live session its
next fire would then take the restore path and spawn a duplicate
auto-approved session onto the live conversation.
"""
with self._dispatch_lock:
if owner is not None and self._dispatch_fns.get(ws_id) is not owner:
return
self._dispatch_fns.pop(ws_id, None)
def get_dispatch_fn(self, ws_id: str) -> Callable[[dict[str, Any], str], None] | None:
@@ -367,8 +523,9 @@ class WatchRunner:
return self._dispatch_fns.get(ws_id)
def forget_terminal_dispatched(self, watch_id: str) -> None:
"""Discard ``watch_id`` from the pending-terminal-dispatched
set if present. Called by paths that take a watch out of
"""Discard ``watch_id`` from the runner's per-watch transient
state (terminal-dispatched set AND any held pending delivery).
Called by paths that take a watch out of
:meth:`StorageBackend.list_due_watches` view independent of
the runner's own poll (most importantly the user-cancel path
in :meth:`ChatSession._exec_watch`). Without this, a
@@ -377,10 +534,21 @@ class WatchRunner:
the user-cancel writes ``next_poll=''`` which excludes the
row from ``list_due_watches``, so the retry-deactivate branch
at the top of :meth:`_poll_watch` never fires to clear the
entry.
entry. Held reminders are dropped for the same reason: a
cancelled watch's row leaves the due view, so its pending
re-delivery would never be retried and would leak.
Call this AFTER the row write that takes the watch out of the
active view: the delivery paths re-check ``is_watch_active``
before stashing or dispatching, so with the write already
visible a racing poll thread drops its own hold instead of
re-stashing behind this clear. (The residual
check-before-write / stash-after-clear interleaving is mopped
up by :meth:`_sweep_cancelled_holds` within one tick.)
"""
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
self._clear_pending_delivery(watch_id)
# -- Main loop -----------------------------------------------------------
@@ -399,6 +567,7 @@ class WatchRunner:
def _tick(self) -> None:
if self._storage is None:
return
self._sweep_cancelled_holds()
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
due = self._storage.list_due_watches(now)
for watch_row in due:
@@ -408,13 +577,60 @@ class WatchRunner:
row_node = watch_row.get("node_id", "")
if row_node and row_node != self._node_id:
continue
watch_id = str(watch_row.get("watch_id", ""))
with self._in_flight_lock:
if watch_id in self._in_flight:
# Still polling from a previous tick (slow command) —
# the row keeps listing as due until its update
# commits; don't double-poll it.
continue
self._in_flight.add(watch_id)
if not self._poll_slots.acquire(blocking=False):
# Pool saturated: the remaining due rows stay due and the
# next tick retries them — nothing is dropped, delivery is
# just deferred by up to ``check_interval``.
with self._in_flight_lock:
self._in_flight.discard(watch_id)
log.debug("watch_runner.poll_slots_saturated")
break
try:
self._poll_watch(watch_row)
threading.Thread(
target=self._poll_one,
args=(watch_row,),
daemon=True,
name=f"watch-poll-{watch_id[:8]}",
).start()
except Exception:
log.exception(
"watch_runner.poll_error",
extra={"watch_id": watch_row.get("watch_id")},
)
# Spawn failure (e.g. OS thread exhaustion) must not leak
# the slot or the in-flight entry, and must NOT abort the
# rest of this tick — the failed watch stays due and
# retries next tick while its siblings still get polled
# (a raise here would unwind out of the un-guarded
# due-row loop and skip every remaining due watch).
with self._in_flight_lock:
self._in_flight.discard(watch_id)
self._poll_slots.release()
log.exception("watch_runner.poll_spawn_failed", extra={"watch_id": watch_id})
continue
def _poll_one(self, watch_row: dict[str, Any]) -> None:
"""Run one ``_poll_watch`` on a pool thread.
Always releases the slot and the in-flight entry, even on an
unexpected raise a leaked slot would shrink the pool for the
rest of the process lifetime.
"""
try:
self._poll_watch(watch_row)
except Exception:
log.exception(
"watch_runner.poll_error",
extra={"watch_id": watch_row.get("watch_id")},
)
finally:
with self._in_flight_lock:
self._in_flight.discard(str(watch_row.get("watch_id", "")))
self._poll_slots.release()
def _poll_watch(self, watch_row: dict[str, Any]) -> None:
watch_id = watch_row["watch_id"]
@@ -437,10 +653,33 @@ class WatchRunner:
self._storage.update_watch(watch_id, active=False, next_poll="")
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
# A held reminder for an already-dispatched watch means a
# re-delivery SUCCEEDED but its row-commit raised (leaving
# the id in both sets). The reminder is delivered, so drop
# the hold here — otherwise, once we deactivate the row it
# never re-lists and the entry would leak forever.
self._clear_pending_delivery(watch_id)
except Exception:
log.exception("watch_runner.retry_deactivate_failed", extra={"watch_id": watch_id})
return
# Held-reminder re-delivery: a prior poll fired but couldn't reach
# this watch's workstream. Re-DELIVER the stashed reminder without
# re-running the command (so a stop_on match that was momentarily
# true isn't lost to a fresh run), bounded by MAX_DELIVERY_ATTEMPTS.
# The per-watch_id in-flight gate in _tick serialises POLLERS for
# this watch_id; the user-cancel path's forget_terminal_dispatched
# runs on a worker thread and CAN race this peek — which is why
# _redeliver_pending re-checks the row's active state before
# dispatching, the hold paths re-check before stashing, and
# _sweep_cancelled_holds mops up any stash that still lands after
# a cancel's clear.
with self._pending_delivery_lock:
pending = self._pending_delivery.get(watch_id)
if pending is not None:
self._redeliver_pending(watch_row, pending)
return
# Safety check
blocked = is_command_blocked(command)
if blocked:
@@ -474,6 +713,23 @@ class WatchRunner:
now = datetime.now(UTC)
now_str = now.strftime("%Y-%m-%dT%H:%M:%S")
# Build the row update this poll intends to commit, UP FRONT, so a
# delivery failure can stash it verbatim alongside the reminder
# (see :meth:`_redeliver_pending`) instead of recomputing it at
# re-delivery time.
update_fields: dict[str, Any] = {
"poll_count": poll_count,
"last_output": output,
"last_exit_code": exit_code,
"last_poll": now_str,
}
if is_final:
update_fields["active"] = False
update_fields["next_poll"] = ""
else:
next_poll = now + timedelta(seconds=watch_row["interval_secs"])
update_fields["next_poll"] = next_poll.strftime("%Y-%m-%dT%H:%M:%S")
# Dispatch before committing the row update. Belt-and-braces
# given the rest of the fix (closure no longer wires a
# ``valid_until`` predicate, cancel-by-name uses
@@ -485,7 +741,12 @@ class WatchRunner:
# ``_terminal_dispatched`` guard above it also bounds the
# duplicate-fire blast radius if the row write fails after the
# reminder shipped.
if fired or is_final:
#
# ``fired ⟹ is_final`` (see the assignment above), so reminders
# are built ONLY for terminal polls — the held-delivery machinery
# relies on that (a committed ``update_fields`` always
# deactivates).
if is_final:
# Compute elapsed from created time
elapsed_secs = 0.0
if created:
@@ -506,34 +767,34 @@ class WatchRunner:
is_final=is_final,
reason=reason,
)
self._dispatch_result(ws_id, reminder, watch_id)
if is_final:
# Mark BEFORE the row write so a raise below routes the
# next tick into the retry-deactivate branch instead of
# re-firing the reminder.
with self._terminal_dispatched_lock:
self._terminal_dispatched.add(watch_id)
# Update DB
update_fields: dict[str, Any] = {
"poll_count": poll_count,
"last_output": output,
"last_exit_code": exit_code,
"last_poll": now_str,
}
if is_final:
update_fields["active"] = False
update_fields["next_poll"] = ""
try:
delivered = self._dispatch_result(ws_id, reminder, watch_id)
except WatchWorkstreamUnrestorable:
# Permanent: the workstream can never be restored (corrupt
# persona stamp / history gone). Stash BEFORE the abandon
# write — not for delivery (there is nowhere to deliver),
# but so a failing deactivation write routes the next tick
# into the redeliver path (which retries the WRITE) instead
# of the still-active row re-listing into a fresh command
# run every tick with the budget never advancing. On a
# successful write, _abandon_delivery clears the stash
# immediately (write-then-clear).
self._stash_pending_delivery(watch_id, reminder, update_fields, attempts=1)
self._abandon_delivery(watch_id, ws_id, update_fields, reason="unrestorable")
return
if not delivered:
# Transiently undeliverable (ws evicted + slots busy, or
# restore admission deferred). HOLD the built reminder +
# its intended row update for re-delivery on the capped
# retry cadence, advancing next_poll and durably charging
# this fire's poll — no baseline advance and no command
# re-run (which would lose a transient stop_on match);
# see :meth:`_hold_delivery` for why the charge commits.
self._hold_delivery(watch_row, reminder, update_fields, attempts=1)
return
self._commit_terminal_update(watch_id, update_fields)
else:
next_poll = now + timedelta(seconds=watch_row["interval_secs"])
update_fields["next_poll"] = next_poll.strftime("%Y-%m-%dT%H:%M:%S")
self._storage.update_watch(watch_id, **update_fields)
if is_final:
# Row write committed; the retry-deactivate branch will
# never be reached for this watch_id.
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
self._storage.update_watch(watch_id, **update_fields)
log.debug(
"watch_runner.polled",
@@ -568,38 +829,395 @@ class WatchRunner:
except Exception as exc:
return f"[command failed: {exc}]", -1
def _dispatch_result(self, ws_id: str, reminder: dict[str, Any], watch_id: str) -> None:
def _try_dispatch_fn(self, ws_id: str, reminder: dict[str, Any], watch_id: str) -> bool | None:
"""Deliver via the registered dispatch fn, if one exists.
Returns ``True`` (delivered), ``False`` (a fn is registered but it
raised the ws is live, so the caller must NOT restore, which
would spawn a duplicate session), or ``None`` (no fn registered
the ws may be evicted and the caller should try to restore).
"""
with self._dispatch_lock:
fn = self._dispatch_fns.get(ws_id)
if fn is None:
return None
try:
fn(reminder, watch_id)
return True
except Exception:
log.exception("watch_runner.dispatch_error", extra={"ws_id": ws_id})
return False
def _dispatch_result(self, ws_id: str, reminder: dict[str, Any], watch_id: str) -> bool:
"""Deliver a watch result to the owning workstream.
``reminder`` is the structured dict produced by
:func:`build_watch_reminder` ``text`` is the formatted body
(matched by the dispatch closure's :func:`sanitize_payload`
pass) and the optional fields ride as queue-entry metadata.
Returns ``True`` iff a dispatch closure ran without raising the
registered one, or the one the restore path produced. ``False``
means the reminder was not delivered for a TRANSIENT reason (no fn
+ slots busy, restore admission deferred, or a live fn raised); the
caller HOLDS the reminder and re-delivers on the capped retry
cadence. Raises
:class:`WatchWorkstreamUnrestorable` for a PERMANENT failure so the
caller can deactivate the watch instead of retrying.
"""
with self._dispatch_lock:
fn = self._dispatch_fns.get(ws_id)
delivered = self._try_dispatch_fn(ws_id, reminder, watch_id)
if delivered is not None:
return delivered
if fn is not None:
try:
fn(reminder, watch_id)
return
except Exception:
log.exception("watch_runner.dispatch_error", extra={"ws_id": ws_id})
# No dispatch fn — the workstream may be evicted. Admit a restore
# only when this ws isn't already restoring and the concurrent-
# restore cap has room; hold ``_restore_lock`` for that fast check
# ONLY (never across the slow restore, which would pin this poll
# slot). A rejected poll defers (holds + re-delivers on the
# capped retry cadence).
if self._restore_fn is None:
log.warning(
"watch_runner.dispatch_failed",
extra={"ws_id": ws_id, "reason": "no dispatch fn and no restore_fn"},
)
return False
# Workstream may be evicted — try to restore
if self._restore_fn is not None:
deliver_after_admission = False
with self._restore_lock:
# Re-check: a restore that completed while we waited for the
# lock may already have registered a fn for this ws_id. This
# is a PRESENCE check only — the dispatch closure itself can
# block (it takes ``ws._lock`` and may spawn a wake thread),
# and running it here would serialise every restore admission
# on the node behind one delivery. (Lock order: _restore_lock
# → _dispatch_lock via get_dispatch_fn; nothing takes them in
# the reverse order.)
if self.get_dispatch_fn(ws_id) is not None:
deliver_after_admission = True
else:
# Alert on admission entries past the stall threshold —
# a restore wedged inside ``restore_fn`` (its poll thread
# is blocked, so the release in the ``finally`` below can
# never run) holds this capacity for the process
# lifetime. Detection ONLY: see
# :data:`RESTORE_STALL_ALERT_SECS` for why reclaiming the
# entry would make the failure strictly worse.
now = time.monotonic()
for rid, started in self._restoring.items():
if now - started > RESTORE_STALL_ALERT_SECS:
log.error(
"watch_runner.restore_admission_wedged",
extra={"ws_id": rid, "stalled_secs": round(now - started, 1)},
)
if ws_id in self._restoring or len(self._restoring) >= MAX_CONCURRENT_RESTORES:
# Same-ws restore already running, or the pool is at
# its restore cap — defer rather than block a poll
# slot.
return False
self._restoring[ws_id] = now
if deliver_after_admission:
# The fn appeared while we waited: deliver OUTSIDE the lock.
# ``None`` (it vanished again — an eviction race) defers like
# any other transient.
return bool(self._try_dispatch_fn(ws_id, reminder, watch_id))
try:
restored_fn = self._restore_fn(ws_id) # may raise WatchWorkstreamUnrestorable
except WatchWorkstreamUnrestorable:
raise # permanent — caller deactivates the watch
except Exception:
log.exception("watch_runner.restore_error", extra={"ws_id": ws_id})
return False
finally:
with self._restore_lock:
self._restoring.pop(ws_id, None)
if restored_fn is not None:
try:
restored_fn = self._restore_fn(ws_id)
if restored_fn is not None:
restored_fn(reminder, watch_id)
return
restored_fn(reminder, watch_id)
return True
except Exception:
log.exception("watch_runner.restore_error", extra={"ws_id": ws_id})
log.exception("watch_runner.restore_dispatch_error", extra={"ws_id": ws_id})
return False
log.warning(
"watch_runner.dispatch_failed",
extra={"ws_id": ws_id, "reason": "no dispatch function and restore failed"},
extra={"ws_id": ws_id, "reason": "restore produced no dispatch fn"},
)
return False
# -- Held-reminder re-delivery -------------------------------------------
#
# Everything held here is a TERMINAL fire (``fired ⟹ is_final`` in
# ``_poll_watch``, and reminders are built only for final polls), so a
# committed ``update_fields`` always deactivates the row. The delivery
# outcome state machine — commit-with-terminal-mark, abandon, hold —
# lives in the three helpers below so ``_poll_watch`` (fresh fire) and
# ``_redeliver_pending`` can't drift apart on the fragile ordering.
def _watch_still_active(self, watch_id: str) -> bool:
"""``True`` unless the row has CLEANLY left the active view (user
cancel / deletion). A storage error biases toward ``True``:
delivery paths keep retrying on a blip bounded by their own
attempt and poll budgets rather than dropping a fire.
"""
try:
return bool(self._storage.is_watch_active(watch_id))
except Exception:
return True
def _stash_pending_delivery(
self,
watch_id: str,
reminder: dict[str, Any],
update_fields: dict[str, Any],
*,
attempts: int,
) -> None:
"""Hold ``reminder`` + its intended row update for re-delivery."""
with self._pending_delivery_lock:
self._pending_delivery[watch_id] = {
"reminder": reminder,
"update_fields": update_fields,
"attempts": attempts,
}
def _clear_pending_delivery(self, watch_id: str) -> None:
with self._pending_delivery_lock:
self._pending_delivery.pop(watch_id, None)
def _commit_terminal_update(self, watch_id: str, update_fields: dict[str, Any]) -> None:
"""Commit a DELIVERED terminal fire's row update with the
``_terminal_dispatched`` mark held across the write: if the write
raises, the next tick routes into the retry-deactivate branch at
the top of :meth:`_poll_watch` instead of re-firing a reminder the
model already saw.
"""
with self._terminal_dispatched_lock:
self._terminal_dispatched.add(watch_id)
self._storage.update_watch(watch_id, **update_fields)
# Row write committed; the retry-deactivate branch will never be
# reached for this watch_id.
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
def _abandon_delivery(
self,
watch_id: str,
ws_id: str,
update_fields: dict[str, Any],
*,
reason: str,
attempts: int | None = None,
) -> None:
"""Give up on delivering this fire: commit the intended row
update (deactivating the terminal watch), then drop any held
reminder. Write-then-clear: a failed commit leaves the hold in
place, so the next tick retries via the REDELIVER path (dispatch
same terminal outcome retry this write) without re-running
the command clearing first would let the row re-list into a
fresh command run every attempt-budget cycle, forever, whenever
storage can read but not write (e.g. disk-full SQLite), with the
poll budget never advancing. The clear itself is pure in-memory
and cannot fail after a successful commit, so no ordering leaks
the hold.
"""
self._storage.update_watch(watch_id, **update_fields)
self._clear_pending_delivery(watch_id)
extra: dict[str, Any] = {"watch_id": watch_id, "ws_id": ws_id, "reason": reason}
if attempts is not None:
extra["attempts"] = attempts
log.error("watch_runner.delivery_abandoned", extra=extra)
def _hold_delivery(
self,
watch_row: dict[str, Any],
reminder: dict[str, Any],
update_fields: dict[str, Any],
*,
attempts: int,
) -> None:
"""Stash the reminder + intended update and advance ``next_poll``
(by the capped retry delay) so the row re-lists for re-delivery
soon without advancing its baseline or running its command.
Re-delivery is a cheap in-memory dispatch, so it retries on
``min(interval_secs, DELIVERY_RETRY_CAP_SECS)`` rather than the
watch's own interval: a daily watch must not sit on its fired
reminder for 24 h because a restore slot was briefly busy.
The poll charge (``update_fields["poll_count"]``) is committed
alongside ``next_poll``: the hold itself lives only in this
process, so a restart mid-hold re-lists the row and re-runs the
(possibly side-effectful) command with the charge durable those
re-runs stay bounded by the watch's own ``max_polls``, matching
the in-memory exhaustion path, plus at most ONE regeneration run
per restart when the held fire had already spent the budget (the
re-list runs the command before the cap check so the lost
reminder is regenerated rather than silently dropped). The
baseline (``last_output``) stays uncommitted so a delta-style
``stop_on`` re-detects the change the model never saw.
Dropped instead when the row has left the active view: the user
cancelled while this fire was in flight, and a stash landing
after the cancel path's :meth:`forget_terminal_dispatched` would
leak the hold forever (an inactive row never re-lists to retry
it) and violate that method's drop guarantee.
"""
watch_id = watch_row["watch_id"]
ws_id = watch_row["ws_id"]
if not self._watch_still_active(watch_id):
self._clear_pending_delivery(watch_id)
log.info(
"watch_runner.hold_dropped_cancelled",
extra={"watch_id": watch_id, "ws_id": ws_id, "attempts": attempts},
)
return
retry_secs = min(int(watch_row["interval_secs"]), DELIVERY_RETRY_CAP_SECS)
self._stash_pending_delivery(watch_id, reminder, update_fields, attempts=attempts)
retry_poll = _iso_in(retry_secs)
self._storage.update_watch(
watch_id,
next_poll=retry_poll,
poll_count=int(update_fields["poll_count"]),
)
log.warning(
"watch_runner.delivery_deferred",
extra={
"watch_id": watch_id,
"ws_id": ws_id,
"attempts": attempts,
"next_retry": retry_poll,
},
)
def _redeliver_pending(self, watch_row: dict[str, Any], pending: dict[str, Any]) -> None:
"""Re-attempt delivery of a held reminder WITHOUT re-running the
command.
Five outcomes:
* **Watch cancelled** the user-cancel path raced the due
listing. Drop the hold and deliver nothing: cancel's
:meth:`forget_terminal_dispatched` promises the held reminder
is dropped, and delivering here could even RESTORE a session
for a watch the user just cancelled.
* **Delivered** commit the row update stashed at fire time (which
deactivates the terminal watch) and clear the hold. The hold is
cleared BEFORE the commit so that a commit failure can't strand
it (the row then re-lists and the ``already_dispatched`` guard
finishes deactivation).
* **Permanent failure** (:class:`WatchWorkstreamUnrestorable`)
deactivate the watch now and drop the reminder; it can never be
delivered.
* **Transient failure past** :data:`MAX_DELIVERY_ATTEMPTS`, poll
budget remaining drop the held reminder, charge ONE poll to the
watch's ``max_polls`` budget, and leave it ACTIVE on its normal
cadence: a temporary cause (restore slots saturated under load)
doesn't silently turn the watch off, it re-fires and re-attempts
delivery next interval. The baseline (``last_output``) is
deliberately NOT committed, so a delta-style ``stop_on`` re-fires
on the same change the model never saw.
* **Transient exhaustion with the poll budget spent** commit the
held update (deactivates). Without this bound a persistently
unreachable workstream would re-run its command every interval
forever, past the user's own ``max_polls``.
"""
watch_id = watch_row["watch_id"]
ws_id = watch_row["ws_id"]
reminder = pending["reminder"]
update_fields = pending["update_fields"]
if not self._watch_still_active(watch_id):
self._clear_pending_delivery(watch_id)
log.info(
"watch_runner.redelivery_dropped_cancelled",
extra={"watch_id": watch_id, "ws_id": ws_id},
)
return
try:
delivered = self._dispatch_result(ws_id, reminder, watch_id)
except WatchWorkstreamUnrestorable:
self._abandon_delivery(watch_id, ws_id, update_fields, reason="unrestorable")
return
if delivered:
# Clear the hold FIRST: if the commit below raises, the row
# re-lists and the ``already_dispatched`` branch deactivates it
# — with the hold already gone there's nothing to leak.
self._clear_pending_delivery(watch_id)
self._commit_terminal_update(watch_id, update_fields)
log.info(
"watch_runner.delivery_recovered",
extra={"watch_id": watch_id, "ws_id": ws_id},
)
return
attempts = int(pending["attempts"]) + 1
if attempts >= MAX_DELIVERY_ATTEMPTS:
poll_count = int(update_fields.get("poll_count", 0))
max_polls = int(watch_row.get("max_polls", DEFAULT_MAX_POLLS))
if poll_count >= max_polls:
# Poll budget spent — deactivate rather than re-run the
# command forever against an unreachable workstream.
self._abandon_delivery(
watch_id,
ws_id,
update_fields,
reason="poll_budget_exhausted",
attempts=attempts,
)
return
# Budget remains: charge this cycle's poll, then drop the hold
# and let the watch re-fire on its own cadence. next_poll uses
# the FULL interval (a fresh command cycle, not a cheap
# re-delivery) and the baseline stays uncommitted so the fire
# re-detects. Write-then-clear, mirroring _abandon_delivery: a
# failed charge commit keeps the hold, so the next tick retries
# THIS branch instead of the row re-listing into a fresh
# command run with the budget never advancing.
self._storage.update_watch(
watch_id,
poll_count=poll_count,
next_poll=_iso_in(int(watch_row["interval_secs"])),
)
self._clear_pending_delivery(watch_id)
log.warning(
"watch_runner.delivery_abandoned",
extra={
"watch_id": watch_id,
"ws_id": ws_id,
"attempts": attempts,
"reason": "transient_exhausted",
"watch_active": True,
"poll_count": poll_count,
"max_polls": max_polls,
},
)
return
self._hold_delivery(watch_row, reminder, update_fields, attempts=attempts)
def _sweep_cancelled_holds(self) -> None:
"""Drop held deliveries whose rows have left the active view.
The cancel paths call :meth:`forget_terminal_dispatched`, but a
poll thread that already passed its own active re-check can
re-stash a hold microseconds AFTER that clear no ordering
between the cancel's row write and the in-memory stash can
prevent it without a per-watch lock spanning storage I/O. An
inactive row never re-lists, so nothing else would ever retry or
drop such an entry; this tick-time sweep bounds the leak (and
any post-cancel redelivery) to one ``check_interval``. Iterates
only currently-held ids holds are rare and short-lived so
the steady-state per-tick cost is zero storage reads.
"""
with self._pending_delivery_lock:
held_ids = list(self._pending_delivery)
for watch_id in held_ids:
if not self._watch_still_active(watch_id):
self._clear_pending_delivery(watch_id)
log.info("watch_runner.hold_swept_cancelled", extra={"watch_id": watch_id})
def _deactivate_watch(self, watch_id: str) -> None:
self._storage.update_watch(watch_id, active=False, next_poll="")
+236 -54
View File
@@ -33,7 +33,7 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Iterable
from collections.abc import Callable, Iterable
from sse_starlette import EventSourceResponse
from starlette.applications import Starlette
@@ -55,6 +55,7 @@ from turnstone.core.auth import (
_DenyFilter,
jwt_version_slot,
)
from turnstone.core.idle_nudge_watcher import wake_workstream_if_pending
from turnstone.core.log import get_logger
from turnstone.core.metrics import metrics as _metrics
from turnstone.core.ratelimit import resolve_client_ip
@@ -772,6 +773,19 @@ def _interactive_events_replay(
yield {"type": "intent_verdict", **v}
def _watch_fire_wake_fn(ws: Workstream) -> Callable[[], object]:
"""Wake closure for watch fires, for ``ChatSession.set_watch_runner``.
Closes over the Workstream OBJECT, never its id: after an
eviction+restore the manager tracks the workstream under a fresh id
while the watch rows keep the resumed session's ``_ws_id``, so an
id-keyed ``manager.get`` lookup at fire time would miss. Shared by
every ``set_watch_runner`` site (create, reopen, watch-restore,
CLI ``--resume``) so they can't drift on that subtlety.
"""
return lambda: wake_workstream_if_pending(ws, trigger="watch-fire")
def _interactive_open_post_load(request: Request, ws: Workstream) -> None:
"""Post-load hook for the lifted interactive ``open`` body.
@@ -794,6 +808,13 @@ def _interactive_open_post_load(request: Request, ws: Workstream) -> None:
handler-side emission is the load-bearing path on interactive;
``InteractiveAdapter.emit_rehydrated`` is a no-op stub
precisely because this enqueue lives here.
4. Re-wire the watch dispatch registration. The workstream's
previous ``close()`` removed its registration, and nothing on
the ``open`` path restored it so a watch firing on a
REOPENED, actively-viewed workstream found no dispatch fn and
took the restore path, spawning a duplicate auto-approved
session that raced turns into the same conversation the live
one was showing.
"""
from turnstone.core.memory import get_workstream_display_name
@@ -803,6 +824,12 @@ def _interactive_open_post_load(request: Request, ws: Workstream) -> None:
if isinstance(ui, WebUI) and session is not None and session.messages:
ui._enqueue({"type": "clear_ui"})
runner = getattr(request.app.state, "watch_runner", None)
if runner is not None and session is not None:
# ``mgr.open`` already resumed, so this keys the registration on
# the adopted id (and ``resume()`` re-registers by itself anyway).
session.set_watch_runner(runner, wake_fn=_watch_fire_wake_fn(ws))
gq: queue.Queue[dict[str, Any]] | None = getattr(request.app.state, "global_queue", None)
if gq is not None:
with contextlib.suppress(queue.Full):
@@ -2104,7 +2131,7 @@ async def _interactive_create_post_install(
ws.ui.auto_approve = True
runner = getattr(request.app.state, "watch_runner", None)
if runner and ws.session:
ws.session.set_watch_runner(runner)
ws.session.set_watch_runner(runner, wake_fn=_watch_fire_wake_fn(ws))
gq: queue.Queue[dict[str, Any]] = request.app.state.global_queue
# Emit ``ws_created`` on the global queue for SSE consumers
# (console). Held until past attachment validation in the
@@ -2228,7 +2255,9 @@ async def _interactive_create_post_install(
# Initial-message worker thread.
initial_message = body.get("initial_message", "").strip()
initial_message_status = ""
if initial_message and ws.session is not None:
from turnstone.core import session_worker
from turnstone.core.attachments import (
resolve_staged_attachments as _resolve_staged,
)
@@ -2236,20 +2265,13 @@ async def _interactive_create_post_install(
session = ws.session
send_id = uuid.uuid4().hex
resolved_atts: list[Any] = []
staged_ord: list[str] = []
if attachment_ids:
# Resolve (peek) the staged uploads, then drain them from the buffer
# now. The inlined first turn is their only consumer and it always
# commits at create (no queue rejection by construction), so leaving
# them staged would let the freshly-opened pane's rehydrate race the
# worker's write-time drain and paint them as still-pending composer
# chips. ``_append_user_turn``'s own per-id discard then no-ops.
resolved_atts, _ord, _drop = _resolve_staged(attachment_ids, ws.id, uid)
if _ord:
from turnstone.core.attachment_buffer import get_attachment_buffer
_buf = get_attachment_buffer()
for _aid in _ord:
_buf.discard(_aid, ws_id=ws.id, user_id=uid)
# Resolve (peek) the staged uploads. The buffer DRAIN happens
# after the dispatch below, and only on the spawn path — the
# enqueue path can't deliver attachments, so there they must
# stay staged (see ``_enqueue_init``).
resolved_atts, staged_ord, _drop = _resolve_staged(attachment_ids, ws.id, uid)
def _run_initial() -> None:
try:
@@ -2268,31 +2290,89 @@ async def _interactive_create_post_install(
_fire_notify_targets(ws, last_content)
except Exception:
log.warning("notify_completion.hook_error", ws_id=ws.id, exc_info=True)
with ws._lock:
# Only clear the flag if THIS thread is still the current
# worker — a force-cancel abandons this thread and a
# follow-up send may already have spawned a successor; an
# abandoned initial-send worker finishing late must not
# clobber that successor's running flag (mirrors the guard
# in session_worker._runner).
if ws.worker_thread is threading.current_thread():
ws._worker_running = False
# Inlined rather than via ``session_worker.send`` because at
# workstream creation no live worker can exist by
# construction — the enqueue branch of the shared dispatch
# is dead code here. ``_worker_running`` + ``ws.worker_thread``
# are set together under ``ws._lock`` so a path-keyed send
# arriving immediately after creation observes the running
# state via the shared session_worker gate instead of racing
# into a parallel worker.
with ws._lock:
ws._worker_running = True
t = threading.Thread(target=_run_initial, daemon=True, name=f"ws-init-{ws.id[:8]}")
ws.worker_thread = t
t.start()
init_enqueued = False
return {"resumed": resumed, "message_count": message_count}
def _enqueue_init() -> None:
# Reached only if a worker already owns this freshly-created ws
# — possible solely with a caller-supplied ws_id raced by a
# concurrent /send. Don't drop the user's first message: queue
# its TEXT as an interjection so the live worker delivers it.
# Attachments can't ride the interjection seam (queue_message
# rejects them), so they stay STAGED instead (the drain below
# is skipped on this branch): the composer keeps showing them
# as pending chips and the user's next send delivers them.
#
# No try/except: ``queue.Full`` must reach
# ``session_worker.send``'s backpressure branch so it returns
# ``False`` — the same surface the /send path reports as
# ``queue_full`` — instead of this create responding as if the
# message were delivered.
nonlocal init_enqueued
init_enqueued = True
session.queue_message(initial_message)
if resolved_atts:
log.warning(
"ws_init.live_worker_at_creation ws=%s%d attachment(s) left staged "
"(cannot ride the interjection seam); message text queued, attachments "
"stay in the composer for the next send",
ws.id[:8],
len(resolved_atts),
)
# Routed through ``session_worker.send`` so the init worker
# inherits ``_runner``'s ownership-clear wake backstop
# (``_retry_pending_wake``): a nudge enqueued during the initial
# send (e.g. a watch fire on a schedule-created workstream) no
# longer strands until the next user message. The enqueue branch
# is unreachable by construction (no worker can own a ws at
# creation) unless a caller-supplied ws_id is raced — hence
# ``_enqueue_init`` preserves the message and leaves the staged
# attachments recoverable instead of assuming the branch is dead.
init_ok = session_worker.send(
ws,
enqueue=_enqueue_init,
run=_run_initial,
thread_name=f"ws-init-{ws.id[:8]}",
)
if not init_ok:
# Two refusal shapes: the raced live worker's interjection
# queue rejected the text (``init_enqueued=True`` — queue.Full,
# the condition /send surfaces as ``queue_full``), or ``send``
# refused outright because the workstream was closed under our
# feet (``init_enqueued=False`` — a create raced by an off-loop
# close). Either way the workstream exists and the first
# message was NOT delivered; say so instead of answering as if
# it were. Attachments stay staged on both paths (the drain
# below is gated on ``init_ok``) so the composer chips survive
# for the user's retry.
initial_message_status = "queue_full" if init_enqueued else "refused_closed"
log.error(
"ws_init.initial_message_dropped ws=%s (%s)",
ws.id[:8],
initial_message_status,
)
if staged_ord and not init_enqueued and init_ok:
# Spawn path took the message: drain the staged copies NOW,
# before this handler returns — the pane's rehydrate can only
# start after it receives this response, so it can never
# observe the consumed uploads as still-pending composer
# chips. (``enqueue`` runs synchronously inside ``send``, so
# ``init_enqueued`` is settled here.) ``_append_user_turn``'s
# own per-id discard then no-ops.
from turnstone.core.attachment_buffer import get_attachment_buffer
_buf = get_attachment_buffer()
for _aid in staged_ord:
_buf.discard(_aid, ws_id=ws.id, user_id=uid)
out: dict[str, Any] = {"resumed": resumed, "message_count": message_count}
if initial_message_status:
# Only present when the initial message was NOT delivered — the
# factory passes it through to the response so API clients don't
# read the 200 as "first message accepted".
out["initial_message_status"] = initial_message_status
return out
def _audit_workstream_created(
@@ -2500,6 +2580,13 @@ async def cancel_watch(request: Request) -> JSONResponse:
if watch_node and node_id and watch_node != node_id:
return JSONResponse({"error": "Watch belongs to another node"}, status_code=403)
storage.update_watch(watch_id, active=False, next_poll="")
# AFTER the row write, per forget_terminal_dispatched's ordering
# contract: drop the runner's transient state for this id (held
# reminder / terminal-dispatched mark) — the inactive row never
# re-lists, so nothing else would ever clear it.
runner = getattr(request.app.state, "watch_runner", None)
if runner is not None:
runner.forget_terminal_dispatched(watch_id)
return JSONResponse({"status": "ok", "watch_id": watch_id})
@@ -4077,7 +4164,11 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
if tls_client is not None:
await tls_client.stop_renewal()
if app.state.watch_runner:
app.state.watch_runner.stop()
# ``stop()`` drains in-flight polls (bounded, but up to ~35 s) —
# run it off the event loop like the state-writer below so SSE
# teardown and the remaining shutdown steps aren't frozen behind
# a watch command that's still finishing.
await asyncio.to_thread(app.state.watch_runner.stop)
from turnstone.core.idle_nudge_watcher import shutdown_idle_nudge_watchers
shutdown_idle_nudge_watchers(app)
@@ -4964,7 +5055,7 @@ def main() -> None:
# Create WatchRunner (periodic command polling, server-level)
from turnstone.core.storage import get_storage as _get_storage
from turnstone.core.watch import WatchRunner
from turnstone.core.watch import WatchRunner, WatchWorkstreamUnrestorable
# Create session manager first (watch restore_fn captures it).
interactive_adapter = InteractiveAdapter(
@@ -5023,25 +5114,110 @@ def main() -> None:
on the rehydrated session, so ``WatchRunner._dispatch_result`` can
re-deliver the current message into the rehydrated workstream's
:class:`NudgeQueue` without a second pass through ``restore_fn``.
Failure taxonomy: two failures are PERMANENT the persona-stamp
pre-read raising (corrupt stamp, nothing created yet) and
``resume()`` returning ``False`` (no stored turns: the target's
history is gone, so there is nothing to deliver into and every
retry would rebuild this shell just to fail again). Everything
else after ``manager.create`` is treated as transient the
half-built shell is closed so a failed attempt can't leak a
``max_active`` slot, and the runner holds the reminder and
retries, bounded by the watch's own poll budget. Deliberately
NOT mapping post-create ``ValueError`` to permanent: ``resume``
can raise it for reasons beyond the corrupt-stamp contract, and
a misclassification here silently kills the user's watch.
"""
try:
ws = manager.create(user_id="", name="watch-restore", **_resume_persona_kwargs(ws_id))
persona_kwargs = _resume_persona_kwargs(ws_id)
except ValueError as exc:
# PERMANENT: corrupt persona stamp. Refuse to run the watch
# under an envelope the operator didn't choose (it would be
# unattended AND auto-approved), and signal the runner to stop
# retrying and deactivate the watch rather than burn the whole
# attempt budget on a cause that can't clear on its own.
log.warning("watch_restore: corrupt persona stamp on ws %s", ws_id, exc_info=True)
raise WatchWorkstreamUnrestorable(ws_id) from exc
try:
ws = manager.create(user_id="", name="watch-restore", **persona_kwargs)
except RuntimeError:
# TRANSIENT: all restore slots active right now. Return None so
# the runner holds the reminder and retries on a later tick.
log.warning("watch_restore: cannot restore ws %s (all slots active)", ws_id)
return None
try:
# Restored workstreams run unattended — auto-approve tool calls
# to avoid blocking forever on approval with no connected user.
if isinstance(ws.ui, WebUI):
ws.ui.auto_approve = True
if ws.session:
ws.session.resume(ws_id)
ws.session.set_watch_runner(_watch_runner)
return _watch_runner.get_dispatch_fn(ws.session._ws_id)
except RuntimeError:
log.warning("watch_restore: cannot restore ws %s (all slots active)", ws_id)
except ValueError:
# Corrupt persona stamp — refuse to run the watch under an
# envelope the operator didn't choose (it would be unattended
# AND auto-approved); the watch stays queued for a manual open.
log.warning("watch_restore: corrupt persona stamp on ws %s", ws_id, exc_info=True)
return None
if ws.session is None:
raise RuntimeError("created workstream has no session")
if not ws.session.resume(ws_id):
# ``resume``'s turn loader swallows storage errors into []
# (memory.load_message_turns), so False here is EITHER
# "history is gone" (permanent — without this check the
# fresh session keeps its own fresh ``_ws_id``, the
# registration below keys on THAT, and the reminder would
# be "delivered" into a blank, orphaned, auto-approved
# session while the watch deactivates as delivered) OR a
# transient read blip. Re-probe with the RAISING storage
# call before declaring permanence: misclassifying a blip
# silently kills the user's watch and drops the fired
# reminder.
with contextlib.suppress(Exception):
manager.close(ws.id)
try:
turns_exist = bool(_get_storage().load_message_turns(ws_id, checkpointed=True))
except Exception:
log.warning(
"watch_restore: turns probe failed for ws %s (treating as transient)",
ws_id,
exc_info=True,
)
return None
if turns_exist:
# Rows exist but resume()'s read came back empty — a
# blip. Retry on the held-delivery cadence.
log.warning("watch_restore: empty resume read for ws %s (blip)", ws_id)
return None
log.warning("watch_restore: ws %s has no stored turns", ws_id)
raise WatchWorkstreamUnrestorable(ws_id)
# A live registration may have appeared while this shell was
# being built — the user reopening the workstream mid-restore
# (``mgr.open`` + post-load registers their PANE). The pane
# wins: deliver into it and close the redundant shell.
# Registering ours would silently clobber the pane's — every
# later fire would run unattended in the shell while the user
# watches a conversation that never shows its watch results.
# (A pane registration landing in the microseconds between
# this check and the set below can still be clobbered; that
# residue requires the reopen to race a window ~10^6 times
# narrower than the restore itself.)
existing = _watch_runner.get_dispatch_fn(ws_id)
if existing is not None:
log.info("watch_restore: live registration appeared for ws %s — yielding", ws_id)
with contextlib.suppress(Exception):
manager.close(ws.id)
return existing
# ``ws`` is the freshly created workstream (manager-tracked id)
# even though the session resumed the original ``ws_id`` — the
# wake must target the live Workstream object, and firing it
# is what lets an unattended restore actually RUN the watch
# result (auto_approve above exists for exactly that turn).
ws.session.set_watch_runner(_watch_runner, wake_fn=_watch_fire_wake_fn(ws))
return _watch_runner.get_dispatch_fn(ws.session._ws_id)
except WatchWorkstreamUnrestorable:
raise # shell already closed at the raise site above
except Exception:
# TRANSIENT: the shell exists but never became the watch's live
# target — close it (untrack + mark closed) so the failed
# attempt doesn't hold a max_active slot forever.
log.warning("watch_restore: resume failed for ws %s", ws_id, exc_info=True)
with contextlib.suppress(Exception):
manager.close(ws.id)
return None
_watch_runner = WatchRunner(
storage=_get_storage(),
@@ -5067,10 +5243,16 @@ def main() -> None:
if args.skip_permissions or config_store.get("tools.skip_permissions"):
ws.ui.auto_approve = True
assert ws.session is not None
ws.session.set_watch_runner(_watch_runner)
if not ws.session.resume(target_id):
log.error("Workstream '%s' has no messages.", args.resume)
sys.exit(1)
# AFTER the successful resume (mirroring the restore fn's order),
# so the registration keys on the adopted ``target_id`` — the id
# the session's watch rows are stamped with. Registered before
# resume, the registry key would be the create-time id no watch
# row references, and every fire would restore a SECOND
# auto-approved session onto the operator's live conversation.
ws.session.set_watch_runner(_watch_runner, wake_fn=_watch_fire_wake_fn(ws))
log.info("Resumed workstream %s (%d messages)", target_id, len(ws.session.messages))
# Record detected model and judge status in metrics