Files
turnstone/tests/test_coord_rich_ws_state_payload.py
T
Patrick Buckley 480a1426b3 Fail-closed history-commit handoff (#1005)
* fix(session): fail-closed history-commit handoff (#981)

The deleted-workstream discovery is now a terminal, ws_id-keyed latch:
keyed conversation commits refuse admission once the durable parent is
gone (convergence finalizers and force-abandon are exempt), history
handoff refuses to mint a proof token so /history fails closed with a
503 instead of silently wiping the pane, and the SSE stream carries a
workstream_gone resync reason. Discarded commits leave a forensic log
of commit keys and roles, never content.

Conversation rows gain a commit_key (migration 071): keyed saves are
idempotent under retry, validated against the full commit identity, and
refused when they would cross a workstream deletion. The prune orphan
category now requires a NULL alias plus a two-hour updated grace, with
cutoffs computed at discovery time and carried into both dialects'
rechecks.

The mid-turn interjection queue is owner-partitioned with no per-site
mode flags: pops take the acting principal's and unowned rows, other
participants' rows are structurally retained, and enforcement lives at
queue admission plus the shared before_spawn gates. The retraction
ledger is bounded by open pop windows: pops open a window atomically
with the queue delete, restores close their ids atomically with the
ledger consume, every other exit closes through one helper, and misses
for unheld ids record nothing. The workstream-gone latch refuses
unattended wakes at all three gates (watcher spawn, claim, delivery
pre-pop), and the retry dispatcher regained its pre-envelope
cancel/error convergence net.

Persistence-state reporting derives through the session bound to each
UI instead of a registry lookup by id that failed open to healthy
during tombstone retention. The dashboard roster no longer re-inserts
ghost entries from trailing activity events, the history tool-outcome
scan tolerates interleaved non-turn rows, and the shared
handoff-deadline handle owns its own retirement.

Single-sourced across call sites: keyed-commit row values, attachment
save wrappers, tail-truncation and conflict-resolution bodies for both
storage dialects; worker-slot lifecycle field sets; the direct-commit
admission frame; queued-row layout accessors; the string-aware comment
stripper shared by every JS harness suite.

Refs #981 #964

* fix(session): sweep handoff fixes to their sibling surfaces

The interactive replay loop treated a system row as a tool-batch
boundary, so every tool result after an interleaved row vanished from
that pane while the coordinator rendered the same history correctly.
Only a conversational turn ends the batch window now, matching the
shared outcome index.

Accepted user turns clear the composer's attachment chips on the same
viewer policy that settles optimistic bubbles rather than on having
matched a local bubble, so a workstream created with an upload no
longer keeps a chip for an attachment the create dispatch already
consumed. The coordinator's raced-Stop arm emits the stream-end hook it
inherits alongside the idle state, leaving no unfinalized bubble or
unflushed tool output. Ending a session surfaces a failure toast when
the request never lands or answers with a non-JSON body.

The per-second persistence reconcile now probes each session without
blocking: a workstream whose generation and handoff locks are held is
skipped until the next pass instead of contending the locks every
commit needs. The one-shot repair that gates workstream creation at
capacity keeps a definite probe — it has no next pass, and the sessions
likeliest to be contended are the ones whose unresolved journals
emptied its candidate list.

Single-sourced: the attachment lane builds its conversation row through
the shared commit-identity builder; the ordinary worker exit releases
its slot through the lifecycle owner; both operator surfaces snapshot
their counters through one non-consuming helper; the replay preamble
loses its per-kind wrappers and its config hook; the browser harness
suites share one brace walker; and each in-flight history attempt is
one record carrying both its abort controller and its deadline.

Refs #981 #964
2026-08-11 04:18:36 -07:00

738 lines
30 KiB
Python

"""Tests for the rich ``ws_state`` payload on coord (Stage 2 follow-up).
Pre-lift coord's ``ConsoleCoordinatorUI`` populated none of the per-ws
metric fields ``SessionUIBase`` defines (``_ws_prompt_tokens`` /
``_ws_context_ratio`` / ``_ws_current_activity`` / ``_ws_turn_content``)
and the ``coord_adapter.emit_state`` broadcast was state-only —
``tokens=0`` / ``content=""`` were hardcoded into
``collector.emit_console_ws_state``. The lift turned ``on_status`` /
``on_content_token`` / ``on_thinking_*`` / ``on_tool_result`` into
shared bodies on :class:`SessionUIBase` so coord populates the same
fields, then enriched ``coord_adapter.emit_state`` to read them under
lock and pass through to the cluster collector with the rich kwargs.
The cluster dashboard's coord rows now render with the same
tokens / activity / content / context_ratio fields interactive rows do.
"""
from __future__ import annotations
import threading
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
from turnstone.core.workstream import WorkstreamState
# ---------------------------------------------------------------------------
# Per-ws metric writes — lifted to SessionUIBase, both subclasses inherit
# ---------------------------------------------------------------------------
def _patch_get_storage(storage: Any):
return patch("turnstone.core.storage._registry.get_storage", return_value=storage)
def test_coord_on_status_writes_per_ws_metrics() -> None:
"""Pre-lift coord ``on_status`` was an enqueue-only stub — ``_ws_*``
fields stayed at their initial zero values regardless of token usage.
Post-lift coord inherits SessionUIBase's body, so token counters and
context ratio populate just like interactive."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
with _patch_get_storage(MagicMock()):
ui.on_status(
{"prompt_tokens": 100, "completion_tokens": 50},
context_window=1000,
effort="medium",
)
assert ui._ws_prompt_tokens == 100
assert ui._ws_completion_tokens == 50
assert ui._ws_context_ratio == pytest.approx(0.15)
def test_coord_on_status_persists_usage_event() -> None:
"""Pre-lift coord didn't persist usage_event rows — only WebUI did.
Lift extends usage tracking to coord so governance dashboards see
coordinator token consumption alongside interactive."""
storage = MagicMock()
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
with _patch_get_storage(storage):
ui.on_status(
{"prompt_tokens": 7, "completion_tokens": 3, "model": "gpt-x"},
context_window=200,
effort="low",
)
storage.record_usage_event.assert_called_once()
kwargs = storage.record_usage_event.call_args.kwargs
assert kwargs["ws_id"] == "coord-ws"
assert kwargs["user_id"] == "u1"
assert kwargs["model"] == "gpt-x"
assert kwargs["prompt_tokens"] == 7
assert kwargs["completion_tokens"] == 3
def test_coord_on_aux_usage_persists_usage_event() -> None:
"""Auxiliary LLM calls (plan/task sub-agents, compaction, web-fetch
summarisation, title gen) bypass ``on_status`` entirely. ``on_aux_usage``
is what gets their token spend onto the governance dashboard."""
storage = MagicMock()
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
with _patch_get_storage(storage):
ui.on_aux_usage(
{
"prompt_tokens": 500,
"completion_tokens": 40,
"cache_creation_tokens": 12,
"cache_read_tokens": 8,
"model": "plan-model",
}
)
storage.record_usage_event.assert_called_once()
kwargs = storage.record_usage_event.call_args.kwargs
assert kwargs["ws_id"] == "coord-ws"
assert kwargs["user_id"] == "u1"
assert kwargs["model"] == "plan-model"
assert kwargs["prompt_tokens"] == 500
assert kwargs["completion_tokens"] == 40
assert kwargs["cache_creation_tokens"] == 12
assert kwargs["cache_read_tokens"] == 8
# Tools a sub-agent calls internally are its own tally, not this ws's.
assert kwargs["tool_calls_count"] == 0
def test_coord_on_aux_usage_leaves_live_counters_untouched() -> None:
"""Unlike ``on_status``, ``on_aux_usage`` must NOT fold the auxiliary
prompt into the live per-ws context gauge — that tracks the main
conversation's window, and an agent/compaction prompt isn't it."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
with _patch_get_storage(MagicMock()):
ui.on_aux_usage({"prompt_tokens": 9999, "completion_tokens": 9999})
assert ui._ws_prompt_tokens == 0
assert ui._ws_completion_tokens == 0
assert ui._ws_context_ratio == 0.0
def test_coord_on_content_token_accumulates(monkeypatch: pytest.MonkeyPatch) -> None:
"""Pre-lift coord ``on_content_token`` only enqueued; lift turns it
into the same per-ws accumulator WebUI uses so the collector
broadcast can piggyback the joined turn content on the IDLE
state-change event.
Batch window forced to 0 (per-token flush) — pins the accumulator
wiring, not the batching cadence (test_sse_token_batching.py)."""
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_content_token("Hello ")
ui.on_content_token("world")
assert ui._ws_turn_content == ["Hello ", "world"]
assert ui._ws_turn_content_size == len("Hello world")
def test_coord_on_content_token_caps_at_ceiling() -> None:
"""Same content cap interactive enforces — keeps a runaway turn from
ballooning the cluster broadcast event past listener queue size."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
chunk = "x" * 1024
rounds = (_MAX_TURN_CONTENT_CHARS // 1024) + 50
for _ in range(rounds):
ui.on_content_token(chunk)
# Cap is enforced at the size check; one over-cap chunk still
# gets in (per the original ``< _MAX``-not-``<=`` semantics) but
# nothing past that lands.
assert ui._ws_turn_content_size <= _MAX_TURN_CONTENT_CHARS + 1024
def test_coord_on_thinking_start_sets_activity() -> None:
"""Live activity tracking — coord's dashboard row now flips
``activity_state`` to ``"thinking"`` when the model starts."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_thinking_start()
assert ui._ws_current_activity == "Thinking…"
assert ui._ws_activity_state == "thinking"
def test_coord_on_tool_result_clears_activity_and_increments_counters() -> None:
"""Lifted ``on_tool_result`` body increments ``_ws_tool_calls`` /
``_ws_turn_tool_calls`` and clears the activity. Pre-lift coord
just enqueued without touching counters."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui._ws_current_activity = "⚙ bash: ls -la"
ui._ws_activity_state = "tool"
ui.on_tool_result("call-1", "bash", "output")
assert ui._ws_tool_calls == {"bash": 1}
assert ui._ws_turn_tool_calls == 1
assert ui._ws_current_activity == ""
assert ui._ws_activity_state == ""
# ---------------------------------------------------------------------------
# Snapshot helper — drains turn content on IDLE/ERROR
# ---------------------------------------------------------------------------
def test_snapshot_idle_returns_content_and_clears_accumulator() -> None:
"""IDLE snapshot piggybacks the joined assistant content onto the
state-change broadcast (so the dashboard renders the turn without
a storage round-trip), then clears the accumulator for the next
turn."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_content_token("Here's ")
ui.on_content_token("the result.")
payload = ui.snapshot_and_consume_state_payload("idle")
assert payload["content"] == "Here's the result."
assert ui._ws_turn_content == []
assert ui._ws_turn_content_size == 0
def test_snapshot_error_clears_accumulator_without_emitting_content() -> None:
"""ERROR clears the partial content (the turn's broken; nothing to
render) but the broadcast itself doesn't carry it — the state
transition is what matters."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_content_token("partial...")
payload = ui.snapshot_and_consume_state_payload("error")
assert payload["content"] == ""
assert ui._ws_turn_content == []
def test_snapshot_thinking_does_not_touch_accumulator() -> None:
"""Mid-turn state transitions (running / thinking / attention)
don't drain the accumulator — only IDLE / ERROR are terminal."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui.on_content_token("partial mid-turn")
payload = ui.snapshot_and_consume_state_payload("thinking")
assert payload["content"] == ""
# Accumulator preserved.
assert ui._ws_turn_content == ["partial mid-turn"]
def test_snapshot_carries_token_and_activity_snapshot() -> None:
"""Snapshot reads tokens / context_ratio / activity under one lock
acquisition so concurrent on_status / on_thinking_start writes
don't tear the snapshot."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
with _patch_get_storage(MagicMock()):
ui.on_status(
{"prompt_tokens": 80, "completion_tokens": 20},
context_window=400,
effort="medium",
)
ui.on_thinking_start() # sets activity = "Thinking…"
payload = ui.snapshot_and_consume_state_payload("running")
assert payload["tokens"] == 100
assert payload["context_ratio"] == pytest.approx(0.25)
assert payload["activity"] == "Thinking…"
assert payload["activity_state"] == "thinking"
# ---------------------------------------------------------------------------
# Coord adapter — passes rich payload to collector
# ---------------------------------------------------------------------------
class _FakeCollectorRecorder:
"""Captures emit_console_ws_state calls so we can assert on the
rich kwargs the lifted coord_adapter.emit_state passes through."""
def __init__(self) -> None:
self.state_calls: list[dict[str, Any]] = []
self.activity_calls: list[dict[str, Any]] = []
def emit_console_ws_state(
self,
ws_id: str,
state: str,
*,
tokens: int = 0,
context_ratio: float = 0.0,
activity: str = "",
activity_state: str = "",
content: str = "",
persistence_state: str = "healthy",
) -> None:
self.state_calls.append(
{
"ws_id": ws_id,
"state": state,
"tokens": tokens,
"context_ratio": context_ratio,
"activity": activity,
"activity_state": activity_state,
"content": content,
"persistence_state": persistence_state,
}
)
def update_console_ws_activity(self, ws_id: str, *, activity: str, activity_state: str) -> None:
self.activity_calls.append(
{"ws_id": ws_id, "activity": activity, "activity_state": activity_state}
)
def emit_console_ws_created(self, *_a: Any, **_kw: Any) -> None:
pass
def emit_console_ws_closed(self, *_a: Any, **_kw: Any) -> None:
pass
def emit_console_ws_rename(self, *_a: Any, **_kw: Any) -> None:
pass
def ensure_console_pseudo_node(self) -> None:
pass
def _build_adapter_and_ws(ws_id: str = "coord-ws-1") -> tuple[Any, Any, _FakeCollectorRecorder]:
"""Construct a minimal adapter + Workstream + UI for emit_state tests.
Skips the full SessionManager wire-up — the adapter's ``emit_state``
only reads ``ws.id`` and ``ws.ui``, so a real ``Workstream`` with
a populated ``ConsoleCoordinatorUI`` is enough.
"""
from turnstone.console.coordinator_adapter import CoordinatorAdapter
from turnstone.core.workstream import Workstream
recorder = _FakeCollectorRecorder()
adapter = CoordinatorAdapter(
collector=recorder, # type: ignore[arg-type]
ui_factory=lambda ws: ConsoleCoordinatorUI(ws_id=ws.id, user_id=ws.user_id),
session_factory=lambda ws: MagicMock(),
)
ws = Workstream(id=ws_id, user_id="u1", name="my-coord")
ws.ui = ConsoleCoordinatorUI(ws_id=ws_id, user_id="u1")
return adapter, ws, recorder
def test_coord_adapter_emit_state_passes_rich_payload_to_collector() -> None:
"""Pre-lift coord_adapter.emit_state called collector with state-only;
post-lift it reads the UI's per-ws snapshot under lock and passes
tokens / context_ratio / activity / content kwargs through."""
adapter, ws, recorder = _build_adapter_and_ws()
with _patch_get_storage(MagicMock()):
ws.ui.on_status(
{"prompt_tokens": 60, "completion_tokens": 40},
context_window=400,
effort="medium",
)
ws.ui.on_content_token("partial answer")
ws.ui.on_thinking_start()
adapter.emit_state(ws, WorkstreamState.RUNNING)
assert len(recorder.state_calls) == 1
call = recorder.state_calls[0]
assert call["ws_id"] == ws.id
assert call["state"] == "running"
assert call["tokens"] == 100
assert call["context_ratio"] == pytest.approx(0.25)
assert call["activity"] == "Thinking…"
assert call["activity_state"] == "thinking"
# Mid-turn (RUNNING) — content stays accumulated for the eventual IDLE drain.
assert call["content"] == ""
assert call["persistence_state"] == "healthy"
def test_coord_adapter_emit_state_idle_drains_content() -> None:
"""IDLE state-change drains the turn-content accumulator and
piggybacks the joined content on the broadcast — same shape WebUI
uses on global_queue. Subsequent emit_state must see the
accumulator cleared."""
adapter, ws, recorder = _build_adapter_and_ws()
ws.ui.on_content_token("Here's ")
ws.ui.on_content_token("the result.")
adapter.emit_state(ws, WorkstreamState.IDLE)
assert len(recorder.state_calls) == 1
assert recorder.state_calls[0]["content"] == "Here's the result."
# Accumulator drained — next emit_state sees nothing carried over.
adapter.emit_state(ws, WorkstreamState.IDLE)
assert recorder.state_calls[1]["content"] == ""
def test_coord_adapter_emit_state_handles_missing_ui_defensively() -> None:
"""``ws.ui`` can be ``None`` mid-eviction; emit_state still
broadcasts the state-change with empty rich fields so the
dashboard's coord row still flips state instead of going stale."""
adapter, ws, recorder = _build_adapter_and_ws()
ws.ui = None # simulate teardown race
adapter.emit_state(ws, WorkstreamState.RUNNING)
assert len(recorder.state_calls) == 1
call = recorder.state_calls[0]
assert call["state"] == "running"
assert call["tokens"] == 0
assert call["content"] == ""
# ---------------------------------------------------------------------------
# Coord activity broadcast — UI fans out directly to the collector
# ---------------------------------------------------------------------------
def test_coord_ui_broadcast_activity_calls_collector() -> None:
"""Live activity transitions on coord (between state changes) reach
the cluster collector via the new ``update_console_ws_activity``
method. WebUI's analog goes via the global SSE queue; coord's
UI calls the collector directly since the console isn't a node."""
recorder = _FakeCollectorRecorder()
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ConsoleCoordinatorUI._collector = recorder # type: ignore[assignment]
try:
ui.on_thinking_start() # base impl calls _broadcast_activity
assert len(recorder.activity_calls) == 1
call = recorder.activity_calls[0]
assert call["ws_id"] == "coord-ws"
assert call["activity"] == "Thinking…"
assert call["activity_state"] == "thinking"
finally:
ConsoleCoordinatorUI._collector = None
def test_coord_ui_broadcast_activity_swallows_collector_failure() -> None:
"""A flaky collector must NOT block the worker thread — activity
fan-out is observational, the worker keeps running on collector
failure."""
recorder = MagicMock()
recorder.update_console_ws_activity.side_effect = RuntimeError("collector dead")
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ConsoleCoordinatorUI._collector = recorder
try:
ui.on_thinking_start() # must not raise
recorder.update_console_ws_activity.assert_called_once()
finally:
ConsoleCoordinatorUI._collector = None
def test_coord_ui_broadcast_activity_no_op_when_collector_unset() -> None:
"""Tests / tooling that don't wire a collector shouldn't crash."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ConsoleCoordinatorUI._collector = None
ui.on_thinking_start() # must not raise
def test_coord_ui_persistence_refresh_updates_cluster_without_transcript_event() -> None:
recorder = _FakeCollectorRecorder()
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
session = MagicMock()
session.conversation_persistence_status = lambda: {"state": "retrying"}
ui.bind_session(session)
ws = MagicMock()
ws.state.value = "error"
# The registry row deliberately disagrees: the persistence field must
# come from the BOUND session, so a registry miss or an id-reuse
# replacement row can never report another session's journal.
ws.session.conversation_persistence_status = lambda: {"state": "healthy"}
mgr = MagicMock()
mgr.get.return_value = ws
ConsoleCoordinatorUI._collector = recorder # type: ignore[assignment]
ConsoleCoordinatorUI._coord_mgr = mgr
try:
ui.on_persistence_state_changed()
finally:
ConsoleCoordinatorUI._collector = None
ConsoleCoordinatorUI._coord_mgr = None
assert recorder.state_calls[0]["state"] == "error"
assert recorder.state_calls[0]["persistence_state"] == "retrying"
assert ui._event_id == 0, "operator refresh must not enter the per-workstream SSE stream"
def test_coord_ui_broadcast_activity_failure_does_not_strand_dedup() -> None:
"""Regression for the Copilot finding on PR #420: post-fix the
dedup state ``_last_broadcast_activity`` is updated **only after**
a successful collector call. If the collector raises mid-broadcast
on tick #1, tick #2 with the same activity tuple must still
attempt the broadcast (otherwise a transient collector failure
would strand the dashboard's coord row at the pre-failure
activity until the activity actually changes). Pre-fix the
dedup state was assigned inside the lock before the collector
call, so the failed broadcast still updated it and tick #2
silently no-op'd."""
recorder = MagicMock()
# First call fails (transient collector outage); second call succeeds.
recorder.update_console_ws_activity.side_effect = [
RuntimeError("collector dead"),
None,
]
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ConsoleCoordinatorUI._collector = recorder
try:
# Tick #1 — collector raises; dedup state must NOT update.
ui.on_thinking_start()
assert ui._last_broadcast_activity is None, (
"dedup state was updated despite a failed collector call — "
"next identical tick would be silently suppressed"
)
# Tick #2 — same activity tuple. Pre-fix this would no-op
# (because dedup state was already (Thinking…, thinking)).
# Post-fix it retries; collector succeeds; dedup state lands.
ui.on_thinking_start()
assert recorder.update_console_ws_activity.call_count == 2, (
"second tick was deduped despite the first call failing"
)
assert ui._last_broadcast_activity == ("Thinking…", "thinking")
finally:
ConsoleCoordinatorUI._collector = None
def test_coord_ui_broadcast_activity_dedup_skips_identical_after_success() -> None:
"""Happy-path dedup: after a successful broadcast, the next identical
tick is deduped — the cluster collector lock is not re-acquired
for a no-op write. This is the perf optimization the dedup is
there for; the regression test above checks the failure-recovery
invariant doesn't break it."""
recorder = MagicMock()
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ConsoleCoordinatorUI._collector = recorder
try:
ui.on_thinking_start() # tick 1 — fires
ui.on_thinking_start() # tick 2 — same tuple, deduped
ui.on_thinking_start() # tick 3 — same tuple, deduped
assert recorder.update_console_ws_activity.call_count == 1
assert ui._last_broadcast_activity == ("Thinking…", "thinking")
finally:
ConsoleCoordinatorUI._collector = None
# ---------------------------------------------------------------------------
# Spawn metrics — coord wires its own hook
# ---------------------------------------------------------------------------
def test_coord_spawn_metrics_increments_messages_and_resets_tool_count() -> None:
"""Coord's ``_coord_spawn_metrics`` mirrors interactive's per-spawn
counter writes (sans the Prometheus call) so the rich ``ws_state``
broadcast renders the same per-turn shape."""
from turnstone.console.server import _coord_spawn_metrics
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui._ws_messages = 5
ui._ws_turn_tool_calls = 3
_coord_spawn_metrics(ui)
assert ui._ws_messages == 6
assert ui._ws_turn_tool_calls == 0
def test_coord_spawn_metrics_tolerates_ui_without_counters() -> None:
"""A SessionUI subclass without the per-ws counters shouldn't trip
the hook — defensive guard mirrors the interactive analog."""
from turnstone.console.server import _coord_spawn_metrics
class _StubUI:
pass
_coord_spawn_metrics(_StubUI()) # must not raise
# ---------------------------------------------------------------------------
# Snapshot lock — single-acquisition guarantee
# ---------------------------------------------------------------------------
def test_snapshot_acquires_ws_lock_exactly_once() -> None:
"""Snapshot must read all four fields under a single lock acquisition
so concurrent on_status / on_thinking_start writes can't tear the
payload."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
acquire_count = 0
inner = ui._ws_lock
class _CountingLock:
def __enter__(self) -> None:
nonlocal acquire_count
acquire_count += 1
inner.acquire()
def __exit__(self, *a: Any) -> None:
inner.release()
def acquire(self, *a: Any, **kw: Any) -> bool:
return inner.acquire(*a, **kw)
def release(self) -> None:
inner.release()
ui._ws_lock = _CountingLock() # type: ignore[assignment]
ui.snapshot_and_consume_state_payload("idle")
assert acquire_count == 1, (
f"snapshot acquired _ws_lock {acquire_count} times; concurrent "
"writes could tear the rich payload"
)
# ---------------------------------------------------------------------------
# Concurrency — snapshot under load
# ---------------------------------------------------------------------------
def test_snapshot_under_concurrent_writes_does_not_crash() -> None:
"""Sanity stress: snapshot reads while on_status / on_thinking_start /
on_content_token write concurrently. Reader cycles through
``("running", "idle", "error")`` so the IDLE/ERROR drain branches
that mutate ``_ws_turn_content`` actually get exercised against
concurrent appends — running-only would only hit the read-only
snapshot path. Each thread's exception (if any) is captured + raised
on join so a silent worker crash can't slip through as a bare
deadlock-check pass."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
writer_exc: list[Exception] = []
reader_exc: list[Exception] = []
def _writer() -> None:
try:
with _patch_get_storage(MagicMock()):
for i in range(50):
ui.on_status(
{"prompt_tokens": i, "completion_tokens": i},
context_window=1000,
effort="low",
)
ui.on_content_token(f"chunk-{i}")
ui.on_thinking_start()
except Exception as exc: # noqa: BLE001 — surface to main thread
writer_exc.append(exc)
def _reader() -> None:
try:
states = ("running", "idle", "error")
for i in range(50):
ui.snapshot_and_consume_state_payload(states[i % len(states)])
except Exception as exc: # noqa: BLE001 — surface to main thread
reader_exc.append(exc)
writer = threading.Thread(target=_writer)
reader = threading.Thread(target=_reader)
writer.start()
reader.start()
writer.join(timeout=5)
reader.join(timeout=5)
assert not writer.is_alive(), "writer thread deadlocked"
assert not reader.is_alive(), "reader thread deadlocked"
assert not writer_exc, f"writer raised: {writer_exc[0]!r}"
assert not reader_exc, f"reader raised: {reader_exc[0]!r}"
def test_coord_on_stream_end_clears_activity() -> None:
"""Lifted ``on_stream_end`` body clears ``_ws_current_activity``
and ``_ws_activity_state`` so the dashboard's coord row stops
showing the stale 'Thinking…' indicator after the stream
finishes. Pre-lift coord just enqueued ``stream_end`` without
touching activity — this test pins the new clear path so a
future re-stub doesn't silently re-introduce a stuck activity
indicator."""
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
ui._ws_current_activity = "Thinking…"
ui._ws_activity_state = "thinking"
ui.on_stream_end()
assert ui._ws_current_activity == ""
assert ui._ws_activity_state == ""
# ---------------------------------------------------------------------------
# WebUI override semantics still preserved
# ---------------------------------------------------------------------------
def test_webui_on_status_still_records_prometheus_metrics() -> None:
"""The lift moves the per-ws writes to SessionUIBase but WebUI's
override must still fire ``_metrics.record_*`` (Prometheus on the
node /metrics endpoint). Regression guard against a future refactor
accidentally dropping the override."""
import queue
from turnstone.server import WebUI
WebUI._global_queue = queue.Queue()
try:
ui = WebUI(ws_id="ws-int", user_id="u1")
with patch("turnstone.server._metrics") as mock_metrics, _patch_get_storage(MagicMock()):
ui.on_status(
{"prompt_tokens": 10, "completion_tokens": 5},
context_window=200,
effort="low",
)
mock_metrics.record_tokens.assert_called_once_with(10, 5)
mock_metrics.record_cache_tokens.assert_called_once()
mock_metrics.record_context_ratio.assert_called_once()
finally:
WebUI._global_queue = None
def test_webui_on_aux_usage_records_prometheus_metrics() -> None:
"""WebUI.on_aux_usage must feed ``_metrics.record_*`` so auxiliary
(sub-agent / compaction / utility) tokens land in
``turnstone_tokens_total``, not just main-loop turns. Regression guard
mirroring ``test_webui_on_status_still_records_prometheus_metrics`` —
without it, a refactor dropping the override would silently stop
counting aux tokens with nothing failing."""
import queue
from turnstone.server import WebUI
WebUI._global_queue = queue.Queue()
try:
ui = WebUI(ws_id="ws-int", user_id="u1")
with patch("turnstone.server._metrics") as mock_metrics, _patch_get_storage(MagicMock()):
ui.on_aux_usage(
{
"prompt_tokens": 64,
"completion_tokens": 8,
"cache_creation_tokens": 3,
"cache_read_tokens": 5,
}
)
mock_metrics.record_tokens.assert_called_once_with(64, 8)
mock_metrics.record_cache_tokens.assert_called_once_with(3, 5)
# Unlike on_status, aux usage must NOT touch the context-ratio gauge.
mock_metrics.record_context_ratio.assert_not_called()
finally:
WebUI._global_queue = None
def test_webui_on_tool_result_still_records_prometheus_tool_call() -> None:
"""Same as above for ``on_tool_result``."""
import queue
from turnstone.server import WebUI
WebUI._global_queue = queue.Queue()
try:
ui = WebUI(ws_id="ws-int", user_id="u1")
with patch("turnstone.server._metrics") as mock_metrics:
ui.on_tool_result("call-1", "bash", "output")
mock_metrics.record_tool_call.assert_called_once_with("bash")
# Per-ws counter writes happened too (inherited from base).
assert ui._ws_tool_calls == {"bash": 1}
assert ui._ws_turn_tool_calls == 1
finally:
WebUI._global_queue = None
def test_webui_accepted_tool_projection_is_metrics_free() -> None:
"""Final guarded replacement must not count the executor receipt twice."""
import queue
from turnstone.server import WebUI
WebUI._global_queue = queue.Queue()
try:
ui = WebUI(ws_id="ws-int", user_id="u1")
ui._ws_tool_calls = {"bash": 1}
ui._ws_turn_tool_calls = 1
with patch("turnstone.server._metrics") as mock_metrics:
event_id = ui.on_tool_turn_accepted(
"call-1",
"bash",
"guarded output",
is_error=True,
effect_status="unknown",
)
assert event_id == 1
mock_metrics.record_tool_call.assert_not_called()
assert ui._ws_tool_calls == {"bash": 1}
assert ui._ws_turn_tool_calls == 1
projected = ui._event_buffer[-1][1]
assert projected["accepted"] is True
assert projected["effect_status"] == "unknown"
finally:
WebUI._global_queue = None