Patrick Buckley 830305555d fix(sse): address PR #561 review round 2
5 follow-up comments from Copilot, all valid:

1. **CRITICAL — snap_seq race with split writer** (concurrency, 001).
   Round-1's fix lifted snapshot capture into
   register_listener_with_replay under nested locks, but the
   WRITER side (on_content_token / on_reasoning_token) still
   released _ws_lock before calling _enqueue (which bumps
   _event_id under _listeners_lock).  A reader could
   interleave between writer's release and writer's _enqueue:
   capture inflight WITH the new text, read STALE _event_id,
   return snap_seq < new_event_id.  The new event's live emit
   then has _seq > snap_seq, slips past the dedup filter, and
   double-renders text the snapshot already contained.

   Fix: move self._enqueue(...) INSIDE the with self._ws_lock:
   block in both token writers.  The inflight mutation and the
   _event_id advancement are now atomic against any snapshot
   reader.  Lock order _ws_lock (outer) → _listeners_lock
   (inner via _enqueue) matches the snapshot helpers, so no
   deadlock.  Fan-out's put_nowait calls happen under
   _ws_lock for token writers — microsecond cost per listener,
   acceptable for the correctness guarantee.

2. **NIT — stale comment ref to buffered[-1]._event_id** (docs, 002).
   The comment referenced a local var (buffered) that lives in
   register_listener_with_replay, not in the events handler.
   Reworded to describe the cutoff in terms of the last replayed
   event id and the atomic-against-writers registration.

3. **MODERATE — 401 branch leaves reconnect loop** (bug, 003).
   The coord's onerror schedules a 5 s CLOSED-state recovery timer
   unconditionally.  In the 401-expired-session branch we close
   evtSource and showLogin — but the timer still fires 5 s later,
   observes !evtSource, and calls scheduleReconnect(),
   which opens a new EventSource that 401s again → infinite
   reconnect loop while the login overlay is up.  Fix: cancel
   reconnectTimer in the 401 branch.

4. **MODERATE — race test was vacuous** (test_coverage, 004).
   The previous regression test drained the listener queue after
   register_listener_with_replay returned, but the helper
   doesn't backfill buffered events into the queue, so the loop
   was almost always a no-op and the assertion never executed.
   Rewrote with a monkey-patched _enqueue that sleeps 50 ms
   before bumping _event_id — widens the race window
   deterministically.  Verified: the test FAILS on pre-fix code
   (snap.content has marker but snap.seq=0 < final_event_id=1)
   and PASSES on post-fix code (writer holds _ws_lock through
   _enqueue, so the reader blocks until writer fully done).
   Also pinned the no-backfill contract so a future change adding
   listener-queue backfill remembers to keep snap_seq the
   high-water mark.

5. **NIT — except Exception too broad in test** (best_practices, 005).
   Tightened except Exception: to except queue.Empty: so
   unexpected exceptions aren't silently swallowed in the drain
   loop.

Tests:
- 86 tests in test_sse_reconnect_replay.py + test_session_ui_base.py
  pass (existing 84 + 2 new race regressions).
- Full non-live suite: 6347 passed, 15 skipped, no regressions.
- Ruff + mypy clean on changed .py files; JS parses.
2026-05-22 19:34:52 -07:00

Turnstone

CI PyPI Python License

Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.

Turnstone coordinator — parallel tool batches with judge-graded approval and child workstream tracking

Named after the Ruddy Turnstone (Arenaria interpres) — a shorebird that flips stones to discover what's hiding underneath.

Release Tracks

Track Install Docker Description
Stable pip install turnstone ghcr.io/turnstonelabs/turnstone:stable Production-grade. Bugfixes only.
Experimental pip install turnstone --pre ghcr.io/turnstonelabs/turnstone:experimental New features. May have rough edges.

See docs/releasing.md for the full release process.

What it does

Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.

  • Interactive sessions — terminal CLI or browser UI with parallel workstreams
  • Cluster dashboard — real-time view of all nodes and workstreams with console routing proxy
  • Intent validation — LLM judge evaluates every tool call with risk assessments and evidence
  • Governance — RBAC, OIDC SSO, tool policies, skills, usage tracking, audit logs
  • Multi-provider — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), Anthropic Messages API, and Google Gemini
  • MCP support — external tool servers with native deferred loading (Anthropic/OpenAI) or BM25 fallback

Turnstone system architecture

Quickstart

pip install turnstone

# Terminal REPL
turnstone --base-url http://localhost:8000/v1

# Browser UI
turnstone-server --port 8080 --base-url http://localhost:8000/v1

# Cluster dashboard
pip install turnstone[console]
turnstone-console --port 8090

For PostgreSQL (recommended for production):

pip install turnstone[postgres]
export TURNSTONE_DB_BACKEND=postgresql
export TURNSTONE_DB_URL="postgresql+psycopg://user:pass@localhost:5432/turnstone"
turnstone-server --port 8080 --base-url http://localhost:8000/v1

Docker

cp .env.example .env  # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose --profile production up

See QUICKSTART.md for the bootstrap wizard and docs/docker.md for Docker configuration and profiles.

Programmatic (SDK)

from turnstone.sdk import TurnstoneServer

with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
    ws = client.create_workstream(name="demo")
    result = client.send_and_wait("Analyze the error logs", ws.ws_id, auto_approve=True)
    print(result.content)

Tools

Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via MCP with native deferred loading. See docs/tools.md for the full reference and docs/mcp-registry.md for MCP configuration.

Architecture

Single-node: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.

Multi-node: Client → Console (rendezvous routing proxy) → Server nodes. The console picks the target node for each workstream via rendezvous (HRW) hashing over the live service registry — pure function of (ws_id, live_nodes), no stored bucket state, deterministic across readers. A node join or drop only re-routes the keys that score highest on the affected node.

Component Purpose
turnstone Terminal CLI (REPL)
turnstone-server Web UI + REST API + SSE events
turnstone-console Cluster dashboard + routing proxy + admin panel
turnstone-channel Channel gateway (Discord and Slack adapters)
turnstone-admin User/token management CLI
turnstone-eval Eval harness for prompt/tool optimization
turnstone-bootstrap LLM-guided setup wizard

Diagrams

UML diagrams in docs/diagrams/:

Diagram Description
System Context Components and external dependencies
Package Structure Python modules and dependency graph
Core Engine SessionUI, ChatSession, LLMProvider
Conversation Turn Message lifecycle through the engine
Tool Pipeline Prepare / approve / execute
Workstream States State machine transitions
Console Data Flow Dashboard data collection
Deployment Docker Compose topology
Auth JWT, scopes, login flows
Channels Discord / Slack adapters + routing
Judge Intent validation pipeline
OIDC SSO authorization code flow

Documentation

Topic Link
Configuration reference docs/settings.md
API reference docs/api-reference.md
Docker deployment docs/docker.md
Intent validation (judge) docs/judge.md
Governance & RBAC docs/governance.md
OIDC SSO docs/oidc.md
TLS / mTLS docs/tls.md
Channel integrations docs/channels.md
Console dashboard docs/console.md
Eval harness docs/eval.md
Tools reference docs/tools.md
MCP integration docs/mcp-registry.md

Requirements

  • Python 3.11+
  • An OpenAI-compatible API endpoint, Anthropic API key, or Google Gemini API key
  • Optional: PostgreSQL (pip install turnstone[postgres]), Anthropic (pip install turnstone[anthropic])
  • Git LFS for cloning (diagram PNGs)

License

Business Source License 1.1 — free for all use except hosting as a managed service. Converts to Apache 2.0 on 2030-03-01.

S
Description
No description provided
Readme Apache-2.0 133 MiB
Languages
Python 87.8%
JavaScript 8.6%
CSS 1.8%
HTML 0.9%
TypeScript 0.5%
Other 0.3%