Retire two polling patterns in coord that have clean event sources. PR 2 of 3 in the coord-completion stack; sits on top of PR #505 (reactive node discovery via PG LISTEN/NOTIFY). `wait_for_workstream` (coord's block-wait tool) polled storage every 0.5 s in a worker thread regardless of whether anything had changed — a 600 s wait incurred ~2400 round-trips. Now subscribes to a new in-process `ChildEventBus` (`turnstone/core/child_event_bus.py`) and blocks on `threading.Event.wait(min(remaining, WAIT_HEARTBEAT_INTERVAL))`: - `CoordinatorAdapter` owns the bus; `_dispatch_child_event` calls `bus.notify(child_ws_id)` after each `_enqueue_on_ui` for the state-class branch (cluster_state, ws_closed, ws_rename, intent_verdict, approval_resolved, approve_request). - Wait loop clears the Event BEFORE the storage snapshot to close the subscribe/check race; a notify between clear and the next `wait()` leaves the Event set so the loop re-reads without losing the wake-up. - 2 s heartbeat cap preserves the existing `wait_progress` SSE cadence for the sidebar UI while cutting SSE traffic ~4x vs the pre-bus 500 ms cadence in the quiescent case. - Worst-case completion latency is 2 s (vs pre-bus 0.5 s) because `set_state` buffers non-ERROR writes through `StateWriter` (async-flushed) while `emit_state` fans out immediately — a bus-driven wake can beat the flusher and read pre-transition state, then re-block until heartbeat. Deliberate trade-off; the SSE-traffic reduction outweighs the regression on the most common terminal transition. - Defense-in-depth: ownership-filter `cleaned` to own-subtree before `register_waiter` so a foreign ws_id passed by an untrusted coord LLM (prompt injection) can't observe wake-up timing as a side channel. Predicate (`_row_in_own_subtree`) requires both `parent_ws_id == coord_ws_id` AND `user_id == coord_user_id` parity — same gate strength as the existing `_is_own_subtree` mutating-op guard, so a corrupted / cross-tenant `parent_ws_id` alone can't satisfy it. Shared with `_snapshot_all` so the snapshot's `denied` shape stays in lockstep with the bus filter (Copilot review on #506). Coord idle-cleanup thread polled the storage scan every `check_every` seconds (~30 s on default 2 h timeout) even when no coord was anywhere near idle. Now subscribes to `SessionManager._state_subscribers` with a `tick_now` event and blocks on `tick_now.wait(check_every)` — any state change wakes the sweeper without waiting a full interval, AND the timeout still fires the periodic sweep for the DB-orphan-only case. A `min_sweep_interval=5 s` floor bounds DB-call traffic at ~0.2/s under sustained activity so the loop can't tight-spin `close_idle` at the rate of its own DB latency (6x improvement over the pre-refactor fixed 30 s cadence under any activity, and prompt state-change-driven wakes when below the floor). `CoordinatorClient` constructor takes `child_event_bus` as a required kwarg — there's no external SDK shape to preserve and keeping it optional would silently mask a wiring bug in any future caller. Tests construct their own `ChildEventBus()` per fixture. Tests: 16 unit tests for `ChildEventBus` (register / unregister symmetry, multi-waiter fan-out, multi-child waiter, subscribe/check race, concurrent register / notify smoke); 7 new adapter tests (bus notify fires for all 6 state-class events, drops for unknown child / wrong ws_id); 7 new coord-client wait tests (subscribe- after-terminal, notify wakes, unrelated notify doesn't wake, heartbeat fires without notify, unregister on exit, multi-waiter independence, cross-tenant denial via the user_id-parity filter); 8 idle-cleanup tests (initial sweep, heartbeat cadence, exception swallowing, stop_event clean exit, state-change wake, subscriber cleanup, mid-sweep wake, `min_sweep_interval` floor). All pass; ruff + mypy clean. Full non-live suite: 6227 passed (+2 vs prior baseline).
Turnstone
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Named after the Ruddy Turnstone (Arenaria interpres) — a shorebird that flips stones to discover what's hiding underneath.
Release Tracks
| Track | Install | Docker | Description |
|---|---|---|---|
| Stable | pip install turnstone |
ghcr.io/turnstonelabs/turnstone:stable |
Production-grade. Bugfixes only. |
| Experimental | pip install turnstone --pre |
ghcr.io/turnstonelabs/turnstone:experimental |
New features. May have rough edges. |
See docs/releasing.md for the full release process.
What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- Interactive sessions — terminal CLI or browser UI with parallel workstreams
- Cluster dashboard — real-time view of all nodes and workstreams with console routing proxy
- Intent validation — LLM judge evaluates every tool call with risk assessments and evidence
- Governance — RBAC, OIDC SSO, tool policies, skills, usage tracking, audit logs
- Multi-provider — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), Anthropic Messages API, and Google Gemini
- MCP support — external tool servers with native deferred loading (Anthropic/OpenAI) or BM25 fallback
Quickstart
pip install turnstone
# Terminal REPL
turnstone --base-url http://localhost:8000/v1
# Browser UI
turnstone-server --port 8080 --base-url http://localhost:8000/v1
# Cluster dashboard
pip install turnstone[console]
turnstone-console --port 8090
For PostgreSQL (recommended for production):
pip install turnstone[postgres]
export TURNSTONE_DB_BACKEND=postgresql
export TURNSTONE_DB_URL="postgresql+psycopg://user:pass@localhost:5432/turnstone"
turnstone-server --port 8080 --base-url http://localhost:8000/v1
Docker
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose --profile production up
See QUICKSTART.md for the bootstrap wizard and docs/docker.md for Docker configuration and profiles.
Programmatic (SDK)
from turnstone.sdk import TurnstoneServer
with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
ws = client.create_workstream(name="demo")
result = client.send_and_wait("Analyze the error logs", ws.ws_id, auto_approve=True)
print(result.content)
Tools
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via MCP with native deferred loading. See docs/tools.md for the full reference and docs/mcp-registry.md for MCP configuration.
Architecture
Single-node: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
Multi-node: Client → Console (rendezvous routing proxy) → Server nodes. The console picks the target node for each workstream via rendezvous (HRW) hashing over the live service registry — pure function of (ws_id, live_nodes), no stored bucket state, deterministic across readers. A node join or drop only re-routes the keys that score highest on the affected node.
| Component | Purpose |
|---|---|
turnstone |
Terminal CLI (REPL) |
turnstone-server |
Web UI + REST API + SSE events |
turnstone-console |
Cluster dashboard + routing proxy + admin panel |
turnstone-channel |
Channel gateway (Discord and Slack adapters) |
turnstone-admin |
User/token management CLI |
turnstone-eval |
Eval harness for prompt/tool optimization |
turnstone-bootstrap |
LLM-guided setup wizard |
Diagrams
UML diagrams in docs/diagrams/:
| Diagram | Description |
|---|---|
| System Context | Components and external dependencies |
| Package Structure | Python modules and dependency graph |
| Core Engine | SessionUI, ChatSession, LLMProvider |
| Conversation Turn | Message lifecycle through the engine |
| Tool Pipeline | Prepare / approve / execute |
| Workstream States | State machine transitions |
| Console Data Flow | Dashboard data collection |
| Deployment | Docker Compose topology |
| Auth | JWT, scopes, login flows |
| Channels | Discord / Slack adapters + routing |
| Judge | Intent validation pipeline |
| OIDC | SSO authorization code flow |
Documentation
| Topic | Link |
|---|---|
| Configuration reference | docs/settings.md |
| API reference | docs/api-reference.md |
| Docker deployment | docs/docker.md |
| Intent validation (judge) | docs/judge.md |
| Governance & RBAC | docs/governance.md |
| OIDC SSO | docs/oidc.md |
| TLS / mTLS | docs/tls.md |
| Channel integrations | docs/channels.md |
| Console dashboard | docs/console.md |
| Eval harness | docs/eval.md |
| Tools reference | docs/tools.md |
| MCP integration | docs/mcp-registry.md |
Requirements
- Python 3.11+
- An OpenAI-compatible API endpoint, Anthropic API key, or Google Gemini API key
- Optional: PostgreSQL (
pip install turnstone[postgres]), Anthropic (pip install turnstone[anthropic]) - Git LFS for cloning (diagram PNGs)
License
Business Source License 1.1 — free for all use except hosting as a managed service. Converts to Apache 2.0 on 2030-03-01.
