turnstone/docs/architecture.md

Turnstone Architecture

Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or Anthropic's native Messages API via pluggable provider adapters, and gives the model 16 built-in tools plus external tools via MCP (Model Context Protocol) for reading, writing, searching, and executing code.

The core design principle is a UI-agnostic engine with pluggable frontends. The engine (ChatSession) drives the conversation loop -- streaming, tool dispatch, retry, compaction -- while every user-facing interaction is delegated through the SessionUI protocol. Any frontend implements that protocol and plugs in.

Entry Points

Command	Module	Frontend	Purpose
turnstone	turnstone.cli	TerminalUI	Interactive terminal REPL
turnstone-server	turnstone.server	WebUI	Browser-based chat (HTTP + SSE)
turnstone-console	turnstone.console.server	ClusterCollector	Cluster dashboard (aggregates all nodes)
turnstone-eval	turnstone.eval	NullUI	Headless evaluation and prompt optimization
turnstone-channel	turnstone.channels.cli	ChannelAdapter	Channel gateway (Discord, Slack, etc.)
turnstone-admin	turnstone.admin	—	Offline user and API token management
turnstone-bootstrap	turnstone.bootstrap	—	LLM-guided setup wizard

---

Module Map

turnstone/
  cli.py              Terminal frontend (TerminalUI, WorkstreamTerminalUI, REPL)
  server.py           Web frontend (WebUI, HTTP handler, static-file serving)
  eval.py             Evaluation harness (HeadlessSession, scoring, prompt optimization)
  core/
    session.py        ChatSession engine, SessionUI protocol, tool dispatch
    providers/        LLM provider adapters (pluggable backend layer)
      _protocol.py    LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
      _openai.py      OpenAIProvider facade (re-exports Chat/Responses providers)
      _openai_chat.py       OpenAIChatCompletionsProvider — vLLM, llama.cpp, local compatible APIs
      _openai_responses.py  OpenAIResponsesProvider — commercial OpenAI Responses API
      _openai_common.py     Shared ModelCapabilities table + helpers
      _anthropic.py   AnthropicProvider — Anthropic Messages API, native streaming, thinking
      _google.py      GoogleProvider — Google Gemini via OpenAI-compat endpoint
      __init__.py     create_provider() + create_client() factory functions
    workstream.py     Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
    tools.py          Tool schema loader (JSON -> OpenAI function-calling format)
    mcp_client.py     MCPClientManager — MCP server connections, tool discovery, dynamic refresh
    tool_search.py    Dynamic tool search — BM25 index, session-scoped tool visibility
    watch.py          WatchRunner daemon — periodic command polling, condition DSL, result dispatch
    judge.py          Intent validation — heuristic rules + LLM judge, advisory verdicts
    model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
    memory.py         Persistence facade + structured memory API (delegates to storage backend)
    config.py         Config file loader (config.toml), apply_config(), warn_migrated_settings()
    config_store.py   ConfigStore — database-backed settings with in-memory cache, thread-safe get/set
    settings_registry.py  SettingDef catalog (~40 settings), validation, type coercion, serialization
    storage/          Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
    metrics.py        Prometheus-compatible metrics collector (MetricsCollector)
    healthcheck.py    BackendHealthMonitor — periodic probe + circuit breaker
    ratelimit.py      Per-IP token-bucket rate limiter (RateLimiter, TokenBucket)
    edit.py           File edit utilities (find_occurrences, pick_nearest)
    safety.py         Command safety validation (blocked patterns, sanitization)
    web.py            Web utilities (HTML stripping, SSRF prevention)
  api/
    schemas.py        Shared Pydantic v2 models (auth, errors, WorkstreamState)
    server_schemas.py Server endpoint request/response models
    console_schemas.py Console endpoint request/response models
    openapi.py        OpenAPI 3.1 spec builder
    server_spec.py    Server endpoint catalog → build_server_spec()
    console_spec.py   Console endpoint catalog → build_console_spec()
    docs.py           /openapi.json + /docs (Swagger UI) handler factories
  sdk/
    server.py         AsyncTurnstoneServer + TurnstoneServer (HTTP client)
    console.py        AsyncTurnstoneConsole + TurnstoneConsole (HTTP client)
    events.py         27 SSE event dataclasses with type registry
    _base.py          Shared httpx async client, auth, error handling
    _sync.py          Background event loop for sync wrappers
    _types.py         TurnResult + TurnstoneAPIError
  console/
    collector.py      ClusterCollector — aggregates state from all nodes via SSE
    scheduler.py      TaskScheduler — background cron/at scheduler, dispatches via HTTP
    server.py         Cluster dashboard HTTP server + SSE + CLI entry point
    static/           Cluster dashboard web UI (page-specific HTML, CSS, JS)
  channels/
    cli.py            Unified channel gateway entry point (turnstone-channel)
    _protocol.py      ChannelAdapter protocol
    _routing.py       ChannelRouter — channel/thread ↔ workstream mapping via HTTP
    _config.py        Base ChannelConfig dataclass
    discord/          Discord adapter (bot, cog, views, streaming, config)
    slack/            Slack adapter (Socket Mode bot, DM routing, approval buttons)
  shared_static/      Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
    katex-0.17.0/    Vendored KaTeX math rendering library (MIT, woff2 fonts)
  ui/
    colors.py         ANSI color constants with NO_COLOR support
    markdown.py       Streaming terminal markdown renderer (line-buffered)
    spinner.py        Braille character spinner (daemon thread)
    static/
      index.html      Single-page app shell (links to CSS and JS)
      style.css       Page-specific UI styles (dashboard, markdown elements, approval blocks)
      renderer.js     Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
      app.js          Split-pane UI (Pane class, binary layout tree, SSE, tool approval)
  tools/
    *.json            16 tool schemas (OpenAI function-calling format + turnstone metadata)

Both UIs share a common design system extracted into turnstone/shared_static/: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports base.css and the shared JS modules at /shared/, then adds only page-specific code at /static/.

---

Core Loop

See also: Conversation Turn diagram

A user message flows through the system as follows:

 User input
     |
     v
 ChatSession.send(user_input)
     |
     v
 _full_messages()  ------------>  system_messages + self.messages
     |
     v
 _emit_state("thinking")
     |
     v
 _create_stream_with_retry()  ---->  provider.create_streaming(client, model, messages, ...)
     |                                  up to 3 retries (4 total attempts), exponential backoff
     v
 _stream_response(stream)  -------->  dispatch tokens to UI:
     |                                  on_reasoning_token() / on_content_token()
     |                                  accumulate tool_calls from deltas
     |                                  track finish_reason
     |                                  _check_cancelled() per chunk (cooperative cancel)
     v
 finish_reason check:
     +--- "length"  --> warn, discard partial tool_calls
     +--- "content_filter" --> warn
     v
 tool_calls present?
     |
     +--- No ---> _print_status_line() -> _emit_state("idle") -> return
     |
     +--- Yes --> _emit_state("running")
                    |
                    v
                  _execute_tools(tool_calls)  <--- three-phase pipeline (see below)
                    |
                    v
                  append tool results to self.messages
                    |
                    v
                  loop back to _full_messages()

Tool Execution Pipeline

See also: Tool Pipeline diagram

Tool execution is a three-phase process:

Phase 1: PREPARE (serial)
  For each tool_call:
    _prepare_tool(tc)
      -> parse JSON arguments (with regex fallback for malformed JSON)
      -> dispatch to _prepare_{tool_name}(call_id, args)
      -> validate inputs, build preview text
      -> return item dict with: header, preview, needs_approval, execute fn

Phase 2: APPROVE (serial, blocking)
  _emit_state("attention")
  ui.approve_tools(items)
    -> display all headers and previews
    -> if any need approval and not auto_approve: prompt user
    -> return (approved, feedback)
  _emit_state("running")

Phase 3: EXECUTE (parallel)
  _check_cancelled()  <-- cancellation checkpoint before execution starts
  if len(items) == 1:
    run_one(items[0])
  else:
    ThreadPoolExecutor(max_workers=4).map(run_one, items)
  Bash tool streams stdout line-by-line via ui.on_tool_output_chunk(call_id, line)
    (cancel_event also checked per line — kills process group on cancel)
  Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
  call_id links tool_info items → streaming chunks → final result

State Transitions

The engine emits state changes via _emit_state() which calls ui.on_state_change(state). Frontends use these to update indicators (spinner, tab badges, status line).

  send() called
      |
      v
  "thinking"  --->  streaming response
      |
      v
  "running"   --->  tool execution
      |
      v
  "attention"  --->  waiting for user approval
      |
      v
  "running"   --->  executing approved tools
      |
      v
  "idle"       --->  no more tool calls, turn complete
      |
  (or "error"  --->  exception or KeyboardInterrupt)

  cancel() may be called from any state. It sets a cooperative flag
  checked at each streaming chunk, before tool execution, and inside
  bash commands. The session transitions to "idle" with partial
  content preserved, emitting on_info("[Generation cancelled]").

---

SessionUI Protocol

See also: Core Engine Classes diagram

Defined in turnstone.core.session.SessionUI as a typing.Protocol with 15 methods. Every frontend must implement all of them.

class SessionUI(Protocol):
    def on_turn_start(self) -> None: ...
    def on_turn_committed(self) -> None: ...
    def on_thinking_start(self) -> None: ...
    def on_thinking_stop(self) -> None: ...
    def on_reasoning_token(self, text: str) -> None: ...
    def on_content_token(self, text: str) -> None: ...
    def on_stream_end(self) -> None: ...
    def approve_tools(self, items: list[dict]) -> tuple[bool, str | None]: ...
    def on_tool_result(self, call_id: str, name: str, output: str, *, is_error: bool = False) -> None: ...
    def on_tool_output_chunk(self, call_id: str, chunk: str) -> None: ...
    def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
    def on_info(self, message: str) -> None: ...
    def on_error(self, message: str) -> None: ...
    def on_state_change(self, state: str) -> None: ...
    def on_rename(self, name: str) -> None: ...  # propagate alias to tab/UI label

on_turn_start fires at the top of each iteration of the send-loop; on_turn_committed fires immediately after messages.append(assistant_msg). SessionUIBase uses both to reset the per-turn inflight buffers (_ws_inflight_content / _ws_inflight_reasoning / _ws_inflight_seq) that fuel the SSE refresh-resume in_progress_snapshot event — see the per-workstream events stream in docs/api-reference.md.

on_rename is called by the /name command (on success) and after a successful /resume (if the resumed session has an alias or title). WebUI.on_rename broadcasts a ws_rename event on the global SSE channel and updates the in-memory Workstream.name; TerminalUI.on_rename is a no-op.

Three Implementations

Class	Module	Notes
TerminalUI	turnstone.cli	ANSI colors, MarkdownRenderer, Spinner, readline-based input() for approval
WebUI	turnstone.server	SSE event queue per workstream + global broadcast, threading.Event for blocking on approval. on_state_change sends to both per-workstream and global SSE (the browser UI uses per-workstream state_change events to manage busy/idle transitions; stream_end only finalizes markdown rendering).
NullUI	turnstone.eval	Discards all output; approve_tools always returns (True, None)

WorkstreamTerminalUI

WorkstreamTerminalUI (in turnstone.cli) extends TerminalUI with workstream awareness:

• Output buffering: When in background (is_foreground is False), tokens are appended to _output_buffer instead of written to stdout. When the user switches to this workstream, flush_buffer() replays them.

• Approval blocking: approve_tools() calls _fg_event.wait() when in background, blocking the worker thread until the workstream is foregrounded. This ensures the user sees the approval prompt in the correct context.

• Foreground/background toggle: set_foreground(bool) sets or clears _fg_event (a threading.Event). The manager calls this during /ws <N> switches.

---

Workstream Architecture

Workstreams are parallel, independent chat sessions. Each has its own ChatSession, SessionUI, message history, and worker thread.

WorkstreamState

See also: Workstream States diagram

Defined in turnstone.core.workstream.WorkstreamState (5 states):

IDLE       waiting for user input
THINKING   LLM is streaming a response
RUNNING    tools are executing
ATTENTION  blocked on user approval or plan review
ERROR      last operation failed

Data Model

@dataclass
class Workstream:
    id: str                              # uuid hex, 8 chars
    name: str                            # user-visible label
    state: WorkstreamState               # current state
    session: ChatSession | None          # the conversation engine
    ui: SessionUI | None                 # frontend adapter
    worker_thread: threading.Thread | None
    error_message: str
    last_active: float                   # time.monotonic() timestamp, updated on every state change
    _lock: threading.Lock                # per-workstream state lock

WorkstreamManager

class WorkstreamManager:
    MAX_WORKSTREAMS = 10

    def __init__(self, session_factory: Callable[[SessionUI], ChatSession]): ...
    def create(self, name="", ui_factory=None) -> Workstream: ...
    def close(self, ws_id: str) -> bool: ...
    def close_idle(self, max_age_seconds: float) -> list[str]: ...  # auto-close stale IDLE workstreams
    def get(self, ws_id: str) -> Workstream | None: ...
    def get_active(self) -> Workstream | None: ...
    def list_all(self) -> list[Workstream]: ...
    def switch(self, ws_id: str) -> Workstream | None: ...
    def switch_by_index(self, index: int) -> Workstream | None: ...
    def set_state(self, ws_id, state, error_msg=""): ...  # updates last_active

The session_factory pattern decouples session creation from configuration. The factory captures shared config (client, model, temperature, etc.) and accepts only a SessionUI, so the manager can create sessions without knowing API details.

Idle Workstream Lifecycle

The web server runs a background _idle_cleanup_thread (daemon) that calls WorkstreamManager.close_idle() periodically (every timeout / 4, max 5 min). Any IDLE workstream whose last_active is older than the configured timeout is closed; non-IDLE workstreams (THINKING, RUNNING, ATTENTION, ERROR) are never touched. The last workstream is always preserved even if expired. On close, a ws_closed event is broadcast on the global SSE channel so browser clients remove the tab immediately. Controlled by --workstream-idle-timeout (default: 120 minutes, 0 = disable).

Workstream eviction at capacity: When WorkstreamManager.create() would exceed max_workstreams (configurable via [server].max_workstreams, default 50), the oldest IDLE workstream is automatically evicted to make room. The turnstone_workstreams_evicted_total counter is incremented on each eviction. If no IDLE workstream is available the create request fails as before.

CLI Workstreams

• /ws list -- show all workstreams with state indicators
• /ws new [name] -- create a new workstream and switch to it
• /ws <N> -- switch to workstream by 1-based index
• /ws close [N] -- close a workstream
• /ws rename <name> -- rename the active workstream

Background notifications: when a background workstream enters ATTENTION state, _bg_attention_notify writes an ANSI escape sequence to stderr (overwrites the line above the prompt) with the workstream name.

Status line: _print_ws_status_line() shows a compact status of all non-idle background workstreams above the input prompt.

Web Workstreams

• Tab bar: Each workstream renders as a tab with a colored state indicator (CSS @keyframes pulse animation per state). Clicking a tab switches the focused pane's workstream (or focuses an existing pane showing that ws).
• Split panes: The UI supports tiling multiple workstreams side-by-side or stacked via a binary layout tree. Each Pane instance encapsulates its own SSE connection, message area, input, and state (busy, approval, streaming). Split via right-click context menu, pane header buttons, or keyboard (Ctrl+\, Ctrl+Shift+\). Max 6 panes; no duplicate workstreams across panes. Layout persisted to localStorage.
• Per-pane SSE: Pane.connectSSE(wsId) opens /v1/api/workstreams/{ws_id}/events for each pane's event stream independently.
• Global SSE: connectGlobalSSE() opens /v1/api/events/global which receives ws_state broadcasts from all workstreams, used to update tab indicators and pane headers without switching.
• New tab / close: POST /v1/api/workstreams/new, POST /v1/api/workstreams/{ws_id}/close.

Thread Safety

• WorkstreamManager._lock: guards _workstreams dict and _order list on all create/close/switch/list operations.
• Workstream._lock: guards per-workstream state mutations in set_state().
• WorkstreamTerminalUI._print_lock: guards _output_buffer access.
• WorkstreamTerminalUI._fg_event: threading.Event that blocks background approval until the workstream is foregrounded.

---

Tool System

Schema Format

Each tool is a JSON file in turnstone/tools/. The file contains an OpenAI function-calling schema (name, description, parameters) plus optional turnstone metadata keys:

Metadata Key	Type	Meaning
task_agent	bool	Include this tool when running as a task sub-agent
auto_approve	bool	Tool is read-only; skip user approval
primary_key	str	Fallback argument name for bare-string JSON recovery

Example (read_file.json):

{
  "name": "read_file",
  "description": "Read the contents of a file. ...",
  "parameters": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "..." },
      "offset": { "type": "integer", "description": "..." },
      "limit": { "type": "integer", "description": "..." }
    },
    "required": ["path"]
  },
  "task_agent": true,
  "auto_approve": true,
  "primary_key": "path"
}

At import time, turnstone.core.tools._load_tools() strips the metadata keys from each schema and builds:

• TOOLS -- list of {"type": "function", "function": {...}} dicts for the API
• TASK_AGENT_TOOLS -- subset with task_agent: true
• TASK_AUTO_TOOLS -- set of tool names with auto_approve: true
• PRIMARY_KEY_MAP -- {name: primary_key} for JSON fallback recovery
• merge_mcp_tools(builtin, mcp_tools) -- merges built-in + MCP tools at session init

16 Tools by Category

Read-only (auto-approve):

• read_file -- read file contents with optional offset/limit
• diff_file -- show diff between two files / versions
• search -- ripgrep-based codebase search
• recall -- search conversation history
• read_resource -- read an MCP resource by URI

Write (requires approval):

• bash -- execute shell commands (with safety checks via turnstone.core.safety)
• write_file -- create or overwrite a file
• edit_file -- string replacement in an existing file (requires prior read_file)
• web_fetch -- fetch a URL (with SSRF protection via turnstone.core.web)
• web_search -- search the web (provider-native for Anthropic/OpenAI, self-hosted SearxNG fallback for local models)
• notify -- send a user-facing notification (Discord/Slack, optional reply routing)
• watch -- schedule a recurring poll with condition DSL

Agent (delegated sub-sessions):

• task_agent -- delegate to a sub-agent with full tool access (TASK_AGENT_TOOLS)

Memory / skills / prompts:

• memory -- save, search, delete, or list memories (typed and scoped)
• skill -- invoke a skill (governed, versioned procedure)
• use_prompt -- fetch and apply a prompt template

The tool name uses the _agent suffix — bare task collides with chat-template channels on some local models.

Prepare / Execute Pattern

Every tool has a _prepare_{name} method and a corresponding _exec_{name} method on ChatSession:

_prepare_bash(call_id, args)   -> item dict with execute=self._exec_bash
_prepare_read_file(call_id, args) -> item dict with execute=self._exec_read_file
...

The prepare method validates inputs and builds the preview. The item dict carries the validated data and a reference to the execute function. This separation allows the UI to show previews before any side effects occur.

Agent Tools

task_agent invokes _run_agent(), which runs a multi-turn loop with a subset of tools and its own system prompt. The sub-agent runs independently, then returns the final content as the tool result.

• task_agent: uses self._task_tools (TASK_AGENT_TOOLS + MCP tools)
• Turn limit: controlled by agent_max_turns (default: -1, unlimited). When a limit is set and reached, the agent is forced to synthesize a final response without tools. When unlimited, the loop only exits when the model stops calling tools or hits finish_reason: "length".
• Retry: each API call in the agent loop uses the same retry+backoff logic as the main _create_stream_with_retry().
• Finish reason handling: finish_reason: "length" stops the agent early and returns whatever content was generated. finish_reason: "content_filter" returns a placeholder.

MCP Tool Integration

MCPClientManager (turnstone/core/mcp_client.py) connects to external MCP servers and exposes their tools alongside built-in tools. The MCP SDK is fully async; turnstone bridges this with a background asyncio event loop in a daemon thread.

Configuration sources: MCP servers can be defined in config files (TOML/JSON) or in the database via the admin UI. Database-backed definitions are managed through the console admin panel's MCP Servers tab and stored in the mcp_servers table. On startup, load_mcp_config(storage=) uses first-match-wins priority: DB rows (if any enabled) take precedence over config files. The console can trigger a cluster-wide reload (POST /_internal/mcp-reload) that causes each node to call reconcile_sync(), which diffs the running MCP connections against the current DB state and adds, removes, or reconnects servers as needed.

Lifecycle:

1. create_mcp_client() reads server configs from TOML/JSON and database
2. MCPClientManager.start() launches the background event loop thread
3. _connect_all() connects to each server (stdio subprocess or HTTP), runs initialize() + list_tools(), converts schemas to OpenAI format, detects tools.listChanged capability for push notification support
4. ChatSession.__init__ receives the manager, builds self._tools (built-in + MCP), and registers a listener callback for tool-change notifications
5. _prepare_tool() routes MCP tools to _prepare_mcp_tool() / _exec_mcp_tool()
6. _exec_mcp_tool() calls call_tool_sync() which dispatches to the async loop via asyncio.run_coroutine_threadsafe()

Tool refresh: Two mechanisms keep tools up-to-date without restart:

• Push: Servers declaring tools.listChanged send ToolListChangedNotification; the registered message_handler triggers immediate single-server refresh.
• Manual: /mcp refresh [server] calls refresh_sync() for on-demand refresh (also attempts reconnection for disconnected servers).

When tools change, _rebuild_tools() creates new _tools/_tool_map objects (copy-on-write for thread safety) and notifies listener callbacks. Each ChatSession rebuilds its _tools and _task_tools lists and reconstructs ToolSearchManager (preserving expanded tools).

Tool naming: mcp__{server}__{tool} — double underscore delimiter, validated at connection time (server names with __ are rejected).

Resilience: Each MCP server has an independent circuit breaker that opens after 3 consecutive transport failures (timeouts, broken pipes, connection resets). Cooldown uses capped exponential backoff (30 s base, 5 min max) with per-server jitter to avoid thundering herd. Protocol-level errors (McpError) from a healthy connection do not trip the breaker. When the cooldown expires (half-open), the next operation attempt triggers automatic reconnection. Manual /mcp refresh also clears the circuit on success. All sync bridge methods (call_tool_sync, read_resource_sync, get_prompt_sync, refresh_sync) cancel orphaned futures on timeout to prevent coroutine accumulation on the background event loop. Push notification refreshes are debounced (5 s per server) to protect against notification storms. Operators can force a catalog refresh or full reconnect from the admin panel; reconnects clear the circuit breaker and run a fresh handshake. Transport stream references are pre-closed before stack teardown to work around the MCP SDK's anyio cancel-scope CPU busy-loop (SDK #2147).

Error isolation: Per-server connection/refresh failures are caught and logged; other servers are unaffected. Tool execution errors return error strings to the LLM rather than crashing the session.

Registry discovery: The console admin panel provides a registry discovery surface backed by the official MCP Registry (registry.modelcontextprotocol.io). MCPRegistryClient (turnstone/core/mcp_registry.py) is a standalone httpx async client that queries the registry's v0.1 API for server discovery. Search results are annotated with installed status by cross-referencing the mcp_servers table. Installation creates a DB row with registry_name, registry_version, and registry_meta columns (migration 019), then triggers cluster-wide node reload via _notify_nodes_mcp_reload(). The registry URL is configurable via the mcp.registry_url setting for enterprise/private registries.

Provider Adapter Layer

See also: Core Engine Classes diagram

ChatSession is provider-agnostic — it delegates all LLM communication to an LLMProvider protocol (turnstone/core/providers/_protocol.py). Internally, messages use an OpenAI-like format; each provider translates at the API boundary.

ChatSession
    |
    v
LLMProvider (protocol)
    |
    +--- OpenAIProvider  --- OpenAI, vLLM, llama.cpp, any /v1/chat/completions API
    +--- AnthropicProvider --- Anthropic Messages API (native streaming, thinking)
    +--- GoogleProvider  --- Google Gemini via /v1beta/openai/ (extends OpenAIProvider)

Protocol methods:

Method	Purpose
create_streaming()	Streaming request, yields normalized StreamChunk objects
create_completion()	Non-streaming request, returns CompletionResult
get_capabilities()	Per-model flags (ModelCapabilities)
convert_tools()	Translate OpenAI tool schemas to provider format
retryable_error_names	Exception class names that trigger retry
extract_reasoning_text()	Walk stored provider_blocks, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic thinking, OpenAI Responses reasoning, OpenAI Chat synthetic reasoning_text)

Normalized data types:

Type	Fields
StreamChunk	content_delta, reasoning_delta, tool_call_deltas, info_delta, usage, finish_reason, provider_blocks
CompletionResult	content, tool_calls, finish_reason, usage, provider_blocks
ModelCapabilities	context_window, max_output_tokens, supports_temperature, token_param, thinking_mode, supports_effort, supports_web_search, supports_tool_search, supports_vision, supports_reasoning_replay
UsageInfo	prompt_tokens, completion_tokens, total_tokens, cache_creation_tokens, cache_read_tokens

OpenAIProvider (_openai.py): passes messages through unchanged (they are already in OpenAI format), including multi-part content blocks (text + images) in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4, O-series, and search models (gpt-5-search-api) — all with supports_vision. For search models, injects web_search_options and removes the web_search function tool (the model always searches). Citations from url_citation annotations are formatted as footnotes. Extended prompt cache retention (prompt_cache_retention: "24h") is enabled for GPT-5.x models at no additional cost. Cached token counts are extracted from usage.prompt_tokens_details.cached_tokens. Unknown models (local servers) get permissive defaults with supports_vision=False and use SearxNG for web search.

AnthropicProvider (_anthropic.py): converts OpenAI-format messages to Anthropic content blocks, maps system/developer roles to the system parameter, groups consecutive tool result messages into user-role content blocks (converting image_url parts to Anthropic's image source format), and translates tool schemas from OpenAI function-calling format to Anthropic's input_schema format. Supports both manual and adaptive thinking modes, with effort parameter support for models like Claude Opus 4.6 and Sonnet 4.6. Replaces the web_search function tool with Anthropic's native web_search_20250305 server-side tool — Claude decides when to search, the API executes it, and results stream back as server_tool_use / web_search_tool_result content blocks (emitted as info_delta for UI display). Automatic prompt caching is enabled via top-level cache_control: {"type": "ephemeral"} — the API places the cache breakpoint on the last cacheable block and advances it as conversations grow (90% input cost reduction on cache hits, 1.25x write on first turn). Cache metrics (cache_creation_input_tokens, cache_read_input_tokens) are extracted from both streaming and non-streaming responses. The anthropic SDK is imported lazily so it remains an optional dependency (pip install turnstone[anthropic]).

GoogleProvider (_google.py): extends OpenAIChatCompletionsProvider for the Gemini /v1beta/openai/ endpoint. Uses a single default ModelCapabilities (2M context window, 65K max output tokens, token_param=max_tokens) since Google updates models frequently. No static per-model capability table. Google's endpoint is wire-compatible with the OpenAI SDK, so no extra dependency is needed.

Factory functions (__init__.py): create_provider(name) returns a singleton provider instance (thread-safe). create_client(name, base_url, api_key) creates the appropriate SDK client.

Multi-Model Registry

ModelRegistry (turnstone/core/model_registry.py) manages named model configurations so workstreams can use different LLM backends.

Config format:

[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
# provider defaults to "openai"

[models.claude]
provider = "anthropic"
api_key = "sk-ant-..."
model = "claude-opus-4-6"
context_window = 200000

[models.openai]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "gpt-5"
context_window = 400000

[models.gemini]
provider = "google"
model = "gemini-2.5-pro"

[model]
default = "local"
fallback = ["claude", "openai"]
agent_model = "claude"

Each [models.*] entry produces a ModelConfig with a provider field (default: "openai"). Supported values: "openai", "anthropic", "google", and "openai-compatible".

Per-model sampling overrides: Each model can specify temperature, max_tokens, and reasoning_effort to override the global defaults from ConfigStore. When unset (NULL), the global default is used.

Per-model reasoning persistence: Two booleans on model_definitions (migration 052) control how reasoning text round-trips:

• surface_persisted_reasoning (default True) — gates whether stored reasoning text is surfaced on /history payloads for UI rehydration. Storage of reasoning bytes happens regardless of this flag — they ride in provider_data independently. Phase-1 admin UI label "Surface persisted reasoning."
• replay_reasoning_to_model (default False) — gates whether stored reasoning blocks are sent back to the provider on subsequent turns. Capability-gated: ModelCapabilities.supports_reasoning_replay must also be True for the wire path to actually replay (canonical OpenAI gpt-5*/o-series and Anthropic Claude entries set it; unknown / local- server models default to False).

Three reasoning paths are recognised:

Path	Provider	Capture	Persist	Replay
1	Anthropic Messages API	thinking_delta	provider_blocks (type="thinking")	Verbatim via _provider_content
2	OpenAI Responses (gpt-5*, o-series)	response.reasoning_text.delta events	provider_blocks (type="reasoning") — only when include=["reasoning.encrypted_content"]	ResponseReasoningItemParam input items
3	OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat)	delta.reasoning_content Pydantic extras	Synthetic {type: "reasoning_text", text, source} block stamped at end-of-stream	None — no API surface for replay on Chat Completions

Cross-provider safety is enforced by ANTHROPIC_VALID_BLOCK_TYPES (a shape filter in _anthropic.py:_convert_messages): foreign blocks (OpenAI reasoning, synthetic reasoning_text) fall through to the text+tool_calls rebuild path rather than reaching Anthropic's input boundary as malformed content.

[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
temperature = 0.7
max_tokens = 8192

[models.o3]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "o3"
reasoning_effort = "high"
# temperature omitted — uses global default

An optional [models.*.capabilities] sub-table overrides per-model ModelCapabilities flags (useful for local models whose capabilities cannot be detected programmatically):

[models.qwen-vl]
base_url = "http://localhost:8000/v1"
model = "qwen-3.5-vl"

[models.qwen-vl.capabilities]
supports_vision = true

Database model definitions: On server entry points, models can also be defined in the model_definitions table (admin Models tab). DB models support the same per-model sampling overrides. Config.toml models override DB models with the same alias in-memory (the DB rows are never modified).

Lifecycle:

1. load_model_registry() loads DB model definitions (if storage available), then overlays [models.*] from config.toml, then builds a "default" entry from CLI --base-url/--model/--api-key args
2. The registry is passed to the session factory closure in both cli.py and server.py; each workstream resolves its model on creation
3. ModelRegistry.get_client() lazily creates SDK client instances via create_client() — OpenAI for the openai provider, Anthropic for the anthropic provider (thread-safe via _client_lock)
4. ModelRegistry.get_provider() lazily creates LLMProvider instances via create_provider() (also cached and thread-safe)
5. /model command shows available models; /model <alias> switches the active workstream's client, model, context window, and per-model sampling parameters
6. _create_stream_with_retry() tries the primary model, then each fallback alias in order if the primary is unreachable
7. _run_agent() resolves registry.agent_model (if set) for task sub-agents, allowing a cheaper model for autonomous loops

Per-workstream selection: POST /v1/api/workstreams/new accepts an optional "model" field, along with skill (skill name) which can override the model before workstream creation.

Tool Output Truncation

Tool execution results (bash, read_file, search) are truncated by _truncate_output() when they exceed tool_truncation characters. Truncation preserves the first half and last half of the output, with a message in between:

... [N chars truncated — output exceeded LIMIT char limit] ...

The default limit is 50% of the context window in characters (computed as context_window * chars_per_token * 0.5). For a 131K context window this is ~262K characters. Override with --tool-truncation <chars>.

This truncation message is visible to the model, so it knows output was cut.

---

Persistence

Storage Architecture

Persistence is managed by the turnstone.core.storage package — a pluggable backend behind a StorageBackend protocol. The memory.py facade provides backward-compatible module-level functions that delegate to the active backend.

session.py / server.py / cli.py
        ↓
    memory.py  (facade — silent-failure wrappers)
        ↓
    storage._registry  (singleton factory)
        ↓
  ┌─────────────┐    ┌──────────────────┐
  │ SQLiteBackend │    │ PostgreSQLBackend │
  │ (FTS5 search) │    │ (tsvector/ILIKE)  │
  └─────────────┘    └──────────────────┘
        ↓                     ↓
    storage._schema  (SQLAlchemy Core tables — single source of truth)
        ↓
    storage._migrate  (programmatic Alembic)

SQLite is the default (zero-config, single file at .turnstone.db). PostgreSQL is the production backend (connection pooling, tsvector full-text search). Select via [database] in config.toml, CLI flags, or environment variables (TURNSTONE_DB_BACKEND, TURNSTONE_DB_URL).

Schema migrations are managed by Alembic and run automatically on startup. Existing SQLite databases created before the migration system are auto-stamped at the baseline revision.

Tables

memories
  key      TEXT PRIMARY KEY
  value    TEXT NOT NULL
  created  TEXT NOT NULL
  updated  TEXT NOT NULL

workstreams
  ws_id       TEXT PRIMARY KEY
  node_id     TEXT NOT NULL
  name        TEXT NOT NULL
  state       TEXT NOT NULL DEFAULT 'idle'
  alias       TEXT UNIQUE            -- user-assigned short name (nullable)
  title       TEXT                   -- LLM-generated title (nullable)
  created     TEXT NOT NULL
  updated     TEXT NOT NULL          -- bumped on every save_message()

conversations
  id            INTEGER PRIMARY KEY AUTOINCREMENT
  ws_id         TEXT NOT NULL
  timestamp     TEXT NOT NULL
  role          TEXT NOT NULL        -- user | assistant | tool_call | tool_result
  content       TEXT
  tool_name     TEXT
  tool_args     TEXT
  tool_call_id  TEXT                 -- links tool_call ↔ tool_result for resume
  provider_data TEXT                 -- raw provider content (e.g. Anthropic encrypted)

workstream_config
  ws_id       TEXT NOT NULL          -- composite PK with key
  key         TEXT NOT NULL
  value       TEXT

conversations_fts                    -- SQLite FTS5 virtual table (optional)
  content     (content=conversations, content_rowid=id)

Table definitions live in storage/_schema.py (SQLAlchemy Core Table objects) and are the single source of truth for both backends and Alembic migrations.

StorageBackend Protocol

Method	Purpose
register_workstream(ws_id, node_id, name, state)	Create a workstreams row (no-op if exists)
save_message(ws_id, role, content, ...)	Log a message to conversations
load_messages(ws_id)	Reconstruct OpenAI message format from DB rows
list_workstreams_with_history(limit)	List workstreams with >=1 message, ordered by updated DESC
delete_workstream(ws_id)	Delete workstream and cascade conversations + config
prune_workstreams(retention_days)	Remove empty workstreams and old unnamed workstreams
resolve_workstream(alias_or_id)	Resolve alias, exact id, or id prefix to full ws_id
save_workstream_config(ws_id, config)	Persist workstream configuration key/value pairs
load_workstream_config(ws_id)	Retrieve workstream configuration
set_workstream_alias(ws_id, alias)	Set user-friendly alias (returns False if taken)
get_workstream_display_name(ws_id)	Return alias if set, else title, else None
update_workstream_title(ws_id, title)	Set/update LLM-generated title
update_workstream_state(ws_id, state)	Update workstream state and bump timestamp
update_workstream_name(ws_id, name)	Update workstream display name
list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id)	List workstreams, optionally filtered by node, parent, kind, or owning user
kv_get(key) / kv_set(key, value) / kv_delete(key)	Generic key-value store (backs memories table)
kv_list() / kv_search(query)	List or search key-value pairs
search_history(query, limit)	Full-text search (FTS5 on SQLite, tsvector on PostgreSQL)
search_history_recent(limit)	Return most recent messages
close()	Release resources (connection pool, engine)

Database Configuration

[database]
backend = "sqlite"                  # "sqlite" | "postgresql"
path = ".turnstone.db"              # SQLite file path
url = ""                            # PostgreSQL connection URL
pool_size = 2                       # PostgreSQL connection pool size (per process)

Environment variables: TURNSTONE_DB_BACKEND, TURNSTONE_DB_URL, TURNSTONE_DB_PATH, TURNSTONE_DB_POOL_SIZE.

The default pool is intentionally small (2 base + 3 overflow = 5 per process) because all database operations are short-burst queries that hold connections for milliseconds. For clusters with many nodes sharing a PostgreSQL instance, use PgBouncer in transaction pooling mode.

Persistence and Resume

ws_id is the sole persistent identity for both routing and conversation history. There is no separate session_id — the workstreams table holds alias, title, and state alongside the routing fields (node_id, name). Messages are saved to conversations (keyed by ws_id) as they happen via save_message(). Workstream state changes are tracked via update_workstream_state().

Auto-titling: After the first complete exchange (user message + assistant response), a background thread calls the LLM with a title-generation prompt (reasoning_effort: "low", max_completion_tokens: 200). The generated title (3-8 words) is stored in workstreams.title.

Resume flow: ChatSession.resume(ws_id) calls load_messages() which reconstructs the OpenAI message format from database rows:

• user and assistant rows map directly
• Consecutive tool_call rows are grouped into one assistant message's tool_calls array, paired with subsequent tool_result rows via tool_call_id (or positional matching for legacy data)
• Interrupted conversation repair: If the last assistant message has tool_calls but fewer tool results than expected (conversation was interrupted mid-execution), the incomplete turn is stripped so the LLM can re-generate cleanly
• The ChatSession adopts the resumed _ws_id, so new messages continue in the same workstream

Config persistence: LLM-affecting parameters (temperature, reasoning_effort, max_tokens, instructions, creative_mode) are persisted to the workstream_config table on creation and whenever changed via slash commands. resume() restores these values so resumed workstreams behave identically to the original.

/clear vs /new: /clear wipes in-memory context but preserves messages in the database for future resume. /new starts a fresh workstream (new _ws_id), leaving the old workstream resumable.

Resolution: resolve_workstream() accepts aliases, exact workstream IDs, or ID prefixes, enabling turnstone --resume refactor or /resume abc12.

Workstream listing: list_workstreams_with_history() only returns workstreams that have at least one saved message (WHERE EXISTS on conversations). Workstreams registered but never used (e.g., from process startup) are invisible until a message is sent.

Workstream pruning: prune_workstreams(retention_days, log_fn) runs once at startup (CLI and server). It removes:

• Workstreams with no messages (orphaned registrations)
• Unnamed workstreams (alias IS NULL) older than retention_days days (default 90)

Named (aliased) workstreams are never age-pruned. Configure with --retention-days N (0 = disable age pruning).

---

Error Handling and Retry

API Retry

ChatSession._create_stream_with_retry() (streaming path) and the agent _api_call() (non-streaming) both use the same retry pattern:

• Retries: 4 total attempts (1 initial + 3 retries, _MAX_RETRIES = 3)
• Backoff: exponential, base 1 second (delay = 1s * 2^attempt)
• Retryable errors: RateLimitError, APITimeoutError, APIConnectionError, InternalServerError, ServiceUnavailableError, APIError (matched by class name to avoid importing backend-specific exception hierarchies)
• On retry: ui.on_info() notification
• On final failure: exception propagates

_compact_messages() also wraps its non-streaming API call in the same retry loop.

Finish Reason Handling

_stream_response() tracks finish_reason from the final streaming chunk:

• "length": warns via ui.on_error() that the response was truncated. Any partial tool calls are discarded (their JSON would be malformed), causing the send() loop to exit cleanly.
• "content_filter": warns via ui.on_error() that the response was blocked.

Agent sub-sessions (_run_agent()) check finish_reason on each non-streaming response and stop the agent early on "length" or "content_filter".

_compact_messages() checks finish_reason on the compaction response and warns if the summary was truncated.

State Emission on Errors

• send() catches KeyboardInterrupt and generic Exception: calls _emit_state("error") before re-raising
• On interrupt: partial tool results and the originating assistant message are popped from self.messages to keep state consistent

Web UI Resilience

• SSE reconnect: both connectContentSSE() and connectGlobalSSE() use exponential backoff on onerror -- starting at 1 second, doubling on each failure, capped at 30 seconds. On successful message, delay resets to 1s.
• Disconnection indicator: #status-bar.disconnected class turns the status text red and shows "Reconnecting..."
• Fetch error handling: all fetch() calls use .catch() to prevent unhandled promise rejections
• Pending approval across tab switches: WebUI._pending_approval stores the approve_request event payload while the session is blocked waiting for user response. On tab switch / reconnect the pane reloads history via REST GET /history and then reconnects SSE; the live approval event is re-injected. The server-side project_history_messages projection marks the trailing orphan tool-call turn "pending": true so replayHistory skips the false ✓ approved badge; the live approval UI is rendered by the re-injected event instead.
• Browser history integration: history.pushState is called in switchTab() with {turnstone: 'workstream', wsId}. The initial state is seeded with history.replaceState({turnstone: 'dashboard'}) on load. The popstate listener restores the correct tab or shows the dashboard, guarded by _historyNavigation = true to prevent re-entrant pushState.
• Pane focus: mousedown and focusin events on pane containers update focusedPaneId. Approval shortcuts (y/n/a) apply to the focused pane. Ctrl+Alt+Arrow cycles focus between panes.

Eval Resilience

_run_single_test(): wraps session.send_headless() in a retry loop (3 attempts) to avoid transient API errors from poisoning evaluation scores.

Health Monitor & Circuit Breaker

BackendHealthMonitor (turnstone/core/healthcheck.py) runs a daemon thread that probes the LLM backend by calling client.models.list() every backend_probe_interval seconds (default 30). Probe results drive a three-state circuit breaker:

CLOSED  ──(N consecutive failures)──>  OPEN
OPEN    ──(cooldown expires)────────>  HALF_OPEN
HALF_OPEN ──(probe succeeds)────────>  CLOSED
HALF_OPEN ──(probe fails)──────────>  OPEN

• record_success() / record_failure() update _consecutive_failures and transition the _state (CircuitState enum: CLOSED, OPEN, HALF_OPEN).
• acquire_request_permit() returns False when the circuit is OPEN or when in HALF_OPEN and the single probe permit has already been consumed. Causes ChatSession._create_stream_with_retry to skip the backend and surface an error immediately.
• The /health endpoint reads the monitor's state: "status": "ok" when the circuit is closed, "status": "degraded" when open or half-open.

Rate Limiting

RateLimiter (turnstone/core/ratelimit.py) enforces per-client-IP request limits using a token-bucket algorithm. Each IP gets a TokenBucket with requests_per_second (refill rate) and burst (bucket capacity) from [ratelimit] config.

• Applied via RateLimitMiddleware after authentication but before route dispatch.
• /health and /metrics are exempt (monitoring must always be reachable).
• X-Forwarded-For support: when trusted_proxies is configured (comma-separated CIDRs), the middleware parses the X-Forwarded-For header using the rightmost-untrusted approach. IPv4-mapped IPv6 addresses are normalized. The direct client IP must be in the trusted set before XFF is considered.
• On limit exceeded: HTTP 429 with Retry-After header and JSON body {"error": "Rate limit exceeded", "retry_after": N}.
• The turnstone_ratelimit_rejected_total counter is incremented on each rejection.

---

User Identity and Authentication

Turnstone supports three authentication mechanisms, unified behind an AuthResult dataclass that carries user_id, scopes, and token_source:

1. API tokens — database-backed, prefixed ts_, stored as SHA-256 hashes in the api_tokens table. Can be exchanged for JWTs via POST /v1/api/auth/login.
2. JWTs — short-lived HMAC-SHA256 session tokens (default 24h) issued after successful credential validation. Contain sub (user_id), scopes, and src (origin) in claims.

Scope Model

Three hierarchical scopes control endpoint access:

Scope	Grants	Endpoints
read	SSE streams, workstream listing, history	GET endpoints
write	read + send, command, workstream create/close	POST to /api/workstreams/{ws_id}/send, /api/command, etc.
approve	write + tool approval, admin operations	POST to /api/workstreams/{ws_id}/approve, /api/admin/*

Middleware Flow

AuthMiddleware (ASGI) intercepts every request:

1. Public path check — /, /static/*, /shared/*, /health, /metrics, /openapi.json, /docs, /api/auth/*, and /api/auth/setup are always allowed.
2. Token extraction — Authorization: Bearer <token> header first, then turnstone_auth cookie as fallback.
3. Token type detection — dots in the token indicate JWT; ts_ prefix indicates API token.
4. Validation — JWT signature check or API token hash lookup in storage.
5. Scope check — required_scope(method, path) determines the minimum scope; the request is rejected with 403 if the token lacks it.
6. Context propagation — on success, ctx_user_id is set so structured logging includes the authenticated identity on every log event.

Architecture Split

• Console is the auth management hub — it hosts the admin endpoints for creating users, issuing API tokens, and managing channel mappings. User records and token hashes live in the shared storage backend. The console dashboard includes an admin panel (18 tabs) for managing credentials, governance, MCP servers, models, node metadata, and runtime settings through the browser.
• Server is a JWT validator only — it validates tokens on each request but never creates users or tokens. Both processes share the same jwt_secret (via TURNSTONE_JWT_SECRET env var or [auth].jwt_secret config).
• First-time setup — both server and console expose POST /v1/api/auth/setup, a public endpoint that creates the initial admin user when no users exist. This avoids the chicken-and-egg problem of needing approve scope to create the first user via /api/admin/users.

Auth Storage Tables

Three tables in storage/_schema.py support identity:

users
  user_id        TEXT PRIMARY KEY
  username       TEXT NOT NULL UNIQUE
  display_name   TEXT NOT NULL
  password_hash  TEXT NOT NULL       -- bcrypt
  created        TEXT NOT NULL

api_tokens
  token_id       TEXT PRIMARY KEY
  token_hash     TEXT NOT NULL UNIQUE  -- SHA-256 of raw token
  token_prefix   TEXT NOT NULL         -- first 8 chars for display
  user_id        TEXT NOT NULL
  name           TEXT NOT NULL         -- human-readable label
  scopes         TEXT NOT NULL         -- comma-separated
  created        TEXT NOT NULL
  expires        TEXT                  -- optional expiry timestamp

channel_users
  channel_type      TEXT NOT NULL      -- e.g. "slack", "discord"
  channel_user_id   TEXT NOT NULL      -- platform-specific user ID
  user_id           TEXT NOT NULL      -- FK to users
  PRIMARY KEY (channel_type, channel_user_id)

See docs/security.md for full security details including token lifecycle, password hashing, and deployment hardening.

---

Threading Model

CLI

Main thread          Spinner thread (daemon)       ThreadPoolExecutor
+--------------+     +------------------+          +-----------------+
| REPL loop    |     | Braille animation|          | Tool execution  |
| input() ->   |     | 80ms tick to     |          | max_workers=4   |
|   send() ->  |     | stderr           |          | parallel tools  |
|   stream  -> |     | started/stopped  |          | run concurrently|
|   tools   -> |     | by TerminalUI    |          |                 |
+--------------+     +------------------+          +-----------------+
       |                    ^                              ^
       +-- on_thinking_start/stop -------------------------+
       +-- _execute_tools ---------------------------------+

Key constraint: input() blocks the main thread. The spinner writes to stderr so it does not interfere with readline. Tool execution may use a ThreadPoolExecutor with up to 4 workers for parallel tool calls.

Server

Starlette ASGI app (served by uvicorn)
  |
  +-- Async request handlers (all under /v1/ prefix)
  |     POST /v1/api/workstreams/{ws_id}/send    -> starts worker thread per workstream
  |     POST /v1/api/workstreams/{ws_id}/approve -> unblocks WebUI._approval_event
  |     POST /v1/api/workstreams/new             -> creates workstream + worker
  |     GET  /v1/api/workstreams/{ws_id}/events  -> SSE via EventSourceResponse (per workstream)
  |     GET  /v1/api/events/global               -> SSE via EventSourceResponse (fan-out)
  |
  +-- ASGI middleware stack
  |     MetricsMiddleware -> CORSMiddleware -> AuthMiddleware -> RateLimitMiddleware
  |
  +-- Worker thread per workstream (daemon)
  |     Runs session.send() synchronously -- ChatSession is fully blocking
  |     Blocks on WebUI._approval_event (threading.Event)
  |
  +-- Background daemon threads
        Global SSE fan-out: reads global_queue, copies to per-client queues
        Idle cleanup: closes stale workstreams, cleans rate limiter buckets

Starlette handles all HTTP routing, CORS, and middleware. uvicorn runs the ASGI application with async request handling. All API endpoints live under the /v1/ prefix via a Starlette Mount. An OpenAPI 3.1 spec is generated from Pydantic v2 models and served at /openapi.json; Swagger UI is available at /docs. SSE endpoints use EventSourceResponse from sse-starlette with async generators that bridge sync queue.Queue via asyncio.get_running_loop().run_in_executor().

ChatSession.send() remains synchronous, running in daemon worker threads. WebUI keeps threading.Event and queue.Queue primitives (unchanged from the sync era). The _global_fanout_thread and _idle_cleanup_thread remain as daemon threads since they interact with sync primitives. A lifespan context manager handles startup/shutdown (health monitor, MCP client, registry).

Each workstream's WebUI has:

• _listeners (per-client SSE queues, fan-out on _enqueue())
• _approval_event (threading.Event for blocking)
• _global_queue (class variable, shared, for state broadcasts)

The SSE handlers bridge these sync queues to async via run_in_executor(), polling queue.Queue.get(timeout=1) while sse-starlette handles keepalive pings automatically.

Workstream Threading (CLI)

Main thread                  Background workstream thread
+------------------+         +---------------------------+
| REPL input()     |         | session.send()            |
| /ws commands     |         | streams response          |
| active workstream|         | executes tools            |
| send() inline   |         | approve_tools() ->        |
+------------------+         |   _fg_event.wait() BLOCKS |
       |                     +---------------------------+
       |                                ^
       +-- /ws <N> switch ------------->|
       |   old.set_foreground(False)    |
       |   new.set_foreground(True)     |
       |   new.flush_buffer()           |
       +-- _fg_event.set() unblocks --->+

When a background workstream needs approval, its WorkstreamTerminalUI calls _fg_event.wait(), which blocks the worker thread until the user switches to that workstream. The _bg_attention_notify callback writes a bell + status line to stderr to alert the user.

Cluster Console

Monitoring (2 daemon threads)        Control + Proxy (async Starlette)
+------------------+                 +----------------------------+
| Node discovery   |                 | POST /v1/api/cluster/      |
| Service registry |                 |   workstreams/new          |
| every 60 seconds |                 |   → POST to target server  |
+------------------+                 +----------------------------+
| SSE manager      |                 | GET /node/{node_id}/       |
| asyncio loop     |                 |   → httpx.AsyncClient      |
| 1 task per node  |                 |     proxy to server_url    |
| /events/global   |                 | GET /node/{id}/v1/api/workstreams/{ws_id}/events |
| snapshot+deltas  |                 |   → SSE stream proxy                              |
+------------------+                 | POST /node/{id}/v1/api/workstreams/{ws_id}/send   |
                                     |   → forwarded to server                           |
                                     +----------------------------+

The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE endpoint uses EventSourceResponse with the same listener queue pattern as the main server. ClusterCollector runs two daemon threads: a discovery loop that queries the service registry every 60 seconds, and an SSE manager that runs a single asyncio event loop multiplexing persistent SSE connections to all nodes via GET /v1/api/events/global. Each node delivers a full snapshot on connect followed by real-time delta events — state changes, health transitions, and aggregate metrics arrive sub-second instead of on a 15-second poll cycle.

The console has two write-path capabilities:

1. Workstream creation — sends HTTP requests to target server nodes to create workstreams. Auto-selects the node with the most available capacity if no target is specified. When a skill field is present, the server resolves the skill BEFORE mgr.create() (applying the model override to the creation request) and snapshot-applies remaining settings (auto-approve, token budget, temperature, etc.) to the workstream config AFTER creation.

2. Reverse proxy — serves each node's server UI through the console port at /node/{node_id}/. Uses httpx.AsyncClient to proxy HTTP and SSE traffic. A JS shim is injected into the server's app.js to override fetch() and EventSource(), routing root-relative URLs through the proxy prefix. This eliminates the need for direct network access to individual server nodes.

The console also performs version drift detection — flagging when nodes report different versions via the /health endpoint. The overview API includes version_drift and versions fields; the dashboard shows a yellow warning indicator when versions diverge.

Clicking a workstream row in the console opens the proxied server UI at /node/{node_id}/?ws_id=<id> — the server's JS parses this on load and auto-selects the workstream. See docs/console.md for the full API reference.

---

Conversation Compaction

When the prompt exceeds auto_compact_pct of the context window (default: 80%, configurable via --auto-compact-pct), ChatSession auto-compacts by summarizing the entire conversation into a structured summary (_compact_messages). The summary model call uses compact_max_tokens (default: 32768, configurable via --compact-max-tokens). The summary preserves:

• Decisions made (architecture, libraries, approaches)
• Files read, created, or modified
• Exact identifiers, paths, and code snippets
• Important tool results
• Open tasks
• User preferences

After compaction, _read_files is cleared to force re-reads before edits, since file contents are no longer in the message history.

---

Client SDK

See also: SDK Architecture diagram | SDK Documentation

The turnstone/sdk/ package provides typed HTTP clients for programmatic access to both the server and console APIs. It wraps REST endpoints with methods that return Pydantic models, and SSE endpoints with async/sync iterators that yield typed event dataclasses.

Two client pairs (sync + async):

• TurnstoneServer / AsyncTurnstoneServer — server API (workstreams, chat, streaming)
• TurnstoneConsole / AsyncTurnstoneConsole — console API (cluster overview, nodes, workstreams)

Design: async-first with thin sync wrappers. _BaseClient provides httpx setup, auth headers, _request() (REST) and _stream_sse() (SSE). Sync clients delegate through _SyncRunner which maintains a persistent background event loop on a daemon thread.

Event types: 38 standalone dataclasses in events.py with a type-registry dispatch (from_json() on each event). Events are decoupled from server internals — the SDK parses SSE frames directly from the /v1/api/events streams.

TypeScript SDK: sdk/typescript/ — separate npm package with the same API surface. Zero browser dependencies, SSE via fetch + ReadableStream parsing.

# Python quick start
from turnstone.sdk import TurnstoneServer

with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
    ws = client.create_workstream(name="demo")
    result = client.send_and_wait("Hello!", ws.ws_id)
    print(result.content)

---

Channel Integrations

See also: Channel Integrations guide

The turnstone-channel gateway connects external messaging platforms (Discord and Slack today, with an adapter protocol for future platforms) to the turnstone cluster via HTTP. Each platform adapter implements the ChannelAdapter protocol and translates between platform-native events and turnstone server API calls.

The ChannelRouter manages bidirectional routing: it maps platform channel/thread IDs to turnstone workstream IDs, handles workstream creation and stale-route recovery, and resolves platform users to turnstone identities via the channel_users table. When an evicted workstream is reactivated, the router uses atomic resume via the resume_ws field on the workstream creation request — the server resumes the old workstream's conversation during creation in a single HTTP request, eliminating ordering fragility.

Discord and Slack adapters ship today. See channels.md for setup instructions, configuration reference, and the adapter development guide.

Notification Subsystem

The notify tool enables the LLM to send notifications to users or channels directly. The server calls the channel gateway directly over HTTP for lower latency: _exec_notify() queries the services database table for healthy channel gateways (heartbeat within 120 seconds), authenticates with a service JWT (aud: turnstone-channel), and POSTs to POST /v1/api/notify on the first healthy gateway. The payload includes the originating ws_id for reply routing. The gateway validates the JWT, resolves the target (username lookup via channel_users or direct channel_type+channel_id), and delegates to ChannelAdapter.send_notification() which sends the message and tracks the outgoing message ID → (ws_id, target_user_id) mapping. Delivery retries up to 3 times with backoff, re-querying the service registry on each attempt. See Notification Flow diagram.

Bidirectional replies: When a user replies to a notification DM, the channel adapter (Discord or Slack) looks up the originating ws_id from the tracked message ID, verifies the replying user matches the notification recipient, and routes the reply to the workstream via router.send_message(). The workstream's response is forwarded back to the DM via a temporary entry in _notify_reply_channels. On TurnCompleteEvent, the response message is itself tracked for further replies, enabling multi-turn DM conversations without requiring the user to open the web UI. Tracking entries are capped at 100 (FIFO eviction) and cleaned up on workstream close.

---

Governance

See also: Governance documentation | Governance Architecture diagram

Turnstone governance extends the Phase 1 auth system with role-based access control (RBAC), tool execution policies, skills, usage tracking, and audit logging. The permission model has two layers: legacy scopes (read, write, approve) checked by AuthMiddleware, and 15 granular permissions checked per-endpoint by require_permission(). Three built-in roles (admin, operator, viewer) are seeded by migration 008; custom roles can be created with any permission subset. JWTs carry both scopes and permissions claims for backward compatibility.

Tool policies use glob pattern matching (fnmatch) with priority-ordered first-match-wins evaluation to control tool execution (allow/deny/ask). Skills provide reusable system messages with {{variable}} substitution plus session configuration (model, temperature, auto-approve, token budget, etc.). Usage events are recorded per-LLM-request for token accounting. An append-only audit log captures all admin mutations.

Skills are snapshot-applied once at workstream creation — not a live binding. The prompt_templates table (which stores skills) supports auto-versioning, and workstreams record which skill and version spawned them. Token budget enforcement tracks consumption in session.send() with 80% warning and 100% approval gate via the __budget_override__ synthetic tool name.

The console admin panel exposes these capabilities as 18 permission-gated tabs: Users, API Tokens, Channels, Schedules, Watches, Roles, Policies, Prompts, Judge, Skills, MCP Servers, Usage, Audit, Memories, Models, Nodes, Settings, and TLS. Both Python and TypeScript SDKs expose governance methods on the console client.

Intent Validation

See also: Intent Validation guide | Judge Architecture diagram

Intent validation provides advisory risk assessments for tool calls that require human approval. The system runs a two-tier evaluation pipeline implemented in turnstone/core/judge.py:

1. Heuristic tier (synchronous, sub-millisecond) -- A priority-ordered rule table using fnmatch tool patterns and regex argument patterns. Four severity levels: critical (deny), high (review), medium (review), low (approve). First match wins. The heuristic verdict is attached to the approve_request SSE event immediately.

2. LLM judge tier (asynchronous, daemon thread) -- A multi-turn evaluation where the judge LLM receives conversation context and tool call details, optionally uses read_file/list_directory to gather evidence (with security-hardened path blocking), and produces a structured JSON verdict. If the LLM verdict has higher confidence than the heuristic, it replaces it via an intent_verdict SSE event.

The judge is session-scoped (IntentJudge), lazy-initialized on first approval, and configured via the [judge] config section or --judge CLI flags. By default it uses self-consistency (same model), but supports cross-model and cross-provider configurations. Task sub-agents are exempt. All verdicts are persisted to the intent_verdicts table (migration 012) with the user's final decision, enabling future calibration. The console exposes GET /v1/api/admin/verdicts for audit queries (requires admin.judge permission).