turnstone/docs/architecture.md

Turnstone Architecture

Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or Anthropic's native Messages API via pluggable provider adapters, and gives the model 14 built-in tools plus external tools via MCP (Model Context Protocol) for reading, writing, searching, planning, and executing code.

The core design principle is a UI-agnostic engine with pluggable frontends. The engine (ChatSession) drives the conversation loop -- streaming, tool dispatch, retry, compaction -- while every user-facing interaction is delegated through the SessionUI protocol. Any frontend implements that protocol and plugs in.

Entry Points

Command	Module	Frontend	Purpose
turnstone	turnstone.cli	TerminalUI	Interactive terminal REPL
turnstone-server	turnstone.server	WebUI	Browser-based chat (HTTP + SSE)
turnstone-bridge	turnstone.mq.bridge	Bridge	Message queue ↔ HTTP API bridge
turnstone-console	turnstone.console.server	ClusterCollector	Cluster dashboard (aggregates all nodes)
turnstone-eval	turnstone.eval	NullUI	Headless evaluation and prompt optimization

---

Module Map

turnstone/
  cli.py              Terminal frontend (TerminalUI, WorkstreamTerminalUI, REPL)
  server.py           Web frontend (WebUI, HTTP handler, static-file serving)
  eval.py             Evaluation harness (HeadlessSession, scoring, prompt optimization)
  core/
    session.py        ChatSession engine, SessionUI protocol, tool dispatch
    providers/        LLM provider adapters (pluggable backend layer)
      _protocol.py    LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
      _openai.py      OpenAIProvider — OpenAI, vLLM, llama.cpp, any compatible API
      _anthropic.py   AnthropicProvider — Anthropic Messages API, native streaming, thinking
      __init__.py     create_provider() + create_client() factory functions
    workstream.py     Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
    tools.py          Tool schema loader (JSON -> OpenAI function-calling format)
    mcp_client.py     MCPClientManager — MCP server connections, tool discovery, async-sync bridge
    model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
    memory.py         Persistence facade (delegates to storage backend)
    storage/          Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
    metrics.py        Prometheus-compatible metrics collector (MetricsCollector)
    healthcheck.py    BackendHealthMonitor — periodic probe + circuit breaker
    ratelimit.py      Per-IP token-bucket rate limiter (RateLimiter, TokenBucket)
    edit.py           File edit utilities (find_occurrences, pick_nearest)
    safety.py         Command safety validation (blocked patterns, sanitization)
    sandbox.py        Math code sandboxing (AST validation, subprocess execution)
    web.py            Web utilities (HTML stripping, SSRF prevention)
  api/
    schemas.py        Shared Pydantic v2 models (auth, errors, WorkstreamState)
    server_schemas.py Server endpoint request/response models
    console_schemas.py Console endpoint request/response models
    openapi.py        OpenAPI 3.1 spec builder
    server_spec.py    Server endpoint catalog → build_server_spec()
    console_spec.py   Console endpoint catalog → build_console_spec()
    docs.py           /openapi.json + /docs (Swagger UI) handler factories
  sdk/
    server.py         AsyncTurnstoneServer + TurnstoneServer (HTTP client)
    console.py        AsyncTurnstoneConsole + TurnstoneConsole (HTTP client)
    events.py         27 SSE event dataclasses with type registry
    _base.py          Shared httpx async client, auth, error handling
    _sync.py          Background event loop for sync wrappers
    _types.py         TurnResult + TurnstoneAPIError
  mq/
    protocol.py       Inbound/outbound message dataclasses (JSON serialization)
    broker.py         Abstract MessageBroker protocol + RedisBroker
    bridge.py         Bridge service (queue ↔ turnstone-server HTTP API)
    client.py         TurnstoneClient library + TurnResult for MQ-based access
  console/
    collector.py      ClusterCollector — aggregates state from all nodes via Redis + HTTP
    server.py         Cluster dashboard HTTP server + SSE + CLI entry point
    static/           Cluster dashboard web UI (page-specific HTML, CSS, JS)
  shared_static/      Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
  ui/
    colors.py         ANSI color constants with NO_COLOR support
    markdown.py       Streaming terminal markdown renderer (line-buffered)
    spinner.py        Braille character spinner (daemon thread)
    static/
      index.html      Single-page app shell (links to CSS and JS)
      style.css       Page-specific UI styles (dashboard layout, approval blocks)
      app.js          Page-specific client-side JavaScript (SSE, workstreams, markdown)
  tools/
    *.json            14 tool schemas (OpenAI function-calling format + turnstone metadata)

Both UIs share a common design system extracted into turnstone/shared_static/: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports base.css and the shared JS modules at /shared/, then adds only page-specific code at /static/.

---

Core Loop

See also: Conversation Turn diagram

A user message flows through the system as follows:

 User input
     |
     v
 ChatSession.send(user_input)
     |
     v
 _full_messages()  ------------>  system_messages + self.messages
     |
     v
 _emit_state("thinking")
     |
     v
 _create_stream_with_retry()  ---->  provider.create_streaming(client, model, messages, ...)
     |                                  up to 3 retries (4 total attempts), exponential backoff
     v
 _stream_response(stream)  -------->  dispatch tokens to UI:
     |                                  on_reasoning_token() / on_content_token()
     |                                  accumulate tool_calls from deltas
     |                                  track finish_reason
     v
 finish_reason check:
     +--- "length"  --> warn, discard partial tool_calls
     +--- "content_filter" --> warn
     v
 tool_calls present?
     |
     +--- No ---> _print_status_line() -> _emit_state("idle") -> return
     |
     +--- Yes --> _emit_state("running")
                    |
                    v
                  _execute_tools(tool_calls)  <--- three-phase pipeline (see below)
                    |
                    v
                  append tool results to self.messages
                    |
                    v
                  loop back to _full_messages()

Tool Execution Pipeline

See also: Tool Pipeline diagram

Tool execution is a three-phase process:

Phase 1: PREPARE (serial)
  For each tool_call:
    _prepare_tool(tc)
      -> parse JSON arguments (with regex fallback for malformed JSON)
      -> dispatch to _prepare_{tool_name}(call_id, args)
      -> validate inputs, build preview text
      -> return item dict with: header, preview, needs_approval, execute fn

Phase 2: APPROVE (serial, blocking)
  _emit_state("attention")
  ui.approve_tools(items)
    -> display all headers and previews
    -> if any need approval and not auto_approve: prompt user
    -> return (approved, feedback)
  _emit_state("running")

Phase 3: EXECUTE (parallel)
  if len(items) == 1:
    run_one(items[0])
  else:
    ThreadPoolExecutor(max_workers=4).map(run_one, items)
  Bash tool streams stdout line-by-line via ui.on_tool_output_chunk(call_id, line)
  Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
  call_id links tool_info items → streaming chunks → final result
  For plan tool: post-execution gate via ui.on_plan_review()

State Transitions

The engine emits state changes via _emit_state() which calls ui.on_state_change(state). Frontends use these to update indicators (spinner, tab badges, status line).

  send() called
      |
      v
  "thinking"  --->  streaming response
      |
      v
  "running"   --->  tool execution
      |
      v
  "attention"  --->  waiting for user approval / plan review
      |
      v
  "running"   --->  executing approved tools
      |
      v
  "idle"       --->  no more tool calls, turn complete
      |
  (or "error"  --->  exception or KeyboardInterrupt)

---

SessionUI Protocol

See also: Core Engine Classes diagram

Defined in turnstone.core.session.SessionUI as a typing.Protocol with 14 methods. Every frontend must implement all of them.

class SessionUI(Protocol):
    def on_thinking_start(self) -> None: ...
    def on_thinking_stop(self) -> None: ...
    def on_reasoning_token(self, text: str) -> None: ...
    def on_content_token(self, text: str) -> None: ...
    def on_stream_end(self) -> None: ...
    def approve_tools(self, items: list[dict]) -> tuple[bool, str | None]: ...
    def on_tool_result(self, call_id: str, name: str, output: str) -> None: ...
    def on_tool_output_chunk(self, call_id: str, chunk: str) -> None: ...
    def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
    def on_plan_review(self, content: str) -> str: ...
    def on_info(self, message: str) -> None: ...
    def on_error(self, message: str) -> None: ...
    def on_state_change(self, state: str) -> None: ...
    def on_rename(self, name: str) -> None: ...  # propagate alias to tab/UI label

on_rename is called by the /name command (on success) and after a successful /resume (if the resumed session has an alias or title). WebUI.on_rename broadcasts a ws_rename event on the global SSE channel and updates the in-memory Workstream.name; TerminalUI.on_rename is a no-op.

Three Implementations

Class	Module	Notes
TerminalUI	turnstone.cli	ANSI colors, MarkdownRenderer, Spinner, readline-based input() for approval
WebUI	turnstone.server	SSE event queue per workstream, threading.Event for blocking on approval/plan
NullUI	turnstone.eval	Discards all output; approve_tools always returns (True, None)

WorkstreamTerminalUI

WorkstreamTerminalUI (in turnstone.cli) extends TerminalUI with workstream awareness:

• Output buffering: When in background (is_foreground is False), tokens are appended to _output_buffer instead of written to stdout. When the user switches to this workstream, flush_buffer() replays them.

• Approval blocking: approve_tools() and on_plan_review() call _fg_event.wait() when in background, blocking the worker thread until the workstream is foregrounded. This ensures the user sees the approval prompt in the correct context.

• Foreground/background toggle: set_foreground(bool) sets or clears _fg_event (a threading.Event). The manager calls this during /ws <N> switches.

---

Workstream Architecture

Workstreams are parallel, independent chat sessions. Each has its own ChatSession, SessionUI, message history, and worker thread.

WorkstreamState

See also: Workstream States diagram

Defined in turnstone.core.workstream.WorkstreamState (5 states):

IDLE       waiting for user input
THINKING   LLM is streaming a response
RUNNING    tools are executing
ATTENTION  blocked on user approval or plan review
ERROR      last operation failed

Data Model

@dataclass
class Workstream:
    id: str                              # uuid hex, 8 chars
    name: str                            # user-visible label
    state: WorkstreamState               # current state
    session: ChatSession | None          # the conversation engine
    ui: SessionUI | None                 # frontend adapter
    worker_thread: threading.Thread | None
    error_message: str
    last_active: float                   # time.monotonic() timestamp, updated on every state change
    _lock: threading.Lock                # per-workstream state lock

WorkstreamManager

class WorkstreamManager:
    MAX_WORKSTREAMS = 10

    def __init__(self, session_factory: Callable[[SessionUI], ChatSession]): ...
    def create(self, name="", ui_factory=None) -> Workstream: ...
    def close(self, ws_id: str) -> bool: ...
    def close_idle(self, max_age_seconds: float) -> list[str]: ...  # auto-close stale IDLE workstreams
    def get(self, ws_id: str) -> Workstream | None: ...
    def get_active(self) -> Workstream | None: ...
    def list_all(self) -> list[Workstream]: ...
    def switch(self, ws_id: str) -> Workstream | None: ...
    def switch_by_index(self, index: int) -> Workstream | None: ...
    def set_state(self, ws_id, state, error_msg=""): ...  # updates last_active

The session_factory pattern decouples session creation from configuration. The factory captures shared config (client, model, temperature, etc.) and accepts only a SessionUI, so the manager can create sessions without knowing API details.

Idle Workstream Lifecycle

The web server runs a background _idle_cleanup_thread (daemon) that calls WorkstreamManager.close_idle() periodically (every timeout / 4, max 5 min). Any IDLE workstream whose last_active is older than the configured timeout is closed; non-IDLE workstreams (THINKING, RUNNING, ATTENTION, ERROR) are never touched. The last workstream is always preserved even if expired. On close, a ws_closed event is broadcast on the global SSE channel so browser clients remove the tab immediately. Controlled by --workstream-idle-timeout (default: 120 minutes, 0 = disable).

Workstream eviction at capacity: When WorkstreamManager.create() would exceed max_workstreams (configurable via [server].max_workstreams, default 10), the oldest IDLE workstream is automatically evicted to make room. The turnstone_workstreams_evicted_total counter is incremented on each eviction. If no IDLE workstream is available the create request fails as before.

CLI Workstreams

• /ws list -- show all workstreams with state indicators
• /ws new [name] -- create a new workstream and switch to it
• /ws <N> -- switch to workstream by 1-based index
• /ws close [N] -- close a workstream
• /ws rename <name> -- rename the active workstream

Background notifications: when a background workstream enters ATTENTION state, _bg_attention_notify writes an ANSI escape sequence to stderr (overwrites the line above the prompt) with the workstream name.

Status line: _print_ws_status_line() shows a compact status of all non-idle background workstreams above the input prompt.

Web Workstreams

• Tab bar: Each workstream renders as a tab with a colored state indicator (CSS @keyframes pulse animation per state).
• Per-tab SSE: connectContentSSE(wsId) opens /v1/api/events?ws_id=<id> for the active tab's event stream.
• Global SSE: connectGlobalSSE() opens /v1/api/events/global which receives ws_state broadcasts from all workstreams, used to update tab indicators without switching.
• New tab / close: POST /v1/api/workstreams/new, POST /v1/api/workstreams/close.

Thread Safety

• WorkstreamManager._lock: guards _workstreams dict and _order list on all create/close/switch/list operations.
• Workstream._lock: guards per-workstream state mutations in set_state().
• WorkstreamTerminalUI._print_lock: guards _output_buffer access.
• WorkstreamTerminalUI._fg_event: threading.Event that blocks background approval until the workstream is foregrounded.

---

Tool System

Schema Format

Each tool is a JSON file in turnstone/tools/. The file contains an OpenAI function-calling schema (name, description, parameters) plus optional turnstone metadata keys:

Metadata Key	Type	Meaning
agent	bool	Include this tool when running as a plan/task sub-agent
task_agent	bool	Include this tool when running as a task sub-agent
auto_approve	bool	Tool is read-only; skip user approval
primary_key	str	Fallback argument name for bare-string JSON recovery

Example (read_file.json):

{
  "name": "read_file",
  "description": "Read the contents of a file. ...",
  "parameters": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "..." },
      "offset": { "type": "integer", "description": "..." },
      "limit": { "type": "integer", "description": "..." }
    },
    "required": ["path"]
  },
  "agent": true,
  "task_agent": true,
  "auto_approve": true,
  "primary_key": "path"
}

At import time, turnstone.core.tools._load_tools() strips the metadata keys from each schema and builds:

• TOOLS -- list of {"type": "function", "function": {...}} dicts for the API
• AGENT_TOOLS -- subset with agent: true
• TASK_AGENT_TOOLS -- subset with task_agent: true
• AGENT_AUTO_TOOLS / TASK_AUTO_TOOLS -- sets of tool names with auto_approve: true
• PRIMARY_KEY_MAP -- {name: primary_key} for JSON fallback recovery
• merge_mcp_tools(builtin, mcp_tools) -- merges built-in + MCP tools at session init

14 Tools by Category

Read-only (auto-approve):

• read_file -- read file contents with optional offset/limit
• search -- ripgrep-based codebase search
• man -- read man pages
• recall -- retrieve stored memories

Write (requires approval):

• bash -- execute shell commands (with safety checks via turnstone.core.safety)
• write_file -- create or overwrite a file
• edit_file -- string replacement in an existing file (requires prior read_file)
• math -- execute Python in sandboxed subprocess (via turnstone.core.sandbox)
• web_fetch -- fetch a URL (with SSRF protection via turnstone.core.web)
• web_search -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models)

Agent (delegated sub-sessions):

• task -- delegate to a sub-agent with full tool access (TASK_AGENT_TOOLS)
• plan -- explore codebase and write a structured plan (AGENT_TOOLS)

Memory (persistent key-value store):

• remember -- save a fact
• forget -- delete a fact

Prepare / Execute Pattern

Every tool has a _prepare_{name} method and a corresponding _exec_{name} method on ChatSession:

_prepare_bash(call_id, args)   -> item dict with execute=self._exec_bash
_prepare_read_file(call_id, args) -> item dict with execute=self._exec_read_file
...

The prepare method validates inputs and builds the preview. The item dict carries the validated data and a reference to the execute function. This separation allows the UI to show previews before any side effects occur.

Agent Tools

task and plan invoke _run_agent(), which runs a multi-turn loop with a subset of tools and its own system prompt. The sub-agent runs independently, then returns the final content as the tool result.

• task: uses self._task_tools (TASK_AGENT_TOOLS + MCP tools)
• plan: uses self._agent_tools (AGENT_TOOLS + MCP tools). Writes output to .plan-<session_id>.md — unique per ChatSession so concurrent workstreams don't collide. On repeat invocations the prior plan tool call and its result are forwarded from self.messages so the agent refines the existing plan rather than starting over. Planning instructions are injected as a developer message prepended to the agent's conversation.
• Turn limit: controlled by agent_max_turns (default: -1, unlimited). When a limit is set and reached, the agent is forced to synthesize a final response without tools. When unlimited, the loop only exits when the model stops calling tools or hits finish_reason: "length".
• Retry: each API call in the agent loop uses the same retry+backoff logic as the main _create_stream_with_retry().
• Finish reason handling: finish_reason: "length" stops the agent early and returns whatever content was generated. finish_reason: "content_filter" returns a placeholder.

MCP Tool Integration

MCPClientManager (turnstone/core/mcp_client.py) connects to external MCP servers and exposes their tools alongside built-in tools. The MCP SDK is fully async; turnstone bridges this with a background asyncio event loop in a daemon thread.

Lifecycle:

1. create_mcp_client() reads server configs from TOML or JSON
2. MCPClientManager.start() launches the background event loop thread
3. _connect_all() connects to each server (stdio subprocess or HTTP), runs initialize() + list_tools(), converts schemas to OpenAI format
4. ChatSession.__init__ receives the manager and builds self._tools (built-in + MCP)
5. _prepare_tool() routes MCP tools to _prepare_mcp_tool() / _exec_mcp_tool()
6. _exec_mcp_tool() calls call_tool_sync() which dispatches to the async loop via asyncio.run_coroutine_threadsafe()

Tool naming: mcp__{server}__{tool} — double underscore delimiter, validated at connection time (server names with __ are rejected).

Error isolation: Per-server connection failures are caught and logged; other servers still connect. Tool execution errors return error strings to the LLM rather than crashing the session.

Provider Adapter Layer

See also: Core Engine Classes diagram

ChatSession is provider-agnostic — it delegates all LLM communication to an LLMProvider protocol (turnstone/core/providers/_protocol.py). Internally, messages use an OpenAI-like format; each provider translates at the API boundary.

ChatSession
    |
    v
LLMProvider (protocol)
    |
    +--- OpenAIProvider  --- OpenAI, vLLM, llama.cpp, any /v1/chat/completions API
    +--- AnthropicProvider --- Anthropic Messages API (native streaming, thinking)

Protocol methods:

Method	Purpose
create_streaming()	Streaming request, yields normalized StreamChunk objects
create_completion()	Non-streaming request, returns CompletionResult
get_capabilities()	Per-model flags (ModelCapabilities)
convert_tools()	Translate OpenAI tool schemas to provider format
retryable_error_names	Exception class names that trigger retry

Normalized data types:

Type	Fields
StreamChunk	content_delta, reasoning_delta, tool_call_deltas, info_delta, usage, finish_reason
CompletionResult	content, tool_calls, finish_reason, usage
ModelCapabilities	context_window, max_output_tokens, supports_temperature, token_param, thinking_mode, supports_effort, supports_web_search
UsageInfo	prompt_tokens, completion_tokens, total_tokens

OpenAIProvider (_openai.py): passes messages through unchanged (they are already in OpenAI format). Model capability lookup table covers GPT-5/5.1/5.2, O-series, and search models (gpt-5-search-api). For search models, injects web_search_options and removes the web_search function tool (the model always searches). Citations from url_citation annotations are formatted as footnotes. Unknown models (local servers) get permissive defaults and use Tavily for web search.

AnthropicProvider (_anthropic.py): converts OpenAI-format messages to Anthropic content blocks, maps system/developer roles to the system parameter, groups consecutive tool result messages into user-role content blocks, and translates tool schemas from OpenAI function-calling format to Anthropic's input_schema format. Supports both manual and adaptive thinking modes, with effort parameter support for models like Claude Opus 4.6 and Sonnet 4.6. Replaces the web_search function tool with Anthropic's native web_search_20250305 server-side tool — Claude decides when to search, the API executes it, and results stream back as server_tool_use / web_search_tool_result content blocks (emitted as info_delta for UI display). The anthropic SDK is imported lazily so it remains an optional dependency (pip install turnstone[anthropic]).

Factory functions (__init__.py): create_provider(name) returns a singleton provider instance (thread-safe). create_client(name, base_url, api_key) creates the appropriate SDK client.

Multi-Model Registry

ModelRegistry (turnstone/core/model_registry.py) manages named model configurations so workstreams can use different LLM backends.

Config format:

[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
# provider defaults to "openai"

[models.claude]
provider = "anthropic"
api_key = "sk-ant-..."
model = "claude-opus-4-6"
context_window = 200000

[models.openai]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "gpt-5"
context_window = 400000

[model]
default = "local"
fallback = ["claude", "openai"]
agent_model = "claude"

Each [models.*] entry produces a ModelConfig with a provider field (default: "openai"). Supported values: "openai" and "anthropic".

Lifecycle:

1. load_model_registry() reads [models.*] sections from config.toml and builds a "default" entry from CLI --base-url/--model/--api-key args
2. The registry is passed to the session factory closure in both cli.py and server.py; each workstream resolves its model on creation
3. ModelRegistry.get_client() lazily creates SDK client instances via create_client() — OpenAI for the openai provider, Anthropic for the anthropic provider (thread-safe via _client_lock)
4. ModelRegistry.get_provider() lazily creates LLMProvider instances via create_provider() (also cached and thread-safe)
5. /model command shows available models; /model <alias> switches the active workstream's client, model, and context window
6. _create_stream_with_retry() tries the primary model, then each fallback alias in order if the primary is unreachable
7. _run_agent() resolves registry.agent_model (if set) for plan/task sub-agents, allowing a cheaper model for autonomous loops

Per-workstream selection: POST /v1/api/workstreams/new accepts an optional "model" field. The bridge CreateWorkstreamMessage carries the same field through the MQ protocol.

Tool Output Truncation

Tool execution results (bash, read_file, search, math, man) are truncated by _truncate_output() when they exceed tool_truncation characters. Truncation preserves the first half and last half of the output, with a message in between:

... [N chars truncated — output exceeded LIMIT char limit] ...

The default limit is 50% of the context window in characters (computed as context_window * chars_per_token * 0.5). For a 131K context window this is ~262K characters. Override with --tool-truncation <chars>.

This truncation message is visible to the model, so it knows output was cut.

---

Persistence

Storage Architecture

Persistence is managed by the turnstone.core.storage package — a pluggable backend behind a StorageBackend protocol. The memory.py facade provides backward-compatible module-level functions that delegate to the active backend.

session.py / server.py / cli.py
        ↓
    memory.py  (facade — silent-failure wrappers)
        ↓
    storage._registry  (singleton factory)
        ↓
  ┌─────────────┐    ┌──────────────────┐
  │ SQLiteBackend │    │ PostgreSQLBackend │
  │ (FTS5 search) │    │ (tsvector/ILIKE)  │
  └─────────────┘    └──────────────────┘
        ↓                     ↓
    storage._schema  (SQLAlchemy Core tables — single source of truth)
        ↓
    storage._migrate  (programmatic Alembic)

SQLite is the default (zero-config, single file at .turnstone.db). PostgreSQL is the production backend (connection pooling, tsvector full-text search). Select via [database] in config.toml, CLI flags, or environment variables (TURNSTONE_DB_BACKEND, TURNSTONE_DB_URL).

Schema migrations are managed by Alembic and run automatically on startup. Existing SQLite databases created before the migration system are auto-stamped at the baseline revision.

Tables

memories
  key      TEXT PRIMARY KEY
  value    TEXT NOT NULL
  created  TEXT NOT NULL
  updated  TEXT NOT NULL

sessions
  session_id  TEXT PRIMARY KEY
  alias       TEXT UNIQUE            -- user-assigned short name (nullable)
  title       TEXT                   -- LLM-generated title (nullable)
  created     TEXT NOT NULL
  updated     TEXT NOT NULL          -- bumped on every save_message()

conversations
  id            INTEGER PRIMARY KEY AUTOINCREMENT
  session_id    TEXT NOT NULL
  timestamp     TEXT NOT NULL
  role          TEXT NOT NULL        -- user | assistant | tool_call | tool_result
  content       TEXT
  tool_name     TEXT
  tool_args     TEXT
  tool_call_id  TEXT                 -- links tool_call ↔ tool_result for resume
  provider_data TEXT                 -- raw provider content (e.g. Anthropic encrypted)

session_config
  session_id  TEXT NOT NULL          -- composite PK with key
  key         TEXT NOT NULL
  value       TEXT

conversations_fts                    -- SQLite FTS5 virtual table (optional)
  content     (content=conversations, content_rowid=id)

Table definitions live in storage/_schema.py (SQLAlchemy Core Table objects) and are the single source of truth for both backends and Alembic migrations.

StorageBackend Protocol

Method	Purpose
register_session(session_id, title)	Create a sessions row (no-op if exists)
save_message(session_id, role, content, ...)	Log a message to conversations
load_session_messages(session_id)	Reconstruct OpenAI message format from DB rows
list_sessions(limit)	List sessions with >=1 message, ordered by updated DESC
delete_session(session_id)	Delete session and all its messages
prune_sessions(retention_days)	Remove empty sessions and old unnamed sessions
resolve_session(alias_or_id)	Resolve alias, exact id, or id prefix to full session_id
save_session_config(session_id, config)	Persist session configuration key/value pairs
load_session_config(session_id)	Retrieve session configuration
set_session_alias(session_id, alias)	Set user-friendly alias (returns False if taken)
get_session_name(session_id)	Return alias if set, else title, else None
update_session_title(session_id, title)	Set/update LLM-generated title
kv_get(key) / kv_set(key, value) / kv_delete(key)	Generic key-value store (backs memories table)
kv_list() / kv_search(query)	List or search key-value pairs
search_history(query, limit)	Full-text search (FTS5 on SQLite, tsvector on PostgreSQL)
search_history_recent(limit)	Return most recent messages
close()	Release resources (connection pool, engine)

Database Configuration

[database]
backend = "sqlite"                  # "sqlite" | "postgresql"
path = ".turnstone.db"              # SQLite file path
url = ""                            # PostgreSQL connection URL
pool_size = 5                       # PostgreSQL connection pool size

Environment variables: TURNSTONE_DB_BACKEND, TURNSTONE_DB_URL, TURNSTONE_DB_PATH.

Session Persistence and Resume

Each ChatSession generates a 12-char hex _session_id on creation and registers it in the sessions table. Messages are saved to conversations as they happen via save_message().

Auto-titling: After the first complete exchange (user message + assistant response), a background thread calls the LLM with a title-generation prompt (reasoning_effort: "low", max_completion_tokens: 200). The generated title (3-8 words) is stored in sessions.title.

Resume flow: ChatSession.resume_session(session_id) calls load_session_messages() which reconstructs the OpenAI message format from database rows:

• user and assistant rows map directly
• Consecutive tool_call rows are grouped into one assistant message's tool_calls array, paired with subsequent tool_result rows via tool_call_id (or positional matching for legacy data)
• Interrupted session repair: If the last assistant message has tool_calls but fewer tool results than expected (session was interrupted mid-execution), the incomplete turn is stripped so the LLM can re-generate cleanly
• The session adopts the old _session_id, so new messages continue in the same session

Config persistence: LLM-affecting parameters (temperature, reasoning_effort, max_tokens, instructions, creative_mode) are persisted to the session_config table on creation and whenever changed via slash commands. resume_session() restores these values so resumed sessions behave identically to the original.

/clear vs /new: /clear wipes in-memory context but preserves messages in the database for future resume. /new starts a fresh session (new _session_id), leaving the old session resumable.

Resolution: resolve_session() accepts aliases, exact session IDs, or session ID prefixes, enabling turnstone --resume refactor or /resume abc12.

Session listing: list_sessions() only returns sessions that have at least one saved message (WHERE EXISTS on conversations). Sessions registered but never used (e.g., from process startup) are invisible until a message is sent.

Session pruning: prune_sessions(retention_days, log_fn) runs once at startup (CLI and server). It removes:

• Sessions with no messages (orphaned registrations)
• Unnamed sessions (alias IS NULL) older than retention_days days (default 90)

Named (aliased) sessions are never age-pruned. Configure with --session-retention-days N (0 = disable age pruning).

---

Error Handling and Retry

API Retry

ChatSession._create_stream_with_retry() (streaming path) and the agent _api_call() (non-streaming) both use the same retry pattern:

• Retries: 4 total attempts (1 initial + 3 retries, _MAX_RETRIES = 3)
• Backoff: exponential, base 1 second (delay = 1s * 2^attempt)
• Retryable errors: RateLimitError, APITimeoutError, APIConnectionError, InternalServerError, ServiceUnavailableError, APIError (matched by class name to avoid importing backend-specific exception hierarchies)
• On retry: ui.on_info() notification
• On final failure: exception propagates

_compact_messages() also wraps its non-streaming API call in the same retry loop.

Finish Reason Handling

_stream_response() tracks finish_reason from the final streaming chunk:

• "length": warns via ui.on_error() that the response was truncated. Any partial tool calls are discarded (their JSON would be malformed), causing the send() loop to exit cleanly.
• "content_filter": warns via ui.on_error() that the response was blocked.

Agent sub-sessions (_run_agent()) check finish_reason on each non-streaming response and stop the agent early on "length" or "content_filter".

_compact_messages() checks finish_reason on the compaction response and warns if the summary was truncated.

State Emission on Errors

• send() catches KeyboardInterrupt and generic Exception: calls _emit_state("error") before re-raising
• On interrupt: partial tool results and the originating assistant message are popped from self.messages to keep state consistent

Web UI Resilience

• SSE reconnect: both connectContentSSE() and connectGlobalSSE() use exponential backoff on onerror -- starting at 1 second, doubling on each failure, capped at 30 seconds. On successful message, delay resets to 1s.
• Disconnection indicator: #status-bar.disconnected class turns the status text red and shows "Reconnecting..."
• Fetch error handling: all fetch() calls use .catch() to prevent unhandled promise rejections
• Pending approval across tab switches: WebUI._pending_approval stores the approve_request event payload while the session is blocked waiting for user response. On SSE reconnect (e.g., switching back to the tab), the event is re-injected after history replay. _build_history marks the pending tool call as "pending": true so replayHistory skips the false ✓ approved badge; the live approval UI is rendered by the re-injected event instead.
• Browser history integration: history.pushState is called in switchTab() with {turnstone: 'workstream', wsId}. The initial state is seeded with history.replaceState({turnstone: 'dashboard'}) on load. The popstate listener restores the correct tab or shows the dashboard, guarded by _historyNavigation = true to prevent re-entrant pushState.

Eval Resilience

_run_single_test(): wraps session.send_headless() in a retry loop (3 attempts) to avoid transient API errors from poisoning evaluation scores.

Health Monitor & Circuit Breaker

BackendHealthMonitor (turnstone/core/healthcheck.py) runs a daemon thread that probes the LLM backend by calling client.models.list() every backend_probe_interval seconds (default 30). Probe results drive a three-state circuit breaker:

CLOSED  ──(N consecutive failures)──>  OPEN
OPEN    ──(cooldown expires)────────>  HALF_OPEN
HALF_OPEN ──(probe succeeds)────────>  CLOSED
HALF_OPEN ──(probe fails)──────────>  OPEN

• record_success() / record_failure() update _consecutive_failures and transition the _state (CircuitState enum: CLOSED, OPEN, HALF_OPEN).
• acquire_request_permit() returns False when the circuit is OPEN or when in HALF_OPEN and the single probe permit has already been consumed. Causes ChatSession._create_stream_with_retry to skip the backend and surface an error immediately.
• The /health endpoint reads the monitor's state: "status": "ok" when the circuit is closed, "status": "degraded" when open or half-open.

Rate Limiting

RateLimiter (turnstone/core/ratelimit.py) enforces per-client-IP request limits using a token-bucket algorithm. Each IP gets a TokenBucket with requests_per_second (refill rate) and burst (bucket capacity) from [ratelimit] config.

• Applied via RateLimitMiddleware after authentication but before route dispatch.
• /health and /metrics are exempt (monitoring must always be reachable).
• X-Forwarded-For support: when trusted_proxies is configured (comma-separated CIDRs), the middleware parses the X-Forwarded-For header using the rightmost-untrusted approach. IPv4-mapped IPv6 addresses are normalized. The direct client IP must be in the trusted set before XFF is considered.
• On limit exceeded: HTTP 429 with Retry-After header and JSON body {"error": "Rate limit exceeded", "retry_after": N}.
• The turnstone_ratelimit_rejected_total counter is incremented on each rejection.

---

Threading Model

CLI

Main thread          Spinner thread (daemon)       ThreadPoolExecutor
+--------------+     +------------------+          +-----------------+
| REPL loop    |     | Braille animation|          | Tool execution  |
| input() ->   |     | 80ms tick to     |          | max_workers=4   |
|   send() ->  |     | stderr           |          | parallel tools  |
|   stream  -> |     | started/stopped  |          | run concurrently|
|   tools   -> |     | by TerminalUI    |          |                 |
+--------------+     +------------------+          +-----------------+
       |                    ^                              ^
       +-- on_thinking_start/stop -------------------------+
       +-- _execute_tools ---------------------------------+

Key constraint: input() blocks the main thread. The spinner writes to stderr so it does not interfere with readline. Tool execution may use a ThreadPoolExecutor with up to 4 workers for parallel tool calls.

Server

Starlette ASGI app (served by uvicorn)
  |
  +-- Async request handlers (all under /v1/ prefix)
  |     POST /v1/api/send      -> starts worker thread per workstream
  |     POST /v1/api/approve   -> unblocks WebUI._approval_event
  |     POST /v1/api/plan      -> unblocks WebUI._plan_event
  |     POST /v1/api/workstreams/new -> creates workstream + worker
  |     GET  /v1/api/events    -> SSE via EventSourceResponse (per workstream)
  |     GET  /v1/api/events/global -> SSE via EventSourceResponse (fan-out)
  |
  +-- ASGI middleware stack
  |     MetricsMiddleware -> CORSMiddleware -> AuthMiddleware -> RateLimitMiddleware
  |
  +-- Worker thread per workstream (daemon)
  |     Runs session.send() synchronously -- ChatSession is fully blocking
  |     Blocks on WebUI._approval_event / _plan_event (threading.Event)
  |
  +-- Background daemon threads
        Global SSE fan-out: reads global_queue, copies to per-client queues
        Idle cleanup: closes stale workstreams, cleans rate limiter buckets

Starlette handles all HTTP routing, CORS, and middleware. uvicorn runs the ASGI application with async request handling. All API endpoints live under the /v1/ prefix via a Starlette Mount. An OpenAPI 3.1 spec is generated from Pydantic v2 models and served at /openapi.json; Swagger UI is available at /docs. SSE endpoints use EventSourceResponse from sse-starlette with async generators that bridge sync queue.Queue via asyncio.get_running_loop().run_in_executor().

ChatSession.send() remains synchronous, running in daemon worker threads. WebUI keeps threading.Event and queue.Queue primitives (unchanged from the sync era). The _global_fanout_thread and _idle_cleanup_thread remain as daemon threads since they interact with sync primitives. A lifespan context manager handles startup/shutdown (health monitor, MCP client, registry).

Each workstream's WebUI has:

• _event_queue (per-workstream SSE events, queue.Queue)
• _approval_event / _plan_event (threading.Event for blocking)
• _global_queue (class variable, shared, for state broadcasts)

The SSE handlers bridge these sync queues to async via run_in_executor(), polling queue.Queue.get(timeout=1) while sse-starlette handles keepalive pings automatically.

Workstream Threading (CLI)

Main thread                  Background workstream thread
+------------------+         +---------------------------+
| REPL input()     |         | session.send()            |
| /ws commands     |         | streams response          |
| active workstream|         | executes tools            |
| send() inline   |         | approve_tools() ->        |
+------------------+         |   _fg_event.wait() BLOCKS |
       |                     +---------------------------+
       |                                ^
       +-- /ws <N> switch ------------->|
       |   old.set_foreground(False)    |
       |   new.set_foreground(True)     |
       |   new.flush_buffer()           |
       +-- _fg_event.set() unblocks --->+

When a background workstream needs approval, its WorkstreamTerminalUI calls _fg_event.wait(), which blocks the worker thread until the user switches to that workstream. The _bg_attention_notify callback writes a bell + status line to stderr to alert the user.

Message Queue Bridge

Main thread              Global SSE thread         Per-WS SSE threads (×N)
+------------------+     +------------------+      +-------------------+
| Inbound loop     |     | GET /events/glob |      | GET /events?ws_id |
| BLPOP on Redis   |     | Parse SSE via    |      | Parse SSE via     |
|                  |     |   httpx-sse      |      |   httpx-sse       |
| Dispatch to      |     | Forward state    |      | Forward content,  |
|   handler        |     |   changes        |      |   tool results    |
| POST to server   |     | Detect turn      |      | Handle approval   |
| Publish ACK      |     |   completion     |      |   forwarding      |
+------------------+     +------------------+      +-------------------+
       |                         |                         |
       +-- Redis inbound queue   +-- Redis pub/sub         +-- Redis pub/sub
           (RPUSH/BLPOP)             (PUBLISH)                 (PUBLISH)
                                                               + response queue
                                                                 (BLPOP on
                                                                  approval)

Approval flow: When a per-WS SSE thread receives an approve_request, it checks the workstream's auto_approve_tools set. If all requested tools are in the set, the bridge auto-approves via POST /v1/api/approve. Otherwise, it publishes an ApprovalRequestEvent to the outbound channel with a request_id, then blocks on BLPOP of a Redis response queue (turnstone:resp:{request_id}) until the client pushes a response or the approval timeout (default 300s) expires.

Completion detection: The bridge tracks which correlation_id maps to which ws_id for active sends. When the global SSE reports ws_state → idle for a tracked workstream, the bridge emits a synthetic TurnCompleteEvent with the correlation ID.

Multi-node routing: Each bridge has a node_id (defaults to hostname) and BLPOPs from both turnstone:inbound:{node_id} (directed, priority) and turnstone:inbound (shared). Messages with target_node set are pushed to the target's per-node queue. Messages for existing workstreams are auto-routed via turnstone:ws:{ws_id} ownership keys in Redis. If a bridge picks up a shared-queue message for a workstream owned by another node, it re-routes to that node's queue (1 extra hop). Bridges publish heartbeats to turnstone:node:{node_id} with configurable TTL for node discovery.

Cluster Console

Monitoring (3 daemon threads)        Control + Proxy (async Starlette)
+------------------+                 +----------------------------+
| Event subscriber |                 | POST /v1/api/cluster/      |
| SUBSCRIBE on     |                 |   workstreams/new          |
| events:cluster   |                 |   → LPUSH to Redis         |
+------------------+                 |     inbound:{node_id}      |
| Node discovery   |                 +----------------------------+
| SCAN node:* keys |                 | GET /node/{node_id}/       |
| every 15 seconds |                 |   → httpx.AsyncClient      |
+------------------+                 |     proxy to server_url    |
| Poll loop        |                 | GET /node/{id}/v1/api/events |
| GET /v1/api/dash |                 |   → SSE stream proxy       |
| GET /health      |                 | POST /node/{id}/v1/api/send  |
| ThreadPoolExec   |                 |   → forwarded to server    |
+------------------+                 +----------------------------+

The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE endpoint uses EventSourceResponse with the same listener queue pattern as the main server. ClusterCollector's background threads (event subscriber, node discovery, poll loop) use sync Redis clients and ThreadPoolExecutor for parallel HTTP polling.

The console has two write-path capabilities:

1. Workstream creation — pushes CreateWorkstreamMessage to Redis inbound queues targeting specific nodes. The bridge on each node picks up the message and creates the workstream on the local server. Auto-selects the node with the most available capacity if no target is specified.

2. Reverse proxy — serves each node's server UI through the console port at /node/{node_id}/. Uses httpx.AsyncClient to proxy HTTP and SSE traffic. A JS shim is injected into the server's app.js to override fetch() and EventSource(), routing root-relative URLs through the proxy prefix. This eliminates the need for direct network access to individual server nodes.

The console also performs version drift detection — flagging when nodes report different versions via the /health endpoint. The overview API includes version_drift and versions fields; the dashboard shows a yellow warning indicator when versions diverge.

Clicking a workstream row in the console opens the proxied server UI at /node/{node_id}/?ws_id=<id> — the server's JS parses this on load and auto-selects the workstream. See docs/console.md for the full API reference.

---

Conversation Compaction

When the prompt exceeds auto_compact_pct of the context window (default: 80%, configurable via --auto-compact-pct), ChatSession auto-compacts by summarizing the entire conversation into a structured summary (_compact_messages). The summary model call uses compact_max_tokens (default: 32768, configurable via --compact-max-tokens). The summary preserves:

• Decisions made (architecture, libraries, approaches)
• Files read, created, or modified
• Exact identifiers, paths, and code snippets
• Important tool results
• Open tasks
• User preferences

After compaction, _read_files is cleared to force re-reads before edits, since file contents are no longer in the message history.

---

Client SDK

See also: SDK Architecture diagram | SDK Documentation

The turnstone/sdk/ package provides typed HTTP clients for programmatic access to both the server and console APIs. It wraps REST endpoints with methods that return Pydantic models, and SSE endpoints with async/sync iterators that yield typed event dataclasses.

Two client pairs (sync + async):

• TurnstoneServer / AsyncTurnstoneServer — server API (workstreams, chat, streaming, sessions)
• TurnstoneConsole / AsyncTurnstoneConsole — console API (cluster overview, nodes, workstreams)

Design: async-first with thin sync wrappers. _BaseClient provides httpx setup, auth headers, _request() (REST) and _stream_sse() (SSE). Sync clients delegate through _SyncRunner which maintains a persistent background event loop on a daemon thread.

Event types: 27 standalone dataclasses in events.py with a type-registry pattern matching OutboundEvent.from_json() from mq/protocol.py. Events are decoupled from the MQ package so SDK consumers don't need the redis dependency.

TypeScript SDK: sdk/typescript/ — separate npm package with the same API surface. Zero browser dependencies, SSE via fetch + ReadableStream parsing.

# Python quick start
from turnstone.sdk import TurnstoneServer

with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
    ws = client.create_workstream(name="demo")
    result = client.send_and_wait("Hello!", ws.ws_id)
    print(result.content)