turnstone/docs/architecture.md

# Turnstone Architecture

Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or
Anthropic's native Messages API via pluggable provider adapters, and gives the
model 19 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, planning, and executing code.

The core design principle is a **UI-agnostic engine with pluggable frontends**.
The engine (`ChatSession`) drives the conversation loop -- streaming, tool
dispatch, retry, compaction -- while every user-facing interaction is delegated
through the `SessionUI` protocol. Any frontend implements that protocol and
plugs in.

## Entry Points

| Command | Module | Frontend | Purpose |
|---------|--------|----------|---------|
| `turnstone` | `turnstone.cli` | `TerminalUI` | Interactive terminal REPL |
| `turnstone-server` | `turnstone.server` | `WebUI` | Browser-based chat (HTTP + SSE) |
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) |
| `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management |
| `turnstone-bootstrap` | `turnstone.bootstrap` | — | LLM-guided setup wizard |

---

## Module Map

```
turnstone/
  cli.py              Terminal frontend (TerminalUI, WorkstreamTerminalUI, REPL)
  server.py           Web frontend (WebUI, HTTP handler, static-file serving)
  eval.py             Evaluation harness (HeadlessSession, scoring, prompt optimization)
  core/
    session.py        ChatSession engine, SessionUI protocol, tool dispatch
    providers/        LLM provider adapters (pluggable backend layer)
      _protocol.py    LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
      _openai.py      OpenAIProvider facade (re-exports Chat/Responses providers)
      _openai_chat.py       OpenAIChatCompletionsProvider — vLLM, llama.cpp, local compatible APIs
      _openai_responses.py  OpenAIResponsesProvider — commercial OpenAI Responses API
      _openai_common.py     Shared ModelCapabilities table + helpers
      _anthropic.py   AnthropicProvider — Anthropic Messages API, native streaming, thinking
      _google.py      GoogleProvider — Google Gemini via OpenAI-compat endpoint
      __init__.py     create_provider() + create_client() factory functions
    workstream.py     Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
    tools.py          Tool schema loader (JSON -> OpenAI function-calling format)
    mcp_client.py     MCPClientManager — MCP server connections, tool discovery, dynamic refresh
    tool_search.py    Dynamic tool search — BM25 index, session-scoped tool visibility
    watch.py          WatchRunner daemon — periodic command polling, condition DSL, result dispatch
    judge.py          Intent validation — heuristic rules + LLM judge, advisory verdicts
    model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
    memory.py         Persistence facade + structured memory API (delegates to storage backend)
    config.py         Config file loader (config.toml), apply_config(), warn_migrated_settings()
    config_store.py   ConfigStore — database-backed settings with in-memory cache, thread-safe get/set
    settings_registry.py  SettingDef catalog (~40 settings), validation, type coercion, serialization
    storage/          Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
    metrics.py        Prometheus-compatible metrics collector (MetricsCollector)
    healthcheck.py    BackendHealthMonitor — periodic probe + circuit breaker
    ratelimit.py      Per-IP token-bucket rate limiter (RateLimiter, TokenBucket)
    edit.py           File edit utilities (find_occurrences, pick_nearest)
    safety.py         Command safety validation (blocked patterns, sanitization)
    sandbox.py        Math code sandboxing (AST validation, subprocess execution)
    web.py            Web utilities (HTML stripping, SSRF prevention)
  api/
    schemas.py        Shared Pydantic v2 models (auth, errors, WorkstreamState)
    server_schemas.py Server endpoint request/response models
    console_schemas.py Console endpoint request/response models
    openapi.py        OpenAPI 3.1 spec builder
    server_spec.py    Server endpoint catalog → build_server_spec()
    console_spec.py   Console endpoint catalog → build_console_spec()
    docs.py           /openapi.json + /docs (Swagger UI) handler factories
  sdk/
    server.py         AsyncTurnstoneServer + TurnstoneServer (HTTP client)
    console.py        AsyncTurnstoneConsole + TurnstoneConsole (HTTP client)
    events.py         27 SSE event dataclasses with type registry
    _base.py          Shared httpx async client, auth, error handling
    _sync.py          Background event loop for sync wrappers
    _types.py         TurnResult + TurnstoneAPIError
  console/
    collector.py      ClusterCollector — aggregates state from all nodes via SSE
    scheduler.py      TaskScheduler — background cron/at scheduler, dispatches via HTTP
    server.py         Cluster dashboard HTTP server + SSE + CLI entry point
    static/           Cluster dashboard web UI (page-specific HTML, CSS, JS)
  channels/
    cli.py            Unified channel gateway entry point (turnstone-channel)
    _protocol.py      ChannelAdapter protocol
    _routing.py       ChannelRouter — channel/thread ↔ workstream mapping via HTTP
    _config.py        Base ChannelConfig dataclass
    discord/          Discord adapter (bot, cog, views, streaming, config)
    slack/            Slack adapter (Socket Mode bot, DM routing, approval buttons)
  shared_static/      Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
    katex-0.17.0/    Vendored KaTeX math rendering library (MIT, woff2 fonts)
  ui/
    colors.py         ANSI color constants with NO_COLOR support
    markdown.py       Streaming terminal markdown renderer (line-buffered)
    spinner.py        Braille character spinner (daemon thread)
    static/
      index.html      Single-page app shell (links to CSS and JS)
      style.css       Page-specific UI styles (dashboard, markdown elements, approval blocks)
      renderer.js     Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
      app.js          Split-pane UI (Pane class, binary layout tree, SSE, tool approval)
  tools/
    *.json            19 tool schemas (OpenAI function-calling format + turnstone metadata)
```

Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.

---

## Core Loop

> See also: [Conversation Turn diagram](diagrams/png/04-conversation-turn.png)

A user message flows through the system as follows:

```
 User input
     |
     v
 ChatSession.send(user_input)
     |
     v
 _full_messages()  ------------>  system_messages + self.messages
     |
     v
 _emit_state("thinking")
     |
     v
 _create_stream_with_retry()  ---->  provider.create_streaming(client, model, messages, ...)
     |                                  up to 3 retries (4 total attempts), exponential backoff
     v
 _stream_response(stream)  -------->  dispatch tokens to UI:
     |                                  on_reasoning_token() / on_content_token()
     |                                  accumulate tool_calls from deltas
     |                                  track finish_reason
     |                                  _check_cancelled() per chunk (cooperative cancel)
     v
 finish_reason check:
     +--- "length"  --> warn, discard partial tool_calls
     +--- "content_filter" --> warn
     v
 tool_calls present?
     |
     +--- No ---> _print_status_line() -> _emit_state("idle") -> return
     |
     +--- Yes --> _emit_state("running")
                    |
                    v
                  _execute_tools(tool_calls)  <--- three-phase pipeline (see below)
                    |
                    v
                  append tool results to self.messages
                    |
                    v
                  loop back to _full_messages()
```

### Tool Execution Pipeline

> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)

Tool execution is a three-phase process:

```
Phase 1: PREPARE (serial)
  For each tool_call:
    _prepare_tool(tc)
      -> parse JSON arguments (with regex fallback for malformed JSON)
      -> dispatch to _prepare_{tool_name}(call_id, args)
      -> validate inputs, build preview text
      -> return item dict with: header, preview, needs_approval, execute fn

Phase 2: APPROVE (serial, blocking)
  _emit_state("attention")
  ui.approve_tools(items)
    -> display all headers and previews
    -> if any need approval and not auto_approve: prompt user
    -> return (approved, feedback)
  _emit_state("running")

Phase 3: EXECUTE (parallel)
  _check_cancelled()  <-- cancellation checkpoint before execution starts
  if len(items) == 1:
    run_one(items[0])
  else:
    ThreadPoolExecutor(max_workers=4).map(run_one, items)
  Bash tool streams stdout line-by-line via ui.on_tool_output_chunk(call_id, line)
    (cancel_event also checked per line — kills process group on cancel)
  Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
  call_id links tool_info items → streaming chunks → final result
  For plan tool: post-execution gate via ui.on_plan_review()
```

### State Transitions

The engine emits state changes via `_emit_state()` which calls
`ui.on_state_change(state)`. Frontends use these to update indicators
(spinner, tab badges, status line).

```
  send() called
      |
      v
  "thinking"  --->  streaming response
      |
      v
  "running"   --->  tool execution
      |
      v
  "attention"  --->  waiting for user approval / plan review
      |
      v
  "running"   --->  executing approved tools
      |
      v
  "idle"       --->  no more tool calls, turn complete
      |
  (or "error"  --->  exception or KeyboardInterrupt)

  cancel() may be called from any state. It sets a cooperative flag
  checked at each streaming chunk, before tool execution, and inside
  bash commands. The session transitions to "idle" with partial
  content preserved, emitting on_info("[Generation cancelled]").
```

---

## SessionUI Protocol

> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)

Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 16
methods. Every frontend must implement all of them.

```python
class SessionUI(Protocol):
    def on_turn_start(self) -> None: ...
    def on_turn_committed(self) -> None: ...
    def on_thinking_start(self) -> None: ...
    def on_thinking_stop(self) -> None: ...
    def on_reasoning_token(self, text: str) -> None: ...
    def on_content_token(self, text: str) -> None: ...
    def on_stream_end(self) -> None: ...
    def approve_tools(self, items: list[dict]) -> tuple[bool, str | None]: ...
    def on_tool_result(self, call_id: str, name: str, output: str, *, is_error: bool = False) -> None: ...
    def on_tool_output_chunk(self, call_id: str, chunk: str) -> None: ...
    def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
    def on_plan_review(self, content: str) -> str: ...
    def on_info(self, message: str) -> None: ...
    def on_error(self, message: str) -> None: ...
    def on_state_change(self, state: str) -> None: ...
    def on_rename(self, name: str) -> None: ...  # propagate alias to tab/UI label
```

`on_turn_start` fires at the top of each iteration of the send-loop;
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
`SessionUIBase` uses both to reset the per-turn inflight buffers
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
the per-workstream events stream in
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).

`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.

### Three Implementations

| Class | Module | Notes |
|-------|--------|-------|
| `TerminalUI` | `turnstone.cli` | ANSI colors, `MarkdownRenderer`, `Spinner`, readline-based `input()` for approval |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
| `NullUI` | `turnstone.eval` | Discards all output; `approve_tools` always returns `(True, None)` |

### WorkstreamTerminalUI

`WorkstreamTerminalUI` (in `turnstone.cli`) extends `TerminalUI` with workstream
awareness:

- **Output buffering**: When in background (`is_foreground` is False), tokens
  are appended to `_output_buffer` instead of written to stdout. When the user
  switches to this workstream, `flush_buffer()` replays them.

- **Approval blocking**: `approve_tools()` and `on_plan_review()` call
  `_fg_event.wait()` when in background, blocking the worker thread until the
  workstream is foregrounded. This ensures the user sees the approval prompt
  in the correct context.

- **Foreground/background toggle**: `set_foreground(bool)` sets or clears
  `_fg_event` (a `threading.Event`). The manager calls this during `/ws <N>`
  switches.

---

## Workstream Architecture

Workstreams are parallel, independent chat sessions. Each has its own
`ChatSession`, `SessionUI`, message history, and worker thread.

### WorkstreamState

> See also: [Workstream States diagram](diagrams/png/09-workstream-states.png)

Defined in `turnstone.core.workstream.WorkstreamState` (5 states):

```
IDLE       waiting for user input
THINKING   LLM is streaming a response
RUNNING    tools are executing
ATTENTION  blocked on user approval or plan review
ERROR      last operation failed
```

### Data Model

```python
@dataclass
class Workstream:
    id: str                              # uuid hex, 8 chars
    name: str                            # user-visible label
    state: WorkstreamState               # current state
    session: ChatSession | None          # the conversation engine
    ui: SessionUI | None                 # frontend adapter
    worker_thread: threading.Thread | None
    error_message: str
    last_active: float                   # time.monotonic() timestamp, updated on every state change
    _lock: threading.Lock                # per-workstream state lock
```

### WorkstreamManager

```python
class WorkstreamManager:
    MAX_WORKSTREAMS = 10

    def __init__(self, session_factory: Callable[[SessionUI], ChatSession]): ...
    def create(self, name="", ui_factory=None) -> Workstream: ...
    def close(self, ws_id: str) -> bool: ...
    def close_idle(self, max_age_seconds: float) -> list[str]: ...  # auto-close stale IDLE workstreams
    def get(self, ws_id: str) -> Workstream | None: ...
    def get_active(self) -> Workstream | None: ...
    def list_all(self) -> list[Workstream]: ...
    def switch(self, ws_id: str) -> Workstream | None: ...
    def switch_by_index(self, index: int) -> Workstream | None: ...
    def set_state(self, ws_id, state, error_msg=""): ...  # updates last_active
```

The `session_factory` pattern decouples session creation from configuration.
The factory captures shared config (client, model, temperature, etc.) and
accepts only a `SessionUI`, so the manager can create sessions without knowing
API details.

### Idle Workstream Lifecycle

The web server runs a background `_idle_cleanup_thread` (daemon) that calls
`WorkstreamManager.close_idle()` periodically (every `timeout / 4`, max 5 min).
Any IDLE workstream whose `last_active` is older than the configured timeout is
closed; non-IDLE workstreams (THINKING, RUNNING, ATTENTION, ERROR) are never
touched. The last workstream is always preserved even if expired. On close, a
`ws_closed` event is broadcast on the global SSE channel so browser clients
remove the tab immediately. Controlled by `--workstream-idle-timeout` (default:
120 minutes, 0 = disable).

**Workstream eviction at capacity:** When `WorkstreamManager.create()` would
exceed `max_workstreams` (configurable via `[server].max_workstreams`, default
50), the oldest IDLE workstream is automatically evicted to make room. The
`turnstone_workstreams_evicted_total` counter is incremented on each eviction.
If no IDLE workstream is available the create request fails as before.

### CLI Workstreams

- `/ws list` -- show all workstreams with state indicators
- `/ws new [name]` -- create a new workstream and switch to it
- `/ws <N>` -- switch to workstream by 1-based index
- `/ws close [N]` -- close a workstream
- `/ws rename <name>` -- rename the active workstream

Background notifications: when a background workstream enters `ATTENTION`
state, `_bg_attention_notify` writes an ANSI escape sequence to stderr
(overwrites the line above the prompt) with the workstream name.

Status line: `_print_ws_status_line()` shows a compact status of all
non-idle background workstreams above the input prompt.

### Web Workstreams

- **Tab bar**: Each workstream renders as a tab with a colored state indicator
  (CSS `@keyframes pulse` animation per state). Clicking a tab switches the
  focused pane's workstream (or focuses an existing pane showing that ws).
- **Split panes**: The UI supports tiling multiple workstreams side-by-side or
  stacked via a binary layout tree. Each `Pane` instance encapsulates its own
  SSE connection, message area, input, and state (busy, approval, streaming).
  Split via right-click context menu, pane header buttons, or keyboard
  (`Ctrl+\`, `Ctrl+Shift+\`). Max 6 panes; no duplicate workstreams across panes.
  Layout persisted to `localStorage`.
- **Per-pane SSE**: `Pane.connectSSE(wsId)` opens
  `/v1/api/workstreams/{ws_id}/events` for each pane's event stream independently.
- **Global SSE**: `connectGlobalSSE()` opens `/v1/api/events/global` which
  receives `ws_state` broadcasts from all workstreams, used to update tab
  indicators and pane headers without switching.
- **New tab / close**: POST `/v1/api/workstreams/new`, POST `/v1/api/workstreams/{ws_id}/close`.

### Thread Safety

- `WorkstreamManager._lock`: guards `_workstreams` dict and `_order` list on
  all create/close/switch/list operations.
- `Workstream._lock`: guards per-workstream state mutations in `set_state()`.
- `WorkstreamTerminalUI._print_lock`: guards `_output_buffer` access.
- `WorkstreamTerminalUI._fg_event`: `threading.Event` that blocks background
  approval until the workstream is foregrounded.

---

## Tool System

### Schema Format

Each tool is a JSON file in `turnstone/tools/`. The file contains an OpenAI
function-calling schema (`name`, `description`, `parameters`) plus optional
turnstone metadata keys:

| Metadata Key | Type | Meaning |
|-------------|------|---------|
| `agent` | `bool` | Include this tool when running as a plan/task sub-agent |
| `task_agent` | `bool` | Include this tool when running as a task sub-agent |
| `auto_approve` | `bool` | Tool is read-only; skip user approval |
| `primary_key` | `str` | Fallback argument name for bare-string JSON recovery |

Example (`read_file.json`):

```json
{
  "name": "read_file",
  "description": "Read the contents of a file. ...",
  "parameters": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "..." },
      "offset": { "type": "integer", "description": "..." },
      "limit": { "type": "integer", "description": "..." }
    },
    "required": ["path"]
  },
  "agent": true,
  "task_agent": true,
  "auto_approve": true,
  "primary_key": "path"
}
```

At import time, `turnstone.core.tools._load_tools()` strips the metadata keys
from each schema and builds:

- `TOOLS` -- list of `{"type": "function", "function": {...}}` dicts for the API
- `AGENT_TOOLS` -- subset with `agent: true`
- `TASK_AGENT_TOOLS` -- subset with `task_agent: true`
- `AGENT_AUTO_TOOLS` / `TASK_AUTO_TOOLS` -- sets of tool names with `auto_approve: true`
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
- `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init

### 19 Tools by Category

**Read-only (auto-approve)**:
- `read_file` -- read file contents with optional offset/limit
- `diff_file` -- show diff between two files / versions
- `search` -- ripgrep-based codebase search
- `man` -- read man pages
- `recall` -- search conversation history
- `read_resource` -- read an MCP resource by URI

**Write (requires approval)**:
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
- `write_file` -- create or overwrite a file
- `edit_file` -- string replacement in an existing file (requires prior `read_file`)
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, self-hosted SearxNG fallback for local models)
- `notify` -- send a user-facing notification (Discord/Slack, optional reply routing)
- `watch` -- schedule a recurring poll with condition DSL

**Agent (delegated sub-sessions)**:
- `task_agent` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan_agent` -- explore codebase and write a structured plan (`AGENT_TOOLS`)

**Memory / skills / prompts**:
- `memory` -- save, search, delete, or list memories (typed and scoped)
- `skill` -- invoke a skill (governed, versioned procedure)
- `use_prompt` -- fetch and apply a prompt template

Tool names are `plan_agent` / `task_agent` (not `plan` / `task`); bare words
collide with chat-template channels on some local models.

### Prepare / Execute Pattern

Every tool has a `_prepare_{name}` method and a corresponding `_exec_{name}`
method on `ChatSession`:

```
_prepare_bash(call_id, args)   -> item dict with execute=self._exec_bash
_prepare_read_file(call_id, args) -> item dict with execute=self._exec_read_file
...
```

The prepare method validates inputs and builds the preview. The item dict
carries the validated data and a reference to the execute function. This
separation allows the UI to show previews before any side effects occur.

### Agent Tools

`task_agent` and `plan_agent` invoke `_run_agent()`, which runs a multi-turn
loop with a subset of tools and its own system prompt. The sub-agent runs
independently, then returns the final content as the tool result.

- **task_agent**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan_agent**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
  to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
  don't collide. On repeat invocations the prior `plan_agent` tool call and its result
  are forwarded from `self.messages` so the agent refines the existing plan rather
  than starting over. Planning instructions are injected as a developer message
  prepended to the agent's conversation.
- **Turn limit**: controlled by `agent_max_turns` (default: `-1`, unlimited).
  When a limit is set and reached, the agent is forced to synthesize a final
  response without tools. When unlimited, the loop only exits when the model
  stops calling tools or hits `finish_reason: "length"`.
- **Retry**: each API call in the agent loop uses the same retry+backoff logic
  as the main `_create_stream_with_retry()`.
- **Finish reason handling**: `finish_reason: "length"` stops the agent early
  and returns whatever content was generated. `finish_reason: "content_filter"`
  returns a placeholder.

### MCP Tool Integration

`MCPClientManager` (`turnstone/core/mcp_client.py`) connects to external MCP servers
and exposes their tools alongside built-in tools. The MCP SDK is fully async; turnstone
bridges this with a background asyncio event loop in a daemon thread.

**Configuration sources:** MCP servers can be defined in config files (TOML/JSON)
or in the database via the admin UI. Database-backed definitions are managed
through the console admin panel's MCP Servers tab and stored in the
`mcp_servers` table. On startup, `load_mcp_config(storage=)` uses
first-match-wins priority: DB rows (if any enabled) take precedence over
config files. The console can trigger a cluster-wide reload (`POST
/_internal/mcp-reload`) that causes each node to call `reconcile_sync()`,
which diffs the running MCP connections against the current DB state and
adds, removes, or reconnects servers as needed.

**Lifecycle:**
1. `create_mcp_client()` reads server configs from TOML/JSON and database
2. `MCPClientManager.start()` launches the background event loop thread
3. `_connect_all()` connects to each server (stdio subprocess or HTTP), runs
   `initialize()` + `list_tools()`, converts schemas to OpenAI format, detects
   `tools.listChanged` capability for push notification support
4. `ChatSession.__init__` receives the manager, builds `self._tools` (built-in + MCP),
   and registers a listener callback for tool-change notifications
5. `_prepare_tool()` routes MCP tools to `_prepare_mcp_tool()` / `_exec_mcp_tool()`
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
   via `asyncio.run_coroutine_threadsafe()`

**Tool refresh:** Two mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
  the registered `message_handler` triggers immediate single-server refresh.
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
  (also attempts reconnection for disconnected servers).

When tools change, `_rebuild_tools()` creates new `_tools`/`_tool_map` objects
(copy-on-write for thread safety) and notifies listener callbacks. Each `ChatSession`
rebuilds its merged tool lists and reconstructs `ToolSearchManager` (preserving
expanded tools).

**Tool naming:** `mcp__{server}__{tool}` — double underscore delimiter, validated
at connection time (server names with `__` are rejected).

**Resilience:** Each MCP server has an independent circuit breaker that opens
after 3 consecutive transport failures (timeouts, broken pipes, connection
resets). Cooldown uses capped exponential backoff (30 s base, 5 min max) with
per-server jitter to avoid thundering herd. Protocol-level errors (`McpError`)
from a healthy connection do not trip the breaker. When the cooldown expires
(half-open), the next operation attempt triggers automatic reconnection. Manual
`/mcp refresh` also clears the circuit on success. All sync bridge methods
(`call_tool_sync`, `read_resource_sync`, `get_prompt_sync`, `refresh_sync`)
cancel orphaned futures on timeout to prevent coroutine accumulation on the
background event loop. Push notification refreshes are debounced (5 s per
server) to protect against notification storms. Operators can force a
catalog refresh or full reconnect from the admin panel; reconnects clear
the circuit breaker and run a fresh handshake. Transport stream references
are pre-closed before stack teardown to work around the MCP SDK's anyio
cancel-scope CPU busy-loop (SDK #2147).

**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
rather than crashing the session.

**Registry discovery:** The console admin panel provides a registry discovery
surface backed by the official MCP Registry (registry.modelcontextprotocol.io).
`MCPRegistryClient` (`turnstone/core/mcp_registry.py`) is a standalone httpx
async client that queries the registry's v0.1 API for server discovery. Search
results are annotated with installed status by cross-referencing the
`mcp_servers` table. Installation creates a DB row with `registry_name`,
`registry_version`, and `registry_meta` columns (migration 019), then triggers
cluster-wide node reload via `_notify_nodes_mcp_reload()`. The registry URL is
configurable via the `mcp.registry_url` setting for enterprise/private
registries.

### Provider Adapter Layer

> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)

`ChatSession` is provider-agnostic — it delegates all LLM communication to an
`LLMProvider` protocol (`turnstone/core/providers/_protocol.py`). Internally,
messages use an OpenAI-like format; each provider translates at the API boundary.

```
ChatSession
    |
    v
LLMProvider (protocol)
    |
    +--- OpenAIProvider  --- OpenAI, vLLM, llama.cpp, any /v1/chat/completions API
    +--- AnthropicProvider --- Anthropic Messages API (native streaming, thinking)
    +--- GoogleProvider  --- Google Gemini via /v1beta/openai/ (extends OpenAIProvider)
```

**Protocol methods:**

| Method | Purpose |
|--------|---------|
| `create_streaming()` | Streaming request, yields normalized `StreamChunk` objects |
| `create_completion()` | Non-streaming request, returns `CompletionResult` |
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |

**Normalized data types:**

| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |

**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
already in OpenAI format), including multi-part content blocks (text + images)
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
For search models, injects `web_search_options` and removes the `web_search`
function tool (the model always searches). Citations from `url_citation`
annotations are formatted as footnotes. Extended prompt cache retention
(`prompt_cache_retention: "24h"`) is enabled for GPT-5.x models at no
additional cost. Cached token counts are extracted from
`usage.prompt_tokens_details.cached_tokens`. Unknown models (local servers) get
permissive defaults with `supports_vision=False` and use SearxNG for web search.

**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
Anthropic content blocks, maps `system`/`developer` roles to the `system`
parameter, groups consecutive `tool` result messages into user-role content
blocks (converting `image_url` parts to Anthropic's `image` source format),
and translates tool schemas from OpenAI function-calling format to
Anthropic's `input_schema` format. Supports both manual and adaptive thinking
modes, with effort parameter support for models like Claude Opus 4.6 and
Sonnet 4.6. Replaces the `web_search` function tool with Anthropic's native
`web_search_20250305` server-side tool — Claude decides when to search, the
API executes it, and results stream back as `server_tool_use` /
`web_search_tool_result` content blocks (emitted as `info_delta` for UI
display). Automatic prompt caching is enabled via top-level `cache_control:
{"type": "ephemeral"}` — the API places the cache breakpoint on the last
cacheable block and advances it as conversations grow (90% input cost
reduction on cache hits, 1.25x write on first turn). Cache metrics
(`cache_creation_input_tokens`, `cache_read_input_tokens`) are extracted from
both streaming and non-streaming responses. The `anthropic` SDK is imported
lazily so it remains an optional dependency (`pip install
turnstone[anthropic]`).

**GoogleProvider** (`_google.py`): extends `OpenAIChatCompletionsProvider` for
the Gemini `/v1beta/openai/` endpoint. Uses a single default
`ModelCapabilities` (2M context window, 65K max output tokens,
`token_param=max_tokens`) since Google updates models frequently. No static
per-model capability table. Google's endpoint is wire-compatible with the
OpenAI SDK, so no extra dependency is needed.

**Factory functions** (`__init__.py`): `create_provider(name)` returns a
singleton provider instance (thread-safe). `create_client(name, base_url,
api_key)` creates the appropriate SDK client.

### Multi-Model Registry

`ModelRegistry` (`turnstone/core/model_registry.py`) manages named model
configurations so workstreams can use different LLM backends.

**Config format:**
```toml
[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
# provider defaults to "openai"

[models.claude]
provider = "anthropic"
api_key = "sk-ant-..."
model = "claude-opus-4-6"
context_window = 200000

[models.openai]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "gpt-5"
context_window = 400000

[models.gemini]
provider = "google"
model = "gemini-2.5-pro"

[model]
default = "local"
fallback = ["claude", "openai"]
agent_model = "claude"
```

Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
(default: `"openai"`). Supported values: `"openai"`, `"anthropic"`, `"google"`,
and `"openai-compatible"`.

**Per-model sampling overrides:** Each model can specify `temperature`,
`max_tokens`, and `reasoning_effort` to override the global defaults from
ConfigStore. When unset (`NULL`), the global default is used.

**Per-model reasoning persistence:** Two booleans on `model_definitions`
(migration 052) control how reasoning text round-trips:

* `surface_persisted_reasoning` (default `True`) — gates whether stored
  reasoning text is surfaced on `/history` payloads for UI rehydration.
  **Storage of reasoning bytes happens regardless of this flag** — they
  ride in `provider_data` independently. Phase-1 admin UI label "Surface
  persisted reasoning."
* `replay_reasoning_to_model` (default `False`) — gates whether stored
  reasoning blocks are sent back to the provider on subsequent turns.
  Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
  also be `True` for the wire path to actually replay (canonical OpenAI
  gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
  server models default to `False`).

Three reasoning paths are recognised:

| Path | Provider | Capture | Persist | Replay |
|------|----------|---------|---------|--------|
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |

Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
text+tool_calls rebuild path rather than reaching Anthropic's input
boundary as malformed content.

```toml
[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
temperature = 0.7
max_tokens = 8192

[models.o3]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "o3"
reasoning_effort = "high"
# temperature omitted — uses global default
```

An optional `[models.*.capabilities]` sub-table overrides per-model
`ModelCapabilities` flags (useful for local models whose capabilities
cannot be detected programmatically):

```toml
[models.qwen-vl]
base_url = "http://localhost:8000/v1"
model = "qwen-3.5-vl"

[models.qwen-vl.capabilities]
supports_vision = true
```

**Database model definitions:** On server entry points, models can also be
defined in the `model_definitions` table (admin Models tab). DB models support
the same per-model sampling overrides. Config.toml models override DB models
with the same alias in-memory (the DB rows are never modified).

**Lifecycle:**
1. `load_model_registry()` loads DB model definitions (if storage available),
   then overlays `[models.*]` from config.toml, then builds a `"default"` entry
   from CLI `--base-url`/`--model`/`--api-key` args
2. The registry is passed to the session factory closure in both `cli.py` and
   `server.py`; each workstream resolves its model on creation
3. `ModelRegistry.get_client()` lazily creates SDK client instances via
   `create_client()` — `OpenAI` for the openai provider, `Anthropic` for
   the anthropic provider (thread-safe via `_client_lock`)
4. `ModelRegistry.get_provider()` lazily creates `LLMProvider` instances via
   `create_provider()` (also cached and thread-safe)
5. `/model` command shows available models; `/model <alias>` switches the
   active workstream's client, model, context window, and per-model sampling
   parameters
6. `_create_stream_with_retry()` tries the primary model, then each fallback
   alias in order if the primary is unreachable
7. `_run_agent()` resolves `registry.agent_model` (if set) for plan/task
   sub-agents, allowing a cheaper model for autonomous loops

**Per-workstream selection:** `POST /v1/api/workstreams/new` accepts an optional
`"model"` field, along with `skill` (skill name)
which can override the model before workstream creation.

### Tool Output Truncation

Tool execution results (bash, read_file, search, math, man) are truncated by
`_truncate_output()` when they exceed `tool_truncation` characters. Truncation
preserves the first half and last half of the output, with a message in
between:

```
... [N chars truncated — output exceeded LIMIT char limit] ...
```

The default limit is 50% of the context window in characters (computed as
`context_window * chars_per_token * 0.5`). For a 131K context window this is
~262K characters. Override with `--tool-truncation <chars>`.

This truncation message is visible to the model, so it knows output was cut.

---

## Persistence

### Storage Architecture

Persistence is managed by the `turnstone.core.storage` package — a pluggable
backend behind a `StorageBackend` protocol. The `memory.py` facade provides
backward-compatible module-level functions that delegate to the active backend.

```
session.py / server.py / cli.py
        ↓
    memory.py  (facade — silent-failure wrappers)
        ↓
    storage._registry  (singleton factory)
        ↓
  ┌─────────────┐    ┌──────────────────┐
  │ SQLiteBackend │    │ PostgreSQLBackend │
  │ (FTS5 search) │    │ (tsvector/ILIKE)  │
  └─────────────┘    └──────────────────┘
        ↓                     ↓
    storage._schema  (SQLAlchemy Core tables — single source of truth)
        ↓
    storage._migrate  (programmatic Alembic)
```

**SQLite** is the default (zero-config, single file at `.turnstone.db`).
**PostgreSQL** is the production backend (connection pooling, `tsvector`
full-text search). Select via `[database]` in `config.toml`, CLI flags, or
environment variables (`TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`).

Schema migrations are managed by Alembic and run automatically on startup.
Existing SQLite databases created before the migration system are auto-stamped
at the baseline revision.

### Tables

```sql
memories
  key      TEXT PRIMARY KEY
  value    TEXT NOT NULL
  created  TEXT NOT NULL
  updated  TEXT NOT NULL

workstreams
  ws_id       TEXT PRIMARY KEY
  node_id     TEXT NOT NULL
  name        TEXT NOT NULL
  state       TEXT NOT NULL DEFAULT 'idle'
  alias       TEXT UNIQUE            -- user-assigned short name (nullable)
  title       TEXT                   -- LLM-generated title (nullable)
  created     TEXT NOT NULL
  updated     TEXT NOT NULL          -- bumped on every save_message()

conversations
  id            INTEGER PRIMARY KEY AUTOINCREMENT
  ws_id         TEXT NOT NULL
  timestamp     TEXT NOT NULL
  role          TEXT NOT NULL        -- user | assistant | tool_call | tool_result
  content       TEXT
  tool_name     TEXT
  tool_args     TEXT
  tool_call_id  TEXT                 -- links tool_call ↔ tool_result for resume
  provider_data TEXT                 -- raw provider content (e.g. Anthropic encrypted)

workstream_config
  ws_id       TEXT NOT NULL          -- composite PK with key
  key         TEXT NOT NULL
  value       TEXT

conversations_fts                    -- SQLite FTS5 virtual table (optional)
  content     (content=conversations, content_rowid=id)
```

Table definitions live in `storage/_schema.py` (SQLAlchemy Core `Table` objects)
and are the single source of truth for both backends and Alembic migrations.

### StorageBackend Protocol

| Method | Purpose |
|--------|---------|
| `register_workstream(ws_id, node_id, name, state)` | Create a workstreams row (no-op if exists) |
| `save_message(ws_id, role, content, ...)` | Log a message to conversations |
| `load_messages(ws_id)` | Reconstruct OpenAI message format from DB rows |
| `list_workstreams_with_history(limit)` | List workstreams with >=1 message, ordered by updated DESC |
| `delete_workstream(ws_id)` | Delete workstream and cascade conversations + config |
| `prune_workstreams(retention_days)` | Remove empty workstreams and old unnamed workstreams |
| `resolve_workstream(alias_or_id)` | Resolve alias, exact id, or id prefix to full ws_id |
| `save_workstream_config(ws_id, config)` | Persist workstream configuration key/value pairs |
| `load_workstream_config(ws_id)` | Retrieve workstream configuration |
| `set_workstream_alias(ws_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_workstream_display_name(ws_id)` | Return alias if set, else title, else None |
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
| `update_workstream_name(ws_id, name)` | Update workstream display name |
| `list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id)` | List workstreams, optionally filtered by node, parent, kind, or owning user |
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
| `search_history(query, limit)` | Full-text search (FTS5 on SQLite, tsvector on PostgreSQL) |
| `search_history_recent(limit)` | Return most recent messages |
| `close()` | Release resources (connection pool, engine) |

### Database Configuration

```toml
[database]
backend = "sqlite"                  # "sqlite" | "postgresql"
path = ".turnstone.db"              # SQLite file path
url = ""                            # PostgreSQL connection URL
pool_size = 2                       # PostgreSQL connection pool size (per process)
```

Environment variables: `TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`, `TURNSTONE_DB_PATH`,
`TURNSTONE_DB_POOL_SIZE`.

The default pool is intentionally small (2 base + 3 overflow = 5 per process)
because all database operations are short-burst queries that hold connections for
milliseconds. For clusters with many nodes sharing a PostgreSQL instance, use
[PgBouncer](pgbouncer.md) in transaction pooling mode.

### Persistence and Resume

`ws_id` is the sole persistent identity for both routing and conversation
history. There is no separate `session_id` — the `workstreams` table holds
alias, title, and state alongside the routing fields (`node_id`, `name`).
Messages are saved to `conversations` (keyed by `ws_id`) as they happen
via `save_message()`. Workstream state changes are tracked via
`update_workstream_state()`.

**Auto-titling:** After the first complete exchange (user message + assistant
response), a background thread calls the LLM with a title-generation prompt
(`reasoning_effort: "low"`, `max_completion_tokens: 200`). The generated
title (3-8 words) is stored in `workstreams.title`.

**Resume flow:** `ChatSession.resume(ws_id)` calls `load_messages()` which
reconstructs the OpenAI message format from database rows:

- `user` and `assistant` rows map directly
- Consecutive `tool_call` rows are grouped into one assistant message's
  `tool_calls` array, paired with subsequent `tool_result` rows via
  `tool_call_id` (or positional matching for legacy data)
- **Interrupted conversation repair:** If the last assistant message has
  `tool_calls` but fewer tool results than expected (conversation was
  interrupted mid-execution), the incomplete turn is stripped so the
  LLM can re-generate cleanly
- The `ChatSession` adopts the resumed `_ws_id`, so new messages continue
  in the same workstream

**Config persistence:** LLM-affecting parameters (`temperature`,
`reasoning_effort`, `max_tokens`, `instructions`, `creative_mode`) are
persisted to the `workstream_config` table on creation and whenever changed
via slash commands. `resume()` restores these values so resumed workstreams
behave identically to the original.

**`/clear` vs `/new`:** `/clear` wipes in-memory context but preserves
messages in the database for future resume. `/new` starts a fresh workstream
(new `_ws_id`), leaving the old workstream resumable.

**Resolution:** `resolve_workstream()` accepts aliases, exact workstream IDs,
or ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.

**Workstream listing:** `list_workstreams_with_history()` only returns
workstreams that have at least one saved message (`WHERE EXISTS` on
`conversations`). Workstreams registered but never used (e.g., from process
startup) are invisible until a message is sent.

**Workstream pruning:** `prune_workstreams(retention_days, log_fn)` runs once
at startup (CLI and server). It removes:
- Workstreams with no messages (orphaned registrations)
- Unnamed workstreams (`alias IS NULL`) older than `retention_days` days (default 90)

Named (aliased) workstreams are never age-pruned. Configure with
`--retention-days N` (0 = disable age pruning).

---

## Error Handling and Retry

### API Retry

`ChatSession._create_stream_with_retry()` (streaming path) and the agent
`_api_call()` (non-streaming) both use the same retry pattern:

- **Retries**: 4 total attempts (1 initial + 3 retries, `_MAX_RETRIES = 3`)
- **Backoff**: exponential, base 1 second (`delay = 1s * 2^attempt`)
- **Retryable errors**: `RateLimitError`, `APITimeoutError`,
  `APIConnectionError`, `InternalServerError`, `ServiceUnavailableError`,
  `APIError` (matched by class name to avoid importing backend-specific
  exception hierarchies)
- On retry: `ui.on_info()` notification
- On final failure: exception propagates

`_compact_messages()` also wraps its non-streaming API call in the same
retry loop.

### Finish Reason Handling

`_stream_response()` tracks `finish_reason` from the final streaming chunk:

- **`"length"`**: warns via `ui.on_error()` that the response was truncated.
  Any partial tool calls are discarded (their JSON would be malformed),
  causing the `send()` loop to exit cleanly.
- **`"content_filter"`**: warns via `ui.on_error()` that the response was
  blocked.

Agent sub-sessions (`_run_agent()`) check `finish_reason` on each
non-streaming response and stop the agent early on `"length"` or
`"content_filter"`.

`_compact_messages()` checks `finish_reason` on the compaction response and
warns if the summary was truncated.

### State Emission on Errors

- `send()` catches `KeyboardInterrupt` and generic `Exception`: calls
  `_emit_state("error")` before re-raising
- On interrupt: partial tool results and the originating assistant message
  are popped from `self.messages` to keep state consistent

### Web UI Resilience

- **SSE reconnect**: both `connectContentSSE()` and `connectGlobalSSE()` use
  exponential backoff on `onerror` -- starting at 1 second, doubling on each
  failure, capped at 30 seconds. On successful message, delay resets to 1s.
- **Disconnection indicator**: `#status-bar.disconnected` class turns the
  status text red and shows "Reconnecting..."
- **Fetch error handling**: all `fetch()` calls use `.catch()` to prevent
  unhandled promise rejections
- **Pending approval across tab switches**: `WebUI._pending_approval` stores
  the `approve_request` event payload while the session is blocked waiting
  for user response. On tab switch / reconnect the pane reloads history via
  REST `GET /history` and then reconnects SSE; the live approval event is
  re-injected. The server-side `project_history_messages` projection marks
  the trailing orphan tool-call turn `"pending": true` so `replayHistory`
  skips the false `✓ approved` badge; the live approval UI is rendered by
  the re-injected event instead.
- **Browser history integration**: `history.pushState` is called in
  `switchTab()` with `{turnstone: 'workstream', wsId}`. The initial state is
  seeded with `history.replaceState({turnstone: 'dashboard'})` on load. The
  `popstate` listener restores the correct tab or shows the dashboard,
  guarded by `_historyNavigation = true` to prevent re-entrant pushState.
- **Pane focus**: `mousedown` and `focusin` events on pane containers update
  `focusedPaneId`. Approval shortcuts (y/n/a) apply to the focused pane.
  `Ctrl+Alt+Arrow` cycles focus between panes.

### Eval Resilience

`_run_single_test()`: wraps `session.send_headless()` in a retry loop (3
attempts) to avoid transient API errors from poisoning evaluation scores.

### Health Monitor & Circuit Breaker

`BackendHealthMonitor` (`turnstone/core/healthcheck.py`) runs a daemon thread
that probes the LLM backend by calling `client.models.list()` every
`backend_probe_interval` seconds (default 30). Probe results drive a three-state
circuit breaker:

```
CLOSED  ──(N consecutive failures)──>  OPEN
OPEN    ──(cooldown expires)────────>  HALF_OPEN
HALF_OPEN ──(probe succeeds)────────>  CLOSED
HALF_OPEN ──(probe fails)──────────>  OPEN
```

- `record_success()` / `record_failure()` update `_consecutive_failures` and
  transition the `_state` (`CircuitState` enum: `CLOSED`, `OPEN`, `HALF_OPEN`).
- `acquire_request_permit()` returns `False` when the circuit is `OPEN` or when
  in `HALF_OPEN` and the single probe permit has already been consumed. Causes
  `ChatSession._create_stream_with_retry` to skip the backend and surface an
  error immediately.
- The `/health` endpoint reads the monitor's state: `"status": "ok"` when the
  circuit is closed, `"status": "degraded"` when open or half-open.

### Rate Limiting

`RateLimiter` (`turnstone/core/ratelimit.py`) enforces per-client-IP request
limits using a token-bucket algorithm. Each IP gets a `TokenBucket` with
`requests_per_second` (refill rate) and `burst` (bucket capacity) from
`[ratelimit]` config.

- Applied via `RateLimitMiddleware` after authentication but before route dispatch.
- `/health` and `/metrics` are exempt (monitoring must always be reachable).
- **X-Forwarded-For support**: when `trusted_proxies` is configured (comma-separated
  CIDRs), the middleware parses the `X-Forwarded-For` header using the
  rightmost-untrusted approach. IPv4-mapped IPv6 addresses are normalized.
  The direct client IP must be in the trusted set before XFF is considered.
- On limit exceeded: HTTP 429 with `Retry-After` header and JSON body
  `{"error": "Rate limit exceeded", "retry_after": N}`.
- The `turnstone_ratelimit_rejected_total` counter is incremented on each
  rejection.

---

## User Identity and Authentication

Turnstone supports three authentication mechanisms, unified behind an
`AuthResult` dataclass that carries `user_id`, `scopes`, and `token_source`:

1. **API tokens** — database-backed, prefixed `ts_`, stored as SHA-256 hashes
   in the `api_tokens` table. Can be exchanged for JWTs via
   `POST /v1/api/auth/login`.
2. **JWTs** — short-lived HMAC-SHA256 session tokens (default 24h) issued after
   successful credential validation. Contain `sub` (user_id), `scopes`, and
   `src` (origin) in claims.

### Scope Model

Three hierarchical scopes control endpoint access:

| Scope | Grants | Endpoints |
|-------|--------|-----------|
| `read` | SSE streams, workstream listing, history | GET endpoints |
| `write` | `read` + send, command, workstream create/close | POST to `/api/workstreams/{ws_id}/send`, `/api/command`, etc. |
| `approve` | `write` + tool approval, admin operations | POST to `/api/workstreams/{ws_id}/approve`, `/api/admin/*` |

### Middleware Flow

`AuthMiddleware` (ASGI) intercepts every request:

1. **Public path check** — `/`, `/static/*`, `/shared/*`, `/health`,
   `/metrics`, `/openapi.json`, `/docs`, `/api/auth/*`, and `/api/auth/setup`
   are always allowed.
2. **Token extraction** — `Authorization: Bearer <token>` header first, then
   `turnstone_auth` cookie as fallback.
3. **Token type detection** — dots in the token indicate JWT; `ts_` prefix
   indicates API token.
4. **Validation** — JWT signature check or API token hash lookup in storage.
5. **Scope check** — `required_scope(method, path)` determines the minimum
   scope; the request is rejected with 403 if the token lacks it.
6. **Context propagation** — on success, `ctx_user_id` is set so structured
   logging includes the authenticated identity on every log event.

### Architecture Split

- **Console** is the auth management hub — it hosts the admin endpoints for
  creating users, issuing API tokens, and managing channel mappings. User
  records and token hashes live in the shared storage backend. The console
  dashboard includes an **admin panel** (18 tabs) for managing
  credentials, governance, MCP servers, models, node metadata, and runtime
  settings through the browser.
- **Server** is a JWT validator only — it validates tokens on each request but
  never creates users or tokens. Both processes share the same `jwt_secret`
  (via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config).
- **First-time setup** — both server and console expose
  `POST /v1/api/auth/setup`, a public endpoint that creates the initial admin
  user when no users exist. This avoids the chicken-and-egg problem of needing
  `approve` scope to create the first user via `/api/admin/users`.

### Auth Storage Tables

Three tables in `storage/_schema.py` support identity:

```sql
users
  user_id        TEXT PRIMARY KEY
  username       TEXT NOT NULL UNIQUE
  display_name   TEXT NOT NULL
  password_hash  TEXT NOT NULL       -- bcrypt
  created        TEXT NOT NULL

api_tokens
  token_id       TEXT PRIMARY KEY
  token_hash     TEXT NOT NULL UNIQUE  -- SHA-256 of raw token
  token_prefix   TEXT NOT NULL         -- first 8 chars for display
  user_id        TEXT NOT NULL
  name           TEXT NOT NULL         -- human-readable label
  scopes         TEXT NOT NULL         -- comma-separated
  created        TEXT NOT NULL
  expires        TEXT                  -- optional expiry timestamp

channel_users
  channel_type      TEXT NOT NULL      -- e.g. "slack", "discord"
  channel_user_id   TEXT NOT NULL      -- platform-specific user ID
  user_id           TEXT NOT NULL      -- FK to users
  PRIMARY KEY (channel_type, channel_user_id)
```

See [docs/security.md](security.md) for full security details including token
lifecycle, password hashing, and deployment hardening.

---

## Threading Model

### CLI

```
Main thread          Spinner thread (daemon)       ThreadPoolExecutor
+--------------+     +------------------+          +-----------------+
| REPL loop    |     | Braille animation|          | Tool execution  |
| input() ->   |     | 80ms tick to     |          | max_workers=4   |
|   send() ->  |     | stderr           |          | parallel tools  |
|   stream  -> |     | started/stopped  |          | run concurrently|
|   tools   -> |     | by TerminalUI    |          |                 |
+--------------+     +------------------+          +-----------------+
       |                    ^                              ^
       +-- on_thinking_start/stop -------------------------+
       +-- _execute_tools ---------------------------------+
```

Key constraint: `input()` blocks the main thread. The spinner writes to
stderr so it does not interfere with readline. Tool execution may use a
`ThreadPoolExecutor` with up to 4 workers for parallel tool calls.

### Server

```
Starlette ASGI app (served by uvicorn)
  |
  +-- Async request handlers (all under /v1/ prefix)
  |     POST /v1/api/workstreams/{ws_id}/send    -> starts worker thread per workstream
  |     POST /v1/api/workstreams/{ws_id}/approve -> unblocks WebUI._approval_event
  |     POST /v1/api/plan                        -> unblocks WebUI._plan_event
  |     POST /v1/api/workstreams/new             -> creates workstream + worker
  |     GET  /v1/api/workstreams/{ws_id}/events  -> SSE via EventSourceResponse (per workstream)
  |     GET  /v1/api/events/global               -> SSE via EventSourceResponse (fan-out)
  |
  +-- ASGI middleware stack
  |     MetricsMiddleware -> CORSMiddleware -> AuthMiddleware -> RateLimitMiddleware
  |
  +-- Worker thread per workstream (daemon)
  |     Runs session.send() synchronously -- ChatSession is fully blocking
  |     Blocks on WebUI._approval_event / _plan_event (threading.Event)
  |
  +-- Background daemon threads
        Global SSE fan-out: reads global_queue, copies to per-client queues
        Idle cleanup: closes stale workstreams, cleans rate limiter buckets
```

Starlette handles all HTTP routing, CORS, and middleware. uvicorn runs
the ASGI application with async request handling. All API endpoints live
under the `/v1/` prefix via a Starlette `Mount`. An OpenAPI 3.1 spec is
generated from Pydantic v2 models and served at `/openapi.json`; Swagger
UI is available at `/docs`. SSE endpoints use `EventSourceResponse` from
`sse-starlette` with async generators that bridge sync `queue.Queue` via
`asyncio.get_running_loop().run_in_executor()`.

`ChatSession.send()` remains synchronous, running in daemon worker threads.
WebUI keeps `threading.Event` and `queue.Queue` primitives (unchanged from
the sync era). The `_global_fanout_thread` and `_idle_cleanup_thread` remain
as daemon threads since they interact with sync primitives. A lifespan
context manager handles startup/shutdown (health monitor, MCP client,
registry).

Each workstream's `WebUI` has:
- `_listeners` (per-client SSE queues, fan-out on `_enqueue()`)
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
- `_global_queue` (class variable, shared, for state broadcasts)

The SSE handlers bridge these sync queues to async via
`run_in_executor()`, polling `queue.Queue.get(timeout=1)` while
`sse-starlette` handles keepalive pings automatically.

### Workstream Threading (CLI)

```
Main thread                  Background workstream thread
+------------------+         +---------------------------+
| REPL input()     |         | session.send()            |
| /ws commands     |         | streams response          |
| active workstream|         | executes tools            |
| send() inline   |         | approve_tools() ->        |
+------------------+         |   _fg_event.wait() BLOCKS |
       |                     +---------------------------+
       |                                ^
       +-- /ws <N> switch ------------->|
       |   old.set_foreground(False)    |
       |   new.set_foreground(True)     |
       |   new.flush_buffer()           |
       +-- _fg_event.set() unblocks --->+
```

When a background workstream needs approval, its `WorkstreamTerminalUI`
calls `_fg_event.wait()`, which blocks the worker thread until the user
switches to that workstream. The `_bg_attention_notify` callback writes a
bell + status line to stderr to alert the user.

### Cluster Console

```
Monitoring (2 daemon threads)        Control + Proxy (async Starlette)
+------------------+                 +----------------------------+
| Node discovery   |                 | POST /v1/api/cluster/      |
| Service registry |                 |   workstreams/new          |
| every 60 seconds |                 |   → POST to target server  |
+------------------+                 +----------------------------+
| SSE manager      |                 | GET /node/{node_id}/       |
| asyncio loop     |                 |   → httpx.AsyncClient      |
| 1 task per node  |                 |     proxy to server_url    |
| /events/global   |                 | GET /node/{id}/v1/api/workstreams/{ws_id}/events |
| snapshot+deltas  |                 |   → SSE stream proxy                              |
+------------------+                 | POST /node/{id}/v1/api/workstreams/{ws_id}/send   |
                                     |   → forwarded to server                           |
                                     +----------------------------+
```

The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE
endpoint uses `EventSourceResponse` with the same listener queue pattern as
the main server. `ClusterCollector` runs two daemon threads: a discovery loop
that queries the service registry every 60 seconds, and an SSE manager that
runs a single asyncio event loop multiplexing persistent SSE connections to
all nodes via `GET /v1/api/events/global`. Each node delivers a full snapshot
on connect followed by real-time delta events — state changes, health
transitions, and aggregate metrics arrive sub-second instead of on a 15-second
poll cycle.

The console has two write-path capabilities:

1. **Workstream creation** — sends HTTP requests to target server nodes
   to create workstreams. Auto-selects the node with
   the most available capacity if no target is specified. When a `skill`
   field is present, the server resolves the skill BEFORE `mgr.create()`
   (applying the model override to the creation request) and snapshot-applies
   remaining settings (auto-approve, token budget, temperature, etc.) to the
   workstream config AFTER creation.

2. **Reverse proxy** — serves each node's server UI through the console port at
   `/node/{node_id}/`. Uses `httpx.AsyncClient` to proxy HTTP and SSE traffic.
   A JS shim is injected into the server's `app.js` to override `fetch()` and
   `EventSource()`, routing root-relative URLs through the proxy prefix. This
   eliminates the need for direct network access to individual server nodes.

The console also performs **version drift detection** — flagging when nodes
report different versions via the `/health` endpoint. The overview API includes
`version_drift` and `versions` fields; the dashboard shows a yellow warning
indicator when versions diverge.

Clicking a workstream row in the console opens the proxied server UI at
`/node/{node_id}/?ws_id=<id>` — the server's JS parses this on load and
auto-selects the workstream. See [docs/console.md](console.md) for the full
API reference.

---

## Conversation Compaction

When the prompt exceeds `auto_compact_pct` of the context window (default:
80%, configurable via `--auto-compact-pct`), `ChatSession` auto-compacts by
summarizing the entire conversation into a structured summary
(`_compact_messages`). The summary model call uses `compact_max_tokens`
(default: 32768, configurable via `--compact-max-tokens`). The summary
preserves:

- Decisions made (architecture, libraries, approaches)
- Files read, created, or modified
- Exact identifiers, paths, and code snippets
- Important tool results
- Open tasks
- User preferences

After compaction, `_read_files` is cleared to force re-reads before edits,
since file contents are no longer in the message history.

---

## Client SDK

> See also: [SDK Architecture diagram](diagrams/png/13-sdk-architecture.png) | [SDK Documentation](sdk.md)

The `turnstone/sdk/` package provides typed HTTP clients for programmatic access
to both the server and console APIs. It wraps REST endpoints with methods that
return Pydantic models, and SSE endpoints with async/sync iterators that yield
typed event dataclasses.

**Two client pairs** (sync + async):

- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming)
- `TurnstoneConsole` / `AsyncTurnstoneConsole` — console API (cluster overview, nodes, workstreams)

**Design**: async-first with thin sync wrappers. `_BaseClient` provides httpx
setup, auth headers, `_request()` (REST) and `_stream_sse()` (SSE). Sync
clients delegate through `_SyncRunner` which maintains a persistent background
event loop on a daemon thread.

**Event types**: 38 standalone dataclasses in `events.py` with a type-registry
dispatch (`from_json()` on each event). Events are decoupled from server
internals — the SDK parses SSE frames directly from the `/v1/api/events`
streams.

**TypeScript SDK**: `sdk/typescript/` — separate npm package with the same API
surface. Zero browser dependencies, SSE via `fetch` + `ReadableStream` parsing.

```python
# Python quick start
from turnstone.sdk import TurnstoneServer

with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
    ws = client.create_workstream(name="demo")
    result = client.send_and_wait("Hello!", ws.ws_id)
    print(result.content)
```

---

## Channel Integrations

> See also: [Channel Integrations guide](channels.md)

The `turnstone-channel` gateway connects external messaging platforms
(Discord and Slack today, with an adapter protocol for future platforms) to
the turnstone cluster via HTTP. Each
platform adapter implements the `ChannelAdapter` protocol and translates
between platform-native events and turnstone server API calls.

The `ChannelRouter` manages bidirectional routing: it maps platform
channel/thread IDs to turnstone workstream IDs, handles workstream
creation and stale-route recovery, and resolves platform users to
turnstone identities via the `channel_users` table. When an evicted
workstream is reactivated, the router uses atomic resume via the
`resume_ws` field on the workstream creation request — the server resumes
the old workstream's conversation during creation in a single HTTP
request, eliminating ordering fragility.

Discord and Slack adapters ship today. See [channels.md](channels.md) for
setup instructions, configuration reference, and the adapter development
guide.

### Notification Subsystem

The `notify` tool enables the LLM to send notifications to users or
channels directly. The server calls the channel gateway
directly over HTTP for lower latency: `_exec_notify()` queries the
`services` database table for healthy channel gateways (heartbeat within
120 seconds), authenticates with a service JWT (`aud: turnstone-channel`),
and POSTs to `POST /v1/api/notify` on the first healthy gateway. The
payload includes the originating `ws_id` for reply routing. The gateway
validates the JWT, resolves the target (username lookup via
`channel_users` or direct `channel_type`+`channel_id`), and delegates to
`ChannelAdapter.send_notification()` which sends the message and tracks
the outgoing message ID → `(ws_id, target_user_id)` mapping. Delivery
retries up to 3 times with backoff, re-querying the service registry on
each attempt. See [Notification Flow diagram](diagrams/png/17-notify-flow.png).

**Bidirectional replies:** When a user replies to a notification DM, the
channel adapter (Discord or Slack) looks up the originating `ws_id` from the
tracked message ID, verifies the replying user matches the notification
recipient, and routes the reply to the workstream via `router.send_message()`.
The workstream's response is forwarded back to the DM via a temporary entry
in `_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
itself tracked for further replies, enabling multi-turn DM conversations
without requiring the user to open the web UI. Tracking entries are capped
at 100 (FIFO eviction) and cleaned up on workstream close.

---

## Governance

> See also: [Governance documentation](governance.md) | [Governance Architecture diagram](diagrams/19-governance-architecture.puml)

Turnstone governance extends the Phase 1 auth system with role-based access
control (RBAC), tool execution policies, skills, usage tracking,
and audit logging. The permission model has two layers: legacy scopes
(`read`, `write`, `approve`) checked by `AuthMiddleware`, and 15 granular
permissions checked per-endpoint by `require_permission()`. Three built-in
roles (admin, operator, viewer) are seeded by migration 008; custom roles
can be created with any permission subset. JWTs carry both `scopes` and
`permissions` claims for backward compatibility.

Tool policies use glob pattern matching (`fnmatch`) with priority-ordered
first-match-wins evaluation to control tool execution (allow/deny/ask).
Skills provide reusable system messages with `{{variable}}` substitution
plus session configuration (model, temperature, auto-approve, token budget,
etc.). Usage events are recorded per-LLM-request for token accounting.
An append-only audit log captures all admin mutations.

Skills are snapshot-applied once at workstream creation — not a live binding.
The `prompt_templates` table (which stores skills) supports auto-versioning,
and workstreams record which skill and version spawned them. Token budget
enforcement tracks consumption in `session.send()` with 80% warning and
100% approval gate via the `__budget_override__` synthetic tool name.

The console admin panel exposes these capabilities as 18 permission-gated
tabs: Users, API Tokens, Channels, Schedules, Watches, Roles, Policies,
Prompts, Judge, Skills, MCP Servers, Usage, Audit, Memories, Models, Nodes,
Settings, and TLS.
Both Python and TypeScript SDKs expose governance methods on the console
client.

## Intent Validation

> See also: [Intent Validation guide](judge.md) | [Judge Architecture diagram](diagrams/png/22-judge-architecture.png)

Intent validation provides advisory risk assessments for tool calls that
require human approval. The system runs a two-tier evaluation pipeline
implemented in `turnstone/core/judge.py`:

1. **Heuristic tier** (synchronous, sub-millisecond) -- A priority-ordered
   rule table using fnmatch tool patterns and regex argument patterns. Four
   severity levels: critical (deny), high (review), medium (review), low
   (approve). First match wins. The heuristic verdict is attached to the
   `approve_request` SSE event immediately.

2. **LLM judge tier** (asynchronous, daemon thread) -- A multi-turn evaluation
   where the judge LLM receives conversation context and tool call details,
   optionally uses `read_file`/`list_directory` to gather evidence (with
   security-hardened path blocking), and produces a structured JSON verdict.
   If the LLM verdict has higher confidence than the heuristic, it replaces
   it via an `intent_verdict` SSE event.

The judge is session-scoped (`IntentJudge`), lazy-initialized on first
approval, and configured via the `[judge]` config section or `--judge` CLI
flags. By default it uses self-consistency (same model), but supports
cross-model and cross-provider configurations. Sub-agents (plan, task)
are exempt. All verdicts are persisted to the `intent_verdicts` table
(migration 012) with the user's final decision, enabling future calibration.
The console exposes `GET /v1/api/admin/verdicts` for audit queries
(requires `admin.judge` permission).