turnstone/docs/architecture.md

# Turnstone Architecture

Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or
Anthropic's native Messages API via pluggable provider adapters, and gives the
model 17 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, planning, and executing code.

The core design principle is a **UI-agnostic engine with pluggable frontends**.
The engine (`ChatSession`) drives the conversation loop -- streaming, tool
dispatch, retry, compaction -- while every user-facing interaction is delegated
through the `SessionUI` protocol. Any frontend implements that protocol and
plugs in.

## Entry Points

| Command | Module | Frontend | Purpose |
|---------|--------|----------|---------|
| `turnstone` | `turnstone.cli` | `TerminalUI` | Interactive terminal REPL |
| `turnstone-server` | `turnstone.server` | `WebUI` | Browser-based chat (HTTP + SSE) |
| `turnstone-bridge` | `turnstone.mq.bridge` | Bridge | Message queue ↔ HTTP API bridge |
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) via Redis MQ |
| `turnstone-admin` | `turnstone.core.admin_cli` | — | Offline user and API token management |

---

## Module Map

```
turnstone/
  cli.py              Terminal frontend (TerminalUI, WorkstreamTerminalUI, REPL)
  server.py           Web frontend (WebUI, HTTP handler, static-file serving)
  eval.py             Evaluation harness (HeadlessSession, scoring, prompt optimization)
  core/
    session.py        ChatSession engine, SessionUI protocol, tool dispatch
    providers/        LLM provider adapters (pluggable backend layer)
      _protocol.py    LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult
      _openai.py      OpenAIProvider — OpenAI, vLLM, llama.cpp, any compatible API
      _anthropic.py   AnthropicProvider — Anthropic Messages API, native streaming, thinking
      __init__.py     create_provider() + create_client() factory functions
    workstream.py     Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
    tools.py          Tool schema loader (JSON -> OpenAI function-calling format)
    mcp_client.py     MCPClientManager — MCP server connections, tool discovery, dynamic refresh, async-sync bridge
    tool_search.py    Dynamic tool search — BM25 index, session-scoped tool visibility
    watch.py          WatchRunner daemon — periodic command polling, condition DSL, result dispatch
    judge.py          Intent validation — heuristic rules + LLM judge, advisory verdicts
    model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
    memory.py         Persistence facade + structured memory API (delegates to storage backend)
    config.py         Config file loader (config.toml), apply_config(), warn_migrated_settings()
    config_store.py   ConfigStore — database-backed settings with in-memory cache, thread-safe get/set
    settings_registry.py  SettingDef catalog (~40 settings), validation, type coercion, serialization
    storage/          Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
    metrics.py        Prometheus-compatible metrics collector (MetricsCollector)
    healthcheck.py    BackendHealthMonitor — periodic probe + circuit breaker
    ratelimit.py      Per-IP token-bucket rate limiter (RateLimiter, TokenBucket)
    edit.py           File edit utilities (find_occurrences, pick_nearest)
    safety.py         Command safety validation (blocked patterns, sanitization)
    sandbox.py        Math code sandboxing (AST validation, subprocess execution)
    web.py            Web utilities (HTML stripping, SSRF prevention)
  api/
    schemas.py        Shared Pydantic v2 models (auth, errors, WorkstreamState)
    server_schemas.py Server endpoint request/response models
    console_schemas.py Console endpoint request/response models
    openapi.py        OpenAPI 3.1 spec builder
    server_spec.py    Server endpoint catalog → build_server_spec()
    console_spec.py   Console endpoint catalog → build_console_spec()
    docs.py           /openapi.json + /docs (Swagger UI) handler factories
  sdk/
    server.py         AsyncTurnstoneServer + TurnstoneServer (HTTP client)
    console.py        AsyncTurnstoneConsole + TurnstoneConsole (HTTP client)
    events.py         27 SSE event dataclasses with type registry
    _base.py          Shared httpx async client, auth, error handling
    _sync.py          Background event loop for sync wrappers
    _types.py         TurnResult + TurnstoneAPIError
  mq/
    protocol.py       Inbound/outbound message dataclasses (JSON serialization)
    broker.py         Abstract MessageBroker protocol + RedisBroker
    bridge.py         Bridge service (queue ↔ turnstone-server HTTP API)
    client.py         TurnstoneClient library + TurnResult for MQ-based access
  console/
    collector.py      ClusterCollector — aggregates state from all nodes via Redis + HTTP
    scheduler.py      TaskScheduler — background cron/at scheduler, dispatches via MQ
    server.py         Cluster dashboard HTTP server + SSE + CLI entry point
    static/           Cluster dashboard web UI (page-specific HTML, CSS, JS)
  channels/
    cli.py            Unified channel gateway entry point (turnstone-channel)
    _protocol.py      ChannelAdapter protocol, ChannelEvent dataclass
    _routing.py       ChannelRouter — channel/thread ↔ workstream mapping via MQ
    _config.py        Base ChannelConfig dataclass
    discord/          Discord adapter (bot, cog, views, streaming, config)
  shared_static/      Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
    katex-0.16.38/    Vendored KaTeX math rendering library (MIT, woff2 fonts)
  ui/
    colors.py         ANSI color constants with NO_COLOR support
    markdown.py       Streaming terminal markdown renderer (line-buffered)
    spinner.py        Braille character spinner (daemon thread)
    static/
      index.html      Single-page app shell (links to CSS and JS)
      style.css       Page-specific UI styles (dashboard, markdown elements, approval blocks)
      renderer.js     Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
      app.js          Page-specific client-side JavaScript (SSE, workstreams, tool approval)
  tools/
    *.json            15 tool schemas (OpenAI function-calling format + turnstone metadata)
```

Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.

---

## Core Loop

> See also: [Conversation Turn diagram](diagrams/png/04-conversation-turn.png)

A user message flows through the system as follows:

```
 User input
     |
     v
 ChatSession.send(user_input)
     |
     v
 _full_messages()  ------------>  system_messages + self.messages
     |
     v
 _emit_state("thinking")
     |
     v
 _create_stream_with_retry()  ---->  provider.create_streaming(client, model, messages, ...)
     |                                  up to 3 retries (4 total attempts), exponential backoff
     v
 _stream_response(stream)  -------->  dispatch tokens to UI:
     |                                  on_reasoning_token() / on_content_token()
     |                                  accumulate tool_calls from deltas
     |                                  track finish_reason
     |                                  _check_cancelled() per chunk (cooperative cancel)
     v
 finish_reason check:
     +--- "length"  --> warn, discard partial tool_calls
     +--- "content_filter" --> warn
     v
 tool_calls present?
     |
     +--- No ---> _print_status_line() -> _emit_state("idle") -> return
     |
     +--- Yes --> _emit_state("running")
                    |
                    v
                  _execute_tools(tool_calls)  <--- three-phase pipeline (see below)
                    |
                    v
                  append tool results to self.messages
                    |
                    v
                  loop back to _full_messages()
```

### Tool Execution Pipeline

> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)

Tool execution is a three-phase process:

```
Phase 1: PREPARE (serial)
  For each tool_call:
    _prepare_tool(tc)
      -> parse JSON arguments (with regex fallback for malformed JSON)
      -> dispatch to _prepare_{tool_name}(call_id, args)
      -> validate inputs, build preview text
      -> return item dict with: header, preview, needs_approval, execute fn

Phase 2: APPROVE (serial, blocking)
  _emit_state("attention")
  ui.approve_tools(items)
    -> display all headers and previews
    -> if any need approval and not auto_approve: prompt user
    -> return (approved, feedback)
  _emit_state("running")

Phase 3: EXECUTE (parallel)
  _check_cancelled()  <-- cancellation checkpoint before execution starts
  if len(items) == 1:
    run_one(items[0])
  else:
    ThreadPoolExecutor(max_workers=4).map(run_one, items)
  Bash tool streams stdout line-by-line via ui.on_tool_output_chunk(call_id, line)
    (cancel_event also checked per line — kills process group on cancel)
  Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
  call_id links tool_info items → streaming chunks → final result
  For plan tool: post-execution gate via ui.on_plan_review()
```

### State Transitions

The engine emits state changes via `_emit_state()` which calls
`ui.on_state_change(state)`. Frontends use these to update indicators
(spinner, tab badges, status line).

```
  send() called
      |
      v
  "thinking"  --->  streaming response
      |
      v
  "running"   --->  tool execution
      |
      v
  "attention"  --->  waiting for user approval / plan review
      |
      v
  "running"   --->  executing approved tools
      |
      v
  "idle"       --->  no more tool calls, turn complete
      |
  (or "error"  --->  exception or KeyboardInterrupt)

  cancel() may be called from any state. It sets a cooperative flag
  checked at each streaming chunk, before tool execution, and inside
  bash commands. The session transitions to "idle" with partial
  content preserved, emitting on_info("[Generation cancelled]").
```

---

## SessionUI Protocol

> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)

Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
methods. Every frontend must implement all of them.

```python
class SessionUI(Protocol):
    def on_thinking_start(self) -> None: ...
    def on_thinking_stop(self) -> None: ...
    def on_reasoning_token(self, text: str) -> None: ...
    def on_content_token(self, text: str) -> None: ...
    def on_stream_end(self) -> None: ...
    def approve_tools(self, items: list[dict]) -> tuple[bool, str | None]: ...
    def on_tool_result(self, call_id: str, name: str, output: str) -> None: ...
    def on_tool_output_chunk(self, call_id: str, chunk: str) -> None: ...
    def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
    def on_plan_review(self, content: str) -> str: ...
    def on_info(self, message: str) -> None: ...
    def on_error(self, message: str) -> None: ...
    def on_state_change(self, state: str) -> None: ...
    def on_rename(self, name: str) -> None: ...  # propagate alias to tab/UI label
```

`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.

### Three Implementations

| Class | Module | Notes |
|-------|--------|-------|
| `TerminalUI` | `turnstone.cli` | ANSI colors, `MarkdownRenderer`, `Spinner`, readline-based `input()` for approval |
| `WebUI` | `turnstone.server` | SSE event queue per workstream, `threading.Event` for blocking on approval/plan |
| `NullUI` | `turnstone.eval` | Discards all output; `approve_tools` always returns `(True, None)` |

### WorkstreamTerminalUI

`WorkstreamTerminalUI` (in `turnstone.cli`) extends `TerminalUI` with workstream
awareness:

- **Output buffering**: When in background (`is_foreground` is False), tokens
  are appended to `_output_buffer` instead of written to stdout. When the user
  switches to this workstream, `flush_buffer()` replays them.

- **Approval blocking**: `approve_tools()` and `on_plan_review()` call
  `_fg_event.wait()` when in background, blocking the worker thread until the
  workstream is foregrounded. This ensures the user sees the approval prompt
  in the correct context.

- **Foreground/background toggle**: `set_foreground(bool)` sets or clears
  `_fg_event` (a `threading.Event`). The manager calls this during `/ws <N>`
  switches.

---

## Workstream Architecture

Workstreams are parallel, independent chat sessions. Each has its own
`ChatSession`, `SessionUI`, message history, and worker thread.

### WorkstreamState

> See also: [Workstream States diagram](diagrams/png/09-workstream-states.png)

Defined in `turnstone.core.workstream.WorkstreamState` (5 states):

```
IDLE       waiting for user input
THINKING   LLM is streaming a response
RUNNING    tools are executing
ATTENTION  blocked on user approval or plan review
ERROR      last operation failed
```

### Data Model

```python
@dataclass
class Workstream:
    id: str                              # uuid hex, 8 chars
    name: str                            # user-visible label
    state: WorkstreamState               # current state
    session: ChatSession | None          # the conversation engine
    ui: SessionUI | None                 # frontend adapter
    worker_thread: threading.Thread | None
    error_message: str
    last_active: float                   # time.monotonic() timestamp, updated on every state change
    _lock: threading.Lock                # per-workstream state lock
```

### WorkstreamManager

```python
class WorkstreamManager:
    MAX_WORKSTREAMS = 10

    def __init__(self, session_factory: Callable[[SessionUI], ChatSession]): ...
    def create(self, name="", ui_factory=None) -> Workstream: ...
    def close(self, ws_id: str) -> bool: ...
    def close_idle(self, max_age_seconds: float) -> list[str]: ...  # auto-close stale IDLE workstreams
    def get(self, ws_id: str) -> Workstream | None: ...
    def get_active(self) -> Workstream | None: ...
    def list_all(self) -> list[Workstream]: ...
    def switch(self, ws_id: str) -> Workstream | None: ...
    def switch_by_index(self, index: int) -> Workstream | None: ...
    def set_state(self, ws_id, state, error_msg=""): ...  # updates last_active
```

The `session_factory` pattern decouples session creation from configuration.
The factory captures shared config (client, model, temperature, etc.) and
accepts only a `SessionUI`, so the manager can create sessions without knowing
API details.

### Idle Workstream Lifecycle

The web server runs a background `_idle_cleanup_thread` (daemon) that calls
`WorkstreamManager.close_idle()` periodically (every `timeout / 4`, max 5 min).
Any IDLE workstream whose `last_active` is older than the configured timeout is
closed; non-IDLE workstreams (THINKING, RUNNING, ATTENTION, ERROR) are never
touched. The last workstream is always preserved even if expired. On close, a
`ws_closed` event is broadcast on the global SSE channel so browser clients
remove the tab immediately. Controlled by `--workstream-idle-timeout` (default:
120 minutes, 0 = disable).

**Workstream eviction at capacity:** When `WorkstreamManager.create()` would
exceed `max_workstreams` (configurable via `[server].max_workstreams`, default
10), the oldest IDLE workstream is automatically evicted to make room. The
`turnstone_workstreams_evicted_total` counter is incremented on each eviction.
If no IDLE workstream is available the create request fails as before.

### CLI Workstreams

- `/ws list` -- show all workstreams with state indicators
- `/ws new [name]` -- create a new workstream and switch to it
- `/ws <N>` -- switch to workstream by 1-based index
- `/ws close [N]` -- close a workstream
- `/ws rename <name>` -- rename the active workstream

Background notifications: when a background workstream enters `ATTENTION`
state, `_bg_attention_notify` writes an ANSI escape sequence to stderr
(overwrites the line above the prompt) with the workstream name.

Status line: `_print_ws_status_line()` shows a compact status of all
non-idle background workstreams above the input prompt.

### Web Workstreams

- **Tab bar**: Each workstream renders as a tab with a colored state indicator
  (CSS `@keyframes pulse` animation per state).
- **Per-tab SSE**: `connectContentSSE(wsId)` opens
  `/v1/api/events?ws_id=<id>` for the active tab's event stream.
- **Global SSE**: `connectGlobalSSE()` opens `/v1/api/events/global` which
  receives `ws_state` broadcasts from all workstreams, used to update tab
  indicators without switching.
- **New tab / close**: POST `/v1/api/workstreams/new`, POST `/v1/api/workstreams/close`.

### Thread Safety

- `WorkstreamManager._lock`: guards `_workstreams` dict and `_order` list on
  all create/close/switch/list operations.
- `Workstream._lock`: guards per-workstream state mutations in `set_state()`.
- `WorkstreamTerminalUI._print_lock`: guards `_output_buffer` access.
- `WorkstreamTerminalUI._fg_event`: `threading.Event` that blocks background
  approval until the workstream is foregrounded.

---

## Tool System

### Schema Format

Each tool is a JSON file in `turnstone/tools/`. The file contains an OpenAI
function-calling schema (`name`, `description`, `parameters`) plus optional
turnstone metadata keys:

| Metadata Key | Type | Meaning |
|-------------|------|---------|
| `agent` | `bool` | Include this tool when running as a plan/task sub-agent |
| `task_agent` | `bool` | Include this tool when running as a task sub-agent |
| `auto_approve` | `bool` | Tool is read-only; skip user approval |
| `primary_key` | `str` | Fallback argument name for bare-string JSON recovery |

Example (`read_file.json`):

```json
{
  "name": "read_file",
  "description": "Read the contents of a file. ...",
  "parameters": {
    "type": "object",
    "properties": {
      "path": { "type": "string", "description": "..." },
      "offset": { "type": "integer", "description": "..." },
      "limit": { "type": "integer", "description": "..." }
    },
    "required": ["path"]
  },
  "agent": true,
  "task_agent": true,
  "auto_approve": true,
  "primary_key": "path"
}
```

At import time, `turnstone.core.tools._load_tools()` strips the metadata keys
from each schema and builds:

- `TOOLS` -- list of `{"type": "function", "function": {...}}` dicts for the API
- `AGENT_TOOLS` -- subset with `agent: true`
- `TASK_AGENT_TOOLS` -- subset with `task_agent: true`
- `AGENT_AUTO_TOOLS` / `TASK_AUTO_TOOLS` -- sets of tool names with `auto_approve: true`
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
- `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init

### 13 Tools by Category

**Read-only (auto-approve)**:
- `read_file` -- read file contents with optional offset/limit
- `search` -- ripgrep-based codebase search
- `man` -- read man pages
- `recall` -- search conversation history

**Write (requires approval)**:
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
- `write_file` -- create or overwrite a file
- `edit_file` -- string replacement in an existing file (requires prior `read_file`)
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models)

**Agent (delegated sub-sessions)**:
- `task` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan` -- explore codebase and write a structured plan (`AGENT_TOOLS`)

**Memory (structured persistent store)**:
- `memory` -- save, search, delete, or list memories (typed and scoped)

### Prepare / Execute Pattern

Every tool has a `_prepare_{name}` method and a corresponding `_exec_{name}`
method on `ChatSession`:

```
_prepare_bash(call_id, args)   -> item dict with execute=self._exec_bash
_prepare_read_file(call_id, args) -> item dict with execute=self._exec_read_file
...
```

The prepare method validates inputs and builds the preview. The item dict
carries the validated data and a reference to the execute function. This
separation allows the UI to show previews before any side effects occur.

### Agent Tools

`task` and `plan` invoke `_run_agent()`, which runs a multi-turn loop with
a subset of tools and its own system prompt. The sub-agent runs
independently, then returns the final content as the tool result.

- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
  to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
  don't collide. On repeat invocations the prior `plan` tool call and its result
  are forwarded from `self.messages` so the agent refines the existing plan rather
  than starting over. Planning instructions are injected as a developer message
  prepended to the agent's conversation.
- **Turn limit**: controlled by `agent_max_turns` (default: `-1`, unlimited).
  When a limit is set and reached, the agent is forced to synthesize a final
  response without tools. When unlimited, the loop only exits when the model
  stops calling tools or hits `finish_reason: "length"`.
- **Retry**: each API call in the agent loop uses the same retry+backoff logic
  as the main `_create_stream_with_retry()`.
- **Finish reason handling**: `finish_reason: "length"` stops the agent early
  and returns whatever content was generated. `finish_reason: "content_filter"`
  returns a placeholder.

### MCP Tool Integration

`MCPClientManager` (`turnstone/core/mcp_client.py`) connects to external MCP servers
and exposes their tools alongside built-in tools. The MCP SDK is fully async; turnstone
bridges this with a background asyncio event loop in a daemon thread.

**Configuration sources:** MCP servers can be defined in config files (TOML/JSON)
or in the database via the admin UI. Database-backed definitions are managed
through the console admin panel's MCP Servers tab and stored in the
`mcp_servers` table. On startup, `load_mcp_config(storage=)` uses
first-match-wins priority: DB rows (if any enabled) take precedence over
config files. The console can trigger a cluster-wide reload (`POST
/_internal/mcp-reload`) that causes each node to call `reconcile_sync()`,
which diffs the running MCP connections against the current DB state and
adds, removes, or reconnects servers as needed.

**Lifecycle:**
1. `create_mcp_client()` reads server configs from TOML/JSON and database
2. `MCPClientManager.start()` launches the background event loop thread
3. `_connect_all()` connects to each server (stdio subprocess or HTTP), runs
   `initialize()` + `list_tools()`, converts schemas to OpenAI format, detects
   `tools.listChanged` capability for push notification support
4. `ChatSession.__init__` receives the manager, builds `self._tools` (built-in + MCP),
   and registers a listener callback for tool-change notifications
5. `_prepare_tool()` routes MCP tools to `_prepare_mcp_tool()` / `_exec_mcp_tool()`
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
   via `asyncio.run_coroutine_threadsafe()`

**Tool refresh:** Three mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
  the registered `message_handler` triggers immediate single-server refresh.
- **Periodic:** Servers without push support are polled on a staggered interval
  (default 4 h, configurable via `[mcp] refresh_interval` or `--mcp-refresh-interval`).
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
  (also attempts reconnection for disconnected servers).

When tools change, `_rebuild_tools()` creates new `_tools`/`_tool_map` objects
(copy-on-write for thread safety) and notifies listener callbacks. Each `ChatSession`
rebuilds its merged tool lists and reconstructs `ToolSearchManager` (preserving
expanded tools).

**Tool naming:** `mcp__{server}__{tool}` — double underscore delimiter, validated
at connection time (server names with `__` are rejected).

**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
rather than crashing the session.

### Provider Adapter Layer

> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)

`ChatSession` is provider-agnostic — it delegates all LLM communication to an
`LLMProvider` protocol (`turnstone/core/providers/_protocol.py`). Internally,
messages use an OpenAI-like format; each provider translates at the API boundary.

```
ChatSession
    |
    v
LLMProvider (protocol)
    |
    +--- OpenAIProvider  --- OpenAI, vLLM, llama.cpp, any /v1/chat/completions API
    +--- AnthropicProvider --- Anthropic Messages API (native streaming, thinking)
```

**Protocol methods:**

| Method | Purpose |
|--------|---------|
| `create_streaming()` | Streaming request, yields normalized `StreamChunk` objects |
| `create_completion()` | Non-streaming request, returns `CompletionResult` |
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |

**Normalized data types:**

| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens` |

**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
already in OpenAI format), including multi-part content blocks (text + images)
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
For search models, injects `web_search_options` and removes the `web_search`
function tool (the model always searches). Citations from `url_citation`
annotations are formatted as footnotes. Unknown models (local servers) get
permissive defaults with `supports_vision=False` and use Tavily for web search.

**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
Anthropic content blocks, maps `system`/`developer` roles to the `system`
parameter, groups consecutive `tool` result messages into user-role content
blocks (converting `image_url` parts to Anthropic's `image` source format),
and translates tool schemas from OpenAI function-calling format to
Anthropic's `input_schema` format. Supports both manual and adaptive thinking
modes, with effort parameter support for models like Claude Opus 4.6 and
Sonnet 4.6. Replaces the `web_search` function tool with Anthropic's native
`web_search_20250305` server-side tool — Claude decides when to search, the
API executes it, and results stream back as `server_tool_use` /
`web_search_tool_result` content blocks (emitted as `info_delta` for UI
display). The `anthropic` SDK is imported lazily so it remains an optional
dependency (`pip install turnstone[anthropic]`).

**Factory functions** (`__init__.py`): `create_provider(name)` returns a
singleton provider instance (thread-safe). `create_client(name, base_url,
api_key)` creates the appropriate SDK client.

### Multi-Model Registry

`ModelRegistry` (`turnstone/core/model_registry.py`) manages named model
configurations so workstreams can use different LLM backends.

**Config format:**
```toml
[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
# provider defaults to "openai"

[models.claude]
provider = "anthropic"
api_key = "sk-ant-..."
model = "claude-opus-4-6"
context_window = 200000

[models.openai]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "gpt-5"
context_window = 400000

[model]
default = "local"
fallback = ["claude", "openai"]
agent_model = "claude"
```

Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
(default: `"openai"`). Supported values: `"openai"` and `"anthropic"`.
An optional `[models.*.capabilities]` sub-table overrides per-model
`ModelCapabilities` flags (useful for local models whose capabilities
cannot be detected programmatically):

```toml
[models.qwen-vl]
base_url = "http://localhost:8000/v1"
model = "qwen-3.5-vl"

[models.qwen-vl.capabilities]
supports_vision = true
```

**Lifecycle:**
1. `load_model_registry()` reads `[models.*]` sections from config.toml and
   builds a `"default"` entry from CLI `--base-url`/`--model`/`--api-key` args
2. The registry is passed to the session factory closure in both `cli.py` and
   `server.py`; each workstream resolves its model on creation
3. `ModelRegistry.get_client()` lazily creates SDK client instances via
   `create_client()` — `OpenAI` for the openai provider, `Anthropic` for
   the anthropic provider (thread-safe via `_client_lock`)
4. `ModelRegistry.get_provider()` lazily creates `LLMProvider` instances via
   `create_provider()` (also cached and thread-safe)
5. `/model` command shows available models; `/model <alias>` switches the
   active workstream's client, model, and context window
6. `_create_stream_with_retry()` tries the primary model, then each fallback
   alias in order if the primary is unreachable
7. `_run_agent()` resolves `registry.agent_model` (if set) for plan/task
   sub-agents, allowing a cheaper model for autonomous loops

**Per-workstream selection:** `POST /v1/api/workstreams/new` accepts an optional
`"model"` field. The bridge `CreateWorkstreamMessage` carries the same field
through the MQ protocol, along with `ws_template` (workstream template name)
which can override the model before workstream creation.

### Tool Output Truncation

Tool execution results (bash, read_file, search, math, man) are truncated by
`_truncate_output()` when they exceed `tool_truncation` characters. Truncation
preserves the first half and last half of the output, with a message in
between:

```
... [N chars truncated — output exceeded LIMIT char limit] ...
```

The default limit is 50% of the context window in characters (computed as
`context_window * chars_per_token * 0.5`). For a 131K context window this is
~262K characters. Override with `--tool-truncation <chars>`.

This truncation message is visible to the model, so it knows output was cut.

---

## Persistence

### Storage Architecture

Persistence is managed by the `turnstone.core.storage` package — a pluggable
backend behind a `StorageBackend` protocol. The `memory.py` facade provides
backward-compatible module-level functions that delegate to the active backend.

```
session.py / server.py / cli.py
        ↓
    memory.py  (facade — silent-failure wrappers)
        ↓
    storage._registry  (singleton factory)
        ↓
  ┌─────────────┐    ┌──────────────────┐
  │ SQLiteBackend │    │ PostgreSQLBackend │
  │ (FTS5 search) │    │ (tsvector/ILIKE)  │
  └─────────────┘    └──────────────────┘
        ↓                     ↓
    storage._schema  (SQLAlchemy Core tables — single source of truth)
        ↓
    storage._migrate  (programmatic Alembic)
```

**SQLite** is the default (zero-config, single file at `.turnstone.db`).
**PostgreSQL** is the production backend (connection pooling, `tsvector`
full-text search). Select via `[database]` in `config.toml`, CLI flags, or
environment variables (`TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`).

Schema migrations are managed by Alembic and run automatically on startup.
Existing SQLite databases created before the migration system are auto-stamped
at the baseline revision.

### Tables

```sql
memories
  key      TEXT PRIMARY KEY
  value    TEXT NOT NULL
  created  TEXT NOT NULL
  updated  TEXT NOT NULL

workstreams
  ws_id       TEXT PRIMARY KEY
  node_id     TEXT NOT NULL
  name        TEXT NOT NULL
  state       TEXT NOT NULL DEFAULT 'idle'
  alias       TEXT UNIQUE            -- user-assigned short name (nullable)
  title       TEXT                   -- LLM-generated title (nullable)
  created     TEXT NOT NULL
  updated     TEXT NOT NULL          -- bumped on every save_message()

conversations
  id            INTEGER PRIMARY KEY AUTOINCREMENT
  ws_id         TEXT NOT NULL
  timestamp     TEXT NOT NULL
  role          TEXT NOT NULL        -- user | assistant | tool_call | tool_result
  content       TEXT
  tool_name     TEXT
  tool_args     TEXT
  tool_call_id  TEXT                 -- links tool_call ↔ tool_result for resume
  provider_data TEXT                 -- raw provider content (e.g. Anthropic encrypted)

workstream_config
  ws_id       TEXT NOT NULL          -- composite PK with key
  key         TEXT NOT NULL
  value       TEXT

conversations_fts                    -- SQLite FTS5 virtual table (optional)
  content     (content=conversations, content_rowid=id)
```

Table definitions live in `storage/_schema.py` (SQLAlchemy Core `Table` objects)
and are the single source of truth for both backends and Alembic migrations.

### StorageBackend Protocol

| Method | Purpose |
|--------|---------|
| `register_workstream(ws_id, node_id, name, state)` | Create a workstreams row (no-op if exists) |
| `save_message(ws_id, role, content, ...)` | Log a message to conversations |
| `load_messages(ws_id)` | Reconstruct OpenAI message format from DB rows |
| `list_workstreams_with_history(limit)` | List workstreams with >=1 message, ordered by updated DESC |
| `delete_workstream(ws_id)` | Delete workstream and cascade conversations + config |
| `prune_workstreams(retention_days)` | Remove empty workstreams and old unnamed workstreams |
| `resolve_workstream(alias_or_id)` | Resolve alias, exact id, or id prefix to full ws_id |
| `save_workstream_config(ws_id, config)` | Persist workstream configuration key/value pairs |
| `load_workstream_config(ws_id)` | Retrieve workstream configuration |
| `set_workstream_alias(ws_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_workstream_display_name(ws_id)` | Return alias if set, else title, else None |
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
| `update_workstream_name(ws_id, name)` | Update workstream display name |
| `list_workstreams(node_id, limit)` | List workstreams, optionally by node |
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
| `search_history(query, limit)` | Full-text search (FTS5 on SQLite, tsvector on PostgreSQL) |
| `search_history_recent(limit)` | Return most recent messages |
| `close()` | Release resources (connection pool, engine) |

### Database Configuration

```toml
[database]
backend = "sqlite"                  # "sqlite" | "postgresql"
path = ".turnstone.db"              # SQLite file path
url = ""                            # PostgreSQL connection URL
pool_size = 5                       # PostgreSQL connection pool size
```

Environment variables: `TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`, `TURNSTONE_DB_PATH`.

### Persistence and Resume

`ws_id` is the sole persistent identity for both routing and conversation
history. There is no separate `session_id` — the `workstreams` table holds
alias, title, and state alongside the routing fields (`node_id`, `name`).
Messages are saved to `conversations` (keyed by `ws_id`) as they happen
via `save_message()`. Workstream state changes are tracked via
`update_workstream_state()`.

**Auto-titling:** After the first complete exchange (user message + assistant
response), a background thread calls the LLM with a title-generation prompt
(`reasoning_effort: "low"`, `max_completion_tokens: 200`). The generated
title (3-8 words) is stored in `workstreams.title`.

**Resume flow:** `ChatSession.resume(ws_id)` calls `load_messages()` which
reconstructs the OpenAI message format from database rows:

- `user` and `assistant` rows map directly
- Consecutive `tool_call` rows are grouped into one assistant message's
  `tool_calls` array, paired with subsequent `tool_result` rows via
  `tool_call_id` (or positional matching for legacy data)
- **Interrupted conversation repair:** If the last assistant message has
  `tool_calls` but fewer tool results than expected (conversation was
  interrupted mid-execution), the incomplete turn is stripped so the
  LLM can re-generate cleanly
- The `ChatSession` adopts the resumed `_ws_id`, so new messages continue
  in the same workstream

**Config persistence:** LLM-affecting parameters (`temperature`,
`reasoning_effort`, `max_tokens`, `instructions`, `creative_mode`) are
persisted to the `workstream_config` table on creation and whenever changed
via slash commands. `resume()` restores these values so resumed workstreams
behave identically to the original.

**`/clear` vs `/new`:** `/clear` wipes in-memory context but preserves
messages in the database for future resume. `/new` starts a fresh workstream
(new `_ws_id`), leaving the old workstream resumable.

**Resolution:** `resolve_workstream()` accepts aliases, exact workstream IDs,
or ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.

**Workstream listing:** `list_workstreams_with_history()` only returns
workstreams that have at least one saved message (`WHERE EXISTS` on
`conversations`). Workstreams registered but never used (e.g., from process
startup) are invisible until a message is sent.

**Workstream pruning:** `prune_workstreams(retention_days, log_fn)` runs once
at startup (CLI and server). It removes:
- Workstreams with no messages (orphaned registrations)
- Unnamed workstreams (`alias IS NULL`) older than `retention_days` days (default 90)

Named (aliased) workstreams are never age-pruned. Configure with
`--retention-days N` (0 = disable age pruning).

---

## Error Handling and Retry

### API Retry

`ChatSession._create_stream_with_retry()` (streaming path) and the agent
`_api_call()` (non-streaming) both use the same retry pattern:

- **Retries**: 4 total attempts (1 initial + 3 retries, `_MAX_RETRIES = 3`)
- **Backoff**: exponential, base 1 second (`delay = 1s * 2^attempt`)
- **Retryable errors**: `RateLimitError`, `APITimeoutError`,
  `APIConnectionError`, `InternalServerError`, `ServiceUnavailableError`,
  `APIError` (matched by class name to avoid importing backend-specific
  exception hierarchies)
- On retry: `ui.on_info()` notification
- On final failure: exception propagates

`_compact_messages()` also wraps its non-streaming API call in the same
retry loop.

### Finish Reason Handling

`_stream_response()` tracks `finish_reason` from the final streaming chunk:

- **`"length"`**: warns via `ui.on_error()` that the response was truncated.
  Any partial tool calls are discarded (their JSON would be malformed),
  causing the `send()` loop to exit cleanly.
- **`"content_filter"`**: warns via `ui.on_error()` that the response was
  blocked.

Agent sub-sessions (`_run_agent()`) check `finish_reason` on each
non-streaming response and stop the agent early on `"length"` or
`"content_filter"`.

`_compact_messages()` checks `finish_reason` on the compaction response and
warns if the summary was truncated.

### State Emission on Errors

- `send()` catches `KeyboardInterrupt` and generic `Exception`: calls
  `_emit_state("error")` before re-raising
- On interrupt: partial tool results and the originating assistant message
  are popped from `self.messages` to keep state consistent

### Web UI Resilience

- **SSE reconnect**: both `connectContentSSE()` and `connectGlobalSSE()` use
  exponential backoff on `onerror` -- starting at 1 second, doubling on each
  failure, capped at 30 seconds. On successful message, delay resets to 1s.
- **Disconnection indicator**: `#status-bar.disconnected` class turns the
  status text red and shows "Reconnecting..."
- **Fetch error handling**: all `fetch()` calls use `.catch()` to prevent
  unhandled promise rejections
- **Pending approval across tab switches**: `WebUI._pending_approval` stores
  the `approve_request` event payload while the session is blocked waiting
  for user response. On SSE reconnect (e.g., switching back to the tab),
  the event is re-injected after history replay. `_build_history` marks the
  pending tool call as `"pending": true` so `replayHistory` skips the
  false `✓ approved` badge; the live approval UI is rendered by the
  re-injected event instead.
- **Browser history integration**: `history.pushState` is called in
  `switchTab()` with `{turnstone: 'workstream', wsId}`. The initial state is
  seeded with `history.replaceState({turnstone: 'dashboard'})` on load. The
  `popstate` listener restores the correct tab or shows the dashboard,
  guarded by `_historyNavigation = true` to prevent re-entrant pushState.

### Eval Resilience

`_run_single_test()`: wraps `session.send_headless()` in a retry loop (3
attempts) to avoid transient API errors from poisoning evaluation scores.

### Health Monitor & Circuit Breaker

`BackendHealthMonitor` (`turnstone/core/healthcheck.py`) runs a daemon thread
that probes the LLM backend by calling `client.models.list()` every
`backend_probe_interval` seconds (default 30). Probe results drive a three-state
circuit breaker:

```
CLOSED  ──(N consecutive failures)──>  OPEN
OPEN    ──(cooldown expires)────────>  HALF_OPEN
HALF_OPEN ──(probe succeeds)────────>  CLOSED
HALF_OPEN ──(probe fails)──────────>  OPEN
```

- `record_success()` / `record_failure()` update `_consecutive_failures` and
  transition the `_state` (`CircuitState` enum: `CLOSED`, `OPEN`, `HALF_OPEN`).
- `acquire_request_permit()` returns `False` when the circuit is `OPEN` or when
  in `HALF_OPEN` and the single probe permit has already been consumed. Causes
  `ChatSession._create_stream_with_retry` to skip the backend and surface an
  error immediately.
- The `/health` endpoint reads the monitor's state: `"status": "ok"` when the
  circuit is closed, `"status": "degraded"` when open or half-open.

### Rate Limiting

`RateLimiter` (`turnstone/core/ratelimit.py`) enforces per-client-IP request
limits using a token-bucket algorithm. Each IP gets a `TokenBucket` with
`requests_per_second` (refill rate) and `burst` (bucket capacity) from
`[ratelimit]` config.

- Applied via `RateLimitMiddleware` after authentication but before route dispatch.
- `/health` and `/metrics` are exempt (monitoring must always be reachable).
- **X-Forwarded-For support**: when `trusted_proxies` is configured (comma-separated
  CIDRs), the middleware parses the `X-Forwarded-For` header using the
  rightmost-untrusted approach. IPv4-mapped IPv6 addresses are normalized.
  The direct client IP must be in the trusted set before XFF is considered.
- On limit exceeded: HTTP 429 with `Retry-After` header and JSON body
  `{"error": "Rate limit exceeded", "retry_after": N}`.
- The `turnstone_ratelimit_rejected_total` counter is incremented on each
  rejection.

---

## User Identity and Authentication

Turnstone supports three authentication mechanisms, unified behind an
`AuthResult` dataclass that carries `user_id`, `scopes`, and `token_source`:

1. **Config-file tokens** — static secrets in `config.toml` `[[auth.tokens]]`
   or the `TURNSTONE_AUTH_TOKEN` env var. Validated in-memory via
   `hmac.compare_digest`. Map to scopes through their role (`read` or `full`).
2. **API tokens** — database-backed, prefixed `ts_`, stored as SHA-256 hashes
   in the `api_tokens` table. Can be exchanged for JWTs via
   `POST /v1/api/auth/login`.
3. **JWTs** — short-lived HMAC-SHA256 session tokens (default 24h) issued after
   successful credential validation. Contain `sub` (user_id), `scopes`, and
   `src` (origin) in claims.

### Scope Model

Three hierarchical scopes control endpoint access:

| Scope | Grants | Endpoints |
|-------|--------|-----------|
| `read` | SSE streams, workstream listing, history | GET endpoints |
| `write` | `read` + send, command, workstream create/close | POST to `/api/send`, `/api/command`, etc. |
| `approve` | `write` + tool approval, admin operations | POST to `/api/approve`, `/api/admin/*` |

### Middleware Flow

`AuthMiddleware` (ASGI) intercepts every request:

1. **Public path check** — `/`, `/static/*`, `/shared/*`, `/health`,
   `/metrics`, `/openapi.json`, `/docs`, `/api/auth/*`, and `/api/auth/setup`
   are always allowed.
2. **Token extraction** — `Authorization: Bearer <token>` header first, then
   `turnstone_auth` cookie as fallback.
3. **Token type detection** — dots in the token indicate JWT; `ts_` prefix
   indicates API token; otherwise config-file token.
4. **Validation** — JWT signature check, API token hash lookup in storage, or
   config-token hmac comparison.
5. **Scope check** — `required_scope(method, path)` determines the minimum
   scope; the request is rejected with 403 if the token lacks it.
6. **Context propagation** — on success, `ctx_user_id` is set so structured
   logging includes the authenticated identity on every log event.

### Architecture Split

- **Console** is the auth management hub — it hosts the admin endpoints for
  creating users, issuing API tokens, and managing channel mappings. User
  records and token hashes live in the shared storage backend. The console
  dashboard includes an **admin panel** (14 tabs) for managing
  credentials, governance, MCP servers, and runtime settings through the browser.
- **Server** is a JWT validator only — it validates tokens on each request but
  never creates users or tokens. Both processes share the same `jwt_secret`
  (via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config).
- **First-time setup** — both server and console expose
  `POST /v1/api/auth/setup`, a public endpoint that creates the initial admin
  user when no users exist. This avoids the chicken-and-egg problem of needing
  `approve` scope to create the first user via `/api/admin/users`.

### Auth Storage Tables

Three tables in `storage/_schema.py` support identity:

```sql
users
  user_id        TEXT PRIMARY KEY
  username       TEXT NOT NULL UNIQUE
  display_name   TEXT NOT NULL
  password_hash  TEXT NOT NULL       -- bcrypt
  created        TEXT NOT NULL

api_tokens
  token_id       TEXT PRIMARY KEY
  token_hash     TEXT NOT NULL UNIQUE  -- SHA-256 of raw token
  token_prefix   TEXT NOT NULL         -- first 8 chars for display
  user_id        TEXT NOT NULL
  name           TEXT NOT NULL         -- human-readable label
  scopes         TEXT NOT NULL         -- comma-separated
  created        TEXT NOT NULL
  expires        TEXT                  -- optional expiry timestamp

channel_users
  channel_type      TEXT NOT NULL      -- e.g. "slack", "discord"
  channel_user_id   TEXT NOT NULL      -- platform-specific user ID
  user_id           TEXT NOT NULL      -- FK to users
  PRIMARY KEY (channel_type, channel_user_id)
```

See [docs/security.md](security.md) for full security details including token
lifecycle, password hashing, and deployment hardening.

---

## Threading Model

### CLI

```
Main thread          Spinner thread (daemon)       ThreadPoolExecutor
+--------------+     +------------------+          +-----------------+
| REPL loop    |     | Braille animation|          | Tool execution  |
| input() ->   |     | 80ms tick to     |          | max_workers=4   |
|   send() ->  |     | stderr           |          | parallel tools  |
|   stream  -> |     | started/stopped  |          | run concurrently|
|   tools   -> |     | by TerminalUI    |          |                 |
+--------------+     +------------------+          +-----------------+
       |                    ^                              ^
       +-- on_thinking_start/stop -------------------------+
       +-- _execute_tools ---------------------------------+
```

Key constraint: `input()` blocks the main thread. The spinner writes to
stderr so it does not interfere with readline. Tool execution may use a
`ThreadPoolExecutor` with up to 4 workers for parallel tool calls.

### Server

```
Starlette ASGI app (served by uvicorn)
  |
  +-- Async request handlers (all under /v1/ prefix)
  |     POST /v1/api/send      -> starts worker thread per workstream
  |     POST /v1/api/approve   -> unblocks WebUI._approval_event
  |     POST /v1/api/plan      -> unblocks WebUI._plan_event
  |     POST /v1/api/workstreams/new -> creates workstream + worker
  |     GET  /v1/api/events    -> SSE via EventSourceResponse (per workstream)
  |     GET  /v1/api/events/global -> SSE via EventSourceResponse (fan-out)
  |
  +-- ASGI middleware stack
  |     MetricsMiddleware -> CORSMiddleware -> AuthMiddleware -> RateLimitMiddleware
  |
  +-- Worker thread per workstream (daemon)
  |     Runs session.send() synchronously -- ChatSession is fully blocking
  |     Blocks on WebUI._approval_event / _plan_event (threading.Event)
  |
  +-- Background daemon threads
        Global SSE fan-out: reads global_queue, copies to per-client queues
        Idle cleanup: closes stale workstreams, cleans rate limiter buckets
```

Starlette handles all HTTP routing, CORS, and middleware. uvicorn runs
the ASGI application with async request handling. All API endpoints live
under the `/v1/` prefix via a Starlette `Mount`. An OpenAPI 3.1 spec is
generated from Pydantic v2 models and served at `/openapi.json`; Swagger
UI is available at `/docs`. SSE endpoints use `EventSourceResponse` from
`sse-starlette` with async generators that bridge sync `queue.Queue` via
`asyncio.get_running_loop().run_in_executor()`.

`ChatSession.send()` remains synchronous, running in daemon worker threads.
WebUI keeps `threading.Event` and `queue.Queue` primitives (unchanged from
the sync era). The `_global_fanout_thread` and `_idle_cleanup_thread` remain
as daemon threads since they interact with sync primitives. A lifespan
context manager handles startup/shutdown (health monitor, MCP client,
registry).

Each workstream's `WebUI` has:
- `_listeners` (per-client SSE queues, fan-out on `_enqueue()`)
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
- `_global_queue` (class variable, shared, for state broadcasts)

The SSE handlers bridge these sync queues to async via
`run_in_executor()`, polling `queue.Queue.get(timeout=1)` while
`sse-starlette` handles keepalive pings automatically.

### Workstream Threading (CLI)

```
Main thread                  Background workstream thread
+------------------+         +---------------------------+
| REPL input()     |         | session.send()            |
| /ws commands     |         | streams response          |
| active workstream|         | executes tools            |
| send() inline   |         | approve_tools() ->        |
+------------------+         |   _fg_event.wait() BLOCKS |
       |                     +---------------------------+
       |                                ^
       +-- /ws <N> switch ------------->|
       |   old.set_foreground(False)    |
       |   new.set_foreground(True)     |
       |   new.flush_buffer()           |
       +-- _fg_event.set() unblocks --->+
```

When a background workstream needs approval, its `WorkstreamTerminalUI`
calls `_fg_event.wait()`, which blocks the worker thread until the user
switches to that workstream. The `_bg_attention_notify` callback writes a
bell + status line to stderr to alert the user.

### Message Queue Bridge

```
Main thread              Global SSE thread         Per-WS SSE threads (×N)
+------------------+     +------------------+      +-------------------+
| Inbound loop     |     | GET /events/glob |      | GET /events?ws_id |
| BLPOP on Redis   |     | Parse SSE via    |      | Parse SSE via     |
|                  |     |   httpx-sse      |      |   httpx-sse       |
| Dispatch to      |     | Forward state    |      | Forward content,  |
|   handler        |     |   changes        |      |   tool results    |
| POST to server   |     | Detect turn      |      | Handle approval   |
| Publish ACK      |     |   completion     |      |   forwarding      |
+------------------+     +------------------+      +-------------------+
       |                         |                         |
       +-- Redis inbound queue   +-- Redis pub/sub         +-- Redis pub/sub
           (RPUSH/BLPOP)             (PUBLISH)                 (PUBLISH)
                                                               + response queue
                                                                 (BLPOP on
                                                                  approval)
```

**Approval flow:** When a per-WS SSE thread receives an `approve_request`, it checks
the workstream's `auto_approve_tools` set. If all requested tools are in the set, the
bridge auto-approves via `POST /v1/api/approve`. Otherwise, it publishes an
`ApprovalRequestEvent` to the outbound channel with a `request_id`, then blocks on
`BLPOP` of a Redis response queue (`turnstone:resp:{request_id}`) until the client pushes
a response or the approval timeout (default 3600s / 1 hour) expires.

**Cancellation:** The `CancelMessage` (type `"cancel"`) is a routed inbound message.
The bridge dispatches it to `POST /v1/api/cancel` on the server owning the workstream,
which sets the cooperative cancel flag and unblocks any pending approval/plan waits.

**Completion detection:** The bridge tracks which `correlation_id` maps to which
`ws_id` for active sends. Content tokens from the per-ws SSE stream are accumulated
in a per-workstream buffer (`_ws_content_buffer`, capped at 256 KB). When the global
SSE reports `ws_state → idle` for a tracked workstream, the bridge emits a synthetic
`TurnCompleteEvent` carrying the correlation ID and the accumulated response text in
`content`. This catch-up mechanism lets downstream consumers (e.g. the Discord bot)
recover the full response even when individual `ContentEvent`s were missed due to the
race between the two independent SSE connections.

**Multi-node routing:** Each bridge retrieves its `node_id` from the server's
`/health` endpoint on startup (with exponential backoff retry). The server
generates the `node_id` (`{hostname}_{4hex}`) and is the sole authority for
node identity. The bridge BLPOPs
from both `turnstone:inbound:{node_id}` (directed, priority) and `turnstone:inbound` (shared).
Messages with `target_node` set are pushed to the target's per-node queue. Messages
for existing workstreams are auto-routed via `turnstone:ws:{ws_id}` ownership keys in Redis.
If a bridge picks up a shared-queue message for a workstream owned by another node, it
re-routes to that node's queue (1 extra hop). Bridges publish heartbeats to
`turnstone:node:{node_id}` with configurable TTL for node discovery.
On startup, `_recover_workstreams` re-registers ownership of existing
workstreams and publishes `WorkstreamCreatedEvent` to the cluster channel
so the console collector picks them up immediately.

### Cluster Console

```
Monitoring (3 daemon threads)        Control + Proxy (async Starlette)
+------------------+                 +----------------------------+
| Event subscriber |                 | POST /v1/api/cluster/      |
| SUBSCRIBE on     |                 |   workstreams/new          |
| events:cluster   |                 |   → LPUSH to Redis         |
+------------------+                 |     inbound:{node_id}      |
| Node discovery   |                 +----------------------------+
| SCAN node:* keys |                 | GET /node/{node_id}/       |
| every 15 seconds |                 |   → httpx.AsyncClient      |
+------------------+                 |     proxy to server_url    |
| Poll loop        |                 | GET /node/{id}/v1/api/events |
| GET /v1/api/dash |                 |   → SSE stream proxy       |
| GET /health      |                 | POST /node/{id}/v1/api/send  |
| ThreadPoolExec   |                 |   → forwarded to server    |
+------------------+                 +----------------------------+
```

The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE
endpoint uses `EventSourceResponse` with the same listener queue pattern as
the main server. `ClusterCollector`'s background threads (event subscriber,
node discovery, poll loop) use sync Redis clients and `ThreadPoolExecutor`
for parallel HTTP polling. The poll loop diffs workstream IDs between poll
cycles and fans out synthetic `ws_created`/`ws_closed` SSE events for any
changes, ensuring browser clients stay in sync even when real-time cluster
events are missed (e.g. bridge startup recovery).

The console has two write-path capabilities:

1. **Workstream creation** — pushes `CreateWorkstreamMessage` to Redis inbound
   queues targeting specific nodes. The bridge on each node picks up the message
   and creates the workstream on the local server. Auto-selects the node with
   the most available capacity if no target is specified. When a `ws_template`
   field is present, the server resolves the template BEFORE `mgr.create()`
   (applying the model override to the creation request) and snapshot-applies
   remaining settings (auto-approve, token budget, temperature, etc.) to the
   workstream config AFTER creation.

2. **Reverse proxy** — serves each node's server UI through the console port at
   `/node/{node_id}/`. Uses `httpx.AsyncClient` to proxy HTTP and SSE traffic.
   A JS shim is injected into the server's `app.js` to override `fetch()` and
   `EventSource()`, routing root-relative URLs through the proxy prefix. This
   eliminates the need for direct network access to individual server nodes.

The console also performs **version drift detection** — flagging when nodes
report different versions via the `/health` endpoint. The overview API includes
`version_drift` and `versions` fields; the dashboard shows a yellow warning
indicator when versions diverge.

Clicking a workstream row in the console opens the proxied server UI at
`/node/{node_id}/?ws_id=<id>` — the server's JS parses this on load and
auto-selects the workstream. See [docs/console.md](console.md) for the full
API reference.

---

## Conversation Compaction

When the prompt exceeds `auto_compact_pct` of the context window (default:
80%, configurable via `--auto-compact-pct`), `ChatSession` auto-compacts by
summarizing the entire conversation into a structured summary
(`_compact_messages`). The summary model call uses `compact_max_tokens`
(default: 32768, configurable via `--compact-max-tokens`). The summary
preserves:

- Decisions made (architecture, libraries, approaches)
- Files read, created, or modified
- Exact identifiers, paths, and code snippets
- Important tool results
- Open tasks
- User preferences

After compaction, `_read_files` is cleared to force re-reads before edits,
since file contents are no longer in the message history.

---

## Client SDK

> See also: [SDK Architecture diagram](diagrams/png/13-sdk-architecture.png) | [SDK Documentation](sdk.md)

The `turnstone/sdk/` package provides typed HTTP clients for programmatic access
to both the server and console APIs. It wraps REST endpoints with methods that
return Pydantic models, and SSE endpoints with async/sync iterators that yield
typed event dataclasses.

**Two client pairs** (sync + async):

- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming)
- `TurnstoneConsole` / `AsyncTurnstoneConsole` — console API (cluster overview, nodes, workstreams)

**Design**: async-first with thin sync wrappers. `_BaseClient` provides httpx
setup, auth headers, `_request()` (REST) and `_stream_sse()` (SSE). Sync
clients delegate through `_SyncRunner` which maintains a persistent background
event loop on a daemon thread.

**Event types**: 27 standalone dataclasses in `events.py` with a type-registry
pattern matching `OutboundEvent.from_json()` from `mq/protocol.py`. Events are
decoupled from the MQ package so SDK consumers don't need the `redis` dependency.

**TypeScript SDK**: `sdk/typescript/` — separate npm package with the same API
surface. Zero browser dependencies, SSE via `fetch` + `ReadableStream` parsing.

```python
# Python quick start
from turnstone.sdk import TurnstoneServer

with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
    ws = client.create_workstream(name="demo")
    result = client.send_and_wait("Hello!", ws.ws_id)
    print(result.content)
```

---

## Channel Integrations

> See also: [Channel Integrations guide](channels.md)

The `turnstone-channel` gateway bridges external messaging platforms
(Discord, Slack, Teams) to the turnstone cluster via Redis MQ. Each
platform adapter implements the `ChannelAdapter` protocol and translates
between platform-native events and turnstone MQ messages.

The `ChannelRouter` manages bidirectional routing: it maps platform
channel/thread IDs to turnstone workstream IDs, handles workstream
creation and stale-route recovery, and resolves platform users to
turnstone identities via the `channel_users` table. When an evicted
workstream is reactivated, the router uses atomic resume via the
`resume_ws` field on `CreateWorkstreamMessage` — the server resumes
the old workstream's conversation during creation in a single HTTP
request, eliminating ordering fragility. The bridge emits a
`WorkstreamResumedEvent` to confirm success.

Discord ships as the first adapter. See [channels.md](channels.md) for
setup instructions, configuration reference, and the adapter development
guide.

### Notification Subsystem

The `notify` tool enables the LLM to send notifications to users or
channels without going through MQ. The server calls the channel gateway
directly over HTTP for lower latency: `_exec_notify()` queries the
`services` database table for healthy channel gateways (heartbeat within
120 seconds), authenticates with a service JWT (`aud: turnstone-channel`),
and POSTs to `POST /v1/api/notify` on the first healthy gateway. The
payload includes the originating `ws_id` for reply routing. The gateway
validates the JWT, resolves the target (username lookup via
`channel_users` or direct `channel_type`+`channel_id`), and delegates to
`ChannelAdapter.send_notification()` which sends the message and tracks
the outgoing message ID → `(ws_id, target_user_id)` mapping. Delivery
retries up to 3 times with backoff, re-querying the service registry on
each attempt. See [Notification Flow diagram](diagrams/png/17-notify-flow.png).

**Bidirectional replies:** When a user replies to a notification DM, the
Discord bot looks up the originating `ws_id` from the tracked message ID,
verifies the replying user matches the notification recipient, and routes
the reply to the workstream via `router.send_message()`. The workstream's
response is forwarded back to the DM via a temporary entry in
`_notify_reply_channels`. On `TurnCompleteEvent`, the response message is
itself tracked for further replies, enabling multi-turn DM conversations
without requiring the user to open the web UI. Tracking entries are capped
at 100 (FIFO eviction) and cleaned up on workstream close.

---

## Governance

> See also: [Governance documentation](governance.md) | [Governance Architecture diagram](diagrams/19-governance-architecture.puml)

Turnstone governance extends the Phase 1 auth system with role-based access
control (RBAC), tool execution policies, prompt templates, usage tracking,
and audit logging. The permission model has two layers: legacy scopes
(`read`, `write`, `approve`) checked by `AuthMiddleware`, and 15 granular
permissions checked per-endpoint by `require_permission()`. Three built-in
roles (admin, operator, viewer) are seeded by migration 008; custom roles
can be created with any permission subset. JWTs carry both `scopes` and
`permissions` claims for backward compatibility.

Tool policies use glob pattern matching (`fnmatch`) with priority-ordered
first-match-wins evaluation to control tool execution (allow/deny/ask).
Prompt templates provide reusable system messages with `{{variable}}`
substitution. Usage events are recorded per-LLM-request for token
accounting. An append-only audit log captures all admin mutations.

Workstream templates build on top of prompt templates as complete behavioral
profiles applied at workstream creation. While prompt templates inject system
message text, workstream templates define model, temperature, reasoning effort,
max tokens, auto-approve policy, token budget, and agent max turns. Templates
are snapshot-applied once at creation — not a live binding. The
`workstream_templates` table (migration 011) supports auto-versioning, and
workstreams record which template and version spawned them. Token budget
enforcement tracks consumption in `session.send()` with 80% warning and
100% approval gate via the `__budget_override__` synthetic tool name.

The console admin panel adds 6 governance tabs (Roles, Policies, Templates,
WS Templates, Usage, Audit), a Memories tab, a Settings tab (form-based
editor for all ConfigStore settings), and an MCP Servers tab (database-backed
server definitions with live connection status and cluster-wide reload) for a
total of 14 tabs, all permission-gated.
Both Python and TypeScript SDKs expose governance methods on the console
client.

## Intent Validation

> See also: [Intent Validation guide](judge.md) | [Judge Architecture diagram](diagrams/png/22-judge-architecture.png)

Intent validation provides advisory risk assessments for tool calls that
require human approval. The system runs a two-tier evaluation pipeline
implemented in `turnstone/core/judge.py`:

1. **Heuristic tier** (synchronous, sub-millisecond) -- A priority-ordered
   rule table using fnmatch tool patterns and regex argument patterns. Four
   severity levels: critical (deny), high (review), medium (review), low
   (approve). First match wins. The heuristic verdict is attached to the
   `approve_request` SSE event immediately.

2. **LLM judge tier** (asynchronous, daemon thread) -- A multi-turn evaluation
   where the judge LLM receives conversation context and tool call details,
   optionally uses `read_file`/`list_directory` to gather evidence (with
   security-hardened path blocking), and produces a structured JSON verdict.
   If the LLM verdict has higher confidence than the heuristic, it replaces
   it via an `intent_verdict` SSE event.

The judge is session-scoped (`IntentJudge`), lazy-initialized on first
approval, and configured via the `[judge]` config section or `--judge` CLI
flags. By default it uses self-consistency (same model), but supports
cross-model and cross-provider configurations. Sub-agents (plan, task)
are exempt. All verdicts are persisted to the `intent_verdicts` table
(migration 012) with the user's final decision, enabling future calibration.
The console exposes `GET /v1/api/admin/verdicts` for audit queries
(requires `admin.judge` permission).