ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

refactor(session): make ModelLane the provider boundary (#979) (#989)

Browse Source 
* refactor(session): make ModelLane the provider boundary (#979)

## Summary

This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.

- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.

## Validation

- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean

Closes #979.

* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8

Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.

Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed

* fix(test): assert canonical model registry URLs

Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.

Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
This commit is contained in:
Patrick Buckley
2026-08-08 16:13:35 -07:00
committed by GitHub
parent f138784ba3
commit 7a06f5e8bc
179 changed files with 37282 additions and 6049 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api-reference.md
+334 -101
View File
@@ -251,7 +251,10 @@ not recognized.
#### Connection lifecycle
1. **`connected`** -- sent immediately on connect.
1. **`connected`** -- sent in the synthetic replay for a fresh connection (and
after an announced replay gap). A cursor reconnect whose buffered gap is
fully covered receives only the missing buffered events, so this preamble is
not duplicated.
```json
{
@@ -262,24 +265,34 @@ not recognized.
}
```
`skip_permissions` reflects the workstream's current auto-approve state. It is
`true` if the server was started with `--skip-permissions` or if the user chose
"Always approve" via the approval prompt during the session.
`skip_permissions` reflects the workstream's blanket auto-approve state. It is
`true` if the server was started with `--skip-permissions` or the workstream was
created with blanket approval. "Approve + Always" now remembers only the tool
names from the resolved cycle and does not flip this field.
2. **`history`** -- replays the full conversation history so the client can
rebuild its UI.
2. **REST history bootstrap** -- the SSE stream does not carry the full
transcript. Before opening a pane's initial event stream, fetch
`GET /v1/api/workstreams/{ws_id}/history?limit=100` (limit is clamped to
1--500). This also works for a saved workstream that is not loaded in the
manager.
```json
{
"type": "history",
"ws_id": "abc123",
"messages": [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi there!", "tool_calls": null},
{"role": "tool", "content": "..."}
]
],
"cursor": null
}
```
`cursor` is normally `null`. When history intentionally trims a still-running
trailing turn that the event ring can reconstruct, open the SSE URL with
`?last_event_id=<cursor>` (or send `Last-Event-ID`) so the buffered delta fills
that turn without double-rendering it.
Each message in the `messages` array has:
| Field | Type | Description |
@@ -298,8 +311,8 @@ Each entry in `tool_calls`:
#### Streaming events
After the initial `connected` and `history` frames, the server streams
real-time events as the model generates a response:
After the synthetic replay or cursor delta, the server streams real-time events
as the model generates a response:
**`thinking_start`** -- the model has begun generating (shown as a spinner).
@@ -350,7 +363,7 @@ state without waiting for the next live transition).
content + reasoning text-so-far when this client connects mid-stream.
Lets a refreshing browser tab restore partial assistant text immediately
instead of waiting for the response to complete. Yielded once after the
kind-specific replay phase (history + pending), only when at least one
kind-specific replay preamble and pending-cycle snapshot, only when at least one
of `content` / `reasoning` is non-empty. Both halves render into the same
assistant bubble the live `content` / `reasoning` events would target;
clients should treat the snapshot as idempotent (skip overwrite if the
@@ -391,11 +404,15 @@ action required).
```
**`approve_request`** -- one or more tool calls that require user approval. The
client must respond via `POST /v1/api/workstreams/{ws_id}/approve`.
client must respond via `POST /v1/api/workstreams/{ws_id}/approve`. Parallel
task agents can leave several approval rounds pending on one workstream at the
same time, so clients should echo the event's `cycle_id` (or one member
`call_id`) when resolving it.
```json
{
"type": "approve_request",
"cycle_id": "cycle_789",
"items": [
{
"call_id": "call_def456",
@@ -410,6 +427,24 @@ client must respond via `POST /v1/api/workstreams/{ws_id}/approve`.
}
```
`cycle_id` identifies this approval round. It is stable across reconnect
replay and is also carried by the corresponding `approval_resolved` event.
**`approval_resolved`** -- one identified approval cycle was answered. Clients
use `cycle_id` (or `call_ids`) to dismiss only that prompt when several remain
live.
```json
{
"type": "approval_resolved",
"cycle_id": "cycle_789",
"call_ids": ["call_def456"],
"approved": true,
"feedback": "",
"always": false
}
```
Each item in `items` (shared by `tool_info` and `approve_request`):
| Field | Type | Description |
@@ -520,19 +555,22 @@ processing.
{"type": "busy_error", "message": "Already processing a request. Please wait."}
```
**`clear_ui`** -- instructs the client to clear all displayed messages (sent
after `/clear` or `/new` commands).
**`clear_ui`** -- instructs the client to clear displayed messages and re-fetch
history after an identity or transcript-boundary change, including `/clear`,
dedicated rewind/retry, successful fork publication, and opening saved history.
```json
{"type": "clear_ui"}
```
**`cancelled`** -- a cancel request was acknowledged (via the Stop button or
`POST /v1/api/workstreams/{ws_id}/cancel`). This signals that cancellation is in progress, not
that it is complete. The worker thread may still be finishing — wait for
`stream_end` before transitioning to a ready state. The client should clear
any in-progress assistant rendering but not re-enable the send button until
`stream_end` arrives.
`POST /v1/api/workstreams/{ws_id}/cancel`). This signals that cancellation is in
progress, not that it is complete. The worker thread may still be finishing.
Clear any in-progress assistant rendering, but keep the composer disabled until
the workstream emits a terminal `state_change` (`idle` in the normal cancel
path, or `error`). `stream_end` only closes assistant rendering: it may already
have arrived before Stop reaches an approval or tool phase, so it is not a
cancellation-completion signal.
```json
{"type": "cancelled"}
@@ -600,13 +638,30 @@ Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
the kind-specific replay (`connected` + `status` + `history` + pending
approval / plan for interactive; `connected` + `status` + pending for coord)
followed by a `state_change` carrying the current worker state and an
optional `in_progress_snapshot` carrying any partial content / reasoning
buffered for the in-progress turn — so a mid-stream refresh restores both
the busy-mode UI and the partial assistant text without waiting for the
response to complete.
either the event-ring delta after its cursor or a synthetic recovery replay.
The synthetic replay includes `connected`, cached `status`, every pending
approval cycle, the current `state_change`, and an optional
`in_progress_snapshot` with partial content/reasoning. Conversation history
stays on the REST `/history` endpoint.
---
### `GET /v1/api/workstreams/{ws_id}/history`
Returns the tail of the reconstructed conversation without opening the
workstream. The endpoint works for a live session and for a saved workstream
that is not loaded in the manager. Cross-kind, tenant, and private-project
visibility checks run before storage reconstruction.
| Query parameter | Type | Default | Description |
|-----------------|------|---------|-------------|
| `limit` | integer | `100` | Tail row limit, clamped to 1--500 |
The response is `{"ws_id": ..., "messages": [...], "cursor": ...}` using the
message shape documented in the event-stream bootstrap above. `cursor` is
normally `null`; when non-null, open `/events?last_event_id=<cursor>` so the
ring replays the deliberately trimmed in-progress tail. A missing, invisible,
or wrong-kind workstream returns the endpoint's ordinary `404` shape.
---
@@ -840,26 +895,41 @@ an `approve_request` event for the given workstream.
**Request body:**
```json
{"approved": true, "feedback": null, "always": false}
{
"approved": true,
"feedback": null,
"always": false,
"cycle_id": "cycle_789"
}
```
| Field | Type | Required | Description |
|------------|-------------|----------|--------------------------------------------------|
| `approved` | bool | yes | `true` to approve, `false` to deny |
| `feedback` | string/null | no | Optional feedback text (sent as denial reason) |
| `always` | bool | no | If `true` and `approved`, enables auto-approve |
| Field | Type | Required | Description |
|------------|-------------|----------|---------------------------------------------------------------|
| `approved` | bool | yes | `true` to approve, `false` to deny |
| `feedback` | string/null | no | Optional feedback text (sent as denial reason) |
| `always` | bool | no | If approved, remember this round's tool names for this session |
| `cycle_id` | string | no | Resolve this exact approval round |
| `call_id` | string | no | Resolve the round containing this tool call |
When `always` is `true` and `approved` is `true`, the workstream's WebUI
instance sets `auto_approve = True`, causing all subsequent tool calls to be
automatically approved without prompting.
adds the tool names from the resolved round to its per-tool auto-approve set.
It does not enable blanket approval for unrelated tools.
Use `cycle_id` when possible. `call_id` is useful for a UI organized around
individual tool rows. If neither selector is supplied, the oldest unresolved
round is selected for compatibility with older clients. A selector that no
longer matches returns `409` with the current oldest `current_cycle_id` and
`current_call_id`; the server never silently redirects a stale click to another
round.
**Response:**
```json
{"status": "ok"}
{"status": "ok", "cycle_id": "cycle_789"}
```
**Error:** `404` with `{"error": "Unknown workstream"}` if `ws_id` is invalid.
`cycle_id` is `null` if no pending round was resolved. An invalid workstream
returns `404`; a stale `cycle_id` or `call_id` returns `409`.
---
@@ -927,13 +997,19 @@ SSE stream / in `/history`).
| `command` | string | yes | The slash command (e.g. `/clear`) |
| `ws_id` | string | yes | Target workstream ID |
If the command is `/clear`, `/new`, or `/resume`, the server pushes a
`clear_ui` SSE event to instruct the client to reset its message display and
re-fetch the transcript via `GET .../history` (there is no SSE event that
carries the messages themselves). These follow-ups are emitted by the
command worker itself, so they fire even when the endpoint already answered
`/clear` pushes a `clear_ui` SSE event to instruct the client to reset its
message display and re-fetch the transcript via `GET .../history` (there is no
SSE event that carries the messages themselves). The follow-up is emitted by
the command worker itself, so it fires even when the endpoint already answered
`{"status": "running"}`.
The remote command surface deliberately rejects lifecycle helpers
`/new`, `/workstreams`, `/resume`, and `/delete`; those commands are local-CLI
only because their legacy implementations enumerate or mutate storage without
the HTTP tenancy gates. Remote callers must use the dedicated create, open,
fork (`resume_ws` on create), close, and delete endpoints. `/rewind` and
`/retry` likewise use their path-keyed endpoints rather than `/command`.
**Response:**
```json
@@ -955,26 +1031,32 @@ or `{"status": "running"}` as above.
### `POST /v1/api/workstreams/{ws_id}/cancel`
Cancels the active generation in a workstream. Sets a cooperative cancellation
flag that is checked at multiple points in the generation loop (per streaming
chunk, before tool execution, inside bash commands). Also closes the underlying
HTTP stream to the LLM provider, unblocking any pending read immediately.
The session transitions to `idle` state and preserves any partial content
already streamed.
Cancels the workstream's current generation. Stop propagates to the primary or
fallback model stream, model-backed attachment processing, parallel task-agent
and foreground tool model calls, intent and output-guard judges, tracked bash
subprocesses, and every approval cycle owned by that generation. Pending plan
review is rejected as well. The worker preserves any assistant content already
streamed and synthesizes honest cancelled tool results where needed so the
saved conversation remains replayable.
If the workstream is waiting for tool approval or plan review, the pending
prompt is automatically denied/rejected to unblock the worker thread.
The cooperative response is immediate: `status: ok` acknowledges the request,
not completion. A running workstream emits `cancelled`, then transitions to
`idle` after its worker unwinds. Depending on where Stop arrived,
`stream_end` may have been emitted before the cancel request or may arrive while
the worker is unwinding; clients use the terminal `state_change`, not
`stream_end`, to become ready. An idle cancel is a harmless no-op and emits no
misleading cancellation event. Detached background shells and watches are
independent resources and are not stopped by this endpoint.
Calling this endpoint when the workstream is already idle is a harmless no-op.
**Force cancel:** When `force` is `true`, the server abandons the stuck worker
thread immediately and transitions the workstream to `idle`. The abandoned
thread continues to wind down in the background (killing any running
subprocesses and exiting at the next cancellation checkpoint). During this
wind-down it may emit a final `stream_end` event which the server suppresses
for the orphaned thread. Use force cancel when cooperative cancel has not
resolved within a few seconds — the web UI offers this as a "Force Stop"
button automatically.
**Force cancel:** When `force` is `true`, the server releases the stuck worker
slot immediately, emits `stream_end`/`idle`, and lets a successor turn start.
The abandoned daemon still owns its already-started external effects until it
reaches a cancellation checkpoint. Send/model generations are fenced from late
history and UI publication, but quick slash-command workers do not yet have
generation checkpoints and may finish an in-place mutation concurrently with a
successor. Use force cancel only when cooperative cancellation has not resolved
within a few seconds — it is recovery from a wedged worker, not confirmation
that every in-flight external side effect was rolled back.
**Path parameters:**
@@ -992,12 +1074,26 @@ button automatically.
|--------|--------|----------|----------------------|
| `force`| bool | no | Abandon stuck worker immediately (default: `false`) |
The body is optional. Because cancel is a recovery verb, an empty or malformed
JSON body is treated as `force: false` rather than blocking Stop.
**Response:**
```json
{"status": "ok"}
{
"status": "ok",
"dropped": {
"was_running": true,
"pending_approval": {"tool_names": ["bash"], "call_id": "call_abc123"},
"queued_messages": {"count": 1, "first_preview": "follow up after the build"}
}
}
```
`dropped` is a best-effort, credential-redacted snapshot of affected pending
work. Fields are omitted when they were not observable. Coordinator sessions
currently return an empty object.
**Error responses:**
| Status | Body | Condition |
@@ -1009,7 +1105,8 @@ button automatically.
### `POST /v1/api/workstreams/new`
Creates a new workstream. The server supports up to 10 concurrent workstreams.
Creates a new workstream, subject to the configured
`server.max_workstreams` capacity.
The endpoint accepts **either** `application/json` (legacy shape) **or**
`multipart/form-data` when you want to upload attachments at creation
@@ -1017,50 +1114,118 @@ time. Multipart requests carry one `meta` field containing the JSON body
shown below plus zero-or-more `file` parts; each file is validated and
reserved onto the new workstream's first turn before the dispatch worker
runs, so queued multimodal turns cannot lose files to racing sends. If
validation fails the fresh workstream is rolled back so no orphan rows
leak.
validation fails the fresh workstream is rolled back so no published row or
phantom create/close event leaks.
**Request body:**
```json
{"name": "my-ws", "model": "openai"}
{"name": "my-ws", "model": "openai", "initial_message": "Start the review"}
```
All fields are optional. The body can be empty or an empty JSON object.
All fields are optional; send an empty JSON object for a defaults-only create.
An absent or malformed JSON body returns `400`.
| Field | Type | Default | Description |
|------------------|--------|---------|----------------------------------------------------------------|
| `name` | string | auto | Workstream display name |
| `model` | string | default | Model alias from the registry (`[models.*]`) |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
| `skill` | string | "" | Skill name. Applies content (system prompt), model, temperature, reasoning effort, max tokens, auto-approve policy, token budget, and other session config from the skill. Returns 400 if not found or disabled. Ignored when `resume_ws` is set (resumed sessions restore their own skill). |
| `persona` | string | "" | Persona slug. Resolved and snapshotted into the workstream at creation; empty selects the kind's default. |
| `judge_model` | string | "" | Optional model alias for the judge (overrides default judge model for this workstream) |
| Field | Type | Default | Description |
|-------------------|---------------|---------|----------------------------------------------------------------|
| `name` | string | auto | Workstream display name |
| `model` | string | default | Model alias from the registry |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `auto_approve_tools` | string/array | `""` | Tool names to auto-approve even when `auto_approve` is false; accepts comma-separated text or an array |
| `user_id` | string | `""` | Owner override honored only for a trusted `console` service identity carrying the `service` scope; ordinary callers remain bound to their authenticated identity |
| `resume_ws` | string | `""` | Source workstream ID or alias to fork atomically into this new ID |
| `skill` | string | `""` | Skill name. Applies its system prompt and session configuration. Returns 400 if missing/disabled; ignored for a fork because the source configuration is cloned. |
| `persona` | string | `""` | Persona slug; empty selects the kind's default. A fork keeps the source persona. |
| `judge_model` | string | `""` | Optional judge model alias |
| `initial_message` | string | `""` | First user message to dispatch after publication |
| `ws_id` | 32-hex string | generated | Caller-selected destination ID; required by the cluster multipart routing path |
| `project_id` | string/null | none | Project to attach. A fork always inherits the source's effective project. |
| `notify_targets` | string/array | `[]` | Completion-notification targets |
| `client_type` | string | `web` | Client surface label (`web`, `cli`, `chat`, or `scheduled`) |
| `parent_ws_id` | string/null | none | Owning coordinator ID for a coordinator-spawned child |
> **Skill behavior:** When `skill` is specified, the skill's content is injected as a system message and its session config fields (model, temperature, auto-approve, token budget, etc.) override system defaults for the new workstream.
#### Fork behavior (`resume_ws`)
Despite the compatibility field name, `resume_ws` does not reopen or move the
source workstream. It creates a distinct destination ID and atomically clones
the source's checkpoint-bounded conversation, saved session configuration,
persona, effective project, and attachment references. The source remains
unchanged. Use `POST .../{ws_id}/open` when you want to rehydrate the original
ID instead.
The clone transaction rechecks source visibility, private-project membership
and attachability, persona/project construction context, destination ownership
and emptiness, and attachment integrity. A caller cannot use `project_id` to
re-file or declassify the fork. Uploads cannot be combined with `resume_ws`;
fork first, then use the ordinary attachment endpoint. Concurrent source-history
writes serialize wholly before or after the clone snapshot; access,
construction-context, or destination conflicts fail the whole fork rather than
publishing a mixed result.
#### Publication and rollback
Creation first reserves the ID durably with internal state `creating`. That
reservation is hidden from list, saved, resolve, open, and cluster-event
surfaces while the session is constructed, uploads are validated, and an
optional fork transaction commits. The final durable `creating` to `idle`
compare-and-set happens before `ws_created`, audit, initial-message dispatch,
or any state event. A normal pre-publication failure immediately and
conditionally deletes the exact token-bearing reservation and emits no
lifecycle event; if cleanup itself fails, the original HTTP error is retained
and `ws.create.rollback_failed` is logged, leaving the row hidden rather than
advertising a half-create.
Long-lived server and console processes also run hidden-reservation recovery at
boot and every five minutes, independently of ordinary idle eviction. It only
considers rows still in internal `state='creating'` and older than two hours,
excluding IDs currently loaded or pending in the manager. A live remote owner
protects its rows; the current process's stable node ID does not self-protect,
so a restart can recover its predecessor's residue. Failure to establish
service liveness, or a storage failure, deletes nothing. Eligible rows are
atomically hard-deleted with their dependent records and attachment refcounts;
an eligible tokenless legacy or corrupt reservation is locked, recovered, and
logged as a warning. Retention pruning leaves `creating` rows to this path. The
value is not a live `WorkstreamState`, and recovery neither publishes nor
closes it.
**Response (success):**
```json
{"ws_id": "ghi789", "name": "ws-3", "resumed": false, "message_count": 0}
{
"ws_id": "ghi789",
"name": "ws-3",
"resumed": false,
"message_count": 0,
"attachment_ids": []
}
```
| Field | Type | Description |
|-----------------|--------|-----------------------------------------------------|
| `ws_id` | string | Unique ID of the new workstream |
| `name` | string | Auto-generated workstream name |
| `resumed` | bool | Whether a previous session was successfully resumed |
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
| `name` | string | Assigned workstream display name |
| `resumed` | bool | Whether the requested source was successfully forked |
| `message_count` | int | Messages cloned into the destination (0 if fresh/empty) |
| `attachment_ids` | string[] | Attachments saved by this create request |
| `initial_message_status` | string | Present ONLY when the workstream was created but its `initial_message` could not be delivered: `"queue_full"` (a raced live worker's interjection queue was at capacity — resend via `/send`; any uploads stay staged) or `"refused_closed"` (the workstream was closed mid-create). Absent whenever the message was dispatched. |
**Error (limit reached):**
For compatibility, `resumed: true` means the requested fork completed; the
source was not resumed in place.
```json
{"error": "Maximum of 10 workstreams reached"}
```
**Selected errors:**
Status code: `400`
| Status | Condition |
|--------|-----------|
| 400 | Invalid body/upload/persona/skill, attachments combined with `resume_ws`, or required project missing |
| 403 | Destination project attach denied |
| 404 | Fork source missing or not visible (same shape prevents an existence oracle) |
| 409 | Caller-selected ID collision, source availability/construction context changed during fork, or destination reservation was superseded |
| 413 | Upload exceeds the configured request/file cap |
| 429 | Workstream manager is at capacity; retry after capacity frees |
| 503 | Storage/factory/model configuration unavailable, or the fork transaction failed operationally |
| 500 | Unexpected create failure; response includes a correlation ID for server logs |
---
@@ -1229,6 +1394,8 @@ Permanently delete a saved workstream and all its messages from storage.
Load a saved workstream into memory with its original `ws_id`. If the
workstream is already loaded, returns immediately with `already_loaded: true`.
An internal `creating` reservation is not openable and returns the ordinary
not-found shape until publication completes.
**Path parameters:**
@@ -2132,7 +2299,7 @@ Status code: `200` with an empty body.
| Condition | Behavior |
|------------------------------------|------------------------------------------------------------|
| Malformed or unparseable JSON body | Treated as an empty dict `{}`; missing fields use defaults |
| Malformed, absent, or non-object body on an endpoint that requires a JSON object | `400`; cancel is the deliberate recovery-verb exception and treats it as `force: false` |
| Unknown `ws_id` | `404` with `{"error": "Unknown workstream"}` |
| Unknown path (GET or POST) | `404` with plain-text body `Not found` |
| Empty `message` on `/v1/api/workstreams/{ws_id}/send` | `400` with `{"error": "Empty message"}` |
@@ -2175,10 +2342,26 @@ reconnection:
| Maximum delay | 30 seconds |
| Reset | Delay resets to 1 second on first success |
On reconnect, the server replays the full conversation history via the
`history` event, so the client can rebuild its UI state without data loss. The
same reconnection strategy applies to both the per-workstream SSE stream
(`/v1/api/workstreams/{ws_id}/events`) and the global state stream (`/v1/api/events/global`).
Per-workstream events carry monotonic SSE IDs and are retained in a bounded
ring. Native `Last-Event-ID` and the `?last_event_id=N` query fallback both
resume after the last applied event. If the ring covers the gap, only missing
events are replayed. If it does not, the server emits:
```json
{
"type": "replay_truncated",
"ws_id": "abc123",
"lost_count": 4,
"earliest_available_id": 91
}
```
The clients then refetch `/history`, adopt its optional resume cursor, and
reconnect; an in-progress snapshot covers partial text on the synthetic path.
This REST snapshot plus cursor/delta split prevents both missing turns and
double-rendering across refreshes, ring eviction, and process restart. The
global state stream has its own snapshot/replay floor rather than conversation
history.
---
@@ -2310,34 +2493,84 @@ gateway) talk to the console instead of individual server nodes.
### `POST /v1/api/route/workstreams/new`
Create a workstream via rendezvous routing. The console generates the `ws_id`,
routes to the rendezvous-selected node, and includes `node_url` in the
response for direct SSE connections.
Create a workstream through the console routing layer. The JSON body accepts
the ordinary create fields plus `target_node`:
### `POST /v1/api/route/send`
| Field | Routing behavior |
|-------|------------------|
| `ws_id` | Optional 32-hex destination. When present, it is preserved and used as the rendezvous key, including on a fork. A 503 never replaces a caller-selected ID. |
| `resume_ws` | Optional source ID or saved alias for an atomic fork. The console resolves aliases to the canonical source ID before routing and forwards that canonical value. When no destination `ws_id` is supplied, the source is the placement key. |
| `target_node` | Optional node ID hint. When neither `ws_id` nor `resume_ws` selects placement, the console generates a destination whose rendezvous owner is this live node. |
Proxy a message to the workstream's assigned server node.
Without any placement field, the console generates a destination ID and routes
it by rendezvous. Multipart callers must pre-allocate the destination and put
the **same** 32-hex value in both `?ws_id=<32-hex>` and the multipart
`meta.ws_id` field. The query value selects the target node; the console
buffers the body, parses only `meta` to require the same destination ID, then
forwards the original bytes and boundary unchanged. The node uses `meta.ws_id`
as the destination identity.
### `POST /v1/api/route/approve`
The response extends the node create response with three required fields:
`node_url`, authoritative `node_id`, and `routing_strategy`.
`routing_strategy` is `rendezvous` for generated, explicit JSON, and multipart
destination IDs; `target_node` when the console generated an ID for a requested
node; or `resume` only when an atomic fork was placed by its canonical source
ID. The node-returned destination `ws_id` is authoritative for the response,
storage binding lookup, and audit record; the fork source is never reported as
the created destination.
Proxy an approval response to the workstream's assigned server node.
The JSON body must be an object. `ws_id`, `resume_ws`, and `target_node` must be
strings when supplied; malformed placement fields return `400`. A missing fork
source returns the same generic `404` as other missing workstreams. If a node
returns `200` without an object containing a valid destination `ws_id`, the
console returns a bounded `502` instead of exposing or trusting the malformed
payload.
### `POST /v1/api/route/cancel`
### `GET /v1/api/route/workstreams/{ws_id}/live`
Cancel generation on a workstream.
Probe the rendezvous-selected owner without opening or rehydrating the
workstream. The console asks that node's manager-authoritative active list and
returns only:
```json
{"ws_id": "abc123", "live": true}
```
Missing, unloaded, still-`creating`, and caller-invisible workstreams all
produce `live: false`. Routing, upstream, and authorization uncertainty returns
an error instead of a false miss, so callers can preserve an existing route.
### `POST /v1/api/route/workstreams/{ws_id}/send`
Proxy a message to the workstream's assigned server node. `DELETE` on the same
path dequeues a queued send.
### `POST /v1/api/route/workstreams/{ws_id}/approve`
Proxy an approval response, including optional `cycle_id` / `call_id`, to the
workstream's assigned server node.
### `POST /v1/api/route/workstreams/{ws_id}/cancel`
Cancel generation on a workstream. The request and response have the same
`force` / `dropped` shape as the node endpoint.
### `POST /v1/api/route/command`
Send a slash command to a workstream.
Send a conversation-local slash command. This legacy route still takes
`ws_id` in the JSON body.
### `POST /v1/api/route/plan`
### `POST /v1/api/route/workstreams/{ws_id}/{rewind|retry}`
Send plan review feedback to a workstream.
Proxy a dedicated conversation-modification request.
### `POST /v1/api/route/workstreams/close`
### `POST /v1/api/route/workstreams/{ws_id}/close`
Close a workstream.
The console also exposes path-keyed routed attachment endpoints and
`POST /v1/api/route/workstreams/delete` for coordinator-driven hard deletion.
### `GET /v1/api/route?ws_id=X`
Look up which server node owns a workstream. Returns `{"node_url": "...", "node_id": "..."}`.
docs/architecture.md
+543 -185
View File
File diff suppressed because it is too large Load Diff
docs/channels.md
+22 -16
View File
@@ -195,13 +195,17 @@ both and the gateway hosts both adapters in one process.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the server emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
- If a persisted channel route is no longer active on its owning node, the
router asks the create endpoint to fork the old workstream into a new ID via
`resume_ws`. The saved source can still resolve normally; its
checkpoint-bounded history, configuration, persona, effective project, and
attachment references are cloned before the channel route is repointed. The
old route remains durable until the replacement (and any initial message)
succeeds. If the create endpoint returns the ordinary
source-not-found response *and* a fresh authoritative storage lookup confirms
that the source is gone, the router retries once without `resume_ws` and
starts a fresh conversation. Other access, conflict, routing, and storage
failures remain visible rather than silently discarding history.
### Slash Commands
@@ -284,15 +288,17 @@ See [Security: Database Schema](security.md#database-schema) for the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route and creates a new workstream with the old `ws_id`
as `resume_ws` on the creation request. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The channel receives a `WorkstreamResumedEvent`, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
3. **Eviction** — the server evicts an idle workstream for capacity. Its saved
source row and channel route remain durable, and the thread stays open.
4. **Reactivation** — the next message resolves the saved route and probes
whether that workstream is live on its owning node. If it is not, the router
creates a distinct workstream with the old `ws_id` as `resume_ws`. The
create response confirms the fork and message count; there is no separate
resume command or channel-specific resumed event. Only after the replacement
succeeds does the router swap the persisted route. If the source was deleted
or pruned, an exact source-not-found response plus a second authoritative
storage miss triggers one fresh-create retry; other fork failures leave the
old route intact and are surfaced normally.
5. **Close** — `/close` command closes the workstream via HTTP, deletes the
route, unsubscribes from events, and archives the Discord thread.
docs/console.md
+90 -36
View File
@@ -174,17 +174,32 @@ Request:
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5"
"model": "gpt-5",
"project_id": "proj_analytics",
"initial_message": "Profile the slow query"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **`"pool"`** — compatibility alias for automatic placement on the reachable node with the most headroom.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
- `judge_model` — optional judge-model alias for this workstream.
- `initial_message` — first message dispatched after the workstream is published.
- `skill` — enabled profile/skill to snapshot onto a fresh workstream.
- `persona` — enabled persona slug; empty uses the interactive default.
- `project_id` — project to attach, subject to the target node's membership gate.
- `resume_ws` — source ID to **fork** atomically into a new workstream. The
source remains unchanged; its checkpoint-bounded history, configuration,
persona, project, and attachment references are copied transactionally.
The endpoint also accepts the same multipart create shape as a node: one
JSON-encoded `meta` field plus up to ten `file` parts. Files require an
`initial_message` in the dashboard launcher. Files cannot be combined with
`resume_ws`; fork first and upload on the new workstream.
Response:
@@ -196,7 +211,19 @@ Response:
}
```
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
The response is returned only after the target node has durably published the
workstream. Its hidden `creating` reservation has already crossed to `idle`,
and the node emitted `ws_created` before any initial-message state event. The
cluster SSE event may therefore arrive before or after the HTTP response;
clients should reconcile both by the returned `correlation_id`/workstream ID
rather than treating them as two creates.
For safety, the console masks most target-node failures as the opaque `502`
shape `{"error":"Dispatch to node <node_id> failed"}` instead of reflecting
arbitrary node text or retry-triggering 401/429 responses. The coded
`server.require_project` refusal is the exception and remains a `400` with
actionable wording. Consult the target node's logs for the underlying create
correlation when a reachable node returns a masked 502.
### `GET /v1/api/cluster/events`
@@ -310,8 +337,8 @@ The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
| `write` | Non-approval mutations: send, create/open/close/delete, cancel, attachments, rewind, and retry |
| `approve` | Tool-approval and admin HTTP surfaces (with their additional RBAC permission checks) |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
@@ -348,52 +375,73 @@ SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). Ordinary users are re-minted with `src="console-proxy"`; coordinator tokens retain `src="coordinator"` plus `coord_ws_id`, and only the validated console service identity with `service` scope retains `src="console"` for trusted owner forwarding. When no user context is available, the proxy falls back to a `ServiceTokenManager` identity `console-proxy` carrying `src="console"` and `{read, write, approve, service}` scopes. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The web UI has five views, toggled client-side:
The console uses an L-shaped application shell: a collapsible navigation rail,
a tab bar, and a pane host. On mobile the rail becomes an off-canvas drawer.
The rail is fed by the cluster SSE snapshot and shows:
### 1. Cluster Overview (landing)
- state/count filters and the live compute-node list, including version drift;
- active coordinator and interactive workstreams, nested under their
coordinator parent and grouped by project when project metadata is visible;
- permission-filtered Manage groups that open the singleton Admin pane.
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
Coordinator and interactive conversations open as tabs inside the same shell.
Interactive panes use the owning node's console proxy, so users do not need
direct network access to compute-node ports. Split-right and split-down actions
can display several panes at once. Closing a pane removes only that tab; use the
pane menu's explicit close or delete action to change the workstream lifecycle.
### 2. Node Drill-down
### Dashboard pane
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
The home view is coordinator-first. It contains the persistent workstream
launcher plus the saved-sessions list. Selecting a state count opens the
filtered workstream table inside the same Dashboard pane; selecting a compute
node opens its proxied node surface. Cluster SSE updates keep rail state,
workstream rows, and tab state glyphs synchronized.
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
### Workstream launcher
### 3. Filtered Workstreams
The landing-page composer starts a workstream with an optional initial task and
attachments. When the caller can create both kinds, a Coordinator / Interactive
toggle selects the target kind. Its options include:
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Node placement** — "Least loaded" picks the reachable node with the most
headroom, or "Specific node" pins the create to a node from the live list.
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Skill** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Project** — optional project filing. Private projects require owner/member access. A coordinator child inherits its parent's project unless explicitly routed to another attachable project.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
- **Judge Model** — optional text input for the judge model alias (overrides the default judge model for this workstream).
- **Model** — optional selector populated from the target model registry.
- **Judge Model** — optional selector for the judge alias (overrides the default
judge model for this workstream).
Keyboard shortcuts: Ctrl+Shift+R (refresh title), Ctrl+Shift+E (edit title), Ctrl+Shift+F (fork), Ctrl+Shift+X (delete). Press ? for full shortcut help.
Interactive launches additionally expose node strategy / node selection.
Submitting uses `POST /v1/api/cluster/workstreams/new`; coordinator launches use
the console's coordinator create surface. A toast confirms the committed
create, while SSE updates the dashboard and opens the resulting pane.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
### Saved and filtered sessions
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
Saved coordinator and interactive sessions share one list with kind and persona
labels, filtering, pagination, and multi-select deletion. Opening a saved
coordinator rehydrates it in the console; opening a saved interactive session
resolves its node, calls `open`, and then connects the node-proxied pane.
### 5. Admin Panel
The filtered live table carries STATE, NAME, MODEL, NODE, TASK, TOKENS, and CTX
columns. The browser maintains a local `clusterState` initialized from the
cluster snapshot and updated incrementally by SSE; the filtered view normally
renders from that state without another API round trip.
### Admin pane
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
@@ -405,8 +453,11 @@ Audit tabs, and [Settings](settings.md) for the database-backed
configuration editor.
The **Channels** tab links users to either a Discord or Slack account
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, the **Nodes** tab edits per-node
via a per-row channel-type selector. The **Models** tab is a CRUD
editor for `model_definitions`, including static and dynamic backend-auth
modes. Model edits rebind existing workstreams at their next send while
in-flight requests keep their original definition snapshot; see
[Settings](settings.md#model-definition-reloads) for the full contract. The **Nodes** tab edits per-node
metadata, and the **TLS** tab manages CA and leaf certificates for the
internal mTLS fabric. The **Settings** tab edits ConfigStore values
live; edits apply without restart.
@@ -504,7 +555,7 @@ Run history is automatically pruned (runs older than 90 days) approximately once
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `pool` | Compatibility alias for the reachable node with the most headroom |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
@@ -664,4 +715,7 @@ turnstone-server --port 8080
turnstone-console --port 8090
```
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams from
the persistent Dashboard launcher. Selecting a workstream opens a coordinator
or node-proxied interactive pane in the console shell — no direct access to
server ports is required.
docs/coordinator-api-tour.md
+22 -11
View File
@@ -112,8 +112,8 @@ with a `type` field. The recurring shapes a UI has to handle:
| `stream_end` | End of a single provider stream | — |
| `tool_result` | A tool call completed (success or error) | `call_id`, `name`, `output`, `is_error?` |
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `approve_request` | One approval cycle needs operator action; several cycles may coexist | `cycle_id`, `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | One identified approval cycle was answered | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
@@ -129,8 +129,8 @@ with a `type` field. The recurring shapes a UI has to handle:
| `info` / `error` | Operational messages | `message` |
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
one `approve_request` snapshot for every unresolved approval cycle, keyed by
the same stable `cycle_id`, plus any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
@@ -324,24 +324,35 @@ uses the cascade-mutation shape and how it differs from the
The `approve` endpoint is what resolves an `approve_request` SSE
event. The coordinator's worker thread is blocked inside
`ui.approve_tools` waiting for this POST.
`ui.approve_tools` waiting for this POST. Parallel task agents can leave
several approval cycles live at once, so current clients echo the event's
`cycle_id` (or a member `call_id`). A selector-less request resolves the oldest
cycle for compatibility.
```http
POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": false}
{"approved": true, "feedback": null, "always": false, "cycle_id": "cycle_789"}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
{"approved": true, "feedback": null, "always": true} // remember this cycle's tool names
```
`cancel` drops the coordinator's in-flight generation and, for a
coordinator, auto-cascades the cancel to its direct children:
Success returns `{"status": "ok", "cycle_id": "cycle_789"}`. A stale selector
returns `409` with the currently oldest cycle/call IDs. `always` remembers only
the tool names in the cycle that actually resolved; it does not enable blanket
approval.
`cancel` requests cooperative cancellation of the coordinator's in-flight
generation and auto-cascades to its direct children:
`cancel_workstream` is dispatched through the routing proxy for
every direct child in the registry. The coordinator itself is left
idle and open for a fresh `send`:
every direct child in the registry. The HTTP acknowledgement is immediate;
the worker becomes idle after unwinding. Pass `{"force": true}` only to release
a wedged worker slot immediately. The coordinator itself remains open for a
fresh `send`:
```http
POST /v1/api/workstreams/{ws_id}/cancel
{}
{"status": "ok", "dropped": {}}
```
---
docs/diagrams/01-system-context.puml
+11 -9
View File
@@ -13,7 +13,7 @@ cloud "LLM Providers" as llm {
component [OpenAI-compatible API\n(OpenAI, vLLM, llama.cpp)] as llm_openai
component [Anthropic Messages API] as llm_anthropic
}
database "SQLite\n(.turnstone.db)" as sqlite
database "SQLite / PostgreSQL\n(durable state)" as storage
' Turnstone System Boundary
package "Turnstone Platform" {
@@ -33,24 +33,26 @@ eval_user --> eval : Python API
' Internal connections
cli --> llm : LLM Provider API\n(via provider adapters)
cli --> sqlite : SQLite
cli --> storage : persistence
server --> llm : LLM Provider API\n(via provider adapters)
server --> sqlite : SQLite
server --> storage : persistence
eval --> llm : LLM Provider API\n(non-streaming)
eval --> sqlite : SQLite
eval --> storage : persistence
console --> server : HTTP proxy\n(hash-ring bucket lookup,\nproxy /node/{id}/* traffic)
console --> server : HTTP routing/UI proxy + cluster SSE\n(FNV-1a rendezvous placement,\nproxy /node/{id}/* traffic)
channel --> server : HTTP + SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
channel --> console : multi-node route/create/live/send/approve
channel --> server : direct mode + owning-node SSE\n(POST /v1/api/workstreams/{ws_id}/send,\nGET /v1/api/workstreams/{ws_id}/events)
' Notes
note right of console
Multi-node router:
- Hash-ring bucket lookup
- FNV-1a rendezvous placement
- Proxies create/send/approve
- Direct SSE from client to node
- HTTP polling for dashboard
- Collector aggregates node SSE
- Browser dashboard receives console SSE fanout
- /node/{id} proxies pane HTTP + SSE
end note
@enduml
docs/diagrams/02-package-structure.puml
+37 -10
View File
@@ -18,6 +18,7 @@ skinparam component {
package "Entry Points" <<Rectangle>> {
component [cli.py\nturnstone] as cli <<entry>>
component [server.py\nturnstone-server] as server <<entry>>
component [console/server.py\nturnstone-console] as consoleentry <<entry>>
component [eval.py\nturnstone-eval] as eval <<entry>>
component [admin.py\nturnstone-admin] as admin <<entry>>
component [bootstrap.py\nturnstone-bootstrap] as bootstrap <<entry>>
@@ -25,9 +26,16 @@ package "Entry Points" <<Rectangle>> {
' Core engine
package "turnstone/core/" <<Rectangle>> {
component [session.py\nChatSession, SessionUI] as session <<core>>
component [session.py\nChatSession, SessionUI\ngeneration-fenced turn loop] as session <<core>>
component [session_manager.py\nSessionManager\nshared lifecycle invariants] as sessionmanager <<core>>
component [adapters/\ninteractive + coordinator\nconstruction/event policies] as adapters <<core>>
component [model_turn.py\nModelLane, model_turn()\nlower / sample / re-ingest] as modelturn <<core>>
component [trajectory.py\ncanonical Turn IR] as trajectory <<core>>
component [lowering.py\nprovider-wire lowering] as lowering <<core>>
component [state_writer.py\nordered durable state tail] as statewriter <<core>>
component [model_backend_auth.py\nper-call backend credentials] as modelauth <<core>>
component [providers/\nLLMProvider, OpenAI, Anthropic, Google] as providers <<core>>
component [workstream.py\nWorkstreamManager] as workstream <<core>>
component [workstream.py\nWorkstream types + state] as workstream <<core>>
component [tools.py\nTool loader] as tools <<core>>
component [memory.py\nPersistence facade] as memory <<core>>
component [storage/\nStorageBackend protocol\nSQLite + PostgreSQL] as storage <<core>>
@@ -79,18 +87,18 @@ package "turnstone/api/" <<Rectangle>> {
package "turnstone/sdk/" <<Rectangle>> {
component [server.py\nTurnstoneServer (sync+async)] as sdkserver <<sdk>>
component [console.py\nTurnstoneConsole (sync+async)] as sdkconsole <<sdk>>
component [events.py\n27 SSE event types] as sdkevents <<sdk>>
component [events.py\nTyped SSE event stream] as sdkevents <<sdk>>
component [_base.py\nhttpx client base] as sdkbase <<sdk>>
}
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n19 tool schemas] as schemas <<artifact>>
component [*.json\nBuilt-in tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
cli --> session
cli --> workstream
cli --> sessionmanager
cli --> config
cli --> memory
cli --> colors
@@ -99,7 +107,8 @@ cli --> spinner
cli --> tools
server --> session
server --> workstream
server --> sessionmanager
server --> adapters
server --> config
server --> memory
server --> metrics
@@ -113,11 +122,26 @@ eval --> memory
eval --> config
eval --> tools
consoleentry --> sessionmanager
consoleentry --> adapters
consoleentry --> consoleserver
admin --> auth
bootstrap --> providers
' Core internal deps
session --> providers
sessionmanager --> workstream
sessionmanager --> adapters
sessionmanager --> storage
adapters --> session : constructs
session --> modelturn
session --> trajectory
session --> lowering
session --> statewriter
session --> modelauth
modelturn --> providers
modelturn --> trajectory
modelturn --> lowering
session --> tools
session --> memory
memory --> storage
@@ -129,6 +153,7 @@ session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
modelturn --> registry : coherent snapshot
healthcheck --> metrics
mcp --> config
registry --> config
@@ -138,15 +163,17 @@ tools --> schemas
gateway --> discordbot
gateway --> slackbot
gateway --> router
discordbot --> sdkserver : HTTP + SSE
slackbot --> sdkserver : HTTP + SSE
discordbot --> sdkserver : direct HTTP + node SSE
slackbot --> sdkserver : direct HTTP + node SSE
router --> sdkserver : single-node/direct mode
router --> sdkconsole : multi-node route/create/live
router --> storage : channel_routes
' Console dependencies
consoleserver --> collector
consoleserver --> config
consoleserver --> auth
collector --> server : HTTP polling
collector --> server : discovery HTTP + cluster SSE aggregation
' API dependencies
serverspec --> openapi
docs/diagrams/03-core-engine-classes.puml
+120 -25
View File
@@ -32,7 +32,7 @@ class "TerminalUI" as TerminalUI {
class "WorkstreamTerminalUI" as WsTermUI {
- _output_buffer: list[tuple]
- ws_id: str
- manager: WorkstreamManager
- manager: SessionManager
+ flush_buffer()
--
Buffers output when workstream
@@ -41,14 +41,14 @@ class "WorkstreamTerminalUI" as WsTermUI {
class "WebUI" as WebUI {
- _listeners: list[Queue]
- _approval_event: Event
- _approval_cycles: dict[str, ApprovalCycle]
- _ws_prompt_tokens: int
- _ws_tool_calls: dict
+ resolve_approval(approved, feedback)
+ resolve_approval(approved, feedback, cycle_id?, call_id?)
--
Enqueues JSON events for SSE.
Blocks on threading.Event for
approval.
Concurrent approval cycles each own
a threading.Event and result slot.
SSE handlers bridge Queue to
async via run_in_executor().
--
@@ -126,25 +126,75 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_reasoning_replay: bool
}
class "ModelLane" as ModelLane <<frozen>> {
+ provider: LLMProvider
+ client: Any
+ model: str
+ alias: str
+ capabilities: ModelCapabilities
+ extra_params: dict | None
+ registry: ModelRegistry | None
+ backend_auth_config: ModelConfig | None
+ backend_auth_resolver: Callable | None
}
class "ResolvedModelBinding" as ResolvedBinding <<frozen>> {
+ lane: ModelLane
+ config: ModelConfig | None
+ registry_generation: int
}
class "ModelTurnResult" as ModelTurnResult <<frozen>> {
+ turn: Turn
+ tool_calls: list[dict]
+ finish_reason: str
+ usage: UsageInfo | None
+ wire_msgs: list[dict] | None
+ producer: str
+ serving_model: str
}
class "model_turn()" as ModelTurnFn {
Turn IR → lower → provider stream
→ drain → canonical assistant Turn
--
core/model_turn.py
}
class "Backend auth resolver" as BackendAuth {
+ resolve_model_backend_auth_token(...)
--
Resolves static / Entra OBO /
Entra app / RFC 8693 per call.
Dynamic failure can fail closed.
--
core/model_backend_auth.py
}
' ChatSession
class "ChatSession" as ChatSession {
- client: Any
- provider: LLMProvider
- model: str
- _model_binding: ResolvedModelBinding
- _model_binding_lock: Lock
- ui: SessionUI
- messages: list[dict]
- messages: list[Turn]
- _msg_tokens: list[int]
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
- _generation: int
- _cancel_event: Event
- _durability_next_ticket: int
+ model_alias: str | None {property}
- _tools: list[dict]
- _task_tools: list[dict]
- _read_files: set[str]
- system_messages: list[dict]
--
+ send(user_input: str)
+ send(user_input: str, ..., acting_user_id: str | None)
+ cancel()
+ compact_now() → bool
+ fork_from_storage(source_ws_id, principal_id, ...)
+ handle_command(command: str)
+ resume(ws_id: str)
- _save_config()
@@ -162,8 +212,10 @@ class "ChatSession" as ChatSession {
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
- _compact_messages(auto: bool, my_generation: int)
- _commit_for_generation(generation, commit)
- _publish_for_generation(generation, publish)
- _full_messages() → list[Turn]
- _update_token_table(msg)
- _emit_state(state: str)
- _generate_title()
@@ -180,15 +232,36 @@ class "HeadlessSession" as HeadlessSession {
records all tool calls
}
' WorkstreamManager
class "WorkstreamManager" as WsMgr {
- _session_factory: Callable[[SessionUI], ChatSession]
' SessionManager
interface "SessionKindAdapter" as KindAdapter <<Protocol>> {
+ kind: WorkstreamKind
+ build_ui(ws) → SessionUI
+ build_session(ws, ...) → ChatSession
+ cleanup_ui(ws)
}
interface "SessionEventEmitter" as EventEmitter <<Protocol>> {
+ emit_created(ws)
+ emit_rehydrated(ws)
+ emit_state(ws, state)
+ emit_closed(ws_id, reason, name)
}
class "SessionManager" as SessionMgr {
- _adapter: SessionKindAdapter
- _storage: StorageBackend
- _workstreams: dict[str, Workstream]
- _pending_creates: dict[str, Workstream]
- _retiring_ids: set[str]
- _state_writer: StateWriter | None
- _order: list[str]
- _active_id: str
- _on_state_change: Callable
--
+ create(name, ui_factory) → Workstream
+ create(user_id, name, ..., defer_emit_created) → Workstream
+ commit_create(ws) → bool
+ discard(ws, ...) → bool
+ open(ws_id) → Workstream | None
+ delete(ws_id) → bool
+ close(ws_id)
+ get(ws_id) → Workstream
+ get_active() → Workstream
@@ -203,11 +276,17 @@ class "Workstream" as Ws <<dataclass>> {
+ id: str
+ name: str
+ state: WorkstreamState
+ session: ChatSession
+ ui: SessionUI
+ worker_thread: Thread
+ session: ChatSession | None
+ ui: SessionUI | None
+ worker_thread: Thread | None
+ error_message: str
+ last_active: float
+ kind: WorkstreamKind
+ user_id: str
+ parent_ws_id: str | None
+ project_id: str | None
- _fork_reservation_token: str
- _closed: bool
- _lock: Lock
}
@@ -283,7 +362,7 @@ class "ModelRegistry" as ModelReg {
+ fallback: list[str]
+ agent_model: str | None
--
+ resolve(alias) → (client, model, config)
+ resolve_binding(alias) → (client, model, config, provider, generation)
+ get_client(alias) → Any
+ get_provider(alias) → LLMProvider
+ has_alias(alias) → bool
@@ -306,6 +385,9 @@ class "ModelConfig" as ModelCfg <<frozen>> {
+ temperature: float | None
+ max_tokens: int | None
+ reasoning_effort: str | None
+ auth_mode: str
+ obo_audience: str
+ obo_scopes: str
}
' Circuit breaker state
@@ -375,22 +457,33 @@ LLMProvider <|.. AnthropicProv
OpenAIProv <|-- GoogleProv
ChatSession --> SessionUI : uses
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> ResolvedBinding : owns coherent snapshot
ChatSession --> ModelTurnFn : every model-backed role
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
WsMgr --> "*" Ws : manages
SessionMgr --> "*" Ws : manages
SessionMgr --> KindAdapter : delegates construction
SessionMgr --> EventEmitter : lifecycle fan-out
Ws --> "1" ChatSession : wraps
Ws --> "1" SessionUI : wraps
Ws --> "1" WsState : has
WsMgr ..> ChatSession : creates via\nsession_factory(ui, model_alias)
KindAdapter ..> ChatSession : constructs
ModelReg --> "*" ModelCfg : holds
ModelReg --> "*" LLMProvider : caches
LLMProvider --> ModelCaps : returns
ModelReg --> ResolvedBinding : resolves atomically
ResolvedBinding --> ModelLane
ModelLane --> LLMProvider
ModelLane --> ModelCaps
ModelLane --> ModelCfg : auth/config snapshot
ModelTurnFn --> ModelLane
ModelTurnFn --> ModelTurnResult
ModelTurnFn ..> BackendAuth : per-call resolver
ChatSession --> HealthMon : checks circuit
HealthMon --> "1" CircuitState : has
@@ -403,7 +496,9 @@ note bottom of ChatSession
Provider-agnostic — delegates all LLM
communication to LLMProvider adapters.
core/session.py (~2700 lines)
Every live/durable publication is fenced by
its generation. Model calls use immutable lanes;
provider-wire mutation stays at lowering.
end note
@enduml
docs/diagrams/04-conversation-turn.puml
+132 -170
View File
@@ -1,183 +1,145 @@
@startuml
!theme plain
title Turnstone — Conversation Turn Lifecycle
title Turnstone — Generation-Fenced Conversation Turn
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "User /\nHTTP Client" as User
participant "ChatSession" as CS
participant "SessionUI" as UI
participant "LLMProvider\n(OpenAI / Anthropic)" as LLM
participant "Tool Executor\n(ThreadPool)" as TP
database "SQLite" as DB
participant "HTTP / CLI\ncaller" as User
participant "SessionManager" as Manager
participant "ChatSession" as Session
participant "SessionUIBase" as UI
participant "model_turn()\n+ lowering" as Plant
participant "LLM provider" as Provider
participant "Tool workers" as Tools
database "StorageBackend\n(SQLite / PostgreSQL)" as Storage
== User Input ==
== Admission and generation claim ==
User -> CS : send(user_input)
activate CS
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(ws_id, "user", input)
== LLM Call Loop ==
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
CS -> LLM : provider.create_streaming(\n client, model, messages, tools, ...)\n (normalized StreamChunk iterator)
activate LLM
note right of CS
Retry up to 3× on transient errors:
RateLimitError, APITimeoutError,
APIConnectionError, InternalServerError,
ServiceUnavailableError, APIError
Backoff: 1s, 2s, 4s
end note
== Streaming Response ==
loop for each chunk in stream
LLM --> CS : delta
note right of CS
on_thinking_stop() called on first
delta token via _stop_spinner_once()
end note
alt reasoning_content present
CS -> UI : on_reasoning_token(text)
else content present
CS -> UI : on_content_token(text)
else tool_call delta
CS -> CS : accumulate in tool_calls_acc
else info_delta present
CS -> UI : on_info(text)\n(e.g. server-side web search status)
end
end
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
LLM --> CS : stream complete (usage stats)
deactivate LLM
CS -> UI : on_thinking_stop() (no-op guard: already called by _stop_spinner_once)
CS -> UI : on_stream_end()
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
alt no tool_calls
CS -> UI : on_status(usage, context_window, effort)
opt prompt_tokens > context_window × auto_compact_pct
CS -> CS : _compact_messages(auto=True)
CS -> LLM : Non-streaming summarization call
CS -> CS : Replace messages with [summary]
end
opt first exchange & no title
CS -> CS : Background thread: _generate_title()
end
CS -> UI : on_state_change("idle")
CS --> User : return
else has tool_calls
CS -> UI : on_state_change("running")
== Phase 1: Prepare ==
CS -> CS : [_prepare_tool(tc) for tc in tool_calls]\nParse JSON args, validate,\nbuild preview + header
== Phase 2: Approve ==
CS -> UI : on_state_change("attention")
CS -> UI : approve_tools(items)
activate UI
note right of UI
TerminalUI: input() prompt
WebUI: _approval_event.wait()
NullUI: returns (True, None)
end note
UI --> CS : (approved: bool, feedback: str?)
deactivate UI
CS -> UI : on_state_change("running")
== Phase 3: Execute ==
CS -> TP : ThreadPoolExecutor(max_workers=4)\nrun_one(item) for each tool
activate TP
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task → _run_agent() sub-loop
web_fetch → httpx + LLM summarize
web_search → provider-native or SearxNG fallback
memory/recall → SQLite
end note
note right of TP
bash: on_tool_output_chunk(call_id, line)
called per stdout line,
then on_tool_result(call_id, name, output, is_error).
is_error=True when execution failed.
call_id routes chunks/results to correct
tool div during parallel execution.
Other tools: on_tool_result() only.
end note
TP --> CS : [(call_id, output), ...]
deactivate TP
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
CS -> CS : messages.append({role: "user", content: feedback})
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
User -> Manager : dispatch send on one Workstream
Manager -> Session : bind_acting_user(principal)\nsend(text, attachments, send_id)
activate Session
Session -> Session : refresh immutable ResolvedModelBinding
opt token budget exhausted
Session -> UI : approve_tools(__budget_override__)
note right of UI
This gate precedes a generation claim but carries
a monotonic cancellation witness. Stop cannot be
mistaken for a budget-policy denial.
end note
end
deactivate CS
Session -> Session : _claim_generation() → generation N\ninstall fresh cancel event
Session -> Session : plan memory / participant context
Session -> Storage : ordered durable batch:\nappend canonical user Turn + metadata
note over Session, Storage
_commit_for_generation(N) admits bounded live mutations under the
generation lock, then executes immutable persistence closures in FIFO
ticket order. A force successor either follows the whole commit or
prevents it; storage I/O never holds the lifecycle lock.
end note
opt already over the hard context ceiling
Session -> Session : compact before first model call\n(preserve the new user turn)
end
== Model / tool loop ==
loop until final answer and no queued input
Session -> UI : on_turn_start()\nreset per-stream replay buffers
Session -> UI : state = thinking\non_thinking_start()
Session -> Session : _stream_response(N)\nretry + fallback policy
Session -> Plant : model_turn(active ModelLane, Turns,\n tools, cancel_ref, on_chunk)
activate Plant
Plant -> Plant : canonical Turns → provider wire\nrestore ids + repair + lane-specific fold
Plant -> Plant : resolve per-call backend credential\nfrom lane's pinned ModelConfig
Plant -> Provider : create_streaming(...)
activate Provider
loop normalized stream chunks
Provider --> Plant : StreamChunk
Plant --> Session : on_chunk(StreamChunk)
Session -> Session : check cancel event + generation N
Session -> UI : reasoning / content / info token
end
Provider --> Plant : finish + usage + native blocks
deactivate Provider
Plant -> Plant : drain + re-ingest assistant Turn\nwith serving-lane provenance
Plant --> Session : ModelTurnResult
deactivate Plant
Session -> UI : on_stream_end()
Session -> Session : generation-fenced result commit:\nappend assistant Turn + token accounting
Session -> UI : on_turn_committed()
Session -> Storage : ordered durable assistant row\n(content + tool mirror + native lane)
alt no tool calls
opt over soft threshold
Session -> Session : cooperative / end-of-turn compaction
Session -> Storage : append checkpoint summary marker\nwith source watermark
note right of Storage
Full history remains durable. Resume loads
[summary] + rows after the checkpoint.
end note
opt model stopped for compaction
Session -> Storage : append synthetic compaction_resume Turn
end
end
alt queued messages drained
Session -> Storage : append combined queued user Turn
else truly complete
Session -> UI : state = idle
end
else tool calls present
Session -> UI : state = running
Session -> Session : prepare items + previews\nattach cancellation witnesses
opt one or more items require a human
Session -> UI : approve_tools(items)\nregister independent ApprovalCycle
note right of UI
Parallel agents may own concurrent cycles.
cycle_id / call_id routes exactly one decision;
Smart Approvals may clear qualifying items.
end note
User -> UI : approve / deny selected cycle
UI --> Session : decision + optional feedback
end
Session -> Tools : execute admitted items in parallel
activate Tools
Tools --> UI : chunks + result card\nwith effect disposition
Tools --> Session : outputs / errors / effect statuses
deactivate Tools
Session -> Session : output-guard evaluation\nthen generation N re-check
opt compaction owed before result sizing
Session -> Session : compact, preserving assistant tool-call Turn
Session -> Storage : append checkpoint marker
end
Session -> Session : one generation-fenced batch:\nappend all Tool Turns, advisories, feedback
Session -> Storage : FIFO durable tool rows + metadata
end
end
== Stop / force-successor boundary ==
User -> Session : cancel()
Session -> Session : atomically set generation event; snapshot\nmain stream, child scopes, judges, subprocesses
Session -> Provider : close live stream handle
Session -> Tools : abort child scopes + kill subprocess groups
Session -> UI : resolve only cancelled operation's\napproval cycles
note over Session, Storage
Every later publish/commit checks generation ownership. An abandoned
worker may unwind, but cannot append Turns, overwrite state, resolve a
successor approval, or repaint the successor UI. Observed tool effects
are preserved as controller-authored cancellation receipts; unreviewed
tool bytes are not laundered into model context.
end note
deactivate Session
@enduml
docs/diagrams/05-tool-pipeline.puml
+86 -103
View File
@@ -1,134 +1,117 @@
@startuml
!theme plain
title Turnstone — Tool Execution Pipeline (Three Phases)
title Turnstone — Tool Pipeline: Prepare, Approve, Execute, Fold
start
partition "Phase 1: Prepare" #E8F5E9 {
:Receive tool_calls list from LLM response;
partition "Phase 1 — Prepare and assess" #E8F5E9 {
:Receive tool calls from one assistant Turn;
:Capture the generation's cancel event\nand acting principal;
while (more tool_calls?) is (yes)
:Extract call_id, func_name, raw_args;
if (json.loads(raw_args) succeeds?) then (yes)
:parsed_args = JSON dict;
while (more tool calls?) is (yes)
:Parse arguments and dispatch to\nthe tool-specific preparer;
if (preparation succeeds?) then (yes)
:Build item: call_id, name, header, preview,\nneeds_approval, execute closure;
else (no)
:Fallback 1: regex extraction;
if (regex found keys?) then (yes)
:parsed_args = extracted dict;
else (no)
:Fallback 2: bare string →\nPRIMARY_KEY_MAP[func_name];
endif
:Build an error item for this call only;\nkeep sibling calls valid;
endif
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 built-in + tool_search):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ diff_file │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve │
│ web_search │ ✗ Auto-approve │
│ tool_search │ ✗ Auto-approve │
│ task_agent │ ✓ Yes │
│ memory │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
│ watch │ ✓ create only │
│ skill │ ✓ load only │
│ read_resource │ ✓ Yes │
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
{call_id, func_name, header,
preview, needs_approval,
approval_label, execute: Callable};
:Attach operation-local cancellation witness\nand pinned principal;
endwhile (no)
:Reject only unsafe ordering shapes\n(for example tasks read + write in one batch);
:Run heuristic intent assessment immediately;
:Start generation-pinned LLM judge in background;
:Stamp one immutable Smart Approval\nsettings snapshot on the batch;
note right
Preparation is per-call isolated: one bad preparer
becomes one error Tool Turn rather than orphaning the
assistant's entire tool-call set.
end note
}
partition "Phase 2: Approve" #FFF3E0 {
if (any items need approval?) then (yes)
:_emit_state("attention");
:ui.approve_tools(items);
partition "Phase 2 — Approval cycle" #FFF3E0 {
:Apply explicit bypasses:\nskill / always / policy / blanket;
if (Smart Approvals enabled?) then (yes)
:Wait within the batch's bounded judge deadline;
:Auto-approve only LLM approve verdicts\nat or above the captured threshold;
endif
if (human-gated items remain?) then (yes)
:Acquire approval-publication lease;
:Register independent ApprovalCycle\n(cycle_id, call_ids, event, result);
:Publish approve_request + heuristic verdicts;
note right
**auto_approve check is handled
internally by ui.approve_tools()**
**TerminalUI**: Print headers/previews,
prompt [y/n/a, optional message]
If user chose "always":
Add pending tool names to auto_approve_tools
(auto-approve these tool types going forward)
**WebUI**: Enqueue approve_request,
block on _approval_event.wait()
**NullUI**: Return (True, None)
Parallel task agents can hold several cycles at once.
A decision selects one cycle_id / call_id (or the oldest
cycle for a legacy selector-less client). Double resolve
is a guarded no-op; one cycle cannot wake a sibling.
end note
if (user approved?) then (yes)
:_emit_state("running");
else (denied)
:Mark all pending items as denied;
:denial_msg = "Denied by user";
:_emit_state("running");
if (operator approves?) then (yes)
:Record decision and optional feedback;
else (denies / policy blocks)
:Mark only pending items denied;\nEffectStatus = none;
endif
else (all auto-approved)
:ui enqueues tool_info event\n(no blocking);
:Publish approval_resolved;\nunregister this cycle;
else (all bypassed / auto-approved)
:Publish tool_info with the exact\nauto-approve reason per item;
endif
if (owning operation cancelled?) then (yes)
:Cancel only cycles carrying that witness;
:Stage every unstarted call as\nEffectStatus = none;
stop
endif
}
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
:Execute in parallel:\nThreadPoolExecutor(max_workers=4)\npool.map(run_one, items);
partition "Phase 3 — Execute" #E3F2FD {
:Generation + cancellation checkpoint;
if (batch requires serial ordering?) then (yes)
:Execute in provider order;
else (no)
:Execute via bounded ThreadPoolExecutor;
endif
note right
**run_one(item):**
if item.error → return error string
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: SearxNG JSON GET (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
Each worker marks its call started only after the final
generation/cancel check. A missing result after that edge is
conservatively unknown; an unstarted call is definitively none.
end note
:Collect results: [(call_id, output), ...];
:Stream tool chunks to the matching call card;
:Capture result / error / preview and effect disposition;
:_truncate_output() on each result\n(max context_window × chars_per_token × 0.5 chars\ndefault: ~context_window × 2 chars);
if (Stop interrupts execution?) then (yes)
:Abort child model scopes and subprocess groups;
:Synthesize cancellation receipts;
note right
EffectStatus vocabulary:
committed / none / unknown /
partial / rolled_back.
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
:ui.on_tool_result(call_id, name, output, is_error) for each;
Observed but unreviewed bytes are omitted from the
model-facing receipt; effect truth is retained.
end note
endif
}
:Return (results, user_feedback);
partition "Phase 4 — Guard and atomic fold" #F3E5F5 {
if (compaction already owed?) then (yes)
:Compact before sizing/folding results;\npreserve the assistant tool-call Turn;
endif
:Truncate each result against the remaining shared budget;
:Run heuristic + optional LLM output guard;
:Re-check generation after guard work;
:Under one generation commit, append the complete\nTool Turn block + advisories + feedback;
:Persist rows and effect/preview metadata\non the ordered durability lane;
:Return results to the next model turn;
}
stop
@enduml
docs/diagrams/09-workstream-states.puml
+68 -12
View File
@@ -3,6 +3,7 @@
title Turnstone — Workstream State Machine
skinparam state {
BackgroundColor<<lifecycle>> #ECEFF1
BackgroundColor<<idle>> #E8F5E9
BackgroundColor<<thinking>> #E3F2FD
BackgroundColor<<running>> #FFF3E0
@@ -10,13 +11,18 @@ skinparam state {
BackgroundColor<<error>> #FFCDD2
}
state "CREATING (persisted only)" as creating <<lifecycle>> : Hidden durable reservation.\nNot returned by ordinary list/open/history.
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
state "CLOSED (persisted only)" as closed <<lifecycle>> : Unloaded, explicitly reopenable row.\nNot a live WorkstreamState member.
[*] --> idle : Session created
[*] --> creating : register exact incarnation\nstate="creating"
creating --> idle : finalize + publish create\nemit ws_created
creating --> [*] : immediate exact-token rollback\n(no lifecycle birth emitted)
creating --> [*] : stale >2h recovery\natomic hard delete; no close event
idle --> thinking : send() called\n_emit_state("thinking")
@@ -38,6 +44,14 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
idle --> closed : close / eviction
error --> closed : close
thinking --> closed : close
running --> closed : close
attention --> closed : close
closed --> [*] : hard delete
closed --> idle : open / rehydrate
thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
@@ -45,33 +59,75 @@ running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
note left of idle
**Cancel escalation:**
1. **Cooperative**: cancel() sets event + closes
SDK stream → worker exits at next checkpoint
2. **Force**: force=true abandons the worker
thread, emits stream_end immediately.
Orphaned thread still kills subprocesses
but skips message mutations (generation
counter prevents stale writes).
**Generation-scoped Stop:**
• Sets the active generation event.
• Closes its SDK stream; aborts child model
scopes and judges; kills subprocess groups.
• Sweeps every approval cycle owned by the
cancelled workstream operation.
• Every later send/model live or durable commit
re-checks generation ownership.
**force=true:** also abandons the stuck worker
slot and emits stream_end + IDLE immediately.
An orphaned send/model generation may unwind
but cannot publish into a successor generation.
Quick slash-command workers are a best-effort
escape hatch: without generation checkpoints,
one may finish an in-place mutation concurrently.
**Capacity eviction:** an IDLE candidate is only
a hint. Per-ID + object lifecycle lanes and the
workstream lock revalidate it as worker- and
send-barrier-free,
then install a terminal claim before slot swap.
end note
note right of thinking
**Emitted via:**
session._emit_state(state)
→ ui.on_state_change(state)
→ SessionManager state tail
**Propagation:**
• WebUI → global SSE queue (ws_state)
• Console → HTTP polling picks up state
• CLI → WorkstreamManager.set_state()
• Console → cluster event / HTTP state
• CLI → SessionManager.set_state()
Non-terminal persistence may use StateWriter;
a per-id tail orders storage + subscribers and
prevents a late state from overwriting CLOSED.
end note
note left of attention
**Blocking mechanisms:**
• TerminalUI: input() prompt
• WebUI: threading.Event.wait()
• WebUI: one Event per ApprovalCycle
• ChannelBot: SSE event + Discord button
• NullUI: auto-approve (never reaches)
end note
note right of creating
CREATING and CLOSED are storage lifecycle
values, not members of WorkstreamState. The
live enum remains IDLE / THINKING / RUNNING /
ATTENTION / ERROR.
**Crash-abandoned CREATING recovery:**
• Boot pass, then every 5 min even when idle
eviction is disabled.
• Only rows >2h old; manager loaded/pending
IDs and live remote owners are protected.
• The current stable node ID is not a live-owner
exemption, allowing restart recovery.
• Unknown liveness/storage fails closed. Deletion
is atomic across dependents and attachment refs.
• Tokenless legacy/corrupt rows are locked,
reaped, and logged with a warning.
A loaded hard delete closes publication, drains
admitted session durability + state tails, then
conditionally removes the exact durable token.
end note
@enduml
docs/diagrams/12-deployment.puml
+2 -2
View File
@@ -31,7 +31,7 @@ node "Docker Host" as host {
Command: turnstone-console
--port 8090
Depends: server
Hash-ring router for
FNV-1a rendezvous router for
multi-node clusters
end note
}
@@ -69,7 +69,7 @@ apiclient --> server : HTTP + SSE\nport 8080
' Internal connections
server --> llm_api : OpenAI API\n(HTTPS/HTTP)
console --> server : HTTP proxy\n(hash-ring lookup,\nproxy /node/{id}/*)
console --> server : HTTP proxy\n(FNV-1a rendezvous placement,\nproxy /node/{id}/*)
' Database connections (production/cluster profiles)
server ..> pgbouncer : PostgreSQL\n(pool_size=2)
docs/diagrams/14-storage-architecture.puml
+150 -129
View File
@@ -1,170 +1,191 @@
@startuml
!theme plain
title Turnstone — Storage Architecture
title Turnstone — Storage, Deferred Create, Fork, and Checkpoint Architecture
skinparam class {
BackgroundColor<<protocol>> #E8EAF6
BackgroundColor<<sqlite>> #C8E6C9
BackgroundColor<<postgres>> #B3E5FC
BackgroundColor<<facade>> #FFF9C4
BackgroundColor<<migration>> #FFE0B2
BackgroundColor<<lifecycle>> #FFF9C4
BackgroundColor<<schema>> #F3E5F5
BackgroundColor<<helper>> #FFE0B2
}
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit, *, parent_ws_id, kind, user_id) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
+kv_list() → list[(str, str)]
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
' -- Backends --
class "SQLiteBackend" as SQLite <<sqlite>> {
-_engine: sa.Engine
-_fts5_available: bool
+__init__(path: str)
interface "StorageBackend" as Storage <<protocol>> {
+ load_message_turns(ws_id, checkpointed=True) → list[Turn]
+ save_message(ws_id, role, content, metadata...)
+ clone_workstream(source, destination, principal, expected_session) → ForkCloneSnapshot
--
FTS5 full-text search
Default pool, check_same_thread=False
+ register_workstream(..., state, reservation_token) → bool
+ ensure_workstream_incarnation_snapshot(ws_id) → row + token
+ finalize_deferred_create(ws_id, token, config...) → bool
+ publish_deferred_create(ws_id, token) → bool
+ delete_workstream_if_fork_reserved(ws_id, token) → bool
+ delete_stale_creating_reservations(...) → list[ws_id]
+ update_workstream_state(ws_id, state)
+ delete_workstream(ws_id) → bool
--
+ attachment / project / memory / auth / governance APIs
}
class "SQLiteBackend" as SQLite <<sqlite>> {
- _engine: sa.Engine
- _fts5_available: bool
--
Fork clone: BEGIN IMMEDIATE
FTS5 refresh in same transaction
}
class "PostgreSQLBackend" as PG <<postgres>> {
-_engine: sa.Engine
+__init__(url: str, pool_size: int = 2,\n max_overflow: int = 3)
- _engine: sa.Engine
--
tsvector + ILIKE search
Connection pooling (5 max per process)
Fork clone: SERIALIZABLE + row locks
Retry SQLSTATE 40001 / 40P01
DML success uses RETURNING rows
}
' -- Schema --
class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+workstreams: Table (node_id, alias, title,\n state, skill_id)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
+scheduled_tasks: Table (..., skill)
class "_utils.py" as Utils <<helper>> {
+ reconstruct_turns(rows) → list[Turn]
+ recover_trajectory(turns) → list[Turn]
+ reconstruct_turns_checkpointed(...)
+ retain_attachment_refs(conn, ids)
+ release_attachment_refs(conn, ids)
+ clone_workstream_transaction(...) → ForkCloneSnapshot
}
class "ForkCloneExpectation" as Expectation <<lifecycle>> {
+ persona_config
+ project_id / name / writable
+ source_reservation_token
+ destination_reservation_token
}
class "ForkCloneSnapshot" as Snapshot <<lifecycle>> {
+ turns: tuple[Turn, ...]
+ config: dict[str, str]
+ project_id: str | None
}
class "workstreams" as Workstreams <<schema>> {
ws_id PK
state: creating | live state | closed
user_id, node_id, kind, parent_ws_id
project_id, persona, alias, title
}
class "conversations" as Conversations <<schema>> {
canonical persisted Turn rows
provider_data + tool_calls mirror
event_id, source, is_error, meta
attachment-id ref list
--
SQLAlchemy Core
Single source of truth
compaction marker:
source="compaction"
meta.watermark=<folded row id>
}
' -- Migration --
class "_migrate.py" as Migrate <<migration>> {
+run_migrations(storage, backend)
-_bootstrap_existing_sqlite()
--
Programmatic Alembic
Auto-bootstrap existing DBs
class "workstream_config" as WorkstreamConfig <<schema>> {
PK (ws_id, key)
stamped persona/session config
private durable incarnation fence:
__fork_destination_reservation
}
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
class "workstream_attachments" as Attachments <<schema>> {
content-addressed blob
attachment_id, bytes, kind
refcount
}
' -- Registry --
class "_registry.py" as Registry {
-_storage: StorageBackend | None
+init_storage(backend, path, url) → StorageBackend
+get_storage() → StorageBackend
+reset_storage()
--
Auto-initializes SQLite
if not configured
class "projects + project_members" as Projects <<schema>> {
visibility / owner / membership
active project-memory envelope
}
' -- Facade --
class "memory.py" as Facade <<facade>> {
+save_message()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all delegated functions)
--
Thin delegation to
get_storage()
Silent failure behavior
class "SessionManager" as Manager <<lifecycle>> {
+ create(..., defer_emit_created)
+ commit_create(ws)
+ discard(ws)
+ reap_stale_creating_reservations(max_age=2h)
+ open / close / delete
}
' -- Consumers --
class "session.py\nChatSession" as Session {
class "ChatSession" as Session <<lifecycle>> {
+ append canonical Turns
+ compact / resume checkpoint
+ fork_from_storage(...)
}
class "server.py\nWeb UI" as Server {
}
SQLite ..|> Storage
PG ..|> Storage
SQLite --> Utils
PG --> Utils
class "cli.py\nTerminal" as CLI {
}
Storage --> Workstreams
Storage --> Conversations
Storage --> WorkstreamConfig
Storage --> Attachments
Storage --> Projects
' -- Relationships --
SQLite ..|> SB
PG ..|> SB
Manager --> Storage : lifecycle reservation + state
Session --> Storage : turn durability + resume
Session --> Expectation : construction witness
Storage --> Snapshot : atomic clone result
Expectation --> Utils : checked inside transaction
Utils --> Snapshot : builds
SQLite --> Schema : uses
PG --> Schema : uses
note right of Manager
**Deferred create publication**
1. INSERT workstream as state="creating" and store a fresh
private token in the same transaction.
2. Construct UI/session and run attachment/fork gates while
ordinary list/open/history reads exclude the row.
3. finalize_deferred_create atomically applies config/alias.
4. publish_deferred_create compare-and-swaps creating → idle.
5. Only then emit ws_created.
Registry --> SB : creates
Registry --> Migrate : calls
Migrate --> Versions : applies
Migrate --> Schema : references
Facade --> Registry : get_storage()
Session --> Facade : imports
Server --> Facade : imports
CLI --> Facade : imports
' -- Config --
note right of Registry
[database]
backend = "sqlite" | "postgresql"
url = "postgresql+psycopg://..."
path = ".turnstone.db"
pool_size = 2 (+ 3 overflow)
Any normal prepublication failure immediately calls exact token-checked
deletion. The token survives publication as the row's incarnation fence:
rollback or later hard delete can never ABA-delete a replacement row.
A legacy row acquires the same private token atomically when rehydrate,
delete, or fork preflight takes its authoritative snapshot. Loaded hard
delete drains admitted session durability before its token-checked delete.
end note
note bottom of SQLite
Default backend.
Zero-config for
single-node / dev.
note left of Manager
**Crash-abandoned hidden-create recovery**
• Boot pass; long-lived processes repeat every 5 min,
even when ordinary idle eviction is disabled.
• Candidates remain state="creating", are >2h old,
and are absent from the manager loaded/pending set.
• Live remote owners are protected. The current stable
node ID does not self-protect, enabling restart recovery.
• Unknown liveness or storage failure deletes nothing.
• One transaction rechecks state, age, and token, then
hard-deletes dependents and releases attachment refs.
• Tokenless legacy/corrupt rows use their locked durable
row as the incarnation fence and log a warning.
• Retention pruning excludes creating rows. Recovery never
closes or publishes them as live WorkstreamState values.
end note
note bottom of PG
Production backend.
Multi-node / Docker default.
Use PgBouncer (transaction mode)
for clusters > 50 nodes.
note bottom of Utils
**Atomic fork clone**
• Reject a provisional source; compare the source incarnation captured
by canonical preflight; re-authorize project visibility and compare the
live session envelope inside the transaction.
• Require a same-owner, empty destination still in creating state
with the exact reservation token.
• Copy the checkpoint-bounded canonical trajectory and config;
retain every referenced attachment or roll everything back.
• Preserve/rebase a valid compaction checkpoint watermark and
return the exact snapshot installed into the live destination.
end note
note bottom of Conversations
Full transcript rows are never deleted by compaction. Normal resume
loads the latest valid [summary] + rows after its watermark; audit and
export can request the full marker-free history.
end note
@enduml
docs/diagrams/15-auth-architecture.puml
+129 -166
View File
@@ -1,190 +1,153 @@
@startuml
!theme plain
title Turnstone — Authentication Architecture
title Turnstone — User Authentication and Model-Backend Credentials
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<token>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
BackgroundColor<<runtime>> #FFE0B2
BackgroundColor<<model>> #F3E5F5
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
package "Request identity" {
class "AuthMiddleware / check_request()" as RequestAuth <<core>> {
Extract bearer or HttpOnly cookie
Validate audience + expiry
Check scope / permission
Publish AuthResult in request state
}
class "AuthResult" as AuthResult <<core>> {
+ user_id: str
+ scopes: frozenset[str]
+ permissions: frozenset[str]
+ token_source: str
}
class "JWT" as JWT <<token>> {
HS256, sub, aud, iat, exp
console proxy mints short-lived
server-audience identity
}
class "API / config token" as ApiToken <<token>> {
ts_* token: SHA-256 DB lookup
config token: constant-time compare
}
class "users / roles / api_tokens" as UserTables <<storage>> {
password hash + token hash
role-derived permissions
}
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
package "Immutable model binding" {
class "ModelRegistry" as Registry <<model>> {
+ resolve_binding(alias)
+ generation: int
--
Atomically resolves client, provider,
model, ModelConfig, generation.
}
class "ModelConfig snapshot" as ModelConfig <<model>> {
+ alias / provider / endpoint / static key
+ auth_mode
+ obo_audience
+ obo_scopes
--
static | entra_obo | entra_app | rfc8693_obo
}
class "ModelLane" as Lane <<model>> {
+ client / provider / model / capabilities
+ backend_auth_config: ModelConfig
+ backend_auth_resolver: Callable
}
class "Model definitions" as ModelTable <<storage>> {
DB + config-file definitions
encrypted protected fields
}
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
package "Per-call credential resolution" {
class "resolve_model_backend_auth_token()" as Resolver <<runtime>> {
+ alias + pinned ModelConfig
+ initiating principal_id
+ ConfigStore + mint client
→ dynamic token | None | fail closed
}
class "Model mint client" as Mint <<runtime>> {
+ mint_model_obo_token_sync(...)
+ mint_app_token_sync(...)
--
Cached by alias / principal / grant leg;
retains refusal cause for diagnostics.
}
class "OIDC / OBO protected state" as OBOState <<storage>> {
encrypted user refresh credential
deployment Fernet key
configured grant profile
}
class "lane_call_client()" as CallClient <<runtime>> {
cancel check before mint
resolve once per plant call
cancel check after mint
client.with_options(api_key=token)
}
class "Provider SDK request" as ProviderCall <<runtime>> {
Anthropic: x-api-key
OpenAI-style: Authorization Bearer
}
}
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
RequestAuth --> JWT : validates
RequestAuth --> ApiToken : validates
RequestAuth --> UserTables : lookup + permissions
RequestAuth --> AuthResult : returns
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
ModelTable --> Registry : load / hot reload
Registry --> ModelConfig : immutable snapshot
Registry --> Lane : coherent binding
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
AuthResult --> Resolver : initiating principal
Lane --> Resolver : callable + pinned config
Resolver --> Mint : dynamic modes only
Mint --> OBOState : decrypt / grant policy
CallClient --> Lane
CallClient --> Resolver
CallClient --> ProviderCall : cloned SDK client
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/workstreams/{ws_id}/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
note right of Resolver
**Mode policy**
• static: return None; registry client's explicit key remains.
• entra_obo / rfc8693_obo: require an effective principal. HTTP
turns pin the authenticated initiator; single-user internal lanes
may use their session owner. Never borrow another generation's identity.
• entra_app: use deployment app identity, no user required.
• rfc8693_obo alone sends obo_scopes; each dynamic mode is paired
with its required Entra or RFC 8693 grant profile.
end note
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
note bottom of CallClient
Dynamic credentials are minted at dispatch, not cached in the registry
snapshot. Endpoint, audience, scopes, auth mode, and static-key presence stay
pinned to the same ModelConfig generation as the SDK client. The global
model.auth_fail_closed policy is read live on every mint. A Stop that wins
before or during mint prevents model bytes from being sent afterward.
end note
note left of JWT
**Console Proxy Token Minting**
When proxying requests to server nodes:
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
2. Proxy mints new JWT (aud: turnstone-server)
with real user_id, scopes, permissions
3. src: "console-proxy" for audit traceability
4. 5-minute expiry (fresh per request)
5. Fallback: ServiceTokenManager if no user context
note bottom of ProviderCall
If minting fails, a configured fail-closed deployment or a keyless alias
raises BackendAuthUnavailableError. A dynamic alias with an explicit static
key may fall back only when policy allows. Authentication refusal is not a
backend-health failure and does not walk to a static fallback model.
end note
@enduml
docs/diagrams/16-channel-architecture.puml
+30 -14
View File
@@ -82,10 +82,12 @@ class "DiscordBot" as Bot <<service>> {
}
class "ChannelRouter" as Router <<service>> {
+resolve_route(platform, channel_id)
-> ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
+get_or_create_workstream(channel_type, channel_id)
+_is_ws_live(ws_id)
+send_message(ws_id, message)
+send_approval(ws_id, ...)
+lookup_ws_id(channel_type, channel_id)
+resolve_user(channel_type, channel_user_id)
-> user_id | None
--
Maps channels -> workstreams
@@ -93,6 +95,16 @@ class "ChannelRouter" as Router <<service>> {
Caches routes in memory
}
class "turnstone-console router" as ConsoleRouter <<server>> {
POST /v1/api/route/workstreams/new
GET /v1/api/route/workstreams/{ws_id}/live
POST /v1/api/route/workstreams/{ws_id}/send
POST /v1/api/route/workstreams/{ws_id}/approve
GET /v1/api/route?ws_id=...
--
Multi-node rendezvous + durable overrides
}
' -- Server --
class "turnstone-server" as Server <<server>> {
POST /v1/api/workstreams/{ws_id}/send
@@ -148,7 +160,9 @@ Bot --> Router : on_message\non_interaction
Router --> CU : resolve identity
Router --> CR : resolve / register route
Router --> Server : POST /v1/api/workstreams/{ws_id}/send\nPOST /v1/api/workstreams/{ws_id}/approve\nPOST /v1/api/workstreams/new
Router --> Server : single-node/direct mode\ncreate + send + approve
Router --> ConsoleRouter : multi-node mode\nroute create/live/send/approve/lookup
ConsoleRouter --> Server : routed HTTP to owning node
Bot --> Server : GET /v1/api/workstreams/{ws_id}/events\n(SSE via httpx-sse)
Server --> Bot : SSE event stream
@@ -156,7 +170,7 @@ Bot --> Discord : reply / embed\nbutton callback
Slack --> SlackBot : socket-mode\nevents
SlackBot --> Router : on_message / on_action
SlackBot --> Server : POST /v1/api/workstreams/{ws_id}/send\nGET /v1/api/workstreams/{ws_id}/events
SlackBot --> Server : owning-node SSE after route lookup
SlackBot --> Slack : post / update\nBlock Kit button callbacks
Teams .[hidden]. Slack
@@ -175,19 +189,21 @@ note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter resolves channel -> ws_id
(or creates new workstream)
3. ChannelRouter gets or creates channel -> ws_id
(direct server or multi-node console router)
4. ChannelRouter resolves platform user -> user_id
via channel_users table
5. Router sends POST /v1/api/workstreams/{ws_id}/send to server
5. Router sends through the configured server/console SDK
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no active SSE listener)
2. Existing ws_id reused directly from route
**Stale-route recovery (evicted workstreams)**
1. Route health check reports the old ws unavailable
2. Existing ws_id becomes the fork source
3. POST /v1/api/workstreams/new with
resume_ws=<ws_id>
4. Server resumes atomically during creation
5. SSE emits WorkstreamResumedEvent -> thread
4. Server atomically clones source history/config/
persona/project/attachment refs into a new ws_id
5. Router stores the new destination route; source is unchanged
6. If the source was pruned, retry one fresh create
end note
note right of Server
docs/diagrams/18-watch-architecture.puml
+2 -2
View File
@@ -11,7 +11,7 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
@@ -130,7 +130,7 @@ note right : action="cancel" (auto-approve)
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures WorkstreamManager
2. restore_fn closure captures SessionManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
docs/diagrams/22-judge-architecture.puml
+100 -195
View File
@@ -1,218 +1,123 @@
@startuml
!theme plain
title Turnstone — Intent Validation (Judge) Architecture
title Turnstone — Intent Judge, Concurrent Approval Cycles, and Output Guard
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<judge>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<fs>> #F5F5F5
}
skinparam sequenceArrowThickness 1.5
skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "ChatSession\n(session.py)" as Session <<session>>
participant "IntentJudge\n(judge.py)" as Judge <<judge>>
participant "LLM Provider\n(provider)" as LLM <<judge>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
participant "Filesystem" as FS <<fs>>
participant "ChatSession\ngeneration N" as Session
participant "SessionUIBase" as UI
participant "IntentJudge" as Judge
participant "model_turn()\n(pinned ModelLane)" as Model
participant "Operator / client" as Operator
participant "OutputGuardJudge" as Guard
database "StorageBackend" as Storage
== Tool Call Requires Approval ==
== Intent assessment begins during preparation ==
Session -> Session : _prepare_tool_calls()
note right
Tool calls parsed from
LLM response. Auto-approved
tools dispatched immediately.
Remaining items need approval.
Session -> Session : prepare each tool item independently\nattach principal + cancel witness
Session -> Judge : evaluate(items, callback, cancel_ref)
activate Judge
Judge -> Judge : synchronous heuristic verdict\nfor each call (first matching rule)
Judge --> Session : heuristic verdicts + daemon cancel event
Session -> UI : cache / publish heuristic assessments
Session -> Storage : persist heuristic intent verdicts
note over Judge, Model
The judge owns an immutable resolved binding. Registry/config generations
are freshness watermarks: an effective lane change replaces the judge for
the next batch, while in-flight work keeps the lane it started with.
Dynamic backend auth is resolved for this batch's initiating principal.
end note
Session -> Session : _evaluate_intent(pending_items)
== Tier 1: Heuristic (synchronous, sub-ms) ==
Session -> Judge : evaluate(items, messages, callback)
Judge -> Judge : evaluate_heuristic()\nfor each item
note right
**36 rules (first match wins):**
Critical (0.90, deny): rm /, mkfs,
dd, pipe-to-shell, chmod 777 /,
write/edit /etc/ .ssh/,
download-then-execute chains
High (0.80, review): sudo, kill -9,
destructive git, DROP TABLE,
secrets, HTTP mutations, ssh/scp,
browser+data-export, transitive
install, control-plane mutation
Medium (0.70, review): content
ingestion, interpreter exec,
cloud CLI mutations, pkg install,
write_file, MCP tools, docker ops
Low (0.85, approve): read_file,
list_directory, search, recall,
tool_search, read_resource,
web_search, read-only bash
Default: medium, 0.50, review
end note
Judge --> Session : heuristic_verdicts[]
Session -> Session : attach _heuristic_verdict\nto each pending item
Session -> UI : SSE: approve_request\n{items: [{verdict: ...}],\n judge_pending: true}
note right
Heuristic verdict displayed
immediately as risk badge.
Spinner shown while LLM
judge evaluates.
end note
Session -> Storage : create_intent_verdict()\nfor each heuristic verdict
== Tier 2: LLM Judge (daemon thread, async) ==
Judge -> Judge : spawn daemon thread\n"intent-judge"
note over Judge, LLM
**Context preparation:**
1. FIFO-truncate conversation history
to max_context_ratio of context window
2. Append tool call details as user message
3. System prompt defines judge role + JSON schema
end note
loop up to 3 turns (timeout budget)
Judge -> LLM : model_turn(lane, judge_turns,\ntools=[read_file, list_directory])\nvia drained create_streaming
LLM --> Judge : ModelTurnResult
alt tool_calls present (turn < 3)
Judge -> Judge : _exec_read_only_tool()
note right
**Security hardening:**
Blocked: /etc/, /root/,
/proc/, /sys/, /dev/,
.ssh, .gnupg, .aws,
*.pem, *.key, *.p12
File cap: 32KB
Dir cap: 200 entries
end note
Judge -> FS : read_file / list_directory
FS --> Judge : file contents
Judge -> Judge : append tool result\nto judge_messages
else text response (final verdict)
Judge -> Judge : _parse_verdict()
note right
**4-stage JSON parsing:**
1. Direct JSON.loads
2. Markdown code block
3. Brace-counting
4. Regex field extraction
end note
par LLM judge daemon
loop bounded turns / deadline
Judge -> Model : model_turn(judge lane, canonical Turns,\nread-only evidence tools, cancel_ref)
Model --> Judge : ModelTurnResult
alt evidence tool requested
Judge -> Judge : execute bounded read_file / list_directory
else verdict text
Judge -> Judge : parse + arbitrate against heuristic
end
end
Judge --> UI : on_intent_verdict(verdict, judge generation)
UI -> Storage : persist LLM verdict / audit update
else approval path continues
Session -> UI : approve_tools(items) with one\nSmart Approval config snapshot
end
== Tier 3: Arbitration ==
== Policy, Smart Approval, and human gate ==
Judge -> Judge : compare confidence:\nLLM vs heuristic
note right
Only deliver LLM verdict
if confidence > heuristic.
Otherwise heuristic stands.
end note
alt LLM confidence > heuristic confidence
Judge -> Session : callback(llm_verdict)
Session -> UI : SSE: intent_verdict\n{tier: "llm", ...}
note right
UI replaces heuristic badge
with LLM verdict. Spinner
resolves to final assessment.
end note
Session -> Storage : create_intent_verdict()\nfor LLM verdict
UI -> UI : apply explicit policy / skill / always / blanket bypasses
opt Smart Approvals enabled
UI -> UI : wait within captured deadline for this batch's verdicts
UI -> UI : auto-approve only recommendation=approve\nand confidence >= captured threshold
UI -> Storage : persist auto-approval reason and decision
end
== User Decision ==
UI -> Session : resolve_approval(\napproved, feedback)
Session -> Storage : update_intent_verdict(\nverdict_id, user_decision)
note right
All tracked verdicts
(heuristic + LLM) updated
with "approved" or "denied".
Swap-and-clear avoids racing
with daemon judge thread.
end note
== Tool Execution ==
Session -> Session : _execute_tools()
note right
Tools execute with
user approval.
end note
== Output Guard (synchronous, time-budgeted) ==
Session -> Session : _evaluate_output()\nfor each tool result
note right
**Priority-ordered checks (5s budget):**
P1: Prompt injection (role injection,
override phrases, instruction tags)
P2: Credential leakage (API keys,
PEM blocks, connection strings)
P3: Encoded payloads (data URIs,
hex shellcode)
P4: Adversarial URLs (cloud metadata,
credential query params)
P5: System info disclosure (private
IPs, sensitive paths)
Annotates + optionally redacts.
Does NOT gate.
end note
alt output_warning flags detected
Session -> UI : SSE: output_warning\n{call_id, risk_level, flags,\nfunc_name, redacted}
note right
Credential values replaced
with [REDACTED:<type>] before
output enters conversation.
sanitized text excluded from
SSE payload (defense in depth).
end note
UI -> Storage : record_output_assessment()\nfire-and-forget persistence
note right
Stored: flags, risk_level,
annotations, output_length,
redacted (bool). Raw tool
output is never stored.
end note
alt human-gated items remain
UI -> UI : acquire publication lease; register ApprovalCycle\n(cycle_id, call_ids, event, result, witnesses)
UI -> Operator : approve_request with cycle_id + item verdicts
Operator -> UI : approve / deny by cycle_id or call_id
UI -> UI : atomically claim exactly one unresolved cycle
UI -> Operator : approval_resolved
UI --> Session : decision + optional feedback
UI -> Storage : stamp tracked verdicts with operator decision
else every item bypassed / auto-approved
UI -> Operator : tool_info with exact auto_approve_reason
UI --> Session : approved
end
== Lifecycle ==
note right of UI
Parallel task agents may register several ApprovalCycles. Each cycle owns
its own Event and result slot. A legacy selector-less decision targets the
oldest cycle; double resolution is a no-op. Cached LLM verdicts carry their
judge generation, so reused provider call ids cannot satisfy a new cycle.
end note
note over Session, Judge
**Lazy initialization:**
IntentJudge created on first approval if judge_config.enabled.
Re-uses session's provider/client by default (self-consistency).
Cross-model: separate provider/client from [judge] config.
== Cancellation boundary ==
**Sub-agent exemption:**
Task sub-agents skip intent validation entirely.
opt Stop / close / force-successor
Session -> Judge : abort all judge events owned by the cancelled operation
Session -> UI : resolve_all_approvals(False, "cancelled")
UI -> UI : block new admission leases; wait for admitted bundles;\nclaim only cycles whose cancellation witness is aborted
UI -> Operator : one cancelled resolution per claimed cycle
note over Session, UI
A Stop can win before cycle registration, during publication, or while a
click resolves. The witness + admission sweep makes exactly one terminal
outcome visible; a successor generation's new cycle is not swept.
end note
end
**Output guard:**
Runs when judge_config.output_guard is true (default).
Credential redaction when judge_config.redact_secrets is true.
note over Judge
Normal operator resolution does not necessarily cancel judge inference.
With cancel_on_approval=false, the daemon finishes and late verdicts remain
auditable. With it enabled, the batch event stops remaining judge work.
end note
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store risk_level,
scan_report, scan_version for install-time risk assessment.
deactivate Judge
== Tool output guard ==
Session -> Session : execute admitted tools; truncate each result
Session -> Guard : evaluate(result, tool context, cancel event)
activate Guard
Guard -> Guard : heuristic checks first
opt LLM guard enabled and time remains
Guard -> Model : model_turn(output-guard lane, bounded prompt, cancel_ref)
Model --> Guard : structured verdict
end
Guard --> Session : assessment / redaction / warning
deactivate Guard
Session -> Session : re-check generation N before folding result
Session -> UI : output warning (no raw secret payload)
Session -> Storage : persist assessment + guarded Tool Turn metadata
note over Guard, Storage
Output-guard objects also pin model/config lanes. Replacement retires the
old object but lets admitted evaluations drain before its private client is
closed. A cancelled or superseded evaluation cannot fold into the successor
trajectory. Raw pre-redaction secrets are never stored in assessment rows.
end note
@enduml
docs/diagrams/23-memory-architecture.puml
+1 -1
View File
@@ -13,7 +13,7 @@ skinparam participant {
participant "ChatSession\n(session.py)" as Session <<session>>
participant "MemoryFacade\n(memory.py)" as Facade <<facade>>
participant "MemoryRelevance\n(memory_relevance.py)" as Relevance <<facade>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "Server API\n(server.py)" as API <<api>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
docs/diagrams/24-settings-architecture.puml
+8 -6
View File
@@ -13,7 +13,7 @@ skinparam participant {
participant "Server\n(main)" as Server <<session>>
participant "ConfigStore\n(config_store.py)" as Store <<config>>
participant "SettingsRegistry\n(settings_registry.py)" as Registry <<config>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "StorageBackend\n(SQLite / PostgreSQL)" as Storage <<storage>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
participant "ChatSession\n(session.py)" as Session <<session>>
@@ -57,9 +57,10 @@ else key not in cache
Store --> Session : default value
end
note right of Session
Settings are captured once
at workstream creation.
Not re-read on every turn.
Most session settings are captured once
at workstream creation. Documented live readers
(including model.auth_fail_closed per mint)
apply immediately.
end note
== Phase 3: Admin API — List / Schema ==
@@ -128,8 +129,9 @@ Store -> Storage : get_system_settings_bulk(node_id)
Storage --> Store : all settings
Store -> Store : rebuild cache,\nswap atomically,\nincrement _version
note right
Existing sessions: unchanged
(frozen at creation time).
Most existing-session settings are unchanged
(frozen at creation time); documented
live readers apply immediately.
New sessions: pick up
updated values immediately.
end note
docs/diagrams/architecture-overview.svg
+5 -5
View File
@@ -88,7 +88,7 @@
<rect x="300" y="151" width="160" height="2" fill="#161b22"/>
<text x="380" y="178" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<line x1="318" y1="190" x2="442" y2="190" stroke="#30363d" stroke-width="1"/>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">hash-ring router</text>
<text x="380" y="208" text-anchor="middle" fill="#8b949e" font-size="9">FNV-1a rendezvous router</text>
<text x="380" y="223" text-anchor="middle" fill="#8b949e" font-size="9">cluster dashboard</text>
<text x="380" y="238" text-anchor="middle" fill="#8b949e" font-size="9">reverse proxy</text>
<line x1="318" y1="250" x2="442" y2="250" stroke="#30363d" stroke-width="1"/>
@@ -115,7 +115,7 @@
<rect x="608" y="162" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="180" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
<text x="700" y="206" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
</g>
<!-- Node B -->
@@ -130,7 +130,7 @@
<rect x="608" y="282" width="184" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="700" y="300" text-anchor="middle" fill="#8b949e" font-size="9">turnstone-server :8080</text>
<!-- Tools label -->
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">19 tools + MCP</text>
<text x="700" y="326" text-anchor="middle" fill="#484f58" font-size="8">model lanes + tools / MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
@@ -171,7 +171,7 @@
<rect x="590" y="450" width="220" height="5" fill="#bc8cff"/>
<rect x="590" y="453" width="220" height="2" fill="#161b22"/>
<text x="700" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
<text x="700" y="492" text-anchor="middle" fill="#8b949e" font-size="9">workstreams, turns, config, auth</text>
</g>
<text x="700" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
@@ -239,7 +239,7 @@
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#3fb950" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (hash-ring bucket lookup)</text>
<text x="54" y="477" fill="#484f58" font-size="9">control plane: client &#x2192; console &#x2192; server node (FNV-1a rendezvous placement)</text>
<circle cx="44" cy="494" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">data plane: client &#x2192; server node (direct SSE, node_url from create response)</text>
<circle cx="44" cy="514" r="3" fill="#f47067" opacity="0.6"/>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

docs/diagrams/png/01-system-context.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:881a8b9bce67b5af9a52d5e50deaa72351cd99c76f18aad5caeb2b61131ca1af
size 119798
oid sha256:b8c1460440784f07e30afea32d4ee17687627df46a24003d761ad79c2676a361
size 169499
docs/diagrams/png/02-package-structure.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
oid sha256:66847ccdf10ef2bd04e93bc0d3924a56ce28462ec9e76a383b53aee4500755e8
size 631799
docs/diagrams/png/03-core-engine-classes.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
oid sha256:e1431edf3891785b922c52b7897e3af5d39ba9973a815f892d6afdb762c5297b
size 612662
docs/diagrams/png/04-conversation-turn.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d9c7769a600c38e6387390e6c42db8152e0f80c31d17b2218f7f636b71c7b868
size 355459
oid sha256:79299b25ccc10484af13684a89ed9457abb9bc1781604bf6a9000ccb84362e55
size 311107
docs/diagrams/png/05-tool-pipeline.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:23ca090b5656baaf70820cbe4ab6c27f0a3a02e18b4db0695614cf9489c23980
size 281440
oid sha256:1b3b7b745f6006ee73d4b31fa598faa0d71ffb74ce349ab08eb3ce09ded506c3
size 266294
docs/diagrams/png/09-workstream-states.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:04d2069a9b5155ad1e7d842147fd78535ad9106d6856520439c33a9868a47499
size 156694
oid sha256:59dc8f92ca83c4354d089b75c6d5075d4a808277150b271c868dab99b3ac02ac
size 333165
docs/diagrams/png/12-deployment.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a872556d111185f4531d1b68ee892b4ce5042d7ccf277e2cad08beb6932c9803
size 191144
oid sha256:16e3f3bfa0a6af637f7a9fb6765d594eb598428679c88a429c096c3dbae931e4
size 181185
docs/diagrams/png/14-storage-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b047cdc318c505f0f0895a65e14c5cc7552716053055cca57fa0a77db150e618
size 255458
oid sha256:1a510eeaabf4ed8dab3b268c8f6bb5b7fef629a664361f7fb5614a6b498db36e
size 294415
docs/diagrams/png/15-auth-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
oid sha256:ea86c6b6c68ed96f6fd18543d2e7a873f4715332cc3a7d4df167392f668a2de7
size 232403
docs/diagrams/png/16-channel-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21
size 415473
oid sha256:edf02b97e1e1287ebba9e74b9858474dcda42e5542656e505ea133a5b2416f47
size 402992
docs/diagrams/png/18-watch-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
oid sha256:aa9ca9a367c79159a26d1ec544b20fc7118a49082d72f7ee0edbaa85608d49fc
size 238991
docs/diagrams/png/22-judge-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79a690c466a5d6f6d4292d78a27b9474e9e9c1373fa80e17dfe37700238c8af8
size 382508
oid sha256:dde4f417956534a70b0e37b24cfe9de383897780669c9da81833c8084d572dfe
size 269928
docs/diagrams/png/23-memory-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c89628ed917dfd576c1af75c68fe5fed9beadaaee9dcea7aa7a1643867c4f1b9
size 344323
oid sha256:bd7fe8bf5c2b56b075453a316e54d61214b0ae912517d9cad6c1e88785aac722
size 300010
docs/diagrams/png/24-settings-architecture.png
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06fe076f0835a891e00afc804fd1805196ebde9fc0d34998c7873e87287f982b
size 346887
oid sha256:0455e0dec36ebb8bcfdadcf327a1dd24ddbdcdd8df211c08918180842497727e
size 318681
docs/docker.md
+6
View File
@@ -186,6 +186,12 @@ overrides.
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
> **Lifecycle upgrade:** the release that introduces hidden deferred-create
> reservations must be deployed as a coordinated cohort across every server
> sharing PostgreSQL; older processes do not understand `state='creating'`.
> Drain create traffic until the cohort is upgraded. See
> [PgBouncer: deferred workstream creation](pgbouncer.md#upgrade-note-deferred-workstream-creation).
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
docs/governance.md
+8 -5
View File
@@ -16,15 +16,18 @@ The permission model has two layers:
2. **Permissions** (granular) — named permission strings checked per-endpoint by
`require_permission()`.
**Built-in roles** (seeded by migration 008):
**Built-in roles** (seeded by migration 008 and extended by later feature
migrations):
| Role | Permissions |
|------|-------------|
| admin | read, write, approve, admin.users, admin.roles, admin.orgs, admin.policies, admin.skills, admin.audit, admin.usage, admin.schedules, admin.watches, tools.approve, workstreams.create, workstreams.close |
| operator | read, write, workstreams.create, workstreams.close |
| admin | Admin-default baseline: ordinary admin, lifecycle, tool-approval, coordinator, project, and persona capabilities. Explicit opt-in capabilities such as `model.skills.write` remain ungranted. |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
Custom roles can be created with any subset of the valid permissions. Built-in
role permission overrides can grant or revoke individual capabilities, so the
admin console is authoritative for the effective set on a deployment.
The `persona.create` / `persona.read` / `persona.write` family gates
persona administration; migration `063` seeds all three onto
`builtin-admin`, and any role can be granted them through the standard
@@ -67,7 +70,7 @@ etc.) since workstream templates were merged into the skills system in v0.8.0.
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--skill <name>` CLI flag, `skill` field on
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
`POST /v1/api/workstreams/new`, console launcher dropdown, scheduled task
config, and channel adapter config. An explicit skill *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
docs/judge.md
+50 -24
View File
@@ -17,7 +17,9 @@ evaluation:
read-only tool access. Runs on a daemon thread and delivers its verdict
progressively.
The verdict is purely advisory -- the user always makes the final decision.
The verdict is advisory by default. The opt-in Smart Approvals mode can use a
completed, high-confidence LLM `approve` verdict to make the decision
automatically under the fail-closed rules below.
The heuristic verdict is attached to the `approve_request` SSE event immediately.
The LLM verdict arrives later via an `intent_verdict` SSE event, allowing the
@@ -40,27 +42,41 @@ api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # seconds (generous for local models)
timeout = 120.0 # per judge turn; also caps the Smart Approvals wait
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
```
### Smart Approvals
With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.
With `smart_approvals = true` (off by default), a pending batch is approved
automatically — no operator prompt — only when **every** call has a completed
LLM verdict recommending `approve` at or above `confidence_threshold`. The
decision is batch-atomic: one uncertain sibling sends the entire parallel batch
to a human rather than executing the safe-looking subset piecemeal.
Every other outcome reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), a
missing/duplicate call ID, an unjudged sibling, and any call the deterministic
heuristic rules explicitly flagged `deny` or `critical`. That heuristic floor
blocks only explicit danger verdicts — it is **not** a general "never lower the
heuristic" rule. The heuristic's default for an unmatched tool is `review`, and
letting a confident LLM upgrade that default is the feature's purpose.
The Smart Approvals enabled flag, threshold, and bounded verdict wait are
captured as one immutable snapshot when each gate batch starts. A settings
reload takes effect on the next batch, while concurrent main-loop and
task-agent gates cannot mix fields from different reload generations. Stop
wakes a batch still waiting for verdicts and is linearized against the final
auto-approval commit: if Stop wins, no `smart_approval` decision or audit row
is recorded for tools that did not cross the gate.
The verdict wait is capped by the snapshot's `judge.timeout`; the judge may
continue evaluating advisory verdicts after that gate falls back to a human.
Requires the judge to be enabled. Auto-approved calls are tagged
`smart_approval` in the dashboard and audit trail. Smart Approvals applies to
the web and coordinator surfaces, not the interactive CLI.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
@@ -235,14 +251,15 @@ calls for approval, it calls `_evaluate_intent()` which:
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.
still stream to the UI and persist, stamped with the decision. A newer main-loop
batch, session close, or explicit Stop retires the old generation; unfinished
items degrade to `llm_fallback` verdicts. A judge/model binding edit prevents
reuse on the next batch, while already-started calls stay pinned to the binding
they began with. With `cancel_on_approval = true`, an ordinary gate decision
additionally aborts the remainder immediately, trading verdict completeness
for inference savings — recommended when the judge shares a single local
inference backend with the session model. Explicit Stop always cancels every
live judge generation, regardless of this preference.
Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
@@ -258,6 +275,15 @@ siblings would otherwise make each other's verdicts look stale); per-cycle
generation checks enforce staleness instead, and `judge.cancel_on_approval`
fires per gate exactly like the main loop.
Several parallel task agents can therefore leave several approval cycles live
on one workstream. Each cycle owns its event, result, verdict set, and
`cycle_id`; a decision targets exactly one cycle by `cycle_id` or member
`call_id` (selector-less legacy clients resolve the oldest). Workstream Stop or
close performs a workstream-wide denial sweep over all cycles belonging to the
cancelled operation. A force-cancel successor's newly registered cycle carries
a fresh operation witness and is not accidentally denied by the predecessor's
late sweep.
---
## Storage and Audit
docs/personas.md
+2 -2
View File
@@ -46,8 +46,8 @@ reads only the stamp:
`creative_mode` set are converted by migration `063` into full
`writer` stamps, so they resume as writing sessions rather than as
legacy defaults.
- Forking (`resume_ws` on create) resumes the source's stamped persona; the
fork does not re-resolve.
- Forking (`resume_ws` on create) clones the source's stamped persona into the
new workstream; the fork does not re-resolve it.
## Seed personas
docs/pgbouncer.md
+44 -8
View File
@@ -15,11 +15,19 @@ down to a small number of real database connections.
## Why PgBouncer works well with turnstone
All turnstone database operations are short-burst queries: acquire a
connection, execute 1–3 statements, commit, release. No operation holds
a connection for more than a few milliseconds. This makes **transaction
pooling mode** ideal — PgBouncer assigns a real connection only for the
duration of each transaction, then returns it to the pool.
Most turnstone database operations are short-burst queries: acquire a
connection, execute a small transaction, commit, release. Workstream forks are
the deliberate exception: they clone the source's checkpoint-bounded history
and configuration and retain its attachment references in one transaction.
PostgreSQL runs that clone at `SERIALIZABLE` isolation and retries serialization
or deadlock conflicts as a whole. A large fork can therefore hold its assigned
server connection longer than an ordinary message write.
This still makes **transaction pooling mode** the right fit — no operation
depends on server-session state, and PgBouncer returns the connection as soon
as the transaction finishes. Size and monitor the server pool with concurrent
fork traffic in mind rather than assuming every transaction completes in a few
milliseconds.
| Cluster size | Client connections (max) | PgBouncer server connections needed |
|--------------|------------------------|-------------------------------------|
@@ -143,9 +151,11 @@ PgBouncer (which then multiplexes to PostgreSQL):
| `TURNSTONE_DB_URL` | — | Connection URL (point at PgBouncer, not PostgreSQL directly) |
The default pool of 2 + 3 overflow = 5 connections per process is
intentionally small to support large clusters. You should not need to
increase this — turnstone's database operations are all short-burst
context-managed queries that hold connections for milliseconds.
intentionally small to support large clusters. Most deployments should not
need to increase it. If operators create many large forks concurrently, watch
PgBouncer's `cl_waiting` and PostgreSQL transaction latency before changing
the per-process pool; adding client-side connections cannot help once the
PgBouncer server pool is saturated.
SQLAlchemy `pool_pre_ping` is enabled, so stale connections (e.g. after
PgBouncer restarts) are automatically detected and replaced.
@@ -177,6 +187,32 @@ Key metrics to watch:
- **`sv_active`** — active server (PostgreSQL) connections. Should stay
below PostgreSQL `max_connections`.
Short `cl_waiting` spikes during large workstream forks can be normal. Sustained
waiters accompanied by long serializable transactions indicate fork/storage
load, not an SSE or HTTP client-pool problem.
---
## Upgrade note: deferred workstream creation
The workstream lifecycle now uses durable, hidden `state='creating'`
reservations while session construction, upload validation, and optional fork
cloning complete. Older server processes do not understand that private state:
against the same database they may resolve, list, open, or prune a reservation
before its new owner publishes it.
For the upgrade that introduces deferred creation, drain create traffic and
upgrade all server processes sharing the database as one cohort. Do not resume
creates until no older server process remains. The change needs no manual
schema migration, but it is not safe to treat mixed lifecycle implementations
as an ordinary rolling-upgrade state.
A `creating` row should be transient and absent from normal APIs and cluster
events. If one persists after a process crash, inspect the corresponding
`ws.create.*` and `session_mgr.commit_create.*` logs before cleanup. Do not
promote it to `idle` manually: its history, configuration, attachment
references, or lifecycle publication may be incomplete.
---
## Troubleshooting
docs/sdk.md
+43 -20
View File
@@ -50,6 +50,7 @@ with TurnstoneServer("http://localhost:8080") as client:
import asyncio
from turnstone.sdk import AsyncTurnstoneServer
async def main():
async with AsyncTurnstoneServer("http://localhost:8080") as client:
await client.login(username="alice", password="s3cret")
@@ -58,6 +59,7 @@ async def main():
if event.type == "content":
print(event.text, end="", flush=True)
asyncio.run(main())
```
@@ -69,16 +71,16 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
|----------|--------|---------|
| **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` |
| | `dashboard()` | `DashboardResponse` |
| | `create_workstream(*, name, model, auto_approve, skill, persona, initial_message, attachments)` | `CreateWorkstreamResponse` |
| | `create_workstream(*, name, model, auto_approve, resume_ws, skill, persona, initial_message, project_id, attachments, ...)` | `CreateWorkstreamResponse` |
| | `close_workstream(ws_id)` | `StatusResponse` |
| **Attachments** | `upload_attachment(ws_id, filename, data, *, mime_type=...)` | `UploadAttachmentResponse` |
| | `list_attachments(ws_id)` | `ListAttachmentsResponse` |
| | `get_attachment_content(ws_id, attachment_id)` | `bytes` |
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `approve(*, ws_id, approved, feedback, always, cycle_id, call_id)` | `ApproveResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `CancelResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
@@ -100,7 +102,7 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `snapshot()` | `ClusterSnapshotResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona)` | `ConsoleCreateWsResponse` |
| | `create_workstream(*, node_id, name, model, initial_message, skill, persona, resume_ws)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
@@ -125,11 +127,10 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| Type | Class | Key Fields |
|------|-------|------------|
| `connected` | `ConnectedEvent` | `model`, `model_alias`, `skip_permissions` |
| `history` | `HistoryEvent` | `messages` |
| `content` | `ContentEvent` | `text` |
| `reasoning` | `ReasoningEvent` | `text` |
| `tool_info` | `ToolInfoEvent` | `items` |
| `approve_request` | `ApproveRequestEvent` | `items` |
| `approve_request` | `ApproveRequestEvent` | `cycle_id`, `items` |
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error` |
| `tool_output_chunk` | `ToolOutputChunkEvent` | `call_id`, `chunk` |
| `status` | `StatusEvent` | `prompt_tokens`, `total_tokens`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
@@ -138,9 +139,15 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state` ∈ `running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `approval_resolved` | `ApprovalResolvedEvent` | `cycle_id`, `call_ids`, `approved`, `feedback`, `always` |
| `cancelled` | `CancelledEvent` | — |
Current servers bootstrap conversation history through
`GET /v1/api/workstreams/{ws_id}/history` before the SSE stream; they do not
emit a `history` event. `HistoryEvent` remains deserializable only for
compatibility with older servers. The Python client does not yet expose a
typed helper for this bootstrap endpoint.
**Global events** (from `stream_global_events()`):
| Type | Class | Key Fields |
@@ -168,12 +175,12 @@ The `send_and_wait()` method returns a `TurnResult` that aggregates the full res
```python
result = client.send_and_wait("Hello", ws_id, timeout=60)
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
result.content # Full text response
result.reasoning # Chain-of-thought (if shown)
result.tool_results # List of (tool_name, output) tuples
result.errors # Any error messages
result.ok # True if no errors and not timed out
result.timed_out # True if timeout expired
```
### Attachments
@@ -183,9 +190,7 @@ Upload files to a workstream and attach them to the next user turn:
```python
# Upload separately, then send a message — attachments auto-attach
with open("screenshot.png", "rb") as f:
att = client.upload_attachment(ws.ws_id, "screenshot.png",
f.read(),
mime_type="image/png")
att = client.upload_attachment(ws.ws_id, "screenshot.png", f.read(), mime_type="image/png")
client.send("What's wrong in this screenshot?", ws.ws_id)
# Or attach at workstream-creation time (multipart upload)
@@ -195,9 +200,7 @@ with open("notes.txt", "rb") as f:
ws = client.create_workstream(
name="triage",
initial_message="Summarize the notes",
attachments=[AttachmentUpload(data=f.read(),
filename="notes.txt",
mime_type="text/plain")],
attachments=[AttachmentUpload(data=f.read(), filename="notes.txt", mime_type="text/plain")],
)
```
@@ -206,6 +209,26 @@ Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8),
client so cluster-routed callers bind attachments to the owning node
before the request lands.
### Forking a workstream
`resume_ws` is the API's compatibility name for an atomic fork. It creates a
new workstream ID while the source remains unchanged:
```python
fork = client.create_workstream(
resume_ws=ws.ws_id,
name="analysis-branch",
initial_message="Try the alternative plan.",
)
assert fork.resumed
```
The server transaction clones the source's checkpoint-bounded history, saved
session configuration, persona, project, and attachment references. Do not
combine `resume_ws` with `attachments`; fork first, then upload to the new ID.
To rehydrate the original ID rather than branch it, call the server's
`POST /v1/api/workstreams/{ws_id}/open` endpoint.
### Error Handling
Non-2xx responses raise `TurnstoneAPIError`:
@@ -217,7 +240,7 @@ try:
client.send("hi", "bad_ws_id")
except TurnstoneAPIError as e:
print(e.status_code) # 404
print(e.message) # "Unknown workstream"
print(e.message) # "Unknown workstream"
```
---
docs/security.md
+53 -20
View File
@@ -64,15 +64,17 @@ Scopes are hierarchical — higher scopes imply all lower ones.
### Path-to-scope mapping
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` |
| POST | `/api/workstreams/{ws_id}/{send,cancel,close,delete,open,refresh-title,title,attachments}` | `write` |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` |
| Any | `/api/admin/*` | `approve` |
| Method | Path pattern | Required scope | Additional RBAC gate |
|--------|-------------|----------------|----------------------|
| GET | Any protected path | `read` | Endpoint-specific where documented |
| POST | `/api/command` | `write` | Project tenancy on the target workstream |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` | `workstreams.create` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/close` | `write` | `workstreams.close` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/approve` | `approve` | `tools.approve` or `admin.coordinator` |
| POST | `/api/workstreams/{ws_id}/{rewind,retry}` | `write` | `conversation.modify` |
| POST | Other `/api/workstreams/{ws_id}/...` mutation endpoints | `write` | Project tenancy and endpoint-specific gates |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` | Project tenancy on the target workstream |
| Any | `/api/admin/*` | `approve` | Matching `admin.*` permission |
Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
`/static/*`, `/shared/*`, `/docs`, `/openapi.json`, `/api/auth/login`,
@@ -84,7 +86,7 @@ Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
> See also: [Governance documentation](governance.md)
Scopes provide coarse endpoint-level access control. For finer-grained
enforcement, the governance layer adds 15 named permissions checked
enforcement, the governance layer adds named permissions checked
per-endpoint by `require_permission()`. Permissions are bundled into
roles; users are assigned roles via the `user_roles` join table.
@@ -98,8 +100,8 @@ Three built-in roles are seeded by migration 008:
| Role | Permissions |
|------|-------------|
| admin | All 15 permissions |
| operator | read, write, workstreams.create, workstreams.close |
| admin | Admin-default baseline (all ordinary admin and lifecycle permissions; explicitly opt-in capabilities remain ungranted) |
| operator | read, write, workstreams.create, workstreams.close, conversation.modify |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
@@ -107,6 +109,34 @@ Role creation and update validate permissions against a static allowlist.
Self-assignment is blocked, and assigning a role requires the caller to
hold a superset of the target role's permissions.
### Workstream lifecycle and project boundaries
The remote `/api/command` endpoint is conversation-local. It refuses
`/new`, `/workstreams`, `/resume`, and `/delete` because those local-CLI
helpers enumerate or mutate storage outside the HTTP resource gates. Remote
clients use the dedicated create, open, close, and delete endpoints instead;
`/rewind` and `/retry` have their own path-keyed, `conversation.modify`-gated
endpoints.
Passing `resume_ws` to create is an atomic **fork**, not an in-place resume.
It requires the ordinary create capability and source visibility. A private
project source is visible only to its workstream creator, project owner/member,
or authorized service-to-service cluster plumbing; denials use a not-found
response so guessed IDs do not become an existence oracle. The caller must also
be allowed to attach a new workstream to the source's current project. The
destination always inherits that effective project — a caller-supplied
`project_id` cannot re-file or declassify the conversation.
The canonical preflight atomically captures (and, for a legacy row, installs) a
private source-incarnation fence. The storage transaction compares that source
fence, rejects provisional sources, repeats the ACL/project check, and verifies
the persona/project construction snapshot, destination ownership and
incarnation, emptiness, and every referenced attachment before committing. A
source replacement, membership, project, persona, or destination-incarnation
race aborts the whole fork. Concurrent source-history writes serialize wholly
before or after the snapshot; no mixed or partially authorized history or
attachment reference becomes visible.
---
## Login Flows
@@ -441,12 +471,15 @@ Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
- **Permission forwarding** — granular RBAC permissions from the
console JWT are carried through to the server.
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
distinguish proxied requests from direct logins in audit logs.
For ordinary users the JWT `src` claim is set to `"console-proxy"`, allowing
servers to distinguish proxied requests from direct logins in audit logs.
Coordinator tokens retain `src="coordinator"` and their signed `coord_ws_id`;
the console service identity retains `src="console"` only when its validated
token also carries the unassignable `service` scope.
When no user context is available (auth disabled, or internal requests),
the proxy falls back to a `ServiceTokenManager` with service identity
`console-proxy` and full scopes.
the proxy falls back to a `ServiceTokenManager` with identity `console-proxy`,
`src="console"`, and `{read, write, approve, service}` scopes.
### Service-to-service authentication
@@ -455,8 +488,8 @@ JWTs when communicating with server nodes:
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
| Console collector | `console-collector` | `read`, `service` | `turnstone-server` | Node health polling and global event collection |
| Console proxy (fallback) | `console-proxy` | `read`, `write`, `approve`, `service` | `turnstone-server` | Proxied API calls when no user context |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
@@ -468,8 +501,8 @@ When the console creates a workstream (the normal path), the
authenticated user's `user_id` is forwarded in the HTTP payload when
calling the server's `POST /v1/api/workstreams/new`. The server
accepts a `user_id` from the request body **only when the caller is a
trusted service** — identified by `token_source` matching
`console-proxy` or `console`. Regular API callers cannot
trusted service** — identified by `token_source="console"` together with the
unassignable `service` scope. `console-proxy`, coordinator, and regular API callers cannot
override `user_id`; the server always uses their JWT identity.
Note that the channel gateway uses a distinct JWT audience
docs/settings.md
+39 -3
View File
@@ -134,6 +134,15 @@ delegated-mode rows and memo entries. `entra_app` rows belong to the shared
revocation, an already-minted app bearer remains usable until its recorded
expiry.
Each model call resolves its dynamic credential against the immutable model
definition snapshot that supplied that call's provider, client, endpoint, and
model ID. An admin edit can therefore never pair an old `base_url` with a new
audience, grant mode, or static-key fallback input. The principal and token
remain per-call/live; the connection and model-owned auth configuration move
together as one binding on the next operation. The deployment-wide
`model.auth_fail_closed` switch is intentionally read live on every mint, so an
operator can tighten fallback policy immediately without rebuilding sessions.
`obo_audience` and `obo_scopes` are literal and capped at 2048 characters
each. Environment-variable expansion is deliberately not applied, so the
allow-list decision cannot vary by node or expand beyond the persisted
@@ -217,7 +226,7 @@ initialization:
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `judge` | enabled, model, provider, base_url, api_key, smart_approvals, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
| `interface` | close_tab_action, theme |
| `skills` | discovery_url |
| `memory` | relevance_k, fetch_limit, max_content, nudge_cooldown, nudges |
@@ -421,10 +430,37 @@ reload.
**Behavior after reload:**
- New workstreams pick up updated values immediately (via `session_factory`)
- Existing sessions keep their frozen configuration (settings are captured at
workstream creation time, not read on every turn)
- Most workstream/session settings remain the snapshot captured at creation or
resume. Component docs call out deliberate live-read exceptions; for
example, Smart Approval settings are snapshotted coherently at the start of
each approval batch.
- Settings marked `restart_required=True` need a server restart to take effect
### Model-definition reloads
The Models tab has a separate live-reload contract from ordinary ConfigStore
settings. Existing sessions remember the concrete registry generation that
supplied their active alias and re-resolve that alias at the start of the next
send. Endpoint, provider, backend model ID, capabilities, extra parameters, and
backend-auth configuration are replaced as one immutable binding. In-flight
turns, judges, and task agents finish or cancel against the binding they
started with; an admin edit never tears one request across two definitions.
Sampling and other saved workstream configuration remain workstream state. A
model-definition edit does not silently rewrite a live workstream's chosen
temperature, reasoning effort, max tokens, skill, or persona. Use
`/model <alias>` (or create/fork a workstream) when an explicit session-level
model switch is intended.
If a live workstream's alias is deleted, its next send first attempts the
configured fallback chain. Without a usable fallback, the operator-facing
error names the removed alias and points interactive users to `/model`; adding
the alias back causes the next send to rebind without a process restart. If a
replacement client cannot be constructed, Turnstone logs one
`session.model_refresh_client_construction_failed` warning per registry
generation and retries only after another model reload, avoiding a rebuild
storm on every send.
---
## Migration from config.toml
docs/skills/import-conversation-history/SKILL.md
+106 -24
View File
@@ -1,7 +1,7 @@
---
name: import-conversation-history
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
version: 1.1.0
---
# Importing Conversation History into Turnstone
@@ -12,7 +12,7 @@ Source formats vary; the destination does not. Your job is to translate whatever
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
1. **Archive or resumable?** An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
@@ -25,13 +25,13 @@ Two tables carry the conversation:
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. The router hashes the **full ID** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `state` | yes | Register as `"closed"` while importing. Leave it closed for an archive; explicitly open it after commit for a resumable import. Never set `"running"` or `"creating"` directly. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `node_id` | no | Nullable creation-time service/liveness hint. It is not the routing key or durable owner and may become stale after membership changes. Let a routed create stamp it; a direct shared-storage import may leave it NULL. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
@@ -55,25 +55,65 @@ The internal format is **OpenAI-shaped**, even when the source was Anthropic or
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over
the **full `ws_id`** and the current live server set. For each node, Turnstone
computes 32-bit FNV-1a over the node ID, a NUL separator, and the full
workstream ID; it then applies the node weight and selects the highest score.
A live per-workstream override takes precedence.
- The live set comes from recent `services` heartbeats. Placement can therefore
change when nodes join, leave, change weight, or an override changes. There
is no stable prefix-derived placement to pre-compute or persist.
- `workstreams.node_id` is stamped at creation and is not updated as HRW
placement changes. It supports display and liveness-safe cleanup; the console
router does not use it as the ordinary ownership decision.
- For multi-node imports, create through the console routing proxy when the
lifecycle must be published, or write the history once through the cluster's
configured **shared storage backend**. Never partition rows across node-local
databases by ID prefix or by a one-time HRW result: a later membership change
can route the same full ID to another node.
- For single-node imports, HRW placement is degenerate; any valid `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
### 1. Quiesced storage import (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
Use the current `turnstone.core.storage.StorageBackend` protocol against the
same shared backend as the cluster. The destination must remain absent from all
in-memory session managers while rows are changing: a loaded `ChatSession`
holds its own trajectory and will not observe conversation rows inserted behind
it.
The safe sequence is:
1. Normalize and validate the complete source transcript before writing.
2. Call `register_workstream(..., state="closed")` and require a `True` return;
`False` means the caller-selected ID already exists, so abort rather than
appending to an unrelated workstream.
3. Insert the ordered conversation rows and attachment references.
4. Load the saved rows back and run the validation checklist below.
5. Leave an archive closed. For a resumable import, only now invoke the normal
`POST /v1/api/workstreams/{ws_id}/open` endpoint on the currently routed
node so the session hydrates from the complete transcript.
Do **not** create the destination through the web/SDK create endpoint before a
direct bulk import. Create publishes an empty live session. If that already
happened, close the workstream and confirm the manager-authoritative live probe
returns false before writing, then explicitly open it again after validation.
For attachment-free history, `save_messages_bulk(rows)` is the canonical
single-transaction insert primitive and bypasses the LLM round-trip entirely.
New attachment bytes require the per-row path described under
[Attachments](#attachments).
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
from turnstone.core.storage import get_storage # initialized by the host/import entry point
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage = get_storage()
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
inserted = storage.register_workstream(
ws_id=ws_id,
user_id=user_id,
name=name,
@@ -81,6 +121,8 @@ storage.create_workstream( # or whatever the project's exposed creator is — c
kind="interactive",
...
)
if not inserted:
raise RuntimeError(f"destination already exists: {ws_id}")
storage.save_messages_bulk([
{"ws_id": ws_id, "role": "user", "content": "Hello"},
@@ -94,7 +136,19 @@ storage.save_messages_bulk([
])
```
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column
internally, so you don't need to compute them per row. Verify the exact
`register_workstream` and message signatures in
`turnstone/core/storage/_protocol.py`; the Storage protocol, not the physical
table layout, is the source of truth.
**Multi-node note:** this path assumes `get_storage()` is connected to the
cluster's shared backend. Do not open a node-local database selected from the
current HRW result, and do not pre-create a live session through the console
routing proxy. After the shared-storage import commits, resolve the current
route and open the closed workstream on that node. Any stored `node_id`
describes creation-time placement, not a permanent shard that should receive a
separate copy.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
@@ -181,27 +235,48 @@ If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
- **Blob identity**: `attachment_id` is the lowercase SHA-256 hex digest of the
bytes. `workstream_attachments` stores that content-addressed blob and its
refcount; it has no workstream or message foreign key.
- **Message link**: the sole message-to-blob link is the ordered JSON ID list in
`conversations.attachments`.
- **No persisted staging lifecycle**: pending upload bytes live only in a
node's in-memory attachment buffer. The old persisted
`pending → reserved → consumed` lifecycle does not apply to storage imports.
Two import paths:
For new attachment bytes, preserve row order by calling `save_message()` for
each turn. It returns the `conversations.id`; for every attachment referenced by
that turn, call `save_attachment()` with its content hash and bytes, then call
`set_message_attachments(ws_id, message_id, ordered_ids)`. Each
`save_attachment()` call accounts for one reference, while
`set_message_attachments()` records the ordered link.
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
`save_messages_bulk(..., attachment_ids=[...])` is appropriate only when those
content-addressed blobs already exist: the bulk transaction retains their
references and writes the ordered lists. Do not first call `save_attachment()`
for a new reference and then pass the same reference to `save_messages_bulk()`;
both paths retain it and would double-count the refcount.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
SDK multipart create remains useful only for attachments on a new first turn;
it publishes a live session and is not the full-history import path.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] The workstream remained closed and absent from every live manager while rows were written; archives stay closed and resumable imports are opened only after validation.
- [ ] `workstreams` row exists with the right `user_id` and `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Every attachment ID is the SHA-256 of its stored bytes; each turn's ordered IDs are in `conversations.attachments`, and blob refcounts match message references.
- [ ] If multi-node: the row is in shared storage and the node selected by
`ConsoleRouter.route(ws_id)` from the current live set can load it.
`workstreams.node_id`, when present, is treated as a creation-time hint rather
than asserted equal to the current HRW result.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
@@ -211,15 +286,20 @@ Before declaring success, verify:
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- **Don't shard imported rows by an ID prefix or a one-time HRW result.** HRW
uses the full ID and live membership; placement may move. In a cluster, write
one copy to shared storage and let request routing select the live node.
## Quick Reference
| Task | Path |
|---|---|
| Generate ws_id | `secrets.token_hex(16)` |
| Bulk insert messages | `Storage.save_messages_bulk(rows)` |
| Multi-node placement | Full-ID 32-bit FNV-1a HRW over live servers; store rows once in shared storage |
| Bulk insert attachment-free messages | `Storage.save_messages_bulk(rows)` |
| Attach new bytes | `save_message()` → `save_attachment()` per reference → `set_message_attachments()` |
| Archive (read-only) | `state="closed"`, skip `provider_data` |
| Resumable | `state="idle"`, populate `provider_data` if same provider |
| Resumable | Register closed, import and validate while unloaded, then explicitly open; populate `provider_data` if same provider |
| Tool call id | OpenAI shape: `{"id": ..., "type": "function", "function": {"name": ..., "arguments": "<json string>"}}` |
| Tool result row | `role="tool"`, `tool_name`, `tool_call_id`, `content` |
| Source role → Turnstone role | See "Role Mapping" table |
@@ -228,6 +308,8 @@ Before declaring success, verify:
## Files to read before writing the importer
- `turnstone/core/storage/_schema.py` — authoritative table definitions.
- `turnstone/core/storage/_protocol.py` — `save_message`, `save_messages_bulk`, `load_messages` signatures.
- `turnstone/core/storage/_protocol.py` — `register_workstream`, message, attachment, and load signatures.
- `turnstone/core/rendezvous.py` — authoritative full-ID FNV-1a HRW scoring.
- `turnstone/console/router.py` — live-node discovery, override precedence, and routing behavior.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
docs/tools.md
+49 -13
View File
@@ -1,9 +1,10 @@
# Tools Reference
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
Turnstone exposes a role-specific built-in tool surface plus any configured MCP
tools through provider-native or OpenAI-compatible function calling. Built-in
schemas live under `turnstone/tools/` and are loaded by
`turnstone/core/tools.py`; metadata selects the interactive, coordinator, and
task-agent subsets. MCP tools are discovered from configured servers by
`turnstone/core/mcp_client.py`.
---
@@ -50,10 +51,10 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | The complete loaded built-in union. Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of the built-in union. Used by tool search to distinguish built-ins from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -62,7 +63,10 @@ set lives in `_META_KEYS` in `turnstone/core/tools.py`):
> See also: [Tool Pipeline diagram](diagrams/png/05-tool-pipeline.png)
Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools()`:
Tool handling spans a four-phase pipeline. `ChatSession._execute_tools()` owns
prepare, approval, and execution (phases 1–3); after it returns, the owning
conversation loop guards the observed results and folds them into the
trajectory (phase 4).
### Phase 1: Prepare
@@ -71,9 +75,8 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 17
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Dispatches to the matching `_prepare_{func_name}()` handler, the synthetic
`tool_search` fallback, or the generic `_prepare_mcp_tool()` handler.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -82,7 +85,10 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
### Phase 2: Approve
All prepared items are sent to the UI via `ui.approve_tools(items)`.
Prepared items are sent to the UI via `ui.approve_tools(items)`. Several
parallel task agents may leave independent `ApprovalCycle` objects pending on
one workstream; each round owns a `cycle_id`, event, result, and verdict set.
Remote clients resolve the exact round by `cycle_id` (or a member `call_id`).
- The UI displays each tool's header and preview to the user.
- Items where `needs_approval` is `False` (auto-approved tools) are shown
@@ -94,6 +100,10 @@ All prepared items are sent to the UI via `ui.approve_tools(items)`.
prompt). This is per-tool, not blanket.
- If `auto_approve` is `True` on the session (via `--skip-permissions` or workstream
template), all tools are approved automatically.
- When Smart Approvals are enabled, one immutable judge/settings snapshot is
stamped onto the whole batch. The batch auto-approves only when every gated
item has a qualifying verdict; partial or mixed qualification fails closed to
the human prompt. Stop is linearized against that terminal decision.
### Phase 3: Execute
@@ -113,6 +123,28 @@ Each item's `execute` callable is invoked:
denials are tracked separately. This removes the need for text-prefix heuristics.
Other tools deliver results atomically via
`ui.on_tool_result(call_id, name, output, is_error=...)` only.
Stop propagates to child model scopes, judges, tracked subprocess groups, and
the approval cycles owned by the cancelled operation. Calls that definitely
never started receive `EffectStatus.none`; an interrupted call whose external
outcome was not observed receives `unknown`, `partial`, or `rolled_back` as
appropriate. These typed receipts preserve effect truth across storage/replay
without exposing unreviewed model output as a tool result.
### Phase 4: Guard and atomic fold
After `_execute_tools()` returns, the main `send()` loop compacts/truncates
completed results to the remaining shared budget and then runs the heuristic
and optional LLM output guard. The task-agent loop deliberately guards the
observed raw output before applying its size cap, so truncation cannot hide a
sensitive result from that check.
After guard work, the owning loop rechecks generation ownership. On the main
conversation path, one generation-fenced commit appends the complete
tool-result block, advisories, feedback, and queued user turns; its durable
records run in FIFO order outside the lifecycle lock. A force-cancelled
predecessor can therefore finish external cleanup, but cannot fold late results
into its successor's trajectory.
---
## Tool Approval Flow
@@ -577,7 +609,11 @@ pre-configure skills at workstream creation.
---
## Summary Table
## Interactive Tool Summary
This table describes the ordinary interactive surface. Coordinator sessions
receive their delegation/lifecycle tools instead, and task agents receive the
metadata-selected `TASK_AGENT_TOOLS` subset.
| Tool | Category | Auto-approve | task_agent | primary_key |
|--------------|------------|--------------|------------|-------------|
@@ -698,7 +734,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 17 built-in tools via
4. **Merging**: MCP tools are appended after the role's built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see