Patrick Buckley
480a1426b3
* fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964
20 KiB
Coordinator API tour
Turnstone's coordinator workstream is a session hosted on the console whose job is to orchestrate other workstreams. It runs an LLM that can spawn child workstreams on any node, watch their progress, wait for them to finish, steer them mid-flight, and tear them down. This doc walks the full lifecycle — one request, one response, and the relevant SSE events at each step.
Aimed at integrators driving a coordinator from a custom UI or SDK
without reverse-engineering the built-in console page. The shapes
here match the live OpenAPI spec served at /openapi.json and
rendered at /docs on every turnstone-console process. Every
step references the operation id from that spec so doc updates track
schema changes.
Auth throughout. Every endpoint below sits behind bearer-token auth and the
admin.coordinatorpermission. A session-scoped JWT is minted per login (see docs/oidc.md / docs/security.md); a service token may call the read paths but destructive governance paths (/restrict,/close_all_children) require the explicitadmin.coordinatorgrant — a service-token owner match isn't enough.
The 9 steps
URL convergence (1.5.0). Pre-1.5 coord-only endpoints lived under
/v1/api/coordinator/.... The Stage 2 verb-shape lift consolidated coord and interactive onto the unified/v1/api/workstreams/{ws_id}/<verb>tree; coord still distinguishes itself via thekind=coordinatorrow classifier rather than a separate URL space. The endpoints below reflect the post-lift surface served byturnstone-console.
| # | Action | Operation |
|---|---|---|
| 1 | Create | POST /v1/api/workstreams/new |
| 2 | Bootstrap history + subscribe | GET .../history, then GET .../events (SSE) |
| 3 | Send a user message | POST /v1/api/workstreams/{ws_id}/send |
| 4 | Inspect children | GET /v1/api/workstreams/{ws_id}/children |
| 5 | Inspect one workstream | GET /v1/api/cluster/ws/{ws_id}/detail |
| 6 | Wait for fan-out | model-side tool wait_for_workstream |
| 7 | Govern | POST /v1/api/workstreams/{ws_id}/trust |
POST /v1/api/workstreams/{ws_id}/restrict |
||
POST /v1/api/workstreams/{ws_id}/close_all_children |
||
| 8 | Approve / cancel | POST /v1/api/workstreams/{ws_id}/approve |
POST /v1/api/workstreams/{ws_id}/cancel |
||
| 9 | Close | POST /v1/api/workstreams/{ws_id}/close |
Refer to /openapi.json (Swagger UI at /docs) on any
turnstone-console process for the authoritative operation ids and
schemas. Coordinator-only verbs (/children, /trust, /restrict,
/close_all_children) 404 against kind=interactive
rows; the shared verbs (/send, /approve, /cancel, /events,
/history, /open, /close, etc.) work on both kinds.
1. Create a coordinator
POST /v1/api/workstreams/new
Content-Type: application/json
Authorization: Bearer <token>
{
"name": "release-coord",
"skill": "engineer-orchestrator",
"initial_message": "audit /auth for CSRF handling across all active routes"
}
HTTP/1.1 201 Created
Content-Type: application/json
{"ws_id": "a1b2c3d4e5f6...", "name": "release-coord"}
All three body fields are optional — an empty body still creates a
coordinator with an auto-generated name and no initial message.
Returns 503 with a remediation message when the cluster isn't
configured with a coordinator model; see
coordinator.model_alias to set one.
SSE implication: the ws_created event fires on the cluster-wide
stream (/v1/api/cluster/events) once the row is committed. Per-ws
subscribers (step 2) see the session warm up as token traffic starts.
2. Bootstrap history, then subscribe to the event stream
Read and render history before opening the initial stream:
GET /v1/api/workstreams/{ws_id}/history?limit=100 HTTP/1.1
Authorization: Bearer <token>
For a loaded coordinator, messages is the requested tail of one total
accepted conversation-row prefix: user, assistant, tool, and system rows,
including projected compaction checkpoints and cancellation-generated markers.
The response's optional cursor and handoff_token belong to that exact
render. Pass both once on the initial stream URL:
GET /v1/api/workstreams/{ws_id}/events?last_event_id={cursor}&history_token={handoff_token}&user_turn=1&tool_turn=1 HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
Omit either query parameter when its history field is null. A handoff token
is opaque and process-local: do not parse, persist, or reuse it. Admission of a
later conversation row changes the token; durable acknowledgement does not. If
history returns 503 {"error":"History temporarily unavailable"}, the response
is not authoritative: retain the current transcript, do not open a tokenless
replacement stream, and retry the read.
One persistent SSE connection per browser tab / SDK caller — the
console fans each event out to every listener queue (cap 500 events
per queue, put_nowait drop on overflow). Events come in flat JSON
with a type field. The recurring shapes a UI has to handle:
type |
Emitted when | Payload highlights |
|---|---|---|
thinking_start / thinking_stop |
Model has entered / exited a reasoning block | — |
reasoning |
Reasoning-token stream chunk (when the model exposes it) | text |
content |
Assistant-content stream chunk | text |
stream_end |
End of a single provider stream | — |
tool_result |
A tool call completed; capable panes also receive the accepted-history replacement | call_id, name, output, is_error?, accepted?, _event_id?, preview?, effect_status? |
tool_output_chunk |
Streaming tool output (e.g. long bash command) | call_id, chunk |
approve_request |
One approval cycle needs operator action; several cycles may coexist | cycle_id, items: [{call_id, header, preview, func_name, approval_label, needs_approval}] |
approval_resolved |
One identified approval cycle was answered | cycle_id, call_ids, approved, feedback, always |
state_change |
Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | state ∈ running, thinking, attention, idle, error |
in_progress_snapshot |
One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | content, reasoning |
status |
Token usage + context-window snapshot (fires on every streaming tick) | prompt_tokens, completion_tokens, total_tokens, context_window, pct, effort, cache_creation_tokens, cache_read_tokens |
rename |
Session's display name changed | name |
intent_verdict |
Intent judge produced a verdict on a pending tool call | risk_level, recommendation, reasons |
output_warning |
Output guard flagged a tool result | call_id, risk_level, flags |
child_ws_created |
A direct child of this coord was just created (fan-out from the cluster bus) | child_ws_id, node_id, name, parent_ws_id (ws_id in the envelope is always the coord's own id) |
child_ws_state |
A direct child transitioned state | child_ws_id, state |
child_ws_closed |
A direct child closed | child_ws_id |
child_ws_rename |
A direct child's name changed | child_ws_id, name |
wait_started / wait_progress / wait_ended |
wait_for_workstream tool lifecycle (see §6) |
call_id, ws_ids, elapsed, results, complete |
batch_started / batch_ended |
spawn_batch / close_all_children tool lifecycle |
call_id, op, total/succeeded/denied/closed/failed/skipped |
info / error |
Operational messages | message |
history_resync |
The rendered history token no longer names the accepted row prefix | ws_id, reason |
Reconnection contract: a freshly-opened SSE connection receives
one approve_request snapshot for every unresolved approval cycle, keyed by
the same stable cycle_id, plus any in-flight wait_* / batch_*
indicator, the worker's current state_change, and an
in_progress_snapshot carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
history_resync is stronger than a numeric replay gap. The server closes that
stream; fetch and render /history again, then open a new stream with its new
cursor/token pair. The API and SDK expose these primitives but deliberately do
not choose a reconnect policy for callers.
3. Send the first user message
POST /v1/api/workstreams/{ws_id}/send
Content-Type: application/json
{"message": "audit /auth for CSRF handling across all active routes"}
HTTP/1.1 200 OK
{"status": "ok"}
The message is queued for the worker thread at its next tool-result
seam (so you can send follow-ups mid-conversation without corrupting
the in-progress turn). On the SSE stream you'll see state_change
→ thinking_start → streaming reasoning / content / tool_result
events, finishing with state_change → idle or an
approve_request when the model invokes a gated tool.
4. Inspect direct children
GET /v1/api/workstreams/{ws_id}/children HTTP/1.1
{
"items": [
{"ws_id": "d4e5f6...", "name": "csrf-audit", "state": "running", "node_id": "gpu-3"},
{"ws_id": "e1f2a3...", "name": "xss-audit", "state": "idle", "node_id": "gpu-1"}
],
"truncated": false
}
The response key is items, not children — the endpoint shape
follows the cluster-wide workstream-list idiom rather than the
coordinator list_workstreams tool's (which uses children).
Rows include every state stored for the parent (running, idle,
closed, ...); the endpoint does not accept a state query param,
so clients should inspect each row's state field and filter
locally if they want to hide closed/deleted children. Nested
coordinator rows are dropped server-side so only interactive
descendants appear.
5. Inspect one workstream (storage + live block + tail)
GET /v1/api/cluster/ws/{ws_id}/detail?message_limit=20 HTTP/1.1
{
"persisted": { "ws_id": "...", "state": "running", "parent_ws_id": "...", "kind": "interactive", ... },
"live": { "state": "thinking", "tokens": 12843, "activity": "...", "pending_approval": null },
"tail": [ {"role": "assistant", "content": "...", "tokens": 128}, ... ]
}
Works for any workstream the caller has admin.cluster.inspect on,
not just children of a single coordinator — useful for a cluster
admin panel watching multiple coordinators at once. live is
null when the owning node is unreachable or has dropped the row
from its dashboard cache; callers should degrade gracefully, not
treat it as an error.
For fan-out views, prefer
GET /v1/api/cluster/ws/live?ids=a,b,c — it
collapses N per-row round-trips into one, returning the live block
for every id in a {results, denied, truncated} envelope.
6. Wait for fan-out (wait_for_workstream)
wait_for_workstream is a model-side tool, not an HTTP endpoint
— the coordinator's LLM invokes it with a list of child ws_ids, the
session's worker thread blocks inside the tool, and a sequence of
wait_started / wait_progress / wait_ended SSE events is emitted
for the UI to drive a "waiting on N children" indicator.
Key properties:
- Caps — up to 32 ws_ids per call, up to 600 seconds per call. A coordinator that needs to wait on more children re-invokes the tool with a fresh timeout.
- Modes —
mode="any"returns as soon as one child reaches a real terminal state (idle/error/closed/deleted);mode="all"waits for every polled child to reach a real terminal state. - Progress throttling — the poll loop runs every 500 ms but the SSE emission is diff-on-state-change plus a 5-second heartbeat. A 600 s wait generates O(dozens) of progress events, not 1200.
- Unresolvable ids — ws_ids are validated up front (exactly
32 hex chars; copy them verbatim): a malformed id fails the call
immediately with did-you-mean suggestions and a roster of the
coord's children. An id the caller doesn't own, a missing row, or
a child hard-deleted mid-wait is reported as
state="not_found"and aborts the wait on the tick that observes it (top-levelerror/not_found/childrenfields,complete=false) — the LLM should fix the id and re-issue, not conclude the child died. Foreign and missing collapse into one shape, so the wait can't be used as an existence oracle.
Prefer wait_for_workstream over polling inspect_workstream in a
loop — a wait consumes one assistant turn regardless of how long the
children take, whereas each inspect_workstream poll costs a full
turn (plus judge, plus tokens). On a fan-out of 3+ children this
rounds to a 10× token-efficiency win.
7. Governance — trust, restrict, close_all_children
These three endpoints let an operator steer a live coordinator session
mid-flight. All three emit an audit event tagged
coordinator.<action> via the dedicated audit executor so a cascade
burst can't starve audit writes.
POST /trust — auto-approve own-subtree sends
POST /v1/api/workstreams/{ws_id}/trust
{"send": true}
Flips trust_send=true on the live session. Subsequent
send_to_workstream calls that target a ws_id in the coordinator's
own subtree skip the approval prompt; foreign ws_ids and other tool
calls still go through the normal flow. Requires both
admin.coordinator AND coordinator.trust.send permissions (the
second grants a service token the opt-in it otherwise wouldn't get).
POST /restrict — revoke tool access mid-session
POST /v1/api/workstreams/{ws_id}/restrict
{"revoke": ["spawn_workstream", "delete_workstream"]}
Unions the names into the session's revoked-tools set. Additive and idempotent — calling twice with overlapping lists converges to the union. Revocations don't survive a session close/reopen; operators opt in per session. Cap 256 tool names per request, 128 chars each.
POST /close_all_children — soft-close the direct fan-out
POST /v1/api/workstreams/{ws_id}/close_all_children
{"reason": "audit round complete"}
Response:
{"status": "ok", "closed": ["c-1", "c-2"], "failed": [], "skipped": []}
Soft-close cascade bounded by a concurrency semaphore. The reason
(up to 512 chars) propagates into each closed child's audit +
workstream_config for postmortem. The model-facing tool that
pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. This soft-closes; to cancel the
fan-out instead, cancel the coordinator (§8) — a coordinator cancel
auto-cascades to its direct children.
See bulk-endpoints.md for why close_all_children
uses the cascade-mutation shape and how it differs from the
spawn_batch / cluster/ws/live shape.
8. Approve / cancel
The approve endpoint is what resolves an approve_request SSE
event. The coordinator's worker thread is blocked inside
ui.approve_tools waiting for this POST. Parallel task agents can leave
several approval cycles live at once, so current clients echo the event's
cycle_id (or a member call_id). A selector-less request resolves the oldest
cycle for compatibility.
POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": false, "cycle_id": "cycle_789"}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // remember this cycle's tool names
Success returns {"status": "ok", "cycle_id": "cycle_789"}. A stale selector
returns 409 with the currently oldest cycle/call IDs. always remembers only
the tool names in the cycle that actually resolved; it does not enable blanket
approval.
cancel requests cooperative cancellation of the coordinator's in-flight
generation and auto-cascades to its direct children:
cancel_workstream is dispatched through the routing proxy for
every direct child in the registry. The HTTP acknowledgement is immediate;
the worker becomes idle after unwinding. Pass {"force": true} only to release
a wedged worker slot immediately. The coordinator itself remains open for a
fresh send:
POST /v1/api/workstreams/{ws_id}/cancel
{}
{"status": "ok", "dropped": {}}
9. Close
POST /v1/api/workstreams/{ws_id}/close
{}
Soft-closes the session — state persists, children keep running
(wind them down first with close_all_children, or by cancelling
the coordinator, which cascades the cancel to its direct children),
the worker thread exits, SSE streams send a final stream_end and
disconnect. The row is reopenable via
POST /v1/api/workstreams/{ws_id}/open so long as it hasn't been
deleted.
If any accepted live conversation row still requires persistence
reconciliation, close returns 409 {"error":"workstream has unresolved persistence"}. The coordinator remains loaded, its journal is retained, and
no history is discarded; retry after storage recovers.
Further reading
- coordinator-skills.md — writing a skill
that runs on a coordinator session (orchestrator framing,
workflow patterns,
SkillKindclassifier). - bulk-endpoints.md — the two bulk-shape
idioms (
{results, denied, truncated}vs{<bucket>, failed, skipped}) used bycluster/ws/live,spawn_batch, andclose_all_children. - architecture.md — cluster-wide architecture including how coordinator sessions fit next to node-hosted interactive workstreams.
- The live OpenAPI spec (
/openapi.jsonon any console process) and Swagger UI (/docs) — authoritative schemas for every endpoint above.