ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

docs(coordinator): phase 8 PR C — API tour, skills guide, bulk-endpoints contract + wait diagram (#388)

Browse Source 
* docs(coordinator): phase 8 PR C — API tour, skills guide, bulk-endpoints contract

Four deliverables that close out the phase 8 doc debt carried since
phase 1:

- docs/coordinator-api-tour.md — 9-step lifecycle walkthrough
  (create → subscribe → send → inspect children / detail → wait for
  fan-out → govern (trust / restrict / stop_cascade / close_all_children)
  → approve / cancel → close), one request + response per step, every
  SSE event type the UI has to handle, and every operation id cross-
  referenced against the live /openapi.json.  Integrators driving a
  coord session from a custom UI or SDK can work end-to-end from this
  doc without reverse-engineering the console page.

- docs/coordinator-skills.md — writing a SkillKind=COORDINATOR skill.
  Tool-surface diff (13 orchestration tools, no bash / edit / web /
  sub-agent), persona diff (orchestrator vs maker, composing on
  base_coordinator.md), SkillKind enum + migration 044, task_list
  integration, ws_id handling, wait vs inspect cost profile, three
  orchestration patterns (delegate-and-summarise, fan-out-and-
  synthesise, plan-then-delegate), testing surface.

- docs/bulk-endpoints.md — codifies the two shape idioms that shipped
  across phases 6–8: {results, denied, truncated} for bulk-read /
  bulk-create-with-payload (cluster/ws/live, spawn_batch); {<bucket>,
  failed, skipped} for cascade-mutation (stop_cascade,
  close_all_children).  Picks-by-semantics guidance so the next bulk
  endpoint author doesn't coin a third shape.

- docs/diagrams/27-coordinator-wait-for-workstream.puml + rendered
  PNG — sequence diagram covering spawn → wait (blocking, with
  bounded progress emission) → inspect → close.  Embedded in the
  API tour doc's §6 so the "why is my coord session blocking?"
  question has a visible answer.

No code changes.  All operation ids in the API tour verified against
a live build of the console spec; all markdown internal links
resolve; PlantUML renders clean on the system plantuml jar.

* docs(coordinator): address PR #388 copilot review

- api-tour.md child-event payload keys: events stamp `ws_id` as the
  coord's own id and carry the child's id separately as
  `child_ws_id`.  Doc previously listed `ws_id` as the child
  identifier on all four child_ws_* events, which would send SDK /
  UI implementers parsing the wrong field.
- api-tour.md SSE table: add the `status` event emitted by
  ConsoleCoordinatorUI.on_status (token usage + context_window +
  effort snapshot; fires on every streaming tick).  Previously
  omitted from the "every event type a UI has to handle" list.
- api-tour.md /children response key: server returns `{items,
  truncated}`, not `{children, truncated}`.  Also drop the
  `state=closed` query-param claim — the endpoint has no state
  filter; clients filter locally on the returned `state` field.
- skills.md task_list shape: the persisted row uses `id` (not
  `task_id` — the input schema uses `task_id`, the row uses `id`),
  has `child_ws_id` / `created` / `updated` (no `notes` field),
  and supports a 5th `reorder` action alongside add/update/remove/
  list.  Adds the parallel-dispatch caveat from the tool
  description.
- skills.md tenant-guard behaviour: foreign / hallucinated ws_ids
  don't return an empty result — they return explicit
  error/not-found/denied shapes that differ by op (mutating ops
  return `{error, status: 404}`; inspect returns `{error}`; wait
  reports state=denied).  Important distinction — a skill that
  expects empty on mismatch will mishandle every single case.

Docs-only; no code / schema / SDK changes.  All internal links
still resolve.
This commit is contained in:
Patrick Buckley
2026-04-19 10:15:08 -07:00
committed by GitHub
parent 67aaa236e8
commit a76d93b6c6
5 changed files with 1022 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/bulk-endpoints.md
+237
View File
@@ -0,0 +1,237 @@
# Bulk endpoint shape contract
Turnstone exposes several endpoints and tool calls that take multiple
ids and return a per-id outcome. Over the last few phases two
**distinct** response shapes have settled, one per semantic category.
This doc codifies both so a future endpoint author can pick the right
shape by semantics instead of by coin-flip.
Existing bulk endpoints at time of writing:
| Endpoint / tool | Category | Response shape |
|---------------------------------------------------------|--------------------------|------------------------------------------|
| `GET /v1/api/cluster/ws/live?ids=a,b,c` | bulk read | `{results, denied, truncated}` |
| model tool `spawn_batch` | bulk create (per-item) | `{results, denied}` |
| `POST /v1/api/coordinator/{ws_id}/stop_cascade` | cascade mutation | `{cancelled, failed, skipped}` |
| `POST /v1/api/coordinator/{ws_id}/close_all_children` | cascade mutation | `{closed, failed, skipped}` |
---
## Why two shapes
The ask-to-outcome mapping is fundamentally different between the
two categories, and a one-size-fits-all envelope ends up papering
over distinctions the caller genuinely needs to branch on.
**Bulk read / bulk create-with-payload.** Each input id (or batch
index) carries a *request-side* concept — "give me the live block
for this ws_id" or "spawn a child with this spec" — and each
successful output carries a *payload* — the live block, or the new
workstream's identifying triple. The interesting distinction on
failure is *ownership / validation* (caller can't see that id, spec
was malformed) — independent of the storage state.
**Cascade mutation.** The action is uniform across every id (cancel
this subtree, close this child). The interesting distinctions on
outcome are *did it reach the terminal state?* (succeeded / already
was there / the dispatch itself failed) — driven by the storage
state plus transport reliability, not by the caller's input.
Trying to unify these forces either:
- a stateless `denied` bucket that has to carry "already gone"
*and* "you don't have permission" *and* "transport failed" with a
separate reason string — reviewers end up string-matching to branch.
- or a per-item-payload map for cascade mutations where every
successful value is the same sentinel — carrier with no payload.
So: two shapes, one per category. The rest of this doc spells out
each.
---
## Shape A — bulk read / bulk create-with-payload
```json
{
"results": { "<key>": <value-or-null>, ... },
"denied": [ "<key>", ... ],
"truncated": false
}
```
**`results`** is a key-indexed map of the positive-path payload.
The key is the input id for read endpoints (`cluster/ws/live` uses
the ws_id), or the input-array index (stringified) for create
endpoints that want ordering preserved (`spawn_batch` uses `"0"`,
`"1"`, ...). The value is whatever the endpoint produces per
success — a live block, a `{ws_id, name, node_id, status}` triple,
etc. A `null` value (read endpoints only) means "the id existed and
you own it, but the live block wasn't available" — distinct from
"denied".
**`denied`** is the negative-path list. For read endpoints it's a
flat list of ids (preserves input order so callers can re-zip
against their input). For create endpoints with per-item payloads
it's a list of `{idx, reason}` objects (`spawn_batch`'s validation
and spawn-error rows; also the operator-reject surface when per-item
selective-deny ships). Include every reason that's *not* the
positive path — authz, ownership, validation, already-consumed,
spawn failure — so callers don't branch on status codes.
**`truncated`** is a boolean set to `true` when the server's
per-endpoint input cap was exceeded and the tail was dropped. The
endpoint docs each spell out the cap (50 for `cluster/ws/live`).
`spawn_batch` hard-errors on overflow instead of silently
truncating — it omits the field entirely rather than carry a
permanently-false flag.
### Example — `cluster/ws/live`
```http
GET /v1/api/cluster/ws/live?ids=a1b2,c3d4,nonexistent,foreign HTTP/1.1
```
```json
{
"results": {
"a1b2": {"state": "running", "tokens": 12843, "activity": "..."},
"c3d4": null
},
"denied": ["nonexistent", "foreign"],
"truncated": false
}
```
Callers that need ordered output zip their original id list against
this map; ids in `denied` drop out of the zip cleanly. A live-block
`null` doesn't route to `denied` — the row exists and the caller
owns it; the node is just currently unreachable.
### Example — `spawn_batch`
```json
{
"results": {
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3", "status": 200},
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1", "status": 200}
},
"denied": [
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
]
}
```
Indexes are stringified to keep the envelope JSON-safe and
consistently-typed across the read and create cases.
---
## Shape B — cascade mutation
```json
{
"status": "ok",
"<bucket>": [ "<ws_id>", ... ],
"failed": [ "<ws_id>", ... ],
"skipped": [ "<ws_id>", ... ]
}
```
Where `<bucket>` is the endpoint-specific name for "succeeded" —
`cancelled` for `stop_cascade`, `closed` for `close_all_children`.
The three buckets partition the input set exactly once:
| Bucket | Meaning |
|---------------|-------------------------------------------------------------------------------|
| `<bucket>` | Action dispatch accepted; target reached the intended terminal state. |
| `failed` | Dispatch returned a non-404 error (transport issue, upstream 5xx, exception). |
| `skipped` | Upstream 404 — stale registry entry, row already deleted, or peer gone. |
The split between `failed` and `skipped` is load-bearing. `failed`
is actionable — the operator may want to retry, or the cascade may
be partial. `skipped` is pre-resolved — the target is already in
the terminal state the cascade was aiming at, so it's neither a
win to report nor a fault to fix.
### Example — `stop_cascade`
```json
{
"status": "ok",
"cancelled": ["child-1", "child-3"],
"failed": [],
"skipped": ["child-2"]
}
```
A subsequent retry would target only `failed` ids, not `skipped`
ones — the latter are already done.
### Example — `close_all_children`
```json
{
"status": "ok",
"closed": ["child-1", "child-3"],
"failed": ["child-2"],
"skipped": []
}
```
Same partition, different success-bucket name. When `coord_client`
is unavailable (session loaded but no HTTP client attached — a
construction bug) every id goes to `failed` so the operator notices
rather than getting a silent all-skipped response.
---
## Guidance for future bulk endpoints
1. **Pick by semantics, not by "what shape is nearby."**
- Mutation that's uniform across ids + terminal-state outcome? →
**Shape B** (cascade mutation).
- Read or create where the input id carries payload, or where the
denial axis is independent of storage state? → **Shape A**
(bulk read / bulk create-with-payload).
2. **Cap the input.** Both shapes assume a bounded input — the
server rejects or silently truncates past the cap. Document the
cap in the endpoint's OpenAPI description. Shape A uses
`truncated: true` on quiet truncation; Shape B hard-errors on
overflow.
3. **Match existing bucket names for the same semantic.** Use
`failed` and `skipped` verbatim in Shape B — the per-endpoint
success bucket is the only slot that varies. Use `results` and
`denied` verbatim in Shape A; the per-endpoint `<key>` /
`<value>` types vary.
4. **Audit the verbose shape.** Both endpoints emit a corresponding
audit event with the full before/after bucket lists — the SSE
stream and the in-process response give live feedback, but a
postmortem operator will read the audit row. Use
`_emit_coord_audit` (coordinator-scoped) or `record_audit`
directly; don't inline.
5. **Don't mix shapes within one endpoint.** If a bulk endpoint
wants both partial-success creation AND per-item failure reasons
(like `spawn_batch` with its `{idx, reason}` denial rows), that's
Shape A with a richer denial element — not a blend with Shape B.
---
## History
- **Phase 6** shipped `cluster/ws/live` as the first Shape A endpoint
(`{results, denied, truncated}`).
- **Phase 7** shipped `stop_cascade` as the first Shape B endpoint
(`{cancelled, failed, skipped}`).
- **Phase 8 PR A** shipped `spawn_batch` (Shape A, keyed by idx) and
`close_all_children` (Shape B, twin of `stop_cascade`), which
crystallised the two-shape-per-semantic-category policy codified
here.
Before adding a third shape, read this doc and argue for why the
new surface doesn't fit either A or B. Two idioms in the cluster
API is a finite operator tax; three is one too many.
docs/coordinator-api-tour.md
+372
View File
@@ -0,0 +1,372 @@
# Coordinator API tour
Turnstone's **coordinator workstream** is a session hosted on the
console whose job is to orchestrate other workstreams. It runs an LLM
that can spawn child workstreams on any node, watch their progress,
wait for them to finish, steer them mid-flight, and tear them down.
This doc walks the full lifecycle — one request, one response, and the
relevant SSE events at each step.
Aimed at integrators driving a coordinator from a custom UI or SDK
without reverse-engineering the built-in console page. The shapes
here match the live OpenAPI spec served at `/openapi.json` and
rendered at `/docs` on every `turnstone-console` process. Every
step references the operation id from that spec so doc updates track
schema changes.
> **Auth throughout.** Every endpoint below sits behind bearer-token
> auth and the `admin.coordinator` permission. A session-scoped JWT
> is minted per login (see [docs/oidc.md](oidc.md) / [docs/security.md](security.md));
> a service token may call the read paths but destructive governance
> paths (`/restrict`, `/stop_cascade`, `/close_all_children`) require
> the explicit `admin.coordinator` grant — a service-token owner
> match isn't enough.
---
## The 9 steps
| # | Action | Operation | Operation id |
|---|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------|
| 1 | Create | `POST /v1/api/coordinator/new` | `v1_api_coordinator_new_post` |
| 2 | Subscribe to events | `GET /v1/api/coordinator/{ws_id}/events` (SSE) | `v1_api_coordinator_{ws_id}_events_get` |
| 3 | Send a user message | `POST /v1/api/coordinator/{ws_id}/send` | `v1_api_coordinator_{ws_id}_send_post` |
| 4 | Inspect children | `GET /v1/api/coordinator/{ws_id}/children` | `v1_api_coordinator_{ws_id}_children_get` |
| 5 | Inspect one workstream | `GET /v1/api/cluster/ws/{ws_id}/detail` | `v1_api_cluster_ws_{ws_id}_detail_get` |
| 6 | Wait for fan-out | model-side tool `wait_for_workstream` | — (tool call, not HTTP) |
| 7 | Govern | `POST /v1/api/coordinator/{ws_id}/trust` | `v1_api_coordinator_{ws_id}_trust_post` |
| | | `POST /v1/api/coordinator/{ws_id}/restrict` | `v1_api_coordinator_{ws_id}_restrict_post` |
| | | `POST /v1/api/coordinator/{ws_id}/stop_cascade` | `v1_api_coordinator_{ws_id}_stop_cascade_post` |
| | | `POST /v1/api/coordinator/{ws_id}/close_all_children` | `v1_api_coordinator_{ws_id}_close_all_children_post` |
| 8 | Approve / cancel | `POST /v1/api/coordinator/{ws_id}/approve` | `v1_api_coordinator_{ws_id}_approve_post` |
| | | `POST /v1/api/coordinator/{ws_id}/cancel` | `v1_api_coordinator_{ws_id}_cancel_post` |
| 9 | Close | `POST /v1/api/coordinator/{ws_id}/close` | `v1_api_coordinator_{ws_id}_close_post` |
---
## 1. Create a coordinator
```http
POST /v1/api/coordinator/new
Content-Type: application/json
Authorization: Bearer <token>
{
"name": "release-coord",
"skill": "engineer-orchestrator",
"initial_message": "audit /auth for CSRF handling across all active routes"
}
```
```http
HTTP/1.1 201 Created
Content-Type: application/json
```
All three body fields are optional — an empty body still creates a
coordinator with an auto-generated name and no initial message.
Returns **503** with a remediation message when the cluster isn't
configured with a coordinator model; see
[`coordinator.model_alias`](settings.md) to set one.
**SSE implication:** the `ws_created` event fires on the cluster-wide
stream (`/v1/api/cluster/events`) once the row is committed. Per-ws
subscribers (step 2) see the session warm up as token traffic starts.
---
## 2. Subscribe to the per-coordinator event stream
```http
GET /v1/api/coordinator/{ws_id}/events HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
```
One persistent SSE connection per browser tab / SDK caller — the
console fans each event out to every listener queue (cap 500 events
per queue, put_nowait drop on overflow). Events come in flat JSON
with a `type` field. The recurring shapes a UI has to handle:
| `type` | Emitted when | Payload highlights |
|---------------------|--------------------------------------------------------------------------------------------|--------------------|
| `thinking_start` / `thinking_stop` | Model has entered / exited a reasoning block | — |
| `reasoning` | Reasoning-token stream chunk (when the model exposes it) | `text` |
| `content` | Assistant-content stream chunk | `text` |
| `stream_end` | End of a single provider stream | — |
| `tool_result` | A tool call completed (success or error) | `call_id`, `name`, `output`, `is_error?` |
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
| `output_warning` | Output guard flagged a tool result | `call_id`, `risk_level`, `flags` |
| `child_ws_created` | A direct child of this coord was just created (fan-out from the cluster bus) | `child_ws_id`, `node_id`, `name`, `parent_ws_id` (`ws_id` in the envelope is always the coord's own id) |
| `child_ws_state` | A direct child transitioned state | `child_ws_id`, `state` |
| `child_ws_closed` | A direct child closed | `child_ws_id` |
| `child_ws_rename` | A direct child's name changed | `child_ws_id`, `name` |
| `wait_started` / `wait_progress` / `wait_ended` | `wait_for_workstream` tool lifecycle (see §6) | `call_id`, `ws_ids`, `elapsed`, `results`, `complete` |
| `batch_started` / `batch_ended` | `spawn_batch` / `close_all_children` tool lifecycle | `call_id`, `op`, `total`/`succeeded`/`denied`/`closed`/`failed`/`skipped` |
| `info` / `error` | Operational messages | `message` |
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
indicator — so a tab refresh mid-approval doesn't strand the
operator.
---
## 3. Send the first user message
```http
POST /v1/api/coordinator/{ws_id}/send
Content-Type: application/json
{"message": "audit /auth for CSRF handling across all active routes"}
```
```http
HTTP/1.1 200 OK
{"status": "ok"}
```
The message is queued for the worker thread at its next tool-result
seam (so you can send follow-ups mid-conversation without corrupting
the in-progress turn). On the SSE stream you'll see `state_change`
→ `thinking_start` → streaming `reasoning` / `content` / `tool_result`
events, finishing with `state_change → idle` or an
`approve_request` when the model invokes a gated tool.
---
## 4. Inspect direct children
```http
GET /v1/api/coordinator/{ws_id}/children HTTP/1.1
```
```json
{
"items": [
{"ws_id": "d4e5f6...", "name": "csrf-audit", "state": "running", "node_id": "gpu-3"},
{"ws_id": "e1f2a3...", "name": "xss-audit", "state": "idle", "node_id": "gpu-1"}
],
"truncated": false
}
```
The response key is `items`, not `children` — the endpoint shape
follows the cluster-wide workstream-list idiom rather than the
coordinator `list_workstreams` tool's (which uses `children`).
Rows include every state stored for the parent (`running`, `idle`,
`closed`, ...); the endpoint does not accept a state query param,
so clients should inspect each row's `state` field and filter
locally if they want to hide closed/deleted children. Nested
coordinator rows are dropped server-side so only interactive
descendants appear.
---
## 5. Inspect one workstream (storage + live block + tail)
```http
GET /v1/api/cluster/ws/{ws_id}/detail?message_limit=20 HTTP/1.1
```
```json
{
"persisted": { "ws_id": "...", "state": "running", "parent_ws_id": "...", "kind": "interactive", ... },
"live": { "state": "thinking", "tokens": 12843, "activity": "...", "pending_approval": null },
"tail": [ {"role": "assistant", "content": "...", "tokens": 128}, ... ]
}
```
Works for any workstream the caller has `admin.cluster.inspect` on,
not just children of a single coordinator — useful for a cluster
admin panel watching multiple coordinators at once. `live` is
`null` when the owning node is unreachable or has dropped the row
from its dashboard cache; callers should degrade gracefully, not
treat it as an error.
For fan-out views, prefer
[`GET /v1/api/cluster/ws/live?ids=a,b,c`](bulk-endpoints.md) — it
collapses N per-row round-trips into one, returning the live block
for every id in a `{results, denied, truncated}` envelope.
---
## 6. Wait for fan-out (`wait_for_workstream`)
`wait_for_workstream` is a **model-side tool**, not an HTTP endpoint
— the coordinator's LLM invokes it with a list of child ws_ids, the
session's worker thread blocks inside the tool, and a sequence of
`wait_started` / `wait_progress` / `wait_ended` SSE events is emitted
for the UI to drive a "waiting on N children" indicator.
![wait_for_workstream sequence](diagrams/png/27-coordinator-wait-for-workstream.png)
Key properties:
- **Caps** — up to 32 ws_ids per call, up to 600 seconds per call.
A coordinator that needs to wait on more children re-invokes the
tool with a fresh timeout.
- **Modes** — `mode="any"` returns as soon as one child reaches a
real terminal state (`idle` / `error` / `closed` / `deleted`);
`mode="all"` waits for every polled child.
- **Progress throttling** — the poll loop runs every 500 ms but the
SSE emission is diff-on-state-change plus a 5-second heartbeat. A
600 s wait generates O(dozens) of progress events, not 1200.
- **Denied rows** — an id the caller doesn't own (cross-tenant) or a
missing row is reported as a `denied` state in the results dict;
`mode="any"` won't satisfy on a pure-denied list (the LLM should
treat it as a config error, not a completion).
Prefer `wait_for_workstream` over polling `inspect_workstream` in a
loop — a wait consumes one assistant turn regardless of how long the
children take, whereas each `inspect_workstream` poll costs a full
turn (plus judge, plus tokens). On a fan-out of 3+ children this
rounds to a 10× token-efficiency win.
---
## 7. Governance — trust, restrict, stop_cascade, close_all_children
These four endpoints let an operator steer a live coordinator session
mid-flight. All four emit an audit event tagged
`coordinator.<action>` via the dedicated audit executor so a cascade
burst can't starve audit writes.
### `POST /trust` — auto-approve own-subtree sends
```json
POST /v1/api/coordinator/{ws_id}/trust
{"send": true}
```
Flips `trust_send=true` on the live session. Subsequent
`send_to_workstream` calls that target a ws_id in the coordinator's
own subtree skip the approval prompt; foreign ws_ids and other tool
calls still go through the normal flow. Requires both
`admin.coordinator` AND `coordinator.trust.send` permissions (the
second grants a service token the opt-in it otherwise wouldn't get).
### `POST /restrict` — revoke tool access mid-session
```json
POST /v1/api/coordinator/{ws_id}/restrict
{"revoke": ["spawn_workstream", "delete_workstream"]}
```
Unions the names into the session's revoked-tools set. Additive and
idempotent — calling twice with overlapping lists converges to the
union. Revocations don't survive a session close/reopen; operators
opt in per session. Cap 256 tool names per request, 128 chars each.
### `POST /stop_cascade` — cancel the subtree
```json
POST /v1/api/coordinator/{ws_id}/stop_cascade
{}
```
Cancels the coordinator's in-flight generation AND dispatches
`cancel_workstream` through the routing proxy for every direct
child in the in-memory registry. Returns:
```json
{"status": "ok", "cancelled": ["child-1", "child-3"], "failed": [], "skipped": ["child-2"]}
```
Response uses the [cascade-mutation bulk shape](bulk-endpoints.md):
`cancelled` = accepted, `failed` = dispatch error worth retrying,
`skipped` = upstream 404 (already gone — stale registry entry or
the row was deleted between snapshot and dispatch). Grandchildren
aren't touched directly; they sit behind their parent's cancel and
propagate via the child's SSE stream.
### `POST /close_all_children` — soft-close the direct fan-out
```json
POST /v1/api/coordinator/{ws_id}/close_all_children
{"reason": "audit round complete"}
```
Response:
```json
{"status": "ok", "closed": ["c-1", "c-2"], "failed": [], "skipped": []}
```
Soft-close cascade bounded by the same semaphore as `stop_cascade`.
The `reason` (up to 512 chars) propagates into each closed child's
audit + `workstream_config` for postmortem. Unlike `stop_cascade`
this does NOT recurse into grandchildren — the model-facing tool
that pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. For a full-subtree teardown, use
`stop_cascade`.
See [bulk-endpoints.md](bulk-endpoints.md) for why both endpoints
share the cascade-mutation shape and how it differs from the
`spawn_batch` / `cluster/ws/live` shape.
---
## 8. Approve / cancel
The `approve` endpoint is what resolves an `approve_request` SSE
event. The coordinator's worker thread is blocked inside
`ui.approve_tools` waiting for this POST.
```json
POST /v1/api/coordinator/{ws_id}/approve
{"approved": true, "feedback": null, "always": false}
{"approved": false, "feedback": "spawn count looks too high — try 3 not 10"}
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
```
`cancel` drops the in-flight generation but leaves the coordinator
idle and open for a fresh `send`:
```json
POST /v1/api/coordinator/{ws_id}/cancel
{}
```
---
## 9. Close
```json
POST /v1/api/coordinator/{ws_id}/close
{}
```
Soft-closes the session — state persists, children keep running (use
`close_all_children` or `stop_cascade` first to wind them down), the
worker thread exits, SSE streams send a final `stream_end` and
disconnect. The row is reopenable via
`POST /v1/api/coordinator/{ws_id}/open` so long as it hasn't been
deleted.
---
## Further reading
- [coordinator-skills.md](coordinator-skills.md) — writing a skill
that runs on a coordinator session (orchestrator persona,
workflow patterns, `SkillKind` classifier).
- [bulk-endpoints.md](bulk-endpoints.md) — the two bulk-shape
idioms (`{results, denied, truncated}` vs
`{<bucket>, failed, skipped}`) used by `cluster/ws/live`,
`spawn_batch`, `stop_cascade`, and `close_all_children`.
- [architecture.md](architecture.md) — cluster-wide architecture
including how coordinator sessions fit next to node-hosted
interactive workstreams.
- The live OpenAPI spec (`/openapi.json` on any console process)
and Swagger UI (`/docs`) — authoritative schemas for every
endpoint above.
endpoint above.
docs/coordinator-skills.md
+321
View File
@@ -0,0 +1,321 @@
# Writing a coordinator-specific skill
Skills are prompt-level personas that steer a Turnstone session
toward a narrow task. Most skills target **interactive** sessions —
the single-workstream "do this thing" surface where the model wields
`bash`, `edit_file`, `web_fetch`, and the rest of the maker toolset.
A **coordinator skill** is different. It runs on a session whose job
is to orchestrate other sessions. The toolset is smaller and
narrower, the persona is an orchestrator instead of a maker, and the
success metric is "did the plan resolve" instead of "did the code
compile". This doc covers the differences a skill author has to
care about.
---
## The two-surface model
A row in `prompt_templates` carries a `kind` column (see
[`turnstone/core/skill_kind.py`](../turnstone/core/skill_kind.py);
migration 044 added the column). Three values:
| `SkillKind` enum | Stored as | Visible in |
|----------------------|-----------------------------|---------------------------------------------------------------------------|
| `SkillKind.INTERACTIVE` | `"interactive"` | Only the interactive-session activation path. `list_skills` on a coord won't show it. |
| `SkillKind.COORDINATOR` | `"coordinator"` | Only the coordinator's `list_skills` tool. Hidden from interactive activation pickers. |
| `SkillKind.ANY` | `"any"` | Both surfaces. Default for legacy rows predating the classifier. |
The `kind` field is a `StrEnum` — drop-in ``str`` compatible — so
DB rows, JSON payloads, and `==` comparisons all work without
translation at the edge.
When a coordinator calls `list_skills`, the SQL filter narrows to
`kind IN ('coordinator', 'any')`. When an interactive session picks
a skill at activation, the filter narrows to
`kind IN ('interactive', 'any')`. A skill author tags once at
creation; the two surfaces stay partitioned without any
per-call filtering on the LLM side.
**Tagging a new skill as coordinator-only** — set `kind` to
`SkillKind.COORDINATOR` (or the literal string `"coordinator"`) when
you POST to `/v1/api/admin/skills`. Existing rows default to
`SkillKind.ANY`; bump them to `COORDINATOR` if you've rewritten the
prompt around the orchestrator toolset.
---
## Tool surface differences
Coordinator sessions receive a **fixed** tool set, defined in
`turnstone/core/tools.py` as `COORDINATOR_TOOLS`. Nothing a skill
or MCP config can do adds to it. Current members:
| Tool | Category | Notes |
|---------------------------|-----------------|---------------------------------------------------------------------|
| `spawn_workstream` | delegate | Create one child. Requires approval. |
| `spawn_batch` | delegate | Create up to 10 children in one approval. Partial-success shape. |
| `inspect_workstream` | observe | Read state + tail of one child. Auto-approved (no mutation). |
| `list_workstreams` | observe | List the direct children (same shape as `/children` endpoint). |
| `wait_for_workstream` | block | Block until one/all listed children hit a terminal state. |
| `send_to_workstream` | steer | Queue a follow-up message to a running child. |
| `close_workstream` | wind-down | Soft-close one child. Requires approval. |
| `close_all_children` | wind-down | Soft-close every direct child in one approval. Partial-success shape. |
| `cancel_workstream` | wind-down | Drop the in-flight generation; leaves child idle for a fresh send. |
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `list_skills` | discover | Coordinator-visible skills only (SkillKind filter above). |
| `task_list` | plan | Orchestrator-only scratchpad. Children don't see it. |
Explicitly **not** in the coordinator set:
- `bash` / `edit_file` / `write_file` / `append_file` / `diff_file` — no local FS.
- `read_file` / `search` — no local FS reads.
- `web_fetch` / `web_search` — no direct web access.
- `task_agent` / `plan_agent` — sub-agent tools are zeroed on coord sessions.
- `memory` / `recall` / `notify` / `watch` / `read_resource` / `use_prompt` / `skill` — the orchestrator's "memory" is its children's outputs; these UX / persistence tools belong to interactive sessions.
If your skill needs a coordinator to "run a command" or "read a
file", write the delegate pattern instead: spawn a child with an
appropriate skill, `wait_for_workstream`, then `inspect_workstream`
for the output. The coordinator stays the orchestrator.
---
## Persona differences
Interactive skills compose on top of `base_interactive.md` — a
"maker" persona: get the work done, use the tools, edit the code,
close the loop.
Coordinator skills compose on top of
[`base_coordinator.md`](../turnstone/prompts/base_coordinator.md) —
an "orchestrator" persona: decompose, delegate, monitor, synthesise.
The base text is short but sets the tone every coordinator skill
inherits:
> You are a coordinator on a small, focused infrastructure team.
> Your role is to orchestrate work across the cluster... You do
> not edit files, run shell commands, browse the web, or manipulate
> the codebase directly. Children do that.
Write your skill's system prompt to *add* task-specific orchestration
hints on top — don't re-explain the role, don't paste tool JSON,
don't try to override the "no direct action" contract. Keep the
additions to: (a) the specific kind of work this skill delegates;
(b) the preferred skill tags for children; (c) the synthesis shape
the skill should end on.
---
## `task_list` integration
`task_list` is the coordinator's scratchpad — a persisted, ordered
list of rows with fields `{id, title, status, child_ws_id, created,
updated}` that only this coordinator sees. Children don't see it;
the user does via the sidebar. Five actions: `add`, `update`,
`remove`, `reorder`, `list` (only `list` is auto-approved; the
mutators go through the approval flow).
The input schema refers to rows by `task_id`; the persisted row
object exposes the same id as `id`. The `child_ws_id` field is a
free-form label the skill sets to link a task to a spawned
workstream — it is NOT validated against the workstreams table, so
a skill can set it to a placeholder before `spawn_workstream`
returns or keep it pointing at a closed child for later audit.
A skill's initial prompt can seed the task list by calling
`task_list(action="add", title=...)` as its very first tool calls —
the user gets a visible plan before any child is spawned, and the
coordinator's future self has something concrete to iterate on.
Status transitions (`pending` → `in_progress` → `done` / `blocked`)
are the skill's main feedback loop: mutate the task when the child
covering it finishes, not when the child starts. Use
`task_list(action="update", task_id=..., child_ws_id=<ws_id>)` to
link a task to the child that owns it once spawn returns.
A final gotcha: parallel tool dispatch does NOT serialise reads
after writes in the same batch. If a skill issues an `update` and
a `list` in one parallel tool batch, the `list` response may reflect
the pre-update state. Dispatch mutate and list serially (one
tool_use turn each) when the list must observe the mutation.
Keep the tasks coarse-grained — one per child, roughly. A 20-task
list for a 3-child fan-out is noise; a 1-task list for a 5-child
fan-out loses the plan. The sidebar renders tasks as the operator's
mental model of "what the coord thinks it's doing".
---
## Referencing children by `ws_id`
Every ws_id returned by `spawn_workstream` / `spawn_batch` is a
**full 32-char hex string**. The skill's system prompt must not
invent ws_ids — a model that hallucinates `"child-1"` or `"ws-abc"`
hits the tenant guard in `CoordinatorClient._is_own_subtree`, which
validates ws_id against `parent_ws_id=coord_ws_id` AND
`user_id=owner` in storage. The rejection shape varies by tool:
- **Mutating ops** (`send_to_workstream`, `close_workstream`,
`cancel_workstream`, `delete_workstream`) return
`{"error": "workstream not in coordinator subtree: <ws_id>", "status": 404}`
— the skill should treat this as a tool error, not an empty result.
- **`inspect_workstream`** returns `{"error": "workstream not found: <ws_id>"}`
(same shape as a genuinely missing row, so the guard can't be
used as an existence oracle).
- **`wait_for_workstream`** reports the offending id with
`state="denied"` in its `results` dict; `mode="any"` won't
satisfy on a pure-denied list, so a hallucinated id won't trick
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"ws_id": "...", "name": "...",
"node_id": "...", "status": 200}`; the model should extract the
ws_id and pass it to `inspect_workstream` / `wait_for_workstream` /
`send_to_workstream` / `close_workstream` verbatim.
A UI that wants human-readable identifiers should render the `name`
field and keep the ws_id as the click-through key.
---
## `wait_for_workstream` vs `inspect_workstream`
Two distinct semantics, different cost profiles:
- **`wait_for_workstream(ws_ids=[...], timeout=60, mode="any")`** —
blocks inside a single tool call until one (or all, for `mode="all"`)
of the listed children reaches a terminal state (`idle`, `error`,
`closed`, `deleted`). The worker thread blocks up to `timeout`
seconds; the assistant turn remains a single round-trip regardless
of how long the wait actually takes. Prefer this for "the plan
needs child X to finish before the next step."
- **`inspect_workstream(ws_id=...)`** — single read of the child's
state + tail. Costs a full assistant turn (judge, tokens, stream).
Prefer this for "what does the final message say?" after the child
has already resolved (via `wait_for_workstream` or a known
transition).
Rule of thumb: wait once for a fan-out, then inspect once per
child for the content. A loop of inspect-every-few-seconds is a
token-burning antipattern — on 3+ children it rounds to a 10×
efficiency hit over a wait+inspect pair.
---
## Common coordinator patterns
Three patterns cover most coordinator skills. Pick the one that
matches the task, or combine them deliberately.
### Pattern 1 — delegate-and-summarise
One specialist child, one focused brief, one synthesis message back
to the user. Appropriate when the user's request is "run the thing
and tell me what happened" and the work fits in one workstream.
```
task_list(action='add', title='audit /auth for CSRF')
spawn_workstream(skill='engineer', initial_message='audit /auth ...')
wait_for_workstream(ws_ids=[<child>], timeout=300)
inspect_workstream(ws_id=<child>)
→ synthesise the final message into a user-facing response
task_list(action='update', task_id='t_01', status='done')
close_workstream(ws_id=<child>, reason='audit complete')
```
### Pattern 2 — fan-out-and-synthesise
N children running in parallel, each with a distinct brief, all
waited-on together, then synthesised. Appropriate when the user's
request naturally decomposes into independent subtasks.
```
task_list seeds:
t_01 benchmark Anthropic 4.7 latency on summarisation
t_02 benchmark OpenAI GPT-5.2 latency on summarisation
t_03 benchmark Gemini 2.5 latency on summarisation
spawn_batch(children=[...3 briefs...])
wait_for_workstream(ws_ids=[c1, c2, c3], mode='all', timeout=600)
inspect_workstream(ws_id=c1); ...(c2); ...(c3)
→ synthesise head-to-head comparison
task_list → all done
close_all_children(reason='benchmark complete')
```
Prefer `spawn_batch` over 3 individual `spawn_workstream` calls —
one approval instead of three, one audit trail, deterministic
sibling ordering. Pair with `wait_for_workstream(mode='all')` and
`close_all_children(reason=...)` to wind the fan-out down in one
approval each.
### Pattern 3 — plan-then-delegate
The coordinator first uses its own reasoning to carve the plan,
records it in `task_list`, then spawns children that each own one
task. Appropriate when the user's request is "figure out how to X"
and the coordinator's planning step is itself valuable.
```
→ coord reasons about the shape of the work
task_list(action='add', title='...') × N # the plan, visible in the sidebar
for task in tasks:
spawn_workstream(skill=..., initial_message=task.brief)
task_list(action='update', task_id=task.id, notes='ws=<child_ws_id>')
wait_for_workstream(ws_ids=[...], mode='all', timeout=...)
for child in children:
inspect_workstream(ws_id=child)
task_list(action='update', task_id=..., status='done', notes='result summary')
→ synthesise
```
The key distinction from Pattern 2: the plan is an artifact the user
can see and interact with (via the sidebar). If the coordinator's
reasoning-pass was wrong about the decomposition, the user can
course-correct before any child runs.
---
## Testing a coordinator skill
Coordinator sessions are hosted on the console, not on a node.
Integration tests that drive a real coord session live under
`tests/test_coordinator_end_to_end.py` — they spin a console with
an in-memory SQLite backend and a fake upstream node, then drive
the session through its HTTP surface.
For a new coordinator skill:
1. Write the skill prompt as a string and pass it to the
`coord_session` fixture's `skill=` kwarg (see
`tests/test_coordinator_tools.py` for the pattern).
2. Build a small fake cluster: one node + two children via
`mgr.register_children(coord.id, ["child-1", "child-2"])`.
3. Drive the session with seeded tool_call dicts matching the
provider layer's shape. The unit-level tests in
`tests/test_coordinator_tools.py` show the helper (`_tc(name,
args, call_id)`).
4. Assert the skill's decision shape — which tools fire in what
order, what the task_list looks like at the end, which
`_error` reasons appear on the denied-path.
A full end-to-end test isn't required for every skill; a
prepare-step unit test that asserts "given this initial message, the
first tool call is X with Y args" is usually sufficient to catch
persona drift without a real LLM in the loop.
---
## Further reading
- [coordinator-api-tour.md](coordinator-api-tour.md) — the HTTP
surface every coordinator skill indirectly drives.
- [bulk-endpoints.md](bulk-endpoints.md) — the response shape
`spawn_batch` and `close_all_children` use, so your skill can
parse results / denied arrays correctly.
- [governance.md](governance.md) — the broader governance surface
(`/trust`, `/restrict`, `/stop_cascade`, role-based permissions)
that wraps every coord session.
- [settings.md](settings.md) — `coordinator.model_alias` and
`coordinator.reasoning_effort` settings that gate which LLM runs
the coordinator session at all.
docs/diagrams/27-coordinator-wait-for-workstream.puml
+89
View File
@@ -0,0 +1,89 @@
@startuml
title Turnstone - coordinator wait_for_workstream lifecycle
skinparam sequenceArrowThickness 1.5
skinparam noteBackgroundColor #FDF6E3
participant "Coordinator\nLLM" as LLM
participant "ChatSession\n(worker thread)" as CS
participant "CoordinatorClient" as CC
participant "SessionUI\n(SSE fanout)" as UI
participant "Console routing\nproxy" as RP
database "Storage\n(workstreams row)" as DB
participant "Child\nnode" as NODE
== Spawn ==
LLM -> CS : tool_call spawn_workstream(...)
activate CS
CS -> CC : spawn(initial_message=...,\nparent_ws_id=coord, user_id=...)
CC -> RP : POST /v1/api/route/workstreams/new
RP -> NODE : dispatch (rendezvous)
NODE -> DB : insert workstreams row\nstate='running'
RP --> CC : {ws_id, node_id, name, status: 200}
CC --> CS : {ws_id, ...}
CS -> UI : on_tool_result\n("spawn_workstream", ws_id)
deactivate CS
note right of LLM
Model now knows the child ws_id.
It can inspect / send / wait, and
the parent registry tracks it.
end note
== Wait (blocking) ==
LLM -> CS : tool_call wait_for_workstream\n(ws_ids=[child], mode="any", timeout=60)
activate CS
CS -> CS : _prepare_wait_for_workstream\n(validate ws_ids, timeout, mode)
CS -> UI : emit wait_started\n{call_id, ws_ids, mode, timeout}
CS -> CC : wait_for_workstream(ws_ids, timeout,\nmode, progress_callback)
activate CC
loop every 500ms up to timeout
CC -> DB : read workstreams row(s)
DB --> CC : {state, updated, tokens, ...}
alt state in {idle, error, closed, deleted}
note over CC
real-terminal state ->
completion condition met
end note
else still running / thinking / attention
CC -> CS : progress_callback(snap)\n(diff-on-change or 5s heartbeat)
CS -> UI : emit wait_progress\n{call_id, elapsed, results?}
end
end
CC --> CS : {complete, elapsed,\nresults: {ws_id: snap}}
deactivate CC
CS -> UI : emit wait_ended\n{call_id, complete, elapsed, results}
CS -> UI : on_tool_result\n("wait_for_workstream",\n"complete after Ns (R/N resolved)")
CS --> LLM : tool_result (full results dict)
deactivate CS
note left of UI
Sidebar "waiting on N children" indicator
keys on call_id - started / progress / ended
scope to a single wait invocation so
nested waits render independent badges.
end note
== After wait: inspect + close ==
LLM -> CS : tool_call inspect_workstream(ws_id=child)
CS -> CC : inspect(ws_id)
CC -> DB : read row + tail
CC --> CS : {state, messages, tokens, ...}
CS --> LLM : tool_result (serialised)
LLM -> CS : tool_call close_workstream\n(ws_id=child, reason="...")
CS -> CC : close_workstream(ws_id, reason)
CC -> RP : POST /v1/api/route/workstreams/close
RP -> NODE : dispatch
NODE -> DB : state='closed',\nclose_reason='...'
RP --> CC : {status: 200}
CC --> CS : {closed: true, status: 200, reason: ...}
CS --> LLM : tool_result
@enduml
docs/diagrams/png/27-coordinator-wait-for-workstream.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aa12d81dc578f7e65bf4df3152b3de1736289c422f83d0b0cd32107726722357
size 172028