Files
turnstone/docs/coordinator-skills.md
T
Patrick Buckley 776430d860 feat(cancel): honest cancellation dispositions + coordinator subtree propagation
A cancelled agent previously discarded its own ledger and reported a bare
"(task interrupted by user)" — fabricating the *outcome* (read downstream
as "nothing happened"), which invites a double-send as readily as a
dropped record causes an orphan. Make the fold-back honest, and propagate
an owner's cancel down the coordinator subtree.

- task_agent (single + parallel): on cancel, fold back a deterministic
  disposition built from the agent's in-memory ledger — actions completed,
  the in-flight action flagged outcome-UNKNOWN, and not-started calls —
  instead of the opaque interrupted string.
- coordinator cancel now auto-propagates to its direct children via a
  post_cancel hook on the shared cancel handler (cooperative fan-out; no
  blocking drain).
- synthesized cancelled tool results now read outcome-UNKNOWN rather than
  implying the call never ran.
- remove the now-redundant stop_cascade operator endpoint (handler, route,
  OpenAPI spec + schema, tests, docs); a coordinator cancel supersedes it.
2026-06-26 03:28:06 -07:00

17 KiB
Raw Blame History

Writing a coordinator-specific skill

Skills are prompt-level personas that steer a Turnstone session toward a narrow task. Most skills target interactive sessions — the single-workstream "do this thing" surface where the model wields bash, edit_file, web_fetch, and the rest of the maker toolset.

A coordinator skill is different. It runs on a session whose job is to orchestrate other sessions. The toolset is smaller and narrower, the persona is an orchestrator instead of a maker, and the success metric is "did the plan resolve" instead of "did the code compile". This doc covers the differences a skill author has to care about.


kind — authored audience metadata

A row in prompt_templates carries a kind column (see turnstone/core/skill_kind.py; migration 044 added the column). Three values:

SkillKind enum Stored as Meaning
SkillKind.INTERACTIVE "interactive" Authored for the interactive maker persona (single-workstream "do this").
SkillKind.COORDINATOR "coordinator" Authored for the orchestrator persona (delegate, monitor, synthesise).
SkillKind.ANY "any" Either surface (or audience-neutral). Default on create.

The kind field is a StrEnum — drop-in str compatible — so DB rows, JSON payloads, and == comparisons all work without translation at the edge.

kind is metadata, not an enforcement boundary. The model can skills(action='find') across every kind from any session, get any row by name, and load any visible skill regardless of session kind. Actual runtime capability is gated by allowed_tools + auto_approve on the skill and the operator's approval card on every load / spawn_workstream(skill=...) decision — kind doesn't add or remove any of that. It's a sorting / grouping / search-narrowing hint.

The opt-in filter is on skills(action='find', kind='coordinator') (or 'interactive') — pass it when you want to narrow a catalog browse to a specific authored audience. Omitting it (or passing kind='any') returns the full catalog. When supplied, the storage filter widens to [<kind>, 'any'] so audience-neutral rows remain visible inside the narrowed view.

Tagging a new skill as coordinator-targeted — set kind to SkillKind.COORDINATOR (or the literal "coordinator") when you skills(action='create', kind='coordinator', ...) or POST to /v1/api/admin/skills. Use this to signal intent to other skill authors and to make the orchestrator-targeted catalog easy to browse — not to hide the skill from interactive sessions. Existing rows default to SkillKind.ANY; bump them to COORDINATOR if you've rewritten the prompt around the orchestrator toolset and want the kind filter to surface them as such.


Tool surface differences

Coordinator sessions receive a fixed tool set, defined in turnstone/core/tools.py as COORDINATOR_TOOLS. Nothing a skill or MCP config can do adds to it. Current members:

Tool Category Notes
spawn_workstream delegate Create one child. Requires approval.
spawn_batch delegate Create up to 10 children in one approval. Partial-success shape.
inspect_workstream observe Read state + tail of one child. Auto-approved (no mutation).
list_workstreams observe List the direct children (same shape as /children endpoint).
wait_for_workstream block Block until one/all listed children hit a terminal state.
send_to_workstream steer Queue a follow-up message to a running child.
close_workstream wind-down Soft-close one child. Requires approval.
close_all_children wind-down Soft-close every direct child in one approval. Partial-success shape.
cancel_workstream wind-down Drop the in-flight generation; leaves child idle for a fresh send.
delete_workstream wind-down Hard-delete one child. Requires approval.
list_nodes discover Enumerate live cluster nodes + capabilities.
skills (action=find) discover Browse the skill catalog; opt-in kind filter narrows by audience.
memory persist Durable orchestration memory (coordinator scope, per-user — survives across coordinator sessions).
notify broadcast Post a status update to a human channel at a narrative beat.
tasks plan Orchestrator-only scratchpad. Children don't see it.

Explicitly not in the coordinator set:

  • bash / edit_file / write_file / append_file / diff_file — no local FS.
  • read_file / search — no local FS reads.
  • web_fetch / web_search — no direct web access.
  • task_agent — sub-agent tool is zeroed on coord sessions.
  • recall / watch / read_resource / use_prompt — UX / persistence tools that belong to interactive sessions. The dual-kind memory / skills / notify tools are available on both kinds (see the table above).

If your skill needs a coordinator to "run a command" or "read a file", write the delegate pattern instead: spawn a child with an appropriate skill, wait_for_workstream, then inspect_workstream for the output. The coordinator stays the orchestrator.


Persona differences

Interactive skills compose on top of base_interactive.md — a "maker" persona: get the work done, use the tools, edit the code, close the loop.

Coordinator skills compose on top of base_coordinator.md — an "orchestrator" persona: decompose, delegate, monitor, synthesise. The base text is short but sets the tone every coordinator skill inherits:

You are a coordinator on a small, focused infrastructure team. Your role is to orchestrate work across the cluster... You do not edit files, run shell commands, browse the web, or manipulate the codebase directly. Children do that.

Write your skill's system prompt to add task-specific orchestration hints on top — don't re-explain the role, don't paste tool JSON, don't try to override the "no direct action" contract. Keep the additions to: (a) the specific kind of work this skill delegates; (b) the preferred skill tags for children; (c) the synthesis shape the skill should end on.


tasks integration

tasks is the coordinator's scratchpad — a persisted, ordered list of rows with fields {id, title, status, child_ws_id, created, updated} that only this coordinator sees. Children don't see it; the user does via the sidebar. Five actions: add, update, remove, reorder, list (only list is auto-approved; the mutators go through the approval flow).

The input schema refers to rows by task_id; the persisted row object exposes the same id as id. The child_ws_id field is a free-form label the skill sets to link a task to a spawned workstream — it is NOT validated against the workstreams table, so a skill can set it to a placeholder before spawn_workstream returns or keep it pointing at a closed child for later audit.

A skill's initial prompt can seed the task list by calling tasks(action="add", title=...) as its very first tool calls — the user gets a visible plan before any child is spawned, and the coordinator's future self has something concrete to iterate on. Status transitions (pendingin_progressdone / blocked) are the skill's main feedback loop: mutate the task when the child covering it finishes, not when the child starts. Use tasks(action="update", task_id=..., child_ws_id=<ws_id>) to link a task to the child that owns it once spawn returns.

A final gotcha: parallel tool dispatch does NOT serialise reads after writes in the same batch. If a skill issues an update and a list in one parallel tool batch, the list response may reflect the pre-update state. Dispatch mutate and list serially (one tool_use turn each) when the list must observe the mutation.

Keep the tasks coarse-grained — one per child, roughly. A 20-task list for a 3-child fan-out is noise; a 1-task list for a 5-child fan-out loses the plan. The sidebar renders tasks as the operator's mental model of "what the coord thinks it's doing".


Referencing children by ws_id

Every ws_id returned by spawn_workstream / spawn_batch is a full 32-char hex string. The skill's system prompt must not invent ws_ids — a model that hallucinates "child-1" or "ws-abc" hits the tenant guard in CoordinatorClient._is_own_subtree, which validates ws_id against parent_ws_id=coord_ws_id AND user_id=owner in storage. The rejection shape is uniform and recovery-oriented:

  • Mutating ops (send_to_workstream, close_workstream, cancel_workstream, delete_workstream) and inspect_workstream return {"error": "no workstream matching '<ref>' among your children; …", "status": 404, "ws_id": "<ref>", "did_you_mean": [...], "children": [...], "children_truncated": bool} — a did-you-mean (edit distance ≤ 3 against the coord's own children, which catches the garbled-hex incident class: a 32-char id whose aaa run collapsed to a) plus a roster of the coord's children. A ref that matches a child's display NAME is called out explicitly with the right id (names are mutable labels, not addresses). Foreign and nonexistent ids produce the same payload (no existence oracle), every hint references only the coord's own children, and near-miss ids are never auto-resolved — the skill should fix the id and re-issue, not treat the child as dead.
  • wait_for_workstream validates ids before waiting: a malformed id fails the whole call immediately (invalid_ws_ids carries the per-id payloads above, elapsed=0); a well-formed id that is foreign, nonexistent, or hard-deleted mid-wait surfaces as state="not_found" and aborts the wait on that tick with top-level error / not_found / children fields. complete=true therefore means every polled lane really finished — an unobservable id can neither burn the timeout nor ride along to a "complete" result.

Pattern: capture each spawn result in the next tool call's input. The JSON tool-result carries {"child_ws_id": "...", "name": "...", "node_id": "...", "routing_strategy": "..."}; the model should extract the child_ws_id and pass it as ws_id (or in the ws_ids list) to inspect_workstream / wait_for_workstream / send_to_workstream / close_workstream verbatim. The asymmetry — spawn returns child_ws_id but the other tools accept ws_id / ws_ids — is intentional: it defuses a coordinator-LLM recency bias where seeing ws_id in a spawn return primed re-spawn loops instead of progression to the wait phase.

A UI that wants human-readable identifiers should render the name field and keep the workstream id as the click-through key — note that the id value is the same regardless of whether it arrived under the child_ws_id key (spawn return) or the ws_id key (every other tool's input/output); only the field name differs.


wait_for_workstream vs inspect_workstream

Two distinct semantics, different cost profiles:

  • wait_for_workstream(ws_ids=[...], timeout=60, mode="any") — blocks inside a single tool call until one (or all, for mode="all") of the listed children reaches a terminal state (idle, error, closed, deleted). The worker thread blocks up to timeout seconds; the assistant turn remains a single round-trip regardless of how long the wait actually takes. Prefer this for "the plan needs child X to finish before the next step."
  • inspect_workstream(ws_id=...) — single read of the child's state + tail. Costs a full assistant turn (judge, tokens, stream). Prefer this for "what does the final message say?" after the child has already resolved (via wait_for_workstream or a known transition).

Rule of thumb: wait once for a fan-out, then inspect once per child for the content. A loop of inspect-every-few-seconds is a token-burning antipattern — on 3+ children it rounds to a 10× efficiency hit over a wait+inspect pair.


Common coordinator patterns

Three patterns cover most coordinator skills. Pick the one that matches the task, or combine them deliberately.

Pattern 1 — delegate-and-summarise

One specialist child, one focused brief, one synthesis message back to the user. Appropriate when the user's request is "run the thing and tell me what happened" and the work fits in one workstream.

tasks(action='add', title='audit /auth for CSRF')
spawn_workstream(skill='engineer', initial_message='audit /auth ...')
wait_for_workstream(ws_ids=[<child>], timeout=300)
inspect_workstream(ws_id=<child>)
→ synthesise the final message into a user-facing response
tasks(action='update', task_id='t_01', status='done')
close_workstream(ws_id=<child>, reason='audit complete')

Pattern 2 — fan-out-and-synthesise

N children running in parallel, each with a distinct brief, all waited-on together, then synthesised. Appropriate when the user's request naturally decomposes into independent subtasks.

tasks seeds:
  t_01 benchmark Anthropic 4.7 latency on summarisation
  t_02 benchmark OpenAI GPT-5.2 latency on summarisation
  t_03 benchmark Gemini 2.5 latency on summarisation
spawn_batch(children=[...3 briefs...])
wait_for_workstream(ws_ids=[c1, c2, c3], mode='all', timeout=600)
inspect_workstream(ws_id=c1); ...(c2); ...(c3)
→ synthesise head-to-head comparison
tasks → all done
close_all_children(reason='benchmark complete')

Prefer spawn_batch over 3 individual spawn_workstream calls — one approval instead of three, one audit trail, deterministic sibling ordering. Pair with wait_for_workstream(mode='all') and close_all_children(reason=...) to wind the fan-out down in one approval each.

Pattern 3 — plan-then-delegate

The coordinator first uses its own reasoning to carve the plan, records it in tasks, then spawns children that each own one task. Appropriate when the user's request is "figure out how to X" and the coordinator's planning step is itself valuable.

→ coord reasons about the shape of the work
tasks(action='add', title='...') × N   # the plan, visible in the sidebar
for task in tasks:
    spawn_workstream(skill=..., initial_message=task.brief)
    tasks(action='update', task_id=task.id, notes='ws=<child_ws_id>')
wait_for_workstream(ws_ids=[...], mode='all', timeout=...)
for child in children:
    inspect_workstream(ws_id=child)
    tasks(action='update', task_id=..., status='done', notes='result summary')
→ synthesise

The key distinction from Pattern 2: the plan is an artifact the user can see and interact with (via the sidebar). If the coordinator's reasoning-pass was wrong about the decomposition, the user can course-correct before any child runs.


Testing a coordinator skill

Coordinator sessions are hosted on the console, not on a node. Integration tests that drive a real coord session live under tests/test_coordinator_end_to_end.py — they spin a console with an in-memory SQLite backend and a fake upstream node, then drive the session through its HTTP surface.

For a new coordinator skill:

  1. Write the skill prompt as a string and pass it to the coord_session fixture's skill= kwarg (see tests/test_coordinator_tools.py for the pattern).
  2. Build a small fake cluster: one node + two children via the _seed_children helper in tests/_coord_test_helpers.py (_seed_children(mgr._adapter, coord.id, ["child-1", "child-2"])).
  3. Drive the session with seeded tool_call dicts matching the provider layer's shape. The unit-level tests in tests/test_coordinator_tools.py show the helper (_tc(name, args, call_id)).
  4. Assert the skill's decision shape — which tools fire in what order, what the tasks looks like at the end, which _error reasons appear on the denied-path.

A full end-to-end test isn't required for every skill; a prepare-step unit test that asserts "given this initial message, the first tool call is X with Y args" is usually sufficient to catch persona drift without a real LLM in the loop.


Further reading

  • coordinator-api-tour.md — the HTTP surface every coordinator skill indirectly drives.
  • bulk-endpoints.md — the response shape spawn_batch and close_all_children use, so your skill can parse results / denied arrays correctly.
  • governance.md — the broader governance surface (/trust, /restrict, role-based permissions) that wraps every coord session.
  • settings.mdcoordinator.model_alias and coordinator.reasoning_effort settings that gate which LLM runs the coordinator session at all.