Spec models now describe what the endpoints do: ListPersonasResponse
declares the tool_inventory the shelf depends on, both console create
models declare persona, CreatePersonaRequest declares org_id, and
UpdatePersonaRequest documents the null-vs-absent split (null clears
base_prompt/tool_allowlist, null on flags/kinds is ignored). Console
OpenAPI regenerated.
Protocol contracts match the implementations: update_persona's return
covers the no-op case, create_persona's raises-list is complete, and
both extended row-shape docstrings gain their tail columns plus the
append-only rule. The workstreams.persona comments say slug, not
display name.
Page corrections from the docs review: personas.md documents the
creative_mode-to-writer migration conversion, the mid-session /resume
MCP-lever behavior, visibility-based nudge gating, the soft-set
prompt-cache cost, and the executive tool list — and drops internal
jargon. The changelog entry moves under [Unreleased] with the house
breaking-marker style and the auto-conversion note. coordinator-skills
and the API tour stop using persona to mean framing; governance,
api-reference, sdk, console, tools, and memory pick up the new
permission family, endpoints, kwargs, picker, and lever caveats.
The persona resolved at creation is snapshotted into workstream_config
(five keys, all-or-none) and applied ONLY from the stamp — the personas
table is never read post-create, so edits/archives never touch existing
workstreams, and a corrupt stamp fails construction loudly instead of
silently reverting to a default envelope. Legacy pre-063 workstreams
carry no keys and keep today's behavior byte-for-byte.
The four levers (turnstone/core/personas.py holds the codec):
1. Base override — compose_system_message(base_override=...) replaces
exactly the BASE module; ENV/CONTEXT/TOOLS/POLICIES keep composing so
mandatory prompt policies ride on top of every persona. This also
closes the old /creative hole where the fork bypassed composition
(no CONTEXT, no DB policies).
2. Tool visibility — the allowlist intersects both the composition name
set (TOOLS block self-suppresses, tool-gated policies drop, the
memory advisory drops) and the END of _get_active_tools so the wire
never advertises hidden tools. tool_search in the set = soft
(discovered tools union with the allowlist via the session's
expanded-names set); absent = hard (the whole pathway is disabled,
covering provider-native defer_loading, which has no synthetic name
to filter). Persona sets force client-side tool search.
3. MCP gate (session-wide) — an MCP-off persona drops the client
reference at construction: no merge into _tools OR _task_tools, no
listeners, refresh callbacks inert, resource/prompt catalogs gone.
4. Memory (own hands only) — no recall injection, memory-directed
nudges suppressed (MEMORY_NUDGE_TYPES; behavioural nudges keep
firing), memory tool hidden. _task_tools is NOT filtered; compaction
spill/markers are never persona-gated, and the post-compaction recall
pointer is emitted only when the recall tool is actually visible.
Threading: the create handler resolves once (explicit name -> 400 on
unknown/disabled/kind-mismatch; empty -> the kind's default; pre-seed DB
-> unstamped legacy) and stamps via constructor kwargs + config keys +
the workstreams.persona column; SessionManager.open threads the stamp
pre-construction exactly like the saved model alias. Non-fork resume
adopts the target's stamp so _save_config can't clobber it. spawn /
spawn_batch gain a persona arg with prep-time validation (children are
interactive-kind; omitted = kind default, never the parent's). Python +
TS SDKs, OpenAPI specs, the picker feed GET /v1/api/personas (authed, no
perm), and console admin CRUD /api/admin/personas (persona.* perms,
archive-only — no DELETE) round out the surface.
BREAKING: /creative is removed (the REPL command now points at the
writer persona); turnstone --persona <name> is the replacement. Also
fixes the CLI session factory, which TypeErrored on the project_id
kwarg the shared InteractiveAdapter passes unconditionally.
* feat(projects): governed project containers — memory scope, grouping, manage UI
A workstream can attach to a project: a first-class, shareable resource
container that owns a `project` memory scope, groups conversations, and is
managed from the console.
Storage / migration 062: projects + project_members tables, workstreams.
project_id, and the memory type default project→general; grants
project.{create,read,write,delete} (admin-default).
Recall + writes: project memory is recalled iff the workstream is attached AND
the user has access (owner ∨ member ∨ public-for-read), resolved once at session
construction; coordinators recall it too. New saves default to the project when
attached + writable; the save and delete paths are write-gated; deleting a
project purges its scoped memory; archived projects aren't recalled.
Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a
single-fetch resolver); visibility changes, member management, and delete are
owner-only.
API: project CRUD routes on both the server and console; project_id threaded
through workstream creation, spawn inheritance, the cluster-create proxy, the
dashboard / snapshot / coordinator row builders, and the collector deltas.
UI: a project picker with an inline "+ New project" creator in every creation
box (console launcher + standalone dialog + dashboard); group-by-project in the
rail; a project badge in the composer and on dashboard rows; a console manage
tab (list + create/edit + members shelves). The admin Memories view gains
coordinator/project scope filters and human scope labels (name, not hex). The
memory tool schema documents the project scope and the attach-aware default.
* fix(projects): client refresh hardening, creator race guard, SDK project_id
Addresses PR #724 review feedback plus two bugs found while validating it.
- projects.js refreshProjects: a non-OK status (e.g. 403 when the caller
lacks project.read) or a network/parse error no longer blanks the cache
or masquerades as "no projects" -- the prior cache is preserved, the
failure is recorded (new projectsError()) and warned. Honors the
long-standing "a transient error can't blank the rail" docstring.
- projects.js _fp: the fingerprint separators were raw control bytes,
which made git treat the whole file as binary (no reviewable diff).
Rewritten as escape sequences instead of raw bytes -- behavior is
byte-identical at runtime.
- project_creator.js: createProject() could reject unhandled (authFetch
throws on network/401; r.json() throws on a non-JSON body), leaving the
widget stuck busy/disabled. Added a .catch, plus a generation guard so a
create whose widget was cancelled/reopened mid-flight drops its result
instead of selecting a project the user backed out of.
- types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo /
DashboardWorkstream to match the server schemas (was SDK-invisible).
- test_project_api.py: move side-effecting HTTP calls out of asserts so
the requests run even under python -O.
* fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators
_fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.
- Remove the unused PerceptionUnavailableError (never raised/caught/imported).
- Reword the now-shipped 'Phase 3' placeholder comments on the Anthropic + OpenAI-Responses audio paths to describe the live upstream STT/perception fallback (these placeholders are defensive, not pending work).
- Clarify the no-vision image fall-through comment (fires when perception is unconfigured OR can't see, not only the former).
- Type AttachmentInfo.kind as the image|text|pdf|audio union in the TS SDK.
- extract_pdf_text: append the truncation marker only when there's actual text, so a scanned PDF over the page cap returns '' (-> placeholder) instead of a content-free document part.
- composer: accept pdf/audio in the upload picker; client-side kind
inference for the optimistic chip (server classify_upload stays
authoritative)
- admin Models tab: supports_pdf + supports_audio_input toggles (flow
through the field-aware capabilities merge into ModelCapabilities, so
flipping supports_audio_input on an omni alias enables native input_audio)
- docs: AttachmentInfo.kind, AttachmentUpload, and the TS SDK note pdf/audio
* chore: relicense BUSL-1.1 -> Apache 2.0 for 1.6.0
Flips every license artifact in the tree; 1.5.x and earlier remain
BUSL-1.1 per their release-time LICENSE files. Contributor consent
record: #548 (rationale: #546).
- LICENSE: canonical Apache 2.0 text
- NOTICE: new; copyright line + pointer to THIRD-PARTY-NOTICES
- pyproject.toml: SPDX expression + explicit license-files trio
- Dockerfile: COPY the license trio (hatchling needs them at build)
- THIRD-PARTY-NOTICES: BUSL line reworded; bundled-version drift
fixed (KaTeX 0.17.0, Mermaid 11.15.0, hls.js 1.6.16)
- README badge + License section, CONTRIBUTING inbound-license line,
TS SDK package(+lock), example pyproject
- docs/pgbouncer.md: drop stray ':' introduced in #353
* docs: add CONTRIBUTORS.md
* chore: drop LICENSE leading blank line
The apache.org LICENSE-2.0.txt begins with a newline; the SPDX
canonical text and GitHub license templates do not. Use the
conventional form — detection is whitespace-normalized either way.
- A1: add a shared `operator-context` marker to every operator-context row in
both UIs; the retry-skip walk keys on it, so a trailing watch-result /
guard-finding / idle-children card no longer makes retry regenerate the
wrong turn. Pinned by source-grep tests + a headless-DOM self-test.
- A2: wrap get_content's two sync DB gates (get_attachment + the unbounded
ws-scoped attachment_referenced_in_ws LIKE scan) in asyncio.to_thread so a
long scan can't stall the event loop — matching the module's convention.
- A3: add the HistoryEvent attachments docstring bullet; drop the dead
`interjection` class; document the interactive-only system-context label;
correct the SDK attachments-meta docs to {kind, filename, mime_type}
(size_bytes is not carried through the history projection).
Operator-context system turns (watch results, output-guard findings, idle
children, user interjections) carried their kind (_source) and a flattened
text content, but the structured per-kind fields were dropped at every
persist/deliver boundary — so the UI rendered every kind as one generic
operator bubble and the structured watch-result card was lost.
Wire the structured meta through as the single source of truth:
- Storage: new conversations.meta JSON column (migration 060); threaded
through save_message/save_messages_bulk (facade + protocol + both backends)
and rehydrated in reconstruct_turns onto Turn.meta.extra["source_meta"].
- Canonical: make_system_turn carries meta as one _source_meta dict;
turn_from_dict/turn_to_dict bridge it to/from Turn.meta.extra.
- Live + history: widen on_system_turn(content, source, meta) across all
impls + the SSE payload; surface _source_meta -> meta in the /history
projection. SDK HistoryEvent docs note the field.
- Producers derive both the model-facing content text AND the card from one
meta dict, so they cannot drift: render_output_guard_text, build_watch_
reminder carrying output, idle_children and user_interjection metadata.
- Frontend: addSystemContext / renderSystemTurn dispatch by source to the
watch-result, guard-finding, idle-children, and queued-message cards in
both the interactive and coordinator panes; every untrusted field renders
via textContent.
The meta is a leading-underscore key, stripped before the wire (sanitize_
messages and the native mid-conversation path copy only role+content), so the
per-provider wire payloads stay byte-identical. Additive column, no backfill:
operator turns predating it reload as plain text bubbles.
A deep-dive review of the branch surfaced a budget regression, SDK doc
drift, dead code, and stale docstrings. Each was boundary-spiked before
fixing.
- R1 (regression): by-reference document attachments were invisible to the
token budget — _msg_text_chars returned 0 doc_chars for a
{type:document,attachment_id} placeholder, and the comment's claim that
the budget "lands at calibration" was false (calibration discards
doc_chars). Thread the doc size through _attachments_meta (size_bytes, at
both the live-append and reconstruct build sites) and count it in
_msg_text_chars, guarded against double-counting the inline form.
Regression test added.
- F1: the history-DTO schema description and the TS HistoryEvent docstring
still advertised the removed reminders/advisories keys and omitted the
system role; corrected server_schemas.py + sdk/typescript/src/events.ts to
match the shipped shape. The committed OpenAPI JSON snapshots were already
~679 lines stale on main; their regen is left to its own chore branch.
- D1: removed AttachmentBuffer.take() — dead (no production caller; the
commit path uses discard()) and scope-weak (ws_id only, unlike its
siblings) — with its test and the now-orphaned Iterable import.
- O1: 4 docstrings referenced the moved ChatSession._fold_system_turns →
lowering.fold_system_turns.
`man` and `math` duplicated capabilities already reachable through
`bash`; `plan_agent` is better expressed as a `task_agent` running a
planning skill, and carried a large amount of special-case machinery
(plan-review gate, refinement loop, per-kind model routing). Removing
all three shrinks the tool surface and cuts per-call token cost.
Also removed, as dead-once-the-tools-are-gone:
- the `math` sandbox executor (`turnstone.core.sandbox`) and its
`[sandbox]` extra; the eval analyst now runs bash-only
- the read-only `AGENT_TOOLS` sub-agent tool set and the `agent`
tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained)
- the plan-review protocol end to end: the `on_plan_review` UI hook,
`resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`,
the `plan_review`/`plan_resolved` SSE events, and their Python SDK /
TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings
- the `model.plan_alias` / `model.plan_effort` settings and the
registry `plan_model` / `plan_effort` routing fields
TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged.
BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the
plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings
from the experimental 1.6 line.
GET /v1/api/workstreams/saved returned only ws_id/alias/title/created/
updated/message_count — too little to drive the planned saved-list table
redesign. Add seven fields, all sourced from already-persisted data (no
migration):
- state, kind, node_id: columns on the workstreams table
- model_alias, launch_skill: from workstream_config via LEFT JOIN
- child_count: COUNT of child workstreams via parent_ws_id
- context_tokens: most recent usage_events prompt size for the workstream
- context_ratio: context-window occupancy (context_tokens / model context
window), computed in the handler so the NULL / zero-window cases stay
explicit and identical across both storage backends
context_window comes from a model_definitions join; aliases defined only in
config.toml are absent there, so context_ratio degrades to 0.0 rather than
reporting bogus occupancy. The Python SDK reuses the Pydantic model; the
TypeScript SDK OpenAPI snapshot and hand-maintained interface are updated.
Tests cover the new storage columns (including NULL-when-absent), the
handler ratio math + zero-window degradation, and the SDK enriched
round-trip.
openapi-server.json / openapi-console.json had drifted well behind
build_server_spec() / build_console_spec() — the committed snapshots are
regenerated periodically (via sdk/typescript/scripts/generate-types.py)
rather than on every schema-changing PR, so accumulated additions (skill
parsing, pending-approval items, model-definition CRUD, etc.) had not been
captured. This resyncs both with no code changes.
Lift the conversation-modifying /rewind and /retry verbs out of the body-keyed POST /v1/api/command into path-keyed POST /v1/api/workstreams/{ws_id}/rewind ({turns:N}) and /retry, as make_rewind_handler/make_retry_handler in SharedSessionVerbHandlers (template: make_close_handler/make_cancel_handler), wired on both interactive and coordinator kinds. Closes the last unlifted conversation-modifying surface — coordinator workstreams gain rewind/retry where they had none — and removes the surviving exception to the post-#422 path-keyed URL convention.
Handler shape: auth gate (coord -> admin.coordinator via permission_gate; interactive -> conversation.modify via accepted_permissions) -> busy-gate -> session.rewind(n)/retry() -> always emit clear_ui (incl. rewind-to-zero, carries #503) -> audit (conversation.rewind/retry on both kinds). Retry re-dispatch reuses the shared session_worker.send via a per-kind dispatch_retry closure (hard-reject on busy), not a third hand-rolled thread.
The web /command handler now rejects /rewind+/retry with a pointer to the path-keyed endpoint (BREAKING; 1.6.0aN-tolerant); session.handle_command's branches stay for the terminal CLI. auth.py adds the verbs to both write suffix-sets; Python + TS SDKs, OpenAPI (RewindRequest + server/console specs), the /route/ proxy mounts + audit actions, and coordinator_client all gain them.
Interactive frontend (app.js): the 3 /command POST sites + the hand-typed-slash reroute now hit the path-keyed endpoints; the bare .msg.user rewind selector is kept (matches the server's _find_turn_boundaries, which counts system-nudge user turns). The coordinator frontend rewind UX lands in a follow-up commit (browser-verified).
Tests: route-walk mount/order, /route/ audit rows, required_scope, OpenAPI catalog, SDK body-inspection, and HTTP-level handler behavior (busy-gate, turns validation, clear_ui emit, retry dispatch, audit invocation + swallow).
PR #498 round-robin review surfaced 5 findings. 4 applied; 1 rejected
with rationale.
Applied
* **Copilot finding 5** (history_decoration.py:341): dispatcher
inspected only ``provider_content[0]['type']``. OpenAI Responses
captures EVERY ``output_item.done`` event into ``provider_blocks``
(not just reasoning) — in practice the order is
``[reasoning, message, ...]`` but the API doesn't guarantee that;
a hypothetical ``[message, reasoning]`` ordering would silently
drop the reasoning under an index-only check. Now walks the list
for the first block whose type is in ``_BLOCK_TYPE_PROVIDER_FACTORY``,
then dispatches the WHOLE list to that provider's extractor. Each
provider's extractor already filters internally by its own block
type, so passing the full list is correct. Regression test added
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
* **Copilot finding 3** (migration 052 docstring): the previous
review-fix wave used sed to rename ``persist_reasoning`` →
``surface_persisted_reasoning`` everywhere, which mangled a
historical reference in the migration docstring ("The earlier name
``surface_persisted_reasoning`` was renamed..."). Restored to
point at the actual pre-rename name (``persist_reasoning``).
* **Copilot finding 4** (sdk/typescript/src/events.ts:26):
``HistoryEvent`` JSDoc still referenced ``persist_reasoning`` —
the sed rename only walked ``turnstone/`` and ``tests/``, missing
the TypeScript SDK. Updated to ``surface_persisted_reasoning``.
Also widened the comment to cover all three reasoning-bearing
block types (Anthropic ``thinking``, OpenAI Responses ``reasoning``,
synthetic ``reasoning_text``) instead of mentioning only Anthropic.
* **github-code-quality finding** (session.py:1120): ``_resolve_server_type``
had a bare ``except Exception: pass``. Replaced with a
``log.debug(..., exc_info=True)`` + explanatory comment. Behaviour
unchanged (still returns ``""`` on any lookup failure); failures
are now observable under DEBUG triage.
Rejected (with rationale)
* **github-code-quality finding** (_protocol.py:265):
``extract_reasoning_text``'s body is ``...`` per ``LLMProvider``
Protocol convention. Every method in the file uses ``...`` (PEP
544 idiomatic Protocol style). Changing only this one to
``raise NotImplementedError`` would be inconsistent with the rest
of the file. CodeQL's "statement has no effect" warning is
technically correct for ``...`` as a standalone expression but
ignores the documented Python Protocol convention. No fix.
Docs sync
* docs/api-reference.md: ``history`` SSE event message-shape table
gains the optional ``reasoning`` field.
* docs/architecture.md: ``ModelCapabilities`` row in the type table
gains ``supports_reasoning_replay``; ``StreamChunk`` and
``CompletionResult`` rows gain the existing ``provider_blocks``
field (was missing pre-PR). New "Per-model reasoning persistence"
subsection under the Models config section, documenting the two
flags + capability gate + three reasoning paths + cross-provider
shape filter.
* docs/settings.md: new "Reasoning persistence (per-model)"
subsection with the two-flag table and capability-gate note.
* docs/diagrams/03-core-engine-classes.puml: ``LLMProvider`` interface
adds ``extract_reasoning_text`` + the new ``replay_reasoning_to_model``
kwarg; ``ModelCapabilities`` class adds ``supports_reasoning_replay``.
PNG regenerated.
Lint + test gate
* ruff check + ruff format clean.
* mypy clean (191 source files).
* pytest -m 'not live' — 6116 passed (3 deselected), +1 net new test
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
Surface stored Anthropic thinking blocks on /history responses so
refreshing the page rehydrates the reasoning bubble. Wire payloads
unchanged. Per-model operator knobs added to model_definitions for
both UI rehydration and (Phase 2) wire-build replay.
Why now: reasoning is already round-tripped via _provider_content for
Anthropic-with-thinking turns, but never surfaces on the history wire,
so a tab reload showed only the final answer with no rationale.
Operators also have no per-model lever to opt out of UI display or to
opt in to replay-to-model on subsequent calls.
What this change does
* Migration 052 adds two boolean columns to model_definitions:
persist_reasoning (default 1) controls UI rehydration; replay_
reasoning_to_model (default 0) reserved for Phase 2's wire-build
shape filter. Mirrors the enabled column pattern (NOT NULL +
integer server_default).
* LLMProvider Protocol gains extract_reasoning_text(provider_blocks)
with concrete impls on AnthropicProvider (walks type=='thinking'
blocks, joins with newline, caps at 64 KiB) and no-op stubs on
OpenAIChatCompletionsProvider + OpenAIResponsesProvider. Google
inherits the no-op via OpenAIChat. Phase 3 will wire the OpenAI
Responses extractor once include=['reasoning.encrypted_content']
is requested.
* turnstone.core.history_decoration gains a structural dispatcher
extract_reasoning_text_from_provider_content keyed off the first
block's type field (Anthropic 'thinking' / OpenAI Responses
'reasoning' / Gemini 'thought' are non-overlapping by API design).
Both history surfaces use it: _build_history calls the dispatcher
directly (the SSE-replay path builds entry dicts from scratch),
and the lifted make_history_handler runs the list-helper variant
in the existing to_thread block.
* make_history_handler resolves persist_reasoning via three tiers:
live session -> workstream_config.model_alias (the same key
SessionManager uses to rehydrate the original model after process
restart) -> conservative True default. Operator flag-flip takes
effect uniformly on both warm and cold workstreams.
* Frontend: app.js replayHistory and coordinator.js role==='assistant'
branch each call the existing reasoning-bubble construction (for
app.js, the document.createElement pattern from the live SSE
handler; for coord, the appendMsg('reasoning') helper) when
msg.reasoning is non-empty. Reasoning bubbles render before the
content bubble, matching live SSE order.
* Admin UI: two checkboxes ('Persist reasoning', 'Replay reasoning
to model') in the model edit modal, plus override-pill display in
the model row when set to non-default values.
What is intentionally out of scope
* Phase 2 -- ANTHROPIC_VALID_BLOCK_TYPES shape filter at
_anthropic.py:312-316, _convert_messages replay_reasoning_to_model
parameter, thinking-strip branch, _msg_text_chars token-calibration
extension. The replay flag is stored but not consumed on the wire.
* Phase 3 -- OpenAI Responses include=['reasoning.encrypted_content'],
Gemini include_thoughts spike, ModelCapabilities.supports_
reasoning_replay.
* Phase 4 -- Local-model / chat-template reasoning persistence
(session.py:3486 reasoning_parts accumulator).
Tests
* AnthropicProvider.extract_reasoning_text -- 13 unit tests covering
None / empty / mixed / multi-block / cap / malformed / non-list
inputs plus other-provider no-op verification (real provider
instances, no mocks).
* extract_reasoning_for_history -- 10 dispatcher tests including
block-type discriminator routing (thinking vs reasoning vs
unknown), strip-when-flag-false, empty / non-dict guards, and
cross-role isolation.
* _build_history -- 6 boundary tests through the real Anthropic
extractor with stub sessions, including the registry-lookup
failure default-True branch.
* make_history_handler -- 5 round-trip tests through real storage:
the storage layer's reconstruct_messages decodes provider_data
into _provider_content, and the helper extracts through the real
AnthropicProvider. Includes the live-session flag honoring path,
the cold-workstream workstream_config lookup path, and the
no-alias default-True fallback path.
* Audit-log discipline -- 4 structural mock-and-assert tests that
capture every Logger.info / warning / error call across the
pipeline (extractor, dispatcher, list-helper, _build_history)
and assert no captured payload contains a marker reasoning string.
* model_definitions storage -- 6 round-trip tests: default flags,
explicit create with both flags, individual update of each flag,
and list-includes-flags assertion.
* model_registry -- 4 tests: dataclass defaults, dataclass with
explicit flags, DB-row-mapping with both flags, and pre-052
legacy-row default-fallback.
Edge cases pinned by the test suite
* Pre-052 DB rows missing the new columns degrade to dataclass
defaults (test_db_reasoning_flags_default_when_absent).
* Live session in memory has its flag honored (test_history_handler_
with_persist_flag_false_via_live_session).
* Cold workstream resolves the flag via workstream_config +
app.state.registry (test_history_handler_cold_workstream_resolves_
via_workstream_config) -- this closes the gap where a process
restart would have silently un-honored an operator flag-flip.
* Cold workstream without persisted model_alias falls through to
default True (test_history_handler_cold_workstream_no_alias_
defaults_true).
* Foreign / unknown / missing block types degrade silently to no
reasoning field rather than misroute or crash.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6030 passed (3 deselected).
Refreshing a coordinator or interactive workstream pane while the LLM
is mid-stream now restores the partial assistant text + reasoning
immediately and flips the composer back to stop-mode, instead of
showing nothing until the response completes.
Per-turn inflight buffers (`_ws_inflight_content`, `_ws_inflight_reasoning`,
`_ws_inflight_seq`) on `SessionUIBase` are kept separate from the
existing multi-turn `_ws_turn_content` buffer that drives the
dashboard's IDLE-piggyback payload. New `on_turn_start` (top of
send-loop, defensive) and `on_turn_committed` (right after
`messages.append(assistant_msg)`, primary) lifecycle hooks reset
inflight at turn boundaries. The seq counter is monotonic across
turns so a long-lived subscriber's `snap_seq` cutoff stays valid for
the lifetime of the connection — resetting per-turn would silently
drop turn N+1's first M tokens (M = whatever was streamed pre-snapshot
in turn N).
`snapshot_and_consume_state_payload` also drains inflight at idle/error
so cancel and exception paths don't leak stale text. New
`register_listener_with_in_progress_snapshot` atomically registers a
listener and snapshots the inflight buffers; `make_events_handler`
emits a `state_change` event (so the JS busy machine flips to
stop-mode) followed by a one-shot `in_progress_snapshot` after the
kind-specific replay, then strips the internal `_seq` field from
yielded live events while filtering against `snap_seq`. A per-listener
shallow `dict` copy in the live drain prevents the multi-tab race
where one listener's `del event["_seq"]` would corrupt another
listener's filter view.
`_synthesize_cancelled_results` now emits synthetic `on_tool_result`
events for each cancelled tool so live coord tabs can drop the
newly-additive `coord-tool-batch--running` indicator cleanly. The
indicator now coexists with `--auto`/`--approved` (applied on
`tool_info` and `approval_resolved` approved; removed when every row
in the batch has a result), making live tool execution visually
parallel to the replay-time orphan rendering.
Frontend handlers in `app.js` (interactive) and `coordinator.js` (coord)
absorb EventSource auto-reconnect re-replays via a length-based
prefix check on the in-progress buffer. New `InProgressSnapshotEvent`
+ `StateChangeEvent` dataclasses in the Python and TypeScript SDKs
with type guards.
`_MAX_TURN_CONTENT_CHARS` lifted 256 KiB → 512 KiB (single constant
for both buffers — headroom for current commercial models).
Regression tests cover race-free composition under concurrent writers,
seq-filter dedup invariants, the cross-turn seq monotonic invariant,
idle/error inflight drain, synthesized `on_tool_result` on cancel
(including UI-hook failure isolation), and the multi-listener
shared-dict invariant.
Four themes from a coordinator-feature shakedown:
1. Correctness fixes (return shapes / examples / behavior)
- tools_coordinator.md: drop fake skill names from spawn examples;
fix wrong kwarg ``node_id=`` → ``target_node=``.
- wait_for_workstream.json: document ``message`` + ``truncated``
per-ws fields (always enriched in the client; the JSON shape
lagged the docstring).
- cancel_workstream.json: document the conditional ``dropped``
payload — ``was_running`` always present when ``dropped`` is,
``pending_approval`` and ``queued_messages`` conditional sub-shapes.
- spawn_workstream.json: document full return shape including
``routing_strategy ∈ {rendezvous, target_node, resume}`` and
``status``.
- close_all_children.json: clarify ``skipped`` covers BOTH
hard-deleted children AND already-closed-and-evicted children
(wire shape doesn't distinguish); drop incorrect "echoed back
in response" claim — server returns ``{status, closed, failed,
skipped}``, never echoes ``reason``.
- console/server.py: comment in ``_fanout_on_children`` clarifying
that the 400 "No session" branch fires for cancel-cascade
callers and is unreachable from close_all_children (close
handler 404s instead).
- coordinator_client._utc_now_iso(): switch to bare ISO format
matching the rest of the storage row format used in the codebase.
2. Tightened the 11 longest tool descriptions (~23% cut on the
coord set). Removed ALL-CAPS emphasis, normalised em-dashes,
dropped informal phrasing. No new claims.
3. Removed static approval annotations from descriptions.
Approval is governed at runtime by the unified ``approve_tools``
body and admin-defined ``tool_policies`` (#436); static
"Auto-approved" / "Approval required" / per-action approval
tags become a stale signal. Field names (``pending_approval``)
and operational verb behaviour ("cancel unblocks pending
approvals") stay.
4. Renamed ``task_list`` coord tool → ``tasks``. The previous name
compounded the bare word ``task`` (which collides with chat-template
channels on local models — same reason ``task_agent`` carries
the suffix); the plural form sidesteps the collision and reads
more accurately, since the tool acts on the whole list rather
than a single task. Sweep covers tool JSON, Python methods (5
client methods + 2 session methods + 1 helper + 1 constant),
audit event name (``task_list.update`` → ``tasks.update``), log
tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``),
frontend SSE event matcher, prompts, docs, and tests. CHANGELOG
entry added.
Plus: dropped the ENV block (Output Environment / Available
rendering / Formatting principles) from coordinator system
prompts. Coordinators orchestrate rather than render rich output
to the user, so the rendering capability matrix is not actionable
for them. Coord prompt drops ~29% (6309 → 4493 chars).
SDK regeneration via ``generate-types.py`` updates both
``openapi-console.json`` (the rename's downstream change) and
``openapi-server.json`` (PR #436 drift — its merge added
``pending_approval_detail`` + ``recent_auto_approvals`` fields to
the Python schemas but didn't regenerate the JSON artifact).
## Behavior changes (operator-visible)
- Audit event name: ``task_list.update`` → ``tasks.update``.
Audit dashboards / SIEM filters / log greps that pinned the old
prefix should update.
- SSE ``tool_result`` events now ship ``name="tasks"`` for the
scratchpad tool. The bundled coord-tree UI is updated atomically;
external consumers reading SSE events by tool name need to update.
- Existing task envelopes in production storage have ``+00:00``
timestamps from the old ``_utc_now_iso``. New writes are bare;
old rows are not backfilled. Within an envelope you may briefly
see mixed formats until each row is re-touched. No code path
string-compares timestamps within an envelope, so this is
cosmetic.
## Validation
- ``ruff check`` + ``ruff format --check`` clean
- ``mypy turnstone/`` clean (175 source files)
- ``pytest -m "not live"`` — 4679 passed, 3 deselected
Copilot review on PR #422 flagged that the DELETE-on-send (dequeue)
EndpointSpec declared no request_model, so the generated OpenAPI
showed no requestBody for an operation that *requires* a JSON body
with ``msg_id`` and 400s when it's missing.
- Add ``DequeueRequest`` to ``server_schemas.py`` with the single
required ``msg_id: str`` field.
- Wire ``request_model=DequeueRequest`` and ``response_model=
StatusResponse`` on the DELETE EndpointSpec; trim the now-redundant
inline body example from the description.
- Re-import the schema in ``server_spec.py`` and add the entry to
``_ALL_MODELS`` so the OpenAPI components list carries it.
- Regenerate ``openapi-server.json``.
Sibling thread on the close EndpointSpec was already addressed in
4000ae2 (request_model=CloseWorkstreamRequest).
4558 tests passing under ``-m "not live"``; ruff + mypy clean.
Copilot caught three real issues in PR #422 review, all clustered
around the close request body contract:
1. The interactive close handler runs with
``supports_close_reason=True``, which calls
``read_json_or_400(request)`` — an empty / non-JSON body returns
``400 {"error": "Invalid JSON body"}``. The previous SDK fix
sent NO body via ``json_body=None``, which would 400 against a
real server. The mock-transport test silently masked it because
the mock answered without inspecting the body.
2. The doc said the body was empty (or ``{}``), with no mention
of the optional ``reason`` field, its 512-byte cap, or the
credential-redaction guard.
3. The Pydantic schema for close was deleted outright; OpenAPI
and SDKs lost their typed shape for the optional ``reason``.
Changes:
- ``turnstone/api/server_schemas.py``: reintroduce
``CloseWorkstreamRequest`` with a single optional
``reason: str | None = None`` field. Docstring documents the
must-be-valid-JSON contract and notes that coord ignores the body
(``supports_close_reason=False``).
- ``turnstone/api/server_spec.py``: re-import the schema, point the
close ``EndpointSpec`` at it via ``request_model=``, restore the
``_ALL_MODELS`` entry. OpenAPI JSON regenerated.
- ``turnstone/sdk/server.py``: ``close_workstream`` (sync + async)
gains an optional ``reason: str | None = None`` parameter and
always sends ``json_body={}`` (or ``{"reason": ...}``) so the
body is never empty. Adds a regression test
(``test_close_workstream_sends_valid_json_body``) that inspects the
raw transport content rather than relying on a path-keyed mock —
the kind of check that would have caught this bug pre-merge.
- ``sdk/typescript/src/server.ts``: ``closeWorkstream`` gains an
optional ``opts.reason`` parameter; reintroduce
``CloseWorkstreamRequest`` interface in ``types.ts`` and re-export
from ``index.ts``.
- ``docs/api-reference.md``: close section documents the JSON-body
requirement, the ``reason`` field, the 512-byte cap, the
multibyte-safe behavior, the credential-redaction guard, and the
non-string-coercion path.
- ``CHANGELOG.md``: amend the 1.5.0 BREAKING block to reflect the
schema reintroduction (slim form, ``reason`` optional) instead of
the prior "removed outright" claim.
4558 tests passing under ``-m "not live"`` (was 4557 — +1 from the
regression test). ruff + mypy clean.
Reviewer caught real misses on the consumer-swap claim:
- TypeScript SDK still defined and re-exported `CloseWorkstreamRequest`
(types.ts + index.ts) — drop both. Now matches the Python-side
removal.
- Four `tests/test_auth.py` cases (`test_write_full_token_ok`,
`test_approve_full_token_ok`, `test_bearer_takes_precedence_over_cookie`,
`test_cookie_full_on_write_ok`) were tautological after the legacy
URL removal: they posted to `/api/send` / `/api/approve` and asserted
`allowed is True`, but those paths now classify as `read` so a read
token would also pass — they no longer tested the write/approve
scope enforcement. Swap to path-keyed URLs to restore the original
intent.
- `is_public_path("/api/send")` test renamed + retargeted to a
path-keyed URL.
Doc-table drift the previous commit missed:
- `docs/security.md` path-to-scope mapping rewritten for the
path-keyed verb family (write set, DELETE-on-/send dequeue,
per-ws_id approve).
- `docs/architecture.md` scope-model row text swap from `/api/send`
/ `/api/approve` to the path-keyed equivalents.
- `docs/diagrams/01-system-context.puml` channel→server edge label
swap.
- `docs/diagrams/15-auth-architecture.puml` scope class swap.
Cosmetic comment-only stragglers:
- `tests/test_session_worker.py` module docstring URL update.
- `tests/test_ratelimit.py` ~11 `/api/send` fixture-key strings
retargeted to `/api/workstreams/abc/send` so the URL fixtures
reflect the post-1.5 surface (rate limiter is path-agnostic; the
swap is purely cosmetic).
4557 tests still passing under -m "not live"; ruff + mypy clean.
CHANGELOG [Unreleased] / Removed (BREAKING — 1.5.0) block calling out
the legacy URL family removal with the swap table. Doc passes on
api-reference.md (per-endpoint sections rewritten with path
parameters and slimmer body shapes), architecture.md (handler-list
diagram and console-proxy URL example), console.md (URL-rewriting
JS shim docstring + SSE proxy example), and the two PlantUML
diagrams (11-console-data-flow, 16-channel-architecture).
Also picks up two test-side stragglers from step 5 that referenced
the legacy adapters in a docstring + a stale /v1/api/events SSE
test: turn into path-keyed equivalents. OpenAPI JSON dump regenerated
to reflect the catalog edits from step 3.
After this commit:
- 4557 tests passing under -m "not live"
- ruff + mypy clean on turnstone/ tests/ sdk/
- grep for "/v1/api/send", "/v1/api/approve", "/v1/api/cancel",
"/v1/api/workstreams/close" returns zero hits across turnstone/
sdk/ docs/ tests/ (excluding CHANGELOG.md, which intentionally
documents the old shape).
- grep for make_legacy_body_keyed_adapter, make_legacy_query_keyed_adapter,
_make_method_dispatch, close_legacy returns zero hits.
All in-tree consumers of the legacy /v1/api/send | /approve | /cancel |
events?ws_id= | /workstreams/close URLs now hit the path-keyed shape
under /v1/api/workstreams/{ws_id}/<verb>. Bodies drop ws_id (the path
provides it). The SSE event stream URL likewise moves to the path-keyed
form; channel adapters drop the params={"ws_id": ...} kwarg on
aconnect_sse.
Touched:
- turnstone/ui/static/app.js: 7 call sites (send×3, dequeue, approve,
cancel, close + EventSource SSE URL).
- turnstone/sdk/server.py (Python SDK): close_workstream, send,
approve, cancel, stream_events, send_and_wait's internal SSE
consumer.
- sdk/typescript/src/server.ts: closeWorkstream, send, approve,
cancel, streamEvents + sendAndWait's internal SSE consumer.
- turnstone/sdk/console.py: route_send, route_approve, route_close,
route_cancel — proxy URLs swap to /v1/api/route/workstreams/{ws_id}/<verb>.
route_plan_feedback / route_command remain body-keyed (out of scope).
- turnstone/console/server.py:
- Proxy mount table swaps the four legacy /api/route/{send,approve,
cancel,workstreams/close} mounts for path-keyed equivalents under
/api/route/workstreams/{ws_id}/<verb>; /send accepts both POST
and DELETE for dequeue.
- route_proxy reads ws_id from path_params (with body-fallback for
the surviving plan/command body-keyed mounts), uses
client.request(request.method, ...) so DELETE on /send proxies
through correctly, and audits DELETE-on-/send as a separate
"route.workstream.dequeue" action via _ROUTE_PROXY_AUDIT_ACTIONS.
- Internal `method` variable renamed to `verb` to avoid confusion
with HTTP method now that the two diverge.
- turnstone/channels/_sse.py: SSE URL builder swaps to path-keyed.
- turnstone/channels/{discord,slack}/bot.py: docstring URL updates.
- turnstone/server.py, turnstone/core/session_worker.py,
turnstone/sdk/events.py, turnstone/api/server_spec.py: comment /
docstring URL updates only.
Test fixtures still reference legacy URLs and will be swapped in step
5 of this PR.
The /v1/api/dashboard endpoint was the last workstream-listing surface
keyed on `id` rather than `ws_id`. The Stage 2 list-verb lift converged
the active list (`/v1/api/workstreams`) and saved list
(`/v1/api/workstreams/saved`) on `ws_id` but explicitly left dashboard
alone to keep that PR's diff focused. This lands the same rename on
the remaining endpoint so v1 row shape is consistent across the family.
Scope kept narrow:
- Pydantic `DashboardWorkstream` and TS SDK `DashboardWorkstream`
interface both rename `id: str/string` → `ws_id`.
- The bundled web UI (`turnstone/ui/static/app.js`) is the only consumer
reading `dashboard.workstreams[].id` and is updated atomically.
- Console `_fetch_live_block` (cluster-inspect's projection over a
remote node's dashboard payload at `turnstone/console/server.py`)
flips its `entry.get("id")` lookup to `entry.get("ws_id")`.
- Drive-by: stale `id` example in `docs/api-reference.md` for the
earlier `/v1/api/workstreams` rename also fixed.
`_build_node_snapshot` (the global-events SSE node_snapshot payload
consumed by the cluster collector) deliberately stays on `id` — it's
part of a separate cluster-row family (collector → cluster_workstreams
→ console UI) that is internally consistent on `id` and would need its
own coordinated sweep. CHANGELOG documents the bounded blast radius.
Tests: 4554 passing (-m "not live"). ruff + mypy clean.
Server-side history endpoint declared ``error_codes=[404]`` but the
lifted ``make_history_handler`` factory can also return:
- ``400`` on empty ``ws_id`` (defensive — Starlette routing makes
it unreachable in practice, but the factory has the branch).
- ``500`` on the ``cfg.list_kind is None`` misconfig gate added in
the /review fix-up (defense-in-depth fail-loud; both production
cfgs wire ``list_kind`` so the gate doesn't fire today).
- ``503`` via ``cfg.manager_lookup`` when the kind's manager isn't
available (interactive's lookup never returns 503; coord's can).
Updated ``server_spec.py`` to ``[400, 404, 500, 503]`` per Copilot's
suggestion — matches the existing detail entry's shape so the two
endpoints document the same possible-error envelope.
Caught the parallel asymmetry on ``console_spec.py``: history was
``[403, 404, 503]`` but the lifted factory's misconfig + empty-
ws_id branches reach coord too. Updated to
``[400, 403, 404, 500, 503]`` — same factory body, same possible
responses, plus ``403`` from coord's ``admin.coordinator``
permission gate.
Regenerated ``openapi-{server,console}.json``. No code changes;
spec metadata only. Tests + lint + mypy unchanged.
Last verb-shape lift before v1.5.0 stable can tag. Adds two new
factories to ``turnstone/core/session_routes.py``:
- ``make_history_handler(cfg)`` — body lifted from coord's
``coordinator_history`` near-verbatim. ``?limit=`` query param
defaults to 100, clamps to [1, 500], malformed values fall back
to 100. Storage operations (``get_workstream`` on the
storage-fallback path, ``load_messages`` for the row read) now
run via ``asyncio.to_thread`` (was inline pre-lift on coord).
- ``make_detail_handler(cfg)`` — body lifted from coord's
``coordinator_detail``. Lazy-rehydrates a closed/evicted
workstream via ``mgr.open()`` on miss; mirrors
:func:`make_open_handler`'s exception envelope (``ValueError``
→ 503 with the session-factory's remediation text; bare
``Exception`` → correlation_id'd 500 with the per-kind noun
via ``cfg.audit_action_prefix``).
NO new ``SessionEndpointConfig`` fields — the factories reuse
``permission_gate``, ``manager_lookup``, ``not_found_label``,
``audit_action_prefix``, and (for history's storage-fallback
kind check) ``list_kind`` — all already wired by both production
lifespans for the list/saved factories.
Coord side: ``coordinator_history`` and ``coordinator_detail``
standalone handler bodies removed from ``console/server.py``;
``register_session_routes`` now wires
``history=make_history_handler(coord_endpoint_config)`` and
``detail=make_detail_handler(coord_endpoint_config)``.
Interactive side: GAINS both endpoints as a feature gain. Pre-lift
interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no
``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to
subscribe to ``/events`` SSE just to read display fields or
message rows. The same lifted factories are wired with the
interactive endpoint config; cross-kind isolation is preserved on
both sides (history via ``cfg.list_kind`` storage-fallback gate
+ fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal
kind check).
Pydantic schemas: ``CoordinatorDetailResponse`` /
``CoordinatorHistoryResponse`` removed from ``console_schemas.py``;
``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added
to ``server_schemas.py`` (mirrors the list lift's pattern for
``WorkstreamInfo``). Both server and console OpenAPI specs
reference the unified schemas; ``server_spec.py`` gains
``EndpointSpec`` entries for the new interactive endpoints. TS
SDK gains both interfaces in ``sdk/typescript/src/types.ts``;
``openapi-{server,console}.json`` regenerated.
Tests: 6 new coord regression/parity tests in
``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404
on storage fallback, storage-only history, detail 503 on
session-factory misconfig, detail 500 with correlation_id on
unexpected rehydrate failure, history swallows
``load_messages`` exception → 200 with empty messages). 10 new
interactive parity tests in ``test_workstream_endpoints.py``
(``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new
openapi spec test pinning the server-side ``?limit=`` query param.
Total: ``4490 → 4491`` after the new exception-swallow
regression test landed. ``ruff check`` clean, ``mypy`` clean on
touched files.
/review pipeline (4 finders → verify → dedupe) caught 1 Minor
defense-in-depth (bug-1/sec-1, merged: ``make_history_handler``
fail-closed gate when ``cfg.list_kind is None``, mirroring
``make_saved_handler``'s same gate) + 1 Minor test-helper rename
(q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``)
+ 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK
mention, q-4 missing exception-swallow regression test, q-5
misleading test comment) — all addressed in the same commit.
New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)``
factories in ``turnstone/core/session_routes.py`` replace four
pre-lift bodies (interactive ``list_workstreams`` +
``list_saved_workstreams``; coord ``coordinator_list`` +
``coordinator_saved``). Same factory + capability-flag pattern as
the merged cancel / open / events / create lifts.
Four new ``SessionEndpointConfig`` fields:
- ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup
``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires
``get_workstream_display_names`` (new bulk helper added on the
storage layer + memory.py); the lifted body resolves every active
row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the
pre-lift N+1 (one SELECT per row).
- ``list_kind: WorkstreamKind | None`` — explicit kind classifier
for the saved-list storage filter. Replaces the initial draft's
``audit_action_prefix == "coordinator"`` string compare which
would have silently leaked INTERACTIVE rows for any future kind
whose audit prefix didn't match. Required when a kind mounts
list/saved; misconfig surfaces as a 500 with a clear log line.
- ``saved_state_filter: str | None`` — coord wires ``"closed"``;
interactive wires ``None``.
- ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only
defence-in-depth filter that excludes ws_ids in the warm pool.
Behaviour changes (all observable in CHANGELOG):
- **Active-list row shape converges on always-include** ``{ws_id,
name, state, kind, parent_ws_id, user_id}``. Interactive renames
``id`` → ``ws_id``; both kinds populate every field (coord adds
kind + parent_ws_id; interactive adds user_id).
- **Top-level response key converges on ``"workstreams"``** on
both endpoints. Coord ``coordinators`` key removed — coord is a
1.5.0aN-only surface (never shipped stable) so the convergence
has no compat shim; SDK / frontend consumers swap once.
- **Storage + manager-lock work moved off the event loop on
interactive**. ``list_workstreams_with_history`` runs through
``asyncio.to_thread`` on both kinds (matches coord's pre-existing
perf-2 pattern from the saved-coordinators review); ``mgr.list_all``
+ per-row work also offloaded.
- **N+1 storage round-trips on /v1/api/workstreams eliminated**.
Pre-lift interactive resolved the alias for every active row in a
separate SELECT (up to 50 round-trips per dashboard refresh on a
saturated node). Lifted body issues one bulk SELECT.
Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``,
``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and
``CoordinatorListResponse`` removed (folded into the unified
``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec
snapshots regenerated. TS SDK types updated (``WorkstreamInfo``
interface gains ws_id + the always-include fields); TS test
mock + assertion updated to match.
``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope
and still returns rows keyed on ``id``. Tracked as a separate
cleanup PR (tombstone-note added at the dashboard handler).
/review pipeline run; the four Major findings + one Minor + six
nits all addressed in the same commit:
- M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) →
renamed + fields added.
- M2: TS SDK test masked the type-mismatch with stale mock → updated.
- M3: N+1 alias resolution on active list → bulk
``get_workstream_display_names`` helper + ``list_resolve_titles``
bulk cfg hook.
- M4: Missing interactive parity regression test for unified row
shape → mirror of coord's added in test_server_authz.py.
- Mi1: ``audit_action_prefix`` string-compare deriving kind →
explicit ``cfg.list_kind: WorkstreamKind`` field.
- Six nits: redundant inner asyncio import, forward-ref quotes on
Awaitable, duplicated frontend comments, dashboard ``id`` field
has no tombstone-note, empty-coord_mgr short-circuit on
``saved_loaded_lookup``.
4512 tests passing; ruff + mypy clean.
New ``make_create_handler(cfg, *, audit_emit=None)`` factory in
``turnstone/core/session_routes.py`` consumes five new ``SessionEndpointConfig``
fields (``create_supports_attachments``, ``create_supports_user_id_override``,
``create_validate_request``, ``create_build_kwargs``, ``create_post_install``)
and replaces both ``create_workstream`` and ``coordinator_create`` bodies.
Same factory + capability-flag pattern as the merged cancel / open / events
lifts. ``_validate_and_save_uploaded_files`` lifted to
``turnstone.core.attachments`` so both processes call one kind-agnostic
implementation.
Coord parity gains (§ Post-P3 reckoning item #1 + carry-forward):
- Create-time attachments: multipart parsing, validate+save+rollback,
``attachment_ids`` on the response. Coord adapter ``send`` doesn't yet
reserve attachments at create time, so the rows save as pending and the
next ``/send`` picks them up via the standard send-with-attachments path.
- Disabled-skill rejection (matches interactive's pre-lift gate).
- Always-include response shape ``{ws_id, name, resumed, message_count,
attachment_ids}`` populated with default ``False``/``0``/``[]`` on the
fields coord doesn't fill.
- 200 status (was 201).
- Audit-emit failures swallow + warning log instead of 500.
Both kinds converge on the manager-at-capacity 429, factory-misconfig 503,
and correlation_id'd 500 for unexpected ``mgr.create`` failure (interactive
lifted up to coord's safer error envelope).
Three /review fixes folded in:
- ``notify_targets`` malformed input gates at the validator (400) instead
of bubbling out of post_install as a 500 — pre-fix the workstream had
already been created + audited + broadcast by the time the validation
raised.
- Skill-lookup storage failures now share the correlation_id'd 500 path
with ``mgr.create`` (was masquerading as 400 "Skill not found").
- Whitespace-only ``skill`` field treated as empty (matches pre-lift coord).
CHANGELOG entry under [Unreleased] documents every observable behaviour
change. OpenAPI spec regenerated. Three new coord regression tests
(create-time-attachments save pending rows, always-include parity fields,
disabled-skill rejection) plus one interactive regression test
(notify_targets 400). 4500 tests passing.
Five fixes from Copilot's review of Stage 2 P1.5 — all preserve
behaviour, narrow docstring claims, and round out the response shape:
* **session_routes.py:supports_attachments docstring** — claimed
the handler "accepts only ``{"message": ...}``" when ``False``,
but the implementation silently ignores ``attachment_ids``
rather than rejecting. Updated wording to say the
attachment-resolution block short-circuits and any
``attachment_ids`` are silently ignored. Behaviour unchanged
(silent-ignore is the right choice for forward compat — clients
passing ``attachment_ids`` speculatively to a not-yet-lit-up
kind shouldn't get a 400).
* **session_routes.py:queue_full response shape** — restored the
always-include guarantee for ``attached_ids`` /
``dropped_attachment_ids``. The queue_full path now returns
``attached_ids: []`` and ``dropped_attachment_ids: list(requested_ids)``
so SDK consumers don't have to branch on status.
* **server.py:_interactive_spawn_metrics guard** — added
``_ws_turn_tool_calls`` to the ``hasattr`` chain. Previously
the guard checked ``_ws_lock`` + ``_ws_messages`` and then
unconditionally assigned ``_ws_turn_tool_calls`` — would
raise on a SessionUI subclass with the first two but not the
third.
* **console_spec.py:coord_send error_codes** — added 409
(the 'session UI not available' branch in
``make_send_handler`` returns 409, but the spec didn't list
it). OpenAPI spec regenerated; TS SDK types refreshed.
* **session_routes.py:tenant_check docstring** — claimed
interactive uses ``_require_ws_access`` with "404 on owner
mismatch", but the helper now delegates to
``resolve_workstream_owner`` which explicitly does NOT enforce
row-level ownership (trusted-team semantics; 404s only on
missing rows). Updated wording to match.
Five new TestCoordinatorAttachments tests in
``tests/test_coordinator_endpoints.py`` exercising the lifted
attachment surface end-to-end on coord:
* upload → list round-trip
* get_content returns raw bytes with text/plain forced for text
* delete removes pending entries and clears them from the listing
* send with attachment_ids consumes pending under the send_id token
* send response carries attached_ids / dropped_attachment_ids even
on plain-text sends (unified shape parity)
The existing ``_coord_endpoint_config`` fixture grew capability
flags to mirror the production console wiring, and ``_make_client``
now mounts the four coord attachment routes via
``make_attachment_handlers``.
OpenAPI specs regenerated; TS SDK bumped to 0.5.0. CHANGELOG entry
under [Unreleased] documents the verb-shape lift, the coord
attachment surface coming online, the response-shape change for
``coordinator_send``, the unification of the three lifted classifier /
lock helpers under ``turnstone.core.attachments``, and the new SDK
helpers.
Stage 2 Priority 0 Step 0.5 follow-on. The handwritten Python
OpenAPI spec already moved to ``/v1/api/workstreams/`` in the URL
sweep commit; this just regenerates ``sdk/typescript/openapi-{server,console}.json``
from those specs so generated TS callers see the new paths.
Bumps the TS SDK to 0.4.0 to flag the URL-shape break for any
1.5.0aN-era consumer of the experimental coord client. Python SDK
needs no change — it never exposed the coord HTTP surface.
TS typecheck + 32 vitest tests pass.
* chore(coord): remove spawn-quota subsystem
The quota gate was operator-level safety per its own comments, not a
security boundary, and never fired in a week of heavy use. Runaway
coordinator spawns are already bounded by max_active slot exhaustion,
which surfaces to the coord LLM as a tool error — same operational
shape, one fewer moving part. Precedes the Stage 1 SessionManager
unification so the coord tool doesn't inherit quota bookkeeping.
Upgraded deployments with the three removed settings persisted will
log three "Skipping invalid setting" warnings on startup and
otherwise degrade cleanly; a follow-up migration to delete the rows
would silence that noise.
* chore(migrations): drop stale coord spawn-quota settings rows (047)
Clears persisted rows for the three ConfigStore keys removed in the
previous commit so upgraded deployments don't log "Skipping invalid
setting" warnings on every startup. Downgrade is a no-op — the rows
were operator-set values, and a rollback to pre-1.5.0 code falls back
to the registry defaults for any key not present.
* feat(coordinator): phase 8 PR B — spawn budget + rate limit + /quota endpoint
Adds two complementary controls so a runaway coordinator can't saturate a
cluster's max_active without anyone noticing:
- **Spawn budget** (hard quota) — cap on concurrently active children.
Default 20 per coord. spawn_workstream returns a tool error guiding
the model to close idle children; spawn_batch routes overflow rows to
`denied[]` with partial-success semantics.
- **Spawn rate limit** (soft pacing) — classic token bucket, defaults
5 tokens/minute with burst 10. A rate-limited spawn surfaces a tool
error carrying `retry after Ns` so the model paces itself. Zero
refill rate is honoured as "disable refill" (bucket still honors the
initial burst).
Shipped infra:
- `turnstone/core/spawn_quota.py` — thread-safe `SpawnBudget` +
`TokenBucket`. 15 unit tests.
- `turnstone/core/session.py` — coord-only state built from settings at
__init__. Shared `_eval_spawn_quota(active)` helper drives both the
single-spawn path (wraps the denial reason in `_coord_tool_error`) and
the batch path (annotates `spec["_error"]`). `_count_active_children`
routes through `coord_client.list_children(include_closed=False)` and
fails *open* on lookup error (budget is operator-safety, not security).
- `POST/GET /v1/api/coordinator/{ws_id}/quota` — partial-update admin
endpoint mirroring the /trust + /restrict shape. Accepts either the
nested `spawn_rate` object or flat aliases — supplying both for the
same field returns 400 so the admin UI can't half-migrate silently.
Overrides are in-memory only (die on session reopen). Audits via
`coordinator.quota.updated` with before/after snapshots.
- Settings: `coordinator.spawn_budget`, `coordinator.spawn_rate.tokens_per_minute`,
`coordinator.spawn_rate.burst` with ranges 1..500 / 0..600 / 1..500.
The range bounds are the single source of truth — the endpoint
validators and Pydantic schema both import from `settings_registry.SETTINGS`
so bumping a cap in one place lights up everywhere.
- OpenAPI: `CoordinatorQuotaRequest` / `CoordinatorQuotaResponse` /
`CoordinatorSpawnRateState` schemas + endpoint specs. TS SDK regenerated.
Tests: +15 unit (SpawnBudget + TokenBucket), +17 endpoint (GET + POST
happy paths, range edges, mixed-body rejection, non-object spawn_rate,
service-token refusal), +11 session-side (budget blocks single spawn,
budget batch partial-success, rate batch partial-success, empty-body
reject, mutator live-update, non-coord session has no quota state).
Deferred (not this PR): per-skill scoping via migration 047 +
`prompt_templates.spawn_budget` column. Count-only storage helper
(opportunistic — list_children at budget ≤ 500 is fine behind a
human-gated approval flow).
* fix(coordinator): address PR #387 copilot review
- Budget undercount: _count_active_children used list_children's
LIMIT-then-Python-filter path, so a fan-out with many recently-closed
children could push live rows past the SQL LIMIT and silently
undercount, leaking spawn slots past the budget. Replace with a new
CoordinatorClient.count_active_children that uses
storage.count_workstreams_by_state (SQL aggregate, no pagination,
sums non-terminal states). Tenant-guarded; fails open on storage
error (budget is operator-safety, not a security gate). New client
tests cover the non-terminal count, the closed/deleted exclusion,
the foreign-parent guard, and the fail-open path.
- Service-token bypass on /quota: both GET and POST used the default
allow_service_bypass=True, so a service token whose user_id matched
the coord owner could read or *raise* spawn capacity without the
explicit admin.coordinator grant. Flip both to
allow_service_bypass=False for consistency with /restrict,
/stop_cascade, and /close_all_children.
- OpenAPI contract leak: CoordinatorSpawnRateState was used for both
the request and response shapes, which let generated SDKs imply
clients could POST tokens_available (a read-only bucket reading the
handler ignores). Split into CoordinatorSpawnRateInput (request:
tokens_per_minute + burst only) and CoordinatorSpawnRateState
(response: adds tokens_available). No runtime behaviour change;
SDKs regenerate with two distinct types.
Drops the _ACTIVE_COUNT_SLACK / _ACTIVE_COUNT_MIN_LIMIT constants in
session.py — no longer needed since the new helper takes no limit
argument. Updates the 5 session-side quota tests to stub
count_active_children instead of list_children.
* feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools
Adds two model-facing batch tools so a coordinator can fan out without burning one approval per child:
- `spawn_batch` — create up to 10 child workstreams in a single approval. Serialised
spawns so sibling ordering (by created_at) stays deterministic. Returns
`{results: {idx: {ws_id, name, node_id, status}}, denied: [{idx, reason}]}`.
Per-item validation / spawn failures surface in `denied[]`; the batch hard-errors
on >10 rather than silent truncation.
- `close_all_children` — soft-close every direct child in one approval. Server-side
Sem(16) fan-out via `coord_client.close_workstream`; `reason` propagates to every
closed child's audit + workstream_config. Response mirrors `stop_cascade`'s cascade
idiom: `{closed, failed, skipped}` where `skipped` is upstream-404 / already-gone.
Shipped infra:
- New console endpoint `POST /v1/api/coordinator/{ws_id}/close_all_children`
(gated `admin.coordinator`, `allow_service_bypass=False`, 512-char reason cap,
`coordinator.closed_all_children` audit).
- Shared `_fanout_on_children` helper — both `stop_cascade` and `close_all_children`
now delegate to it (one place to own the snapshot → semaphore-gather → bucket-split
skeleton).
- `CoordinatorClient.close_all_children(reason)` plus a `_post_url` seam that
`_post` now reuses (no more duplicated transport-error handling).
- `_emit_batch_event` — best-effort SSE emitter modelled on `_emit_wait_event`.
Emits `batch_started` / `batch_ended` pairs keyed by call_id. Throttled
`batch_progress` deferred to a follow-up.
- OpenAPI request + response schemas, endpoint spec entry, TS SDK regenerated.
- Persona doc (`tools_coordinator.md`) covers the two new patterns.
Bulk-endpoint shape policy (codified in PR C later): split by semantic category —
`{results, denied, truncated}` for bulk-read / bulk-create-with-payload (cluster/ws/live,
spawn_batch), `{<bucket>, failed, skipped}` for cascade-mutation (stop_cascade,
close_all_children). No retrofit needed on stop_cascade.
Tests: new `test_coordinator_close_all_children.py` (8 endpoint tests), expanded
`test_coordinator_tools.py` (session-side prepare/exec, coord_client=None guards,
batch SSE events), expanded `test_coordinator_client.py` (route map, client method,
transport errors), tool-count assertions updated.
Deferred (not this PR): per-item selective-deny approval UI, throttled batch_progress
SSE, coordinator-skills doc + bulk-endpoints doc (PR C), spawn budget / rate limit (PR B).
* fix(coordinator): address PR #386 copilot review
- coordinator_client.close_all_children: pass the unformatted path template
as log_path so telemetry aggregates don't fragment per session (ws_id
still lives in the real URL).
- session.py: drop dead spawned_ids accumulator in _exec_spawn_batch —
leftover from an eager-register path that got removed earlier.
- close_all_children tool JSON: document the 512-char server-side cap on
reason and that reason is echoed back in the response payload. Added
maxLength:512 on the schema property so the LLM sees the constraint.
- CoordinatorCloseAllChildrenRequest: add Field(max_length=512) so the
OpenAPI schema reflects the runtime 400-on-overflow constraint.
Patrick Buckley