The #805 server-side fixes (emit-time batching, _ListenerQueue poison,
out-of-band closing) already cover every SSE stream, but the client-side
companions lived only in the interactive pane. Port them to coordinator.js
and extract the drift-prone pure core into a shared module (closes#806).
- shared_static/sse_overflow.js (new): storm-guard constants +
overflowWindowTripped + degradedCooldownStep, imported by both panes so the
trip threshold and cooldown ladder have one source of truth. interactive.js
imports these instead of holding local copies; the two node runtime probes
move to tests/test_sse_overflow_js.py.
- coordinator.js: handle the stream_overflow frame (storm guard -> degraded
catch-up with a doubling cooldown; the reconnect replays from the ring, or
falls to the replay_truncated -> /history floor); add the close-on-hide /
replay-on-show visibilitychange handler plus a document.hidden guard at the
connectSSE chokepoint; add drop-vs-render-wedge counters (onmessage now wraps
the dispatch in try/catch -- the coordinator previously had no wedge guard,
so a handler throw silently poisoned every later turn).
- After a stream gap the children/tasks sidebar re-syncs only when the ring
replay cannot cover it: no resume cursor, a replay_truncated envelope, a gap
beyond the cursor-trust window, or a live event id below the saved cursor (a
process restart reset the counter, which the replay path reports as a false
replay_ok). child_ws_*/task events are ordinary ring entries, so an ordinary
short reconnect heals the sidebar through the live handlers with no REST
rebuild -- a momentary blur/focus under close-on-hide rebuilds nothing.
- Close-session teardown detaches the visibility handler before the close POST
so a hide/show mid-close can't resurrect a dying stream. A replay_truncated
seen mid-stream is deferred (not dropped) and re-synced from /history on the
next idle -- repairing both a ring-evicted gap and a turn stranded by
close-on-hide (stream_end evicted while hidden), matching interactive.js's
_pendingTruncatedResync.
The extraction stops at the pure core: interactive.js's stateful glue is
hard-pinned by its source-assertion suite, so its class-method shape stays put
and the coordinator reimplements the equivalent glue as closure functions.
Tests: new test_sse_overflow_js.py (module exports + the two runtime probes);
coordinator parity + lifecycle pins in test_app_js.py (replay-aware sidebar
refresh, restart detection, truncated-resync deferral, close-session
visibility detach); interactive's moved probes replaced by an extraction pin.
All JS-source suites green.
(cherry picked from commit 2a32211e4a)
HYPOTHESIS.md:
- New appendix entry "Daemons (the recurrent harness)": a daemon as the
regenerative process of concatenated runs — ready-set recurrence,
renewal-reward lifting exactly at regeneration points, accumulation as
what breaks regeneration (cross-cycle provenance meet, renewal events
that reset accumulated risk), and authority under intermittence
(owner contact as a renewal point for authority; TOCTOU at cycle
scale).
- New body section "The loop": the task-dispatching outer loop as the
harness construction applied one level out — the composition
correspondence read at the top level, the daemon as its single-agent
special case, the bare while-loop as the trivial-group harness one
level up. Flagged as a sketch; outer fail-closed/reach-avoid
treatment deferred to later rounds.
- Veto caveat threaded to match: judge-as-veto safety scoped to the
authority lattice, and the nonblocking escape degrades to an
always-enabled safe halt when the principal is unreachable.
- Consistency: Grounding's Asserted tier now covers "The loop";
"always-enabled escalation" -> "escape" (the appendix's own term, now
that the escape has an unattended form); brace the one unbraced \bot
subscript (linter section-B HIT).
PRIMER.md: new plain-language companion — same object, no symbols, the
formal doc wins every disagreement. README's entry link now points at
the primer, which links onward to HYPOTHESIS.md.
(cherry picked from commit d115111756)
highlight.js, KaTeX and Mermaid all run on every surface via the shared
renderer (renderer.js), but their theme/wrapper CSS lived only in
ui/static/style.css. The console and coordinator load /static/style.css from
console/static/ — a different file on a different server — so hljs token spans
fell back to --fg (flat monospace for several releases), and the KaTeX/Mermaid
wrappers lacked their overflow containers, letting wide equations/diagrams
overflow the pane.
Move the hljs theme, .katex-display/.katex-error and the .mermaid-* wrappers
into shared_static/chat.css, which every surface loads via /shared/chat.css.
Restate the mermaid width-clamp for the preview pane (.preview-markdown) too,
since its content isn't a .msg.assistant message.
Drop the redundant background on .msg.assistant pre code.hljs so the <pre>
carries the code surface on every surface — otherwise the console/coordinator
(where the pre is --panel, not --code-bg) showed a darker box inside a lighter
padding band.
(cherry picked from commit 5dcf66c284)
A scheduled task could pin the model and skill of the workstream each
firing creates; it can now also pin its persona and project, so a
schedule can run under, e.g., the researcher persona attached to a
specific project's memory bucket.
The two values live on scheduled_tasks (migration 066, Text NOT NULL
default '') and are passed verbatim to create_workstream at dispatch,
where the node resolves the persona for the workstream kind and gates
the project attach. Empty means "kind-default persona / no project",
resolved late at each firing (mirrors how empty model/skill already
behave) -- existing schedules keep byte-identical dispatch behaviour,
so there is no backfill.
Also fixes a latent bug this feature depends on: admin_create_schedule
read created_by from request.state.user_id, which AuthMiddleware never
sets, so every scheduled task stored created_by=''. It now reads
auth_result.user_id like every other console endpoint. This is now
load-bearing -- the scheduler dispatches under created_by and the node
gates the project attach against it. admin_update_schedule adopts the
editing admin as owner when a project is assigned to a pre-fix orphaned
('') schedule, and re-validates persona/project only when they change
so a since-disabled persona or lost membership does not block unrelated
edits (the node re-checks at dispatch either way).
Wired through: schema + migration (up/down + parity tested), both
storage backends, API schemas, SDK create_workstream and console
create_schedule/update_schedule, scheduler dispatch, and the admin
schedule shelf (persona + project pickers, current value preserved so
an edit cannot silently clear a filtered-out selection).
(cherry picked from commit c328bebecd)
The URL-extraction call hard-coded max_tokens=8192 and rode the "low"
reasoning default, which broke local-inference models whose registry entry
advertises a tighter output limit or a different reasoning config. Inherit
the session/registry max_tokens and reasoning_effort instead (temperature
already was) — the same knobs the main turn uses.
max_tokens is capped to context_window // 4, the ~25% output slice Phase 2
already reserves, matching the main turn's response reserve
(_remaining_token_budget), so a large operator budget can't push
prompt + output past a small context window on strict runtimes.
(cherry picked from commit d5ddc95e9f)
github-code-quality flagged 8 spots where tests imported
turnstone.core.session_ui_base both as `from ... import` and `import ... as
suib` (the alias was only there to monkeypatch the module-level batch
constants). Drop the alias and patch via string target
(`monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", ...)`),
which resolves to the same module global — behavior-identical. The one test
that READS the constant imports the symbol directly. Test-only, no
production change.
(cherry picked from commit 026c646116)
PR #805 review (Copilot + the round-3 finding it corroborates):
- connectSSE opened a new EventSource even when the tab was already hidden
(e.g. a first load in a background tab), where the close-on-hide handler
never fires because there is no open stream to close — so a throttled
hidden tab could still become the slow consumer this PR prevents. Add the
document.hidden guard at the single connect chokepoint, after the wsId
assignment + visibilitychange-handler install (so the show edge reconnects)
and before new EventSource (so nothing opens). The timer callbacks keep
their own pre-checks (the recover beat's also gates failCount); this closes
the fresh-connect path they never covered.
- Fix the stale _ListenerQueue.closing docstring: it claimed the drain loop
checks closing BEFORE poisoned, but the round-2 fix moved that check INSIDE
the poison branch (poisoned+closing -> clean close; a healthy closing queue
drains its tail to the ws_closed sentinel). Wording now matches the code.
(cherry picked from commit dfe09d029b)
A long live session driven by a fast local model (500-2000 tok/s) showed
corrupted / missing spans of assistant text while the backend stayed
healthy. Root cause: on_content_token/on_reasoning_token enqueued one SSE
event per model delta, so the per-listener queue (cap 500) overflowed
against any slow consumer; put_nowait on a full queue silently dropped the
newest event. Once saturated, drops scatter (the consumer keeps freeing
single slots), so the client's lastEventId sails past the holes and
reconnect-replay (eid > last_event_id) can never heal them. A dropped
fence-closer reshapes all downstream markdown -> reads as heavy corruption.
Fix B (primary) - emit-time micro-batching:
Coalesce content/reasoning fragments over a ~25 ms window (or 4 KB) into
one _enqueue, cutting the wire event rate ~10-20x at local-inference
speeds. A batch is assembled before it gets an _event_id, so it is one
ordinary ring entry no cursor can fall inside (unlike the forbidden
in-ring coalesce). Two conditions are load-bearing and pinned:
1. The inflight-buffer append and the enqueue are one _ws_lock section,
so a snapshot's snap_seq stays a true high-water mark for its text.
Splitting them lets a straddling snapshot double-render (the client
content path is a blind +=, no dedup).
2. Every non-token emit flushes the pending batch first, enforced at the
single _enqueue choke point, so stream_end/tool_*/state_change can't
overtake trailing content and repaint it into a new bubble.
Fix A (recovery net) - poison-at-first-overflow:
_ListenerQueue latches `poisoned` atomically at the FIRST rejected put and
refuses every later put, freezing its contents as a contiguous prefix; the
drain loop closes the stream after an id-less stream_overflow frame and the
native EventSource reconnect replays the whole gap from the ring buffer.
Poisoning at the first full (not after N) is required: any deliver-while-
dropping window advances lastEventId past interior holes that reconnect
can't replay. A ws teardown that races the overflow sets an out-of-band
`closing` flag (mark_closing), checked inside the drain loop's poison
branch: a poisoned+closing queue returns clean (no false overflow frame),
while a healthy closing queue still drains its full tail FIFO to the in-band
ws_closed sentinel -- so a slow-but-unpoisoned client never loses the turn's
final content batch + stream_end at teardown.
Client (interactive.js):
- Reconnect storm guard: after 3 overflow closes in 60 s the pane drops to
a degraded catch-up (stop live streaming, "connection is slow" state,
reconnect after a doubling 15->120 s cooldown that resyncs from the ring
or the uncapped /history floor). The cooldown ladder is keyed off a
last-trip timestamp, not the overflow-window array (which the trip
clears), so the escalation survives its own backoff.
- Close-on-hide / replay-on-show: a visibilitychange handler closes the
EventSource on tab-hide (a throttled hidden tab is the likeliest slow
consumer) and reconnects with the saved Last-Event-ID on show. The
factory recovery beat defers when hidden, and giveUp() detaches the
handler, so a dead or backgrounded controller can't reopen a stream.
- Drop-vs-render-wedge counters distinguish this bug (server overflow
closes) from the handler-wedge class (render/finalize throws) in the
field. No global gap-detector: live ids are not strictly monotonic
across concurrent tool+content emit, so a naive id!=last+1 check would
false-positive; recovery is server-signalled instead.
Corrects the stale _resolve_event_buffer_max comment that justified the
50k ring on a "PR-G closes connections on hide" mitigation that never
existed (the close-on-hide handler above is the real one).
Negative-tested (revert the guarantee, confirm the pin fails, restore):
per-token inflight append -> snapshot straddle double-render; removed
choke-point flush -> stream_end split; no poison latch -> silent drops;
top-of-loop closing check -> healthy-close tail loss; missing mark_closing
wiring / drain closing check -> clean close mis-reported as overflow;
_noteStreamOverflow cooldown reset -> ladder never escalates; removed
hidden-tab recovery guard / giveUp handler removal -> hidden-tab reconnect.
(cherry picked from commit 5083f67e96)
Copilot review on PR #804:
- An indented closing fence line (" ```") left its leading spaces as a
trailing whitespace-only line inside the rendered code block: the content
capture runs up to the backtick run and the close-line indent precedes it, so
it was captured as content. Strip a trailing newline PLUS any trailing indent
(/\n[ \t]*$/ instead of /\n$/); a column-0 close is unaffected. Red-green
pinned (content is exactly " x = 1", no trailing whitespace line).
- Correct a stale test docstring claiming the fence open anchor allows "up to 3
spaces" of indent — it allows arbitrary indent (the 4-space case is pinned
separately).
(cherry picked from commit e5e48a788a)
The markdown renderer protects structural blocks with in-band NUL-framed
sentinels (chr(0)+tag+index+chr(0)). escapeHtml preserves U+0000, so
model/tool text could forge sentinels, and recursively-rendered <details>
bodies re-rendered against fresh block arrays and lost their content. This
lands the ordered containment fixes from the render-containment brief.
Fixes (each pinned in tests/test_renderer_js.py; all NUL-sensitive cases also
confirmed in real headless Chrome, which drops a U+0000 token the node harness
preserves):
- B1/B2/B3 — forged sentinels: strip U+0000 at the TOP-LEVEL render entry only
(_fnDepth === 0). renderer.js is the sole NUL producer and every restore
regex is NUL-framed, so removing NUL closes every forgery path (block
duplication/relocation, out-of-range "undefined", cross-container injection)
while generated sentinels in recursive frames survive. Only NUL is stripped,
so a code fence still shows pasted control bytes (ESC/FF/VT/DEL) verbatim.
- B4 — blockquote-in-fence (the common one): the code-fence pass now runs
before the line-based blockquote pass. Its open matches at line start after
optional indent and an optional list marker (`- `, `1. `), and re-emits that
indent+marker before the sentinel so the fence keeps its document position
(a nested-list item stays nested; a fence continuing a footnote definition
keeps the indent its continuation scan needs). A blockquoted fence (`> ```)
is not matched (`>` is neither indent nor a list marker), so the blockquote
pass extracts that `> ` run and its recursion renders the fence. A `> ` line
inside a plain fence stays literal.
- B5 — <details> open anchored to line start (^[ \t]*), so a `<details>`
mentioned mid-line inside inline code no longer starts a block.
- NEW-1 — recursive-frame content loss: <details> extraction runs AFTER fence
protection and restores a fenced body from a saved raw-source array
(codeBlockRaw) back to raw markdown before the recursive render, so
code-in-details renders in-frame instead of restoring to "undefined". Running
after fence also means a </details> shown as example code inside a fence
can't close the block early, and a <details> shown inside a fence stays
literal — no offset-based fence-awareness needed. Inline-code/math in footnote
definitions render via the restore round-trip the undefined-guard enables
(documented at the append site).
- NEW-3 — code blocks gained the <p>SENTINEL</p> unwrap variant DT/BQ/MB/TB
already had, removing a stray empty <p> before a standalone <pre>. The CB
unwrap is whitespace-tolerant so an indented own-line fence (whose indent the
fence pass re-emits) also doesn't leave a stray <p>.
- Defense-in-depth: every restore callback returns the matched sentinel
(inert; the browser drops the NUL) instead of the array's `undefined`.
Non-obvious decisions:
- Control chars are authored as literal \xNN hex escapes (byte-verified: only
\uXXXX decodes to raw bytes in this toolchain; \xNN matches the file's
existing \x00 sentinel convention).
- Open anchors allow arbitrary leading indent (the fence open also allows a
list marker), not CommonMark's ^ {0,3}: the renderer has no indented-code
fallback, so preserving the prior behaviour of matching indented/list-nested
fences beats CommonMark strictness, while still excluding `> ``` and mid-line
forms.
- codeBlockRaw (the <details> raw-fence array) and the restore callbacks are
factored through a _restorer(arr) helper; codeBlockRaw is only populated when
the text contains a <details> tag (its sole reader).
- The entry strip is depth-0-only on purpose: an unconditional strip would
shred the generated sentinels recursive frames carry, foreclosing NEW-1.
Negative-tested (reverted the production line, confirmed the pin fails):
- fence anchor: unanchored swallows a blockquoted fence.
- NEW-1 codeBlockRaw restore: without it, code inside <details> is lost.
Deferred (called out per the brief):
- B6/NEW-4 bidi controls (U+202A–202E, U+2066–2069, U+200E/F) still pass
through unescaped; they are not C0 so the entry strip misses them. Left to a
follow-up — stripping risks corrupting legitimate RTL text and <bdi>
isolation is involved for a string renderer.
(cherry picked from commit a164d61552)
Review follow-up: the helper returned `fallback` verbatim when the name
sanitized to empty, so a future caller passing an unsafe fallback (non-latin-1,
control chars, quote, backslash) could reintroduce the header crash/corruption
the helper exists to prevent. Not reachable today — all call sites pass safe
ASCII literals — but the helper is a shared safety primitive whose contract is
wire-safe output.
Run the fallback through the same cleaning, backed by a safe constant if even
that is empty, so the return is always wire-safe and never filename="". Adds a
test.
(cherry picked from commit 2c5adb7aca)
Attachment `/content`, preview, and workstream-export downloads built the
Content-Disposition `filename="..."` value straight from a user-supplied
name, stripping only quotes and CR/LF. Three input classes still broke the
header:
- Non-latin-1 names (CJK, em dash): Starlette encodes header values as
latin-1 and raised, 500-ing the serving route. (The original get_content
bug.)
- ASCII control bytes (NUL, form-feed, VT, DEL): latin-1-encodable, so they
passed Starlette, but the HTTP server layer rejects control characters in a
header value and 500s one layer later.
- Backslash: the RFC 6266 quoted-pair escape. A trailing backslash escaped
the closing quote and corrupted the download filename (not a 500, but wrong
output; Windows-origin uploads carry it legitimately).
Extract one `latin1_safe_filename()` helper in web_helpers that drops every
non-printable character plus the double-quote and backslash quoted-string
metacharacters, folds any surviving non-latin-1 codepoint to '?', and falls
back to a non-empty name so the header never emits an empty filename. Route
get_content, preview_response_headers, and the export handler through it,
replacing three near-duplicate inline strips.
Adds unit tests for the helper (non-latin-1 fold, control-char and backslash
stripping, per-site fallback) and an endpoint regression test.
(cherry picked from commit fd3aed1eca)
unsplit() closed each doomed (ephemeral) pane via close() — which already
renders/persists/notifies — then repeated that trio, firing intermediate
persist/notify passes mid-operation. A 2-cell split fully collapses inside
close(), so bail there; only a 3+-cell split (or an empty doom list) still
needs the trailing exit + refresh. The all-conversation path is unchanged.
Also reword the cell-chip CSS comment so it names the reversible hide vs
destructive close glyphs, now that an ephemeral pane can show the close glyph
in split mode.
(cherry picked from commit f56fa55929)
The preview pane opens beside the conversation as a split cell. Dismissing
that cell — the per-cell chip, or Unsplit from the other pane — ran
closeCell(), which hides the pane but keeps it in _panes/_order, leaving an
orphan tab with no meaningful reopen (the reopen affordance is the transcript
chip, not the tab bar).
Add an `ephemeral` flag on ShellPane. For an ephemeral pane the cell chip and
Unsplit route to close() — destroying the pane and its tab — and the chip's
glyph/label read as a destructive close rather than a reversible hide. Unsplit
still spares the focused survivor even when it is ephemeral ("keep the focused
pane"). The preview pane sets the flag; conversational panes do not, so an
all-conversation split is unchanged (Unsplit reduces to the prior
_exitLayout(_activeId)).
(cherry picked from commit ace9e034f9)
The wake gate documented exactly one info line per call past its
gates, but a send() refusal (the authoritative under-lock _closed
re-check catching a teardown the gate's lockless peek missed) emitted
nothing — a dropped wake should stay traceable to its trigger, so the
refusal now logs nudge_wake.refused.
The already-dispatched branch's comment claimed a held reminder can
coexist with the terminal mark via a redelivery whose commit raised —
impossible with the current control flow (_redeliver_pending clears
the hold before committing). Reworded to what the clear actually is:
the last line of defense against any coexisting hold leaking forever
once this branch deactivates the row, since inactive rows never
re-list. Test comment updated to match.
(cherry picked from commit cb59afe443)
A session whose _nudge_queue answers has_pending truthily while its
deliver_wake_nudge_from_queue consumes nothing turns the worker-exit
backstop into an infinite respawn loop: the gate passes, the wake
worker no-ops, the exit backstop re-runs the gate, forever.
Mock-backed test sessions riding real Workstreams are exactly that
shape, and one worker on such a pairing is enough to ignite a
wake-thread storm that trips the leaked-thread guard in every
subsequent test. The wake contract requires real drain semantics —
the spawned worker must CONSUME what the gate saw — so the gate now
refuses on type, not just presence.
(cherry picked from commit 7886d3b763)
str | None under-specified the field: the implementation and the TS SDK
union both constrain it to queue_full / refused_closed, and the Literal
projects a proper enum into the generated OpenAPI spec so clients
reject unexpected values. Specs regenerated.
(cherry picked from commit fa1ba2cc01)
Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
already-idle workstreams (no IDLE transition for the watcher to
observe), wired as wake_fn at every set_watch_runner site via the
shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
— after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
ownership clears: IDLE fans out on the worker thread, so
transition-time wakes always landed on the reuse path and no-op'd
(the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
the wake worker's run() closure and only Exception is caught
downstream.
Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
redelivered on min(interval, 60s) without re-running the command,
bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
max_polls across cycles; the poll charge commits durably at hold
time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
cap, presence-only re-check under the lock, detection-only stall
alerts (reclaiming a wedged admission would trade capped degradation
for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
genuinely-missing history (confirmed by a raising storage probe —
the resume loader swallows read blips into []) deactivate the watch
immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
stashing/dispatching, cancel paths write the row BEFORE
forget_terminal_dispatched, the HTTP cancel endpoint clears runner
state, and a per-tick sweep bounds the residual stash-after-clear
interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
read but not write retries the row write instead of re-running the
command every cycle; the fresh-fire unrestorable path stashes before
its deactivation write for the same reason.
Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
rebind now moves it: non-fork resume() and /new go through
_follow_watch_registration (new key live before the old is removed,
never stealing a registration another live session holds), removals
are owner-checked so tearing down a watch-restore shell or a
resumed-away session cannot unregister a live pane, the restore
shell yields to a registration that appears mid-restore, CLI
--resume registers after the successful resume, and both the open
path and the detail-GET lazy rehydrate wire the registration.
Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
teardown path (close, close_idle, evict, delete, discard) funnels
through it — and session_worker.send re-checks under the same lock,
so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
message could not be delivered (queue_full / refused_closed) instead
of reading as success; staged attachments survive for the retry;
/send surfaces a closed workstream as 404 rather than queue_full.
Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.
Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).
(cherry picked from commit e60c19befd)
Review feedback (PR #800): the URL lane hard-capped fetched bodies at
10 MB before kind resolution, making the 32 MiB pdf cap unreachable for
URL targets while path targets honored it. The flat pre-check is gone;
the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values())
- mirroring the path lane's stat pre-check - and the per-kind caps after
resolution stay authoritative.
Also drops a redundant function-local asyncio import in test_console.py.
(cherry picked from commit bbe92faca1)
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.
Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.
(cherry picked from commit 29a4bbf876)
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.
New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.
The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).
Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.
(cherry picked from commit 09abc9d199)
Four follow-ups to the preview pane:
- Probe-mode preflight: the pane preflights src-loaded kinds with
GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
the console reverse proxy forwards HEAD as a full GET, so the old
preflight dragged the whole blob across the node→console hop twice.
Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
store time (declared charset → UTF-8 → cp1252-replace ladder), same
model the web kind already used. The ladder applies only when the
text kind was DECLARED (MIME/extension/override); the bare no-hint
fallback stays strict UTF-8 and NUL bytes still hard-reject, so
binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
data:; font-src data:" — they render with inline styling but cannot
contact their origin site (no viewer IP/traffic disclosure). A
per-pane "Load remote images & styles" checkbox (web previews only,
sticky, not persisted) reloads with ?assets=1 for the permissive
bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
like the conversation pane, with preview-scoped code-block/KaTeX
chrome (the conversation theme is .msg.assistant-scoped).
Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.
(cherry picked from commit 1e2ab91ec2)
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
(cherry picked from commit e010124008)
The discovered-endpoint wrapper converted every OAuthSSRFError to a
bare OIDCError, so a private-resolving endpoint or trusted host got the
non-public message without the allow_private_network remediation even
though the same knob fixes it. Hoist the hint into a module constant
and append it in both wrappers.
Also name "unspecified" in the refused-even-with-opt-in message so
0.0.0.0/:: rejections read unambiguously.
(cherry picked from commit 4350248d8f)
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.
Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.
The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.
(cherry picked from commit 9c74673fd4)
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or
HTTPS://user:pass@host in tool output leaked the password past the
case-sensitive scheme alternation. Compile with IGNORECASE on both
sides of the FE/backend mirror; the structural userinfo requirement
is unchanged. Uppercase-scheme cases added to both test suites.
(cherry picked from commit 19b1a04f17)
Connection-string redaction (the output_guard pattern and its frontend
mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy
dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://,
mysql+pymysql:// — leaked the password through every redaction surface.
The scheme now takes an optional +suffix instead of enumerating drivers.
redactCredentials() also gains a single early-exit prefilter scan ahead
of its sixteen replace passes, for plain-log tool output on card render.
The prefilter is documented and pinned as a superset of the pattern
set's required substrings, so a miss is provably a no-op: new smoke
cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@'
anywhere in the text still redact, alongside the fast-path no-op and
the driver-scheme URLs on both sides of the mirror.
(cherry picked from commit ed2623ff44)
- Proven: name supervisory control's controllability and nonblocking
conditions as the ancestors of gate-early-on-irreversibles and the
always-enabled escalation required behind a learned veto; cite TCSEC
covert-channel analysis (NCSC-TG-030) for the verdict channel.
- Asserted: add the narrow-only rule's influence-side twin (verdict
payloads to the plant selected, never generated).
- New caveat paragraph: a denial is free only in the authority lattice;
in the dynamics it is an input (selection + targeted-liveness
channels), so a learned veto needs a nonblocking escape it cannot
disable, verdict payloads are selected rather than generated with the
symbols/tokens/language thresholds bounding the alphabet, and the
strongest form dissolves the verdict into scheduling over
deterministic checks.
(cherry picked from commit 625218b7b5)
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary
SDK exception text can embed filesystem paths. Echo the exception type
only; log the full exception with traceback at the raise site.
(cherry picked from commit a029849724)
SDK client construction can fail on environment problems the config
never sees (e.g. httpx resolving a certifi CA path deleted by a venv
rebuild). Those escaped as bare exceptions and turned every workstream
open/create into an opaque 500; re-type them as ValueError in
ModelRegistry.get_client so routes answer 503 with the message and the
alias.
(cherry picked from commit 9c2e809b26)
to_tsvector was computed inline over full row content, so one row
whose tsvector exceeds PostgreSQL's 1MB limit aborted every
search_history scan. Cap the FTS input at 250K chars (worst-case
tsvector expansion stays under the limit; giant rows remain findable
by their head). The ILIKE fallback also never ran on postgres: the
failed statement leaves the autobegun transaction aborted, so roll it
back before falling back.
(cherry picked from commit 51ed336989)
Address review on #794. The prior gate deepcopied every agent tool, then compared, then discarded the copy on a no-op render; and its comment framed the equality case as 'no personas' when it also covers an idempotent re-render of unchanged aliases/personas. Compute the target model/persona descriptions from the current (read-only) schema first and only deepcopy when one differs — so a no-op render is genuinely allocation-free, not just fork-free. Behaviour is unchanged: identity preserved when nothing differs, stale text still cleared on reload.
(cherry picked from commit 90663ce695)
_render_agent_tool_descriptions rebuilt self._tools and reassigned it on every session init, deep-copying task_agent even with no model aliases and no personas to inject — the single-model CLI case the docstring says is skipped. This regressed after the persona-discoverability change removed the early 'if self._registry is None: return' guard, breaking the session._tools is INTERACTIVE_TOOLS invariant (test_session_without_mcp).
Gate the reassignment on whether a description actually changed: keep the original tool object when the render is a no-op, fork self._tools only when something was injected. Restores the shared-constant invariant, makes repeated renders idempotent, and preserves clear-stale-on-reload (an emptied registry still takes the changed path).
(cherry picked from commit bd37bcd1ec)
Add .github/FUNDING.yml to enable the native GitHub Sponsor button, plus a Sponsor badge and a Support section in the README. Primary CTA is GitHub Sponsors (eous); PayPal (paypal.me/eousphoros) is offered as a one-off fallback.
(cherry picked from commit a61d454df5)
Review feedback on #792: the "not found or disabled" branch
interpolated the raw input unquoted, so the whitespace-only and
trailing-space inputs the forgiving lookup explicitly handles rendered
invisibly in CLI output and logs. Use {name!r} like the other two
resolver errors already do.
(cherry picked from commit 647939fe4d)
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.
- Inject the live persona catalog (enabled, interactive-kind; children
and sub-agents are always interactive) into the `persona` parameter
description of task_agent / spawn_workstream / spawn_batch, riding
the same render path as the model-alias injection. Rebuilt from the
pristine TOOLS base every render, so repeated renders are idempotent
and archived personas drop out instead of lingering. Entries carry
name + default marker + <=96-char description; names-only past 25
personas. spawn_batch's persona property is nested per-child under
children.items.properties (located via _persona_property, null-safe
against name-colliding MCP tools). Storage-less sessions keep the
base text: the render runs at session construction, so it gates on
is_storage_initialized() rather than get_storage(), which would
auto-init SQLite as a side effect.
- resolve_persona_for_kind - the ONE shared rule behind the HTTP
create handler, CLI --persona, the coordinator spawn precheck, and
task_agent prep - is now forgiving: exact slug, then the lowercased
input (created names are regex-enforced lowercase slugs), then a
case-insensitive display-name match accepted only when unique among
the kind's enabled personas. Duplicates refuse loudly naming the
candidate slugs; a same-label persona of another kind neither blocks
nor wins (the label the caller saw came from a kind-filtered
surface); whitespace-only input never matches blank display names
(display_name defaults to ""). Every failure now enumerates the
kind's valid names, so a stale injected list or a typo self-corrects
on the next attempt.
- The canonical slug is stamped everywhere: task_agent prep rewrites
its arg from the resolved snapshot, _validate_child_persona returns
(canonical, error) and both spawn call sites adopt it - approval
chrome, the wire, and workstream_config never carry a forgiven
variant.
- Create-persona shelf: label hint under Name explaining agents and
the CLI launch the persona by this name (case-insensitive) and the
display name is only a list label. docs/personas.md gains a "How
agents discover personas" section and drops the stale claim that
task_agent has no persona parameter.
Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.
(cherry picked from commit 457b01737a)
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from
the global registry (get_workstream_row / for_request without a storage arg),
which can evaluate the project-tenancy decision against a different or
auto-initialised backend and fail OPEN on a missing project row. Use
request.app.state.auth_storage explicitly, matching cluster_ws_detail and
_resolve_coordinator_or_404; fail closed (404) when it is unset.
- reject_unassignable_scopes now derives its allowed-scope error message from
ASSIGNABLE_SCOPES so validation and the message can't drift.
(cherry picked from commit a40ff249ec)
Workstreams attached to a private project were visible -- including their
conversation content -- to holders of admin.cluster.inspect / admin.coordinator
(both default builtin-admin permissions), defeating the project's confidentiality
boundary. Enforce that a private project's resources are visible only to people
IN the project (owner, workstream creator, or an explicit member), even for admins.
Surfaces closed:
- WorkstreamProjectVisibility bypass narrowed to service scope only (node->console
machine plumbing, re-filtered per-user at the console edge). No human principal
bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This
flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams,
cluster_node_detail, and cluster_snapshot/SSE.
- cluster_ws_detail 404-masks a workstream in a private project the caller can't
see; cluster_ws_live_bulk routes such ids to the denied list (no private-project
oracle).
- Coordinator operator verbs (history/export/detail/send/approve/set_title/open/
children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check
on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks),
the tenant_check now run in make_open_handler before rehydrate, and a
project-visibility check in _coord_attachment_owner. admin.coordinator gates the
surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the
manager-first + coordinator-kind ladder so kind-isolation is preserved.
- service scope is no longer user-assignable: admin_create_token and both
turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an
admin.users holder cannot self-mint a service token and restore the bypass. Service
scope is minted only by ServiceTokenManager / the JWT secret.
- The events/global node proxy (service-elevated cross-tenant firehose) is gated on
admin.cluster.inspect so a plain authenticated user cannot reach it through the
console proxy.
Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds
tests for every surface (visibility predicate + cluster detail/bulk + coordinator
history/export/children/open/attachments + events/global proxy + scope-mint
rejection); inverts the tests that pinned the old admin-bypass contract.
(cherry picked from commit 36419a9809)
* fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
* fix(mcp): route pool transport lifecycles through per-entry owner tasks
Completes the owner-task migration started for the static path: the
oauth_user pool path had the same latent anyio cancel-scope exposure
(host-task-bound scopes entered by short-lived connect tasks; a scope
cancelled after its host finished re-delivers cancellation via
call_soon forever - the 100%-CPU zombie), previously covered only by
the disarm backstop.
Each (user, server) pool entry's transport + ClientSession cms are now
entered, parked, and exited by ONE long-lived owner task
(_pool_transport_owner). The caller keeps building client_kwargs (the
per-user bearer and, when an auth-capture carrier is active, the
httpx_client_factory response hook) so 401/WWW-Authenticate capture
semantics are unchanged. Teardown is the shared one-cancel close
protocol (_teardown_pool_entry: signal before first await, graceful
grace, at most ONE cancel), used by the connect stale-guard, idle/LRU
eviction, and shutdown (parallel signal-then-reap). Unrequested owner
death evicts the session but keeps the entry and its discovered
catalog, matching the existing evict-session-keep-entry semantics the
auth_401 retry relies on.
Discovery still runs in the connecting caller while the transport is
hosted by the owner, so a transport collapse mid-discovery (e.g. the
SDK tearing its task group down on an upstream 401) cancels the OWNER,
not the caller - a bare await on the response stream would hang until
the 30s phase timeout. _await_pool_discovery races each discovery
await against owner completion and converts owner death into a prompt
ConnectionError (the owner is never cancelled there; teardown owns its
lifecycle). Carrier-first failure classification preserves auth_401
semantics for captured 401s.
With no cross-task stack closes left, _safe_close_stack and
_safe_teardown_on_connect_failure are deleted (zero callers).
Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol
(graceful event-before-await close, exactly-one-cancel escalation,
owner-death eviction retaining entry+catalog, caller-cancel-mid-connect
cm-exit guarantee, factory-present-iff-capture, and the
owner-death-during-discovery fast-fail). 1010 mcp tests and the full
8471-test suite green, including the historical cross-task-anyio
sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds;
ruff+mypy clean; zero destroyed-task warnings.
* fix(mcp): harden disarm-sweep loop guard and owner BaseException arm
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
* fix(mcp): extend owner-death discovery fast-fail to the static path
The static connect path had the same exposure the pool's discovery race
closed: discovery runs in the connecting caller while the transport is
hosted by the owner task, so a transport collapse mid-discovery cancels
the OWNER and the caller's bare await on the response stream hung until
the caller-side attempt timeout (~45s) instead of failing promptly.
_await_pool_discovery is renamed to _await_owner_discovery (it is now
path-neutral) and wired into _connect_one_locked's four discovery
awaits. The helper also converts a discovery future that completes
CANCELLED without the race's own reap (an SDK-internal cancellation
shape) into the same ConnectionError, instead of leaking a bare
CancelledError the caller would misread as its own cancellation.
The pool transport owner's BaseException arm gains the same refinement
the static owner received in review: interpreter-level exits
(KeyboardInterrupt, SystemExit) re-raise after delivery to the
readiness future instead of being swallowed.
Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s
hang), and a direct pin on the cancelled-discovery-future conversion.
* fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
* fix(mcp): mirror targeted-catch + finally delivery in the pool owner
Same shape the static owner received in review: the failure arm catches
only (BaseExceptionGroup, Exception), and waiter delivery for anything
else moves to a finally that resolves the readiness future with a clean
ConnectionError before the task unwinds - interpreter exits and
BaseException-derived library escapes propagate exactly once, uncaught,
and the waiter can never be left hanging.
* test(mcp): narrow the escape test's waiter catch to explicit types
* test(mcp): narrow discovery-race waiter catches to explicit types
* refactor(mcp): make reap/synchronization awaits explicit to analyzers
Full-absorb reaps (cancel-then-drain of a future whose outcome is
deliberately consumed) become `await asyncio.gather(x,
return_exceptions=True)` - one line, self-describing, and in the
owner-died discovery reap it is also a small semantic improvement: a
caller cancellation arriving during the reap now propagates instead of
being masked by the ConnectionError. Bare synchronization awaits and
selective suppress blocks in tests keep their raise-through semantics
via throwaway assignment. Applied uniformly across the owner-task
test files, including sites introduced by the static-path PR.
(cherry picked from commit 0c2c534c86)
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
(cherry picked from commit 8ed86ae7ab)
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
(cherry picked from commit ed30e4f0bf)
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
(cherry picked from commit 62f62ae624)
The bounded key prefix pattern (api_key=/secret_key= etc.) avoids
monkey/turkey false positives, but compound keys like secret_access_key
and aws_secret_access_key only matched on the access_key= suffix, leaking
the secret_ / aws_secret_ prefix. Added these as explicit alternations.
Also added bearer_token and secret_token to the token prefix list.
(cherry picked from commit c5ff3147ce)
Bare alternatives (|token, |key) for standalone token= and key= assignments
were re-added. A negative lookbehind (?<![a-zA-Z0-9_]) prevents matching
word-suffixed identifiers like monkey=, turkey=, mytoken=, over_tokenized=.
Also applied the same protection to _RE_QUERY_CRED which had a bare |token
alternative without boundary protection.
(cherry picked from commit bfcfb0c791)
- Replace unbounded [a-zA-Z0-9_]*key= prefix with specific credential
key suffix alternation to avoid false matches on monkey=, turkey=, etc.
Same for *token= (access_token=/auth_token= but not over_tokenized=).
- Hoist redactCredentials() out of per-line diff render loop in
buildConvCmd - 1 call on full text instead of N calls per line,
eliminating ~2800 regex passes worst-case.
- Remove bare 'key' from _RE_QUERY_CRED alternation (too aggressive).
- Add x-api-key / x_api_key to JSON secret key lists in both Python
and JS.
- Add re.IGNORECASE to configurable-mode credential_bearer pattern.
- Fix annotation dedup guard in _check_credentials (was checking flag
name against annotation prose list, always-true dead code).
(cherry picked from commit cbf5c5f3b6)
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:
- Detect and redact single-quoted JSON secrets such as
{'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
access_token=) as a whole rather than chewing the tail into a garbled
"api_[REDACTED:api_key]", while still covering bare key=/token=. A word
boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
the manual value-slicing helper for a capture-group substitution.
The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.
(cherry picked from commit 31a1d5c3ee)
New shared ES6+ module (redact_credentials.js) provides comprehensive
visual credential censorship matching the backend output guard patterns:
- PEM private key blocks, connection strings, Bearer tokens
- OpenAI / GitHub / AWS / Google API key formats
- Query-string and JSON-style credential values
- JSON secret keys (api_key, password, token, authorization, etc.)
- ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.)
Integrated into both frontend surfaces:
- interactive.js: replaces legacy minimal _redactApiKeys function
- conversation.js::buildConvResult (shared substrate, used by coordinator)
- coordinator.js::renderToolOutput fallback paths
Backend parity: added 'authorization' to the JSON secret regex in
output_guard.py so the output guard flags and redacts Authorization
headers in JSON tool output.
Tests: ported the runtime smoke test from the removed _redactApiKeys
to import the new module directly; added redact_credentials.js to the
var-free and const-reassign guard bundles.
(cherry picked from commit 93a7486cc2)
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and
`wire.tool_args_legalized` (DEBUG), and tool arguments are model/user
controlled — they can carry secrets (a token in a bash command, a password in a
connection string) or raw CR/LF that break log lines. Route the preview through
`output_guard.redact_credentials` over the full value first (before the 120-char
cap, so a secret straddling the cut isn't half-shown past the pattern's reach),
then collapse every control char to a space, mirroring `audit._scrub_string`.
Addresses the PR review comments.
(cherry picked from commit 56624f9597)
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.
Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.
Convert `lowering.py` to structlog so the new pass emits structured events.
(cherry picked from commit 16a68ae6d6)
The pane/workstream accelerators only worked on macOS. They were bound to
Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T,
Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab /
switch tab) and never reached the page. macOS browsers own Cmd instead, so
Ctrl was free there and everything appeared to work.
On top of that the shortcuts were declared in three places that had drifted
apart — the "?" overlay, each app.js keydown handler, and the tab-menu
badges in shell.js. The console fell to convTabMenu's node-proxy fallback
lane, which dropped every shortcut badge (and Fork), so its tab menu showed
no accelerators and Ctrl+W there just closed the browser tab.
Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and
make shell.js the single source of truth for the per-pane accelerators: a
stable accel registry drives both the platform-aware badge and one shared
keydown handler that invokes the ACTIVE pane's own menu item, so a badge
can't advertise a chord the handler ignores and each surface contributes
only what it supports (the console omits Fork; it has no fork surface yet).
Each surface's app.js keeps only its global accels (new / switch /
dashboard); the console regains switch + dashboard to match. Mod+W now
uniformly means Close pane (drop the tab, session keeps running), matching
its badge and the universal Ctrl+W convention — previously the standalone's
Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R"
is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused
(macOS transpose / delete-forward).
(cherry picked from commit 2d4cb6fea9)
* Fix send button stuck disabled by pruning orphaned approval cycles
When a DOM wipe (clear_ui / replay_truncated / replaceChildren)
detaches approval card elements while an approve_request event is
processed between the wipe and refetch-restore, the matching
approval_resolved may never arrive. The orphaned cycle entry in
approvalCycles keeps pendingApproval=true and the send button
disabled forever.
The fix adds a pruning pass at the top of _syncApprovalState():
cycles whose blockEls are all .isConnected === false are deleted
from the Map. This runs on every register/resolve/rebuild so
orphans are cleaned up promptly.
Also fixes an ordering bug in showInlineToolBlock discovered
during review: the block element was appended to the DOM after
_registerApprovalCycle, so the new isConnected prune would kill
the just-registered cycle before it took effect.
* Fix comment inaccuracy in showInlineToolBlock append-before-register guard
The comment said 'blockEls.every(el => el.isConnected)' but the
actual prune check is '!blockEls.some(el => el.isConnected)' -
no block elements are connected, not every element.
The wire-payload golden matrix had no anthropic-compatible coverage —
both AnthropicProvider rows are the native lane (compat=False), so the
distinct compat wire shape (reasoning control in
extra_body.chat_template_kwargs, never the native thinking param) was
unfrozen. Add the compat lane across all eight representative fixtures
with a manual-mode capability (the lane has no static table, so caps
ride in as a model definition would supply them) and reasoning_effort=
high: every golden now pins {enable_thinking: true, reasoning_effort:
high} in chat_template_kwargs, asserts the native thinking param is
absent, and preserves temperature (no forced 1.0). _capture gains an
optional caps override to support the no-static-table lane.
Copilot review caught the native Anthropic adaptive ladders (sonnet-5,
fable-5, opus-4.8/4.7/4.6) labeling the none position 'None — sends
adaptive' — raw mode vocabulary the plain-language rule exists to
prevent. But treating the 'adaptive' token identically to the local
lane's 'on' would still misframe it: on these models thinking is always
on and an effort level rides output_config on every graded position
(verified on the wire), so 'thinking stays on' is true everywhere and
not what distinguishes none. The none position uniquely means no effort
is pinned — the model self-regulates it — so it now reads 'None — model
sets effort'. The local adaptive lane's bare toggle keeps 'thinking
stays on' (no effort lever exists there).
Aliased knob positions were labeled after the lowest sibling sharing
their wire token — a toggle-only model rendered 'Max (= minimal)',
implying a minimal-effort downgrade the wire doesn't contain, and with
declared values 'High (= minimal)' while the wire carries high. Each
position now states its delivered level: exact matches stay plain
('Max'), snapped positions say 'Low — sends high', the adaptive none
position warns 'thinking stays on', budget detail stays in the
tooltip. Effort-param placeholder corrected to the real graded keys
(reasoning_effort / reasoning).
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:
- openai-compatible: the flat reasoning_effort param carries the knob
verbatim (effort_passthrough on the lane default); declared values
still snap ordinally, and a declared effort_param still claims the
template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
alongside the toggle whenever reasoning control is engaged — under
the operator's effort_param, else the conventional fallback key
(reasoning_effort); templates that don't reference the kwarg ignore
it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
effort control' (o1-mini) and the ordinal snap is unchanged.
Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (03f82521) the default-escaping writer
could never reproduce — regens no longer churn unrelated lines.
Local-lane model ids are operator-chosen strings (vLLM
--served-model-name), so a prefix collision with a cloud model id
inherited that model's sampling and effort contract: a box named
o3-distill silently lost temperature support, and one named
gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values
it never declared. Both surfaces of the lane now return plain defaults
(OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and
the responses pin via a compat-mode OpenAIResponsesProvider mirroring
AnthropicProvider(compat=True). Everything beyond the defaults is
declared by the operator on the model definition, matching the
anthropic-compatible lane and lookup_model_capabilities' documented
'no static table for local models' contract. The commercial openai
lane (Responses-only) is untouched.
Pre-split tests that reached commercial rows through the chat-class
OpenAIProvider alias now source them from lookup_openai_capabilities;
their subject (registry rows + shared gating helpers) is unchanged.
The except path for a malformed capabilities column appended the row
without the key, so clients had to null-check a field the happy path
guarantees. Initialize each entry with an empty ladder and let the try
block overwrite it — the response schema is stable per row.
Three server.test.ts / server-attachments.test.ts cases still asserted
the pre-verb-lift send shape (POST /v1/api/send with ws_id in the body).
The server route has been POST /v1/api/workstreams/{ws_id}/send (ws_id
in the path, {message} body) since that lift, and the SDK send() was
updated with it — only these expectations rotted. They fail on main
too; unrelated to approvals, folded in here to leave the TS SDK suite
green. Test-only, no SDK source change.
ApproveRequestEvent.cycle_id and ApprovalResolvedEvent.cycle_id /
call_ids were typed required, but they are 1.7 additions: a pre-1.7
server omits them on the wire, so a current SDK talking to an older
node sees undefined. The UI and channel adapters already keep the
legacy no-selector fallback for exactly that case. Mark them optional
so consumer code can't assume a string that may be absent — matching
the Python SDK, whose dataclasses default all three.
The interactive composer had two uncoordinated writers of
sendBtn.disabled: _syncApprovalState wrote `pendingApproval || busy`
directly, while Composer.setBusy wrote it too via _reconcileDisabled.
In queueWhileBusy mode the composer's write re-enables send, so a
state_change to "attention" (exactly the pending-approval state)
firing after an approve_request card rendered raced the approval
disable back off. The `|| busy` term also defeated the "Queue
message…" affordance whenever _syncApprovalState ran mid-turn.
Opt the composer into externalDisable (it keeps rotating the
Send/Queue label + placeholder + stop button, but no longer writes
the flag) and route every axis — live approval cycle, cross-user send
gate, busy — through one App reconciler, _reconcileSendDisabled. Busy
is intentionally not a disable axis here: queueWhileBusy keeps Send
clickable as "Queue" while the agent runs.
Pre-existing on main (the old scattered direct writes had the same
collision); surfaced by the concurrent-cycle review.
The checked-in openapi-server.json / openapi-console.json still
described the removed singular pending_approval_detail field; re-run
generate-types.py so the reference specs carry the
pending_approval_details list + cycle_id. Retire two docstring
references to the deleted singular serializer and one stale comment.
New regression matrix for the release blockers: cross-approval
independence, lost-wakeup at gate entry, FIFO selector-less
resolution, resolve-all sweep, double-resolution no-op, cards/legacy
view tracking, and the generation-exactness set — stale delivery
rejection, Smart-Approvals origin check, purge keep_origin, the
purge-to-register window eviction, late cross-generation "superseded"
stamping, concurrent smart+human gates, and the pre-delivered-verdict
fast path. Plus sub-agent judge wiring (agent_gate off the main
slot, close() firing all generations) and endpoint tests for cycle
pinning and the Approve+Always race guard.
Gate threads run under one shared mock-patch harness — mock.patch
start/stop of the same target from concurrent threads corrupts the
patcher's restore stack — with a sweep-until-dead teardown so the
conftest leak guard can't trip. Existing suites migrate off the
singleton fields to cycle assertions and the
pending_approval_details wire shape.
interactive.js tracks live cycles in a Map: per-cycle action buttons
and feedback, per-cycle optimistic clears and resolved-status pills,
keyboard routed to the oldest (or the focused) cycle, announce-shell
dedupe, and a composer that stays disabled while ANY cycle is live.
Verdict glow is scored per batch — a sibling's verdict neither
recolors the oldest card nor leaves its own card stale.
coordinator.js renders one approval block per pending_approval_details
entry, posts child approve/deny with the block's cycle_id, clears
exactly the resolved cycle on approval_resolved (legacy events without
one clear all), and replays every entry from snapshots.
Track posted approval prompts by (ws_id, cycle_id) — one message per
concurrent cycle — so parallel task agents' prompts resolve
independently. Buttons carry the cycle in their value (Slack) /
footer (Discord); intent verdicts route onto the owning cycle's
message by call_id membership.
The exact-key-then-legacy-fallback lookup (an empty cycle_id from a
pre-multi-cycle server resolves the workstream's single tracked
entry) and the all-cycles sweep are centralised as shared _routing
helpers so the fallback semantics can't drift between adapters.
POST /approve accepts cycle_id / call_id selectors and 409s on stale
selectors with the current cycle's ids so clients re-render instead of
silently resolving an unrelated batch. Selector-less bodies pin the
resolve to the cycle the lookup returned (not "whichever is oldest by
the time the resolve runs"), and Approve+Always names apply only after
the pinned cycle actually resolved — the auto-approve whitelist can no
longer describe a different batch than the one that resolved.
approve_request carries cycle_id; approval_resolved carries cycle_id +
call_ids; SSE reconnect replays every live cycle's card. The console
collector, coordinator UI fan-out, and both SDKs (Python + TS) thread
the cycle correlation through.
BREAKING (1.7): the singular pending_approval_detail field is removed
from dashboard rows, workstream detail, and node snapshots — replaced
by the pending_approval_details list (one entry per live cycle, each
carrying its cycle_id). stable/1.6 keeps the old shape.
task_agent tool calls reached the approval gate judge-blind: no
heuristic verdict on the card, no LLM verdict, no audit row, and Smart
Approvals could never clear them. Run _evaluate_intent on the
sub-agent gate with the sub-agent's own trajectory as judge context —
its task prompt is the delegation contract the operator approved, so
"does this call serve the task" is the right local alignment question.
Sub-agent spawns are agent_gate generations: they never touch the main
loop's supersede slot (parallel siblings would make each other's
verdicts look stale), staleness is enforced per-cycle by the UI's
generation checks instead. Every generation registers in
_judge_cancel_events — kept exact by the judge's new done_callback —
so close() aborts all in-flight daemons; judge.cancel_on_approval
fires per-gate exactly like the main loop. CLI and eval UIs accept
the judge_event delivery kwarg.
The approval pipeline was a per-UI singleton (one card, one Event, one
result slot) multiplexed by N concurrent gates. With parallel task
agents that meant one click could resolve every parked batch with the
same verdict, a sibling's gate entry could eat a just-fired resolution
(3600s "stuck dialog" hang), and Approve+Always could whitelist a
different batch than the one on screen.
Replace the singleton with a registry of ApprovalCycle objects keyed
by cycle_id: per-cycle events/results/decisions/verdict parking,
oldest-first selector-less resolution, guarded double-resolution, a
resolve_all_approvals sweep for cancel/close/worker-recovery, and a
maintained oldest-cycle view in the legacy _pending_approval slot for
boolean-ish consumers.
Verdict bookkeeping is generation-exact end to end: the entry purge
spares verdicts the entering batch's own judge spawn already delivered
(a fast judge no longer stalls the Smart-Approvals wait to its full
budget), registration evicts stale-generation arrivals that land in
the purge-to-register window, recent decisions are generation-tagged
so a stale generation's late verdict stamps "superseded" instead of
stealing a reused call_id's decision, and Smart-Approvals
qualification identity-checks the delivering generation.
Capability-registry corrections verified against the official OpenAI
reasoning guide, the Azure reasoning-models matrix (2026-06 revision),
and the Anthropic models-overview/effort/migration pages (2026-07):
OpenAI (vocabulary confirmed none/minimal/low/medium/high/xhigh — no
"max" level exists; knob max rides the xhigh ceiling via the ordinal
snap):
- o1/o3/o3-mini/o3-pro/o4-mini declare low/medium/high (every o-series
model except o1-mini) — without declared values the session knob was
silently dropped for these models. o1-mini stays effort-free.
- gpt-5.5 default corrected none -> medium (5.5 reasons by default,
unlike 5.1-5.4).
- gpt-5.1-codex-max gets an explicit row: it prefix-matched the
gpt-5.1 row (no xhigh), capping the knob's xhigh at high on the one
model xhigh was introduced for.
Anthropic (effort-page matrix):
- claude-sonnet-5 row added — it previously fell through to
_ANTHROPIC_DEFAULT (manual budgets, 200k ctx, no effort), all wrong:
adaptive-by-default thinking (manual budgets are a 400), sampling
params rejected, 1M ctx / 128k out, effort low..max incl. xhigh.
- claude-sonnet-4-6 gains its documented "max" effort level (knob
xhigh now rides max, not high) and the stale 64k max_output becomes
the documented 128k.
- fable-5 / opus-4-8 / opus-4-7 / opus-4-6 / opus-4-5 rows verified
correct as declared.
Knob semantics completed: resolve_reasoning_effort now forwards the
knob's "none" position verbatim when the model DECLARES an explicit
none level (gpt-5.1+, grok-4.3) — omitting the param there leaves a
reasoning-on server default (gpt-5.5: medium) in charge of a knob
that promises off. Models without a declared none still omit, and
none is never a snap target. Parity harness swaps its synthetic
openai shape for the real gpt-5.5 registry row.
The knob domain grew xhigh/max after the snapping fallbacks were
written, which silently inverted their semantics: off-list meant
"unrecognized string" then, but now usually means "above the model's
ceiling", where falling back to the default tier is directionally
wrong (grok-4.3 at knob max got low; values low/medium/high at knob
xhigh got medium; Anthropic manual mode gave xhigh/max a 4096 budget
while high got 16384).
One rule everywhere now, via snap_reasoning_effort in _protocol:
exact match wins; otherwise the smallest declared level ranking at or
above the knob; above the ceiling, the ceiling. "none" is never a
snap target, and default_reasoning_effort only catches values the
ordinal snap cannot rank.
- resolve_reasoning_effort (flat chat / responses / validated
effort_param lanes) snaps ordinally: xhigh over (low, medium, high)
now sends high; xhigh over DeepSeek-style (high, max) sends max —
matching DeepSeek's official xhigh-to-max aliasing, so a declared
values list now reproduces that contract instead of defeating it.
- _map_reasoning_to_effort (native output_config) rounds up too:
knob xhigh on Opus 4.6 (low, medium, high, max) rides max instead
of silently dropping output_config.
- EFFORT_BUDGET_MAP is monotone across the whole knob domain:
minimal/low 1024 (API floor), medium 4096, high 16384, xhigh 32768,
max 65536. Unknown strings still fall to the 4096 default.
Google defaults are unaffected (ceiling and default coincide at
high); wire goldens unchanged. Parity harness caught the budget
clamp interacting with its own max_tokens during development —
capture budget raised above the largest manual budget.
Proves the effort-ladder projection against the real request path
instead of against the mapping helpers it shares with it. For 22
(provider lane x capability shape) points — both Anthropic lanes,
openai-compatible on both API surfaces, openai, google (default and
template-override hybrid), xai (default and inert-override), and the
DeepSeek/qwen template contracts — every knob position is driven
through the actual provider create_streaming against a recording fake
client, and two invariants are asserted per shape:
1. each ladder token decodes to an expected effort wire subset
(toggle / template effort / flat param / thinking budget /
output_config) that must equal the captured kwargs exactly;
2. two knob positions carry equal tokens iff they produce identical
effort-relevant wire payloads — the grouping promise the UI
annotations lean on.
The RecordingClient SDK-seam stub moves from the wire-payload golden
harness into tests/_wire_capture.py so both suites capture at the same
seam. Verified the harness catches the bug class it was built for:
re-adding xai to _CHAT_LANES fails xai-template-override-inert.
Second external audit round on the ladder. Verified and fixed:
- xai was in _CHAT_LANES on the false premise that XAIProvider
subclasses the chat provider. It subclasses OpenAIResponsesProvider,
whose surface ignores extra_body entirely, so a thinking_mode /
effort_param override never changes an xai request — but the ladder
claimed a template toggle ("on+low") that does not exist on the
wire. xai now projects through the flat channel only, like openai.
- The flat-param suppression rule (a declared effort_param claims the
template channel) was encoded independently in
apply_temperature_and_effort and the ladder. Extracted into
flat_effort_suppressed() in _protocol so the request path and the
projection cannot drift.
- admin_effort_ladder logs the swallowed resolver exception before
returning 400 (a genuine bug would otherwise hide as a silent 400).
- list_available_models reads server_compat with .get() instead of
destructively popping it out of the parsed capabilities dict.
- models_changed SSE now re-annotates the skill launch-config effort
select after invalidating the models cache instead of leaving a
stale ladder until the next keystroke.
- Capabilities JSON textarea placeholder hints the two effort fields
that have no structured control (reasoning_effort_values,
default_reasoning_effort).
The Gemini effort fix (ee9e9c1f) adds a flat reasoning_effort to every
Google chat-completions request at the session default knob — the
golden fixtures now carry it. Only the eight google__* goldens change
(one added key each); other providers' goldens are untouched — the
UPDATE_WIRE_GOLDENS pass also wanted to rewrite twelve passing goldens
with escape-format-only churn (raw em-dash vs \u2014), reverted to
keep the diff semantic.
The one that mattered: /v1/api/models passed the capabilities column —
a JSON STRING (sa.Text) — straight into effort_ladder_for_model, whose
field filter calls .items() on it; the per-row guard swallowed the
AttributeError, so effort_ladder was silently absent from every row and
the sklc annotation could never fire. The endpoint now parses the JSON
and splits the namespaced server_compat exactly like the model_registry
loader, and a regression test seeds a string-capabilities row.
Projection fidelity: effort_ladder_for_model threads api_surface (the
responses surface ignores extra_body — flat-param-only ladder, matching
create_provider's request-time divergence); google/xai route through
the chat-lane projection they actually inherit (_finalize_extra_body +
flat param); the native-Anthropic branch reflects that output_config
gates on supports_effort alone, independent of thinking_mode; budget
clamping to per-request max_tokens is documented as out of scope.
Hardening and symmetry: admin endpoint 400s (not 500s) on non-dict JSON
bodies and gained HTTP tests; the admin edit-load strips effort_param
from the raw JSON only for the lanes whose save path re-adds it, so
non-compat rows can't silently lose a stored key; the empty-model early
return bumps the ladder sequence so stale in-flight responses can't
re-annotate; alias labels only reference positions the target select
actually offers (the skill shelf omits none/minimal); the sklc models
cache no longer pins a rejected promise and is invalidated on the
models_changed event; debounces unified at 500ms;
merge_reasoning_template_kwargs now always returns a fresh dict for
non-empty input; the shared budget constants are public.
Seven knob positions render as seven behaviors in the UI, but the real
ladder depends on the lane and the model: qwen3.6 has two (off/on),
DeepSeek-V4 three, Claude 4.6 five. Operators had no way to see which
positions alias — the confusion class behind silently-equal effort
levels.
providers/effort_ladder.py projects the knob domain through the same
mapping functions the providers use at request time (resolve_reasoning_
effort, reasoning_template_kwargs, the manual budget map — hoisted to a
shared constant so the projection can't drift), yielding
{value, effective} rows where equal tokens promise identical requests.
/v1/api/models rows now carry the ladder (guarded per row), and
POST /v1/api/admin/models/effort-ladder computes it for the admin
modal's unsaved edits.
The admin per-model effort select and the skill launch-config effort
select annotate aliased positions ("Max (= high)", "None (model
default)") with a sends-tooltip; annotations refresh as thinking-mode /
effort-param / capabilities fields change. The ladder describes what
Turnstone sends — server-side templates may alias further (DeepSeek-V4
folds low/medium into its default high tier).
_GOOGLE_DEFAULT declared no reasoning_effort_values, so
resolve_reasoning_effort returned None and the session effort knob was
silently dropped for every Gemini model — the same bug class this
branch fixed on the local lanes. Gemini's OpenAI-compat surface
documents a flat reasoning_effort (2.5: thinking_budget mapping; 3.x:
thinking_level), so declaring values lights up the inherited
chat-completions path.
Values are the safe cross-model set (minimal/low/medium/high): "none"
is excluded because 2.5 Pro and the 3.x family reject disabling
thinking — and the resolver never forwards the knob's none anyway (the
param is omitted, server default applies). Off-list xhigh/max snap to
the declared default high. Encoded from the official compatibility
docs per the static-caps pattern; not live-verified.
Post effort-knob rework, "Enabled" (manual) means knob-controlled —
effort none turns thinking off. Operators who want the pre-#771
always-on behavior (knob never disables) previously had to hand-write
thinking_mode "adaptive" into the raw capabilities JSON. The dropdown
now offers all three representable modes — None / Effort-knob
controlled / Always on — and the edit-load lift captures adaptive
instead of relegating it to raw JSON.
Cross-checked online: Qwen3.6's template documents enable_thinking +
preserve_thinking only — no effort parameter exists (vLLM's flat
reasoning_effort convenience boolean-maps to the same toggle).
DeepSeek-V4 officially accepts reasoning_effort high/max with Think
High as the default thinking tier and low/medium→high, xhigh→max
aliasing — so freeform effort_param passthrough matches the contract
exactly, and a declared values list omitting xhigh/max would make
Think Max unreachable. Live probes on both boxes agree with the
official contracts once the default-tier framing is applied.
Verified findings applied:
- adaptive thinking_mode never knob-disables: the shared mapping now
sends the toggle unconditionally true for adaptive (the native
adaptive branch ignores the knob's none), while manual keeps the
knob-driven contract. Restores the invariant the deleted chat-lane
code upheld.
- a set effort_param suppresses the flat top-level reasoning_effort on
the chat lane: the template channel replaces it — double-sending
could 400 on schema-strict servers and disagree with operator pins.
- admin edit-save no longer drops a stored thinking_param when the
thinking-mode dropdown is empty: the raw-JSON strip now only fires
when a mode value actually round-trips through the dropdown.
- effort_param persistence gated on the local-server lanes so a value
lingering across a provider switch never lands on commercial rows.
- three stale _compat_extra_params references renamed to
merge_reasoning_template_kwargs.
Documented dispositions (no code change): the knob-none-disables flip
on upgrade is intentional and now carries an upgrade note; gateways
fronting real Claude belong on provider=anthropic with a custom
base_url (the compat lane is vLLM-schema-only); nonstandard
thinking_mode strings staying inert is the intended allowlist
contract. The real anthropic provider is unaffected throughout —
official Claude models keep native thinking/output_config.
Hoist the compat-lane injection into _protocol.merge_reasoning_template_kwargs
(next to ModelCapabilities — one implementation for both local-server lanes)
and retire OpenAIChatCompletionsProvider._apply_thinking_mode in its favor:
_finalize_extra_body now receives the session effort knob, so thinking_mode
manual/adaptive maps knob "none" to an explicit thinking_param false
(previously the toggle was unconditionally true) and caps.effort_param
carries the graded effort key on chat completions too. Operator
server_compat pins still win; the Responses surface is untouched (native
reasoning handles effort itself).
The admin Models form grows an "Effort param" field that round-trips like
thinking_param: lifted out of the raw capabilities JSON on edit-load,
re-added on save, cleared by emptying the field.
Verified live against qwen3.6-27b /v1/chat/completions: knob medium streams
reasoning_content, knob none suppresses it.
The compat lane sent no reasoning control at all: vLLM's /v1/messages
has no thinking request field, thinking_mode stayed "none", and the
session effort knob was silently dropped. The reasoning levers live in
the chat template, so fold them into extra_body chat_template_kwargs
(_compat_extra_params): thinking_mode manual/adaptive maps the knob
onto caps.thinking_param (effort "none" = off, mirroring the native
manual-mode contract), and caps.effort_param (new ModelCapabilities
field) carries a graded effort value for gpt-oss-style templates,
validated against reasoning_effort_values when declared. Operator
server_compat entries win on key collision; native thinking params,
temperature forcing, and output_config never fire on compat.
resolve_reasoning_effort moves from _openai_common to _protocol next to
ModelCapabilities — importing it into _anthropic would otherwise cross
provider families.
The admin Models form now shows and round-trips the thinking-mode
dropdown for this lane; the #661 hide was premised on thinking_mode
being inert here, which this change inverts.
Verified live against qwen3.6-27b on vLLM /v1/messages: knob medium
streams a thinking block, knob none suppresses it, an operator pin
beats the knob.
* feat(mcp): autonomous reconnect + liveness for static MCP servers
Static (non-oauth_user) MCP servers had no autonomous reconnect. Every reconnect
path was lazy — a tool dispatch (_cb_auto_reconnect), an operator refresh, or a
config edit — and the MCP SDK's own reconnect is a bounded 2-attempt burst on the
streamable-http GET stream only (verified: mcp 1.28.1), with no backoff and
nothing for the other transports. So a static server that went down and came back
while nobody was dispatching to it stayed disconnected until a dispatch or a
manual reconnect. Worse, a session whose transport dies while idle survives as a
non-None ClientSession with closed streams — nothing evicts it, so even a later
dispatch may not notice until it fails.
Add a static-server health loop on the mcp-loop (started in _connect_all; config
``static_health_check_seconds`` default 30, <= 0 disables):
- Reconnect: a disconnected server (session is None) is reconnected on a capped,
jittered, FOREVER backoff (full jitter, base 1s, cap 60s, no attempt limit) —
a server that returns after a long outage reconnects within ~a minute, and a
permanently-misconfigured one costs at most one attempt per cap. The health
loop owns this clock; the circuit breaker stays the DISPATCH fail-fast gate (a
tool call to a down server errors immediately rather than blocking on the
retry), and the loop keeps breaker state in sync so an open breaker closes on
reconnect.
- Liveness: a connected server is pinged (send_ping) each cadence; a dead-but-
idle one — which nothing else would notice — is evicted so the next tick
reconnects it. This is the core of the "never reconnects" failure.
Serialize _connect_one per server behind a per-name lock (split into a thin
wrapper + _connect_one_locked): the health loop, a dispatch's _cb_auto_reconnect,
and an operator refresh could otherwise interleave teardown/rebuild on the shared
StaticServerState and corrupt it — a latent pre-existing race this also closes.
The body is unchanged (only relocated), so the delicate anyio / wait_for connect
logic is untouched.
Out of scope (follow-up): silent GET-stream / notification death — the SDK stops
the notification stream after 2 attempts while the request path stays alive, so
send_ping is blind to it; the fix is bounded session recycling, which needs a
static in-flight guard first (only PoolEntryState tracks in_flight today).
Tests: backoff bounds (capped / jittered / forever, no overflow), reconnect
success resets backoff + closes breaker, reconnect failure retries forever,
in-flight skip, per-name serialization (no overlap), ping keeps healthy / evicts
dead / evicts on timeout, tick skips oauth_user, connect_all start + disable
gating, clean cancel.
* fix(mcp): harden static-server health loop (review findings)
A max-effort review found 13 concurrency/correctness defects, all from the loop
mutating shared StaticServerState without the interlocks the pool path carries.
Fix all 13:
- In-flight interlock: add StaticServerState.in_flight (parity with
PoolEntryState); _static_session_op increments/decrements around the static
call_tool/read_resource/get_prompt session ops; the ping skips and never
evicts a busy server, so a long tool call can't be torn down mid-flight.
- Dead-transport gating: the ping evicts + trips the breaker only on
_is_dead_transport(exc); an McpError, httpx.PoolTimeout, or plain ping timeout
is "slow, not dead" and only reschedules (matching the dispatch path).
- Session-identity: only evict the exact session that was pinged.
- asyncio.timeout (invariant-18) not wait_for for the ping; 5s->30s; the
timeout-scoped cancel is distinguished from an external shutdown cancel
(which still propagates) via .expired().
- Bounded reconnect: wrap _connect_one in asyncio.timeout so a server that
handshakes then stalls list_tools can't wedge the loop or hold the per-name
lock forever (connect internals untouched).
- Concurrent tick under asyncio.gather with a freshly-read clock for the sleep.
- Cross-path coordination: reconnect_sync/remove_server_sync take the per-name
lock across teardown+rebuild (calling _connect_one_locked directly);
_cb_auto_reconnect reuses a health-established session instead of racing a
redundant reconnect and no longer trips the breaker on lock contention.
- Backoff hygiene on recovery; skip '__' names; on health reconnect clear only
the open-circuit deadline (not the failure count) so a connect-ok/calls-fail
server still escalates to a trip.
Adds 14 tests and adjusts those that assumed the old behavior; suite 195->209.
* fix(mcp): unify static-server reconnect coordination (round-3 review)
A third review round + live testing found 8 issues on the health loop, five
sharing one root: reconnect logic was fragmented across five drivers, each
handling the lock / session-reuse / in_flight / config-recheck / breaker / clock
differently and incompletely. Introduce one primitive and route every
lazy/autonomous driver through it.
_ensure_static_connected(name, cfg) — the single lazy (re)connect path, all under
the per-name lock: config re-check (+ lock-identity re-check, closing the
remove->re-add race) so a removed server is never resurrected; reuse-if-live so a
queued/concurrent driver never tears down and rebuilds a live session (the
observed reconnect storm); in_flight guard so a reconnect can't tear down a
session with a call still in flight on the evicted stack; bounded connect; and the
circuit breaker owned in one place (clear the open-circuit deadline on success per
finding-13, record one failure on real connect failure). Returns the session on
success/reuse, None on a deliberate skip, raises on real failure. Routed through
it: the health loop, a dispatch's _cb_auto_reconnect, and _refresh_all; operator
reconnect_sync stays a deliberate force-rebuild.
Also: fresh-clock deadlines (the stale tick-start clock was landing deadlines in
the past and collapsing the backoff into an every-tick retry storm); loop-death
fix (the tick no longer re-raises a CancelledError found in the gather results —
per-server fallout, not shutdown; the loop returns only when Task.cancelling()
marks a genuine shutdown); dispatch breaker records no failure on a sync-boundary
reconnect timeout (lock contention is not a server failure; real outcomes recorded
once, inside the primitive). Cleanups: extract _teardown_static_session (was
copy-pasted 3x); share _capped_exponential between the breaker cooldown and the
reconnect backoff. Adds 17 tests; test_mcp_client 204->226.
* fix(mcp): coherent timeout hierarchy + round-4 review fixes
A fourth review round on the unified reconnect coordination found 5 correctness
regressions + 1 cleanup, five sharing one root: the inner reconnect attempt bound
(45s) was LONGER than every caller wait (dispatch 30s, remove 15s, reconnect 30s),
so a caller cancelling mid-attempt delivered a bare CancelledError that slipped
past the primitive's `except Exception`.
- Coherent timeout hierarchy: add _STATIC_RECONNECT_CALLER_TIMEOUT_S (> the inner
attempt bound) for the dispatch + operator waits, so the inner asyncio.timeout
always fires first — a clean TimeoutError the primitive converts, cleans up, and
records on the breaker — instead of a caller cancelling a live attempt. Fixes
[0] (half-discovered session left installed, served with a stale catalog) and
[1] (breaker never trips via dispatch).
- Primitive cancel-safe (belt-and-suspenders): its handler is now
`except BaseException`, so even a bare CancelledError drops the partial session
and records the failed attempt before re-raising.
- Operator waits: reconnect_sync / remove_server_sync default timeouts raised above
the reconnect bound; remove_server_sync now CANCELS the pending _remove on
timeout (so it can't later pop a re-added entry and corrupt state) and reports
failure instead of a false 'removed' ([2], [4]).
- in_flight defer is gated on defer_if_busy: autonomous callers (health loop,
_refresh_all) defer, but a DISPATCH reconnects rather than hard-fail a reachable
server with an in-flight sibling ([3]).
- Cleanup: extract _schedule_next_ping (was a copy-pasted triplet in 3 branches) [5].
Adds 6 tests; the lock-contention test now shadows the caller-timeout constant so
it runs in ~1s instead of the full wait.
* fix(mcp): address PR review feedback on static reconnect
- _ensure_static_connected: skip breaker record on CancelledError
(cancel proves nothing about the server; aligns docstring with impl)
- reconnect_sync: add asyncio.timeout wrapper so discovery-phase
stalls get a clean TimeoutError inside the lock
- reconnect_sync: change except Exception to except BaseException
so CancelledError from future.cancel() triggers catalog cleanup
- reconnect_sync: null state.session on failure so a tool-less
session isn't mistaken for a live one
- _static_reconnect_one: use fresh monotonic clock for backoff
gate instead of stale tick-start snapshot; remove dead now param
- _static_health_tick: correct docstring (0.5s clamp prevents
busy-spin, not 'no sleep through short backoff')
- Test: new test for reconnect_sync timeout + catalog cleanup
- Test: update cancelled-attempt test for new CancelledError semantics
- Test: narrow except BaseException to except Exception + type hints
- Test: fix typo 'Understone' -> 'Turnstone'
- Test: remove unused stale_now variable; update mock signatures
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Migration 054 added these columns but they were never populated (the sole
writer hardcodes None) and never read by any query, dashboard, or API.
Drop them so the schema matches reality.
Closes#769
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The docstring claimed the dispatch path plumbs a triggering ws / tool-call id
through a thin wrapper. It does not: _record_pending_consent_best_effort neither
receives nor forwards any id (the mcp dispatch layer never has one), the
proactive sweep has no dispatch, and nothing reads last_ws_id / last_tool_call_id
— they are unpopulated schema columns from migration 054. State that plainly so
the comment doesn't imply data is captured when it isn't (Copilot review nit).
Per-user OAuth (auth_type=oauth_user) token refresh is entirely lazy: a token
is refreshed only when a tool is dispatched or a session binds the acting user,
and a dead refresh token is discovered only when a dispatch fails. That assumes
a human is driving the session, which breaks for autonomous / scheduled work
acting on behalf of an absent user — the token may be expired (latency), in a
transient-failure cooldown (unavailable), or the grant may be dead with nobody
present to re-consent. The only periodic MCP-loop task, idle eviction, actively
tears OBO connections down; nothing keeps tokens warm.
Add a background token-freshness sweep that keeps every consented oauth_user
grant hot WITHOUT keeping connections warm and WITHOUT mutating consent state on
a timer.
Sweep (_user_token_sweep_loop, default 240s):
- Enumerates consented (user, server) grants from the token store and runs the
canonical refresh path for each. Strictly oauth_user-scoped: gates on
_oauth_user_server_names and drives off mcp_user_tokens rows, so a static /
no-auth server — which has neither — is structurally invisible (no DB scan, no
authorization-server round-trip, no MCP-server call). Never connects to the
MCP server; connections stay lazy.
- Observe-only: passes revoke_on_failure=False (new parameter on
get_user_access_token_classified) so a timer NEVER deletes a token, emits
token_revoked, or mutates the shared ambiguous-streak / cooldown. A dead grant
is only surfaced (proactive dashboard pending-consent badge); the
authoritative revoke stays on the lazy-dispatch path where a real user action
justifies it. Because the row survives, a spurious server-wide invalid_grant
(an AS maintenance window) self-heals — the badge is dropped on the tick the
grant works again.
- Keepalive: force-refreshes a grant whose refresh token has sat un-exercised
past user_token_refresh_keepalive_seconds (default 1800s) even while the
access token is still fresh, so a provider that ages out idle refresh tokens
can't expire one between a user's real sessions.
- Surfaces dead grants once per transition, pinning the pair only after a
durable badge write so a failed persist retries rather than being lost. First
sweep runs after a short startup grace so a restart surfaces a downed grant
within seconds, not a full cadence later.
Cadence <= 0 disables the sweep; a positive value is floored (30s) so a
misconfigured tiny cadence can't turn the loop into a busy-loop. Reuses the
per-key refresh lock, so a keepalive force cannot double-refresh against a
concurrent dispatch.
Storage: add list_mcp_user_token_reconcile_targets() returning
(user_id, server_name, COALESCE(last_refreshed, created)) — expiry-unfiltered,
no ciphertext projected — on the protocol, sqlite, and postgres backends.
Tests: the sweep's no-auth invisibility (zero DB / AS calls with no oauth_user
server), observe-only non-destruction (token kept and shared streak untouched on
a background permanent / ambiguous failure), keepalive gating, badge
persist-then-pin retry, self-heal on recovery, cadence clamp / disable, and the
storage enumerator.
- task_agent.json referenced a non-existent `skill(action='search', query=...)`
in two spots. The discovery tool is `skills` and the action is `find`
(`search` is an activation value, not an action), so the guidance would
mislead the model. Corrected both to `skills(action='find', query='...')`,
matching the tool's own error strings.
- _high_risk_skill_denied returned "" (allow) when get_storage() is None — an
asymmetry with the fail-closed lookup-exception path added earlier. A risk
gate that can't verify the tier must DENY, not wave the skill through, so
storage-unavailable (None) now denies too; both paths share one denial.
Two independent multi-agent reviews of the branch (high, then max effort) found
authority-confinement and robustness defects the per-step reviews could not
see. This commit addresses every confirmed finding. task_agent turned out to be
the surface that lagged its siblings on nearly every axis.
Risk gate (most severe):
- task_agent(skill=...) never enforced the high/critical-risk PRINCIPAL-load-
only gate that skills(load) / spawn_workstream / spawn_batch enforce, so a
model could route around it by delegating activation to a sub-agent. Enforce
it inline in _prepare_task on the row already fetched (no re-query, no drift
between get_skill_by_name and get_prompt_template_by_name).
- _high_risk_skill_denied now fails CLOSED on a storage fault: deny, never wave
the skill through. Denying (not returning "") also keeps spawn_batch's per-row
partial-success intact under a transient blip.
- (first round) extracted _high_risk_skill_denied onto spawn_workstream /
spawn_batch, closing the coordinator-side bypass.
Persona confinement (Principle 7 attenuation on the task_agent edge):
- A restrictive persona now attenuates the sub-agent's TOOLS, not just its
identity text — the tool lever is frozen into the item and filtered before
_run_agent.
- Honor ALL FOUR persona levers on the sub-agent, not two: a child persona's
mcp-off and memory-off levers now drop MCP tools (mcp__* + read_resource /
use_prompt) and the memory tool, matching a main session under the persona.
- Cap the sub-agent by the PARENT session's own persona grant too, so a
restricted principal cannot escalate authority by spawning.
- Add persona to the task_agent judge/audit func_args projection (policy +
audit parity with spawn).
- Persona-resolution failures defer to a clean tool error (try/except mirroring
_validate_child_persona) instead of an opaque "internal error".
Substitution / capability:
- substitute_args=False for capability contexts (defaults, task_agent) so a
literal $ARGUMENTS / $N in a body is preserved, not blanked; env vars still
resolve. The literal-$ARGUMENTS scan is deferred behind that guard (skipped on
every capability render).
- Drop the CLAUDE_SKILL_DIR alias (canonical TURNSTONE_SKILL_DIR only). That
name also lives in bash, where turnstone-as-a-node-inside-Claude-Code must not
shadow the host's value; claiming it in the prompt but deferring in bash
diverged the two surfaces (a review finding). turnstone now claims it in
neither surface. The CLAUDE_SESSION_ID / CLAUDE_EFFORT prompt aliases stay
(pure prompt values, no bash-namespace collision).
Skills-as-context:
- DEFAULT (always-on) skills stay in the identity system message — the standing
baseline, never a mid-session cache-bust; only a NAMED applied skill moves to
the user-role capability message. This shrinks the pending model-adherence
eval surface to the named-skill move alone.
Cleanups: consolidate a duplicated rationale comment; correct the now-stale
"task agents are not persona-filtered" note.
PRE-MERGE GATE unchanged: the §7 Q1 model-adherence eval (named-skill move,
this branch vs main) is not runnable in-tree and must clear before merge.
A skill can auto-fire tools (auto_approve + allowed_tools) once loaded, so
letting the model activate a risky skill through skills(action='load') is an
injection-steerable lane that widens authority behind a rubber-stampable
approval. Deny it: high/critical-risk skills are now PRINCIPAL-load-only --
the model gets a clean error pointing the user at /skill, and the operator
loads such skills explicitly (handle_command /skill and cli --skill call
set_skill directly and bypass this gate).
The scanner-computed risk_level is the gate signal -- it already escalates for
the auto_approve + allowed_tools authority the create path warns about -- so no
new column or migration is needed. The check can only DENY, never widen, so it
is safe by construction (HYPOTHESIS.md Principle 7 / design section 5.5).
Remaining step-5 follow-ups, out of scope here: persisting the literal
disable-model-invocation frontmatter field for arbitrary author-marked skills
(needs a column) and deprecating the vestigial variables mechanism.
Step 3 of the skill/persona split: an applied skill (including default skills)
is CAPABILITY context, so its body no longer sits in the identity system
message. It rides its own message (user role) after the identity block, with a
short intro naming the active skill. The <available-skills> discovery catalog
stays in the system message.
Two consequences:
- The cached identity prefix (persona BASE + ENV + POLICIES + catalogs) stays
stable across skills(load): loading/clearing a skill changes only the
trailing capability message, not the identity block.
- The task_agent base (_agent_system_messages) is snapshotted BEFORE the skill
block, so a parent's applied skill no longer leaks into the sub-agent prefix
(the sub-agent supplies its own persona identity and skill via _exec_task).
PRE-MERGE GATE: the design gates this on a model-adherence eval (this branch vs
main) verifying the model follows a skill as well from a context message as it
did from the system message (design section 7 Q1; ASSUMED-neutral, UNVERIFIED).
That eval is not runnable in-tree and MUST clear before this branch merges.
Mechanical structure is pinned by TestSkillContextPlacement.
Deferred follow-up: sub-agent (task_agent) skill-resource materialization, so
${TURNSTONE_SKILL_DIR} stays literal on that path (unchanged since step 1).
Test helpers (_sys_content) now read the full prompt prefix (identity + skill
context) so placement-agnostic assertions keep working.
Before, a task_agent's system identity WAS its skill (skill body concatenated
into the sub-agent's system message), and #683 deliberately gave task_agent no
persona. Now that personas are first-class on every creation/spawn path, make
task_agent consistent: identity comes from a persona, the skill is capability.
- task_agent gains persona= (validated at prep against the interactive kind,
the general-purpose personas a worker can adopt). The resolved base prompt
is frozen into the approval item; _exec_task never re-reads storage.
- Default identity (no persona=) stays _TASK_DEFAULT_IDENTITY. The one-shot,
tool-over-narration operating guidance always layers on top.
- skill= is now CAPABILITY: rendered through the shared pipeline (step 1) and
delivered as a distinct user-role context turn ahead of the task, never
fused into the identity. Consecutive user turns coalesce at the provider
boundary (Anthropic _merge_consecutive), so this is wire-safe.
Updates the task_agent tool schema (persona param; skill reframed as
capability) and flips the persona guard test (task_agent HAS a persona param
now). Sub-agent skill-resource materialization and the interactive
skills->context move remain follow-ups.
Skill-body placeholder substitution diverged by invocation context:
interactive load, default skills, and spawn-child ran the full
render + spec-substitute, while task_agent (_exec_task) ran
_render_template only -- so $ARGUMENTS and ${...} env placeholders
rendered literally on that one path.
Introduce _render_skill_body as the single render+substitute path and
route interactive load, defaults, and task_agent through it, so a skill
reading ${TURNSTONE_EFFORT} or $ARGUMENTS resolves identically wherever
it runs. A sub-agent has no invocation args, so bare $ARGUMENTS and the
positional $N / $ARGUMENTS[N] forms resolve to empty there -- matching
the defaults and spawn-child paths, not the old verbatim passthrough.
- Add ${TURNSTONE_*} as the canonical vendor-neutral spelling for the
env placeholders (SESSION_ID, EFFORT, SKILL_DIR); keep ${CLAUDE_*} as
a permanent back-compat alias so imported skills keep resolving.
- Bash env: export TURNSTONE_SKILL_DIR and SKILL_RESOURCES_DIR
unconditionally, but add CLAUDE_SKILL_DIR only when the host has not
set it, so turnstone does not shadow a real value when it runs as a
node inside Claude Code.
- Materialize skill resources before substituting the body, so
${TURNSTONE_SKILL_DIR} resolves to the concrete bundle path on the
interactive path.
Sub-agent resource materialization and moving identity to a first-class
persona are left to follow-ups; ${TURNSTONE_SKILL_DIR} stays literal on
the task_agent path for now (unchanged from prior behavior).
Address Copilot review: cancel() now signals the worker to stop (a
cancellation Event) and joins only when started, so a test that errors
before the approval registers can't leak the worker or resolve late into a
finished test. The worker reads _pending_approval via getattr so a UI
without it can't crash the thread into a silent death (leaving approve_tools
blocked the full timeout), and only resolves when it actually observed the
registration.
The UI-approval tests drive a blocking approve_tools() by firing
resolve_approval() from a fixed 0.05s threading.Timer. approve_tools does
_approval_event.clear() -> register _pending_approval -> wait(3600s); on a
slow/loaded runner the timer can fire the event's .set() BEFORE that .clear(),
so the wakeup is wiped and approve_tools blocks the full _APPROVAL_WAIT_TIMEOUT
(one hour) -- surfacing as an intermittent CI hang (observed on the 3.12 runner
~15% into the suite; fast runners win the race, so 3.11/3.13 pass the same
commit).
Replace the fixed-delay timer with resolve_when_pending() (tests/conftest.py):
it waits until the approval is actually registered -- which happens AFTER the
clear -- before resolving, so the set can never be lost. The helper mirrors
threading.Timer's start()/cancel() so the surrounding scaffolding is unchanged.
10 sites across 3 files; the verdict-delivery timer (bounded to its own 5s
budget, not a hang) is left as-is.
Validated: the 3 files pass 20/20 under single-CPU stress (taskset -c 0) with
no hang or thread leak.
- Docstrings/help said the treatment skill 'composes into the system
message'. This harness runs on both checkouts (system on main, a context
turn on the placement-refactor branch), so the wording now describes the
natural set_skill composition path without asserting a placement.
- Validate each skill-bearing case's 'skill' shape up front (driver +
CLI) so a malformed dataset fails with a clear error, not a mid-run
KeyError. Pinned by test_rejects_malformed_skill.
Add a two-arm skill-adherence mode to the eval measurement substrate that
measures whether a NAMED skill changes tool-use behaviour, so skill-in-system
(main) can be compared against skill-in-context.
- _run_single_test gains skill/skill_mode: skill_mode builds HeadlessSession
under natural composition (no system_prompt_override) and, for the treatment
arm, seeds the skill into the temp DB and activates it via the real
set_skill path so the skill body folds into the system message under test.
skill_mode defaults False, so the optimizer/measure paths are unchanged.
- Thread skill/skill_mode through _run_and_score_subprocess, _run_iteration
and _run_iteration_parallel (serial + parallel).
- run_skill_adherence: per case, run treatment (skill) vs control (no skill)
n_runs each, score against expected_actions, report per-case lift =
pass_rate(treatment) - pass_rate(control) and the mean lift. The control
isolates the skill's causal effect.
- turnstone-eval --skill-adherence <dataset>: loads a skill-scenario dataset
and prints a treatment/control/lift table.
- eval_skill_adherence.json: authored search-first / test-after-edit /
changelog-update scenarios, chosen so the base model does not do the action
by default.
- tests: plumbing proof (skill folds into system_messages for treatment,
absent for control) + lift-math aggregation.
The success path already extracts message_count/total_usage before the
break, so the session = None rebind was unused dead code (code-quality
review). Remove it; exception/timeout cleanup paths are unchanged.
turnstone-eval was misnamed: it was a prompt optimizer, not a measurement
harness. Split the 3252-line turnstone/eval.py into a strictly one-way
dependency (optimizer -> eval-core; core never imports the optimizer):
- turnstone/eval/core.py measurement substrate — everything up to and
including _run_iteration: provider detection, NullUI, HeadlessSession,
the test runner, score_run, aggregation, and neutral reporting.
- turnstone/eval/cli.py new measure-only `turnstone-eval` — the old
--no-optimize path promoted to the whole job (one _run_iteration call,
then print the summary table).
- turnstone/optimizer.py the UCB self-modify loop and its multi-agent
pipeline (analyst/optimizer/observer/diversifier/tool optimizer), now
`turnstone-optimizer`; imports from eval.core only.
- turnstone/eval/__init__.py re-exports the core public API for
back-compat (score_run, _match_action, _run_iteration, HeadlessSession).
_apply_tool_overrides lives in core (HeadlessSession needs it) rather than
alongside the other tree helpers, so the dependency stays one-way.
Breaking change: `turnstone-eval` now measures; use `turnstone-optimizer`
to optimize. Both code paths are behaviour-preserving — the moved function
bodies are byte-identical.
- Add the missing #admin-personas grid-template so the table lays out as
columns; it was the only admin table without its own template, so every
cell collapsed into one implicit stacked column.
- Base prompt: accurate per-mode placeholders (required on create; blank
keeps a built-in's shipped prompt on edit) plus a client-side required
check on create. The old "empty = the kind's stock base prompt" copy was
wrong now that create rejects a missing base_prompt.
- Set the default persona from the row ("set default", with a scope-default
badge) to match the models table; drop the shelf checkbox and its
create/edit/submit wiring.
ruff format --check flagged two lines: a long persona-picker Field
description in server_schemas.py and a watch-restore create() call in
server.py that fits on one line after the persona-kwargs merge.
Formatting only, no behavior change.
Built-in persona base prompts move from inline DB text / base.md into
prompts/personas/<slug>.md — code-owned, PR-reviewable, drift-proof.
base.md / base_coordinator.md become personas/engineer.md / orchestrator.md.
Prompt source is now explicit in storage instead of inferred in app logic:
a new base_prompt_file column plus CHECK (base_prompt IS NOT NULL OR
base_prompt_file IS NOT NULL) — two nullable columns, never both empty.
Resolution is a coalesce (base_prompt else load(base_prompt_file)), frozen
into the workstream stamp at creation. base_prompt_file marks a persona as
built-in (code-only, un-archivable); an operator override on a built-in is
allowed and wins over the file. "Inherit the kind default" is a
workstream-creation act (is_default), not a persona-row state.
Migration 063:
- seeds reference their file (base_prompt NULL); no runtime file reads —
the backfill's frozen prompt text is inlined as a point-in-time snapshot
so migration history stays self-contained and reproducible.
- every existing workstream is stamped by kind (creative -> writer, else
the kind default), set-based (INSERT..SELECT via temp tables) with the
persona column added after the bulk writes to shorten its lock window.
Storage guards (both backends): operators must supply base_prompt;
built-ins can't be archived or have base_prompt_file set via the API;
clearing an operator persona's only source is rejected.
Follow-ups reviewed alongside (#756): soft-set visibility docstring scoped
to per-process; _apply_persona_snapshot / _current_persona_snapshot own the
stamp round-trip; spawn approval-header args (skill/name/target_node)
flattened+capped like persona; server-side tool injection generalized to
replace-only (client-def gated, incl. the xAI include forwarding). Seed
copy revised (researcher soft; de-costumed prose; engineer de-biased).
New test_schema_parity asserts create_all matches the alembic head.
Closes#683 groundwork; ruff + strict mypy clean, full suite green.
The roster persona merge distinguishes key-absence (pre-persona node in a
rolling upgrade — preserve) from present-but-empty (authoritative
unstamped — accept), so a stale in-memory value can never mask the
snapshot on an immutable field. The node create route caps the persona
slug at 64 like the console proxy, keeping oversized values out of the
storage lookup and the reflected 400 text. The four persona admin
handlers drop their redundant function-local asyncio imports, and the
DELETE-route test moves its request out of the assert statement.
The scribe, researcher, and executive prompts drop the infrastructure-team
costume and the demo-theater close — those framings suit the stock BASE
modules (whose job is today's default engineer/orchestrator behavior) but
narrowed personas meant for general use: a scribe summarizing meeting
notes is not a teammate, and the executive's verdict language works
without corporate staging. The behavioral substance is unchanged —
fidelity discipline for scribe, evidence discipline for researcher,
interrogate/delegate/verdict for executive, with the approvals boundary
still stated plainly.
The writer prompt loses 'use the analysis channel', a harmony-format
holdover from the CLI's single-provider days — the reasoning cue is now
format-neutral. Migration docstrings ride along: the workstreams.persona
column is documented as a slug carrier, and downgrade() now states the
capability-widening consequence of stripping stamps.
Spec models now describe what the endpoints do: ListPersonasResponse
declares the tool_inventory the shelf depends on, both console create
models declare persona, CreatePersonaRequest declares org_id, and
UpdatePersonaRequest documents the null-vs-absent split (null clears
base_prompt/tool_allowlist, null on flags/kinds is ignored). Console
OpenAPI regenerated.
Protocol contracts match the implementations: update_persona's return
covers the no-op case, create_persona's raises-list is complete, and
both extended row-shape docstrings gain their tail columns plus the
append-only rule. The workstreams.persona comments say slug, not
display name.
Page corrections from the docs review: personas.md documents the
creative_mode-to-writer migration conversion, the mid-session /resume
MCP-lever behavior, visibility-based nudge gating, the soft-set
prompt-cache cost, and the executive tool list — and drops internal
jargon. The changelog entry moves under [Unreleased] with the house
breaking-marker style and the auto-conversion note. coordinator-skills
and the API tour stop using persona to mean framing; governance,
api-reference, sdk, console, tools, and memory pick up the new
permission family, endpoints, kwargs, picker, and lever caveats.
The rank guard now derives needs_approval through the real _prepare_tool
on a bash call under an allowlisting persona instead of scripting the
flag, and asserts the approval gate actually fires. The row-shape guard's
source-grep is replaced with behavioral collector tests driving both
ws_created lanes (poll-diff and SSE relay), plus a proxy-forward twin and
a saved-list value assertion that would catch positional column mix-ups.
Receiving-side stamping gets its first HTTP coverage: create with an
explicit persona under workstreams.create only (selection needs no
persona perm), kind-mismatch and unknown-name 400s, omitted-persona
default stamping, the 503 on a failed default lookup, and the clean-None
legacy lane. Resume adoption is pinned end to end: a corrupt target stamp
leaves the session fully intact, an MCP-on stamp is refused when the
client was persona-gated at construction, and an MCP-off stamp drops the
live surface (listeners deregistered, toolsets reset). Soft-set
tool_search expansion recomposes the prompt exactly once; legacy sessions
never recompose.
Compaction legs run real flows now: spill plus the recall-pointer variant
under memory-off with recall visible vs hidden, and a full stamp
surviving compaction-then-resume. Migration 063 gains the downgrade
config-cleanup case (stamps removed, creative_mode preserved) and the
conversion idempotency case (already-stamped creative rows don't crash
the upgrade).
RBAC coverage goes cross-perm: read-only and write-only principals hit
every verb (a wrong-perm-name regression in any handler is now visible),
archive and default-flip succeed through PATCH, persona.* strings
round-trip the role editors and the overrides overlay, and the production
route table is asserted directly (no DELETE registered). Endpoint/storage
fixtures move off migration-seed names; storage hardening tests cover the
size caps, corrupt-row reads, the TypeError-to-ValueError ordering, the
duplicate-name race mapping, and the single-default backstop. Shell
asserts pin the new picker surfaces and drop the last persona-as-kind
wording.
Provider search gating (replace-only): native web search now stands in for
a client web_search def that survived the persona visibility filter — on
both OpenAI surfaces and both injection lanes (web_search_options, the
server_side_tools loop, and _convert_tools' capability lane). A scribe or
any envelope hiding web_search stays search-free on search-capable models;
coordinators and tool-less utility calls stop receiving search too.
Resume stamp discipline: resume() loads config and parses the target's
stamp BEFORE touching session identity/history, so a corrupt stamp raises
with the session intact instead of half-adopting and then 'repairing' the
target's stamp on the next config save. The MCP lever now follows the
stamp on mid-session adoption: an MCP-off stamp drops the live surface in
place (listeners deregistered, toolsets reset); adopting an MCP-on stamp
into a session whose persona gated the client off is refused loudly (the
surface cannot be rebuilt post-construction). The REPL /resume handler
reports these errors instead of crashing the CLI.
Fail-closed default lane: a FAILED default-persona lookup at create is a
503 (routes) / clear exit (CLI) instead of silently degrading to the
unstamped stock envelope; a clean 'no default configured' still creates
legacy. resolve_persona_for_kind reports storage-unavailable distinctly
from unknown-persona.
Soft-set governance: tool_search expansion under a persona visibility set
recomposes the system prompt so tool-gated policy segments land with the
tool they gate. MCP resource/prompt catalogs gate on read_resource /
use_prompt visibility. Spawn judge/audit projections carry persona (the
human approval header already did). Active-list rows carry persona like
their project_id twin.
RBAC catalogs: persona.{create,read,write} join _VALID_PERMISSIONS and
the roles-editor sections, making the documented grant-outward path real.
Storage hardening: default-persona invariants move to a shared _utils
helper (validate + demote) with a pg advisory xact lock serializing
promotions and a post-promote single-default assertion; create maps the
unique-name race to the same ValueError as the pre-check; reads validate
JSON shape loudly (naming the persona); serialize enforces size caps;
field validation runs before invariant checks so malformed input is a 400,
never a TypeError-500. org_id guards explicit null and caps at 64.
Also: base_override='' means 'no override' at the compose boundary;
persona tag flattened/capped before the spawn approval header; /creative
redirect resolves the writer persona before advertising it; memory-nudge
gating unified through _nudges_enabled.
Provider/row-shape tests updated to the new contracts (the old ones
pinned the injection hole and the pre-persona row shape).
Review pass over the branch surfaced real defects, all fixed here with
regression guards:
- Fork-resume (resume_ws) adopts the SOURCE workstream's stamp,
resolved pre-construction so all four levers (including the
construction-time MCP gate) bind the fork; a corrupt source stamp is
a loud 400, an unstamped legacy source forks unstamped — never the
kind default. Watch-restore and CLI --resume thread the stamp the
same way, closing an MCP leak where a restored MCP-off workstream
re-merged the catalog.
- SessionManager.open parses the stamp inside the install guard so a
corrupt stamp releases the reserved slot; a retry reproduces the
loud error instead of 'already tracked'.
- Mid-session resume() adopting a stamp rebuilds the tool_search
pathway to match (hard set drops it, soft set force-constructs it);
soft persona sets survive the global tool-search setting being off.
- Memory nudges gate on actual memory-tool VISIBILITY, not just the
memory lever, so an allowlist that hides the tool also silences the
nudges that point at it; post-compaction resume gets a no-recall
nudge variant when the pointer would dangle.
- Console PATCH: explicit null flags from UpdatePersonaRequest no
longer archive the persona or flip levers on a rename; multi-kind
personas survive a shelf edit; admin list ships the per-kind
tool_inventory so the shelf checklist tracks the server inventory
instead of a hardcoded JS list; admin CRUD moved off the event loop.
- Migration 063 converts legacy creative_mode workstreams to the full
writer stamp (downgrade removes all persona keys).
- REPL: /new passes the persona; /workstreams unpacks the widened row.
- Shared resolve_persona_for_kind is the single eligibility rule for
the HTTP handler, CLI, and spawn precheck; spawn_batch memoizes the
persona lookup; ToolSearchManager.is_expanded gives the visibility
tail an O(1) probe.
docs/personas.md covers the four levers, the resolve-once/stamp-forever
snapshot semantics, the seed matrix, per-surface selection, authoring
rules, and RBAC; architecture.md's config-persistence paragraph swaps
the removed creative_mode for the persona stamp.
The 15 guards from the design brief: the approval path is untouched
under any persona (rank guard); empty-toolset personas compose no tools
block and put zero definitions on the wire; the tool_search escape hatch
is soft when included (discovered tools union with the allowlist) and
hard when omitted (pathway disabled, including native defer_loading);
memory-off suppresses recall injection, the memory tool, and
memory-directed nudges while behavioural nudges and task-agent tools
survive; MCP-off is session-wide and refresh-proof; spawn validates
persona at prep time and never inherits the parent's; task_agent's
schema stays persona-free; the stamp is immutable, survives
SessionManager.open threading, and corrupt stamps fail construction
loudly; mandatory prompt policies compose under every persona; CLI
resolution (extracted to resolve_cli_persona_kwargs for testability)
loads seeds, exits clearly on unknown names, and adopts the resume
target's stamp; persona edits/archives never touch stamped workstreams;
and the row-shape contract twins carry the persona field.
RBAC endpoint coverage: admin CRUD 403s without persona.* and succeeds
with it; the picker feed needs no persona permission and hides archived
personas; invariant violations surface as 400s; DELETE is 405.
Creation surfaces: the console launcher, the server webui new-workstream
dialog, and the dashboard composer all gain a Persona select fed by the
shared personas.js data layer (module cache + fingerprint + never-reject
fetch + window.TurnstonePersonas bridge, cloned from projects.js), kind-
filtered with the kind default preselected so a zero-touch launch is
byte-identical to today.
Authoring: a Personas tab in the console Manage surface (Governance
group, persona.read-gated) with a Service Hatch shelf exposing exactly
the four levers — base prompt, tool-visibility checklist (kind inventory
+ free-text row for MCP/dynamic names; tool_search membership decides
soft vs hard), MCP and memory toggles — plus kinds, default flip, and
archive. No delete action anywhere (archive-only lifecycle).
The workstream wears it: SavedColumns.persona() on both saved tables,
hover/aria labels on the rail rows (raw slug fallback keeps archived
personas labelling their workstreams), and the full row-emitter sweep —
storage projections (list_workstreams tail, get_workstreams_batch,
list_workstreams_with_history) on both backends, the server dict
builders and ws_created events, both _coordinator_rows lanes, the
collector delta + pseudo-node paths, and the console cluster-create
proxy that rebuilds its body.
Naming reclamation: the launcher kind-toggle ids/classes that squatted
on 'persona' (launcher-personas, persona-coordinator/-interactive,
.persona-btn/.persona-led/.persona-tag) are renamed to kind-* before
'persona' becomes user-facing vocabulary, along with the prompts-module
and test wording that used persona to mean kind.
The persona resolved at creation is snapshotted into workstream_config
(five keys, all-or-none) and applied ONLY from the stamp — the personas
table is never read post-create, so edits/archives never touch existing
workstreams, and a corrupt stamp fails construction loudly instead of
silently reverting to a default envelope. Legacy pre-063 workstreams
carry no keys and keep today's behavior byte-for-byte.
The four levers (turnstone/core/personas.py holds the codec):
1. Base override — compose_system_message(base_override=...) replaces
exactly the BASE module; ENV/CONTEXT/TOOLS/POLICIES keep composing so
mandatory prompt policies ride on top of every persona. This also
closes the old /creative hole where the fork bypassed composition
(no CONTEXT, no DB policies).
2. Tool visibility — the allowlist intersects both the composition name
set (TOOLS block self-suppresses, tool-gated policies drop, the
memory advisory drops) and the END of _get_active_tools so the wire
never advertises hidden tools. tool_search in the set = soft
(discovered tools union with the allowlist via the session's
expanded-names set); absent = hard (the whole pathway is disabled,
covering provider-native defer_loading, which has no synthetic name
to filter). Persona sets force client-side tool search.
3. MCP gate (session-wide) — an MCP-off persona drops the client
reference at construction: no merge into _tools OR _task_tools, no
listeners, refresh callbacks inert, resource/prompt catalogs gone.
4. Memory (own hands only) — no recall injection, memory-directed
nudges suppressed (MEMORY_NUDGE_TYPES; behavioural nudges keep
firing), memory tool hidden. _task_tools is NOT filtered; compaction
spill/markers are never persona-gated, and the post-compaction recall
pointer is emitted only when the recall tool is actually visible.
Threading: the create handler resolves once (explicit name -> 400 on
unknown/disabled/kind-mismatch; empty -> the kind's default; pre-seed DB
-> unstamped legacy) and stamps via constructor kwargs + config keys +
the workstreams.persona column; SessionManager.open threads the stamp
pre-construction exactly like the saved model alias. Non-fork resume
adopts the target's stamp so _save_config can't clobber it. spawn /
spawn_batch gain a persona arg with prep-time validation (children are
interactive-kind; omitted = kind default, never the parent's). Python +
TS SDKs, OpenAPI specs, the picker feed GET /v1/api/personas (authed, no
perm), and console admin CRUD /api/admin/personas (persona.* perms,
archive-only — no DELETE) round out the surface.
BREAKING: /creative is removed (the REPL command now points at the
writer persona); turnstone --persona <name> is the replacement. Also
fixes the CLI session factory, which TypeErrored on the project_id
kwarg the shared InteractiveAdapter passes unconditionally.
Adds the personas template shelf (#683): migration 063 creates the
personas table (tri-state tool_allowlist, per-kind is_default, archive
via enabled=0 — no hard delete) plus the workstreams.persona display
column, seeds the six launch personas (engineer/orchestrator as
zero-touch per-kind defaults; scribe/researcher/writer/executive as
curated envelopes), and grants persona.{create,read,write} to
builtin-admin following the 062 pattern.
Storage: list/get/get_by_name/get_default/create/update on both
backends, with the default-persona invariants (exactly one per kind,
single-kind, enabled, not archivable, demote-on-flip) enforced in the
storage layer and the JSON serialization shared via _utils so the
backends cannot drift.
context_window=0 is the documented auto-detect sentinel -- "inherit the
CLI-detected window." The DB loader applies it (row.get(k, 0) or
context_window); the config.toml loader did not (entry.get(k, default)
only substitutes a MISSING key), so an explicit context_window = 0
leaked a literal 0 downstream, zeroing every budget that reads
ModelConfig.context_window -- judge lowering, session compaction. The DB
loader's comment even claimed config.toml shared "the same fallback
chain," which was false.
Match the DB loader at the source. The judge-side _positive_window
coercion stays as defense-in-depth, but its comments no longer misframe
0 as garbage -- it's a valid sentinel, now normalized at load.
Two window-sourcing edge cases the first pass missed:
- config.toml models can carry context_window=0 (that load path lacks
the DB loader's 0-inherit normalization). The getattr guard caught a
missing attribute but not a present 0, which would zero every budget
and make honest_truncate drop everything. A shared _positive_window
helper now coerces any non-positive / non-int window to the next sane
candidate (the session window) then a floor, on every resolution path
in both judges.
- The output-guard judge's session-model fallback keyed off
provider.get_capabilities(), which reports 200k for local models --
the fictitious-window bug the alias path already fixes. It now takes
the session's real (config/registry-aware) window, like IntentJudge.
output_guard_judge shares _CHARS_PER_TOKEN from judge rather than
duplicating it, now that it imports the coercion helper anyway.
The output-guard LLM judge fed the model the full tool output with no
window awareness, so on a small-window local judge a large output
overflowed into an opaque provider error and fell silently to
heuristic-only — the opted-in LLM tier vanished without a trace.
Resolve the judge's real context window from the registry's per-model
config (the static capability table reports 200k for every local
model), and add an up-front oversize guard: when the assembled prompt
would exceed the window, skip the doomed call and record a labelled
llm_error the operator can see instead of a silent no-op. The
heuristic tier runs first, so its verdict still stands.
Also drop the fixed 500-char cap on the tool_args framing field: like
the output under review it now lowers whole, bounded only by the same
window backstop, never by a default clip of a normal argument.
The func_args projection in _evaluate_intent is the intent judge's
entire view of a pending call's arguments, yet it lowered only a
narrow field per tool: edit_file reached the judge as {path} with the
edits stripped, and skills mutations built their projection and never
assigned it, so the judge ruled on {}. A small local judge denied a
legitimate multi-edit edit_file at 95% confidence as "malformed,
missing old_string/new_string" on exactly this gap.
Project the full risk-relevant surface per tool — edits, file content,
timeouts, model overrides, skill risk fields, task status and
ordering, MCP resource URIs and prompt arguments — and add the
read_resource / use_prompt branches that previously fell through to an
empty {}.
Truncation is now a backstop, not a default. Arguments lower whole up
to the judge model's real context window, sourced from the registry's
per-model config rather than the static capability table (which
reports 200k for every local model and would over-budget a small local
judge into overflow). Only a genuine overflow truncates, with an
explicit dropped-character marker; the untruncated arguments always
remain in the trajectory. The verdict's persisted and streamed copy
carries a separate 16 KB backstop against pathological payloads.
A parametrized guard test asserts every gated tool projects a
non-empty argument view, so the silent-starvation failure mode fails
CI instead of shipping.
- typecheck: dropping _acting_user_id from the SessionUI protocol (it made
the attribute required, breaking NullUI's structural match and making
TerminalUI abstract). _emit_state now narrows to SessionUIBase before
assigning — the field belongs to the web-fanout UIs, not the protocol
contract that CLI/eval UIs also satisfy.
- test: two mock queue_message stubs (attachments-endpoint fake,
coordinator adapter double already fixed) needed the new
interjector_user_id kwarg; the endpoint fake was raising TypeError ->
queue_full. Added it and a negative test that a non-SessionUIBase UI is
skipped by _emit_state.
- review (Copilot): _load_persisted_senders now latches _db_senders_loaded
only if self._ws_id still matches the workstream it queried, so a
concurrent resume() can't mark the read done for a workstream whose
senders were never loaded.
- review (Copilot): corrected the acting-user-id comments in three places
— it carries the owner id even single-user (the gate no-ops because it
equals the viewer); it is empty only on unauthenticated lanes / before
first state emit.
Note: the storage-protocol '...' stub flagged by the code-quality bot is
the file's universal convention (262 stubs, zero NotImplementedError);
left as-is for consistency.
Coordinator workstreams will hit the same cross-user issue once MCP is
enabled there, so wire the same protection now.
- CoordinatorAdapter.send takes acting_user_id: binds it on a fresh turn
(so MCP creds + the acting-user signal are correct) and passes it to
queue_message on the interject path (CrossUserInterjectionError block).
The create-time initial dispatch passes the creator's id.
- ConsoleCoordinatorUI.on_state_change includes acting_user_id (mirrors
WebUI); the mid-turn-connect replay already carries it via the shared
make_events_handler. The coordinator's user-facing /send route already
reuses make_send_handler, so it inherited the interjector guard + 409.
- coordinator.js gains the same gate as the interactive pane: tracks the
acting user from state_change, blocks send when busy AND acting !=
viewer, and handles the 409 cleanly.
Drive-by hygiene (requested): the _verdictSig join used a raw U+001F
byte embedded in the source; replaced with String.fromCharCode(0x1f) —
identical runtime, ASCII-clean source (no invisible control char in the
file).
The UX complement to the server-side cross-user interjection block: on a
shared workstream, while another participant's turn is in flight, this
viewer's send button is disabled so they don't click into a 409 (and
can't drive tools under the initiator's credentials).
Backend signal (the linchpin — the acting user was tracked but never
surfaced to clients):
- ChatSession._emit_state pushes the acting user (turn initiator, owner
fallback) onto its UI (_acting_user_id on SessionUIBase).
- server.WebUI.on_state_change includes acting_user_id in the broadcast
state_change event; the mid-turn-connect replay (session_routes) adds
it too, so a client joining mid-turn learns who holds it. Id only (a
uuid the client compares) — no name, no storage lookup on the hot path.
Frontend:
- auth.js retains the opaque user_id from /whoami (ts.user_id) — kept
separate from the display username, used only for id comparison.
- composer.js gains an independent hard-block axis (setSendBlocked /
_reconcileDisabled) so send can be disabled even in queueWhileBusy mode.
- interactive.js tracks the acting user from state_change, blocks send
when busy AND acting_user_id !== the viewer's own id, and handles the
409 as a clean message (reactive fallback for the click-beats-event
race) instead of a generic connection error.
Degrades gracefully: single-user workstreams (acting user == viewer) and
older backends (no acting_user_id) never engage the gate.
A mid-turn interjection folds into the current turn under the
initiator's identity — bind_acting_user deliberately does not rebind
mid-turn — so on a shared workstream a second participant's queued text
would run any tools it triggers under the initiator's MCP (oauth_user)
credentials (confused deputy) and be stamped with the initiator's sender
label (misattribution). Rather than fold it in, reject: queue_message
now takes the authenticated interjector_user_id and raises
CrossUserInterjectionError when it differs from the current acting user.
The send route surfaces it as 409 cross_user_interjection. Only an
authenticated non-acting participant is blocked — self-interjection,
single-user workstreams, and unauthenticated internal lanes (empty id,
e.g. the coordinator adapter) are unaffected.
Cloud multi-agent review of the three follow-up commits surfaced 15
verified defects; this addresses them.
Security / correctness:
- output_guard was blind to the new sender-label trust marker: add
fence.SENDER_LABEL_TAG to the forgery/leak detector and thread a
second trusted nonce (trusted_sender_label_nonce) through
evaluate_output/_check_marker_forgery so a forged or leaked
sender-label block in tool output is flagged like an operator marker.
- Attachment-derived text (PDF extraction, audio transcript, perception
output) bypassed sender-label neutralization because it materializes
after _inject_sender_labels runs; neutralize it at each fallback site.
- _recompute_shared_state now runs on every compose (moved out of the
non-creative branch) so a creative-mode resume can't leave shared
state stale.
- /new and rewind/retry now reset shared state (were leaking the prior
conversation's participant set / keeping a workstream latched 'shared'
after its only second-participant evidence was deleted).
- Non-fork resume and /new remint both trust nonces; carrying a nonce
across a workstream switch would let a token leaked in one forge a
marker in another.
- _senders_dirty is only cleared once the persisted-sender read has
actually landed, so a transient storage error retries within the turn.
- ws_id snapshot guard in _recompute_shared_state discards a result if
resume() swapped workstreams mid-scan (MCP-callback race).
- recall/history search is scoped to the acting sender's visibility;
the shared-workstream declaration now names that exception so the
model doesn't read a filtered 'no results' as 'no record exists'.
Cleanup:
- fork skips the redundant persisted-sender read (its rows were just
bulk-written); _maybe_note_new_participant goes through the single
recompute entrypoint; senders_from_user_meta reuses _source_meta_from_json.
- fence.wrap docstring names the sender-label caller as a third
untrusted-host boundary.
Tests: end-to-end compaction-narrowed resume recovery, hostile
display-name fence break-out, sender-label output-guard leak/forgery,
and the two-nonce independence.
- q-2: document the deliberate username-first display-name precedence in
_resolve_display_name (diverges from auth.py's display_name-first
because sender labels must match the owner-banner identity kind).
- q-3: drop change-lineage comments referencing the separate acting-user
credential fix (tombstone noise once merged).
- q-4: tighten the plain-text attachment assertion from a tolerant
subset check to exact shape + _sender value now that the stamp is
deterministic.
- q-5: drop the contributor-local bare 'etc/' from .gitignore.
sec-1: the [message from <sender>] label was plain text, so a participant
could type a look-alike in their own message and impersonate another
sender to the model. Labels are now wrapped in a nonce-delimited
[start sender-label_<nonce>] ... [end sender-label_<nonce>] fence (new
fence.SENDER_LABEL_TAG, distinct value from the operator nonce) whose
token lives in the cached system prefix; participant content is
neutralized so typed look-alikes are defanged. A new
build_shared_workstream_declaration pins the token as the sole authentic
label.
sec-2 + q-1: the CONTEXT banner no longer embeds behavioral prose. It
carries a terse owner line + shared flag; the attribution rules, the
authenticity declaration, and the (now narrowed) tool-credential claim
move into the shared-workstream declaration. The credential claim is
corrected: per-participant credentials apply to MCP (OAuth) tools only;
built-in tools and skills run under the server/owner identity.
perf-3: _inject_sender_labels resolves each distinct sender's display
name once per call instead of once per turn, capping blocking storage
lookups at one per sender on the uncached error path.
- _known_senders/_shared_workstream are now monotonic: union-only growth,
latched shared flag, seeded once per workstream from a full-history
distinct-sender read (new StorageBackend.list_message_senders) so
compaction narrowing the resumable slice can no longer forget
participants (duplicate join notes) or flip the banner back to
single-user framing (prompt-prefix cache churn).
- Recompute is memoized per turn (invalidated on stamped user-turn
append); system-prompt composition no longer pays an O(n) trajectory
scan on every recompose.
- resume() resets the state: the monotonic guarantees are per
workstream, not per session object.
- resume(fork=True) bulk-persist now carries the user-turn sender stamp
into the fork's meta column (was: _source_meta only, which dropped
attribution for every forked user turn on reopen).
* multiuser chat fixes for identity clarity and obo oauth token selection during tool calls
* added some missing context to the session so that the llm would know what session/project to reference in tool calls
* updated to address copilots issues and excluded a local config folder
* I think this resolves the cicd failures
---------
Co-authored-by: pow3rtool <root@pow3rtools>
- applyRosterSnapshot: null-prototype membership map (a ws id colliding
with an Object.prototype property name would read as always-seen and
dodge eviction) and a stable Object.keys snapshot for the eviction
walk — current-key deletion during for...in is spec-safe, but the
snapshot is self-evidently order-safe and skips inherited keys.
- _streamingRenderApply: drop the tautological typeof guards around the
post-render decorators — both are module-local declarations, and the
surrounding try/catch owns decoration fault tolerance.
Long sessions (5000+ messages, several compactions) degraded steadily
and could stop rendering entirely while the backend stayed healthy.
Four hard failure mechanisms, each sufficient on its own:
- Unguarded event pipeline: one throw escaping onmessage/handleEvent
(e.g. renderMarkdown stack overflow on a few KB of nested "> ")
stranded the streaming refs, so every later delta painted into the
poisoned segment. stream_end now resets segment refs BEFORE the
finalize render with a plain-text fallback (the coordinator pane's
existing pattern); onmessage guards both parse and dispatch;
renderMarkdown is depth-capped with throw-safe footnote-scope
accounting; the streaming buffer is marked rendered only on success.
- Rebuild-vs-live races: clear_ui/replay_truncated re-renders wiped
events painted in the snapshot->replaceChildren window (never
redelivered) and left deltas writing into detached nodes. Rebuilds
now quiesce the event stream behind a token-owned queue flushed
after the render; streaming refs reset on every rebuild path
including refetch FAILURE; a mid-stream replay_truncated defers its
re-sync to the idle edge instead of dropping the repair.
- Ignored recovery floor: the global stream now handles node_snapshot
and replay_truncated. Roster eviction (with a "Session ended" toast
for open panes) happens only from the stream-ordered snapshot; the
REST resync is merge-only and r.ok-gated so a mid-restart 503 body
cannot read as an authoritative empty roster.
- Unbounded growth: _agentCards released on rebuild — deliberately NOT
on transport-only reconnects, which must preserve the maps or the
next child event builds a duplicate card; orphan grace timers
cancelled on full reload/destroy; toast queue capped with duplicate
coalescing; diff previews capped at 400 rendered lines (the
spread-append could throw RangeError before the approval gate
painted) with the omission notice below the scroll box; raw results
clamped at 64KiB.
Per-event O(N) work removed from the hot paths: thinking-indicator
instance ref; near-bottom cached from a passive scroll listener and
re-checked at rAF pin time (a user scroll-up landing in the coalescing
window wins; ResizeObserver re-engages follow after layout changes);
rAF-coalesced outer and per-stream scroll pins; self-healing
call_id->row/stream lookup caches; verdict lookup scoped to the row's
batch; tracked retry holder; queue-controller Set replaces the
whole-transcript idle sweep; rail renders rAF-coalesced; coordinator
child_ws_state ticks routed to single-row updates (full render only on
terminal-boundary crossings) with observer unobserve on replace.
Also: the coordinator SSE-error 401 probe is un-deadened (raw fetch —
authFetch never resolves a 401 — with the body inspected so a
version_mismatch still takes auth.js's upgrade-reload path via the new
noteVersionMismatch export); the console cluster-SSE reconnect timer
is tracked across logout; the mermaid render chain is rejection-proof
per link and paints errors on the containers the failing link had
already claimed.
Measured with scripts/livepass.py --perf (n=3000 history + 20-turn
live storm): full replay 1060ms -> 238ms; re-render cycles 836-1071ms
-> ~94ms flat; chunk path now flat vs transcript size; worst longtask
1080ms -> ~500ms; agent-card retention across rebuilds 4 -> 0.
Known limit (needs a server-side event watermark on /history): a turn
completing inside the refetch window can paint twice after the quiesce
flush — rare, visible, and strictly better than the silent loss it
replaces.
New /perf/livepass.html mounts the real InteractivePane at production
scroll geometry and drives production-shaped SSE events through
handleEvent/replayHistory in real time (no virtual-time budget, no
forced reduced-motion — both corrupt the measurement), reporting:
replayHistory wall time at N messages, per-turn live-storm cost on top
of that transcript, tool_output_chunk throughput, busy/idle churn,
heap + node + agent-card counts across repeated replay cycles (the
detached-DOM leak probe), and longtask counts.
The --perf runner builds, serves, and launches headless Chrome with
--js-flags=--expose-gc and --enable-precise-memory-info so heap
numbers are real floors; the page POSTs its JSON report to
/perf/report. Reports carry a per-attempt run token the runner
validates, so a straggler POST from a killed prior attempt cannot be
misattributed to the next size, and the wait loop polls the Chrome
process so a sandbox startup failure bails to the --no-sandbox
fallback in seconds instead of burning the full timeout.
The workflow_dispatch input is an arbitrary PR number, and the job used
only headRefName to pick the checkout ref. For a fork PR that is a bare
branch name that can collide with a branch in this repo, so the job
(contents:write, ends in git push) would operate on that unrelated
branch. Resolve isCrossRepository alongside headRefName and fail loudly
unless the PR head lives in this repository.
The publish and docker workflows trigger on workflow_run of CI, which
fires for every CI completion — including CI runs for pull requests
from forks — and always executes with this repo's secrets, tokens, and
the pypi environment. The only gate was CI success, so fork-PR CI runs
spawned publish jobs in the upstream context; actions/checkout v7's
fork-checkout refusal was the only thing that stopped one on 2026-06-30.
A fork PR whose head is an upstream-tagged commit would have passed the
tag check and reached the upload with valid OIDC.
Both workflows now require the triggering CI run to be a push event,
from this repository, with head_branch starting with 'v' — CI's push
trigger only matches main/stable/* branches and v* tags, so that is
necessarily a tag run (verified: tag-push runs report the tag name as
head_branch). Checkouts no longer persist the token while the tree's
build backend executes, and publishes are no longer cancellable
mid-upload (a half-uploaded release cannot be re-run cleanly because
PyPI rejects duplicate files).
vendor-js hardening in the same pass: gate on the immutable PR author
instead of github.actor, require a same-repo head before pushing to the
PR branch with contents:write, and pass github.head_ref through env
instead of interpolating it into the script body.
Review finding on #751: the carry invariant omitted the system message
and tool definitions, which ride every request — at shipped defaults
reserve + 2 carries + margin lands exactly at the window, so any real
prompt overhead pushed the post-compaction send over it, and the
overflow backstop re-compacts WITHOUT the carries.
spare now subtracts system_tokens + tool_def_tokens (the same terms the
_estimated_prompt_tokens fallback counts), making
overhead + reserve + carries*budget + margin <= window hold by
construction. Invariant test pinned at shipped defaults with a 4k-token
synthetic prompt; a monotonicity test pins that the term is live; exact-
arithmetic tests isolate the overhead explicitly.
The definition review found the two control-relevant crossings paraphrased:
the model's wind-down spill (recorded on the cooperative advisory, then
handed to the summarizer with everything else) and the user's last message
(clipped to 400 chars in the continuation hint). Both now cross copied.
- carry_spill: when the model stopped because it was advised to wrap up,
its final turn's text is shell-concatenated onto the summary under
'## Wind-down (verbatim)', ahead of '## Continue'. The summarizer still
reads the spill; its paraphrase is no longer the only survivor.
- _carry_budget_chars(carries): ~25% of the window per carry, sized so ALL
concurrent carries fit the spare after the summary output reserve —
spill + hint fire together at the end-of-turn site, and independent
sizing stacked reserve + 2*(cw/4) + margin past the window at default
config. Floored at 2000 chars; oversize content keeps head + tail.
- _truncate_block's marker reports the original size ('truncated — N chars
total'), and a truncated carry adds one line telling the model the full
text remains in history and recall can retrieve it.
- Summary turns carry source="compaction" (in-memory swap and checkpoint
reconstruction); _find_turn_boundaries and _generate_title test the tag
instead of the label string, so a user who literally types
'[Conversation summary]' stays a real turn.
- The send-loop overflow backstop now passes my_generation, closing the
compact-and-swap race every other compaction site already guards.
Tests: tests/test_compaction_crossing.py (tags on both paths, literal-label
boundary, budget arithmetic incl. the double-carry invariant at shipped
defaults, verbatim/truncated carries, spill semantics, forwarding); existing
suites updated for the tagged label turns and the new kwargs.
After a compaction, storage keeps the full transcript and the in-context
summary is a cache over it — recall is the model's re-derivation path back
into the originals. Un-scoped, its results duplicated the live context.
- search_history gains exclude_ws_id/exclude_after: the excluded ws's rows
above the boundary (the live segment, already in context) are dropped in
SQL via one shared fragment; rows at or below it — the summarized-away
past — stay searchable. A never-compacted ws is excluded whole:
everything is live. Other workstreams untouched.
- New get_compaction_checkpoint(ws_id) reads the latest marker's persisted
watermark (distinct from get_compaction_watermark, which computes what a
NEW compaction would use); the meta decoder is single-sourced with the
resume slice (parse_checkpoint_watermark) so the two boundary consumers
cannot drift.
- _exec_recall reads the boundary fresh at execution (a compaction that ran
while the item was queued is respected) and labels own-conversation hits
'(earlier in this conversation, compacted)'. Storage errors degrade to
whole-ws exclusion — less information, never duplicates. Known limit
(documented): a forked session excludes only its own ws, so inherited
parent rows remain searchable — harmless duplication bounded by tenancy.
- NUDGE_COMPACTION_RESUME teaches the path: the summary is a digest, not
the record, and recall can search the compacted portion.
- /history deliberately unchanged: a human browsing history has no context
to duplicate.
Tests: tests/test_recall_compaction_scope.py — checkpoint reads (none /
marker / latest-wins / malformed-as-live), the exclusion matrix, the
composed tenancy+exclusion query with both filters dropping rows, exec
plumbing and labeling, the nudge line; cross-backend.
Renovate opens PRs as a bot actor, which claude-code-action's default
human-actor check rejects — Renovate's dependency-bump PRs were never
getting reviewed.
search_history / search_history_recent searched every workstream's rows
regardless of who asked. Pre-projects that matched the trusted-team
deployment shape; with private projects (062) it became a cross-tenant
read — the recall tool and /history returned private-project rows to
non-members.
Both methods take a keyword-only user_id (protocol, sqlite, postgresql)
scoped by one portable SQL predicate (HISTORY_VISIBILITY_SCOPE_SQL)
mirroring WorkstreamProjectVisibility: a row hides only when its
workstream links to an existing private project and the user is neither
the workstream creator, the project owner, nor a member. Applied in SQL
so limit/offset pagination stays honest; COALESCE guards the
NULL-creator row, which plain <> would leak.
The recall tool pins the scope identity at prepare time (the mcp_user_id
discipline) and fails loudly on an unpinned item; /history scopes to the
acting user; user_id=None (single-user CLI lanes) stays unscoped.
Tests: cross-backend visibility matrix, ws_visible parity pin,
marker-exclusion composition, LIKE-fallback path, prepare-pin plumbing.
The redundant function-local asyncio import in project_resources_endpoint
shadowed the module-level one. resolve_workstream_owner's docstring now
maps the failure modes precisely: a failed ROW lookup is fail-soft 404
(get_workstream_row degrades to None, pre-existing behaviour), while the
fail-closed 403 applies once a row is resolved and the project gate's
storage lookup fails — in-memory workstreams 403 on a gate blip,
not-loaded ones 404 at the row fetch first. Plus ruff-format on the
visibility test file (edited via script, so the local format hook never
saw it).
Max-effort review findings on the visibility feature, worst first:
Leaks — the filter was sound where it ran, but several surfaces never
carried project_id to gate on:
- cluster_snapshot served the raw collector state with no filter at all;
it now gets the same per-request tenancy treatment as its siblings
- console pseudo-node coordinator rows + emit_console_ws_created,
the interactive-create ws_created event, and the poll-diff ws_created
now carry project_id/user_id (parity with their filtered siblings —
a missing field failed open, and a missing user_id over-hid the
creator's own workstreams)
- the SSE snapshot's overview total/state histogram is re-derived from
the filtered rows instead of leaking pre-filter counts
Correctness:
- saved list pages with OFFSET until it fills its 50-row window instead
of filtering after the LIMIT (a caller's own rows at position 51+
used to vanish behind other tenants' private rows); scan capped at 20
pages, logged when hit
- an INHERITED project_id whose project was since deleted no longer
400s coordinator child spawns — the dangling link is dropped; explicit
unknown ids still 400, revoked membership still 403s
- the SSE filter keeps a per-connection unresolved map: a storage blip
suppresses a row without pinning it hidden until reconnect (re-judged
on later events, rate-limited); definitive verdicts settle as before
- bypass principals (service / admin.cluster.inspect) get payloads
untouched — no row drops, no overview rewrite
Consistency and robustness:
- dashboard + saved-list visibility checks moved off the event loop
(executor), matching every sibling site
- list_project_attachments chunks its IN() at 500 ids per statement
- ws_visible/ensure_project_attachable now share one _project_grants
predicate so the tenancy rule can't diverge
- resolve_workstream_owner's docstring states the deliberate
fail-closed trade for project-attached rows during DB outages
- the workstreams-for-project ordering test asserts strict order on a
forced timestamp instead of a vacuous set fallback
ws_visible only treats real strings as project links (a test double or
corrupted value means no-project, not private-and-denied), the mgr-path
project_id is coerced likewise, and the HTTP send path binds the acting
user via a getattr-guarded bind_acting_user call inside the fresh-turn
closure instead of a send() kwarg — per-kind session stubs with explicit
send signatures keep working. Row-shape contract tests (interactive +
coordinator twins) grow the intentional project_id key.
Dashboard saved-sessions lists now carry and render the workstream's
project: SavedWorkstreamInfo gains project_id (the saved projection was
extended in the visibility change), SavedColumns grows a PROJECT column
(name resolved through the shared projects data layer, searchable via
the filter haystack, re-rendered when the async project cache fills),
inserted on both the webui saved-workstreams and console saved-sessions
tables.
Manage → governance → Projects rows are now expandable (same
interaction contract as the Users tab's OIDC panel): a per-project
resources panel lists the project's workstreams (kind/state/updated),
referenced attachments (metadata + ws-scoped download link through the
console's node proxy), and the project-scoped memory count. Backed by
GET /v1/api/projects/{id}/resources (project.read + per-project ACL,
collection off the event loop) over two new storage queries —
list_workstreams_for_project (first consumer of idx_workstreams_project)
and list_project_attachments (conversation ref-list walk, metadata only,
first-referencing ws per blob, pruned blobs skipped).
Workstreams attached to a private project were listed and reachable for
every authenticated user — only the scope tier was checked. Add a
tenancy predicate (WorkstreamProjectVisibility: private → project
owner/members, the workstream's own creator, service scope, or
admin.cluster.inspect; public/dangling/no project → unchanged
trusted-team visibility; membership itself is the grant — deliberately
NOT gated on the project.read capability, which guards the management
API) and apply it at every surface:
- listings: saved sessions (project_id + owner tail-appended to
list_workstreams_with_history on both backends), active list, node
dashboard, console cluster list (pre-pagination via a collector
row_filter so totals stay honest), node detail
- console tier-1 SSE: per-connection snapshot filtering + a hidden-set
for sparse follow-up events; ws_created project lookups run on the
executor, membership changes take effect on reconnect
- row access: resolve_workstream_owner 403s private-project rows for
non-members, covering every interactive ws-scoped verb via
tenant_check (console coordinator lane stays on its privileged
admin.coordinator gate)
- create: ensure_project_attachable gates explicit and parent-inherited
project_id on both create validators (unknown project 400s instead of
minting a dangling link)
Per-user MCP credential resolution was bound once at session construction
to the persisted workstream owner, so on a shared workstream every sender
executed oauth_user tools under the creator's tokens (and saw the
creator's tool catalog). Bind the authenticated initiator of each turn
(send + retry paths) as the session's acting user: dispatch, catalog
merge, visibility gates, and consent flows now follow whoever is driving,
with the owner as fallback for CLI / eval / scheduled / internal turns.
Rebinding swaps the user-scoped tool/resource/prompt listeners (identity
is the (user_id, callback) pair), fire-and-forget primes the acting
user's pools, and rebuilds the merged tool list. Prepared tool items pin
the identity at prepare time so an item pending approval executes under
the user whose turn requested it, not whoever binds later. Queued
mid-turn interjections deliberately do not rebind (no mid-turn
credential switch).
Corrections: the middle-form re-separation names its true mechanism
(restarting specs or refusal-event predicates; within-run retries never
touch F), the standard-Borel aside admits belief-state coordinates, the
drift-slack display binds its variable, effect-record status gains a
`none` value (never launched) distinct from rolled_back and unknown,
and parsing is assigned to the inner readout R with the gate as pure
authorization.
Structure: the trusted principal as the provenance lattice's single
widening writer; two-rank control (authority vs plan) with a
rank-neutrality corollary; the narrow-only rule for learned checks;
pi's never-lower filter joins the deterministic core; gate TOCTOU and
cross-run serialization; a composition law for harness trees (four
correspondences) with delegation as monotone attenuation.
Appendix: new worked entries for resume (journal-before-dispatch),
parallel proposals (the batch gate), derived and durable state (the
provenance meet rule), and ambient authority (per-action capability).
Claims numbered C1-C8; two falsifiers added (certificate compression;
working-set probe anchored in streaming lower bounds).
Grounding: adds Ramadge-Wonham supervisory control, RL shielding, and
Dayan's successor representation; repairs the Positivity/Skolem gloss
and two citation characterizations. All 18 external citations verified
against their sources.
A max-effort review of the branch before pushing surfaced six defects, several
introduced by this branch's own commits. All fixed:
[0]+[3] oauth priming (refined). Fully non-destructive priming never cleared a
genuinely-revoked grant — the dead token stayed "consented", its tools never
entered the catalog, and (bug) the PERMANENT branch returned before arming the
cooldown, so every session re-hit the AS with a dead refresh token. Root cause:
invalid_grant (PERMANENT) is a RELIABLE dead-grant signal (RFC 6749 §5.2), so
deferring its revoke was net-harmful. Renamed the flag revoke_on_dead_grant ->
revoke_ambiguous_escalation: priming now revokes genuinely-dead grants (permanent
/ expired-no-refresh) so the catalog isn't stranded cold behind a phantom token,
and defers ONLY the sustained-UNCLASSIFIABLE (ambiguous) escalation to lazy
dispatch — the case the "don't revoke an unused server's grant on a
misclassification" concern actually applies to. The cooldown is armed before the
ambiguous path, so the deferred case can't hammer the AS either.
[1] server.py. _public_server_status (operator refresh/reconnect endpoints)
didn't forward the new scope, so after per-user scoping every warm oauth_user
server rendered disconnected/empty there. Now passes aggregate=True (operator /
approve-scoped cluster view, matching the admin console).
[5] _is_dead_transport. The widened httpx.TimeoutException swept in
httpx.PoolTimeout — pool saturation, NOT a dead connection — so transient load
would evict a healthy session and trip the shared breaker for all users.
Narrowed to Connect/Read/WriteTimeout (kept NetworkError, RemoteProtocolError).
[8] _is_dead_transport. The exact-message "session terminated" fallback still
fired on a healthy session-owning server's protocol error with that message. The
SDK-synthesized code 32600 is the only deterministic signal (the message is
application-controlled), so match the code ALONE and drop the message fallback.
[11] cleanup. The dead-transport except block was triplicated across
call_tool_sync / read_resource_sync / get_prompt_sync — the exact drift this
branch had to repair. Extracted _record_and_evict_on_dead_transport.
Tests updated/added: prime revokes-permanent / defers-ambiguous (drives the real
resolver both ways); PoolTimeout-is-not-dead; exact-"Session terminated"-message
stays alive; _public_server_status aggregate. 836 test_mcp_* green, ruff + mypy
clean.
Resolves the one regression the user-scoping in 0c28b0ce introduced: the admin
console reaches the read-scoped /mcp-status endpoint via the console proxy with
the ADMIN's forwarded identity, so per-user scoping made oauth_user servers show
as the admin's own (usually empty) pool instead of the cluster-health "in use by
anyone" aggregate.
Add an `aggregate` flag (default False) through get_all_server_status ->
get_server_status -> _oauth_user_server_status. When set, connected + a
representative catalog reflect ANY user's warm pool. internal_mcp_status gates it
on the admin.mcp permission: holders (who already see consent counts + server
config — the proxy forwards permissions via create_jwt, repopulated on validate)
get the aggregate; every other read-scoped caller stays strictly per-user, so the
cross-user catalog leak stays closed. Static-server status is unaffected.
Tests: manager-level aggregate-sees-any-user, and an endpoint-level gating test
asserting admin.mcp -> aggregate=True / read+approve-without-it -> aggregate=False.
Follow-up to f585c47b (review finding #4). _oauth_user_server_status derived
connected + tools/resources/prompts counts from warm[0] — an arbitrary user's
pool entry — and get_all_server_status surfaced that to every read-scoped
caller of /v1/api/_internal/mcp-status, ignoring who was asking. So user B saw
user A's oauth_user server as connected with A's catalog size, over the wire
(connected + the three counts are in _READ_STATUS_PUBLIC_KEYS; user_pools /
auth_type are stripped). Before f585c47b these servers were absent from the
read map entirely.
Thread user_id through get_all_server_status -> get_server_status ->
_oauth_user_server_status; the warm-pool filter now matches uid == user_id, so
connected + counts reflect ONLY the requester's own pool. internal_mcp_status
passes _auth_user_id(request); an empty/absent principal (user_id falsy) sees
oauth_user servers as not-connected. Static-server status is unaffected (the
new param defaults to None and is ignored for them).
Note: the admin console (admin.mcp) reaches this same read endpoint via the
console proxy, which forwards the ADMIN's identity — so an admin now sees an
oauth_user server scoped to their OWN pool (typically not-connected) rather
than the prior any-user aggregate. Server-global health (circuit_open / error /
consecutive_failures) is unchanged, and the consented-users-count is a separate
aggregate. Restoring an aggregate in-use pill for admins (without re-leaking
per-user catalogs) would need a privilege-aware aggregate mode + admin.js
change — deferred.
Tests: updated TestOAuthUserServerStatus to the scoped signature, added the
cross-user isolation regression (user B sees neither A's connected flag nor A's
catalog size) and a no-user-context case.
Follow-up to f585c47b (review finding #5/#6). f585c47b routed
_prime_user_pools through get_user_access_token_classified to refresh expired
tokens at session start (closing the chicken-and-egg where an expired token
stranded the pool). But that resolver also REVOKES a grant (delete_user_token
+ token_revoked audit) on a permanent-classified refresh failure — and priming
runs for EVERY consented server, so a single misclassified AS hiccup (e.g.
invalid_grant during a key-rotation window) could now delete a working grant
for a server the user isn't even using this session. The _prime_one comment
still claimed "priming can never revoke a live grant" — no longer true.
Add revoke_on_dead_grant: bool = True to get_user_access_token_classified. When
False, the four would-revoke sites return refresh_failed_transient with the
token left in place instead of deleting it. _prime_one passes False: priming
still refreshes+persists refreshable tokens (f585c47b's fix intact) but never
revokes — the authoritative revoke stays on the lazy-dispatch path, where the
user actually invokes the tool and a permanent failure means re-consent anyway.
Replaces the vacuous prime test (which fully stubbed the resolver, so its
"never revoke" assertion was meaningless) with a test that drives the REAL
resolver and pins both directions: same permanent failure, same code path,
revoke_on_dead_grant=False keeps the token / =True (lazy default) deletes it.
Follow-up to f585c47b. Three correctness gaps from a max-depth review of
that commit, all in the same dead-transport / session-corpse family it set
out to close.
1. read_resource_sync and get_prompt_sync were left on the old
BrokenPipe/ConnectionReset/EOF-only eviction guard, so a dead
streamable-http transport (McpError(CONNECTION_CLOSED), anyio
ClosedResourceError, server-restarted session) reused the corpse session
forever — the exact restart-hang call_tool_sync already fixes, just for
resources and prompts. Both now route through _is_dead_transport and
evict + trip the breaker like the tool-call path.
2. _is_dead_transport matched a bare "session terminated"/"session not
found" substring, so a healthy session-owning MCP server (game/shell)
rejecting a stale id with those words was misclassified as transport
death — evicting the live session and opening the SHARED per-server
breaker for every user after 3 such rejections. Now anchored on the
SDK's deterministic synthesized code (32600, pinned as a named constant)
with its exact message as a forward-compat fallback. The client never
receives "session not found" for a real dead transport (the SDK discards
the server's 404 body), so the tightening loses no coverage.
3. _is_dead_transport omitted httpx's read/write/close NetworkError leaves
and the whole TimeoutException family (Read/Write/Pool timeouts are NOT
builtin TimeoutError), so a stream that died on an idle read timeout —
the dominant idle-death mode — fell through to "other" and the corpse
was reused. Broadened to httpx.NetworkError | TimeoutException |
RemoteProtocolError (LocalProtocolError, our own bug, stays excluded).
Also: _oauth_user_server_status iterated _user_pool_entries without a
list() snapshot, so a concurrent pool insert/evict on the mcp-loop thread
could raise "dictionary changed size during iteration" and 500 the status
endpoint. Snapshot like the sibling get_all_server_status does.
Adds TestIsDeadTransport (direct classifier unit tests, incl. the
healthy-"session not found"-is-not-dead and httpx-coverage regressions) and
resource/prompt eviction tests. All 8 behavior-change tests fail on the
pre-fix source and pass with the fix.
_stop_retrying calls _is_ctx_overflow with no exception-class gate of its own,
so a retryable 429 whose token-quota text contains an overflow phrase (e.g.
"... maximum number of tokens allowed per minute ...") was treated as a
deterministic overflow and made non-retryable.
Gate _is_ctx_overflow on "not a known backend class": an overflow is never a
recognized error (it arrives as BadRequestError/InternalServerError, neither in
_BACKEND_KNOWN_EXC_NAMES), so excluding known classes can't suppress a real
overflow while keeping a 429 retryable across every caller (the retry gates,
send-loop recovery, chunker, task_agent loop, formatter). _format_backend_error
drops its now-redundant inline class check.
Addresses Copilot review feedback on #740.
A session created under the openai-compatible provider and resumed under the
anthropic-compatible provider (same vLLM model) failed with an opaque
InternalError instead of recovering. Root cause: vLLM returns a context-window
overflow as HTTP 400 BadRequestError on /v1/chat/completions but HTTP 500
InternalServerError on /v1/messages, and the rehydrated resume payload overflowed
the window. The 500 was retried four times then surfaced as a bare class name.
- Detect overflow by message text, not exception class (_is_ctx_overflow),
shared across the fatal-error formatter, both stream-retry gates, the send-loop
recovery, the chunker, and the task_agent loop. Overflow is non-retryable
(deterministic; no backoff). Phrasing is overflow-specific so a token-quota
rate-limit isn't misclassified.
- Proactive pre-send compaction (Layer A): when already over the hard ceiling,
compact once before the first stream so a resume that arrives over-window (or
follows a switch to a smaller-context model, with no prior compaction) doesn't
go out blind. Generation-guarded end to end so an orphaned or superseded send
can never swap the live generation's history.
- Binary-subdivision chunker: an over-window summary batch is split in half and
the partials merged (~log2(N) calls, not one per block); a lone over-window
block is truncated progressively down to a floor before bailing irreducible.
- Cooperative cancellation honored through compaction; send() consumes its own
generation's cancel signal on exit, so a stale cancel can't block a later
idle /compact and a live cancel is never disarmed.
- _format_backend_error surfaces "Context window exceeded ..." instead of an
opaque InternalServerError, and only for unrecognized classes.
- retry/rewind, the continuation hint, and title generation all exclude the
synthetic [Conversation summary] turn so they can't target the label.
- task_agent salvages a sub-agent's partial work on any terminal error (not only
overflow), re-raising only when there is nothing to salvage.
- Add a plain-terms gloss of the claim (shell/plant split up front)
- Add a 'Converged-upon' grounding subsection: independent corroboration
from capabilities, control theory, software architecture, and LM theory
- State provenance as a precondition of the reach-avoid certificate
(CaMeL control/data-flow separation), not just an entry point to police
- Drop redundant 'none' from the effect-record status enum; normalize
minor notation (A_bot, h->N)
- Fix stray backslash-escaped quotes that rendered literally
* fix(memory): atomic single-statement upsert for memory save/update
save_structured_memory used "try INSERT -> catch IntegrityError ->
SELECT + UPDATE". On PostgreSQL a model saving the same key twice in a
turn logged a uq_smem_name_scope violation on the failing INSERT, and the
pattern threw + caught an exception on every update.
Replace it with one statement: a new StorageBackend.upsert_structured_memory
on both backends emitting INSERT ... ON CONFLICT (name, scope, scope_id)
DO UPDATE ... RETURNING. It returns (row, was_update) -- the full saved
row and whether an existing row was updated -- like Django's
update_or_create; was_update is the supplied (fresh) memory_id differing
from the returned id. save_structured_memory is a thin wrapper over it.
description / mem_type of None mean "leave unset": the column default
applies on insert and the stored value is kept on conflict; an explicit
value (including "" / "general") overwrites -- so clearing a description or
setting type back to "general" now persists, where the prior
"if mem_type != 'general'" / "if description" semantics silently dropped it.
The memory tool and the memories HTTP endpoint pass None for omitted fields
and read effective type/scope from the returned row; the HTTP endpoint
returns that row directly (one query, no follow-up SELECT).
Removes the now-unused update_structured_memory primitive and its dead
STRUCTURED_MEMORY_MUTABLE constant. Adds cross-backend storage tests and a
session tool-path test (preserve-on-omit / overwrite-on-explicit), run on
PostgreSQL via --storage-backend -- the save-over-existing path was
previously SQLite-only.
* docs(memory): clarify upsert was_update precondition
Lead the upsert_structured_memory docstring with the behavioral contract
(callers MUST supply a fresh unique memory_id) rather than the internal
id-comparison mechanism, so a future caller can't reuse an existing id and
silently get was_update=False on a real update.
claude-code-review.yml granted pull-requests: read, so the Claude reviewer
ran green but its post step was permission-denied (permission_denials_count:
3) and posted no review on the PR. Bump to pull-requests: write so it can
post the review + inline comments.
claude.yml (the @claude responder) had the same read-only block and would
silently fail to post a reply; widen it to pull-requests + issues: write.
contents stays read -- no repo-push capability is granted. Both workflows
remain gated (the reviewer to same-repo PRs via head.repo.full_name ==
github.repository; the responder to @claude from OWNER/MEMBER/COLLABORATOR),
so write is scoped to already-trusted triggers.
Injected memories ride in the cached system block, so calling
_init_system_messages() on every memory save/update rebuilt the prompt
prefix and busted the provider prompt cache (a full system + history
re-write) -- for a memory the model already holds via the tool result.
memory(save) now only invalidates the per-turn search cache, so an
in-turn memory(search)/(list) still reflects the write; the new memory
folds into the prefix at the next natural recompose or the next session.
Also drop the redundant _init_system_messages() in the /reason handler:
reasoning effort rides in request kwargs (output_config / thinking), not
the composed prompt, so it recomposed to byte-identical output.
Add a chain-level test through the real _exec_memory -> no-recompose path
(asserts prefix unchanged, search cache invalidated, next recompose folds
the memory in). The prior memory tests either drove _init_system_messages
directly or patched it out, so this path was uncovered.
Address the Copilot review on #732 plus a task-agent sub-tool nesting
race surfaced alongside it.
Nesting (web UI):
- A sub-tool step whose task_agent row hasn't painted yet (the 4-wide
tool pool's ordering window) buffers and nests when the row lands,
instead of escaping to a top-level row that looks main-harness-issued.
- A row that never paints (id-correlation mismatch / aborted agent)
escapes its buffered steps back to a visible top-level paint after a
grace window, so steps are never buffered invisibly or leaked.
- The nested card survives the parent row's pending->resolved rebuild; a
call_id reused across turns builds a fresh card rather than stealing the
prior agent's steps.
- tool_info routes through the same nesting path (no duplicate top-level
row); a namespaced sub-tool result no longer grafts onto an unrelated
top-level row.
Denial reasons (backend):
- Preserve the specific denial reason a gate already stamped (operator
feedback, or the matched policy pattern; web and CLI contracts) instead
of clobbering it with a flat "Denied by user" -- in both the sub-agent
and the main tool loop.
Verified with the livepass task_agent harness (race + orphan-escape
scenarios, headless) and unit tests.
Final chunk of the task_agent modernization: rebuild a finished task
agent's card from /history (reload / reopen while the workstream is in
memory) and isolate each sub-agent's file-read tracking.
Recall: _project_agent_steps projects a sub-agent's trajectory into step
items (FIFO-per-call_id pairing via _iter_agent_tool_results, shared with
_cancel_ledger; output/arguments/count capped); _stash_agent_trajectory
keeps them on the UI in an LRU-bounded store; make_history_handler
attaches them as agent_steps to each task_agent tool_call, and
replayHistory/_replayAgentCard rebuild the collapsed card. In-memory only
(durable persistence deferred); a cold/evicted entry renders the flat
parent row ("not retained"), never a fabricated 0-step card.
Read isolation: _read_files (the blind-overwrite guard's memory) is now
per-sub-agent via the _active_read_files contextvar -- _exec_task copies
the parent's set on spawn and merges the agent's reads back on
completion, so a sibling in the 4-wide pool can't suppress another
agent's guard.
Also: _exec_task now self-reports the task_agent tool_result on every
path (the parent loop only reports error/denied results centrally) --
without it the live card never completed and a failed task recorded
is_error=False in the canonical trajectory. is_error flows from
_tool_error_flags to the recalled step; on_info suppression is per-thread
so a parallel sibling tool's progress isn't dropped.
Route a task agent's sub-tool events (tool_pending / approve_request,
tagged with parent_call_id) into a collapsible card under the task_agent
row, replacing the blue on_info turn-legs.
- conversation.js / interactive.js: buildAgentCardBody +
_routeAgentItems / _ensureAgentCard nest steps by parent_call_id.
Collapsed by default (a task agent can run 100+ steps and the parent
fans out many in parallel); the label carries the live count + state.
Auto-expand when a nested approval is pending so the blocking prompt
can't hide behind the toggle.
- session.py / session_ui_base.py: on_agent_step paints auto-tool step
rows; namespace child call_ids by parent so the 4-wide task pool can't
collide on local sequential ids (call_0); suppress sub-agent on_info on
the web pane (no call_id to nest by — the card carries steps + result).
- cli.py: on_agent_step prints a dim step leg (no card on the CLI, which
keeps its on_info).
- livepass.py: task-agent card harness driving the real InteractivePane.
Rebuild the task_agent sub-harness on the canonical Turn trajectory (build list[Turn], lower via dicts_from_turns at the wire boundary) instead of hand-rolled OpenAI dicts; the cancel-ledger helpers read Turns.
Tag each sub-tool's events with parent_call_id via a lock-guarded child registry stamped centrally in SessionUIBase._enqueue, so a later UI can nest a task agent's steps under its card. Getattr-guarded on the session side so CLI/eval/test UIs are unaffected.
Behaviour-preserving (same wire shape, same cancellation semantics); the parent tag is wire-invisible and unconsumed until the frontend card lands.
Compaction swapped a session's in-memory history for a summary but left the full
transcript in storage, so resume() reloaded all of it -- on a long session, or
one switched to a smaller-context model, the rehydrated context overflowed the
model window and deadlocked the first post-resume send.
Persist a `_source="compaction"` marker (summary + watermark) on compaction;
resume rehydrates [summary] + [rows after the watermark] instead of the full
transcript. Full history stays in storage for /history, export, and audit;
markers are filtered from display, search, and export, and rewind/retry
truncation is floored at the marker so the summary's backing is never deleted.
The watermark and search filters count real transcript rows only. No migration.
Address PR #730 review. _summary_input_budget_chars now caps the _MIN_SUMMARY_BUDGET_CHARS floor at the true input capacity (input_tokens), so output reserve + budgeted input + prompt always fit context_window; on a window too small to summarize it returns a sub-floor budget and _pack_blocks bails as irreducible instead of overflowing the summary call. After the half-window output-reserve bound this only affected sub-~2048-token windows, but it was a real edge.
Clarify the _CompactionIrreducibleError docstring: chunked compaction never drops or fabricates whole turns, but a single oversized block is still head/tail-truncated as summary input via _truncate_block.
Add test_budget_never_exceeds_true_input_capacity.
The compaction summary ran as a single model call sized by the per-message
token estimate, which disagreed with the head+tail-capped formatted text, so a
long history could overflow the summary call itself; the old prefix-fit also
silently dropped the most-recent messages.
Summarize the whole selection via _summarize_blocks: greedily pack the
formatted blocks into batches that each fit the summary call's own input budget
(_summary_input_budget_chars), summarize each, and recursively merge the
partials until they collapse to one. The common case (it all fits) stays a
single call. Bail to the existing False path when the input is irreducible
rather than fabricate a summary; a mid-chunk failure leaves messages untouched
(atomic swap only on full success).
Bound the summary output reserve to half the context window
(_summary_output_tokens), used by BOTH the input-budget sizing and the actual
call. compact_max_tokens defaults to the full window (32768); clamped only by
max_output_tokens it reserved the entire context for output, flooring the input
budget so compaction overflowed (or bailed as irreducible) at the
default/small-window config that needs it most. Large windows are unaffected
(compact_max_tokens stays binding).
Guard an empty summary (keep history instead of swapping in nothing and
reporting success). Fold tool-def tokens into the _last_usage-less estimate AND
the post-compaction usage anchor, so the compact-before-truncate budget doesn't
over-state free space by the tool-def count. Single-source the shared
compactor/merge prompt section (_COMPACT_OUTPUT_FORMAT) and the tool-def sizing
(_tool_def_chars/_tool_def_tokens). A just-resumed session (no _last_usage) now
counts tool-def tokens so it doesn't undercount and skip proactive compaction
until its first reply re-anchors the estimate.
Prepush review follow-ups: generation-guard the end-of-turn auto-compaction and
its resume turn so a force-cancel during the slow summary call can't compact or
persist under a new generation (matching the mid-turn and end-of-loop guards);
single-source the soft-threshold predicate (_over_soft) shared by the mid-turn
policy, _compaction_owed, and the end-of-turn check; add tests for the
pre-attempted-compaction guard and the recursion depth ceiling.
Unify truncation and compaction on one provider-anchored fullness measure
(_estimated_prompt_tokens), closing the 80-100% dead zone where tool output
was truncated but compaction never fired. Make compaction cooperative: advise
the model to wrap up and record its plan, compact if it continues, auto-resume
after a cooperative stop, and compact-before-truncate (preserving the in-flight
tool-call turn). Floor auto_compact_pct at 0.1 (invalid 0 -> default 0.8).
The opening hours were a gold/HP death spiral: a fresh hero spent more gold
healing a fight than the kill paid, mid-tier foes out-damaged a starting HP
bar, the map gave no read on where harder foes spawned, and a spent day left
the player idle until the UTC rollover. This eases the on-ramp across both
shipped worlds.
Economy
- Healer drops from 2 to 1 gold per HP, so topping up no longer outruns income.
- Starting purse 20 -> 37: enough to buy the cheapest armor and one potion up
front, a one-point DEF bump plus a heal cushion the player chooses to spend.
Combat
- Tier 2-4 common foes lose 1 ATK each, trimming the burst that could halve or
end a fresh hero in a single bout. Rares and the boss are untouched.
Wayfinding
- The road glyph changes from "=" to a shaded path that reads as one continuous
road in every orientation; "=" only looked right horizontally and broke into
stacked dashes on vertical runs. Cinder's basalt path gets the same treatment
with a crosshatch glyph free in its palette.
Rest
- Sleeping at the inn now rolls a spent adventurer into a fresh day's turns (it
already fully heals). The top-up fires only at zero turns, so it never banks
past the daily cap.
Verified: full suite green (+2 rest tests, ruff/mypy clean); the greedy balance
bot still clears the world 11/12 seeds at an unchanged pace on both packs.
Auto-title generation and manual refresh stopped producing titles on
reasoning models (the cluster serves qwen3.6). The title call capped
max_tokens at 200, so the model's think pass consumed the whole budget and
content came back empty (finish_reason=length) -> the title was skipped.
Both paths share _generate_title, so both broke.
Title path:
- Raise the title completion to 2048 tokens so reasoning finishes and the
title text actually lands.
- Recover the title from content (never reasoning): reuse the canonical
_strip_reasoning (handles <think>/<reasoning>, paired or unclosed) plus a
backstop for the opener-absent </think> shape some templates emit, take
the first non-empty line, and peel a "Title:" label and wrapping
markdown/quote decoration. Internal punctuation is preserved. Cap at 80 to
match the manual-alias bound.
Temperature:
- _utility_completion no longer hard-codes a temperature; it defaults to the
session/registry value the main turn uses. Title (was 0.7/0.3), web-fetch
extraction (was 0.2), and compaction all defer. Hard-coding a constant
fought thinking/no-temp models and silently overrode an explicit [models.*]
temperature; the provider still gates temperature per model.
Tests: title sanitization across think/reasoning variants, truncation, and a
trailing-prose case; utility-completion temperature deferral + explicit
override.
The send POST's `!r.ok` guard threw a bare `send_http_<status>`, which both
send `.catch` handlers render verbatim — so a rejected send surfaced as
"Connection error: send_http_400" instead of the server's reason. Read the
`{error}` body and throw that, falling back to the status code when a wedged
proxy answers non-JSON (502/504 HTML) so it can't become an "Unexpected
token <" error. Applied to interactive and coordinator.
Also correct the queue-controller comments: a dequeue releases no
"server-side reservation" (queued messages are text-only and dequeue_message
just pops the entry), and onAfterDequeue is wired by coordinator too — not
omitted.
The queued-message dismiss DELETE hardcoded /v1/api/workstreams without
the node-proxy prefix, so cancelling a queued message on a proxied
(remote-node) interactive workstream hit the console root, 404'd, and
the message was delivered anyway -- the dismiss silently did nothing.
composer_queue:
- Prefix getBase() onto the dequeue DELETE; interactive passes getBase
(mirrors the attachment controller). Coordinator stays at base "".
- Never remove the card before the server confirms the cancel: removed
-> drop the card; not_found (already drained) -> promote to a sent
bubble + "already sent" notice; 404 (reaped session) -> terminal drop;
error/timeout -> re-enable + "couldn't remove" notice.
- Bound the DELETE with a 15s AbortController (Promise.race fallback when
AbortController is absent) so a wedged node can't freeze the card.
- a11y: aria-disabled (not the real disabled attribute) keeps keyboard
focus on the dismiss control; in-flight state shown via aria-busy + CSS.
consumers (interactive, coordinator):
- Bound the send POST with the same 15s timeout so a pre-bind dismiss
can't strand the card when the POST hangs.
- r.ok guard so a rejected send (4xx/5xx error body) surfaces as an
error instead of being promoted as "delivered".
- Coordinator wires onNotice -> appendText.
Operator-context system turns must follow a user/tool input turn — producers
maintain this via the user/tool drain seams plus the synthetic wake turn, so an
assistant predecessor is unreachable today. Add a fail-loud guard so a future
producer that breaks the invariant surfaces in logs instead of silently splicing
operator markup into the model's own prior output.
Logged, not raised: it degrades to a fold, since the nonce still gates operator
trust regardless of the host turn, so the harm is out-of-distribution voice
rather than a trust breach — disproportionate to crash a turn over.
Address PR review feedback:
- detection_pattern(()) with an empty tag set compiled to an overly-broad regex
(the empty alternation matches any [start ...]/[end ...] run), which would turn
the forgery scanner into a false-positive generator. Reject an empty or
all-empty tag set up front. Not reachable from the sole caller today, but it is
a public, security-relevant helper.
- Clarify build_operator_instruction_declaration's docstring: the trusted region
is delimited by both the start and end markers (each carrying the nonce), not
just the opening marker.
Swap the trust-fence marker shape from <tag_nonce>...</tag_nonce> to
[start tag_nonce]...[end tag_nonce] for both the operator fold (system-reminder)
and the output-guard judge (tool_output). Angle-bracket markup pushed some local
models out of distribution and toward emitting their own turn-structure tokens:
chat templates built around rigid <...>-style structural tokens derail once a
few folded reminders accumulate. The start/end keywords carry no slash (no </ or
[/ closing-tag shape) and read as ordinary text.
Single-source the shape in fence.py (_OPEN_KW/_CLOSE_KW + detection_pattern) so
wrap, neutralize, the forgery/leak detector, and both trust declarations track
one definition. The nonce still rides both boundaries (unforgeable close); the
leak-vs-forgery split and the forge-in / break-out defang are preserved. The
fold is wire-only, so there is no migration; the legacy persisted-envelope
readers keep the old shape.
Add regression tests pinning each trust declaration to fence.wrap's emission so
a future keyword change fails loudly instead of silently desyncing the anchors.
* feat(projects): governed project containers — memory scope, grouping, manage UI
A workstream can attach to a project: a first-class, shareable resource
container that owns a `project` memory scope, groups conversations, and is
managed from the console.
Storage / migration 062: projects + project_members tables, workstreams.
project_id, and the memory type default project→general; grants
project.{create,read,write,delete} (admin-default).
Recall + writes: project memory is recalled iff the workstream is attached AND
the user has access (owner ∨ member ∨ public-for-read), resolved once at session
construction; coordinators recall it too. New saves default to the project when
attached + writable; the save and delete paths are write-gated; deleting a
project purges its scoped memory; archived projects aren't recalled.
Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a
single-fetch resolver); visibility changes, member management, and delete are
owner-only.
API: project CRUD routes on both the server and console; project_id threaded
through workstream creation, spawn inheritance, the cluster-create proxy, the
dashboard / snapshot / coordinator row builders, and the collector deltas.
UI: a project picker with an inline "+ New project" creator in every creation
box (console launcher + standalone dialog + dashboard); group-by-project in the
rail; a project badge in the composer and on dashboard rows; a console manage
tab (list + create/edit + members shelves). The admin Memories view gains
coordinator/project scope filters and human scope labels (name, not hex). The
memory tool schema documents the project scope and the attach-aware default.
* fix(projects): client refresh hardening, creator race guard, SDK project_id
Addresses PR #724 review feedback plus two bugs found while validating it.
- projects.js refreshProjects: a non-OK status (e.g. 403 when the caller
lacks project.read) or a network/parse error no longer blanks the cache
or masquerades as "no projects" -- the prior cache is preserved, the
failure is recorded (new projectsError()) and warned. Honors the
long-standing "a transient error can't blank the rail" docstring.
- projects.js _fp: the fingerprint separators were raw control bytes,
which made git treat the whole file as binary (no reviewable diff).
Rewritten as escape sequences instead of raw bytes -- behavior is
byte-identical at runtime.
- project_creator.js: createProject() could reject unhandled (authFetch
throws on network/401; r.json() throws on a non-JSON body), leaving the
widget stuck busy/disabled. Added a .catch, plus a generation guard so a
create whose widget was cancelled/reopened mid-flight drops its result
instead of selecting a project the user backed out of.
- types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo /
DashboardWorkstream to match the server schemas (was SDK-invisible).
- test_project_api.py: move side-effecting HTTP calls out of asserts so
the requests run even under python -O.
* fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators
_fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.
- Turn.effect_status also catches TypeError: a corrupt non-string meta value
(e.g. a dict that survived into the column) would otherwise crash a consumer
on access, since EffectStatus(non-str) raises TypeError, not ValueError.
Degrade to None, mirroring the meta decoders (Copilot review).
- test_lowering: the wire-repair synth now carries the _effect_status side
channel (stripped before the provider wire) — assert it.
- test_session_mcp_dispatch_error: the _capture stub swallows the new status
kwarg via **_ so it stays signature-compatible with _report_tool_result.
The unknown / none / committed distinction the cancel and timeout paths
carry lived only in the result's free text — a deterministic reader (a
re-issue guard, owner-side compensation) couldn't recover it without
parsing prose. Promote it to a typed EffectStatus on the canonical Turn.
- EffectStatus (committed/none/unknown/partial/rolled_back) rides
TurnMeta.extra["effect_status"] — wire-invisible like the other meta
side channels: the model still reads the body, deterministic code reads
the type.
- Persisted in the role-exclusive conversations.meta column (source_meta
rides SYSTEM turns, effect_status rides TOOL turns), routed by role in
reconstruct_turns. No migration; survives reload for the audit trail.
- Producer seam: _report_tool_result(status=) + a _tool_status dict popped
at the fold, mirroring _tool_error_flags.
- Populated where the disposition is already determined: UNKNOWN at the six
unobserved sites (bash / MCP-tool timeout, bash SIGKILL-cancel, cancel
synthesis, wire-repair) and a precise none/partial/unknown on a cancelled
task agent (shared _cancel_ledger so the typed status and the prose
disposition can't disagree). Ordinary results stay unset.
Only the unknown/none split is load-bearing (HYPOTHESIS.md effect-record
appendix: unknown, never none); the full per-effect reversibility list
stays deferred. Thread A of the effect-record work; Thread B (per-tool
Smart-Approval floor + reversibility surfacing) follows.
A bash command SIGKILL'd at its deadline and a timed-out MCP tool call are
killed / abandoned mid-flight, so their side effects are as unobserved as a
cancelled call's. Both read as a definitive "timed out after Ns", which invites
a blind re-run (a double-send) exactly as a dropped record invites an orphan.
Route both through a shared TIMEOUT_OUTCOME_CLAUSE so they read "Outcome
UNKNOWN ... do not assume it did not run, reconcile before re-issuing" — the
same "unknown, never none" discipline cancellation already follows
(HYPOTHESIS.md effect-record appendix). bash also keeps any partial stdout
captured before the kill, mirroring the cancel path.
Read-only timeouts (search, MCP resource/prompt reads) stay a plain failure:
an idempotent read has nothing to reconcile, so the reconcile advice would be
misleading there.
* docs(hypothesis): gate-placement & effect-record appendix; scope incompressibility; split the two walls
Refinement + expansion pass on the harness hypothesis.
Appendix (new subsections):
- Gate placement (fail-closed, in practice): γ as a pure, effect-free
parse-and-authorize; syntactic / user-authorization / structural-intent
validation; semantic intent as a recursive plant call (a mini-harness),
not a predicate in γ; "before any invocation" sharpened to "before any
effect" — reads aren't free, the parser must not act, the output is an
action too.
- Effect records (what ρ folds back): pins down the
e = (tool_id, action_id, status, effects, time) shape the body referenced
twice but never defined; committed/none/unknown trichotomy + a reversibility
bit, framed explicitly as an open interface, not a result.
Corrections:
- Scope the incompressibility conjecture: split per-step drift by coordinate
(the shell term is a low-complexity designed descent), so the incompressible
part is the plant's, not all of W; add the coarse-functional counter-
possibility (V* is one scalar hitting time, sometimes cheap) and state the
claim conditionally. Walks back the earlier "the dynamics it certifies are
the weights" overclaim.
- Split the second wall: the tape / space-O(L) picture follows from the
autoregressive structure alone; the per-pass TC^0 bound is separate and
weaker; flag that chaining them is a non-sequitur.
Smaller:
- Concrete justification for the standard-Borel assumption.
- Reading-table rows for the C/Y/A/E spaces and for H_ok/B.
- Daemon note: per-cycle hazard compounds, (1-q)^h over the horizon.
- Minimax: well-posedness caveat for sup over the adversary class Π.
- Note that H_cancel refines the body's deliberately coarse H\H_ok.
Notation (consistency linter clean):
- Brace the subscript A_{⊥} in the new table row (was unbraced — GitHub
render hazard the linter guards against).
- Daemon cycle-count N → h, freeing N for the fundamental matrix.
* docs(hypothesis): address Copilot review — plain quotes + 'none' status value
- Effect-record status enum: add `none`, which the prose already treats as a
distinct value ("unknown ... never none"; the committed/none/unknown
trichotomy). Resolves the enum/prose inconsistency — `none` (no effect) is
distinct from `rolled_back` (ran, then undone).
- Drop the two backslash-escaped quotes (the incompressibility walk-back and
the minimax well-posedness caveat) for plain quotes, matching the rest of
the document. GFM strips the backslash, so they rendered fine; the escapes
were just unnecessary and inconsistent.
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor
turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install.
Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only
tool for a running cluster.
- Preflight detects the install kind (docker-compose/systemd/pip/source) from
config.toml + TURNSTONE_* env, with secret redaction.
- Self-configuring brain resolves the cluster's own model from config/env/storage
read-only (no migrations, no create_all), falling back to interactive
selection; the attempt itself is the LLM-backend health check.
- Deterministic version check: installed version, cluster drift via the console's
authoritative /health, and latest upstream stable/experimental (offline-safe).
- Read-only diagnostic tools (read_file, compose/systemd/journal, http_health,
check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint;
no generic shell, so read-only is structural.
- node_health reaches a node the right way for the detected install kind
(exec-into-container for compose, direct HTTP otherwise), overridable per node
for mixed clusters.
- mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of
mislabelling healthy nodes "unreachable".
init_storage gains a backward-compatible create_tables override for read-only
opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/
architecture/docker docs, the bundled compose header, run.sh, and the CI smoke
updated. CHANGELOG deferred.
* fix(doctor): address Copilot + CodeQL review findings on #718
Validated all seven review findings (none false positives) and fixed:
- check_llm_backend now applies the same scheme / metadata-host guard as
http_health (extracted to _assert_safe_http_url), so a model-supplied
base_url can't be steered at the cloud metadata endpoint or a file:// URL.
- node_health no longer double-appends the default port when the operator
passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080).
- node_health install_type enum uses "git-source" to match the label the
rest of the module and the prompt/report show the model (a schema-strict
provider would otherwise reject the value the model is told to use).
- _read_api_creds takes base_url + api_key as a unit from the first config
source that defines either field, then env-fills, instead of splicing the
two across different config files into a pair that exists in no real config.
- _mask_secrets masks assignment-shaped content inside comment lines, so a
commented-out real secret can't leak through read_file / the report; prose
comments (no KEY=value shape) still pass through untouched.
- drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py.
Adds 5 tests; ruff + mypy clean; full doctor suite passes (129).
* docs(hypothesis): clarity pass, GitHub-render fixes, consistency linter
Document (HYPOTHESIS.md):
- split the dense "Formal" definition into labeled subsections
- define the load-bearing terms: certificate (proven witness vs measured
surrogate) and the controller / plant (= M_W) / shell triad
- corrections: three-way drift split (+ r_env), scope the success/safety
collapse to absorbing refusal, unify tau*->tau_H and drop the orphaned bare tau
- calibrations: pin the incompressibility conjecture (still conjectural),
mark the interlingua=certificate identity as figure, soften the two-walls trade
- GitHub math rendering: brace command-subscripts (_\bot -> _{\bot}, etc.) so the
markdown emphasis parser stops breaking inline math; replace R_\# with R_{\sharp}
(\# unescapes to a raw # in GitHub math)
Linter (lint_hypothesis.py):
- deterministic consistency checks A-G; G adds an orphan/redundant-declaration
scan that catches the bare-tau failure mode
- residue guards so tau^star, unbraced _\cmd subscripts, and \# cannot return
* fix(hypothesis): make lint_hypothesis.py pass ruff under py311
- precompute the inline-$ count so no backslash sits inside an f-string
expression (backslashes in f-strings are 3.12+; the project targets 3.11)
- split the one-line import (E401/I001); open HYPOTHESIS.md via a context manager (SIM115)
- console/server.py: replace a stale hard-coded `session_routes.py:852-854`
comment reference (already drifted to make_close_handler's signature) with
a by-name reference to make_close_handler's not-found path.
- test_cancel.py: rename test_marks_most_recent_action_unknown ->
test_marks_in_flight_action_unknown; the disposition marks the first
unanswered (in-flight) call, not the most recent — they merely coincide in
this two-call case.
The multi-stage review of this branch surfaced four major + two minor issues,
three of them in the new cancellation code. All fixed here (bug-3, the stale
generated TS SDK spec, stays deferred — it regenerates out-of-band).
- sec-1: cancelling a coordinator now auto-cascades to its children, but the
cancel route allows the service-scope bypass while the removed stop_cascade
gated the same destructive subtree-cancel at no-bypass — a service token
without admin.coordinator could trigger the cascade. Re-assert the
no-service-bypass gate inside _cascade_cancel_to_children, so a plain cancel
by an under-privileged service token still cancels the coordinator's own
turn but no longer cascades.
- bug-1: _cancelled_agent_disposition took the LAST issued tool call as the
in-flight one. _run_agent executes a turn's calls sequentially, so the
in-flight call is the FIRST unanswered one — taking the last inverted
unknown/none on a multi-call turn (a SIGKILL'd bash mislabelled "not
started", the never-run tail mislabelled UNKNOWN, inviting a re-run of the
destructive call). Fixed to first-unanswered.
- perf-1: the per-child cancel fan-out was awaited inline before the cancel's
200, so a cancel could block for tens of seconds on slow/unreachable
children. Return the fan-out as a response BackgroundTask so it runs after
the 200 (trigger, not drain).
- bug-2: the initial-send worker (_run_initial) cleared _worker_running
unconditionally — the same clobber the session_worker guard just fixed.
Apply the identity guard there too.
- sec-2: restore the per-child cascade audit row (coordinator.cancel_cascaded)
the removed stop_cascade wrote; it had become log-only.
- q-1: extract the shared UNKNOWN-outcome clause (UNOBSERVED_OUTCOME_CLAUSE)
so the wire-repair fallback and the session-layer synthesis can't drift.
Follow-up to the cancellation review — harden how cancel interacts with a
workstream's OWN turn and tools, not just its children and agents.
- wait_for_workstream: the wait loop holds no cancel handle and blocks on the
child-event bus, so a cancelled coordinator parked in a wait stayed pinned
for up to WAIT_MAX_TIMEOUT (600s). Add a cooperative check to the ~2s
progress heartbeat — it raises GenerationCancelled, which propagates out of
the otherwise cancel-blind wait (~2s abort).
- spawn_batch: stop creating the rest of the children once cancel is observed;
already-spawned children stay recorded (they are live, durably parent-linked
workstreams), the remainder are marked not-spawned.
- session worker: only clear _worker_running if this thread is still the
current worker, so a late-finishing abandoned worker (force-cancel) can't
clobber a live successor's flag — which would let a third send spawn a
duplicate worker on the same session.
- bash silent-cancel: a SIGKILL'd silent command now records outcome-UNKNOWN
(is_error, partial output kept) instead of a clean "Cancelled by user." that
read as a successful empty result on replay.
- wire-repair: the last-resort orphan disposition now reads outcome-UNKNOWN,
matching the cooperative-cancel message (unknown, never none).
Deferred: MCP / web_fetch / web_search remain uninterruptible mid-call,
bounded by tool_timeout; only bash is truly preemptible.
A cancelled agent previously discarded its own ledger and reported a bare
"(task interrupted by user)" — fabricating the *outcome* (read downstream
as "nothing happened"), which invites a double-send as readily as a
dropped record causes an orphan. Make the fold-back honest, and propagate
an owner's cancel down the coordinator subtree.
- task_agent (single + parallel): on cancel, fold back a deterministic
disposition built from the agent's in-memory ledger — actions completed,
the in-flight action flagged outcome-UNKNOWN, and not-started calls —
instead of the opaque interrupted string.
- coordinator cancel now auto-propagates to its direct children via a
post_cancel hook on the shared cancel handler (cooperative fan-out; no
blocking drain).
- synthesized cancelled tool results now read outcome-UNKNOWN rather than
implying the call never ran.
- remove the now-redundant stop_cascade operator endpoint (handler, route,
OpenAPI spec + schema, tests, docs); a coordinator cancel supersedes it.
Review follow-up (#717). The bug-1 fix made the transient keep-path retain the
per-(user, server) refresh lock for serialization, so the lock entry now lingers
after a transient failure. When the token then vanishes (missing) or goes
undecryptable, _no_token_result pruned only the backoff entry and left the lock
entry stranded, so mcp_oauth_refresh_locks could grow on that path. Drop both
sibling dicts in _no_token_result (removing the now-redundant explicit
_drop_refresh_lock on the in-lock decrypt return); the regression test asserts
both are pruned on the missing-after-transient path.
Follow-up to #714 (Entra OBO, #682). A refresh failure deleted the user token +
emitted token_revoked regardless of cause, so a transient AS/network blip during
a forced refresh (the live 401-retry path) permanently revoked consent
cluster-wide. Fixing only that, though, opens the dual failure: a genuinely-dead
grant the AS reports in a non-standard shape would now be kept forever and the
user stranded on a retryable error with no re-consent path. This classifies the
failure three ways so each is handled correctly.
Classification (_classify_refresh_failure): MCPOAuthRefreshFailed carries a
_RefreshFailureClass instead of a bool —
- PERMANENT (revoke + re-consent): an explicit dead-grant / re-consent signal —
invalid_grant at any 4xx (400/401/403), invalid_scope, or an OIDC
interaction-required code (interaction_required / login_required /
consent_required / account_selection_required) the AS surfaces.
- TRANSIENT (keep, retry, never escalate): infrastructure (network, 5xx, 429,
malformed body) and operator-fixable codes (invalid_client, invalid_request,
unauthorized_client, unsupported_grant_type, temporarily_unavailable) —
re-consenting the user can't fix a bad client_secret, and an outage must not
revoke consent however long it lasts.
- AMBIGUOUS (keep, but escalate after a run): a 400/401 we can't map to a
standard code. A one-off can't revoke, but an uninterrupted streak past a
threshold escalates to re-consent so a dead grant in a non-standard shape
can't strand the user. Infra transients reset the streak, so an outage never
escalates.
Concurrency: do NOT drop the per-(user,server) refresh lock on the keep-the-token
path. Evicting it while the token is still live let a second concurrent caller
mint a fresh lock and refresh the same token in parallel; with refresh-token
rotation the second send reuses the consumed token, gets invalid_grant, and
spuriously revokes — the exact bug this commit prevents. The async-with still
releases the lock on return; the registry entry is pruned only when the token is
actually refreshed or revoked. Bit SQLite single-node hardest, where the pg
advisory lock is a no-op.
perf: a per-(user,server) cooldown short-circuits the token-endpoint round-trip
for a brief window after a transient failure, so a down AS isn't hit once per
tool call; self-heals when the window expires. Plus the lock-free in-flight key
set that collapses concurrent session-start pool primes (single mcp-loop thread).
dispatch/FE: the transient kind maps to a retryable mcp_refresh_unavailable
structured error (not mcp_consent_required); the FE titles it "Temporarily
unavailable" under a new soft "transient" category (amber, not the red hard-error
styling) in both stylesheets, with no wrong re-consent button.
tests: invalid_client kept (pins the discriminator on the error code, not the 4xx
status), single ambiguous 400 kept, 403 invalid_grant revokes, interaction_required
revokes, ambiguous streak escalates at the threshold, sustained 5xx never escalates
(outage safety), and the cooldown skips the second AS round-trip — all through the
real AS HTTP boundary.
Follow-up review of the #706 on-behalf-of / Entra ID MCP changes (#682).
security (PKCE downgrade): the AS-metadata "assume S256 when
code_challenge_methods_supported is absent" relaxation applied to BOTH the
RFC 8414 oauth-authorization-server document and the OIDC openid-configuration
document. Per RFC 8414 an omitted field on the oauth-authorization-server
document means the AS does NOT support PKCE, so this was fail-open. The client
always sends code_challenge_method=S256, making this discovery check the only
pre-flight that the AS enforces PKCE. Track which document won discovery and
assume S256 only for the OIDC document; the RFC 8414 document now fails closed.
Also log which discovery profile (rfc8414 vs oidc) answered, for operators
debugging an enterprise AS.
bug (consent loss): session-start pool priming called the refreshing token
lookup for every cold oauth_user server. A near-expiry token triggered a
refresh, and a transient refresh failure (network/5xx/429) deletes the token
and emits token_revoked — so a blip during a cold-pool warm (e.g. after a
reboot) silently revoked consent across servers the user wasn't even using.
Priming now reads the token directly and skips missing/near-expiry tokens;
a refresh that may fail stays on the lazy dispatch path.
perf/UX (blocking redirect): the OAuth callback awaited prime_user_server
(default 20s timeout), holding the consent redirect on a slow/unreachable MCP
server. Replaced with fire-and-forget schedule_prime_user_server that schedules
onto the mcp-loop (GC-safe, no unreferenced request-loop task) and returns at
once.
perf: prime a user's pools concurrently under a bound instead of serially, so
one slow upstream can't stall the rest.
hygiene: log (not silently swallow) prime scheduling failures at session start;
add exc_info to the prime-failure warning; guard run_coroutine_threadsafe
against a closed mcp-loop.
tests: per-document S256 + OIDC-fallback discovery cases; pool priming
(non-destructive on near-expiry, skips connected) and bound-token rotation
reconnect.
* This is a collection of little snippits to resolve all the OBO flow problems required to get this talking to entra id for on behalf of user impersonating to protected mcp servers. we make sure turnstone checks these mcp servers on startup, and address some of microsofts opinionated implementations of oauth2/oidc and metadata provided by the identity provider.
* minor token timeout bugfix
---------
Co-authored-by: root <root@pow3rtools>
Replace the amber gauge/needle favicon with a teal up-chevron and amber
dot on a dark-teal field. Applied identically to the console, coordinator,
and standalone UI entry points. Self-contained inline SVG data URI; no
network dependency.
Establishes the appendix pattern (locate a practical concern in the existing
formal objects; read off the discipline rather than inventing machinery) with
cancellation as the first and only worked example. Not the whole model.
Cancellation semantics, derived from objects already on the page:
- Cancel is a signal → lives in s (Markov). The gate closes on it: γ(s,y)=⊥ while
live, which blocks pending actions and all future turns with no new machinery.
- In-flight (past γ) disposition is a trinary on the kind of Q_E: cancellable
(propagate, true end-state), bounded (drain, real e), or opaque/unbounded
(controller fabricates a synthetic "cancelled" e so the loop can halt).
- Load-bearing rule: ρ may fabricate the acknowledgment but not the outcome — an
unobserved outcome is `unknown`, never `none` (double-send vs orphan, same bug
opposite sign).
- New terminal H_cancel ⊆ H\H_ok: non-accepting but safe (outside B), postcondition
"no action past γ after observed; in-flight drained or recorded unknown; ledger
consistent."
- Cooperative not preemptive (observed at next γ check, not on send); recursive
down the task-agent subtree (why task agents are the worst case).
- Compensation is the owner's job (saga, after H_cancel, reads child ledger) — the
cancelled agent can't know if it's needed; it never observed the outcome.
- Design pressure: prefer bounded/instrumented Q_E over opaque, so cancellation and
the ledger stay honest (a bash wrapper converts branch 3 -> branch 1).
Linter: balanced, no new collisions. Two ρ role-flags, both false positives
("authorized action" near ρ, correct usage).
The linter closed mechanical consistency; this review probes meaning, a separate
axis. Twelve findings, several real corrections, all folded.
Correctness:
- Stationarity overclaim: the supermartingale BOUND survives a nonstationary
kernel under uniform conditional drift. Time-homogeneity is needed for V* as a
fixed function, the resolvent/fundamental-matrix identities, and δ-calibration.
- Self-contradiction: "the certificate cannot be proven, only observed" contradicted
the established "a proven inequality certifies" — reworded to "the architecture
does not hand it to you; estimated unless separately certified."
- Citation: the TACL result is LOG-precision → logspace-uniform TC⁰ (verified);
fixed/constant precision is a stronger restriction. Fixed in body and Grounding.
Modeling holes closed:
- Adversary class Π must respect rejection: γ(s,y)=⊥ ⇒ Q_E^α(s,⊥,·)=δ_e0, else the
adversary resurrects refused side effects.
- R must be a syntactic/verified readout, not a semantic solver — otherwise the
L-wall is void (compute could hide in R off the ≤L window).
- e must be an effect record (ledger outcome), not just API bytes, since only ρ
writes external effects into s.
- The displayed M_W(c) freezes endpoint/version/sampler; config changes need a
state-indexed M_{κ(s)} or K_C — the kernel can't silently depend on config in s.
- The final user-visible response/log is itself an effect: an authorized action
through γ, or emitted only after an accepted halt.
- m_t must include a token counter and clock for the cap/timeout to be functions of it.
Residue (omissions a collision-linter can't catch):
- Another γ dropped from the K_C Dirac-special-case list.
- τ* mislabeled as "designed code" → the halt test (H) is the code; τ* is its
emergent hitting time.
- H\H_ok relabeled "non-accepting" (safe refusals outside B; wrong/bad halts
possibly in B), not uniformly "rejecting/fail-closed."
Built and ran a static linter (no model): delimiter/emphasis balance, residue
regexes for everything prior rounds fixed, single-capital collision scan, a
definition check for recently-introduced symbols, and a γ/ρ role-neighborhood
scan. Result:
- All balance checks pass; all 10 residue regexes clean (no regression across
14 rounds); all 12 introduced symbols defined; no display-only symbols.
- γ/ρ scan: one flag, a false positive (the symbol-table cell defines both).
- One real find: G was overloaded — the parser-stop update G(m_t,v) (added in
round 14) collided with the Green/potential operator G. Renamed the stop-update
to \mathsf{step}; the Green operator G is now unique.
This closes the consistency axis deterministically rather than by another review.
Same prior-maxima full-tools review, re-run. Found mostly residue from round-13's
own edits plus longer-standing inconsistencies. Folded all; left the final
signature line alone (it is the author's call, and it is well-formed — see below).
Round-13 residue:
- 𝒴/𝒴_⊥ split half-committed: 𝒴 already includes ⊥, so R:𝒵→𝒴 and A_Y⊆𝒴 (drop _⊥).
- m_t was added to the inner triple with no dynamics: add m_{t+1}=G(m_t,v), define
the stop set Stop and τ=inf{t:m_t∈Stop} in both display and prose.
- No-truncation special case had R=id, ill-typed on a triple: R(c,b,m)=c.
- Safety/success "exactly on safe refusals" overclaimed: they differ on any
B-avoiding non-success run — also safe non-halting / endless safe retry, absent
a.s. absorption into H∪B.
Role residue (γ does authorization/rejection; ρ does response/fold-back):
- "⊥ branch is what ρ rejects" → γ rejects it.
- "ρ validates response as well as the proposal" → ρ validates the response; γ
gated the proposal.
- "fail-closed rejection at ρ" (falsification list) → at the gate γ.
- Symbol table still typed Q_E on authorized a → a∈𝒜_⊥ with the no-op; define e_0.
Longer-standing:
- Stochastic-controller contradiction: stochastic control falsifies the
deterministic special case, not the broader K_C kernel model (round-9 K_C).
- Drift split r=r_shell+r_plant needs an additively separable V̂ or a declared
attribution scheme.
Citation (verified via search, not the reviewer's say-so):
- TC⁰/log-precision → Merrill & Sabharwal, "The Parallelism Tradeoff", TACL 2023;
caveat (added autoregressive steps escape it) → Merrill & Sabharwal, "The
Expressive Power of Transformers with Chain of Thought", ICLR 2024.
A cold reviewer given the complete prior-maxima changelog + full tools ran a
consistency audit of the file (it did not use tools for grounding — the gap was
internal). Found 15 real issues, all folded. No new design flaws; this is
accumulated editing debt from 12 rounds of surgical patches.
Half-applied fixes now propagated:
- Inner-kernel display still showed M_W(c)=Law(c_τ) and R:C→Y_⊥ despite the round-12
triple; made z_t=(c_t,b_t,m_t) primary, R:Z→Y_⊥, M_W=Law(R(z_τ)).
- Append formula still used bare c·v; now suffix_{≤L}(c·v) in the display.
- Tuple still called B a "terminal set"; B is separate (τ_B fires mid-run).
- "halt/ready" survived at line 75 (fixed before only in tuple + table).
Collisions created by added notation:
- γ was both the authorization gate and the RL discount in (I-γP)^{-1}; discount → β.
- B was both the bad set and the dummy measurable set in the pushforward; dummy → A_Y.
- ρ over-credited as the disturbance-rejection margin; for side effects the margin
is γ (consistent with round-12 irreversibility), ρ validates response/fold-back.
Real error in a prior round:
- The round-12 safety/success distinction collapses under absorbing refusal
(Pr(τ_Hok<τ_B) requires reaching H_ok, so it is a success form). Split correctly:
p_succ=Pr(τ_Hok<τ_F), F=B∪(H\Hok); p_safe=Pr(τ_B=∞); they differ on safe refusals.
Typing / hygiene:
- Q_E typed on S×A_⊥ (it is applied to ⊥); 𝒴 declared to include ⊥ (M_W, γ total).
- Controller list omitted γ and mis-listed the readout (specialization-only).
- Defined the previously-bare symbols D={s:E[τ_H]=∞}, μ, the drift r(s), and Π.
- Grounding "verify by measured drift" overstated; a proven inequality certifies,
empirical drift only checks — reconciled with the body.
A cold no-priors review (given a local sandbox it did not use — the remaining
work is judgment, not computation). Mostly editorial/formal; its real catches
again concern round-10/11 additions. Folded the substantive ones, declined the
"extract a smaller core" restructure and the formalism padding.
Substantive:
- Inner kernel: replace round-11's awkward "read c_τ as the buffer" overload with
a clean inner-state triple z_t=(c_t,b_t,m_t) — window, output buffer, parser/
stop state — and M_W(c,·)=Law(R(z_τ)) from z_0=(c,∅,m_0). Strictly cleaner.
- Authorization is the irreversibility boundary: ρ can reject a bad tool RESPONSE
but cannot undo an authorized action's side effects, so γ (not ρ) is the last
line before irreversible effects. And the gate is bypassed if raw y reaches any
sink (tool, logger, browser, remote) before γ.
- Safety ≠ success: p_ok=Pr(τ_Hok<τ_B) is the safety object (refusal permitted);
the stricter success object races H_ok against all failure F=B∪(H\Hok). They
differ exactly on safe refusals.
Precision:
- Foster–Lyapunov positive recurrence needs irreducibility/petite-set hypotheses;
the absorbing-halt case needs only the weaker supermartingale hitting-time bound.
- Name an initial distribution s_0~μ_0. Fix residual "halt/ready" in the table
(round 11 fixed only the tuple).
A JSON-constrained cold review largely validated round 10; its new catches
cluster in round-10's newly-added material.
Fixes (the real ones):
- Terminal-set partition was wrong (a round-10 error): B is NOT a terminal
component — τ_B can fire mid-run. H now splits into accepting (H_ok) and
rejecting/fail-closed (H\H_ok); B is a separate unsafe set for reach-avoid.
- Fail-closed generalized: rejection need not be terminal (reject-then-retry is
valid) — define it as "no unauthorized side effect + land in a safe non-bad
set," with terminal rejection one case. ρ must also validate the tool RESPONSE
e (adversarial/malformed Q_E output), not only the model proposal at γ.
- Sliding-window truncation (round-10) loses transcript: the readout R reads a
separate output buffer, not the truncated c_τ alone.
Precision:
- Deterministic maps are measurable transforms inside the pushforward, not
literally "outside the integral."
- Absorbing halt H vs the separate (non-absorbing) daemon "ready" recurrence.
- "Syntactic soundness is free" qualified: relative to a formal schema and a
correct validator.
- State-ablation falsifies Markovity but cannot establish it (necessary, not
sufficient). Added a readout-typing diagnostic.
A cold no-tools external review (lower trust on world-facts, but its catches are
math-internal and correct) found two real bugs plus rigor gaps.
Bugs fixed:
- Verification-after-side-effect (the important one): the kernel ran e~Q_E(s,y)
then ρ verified, so a tool call's side effect landed before authorization. Add
a deterministic authorization gate γ:S×Y→A_⊥ between model and environment;
Q_E now acts on the authorized action γ(s,y); ρ becomes ρ(s,y,a,e). Fail-closed
is now a property (γ=⊥ ⇒ no-op env ⇒ fold to H\H_ok), not a name.
- Minimax drift display had a free y (introduced round 9): it integrated only
over e while y~M_W(π(s)). Now integrates over both y and e, adversary as a
policy α(s,y) over environment kernels, on the authorized action.
Reframing / rigor:
- Raw halting is cheap: a budget counter k gives V=k as a trivial halting
certificate, so "no certificate by construction" overstated. The missing
guarantee is correct/safe/successful halting (H_ok, B, p_ok).
- Standard Borel spaces (not merely measurable); define H, H_ok (⊆H), B (∩H_ok=∅)
and hitting times τ_A up front; add 𝒜 to the tuple.
- Inner kernel: truncate c·v to suffix_{≤L} at the window edge; M_W is a
probability kernel only via EOS/max-token/timeout/⊥ (else sub-probability +
cemetery).
- Drift: weaker bound δ≤δ̄<ε gives E[τ]≤V̂/(ε-δ̄); distinguish δ_ν (distributional)
from δ_sup (worst-case).
- Injection enters π's inputs (retrieval/pages/tool metadata), not only post-model
Q_E; B needs a side-effect ledger in S. Architectural invariants stated
(model sees only C; outputs are proposals; γ gates side effects; terminals
partitioned). Complexity/LBA material marked heuristic, not definitional.
An LLM-judge verifier is a learned kernel, not deterministic ρ.
A cold external review (same priors, no path-dependence) surfaced three real gaps
the iterative chain missed, plus precision items. Folded in:
Substantive:
- Stochastic controller: the deterministic π,ρ,H are the Dirac special case of a
controller kernel K_C(s,dc) (routing, sampled retries, ensembles, learned
routers). Deterministic is the case worth wanting (localizes randomness); the
split widens, not breaks, under stochastic control.
- Minimax type fix: the adversary chooses a POLICY/kernel, not the realized
sample. Display is now sup over α of ∫ V(ρ(s,y,e)) Q_E^α(s,y,de), not sup over
the post-probability e.
- Reach-avoid security: add a bad set B; injection steers toward B (wrong
acceptance, exfiltration, unauthorized tool use, privilege escalation,
irreversible effects), so security is reach-avoid p_ok=Pr(τ_{H_ok}<τ_B) with a
barrier certificate for B, not liveness. B and H_ok added to the tuple.
- Unconditional V*_ok is infinite under any positive pre-acceptance failure
probability ⇒ the workable object is p_ok (or the regenerative time on restart).
Precision / hygiene:
- Compiler claim scoped to a specific data-flow analysis (not a whole compiler);
add integrability/optional-stopping conditions to the hitting-time bound.
- Formal hygiene: spaces measurable, τ/τ* stopping times, H absorbing.
- Mid-generation tool calls interleave the loops — clean nesting is an
idealization needing a finer state machine.
- Soften SSM ("different", not "tighter"); demote "manifold" to informal
shorthand in the formal section; gloss "all undefined behavior" as "no complete
formal source-language semantics."
Not changed: V* incompressibility (already labeled conjectural in Grounding).
The review's verdict was "Merge." These are its two correct non-blocking nits;
its third nit (stop adding theorems/caveats) is heeded — nothing else changed.
- Grounding: "the compiler's V is free" → "a classical monotone data-flow
analysis gets its V for free." A whole compiler does not get termination for
free; the specific lattice-based analysis does (Kildall).
- Asserted: the Koopman/certificate co-determination "holds only under" →
"is well-posed only under" the spectral assumptions — avoids asserting truth
("holds") for a claim explicitly labeled as not-a-theorem.
Deliberately NOT changed: D → D_H (prose already marks D harness-relative;
subscripting one formula while D stays bare elsewhere would add asymmetry, not
remove it), and no further theorem additions or caveats per the review's note
that more caveating now costs clarity without adding rigor.
The review's verdict was "mergeable"; these are its three optional items plus the
delta-attribution nit.
- δ attribution: sampled-state coverage is an evaluation-protocol property, not a
weights property. Attribute the noise floor / residual risk to the trained
weights, the environment, AND the evaluation distribution.
- reachable(L) is harness-relative too (same reason U_H(L) is): rename to
reachable_H(L) and note the divergent set D is likewise relative to H.
- Split the dense frontier paragraph in two: (1) the SR / fundamental-matrix /
potential-operator identity with its caveats; (2) the speculative interlingua/
certificate thesis. No content change.
- Grounding: add the absorbing-chain fundamental matrix (Kemeny & Snell 1960),
the general-state potential/Green operator (Revuz 1984), and Koopman (Koopman
1931; Lyapunov-from-eigenfunctions, Mauroy & Mezić 2016) to Proven; mark the
Koopman/certificate co-determination (spectral-assumption-dependent) and the
interlingua/certificate identification as Asserted.
- U(L) is harness-relative: tools and decompositions change membership, so rename
to U_H(L) and note the shell's verified tools / decompositions determine what
can be paged or outsourced.
- Countable fundamental matrix: lead with the Neumann series N=Σ Q_tr^n, scope
countable to convergence, and write (I-Q_tr)^{-1} only when the inverse exists;
general-state version is the same series read as the potential (Green) operator.
- Distinguish failure modes for V*_ok: infinite under a formal success predicate
vs undefined if no predicate has been specified.
- Soften the delta "floors" line: mu(D), sampled-coverage, and Var[tau*] drive
the empirical noise floor / residual risk, they are not literal floors of the
drift slack.
- Hedge the Koopman bridge (the last frontier thread): the eigenbasis claim
presumes a diagonalizable, point-spectrum operator — mixing dynamics carry
continuous spectrum and admit no eigenbasis — and the linearizes/certificate-
decomposes coincidence holds only for a V in the span of those eigenfunctions.
Address the round-five review's three precision points (plus the adaptive-adversary
refinement).
- Absorption is finite expected hitting time, not positive recurrence: replace
"positive-recurrent to H" with "reached in finite expected time," domain
{s : E_s[τ_H] < ∞}. Positive recurrence stays reserved for the daemon/
ready-state case (where it is used correctly).
- The fundamental matrix N=(I-Q_tr)^{-1}=Σ Q_tr^n is the finite/countable object;
the formal model lives on general measurable spaces, so add the general-state
potential (Green) operator G=Σ Q_tr^n with G·1=V* where the series converges.
Q_tr now stated as the sub-stochastic kernel restricted to H^c.
- V*_ok is taken on the process where H\H_ok (halting wrong, refusing, failing
closed) is absorbing failure — so a run that fails closed before acceptance
has infinite accepting hitting time unless the spec restarts it. This is the
mechanism by which a U(L) task sends V*_ok → ∞.
- Adaptive adversary: nonstationary Q_{E,n} → time-ordered product; an adaptive
adversary → controlled / game-value operator (not merely time-indexed).
Fold in the two seams flagged after round three, before the next review pass.
- Limit section now states explicitly that its V*=E[τ*|s] certifies *halting*
(reaching H at all), not correct halting; defers V*_ok (expected time to an
accepting H_ok ⊆ H) to the second wall. Removes the latent inconsistency
between the limit section (plain H) and the U(L) refinement (H_ok).
- Frontier section: the discounted successor-representation resolvent
(I-γP)^{-1} presumes a discount γ and fixed P the stopped formulation lacks.
Replace with the correct undiscounted/absorbing object — the fundamental
matrix N=(I-Q_tr)^{-1}, Q_tr the sub-stochastic transient block — whose row
sums N·1 are exactly V*. Converts analogy-dressed-as-identity into a true
identity for the doc's own kernel.
- Mark the "one object seen twice" identity as holding only in the stationary
regime: under the adversarial Q_{E,n} the resolvent/fundamental matrix become
a time-ordered product, so identity in the stationary case, analogy beyond.
Address the round-three review. The substantive one is the V* correction.
- Successful halting vs raw halting (the real conceptual fix): a U(L) task does
NOT make V*=E[τ_H|s] undefined — the chain can still hit H by failing closed,
refusing, or returning a wrong answer. Split H from the accepting set H_ok and
define V*_ok=E[τ_{H_ok}|s]; U(L) blows up V*_ok, not V*. Restate the domain as
dom_{<∞}(V*_ok) ⊆ reachable(L)\D.
- Tools compute, not just store: the L-wall binds *model-mediated* work; work
discharged to a verified external tool (solver, interpreter, compiler) runs
off-context. U(L) now excludes tool-dischargeable work explicitly.
- Readout typing: use the pushforward M_W(c,·)=R_# Law(c_τ) (equivalently the
conditional law); make R total, R: C → Y_⊥, with the ⊥ branch handled by the
fail-closed ρ.
- Adversary/history: a history-conditioning adversary needs that history in s,
else the object is a Markov game requiring further augmentation, not a chain.
- Hedge the LBA claim: "in the variable-L, fixed-precision idealization, the
model-mediated inner computation behaves like a linear-bounded automaton."
Address the three follow-up points on the first review patch.
- Reconcile the model kernel's two types: M_W(c,dy) maps into 𝒴, while the
transformer line writes M_W(c)=Law(c_τ) over contexts. Add the readout R:
𝒴 is either c_τ itself (𝒴=𝒞) or a deterministic readout R(c_τ), with
M_W(c,dy)=Law(R(c_τ)∈dy).
- Separate harness state 𝒮 from model-visible context 𝒞: the L wall binds 𝒞
(the L×d residual stream), not 𝒮. External stores (files, DBs, vector stores,
durable memory) are shell-supplied memory that extends addressable storage but
not the per-pass resident set — every read still routes through the ≤L window.
Retype U(L) accordingly: not data exceeding L (pageable) but irreducible
per-step working set exceeding L (not pageable).
- Make the time-homogeneity assumption explicit at the formal kernel: the
displayed T is the fixed-kernel case; nonstationary/adversarial environments
replace Q_E with a time-indexed kernel Q_{E,n} / admissible family, which the
minimax certificate downstream quantifies over.
Address the accepted points from an external peer review while preserving the
controller/plant thesis and the document's voice (layer, don't flatten).
- Claim: replace the ill-typed `T = ρ ∘ (M_W ∘ π, E)` with the integral
transition kernel over (𝒴,ℰ); add explicit informal/formal split; demote the
residual-stream implementation from definitional to a kept specialization
(M_W as a general learned kernel); weaken "fixpoint searches" to hitting-time
processes with fixpoint as one mode.
- Reading-it: note s is Markov only after state augmentation; mark controller
determinism as conditional on versioned code/config/endpoint/interfaces.
- The limit: rephrase "carries no descent function by construction" to "supplies
no certificate automatically" (a certificate is sufficient, not provided for
free); label V* incompressibility as conjecture, not theorem.
- δ: "measure" → "estimate"; demote empirical δ from certificate to calibrated
risk metric (confounds: bad V̂, coverage, sup not attained, nonstationarity,
non-Markov); certificate only once statistically bounded.
- Cash-out: split "soundness is free" into syntactic soundness (free) vs
semantic adequacy (empirical).
- Qualify the single-pass TC^0 claim (fixed-depth/fixed-precision; log-depth
changes it) in both body and Grounding.
- Add an operational falsification program (state-ablation, determinism audit,
drift calibration, adversarial-environment, boundary-control ablation).
Citations with a proven-vs-asserted split; the orthogonal context-length
tape bound (TC^0 single pass, the U(L) non-haltable region); and a flagged
frontier coda on V* and the semantic interlingua as one object.
A one-formula definition of a harness — a deterministic controller in
closed loop with a stochastic learned plant — and the certificate it
provably can't carry. The headline equation sits at the top of the
README and links through to the full doc.
* feat(deploy): vllm-litellm example — 3-model co-resident shape + HF loader
Update the unified-memory inference example to the validated GB10 Spark shape:
qwen3.6-27B-FP8 (reasoning) + gemma-4-12B-it (perception) + Qwen3-Reranker-4B,
all co-resident on one GPU behind LiteLLM, loaded by HF id into a mounted
HF_HOME cache.
- qwen: MTP spec-decode + runai_streamer (weight load ~166s->1s) + full 256K at
util 0.50 (default KV)
- gemma on the OpenAI lane (audio), reranker direct on :8002/rerank
- sequential startup + page-cache-drop guidance; runai_streamer kept on the big
model only (its buffers break small models' KV budgets)
- README: HF-id loader, DGX Spark (validated) + AMD Strix Halo (ROCm) setup,
tuning notes, troubleshooting
- wheel-check ALLOW entries for the example files (supersedes #687)
* docs(deploy): clarify AMD edits are compose literals (Copilot review)
In the Strix Halo guidance, --max-model-len and --load-format runai_streamer are
hard-coded in docker-compose.yml's vllm-qwen command, not .env vars — say where
to edit them.
* feat(deploy): add vLLM + LiteLLM unified-memory inference example
A docker-compose stack co-residing a reasoning model (Qwen 3.6 27B) and a
perception model (Gemma 4 12B) on one unified-memory accelerator (NVIDIA DGX
Spark / AMD Strix Halo) behind a LiteLLM gateway serving both the Anthropic
/v1/messages and OpenAI /v1/chat/completions routes.
- qwen on the Anthropic lane (vLLM native /v1/messages), full 256K context
- gemma on the OpenAI lane (required for audio input_audio perception)
- sequential startup + page-cache drop for reliable KV provisioning on one card
- README: DGX Spark (validated) + AMD Strix Halo (ROCm) setup + troubleshooting
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(deps): add altair + vl-convert-python viz stack
The standard, ui://-ready visualization stack: one Vega-Lite spec renders to
static SVG via vl-convert (a bundled Rust renderer — no browser, GDAL, or
chromium) and drops into vega-embed for interactive ui:// panels. The first
consumer is the civic-records choropleth map; future ui:// surfaces build on
the same stack.
The dependency closure is fully permissive (BSD-3 + the OFL font + MIT/ISC JS) —
clean for Apache-2.0 and commercial use. Adds a mypy override for the untyped
vl_convert wheel.
* fix(deps): bump pydantic-settings to 2.14.2 (GHSA-4xgf-cpjx-pc3j)
Clears the pip-audit --strict advisory on the transitive pydantic-settings
2.14.1. Pinned as an explicit security floor in [project.dependencies]
(matching the starlette/cryptography CVE-floor convention) even though it is
transitive-only, so the floor is documented and survives re-resolution.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* chore: regenerate uv.lock to pass lock check
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
- The guard snapshotted live threads by `Thread.ident`, but idents are
recycled after a thread exits — a new leaked thread reusing an exited
thread's ident would be mistaken for pre-existing and missed (false
negative). Snapshot the Thread OBJECTS and compare by identity instead.
- Fix the `serve` fixture docstring: the factory returns the ephemeral
port, not the server.
Background daemons, event loops, and test servers that outlived their test
bled into later tests' captured output — an intermittent "I/O operation on
closed file" heisenbug, and the same class behind a past multi-day CI-hang
investigation.
- conftest: a fail-on-leak autouse guard (`_no_leaked_threads`) snapshots
threads at setup and fails any test that leaves one running past teardown,
with an `allow_thread_leak` opt-out — so the next leak is caught in minutes,
not days. Plus `logging.raiseExceptions = False` to mute the benign
logging-vs-capture-teardown race, and shared loop/server teardown helpers
(`stop_loop_thread`, `serve_until_exit`).
- collector (PRODUCT FIX): the node-discovery loop slept uninterruptibly, so
`ClusterCollector.stop()` couldn't join the `console-discovery` thread until
the full interval elapsed — a real shutdown hang in production (up to
`discovery_interval`). It now sleeps on an interruptible Event that `stop()`
sets and `start()` clears.
- test fixtures: docker_healthcheck's HTTP servers, the MCP background event
loops (shutdown_default_executor + close), and the FastMCP uvicorn upstreams
(timeout_graceful_shutdown=0 + force_exit) now tear down cleanly instead of
leaking.
Full non-live suite: 7456 passed, 0 closed-file errors, 0 leaked threads, and
~1.5 min faster (the leaks were dragging it).
Coordinators carry LLM/auto titles like interactive workstreams but had no
way to regenerate or rename them. Port the interactive "Refresh title" (LLM
regenerate) + "Edit title" (manual alias) dropdown actions by lifting the
two handlers — the last shared verbs that weren't yet lifted — and opting
coordinators in.
- session_routes.py: add make_refresh_title_handler / make_set_title_handler
factories (cfg pattern, mirroring make_close_handler). set_title resolves
the workstream BEFORE the alias write and 404s when the kind has no
tenant_check storage gate and the in-memory manager doesn't own it:
set_workstream_alias is a global, kind-unscoped UPDATE, so this prevents
an operator renaming a workstream the coord manager doesn't own (e.g. an
interactive ws via the coord route) and the silent-200 on a bogus id.
- server.py: re-point the interactive bundle to the lifted handlers; drop
the standalone refresh_workstream_title / set_workstream_title.
- console/server.py: wire refresh_title / set_title into the coord bundle
(gated by the existing admin.coordinator operator check).
- shell.js: enable titleVerbs on the coordinator pane's tab menu; the
base-aware lane posts to the console-origin coord routes.
Tests: coord refresh/set-title (regenerate, operator-gate, 404 unknown,
alias store + broadcast, empty, conflict, cross-kind reject); interactive
title tests re-pointed to the lifted handlers for lift-parity; shell.js
coord-menu assertion.
Three points from the PR #676 Copilot review:
- _coord_display_name ran on a lifecycle-event path and called
get_workstream_display_name → get_storage(), which auto-initializes a
SQLite .turnstone.db in the CWD when storage isn't initialized yet —
a stray-file footgun on early-startup / unit-test paths. Add
is_storage_initialized() to the storage registry and skip the DB read
(fall back to ws.name) when storage isn't up. (Copilot's "skip when
ws.name is non-synthetic" suggestion would have broken alias > title >
name, so guard on init state instead.)
- Document, on SessionUIBase, that on_aux_usage (storage/metrics, no
_ws_lock state) and on_rename (queue/locked fan-out) are safe to call
from a concurrent auxiliary thread — the title-gen thread now runs
during streaming, and these are the only two UI hooks it touches. No
behavior change: the methods were already thread-safe (the same path
task_agent sub-agents use); the contract just didn't say so. Add a
matching note at the title-trigger site.
- Note in _coordinator_rows that the secondary `title` field is
best-effort for a live coord outside the limit=200 window (the
user-visible `name` stays correct via the uncapped bulk lookup, and
the window is unreachable in practice — live coords are max_active-
bounded and sort to the top of updated DESC).
Coordinator workstream LLM titles were written to workstreams.title but
never read back, and were rarely generated in the first place:
- Read path: the dashboard's `_coordinator_rows` builder hardcoded
title="" and used the synthetic `ws.name`, so a generated title (or a
user alias) reverted to `ws-xxxx` on every refresh. Interactive rows
resolve via get_workstream_display_name, so the gap was coord-only.
- Write path: the auto-title trigger only fired on a tool-call-free
assistant turn, which coordinators (near-constant tool use) seldom
reach — so the title almost never generated.
Read path:
- Project `title` + `alias` in list_workstreams (appended after user_id so
existing positional fallbacks stay valid). `_coordinator_rows` resolves
the display name (alias > title > name) for both lanes — live names via
the bulk get_workstream_display_names (exact ids, no row cap), persisted
rows from their own _mapping.
- Seed the console pseudo-node fan-out with the resolved display name so a
rehydrated coordinator shows its title in the live tree immediately
(one bulk lookup instead of an N+1 over mgr.list_all()).
Write path:
- Fire auto-title right after the user turn is recorded in send(), gated on
a real (non-wake, non-empty) user message, instead of waiting for the
terminal tool-call-free turn. Applies to interactive + coordinator.
- Snapshot self.messages in _generate_title since it can now run
concurrently with the streaming turn.
Speech-to-text against an omni chat model (e.g. Gemma-4 on vLLM) was
broken end to end:
- The browser records webm/opus, but the omni chat lane only decodes
wav/mp3 (it sniffs the bytes), so every clip came back 400 "Invalid
or unsupported audio file". Transcode the upload to 16 kHz mono WAV
with ffmpeg first, hardened against the untrusted blob:
-protocol_whitelist pipe (no file:/http: SSRF), -vn, and a duration cap.
- The chat STT path calls the raw client and so bypasses the provider's
request shaping. It now forces enable_thinking=false (via the model's
thinking_param): leaving reasoning on costs ~11x latency and returns
empty content on some clips. The prompt precedes the audio part (the
order Gemma documents for transcription) and max_tokens is capped.
Add a streaming variant: POST .../speech-to-text/stream returns the
transcript as plain-text deltas and the composer fills them in live
(~0.3s to first word). The blocking stream is driven from one worker
thread that owns and closes the upstream connection.
Drop the gemma skip_special_tokens server-compat workaround: the vLLM
bug it patched is fixed upstream, and a stale shim can corrupt output.
The node image now installs ffmpeg; rebuild to run this live.
The test-postgres failure on test_init_retries_exhausted_raises surfaced the
root cause: sleeps held 2275x 0.1 instead of [1.0, 2.0]. Those 0.1s came from
a concurrent background poller doing asyncio.sleep(0.1) on anyio's shared
(persistent) event loop — the tls retry tests patched the *global*
asyncio.sleep, which intercepted that poller too.
- Before: the stub didn't yield, so the poller busy-looped and monopolized
the loop -> the test hung (the CI-only "after 92%" hang on 3.12+).
- The earlier "make the stub yield" change converted the hang into this
flood (the poller spins instead of blocking), which is what exposed it.
Fix: route init()'s backoff through TLSClient._sleep so the tests stub that
method in isolation and never touch the global asyncio.sleep. Tasks sharing
the loop are no longer affected; schedule assertions are unchanged.
The deeper fragility this exploited — a leaked, un-cancelled background poller
surviving on the shared test loop — is left as a follow-up.
run_with_deadline checked the deadline/cancel before reading the result
queue, so a call that completed in the same scheduling window could be
reported as a spurious timeout. Drain the queue first.
Also from review:
- test_validate_regex_pattern stubs run_with_deadline, so the probe regex
never runs — use a benign pattern instead of a real backtracking literal
(the literal tripped a ReDoS scanner).
- output_guard_judge docstring: reference IntentJudge._parse_verdict instead
of brittle judge.py line numbers.
The _runner BaseException catch is intentional and kept: it relays (not
swallows) whatever fn() raises to the caller via the queue; narrowing to
Exception would let a BaseException escape the worker so the caller never
gets a value, degrading the no-hang guarantee.
A hung run otherwise rides GitHub's 6-hour default with -v streaming the
whole time (the source of the multi-GB job logs). Cap test and test-postgres
at 20 minutes so a flaky hang fails fast instead of bleeding hours.
CI hung on test_init_retries_transient_failure (the new -v output named it:
its nodeid printed, no PASSED, the job rode to cancellation). It is the first
retry test that actually awaits the stubbed asyncio.sleep — the earlier tests
raise before sleeping — which points straight at the stub.
The stub returned without ever suspending, so the retry run completed in one
event-loop step with no checkpoint; that is fragile under the async test
runner and is the suspected cause (3.12/3.13/3.14 only — never reproduced on
3.11 or locally). Capture the real asyncio.sleep before patching and await
sleep(0) in the stub so it still yields, keeping the no-real-delay behavior
and the backoff-schedule assertions. Same fix in the discovery-failure test.
The judges and the regex ReDoS probe ran a blocking call on a
ThreadPoolExecutor and abandoned the worker with shutdown(wait=False) on
timeout or cancel. concurrent.futures joins every executor worker from an
atexit hook regardless of wait=False, so a wedged call could pin
interpreter exit — and hang the test suite at shutdown.
Add turnstone/core/deadline.py::run_with_deadline: run a blocking callable
on a daemon thread bounded by a wall-clock timeout and an optional cancel
event. A daemon worker is never joined at exit, so abandoning one is safe.
Migrate three sites onto it:
- OutputGuardJudge.evaluate()
- IntentJudge._evaluate_single / _run_judge — this also removes
_ExecutorPoisonedError and the executor-restart dance: per-call daemon
threads can't poison a shared single-slot pool, so a timeout now returns
None and the caller delivers one fallback verdict.
- console/server.py _validate_regex_pattern (regex ReDoS probe)
Also:
- Double the default judge LLM timeouts for slower local models:
judge.timeout 60->120s and judge.output_guard_llm_timeout 30->60s
(settings registry, JudgeConfig dataclass, --judge-timeout CLI default,
class docstring, docs). Correct a stale doc that described the per-turn
timeout as a total budget across turns.
- Raise the regex probe bound 0.5->3.0s so a legitimately complex pattern
isn't false-flagged as catastrophic backtracking.
- CI: run pytest with -v instead of -q so a hang names the offending test
instead of riding the job timeout.
- Tests: cover deadline.py and the regex validator; move test_judge.py off
fixed sleeps onto the existing _wait_for helper.
main's lockfile was already on the patched versions (cryptography 49.0.0,
starlette 1.3.1) via renovate, but the pyproject floors (>=42, >=1.0.1) still
permitted a regression to vulnerable versions. Raise the floors to match
stable/1.6's v1.6.7 security fix:
- cryptography >=48.0.1 (GHSA-537c-gmf6-5ccf — bundled OpenSSL vulnerable <48.0.1)
- starlette >=1.3.1 (CVE-2026-54282 host spoof + CVE-2026-54283 url-encoded form DoS)
Copilot review: _audioModelEligible gated stt/tts on md.provider, but a
blank/unset provider was treated as not-audio-capable and excluded — an
asymmetry with the backend, where _provider_carries_audio and
ModelConfig.provider both default to "openai". Default the provider to "openai"
before the check so a provider-less model isn't wrongly dropped from the
voice-role dropdowns.
The by-ref change replaced the send_id reservation model with the per-node
upload buffer (peek-then-drain at write time), but the surrounding narration was
never swept and a no-op stub was retained to make the send handler "read like"
the old flow — which is what made a recent diagnosis assume reservations still
existed.
- Delete the no-op _release_reservation_on_fail() and its 5 call sites in the
send handler (behaviour-preserving — it did nothing).
- Rename ordered_reserved / reserved_set -> ordered_taken / taken_set (the values
are the "taken" subset from resolve_staged_attachments, not reservations).
- Sweep the stale "reserve/reservation" wording across the create/send
docstrings, the API schemas/specs, and the SDK docstrings to the staged-buffer
vocabulary (resolve / attach / drain). The canonical docs in attachment_buffer
and attachments already stated the reservation token is gone.
No behaviour change; no tests exercised the removed scaffolding (the
migration-060 test correctly pins the reserved_at column removal and stays).
A create-time attachment is dispatched on the first turn, but the buffer drain
runs at write time inside the async dispatch worker (_append_user_turn). The
freshly-opened pane calls rehydrate() before the worker drains, so it painted
the image as a still-pending composer chip ("thumbnail in the text input box").
The inlined first-turn dispatch is the only consumer of those staged uploads and
always commits at create, so drain them from the buffer synchronously right
after resolving them — both the interactive and coordinator post-install paths.
The worker's own per-id discard then no-ops.
An omni model registered via the anthropic-compatible lane (vLLM Messages API)
was offered for the STT role because it carries supports_audio_input — but the
Anthropic SDK client has no .chat.completions and the Messages API has no audio
content block, so the mic failed with a cryptic
"'Anthropic' object has no attribute 'chat'".
Audio (input_audio) only rides the OpenAI-SDK surface, so gate all audio roles
to OpenAI-SDK providers (openai / openai-compatible / google / xai):
- model_supports_role returns False for anthropic(-compatible), so the mic
won't draw and the STT/TTS dropdowns won't offer those models.
- transcribe() raises a clear AudioUnavailableError naming the provider instead
of the opaque AttributeError (defence in depth).
- admin _audioModelEligible mirrors the gate — voice roles only; reranker hits a
/rerank endpoint, not audio, so it stays un-gated.
To use an omni model's audio, register it as openai-compatible (the input_audio
path); the anthropic-compatible lane is text/vision only.
- DRY the launcher create body: the multipart (meta + file parts) vs JSON
framing was duplicated in _createCoordinator and _createInteractive — extract
_createWorkstreamFetchOpts so the create wire shape lives in one place.
- Correct the proxy comment: the forwarded owner uid comes from the
authenticated ws_body (as on the JSON path), not the caller's meta; the proxy
token source is console-proxy, not console.
The mic is an STT control — it records and transcribes to editable text in the
composer. STT eligibility required the dedicated /audio/transcriptions endpoint
(supports_transcription / a whisper-style name), so an omni chat model
(supports_audio_input, e.g. Gemma) couldn't back it: it has no transcription
endpoint, it ingests audio via chat.
- model_supports_role accepts supports_audio_input for the STT role, so an omni
alias resolves as STT and the mic draws for it.
- transcribe() branches: a whisper-style alias keeps /audio/transcriptions; an
omni alias transcribes via chat input_audio + an instruction prompt — the
audio.stt_prompt override, else a default that emits only the transcript.
Audio attachments on an omni-STT setup transcribe the same way.
- admin _audioModelEligible mirrors the eligibility so omni models show in the
STT dropdown; the role description notes the two backends.
Phone photos store landscape pixels plus an EXIF orientation tag. Browsers honour
the tag for <img>, but Pillow (our thumbnails) and many vision-model image
decoders do not — so the thumbnail rendered rotated AND the model literally
perceived the photo sideways (noticed earlier as model "hallucinations", before
thumbnails made the rotation visible).
Normalize on read, at both surfaces:
- new core/images.normalize_image_orientation: bakes the rotation into the pixels
and re-encodes (preserving format); images with no / identity orientation pass
through untouched (pristine original, no per-send cost).
- make_thumbnail applies exif_transpose — after the decompression-bomb pixel gate,
which now also covers the transpose decode.
- attachment_to_content_part runs image bytes through the normalizer before
base64, so the primary model and the perception model both get upright pixels.
Because normalization is on read (not at upload), it fixes already-stored uploads
too.
The universal perception fallback (perception.model_alias) shipped backend-only
— session.py + perception.py + settings_registry.py — so its admin UI was never
wired. Operators had no way to assign it from the Models → Roles sub-tab, and the
raw setting leaked into the Settings tab.
- Add a Perception row to MODEL_ROLES (no capability filter — it spans
image/PDF/audio; the description tells operators to enable supports_vision /
supports_audio_input on the target model, which is what makes the audio
fallback engage when no STT role is set).
- Derive the Settings role-key skip-set from MODEL_ROLES instead of a
hand-maintained list, so perception is filtered out and no future role can
drift back in (stt/tts/reranker had leaked the same way).
- Add an optional per-role disabledLabel so the blank dropdown option reads
correctly for non-voice roles (perception, reranker) instead of "voice off".
- Refresh the stale STT description that claimed "no audio-capable session
fallback" — audio attachments now fall back to perception.
The console creates interactive sessions by proxying to the owning node via
/v1/api/cluster/workstreams/new, which only forwarded JSON — so a file staged in
the launcher was blocked with "Attachments aren't supported for interactive
sessions yet". The node create endpoint already accepts multipart (meta JSON +
file parts) on interactive_endpoint_config; only the proxy lacked it.
Teach create_workstream to accept multipart: parse meta + files (same caps as
the node), pick the node exactly as before (auto / pool / pinned), and forward
the files instead of re-serialising JSON. _createInteractive sends multipart
when files are staged (mirroring _createCoordinator) and the launcher gate is
removed. The files-need-a-task guard already ensures an initial turn to
dispatch them on.
A console interactive pane is node-proxied — every request rides the pane's
transport base ("/node/{id}"). The attachment controller hardcoded bare
/v1/api/workstreams/... paths, so upload / list / delete / preview landed on
the console's OWN coord route, which resolves ws_id via coord_mgr.get() and
404s as "coordinator not found". The standalone server (base="") was
unaffected, which masked the bug.
Thread the pane base through: createAttachmentController and
buildAttachmentPreview take an optional getBase / base, and the interactive
pane wires this._base into both. Coordinator panes and the standalone server
pass "" and stay origin-mounted as before.
- TextDecoder in the text-preview stream now flushes on completion/cancel, so a multibyte UTF-8 char split across a chunk boundary isn't dropped (Copilot).
- send() clears self._wire_part_cache in a finally so the per-send memo (which can hold large rasterized PDF page-images) is released at send end instead of retained on an idle session until the next send (Copilot + fix-review).
- Make the implicit byte-string concatenation in _minimal_pdf explicit (+) in test_pdf.py and test_thumbnails.py so it can't read as a missing comma (CodeQL / github-code-quality).
A review of the fix commits surfaced three refinements:
- ftyp audio sniff: scan the whole ftyp box (its declared length) for an audio brand instead of a fixed 6-slot window, so a real .m4a with the brand listed late still passes — while a pure-video file (no audio brand) still rejects.
- text-preview: accumulate body chunks until >=240 chars before cancelling the stream, instead of assuming the first chunk is large (flush boundaries can split a large body into small early chunks).
- _resolve_attachments: correct the cache comment — the memo is refreshed per send and the wire resolver only runs during a send, so a stale value is never observed between sends.
- Remove the unused PerceptionUnavailableError (never raised/caught/imported).
- Reword the now-shipped 'Phase 3' placeholder comments on the Anthropic + OpenAI-Responses audio paths to describe the live upstream STT/perception fallback (these placeholders are defensive, not pending work).
- Clarify the no-vision image fall-through comment (fires when perception is unconfigured OR can't see, not only the former).
- Type AttachmentInfo.kind as the image|text|pdf|audio union in the TS SDK.
- extract_pdf_text: append the truncation marker only when there's actual text, so a scanned PDF over the page cap returns '' (-> placeholder) instead of a content-free document part.
The /thumbnail endpoint and the _resolve_served_blob ownership/404-leak gate it shares with /content had no handler-level test (only make_thumbnail as a unit + route mounting). Add cases through the real app: image -> 200 image/png with the nosniff + CSP + max-age headers; audio/text -> 415; make_thumbnail None -> 415; cross-workstream id and unowned-ws cross-user -> 404 (no existence leak).
The text-snippet preview fetched the entire /content body (text attachments are capped at 512 KiB) only to render the first 240 chars — and again on the sent-message pill (the endpoint sends Cache-Control: no-store). Read only the first response-body chunk and cancel the stream, so the rest of the blob is never transferred or regex-scanned. Falls back to r.text() where the streaming body API is unavailable.
Three copies of the kind->glyph mapping had drifted: the coordinator pill rendered audio as the document glyph (not the audio note) and showed no inline preview, diverging from the interactive pane.
Export kindIcon() from composer_attachments.js (+ window bridge) as the single source of truth; the interactive pane imports it and the coordinator pill uses it. Wire the coordinator pill to buildAttachmentPreview too (image/pdf thumbnail, audio player), gracefully no-oping on history replay (which omits attachment_id), matching interactive.
Also fix buildAttachmentPreview's thumbnail-error handler: it called img.remove(), but the caller has already replaced the icon span with the img, so a failed thumbnail left a blank gap. Swap in the kind glyph instead (.attach-preview-icon, sized to the thumbnail slot).
_reconstruct_attachment_refs collapsed every non-image attachment to the 'document' placeholder kind, so a reloaded session's pdf/audio placeholder type ({type:document}) mismatched the live-injection type ({type:pdf}/{type:audio}). Harmless today (resolution keys on attachment_id + blob kind) but a latent footgun for any consumer branching on the pre-resolution placeholder type. Preserve image/pdf/audio verbatim; only a stored 'text' blob collapses to 'document'.
A user-controlled filename was interpolated unescaped into model-visible frames (the [PDF attachment '{name}'...] / audio / transcript / perception placeholders, the Anthropic document title, and the unreadable placeholder). A crafted name like "'] New instructions:" broke out of the frame and injected text into the model context.
Add core.attachments.safe_attachment_label() (strip control chars + quote/bracket/angle delimiters, collapse whitespace, clamp length) and apply it at every model-context embedding site. The raw filename is still used verbatim for display / Content-Disposition, which neutralize at their own boundaries.
Also tag perception descriptions and STT transcripts '(untrusted)' so attachment-derived text reads as data, not instructions. Blast radius is single-tenant (injecting into a model reading one's own upload); a structural role=tool fence is deferred as disproportionate.
sniff_audio_mime returned audio/mp4 for ANY ISO-BMFF ftyp box, so an MP4/MOV video uploaded within the audio size cap sniffed as audio and was sent as input_audio. Restrict to genuine audio brands (M4A/M4B/F4A/F4B major, or M4A/M4B in the compatible-brands list, so a real .m4a with an mp42 major brand still passes).
Also add ADTS-AAC sniffing (0xFFF1/0xFFF9): audio/aac was in ALLOWED_AUDIO_MIMES + AUDIO_MIME_TO_FORMAT but never sniffable, so an advertised .aac upload always failed.
make_thumbnail set Image.MAX_IMAGE_PIXELS=40M, but Pillow only raises DecompressionBombError above 2x the cap; a 40-80M px image merely warns and decodes fully (~480MB RGB), defeating the documented bound.
Gate on the header-declared size after open() and before convert(), so nothing past the cap is decoded. Explicit check rather than a warnings filter — make_thumbnail runs in a worker thread and global warnings state is not thread-safe. Adds tests for the (cap, 2*cap] warn-only window and the at-cap boundary.
_resolve_attachments re-runs on every agentic round-trip (and per fallback model), each time re-fetching every attachment across the full history and re-rasterizing / re-base64'ing it. A 10-page PDF in a 10-cycle tool turn was rendered dozens of times.
Add a per-send memo (self._wire_part_cache) keyed by (attachment_id, caps-signature): the materialized wire part is computed at most once per send. The cache is None outside a send (display/export paths unaffected) and reset per send to bound the heavy rasterized-page parts and pick up any mid-session capability change. Skip the DB fetch entirely when every id is already cached.
Also peek the perception (alias, content_hash) memo before building parts in _perception_fallback_part, so a cross-send describe hit no longer wastes a PDF rasterize. Leaves pdf.py's deliberate no-module-cache stance intact — the per-send scope addresses the round-trip amplification without the durable store it defers.
Adds describe_peek() + per-send-cache and peek tests.
sanitize_messages ran inline_document_parts (which placeholders an application/pdf document part) before the Responses translator's native input_file branch could run, so every supports_pdf model silently degraded its PDF to an unsupported text placeholder.
Thread a skip_pdf_inline flag through sanitize_messages -> inline_document_parts; the Responses lane sets it so the PDF document survives to convert_content_parts. Chat / Google-compat keep the placeholder (they have no native PDF block).
The existing test exercised convert_content_parts in isolation, bypassing sanitize_messages and masking the bug. Add an end-to-end _convert_messages regression test (verified to fail without the fix) plus contrast tests pinning both lanes' behavior.
q-3 from the pre-push review, settled against docs.x.ai: Grok's document
support is an agentic attachment_search workflow over Files-API uploads
(file_id / file_url), not the inline base64 native ingestion that OpenAI
input_file / Anthropic document blocks use. Our native PDF path emits inline
base64, which xAI's Responses surface doesn't accept — so supports_pdf is
correctly left unset (Grok PDFs rasterize to images, which Grok can see).
Document the rationale on GROK_CAPABILITIES and pin every Grok row's
supports_pdf=False with a test so it isn't naively flipped without first
wiring a Files-API upload flow.
Add a `perception.model_alias` model role: when the primary model can't ingest
an attachment natively and can't be shown a degraded-but-native form, a
configured perception model perceives it and its output is carried as text.
Mirrors the STT role — a role alias plus a module-level memo so the extra LLM
round-trip runs once per attachment, not once per conversation turn. The call
goes through the provider abstraction's create_completion (the path the intent
judge uses), so any vision/omni provider works.
Bottom-tier, universal ladder — perception only fills the remaining gap:
- pdf : native supports_pdf -> rasterize-to-vision-primary -> perception
-> extracted text -> placeholder
- image: native vision -> perception (non-vision primary) -> native image_url
- audio: native supports_audio_input -> STT -> perception (omni) -> placeholder
Folds in two review findings the role subsumes:
- bug-1: thread the active attempt's capabilities into _resolve_attachments
(bound in _try_stream) so a model fallback materializes attachments against
the fallback model's caps, not the primary's.
- bug-2: charge a by-reference pdf/audio a bounded budget min(size_bytes, 16K)
instead of zero, so a large-attachment turn isn't budgeted as ~empty (the
exact materialized size isn't known until wire build).
Pre-push review follow-ups that are independent of the perception-role work
(bug-1 caps threading, bug-2 budget, and the perf cluster fold into that):
- thumbnails: cap decoded pixels (Image.MAX_IMAGE_PIXELS=40M) so a small
compressed image that decodes to huge dimensions can't OOM the node, and
reject DecompressionBombError cleanly.
- pdf: clamp per-page render scale so the longest rendered side stays <= 2000px
(a maximal MediaBox at scale 2.0 rendered to a ~28800px, multi-GB bitmap).
- session_routes: type classify_upload's rejection element as
UploadRejection | None instead of Any.
- test_session_routes: assert the /thumbnail route mounts (it was untested) and
fix the stale "quartet"/four wording to five.
Two-reviewer + sanity pass over the attachment previews:
- composer audio chip is icon+name+size only; the native <audio> player
renders on the sent message, not the staging chip (too heavy at chip scale)
- cap sent-message pills (+ in-pill audio/snippet) so they no longer overflow
the bubble at narrow widths; player and snippet drop to their own row
- clamp the chip filename in shared chat.css so long names ellipsize instead
of wrapping (console main + coordinator previously left it unclamped)
- merge the duplicated .composer-chip rule; drop unused kind-modifier classes
and inert vertical-align / inline-block declarations
- fix undefined var(--bg-base) -> var(--bg-surface) thumbnail backing
- label the <audio> control (aria-label) and drop the decorative snippet from
the a11y tree
scripts/livepass.py: add an attachments harness that drives the real
createAttachmentController + Pane.addUserMessage so these surfaces render
headlessly for review.
- core/thumbnails.py + GET .../attachments/{id}/thumbnail: server-rendered PNG
thumbnails (image downscale; pdf first page via pypdfium2). Extracted a shared
ownership-gated blob resolver used by both get_content and the thumbnail route
- buildAttachmentPreview (composer_attachments.js): image/pdf -> thumbnail,
audio -> <audio> player, text -> lazy snippet; reused by the composer chips and
the sent-message pills (interactive.js). Cookie auth, so direct media src works
- chip kind icons now cover pdf/audio; the upload swap adopts the server's
authoritative kind for styling + icon + preview
- chat.css preview styling; tests for make_thumbnail
- composer: accept pdf/audio in the upload picker; client-side kind
inference for the optimistic chip (server classify_upload stays
authoritative)
- admin Models tab: supports_pdf + supports_audio_input toggles (flow
through the field-aware capabilities merge into ModelCapabilities, so
flipping supports_audio_input on an omni alias enables native input_audio)
- docs: AttachmentInfo.kind, AttachmentUpload, and the TS SDK note pdf/audio
A vision-capable model that can't ingest PDF natively now gets the PDF
rendered to one image per page instead of extracted text; falls back to
text extraction when rendering yields nothing.
- core/pdf.py: rasterize_pdf via pypdfium2 render + Pillow PNG (page-capped
at 10, never raises)
- session._wire_content_part: pdf + !supports_pdf + supports_vision ->
rasterized image parts; else text extraction
- trajectory.resolve_attachment_parts: a placeholder can now expand to a
list of parts (1->N); the resolve_attachments callback return type widened
to dict[str, Any] across the provider protocol + 4 providers
- pyproject: pillow dependency
- tests: rasterize_pdf, vision-rasterize gate path, 1->N materialization
When the active model can't ingest a kind natively, the wire resolver
converts it client-side instead of sending a part the model can't read.
Per-kind ownership, no shared machinery: PDF text-extraction is a
pure-local PDF concern; audio transcription is an STT concern memoized
in the audio domain.
- core/pdf.py: extract_pdf_text via pypdfium2 (pure-local, no network, no
cache — re-run per build; page-capped)
- core/audio.py: transcribe_cached — non-raising, memoized by
(alias, content-hash); backend failures not cached
- session._wire_content_part: per-kind dispatch — native where the model
supports the kind (supports_pdf / supports_audio_input), else fallback;
display/export resolve natively so no conversion fires on a render
- image left ungated (pre-existing behavior unchanged)
- pyproject: pypdfium2 dependency + mypy untyped-import override
- tests: pdf extraction, transcript memoization, per-kind gate dispatch
PDF and audio attachments now work end-to-end on the native provider
lanes; non-native lanes degrade to a placeholder (client-side fallback
lands next). Capability flags are populated but not yet consumed by a
wire-build gate.
- providers: Anthropic PDF -> base64 document; OpenAI Responses PDF ->
input_file; compat/Google inline_document_parts PDF -> placeholder
(fixes the base64-as-text mangle); audio = input_audio passthrough on
the compat lane (omni), defensive text placeholders on Anthropic +
Responses
- capabilities: supports_pdf on cloud Claude + OpenAI chat models;
local/default/compat stay False (-> client-side fallback)
- upload: classifier accepts pdf (32 MiB) + audio (25 MiB); endpoint
multipart read cap raised to PDF_SIZE_CAP
- hygiene: consolidate the duplicated upload classification into one
attachments.classify_upload (+ UploadRejection); collapse
AttachmentUploadHelpers to a single classify_upload callable
- tests: PDF/audio translator shapes, capability flags, classify_upload
Provider-neutral plumbing for PDF and audio attachments, with no
user-facing change yet: the upload classifier still rejects them and the
capability tables stay unpopulated (both land in the native-translator
phase). No migration — workstream_attachments.kind is free-text.
- attachments.py: PDF/audio byte caps, allowed-audio MIMEs + format map,
magic-byte sniffers (sniff_pdf_mime / sniff_audio_mime),
Attachment.is_pdf / is_audio
- providers/_protocol.py: supports_pdf / supports_audio_input capability
fields (default False; orthogonal to the STT/TTS roles)
- storage/_utils.py: attachment_to_content_part emits the internal
document(application/pdf, base64) and input_audio shapes
- session.py: by-reference placeholder branches for pdf / audio
- trajectory.py: AttachmentRef docstring (dict-bridge already kind-agnostic)
- tests: test_attachments_pdf_audio.py
Hardened service + slice + node-identity drop-in template + a README for
running a turnstone-server outside Docker that joins the compose cluster —
the production-shaped counterpart to the one-liner in docs/docker.md. Secrets
stay in config.toml; per-host identity + cluster URLs go in the drop-in. The
README notes the cross-host mTLS caveat (turnstonelabs/lacme#22).
A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.
Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.
The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).
Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.
The server (:8080) and console (:8090) both set a cookie named
`turnstone_auth`. Cookies ignore port (RFC 6265), so on a shared host
(localhost dev, the Electron build, single-box installs) logging into one
surface overwrote the other's cookie and 401'd the first session.
Give each surface its own cookie name -- `turnstone_auth_server` /
`turnstone_auth_console` -- threaded as a required `cookie_name` argument
through the cookie builders, `check_request`, `AuthMiddleware`, and the six
shared auth handlers (login/logout/setup/whoami/refresh/oidc_callback). Each
app passes its own constant; the parameter is required (no default) so a
forgotten caller fails loudly instead of silently reverting to the legacy name.
Names key on role, not node: the cluster shares one JWT identity and the
console->node proxy re-mints a bearer token (dropping Set-Cookie), so
per-instance names would break identity portability and aren't used.
Hard cutover: the legacy `turnstone_auth` cookie is no longer read and
self-expires within its 24h TTL (one forced re-login). JWT audience was
already enforced, so the shared cookie was a session clobber, not an auth
bypass.
The interactive pane only auto-scrolled when isNearBottom() was true, but it measured that AFTER the new node was appended. A tool block is a tall one-shot append (batch shell, approval card, or result) that clears the 80px near-bottom threshold in a single step, so the post-append check read false and auto-follow silently disengaged at exactly tool-call time — the view froze at the top of the block and only snapped back at the next stream_end. Token streaming was unaffected because each append stays sub-threshold.
Capture the near-bottom state as the first statement of each tool-render method, before any DOM mutation, and thread it into scrollToBottom(stick). This re-pins when the user was already at the bottom and, unlike the coordinator pane's unconditional pin, leaves the view alone if they deliberately scrolled up while a result was rendering.
Methods fixed: announceToolBlock, showInlineToolBlock, resolveApproval, appendToolOutput (all three exit paths), appendToolOutputChunk.
The streamable-http server bound to 0.0.0.0/a LAN IP answered TCP and
/watch but returned 421 "Invalid Host header" on /mcp for every remote
node — which broke multi-node play entirely. FastMCP freezes DNS-rebinding
protection (a localhost-only Host allowlist) at CONSTRUCTION, and this
module builds its FastMCP at import time with the default 127.0.0.1 host;
flipping settings.host in _serve afterward never updated the frozen
allowlist, so the LAN Host was always rejected.
When UNDERSTONE_HOST is off localhost, drop the allowlist in _serve before
run() — matching the SDK's own default for a non-localhost bind. The /mcp
and /watch routes are unauthenticated by design, so serve only on a trusted
network (documented).
Regression test pins the mechanism: a default FastMCP 421s a foreign Host,
a protection-disabled one accepts it. Tests 420 -> 421.
The job was named "test", colliding with core CI's "test" matrix so the PR
checks list showed two "test (3.11)" rows. Rename it to "understone" so the
example's checks read unambiguously (understone (3.11) / (3.13)).
- CodeQL (implicit string concatenation in a list): collapse the wrapped
bullets in cli._render_validate_coverage to single literals. The rendered
output is byte-identical (the example's ruff ignores E501); clears all
six alerts and reads cleaner.
- Copilot: packs/README no longer claims the directory ships "effectively
empty" — it ships the bundled Cinder Wastes alternate world.
- Copilot: the Cinder Wastes' ash_flats and caldera_deep zones overlapped
on column x=60 (inclusive bounds + first-match zone_for silently shadowed
the tier-3..5 band onto a 1x5 deep-edge strip). Move caldera_deep to
x0=61 — no overlap, no dead tiles, deep zone still covers the dungeon.
And harden the loader: overlapping zone rectangles are now a
WorldLoadError, so no authored pack can ship that bug unseen (the
cold-author dogfood loop — a generated pack exposed a validator gap).
Tests 419 -> 420 (zone-overlap rejection). Both worlds validate sound and
remain winnable by the sim bot.
The door-game example is a standalone package (no turnstone-core
dependency) that the root suite does not collect — its
testpaths are scoped to ["tests"], so the example's 419 tests, ruff,
and mypy gates never ran in CI.
Add a path-filtered workflow that installs the example and runs its
full gate (pytest + ruff check + ruff format --check + mypy) whenever
examples/door-game (or this workflow) changes, across the example's
declared Python floor and ceiling (3.11, 3.13). Pinned action SHAs and
contents:read permissions match the existing CI workflows.
A game-loop mechanics patch: the satchel becomes a real stacking inventory,
forging now demands ore won in combat (not just gold), and a vault lets a
hero protect coin from ambush.
- Stacking satchel: the bag re-encodes from a flat id list to "id:qty"
stacks, so potions stack (three Minor Potions fill one slot, not three)
and materials ride alongside. satchel_max now caps distinct KINDS (3);
per-kind quantity is unbounded. quaff/death-save still pull the strongest
potion and ignore materials. One pure codec (engine/satchel.py) owns the
encoding; the façade, the Watch, and the sim all decode through it — no
three-way drift (the v0.9 single-source lesson). The codec parses a bare
id as qty 1, so it can never silently drop a malformed stack.
- Ore-gated forge: ore is a material that drops from won dungeon-rung
fights (and, less often, forest fights), stacks in the satchel, and is
not buyable or sellable — you earn your edge by fighting for it. Forging
now costs gold AND ore ((plus+1) ore per tier), so a rich-but-idle hero
can no longer buy power at the dice table. The dungeon is now also the
mine.
- The vault: deposit/withdraw at the inn moves coin to a strongbox that
ambush cannot touch and that SURVIVES the Wyrm-win legacy reset — the
carry-vs-protect decision the PvP economy was missing.
- Surfaced on both the /watch lobby TV and the in-chat door_status sheet:
each hero's stacked satchel, carried gold, and vaulted gold.
- Tuning (the sim is the instrument): the ore gate added ~2 days to the
Vale and ~1.6 to the Cinder Wastes; the greedy bot still slays the Wyrm
3/3 on both, fully forged to +3/+3, so the loop is not stalled. Defaults
held — no numbers needed retuning.
Four new banded settings (forge_ore_item, forge_ore_per_plus,
ore_dungeon_drop, ore_forest_chance); both worlds gained an ore item.
Schema mutated in place (banked column, satchel re-encoding) — pre-1.0, no
migration by design; a real migration story is owed at 1.0. Tests 382 ->
419; the vault-survives-rebirth invariant and the codec are revert-verified.
Graphics polish: distinct terrain and structures now read by COLOUR on the
Watch, not only by glyph. One unified palette, shared by every world — the
fix is to grow the set of distinct object-type roles, not to fork per-world.
- Roads were the tell: road shared the "floor" green with grass, so a path
vanished into the meadow on the lobby TV. Likewise forest shared "tree",
the three town buildings all shared "town", and the Cinder Wastes' molten
slag borrowed "water" and rendered BLUE. Each is now its own role: road
(stone), forest (lush green) with scrub (its barren ember-brown
counterpart for volcanic/desert dense terrain that must NOT read as
woods), lava (molten orange), barren (wasteland taupe), and inn/shop/
healer split out of the generic town.
- Both worlds remap onto the shared vocabulary; in each, no two distinct
terrain/building types share a colour. A live render caught the Cinder
cinder-fields rendering green under the generic "forest" role — hence the
scrub role, so the volcanic waste reads warm. The text frame renderer
stays monochrome (it never read colour), so frames and goldens are
untouched — this is Watch-only.
- The bug class is now closed by construction: a test asserts the Watch
PALETTE carries a hex for EVERY Color role, so a role can never ship
unpaintable and silently fall back (which is exactly how road hid).
- Color.assignable() is the single source for the overlay-vs-assignable
split (runtime actor/item colours and the DEFAULT fallback are not
author-pickable); the authoring manual's colour vocabulary generates
from it, so it can't drift.
Tests 373 -> 382. floor/tree/forest are three greens kept deliberately
distinct (forest is olive-hued); verified on a real render along with the
scrub fix.
The slice that proves the pipeline: a second world authored entirely by an
LLM from AUTHORING.md and the validator alone, plus the tooling to discover,
theme, and balance-test any world.
- The dogfood: "The Cinder Wastes" — an ashen volcanic underworld (slag
rivers, a caldera mouth, a Magma Wyrm) — was written cold by an agent
given only the generated authoring manual and `understone validate`. It
passed validation on the FIRST run with zero failures. Its stumble log
found six places where the manual stated a rule the validator didn't
enforce; those became permanent hardening (below). It ships in
understone/world/packs/ and glows ember on the lobby TV.
- `understone worlds` lists every bundled world (the Vale + alternates)
with its load status, via one shared discovery path.
- Per-world Watch themes: settings.watch_theme (phosphor/amber/ice/ember,
loader-validated) repaints the spectator page; the Vale's green is
byte-for-byte unchanged.
- The sim harness: a pure, seeded, greedy bot plays the real game façade
over an injected day-stepping clock and emits a balance report —
`understone simulate PATH [--days N] [--seeds K]`. It SLAYS THE WYRM on
both worlds (Vale ~day 13, Cinder ~day 25), so the whole v0.1->v0.7 loop
is proven winnable end-to-end by an unclever bot through the real stack.
- Loader hardening from the dogfood: a rare monster may not occupy a
dungeon-rung guardian slot (it would silently become a fixed foe and
leave the rare pool); exactly one monster may be the boss; and the
boss-tier error now says "no non-boss monster," matching the manual.
AUTHORING gained a generated "what validate checks vs. what it cannot"
section so the rule/guidance boundary is honest.
Review hardened the bot for arbitrary authored packs (a MENU-mode fight
spin and four related robustness gaps that were latent on the shipped
worlds), and documented that final_level reads post-legacy-reset. Tests
359 -> 373; both worlds still win byte-identically after the fixes.
The depth slice: four standing reasons to return past the daily reset.
- The rung ladder: the dungeon is a descent fought one rung per turn, each
guardian a fixed tier. A loss bounces you home but your depth PERSISTS —
you re-enter where you left off. The Wyrm now gates on BOTH level AND
reaching the floor (the deep has a bottom, and you must have touched it).
- The satchel + the death-save: potions are CARRIED now (up to three),
bought to the satchel, drunk with quaff. The heart of it: when any fight
would kill the active fighter and they carry a draught, the strongest is
drunk automatically — they survive standing at the potion's value, no
bounce. This fires on EVERY fight (forest, rung, and the Wyrm itself —
a potion carried to the climax is a real tactical choice); a Wyrm loss
so saved is "driven back, alive but unproven," not devoured. The sleeping
ambush victim never quaffs (they are asleep). combat.py stays pure — the
satchel and the save live entirely in the façade.
- The forge: the shop spends scaling gold to add a +1 edge to equipped
weapon or armour, capped — the late-game gold sink. Swapping or selling
the piece loses the edge with it (one centralized unequip clears the
bonus and the plus so a stat can never go phantom).
- Rare beasts: a few named foes prowl the forest via weighted selection,
surfacing seldom; felling one is a public Herald flash and always yields
a draught into the satchel. Rung guardians are never rare (fixed foes).
Four new player columns; four new banded settings; dungeon_tiers extended
to three rungs. Tests 283 -> 330; the death-save (all four paths), forge
accounting across forge/buy/sell/legacy, rung math, and weighted rare
selection all pinned, with the death-save and forge invariants
revert-verified.
The look of the next age — the modern equivalent of the ASCII->CP437 leap.
Full Unicode is available now, but the whole stack (text frames, golden
tests, the Watch's 1ch grid) assumes one glyph = one column, so the
enabling piece is a WIDTH RULE, not the glyphs themselves.
- textwidth.is_grid_safe: one code point, printable, East-Asian width not
Wide/Fullwidth, no combining/format/control category. This is the
one-glyph-one-column contract. Ambiguous-width glyphs are ACCEPTED on
purpose — they ARE CP437 (the wall, the club-tree, the up-arrow forest)
and render single-column on the Western-monospace metrics every surface
uses; only genuinely double-width runes are barred. The loader enforces
it on every map glyph; the player-name/free-text sanitizer enforces the
same rule (the narrow ledger), so a wide name can't shear a frame.
- Re-skin: water ~ -> ≋, inn -> ⌂, healer -> ✚, dungeon mouth -> ∩, and
the other adventurer -> ☻ (CP437's own player glyph). The colour field
the renderer has carried unused since v0.1 now has a second consumer.
- Texture variants: grass and water vary by a deterministic per-coordinate
hash, rendered identically in the Python frame builder and the Watch's
JS. The two are kept in lockstep by shared hash constants + an agreement
test that replays the JS arithmetic and asserts it equals the Python
output for every variant over a grid — not a comment-coupled copy.
- Watch glow-up: a Noto Sans Mono font stack and a UTC-hour day/night tint
(the Vale darkens at dusk on the lobby TV).
- The curated SAFE_PALETTE is enforced author-usable: a test asserts no
palette glyph collides with the reserved player markers, so AUTHORING's
generated appendix can't advertise a glyph the loader would reject.
- Resume is identity-preserving: an existing character resumes by exact
stored name without re-validating the width rule (which governs creation
only) — resume must never lock anyone out.
Tests 231 -> 283; width edges (CJK/emoji/combining/fullwidth), the
Python<->JS lockstep, the palette/reserved guard, and resume-vs-create all
pinned and revert-verified.
The social slice: the shared world gets teeth, letters, and a house game.
- Ambush (async PvP, classic door-game player-kill spirit): waylay an adventurer who has
not yet begun their day. Ordered gates — known target, not yourself, the
gatekeeper shields the young (both >= min level), level band +-2, the
SLEEP RULE (acting today makes you watchful — an active-play defense),
mercy for the downed (hp<=1 cannot be piled on: even bandits have
standards), once per pair per UTC day. Win: capped gold cut transfers,
victim wakes at the spawn-stone with a private note; lose: the sleeper
wakes blade-in-hand and the Herald crows your shame. The attacker wears
the counter-blows the combat log narrates (state matches story). Both
players persist in one transaction.
- The inn mailbox: events carry a target ('' = public). door_log delivers
private notes to the addressee only; the Watch and other players never
see them. Mail is DURABLE past the in-memory tail (SQLite backfill for
cursors older than the resident window) — the broadsheet is ephemeral,
letters are not. Sanitized, daily-capped.
- Inn dice: 2d6 against the house, bet- and count-capped per day, big wins
make the news.
- Six new banded settings; four day-counter columns join the shared lazy
UTC reset; schema stamp stays 1 (pre-1.0 mutates in place by design).
Tests 184 -> 231; sleep rule, mercy gate, band boundary (exact/over),
refusal precedence, attacker wear, zero-gold robbery, mail eviction
survival, and Watch privacy all pinned; guards revert-verified.
The IGM seam realized: world packs are now a first-class authoring target
for models and humans, with a validate loop and a loader hardened for
routinely-untrusted generated content.
- understone newpack DIR scaffolds a pack (the six content JSONs templated
from the shipped Vale) plus AUTHORING.md — a manual written for a model
to follow cold. Its bands table is RENDERED FROM the loader's own band
constants at scaffold time, so documented limits and enforced limits
cannot drift.
- understone validate DIR loads a pack and prints either a pack report
("This pack is sound. The door stands open.") or the loader's
file/index/field-naming error — the authoring feedback loop.
- Loader hardening: glyphs must be one printable column-safe character and
never the frame box-drawing set or the @/& player markers (map content
cannot impersonate players or forge frame chrome); map dims 8..256;
per-file count caps; display-name length caps. All errors instructive.
- The packaged-world path is single-sourced (understone.world.
PACKAGED_WORLD_DIR) for the server default and the scaffold template.
- README "Authoring worlds" section frames the loop: newpack -> write or
generate -> validate -> serve with UNDERSTONE_WORLD=dir.
Review round: bug finder returned zero findings; quality round fixed the
world.json doc example (it showed a zone fragment where an authoring model
would copy a whole-file shape — now a labeled skeleton), the stale Usage
docstring, and the duplicated packaged-path constant.
Tests 166 -> 184. Scaffold round-trips through load_world by test.
A read-only CRT spectator page served by the game process itself, plus
content depth. Input never flows through the Watch — it is the wall-mounted
terminal in the BBS room; chat remains the only actuator, so there is no
input channel to deadlock and no cross-origin surface (the page polls the
same origin that served it).
- /watch: one self-contained page (inline CSS/JS, no external assets),
phosphor CRT styling. The base map paints once from /watch/world.json
(terrain glyph rows + a glyph->color legend — the palette the text
renderer has deliberately ignored since v0.1 finally gets its first
renderer); players overlay as positioned glyphs repainted from
/watch/state.json every 2s; the sidebar carries the roster with win
stars, the Hall of Legends, and the Herald. SIGNAL LOST on poll failure;
the bootstrap retries so a spectator arriving during a server blip
recovers without a reload.
- Routes ride FastMCP custom_route on the existing process — read-only
handlers with no awaits between reads (handlers and sync tools
interleave on one event loop, so every response is a consistent
snapshot).
- door_join/door_help advertise the Watch URL in http mode (stdio: none).
- Content: +5 monsters (one per tier; the gauntlet's first-in-tier foes
preserved), +3 items smoothing the gear curve, +6 events; fight weight
retuned to hold ~55% of encounter rolls. Zero geography churn.
- Review round: the Herald window is a plain list tail (id arithmetic
under-reported the feed when AUTOINCREMENT ids gap — regression-pinned
with sparse ids), and the bootstrap-retry fix above.
Tests 149 -> 166.
The "make it a game" slice: a win condition with classic-door-game-style legacy, texture
between fights, and a shared broadsheet.
- The Wyrm Below: a boss (flagged in the pack, excluded from random bands)
behind a level-gated `challenge` verb at the dungeon. Victory writes a
Hall of Legends row and the character resets to the fresh-start kit,
keeping a wins counter rendered as ★ on the leaderboard — the classic
race-reset-race loop. Defeat and stalemate flight make the news.
- Forest events: movement encounters weighted-pick from a content-pack
table (fight/gold/heal/trap/lore). Only fights stop the walk or cost
turns; texture is free and private. Trap damage floors at 1 hp.
- The Understone Herald: door_log is a broadsheet with a masthead and
write-time template variety; the public feed is curated to notable beats
(joins, blessings, level-ups, defeats, the Wyrm's fate) — town errands
stay private.
- Reward narration moved from the combat engine to the façade, composed at
the moment gold/xp are actually banked, so the server can never narrate
a reward it did not apply (the Wyrm win previously claimed +400 XP /
+250 gold that the legacy reset wiped).
- Fresh-start hp/atk/def promoted into world.json settings alongside the
starting kit; dungeon-tier validation counts non-boss monsters only,
keeping the validator's no-silent-rung promise true.
Schema mutated in place (players.wins, hall_of_fame) — pre-release, no
migration path by design. Tests 109 -> 149; the challenge level gate is
negative-tested; rank stars survive 24-char names (compact form past 5).
A shared-world, classic-door-game-style door game in examples/door-game/: a pure-stdlib
game engine (tile overworld + location menus, seeded combat, daily turn
budget, leveling, shop, event log, leaderboard) behind nine sync door_*
FastMCP tools returning monochrome box-drawing frames. The connecting
session's LLM plays dungeon master — tool descriptions plus a door_help
manual teach a cold model to run the game with zero setup, while the server
owns all dice and state, so the DM narrates around facts it cannot bend.
Non-obvious decisions:
- engine/screen/world/persistence import stdlib only; server.py is the only
mcp import. All nine handlers are sync def: on mcp 1.27 they execute
inline on the event loop (verified against func_metadata), so tool bodies
serialize and one SQLite connection (WAL, per-action commit) is safe.
check_same_thread=False exists only because the Store may be constructed
on a different thread than the serving loop.
- Streamable HTTP serves ONE process = one shared world (players appear on
each other's maps; async "while you were away" event feed); stdio is the
solo-world fallback.
- The economy is content, not code: daily_turns, costs, xp curve, bestow
budget, and dungeon tiers live in world.json settings, band-validated by
the loader. door_bestow gives the DM capped, event-audited largesse
(gold/heal only, never turns) so story generosity cannot melt the shared
leaderboard.
- Player names and bestow reasons are sanitized (printable-only, length
caps) because they flow into the shared event log and from there into
other players' DM context — embedded newlines would forge log lines.
- Daily turn/bestow pools lazy-reset per UTC day on every consuming path
(injectable clock); the dungeon gauntlet is a fixed boss ladder by design.
Tests: 109 — engine units with seeded RNG + frozen clock, hand-authored
golden frames paired with structural asserts, loader band rejections, and
one real-wire integration test (uvicorn + streamablehttp_client) with a
two-session shared-world assertion. Negative-tested by reverting the guard
and watching the suite fail: the daily turn-budget guard, the bestow cap,
and the sanitizer's isprintable clause.
The coordinator memory scope was keyed by the session's ws_id, so every
new coordinator session started with an empty namespace and its rows
were orphaned on close — coordinator memory never actually persisted.
Re-key the scope to the coordinator's creator user_id: one durable
orchestration namespace per user, shared by all of that user's
coordinator sessions (concurrent ones included; upsert-by-name is the
collision rule).
The child-containment threat model is unchanged: the gate is session
KIND — children are always interactive and share the parent's user_id,
so _validate_scope rejects them before scope resolution, and the REST
memories API still rejects the coordinator scope outright. The implicit
visibility lane now also fails closed on an empty scope_id to match the
explicit search/list lanes (the storage helpers treat a falsy scope_id
as 'no scope_id filter', which would have read every user's rows).
Anonymous coordinators are no longer constructible: ChatSession refuses
kind=COORDINATOR with an empty user_id at the constructor — the single
choke point covering create, rehydration of legacy rows (surfaced by
the open handler as a 503 with remediation text), and any future host —
and the console no longer masks an empty uid as a phantom 'system'
principal when minting coordinator JWTs, per CoordinatorTokenManager's
documented 'sub = the real creator user_id' contract.
Migration 061 carries existing coordinator rows across: rows whose
owning workstream is gone or ownerless are deleted (unreachable under
user keying), same-name collisions within a user keep the newest
updated row (memory_id tiebreak), and survivors re-key to the owner's
user_id.
_buildHandle hard-coded aria-valuemin/max at 10/90 (inherited from the
old ui/static implementation) while the actual drag/keyboard clamp is
_ratioBounds — the cell minimums against the split node's OWN px region
(a 1200px host really clamps at ~17/83; nested splits sit tighter), so
assistive tech was told a wider range than the separator allows.
aria-valuenow/min/max are now all written in _applyLayout's handle loop
from _ratioBounds(h.node) — one writer, refreshed on every drag,
keyboard nudge, and structural change. A bare window resize can stale
the advertised range until the next interaction (no resize listener by
design — % insets make resizes free), still strictly truer than a
constant. The max>=min guard covers a host shrunk below two cell
minimums, where the bounds legitimately cross.
The /coordinator/{ws_id} standalone page is reachable only by direct
URL — all three console navigation sites are shell-fallback else
branches behind openPane. Record that in the sidebar-padding comment
so the scope isn't over-read as a live second surface.
The per-pane ✕/− chip floats at the pane's top-right — exactly where
the coordinator sidebar's toggle row and Children refresh button sit,
so the chip covered them. Pane-hosted coordinators now start the
sidebar content 44px down (padding, not margin, so the column's left
border still runs the full pane height); the standalone coordinator
page has no chip and keeps the 14px default.
Dual designer review (one primed on the branch context, one cold), all
measured findings applied:
- The per-pane chip was a mode-error trap: identical glyph at the
identical locus, reversible in split mode (hide cell) but destructive
single-pane (close pane). Now − hides, ✕ closes, and the close mode
wears a danger hover/focus ring so the irreversible action telegraphs
before the click lands.
- Single-pane chip anchored to the VIEWPORT: an unpositioned section
resolves absolutes to <body>, so the chip only coincidentally landed
near the pane corner. .panes > section.pane is now position:relative
in both modes (all pane-content absolutes verified to anchor to their
own local relative parents).
- Light-theme AA (measured): .shown tab underline 55% mix composited to
2.34:1 -> 80% (~3.7:1 light / ~5:1 dark); focused-cell ring 2.60:1 on
light -> 75% mix override there (dark keeps 55% at 3.75:1).
- Chip: border --hair-2 measured ~1.3:1 (invisible) -> --ink-4; 22px
target under WCAG 2.5.8's 24px floor -> 28px; right offset clears the
message scrollbar gutter; light resting glyph one ink step up.
- Focus bar inset 1px from cell sides (no doubled-accent stripe where
it butted a separator at the T-junction); greyscale font smoothing on
the tail glyphs (subpixel RGB fringed the box-drawing characters).
Rejected with rationale: aria-pressed on the split buttons (they are
one-shot verbs — splitting again nests — not mode toggles).
Four refinements from first live use:
- Per-pane ✕ chip, top-right of every visible pane. Split mode: hide
that cell (closeCell — the tab stays, the sibling absorbs the space).
Single-pane: close the pane outright (withheld from the unclosable
Dashboard). The click decides at click time; the label tracks the
mode. Manager-injected into the pane section — content untouched.
- Coordinator child links open BESIDE the coordinator (openPaneBeside:
split right of the focused cell, seeded with the child pane) instead
of replacing it — the parent stays on screen. Degrades to the plain
focused-cell swap when the split is denied (cap / narrow viewport).
splitFocused() gained an optional explicit-fill parameter for this.
- Tier-1 ws_closed now CLOSES the open interactive pane (tab gone, a
split cell collapses) — the coordinator-closes-its-child flow,
matching the standalone's pane-auto-close. The dead-banner lane
stays for streams that die without a ws_closed (node crash/network),
where the session may still be revivable.
- Paint bug: the focused-cell ring was an inset box-shadow on the
section, which paints in the element's own background layer — UNDER
opaque children touching the edges, so the status bar / composer
strip occluded it. The ring now rides a click-transparent ::after
overlay above pane content; the 2px top bar sits above the ring line.
The livepass shell surface's demo panes grew a .ws-status-bar footer so
the occlusion bug class stays visible to future passes.
Revives the split-pane feature retired with ui/static (step 6), rebuilt
on PaneManager: an optional binary layout tree (null = the one-pane-per-
tab behaviour, unchanged) renders visible panes as %-inset cells — no
reparenting, so live stream DOM, scroll state and media survive layout
changes. Tabs stay global: the active tab is the focused cell, a
backgrounded tab swaps into it, clicking inside a visible pane focuses
its cell, .shown marks visible-unfocused tabs. Separators resize by
pointer-capture drag and arrow keys (role=separator + aria-value*); the
tree persists in the working-set blob and rehydrate prunes leaves whose
pane did not restore. Limits: 6 cells, 200x150 cell minimums, denials
toast the manager's reason.
Affordance: Split right / Split down / Unsplit buttons in the tab-bar
tail replace the redundant [+] (the permanent Dashboard tab is the
launcher) — deliberately no contextmenu override this time. The dead
TS_APP.focusLauncher seam goes with it.
Measured chrome: the focused cell wears a 2px accent top bar (no thin
tinted ring clears 3:1 in both themes) plus a 55%-mix inset ring;
separators rest at --ink-4 with solid-accent hover/drag/focus; .shown
tabs carry an accent underline; the tail cluster is fenced and lifted
to --ink-3.
scripts/livepass.py grows a third surface: shell/livepass.html boots
the real shell.js + pane.js and drives ?split=right|down|three|none
(+ &theme=light), stamping SPLIT-READY-<cells> / SPLIT-FAILED-<reason>.
121 warnings -> 0. Two upstream deprecations get narrowly-scoped
filterwarnings entries (the mcp streamablehttp_client rename — adoption
deliberately rides the v2 migration since the new entry point's call
shape changes again there; the starlette httpx TestClient notice). The
one real RuntimeWarning is fixed at the source: tests that mock
asyncio.run_coroutine_threadsafe handed real coroutines to a stub that
never awaited them, GC-firing 'coroutine was never awaited' inside
whatever unrelated test ran later (the same cross-test bleed mechanism
as the CI closed-stream spew — per-test filterwarnings markers cannot
catch it, which is why two such markers existed and still leaked). A
shared _dispatch_stub now closes real coroutines before returning the
canned future; the obsolete markers are removed.
mcp 2.0.0a1 shipped 2026-06-11 (stable targeted ~2026-07-27). v2 removes
streamablehttp_client, changes the transport tuple arity, and renames
mcp.types fields to snake_case — all of which our client imports. The
maintainers' release note asks downstream packages to add an upper
bound now (their worked example is this exact constraint). Floor stays
at 1.27: nothing newer adds anything our surface needs, and the #2147
shutdown busy-loop we wrap remains unfixed at every released version.
Resolution is unchanged (1.27.2); lockfile re-pinned metadata only.
Copilot review on #661: empty base_url let the SDK fall back to
https://api.anthropic.com, sending compat-shaped requests to the
commercial API. The lane is local-only by definition, and the /v1-strip
edge case already established fail-loudly-over-silent-prod-retarget;
apply the same principle to the empty case. create_client raises an
actionable ValueError; the admin Detect path surfaces it as a clean
error string via probe_model_endpoint's existing handler.
Add provider id "anthropic-compatible": the existing AnthropicProvider
pointed at Anthropic-compatible local servers (vLLM /v1/messages),
mirroring the openai/openai-compatible split. Registry-only — configured
via the admin Models tab or [models.*] toml, not exposed on the bare
--provider flag, so the CLI/server prod-URL defaults are unreachable for
the lane and real-Anthropic behavior is untouched.
Lane behavior (live-verified against vLLM 0.22.1rc1 + DeepSeek-V4-Flash):
- Capability defaults replace the Claude static table: token_param
max_tokens, thinking_mode none, web_search/tool_search/vision off,
reasoning replay on. vLLM rejects Anthropic server-side tool types
(tools require input_schema) and ignores the thinking request param,
so neither is sent; thinking blocks still stream back and round-trip
through the native lane verbatim.
- Reasoning toggles via server_compat extra_body chat_template_kwargs
(first-class vLLM request field; request-level keys beat server
defaults). _build_thinking_and_kwargs forwards non-internal
extra_params as SDK extra_body; thinking_budget_tokens stays internal.
- No temperature force: thinking_mode none skips the Claude-only
temperature=1.0 requirement.
Admin UI: provider option + URL placeholder (base_url without /v1 — the
SDK appends /v1/messages); the server-compat section shows only the
extra-body field for the lane. thinking_mode round-trips through the
form dropdown for every provider except anthropic-compatible, where it
stays in the raw capabilities JSON — the edit-load lift and save restore
use the same predicate so stored overrides are never silently dropped.
Docs: architecture.md gains the lane subsection incl. verified quirks
(thinking param dropped by vLLM; stop_sequences cut inside thinking and
report end_turn; usage has no cache fields; images need a multimodal
model; mid-conversation system turns are per-model opt-in).
Negative-tested: removing the _INTERNAL_EXTRA_PARAMS exclusion fails
test_internal_keys_not_leaked; the live test drives a streamed turn with
the chat_template_kwargs toggle and asserts no reasoning deltas.
Review feedback: (1) gating the drain on a main-thread truthiness check
of _background_tasks could skip cancellation when a spawn queued via
call_soon_threadsafe had not reached the set yet — submit whenever the
loop is RUNNING and snapshot on the loop, where FIFO callback order
guarantees earlier-queued spawns have landed; (2) shutdown stopped the
loop thread but never closed the loop or cleared _loop/_thread, leaking
selector resources for embedders that cycle managers — close + clear
when we own the thread and it actually stopped (loud warning when it
does not); unowned loops (tests wiring _loop directly) stay untouched;
(3) the bare await-in-suppress drain loops become
asyncio.gather(return_exceptions=True) in both the shutdown drain and
the test fixture.
The post-reconnect catalog refresh was scheduled as a bare
asyncio.create_task: no strong reference (the task could be GC'd
mid-flight, so the refresh might silently never run) and no exception
retrieval (failures surfaced as "Task exception was never retrieved"
at GC time — in CI, onto an already-closed pytest capture stream, the
"I/O operation on closed file" spew; a suspected contributor to the
flaky 60-minute CI hangs via cross-test loop/task state bleed).
- _spawn_background(coro, label): tracked-task set + done-callback
that retrieves and logs failures at warning; discard runs LAST so
set-emptiness means "done AND reported"
- shutdown() drains tracked tasks FIRST, so stack teardown can't race
an in-flight refresh; same run_coroutine_threadsafe idiom and
timeouts as the existing close steps
- running_loop_mgr fixture: cancel-pending -> drain -> stop ->
join(5) with a loud assert -> loop.close() (was stop + silent
join(2), never closed)
- the false-property test ("swallows refresh failure" — nothing
swallowed it) now waits for completion and asserts the logged
warning via the patched module logger (structlog; caplog cannot
observe it), polling inside the patch context
Review feedback on the purge's race window: the pre-SELECT re-verify
left a statement-to-statement gap where a concurrent registration could
still lose rows — and the pre-counted refcount release could underflow
when it didn't. Orphan-ness now rides the DELETE itself (correlated
NOT EXISTS) with refcounts released from its RETURNING, so refs are
released for exactly the rows that were deleted. Input is de-duplicated,
IN-lists chunk at the storage layer's 500 convention, and the scan's
per-workstream ref-count loop is now one anti-join pass.
Conversation rows whose workstreams row is gone (historical unregistered
writers; the delete-during-inflight race re-creating rows after
delete_workstream) are invisible cruft that also pins attachment
refcounts. Add a turnstone-admin verb: default = read-only scan report
(ws_id, rows, attachment refs, first/last); --delete [--yes] purges.
- shared find/purge logic in storage/_utils; protocol + both backends
in lockstep (thin wrappers)
- purge re-verifies orphan-ness in-transaction: a ws_id re-registered
between scan and purge is skipped, never deleted
- releases the deleted rows' attachment refcounts through the
delete_workstream GC path and sweeps workstream_config/overrides
- summary reports actual purge results, including the skipped clause
* fix(ui): re-home MCP consent badge on the Manage Connections row
The L-shell renovation retired the standalone settings gear (#settings-btn).
The MCP pending-consent badge anchored to that gear via _refreshConsentBadge,
which null-guarded silently — so since the renovation pending consent requests
had no indicator (the badge was invisible).
Re-home the badge on the rail's Manage row where the MCP/connections surface
lives in both deployments:
- rail.js gains a generic setRowBadge(tabKey, count, label?) hook + a `badge`
builder: a small ⚠-glyph + count chip (never colour alone) using the DS warn
tokens. mountManage registers row + owning-group-head refs and re-applies live
counts across a (re)mount. When the owning group is collapsed, the count also
mirrors onto the group head so a hidden row never hides the signal. rail.js
stays agnostic — it owns the mechanism, the caller owns the meaning.
- shell.js (the ESM bridge) re-exports setRowBadge on window.TS_SHELL so the
classic ui/static/app.js subsystem can drive it without importing the module.
- The standalone consent subsystem keeps its shell-level ownership: _refresh-
ConsentBadge now drives setRowBadge on the Connections tab, fed by both the
loadPendingConsents hydrate/poll load and live onConsentDetected notifications.
- The shared interactive pane host bridges onConsentDetected to the new
window.TS_APP.onConsentDetected seam (undefined on the console, so the console
pane stays a no-op there); panes only notify.
- The dead colour-only gear badge CSS (.settings-consent-badge, red dot) is
removed; the new chip lives in shell.css as token-only .rail-badge so it
flips themes by construction.
Console MCP tab (Extensions > mcp) and standalone Connections tab
(Extensions > connections) both badge correctly. Pins extended in
test_shell_js.py + test_app_js.py.
* fix(ui): drop the unused head ref from the rail badge row map
Review feedback: _rowEls stored each row's group-head element but every
head consumer resolves it through _groupEls; keeping the duplicate DOM
ref made the remount state shape harder to reason about.
Review feedback: the next-case end markers were exact-indentation
string finds that raised a bare ValueError when unmatched. Use
whitespace-tolerant regexes with actionable assertion messages, and
bound the history-replay window structurally (next role branch, with
a generous fallback) instead of a fixed 600 chars.
Review feedback: (1) the Enter keydown re-dispatched through btn.click(),
relying on the disabled-guard to suppress the browser's own
Enter-to-click — preventDefault + direct activation makes the keyboard
path provably single-fire; (2) the branch-scoped Hls instance was
unreachable from the media error handler, leaking its listeners and
loader timers when the player node was replaced with the retry UI —
hoist the ref and destroy it before replacement.
The interactive Pane renders media embeds (buildMediaEmbed / buildPlayButton),
but the Play activation — _loadHls / _isHlsUrl / _activatePlayer and the
click/keydown delegate — stayed behind in the standalone ui/static/app.js as
DOCUMENT-level listeners. The console L-shell mounts the same interactive.js
module but never loads ui/static/app.js, so the Play button was dead in
console-hosted interactive panes.
Lift the activation into shared_static/interactive.js (alongside the existing
buildMediaEmbed/buildPlayButton — media embeds are interactive-pane-only; the
coordinator pane renders none) and wire it as a pane-owned, root-scoped
this.el click/keydown listener, mirroring the approval-keydown pattern the
fork collapse established. The standalone copy is deleted so no duplicate
implementation remains; both deployments now activate through the one shared
handler.
The hls.js vendor is fetched lazily by absolute /shared/ URL (the same
mechanism renderer.js uses for mermaid), and /shared is mounted at the root in
both turnstone/server.py and turnstone/console/server.py, so the vendor —
which ships in shared_static/hls-1.6.16/ — resolves in both deployments with
no HTML change.
Pins: assert the lift + pane-ownership in test_interactive_pane_js.py and the
standalone-stays-clean guard in test_app_js.py.
The live-SSE/history system-turn dedupe (renderedSystemEventIds /
_renderedSystemEventIds) was already in place on both panes and merged
to main (21af6c4 aligned the persisted row event_id with its SSE event;
09e41d1 added the belt-and-braces Set on the coordinator). The existing
pin tests only assert the Set's .has()/.add()/.clear() symbols appear
somewhere in the file, so a refactor that keeps the Set but short-circuits
the live-handler consultation (guard -> false) re-opens the double-render
while the pins stay green.
Scope the new assertions to their blocks: the live system_turn case must
CONSULT and RECORD against the Set, and the history render path
(replayHistory / refetchHistory's system-role branch) must record each
replayed row's event_id. Bounded at the next switch case rather than the
first break; the dedup-skip path itself breaks before the .add(), so a
break-bounded slice would drop the record half.
Verified the new slice checks fail on a dedupe-neutered factory (a
headless-Chrome harness driving the real createCoordinatorPane confirms
that neutering produces two rendered nodes for one event id; intact code
renders one, and the no-event-id legacy path still renders both).
The touch_structured_memories facade and both storage backends were
implemented but had zero call sites, so access_count never moved and
last_accessed never advanced past write time on any deployment.
Wire two touch points:
- proactive composition touches the injected top-k (post-rerank) set,
deduped per turn since _init_system_messages recomposes many times
within a single turn;
- the memory tool's search and get reads touch their returned rows,
counted per call. save/delete/list do not touch.
Touches are best-effort through the facade, which already swallows
storage errors, so a failed touch never breaks composition or a tool
call.
* docs: 1.6.0 changelog — roll up the 1.5→1.6 line for stable
Replaces [Unreleased] with the 1.6.0 section: 320 main-only commits
since the stable/1.5 divergence grouped into theme bullets (license,
trajectory/migration-060, web search, rerank/memory, approvals/judge,
L-shell, shelf, SSE, providers, cluster ops, security). Breaking
changes aggregated up top; migration-060 backup callout reshaped from
discussion #631 for the stable audience.
* docs: add the stable/1.6 track to the changelog preamble
* docs: retire the stable/1.4 track — current + one prior policy
Changelog preamble down to three tracks with the policy stated;
1.4 retirement noted in the 1.6.0 Removed section (final release
v1.4.0; tags/artifacts remain, BUSL-1.1 as shipped). releasing.md
track table, policy bullet, and examples brought up to the 1.6.0
promote cycle — the doc was still describing the 1.4-stable era.
PR #652 review follow-ups:
- not_found snapshot entries now carry the full key set (updated/name
empty) so results[ws_id] is shape-uniform across states; pinned by a
key-set assertion in the sentinel test
- ws_ids param text now distinguishes malformed (fails before any
waiting) from well-formed-but-unobservable (first-tick abort) at
unchanged length — per-param descriptions stay lean by policy
Field incident: the coordinator LLM hand-copied a child ws_id and
collapsed its aaa run to a, producing a 30-char id. inspect said "not
found", wait called it "denied", neither offered recovery, and the model
concluded the child was dead and dropped the lane — silent report
degradation while the child kept working.
- validate model-supplied ws_id args at the tool boundary
(send/close/cancel/delete/inspect/wait): full 32-hex ids pass through
at unchanged storage cost; a child's exact legacy id still resolves;
anything else fails fast with a did-you-mean (capped Levenshtein <=3
over the coord's own children) plus a child roster. Near-misses never
auto-resolve; display names are not addresses (mutable, non-unique) —
a name ref errors with a pointer at the right id
- wait_for_workstream: rename per-entry state "denied" -> "not_found"
with an honest sentinel; malformed refs error before any waiting
(invalid_ws_ids); a well-formed id that is foreign, missing, or
hard-deleted mid-wait aborts the wait on the tick that observes it
instead of burning the timeout (mode=all was unsatisfiable) or riding
along to complete=True (silent lane loss); mode=all completes only
when every id is real-terminal; entries carry the child display name
- one not-found payload across all verbs: foreign and nonexistent stay
byte-identical (no existence oracle), hints reference only the coord's
own children, echoed refs clipped in error strings; invalid_ws_ids and
not_found share one per-ref shape with the roster hoisted top-level
- inspect ownership now requires user_id parity via _row_in_own_subtree,
matching the wait/mutating gates (#506) — closes the forged-parent
cross-tenant read
- session exec serializes the structured recovery payload (results +
did_you_mean + children) on unresolvable-id wait errors instead of
collapsing to the bare error string
- tool JSON descriptions + coordinator docs updated to the new contract;
incident regression test pins the captured aaa-collapse ids
* chore: relicense BUSL-1.1 -> Apache 2.0 for 1.6.0
Flips every license artifact in the tree; 1.5.x and earlier remain
BUSL-1.1 per their release-time LICENSE files. Contributor consent
record: #548 (rationale: #546).
- LICENSE: canonical Apache 2.0 text
- NOTICE: new; copyright line + pointer to THIRD-PARTY-NOTICES
- pyproject.toml: SPDX expression + explicit license-files trio
- Dockerfile: COPY the license trio (hatchling needs them at build)
- THIRD-PARTY-NOTICES: BUSL line reworded; bundled-version drift
fixed (KaTeX 0.17.0, Mermaid 11.15.0, hls.js 1.6.16)
- README badge + License section, CONTRIBUTING inbound-license line,
TS SDK package(+lock), example pyproject
- docs/pgbouncer.md: drop stray ':' introduced in #353
* docs: add CONTRIBUTORS.md
* chore: drop LICENSE leading blank line
The apache.org LICENSE-2.0.txt begins with a newline; the SPDX
canonical text and GitHub license templates do not. Use the
conventional form — detection is whitespace-normalized either way.
Independent re-implementation of the --skip-permissions argparse flag,
OR-ed with the tools.skip_permissions config-store setting at both
consumption sites. Written from the flag's pre-existing spec (the
--help epilog and compose.yaml, which referenced it before #450
existed).
Replaces reverted #450 so that 1.6.0 ships no non-consented
contributions under Apache 2.0. Provenance record in #548.
Copilot round: the 3px literal (carried from the hatch .seg segments)
disagreed with the shared :focus-visible rule, which restates
border-radius as var(--r-sm) — so the corner radius popped on keyboard
focus. One token, no jump.
The dashboard launcher's Coordinator|Interactive radiogroup styled its
active option as a neutral panel highlight — two faint text links that
said nothing about WHAT was being chosen. The active option now takes
the kind vocabulary the rest of the shell already speaks (.ptag.coord
amber / .ptag.int cyan): 15% kind tint, kind-colored label, and a kind
LED dot, in a recessed .seg-style track. Colour is never alone — the
LED + label weight carry the state, and the JS contract (classList
toggle on .active, aria-checked, roving tabindex) is untouched.
Designer-review round on the scroll fix found the harness's dialog-tier
gate silently green: confirm-dialog (and install/coord-delete) markup
lives OUTSIDE #admin-layout, so the fragment extraction never embedded
it — ?open=confirm threw at showConfirmModal and screenshot a normal,
dialog-less page. build() now injects every hatch dialog the fragment
does not already contain, and a driven ?open= that ends with no open
dialog stamps OPEN-FAILED-<state> into the title instead of passing.
Also upstreams the review's probe states: &focuslast=1 focuses the last
shelf-body control (the displaced-dock regression class — only .sh-body
may scroll; head/foot must stay pinned) and &scrolled=bottom shows the
24px scroll tail.
The L-shell height-pins the admin chain and .hatch-host clipped it, so no
box below the pane could scroll: tabs taller than the pane were cut dead,
and the overflow:hidden host doubled as a hidden scroll container that
focus-into-view silently scrolled — visually-hidden toggle/cap/radio
inputs escape the .sh-body scroller (abspos under an unpositioned label),
overhang the shelf, and a Tab keypress shoved the docked hatch off its
head with no scrollbar to recover by.
- .admin-content becomes the manage pane's interior scroller (the #main
precedent); switchAdminTab resets it on real tab changes only
- .hatch-host: overflow hidden -> clip — paint clipping without a scroll
container, so focus can never displace the dock
- position:relative anchors on the three hidden-input labels
(toggle-switch, .sh-body .cap, segmented-option); .settings-toggle
already carried one
- livepass: the console harness wraps the fragment in the REAL L-shell
chain (its bespoke height pin is exactly how this bug class stayed
invisible to the screenshot gates) and gains a ?tall=1/&scrolled=1
scroll state
The wiring-lint test used percent-formatted regex patterns — UP031 under
the ruff 0.15.6 the CI pre-commit pins (the older venv binary let it
through; checked repo-wide against the exact pin now). f-strings with
doubled quantifier braces, plus one over-long fixture line in the
livepass generator split.
Copilot threads, both validated rather than blindly applied:
- closeShelf's scrim-ownership scan now skips detached entries. The
thread's throw scenario doesn't occur on the real removal path (a pane
close detaches an ANCESTOR, so _hostOf still resolves inside the
detached subtree) — but a detached shelf is genuinely not a scrim
owner, so the guard is correct beyond being defensive.
- toast.js drops the popover attribute via removeAttribute instead of
the null assignment. The claim that null leaves popover="null" is
refuted — the IDL is nullable and null removes the attribute (verified
empirically in headless Chrome) — but removeAttribute reads correct
without requiring that spec knowledge.
The livepass harness — the headless-render rig that verified every
converted modal surface and click-drives submits (the dead-Save bug
class) — lived as ad hoc files in /tmp and got wiped once already.
The durable piece is the GENERATOR: the markup is extracted fresh from
the index files at build time (a committed snapshot would drift) and
the stylesheets/scripts are symlinked so edits are live on refresh.
scripts/livepass.py builds both harnesses into /tmp/livepass/ (ui:
all six dialog-tier surfaces incl. the real cards.js batch controller
drive; console: the admin-pane fragment hosting the shelves, with
schedule/model/policy/confirm/token fixtures and the model-save click
drive that flips document.title to PUT-OK-<n>). --serve included;
the chrome screenshot incantation and the ?open= registry are in the
module docstring. Governance fixtures (roles/HR/OGP/memory/skill) are
documented seams for when those surfaces need driving.
The models conversion dropped the legacy onclick= from the submit button
and wired detect/recalibrate/capgrid/thinking in the boot IIFE — but never
the submit itself. submitCreateModel existed with nothing calling it: Save
clicked dead with no error, exactly what a live test surfaced. None of the
gates could see it — the markup lint checks anatomy not wiring, the review
finders verified the submit function's internals, and the livepass renders
never clicked Save.
Audited every id-bearing button inside every dialog.hatch across both apps
for click wiring: model-create-submit was the only true positive (the
batch confirm buttons wire through cards.js's $() prefix helper and
new-ws-submit wires 106 lines from its getElementById — audit false
alarms). The new test_every_hatch_button_is_wired pins the class: direct
getElementById wiring or wiring through the assigned variable, with the
two prefix-built cards.js ids allowlisted; verified it fails against the
pre-fix tree. Livepass now drives the actual click: Save → busy → one PUT
→ shelf closes + toast.
Sixteen-finder review (4 dimensions x 4 subsystem slices) + adversarial
verify: 16/16 findings confirmed, all fixed.
Majors:
- The schedule preview (and create/update via _compute_next_run) 500'd on
syntactically-valid-but-impossible cron dates: croniter.is_valid passes
'0 0 30 2 *' but get_next raises CroniterBadDateError. One _next_cron_runs
helper now owns construction + the guard for both paths; the preview
answers its 200/valid:false contract, and next[] is one shape (the cron
branch now carries the UTC offset the 'at' branch always had).
- The batch-delete results view tore down (exit delete mode, refresh the
stale list) only via the footer Close — header ✕ / Escape / backdrop left
deleted rows on screen and the mode stuck. The teardown moved onto the
dialog's onClose (gated by a resultsShown flag so a pre-delete cancel
keeps the selection), and Close just closes.
- The model shelf's Server-compatibility section was permanently invisible:
the one hidden-attr element still toggled via style.display, which cannot
beat .hatch [hidden] !important — openai-compatible operators could never
reach server type / API surface / extra-body. Now .hidden like its
siblings.
Minors: shelves prune detached entries when a pane closes mid-edit (state
map + Escape-listener leak); the capabilities autofill gains the
_schPreviewSeq stale-response guard; the rename dialog focuses its input
before select() (select() does not move focus per spec — Enter landed on
the ✕); .mcp-install-source-label becomes the fourth protected label
component; nine write-only shelf-handle vars dropped; one alert region
gets one name; orphaned .modal-col-heading CSS, the stale toast z-index
rationale, a dangling divider comment, and a comment chasing the renamed
_submitRoleShelf all cleaned. Regression tests pin the Feb-30 preview,
the create-path guard, and the uniform next[] shape.
Dual review of the dialog-tier work (primed + cold), adjudicated:
- Menu-launched dialogs lost focus return: openPopupMenu's close() removed
the focused item before the action ran, so dialogs captured <body> as
their opener and the close-restore no-op'd. The menu now hands focus to
its return target before invoking the action — repairs every
menu→dialog flow in both apps.
- Rename targeted the wrong workstream: submitEditTitle re-read the
ACTIVE pane's id, so renaming a background tab via its context menu
renamed whichever pane was focused. The dialog now pins its target at
open (pre-existing bug, carried from the legacy overlay).
- Batch-failure rows printed raw HTML error pages verbatim (proxy 502s,
gateway timeouts): bodies are stripped of style/script content and
markup before display, with an HTTP-status fallback.
- The two single confirms behaved differently in flight — delete-ws
closed optimistically while revoke held under the busy lock. Unified on
hold-open-with-busy: failures keep the user's context for retry.
- Batch results: failures announce through the live sh-alert (previously
dead markup), a clean run flips the chrome to the success kind (red
head over '3 deleted, 0 failed' disagreed with the de-dangered foot),
and focus lands on Close after the state swap.
- Ghost-button boundaries measured ~1.2:1 on the foot strip (WCAG
1.4.11): ink-mix borders routed through a variable so kind variants
keep their own border colors; reduced-motion busy gains a static ' …'
cue; dark label-hint opacity brought above the compound 4.5:1 line.
- Revoke voice unified ('Revoke connection', no '?'); batch alertdialogs
gain aria-describedby; dead _settingsTrap machinery removed.
The revoke confirm is no longer a role=dialog/aria-modal overlay with
page-local CSS — it's a hatch dialog-tier alertdialog whose chrome lives
in /shared/hatch.css. The markup pin follows the new shape and the
stylesheet pin list drops the retired #revoke-mcp-overlay rule.
The shared cards.js multi-select controller serves the ui Saved
Workstreams AND the console Saved Coordinators, so the builder and its
two host markups convert as one unit: #ws-delete-dialog and
#coord-delete-dialog are md danger dialogs whose list renders inside the
sh-body (the only scroll region — the 200px inner list cap dies).
Foot grammar: [N selected meta] [Cancel data-close, autofocus] [red
filled "Delete N workstreams"] — the count moves out of the body prose
into the meta and the action label. The fan-out brackets with setBusy
(LED pulse + action lock replace the disabled/'Deleting...' swap); the
results view swaps the action to a neutral Close and hides Cancel (a
Cancel beside a Close is the redundant dismissal pair the foot grammar
forbids). The dormant error region becomes the sh-alert.
The controller's hand-rolled focus trap, prevFocus bookkeeping and
overlay display toggles die — hatch.js owns trap/Escape/backdrop/focus
restore; the post-results Close still hands focus to the section toggle
the bar collapse just rebuilt. The ws-delete-modal-* CSS family leaves
cards.css (only the row treatment survives); the dead window wrappers
(cancelWsDelete/confirmWsDelete + coord twins) and the ui keydown
handler's last legacy-overlay branch go with it.
The ui app has no admin pane host and new-ws is a launcher invokable from
anywhere, so every surface lands on the document-modal DIALOG tier (no
shelves). hatch.css gains the one md dialog width (560) the v1 design
specified; the ui index links hatch.css/hatch.js alongside the other
shared assets.
- New workstream: md create dialog, WS-NEW plate; the fork path keeps its
title/semantics (WS-FORK plate, skill + attach rows hidden via the
hidden attribute). Submit brackets with setBusy; errors land in the
sh-alert with the scroll-into-view rule.
- Rename: a styled prompt() — single field, Enter submits, no plate.
- Delete-workstream + revoke-MCP mirror the console confirm exactly:
danger chrome, prose body, Cancel autofocus, red filled action.
The four hand-rolled focus traps, the Escape/overlay-click dispatch and
the body-overflow lock die (native dialog + hatch.js own all of it); the
global-shortcut handler defers on dialog:modal instead of the overlay-ID
list. The forked legacy modal CSS leaves style.css (~270 lines). The
markup-shape lint now scans both index files, the asset assertion and the
parse-time-bridge guard extend to the ui app.
Cosmetic/consistency findings from the dual shelf review, adjudicated;
accepted items:
- One designation-plate grammar: 2-4 char domain code + closed suffix
vocabulary. SKILL-GH → SKL-IMPORT, PP-* → PPO-*, ROLE-MAP → USR-ROLES,
ROLE-* → ROL-*; HR-/OGP- stay (established judge domain terms).
- Create titles say "New <thing>" (Add model / Add MCP server renamed;
"New prompt" was creating a prompt POLICY); "Edit model" gains the
"— {alias}" suffix every other edit surface carries.
- Shelf primaries uniform single-word Create/Save — the skill shelf's
"Create skill" / "Save config" / "Save changes" collapse.
- label-hint dialect normalized to bare lowercase (wrapping parens
dropped); the em-dash unit form "— UTC" is a different species, kept.
- Placeholder-only format instructions promoted to label-hints (a11y
3.3.2): HR confidence "0.0–1.0", intent "supports {arg_snippet}". The
install dialog's dynamic required asterisk pairs with required on the
control + aria-hidden on the glyph.
- The mcp and schedule adjacent toggle pairs wrap in toggle-stack like
the model shelf; the sch-enabled-row hidden toggle rides inside.
- memory-detail foot grammar: Delete demoted from err-filled to a quiet
destructive text action (.sh-btn--quiet-danger, doubled class so it
beats the later-loaded hatch.css quiet color), Close stays rightmost.
- First-focus: user-roles lands on its first role toggle once rendered,
channel on the type select, builtin-role edit on the first enabled
control, mcp-install autofocuses its primary.
- The watch-cancel confirm's action reads "Stop watch" — no more
Cancel-beside-Cancel.
- A locked skill opens as data-kind=inspect (cyan read-out chrome) and
flips to edit on the in-place unlock re-render.
- mcp-detail healthy node dot pairs with a dim "connected" text token,
mirroring the error-text sibling (state was color-alone).
- etm-scan section converts to the hidden attribute (last
style.display straggler on the skill shelf).
Dual design review of the shelf stack (one primed on intent, one cold)
adjudicated; accepted findings:
- The busy lock held the door for the mouse only: Enter on the focused
primary re-fired submits, scrim clicks closed a shelf mid-flight, and
the dialog tier's native Escape (cancel event) dismissed a busy confirm.
One capture-phase guard + a scrim busy check + cancel interception close
all three for every surface; setBusy now announces aria-busy. Contract
assertions added to the busy test (it previously claimed scrim coverage
it didn't assert).
- The MCP auth radio cards were destroyed by the .sh-body label cadence —
the exact specificity war the toggle-switch/cap exceptions guard
against, missed for .segmented-option. Restated at 0,2,1.
- Light-theme off-state toggle tracks measured ~1.2:1 (WCAG 1.4.11 wants
3:1): real ink fill on light, recessed look kept on dark.
- Failed submits on tall shelves rendered out of view — _showModalError
scrolls the alert into view. user-roles gains the missing busy bracket
and in-shelf errors instead of toast-only.
- Toasts render under the dialog tier's top layer: promote the toast to a
manual popover only while a modal dialog is open (popovers stack above
later dialogs); attribute dropped after so the everyday fade survives.
- Ambient glow now follows the kind accent (cyan/red surfaces no longer
bloom amber); skill editor's content column pins sticky so the SKILL.md
pane stays visible while the meta column scrolls; MCP-detail stacking
re-keyed from viewport to @container pane; dark-body micro-text and the
origin badge brought up to the contrast floor; quiet foot actions
(Validate regex, Detect) lifted from body-copy gray.
With the popover and a shelf both open, Escape closed both at once
(admin.js's popover listener and hatch.js's shelf listener each fired).
The popover listener registers at parse time — always ahead of hatch.js's
first openShelf — so stopImmediatePropagation makes Escape peel one layer
at a time: popover first, shelf on the next press.
With the skill editor converted, nothing renders through the legacy
overlay system any more. Grep-driven deletion:
- admin.js: _modalFocusTrap/_installTrap/_removeTrap, the govOverlays
dispatch table, and the global Escape handler die. The handler's one
still-live block — closing an open settings-help popover — is
extracted into its own small keydown listener (the settings panels
and the shelf form help buttons share that component).
- governance.js: the orphaned template trap/trigger let declarations.
- style.css: the .admin-modal box rules (incl. -wide/-skill), every
.admin-modal-prefixed half of the doubled toggle/segmented/perm/
user-roles selectors (the unscoped twins keep serving the settings
panels and shelf bodies), .admin-details (the shelf uses
details.rawhatch), .modal-buttons/.modal-cancel/.modal-submit,
.modal-section-divider, hr.toggle-group-divider, the now-empty
#…-template-overlay ID rule, and their reduced-motion entries.
Stale comment pointers re-aim at the live rules (.sh-alert,
.sh-body label).
Kept with reason: .modal-columns/.modal-col* — _openMcpDetail still
builds the MCP-detail shelf body with them (their narrow-viewport
stacking rules survive in a rebuilt media block).
Livepass re-verified after the teardown: zero console errors across
create/edit/locked, and the unlock-confirm-over-shelf capture is
pixel-identical to the pre-teardown one.
The last legacy modal pair (create-template + edit-template, ~430 lines
per mode) collapses into one pane-scoped 920px shelf: a hidden skl-id
decides POST vs PUT, the duplicated ctm-/etm- field sets merge into one
skl-/sklc- set, and the skill-spec two-column grid transplants whole into
the scrolling body (content-area rule re-scoped under .sh-body so the
shelf's font:inherit/min-height cadence doesn't flatten it; the mobile
breakpoint becomes a pane container query to match the shelf's own
bottom-sheet degradation).
Everything judgment-bearing carries over: paste-to-parse (now cancelled
via the shelf's onClose so Escape/scrim dismissals abort the inflight
parse too), live {{variable}} detection (re-dressed as match-strip chips
under the textarea, count mirrored into the foot meta when provenance
isn't occupying it), pending-resource rows in create, server-backed
resources + security scan + re-scan in edit, and the runtime-config
field set that stays editable on readonly skills (must keep matching
SKILL_RUNTIME_CONFIG_FIELDS). The readonly lock affordance moves into
the head strip left of the designation plate as a ghost icon button;
origin/locked provenance renders in the foot meta lane (installed/
customized chip + source URL + 'locked — unlock to edit'); the unlock
confirm stacks above the shelf via the native top layer and the
post-unlock re-render mutates the open shelf in place. Progressive
disclosure keeps its <details> semantics on rawhatch chrome, with the
shared-DOM state leaks (disabled spec fields, expanded details) re-armed
on every create open.
Mode-exclusive blocks keep their ctm-/etm- ids (pending vs server
resources, scan) — only true duplicates merged. Inline onclick wiring
moves to a one-shot _skillShelfWire; submits go busy via the shelf LED
lock. The legacy overlay markup and the .skill-lock-btn / .skill-vars-*
rules are deleted; trap machinery teardown follows separately.
Livepass-verified (stubbed authFetch, headless Chrome): create, edit
populated, locked view (disabled spec + editable runtime + scan +
readonly resources), unlock confirm stacked over the shelf, paste
auto-fill, variable chips, pending-resource add/remove.
The memory-detail read-out leaves its body-level overlay for a pane-scoped
inspect shelf (#memory-detail-shelf, cyan, lg) inside #admin-layout. The
detail-grid + content population carries over verbatim; the foot keeps the
Delete action (now a danger sh-btn) wired after the record loads, beside Close.
showMemoryDetailModal/hideMemoryDetailModal keep their names — the row renderer
calls them — and run through window.TurnstoneHatch; the post-delete close check
keys off the dialog's .open. Legacy overlay markup, its _installTrap/Escape
dispatch entries, the trap/trigger lets, and its style.css overlay ID-list row
are deleted.
The three governance create+edit pairs collapse onto pane-scoped lg shelves
inside #admin-layout. Each merges into ONE shelf with a hidden id (plus a
builtin flag on the judge surfaces) deciding the write: PUT for a DB row, an
override-POST for a built-in's first edit, a plain POST for a new row.
Title/tag/data-kind/submit-label flip between create and edit; edit-only chrome
(the prompt-policy Enabled toggle) sits in a hidden-toggled row.
The heuristic-rule and output-guard editors lift out of the Judge tab panel to
sit as direct children of the hatch-host (the shelf's inert containment needs
that); the two built-in-edit entry points share a single populate-and-open
helper. The settings-help-popover buttons carry over verbatim — their document-
delegated toggle is independent of the container. The output-guard "Validate
regex" button moves to the foot as a quiet action while its result strip stays
in the body. Submits go busy via the shelf LED lock; the regex flags input and
the is-credential toggle move onto the field grid. Legacy overlays, their
_installTrap/Escape dispatch entries, the trap-handler/trigger lets, and their
rows in the style.css overlay ID list are deleted.
The governance role surfaces leave their legacy overlays. Create + edit role
collapse into ONE pane-scoped shelf (#role-shelf inside #admin-layout): a hidden
role-id decides POST vs PUT, and title/tag/data-kind/submit-label flip between
"New role"/ROLE-NEW/create/Create and "Edit role — name"/ROLE-EDIT/edit/Save.
The slug-name row is create-only (hidden attr on edit); the display name carries
over and is disabled for builtin rows. The permission checkbox grid renders with
one "role" prefix for both modes — the builtin baseline-vs-rendered diff that
produces {grant, revoke} (and round-trips unknown perms untouched) is preserved
verbatim. Submit goes busy via the shelf LED lock instead of the disable dance.
User-roles becomes an edit shelf carrying its toggle-list population; github-
import a create shelf with the URL as an sh-mono field and the hint folded into
a label-hint span. The public show/hide/submit names the toolbars and row
renderers call are kept — only the bodies are rewired through window.TurnstoneHatch
(handler-time, never at parse time). Legacy overlays, their _installTrap/Escape
dispatch entries, the trap-handler/trigger lets, and their rows in the style.css
overlay ID list are deleted.
The four MCP surfaces leave their legacy overlays. The add/edit server editor
collapses into ONE lg pane-scoped shelf (#mcp-shelf): a hidden mcp-edit-id
decides POST vs PUT, and the title/tag/data-kind/submit-label flip between
"Add MCP server"/MCP-NEW/create/Create and "Edit MCP server — name"/MCP-EDIT/
edit/Save. The transport-conditional stdio/http field groups and the OAuth
subfield block toggle on the hidden attribute instead of style.display (which
.hatch [hidden] enforces); the multitenant-auth segmented radio control carries
over verbatim. The transport/auth onchange and submit move out of inline markup
into _mcpWire (which also installs the audience-autofill listener once).
mcp-import becomes a create shelf with the JSON paste as an sh-mono textarea;
mcp-detail an inspect shelf carrying its setSafeHtml two-column population. The
registry install flow moves onto the document-modal dialog tier per the confirm
precedent: a STATIC #mcp-install-dialog whose summary/source/fields containers
_showInstallMcpModal still populates — the dynamic overlay-shell construction is
gone. _doRegistryInstall is shared by the one-click card path and the dialog
submit, so its busy lock and inline-vs-toast error branch now key off the
dialog's .open. Submits go busy via the LED lock; legacy trap/Escape/ID-list
entries and the trap-handler lets are deleted.
The four admin.js leaf surfaces follow the schedules pilot onto pane-scoped
shelves inside #admin-layout: create-user, create-token, link-channel and the
read-only schedule run history. Each keeps the public show/hide/submit names the
toolbars and row renderers already call (showCreateUserModal, showScheduleRuns,
…) — only the bodies are rewired. Open/close run through window.TurnstoneHatch
(handler-time, never at parse time); submit goes busy via the shelf LED lock
instead of the disable/relabel button dance; errors land in the sh-alert.
The token shelf hands its issued secret to the already-converted token-created
dialog unchanged. Schedule runs is an inspect shelf (cyan, lg width, Close-only
foot) and carries its setSafeHtml run-table population verbatim. The channel
type→placeholder onchange moves out of inline markup into _channelWire. Legacy
overlays, their _installTrap/Escape dispatch entries, the trap-handler lets, and
their rows in the style.css overlay ID list are deleted.
Tool policies join the shelf: one pane-scoped editor for create+edit with a
priority-neighbor read-out computed from the loaded policy list ('evaluates
after deny-rm (900) · before default-ask (0)' — policies run highest-first),
so where a priority lands answers itself while typing. Live tool-pattern
match chips are deferred until a cluster tool-registry endpoint exists.
The reusable confirm and the show-once token dialog move onto the
document-modal hatch tier (native showModal): the confirm keeps its
showConfirmModal(title, message, actionLabel, callback) contract for all
14 call sites, gains the danger chrome (red LED/hairline/title + err-filled
action), and deliberately moves autofocus from the action button to Cancel
— Enter on a fresh destructive confirm no longer fires the action. The
token dialog gets the success chrome + show-once callout, with copy wired
to the primary. Nested confirm-over-shelf now stacks via the top layer;
the z-index 650 special case and four more overlay-ID/dispatch entries die.
Also fixes a real war the livepass caught: display rules on form chrome
(label.toggle-switch's inline-flex) defeat the hidden attribute — hatch
containers now enforce [hidden] with display:none !important.
The model editor moves onto a lg shelf and the hand-written capabilities
JSON requirement dies. Nine LED tiles (tools/streaming/vision/web-search/
temperature/effort/STT/TTS/reranker) display merge(dataclass defaults,
known-model table, explicit overrides) with SPARSE-OVERRIDE persistence:
only keys saved in the row or toggled by the operator are written back, so
known models keep tracking future capability-table updates instead of
being pinned at save time. The known-model lookup that previously dumped
the whole table into the textarea becomes the tile BASELINE refresher with
a provenance banner ('Loaded from the built-in table for X'); the raw JSON
survives as a collapsed advanced hatch holding everything the tiles don't
manage (thinking_display, max_output_tokens, …). supports_rerank is now a
tile — no more hand-JSON to flag a reranker — and drives the Re-calibrate
button + calibration chip (now a foot read-out next to quiet Detect).
Everything regression-prone carries over: server_compat extraction,
write-only api_key sentinel, reranker calibration field re-merge (raw-typed
keys still win), thinking-mode representability guard, detect/calibrate
flows. Inline onclick/onchange wiring moves to the boot IIFE; busy locks
the shelf LED instead of disabling the button. Legacy overlay markup, the
trap/Escape dispatch entries, and the CSS ID-list entry are deleted.
Livepass-verified (stubbed authFetch, headless Chrome): create + edit,
baseline banner, tile extraction, toggle-switch inside .sh-body.
First production surface on the service-hatch shelf. Create + edit collapse
into ONE pane-scoped dialog (#schedule-shelf inside #admin-layout, now the
.hatch-host): a segmented Runs control (Daily/Weekly/Monthly/Interval/Once/
Cron) compiles to schedule_type/cron_expr/at_time — nobody types cron unless
they choose Cron mode, which keeps the raw input as the escape hatch with
the same live read-out. The NEXT RUNS read-out previews the next three
firings through the server's croniter (debounced POST /schedules/preview),
and the foot strip always shows the compiled expression for verification.
Storage is untouched: on edit the saved expression is reverse-parsed back
into the friendly mode when its shape matches (_cronToScheduleMode), else
the editor opens in Cron mode. Notify-row/select-populate helpers carry
over verbatim; submit goes busy via the shelf LED lock instead of button
text swapping. The legacy create/edit overlays, their show/hide/toggle
globals, hand-rolled trap wiring, and their entries in the overlay ID list
and _installTrap/Escape dispatch tables are deleted (-242 lines of markup).
Toggle-switch gets its .sh-body exception in hatch.css (the same
specificity war .admin-modal fights), + .sh-mono utility.
Verified against a stubbed-authFetch livepass harness in headless Chrome:
create/edit/light/busy, weekly reverse-parse round-trip, preview rendering.
One chrome vocabulary (machined head/foot strips on --code-bg, kind LED,
designation plate, center-fading hairline), two mounting points:
- .hatch--shelf: pane-scoped, NON-modal (dialog.show()). Mounts inside the
pane's .hatch-host, docks right, dims only that pane via a lazy sibling
.pane-scrim, and contains focus with inert on the pane's other children —
rail/tabs/other panes stay live. Split panes work by construction; the
bottom-sheet degradation is an @container query on the pane, not a
viewport media query, so a narrow split degrades too. Controller-owned
Escape defers to any document-modal dialog stacked above.
- .hatch--dialog: document-modal (showModal()) confirm/show-once tier;
top layer stacks it above any shelf with no z-index ladder.
Smart-input primitives ship alongside (seg, chips, readout, capgrid,
match-strip, rawhatch, autofill) for the Phase-1 surfaces. data-busy locks
the container while a submit is in flight (LED pulses, dismissal refused).
Light-theme micro-text gets the .tab-menu-key one-step-up contrast pass.
Classic scripts reach the ESM controller via the window.TurnstoneHatch
bridge (toast.js pattern, handler-time only); invariants pinned in
tests/test_hatch_js.py incl. a markup-shape lint over dialog.hatch.
POST /v1/api/admin/schedules/preview validates {schedule_type, cron_expr,
at_time} with the same _validate_schedule_fields the CRUD path uses and
returns the next three croniter firings. Pure compute, no storage touch;
invalid input answers 200 {valid:false, error} because the schedule
editor renders it live while the user types. Registered ahead of the
{task_id} routes so the literal segment wins.
Review feedback: the cancel-path docstrings described undone items as
degrading to 'heuristic fallback verdicts', but the emitted and
persisted tier is llm_fallback (heuristic content relabeled). Aligned
all eight occurrences — including the pre-existing _deliver_fallbacks
docstring — so docs, logs, and audit rows use one vocabulary.
Review follow-up: the ON CONFLICT rationale lived verbatim in three
places (protocol docstring + both backend comments). Keep the prose in
the protocol — the contract's home — and point the backends at it.
Also recommend cancel_on_approval=true in docs for deployments where
the judge shares one local inference backend with the session model.
judge.cancel_on_approval=False (the default) promises the daemon
evaluates every tool call to completion so all verdicts are available
for later review. Two sites conspired to break that: the approval
gate's finally set the cancel event unconditionally the moment a
decision landed, and _evaluate_single's poll loop honors the event
regardless of config — so every item the sequential judge hadn't
reached degraded to a heuristic llm_fallback row. On a 22-call
parallel batch, approving after the third verdict silently downgraded
the other 19; the elaborate late-verdict machinery in
on_intent_verdict was effectively dead code.
Make the event a pure abort signal whose firing policy lives with the
caller: the gate fires it only when cancel_on_approval is enabled,
while generation supersede (next batch) and close() keep firing it
unconditionally, bounding a stale daemon to one batch of real work.
_run_judge drops its own config second-guessing — a fired event always
fast-forwards the remainder to fallbacks (every call still gets
exactly one verdict), and the fallback reason no longer claims 'user
approval' for supersede/close aborts.
The async judge daemon can UPSERT a fallback row — reusing a heuristic
verdict_id from the batch approve_tools is about to bulk-insert —
before the bulk write runs. With a plain INSERT, that single PK
collision aborted the entire statement, and the caller's best-effort
try/except silently discarded every heuristic row in the batch.
Insert ON CONFLICT (verdict_id) DO NOTHING on both backends: siblings
survive a mid-batch collision, and the colliding row keeps the
daemon's llm_fallback tier upgrade instead of regressing to the
heuristic stamp (the documented preferred outcome). Regression test
runs against both storage backends via --storage-backend.
ChatSession._on_verdict guards on judge-generation identity so a stale
verdict can't ride a reused call_id into the Smart-Approvals cache —
but it dropped those verdicts entirely, before persistence. Every
ruling the sequential judge delivered after the next turn began left
intent_verdicts claiming the judge never answered.
Route superseded verdicts to a new persist-only hook
(SessionUIBase.on_superseded_intent_verdict): the row lands with
user_decision="superseded" while every live surface stays untouched
(no SSE, no replay cache, no pending-decision park). The hook is
duck-typed; display-only UIs (CLI/eval) don't define it and keep the
plain drop. upsert_intent_verdict already excludes user_decision from
its on-conflict SET, so a superseded fallback upgrading its heuristic
row in place cannot clobber a decision already stamped there.
The /history decoration layer suppressed intent-verdict rows with
risk_level="none" from the wire payload, on the assumption the client
filtered them anyway. It never did: buildConvVerdict renders a badge
for every verdict it receives, so the live SSE path painted all judge
verdicts while rehydration silently dropped the benign majority — a
22-call parallel batch came back from a restart showing only the 3
flagged calls.
Ship every stored row and let the client render replay exactly as it
rendered the live stream. The output-guard chip pair (showOutputWarning
+ merge-on-clean) already suppresses consistently on BOTH sides and is
unchanged.
attempts < 1 made init() return successfully without fetching CA or
cert — a silent no-op leaving the client uninitialized. Fail fast with
ValueError instead; negative base_delay rejected on the same guard.
A whole-stack restart races every node against the console for the CA
fetch (compose re-enforces depends_on ordering only on `up`): losers
logged one warning and served plain HTTP for their lifetime, while
winners served mTLS that the plain-HTTP container healthcheck could
never probe — leaving "healthy" plaintext nodes and "unhealthy"
working ones.
- TLSClient.init() grows attempts/base_delay retry (server passes 6
attempts, ~31 s backoff) absorbing the boot race; per-attempt CA-fetch
failures log warning + debug traceback instead of error tracebacks.
- healthcheck.py falls back to HTTPS when the plain probe fails,
presenting the node's own cert as the client cert with the cluster CA
pinned; dials localhost because the internal CA issues DNS SANs only.
Default plain-HTTP deployments are unchanged.
- The server writes boot PEMs under a fixed root (TURNSTONE_TLS_PEM_DIR,
default <tmpdir>/turnstone-tls) so the probe can find them; boot
clears stale dirs and refuses a symlinked/foreign-owned root; renewal
rewrites the PEM dir so the probe's client cert never outlives the
served cert.
- /health reports tls: "active"|"fallback" (absent when TLS is
disabled) so a silently downgraded node is observable.
Review feedback (Copilot), both confirmed against source:
- openPopupMenu: when no menu item has focus (a click on a separator or
the menu surface moves focus off the items without closing), ArrowUp's
unguarded modulo landed on the second-to-last item ((-1-1+n)%n == n-2).
Guarded to enter at the bottom; ArrowDown's (-1+1)%n already entered at
the top. Pre-existing in the tab dropdown this helper was extracted
from — the shared chrome means one fix covers both menus.
- The burger and the [+] tail were focusable non-tab children inside the
element PaneManager stamps role=tablist (the [+] violation pre-existed;
the burger doubled it). The tabs now live in their own .tabstrip, which
becomes the tablist PaneManager owns; burger and tail sit outside it in
.tabbar. The strip is also the mobile horizontal scroller, so burger +
[+] stay pinned while tabs scroll.
Verified: 289 static-suite tests, 28 harness self-tests, and both
real-page boot harnesses green; mobile render unchanged.
test_renderer_js drives renderer.js behaviorally through node via
vm.runInThisContext — script semantics, which choke on the import/
export syntax renderer.js and utils.js now carry (all 68 tests failed
at harness setup). The harness now evaluates _demodulize()d source:
imports drop (the shared vm context resolves cross-file bindings as
globals, exactly like the pre-module classic scripts) and export
keywords peel off.
Deliberately NOT switched to dynamic import(): the mermaid harness
pokes renderer-internal state (_mermaidState = 'ready') that script
evaluation exposes but a real module would encapsulate. Module
semantics are covered by test_shell_js's .mjs parse sweep; these
tests pin renderer behavior.
Review follow-ups: the new rail collapse and mobile drawer had no
committed guards (the repo pattern is per-step string assertions in
test_shell_js.py) and openPopupMenu — now load-bearing for both the
tab dropdown and the footer user menu — was unpinned.
- test_rail_collapse_glyph_strip: persistence key, toggle +
aria-controls, class-flip seam, cpill-label/manage-glyph companions,
52px desktop-scoped CSS.
- test_mobile_drawer_off_canvas: burger, scrim, rail-open flip,
pane-activation auto-close, off-canvas translateX + visibility:hidden.
- test_popup_menu_shared_helper: the export + both consumers (the user
menu's prefer-up path included).
- shell.css: the 769/768 media blocks are a matched pair CSS cannot
express as a shared token — both now carry a cross-referencing
change-both comment.
Review finding (critical): the ESM migration made cards.js a deferred
module, but both classic app.js bundles built their saved-list tables at
TOP LEVEL — const COORD_COLUMNS = [SavedColumns.name(), ...] and
const _coordTable/_wsTable = createSavedTable({...}) execute at parse
time, before the window bridges exist. ReferenceError aborted each
bundle before it could define TS_APP.boot, so neither deployment booted.
(The earlier consumer audit caught bare top-level CALLS and IIFE bodies
but excluded declarations — missing initializers with side effects.)
Construction moves into _initSavedCoordTable()/_initSavedWsTable(),
called first from each boot path (substrate modules have evaluated by
then). The two typeof-undefined guards become null-checks (a let
binding passes typeof). Verified end-to-end with real-page load
harnesses: both index.html script chains (real app/admin/governance +
module substrate, network mocked) boot to a mounted shell with zero
uncaught errors — console over loopback HTTP (the coordinator dynamic
import needs real URL resolution), standalone over file://.
Dead-code removal, all provable (no JS creator / no reachable caller):
- interactive.js drops the !this._embedded branches: focus tracking, the
right-click context menu, and the header with split/close buttons all
referenced shell globals (setFocusedPane, splitPane, splitRoot,
showPaneContextMenu, countLeaves, closePane) that exist nowhere since
the step-6 fork collapse — reaching them was a guaranteed
ReferenceError. The embedded flag goes with them (every pane is
L-shell-hosted; pane--embedded is now unconditional), as do the
call-less updateWsName() and the host adapter's getWsName seam.
- ui/static/style.css drops the orphaned split-pane/tab-bar vocabulary:
.ws-tab*, #new-tab-btn, #split-btn, .split-handle, the pane-header/
action-button block, the unused dropdown-in keyframe, and the dead
entries in the reduced-motion list.
- ui/static/app.js drops the retired settings-gear menu remnants (state
vars for a builder that no longer exists + an always-false Escape
guard) and fixes a real crash: hideNewWsModal() focused the removed
#new-tab-btn unguarded, throwing a TypeError on every create/fork
modal close; focus now returns to the shell's [+] new-session button.
- pane.js exports openPopupMenu — items, positioning (flip + clamp),
dismissal, aria-expanded mirroring, and arrow-key roving in one place.
The tab-action dropdown delegates to it, and shell.js's footer user
menu replaces its hand-rolled duplicate (which lacked Tab-close and
arrow roving — it inherits both).
test_interactive_pane_js.py: the embedded-gate pins flip to retired-
symbol pins (the gate is gone, not gated).
utils/toast/kb/cards/auth/renderer/composer/composer_attachments/
composer_queue/status_bar convert from classic scripts (implicit globals,
IIFE wrappers) to ES modules with explicit exports. Parse-time
cross-dependencies become real imports (auth/kb/cards -> utils,
auth/cards -> toast, cards -> auth, renderer -> utils), which deletes the
implicit script-order contract those files relied on. utils stays
import-free (bottom of the graph); its two upward calls (setMarkdown ->
renderer, export -> toast/auth) late-bind through window at call time to
avoid import cycles.
Each module installs a transitional window bridge for the still-classic
bundles (console app/admin/governance, ui app, inline onclick=), which
only touch the globals at boot/event time — verified by a column-0 /
IIFE-body audit of all four consumers, and including the audit-missed
initLogin() that both app.js boot paths call. theme.js stays classic:
deferring it would flash the wrong theme before first paint. Vendored
katex/hljs/mermaid stay classic and lazily typeof-guarded.
interactive.js and shell.js drop their bare-global reads for real imports
(authFetch, showToast, Composer, StatusBar, queue/attachment controllers,
streaming renderer, setMarkdown). The three HTML entries load the
substrate as module tags (same positions, same version_html stamping);
classic admin/governance/app still parse first, modules evaluate before
shell.js calls TS_APP.boot().
Tests: auth/kb/utils move from test_app_js's classic node-check sweep to
test_shell_js's module-semantics sweep, which now covers all 15 shared
modules (sink scan excludes renderer.js, the sanctioned HTML producer;
the no-var ratchet covers the var-free subset). The const-reassign guard
re-includes the converted files plus the shell modules.
The L-shell rail gains its two deferred responsive modes:
- Desktop collapse (user preference, localStorage turnstone_interface.rail):
the rail shrinks to a 52px glyph-only strip — live Tier-1 state glyphs
remain the navigation, cluster pills stack as glyph+count, Manage becomes
one gear row opening the Admin pane, children flatten to peer glyphs.
Title attrs (now set unconditionally) carry the names; aria-labels were
already complete.
- Mobile drawer (max-width 768px): the rail leaves the grid and overlays
off-canvas at full width behind a scrim. Burger in the tab bar opens it
(focus moves into the rail); Escape (focus returns), scrim tap, or any
pane activation closes it. Closed drawer is visibility:hidden so its
buttons leave the Tab order. The collapse preference lies dormant here.
- Tab titles render in an ellipsizing span capped at 240px (48vw mobile)
instead of growing the tab unbounded; tab bar scrolls horizontally on
mobile.
A dead interactive controller's base goes stale once its node loses or
re-homes the ws, but menuBase() returned it first — so the close/delete
404-as-success lanes could silently drop a tab whose session is alive on
the node it re-homed to. Mirror the revive path: when isDead(), lead with
the live Tier-1 node and fall back to the stale base only when the ws is
gone cluster-wide (its 404 then correctly reads as "already closed").
Two reported console bugs, one shared root: a pane can outlive its
session, and nothing brought the two back together.
Reconnect: an interactive pane whose stream died (ws closed/evicted
elsewhere, node restart, re-home) could never reconnect while its tab
existed — openPane() on an existing pane was focus-only, the
controller's connect() is one-shot, and its 5s recovery loop re-dialed
the SAME node forever (infinite 404 polling through the console proxy).
The only workaround was closing the tab before resuming.
- createInteractivePane now tracks terminal failure: 3 consecutive
CLOSED recovery beats -> give up (stream closed, timers + any pending
history load invalidated, status bar "Disconnected", opts.onDead
fired once). host.onStreamOpen (new hook) resets the counter;
isDead()/markDead()/base join the controller surface; onLogin
ignores a dead controller — revive owns recovery, so a deliberately
closed session is never resurrected by a timer.
- PaneManager.openPane fires pane.onReopen(extra) when it targets an
ALREADY-OPEN pane — the explicit-intent signal (saved-list resume,
rail row, child link) that activate() can't carry (hooks no-op on the
active pane, and onActivate also fires on plain tab switches).
getPane() added for cross-cutting lifecycle signals.
- The shell paints a click-to-reconnect banner on give-up — and
immediately on Tier-1 ws_closed via the new
TS_SHELL.notifySessionClosed seam (the console keeps the tab, unlike
the standalone's auto-close, so the conversation stays readable).
Reopen/banner-click revives: tear down the dead controller,
re-resolve through the origin-first POST /open lane, rebuild. The
forced resolve skips BOTH beginConnect fast paths (a stale Tier-1 row
must not bypass /open) while a live node leads the hint chain (an
origin-first /open then reuses a genuinely-live session instead of
loading a duplicate on the old meta node). The standalone lane POSTs
its local /open on revive too — /events 404s on an unloaded ws.
- Coordinator parity: the factory exposes reconnect() (acts only on a
missing/CLOSED stream; OPEN is healthy, CONNECTING is already being
worked) and the pane's onReopen drives it — the saved-list resume
POSTs /open before openPane, so a fresh stream is all it needs.
Tab menu: a node-proxied interactive pane's dropdown gated every verb
on classic globals that only exist in ui/static/app.js, so the console
got a nearly-empty menu whose one surviving verb (Export) hit the
console origin and 404'd. convTabMenu gains a base-aware fallback lane:
verbs POST against the pane's OWN transport base (controller's exact
base -> persisted node hint -> live Tier-1 node; a verb is omitted
while no base is resolvable — never aimed at the wrong origin).
Close/Delete confirm first (window.confirm, the coordinator precedent)
and treat 404 as intent-satisfied (nothing left to stop/delete -> drop
the tab). exportWorkstreamDownload takes the base. The standalone
keeps its globals lane (incl. Fork) byte-identical, and an empty verb
section no longer renders a leading separator.
Verified: 189 JS-pin tests; two headless-Chrome live-DOM harnesses
driving the real modules — console 16/16 (connect -> ws_closed ->
banner -> reopen revives on a new node with the fresh hint -> give-up
stops retrying -> live-node-led resolve), standalone 10/10 (globals
menu intact, revive POSTs /open exactly once, no cluster resolve).
CI lock-check failed: the Fable 5 commit raised the anthropic floor to
>=0.108 in pyproject.toml but uv.lock still recorded >=0.39 / resolved
0.107.1. Regenerate the lock: anthropic 0.107.1 -> 0.108.0, specifier
0.39 -> 0.108 (no transitive changes).
Also address the Copilot review nit: the operator-instruction trust
declaration docstring wrote the fence marker as <system-reminder_<nonce>>;
align it to the emitted and project-standard <system-reminder_{nonce}>
notation.
- claude-fable-5 capability entry: 1M context / 128K output, adaptive
thinking (summarized display), effort low..max incl. xhigh, no
sampling params, web + tool search, vision, reasoning replay, native
mid-conversation system messages
- document the Fable 5 wire quirk at the capability table: an explicit
thinking={"type": "disabled"} is a 400 on this model; the adaptive
branch never emits "disabled", so adaptive-or-omitted is preserved
- widen the native mid-conversation-system comments from opus-4-8-only
to opus-4-8 + fable-5 (protocol, provider, tool_advisory, prompts,
session)
- raise the anthropic SDK floor 0.39 -> 0.108: 0.39 predates every
named kwarg the provider sends (output_config 0.77, top-level
cache_control 0.83, mid-conversation system blocks 0.105); 0.108
adds claude-fable-5
- tests: capability assertions for claude-fable-5 + dated-variant
prefix match
The pattern attribute on the MCP server-name and model-alias inputs used an unescaped hyphen in its character class. Browsers compile the HTML pattern attribute with the RegExp `v` flag, under which a literal `-` must be escaped — the class failed to compile, so the browser silently dropped the constraint and disabled client-side validation (Firefox). Escaping the hyphen leaves the matched set unchanged and consistent with the server-side ^[a-zA-Z0-9._-]+$ validators.
Add an inline-SVG data: URI favicon to the console, coordinator, and ui entry points so page loads no longer 404 on /favicon.ico. A data URI needs no new static route and survives the /node/{id} proxy path rewrite.
All findings validated against source before fixing; behaviour-preserving:
- coordinator.js: drop the unused `stripAnsi` import; `updateStatusBar` early-returns
on a null evt, so `(evt && evt.effort)` is simplified to `evt.effort` (no
redundant guard).
- pane.js `_onTablistKeydown`: drop the dead `let j = i` initial value — every
branch reassigns j before `tabs[j]` is read (the no-match case returns first).
- rail.js: collapse the redundant `ws.parent_ws_id ? "interactive" : "interactive"`
ternary to `ws.kind || "interactive"`. Tier-1 always stamps `kind`
(console/static/app.js defaults it to "interactive"), so the fallback was dead
and the parent-based arm would have mis-tagged a standalone interactive — the
single default mirrors the snapshot's own and is behaviour-identical.
- status_bar.js: correct the stale JSDoc — it's a THREE-cell bar now (tokens /
tools / turns); the model cell moved to the composer chip.
- ui/static/index.html: the tool-approval `a` shortcut help read "Always approve";
align it to the button language "Approve all".
Multi-stage review (find → verify → sanity) of b8914854 found one critical
bug plus four minor + one nit; all confirmed against source and fixed:
- CRITICAL — the interactive launcher's "Specific node" pick was unusable:
selecting a node fired the composer `change` event → onChange →
_applyLauncherFields → _populateLauncherNodes → setOptionChoices, which
rebuilds the <select> and reset it to the placeholder, wiping the selection
the instant it was made (submit then failed "Choose a node…"). Fix:
_populateLauncherNodes snapshots the current pick before the rebuild and
restores it after (setOptionValue does not dispatch `change`, so no loop).
- perf — every interactive open blocked first paint on a POST /open round-trip,
even on the hot rail / active-row paths where the ws is already live. The
pane now connects DIRECTLY when the Tier-1 snapshot already names the owning
node; only the dormant / reload case (snapshot empty) resolves + opens. This
resolves the uniform-vs-gated /open question left open last change; refresh
safety is unchanged (a reload activates before the snapshot lands → nodeForWs
null → resolve path).
- bug — an errored resolve (capacity / no node free) had no in-place retry
(re-clicking the active tab is a no-op); the error status line is now
click-to-retry.
- quality — resolveInteractiveNode surfaced each failure twice (toast + in-pane
line) with drifted wording; dropped the toasts, the in-pane status line is the
single source of truth.
- quality — corrected a setOptionFieldVisible comment that cited a nonexistent
"flex/grid rule" (the row is `display: contents`).
- nit — buildController skips the redundant sessionStorage re-persist when the
resolved node already matches the persisted hint.
Guard tests extended (bug-1 capture/restore, the live-direct path); the
behavioral harnesses were strengthened to fire a real selection rebuild and to
exercise the live-direct vs reload-resolve split that the first round missed.
Workstream-lifecycle bugfixes on the L-shell:
- Node-proxied interactive panes now SURVIVE a browser reload. On first
activate a pane resolves its owning node and (re)opens the session there
before streaming — the node /events stream 404s on a ws not loaded on its
node, so a rehydrated pane could not just connect blind. Resolution is
origin-first via the new TS_APP.resolveInteractiveNode seam (POST /open with
a rendezvous /route fallback). PaneManager now persists a pane's resolved
nodeId as opaque meta and hands it back on rehydrate, so a reload restores
the pane onto the SAME node even before the Tier-1 snapshot has populated —
the exact timing that used to strand it on base="" (the console, not a node).
- Both launcher personas open the new session as a PANE, not a full-page nav
(coordinator -> coordinator pane; interactive -> node-proxied pane); the
full-page nav stays only as the shell-absent fallback. Every interactive
entry point (create, active row, rail, saved row, child link, reload) now
funnels through one resolve-open-connect path, folding away the bespoke
restoreInteractiveSession helper.
- The interactive launcher gains a node-selection strategy (Least loaded |
Specific node, with a live node picker fed from the cluster snapshot) and a
persona-aware task hint — the shared composer no longer shows
"...coordinator orchestrate?" when the interactive persona is selected.
Guards updated to pin the new wiring; the stale console landing test (asserting
the renovation-retired bottom-bar node picker) is corrected to the rail.
The parallel tool-batch kicker strings ("Parallel · N tools", "Evaluating ·
Parallel N", "Running · Parallel N", "⚠ Approval · Parallel N") and the "1/N"
index label were duplicated byte-for-byte across interactive's three render paths
and the coordinator's kicker state machine, with no shared source enforcing the
visual parity the two surfaces require. Extract batchKicker(state, n) +
indexLabel(idx, n) into conversation.js (both files already import it) and route
all 13 sites through them, so a future label tweak can't silently diverge them.
Byte-identical, harness-verified: the rendered kicker + 1/N labels are unchanged.
From the multi-stage review of this session's changes (all minor):
- bug-1: the footer user-menu's deferred document-listener attach now bails if the
menu was already closed (closeUserMenu nulls the cleanup ref), closing a latent
listener-leak window.
- perf-1: fold paintConvTabGlyphs + paintConvTabTitles into one paintConvTabs — a
single findWs per stateful tab per Tier-1 render instead of two scans.
- q-2: remove the dead StatusBar.paint modelEl branch + its modelInfo arg (both
callers dropped it when the model moved to the composer chip) and the orphaned
.ws-sb-model CSS rules.
(q-1, the parallel-head string-helper extraction, follows separately.)
After the stacked composer box, the action row's margin-left:auto on the send
button pushed send to the right edge but left the mic stranded on the left next
to the model chip (the mic inserts before send). When the STT role is confirmed
the action row now gets .has-mic, which puts margin-left:auto on the MIC instead
so mic + send form a right-aligned cluster (send flush after the mic); without
STT, send keeps its own auto margin and sits alone on the right.
Verified (headless): with .has-mic the mic moves from x=427 (by the model chip)
to x=906 (38px left of send at 944); css audit at baseline 10.
Move the interactive + coordinator composers to the mock's layout: a compact
rounded composer box with the borderless textarea on top and the
[+] / model·effort chip / send row below (layout:"stacked"). The bordered inner
textarea and the boxed paperclip are gone — the box is the frame, and the attach
is a plain "+" glyph. Scoped via a composer--chat class (the model-chip hosts)
so the home launcher's taller stacked composer is untouched.
Verified by headless render against the mock; css audit at baseline 10, guards
green.
After the collapse fix, parallel tool calls render but the head read
"TOOL web_fetch + 1 more", which implies the rest are hidden. Match the
coordinator's presentation across all three interactive paths (announce /
inline / replay) + buildToolDiv:
- "Parallel · N tools" kicker (renders "PARALLEL · N TOOLS") instead of "Tool"
- the conv-batch--parallel class (the numbered-row connecting rail)
- per-row "1/N" index labels
so the "+ N more" summary now reads as a label, not hidden calls.
Verified in the real console shell (headless): a 2-call batch shows kicker
"Parallel · 2 tools", rows "1/2"/"2/2", conv-batch--parallel, both calls named,
no JS errors.
The real regression behind "parallel tool calls don't show / tool cards get
overwritten, leaving a thin stripe with a coloured pixel on the left": the
.conv-* convergence put an `overflow:hidden` card (.conv-batch) into the
interactive pane's SCROLLING flex-column message list
(.pane--embedded .pane-messages, overflow-y:auto). An overflow:hidden flex
item's `min-height:auto` resolves to 0, so flexbox squished the tool batch to
~2px (just its left border) once the column filled — while plain .msg blocks
(overflow visible) kept their height. That asymmetry is why the COORDINATOR
(different container) and main's old `.ts-approval` block (a .msg, overflow
visible) were never hit, and why it impacted ALL models — it was never a
local-model id-collision (that earlier theory + fix were reverted).
Fix: pin every message child to flex-shrink:0 so the column scrolls instead of
collapsing cards. Reproduced + verified in the real console shell (headless):
the parallel-bash tool batch went from 2px (collapsed) to 244px (full content)
once a multi-turn conversation fills the column. Guarded in test_conversation_css.
Per the BRIEFING the composer is the sole model location, but the model still
rendered in the per-pane status bar. Add a display-only "model · effort" chip to
the chat composer (next to the attach button) and remove the status-bar model
cell from both the interactive and coordinator panes; the status bar keeps
tokens/tools/turns. The chip repaints from the same model + status events, with
effort silent on the implicit "medium"/none (mirroring the status bar's old
suffix rule). A per-session model/effort PICKER is a separate deferred task
(needs a backend override path).
Verified in a real interactive pane (headless): the chip renders with its em-dash
placeholder, the status-bar model cell is gone (tokens/tools/turns remain), the
send glyph is intact, no JS errors; JS guards green, css audit at baseline 10.
The embedded message list stacked a 5px flex gap ON TOP of each .msg turn box's
4px margin-bottom (~9px of dead space between segments — "too thick"). Trim the
container gap to 2px so segments land at a compact ~6px, matching the mock's
gap-only intent without overriding the shared .msg margin (no specificity-audit
flip).
whoami returned only user_id (an opaque uuid), so the rail footer rendered the
uuid. whoami now resolves the user record by id and returns the human
username/display_name (best-effort — a storage miss just omits it). The client
stores data.username (no fallback to user_id: a uuid is worse than the generic
"account" placeholder). Hardened against a malformed user record (isinstance
dict guard) so a bad row can't 500 whoami; test stubs get_user + asserts the
display name is surfaced.
Local OpenAI-compatible models (e.g. DeepSeek) reuse tool-call ids across turns
(call_0, call_1 each turn). The interactive pane resolved a call's card via a
PANE-WIDE first-match messagesEl.querySelector('[data-call-id=...]'), so a later
turn's tool_result / verdict / warning / output-chunk landed on an EARLIER turn's
card — corrupting it and leaving the current batch's rows empty (a thin stripe).
It also made parallel calls look like they "didn't show" (the head summary sat
over emptied rows).
The pane is strictly serial, so a live result belongs to the MOST-RECENT matching
card. Add a _lastMatch(root, selector) helper and resolve the five live-path
lookups (appendToolOutput x2, appendToolOutputChunk, showOutputWarning,
updateVerdictBadge) to the LAST match instead of the first. The replay/history
path was already scoped to its block and is untouched.
Known limitation: if a model emits ALL parallel calls in ONE turn sharing the
same id, they still collide within the batch — a separate source-level issue.
The chat composers (coordinator + interactive) now render an up-arrow send glyph
instead of the text label, matching the mock. Opt-in via opts.sendGlyph so
creation-form composers keep their text label; the visible glyph is constant
while the textual sendLabel stays the aria-label and drives setBusy's a11y
rotation (setBusy no longer overwrites the glyph).
Verified in a real interactive pane (headless): the send button is the up-arrow
glyph with aria-label "Send message", no JS errors.
Conversational tabs froze at their open-time title — wsTitle(id), which is the
id-slice when the session isn't in the Tier-1 snapshot yet (e.g. a just-restored
saved session) — and never updated, so the tab read as the raw id instead of the
saved name.
Add PaneManager.setTabTitle(paneId, text) (mirrors setTabGlyph: rewrites the tab's
title text node in place + pane.title for a later rebuild) and a
paintConvTabTitles(pm) Tier-1 hook alongside the glyph repaint. It only UPGRADES
a tab to a real name (ws.name || ws.title) — never flickers a known name back to
the id if the ws blips out of a single frame.
Verified in the real console shell (headless): a named ws shows the name at open,
a dormant ws shows the id-slice then upgrades on repaint, no JS errors.
Mock-review batch (4 items):
- Footer: drop the redundant Admin button — Manage already surfaces every
admin tab, so only the theme toggle relocates from the retired header.
- Footer: the user chip now shows the real logged-in user. whoami returns
user_id but _storePermissions only persisted permissions, so the chip was
stuck on the "account" placeholder; it is now stored as ts.username and the
chip repaints once whoami lands (Tier-1 render hook).
- Footer: Log out moves into a click-menu on the user chip (reuses the
.tab-menu popup chrome; the item clicks the hidden #logout-btn so auth.js
stays the single owner of logout and its in-flight-refresh race guards).
- Manage: groups start collapsed instead of auto-expanding the first one —
the rail is a discovery map, not a wall of open links.
Verified end-to-end in the real console shell (headless): chip is a button
showing the user, no admin button in the footer, the menu opens with Log out
which invokes logout, outside-click/Escape close it, no JS errors.
Saved INTERACTIVE sessions opened from the console did nothing useful.
Coordinators rehydrate because their activation POSTs /open first; the
interactive branch only passed the saved DTO's node_id to openPane and
bailed "Session node unknown" when falsy. Even with a node_id nothing
streamed: the per-pane SSE /events 404s on a not-loaded ws and /history
alone does not rehydrate, so the session was never loaded onto a node.
New restoreInteractiveSession (console app.js) is ORIGIN-FIRST: POST /open
to the session's origin node (the DTO node_id, stamped at create) and pin
the pane there. This keeps node affinity and — load-bearing — REUSES a
session already live on its origin instead of loading a duplicate copy
elsewhere; the interactive pane talks directly to /node/{id} for every
verb, so the load-node and the pane-node must match (no split-brain).
Only when the origin is gone (POST /open 404 = not in registry / 502 =
unreachable) do we re-home onto a fresh rendezvous node via
GET /v1/api/route (the router skips dead nodes; persistence is shared
ws_id-keyed Postgres, so any live node is state-safe). Capacity (429) and
permission (403) are surfaced, not silently re-homed. No origin (legacy/
CLI rows) routes straight away. Mirrors the coordinator open-before-
navigate and the standalone dashboardResumeSession; the active-row path
(already-loaded sessions) is untouched.
Bugs surfaced by the live console (the headless harnesses stubbed data, so these
only showed against a real cluster):
- Saved-session AND active/filtered-table row clicks did full-page nav (interactive
-> /node/{node}/?ws_id=, coordinator -> /coordinator/{ws}) instead of opening an
L-shell tab. Route both through window.TS_SHELL.panes.openPane (interactive =
node-proxied pane, coordinator = coordinator pane). Full-page nav stays only as
the shell-absent fallback. (The broader ?ws_id= URL-pattern cleanup is deferred to
its own session.)
- The cluster health pills wrapped ("idle" fell to a second line) in the 266px rail.
Tightened gaps + font + nowrap so all three fit one line (verified at 266px).
- The [+] new-session button was a no-op when the Dashboard was already active
(showHome focuses it, no visible change). It now also focuses the launcher
composer via a new TS_APP.focusLauncher seam — "new session" lands you ready to
type.
- The cluster node list wasn't collapsible (unlike the Manage groups). The "Nodes"
header is now a toggle (button + rotating caret), state persisted across the
rail's Tier-1 re-renders.
Verified: node + prettier clean; CSS audit baseline; 89 JS guards; cluster-pill +
node-collapse render harnesses (pills one-line, toggle hides/shows + caret rotates);
wiring harnesses errs:[] (no regression).
Three shell-spine P3s from the review:
- Consolidated the three near-identical Tier-1 ws-scans (wsTitle / nodeForWs /
stateForWs each re-walked getClusterState -> nodes -> workstreams) into one
findWs(wsId, skipConsole) helper + three thin wrappers. nodeForWs keeps the
console-pseudo-node skip (coordinators live there, must not be node-proxied);
the other two scan all nodes. Also restored the convTabMenu doc comment that an
earlier glyph-helper insertion had orphaned above stateForWs.
- Simplified PaneManager.activate's dead/misleading guard (_activeId===paneId ||
!has, with a nested re-check) to the equivalent `if (!has) return;`.
- rehydrate now counts a pane restored only when openPane actually returns one — an
auth-gated (denied) coordinator pane returned null but still set restored=true,
which would suppress the Dashboard fallback into a blank shell (latent today: the
non-closable Dashboard is always in the persisted set).
Verified: 28 shell guards; mechanism harness 31/31 (activate/rehydrate/gate); both
wiring harnesses errs:[] (the scans drive the verified tab titles / node-proxy /
state glyphs — console running/idle, standalone full menu). node + prettier clean.
Three console front-door P3s from the review:
- Removed dead module state _lastOverviewJson / _lastNodePickerJson (memo caches
for the removed renderStatusBar / renderNodePicker; only declared, never read).
- Dropped the unreachable popstate view==="admin" branch — Admin is a rehydrated
PaneManager pane now, nothing pushes {view:"admin"}, and Back-from-admin already
lands on the dashboard via the home/filtered path.
- _createInteractive: added an else for a 200 without target_node so a server-
contract drift surfaces an error instead of silently stranding the user (the
branch is currently unreachable — the node is validated non-empty server-side —
but it was a silent failure mode).
Verified: node + prettier clean; 28 shell guards; console wiring harness errs:[].
getFocusedPane() has been a permanent `null` stub since the fork collapse
(PaneManager owns focus; interactive.js owns approval keys). That left ~90 LOC of
unreachable pane-dependent code — and it's exactly where the P1 closeTabDropdown
crash hid. Removed all 4 sites:
- the global-keydown Escape-cancel branch + the whole inline-approval keybinding
block (still referencing the retired .ts-approval-feedback / .verdict-* vocab);
every LIVE shortcut (Escape->dashboard, Ctrl+D/T/1-9, Ctrl+Shift+E/F/X, Ctrl+W)
is kept;
- the dashboardSubmit optimistic-echo block (interactive.js echoes its own turn);
- the new-ws modal model prefill (curModel is always "");
- the stub itself.
Also fixed _formatAttachSize: called 4x (chip size + over-cap error) but defined
nowhere -> a ReferenceError that broke file staging (pre-existing on main; flagged
by the review). Defined a local B/KB/MB formatter mirroring composer_attachments's
IIFE-local formatSize.
Verified: node + prettier clean; 61 app guards; standalone harness errs:[]; all
live keyboard-shortcut verbs asserted present after the splice.
The header removal (5e.2e) left the wait_for_workstream progress surface inert:
_waitIndicatorEl() mounts only into the deleted #coord-header and returns null, so
the whole #14 wait-indicator (handleWaitStarted/Progress/Ended, the activeWaits
Map, _renderWaitIndicator, the reconnect-path clear, the 3 SSE switch cases) ran
but rendered nothing. Ripped it (~110 LOC) + the orphaned .coord-wait-indicator
CSS rule. (The observability loss was a deliberate, tested design decision —
test_coordinator_page.py pins the header absence.)
The review's broader coord-chrome dead-CSS list was a false positive on
verification: `.task-row .status-done/.status-blocked` are LIVE (applied via a
dynamic `"status-" + status` class), `ts-spin` is live (coord-chrome.css:209), and
the rest (.coord-tool-*/.judging/.feed-item/.topbar) appear only in prose comments.
Also (review P3-1): clear aria-busy in _unsetBatchRunningIfAllResults so a batch
that completes via tool_result only (judge + gate bypassed, early-paint on) stops
announcing "busy" to screen readers after completion.
Verified: node + prettier clean; 16 coordinator guards; CSS audit baseline;
coord-keys harness all green (approval keys intact after the rip).
Two lower-severity findings from the full-branch review:
- Backend doc divergence (session_routes.py + console/server.py): the comments
justifying interactive's `state=None` saved listing claimed "the storage layer
already excludes state='deleted' tombstones" — but neither storage impl has such
a filter. It's incidentally safe because delete is a HARD delete (no `deleted`
tombstone is ever written), NOT because of a filter. Corrected both comments to
state the real mechanism + flag that a future soft-delete tombstone would need an
explicit `state != 'deleted'` guard here.
- Dead CSS: the entire #cluster-status-bar / .csb-* block (387 lines) was orphaned
— its HTML element + all JS writers were deleted earlier in this branch and
nothing reuses the vocabulary (grep-confirmed across console+shared). Removed the
main block (per-selector verified all-csb before splicing); the 9 residual csb
rules inside two MIXED @media blocks are left as a safe over-keep, matching the
5e.2f dead-CSS method.
Verified: ruff clean; CSS audit at baseline (zero new flips); braces balanced;
prettier clean; console builds errs:[].
The full-branch pre-push review (6 subsystem slices) found 1 P1 + several real P2
bugs; the confirmed functional ones, fixed here:
- P1 (standalone): two LIVE keydown branches called the deleted closeTabDropdown()
-> ReferenceError that silently killed Ctrl+Shift+E/F/X (edit/fork/delete) and
Ctrl+W (close) before reaching the verb. Removed the dead calls.
- coordinator (this branch's step-7 keys): the deny branch fired on any `d` with no
modifier guard, so Cmd+D (bookmark) / Ctrl+D / Alt+D silently DENIED a pending
batch. Early-return on ctrl/meta/alt (Shift+A still resolves).
- shell.js: window.TS_LOGIN was defined AFTER rehydrate(), so a RESTORED
conversational pane silently skipped its re-auth Tier-2 reconnect (onActivate saw
no TS_LOGIN and never re-fired). Moved the fan-out (+ TS_SHELL) above rehydrate.
- shell.js: TS_LOGIN.subscribe had no unsubscribe -> a closed pane leaked its
controller closure across open/close/re-login. Added unsubscribe + call it in
both onClose hooks.
- standalone dashboard: updateTabIndicator dropped its `extra` arg in the fork
collapse, so a watched row's STATE/TOKENS/CTX went stale on every ws_state tick
until a full reload. Ported the in-place row patch from main (sans the retired
.ws-tab indicator).
Also dropped a redundant index.html NODE-gate comment (the CSS rule + loadDashboard
already document it) that had tipped a fragile 4000-char structural guard.
Verified: 105 JS guards green; mechanism (31/31) + coord-keys (all) + wiring
harnesses errs:[]; node + prettier clean.
The merge-gate designer pass on the CONSOLE persona (coordinator + interactive,
caps on) found 0 P1 — ship-quality — and 4 small CSS P2s, applied here:
- P2-1 (non-optional AA): the pending approval card's "APPROVAL NEEDED" kicker was
4.17:1 in light (sub-AA on the operator's primary decision signal). Darken off
raw --warn via a theme-tracking color-mix toward --ink-2 (~5.5:1 light; dark
stays warm + passing). In conversation.css, so both personas benefit.
- P2-2: rail micro-labels (.sec-label/.nlabel/.node-row .ver/.grp-head .gcount)
were --ink-4 <=11px = ~3.9-4.0:1 in light. Light-scoped --ink-3 (~6.5:1), the
same escape hatch as .tab-menu-key; drift versions keep --yellow.
- P2-3: the tab-dropdown separator was imperceptible. Use the MORE-visible hairline
per theme (--hair-2 dark / --hair light) — the designer's suggested tokens were
reversed; corrected against the actual hex values.
- P2-4: the relocated "Reconnecting..." rail-conn read as an alarming top-of-rail
peer of Cluster health. Quiet it (10px, collapses when connected) + --warn (not
the near-error --yellow) on disconnect.
- P3-5: the Manage active-tab marker (inset --hair-2) nearly vanished in light ->
--ink-4 (reads in both themes, still not the amber `.open` of a live session).
Verified: CSS audit at baseline (zero new flips); 33 conversation/shell guards
green; node + prettier clean. Other P3s noted (model-chip disclosure is the locked
"model lives only in the composer" decision; node-row rhythm / verdict-expand minor).
The settings MODAL backdrop (#settings-overlay) died when MCP connections moved to
the Manage > Connections pane (step 6) — the #settings-mcp-* content rules are
reused there, but the overlay wrapper is gone. Remove the closed loop of dead-but-
mutually-alive references: the CSS rule, the stale "settings-overlay" modal-id
array entry, and the guarded getElementById no-op in the settings-close path.
The other fork-collapse dead-code (the getFocusedPane stub + its null-gated
branches, the partially-retired settings-gear) is woven into still-live handlers —
deferred to the merge-gate /review for a systematic sweep with the review findings.
Verified: 0 settings-overlay refs remain; node + prettier clean; CSS audit at
baseline; standalone harness errs:[].
The console twin of the interactive.js approval-key fix: the coordinator's
tool-batch card shows kbd hints (Enter approve / D deny / Shift+A approve-all) but
they did nothing — the keys were never wired (the standalone routed approval keys
through the app.js global keydown + getFocusedPane, retired in the fork collapse).
Add a pane-owned keydown on `root` that resolves the current pending batch:
- _currentPendingBatch() finds the last .conv-batch with a still-pending
[data-needs-approval="1"] row whose actions aren't already disabled — the
in-flight double-fire guard (a second key during the resolve is a no-op).
- Enter -> approve, D/Esc -> deny, Shift+A -> approve-all, routed to the existing
_resolveBatchAction path.
- A focus guard skips when an input/textarea/contenteditable is focused, so the
keys never hijack composer typing (the coordinator has no feedback field, unlike
interactive, so no feedback special-case).
Verified end-to-end against the real coord pane (keydown -> _currentPendingBatch ->
_resolveBatchAction -> approveWorkstream -> postJSON -> authFetch, stubbed at the
HTTP boundary): Enter/D/Shift+A fire the right verb, the double-fire + focus guards
hold, errs:[] + a coordinator JS guard. Live keypress confirm rides the merge gate.
The WORKSTREAMS table showed a NODE column (a multi-node console-ism) on the
single-node standalone server, where every row reads "local". Drop it: remove the
NODE header span + skip the node cell in loadDashboard, and gate just that table to
6 columns by overriding the --dash-grid VARIABLE (not the grid-template-columns
property — so it stays the var's single declaration, no cascade flip), scoped by
id, so the shared --dash-grid and the Saved Workstreams table keep their 7-col
layout. Matches the brief's capability-derived-affordances thesis (the rail drops
Cluster the same way). Designer P2 (pre-existing, not a step-6 regression).
Verified: standalone DOM shows 0 dash-col-node + the saved table intact; CSS audit
at baseline (zero new flips); node + prettier clean.
The brief defers mobile: the rail -> off-canvas drawer matches no current DS scope
(the DS is desktop-only; the console's mobile drawer was retired in step 3b).
Record the decision in-code at the .app layout seam (the brief is local-only) where
a future max-width @media would slot in. Verified narrow viewports (720px wide)
are cramped, not broken — no silent mobile-support claim.
openPane now auth-gates pane CREATION via an optional per-type canOpen predicate
(deny -> no pane; focusing an already-open pane is never re-gated). PaneManager
stays generic — it holds a _gates map and consults canOpen/onDeny; the shell
supplies the gate.
The coordinator type gates on the admin.coordinator scope — the SAME
sessionStorage-backed _hasCoordPermission helper the launcher + saved-list use.
Because every coordinator open path (rail click, child-link, rehydrate, [+]
launcher) routes through openPane, this gates them all at once — closing the gap
where a rail click opened a coordinator pane a user lacked scope for (it then
404'd server-side). Perms live in sessionStorage so they survive a refresh →
rehydrate gates correctly (an operator's persisted coord pane restores, a
non-operator's is skipped). The backend enforces the scope too; this just avoids
opening a doomed pane.
Verified: 31/31 mechanism harness (gate allow/deny/onDeny) + real-stack console
wiring (authorized operator opens the coord pane; a no-permission stub denies a
new coord pane, gateDenied:true, errs:[]) + a shell JS guard + CSS audit baseline.
The right-floated tabbar tail (empty since the scaffold) gets a [+] button that
focuses the persona launcher — the Dashboard pane hosts the unified
coordinator/interactive launcher, and a new session needs a task prompt, so "new
session" composes there. Cross-deployment via window.showHome (both the console
and standalone expose it) with a pm.openPane("dashboard") fallback; reuses the
scaffold's .tab-add styling. Auth stays the launcher's concern (it gates each
persona option), so focusing it is always safe.
Verified: renders in the real shell (standalone harness DOM) + a shell JS guard.
Conversational tabs now show a live shape+colour state glyph (● ◐ ⚠ ✗ ○) instead
of the static ◆/○ placeholders the header removal (5e.2e) left behind — driven by
the SAME Tier-1 source + builder the rail uses, so tab and rail always agree.
- rail.js: export the glyph() builder (one source of truth for the mapping).
- pane.js: ShellPane.stateful + PaneManager.setTabGlyph/statefulTabs — generic
(PaneManager owns no glyph vocabulary; the shell passes the built element).
A stateful pane builds no static glyph; the shell paints a live .ui-glyph.
- shell.js: stateForWs() reads the Tier-1 snapshot; paintConvTabGlyphs() repaints
every stateful tab on each Tier-1 render (subscribed to TS_APP.onRender) + per
pane on activate. Coordinator + interactive panes are now stateful.
- shell.css: .tab .tab-glyph spacing (static + live); live glyphs keep their own
.ui-glyph-* state colour (no .tab .glyph override).
SINGLE WRITER: the tab glyph is written only by the Tier-1 path (the pane's Tier-2
stream drives its body, not the tab) — no two-tier race, no stale open-time
placeholder on reconnect (BRIEFING L144-147). A coordinator-telemetry-parity gap
(open Q#2) would stale tab + rail equally, consistently.
Verified: 27/27 mechanism harness (8 new glyph asserts) + real-stack wiring
harnesses (console coord ui-glyph-running / int ui-glyph-idle; standalone int
ui-glyph-running — matching the stubbed Tier-1 state, errs:[]) + 26 shell JS
guards + CSS audit at baseline.
PaneManager tabs gain a caret opening a generic, keyboard-navigable action
dropdown — recovering the affordances the pane-header removal (5e.2e) dropped.
The mechanism is generic; the item set is pane-type AND deployment derived.
- pane.js: the caret (a <span>, not a nested <button>) + _openTabMenu/_closeTabMenu
— singleton, right-anchored under the caret with overflow flip + viewport clamp,
Arrow/Home/End/Esc/Tab nav, ContextMenu/Shift+F10 + right-click open.
- shell.css: the .tab-menu chrome promoted to the SHARED sheet (both deployments),
recovered from the retired .ws-tab-dropdown design but translated onto the DS
token vocabulary (--panel-2/--hair-2/--ink-*/--err).
- shell.js: convTabMenu wires each type by capability/feature-detection —
coordinator: Export · Close pane · Close workstream (its controller's
closeSession — the Export + end removed from its header land here)
standalone interactive: Refresh/Edit/Fork · Export · Close pane ·
Close workstream · Delete (classic ui/static globals)
console interactive: Export · Close pane (those globals are standalone-only)
admin: Close pane
Three-verb close is load-bearing: Close pane (drop tab) != Close workstream
(stop session) != Delete (destroy + unsave).
Designer-reviewed both personas, dark+light: resting danger cue on Delete (never
colour-alone), elevated --panel-2 surface, accent-wash hover, light key-hint AA,
viewport y-clamp + max-height.
Verified: 19/19 mechanism harness + real-stack wiring harnesses (all three menus,
errs:[]) + 25 shell JS guards + CSS audit at baseline (zero new flips).
The converged .conv-* card advertises y/n/a (+Enter/Esc) kbd hints, but the keys
did nothing in the L-shell: the only handler was the old standalone app.js global
keydown gated on getFocusedPane(), which the fork collapse stubbed to null — so
Approve/Deny/Approve-all were mouse-only (the chips over-promised), and that dead
block also queried the retired .ts-approval-feedback class.
Wire the keys pane-owned on this.el (every embedded L-shell pane), restoring the
pre-regression behavior + using the converged .conv-feedback: when a pending
approval is up, in the feedback field Enter approves (with feedback) / Esc denies
and other keys type; elsewhere y|Enter approve, n|Esc deny, a = approve-all. The
composer is disabled while pending, so the feedback field is the only typing
surface. This fixes both the standalone and the console interactive pane (shared
interactive.js); the coordinator pane's keys (console-only) are a separate
merge-gate item. Verified: node-check, the headless shell harness still builds
clean (errs:[]), 102 JS guards green incl. a new wiring guard. The live keypress
-> resolve confirm rides the owed live-backend pass.
Removes the structurally-dead CSS the L-shell superseded — 794 lines: the tab
bar (.ws-tab*, #tab-bar, #split-btn, .ws-tab-dropdown-*), the binary split-pane
machinery (.split-*, #split-root, .pane-ctx-*), the old approval/verdict card
(.ts-approval-*, .verdict-*, the judge spinner), the fixed .dashboard-overlay,
and the retired appbar/settings-overlay bits — plus their [data-theme=light]
overrides. The standalone now styles its conversation from the shared sheets
(chat/conversation/interactive.css); the dashboard table + saved list were
always shared (base/cards.css).
Method: a conservative token-diff — a rule is dropped only when EVERY selector's
class/id token is absent (word-boundary, comments stripped) from the standalone
runtime (index.html + every JS it loads, incl. the vendored hljs/katex/mermaid
so their runtime-built classes aren't mistaken for dead). Mixed/any-live rules
are kept verbatim (no reformatting), so ~50 dead-but-harmless rules that share a
generic token like `.active` survive — safe over-keep. The markdown / syntax /
math / diagram theme lives ONLY in this style.css (the shared sheets don't carry
it), so the hljs/katex/mermaid families are protected from removal.
Also fixes four dead tab-DOM pokes in app.js (editWorkstreamTitle /
confirmDeleteWorkstream read the title from the workstreams roster now, not the
retired .ws-tab .tab-name; the cancel handlers drop the gone .tab-chevron focus
restore).
Verified: braces balanced (370/370), the headless harness still builds clean
(errs:[]), git diff confirms zero live dashboard/render rules removed, and the
css_specificity_audit (manifest synced to the standalone's new sheet set) shows
the SAME 10 pre-existing findings before/after — zero new cascade flips (removing
a rule for a non-existent selector can't change any live element's cascade).
121 JS guards green, ruff/mypy clean.
The renovation META-GOAL: a standalone turnstone-server now serves the SAME
capability-parameterised L-shell the console serves (caps {cluster:false,
orchestration:false}), collapsing the console/static vs ui/static fork. No server
change was needed — turnstone/server.py already mounts ui/static at /static; this
changes what ui/static CONTAINS.
ui/static/index.html -> the L-shell skeleton: a hidden #header the shell
relocates (status -> rail, theme/logout -> footer), #main as the Dashboard pane
body (launcher + workstreams table + saved list), a one-panel #view-admin hosting
MCP connections (reusing the #settings-mcp-* table ids), the modals, and the caps
block flipped to {cluster:false, orchestration:false, brandSub:server}. The
split-pane chrome (#tab-bar/#split-root/#split-btn), admin.js/governance.js, and
the separate interactive.js module tag are gone (shell.js imports it).
ui/static/app.js -> a single-node TS_APP/TS_ADMIN/showHome provider (-1417 lines):
- TS_APP.{getClusterState, onRender, bucketByParent, boot}: getClusterState
synthesizes a one-node cluster from the flat /v1/api/events/global roster;
boot() is shell-driven (no parse-time auto-run).
- TS_ADMIN: a one-tab Manage IA (Extensions > Connections) whose openTab opens
the Admin pane + renders the MCP table — the floating settings gear is retired.
- The binary split-pane machinery (layout tree, splitPane/renderLayout, tab bar,
context menu, tab dropdown, STANDALONE_HOST, createPane, the gear menu) is
deleted; the keep surfaces (dashboard, global SSE, new-ws modal, MCP
consent/connections, health/theme/kb) are rewired onto PaneManager + the rail
(switchTab/renderTabBar/showDashboard become thin shims; sessions open as
interactive panes).
interactive.js -> the window.InteractivePane bridge is retired (the shell imports
the factory in both deployments; nothing reads the global anymore).
JS guards re-pointed to the L-shell reality (gear/split-pane/window-bridge guards).
126 JS guards green, ruff/mypy clean, node clean. Verified in a headless harness:
the standalone shell builds caps-off (rail = Workspaces + Manage > Connections, no
Cluster), the Dashboard pane adopts #main, TS_APP/TS_ADMIN wired, zero uncaught JS
errors. The owed merge-gate passes (designer both personas + live-backend +
/review) are unchanged.
Two changes so the SAME shell mounts on a standalone turnstone-server (step 6's
META-GOAL: collapse the console/static vs ui/static fork):
- The coordinator pane import is LAZY + gated on caps.orchestration. A static
`import ... from "/static/coordinator/coordinator.js"` 404s on a standalone
server (whose /static is ui/static, no coordinator file) and aborts the whole
shell module. It's now `await import()` inside mountShell, before rehydrate,
registered only when the deployment has orchestration (the console); a
persisted coordinator pane then degrades to a rehydrate skip. mountShell is
now async.
- The interactive pane's nodeId is gated on caps.cluster. Node-proxy transport
only exists in a cluster deployment; on a single-node standalone every session
is LOCAL, so nodeId stays null -> the pane uses base="" (no /node/<id> hop),
even though the synthesized one-node clusterState names a node.
Console behaviour is identical (orchestration:true -> the import runs + the
coordinator registers; cluster:true -> the nodeId ternary's true branch is the
original expression). Verified both personas build clean in a headless harness:
console = [Cluster, Workspaces, Manage] + coordinator registered, no errors;
standalone = caps off, no Cluster, no coordinator, no errors.
Same as the coordinator: in the L-shell the embedded interactive pane's header
content (workstream name + INTERACTIVE persona tag) is redundant — the tab shows
the name and the rail (Workspaces) shows name + state + the INT/COORD persona.
So the embedded pane builds no header and the conversation reclaims the height.
The header build is gated behind !this._embedded (the standalone split-pane,
retired in step 6, keeps its split/close header). The --skip-permissions
SECURITY banner moves off the header to messagesEl (the console host's
warningTarget now matches the default host) so it's preserved, not dropped.
updateWsName null-guards the absent header. Dead .pane--embedded .pane-header /
.pane-ws-name / .pane-persona-tag CSS removed.
Both panes are now header-less in the L-shell — name/state/persona live in the
tab + rail. 103 JS guards green (test_embedded_chrome_is_gated updated for the
no-header reality), node-check clean.
In the L-shell a coordinator is always a pane (no standalone page), and
everything the header showed is redundant: the name + state live in the pane
tab and the rail (Workspaces); end / export moved to the tab dropdown (step 7);
the light/dark toggle is in the rail footer. So drop the header entirely and
give the wasted vertical pixels to the conversation.
buildCoordChrome no longer builds an appbar. The busy/wait indicator
self-disables (its #coord-header mount host is gone; busy shows in the rail
glyph). The per-pane SSE-connection indicator is dropped — reconnect handles
transient drops, matching the interactive pane (which has none); setSseStatus
and the name/state writes are null-guarded. End/export logic stays reachable
for step 7 (exportWorkstreamDownload(wsId); the pane's closeSession() API).
15 coordinator guards green, node-check clean. (Interactive pane header is next
— it hosts the --skip-permissions security banner, which moves to a pane-top
slot so removing the header doesn't drop a security warning.)
The cutover delegated the pane's verdict/warning/status DOM to the shared
conversation.js builders, so three test_app_js guards pinning the old
implementation needed updating to the new reality:
- replayHistory now calls buildConvVerdict(tc.verdict) (was renderVerdictBadge).
- risk normalization moved into the shared builders — the pane carries no raw
`risk_level || "medium"` fallback and builds via buildConvVerdict /
buildConvWarning (which route through normalizeRiskLevel in conversation.js).
- the error pill converged onto .conv-status--error (was .ts-approval-badge--error).
Also dropped the now-dead normalizeRiskLevel import from interactive.js (the
builders own normalization; the pane no longer calls it directly). 123 frontend
JS/CSS guards green.
The emitters switched to .conv-* (conversation.css), so the forked card
vocabularies are now dead (match nothing). Remove them:
- coordinator.css: the whole .coord-tool-* tool-batch construct (-552).
- chat.css: the .ts-approval-* / .ts-verdict-* approval shell (-261); dead
selectors grouped with kept ones in reduced-motion/hover @media blocks were
stripped from the group, not the whole rule.
- interactive.css: the .ts-approval-* / .verdict-* / .output-warning* / tool-div
internals (-436), keeping the .tool-output collapse/stream + .media-* result
subsystem (interactive-only live-execution affordances). Also fixed a latent
malformed-comment bug — the file header had ".ts-approval-*/" whose "*/"
accidentally closed the comment, leaving the rest as stray CSS the browser
silently dropped.
~1249 lines of dead CSS gone. Verified: 0 dead selectors remain across all four
sheets (comments aside), kept anchors present, braces balanced, prettier-clean,
27 JS/CSS guards green. The cards render entirely via conversation.css.
renderApprovalBlock's child-approval pill mapped an unknown/unrecognized
risk_level -> .high (a deliberate "fail-safe over-alert"). Per the user's
2026-06-06 decision, fold it onto the canonical unknown->medium (5e.1b): the
separate "(judge unavailable)" pill already covers the genuinely-unassessed
case, so this path only fired for a malformed risk_level on an otherwise-present
verdict — a rare data edge, not a "we didn't check" signal. Unknown now maps to
.med, consistent with how every other surface (the shared builders via
normalizeRiskLevel) displays it. No remaining inline `|| "medium"` risk
fallbacks in either emitter.
Re-vocabularize interactive.js's approval card onto the shared conversation.js
builders, converging it with the coordinator onto ONE neutral .conv-* card:
buildToolDiv -> buildConvRow + buildConvCmd, renderVerdictBadge ->
buildConvVerdict, _buildOutputWarningEl -> buildConvWarning; showInlineToolBlock
/ announceToolBlock build the .conv-batch shell + head + rows + buildConvActions
(with the inline feedback + recommended glow); resolveApproval / the auto path
-> buildConvStatus; updateVerdictBadge replaces the badge via buildConvVerdict;
the history-replay branch synthesizes the live `item` shape so replay renders
the SAME .conv-row. ~124 ts-approval-* / verdict-* references gone (a final
no-stale-vocab assert guarded it); dead toggleVerdictDetail removed.
The card chrome converges; interactive's richer post-execution result subsystem
(.tool-output collapse/stream + .media-* embeds) is KEPT as-is — those are
live-execution affordances the read-only coordinator history doesn't need.
Block state classes move to the BEM modifiers (.conv-batch--approved/--denied/
--error/--auto); per-pane keybindings (y/n/a) preserved via the builder kbd
hint. The now-dead old card CSS (.coord-tool-*/.ts-approval-*/.verdict-*) is
inert (matches nothing) and is removed in 5e.2f's CSS dedup.
Verified: node --check both emitters; the no-stale asserts; 60 JS guards green
(test_coordinator_page pinned vocab updated coord-tool-batch-> conv-batch). The
builders are behavior-tested (5e.2b). Designer + /review + live-backend run once
at the merge gate.
Re-vocabularize coordinator.js's tool-batch construct onto the shared
conversation.js builders (5e.2b): _renderBatchRow -> buildConvRow,
_appendVerdictLineTo -> buildConvVerdict, _attachOutputWarningChip +
appendGuardFinding -> buildConvWarning, _appendResultToRow -> buildConvResult,
_buildBatchActions -> buildConvActions, _buildStatusPill -> buildConvStatus,
the appendToolBatch shell -> buildConvBatchShell. All 115 .coord-tool-*
references (including the security-critical _resolveBatchAction call_id
selector and the SSE upgrade-in-place handlers) renamed to .conv-* (a final
"no coord-tool- remains" assert guarded the rename). Dead _makeActionButton
removed (buildConvActions replaces it).
The verdict converges on the richer expandable badge; its rationale folds into
the verdict detail (the separate .coord-tool-row-rationale <details> is gone).
The warning rationale is now inline. Coordinator renders via conversation.css
(linked since 5e.2a); the old .coord-tool-* rules in coordinator.css are now
dead and get deleted with the interactive switch. Child-approval block
(.approval-*) untouched here — it converges in 5e.2d.
net -249 lines. Verified: node --check + the no-stray-ref assert; the builders
are behavior-tested (5e.2b, 48 asserts). Holistic both-panes harness + designer
pass land after the interactive switch.
Add the pure leaf DOM builders for the unified `.conv-*` card to
conversation.js (both panes import it): buildConvBatchShell / buildConvRow /
buildConvCmd / buildConvVerdict / buildConvWarning / buildConvButton /
buildConvActions / buildConvStatus / buildConvResult. The builders own only
the DOM + class vocabulary; everything stateful (the toolRows map, idempotent
upgrade-in-place, the early-paint announce shell, SSE routing) stays in each
pane and CALLS these in 5e.2c.
Parameterized by AFFORDANCE, not subclass: buildConvRow takes an indexLabel
(coordinator's parallel idx pill) or defers to buildConvCmd (interactive's
bash `$ cmd` + diff preview); buildConvActions takes per-pane keybinding hints
+ an optional feedback input (interactive) and per-pane resolve callbacks. The
persistent action unifies on "Approve all" (dashed --ok ghost), not the
coordinator's old "Always". Risk routes through normalizeRiskLevel so the
per-site `|| "medium"` fallbacks fold onto the canonical unknown->medium; the
judging spinner withholds the --{risk} class so its stripe stays neutral.
Additive — no emitter calls these yet (the re-vocabularize + delete is 5e.2c).
Verified: 48-assert headless-Chrome behavior harness (DOM shape, crit->critical
normalize, unknown->medium fold, expand toggle, action callbacks, JSON
pretty-print) + tests/test_conversation_js.py extended (11) + node --check.
Author shared_static/conversation.css: ONE neutral `.conv-*` approval-card
vocabulary that both panes will emit, converging the two forked cards
(coordinator's `.coord-tool-*` + interactive's `.ts-approval-*`/`.verdict-*`).
Based on the BRIEFING-blessed `.coord-tool-batch` idiom — neutral surface,
state left-stripe (warn pending / ok approved / err denied), uppercase kicker,
Approve = subtle --ok fill / Approve all = dashed --ok ghost / Deny = --err
(the DS hard-rule: approve uses --ok, never --warn) — with the interactive
affordances folded in (bash `$ cmd`, unified-diff preview, inline feedback,
recommended-button glow, auto-approved tag, the expandable verdict detail).
Converges onto the DS token vocabulary (--ok/--warn/--err, --ink-*, --panel*,
--hair*), not chat.css's legacy --green/--red/--cyan. Self-contained spinner
keyframe (conv-spin) so the sheet doesn't depend on coord-chrome.css's ts-spin,
which the standalone interactive pane never loads.
Additive only — no emitter uses `.conv-*` yet (the re-vocabularize + delete of
the old sheets is 5e.2c). Linked from the console + both standalone pages so
the card is styled the moment 5e.2c switches the emitters over.
Designer-reviewed (rendered both themes): applied the warning-chip wrap fix,
the medium-severity-weight fix (12% mix, not raw --warn-tint), ink-4 -> ink-3
on verdict-detail/tier content for light-mode AA, the bold severity label, and
the neutral judging-row stripe. Guard: tests/test_conversation_css.py (5).
Both panes carried their own risk-level logic that disagreed on the fallback:
interactive's normalizeRiskLevel sent an unknown level to "medium" (and, lacking
the crit/med aliases, rendered a "crit" verdict as medium), while the
coordinator's _riskRank sent unknown to "high". Lift one canonical normalize +
rank into conversation.js and route both panes through it.
- conversation.js: normalizeRiskLevel (aliases crit->critical / med->medium;
unknown -> "medium"), riskRank, maxSeverityItem (keeps the no-verdict -> -1
edge so an unassessed item never wins the max-severity pick).
- interactive.js: import normalizeRiskLevel, drop the local copy (its 3 callers
unchanged); a "crit" verdict now renders critical instead of medium.
- coordinator.js: import maxSeverityItem, drop RISK_SEVERITY / _riskRank /
_maxSeverityItem; an unknown-level item now ranks medium, not high.
- unknown -> medium is the deliberate fallback (per decision), not "high":
medium is the neutral default both panes' displays already used.
- tests: conversation guards for the fallback + aliases + the no-verdict edge;
the two pane guards now check the shared module.
The per-site risk->CSS-class display mappings (coordinator's inline chips and
renderApprovalBlock's deliberate unknown->crit over-alert pill) are left for the
5e.2 vocabulary reconcile.
Stand up shared_static/conversation.js as the deduplicated conversational-pane
substrate both panes import (interactive via ./, the coordinator via /shared/ —
both ES modules since 5e.0). First tenants are the byte-identical duplicates the
in-file comments flagged for the step-5e lift: stripAnsi, the watch-result card
builder, and the system-nudge marker. No visible change — the builders return
the same DOM; each caller still appends + scrolls.
- conversation.js: stripAnsi (null-safe variant), buildWatchResultCard,
buildSystemNudgeMarker.
- interactive.js / coordinator.js: import the three, drop their local copies,
delegate appendWatchResult + the nudge marker through the shared builders.
- stripAnsi unified on the coordinator's null-safe form (interactive's threw on a
non-string arg); identical output for string inputs.
- tests: new test_conversation_js.py pins the module; the two retry-walk guards
now check the watch-result marker in conversation.js (it moved there).
- refreshes interactive.js's header comment, stale since 5e.0 made the
coordinator an ES module too.
Lift coordinator.js off the window.createCoordinatorPane bridge onto a real ESM
export, so the upcoming shared conversational module (5e) is import-consumed on
both sides rather than through a classic window global. The console shell and the
standalone page's bootstrap both import the factory now; zero behaviour change.
- coordinator.js: export the factory, drop the window bridge — no classic
consumer remains (unlike interactive.js, whose ui/static app.js still uses its
global).
- shell.js: import the coordinator factory by URL (mirrors the interactive
import) and call it directly.
- console index.html: stop script-tagging coordinator.js; shell.js's import loads
it (a classic tag chokes on the top-level export).
- coordinator/index.html: the standalone bootstrap becomes a module that imports
the factory — a classic eager IIFE ran before the deferred module loaded it.
- tests: pin the new ESM seam (export, shell import, module bootstrap).
Designer pass on the new console interactive pane. Two clean fixes; the rest of
the findings are scoped to their planned steps (see below).
- Rail Workspaces `.open` marker now tracks the ACTIVE pane instead of being
hardcoded to Dashboard — the rail map and the tab bar were disagreeing about
what's focused (opening a session never moved the rail highlight). PaneManager
gains getActive() + onActiveChange() and fans out on activate/close; the rail
keys `.open` off the active pane's rawId and re-renders on activation (not just
on the next Tier-1 snapshot).
- The active tab's glyph brightens (--ink-4 -> --ink-2) so an open session's `○`
placeholder doesn't read permanently "idle" beside its live (running ●) rail
row. (Tab glyphs go fully live in step 7.)
Deferred (planned elsewhere, not regressions): the tab CLOSE affordance is step 7
(the brief's three-verb `.ws-tab-dropdown`); the interactive/coordinator HEADER
consistency is what the step-5e base lift unifies (a shared header parameterized
by affordances), so partial coordinator-header surgery now would be a half-measure.
Verified: a headless screenshot (rail `.open` now on the active session, slim
header + persona tag render clean) + 107 JS-guard tests (test_shell_js 5d guard),
node --check, prettier, ruff.
Rewire the coordinator's child ws links (deferred from step 4) to open the child
as a node-proxied interactive pane inside the console L-shell, instead of a
full-page new tab to /node/{id}/?ws_id=.
- A delegated click handler on the pane root catches .ws-link (children tree,
renderChildRow) and .coord-ws-link (linkified tool output, renderToolOutput)
clicks; both link types now carry data-ws-id + data-node-id. When a
PaneManager is present it opens openPane('interactive', child_ws_id,
{nodeId: child_node_id}) — the child's OWN node, so its stream proxies to that
node even though the coordinator lives in the console.
- Progressive enhancement: the link's href (/node/{node}/?ws_id=) stays the
standalone fallback — the standalone coordinator page has no PaneManager, so
the new-tab nav stands. No innerHTML introduced (the tool-output linkifier
still returns a string; only data-* attrs were added).
Verified: a harness running the console stack (coordinator.js + shell.js +
interactive.js) — open a coordinator pane, click a child link -> a node-proxied
interactive pane opens (/node/{child_node}/.../events), zero errors; 106
JS-guard tests (test_coordinator_page step-5c guard), node --check, prettier.
Wire the shared interactive Pane (5a) into the console L-shell as a ws_id-keyed,
node-proxied conversational pane.
- shell.js IMPORTS createInteractivePane (interactive is a real ES module, so
the shell consumes it the modern way; the legacy coordinator pane stays on the
window.* seam — the incremental "pulled by the adopting pane" modernization).
registerType('interactive') mirrors the coordinator: build on mount, connect
on activate (idempotent) + login re-arm, deactivate on tab-away (stops
focus-stealing while the stream stays live), destroy on close.
- The node-proxy target is DERIVED from the Tier-1 snapshot (nodeForWs), so a
rehydrated pane needs no persisted node_id; a rail click / child link can pass
{nodeId} as an open-time hint. openPane(type, id, extra) threads that hint to
the factory (not persisted).
- rail.js: interactive session clicks now openPane('interactive', ws.id,
{nodeId: ws.node}) instead of full-page nav to /node/{id}/.
- interactive.css (new, shared): the embedded slim-header layout (scoped to
.pane--embedded so it never collides with the ShellPane's own .pane section —
the brief's namespace watch-out) + the conversational rendering (tool output /
media / MCP-error / verdict / output-guard cards) COPIED from ui/static. The
shared chat.css .msg/.ts-approval base is left untouched, so the coordinator
pane is unaffected; step 5e unifies the vocabularies, and ui/static keeps its
copy for the standalone until step 6.
Verified: an integration harness running the REAL shell.js + rail.js +
interactive.js (register -> rail-open -> embedded chrome -> node-proxy SSE
/node/{id}/.../events -> /history replay into real .msg turns -> destroy, zero
errors) + a screenshot; 105 JS-guard tests (test_shell_js step-5 guard),
node --check, prettier, ruff.
Lift the per-workstream conversational Pane (chat + approval cards + composer +
voice + tool/media/MCP-error/verdict rendering) out of ui/static/app.js into a
new shared ES module shared_static/interactive.js, so BOTH deployments can
mount it: the standalone turnstone-server UI (its split-pane shell stays in
app.js and builds panes via window.InteractivePane) and — next, in step 5b —
the console L-shell over a node-proxied Tier-2 stream.
- Transport seam: a per-pane `base` prefix ("" local, "/node/{id}" proxied)
threads through every request; createInteractivePane derives it from nodeId
(the LOCALITY invariant — an interactive session lives on a cluster node).
- Host seam: the couplings only the surrounding shell knows (workstream name,
focus, stream-error recovery, the --skip-permissions banner target, the MCP
consent badge) route through an injected host adapter; the standalone shell
supplies the real one (refetchWorkstreamsAndReassign + STANDALONE_HOST), the
console factory a Tier-1 / no-op one.
- Embedded chrome: the standalone split-pane affordances (focus tracking,
context menu, split/close buttons) are gated behind !embedded; the embedded
path adds the INTERACTIVE persona tag.
- First legacy pane lifted into a real module: it exports the factory for the
console shell's import and bridges window.* for the still-classic standalone
shell (which builds panes only after the workstream fetch, so the deferred
module has run). coordinator.js + the shared substrate stay classic.
The whole tool-output / media / MCP-error / verdict cluster moved with the Pane
(used only by it); the consent-BADGE subsystem stays in the standalone shell,
reached via host.onConsentDetected.
Verified: 104 JS-guard tests + a headless harness running the real module
(standalone + embedded chrome, node-proxy transport, lifecycle, zero errors),
node --check, prettier, ruff.
A dedicated, formatting-only pass over the renovation's frontend so future edits
inherit a consistent style — LLM/contributor edits pattern-match the surrounding
code, so a clean baseline keeps it clean. Covers every non-conformant
.js/.css/.html under shared_static/ + console/static/ + ui/static/ (vendored
katex/hljs + *.min.* excluded; the other ~22 frontend files were already clean).
No rule, value, or markup-semantic changes — whitespace/wrapping only.
Also fixes a real (browser-tolerated) bug the pass surfaced: a `*/` inside a
coord-chrome.css header comment (`#coord-*/coordinator-class`) closed the CSS
comment early; reworded so the comment is valid.
Files: coord-chrome.css, console/static/{index.html,style.css}, coordinator.css,
shared_static/{auth.js,base.css,chat.css,ui-base.css}, ui/static/index.html.
Four findings from the step-4 designer pass on the coordinator pane.
- P1 (bug): the `end` button ran `window.location.href = "/"`, which inside the
L-shell reloaded the WHOLE console — every other pane destroyed, all their
Tier-2 streams dropped. Thread an `onClose` through the factory; the console
pane passes `() => pm.close(pane.id)` so `end` closes that tab (and runs the
controller teardown via onClose→destroy); the standalone page passes none and
keeps the console redirect.
- P2: the pane root carries both `.pane-body` (overflow:auto) and
`.coord-chrome-root` (flex column), so the generic pane scroller redundantly
wrapped the sticky appbar. `.pane-body.coord-chrome-root { overflow: hidden }`
(scoped to this pane type) — the coord chrome owns its own scroll regions.
- P3: the coordinator tab glyph `●` collided with the rail's running state-dot
vocabulary (a static dot reading as "live"); swap to `◆` (a shape marker that
pairs with dashboard's `◇`), pending the real state-glyph in step 7.
- P3: the destructive `end` button had only a title; add aria-label
"End coordinator session".
Verified via the harness (clicking `end` closes the pane without reloading;
glyph `◆`; aria-label present; zero errors); guards pin the pane-aware close.
test_shell_js + test_coordinator_page (97 green), ruff.
The console can now host coordinator sessions as ws_id-keyed panes alongside
dashboard/admin — step 4 complete (the de-globalization landed in 4a).
- coordinator.js: `buildCoordChrome(root, opts)` builds the coordinator chrome
programmatically (createElement, no innerHTML); the factory builds it on
instantiate, so the SAME factory serves the standalone page and a console pane.
`opts.standalone` adds the page-level bits a pane doesn't want (the Console
back-link, the theme toggle, the shared #toast).
- index.html (standalone): goes thin — a bootstrap calling
createCoordinatorPane(document.body, ws_id, {standalone:true}); the ~500-line
inline <style> is migrated to coord-chrome.css (its lone page-level body rule
scoped to .coord-chrome-root) so the console can load the same chrome CSS.
- shell.js: registerType('coordinator') keyed by ws_id — onMount builds the
controller into the pane body, onActivate opens its Tier-2 SSE once, onClose
destroys it. Plus a window.TS_LOGIN fan-out registry so every pane re-arms its
own stream on re-auth (app.js's single onLoginSuccess becomes one subscriber).
- rail.js: coordinator clicks → openPane('coordinator', ws_id) instead of
full-page nav (interactive sessions stay interim full-page until step 5).
- console/index.html: loads the coordinator controller + chrome CSS + the shared
composer/renderer deps it needs.
Child links → openPane('interactive', ws_id) are deferred to step 5 (the
interactive pane doesn't exist yet); coordinator transport stays console-local
inline (parameterized only when the shared ConversationalPane base is lifted).
Verified end-to-end with a headless harness running the real shell.js + rail.js +
coordinator.js: opening a coordinator pane registers the type, the rail row opens
it, buildCoordChrome populates the pane, the Tier-2 SSE connects, destroy() tears
down — zero uncaught errors; renders cleanly (appbar + chat + children/tasks
sidebar + status bar). test_shell_js + test_coordinator_page (97 green), ruff.
coordinator.js was a page-global IIFE keyed off <html data-ws-id>. Make it
multi-instantiable so the console shell can host coordinator sessions as panes
(one per ws_id) alongside dashboard/admin — the first conversational pane-content.
- IIFE -> `createCoordinatorPane(root, wsId)`: the ~40 module-state vars stay
closure-local (now automatically per-instance), every #coord-* lookup is
root-scoped (27 getElementById -> root.querySelector), ws_id is a constructor arg.
- New lifecycle: `connect` (= init), `destroy` (closes the EventSource + clears the
6 timers + the prune interval + the IntersectionObserver — the IIFE had no
teardown, so a backgrounded pane would leak an SSE and fire into detached DOM),
`onLogin` (re-arm after a 401), `closeSession`.
- Drop the page-global collision points: `window.coordSend`/`coordCloseSession`
-> local fns (the close button binds per-instance; its inline onclick is removed);
`window.onLoginSuccess` -> the returned `onLogin` (the console shell will fan
login out to every pane; standalone keeps the single hook).
- Standalone coordinator page = one pane filling the body: a thin bootstrap calls
`createCoordinatorPane(document.body, ws_id).connect()`.
Console-local transport (the coordinator endpoints) stays inline — coordinators
always live in the console; transport is parameterized only when the shared
ConversationalPane base is lifted (after step 5). The chrome builder, CSS
migration, and console pane registration are 4b.
Verified: node --check; a headless smoke (the real factory instantiates against a
provided root, runs connect()'s snapshot/history/children/tasks/SSE on stubs, then
destroy()s — zero uncaught errors); test_coordinator_page.py (13, incl. a new
factory-shape guard); ruff.
Five findings from the step-3 designer pass; the P3 chevron-rotation (taste) was
skipped — the text-swap is already motion-safe.
- P1: the adopted #view-admin had no inset, so the first admin section-header
butted the tab-bar hairline + rail edge. Add `padding:16px 0 0 16px` on
`.pane-body > #view-admin` (.admin-content keeps its right pad).
- P2: the rail Manage active-marker never seeded from getActiveTab(), so a
PaneManager.rehydrate-restored Admin pane showed no active group/row until a
re-click. mountManage now takes the PaneManager, seeds the marker + expands the
owning group when the Admin pane is already open (new PaneManager.hasPane()).
- P2: the active-row band was byte-identical to the amber `.row.open` of live
sessions (distinct only by a 2px stripe). Give it its own neutral idiom —
`--panel-2` fill + a hairline `inset 2px` marker — so "which admin tab" reads
as different in kind from "which session is live".
- P3: `.gcount` pinned right with `margin-left:auto` (was incidental via flex).
- P3: strip the dangling `role="tabpanel"`/`aria-labelledby="tab-*"` from the 18
adopted admin panels (their sidebar buttons were deleted in 3b); the 9 legit
tabpanels elsewhere are untouched.
Verified via the headless harness (rail / admin-open / rehydrate states) +
test_shell_js.py guards (the aria strip is now pinned).
The rail's Manage groups replaced the in-pane admin sidebar in 3a; this removes
the now-dead markup, JS, and CSS that it leaves behind.
- index.html: drop the #admin-sidebar nav (6 groups / 18 buttons) + the mobile
#admin-sidebar-backdrop; #admin-layout now wraps #admin-content alone.
- admin.js: delete the mobile off-canvas drawer (_mobileSidebarOpen,
_injectMobileToggle, _toggleMobileSidebar, the Escape-to-close + arrow-nav +
resize-sync handlers) and switchAdminTab's now-dead .admin-nav active loop +
breadcrumb write.
- style.css: remove the .admin-sidebar* / .admin-nav* / .admin-mobile-toggle*
rules, the mobile off-canvas @media block, and the dead reduced-motion entries.
- shell.css: drop the .pane-body .admin-sidebar hide rule (nothing to hide now).
.admin-layout / .admin-content / #view-admin stay (the Admin pane adopts them).
Verified: no residual sidebar/mobile refs, CSS braces balanced, admin.js parses,
headless render unchanged, and test_shell_js.py pins the removal.
Admin becomes a singleton pane and the rail's Manage section becomes its
navigation; the in-pane sidebar is retired.
- shell.js registers an `admin` pane type that adopts #view-admin (the 18
tabpanels) on first open; the dashboard pane keeps #main.
- admin.js: new ADMIN_IA seam (window.TS_ADMIN) — the group→tab map, a shared
adminTabAllowed() gate (mirrors the legacy showAdmin permission gate, incl.
the ungated node list), an active-tab subscription, and openTab. showAdmin is
now a thin delegator (openPane('admin') + switchAdminTab); the in-#main view
toggle, breadcrumb write, history push, and mobile-hamburger injection go.
- rail.js: mountManage() builds the six collapsible .grp groups from the seam,
permission-filtered, routing a row click through openTab — never touching
admin DOM.
- app.js: home/drill re-focus the Dashboard pane instead of blanking the moved
#view-admin.
- shell.css: the .grp vocabulary + admin-pane layout (in-pane sidebar hidden,
#view-admin fills the pane).
The legacy #admin-sidebar is hidden via CSS pending its deletion in 3b; this is
the additive, independently-runnable half. Verified with a headless-Chrome
harness driving the real shell.js + rail.js over a stubbed seam, plus the
test_shell_js.py guards (19 passing).
From the designer pass on the live rail + persona launcher:
- rail.js: the version-drift amber now marks only nodes whose version differs from the cluster majority (was painting every node when the cluster drifted — the highlight pointed at everything, so at nothing). Adds a per-node title naming the majority.
- app.js + index.html: the persona toggle honours its role=radiogroup contract — arrow keys move the selection, roving tabindex makes the group a single tab stop (seeded statically + in _setLauncherKind), instead of announcing radios but behaving like plain buttons.
- shell.css: an inset (-2px) :focus-visible ring for the rail rows/pills + persona buttons, so the keyboard focus outline doesn't clip against the 266px rail edge (mirrors .dash-row:focus-visible).
The dashboard body becomes a persona-unified launcher (start a coordinator OR an interactive session from one composer) and the saved list spans both kinds; the redundant active-coordinators table is dropped (the rail covers it now).
Backend — the console /v1/api/workstreams/saved now returns both kinds: session_routes.py extracts _collect_saved_rows (shared by the refactored, behaviour-preserving make_saved_handler) + adds make_unified_saved_handler (merges per-kind queries — run concurrently via asyncio.gather — sorted by updated desc). The operator gate (admin.coordinator) is applied once; no new exposure (operators already see every session). console/server.py mounts it with [coordinator, interactive] cfgs.
Frontend — a persona toggle routes submit by kind: coordinator -> console-local POST /v1/api/workstreams/new; interactive -> node-proxy POST /v1/api/cluster/workstreams/new (auto placement). Each option is scope-gated (admin.coordinator / workstreams.create); attachments stay coordinator-only. The saved list gains a KIND tag column + kind-routed activation (coordinator -> /open + /coordinator; interactive -> /node/{id}/?ws_id=) and stays operator-gated. The active-coordinators table + _renderHomeView/_activeCoordsFromClusterState are removed.
Tests: make_unified_saved_handler coverage (tests/test_saved_handler_unified.py, synthetic fixtures, no DB) + a console-launcher static guard (tests/test_shell_js.py). Reviewed via the multi-stage pipeline; findings applied (client/server gate match, concurrent queries, chip-CSS dedup, static guards, stale-comment cleanup).
The rail's Cluster + Workspaces sections (step-1 stub labels) now render live from the Tier-1 clusterState, and the legacy bottom #cluster-status-bar is retired — the rail replaces it (the L-shell has no bottom bar).
New shared_static/rail.js (ESM): renders Cluster (health pills wired to drillDownByState + a node list with version/drift) and Workspaces (the session tree — coordinators with children nested via the shared _bucketByParent, COORD/INT persona tags, state = shape+colour via ui-base .ui-glyph-*).
app.js exposes a minimal Tier-1 seam on window.TS_APP (getClusterState + onRender + the rail's nav actions); renderFromState fires subscribers. No physical clusterState extraction — the seam closures see the live binding. shell.js builds the Cluster/Workspaces render targets and mounts rail.js before boot so it catches the first snapshot.
Retire the bottom bar: delete the #cluster-status-bar markup + renderStatusBar / renderNodePicker / the node-picker helpers + STATE_ORDER (~310 lines), and the .stale toggles in connectSSE (the rail-conn #status-bar carries connection state now). buildNodeInfoFromSnapshot / recomputeOverview / _bucketByParent stay — the rail reuses them. The dashboard body still carries its active-coordinator table transiently; 2b reshapes it into the persona launcher + unified saved list.
Step 1 of the console renovation: a full-height left rail, a top tab bar, and a generic pane host that shows one pane per tab. Existing console content is hosted unchanged inside it as the default Dashboard pane.
New shared_static ES modules (the first ESM citizens; classic scripts keep loading alongside them): pane.js (PaneManager + ShellPane — typed-window host with openPane/activate/close, sessionStorage rehydrate, a WAI-ARIA tablist with roving tabindex + arrow-key nav, reconcile-in-place tabs); shell.js (builds the rail/tab-bar/pane-host, reparents #main + #status-bar with ids preserved so connectSSE needs no rewire, relocates the header controls into the rail footer, drives the app boot); shell.css (chrome ported from the layout mock to base.css tokens).
console index.html loads the shell module + capability flags + stylesheet; app.js's bottom init is wrapped into window.TS_APP.boot, which the deferred shell module drives (it runs after the classic scripts). Cluster health, the Workspaces tree, admin, and conversational pane types arrive in later steps; the rail sections are labelled stubs.
The interactive pane (app.js) got the event-id dedup that skips an
operator-context system turn already painted from /history when an SSE replay
redelivers it; the coordinator pane (coordinator.js) shares the identical
/history + live system_turn + last_event_id replay seam but was left without
the guard. The backend row/event-id alignment already fixes the actual double
for both panes — this restores the defense-in-depth symmetry.
- Module-scoped renderedSystemEventIds (persists across reconnects like
lastEventId), reset in refetchHistory.
- onmessage tags each event with its SSE id; the live system_turn handler skips
an already-rendered id; the history loop records the ids it paints.
- Parallel static-shape regression test in test_coordinator_page.py.
A first-class operator-context system turn (metacognition nudge, output-guard
finding, interjection, watch result) was persisted stamped with the event-id
counter's PRE-emit value, then its live `on_system_turn` SSE event was emitted
with the post-increment id — so the row sat one below its own event. On an
in-flight-orphan `/history` resume, `_resume_cursor_and_trim` derives the SSE
replay cursor from the row's id; being one low, the replay redelivered the
turn's own `system_turn` event and the frontend (no dedup) painted the
operator bubble twice. Reliable for coordinator-spawned children (opened
mid-task) and self-healing on rehydrate — a non-persisted, live-only double.
- `SessionUIBase._enqueue` returns the monotonic `_event_id` it assigns;
`on_system_turn` returns it; `_append_system_turn` emits the hook first and
persists the row with that id (fallback to the current cursor for non-SSE
UIs / a throwing hook). Now row.event_id == its own SSE event id.
- `project_history_messages` surfaces each row's `event_id` so the frontend
can dedup.
- app.js: tag each SSE event with its id, reset a per-pane rendered-id set on
`replayHistory`, and skip a `system_turn` already painted from `/history`
(belt-and-braces against any future cursor skew).
- Regression tests pin the row/event id alignment, the `/history` emit, and
the FE dedup.
- ruff format on two test modules that had drifted (a stray blank line
and multi-line calls that now fit on one line) — restores a clean
`ruff format --check`.
- test_attachment_buffer: pull `buf.discard(...)` out of the `assert`
expressions into locals so the eviction still runs under `python -O`
(CodeQL: assert statement has a side effect).
The Anthropic SDK provider was the lone first-class provider gated behind
an optional extra, while OpenAI ships in core and Google rides the
OpenAI-compatible path. Fold anthropic, psycopg (postgres), croniter
(console), and lacme (tls) into the base dependency set so a default
`pip install turnstone` yields a complete single- or multi-node
deployment; only the Discord/Slack channel gateways stay optional.
- pyproject: four extras → base deps; `all` is now discord+slack; drop the
redundant croniter from the `test` extra; regenerate uv.lock.
- ci: the postgres test job installs `.[test]` (psycopg is base now).
- providers: `_ensure_anthropic` becomes a thin SDK accessor for
`create_client`; drop the now-redundant eager import-guard calls from
the streaming/completion hot path (anthropic is always present).
- bootstrap: import anthropic directly.
- tests/docs: drop the anthropic importorskips and stale extra-install hints.
The operator-context consolidation persists each metacognition nudge's type as
``_source`` (start / resume / correction / denial / completion / repeat), and
both panes rendered it raw as "operator · start". Add a shared
``operatorSourceLabel`` helper in utils.js (loaded by both UIs) that collapses
the metacognition types to one "metacognition" category and humanizes
``tool_error`` / ``skill_hint``; both panes call the single helper so they can't
drift. Carded kinds (watch / guard / idle / interjection) are unaffected.
The canonical-Turn migration left lowering's fold/drop/repair passes Turn-typed even
though they convert to dicts internally and feed dicts to the translators, so
_prepare_wire_messages round-tripped the whole history Turn->dict->Turn ~7-8x per send
(even on the no-op early-return paths). Make fold_system_turns / drop_empty_user_turns
/ repair_wire_messages dict-native (list[dict]->list[dict]); _prepare_wire_messages now
threads the dict projection _full_messages already produced straight through, with no
Turn round-trip. self.messages stays the canonical Turn trajectory. export.py is
simplified (it converted to dicts immediately after repair anyway). Equivalence-
preserving — test_wire_payload_golden stays byte-identical.
- Buffer (attachment_buffer.py): content-address staged bytes once and track the
per-(ws_id,user_id) references to them, so identical bytes staged from two tabs
dedupe to one copy yet neither scope's send can drop the other's pending upload
(the prior hash-only key let one overwrite the other). Single lock; add a public
clear() that replaces test reaches into the private store.
- GC: lift the byte-identical _release_attachment_refs out of both backends into one
dialect-agnostic storage/_utils.release_attachment_refs with a portable searched-
CASE single-query decrement (was one UPDATE per id in a Python loop).
- _format_messages_for_summary: mark by-reference vision results
({type:image, attachment_id}) as [image], not just inline image_url.
- Security: escape_like() the attachment_referenced_in_ws LIKE needle on both
backends; secrets.compare_digest for the output-guard operator-fence leak check.
- B1: page _backfill_content_addressed_attachments via a composite keyset cursor
(message_id, created, attachment_id) instead of one un-paged fetchall() of every
blob's bytes — bounds peak migration memory regardless of stored blob volume.
Validated on the dev-DB snapshot: upgrade + downgrade clean, 5431 conversations
preserved, blobs deduped + content-addressed.
- B2: _native_from_provider_data strips orphan client tool-call blocks on load when
the row's tool_calls column is empty (the truncated-mid-tool_use legacy hole), so
a same-provider resume can't replay an unanswered tool_use — closing the Anthropic
400 and the Google tool-call resurrection path. Healthy (mirror-holds) rows decode
byte-identically.
- A1: add a shared `operator-context` marker to every operator-context row in
both UIs; the retry-skip walk keys on it, so a trailing watch-result /
guard-finding / idle-children card no longer makes retry regenerate the
wrong turn. Pinned by source-grep tests + a headless-DOM self-test.
- A2: wrap get_content's two sync DB gates (get_attachment + the unbounded
ws-scoped attachment_referenced_in_ws LIKE scan) in asyncio.to_thread so a
long scan can't stall the event loop — matching the module's convention.
- A3: add the HistoryEvent attachments docstring bullet; drop the dead
`interjection` class; document the interactive-only system-context label;
correct the SDK attachments-meta docs to {kind, filename, mime_type}
(size_bytes is not carried through the history projection).
Operator-context system turns (watch results, output-guard findings, idle
children, user interjections) carried their kind (_source) and a flattened
text content, but the structured per-kind fields were dropped at every
persist/deliver boundary — so the UI rendered every kind as one generic
operator bubble and the structured watch-result card was lost.
Wire the structured meta through as the single source of truth:
- Storage: new conversations.meta JSON column (migration 060); threaded
through save_message/save_messages_bulk (facade + protocol + both backends)
and rehydrated in reconstruct_turns onto Turn.meta.extra["source_meta"].
- Canonical: make_system_turn carries meta as one _source_meta dict;
turn_from_dict/turn_to_dict bridge it to/from Turn.meta.extra.
- Live + history: widen on_system_turn(content, source, meta) across all
impls + the SSE payload; surface _source_meta -> meta in the /history
projection. SDK HistoryEvent docs note the field.
- Producers derive both the model-facing content text AND the card from one
meta dict, so they cannot drift: render_output_guard_text, build_watch_
reminder carrying output, idle_children and user_interjection metadata.
- Frontend: addSystemContext / renderSystemTurn dispatch by source to the
watch-result, guard-finding, idle-children, and queued-message cards in
both the interactive and coordinator panes; every untrusted field renders
via textContent.
The meta is a leading-underscore key, stripped before the wire (sanitize_
messages and the native mid-conversation path copy only role+content), so the
per-provider wire payloads stay byte-identical. Additive column, no backfill:
operator turns predating it reload as plain text bubbles.
A deep-dive review of the branch surfaced a budget regression, SDK doc
drift, dead code, and stale docstrings. Each was boundary-spiked before
fixing.
- R1 (regression): by-reference document attachments were invisible to the
token budget — _msg_text_chars returned 0 doc_chars for a
{type:document,attachment_id} placeholder, and the comment's claim that
the budget "lands at calibration" was false (calibration discards
doc_chars). Thread the doc size through _attachments_meta (size_bytes, at
both the live-append and reconstruct build sites) and count it in
_msg_text_chars, guarded against double-counting the inline form.
Regression test added.
- F1: the history-DTO schema description and the TS HistoryEvent docstring
still advertised the removed reminders/advisories keys and omitted the
system role; corrected server_schemas.py + sdk/typescript/src/events.ts to
match the shipped shape. The committed OpenAPI JSON snapshots were already
~679 lines stale on main; their regen is left to its own chore branch.
- D1: removed AttachmentBuffer.take() — dead (no production caller; the
commit path uses discard()) and scope-weak (ws_id only, unlike its
siblings) — with its test and the now-orphaned Iterable import.
- O1: 4 docstrings referenced the moved ChatSession._fold_system_turns →
lowering.fold_system_turns.
The blob store is global content-addressed — identical bytes dedupe across
workstreams and users, so the per-tenant ws_id/user_id scope columns are dead:
nothing reads them, and a committed blob is authorised via the
conversations.attachments ref-list (attachment_referenced_in_ws), not a row
scope. Drop both columns and idx_ws_attachments_ws_id from the schema and from
the save_attachment signature (protocol + both backends + memory wrapper + the
caller); fold the column/index drops and their downgrade into the unshipped
migration 060.
tool_name stays: it is a live denormalised search label (search_history →
recall + /history), not trajectory data — "never rehydrated" held only for the
wire path, which already ignores it.
The by-reference content lane now materializes at the provider translator (the
C layer), not in the session. Each create_streaming / create_completion takes
a resolve_attachments callback and runs materialize_attachments() up front,
expanding {type:kind, attachment_id} placeholders to inline data-URI / document
parts by a content-addressed point-lookup the session hands down
(_resolve_attachments). _full_messages emits placeholders; the dict bridge
carries only placeholders.
RawContentBlock is removed — ContentBlock = TextBlock | AttachmentRef. A
resolved inline part is terminal (the wire payload / display output) and never
re-enters the canonical path, so turn_from_dict drops a stray inline image_url
rather than carrying bytes. resolve_attachment_parts / materialize_attachments
operate on the dict projection. Tool vision output rides by reference too
(_tool_content_by_reference): the turn carries placeholders, the bytes persist
content-addressed. The per-turn token estimate counts a by-ref image as one
fixed image budget; the document char budget lands at send (on resolution).
Wire harness byte-identical (the multipart fixture is a placeholder + a matching
resolver); full non-live suite green (7136).
Non-text content (user uploads, reloaded tool images) rides as AttachmentRef(id,kind)
in the canonical Turn — session.messages carries ids, never bytes. Each output
materializes it to inline data-URI/document parts by point-lookup on the content-
addressed store: the wire (ChatSession._lower_messages_to_wire, in _full_messages),
/history + export (reconstruct_messages resolves), and the per-turn token estimate
(a by-ref image costs one fixed image budget; the doc char budget lands at send).
reconstruct splits: reconstruct_turns = unresolved row→Turn (load_message_turns, the
resume path); reconstruct_messages = resolved dict facade. RawContentBlock is demoted
to the transient carrier for a resolved inline part on the dict↔Turn bridge.
repair_wire_messages / fold_system_turns / drop_empty_user_turns take and
return list[Turn] — the neutral lowering layer (A representation + B validity)
now speaks the canonical type. Their intricate content-merge / orphan-detect
internals run over the dict projection (reading Turn content blocks would only
duplicate turn_to_dict's content logic), so each bridges
dicts_from_turns ↔ turns_from_dicts at its boundary; byte-identical.
ChatSession._prepare_wire_messages lifts the wire dicts into Turns, runs the
lowering passes, and lowers the result back to the dict projection the provider
translators (the C layer) consume — the dict bridge now lives in the wire layer,
not in _full_messages. Export runs the same repair, reordered before the
non-canonical reasoning-content attach (a key the Turn model does not carry).
The provider translators keep their dict input by design: they are the format
layer that emits provider bytes, the vLLM reasoning-attach is a non-canonical
wire concern that sits between lowering and the provider on dicts, and feeding
the converters the lowered projection is equivalent to — and simpler than —
threading Turn content through them. Wire harness byte-identical; full
non-live suite green (7130).
ChatSession.messages flips from list[dict] to list[Turn] — the in-memory
canonical trajectory. Reads migrate to typed fields (turn.role, turn.text,
turn.tool_calls); appends and assignments go through turn_from_dict /
turns_from_dicts; the fork bulk-save and retry's multipart check read via
turn_to_dict. _full_messages lowers Turns→dicts at the wire boundary — the
fold/repair and provider translators still consume dicts until the next slice.
The token-accounting helpers accept a dict or a Turn.
Non-session consumers migrate too: coordinator_idle_observer and eval to typed
fields (mypy-enumerated), and server's last-assistant extractor via turn_to_dict
(an Any-typed call site mypy could not flag). An all-text multipart content
list (the unreadable-attachment placeholder path) now round-trips faithfully
through the adapter (single text block → str, multiple → list).
Tests that inspected session.messages as dicts read it through the
dicts_from_turns / turn_to_dict bridge; those that built it pass dicts through
turns_from_dicts / turn_from_dict. Byte-identical wire harness; full non-live
suite green (7130).
reconstruct_turns is the pure row→Turn deserialize: one positional unpack of
the row tuple, one Turn per row, no wire-validity correction. The scattered
per-role dict-building and the side-channel keys collapse into typed Turn
fields (native ← {producer,blocks}, source ← _source, …); the dead tool_name
column is unpacked but unused. recover_trajectory(turns) is the load-time
trailing-strip policy, lifted out as its own function (one of lowering's three
orphan policies).
reconstruct_messages stays the dict-returning facade for now —
dicts_from_turns(recover_trajectory? · reconstruct_turns) — so every consumer
is unchanged and byte-identical (verified across the storage + reconstruct +
export + wire-payload suites, 7129 green). developer collapses into
Role.SYSTEM (zero writers, wire-identical); a bare-dict provider_data (never a
real native shape — the lane is a block list) no longer round-trips, which the
storage test now reflects.
turn_from_dict / turn_to_dict losslessly bridge the OpenAI-like message dict
(plus its _-prefixed side channels) and the typed Turn, so the migration to
Turn can proceed one boundary at a time: a dict-producing layer can be read as
Turns, and a Turn-holding layer can hand dicts to a not-yet-migrated consumer.
The _ side channels become typed fields: _source→source, _provider_content
(+_producer)→native, _event_id→meta.event_id, _attachments_meta→meta.extra.
A transitional RawContentBlock carries image/document parts verbatim until the
by-reference AttachmentRef wiring (§2/§6) relocates byte-resolution to the
translator; text parts become TextBlock so .text/FTS stays faithful.
turn_to_dict(turn_from_dict(d)) == d for every shape reconstruct and the wire
path emit (test_trajectory). No consumers yet — wiring is the next slices.
reconstruct_messages(repair=True) did two things: strip a trailing incomplete
tool-call turn AND synthesize cancellation results for mid-conversation
orphans. The mid-orphan synth was a near-duplicate of
lowering.repair_wire_messages — same detector, same contiguous insert past
interspersed system turns, same cancellation string — running at the wrong
layer (storage, on every load).
Drop it: load is now trailing-strip only (boot-crash recovery), and the
mid-orphan synth happens once, at send, in lowering.repair_wire_messages — the
single place the wire path fills orphans. The session send path gets it via
_prepare_wire_messages; export, which bypasses that path, now runs
repair_wire_messages itself (otherwise a mid-conversation orphan would
serialize as an unanswered tool_call). The duplicated cancellation string
goes with the synth — CANCELLED_TOOL_RESULT lives only in lowering now.
Safe: a bare mid-orphan is harmless between load and send (token count is
additive, /history reads repair=False, compaction summarizes to text), and
every wire path repairs it. Reconstruct tests updated to the new load
contract; an export mid-orphan test added.
The fold (representation) joins repair (validity) in the shared lowering
sibling module: fold_system_turns / _neutralize_host / _append_text_block /
drop_empty_user_turns move out of ChatSession as free functions.
_prepare_wire_messages now composes the two neutral passes plus repair, so
session.py owns zero wire-shape mutation.
The nonce stays session-minted and session-owned (_envelope_nonce binds three
consumers: the fold, the cached-prefix trust declaration, and the output-guard
forgery check) — lowering borrows it as a parameter and never mints its own.
The capability gate (supports_mid_conversation_system) is a parameter too, so
native-passthrough is unit-testable without monkeypatching a session; the
provider-None case is handled by the caller.
Pure relocation: the fold algorithm, the once-per-host neutralize ordering,
the read-only contract, and drop-after-fold are unchanged. Wire harness
byte-identical; fold unit + _prepare_wire_messages integration tests green.
Synthesizing a cancellation result for an assistant tool_call with no
matching tool result was triplicated across the translators: Anthropic's
verbatim-replay (pc_tool_ids) and rebuild branches, and sanitize_messages
for the OpenAI-compatible lanes (Chat, Responses, Google). The Anthropic
pc_tool_ids branch was also the sole repairer of a native tool_use orphan.
Lift it to one neutral policy — lowering.repair_wire_messages — run once in
ChatSession._prepare_wire_messages before the translator. It reads tool_calls
only, which is sound because the native/tool_calls mirror is enforced at save
(normalize_native_for_save): a verbatim-replay orphan is caught via its
mirrored top-level call. The translators become pure format translation and
carry no orphan synthesis.
The neutral cancellation turn carries is_error=True; Anthropic renders it on
the tool_result block, the OpenAI-compatible tool message has no such field
so sanitize_messages drops it (the C-layer translation of the flag).
sanitize_messages keeps one orphan synth of its own: a back-filled empty-id
tool_call (local servers that omit ids) is id-less when the upstream repair
runs and so invisible to it, so that lane owns its cancellation — preserving
the pre-refactor behavior for local servers.
reconstruct's load-time strip and the runtime-cancel persist-synth are
unchanged. Proven byte-identical against the per-provider wire-payload golden
harness (including a new native_orphan fixture); the harness applies the same
send-side repair the session does.
Freeze the wire payload for an unanswered native tool_use whose id is
mirrored top-level in tool_calls (the P1 invariant). This pins the
verbatim-replay orphan path each provider repairs today — Anthropic via
the provider_content tool_use synthesis, the OpenAI-compatible lane via
sanitize_messages — as the baseline the repair-unification change is
proven byte-identical against.
Replace the persisted pending/reserved/consumed upload lifecycle (and its orphan-sweep
and per-user cap) with a content-addressed, refcounted blob store fronted by the per-node
in-memory pending buffer:
- Upload stages bytes in the buffer (keyed by sha256); send-commit drains the referenced
handles, writes each blob content-addressed (INSERT-OR-IGNORE then refcount += 1, so a
stored blob is born referenced and dedupes across messages/workstreams), and records the
ordered conversations.attachments ref-list — the sole message->blob link.
- reconstruct rebuilds inline image_url/document multipart content from the ref-list,
role-agnostically (so tool-produced images via _exec_read_image now persist + rehydrate
instead of being flattened to text and lost). Output shape unchanged.
- GC is reference counting: delete_messages_after / delete_workstream decrement once per
reference and prune a blob at 0; a deduped blob shared with a kept turn (or another ws)
survives.
- get_content for a committed blob is gated by reference-ownership (the requester owns a
turn in the ws whose ref-list names the id), replacing the dropped ws_id/user_id scope.
- Migration 060 re-keys legacy consumed attachments to their content hash, dedups, sets
refcounts, writes the ref-lists, and drops message_id/reserved_*; pending legacy rows are
dropped (pending now lives only in the buffer).
Both backends symmetric; the reservation methods, cap, and orphan-sweep are removed across
storage/facade/protocol/endpoints/coordinator. Wire harness byte-identical; full suite green.
The content-addressed model writes blob bytes to workstream_attachments only at
send-commit (so every stored blob is born referenced). Pending uploads — between the
upload request and the send that references them — live in this per-node, content-hash-
keyed buffer, scoped to (ws_id, user_id) and bounded by a TTL + total-size ceiling
(OOM-safety, not the removed per-user cap). ws->node affinity (HRW routing) keeps it
process-local. Losing an unsent upload on crash/re-route is acceptable transient state.
This replaces the persisted pending/reserved/consumed lifecycle + orphan-sweep. No
consumers yet — the upload-endpoint rework and the send-commit drain wire onto it as the
attachment cutover lands.
Additive schema for the attachment cutover: workstream_attachments gains refcount +
origin, conversations gains the attachments ref-list column (migration 060 + _schema in
lockstep). Columns sit unused until the cutover, which fills them and retires the
message_id/reserved_* upload-lifecycle in favour of a content-addressed, refcounted blob
store keyed by the conversations ref-list.
Also registers the coordinator test's backend via init_storage: the attachment handlers
resolve storage through the global registry, so a bare SQLiteBackend left get_attachment
hitting a stale default db — latent until the new column made the schema drift bite.
060 now tags legacy bare-list provider_data rows with the {producer, blocks} envelope,
inferring the producer from block types — and the inference yields the exact provider_name
strings the live save writes (anthropic / google / openai / openai-compatible) so a
backfilled row compares equal to a freshly-saved one under the lowering layer's
producer==active rule. Google is keyed on a 'function' block carrying thought_signature;
xAI is byte-identical to OpenAI-Responses in the stored blocks so legacy xAI rows tag as
'openai' (bounded, self-healing). Un-inferable rows are left bare (reconstruct dual-reads
them). Paged like the envelope rewrite. Completes the producer story: 2a tags new rows,
this tags legacy. Sub-commit 2b of the canonical-trajectory storage cut.
Persist provider_data as a {producer, blocks} envelope (producer = the generating
provider's name) so the lowering layer can later replay the native lane verbatim only
to its producer and rebuild from neutral fields for any other.
The envelope is storage-only: prepare_provider_data_for_save runs the P1 mirror on the
bare block list and then wraps; reconstruct_messages dual-reads (new envelope OR legacy
bare list), unwraps to a bare _provider_content list (every consumer's contract), and
surfaces the producer on the stripped-before-wire _producer side channel. The producer
threads the same four save layers as is_error (facade -> protocol -> SQLite + Postgres);
the live assistant save tags it from self._provider.provider_name and the fork carries
_producer. Legacy rows need no migration to keep working (dual-read); the one-shot
backfill that tags them is a follow-up.
Sub-commit 2a of the canonical-trajectory storage cut.
Tool-result error state was an in-memory-only message key, lost on reload. Add an
is_error column to conversations (migration 060, backfilled False) and thread it through
the four save layers (memory facade → StorageBackend protocol → SQLite + PostgreSQL):
save_message/save_messages_bulk persist it, reconstruct_messages emits it on tool rows,
and the session tool-result + synthetic-cancel saves + the fork bulk-copy pass it. It
rides as the last conversations column so reconstruct's row-tuple positions stay stable
(legacy fixtures default False). history_decoration already prefers the persisted flag
over its text heuristic, so reload fidelity improves immediately.
First sub-commit of the canonical-trajectory storage cut (folds into rev 060).
The provider-neutral typed Turn (flat, role-discriminated; uniform tuple[ContentBlock]
content + .text; AttachmentRef by-reference content; ToolCall raw-arg str;
ProviderNative producer-tagged opaque lane; TurnMeta sidecar). In-memory foundation
for the wire-shape narrow waist — no consumers yet; storage deserialization, the
lowering layer, and the provider translators wire onto it in subsequent steps.
A max-tokens truncation mid-tool_use can leave an orphan tool_use in the native lane
(provider_data / _provider_content) with no matching tool_calls; on a same-provider
resume that replays as a tool call with no result and the API rejects it.
normalize_native_for_save strips orphan client tool-call blocks (tool_use /
function_call / function) when tool_calls is empty, applied by save_message and
save_messages_bulk in both backends; strip_orphan_client_tool_blocks enforces the
same mirror in memory at message assembly (the truncation path). The mirror now holds
by construction, so the orphan-repair pass can read tool_calls alone and the Anthropic
pc_tool_ids fallback can be retired.
Captures the exact request kwargs each provider hands to its SDK seam (Anthropic
messages.stream, OpenAI chat/responses create, Google OpenAI-compat) for a
representative set of trajectories, asserted against committed goldens. This is the
behavior-equivalence net the canonical-trajectory wire-shape refactor is proven
against. Regenerate the baseline with UPDATE_WIRE_GOLDENS=1.
Phase-2 review follow-ups:
- sec-1 (major): the skills find-zero hint interpolated the model-supplied
filter values (query/category/tag/…) into the system_reminder, which now
rides a TRUSTED operator system turn (fold fence / native system role). Under
an indirect prompt injection the model could be steered to call
skills(find, query='<directive>', category='nonexistent') so 0 rows match,
laundering the attacker text into operator authority. Drop the filter echo —
the count is harness-derived and the model already knows its own filters.
- q-1 (minor): refresh the stale :func:`escape_wrapper_tags` cross-reference in
metacognition.sanitize_payload's docstring (the function was removed; fold-time
fence.neutralize is the current marker defense).
_skill_hint spliced its guidance into the tool result as a bare <system-reminder>
block — but the operator declaration now tells the model to treat bare markers
as untrusted, silently demoting the hint. Make the hint first-class instead:
- _skill_hint returns the tool result verbatim and queues the guidance via
_queue_tool_advisory("skill_hint", ...); _collect_advisories drains it into a
{role:system, _source:"skill_hint"} turn after the clean result — folded in
the trusted nonce fence for non-native models, inline for native. (Queuing
no-ops mid-wake, like the other tool-channel advisories.)
- skill_hint added to SYSTEM_TURN_SOURCES (an advisory-producer source).
- escape_wrapper_tags removed outright: it was the last consumer, and its job
(defang a marker next to the bare block) is now covered at fold time by
_neutralize_host. The result message rides through verbatim. This also
collapses the two-escaping-mechanism confusion the review flagged.
Tests assert the clean result + the queued/drained hint, plus wake suppression.
Operator context moved to first-class system turns, leaving _reminders written
by nothing and read by nothing. Nulling it (the prior 060 step) left a writable
dead column — a foot-gun inviting accidental reuse. Drop it outright and remove
every reference in one shot so there is no half-alive state:
- migration 060: replace the wholesale null with batch_alter_table drop_column
(per migration 027); downgrade re-adds the empty column to match the 059
schema (the envelope un-wrap stays irreversible).
- _schema.py: remove the column.
- _sqlite / _postgresql: drop the reminders save param, the INSERT/bulk values,
and both SELECT columns.
- reconstruct_messages: the row tuple is now 8/9-tuple (event_id shifts from
index 9 to 8); _utils + the _row test helper updated.
- _protocol / memory save_message: drop the reminders param + docstrings.
- tests: replace the reminders-roundtrip tests with a _source-only file and a
060 drop-column assertion; remove the obsolete legacy-reminders wire test.
No production caller passed reminders=, and the SELECT no longer reads the
column, so an un-migrated DB simply ignores any residual values.
Phase-2 follow-ups to the mid-conversation-system consolidation:
- user_interjection framing (known #2): a queued message that drains mid-turn is
re-framed via render_user_interjection ("The user sent … User message: …") so
the user's words keep USER authority, not operator authority — the regression
mattered most on the native path, where the turn enters as a real role=system
message. Empty/whitespace interjections (e.g. a bare "!!!") are dropped (bug-2).
- empty-content user turns dropped at the wire boundary after the fold
(known #3): the wake pipeline's synthetic empty send("") leaves an empty user
turn on the native path (the nudge stays inline); an empty user message is
invalid on every provider. The drop runs after the fold so the fold-path wake
turn, which the nudge fills, survives.
- leading-system guard (_anthropic): a turn that converts to nothing no longer
lets a system message become messages[0] (the API requires messages[0]=user).
Newly reachable now that the empty-turn drop can expose it on a fresh-session
native wake.
- refresh stale .msg.watch-result comments (the card was removed) to describe
the current operator-bubble rendering.
Phase-1 review follow-ups:
- neutralize() now tolerates whitespace between '<' and the slash ('< /tag',
'< /tag'), matching output_guard's detection regex so a marker can no longer
be detected-but-not-defanged (a leaked-nonce break-out gap).
- Add direct tests for the sec-1 forge-in defence: _neutralize_host defangs a
forged <system-reminder_{nonce}> in both string- and list-content untrusted
hosts before the real fence is appended, and the host is defanged exactly once
so consecutive folds don't corrupt the first appended fence.
PlanReviewView was removed alongside the plan_agent built-in tool (110d44b0),
but its Discord owner-check tests were left behind importing a class that no
longer exists. They raise ImportError wherever discord.py is installed (green in
CI only because discord.py is absent there). Remove the dead test class and the
now-unused send_plan_feedback router mock from the shared bot double.
060 un-wrapped legacy <tool_output> envelopes with a loose guard (open + close)
that could irreversibly mis-rewrite a bare tool row resembling the open, and
entity-decoded the wrapper tags back to live form — re-activating injection the
old escape had neutralised (and downgrade cannot undo it).
- Require the full legacy signature (the exact </tool_output>\n\n<system-
reminder>\n join plus a trailing </system-reminder>), which wrap_tool_result
only ever emitted with advisories. A bare row with a matching close but no
advisory is left byte-for-byte untouched.
- Reverse only & -> & ; leave the wrapper-tag entities escaped so a
previously-defanged injection stays defanged.
Adds false-positive guard tests (open+close without advisory; missing tail).
Both the operator fold and the output-guard judge wrap spans in nonce-delimited
fences, but the two had drifted: the operator path minted a 32-bit nonce reused
per session with no body escaping, while the judge used a 64-bit per-call nonce
plus closing-tag escaping. Extract the shared mechanism (mint/neutralise/wrap)
into turnstone/core/fence.py and put both callers on it so they cannot diverge
again.
- Operator fold (sec-1): 64-bit nonce; fence.wrap neutralises the operator
body's close marker, and _fold_system_turns neutralises the untrusted host
turn's <system-reminder> markers once before the first fold, so a leaked or
guessed per-session nonce still cannot forge a trusted block. Per-session +
cached declaration kept (the declaration pins the exact value, so per-turn
rotation would bust the prompt cache). Marker is now <system-reminder_{nonce}>.
- Judge: refactored onto fence (behaviour-preserving; still per-call).
- Forgery detection: output_guard scans tool output for trust-fence markers —
an exact session-nonce match is HIGH (operator_marker_leak: the token has
leaked and is being replayed), any other marker LOW (operator_marker_forgery).
Removes mint_envelope_nonce / wrap_system_context (folded into fence.wrap).
Replace the two operator-context hacks (the <tool_output>/<system-reminder> content envelope and the transient _reminders side-channel) with one persistent {role: system, _source} trajectory turn. Adds supports_mid_conversation_system (claude-opus-4-8): native models take the turn inline; all others fold it into the preceding turn as a nonce-delimited <system-reminder> block declared in the system prompt as the sole trusted marker. Producers (advisories, metacog nudges, user interjections, idle/watch) emit system turns; the envelope/_reminders machinery, escaping round-trip, replay parser, and reminder SSE events are removed. Eager 060 migration un-wraps legacy envelopes. Net -1662 lines.
Known follow-ups from review (unfixed here): (1) the 060 un-wrap heuristic can irreversibly mis-rewrite bare tool rows that resemble the envelope, so do not run the migration until it is tightened; (2) user_interjection turns lost the user-framing/priority preamble (a regression, and a native-path authority-framing concern); (3) native-path wake nudge can emit empty user content.
- rerank_config.py: the runtime instruction fallback had a dead tail
(`get_rerank_instruction() or str(cs.get(...))` -- the cs.get term can only
return the registry default ""), via a stored_keys() branch that also diverged
from the calibrate CLI / endpoint. Collapse to the sibling idiom
(`cs.get(...) or get_rerank_instruction()`) so the instruction used at
calibration time matches the one used at runtime. Correct the module docstring:
ChatSession is the sole caller (the CLI/endpoint share only the instruction
precedence, not this function).
- session.py: the deferred first-turn memory recompose was gated on the flag
alone, so a synthetic wake send (empty user content -> flag stays False) re-ran
the full compose on every wake before the first real turn. Gate on a non-empty
query too, so wakes don't re-pay it and the real turn still fires exactly once
(+ test).
Accepted as-is: the __init__ compose (kept so system_messages/_agent_system_messages
are valid for early readers; one cheap extra compose per fresh session) and the
orphaned tools.rerank_* config rows (inert -- no read path, never listed or
redacted; a purge migration would collide with the 060 in flight on another branch).
Proactive memory selection scores candidates against the recent-user-message
query (extract_recent_context), but a fresh session composes the system prefix
once in __init__ while self.messages is still empty. That empty query takes the
no-context path: _select_memory_candidates returns recency order and
score_memories returns memories[:k] verbatim -- the 5 most recently UPDATED
memories, with BM25 and the reranker never invoked. send() never recomposes, so
those recency-only memories are what the model sees for the whole session (until
an unrelated event -- skill / MCP / model refresh / resume / memory write /
command -- happens to rebuild the prefix). Net effect: the injected memories are
unrelated to the actual question.
Fix: defer the memory-bearing compose to the first real user turn.
- Track _system_composed_with_context, set once extract_recent_context is
non-empty in _init_system_messages.
- send() recomposes once, right after _append_user_turn, while the flag is still
False -- so the opening turn's memory block is selected (and reranked) against
the real message. The flag then stays True, so the prefix is composed once and
stays cache-stable exactly as before (no per-turn prompt-cache churn).
This is the targeted fix; per-turn memory refresh (so later topic shifts also
re-rank) is the larger tail-injection redesign tracked on another branch. Two
adjacent gaps are left as-is for now: the reranker/BM25 only see content[:200],
and build_memory_context flat-truncates each memory to 500 chars (max_content is
the save cap, not an injection budget).
Tests: flag is False on a fresh session and after a whitespace-only wake turn,
flips True on a real query; send() runs the deferred recompose with the user
message in the query.
The reranker_alias -> model-definition path (added when reranking became a model
role) made the older global endpoint settings redundant. Resolve reranking
solely through the Reranker role and remove the parallel global config.
- Removed settings tools.rerank_url / rerank_model / rerank_api_key, their
config.py getters (+ $TURNSTONE_RERANK_URL / $TURNSTONE_RERANK_MODEL and the
module caches), and the fallback branch in resolve_rerank_client_from. The
resolver now returns a client only when a Reranker model (capability
supports_rerank, base_url = its /rerank endpoint) is selected, else None.
- Kept as global knobs: reranker_alias (the selector), rerank_web_search,
rerank_bm25, rerank_bm25_threshold, and rerank_instruction -- a task-level
query knob (Qwen3-style), not endpoint identity.
- The Settings tab is registry-driven, so the three fields disappear with their
SettingDefs. Updated the Reranker role help, example config, and docs/tools.md.
BREAKING: a reranker configured via [tools] rerank_url (config.toml / env /
Settings tab) no longer works -- add the reranker in the admin Models tab and
pick it under Models -> Roles -> Reranker. No migration: reranking is days old
and disabled by default, so any orphaned tools.rerank_* config rows are inert.
Tests: the resolver covers no-store / no-alias / non-rerank-alias -> None and the
model-definition happy path; the obsolete global-fallback tests are removed.
Adding a reranker model through the admin modal was impossible: Detect
(admin_detect_model) always ran the OpenAI /v1/models probe first and gated
calibration on its `reachable` result. A Cohere/Jina /rerank endpoint can't
answer /v1/models, so it failed both ways -- `$host/v1` passed the probe but
calibration POSTed to the wrong path, `$host/rerank` 404'd the probe outright.
- console/server.py: branch on `supports_rerank` BEFORE the probe and calibrate
the endpoint directly; that round-trip IS the reachability check (there is no
independent /v1/models signal for a rerank-only endpoint). Success
autopopulates the three calibration fields the way context_window does;
calibration failure -> reachable:False + error; empty base_url -> 400 with the
/rerank hint; reachable-but-no-clean-separation -> a note. Drops the now-dead
post-probe calibrate-on-detect block.
Reranker selection stays per-model (reranker_alias -> registry); recalibrate on
a saved reranker was already correct (it calibrates directly). Flagging a model
as a reranker still rides the capabilities JSON (supports_rerank).
Negative-tested: rerank detect skips the probe, autopopulates on success, notes
no-separation, reports unreachable on calibration failure, and 400s on an empty
base_url; the non-rerank detect path is unchanged.
Phase 3. Stores reranker calibration per-model on the model definition's
capabilities (rerank_threshold/rerank_scale/rerank_separated; a non-empty
rerank_scale is the "has been calibrated" marker) instead of a single global
threshold, populated automatically when a reranker endpoint is detected.
- ChatSession._bm25_rerank_threshold precedence: the active reranker model's
calibrated rerank_threshold (when separated) wins; calibrated-but-not-
separated -> 0 (no floor); else the global tools.rerank_bm25_threshold
fallback. Reads the raw caps dict, in-memory per turn.
- Detect (admin_detect_model) calibrates a supports_rerank endpoint and
autopopulates the three fields like context window; the create-model UI shows
a verdict chip (calibrated / no-clean-separation / not-calibrated).
- POST /api/admin/model-definitions/{id}/calibrate backs the Re-calibrate
button; calibration runs off the event loop (run_in_executor, bounded by
asyncio.timeout(90)), persists the fields + refreshes the registry, and is
graceful on a down/slow endpoint (never 500). Shared
merge_calibration_into_caps helper used by the endpoint and the CLI.
- turnstone-admin rerank-calibrate is now per-model: --model <alias> required;
--apply writes that model's caps (not the global setting). A no-separation
result records the marker (calibrated, no floor) consistently across CLI and
endpoint.
The serving lesson stays documented: Qwen3-Reranker needs vLLM --chat-template
or its scores are near-random (live-validated 0.6B + 4B: the calibrated floor
came out 0.95 vs 0.33 for the same task -- why per-model calibration exists).
Negative-tested: the floor-precedence branches (calibrated+separated -> per-
model, calibrated+!separated -> 0, uncalibrated -> global, no-alias -> global,
registry-must-not-be-consulted), endpoint persist/refresh + graceful failure,
detect-skips-non-rerankers, the CLI per-model write, and the caps-merge
preserving supports_rerank. Chip states verified via headless Chrome.
Phase 2 of BM25 reranking (follows #627). Makes the rerank_bm25_threshold floor
usable across reranker models and adds tooling to pick it.
- normalize_scores (rerank.py): map a rerank batch into a 0-1 relevance
probability -- sigmoid when any score falls outside [0,1] (logit endpoints
like bge/TEI), identity otherwise (Cohere/Jina/Qwen already 0-1). Applied in
the _bm25_reranker closure AND calibration so the threshold means the same on
every endpoint. Monotonic, so ranking order is unchanged.
- rerank_calibrate.py + `turnstone-admin rerank-calibrate [--apply]`: probe the
endpoint with labelled relevant/irrelevant groups, normalise, and recommend a
recall-biased floor -- or report "no clean separation" (a mis-served/weak
reranker). A warmup loop absorbs a cold endpoint's first-request compile so
calibration doesn't time out. Validated live against Qwen3-Reranker 0.6B and
4B: the calibrated floor differs sharply per model (~0.95 vs ~0.33 for the
same task) -- exactly why per-endpoint calibration exists.
- rerank_config.py: extract resolve_rerank_client_from(config_store, registry);
the alias/url precedence now lives in one place, shared by ChatSession (which
delegates) and the CLI.
- tools.rerank_instruction (config + setting + client): wrap the query as
<Instruct>:/<Query>: for instruction-aware rerankers (Qwen3) on endpoints that
don't apply the model's own chat template. Docs note the critical vLLM serving
detail: Qwen3-Reranker needs --chat-template or its scores are near-random and
reranking hurts retrieval.
Negative-tested: normalize sigmoid/identity branches, closure-normalises-before-
floor, calibration separation/recall-bias/warmup-absorbs-cold-start, the CLI
apply/no-apply/no-separation paths, and instruction query-wrapping through the
real httpx boundary.
Reuse the shipped Cohere/Jina rerank client as an optional post-process on
the BM25 surfaces (tool search, skill search, memory composition) via one
seam: BM25Index gains an injected reranker + a two-stage search (BM25 recall
top-50 -> rerank -> top-k). No new storage.
Gated on a configured endpoint plus tools.rerank_bm25 (default on, matching
rerank_web_search). tools.rerank_bm25_threshold (default 0.0 = off) is a
relevance FLOOR for proactive memory surfacing: BM25 always returns something,
so without a floor every-turn memory injection spends tokens on the top-k of
whatever lexically matched; the reranker score is what makes a meaningful
"inject nothing" gate possible.
Two reranker modes (BM25Index rerank_filters):
- REORDER (reactive tool/skill search): the reranker must never drop results
-> fall back to BM25 order on empty, backfill omitted pool items, so a
misbehaving endpoint can't silently lose tools.
- FILTER (memory, rerank_filters = threshold > 0): a clean empty/short result
is honoured (inject nothing) -- a deliberate divergence from
web_search._rerank_results.
Parse/endpoint failure is a discrete branch from the floor: an empty result
for non-empty input means an unparseable response (a conforming reranker
scores every doc), so the closure raises RerankError and BM25Index falls back
to BM25 order in BOTH modes -- the floor only acts on valid scores.
Also: cap the rerank client timeout at 15s (the per-turn memory path can't
afford tools.timeout's 120s default); move the Reranker alias to rerank.py
(shared, no import cycle); document the endpoint egress in the rerank_bm25
help, the admin Reranker-role description, and docs/tools.md; add
scripts/bench_bm25_rerank.py (manual, needs a live endpoint) to measure
precision@k/MRR lift and recommend a threshold default.
Negative-tested: reorder fallback-on-empty and omitted-item backfill,
filter-mode honor-empty, singleton-still-floored, the parse-fail RerankError
raise, the >= floor boundary, and pool-position-to-doc-index mapping -- each
guard reverted to confirm its test fails, then restored.
list()-materialize the reranker's output inside the guarded block so a
None / non-iterable / lazily-raising reranker falls back to native order
instead of raising out of web_search, and reject bool indices (an int
subclass) the same way _parse_hits already does.
Drop leftover web_fetch references from the rerank settings and Reranker
role help (reranking is wired into web_search only), and document both
endpoint paths (tools.reranker_alias and tools.rerank_url).
Reranking is delegated to an external Cohere/Jina-compatible /rerank endpoint
(self-hosted vLLM/TEI/llama.cpp, or hosted Cohere/Jina/Voyage); Turnstone runs
no reranker model itself. Disabled until an endpoint is configured.
- core/rerank.py: CohereJinaRerankClient (tolerant of results-wrapped and
bare-list responses) + resolver.
- web_search: rerank the SearxNG result pool by query relevance before top-k,
with a native-order fallback on error; answers/infoboxes untouched.
- Reranker as a model definition: add a model with the supports_rerank
capability and pick it under Models -> Roles -> Reranker
(tools.reranker_alias); takes precedence over the tools.rerank_url settings.
Settings: tools.rerank_url/model/api_key, tools.rerank_web_search,
tools.reranker_alias. Docs: docs/tools.md, turnstone.example.toml.
(web_fetch reranking was evaluated and dropped: for single-document chunk
selection it did not reliably beat head-truncation. Reranking is reserved for
multi-item ranking.)
`man` and `math` duplicated capabilities already reachable through
`bash`; `plan_agent` is better expressed as a `task_agent` running a
planning skill, and carried a large amount of special-case machinery
(plan-review gate, refinement loop, per-kind model routing). Removing
all three shrinks the tool surface and cuts per-call token cost.
Also removed, as dead-once-the-tools-are-gone:
- the `math` sandbox executor (`turnstone.core.sandbox`) and its
`[sandbox]` extra; the eval analyst now runs bash-only
- the read-only `AGENT_TOOLS` sub-agent tool set and the `agent`
tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained)
- the plan-review protocol end to end: the `on_plan_review` UI hook,
`resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`,
the `plan_review`/`plan_resolved` SSE events, and their Python SDK /
TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings
- the `model.plan_alias` / `model.plan_effort` settings and the
registry `plan_model` / `plan_effort` routing fields
TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged.
BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the
plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings
from the experimental 1.6 line.
Rename the web_search tool's `topic` parameter to `category` and expand the
enum to general/news/it/science, mapped to SearxNG `categories=`. The model can
now target the right corpus per query (e.g. `it` for code, `science` for
papers) — useful when generic engines rate-limit. The Tavily-era `finance`
topic (no SearxNG equivalent) is dropped. Threaded consistently through
_prepare_web_search / _exec_web_search / both search clients.
BREAKING: the web_search `topic` argument is now `category`.
The SearxNG change reworded _tool_write_compose's duplicate-skip return and
dropped the "identical content" phrase that test_identical_content_skipped
asserts on, failing CI (which then fail-fast-cancelled the parallel matrix
jobs). Restore the phrase (now covering all three bundled files) and assert
the searxng/settings.yml extraction in test_writes_compose_file.
Drop the Tavily and DuckDuckGo (ddgs) web_search backends for a single
self-hosted SearxNG service bundled into the docker-compose stacks.
Core:
- New SearXNGClient + _format_searxng; rewrite resolve_web_search_client to
(backend, searxng_url, searxng_engines, ...). MCP backend + oauth_user guard
unchanged. _resolve_search_client follows storage -> toml -> env -> default
precedence (explicit "" disables, via ConfigStore.stored_keys()).
- Drop the Tavily-era topic=finance (no SearxNG category); topic is now
general/news.
Settings/config:
- Remove tools.tavily_api_key, get_tavily_key, $TAVILY_API_KEY, [api].tavily_key.
- Add tools.searxng_url (default http://searxng:8080) + tools.searxng_engines,
with get_searxng_url/get_searxng_engines.
Compose + bundled config:
- Internal-only searxng service (no published API port, :ro config, /healthz
healthcheck, persistent searxng-cache volume) in both stacks; bundle
turnstone/deploy/searxng/settings.yml (JSON output on, limiter off).
- Caddy serves the SearxNG web UI on :8444 (dev: localhost-only; prod: opt-in).
- bootstrap extractor + wheel packaging updated.
Deps: drop the ddg extra + ddgs mypy override (regenerates uv.lock, removing the
lxml/h2/brotli transitives).
Docs: tools/docker/architecture/openshell + diagrams + config example + CHANGELOG;
docs/docker.md carries the AGPL-3.0 §13 operator note.
BREAKING: tools.web_search_backend no longer accepts "tavily"/"ddg";
tools.tavily_api_key and the ddg extra are removed. Run the bundled SearxNG (ships
in the compose stacks) or set TURNSTONE_SEARXNG_URL to an external instance.
Closes#545
run.sh autodetects Ubuntu/Debian, Fedora/RHEL, Arch, and WSL; ensures git and
Docker; clones, builds, picks free ports (Caddy prefers 443, Postgres 5432),
generates a .env with a JWT secret and Postgres password, asks how many nodes to
run, and starts the stack. The node count persists via an auto-loaded
compose.override.yaml, so a later plain `docker compose up -d` keeps it; a fresh
clone with no override still starts all 10. Ignores the generated override.
Lead with privacy, local-first, and no-telemetry; demote governance to an
optional team-controls line. Fix the dashboard URL (Caddy on 8443), add the
one-line installer, and link the Discord community.
The bundled production compose mounts ./Caddyfile, so write_compose has to write
it alongside compose.yaml or `docker compose up` fails to start Caddy. Update
the wizard's system prompt for the new model (no profiles, Caddy-fronted
dashboard, current ports).
Without a Discord/Slack token the channel gateway exited, which crash-loops
under `restart: unless-stopped`. Run the HTTP server and service heartbeat with
zero adapters instead — registered and idle — until a token is set. The Slack
token-pair mismatch stays a hard error.
A node refused to start with no model configured and no LLM reachable, so a
fresh cluster couldn't come up to be configured. load_model_registry now
accepts allow_empty and returns an empty registry; ModelRegistry permits the
empty state (default unset); the server passes allow_empty so a node registers
and shows in the console, then picks up models added in the admin UI live. The
CLI keeps failing fast — a REPL with no model is unusable.
Migrations run on every node at boot, and one migration rebuilds an index with
CREATE INDEX CONCURRENTLY, which can't run in a transaction and waits for all
concurrent transactions to drain. The advisory lock that serialises migrations
was held inside an open transaction, so the lock-holder's own idle-in-
transaction connection deadlocked the concurrent build when several nodes
started together. Acquire the lock on an AUTOCOMMIT connection and poll
pg_try_advisory_lock so no waiter pins a snapshot. Adds a Postgres concurrency
regression test (skipped on SQLite).
`docker compose up` from a clone builds one image and brings up the whole stack
— PostgreSQL, console, Caddy, channel, and 10 server nodes — sharing one
Postgres so the console discovers every node. The dashboard is reachable only
through Caddy (HTTP/2 avoids the browser's 6-connection cap on the dashboard's
SSE streams); the console's plain-HTTP port is no longer published. Postgres
binds 127.0.0.1 so a bare-metal turnstone-server can join the cluster — the
bare-metal overlay is folded in and removed. Insecure dev defaults keep it
zero-config; the bundled production stack mirrors the shape but pulls ghcr
images and requires real secrets.
Move the Caddyfile under turnstone/deploy so it ships in the wheel; update docs,
QUICKSTART, and .env.example to match.
The guard tests were appended via heredoc, bypassing the editor's
auto-format; ruff-format collapses one wrapped call onto a single line.
No behaviour change.
The intent-validation judge runs before the approval gate resolves, and
Smart Approvals (judge.smart_approvals) parks approve_tools on the async LLM
verdict for up to judge.timeout — so the tool-call card never reached the UI
until the judge had ruled. An operator could not see a committed call, let
alone Stop it, during that window.
approve_tools now emits a tool_pending event carrying the serialized batch at
the top of the gate, before the tool-policy lookup, the verdict wait, and the
human prompt. It is a UI paint only — no persistence, audit, or verdict
bookkeeping — so it cannot perturb the gate's accounting. The authoritative
tool_info / approve_request / tool_result events that follow upgrade the same
construct in place, keyed by call_id, and the Last-Event-ID replay slice
reconstructs it on reconnect. A ToolPendingEvent joins the SDK registry.
Coordinator: appendToolBatch was already idempotent on call_ids; the new
handler reuses the --running placeholder it already upgrades, with an
"Evaluating" kicker that swaps to "Running" on the auto-approve upgrade.
Interactive: showInlineToolBlock was create-only, so a second card would
duplicate. Added announceToolBlock + _takeAnnouncedBlock to reuse the
announced shell (matched on its call_id set) instead. The announced rail is
dashed amber and must out-specify the .msg.ts-approval--inline cyan-hold
(specificity 0,2,0) — at 0,1,0 it rendered cyan, indistinguishable from a
normal card — so the announced card is the one visually distinct surface in
the stream.
Screen-reader parity: the early paint announces politely through dedicated
off-screen aria-live regions on both surfaces (the messages log is
aria-live=off mid-stream, so the appended shell alone is inaudible), and the
announced shell carries aria-busy until the upgrade clears it. Polite, not
assertive — the human gate keeps its assertive announcement.
Tests cover the gate ordering (tool_pending precedes tool_info and the Smart
Approvals gate) plus string-guards on both UIs' wiring, the announced-rail
specificity, and the screen-reader regions.
risk_level is server-supplied and was interpolated straight into className
and data-risk at three sites — updateVerdictBadge, _buildOutputWarningEl,
renderVerdictBadge — as `risk_level || "medium"`. Whitespace, a stray case,
or a future relaxed-validation value would pass into the class string and
silently break the selectors that updateVerdictBadge, toggleVerdictDetail,
and the d-key handler rely on. It is not an XSS vector (className assignment
is text-typed), but a broken selector is a real failure.
Funnel all three through a normalizeRiskLevel() chokepoint backed by a
{low, medium, high, critical} allowlist; unknown or blank falls back to the
neutral "medium" default. Pre-existing; surfaced while rendering verdict
badges from the early-paint path.
The floor blocks only explicit heuristic deny/critical verdicts — it is
not a general "never lower the heuristic" rule. The heuristic default for
an unmatched tool is `review`, and letting a confident LLM `approve`
upgrade a `review` is the feature's purpose. Matches the implementation
and addresses PR review feedback.
Opt-in judge.smart_approvals (default off): when the intent-validation
LLM judge returns a high-confidence "approve" verdict, the tool batch is
approved automatically with no operator prompt. review/deny recommendations,
low confidence, judge errors (llm_fallback), and a deterministic heuristic
deny/critical finding all still require a human. Requires judge.enabled.
- Batch-atomic: a parallel tool batch auto-approves only if every call
qualifies; one non-qualifying call holds the whole batch for a human.
- Gate: tier==llm + recommendation==approve + confidence >=
judge.confidence_threshold (default raised 0.7 -> 0.95), with a floor
that never clears an explicit heuristic deny/critical verdict.
- approve_tools waits for the async LLM verdicts, finalises the audit
trail (AutoApproveReason.smart_approval), and re-emits verdicts after
the card so the live chip updates; the auto-approved row renders the
LLM verdict rather than the cautious heuristic carry-over.
- judge: always deliver exactly one verdict per call (fallback on error);
reject non-finite confidence so NaN can't clear the bar.
- Drop verdicts from a superseded judge generation so a reused call_id
from a prior turn's still-running daemon can't satisfy the gate's wait.
Config plumbed through the server/console/CLI builders and the live
_judge_cfg; admin Judge tab renders the toggle. Docs + example config
updated. ~35 tests covering the gate matrix, batch-atomicity, the
heuristic floor, audit stamping, the streaming re-emit, NaN/duplicate-id
defenses, and the cross-turn generation guard.
- Surface the degraded state in each node item's aria-label (only
unreachable was included), so screen readers announce it alongside
the visible DEGRADED word.
- Give the trigger an initial aria-label="Nodes" so it isn't an unnamed
control before the first snapshot render populates it; renderNodePicker
overwrites it with the live count/version once data arrives.
The trigger only showed a border on hover/open, so at rest it read as
plain text rather than a control. Add a persistent recessed box (subtle
fill + border) so it's visibly clickable, and brighten the border to
accent on hover and when open.
The always-visible NODES table dominated the coordinator-first landing
page for information most users glance at rarely. Replace it with a
compact node picker in the cluster status bar: the rightmost segment
shows "N nodes" + the cluster version (or a DRIFT chip on mixed
versions), and clicking it opens a dropdown of every compute node with
its live workstream count. Selecting a node navigates to /node/{id}/ —
the same destination the table rows linked to.
The picker reads the same /v1/api/cluster/snapshot + SSE data the table
did (via the retained buildNodeInfoFromSnapshot), so no backend change
was needed; the table was a pure client-side render. The node-grouping
/ prefix-collapsing JS and all the table CSS are removed.
Accessibility / design:
- Status is encoded by shape and colour (round = healthy, diamond =
degraded, square = unreachable), mirroring the .csb-state-dot
vocabulary, plus a spelled-out DEGRADED/DOWN word — colour alone
fails at 7px for color-blind users.
- DRIFT renders as a solid amber chip (dark text on fill) so it reads
as a real alert rather than yellow-on-yellow.
- role="menu"/menuitem (navigation, not selection), aria-haspopup,
aria-expanded; Escape / outside-click / Arrow / Home / End handling;
full node id surfaced via title when the name ellipsizes; menu height
clamped to the viewport so a long list never touches the top edge.
tests/test_console.py: assert the picker markup is served and the old
table markup (#view-overview / #node-table) stays gone.
Address PR review threads:
- _on_renewed now wraps the client-context hot-reload in try/except, so a
load_cert_chain failure can't abort the renewal callback before the
frontend-bundle update (matching the node-side reload hook).
- The startup gc_expired_certs() sweep is contained like the periodic one,
so a malformed legacy cert row can't abort the TLS block and skip the
proxy/collector mTLS client setup that follows it.
Enabling mTLS broke the cluster in three layered ways:
- Service certs were keyed on socket.gethostname() (the container ID) and
never carried the advertised service name as a SAN, so every collector and
routing-proxy handshake failed the hostname check. build_cert_hostnames()
now puts the advertised host first: it becomes the cert's primary domain
(hence a SAN) and a stable store key that survives container recreation.
- lacme's RenewalManager renews everything in the store; with the store shared
cluster-wide, every node renewed every other node's (and every dead
container's) cert — an N×M renewal storm. _SingleDomainStore scopes each
node's sweep to its own cert, and the console adds a periodic GC for the
certs of long-departed nodes.
- uvicorn loads its cert once at boot and never reloads, so renewed certs
never reached the listener and the served cert expired mid-process.
swap_context_cert() hot-swaps renewed material into the live SSL context
(server listener and console client context) via load_cert_chain.
Observability and browser access:
- The collector logged connection/TLS failures at DEBUG, so a persistent
mTLS-verify failure was invisible. It now logs the first failure per node
(reachable->unreachable) at WARNING and stays at DEBUG on retries.
- The console serves plain HTTP (it is the ACME bootstrap endpoint) and no
longer rewrites its advertised URL to https://. Browser->console TLS is
terminated by a reverse proxy: the cluster profile gains a caddy service
(browser h2/HTTPS -> caddy -> console h1.1/HTTP) plus browser-TLS docs.
Tests: tests/test_tls_san_renewal.py, tests/test_collector_reachability.py.
* feat(audio): voice I/O — speech-to-text + text-to-speech via model roles
Browser voice input/output over the OpenAI audio wire protocol, selected
through the existing model-roles system so the same code path serves OpenAI,
vLLM/vLLM-Omni, or any compatible backend — pure registry config, no new
in-process deps. Anthropic has no audio API, so it is capability-gated out of
the audio roles while remaining valid as the agent model.
Backend
- core/audio.py: role resolution + capability gating + transcribe()/synthesize()
over a registry-resolved client. Typed AudioUnavailableError (503) /
AudioBackendError (502 — body masked, SDK detail logged). Optional STT prompt.
- Endpoints POST /v1/api/workstreams/{ws_id}/speech-to-text and POST /v1/api/tts,
registered in v1_routes, write-scoped (direct + proxied), offloaded with
asyncio.to_thread. Silence -> 422; configured-but-failed backend -> masked 502.
- Model roles: audio.stt_model_alias / audio.tts_model_alias / audio.tts_voice /
audio.stt_prompt settings; Models -> Roles entries (capability-gated dropdowns,
"(disabled — voice off)" when unset). /v1/api/models exposes resolved
stt_default_alias / tts_default_alias + per-model capabilities.
- Capabilities: supports_transcription / supports_speech_synthesis on
ModelCapabilities; current OpenAI audio lineup (whisper-1, gpt-4o[-mini]-
transcribe, tts-1[-hd], gpt-4o-mini-tts) registered as known models, with a
name-inference backstop for local/openai-compatible aliases.
Frontend (interactive UI)
- Mic dictation (record -> transcribe -> fill composer for review) and
per-message playback, shown only when the role is configured.
- CSS-mask icon set, aria-pressed + live-region announcements, recording timer,
reduced-motion cue, error-typed toasts + persistent denial, mic disabled while
busy, code/math stripped before TTS.
Tests: new test_audio.py plus STT/TTS endpoint, settings, openapi, available-
models, and OpenAI-lineup capability coverage. ruff + mypy + node --check clean.
* fix(audio): use const for AUDIO_MODEL_HINTS (var-sweep invariant)
Saved Workstreams / Saved Coordinators (shared createSavedTable):
- Re-add the pagination retired by #611, capped at 20 rows/page, in the
shared component so both surfaces stay consistent. The list is fetched
whole and sliced client-side; the delete controller only sees the visible
page so Select-All stays bounded. Page resets on filter/sort, clamps on
shrink, hides on a single page or in delete mode.
- Footer is range-aware ("Showing 1-20 of N"); footer + pager share one
justified row (range left, pager right) so they read as one region.
- Saved rows carry the pointer cursor in the shared cards.css. The console
only set it on .dash-row.has-link, which the shared row builder never adds,
so saved-coordinator rows had fallen back to the default cursor.
Active Coordinators (console home):
- Give the active-coordinators block the full card chrome matching the
server's Workstreams block: a dash-header bar with an "N active / M total"
summary, the shared dash-colheaders band (was missing entirely), the rows,
and a dash-footer count line.
- Share .dash-footer into base.css (was server-only); the server keeps its
bottom-margin override.
- Make both coordinator cards contiguous by dropping the console-only
home-section gap, matching the server which ships both cards contiguous.
Frontend only -- no API, DTO, or migration changes. Pagination logic
covered by a DOM-stub harness; two designer passes applied.
make_history_handler is shared by interactive and coord, so coord
/history already trims the executing in-flight orphan turn and returns
a cursor. But coordinator.js never read it -- it connected fresh, so the
trimmed turn was neither in /history nor delta-replayed and vanished
from the dashboard (a regression vs the prior #610 in-flight render).
Mirror the ui/static/app.js fix in coordinator.js: refetchHistory takes
a seedCursor flag (default false) and seeds lastEventId from hist.cursor
only on the initial-connect path; connectSSE gates ?last_event_id= on
!= null so a cursor of 0 isn't dropped. The clear_ui / replay_truncated
re-render callers leave seedCursor false (they run on a live stream and
must not rewind the live cursor). Adds a coordinator.js static guard.
When the active UI is a MagicMock test double, _ui_event_id() returned
the auto-vivified _event_id mock (getattr finds it, so the None default
never applies). That mock reached the conversations INSERT and failed
to bind ("type 'MagicMock' is not supported"), so save_message raised,
the row was dropped, and tests on the real-storage + mock-UI path broke
(CI: test_session_attachments::test_db_row_stores_text_only).
Coerce a non-int _event_id to None so mock UIs -- and counterless
CLI/eval/placeholder UIs -- stamp NULL (the synthetic-snapshot floor),
matching the documented contract. Production UIs always carry an int,
so behaviour there is unchanged.
Also drop two redundant local `import json` in the new /history
integration tests; the module-level import already covers them.
A fresh browser connect during a parallel tool batch (e.g. several
web_fetch) left completed siblings' tool blocks empty until a manual
refresh: each tool_result SSE event fires the instant a sibling
finishes, but the result messages persist only after the whole batch
returns, so a fresh connect replayed neither the already-fired event
(a fresh connect doesn't replay the ring buffer) nor a /history row.
Route the fresh connect through the same delta replay a reconnect
already uses. Persist the per-ws SSE ring-buffer high-water mark
(_event_id) onto each saved conversation row. /history returns the
committed snapshot up to a resolved-turn-boundary cursor and omits the
trailing executing in-flight turn; the client opens its initial SSE
with that cursor (Last-Event-ID) so the existing replay_ok path
fast-forwards the in-flight turn whole -- tool blocks, results, and
approve/plan prompts all rebuild from the ring buffer.
The cut sits at the last resolved-turn boundary (not max(saved
event_id)), so out-of-order result saves in the post-batch loop can't
move it or strand a sibling. Gated on buffer-liveness (can_replay_from):
reloaded / evicted / awaiting-approval cases keep the in-flight turn in
/history and return a null cursor, falling back to the synthetic
snapshot floor -- preserving the existing in-flight render and never
leaving a turn unrenderable.
- Migration 059: nullable event_id BIGINT on conversations + a
(ws_id, event_id) index (keeps the cold-open high-water reseed a seek).
- save_message(event_id=) across the storage wrapper / protocol /
sqlite / postgres backends; get_max_event_id; reconstruct_messages
surfaces the _event_id side-channel.
- SessionUIBase: reseed _event_id from storage on construction (so the
id space stays monotonic across restarts); can_replay_from() gate.
- make_history_handler: _resume_cursor_and_trim() + cursor in the
response (WorkstreamHistoryResponse.cursor). The shared projection,
export, and coord-rebuild paths are untouched.
- app.js: seed the resume cursor on the initial-connect path only, and
gate the last_event_id param on != null so a cursor of 0 (a brand-new
workstream's first-turn boundary) is not dropped.
Tests: helper, storage round-trip, and seed unit tests; two
make_history_handler integration tests (cursor + orphan-trim when
replayable, null cursor + orphan kept when not); app.js static guards.
Migration applies up and down on SQLite.
strip_html deleted every HTML tag with no separator, gluing paragraphs,
headings, list items, and table cells into a structureless run of text
("<p>a</p><p>b</p>" -> "ab"). This degrades web_fetch, which feeds the
cleaned page to a summarising agent — and it flattens the structure any
downstream chunking/retrieval would rely on.
Block-level tags and <br> now become newlines so structure survives
("<p>a</p><p>b</p>" -> "a\n\nb"); inline tags are still dropped.
The conversion is a single linear tag scan: one pass over `<[^>]++>` with
a possessive quantifier, dispatching each tag name against a frozenset.
This replaces three full-document passes plus a 24-way alternation, and:
- Removes catastrophic backtracking (ReDoS). The earlier `<\s*/?\s*` and
`<\s*br\s*/?\s*>` patterns were quadratic on '<' + a long whitespace
run (~2s at 4k chars); the scan is now linear (~3ms at 1M chars) on the
untrusted, up-to-10MB web_fetch input. The possessive quantifier also
neutralises the pre-existing quadratic in the old `<[^>]+>` pass.
- Matches <br> carrying attributes (e.g. `<br clear="all">`), which the
first cut missed.
Tests cover block separation, inline-tag joining, uppercase tags, <br>
with attributes, lookalike tag names, and a pathological-whitespace
regression guard.
Note (pre-existing, not changed here): in _exec_web_fetch the 10 MB cap is
applied after strip_html, so the stripper sees the full fetched body. With
the scan now linear this is no longer a CPU concern; capping the raw input
before stripping remains a worthwhile defence-in-depth follow-up.
Add a workstream conversation export on three surfaces, all sharing one
serializer (turnstone/core/export.py):
- `turnstone-admin export <ws_id> [--children] [-o FILE|-]` — offline,
direct-DB. `--children` bundles a coordinator's parent conversation
plus one JSON per child into a zip (parent.json + children/<id>.json,
no manifest).
- `GET /v1/api/workstreams/{ws_id}/export` — conversation-only file
download, mounted on both the node (interactive) and console
(coordinator) lifespans via `make_export_handler(cfg)`, reusing the
/history gate ladder (permission_gate, tenant_check, list_kind
cross-kind isolation) so ownership and isolation come for free.
- Web UI — an "Export conversation" item in the interactive per-tab
dropdown (scoped to that tab's workstream) and an Export button on the
coordinator appbar.
Format is OpenAI Chat Completions messages JSON (`{"messages": [...]}`),
built from `sanitize_messages(load_messages(repair=True))`. Persisted
reasoning is surfaced on assistant messages as a flat `reasoning_content`
field (the convention OpenAI-compatible inference servers use) via a
dedicated helper that runs before sanitize strips the internal
_provider_content lane. Attachments ride along as the standard image_url
/ inlined-document content parts.
Lets users get conversations out in a portable interchange format
(backup, fine-tuning datasets, sharing, interop) without lock-in.
Closes#613.
Non-obvious decisions:
- Single format (openai-json); children/zip is CLI-only. The HTTP
endpoint and web UI are conversation-only, keeping the served surface
— and its security surface (no child rows read through the coordinator
handler) — small.
- `reasoning_content`, not the `reasoning` field /history and the
reasoning-replay path use: export targets the chat-completions
convention. Documented in export.py to prevent a "consistency fix".
- list_workstreams exposes no cursor, so the child walk passes an
explicit high limit rather than inheriting the default 100, which
would silently drop a coordinator's children past 100.
- Interactive export lives in the per-tab menu (interactive is
per-tab/pane — avoids focused-workstream ambiguity); the coordinator
is one conversation, so it keeps an appbar button.
Tested: 25 new tests through real storage + handlers (TestClient), incl.
cross-kind isolation 404, misconfig 500, the reasoning + attachment
pipeline, and the coordinator children zip. The shared frontend helper
is verified by a node sandbox harness (re-entrancy guard, button
disable/aria-busy, no-button tab-menu path). Full non-live suite green
(6714 passed); ruff + format + mypy clean; OpenAPI spec updated.
PR #612 review (Copilot): the synthetic `error` event surfaced on a fresh
connect carried no SSE `id:`, so the client's `lastEventId` never advanced.
The client's `error` handler is append-only (not idempotent like
`state_change` / `in_progress_snapshot`), and a terminal-errored idle ws
emits no live event to set a cursor — so a native EventSource reconnect sent
no `Last-Event-ID`, re-ran the fresh path, and appended a DUPLICATE error
bubble on every reconnect cycle (proxy idle-timeout, network blip).
Attach `id: str(snap_seq)` (the registration-time buffer cursor already in
scope) to the surfaced error. The reconnect then sends that `Last-Event-ID`
→ `register_listener_with_replay` returns `replay_ok` (nothing buffered past
snap_seq on an idle ws) → the handler's replay_ok branch skips the synthetic
surface. No duplicate.
Test asserts the surfaced error carries `id: snap_seq`; the existing
`test_handler_replay_ok_does_not_resurface_last_error` pins the
reconnect-skips half.
A browser connecting fresh to a workstream sitting in the error state
saw the error STATE (composer unlock + retry, via the replayed
state_change) but not the error TEXT explaining why — on a fresh connect
there was no source for it. `on_error` is never persisted as a message,
so `/history` can't rebuild it; only the reconnect path (ring buffer)
carried the original `error` event.
Surface the persisted `last_error` in `make_events_handler`'s
fresh/truncated synthetic-replay branch when the workstream is in the
error state. Gated on the error state so a healthy ws skips the storage
read, and confined to the fresh/truncated path — the `replay_ok` branch's
ring buffer already replays the original `error` event, so surfacing here
would double it. The persist (`_record_fatal_error`, sanitized) / clear
(on recovery) lifecycle already exists; this only reconstructs the event
on a fresh connect, reaching parity with reconnect.
Second of the fresh-connect replay-completeness fixes surfaced by the
audit (sibling to the tool-call `pending` fix in this branch). Non-terminal
mid-turn errors (tool parse failures, truncation — not state=error, not
persisted) remain an accepted gap; the queued-message indicator gap is
deferred (needs client-side render-on-replay).
Adds two parity tests: fresh+error → surfaced / fresh+idle → gate skips,
and replay_ok → not double-emitted.
In-flight tool calls did not render when a browser connected fresh to an
in-progress workstream mid-tool-execution; they only reappeared after the
SSE dropped and reconnected.
`project_history_messages` marked the trailing tool-call turn `pending`
from orphan-detection (a tool_call with no result) as a proxy for
"awaiting approval". But an orphan that is *executing* (already approved,
running) is orphan-but-not-awaiting. The renderer skips `pending` turns
because the SSE replay re-emits the interactive approve_request prompt
instead — and during execution `_pending_approval` is None, so nothing
re-emits. The tool call rendered from neither source on a fresh connect,
recovering only on reconnect (ring-buffer replay carries the
tool_info / tool_result events).
Regression from the REST-first history convergence (0ad1ab7f), inherited
by the wire-shape unification (#596): both swapped the `pending` predicate
from the live `_pending_approval` signal to storage orphan-detection,
which diverge exactly during tool execution.
Thread the live awaiting-approval signal from `make_history_handler` into
`project_history_messages` (new `awaiting_approval` param) and gate the
pending mark on it, re-syncing `pending` with the same `_pending_approval`
signal that drives the SSE prompt re-emit. A storage-only / closed ws has
no live session → never pending → trailing orphans render as historical.
Asserted as `dict` to match the detail handler's MagicMock-safe guard.
Adds a projection-level gate test and two handler boundary tests
(execution → renders, awaiting approval → pending). The existing
partial-trailing-turn test now asserts the turn RENDERS, not just that the
row survives — the parity gap that let this regression through.
Replaces the Saved Workstreams (ui/static) and Saved Coordinators
(console/static) card grids with a dense, sortable table that reuses the
active dashboard's row system, via one shared component in
shared_static/cards.js (renderSessionRow, SavedColumns, createSavedTable)
+ cards.css. The two surfaces differ only by column spec (MSGS vs CHILDREN)
and per-app data/ids/delete-request; everything generic is shared.
- NAME flexes to full width (kills the card grid's near-duplicate-name
truncation); client-side filter + sortable headers; scroll-all
(pagination retired); multi-select delete preserved on rows.
- Consumes the enriched saved-list DTO: MODEL, CTX (context-window
occupancy, a frozen last-activity snapshot), SKILL chip, CHILDREN, and a
red left-edge for failed runs.
- Saved rows reuse the dash-table chrome but opt out of the active table's
live-state styling: idle rows aren't dimmed, CTX reads as a snapshot (not
the live gauge), legible zebra + AA-contrast muted text for a long
terminal list, and responsive compact columns keep NAME readable on
narrow viewports.
- a11y: sortable headers exposed to assistive tech (aria-label / aria-sort
+ at-rest carets); footers are live regions.
- Removes the now-dead renderSessionCard + card-grid CSS.
GET /v1/api/workstreams/saved returned only ws_id/alias/title/created/
updated/message_count — too little to drive the planned saved-list table
redesign. Add seven fields, all sourced from already-persisted data (no
migration):
- state, kind, node_id: columns on the workstreams table
- model_alias, launch_skill: from workstream_config via LEFT JOIN
- child_count: COUNT of child workstreams via parent_ws_id
- context_tokens: most recent usage_events prompt size for the workstream
- context_ratio: context-window occupancy (context_tokens / model context
window), computed in the handler so the NULL / zero-window cases stay
explicit and identical across both storage backends
context_window comes from a model_definitions join; aliases defined only in
config.toml are absent there, so context_ratio degrades to 0.0 rather than
reporting bogus occupancy. The Python SDK reuses the Pydantic model; the
TypeScript SDK OpenAPI snapshot and hand-maintained interface are updated.
Tests cover the new storage columns (including NULL-when-absent), the
handler ratio math + zero-window degradation, and the SDK enriched
round-trip.
The Usage dashboard summary cards read the oldest day bucket
(`summary.breakdown[0]`) instead of the window SUM, so every headline
(total/prompt/completion/tool-calls/cache) showed a single day's value —
e.g. 30-day tool-calls reading lower than 7-day. Read `.summary[0]` and
collapse the redundant two-request fetch into one (the response already
carried both `summary` and `breakdown`).
Only the main streaming loop (`on_status`) recorded `usage_events`.
Auxiliary non-streaming calls — title generation, conversation
compaction, web-fetch summarization, and plan/task sub-agents — bypassed
that path and were never counted, undercounting real consumption by a
large factor for agent-heavy workstreams. Add an `on_aux_usage` UI hook
(storage row via a shared `_write_usage_row` helper with `on_status`;
`WebUI` override feeds Prometheus) and route `_utility_completion` and
sub-agent turns through it, attributed to the agent's own model. Judge
token spend remains uncounted — deferred to a follow-up.
GET /v1/api/models blanked default_alias whenever model.default_alias named
an alias absent from the server's live registry — e.g. when a standalone
turnstone-server shares a ConfigStore with a console whose model.default_alias
points at a console-only / DB alias (or the underlying model id rather than
the alias). The interactive dashboard then showed a bare "Default model"
placeholder even though a new workstream launches on a concrete model.
Mirror session_factory's _effective_default_alias / _effective_routing: fall
back to registry.default (which already incorporates a *valid* model.default_alias
override) when the configured alias is unset or foreign, blanking only if
registry.default is itself unresolvable. The endpoint now reports the model
creation actually uses.
openapi-server.json / openapi-console.json had drifted well behind
build_server_spec() / build_console_spec() — the committed snapshots are
regenerated periodically (via sdk/typescript/scripts/generate-types.py)
rather than on every schema-changing PR, so accumulated additions (skill
parsing, pending-approval items, model-definition CRUD, etc.) had not been
captured. This resyncs both with no code changes.
Addresses #603 review: the pagination control is a sibling of the cards
container, so loadDashboard()'s "Loading…" / "Failed to load" states (which
replaceChildren only the cards) left stale Prev/Next visible and still wired
to the previous _wsSavedItems cache — in the error state clicking them would
resurrect the old cards over "Failed to load". Route both transient states
through a _setSavedWsMessage() helper that clears the cards and hides the
pagination in lockstep; a successful load re-renders both via
renderSavedWorkstreams.
The interactive dashboard had drifted from the coordinator launcher in two
ways; this backports both for consistency.
Selectors: the Model / Judge Model dropdowns now show the resolved default
model in the placeholder (e.g. "Default — primary (vendor/primary)") instead
of a generic "Default model" / "Default (agent model)". The server's
GET /v1/api/models now returns judge_default_alias (mirroring the console
endpoint), sourced from the judge.model setting. It is intentionally left
blank when judge.model is unset or points at a disabled/removed alias,
because the judge then follows the per-workstream agent model at runtime
(session_factory: judge_config.model or model) — the UI keeps the honest
"Default (agent model)" wording in that case. This also fixes a latent
mislabel: the judge row previously said "agent model" even when an operator
had configured judge.model.
Pagination: Saved Workstreams now paginates at 24/page (Prev · X / Y · Next),
matching Saved Coordinators — page clamp after deletes, hidden on a single
page and in delete mode, Select-All bounded to the visible page. The empty
branch drops out of delete mode (matching the launcher) so the toolbar can't
linger over an empty grid.
The shared .pagination CSS is lifted from console/static/style.css into
shared/cards.css (loaded by both apps) so the two dashboards keep one source
of truth instead of a third copy. The pagination JS render wiring stays
per-app (it binds per-app DOM ids + controller instances) with a
cross-reference comment to its console twin.
Tests: new tests/test_server_available_models.py pins the judge/model
resolution chain (unset / configured / unknown / whitespace / registry-default
fallback).
The output-guard judge's model (judge.output_guard_model) was only
configurable on the Judge settings tab, while every other model role —
coordinator, intent judge, plan/task agents, channel adapter — lives in
Models → Roles. Add it there as a role (mirroring the intent Judge role)
and skip it on the Judge tab so it renders in exactly one place.
No backend change: the role read/write goes through the generic
/v1/api/admin/settings endpoints, the same path the intent-judge model
role already uses.
- LLM-tier row no longer duplicates the judge reasoning into its
annotations column — reasoning lives in the dedicated reasoning column,
so annotations stays heuristic-only and audit consumers aren't confused.
(The replay merge reads the heuristic row's annotations + the LLM row's
reasoning, never the LLM row's annotations, so this is display-safe.)
- Correct the output-warning chip comment: tier "llm" means the judge
returned a verdict (it may have cleared a heuristic-positive), not that
it owns the displayed finding.
After the heuristic+LLM merge, the LLM verdict no longer "overrides" the regex
verdict; it merges (risk = max, flags = union) and can raise but never lower a
regex finding. Fix the admin Settings help string to match.
Surface the output-guard LLM judge on the inline finding chip and merge it
with the regex heuristic instead of one stage winning outright.
Merge rule (issue #560, "show, annotated"):
- risk_level = max(heuristic, llm); flags = union. The judge can escalate
but never lower a heuristic positive — it evaluates adversarial tool
output, so defeating it must not erase a deterministic regex finding.
Credentials stay heuristic-only and are always redacted.
- The judge's own verdict rides along as a dissent-aware annotation
(judge_risk / confidence / reasoning / judge_model) on the chip in both
the interactive and coordinator UIs, live and on reconnect. One shared
merge_guard_display_payload drives both paths so they cannot drift.
- The model is shown the merged risk + flags but never the judge's "benign"
verdict — a fooled judge must not talk the model out of caution.
Fixes a reconnect bug: a judge that ran but failed wrote a risk="none" row
that won the replay dedup and hid the heuristic finding (it showed live but
vanished on refresh). Failed judges now persist under tier="llm_error",
excluded from the display merge; the max-merge also floors the displayed
risk at the heuristic level so the chip never vanishes.
Also adds a regression test confirming the LLM judge runs on every tool
output, not just heuristic-flagged ones.
Tests: merge unit tests, storage-backed replay regression, live/replay
wire-shape parity, SDK-event drift guard. ruff + mypy clean.
The per-message rewind / edit / retry affordance — the icon glyphs
(.icon-edit / .icon-rewind / .icon-retry) plus the inline edit-in-place
form (.msg-edit-*) and the [data-busy] / .msg-editing states — was
duplicated verbatim in both pane stylesheets: ui/static/style.css
(interactive) and console/static/coordinator/coordinator.css
(coordinator). PR #598 deferred consolidating them to keep that
coord-only change off the shipped interactive stylesheet's cascade.
Move the block into shared_static/chat.css, immediately after the
.msg-actions / .msg-action-btn primitives both panes already share, and
delete both copies (including coordinator.css's now-obsolete FOLLOW-UP
note describing the duplication).
Both index.html files load chat.css before their pane stylesheet, so the
rules land earlier in the cascade; the selectors are unique (defined
nowhere else, confirmed repo-wide) so it is a visual no-op. The block is
moved verbatim — chat.css's sibling rules use a different variable
vocabulary (--r-sm=4px / --font-mono) than the affordance block
(--radius-sm=3px / --font-ui), so renaming would change radii/fonts.
Verified pixel-identical via a headless-Chrome render-diff of both panes,
before vs after, across all four affordance states (edit+rewind, retry,
editing-open, busy): 0 differing pixels.
The capture-phase window scroll listener that dismisses an open overflow
menu also fired for scrolls originating inside the menu itself (the menu can
overflow-y:auto at high browser zoom / short viewports), so a tall menu
closed the instant you tried to scroll it. Skip scroll events whose target
is inside .admin-kebab-menu; page/ancestor scroll still dismisses.
Addresses review feedback on #599.
Admin tables render their per-row actions in an ACTIONS column whose grid
track is fixed-width and inherits `.admin-col`'s overflow:hidden +
text-overflow:ellipsis. Rows with several actions (MCP servers:
refresh/reconnect/edit/del) overflowed the track, so only the first button
showed and the rest were clipped behind a misleading "…". The edit button
was unreachable — the only way to change an MCP server was editing the DB
by hand (#593).
Replace the inline button strips across all 15 admin tables (admin.js +
governance.js) with a shared kebab (⋯) overflow menu:
- _kebabMenu()/_kebabMenuEl() build the menu; a single action degrades to
an inline button. _initKebabMenus() wires open/close, outside-click,
Escape, arrow-key nav and viewport-aware flip-up/flip-left through one
document-level delegated listener. Menu items keep the same data-*
attributes, so the existing per-table click bindings are unchanged.
- Right-align the actions column so the trigger shares an edge with its
menu; pair danger/caution items with warning glyphs (not colour alone);
theme-aware --red-glow/--yellow-glow hover tints.
Fixes#593.
_refreshRetryButton() ran only from finishAssistantStream (live turn ends), so a reloaded coordinator workstream or a clear_ui/replay_truncated re-render after rewind/retry showed assistant turns without the retry button — unlike the interactive replayHistory() path, and inconsistent with the edit/rewind buttons (which DO attach on re-render via appendUserMessageWithAttachments). Call it at the end of refetchHistory() so retry attaches on every render. (Addresses Copilot review on PR #598.)
Give coordinator workstreams the rewind/retry/edit affordance the interactive pane has, completing the #549 verb lift on the frontend. Browser-verified separately (the console won't boot in-sandbox).
- Factor refetchHistory() out of init(): the history-render block moves into a reusable function that clears the message column + tool-tracking state (toolRows / activeBatch) before re-rendering, so a mid-session re-render leaves no stale call_id->row mappings. init() keeps the first-paint-only pending-approval replay + children/tasks/attachments/SSE.
- SSE: case clear_ui re-renders from REST then dispatches the latched edit-and-resend via path-keyed /send; case replay_truncated re-syncs (skipped mid-stream). New _pendingEditSend module latch.
- Per-message affordance mirroring the interactive pane: edit + rewind buttons on every user bubble, a retry button on the last assistant turn (skipped when the turn ended tool-only, gated on .coord-tool-batch). Bare .msg.user turn-count matches the server's _find_turn_boundaries.
- Port the icon-glyph + inline-edit-form CSS into coordinator.css (.msg-actions / .msg-action-btn already live in shared chat.css; consolidating all of it into shared chat.css is a tracked follow-up).
coord-LOCAL mirror (not a shared_static extraction): app.js is class-based over this, coordinator.js is a module closure.
Lift the conversation-modifying /rewind and /retry verbs out of the body-keyed POST /v1/api/command into path-keyed POST /v1/api/workstreams/{ws_id}/rewind ({turns:N}) and /retry, as make_rewind_handler/make_retry_handler in SharedSessionVerbHandlers (template: make_close_handler/make_cancel_handler), wired on both interactive and coordinator kinds. Closes the last unlifted conversation-modifying surface — coordinator workstreams gain rewind/retry where they had none — and removes the surviving exception to the post-#422 path-keyed URL convention.
Handler shape: auth gate (coord -> admin.coordinator via permission_gate; interactive -> conversation.modify via accepted_permissions) -> busy-gate -> session.rewind(n)/retry() -> always emit clear_ui (incl. rewind-to-zero, carries #503) -> audit (conversation.rewind/retry on both kinds). Retry re-dispatch reuses the shared session_worker.send via a per-kind dispatch_retry closure (hard-reject on busy), not a third hand-rolled thread.
The web /command handler now rejects /rewind+/retry with a pointer to the path-keyed endpoint (BREAKING; 1.6.0aN-tolerant); session.handle_command's branches stay for the terminal CLI. auth.py adds the verbs to both write suffix-sets; Python + TS SDKs, OpenAPI (RewindRequest + server/console specs), the /route/ proxy mounts + audit actions, and coordinator_client all gain them.
Interactive frontend (app.js): the 3 /command POST sites + the hand-typed-slash reroute now hit the path-keyed endpoints; the bare .msg.user rewind selector is kept (matches the server's _find_turn_boundaries, which counts system-nudge user turns). The coordinator frontend rewind UX lands in a follow-up commit (browser-verified).
Tests: route-walk mount/order, /route/ audit rows, required_scope, OpenAPI catalog, SDK body-inspection, and HTTP-level handler behavior (busy-gate, turns validation, clear_ui emit, retry dispatch, audit invocation + swallow).
Register `claude-opus-4-8` in the Anthropic capability table. Opus 4.8
shares Opus 4.7's request/response surface exactly — adaptive-thinking
only (`budget_tokens` rejected), sampling params removed, the
low/medium/high/xhigh/max effort levels, `thinking.display` defaulting to
omitted, 1M context, and 128K output — so the entry is a verbatim copy of
the 4.7 row. `_lookup_capabilities` longest-prefix matching then resolves
date-suffixed ids (e.g. `claude-opus-4-8-20260601`) without colliding
with the 4.7 key.
No provider code paths change: the existing 4.7 handling already covers
all of 4.8's behavior. Models are selected via config.toml / the admin
ConfigStore UI, so there is no catalog or dropdown to update.
- _anthropic.py: new claude-opus-4-8 capability entry + effort comment
- tests/test_providers.py: opus 4.8 bare + dated capability tests
- turnstone.example.toml: bump the showcased model example to 4.8
Migrate the coordinator dashboard's init() history rebuild onto the
server-projected wire shape (the prior commit's project_history_messages),
retiring its inline raw-storage-shape handling. Field-sourcing only -- the
batch render + orphan->--running state machine is unchanged:
- callOutcomes reads the server-derived `denied` / `is_error` flags
instead of re-sniffing tool content prefixes;
- tool_calls are read flat (`tc.name` / `tc.arguments`) now that the
projection flattens the nested `function` wrapper;
- user content is a plain string and attachments come from the projected
`attachments` list, replacing the multipart walk + `_attachments_meta`
side-channel read.
Fix two latent reload bugs along the way: coord read `m.reminders` /
`m.source` but the raw shape carried `_reminders` / `_source`, so metacog
reminder bubbles and the system-nudge marker never rendered on a coord
history reload. The projection surfaces both top-level, so coord's
existing (unchanged) render paths now fire.
Repoint the test_coordinator_page deny-classifier guard at the live
`m.denied` / `m.is_error` reads instead of comment prose.
Refs #549.
Collapse the three hand-synced "raw storage -> render shape" projections
into one server-side projection. The projection previously lived in a
test-only `_build_history` (SSE-era reference impl), a client-side JS
normaliser (`history_normalize.js`, the transitional bridge), and coord's
inline `init()` handling -- drifting silently with no parity test.
Add `project_history_messages` to `history_decoration.py` and run it as the
final step of the `make_history_handler` pipeline (load_messages -> decorate
-> extract_reasoning -> project), so `GET /history` emits the canonical
render shape directly: flat tool_calls (with verdict / output_assessment),
top-level source / reminders / attachments, collapsed multipart content,
derived denied / is_error / pending, reasoning, and advisories. Interactive
`replayHistory` now consumes the payload verbatim.
Close two gaps the JS bridge deferred:
- list-content <tool_output> advisory extraction (decorate handles only
string content; the projection extracts list-carrier advisories, then
joins remaining text parts to the string the renderers require);
- orphan->pending marks ONLY the last orphan tool-call turn, so a
mid-conversation cancelled tool still renders instead of vanishing.
Delete `history_normalize.js` (+ its <script> tag and node test) and the
test-only `_build_history` (+ orphaned imports); retarget its direct tests
onto the projection helpers. Update the WorkstreamHistoryResponse
description and the Web UI Resilience architecture note to the projected
shape.
Coord's `init()` still reads the raw side-channels; migrating it to the
projected shape is the next commit, browser-verified separately.
Refs #549.
Two issues on the new REST-first history path (PR #595 review + a user repro):
- Stale history on a fast workstream switch: a slow `_refetchHistory` (e.g. a
large resumed session) could resolve AFTER the pane moved to another ws,
rendering the old ws's history over the new one — and its `.finally`
reconnecting the old stream. Add a per-pane load-generation token:
`_loadHistoryThenConnect` bumps it, and the refetch render, the deferred
`connectSSE`, and the `clear_ui` resend are each discarded when a newer load
supersedes them. Fixes the "open a child workstream, see the previously
resumed workstream's history" repro.
- Stale per-ws SSE replay cursor (Copilot): ws-assign callers set `this.wsId`
before `_loadHistoryThenConnect`, so `connectSSE`'s `wsChanged` is already
false, and `reset()` never cleared `_lastEventId` — so a tab switch / child
open sent ws-A's `last_event_id` to ws-B, mis-triggering the server's
`replay_ok` path and skipping the synthetic replay (connected / status /
in_progress_snapshot). Reset `_lastEventId` + `_lastStatusEvt` in
`_loadHistoryThenConnect` so a ws (re)load always opens a fresh stream;
transient same-ws reconnects (direct `connectSSE`) still reuse the cursor
for `replay_ok`.
test_app_js.py pins the load-generation guard.
Interactive fetched conversation history as an inline SSE `history` event on
every (re)connect — a multi-MB payload — while coord fetches it once via REST
`GET /history` and uses SSE for live deltas only. This converges interactive
onto coord's model so both kinds share one history-delivery pattern, the
prerequisite for lifting the `/command` (rewind/retry) verb to coord.
Backend (server.py, core/session_routes.py):
- `_interactive_events_replay` and the `/command` resume/rewind branches no
longer emit the inline `history` SSE event; the open/create-resume paths
emit `clear_ui` only. The `/events` stream no longer carries conversation
history — REST `GET /history` is the source (acceptable on 1.6.0aN).
- Removed the now-orphaned `events_replay_prepare` hook.
- `_build_history` retained as the canonical wire-shape reference for the
decoration/parity tests (no production callers post-convergence).
Frontend (ui/static/app.js, ui/static/index.html):
- `_loadHistoryThenConnect` fetches REST `/history`, renders, then opens SSE
(mirrors coord's `init()` ordering); wired into the seven ws-assign sites.
- `clear_ui` re-renders via a REST refetch and dispatches the edit-and-resend
latch; `replay_truncated` re-syncs (skipped mid-stream so it cannot clobber
an in-flight turn). The `case "history"` SSE handler is removed.
New shared module (shared_static/history_normalize.js):
- `normalizeHistoryMessages` converts the raw provider-native REST shape
(nested tool_calls, `_source`/`_reminders`/`_attachments_meta` side-channels,
multipart content, no derived flags) into the projected shape `replayHistory`
renders. Pure/DOM-free and node-unit-tested. This is a transitional bridge —
a server-side wire-shape unification (folding this projection back into the
server so interactive, coord, and coord's inline raw-handling collapse onto
one shape) is planned to replace it.
Tests: backend replay-omits-history regression; a node-executed normalizer
projection test (incl. the orphan->pending and denial-propagation edges); and
REST-first wiring guards in test_app_js.py.
notify was interactive-only — a coord with a natural "fan-out complete"
or "batch failed" beat could only post by spawning a child for the
single message, which is a lot of ceremony. Routing is session-kind-
agnostic in _prepare_notify / _exec_notify; this is a metadata flip
that adds the coord flag (plus the explicit interactive flag the loader
needs once coordinator is set) and updates the dual-kind whitelists,
coord tool-set assertions, and skill-author docs accordingly. Adds
two coord-session tests pinning the prepare dispatch contract
(needs_approval=False matches notify.json auto_approve) and the exec
→ channel-gateway path.
The Judge → Settings tab has its own renderer in governance.js separate
from _renderSettingRow in admin.js — it builds card-style rows inline
and was painting ``s.help || s.description`` straight into the row,
which is why the Judge settings still showed long paragraphs inline
after the prior commit on this branch.
Convert renderJudgeSettings to the same shape as the generic Settings
tab:
- Short description (s.description) renders inline below the key — the
always-visible TLDR.
- A ? button next to the key gates the long-form s.help paragraph via
a sibling .settings-help-popover with id ``<key>-help``. Picked up
automatically by the document-delegated click handler in admin.js.
Toggle behavior, save/reset buttons, and Judge-specific save endpoints
are untouched.
Settings-tab buttons assembled in admin.js and the skill-modal buttons in
index.html used two parallel rendering shapes with two binding mechanisms
(per-button addEventListener vs. document delegation). Listeners stacked
on the same DOM whenever the Settings tab re-rendered.
Converge both surfaces on the empty-button + ``data-help-target`` +
document-delegated dispatch pattern:
- ``_renderSettingRow`` emits empty <button> with ``data-help-target`` and
gives the sibling popover a matching ``id``. Per-button listener loop in
``_renderSettings`` removed.
- ``_toggleSettingsHelp`` and ``_closeAllSettingsHelp`` drop their
``.settings-label-col`` fallback branches; ``data-help-target`` reverse
lookup is the only path.
- 12 skill-modal buttons in index.html lose their literal ``?`` text so
every emitter produces empty content.
- ``font-size: 0`` on ``.settings-help-btn`` removed (workaround is no
longer load-bearing now that all emitters are empty).
Also tucks long-form help text in Judge modals behind the same ``?``:
- 8 ``.label-hint`` spans in the Create/Edit Heuristic Rule and Create/Edit
Output Guard Pattern modals (Tool Pattern, Arg Patterns, Confidence,
Pattern Flags) become ``?`` + popover. Short ``.label-hint`` strings
elsewhere in the admin UI are unchanged.
Description rows (the short TLDR under each key) stay inline as before.
The ``?`` popover carries only the long-form ``help`` paragraph and any
``reference_url`` learn-more link.
* feat(rbac): editable builtin role permissions via overlay layer
Adds a ``role_permission_overrides`` table that stores per-(role_id,
permission) grant/revoke deltas, applied on top of the immutable
``roles.permissions`` baseline at permission-load time. Builtin roles
(``builtin-admin/operator/viewer``) become customizable through the
admin Roles UI without losing the "reset to default" guarantee — every
override is auditable and reversible.
Motivating case: ``model.skills.write`` is deliberately default-ungranted
on every role so operators must consciously opt in before a coordinator
session can mutate the skill catalog. Until now there was no UX path to
do that opt-in — the only options were dropping into SQL or running a
fresh migration. The overrides editor closes that gap.
Backend
- Migration 057 + storage methods on both sqlite + postgresql backends
- ``get_user_permissions`` merges baseline ∪ grants − revokes for builtin
rows; custom rows pass through unchanged
- ``GET /v1/api/admin/roles/{id}/effective`` for inspect
- ``PUT /v1/api/admin/roles/{id}/overrides`` for write — admin.roles gated,
audited, validates against ``_VALID_PERMISSIONS``, refuses non-builtin
targets, strips no-op grants/revokes before persisting
- Lockout guard: cannot revoke ``admin.roles`` if doing so would leave
zero users with the permission (returns 409)
- ``coordinator.trust.send`` added to ``_VALID_PERMISSIONS`` — was
seeded into builtin-admin by migration 042 but never registered with
the validator, so the very first round-trip through the editor 400'd
on it. Drift-detection test guards future migrations from recreating
the same gap
Frontend
- Roles tab redesign: chevron + permission-count chip replace the
"..." truncation; expand-on-click drawer groups perms by namespace
with baseline / grant (green +) / revoke (red −) chip variants
- Edit modal opens for builtin rows ("Customize Built-in Role" title);
toggles show baseline-default vs override state; submit diffs against
the rendered toggle universe (not raw baseline) so future taxonomy
drift can't silently strip unknown perms
- "Modified +N/-N" pill on rows with active overrides; "Reset to default"
drawer action clears the override set
- ``_PERMISSION_SECTIONS`` brought up to date with all currently-seeded
perms (admin.coordinator, admin.cluster.inspect, admin.models,
admin.nodes, admin.prompt_policies, conversation.modify,
coordinator.trust.send were missing)
Tests
- 7 storage tests covering set/list/clear/effective + overlay merge into
``get_user_permissions`` for both builtin and custom roles
- 11 endpoint tests covering effective/overrides happy paths, validation,
lockout guard, builtin-only restriction, no-op normalization, list
enrichment
* feat(rbac): enforce workstreams.{create,close} + tools.approve gates
These three permissions were declared in ``_VALID_PERMISSIONS``, seeded
into ``builtin-operator``'s baseline by migration 008/017, surfaced in
the admin Roles UI as toggles, and documented in ``bootstrap.py`` as
the operator role's capabilities — and never enforced anywhere. The
audit that ran out of the overlay PR found zero ``require_permission``
sites for any of them; any authenticated user could create workstreams,
close any workstream, or approve any pending tool regardless of role.
Behaviour change for callers without the perms:
- ``POST /v1/api/workstreams/new`` (node + console proxy variants)
now 403 without ``workstreams.create``
- ``POST /v1/api/workstreams/{ws_id}/close`` (and ``/route/`` proxy)
now 403 without ``workstreams.close``
- ``POST /v1/api/workstreams/{ws_id}/approve`` (and ``/route/`` proxy)
now 403 without ``tools.approve``
The OR-fallback to ``admin.coordinator`` keeps coord sessions spawning
interactive children unblocked without needing operator-style perms.
Service-scoped inter-cluster calls bypass via the existing
``allow_service_bypass`` path on the new ``require_any_permission``
helper. Builtin admin and operator both already carry these perms;
viewer correctly loses workstream create/close/approve (it already
couldn't do those in spirit).
Implementation
- ``require_any_permission`` (core/auth.py) — OR-semantics variant of
``require_permission`` with per-conditional comments documenting the
security policy at the choke point. 403 body names every accepted
perm so operators get an actionable remediation
- ``make_{create,close,approve}_handler`` (core/session_routes.py)
accept ``fallback_permissions: tuple[str, ...]`` — checked only when
``cfg.permission_gate is None`` (interactive case). Coord's
``permission_gate=_require_admin_coordinator`` continues to take
precedence on the coord-config side
- Console-side ``create_workstream`` and ``route_create`` inline the
same OR check before proxying — fail fast on a forbidden request
without burning a cluster round-trip
- ``route_proxy`` adds a verb-scoped gate on ``approve`` and ``close``
only; ``send``/``cancel``/``dequeue``/``command``/``plan`` remain
authenticated-only (pre-existing, out of scope for this audit)
Tests
- New ``TestPermissionGatesOnLifecycle`` (4 tests) in test_server_authz
pinning 403-without-perm + non-403-with-perm at the node lift sites
- New ``TestRouteProxyPermissionGates`` (5 tests) in
test_console_routing_proxy covering 403 paths, OR fallback via
``admin.coordinator``, and that ``send`` remains ungated
- ``_make_jwt`` helpers in test_server_authz, test_close_reason_
persistence, test_server_attachments_on_create updated to embed
operator-shaped perms by default so existing tests continue to
exercise the post-gate logic rather than 403'ing on the new check
Docs
- ``bootstrap.py`` operator role line corrected to list every perm
it actually carries (was missing ``tools.approve`` and
``conversation.modify``)
* fix(rbac): close lockout + escalation gaps in role-overrides editor
Three issues surfaced by /review of the overlay layer and gate uplift —
all in the RBAC/auth surface, treated as zero-days.
**F-1: lockout guard misses the grant-removal path.** PUT-replace
semantics on ``set_role_overrides`` mean an existing grant of
``admin.roles`` (added via override to e.g. builtin-operator) is
silently dropped when the new payload omits it. The previous guard
short-circuited on ``"admin.roles" not in revokes`` and never noticed.
Concrete cluster-bricking scenario: grant admin.roles to operator via
override, unassign builtin-admin, click "Reset to default" on operator
→ all users lose admin.roles, recoverable only via SQL.
The rewritten guard simulates the post-PUT effective set on the target
role directly: if ``(baseline | new_grants) - new_revokes`` lacks
admin.roles AND nobody holds it via another role, refuse the change.
The "via another role" question is answered by one bulk query rather
than the prior O(users × roles) round-trip loop.
**F-3: lockout check blocked the event loop on moderate deployments.**
The prior check called ``storage.list_user_roles`` per user and
``storage.effective_role_permissions`` per (user, role) pair —
synchronous SQL inside an async handler. 200 users × 5 roles = 1000
connection cycles long enough to trip reverse-proxy timeouts on a
permission revoke.
Replaced with ``storage.users_with_permission(perm, *,
exclude_role_id)`` — one join over ``user_roles ⋈ roles`` plus one IN
fetch on overrides for the builtin role ids in the result, folded
in-process. Two queries total, independent of cluster size. The whole
check now runs under ``asyncio.to_thread`` so even the bulk read
doesn't stall the loop.
**F-2 reframed: admin_assign_role's subset check ignored the overlay.**
The check at lines 6321-6328 reads ``target_role.get("permissions",
"")`` (baseline column) when computing the perms it requires the
caller to hold. After this branch, an admin.roles holder can grant
e.g. ``model.skills.write`` to builtin-operator via override; an
admin.users holder (who happens to NOT hold that perm) could then
assign operator to a new user, silently escalating the assignee. The
existing two-person-rule by perm split (admin.roles for catalog edits,
admin.users for assignments) only holds if the assignment-time check
considers the overlay. Switched ``target_perms`` to
``storage.effective_role_permissions(role_id)["effective"]``.
Note: this PR retains the existing model where admin.roles is the
catalog-edit superuser (admin_create_role, admin_update_role, and now
admin_role_overrides all skip the caller-holds-grants check). The
two-person rule against escalation lives at the assignment gate, which
this fix reinforces.
**F-7: delete_role left orphaned override rows.** No FK on
``role_permission_overrides.role_id`` (migration 057 omitted FKs to
match the rest of the governance schema). Added explicit cleanup in
both sqlite + postgresql ``delete_role`` implementations so a
re-seeded role_id (deterministic for builtins on schema reseed) can't
silently inherit stale overrides from the prior occupant.
Tests
- storage: ``test_users_with_permission_bulk`` exercises the new bulk
helper including ``exclude_role_id`` and overlay folding
- storage: ``test_delete_role_cleans_up_overrides`` pins the F-7 fix
- endpoint: ``test_overrides_lockout_guard_blocks_grant_removal`` is
the F-1 reproduction — operator-overlay grants admin.roles, builtin-
admin has it removed, attempting to reset operator's overrides 409s
- endpoint: ``test_assign_role_blocks_escalation_via_overlay_grant``
pins the F-2 reframed fix — overlay-poisoned operator can't be
assigned by a caller missing the overlay perms
* refactor(rbac): cleanup batch from /review (#584)
Five non-security findings folded into one commit so the security
batch stays focused. All consistent with the existing intent of
``feat/builtin-role-overrides``.
**F-4: presence check on ``_effectivePerms``.** ``governance.js`` was
guarding on ``Array.isArray(role.effective) && role.effective.length > 0``,
falling through to splitting ``role.permissions`` (the baseline) when
the array was empty. For a builtin role whose overrides legitimately
revoke every baseline perm, that path silently rendered the baseline
chips with no override indicators — the inspector lied about what the
role can do. ``_enrich_role`` always sets ``effective: []``, so
presence is the right sentinel.
**F-5: JS-side drift detector.** Commit 1 added a Python-side test
asserting ``_VALID_PERMISSIONS`` covers every baseline perm; the
mirror invariant on the frontend went uncaught. A new perm added to
``_VALID_PERMISSIONS`` without a matching entry in
``_PERMISSION_SECTIONS`` becomes silently un-customizable through the
admin UI (the only documented grant/revoke path). Test parses the
JS const out via regex and asserts set-equality both directions —
detects "missing in UI" and "extra in UI" so the toggle catalog and
validator can't fork.
**F-6: bulk enrich for ``admin_list_roles``.** Was ``1 +
2*builtin_count + 1*custom_count`` SELECTs per admin-tab open;
collapsed to one ``IN``-filtered query via new
``storage.effective_role_permissions_bulk(role_ids)``. Implemented
on both sqlite + postgresql backends following the existing
``effective_role_permissions`` shape.
**F-8: rename ``fallback_permissions`` → ``accepted_permissions``.**
The lift body uses ``if cfg.permission_gate / elif accepted_permissions``
— mutually exclusive — so when ``permission_gate`` is None this IS
the primary gate, not a fallback to anything. The "fallback" name
suggested a tier-2-after-tier-1 semantic that didn't exist. Renamed
across ``make_{approve,close,create}_handler`` factories, the three
call sites in ``turnstone/server.py``, and the docstrings.
**F-9: positive lift-level tests for ``admin.coordinator``-only.**
``TestPermissionGatesOnLifecycle`` previously had a single positive
test for ``workstreams.create`` alone, plus negative-403 tests for
each verb without perms. The OR-fallback to ``admin.coordinator``
(which keeps coord sessions spawning interactive children unblocked)
had no positive coverage at the lift code path — only at the proxy,
which exercises a different verb-dict gate. Added three tests
(create / close / approve) that pass ``admin.coordinator`` alone and
assert non-403, so a future tightening of the accepted_permissions
tuple can't silently regress coord-driven child workstreams.
Out of scope: nit perf-4 (event-delegation refactor on
``_renderGovRoles``). ``setSafeHtml`` rebuild is the existing
pattern across every admin tab; rewriting one tab's render path on
this branch would be drive-by inconsistent with the surrounding
codebase. Filed as a separate concern if the Roles tab grows past
the scale where it bites.
* fix(rbac-ui): aria-expanded + row-click on Roles drawer (#585)
Two Copilot review findings on governance.js:
- Expand button was missing aria-expanded — screen readers couldn't
announce drawer state. Now reflects the row's expanded flag.
- Comment said "row + chevron both work" but only the chevron was
wired. Added data-expand-role to the row element too so the
existing handler loop (querySelectorAll on the attribute) picks up
both — clicking anywhere in the role row toggles the drawer.
Edit/Delete handlers already stopPropagation so they aren't
triggered by the row-level click.
* fix(migrations): rebase role_permission_overrides to 058
PR #560 mitigation #1 landed 057_output_assessments_llm_judge.py on
main in parallel; my migration claimed the same number, forking
alembic's head and breaking postgres. Renumbered to 058 and
re-pointed down_revision at 057 so the chain stays linear.
No behaviour change — same DDL. Full sweep clean (6730 passed).
* fix(migrations): update 058 revision strings to match filename
Previous commit (ea86aefc) renamed 057_role_permission_overrides.py to
058_* but the in-file revision = "057" / down_revision = "056"
strings stayed — leftover from when the file shipped as 057. Tests
pass because alembic walks the chain by revision string, and the
strings now correctly read revision = "058" / down_revision = "057"
to make the chain linear with main's 057_output_assessments_llm_judge.
Caught locally before re-running CI; my prior `git mv` + content edit
landed as a staged rename + unstaged modification on the previous
push.
`float("4.20") == 4.2`, so the old version-sort routed `grok-4.20`
under `grok-4.3` despite 4.20 being the newer dated-snapshot line.
Parsing each component as an int via `_version_tuple` makes
`(4, 20) > (4, 3)` as intended.
Applied symmetrically to the openai branch — same shape, same latent
bug against a future `gpt-5.10` vs `gpt-5.2` collision.
Locked in by four tests in `test_provider_xai.py::TestSelectBestModel`.
Spotted by Copilot review on #586.
Adds xAI as a first-class commercial provider through the officially-
documented server-to-server API-key path against https://api.x.ai/v1.
XAIProvider is a thin subclass of OpenAIResponsesProvider; xAI's
Responses surface is OpenAI-shaped, so the only override needed is
_build_kwargs, which merges <tool>_call_output strings into include[]
so xAI's server-side tool outputs (hidden by default) become visible.
GROK_CAPABILITIES covers the five documented chat models; aliases
like grok-4.3-latest resolve via the existing longest-prefix lookup.
Two narrow base-class generalisations earn their keep beyond Grok:
- ModelCapabilities.server_side_tools: tuple[str, ...] drives the
Responses-surface tool injection (previously hardcoded to
web_search). resolve_server_side_tools folds in the legacy
supports_web_search boolean for backward compat.
- extra_headers: dict[str, str] | None threaded through
LLMProvider.create_streaming / create_completion so callers can
pass x-grok-conv-id: <ws_id> for prompt-cache hit-rate. Session-
side population is a follow-up; the plumbing lands here.
Out of scope:
- OAuth (SuperGrok / X-Premium+) — not officially documented.
- Chat Completions surface — deprecated on xAI's comparison page.
- Image / voice / video models.
- argparse --provider xai in cli.py / server.py — Google isn't
there either; both providers configure via config.toml.
Closes#583.
Validated each of the 7 substantive Copilot findings + 5 CodeQL
findings via source spike; applied 7 (CP2-CP7, C3-C5), refuted 2
(CP1, C1+C2) with citation.
* CP2 (session.py:3551) — pre-truncation budget was reused per
output, allowing N parallel tool results to each claim the full
remaining context budget and collectively overflow. Maintain a
running budget that shrinks as each output is sized.
* CP3 (ratelimit.py) — TokenBucket.consume() mutated tokens /
last_refill without a lock; ChatSession._batch_evaluate_outputs
invokes it concurrently from up to 4 worker threads, so the rate
limiter's stated 60-call/min cap was best-effort. Added an
internal threading.Lock that protects consume() and retry_after.
RateLimiter's outer lock still holds for the bucket-dict it owns;
the new lock just makes the class safe-by-default for direct
consumers.
* CP4 (session_ui_base.py:1642) — docstring claimed "llm" rows
were only persisted on success; the session actually persists a
failure row too (with reasoning=error_reason) for audit. Match
the docstring to behavior.
* CP5 (migration 057) — header said "Revises: 055" but
down_revision was "056". Fix header.
* CP6 (_protocol.py:1761) — :func:`intent_verdicts` is a table,
not a callable. Plain reference.
* CP7 (test_output_guard_judge.py:271) — test_client_created_once
was self-contradictory: header comment claimed caching was being
verified, body asserted the patched fake was called once per
evaluate (no caching). The next test
(test_real_lazy_init_caches_real_client) covers actual caching;
drop the misleading one.
* C3-C5 (_extract_json) — three empty except-pass blocks now carry
a justifying one-liner explaining each strategy's expected failure
mode and what falls through.
REFUTED:
* CP1 (output_guard_judge.py:453 catch of TimeoutError vs
concurrent.futures.TimeoutError) — in Python 3.11+ (our minimum,
per pyproject.toml requires-python = ">=3.11") the two are the
same class. Verified at runtime: ``cf.TimeoutError is
TimeoutError`` is True; MRO is (TimeoutError, OSError, Exception,
BaseException, object). The polling loop catches the right
exception.
* C1/C2 (Protocol method bodies and test stubs using ``...``) —
`...` is the idiomatic Python pattern for Protocol method bodies
and one-line stub functions; the surrounding code in
test_model_registry.py uses ``...`` consistently across all stub
methods. CodeQL is flagging a single instance while ignoring
identical patterns nearby.
Adds a second, LLM-driven stage to the output guard so domain-camouflaged
prompt-injection payloads that the regex stage misses (arXiv:2605.22001 —
Llama 3.1 8B evades the existing regex set on ~90% of camouflaged
prompts) get caught before the tool output lands in the assistant's
context.
## Surface
* New `OutputGuardJudge` in `turnstone/core/output_guard_judge.py` —
synchronous, single-shot LLM call. Inlines the alias-resolution +
client-config + JSON-parsing helpers (copied verbatim from
`IntentJudge` at `judge.py:917-969` / `1604-1659`) rather than going
through a shared module — when `IntentJudge` lifts its own helpers,
both copies move together.
* JSON-in-content verdict with a 3-strategy parser (direct / markdown
fence / balanced braces). `IntentJudge` ships a 4th regex-field
fallback; OutputGuardJudge deliberately doesn't, because strategy-4
hits on broken LLM output can extract a "verdict" from the model's
reasoning quote that lands in storage looking identical to a clean
strategy-1 result. Failure of all three returns
`error="unparseable_verdict"` and the heuristic stage stands.
* `OutputJudgeVerdict` is a frozen dataclass with:
`risk_level` (none/low/medium/high — normalises `critical`→`high`
and `info[rmational]`→`low` for IntentJudge-echo safety),
`flags: tuple[str, ...]`, `reasoning`, `confidence: float`
(0.0-1.0, parsed + clamped from the LLM's self-report;
pass-through to audit, no threshold gating), `judge_model`,
`latency_ms`, `error`.
* Real wall-clock timeout via `ThreadPoolExecutor.shutdown(wait=False,
cancel_futures=True)` on the timeout/cancel path — `with ... as ex:`
would block return until the worker drained. 1s `cancel_event`
poll mirrors `IntentJudge._run_judge` at `judge.py:1117-1118`.
* HTTP client lazy-init + reuse for the judge instance's lifetime.
Session-side model swap drops the entire judge, dropping the client
with it.
* Untrusted tool output wrapped in per-call random-nonced
`<tool_output_NONCE>...</tool_output_NONCE>` fence. Closing-tag
substrings in the raw text are case-insensitively backslash-escaped
first (`</tool_output` → `<\/tool_output`) so an attacker can't
break out even if they guess the nonce. System prompt classifies
the fenced region as UNTRUSTED DATA so directives inside are
evaluated as content, not obeyed.
* Judge user prompt carries the heuristic verdict (risk + flags +
annotations), the tool description (looked up from the session's
tools registry), and the tool args (truncated to 500 chars, also
classified UNTRUSTED in the system prompt since they may be
caller-supplied). Lets the judge defer to the regex on credential
leaks and focus on injection signals the regex set misses; also
enables output-vs-request plausibility reasoning.
## Session integration
* `_evaluate_output(call_id, output, func_name, *, tool_args="")` —
heuristic always runs; LLM stage runs when `judge.output_guard_llm`
is enabled. When the LLM produces a usable verdict and the
heuristic didn't detect credentials, the LLM verdict is acted on;
otherwise the heuristic stands.
* Credential redaction is a regex-only signal. When `heuristic.
sanitized` is non-None, the heuristic owns the acted assessment
regardless of what the LLM said — an LLM asked about prompt-
injection can correctly label a credential-bearing output as
"none" risk for injection, but the secret still needs redaction.
* `_batch_evaluate_outputs` runs the per-tool guard concurrently
(4-worker pool) when LLM is enabled and there are ≥2 string
outputs — collapses N×LLM-latency to ⌈N/4⌉×latency on the common
5-20 tool-calls-per-turn turn.
* Per-session `TokenBucket(rate=1.0, burst=60)` caps adversarial
LLM-fan-out cost at 60 calls/min/session.
* Pre-truncation: the per-tool loop truncates output before the
judge sees it, so the judge evaluates exactly what enters the
assistant's context (no wasted tokens on text that won't land).
* Both heuristic and LLM tier rows persisted to `output_assessments`
when the LLM ran (audit completeness); heuristic-only rows skip
when matched-clean to keep the table focused.
## Storage
Migration 057 extends `output_assessments` with five LLM-tier
columns: `tier` (`heuristic` / `llm`, backfilled to `heuristic`),
`reasoning`, `judge_model`, `latency_ms`, `confidence`. Tie-break
on `(created DESC, tier='llm' first)` so downstream consumers see
the acted verdict first when the two rows tie at second resolution.
`StorageBackend.record_output_assessment` + sqlite/pg implementations
+ `SessionUIBase.record_output_assessment` + `SessionUI` protocol +
the test stub overrides (cli, eval, 9 test files) all take the new
LLM-tier kwargs.
## Config surface
Three new judge.* settings in `settings_registry`:
* `judge.output_guard_llm` (bool, default False) — capability gate.
Default off; operators opt in once a small/fast model is pointed
at `output_guard_model`.
* `judge.output_guard_model` (str, default "") — alias for the LLM
stage. Empty inherits the session model (same fallback shape as
`judge.model`).
* `judge.output_guard_llm_timeout` (float, default 30.0, min 1.0) —
wall-clock budget per call.
Both `server.py` and `console/session_factory.py` wire these into
the `JudgeConfig` they hand to `ChatSession`.
## Notes
* No backwards-compatibility shims — the LLM stage is purely additive.
* No reasoning/threshold gating on confidence; it rides as an
audit-only signal per maintainer direction. Surface it in the
`on_output_warning` dict so live UI / cluster broadcast can sort
flagged outputs by judge certainty.
* Tests: 392 lines of judge-only coverage (`test_output_guard_judge.
py`) + 629 lines of session-integration coverage in `test_session.
py`, plus the storage and stub-shape updates.
Two related cleanups landed together because they touch the same surface
(skill-spec uplift PRs #569/#570/#571/#572):
1. Wording: replace "Anthropic spec" / "Anthropic Claude Code skill spec"
with "SKILL.md spec" across admin UI tooltips, code comments, test
docstrings, migration 056's module docstring, and the user-facing
`arguments` description in tools/skills.json. Renames a parser test
`test_anthropic_tags` -> `test_nested_metadata_tags` and consolidates
a parse-API test of the same shape; fixture author renamed
`Anthropic` -> `Acme` to keep the fixture neutral. Legitimate
provider/SDK/API references (provider name, api.anthropic.com,
`_anthropic.py`, capability comments) are intentionally untouched.
2. Admin UX: in the Create + Edit Skill modals, six fields per modal
(Compatibility, Paths, Hide-from-skill-picker, Arguments, Argument
hint, Activation) had long uppercase label-hint spans crammed into
the visible label. Migrated each to the existing
`.settings-help-btn` + `.settings-help-popover` pattern already used
in the Settings tab — short label + inline `?` button that opens a
styled popover with proper `<code>` formatting for technical tokens.
Pattern reuse required two small generalisations in admin.js:
* `_toggleSettingsHelp` now looks up the popover via a new
`data-help-target="<id>"` attribute first, falling back to the
settings-tab `.settings-label-col` ancestor lookup.
* `_closeAllSettingsHelp` mirrors the same dual-path lookup when
resetting `aria-expanded`, so modal buttons don't get stuck on
`aria-expanded="true"` after another popover opens.
* Added a document-delegated click handler that fires only for
buttons with `data-help-target`; existing per-button binding
in the settings-tab render path is unchanged.
CSS: `.settings-help-btn` now paints its `?` via `::after` with the
button's own `font-size: 0`, so prettier-introduced whitespace
inside the new HTML buttons can't off-center the glyph. The same
rule applies to existing admin.js-generated buttons (text content
hidden, pseudo identical). Small additions for
`.settings-help-popover code` / `strong` styling so technical
tokens render with the same monospace pill treatment used elsewhere
in skill UI.
Known follow-ups (intentionally NOT in this PR):
* Migrate the settings-tab `_renderSettingRow` button assembly to the
empty-`<button>` + `data-help-target` form so the per-button
addEventListener loop can be dropped in favour of pure document
delegation, and the `font-size: 0` rule stops being a workaround for
two markup styles.
* The 12 new popover blocks are duplicated verbatim between the
Create and Edit modals (same as the rest of the create/edit modal
pair). A small renderer that emits popovers from a shared data
object would eliminate the drift risk but is unrelated cleanup.
* fix(coord): strip intent-judge verdicts from inspect_workstream output
Coordinator LLMs repeatedly misread `user_decision="policy"` (the label
meaning "auto-approved by an admin policy allow rule") as "blocked,
waiting for policy review" — combined with `recommendation="review"`
(the heuristic judge's risk class, not a workflow state) the verdict
fields read end-to-end as "stuck on policy review" and produced
incorrect cancel-and-respawn reasoning against healthy children.
The blocking signal already lives on `state` (`"attention"`) and the
`live.pending_approval` block, both still in the result. Verdict
history remains queryable through admin / audit surfaces — only the
LLM-facing inspect surface drops them.
Also drops `verdict_count` / `verdicts_by_risk` from the tier-3
skeleton fallback, deletes the now-dead `_serialize_verdicts` helper,
and clears the now-stale `"verdicts": []` keys from 10 fixture sites
that fed `_format_inspect_tiered` test cases.
* fix(coord): correct comment pointer — inline comment, not docstring
Implements the Anthropic Claude Code skill spec's placeholder
substitution end to end. The renderer in ``_substitute_skill_args``
handles every spec form except ``\${CLAUDE_SKILL_DIR}`` (deferred):
* ``\$ARGUMENTS`` — full args string as the user/model typed it
* ``\$ARGUMENTS[N]`` / ``\$N`` — Nth positional arg, ``shlex.split``-parsed
* ``\$<name>`` — named arg from the SKILL.md ``arguments:`` list
* ``\${CLAUDE_SESSION_ID}`` / ``\${CLAUDE_EFFORT}`` — session state
Substitution is single-pass (one combined regex, one ``re.sub``).
Append rule: when args are passed but the body has no bare
``\$ARGUMENTS``, append ``ARGUMENTS: …`` at the end.
## Surface
* Parser: ``arguments:`` (list/space-delim) + ``argument-hint:`` (str)
extracted into ``ParsedSkill``.
* Install: persists both to the pre-allocated columns from migration
056 (PR #574). Install path clamps ``argument_hint`` to 128 chars
to match the admin-create cap (untrusted upstream source).
* Admin: ``CreateSkillRequest`` / ``UpdateSkillRequest`` accept both
fields; create + edit modals get inputs; parse-preview echoes.
* Renderer: ``_substitute_skill_args`` runs AFTER ``_render_template``
in ``_load_skills`` so user-supplied args containing ``{{var}}`` can't
be re-expanded by the legacy renderer.
* Session: ``_skill_arguments`` plumbed through ``__init__``,
``set_skill``, and ``_save_config`` so a resumed workstream re-renders
with the original arg payload.
* Model tool: ``skills(action='load')`` accepts an ``arguments`` string.
Approval label includes a SHA-256 digest of the args so a once-
approved skill name can't grant cover for a future payload; preview
surfaces the args inline.
## ``/review`` findings (addressed)
* ``\${CLAUDE_EFFORT}`` referenced ``self._reasoning_effort`` — wrong
attribute; the real one is ``self.reasoning_effort``. Always rendered
empty. Fixed.
* Two-pass layering let user args containing ``{{var}}`` re-expand.
Render order reversed.
* ``_skill_arguments`` wasn't in ``_save_config`` — resumed workstreams
silently lost their payload. Added.
* Approval label omitted ``arguments``. Digest + preview added.
* Install path didn't bound ``argument_hint``. Clamped.
* Added ``_skill_arg_names`` decode tests + "load same skill,
different args → re-render" invariant test.
## Copilot review findings (addressed)
* ``skills.json`` tool description was inaccurate about ``shlex``
stripping quotes and "empty string disables substitution". Rewrote
to match actual behaviour.
* Named-argument regex was stricter than parser/storage contract.
``arguments: [issue-number]`` would partial-match ``\$issue-number``
as ``\$issue``, leaving ``-number`` as stray text. Broadened the
regex to ``[A-Za-z_][A-Za-z0-9_]*`` AND added validation at
``_skill_arg_names`` decode time so names not matching the regex
are dropped with a warning.
## Tests
* ``tests/test_substitute_skill_args.py`` — placeholder forms,
single-pass guarantee, append-at-end rule, shell-quoted input,
unbalanced-quote fallback, uppercase + underscore-prefix names
* ``tests/test_skill_parser.py::TestArgumentsAndHint`` — parser
extraction
* ``tests/test_skill_parse_api.py`` — HTTP parse-preview echoes
both fields
* ``tests/test_skill_discovery_api.py::test_install_seeds_arguments_and_argument_hint``
— install round-trip
* ``tests/test_skills_tool.py::test_load_forwards_arguments_to_set_skill``
+ ``test_load_same_skill_different_args_triggers_resub`` —
wire path through prepare → exec → set_skill
* ``tests/test_skills_tool.py::TestSkillArgNames`` — storage decode
helper including the hyphen/dot/leading-digit filter
Two findings from Copilot's review of PR #577:
* ``_extract_bool`` int branch: Copilot flagged that ``bool(raw)``
treats any non-zero int as True, so ``disable-model-invocation: 2``
silently disables model invocation without warning the author about
the typo. Tightened to accept only ``0`` and ``1`` as integer
boolean forms — anything else falls back to *default*. Matches
spec (which mentions only 0/1) and the broader principle that
ambiguous input should not coerce silently.
* ``hidden_from_menu`` admin body parse: Copilot flagged that
``bool(body.get("hidden_from_menu", False))`` treats non-empty
strings via Python truthiness, so a malformed client sending
``"false"`` would flip the flag to ``True`` — opposite to obvious
intent. Extracted a ``_parse_strict_bool`` helper that accepts
only Python ``bool`` or int ``0``/``1`` and returns a 400 on
anything else. Applied at both admin create and admin update
sites; the install path remains untouched because it derives the
flag from the typed ``ParsedSkill.user_invocable`` field (not raw
HTTP body).
## Tests
* ``test_other_ints_fall_back_to_default`` — ``2`` and ``-1`` no
longer silently coerce
* ``test_create_skill_hidden_from_menu_string_rejected`` — string
``"false"`` returns 400
* ``test_create_skill_hidden_from_menu_int_zero_and_one_accepted``
— 0 / 1 accepted, 2 rejected with 400
No behaviour change to the main surface — both fixes close latent
type-coerce hazards a malformed input could have exploited.
The Anthropic Claude Code skill spec defines two invocation-control
axes Turnstone was parsing but not consuming:
* ``disable-model-invocation: true`` — model can't autoload this skill
(only user can invoke by name). Stored on ``ParsedSkill`` and
echoed on the parse-preview UI; no install consumer because
Turnstone hardcodes ``activation="named"`` on source-installs
already. The dataclass docstring spells out the no-op so a future
reader doesn't try to wire a translation that's already implicit.
* ``user-invocable: false`` — skill stays available to the model but
disappears from the user-facing picker. Mapped to
``hidden_from_menu=true`` on ``prompt_templates`` (column
pre-allocated by PR #574); consumed by ``list_skills_summary``
(both the standalone-server and console-server impls).
## Surface
* Parser: new ``_extract_bool`` helper accepts every YAML 1.1
boolean spelling (true/false/yes/no/on/off/1/0) plus their quoted
variants — caught by ``/review`` as a real gap, since YAML's
``safe_load`` returns ``int`` for unquoted ``1``/``0`` and ``str``
for the YAML 1.1 spellings when quoted.
* Install handler: derives ``hidden_from_menu`` from
``parsed.user_invocable`` on the source-install path.
* Admin: ``CreateSkillRequest`` / ``UpdateSkillRequest`` accept
``hidden_from_menu``; both modals get a checkbox; the parse-preview
auto-fill flips it when the source SKILL.md sets
``user-invocable: false``.
* Runtime config: ``hidden_from_menu`` joined
``SKILL_RUNTIME_CONFIG_FIELDS`` so admin can override on installed
(readonly) skills — same precedent as ``model`` / ``effort``.
## list_skills_summary shared helper
Two identical implementations of ``list_skills_summary`` had
accreted in ``turnstone/server.py`` and ``turnstone/console/server.py``.
Both needed the new ``hidden_from_menu`` filter, so extracted the
shared body to ``turnstone/core/web_helpers.skill_summary_rows``.
Future spec-uplift fields (e.g. #572's ``argument_hint`` for
autocomplete) only touch one place now.
## Tests
* ``TestInvocationControl`` — bool / quoted / YAML 1.1 / int variants
across both fields
* ``test_install_user_invocable_false_sets_hidden_from_menu`` +
default-unhidden case
* ``test_list_skills_summary_excludes_hidden_from_menu`` — picker
filter, admin tab unaffected
* ``test_update_skill_readonly_hidden_from_menu_allowed`` — admin
can hide/unhide installed skills via PUT (pins the runtime-config
membership invariant)
* Existing parse-API fixture extended with both new fields plus
default-case assertions
The Anthropic Claude Code skill spec defines three frontmatter fields
the parser was previously dropping; this PR wires them through to the
existing storage shape so the SKILL.md author's intent survives the
import.
* ``when_to_use`` — concatenated into ``description`` at parse time
with a ``\n\nWhen to use: `` separator. Kept as its own field on
``ParsedSkill`` so the admin parse-preview UI can surface it
separately.
* ``model`` — passed through to ``create_prompt_template(model=...)``
on the source-install path, seeding the existing
``prompt_templates.model`` column.
* ``effort`` — same shape, translates to the existing
``reasoning_effort`` column at the install handler boundary.
Re-install short-circuits at the source_url dedup, so admin overrides
to either column survive an upstream re-install — covered by a new
``test_reinstall_preserves_admin_model_override`` test that pins the
load-bearing invariant.
## Description length cap
``_MAX_DESCRIPTION_LEN`` exported as ``MAX_SKILL_DESCRIPTION_LEN``
(public name) and raised from 1024 to 1536 to match the spec's
combined ``description`` + ``when_to_use`` listing budget. All five
write surfaces now import the same constant rather than each carrying
their own magic number:
* ``skill_parser.MAX_SKILL_DESCRIPTION_LEN`` — parse-time cap
* ``console_schemas.CreateSkillRequest.description`` — Pydantic
* ``console_schemas.UpdateSkillRequest.description`` — Pydantic
* ``console/server.admin_create_skill`` — handler slice
* ``console/server.admin_update_skill`` — handler slice
* ``core/session._exec_skills_create`` — coordinator tool slice
* ``core/session._exec_skills_update`` — coordinator tool slice
The coordinator sites (last two) were the bug ``/review`` caught:
they still capped at 1024 after the rest of the surface bumped to
1536, so a model-issued ``skills(action='create')`` with a 1025-1536
char description would silently truncate. Sharing the constant
closes that desync.
## when_to_use truncation guard
The ``when_to_use`` concat reserves room for the separator + at
least one character of the appended value; below that budget, the
addition is dropped entirely. Previously the naive concat could
truncate mid-separator and leave the description ending in a
dangling ``\n\nWhen ``.
## Tests
* ``TestWhenToUse`` — concat semantics, no-description fallback,
1536 truncation
* ``TestModelAndEffort`` — extraction + defaults
* ``test_install_seeds_model_and_effort_from_frontmatter`` — install
path persists both columns
* ``test_install_no_model_or_effort_leaves_columns_empty`` — bare
SKILL.md doesn't invent values
* ``test_reinstall_preserves_admin_model_override`` — admin edits
survive an upstream re-install (dedup invariant)
* ``test_parses_full_frontmatter`` / ``test_parses_minimal_frontmatter``
extended with the new field assertions
Three findings from Copilot's review of PR #574:
* `update_prompt_template` (both backends) coerces every other
INTEGER-as-bool field (`is_default`, `auto_approve`, `enabled`) but
not `hidden_from_menu`. Without coercion, a caller updating with
``hidden_from_menu=True`` writes a Python bool to a SQLAlchemy
Integer column, which is driver-dependent on PostgreSQL and a
consistency hazard. Coerce to int alongside the existing trio.
Added a focused round-trip test that updates with ``True`` /
``False`` and asserts the read-back bool transitions.
* `_canonicalize_skill_string_list` docstring describes its own
``None``-collapses-to-``"[]"`` rule but doesn't mention that
`admin_update_skill` intercepts ``None`` before the helper is
called. Added a note documenting the layered contract: the helper
defines normalization (create semantics), the update endpoint
layers no-op semantics on top.
* HTML label hints used Markdown-style ``paths:`` backticks inside
plain HTML, which render as literal backticks in the browser.
Replaced with `<code>paths:</code>` on both the create and edit
modal Paths inputs.
No behaviour change to PR #574's main surface — the bool coercion
addresses a latent bug a future consumer would have hit; the
docstring + HTML fix are purely cosmetic.
Implements PR1 of issue #569 — parser + storage + admin UI for the
Anthropic Claude Code skill spec `paths:` SKILL.md frontmatter field
(glob patterns gating model-initiated autoload). The autoload filter
that consumes `paths` is deferred to a follow-up PR pending the
workstream-CWD design discussion.
Migration 056 bundles three additional columns whose consumer PRs are
filed but not yet implemented:
* `hidden_from_menu` (boolean) — backs the spec's `user-invocable:
false` (issue #571).
* `arguments` (JSON list) — backs spec `arguments:` named arg slots
(issue #572).
* `argument_hint` (string) — autocomplete display string (issue #572).
The deferred columns surface in `SkillInfo` (response) so consumers can
read them, but are deliberately absent from `CreateSkillRequest` and
`UpdateSkillRequest` — the create/update handlers don't yet read them
and advertising a writable field the handler would silently ignore
would be an OpenAPI lie.
Surface
- Parser: `ParsedSkill.paths` populated from frontmatter; accepts the
spec's YAML-list-or-CSV-string shape via the existing
`_extract_list` machinery.
- Storage: 4 new columns on `prompt_templates`; `SKILL_MUTABLE`
extended; `_row_to_dict` calls extended to cast the new bool;
protocol + SQLite + PostgreSQL `create_prompt_template` signatures
threaded.
- HTTP: admin create/update/install/parse handlers plumb `paths`
through. Pydantic schemas extended accordingly.
- Admin UI: `skill-paths` and `etm-paths` inputs on the create + edit
modals; field map and read/write helpers wired across paste-parse,
reset, create-send, edit-load, edit-send, and the readonly-disable
list.
Notable
- `_canonicalize_skill_string_list` collapses the list-or-CSV-or-JSON-
string normalization shared between admin_create_skill and
admin_update_skill. Treats `None` as no-value so a body containing
`{"paths": null}` doesn't CSV-split through `str(None)` and store
the literal `["None"]`. Will back `arguments` once #572 wires its
consumer.
Tests
- Parser: TestPaths covers YAML list, CSV string, empty, full-
frontmatter integration (tests/test_skill_parser.py).
- Storage: round-trip suite covers create + read + update for each
of the four new columns on both backends
(tests/test_storage_skill_spec_uplift.py).
- Helper: focused unit tests for the canonicalizer including the
regression-net case for the null-corruption bug
(tests/test_canonicalize_skill_string_list.py).
- HTTP boundary: extended test_parses_full_frontmatter +
test_parses_minimal_frontmatter to assert `paths` survives the
admin parse endpoint.
* fix(output_guard): harden against domain-camouflaged injection (#560)
Three layered mitigations against the camouflage attack class described in
arXiv:2605.22001 (Pai, May 2026), which demonstrates 90.3% evasion on Llama
3.1 8B and 44.4% on Gemini 2.0 Flash against pattern-based detectors:
- Sub-agent synthesis is now scanned by output_guard at the sub-agent
boundary in _run_agent, in addition to the existing scan at the parent's
tool-result loop. Covers all four return paths (clean exit, truncation,
context-limit recovery, turn-limit forced synthesis), closing the
cross-workstream summary laundering surface.
- Adds pair-of-signals camouflage detection: imperative recommendation
phrase combined with either an authority frame ("consistent with our
risk framework") or a caps action verb (SELL/BUY/TRANSFER/...). New
flag camouflaged_injection at medium risk; deliberately partial — the
paper's augmented-detector approach recovers only ~10% on Llama-class
models, so this is duct-tape pending a semantic-evaluator follow-up.
- Bumps output_guard's wall-clock budget default from 5s to 30s and
exposes it as judge.output_guard_budget_seconds in ConfigStore, so the
expanded regex set has headroom on large tool outputs.
* Fix test_budget_kwarg_is_honored to exercise deadline logic path
The test previously passed an empty string which short-circuited
evaluate_output() before budget_seconds was used. Now uses a non-empty
input and monkeypatches time.monotonic() to deterministically verify
the deadline path is exercised.
Single-$ inline math is too ambiguous in conversational text: currency
amounts ("$5 and $10 each"), shell variables ("$HOME and $PATH"), and
shell prompts all produced false-positive KaTeX spans because the
regex matched any non-$/non-newline span between two dollar signs.
Inline math now requires the unambiguous \(...\) form, which is what
GPT-5 / o-series / Claude with reasoning effort emit by default anyway.
Display math ($$...$$ and \[...\]) is unchanged — the doubled
delimiter has enough mass that ambiguity is not a practical problem.
Three former positive tests are inverted into regression guards so a
future regex change can't quietly resurrect the bug, and new tests
name the currency and env-var cases explicitly. The web env prompt is
updated to advertise \(...\) and to tell the model why $...$ is gone.
5 follow-up comments from Copilot, all valid:
1. **CRITICAL — snap_seq race with split writer** (concurrency, 001).
Round-1's fix lifted snapshot capture into
register_listener_with_replay under nested locks, but the
WRITER side (on_content_token / on_reasoning_token) still
released _ws_lock before calling _enqueue (which bumps
_event_id under _listeners_lock). A reader could
interleave between writer's release and writer's _enqueue:
capture inflight WITH the new text, read STALE _event_id,
return snap_seq < new_event_id. The new event's live emit
then has _seq > snap_seq, slips past the dedup filter, and
double-renders text the snapshot already contained.
Fix: move self._enqueue(...) INSIDE the with self._ws_lock:
block in both token writers. The inflight mutation and the
_event_id advancement are now atomic against any snapshot
reader. Lock order _ws_lock (outer) → _listeners_lock
(inner via _enqueue) matches the snapshot helpers, so no
deadlock. Fan-out's put_nowait calls happen under
_ws_lock for token writers — microsecond cost per listener,
acceptable for the correctness guarantee.
2. **NIT — stale comment ref to buffered[-1]._event_id** (docs, 002).
The comment referenced a local var (buffered) that lives in
register_listener_with_replay, not in the events handler.
Reworded to describe the cutoff in terms of the last replayed
event id and the atomic-against-writers registration.
3. **MODERATE — 401 branch leaves reconnect loop** (bug, 003).
The coord's onerror schedules a 5 s CLOSED-state recovery timer
unconditionally. In the 401-expired-session branch we close
evtSource and showLogin — but the timer still fires 5 s later,
observes !evtSource, and calls scheduleReconnect(),
which opens a new EventSource that 401s again → infinite
reconnect loop while the login overlay is up. Fix: cancel
reconnectTimer in the 401 branch.
4. **MODERATE — race test was vacuous** (test_coverage, 004).
The previous regression test drained the listener queue after
register_listener_with_replay returned, but the helper
doesn't backfill buffered events into the queue, so the loop
was almost always a no-op and the assertion never executed.
Rewrote with a monkey-patched _enqueue that sleeps 50 ms
before bumping _event_id — widens the race window
deterministically. Verified: the test FAILS on pre-fix code
(snap.content has marker but snap.seq=0 < final_event_id=1)
and PASSES on post-fix code (writer holds _ws_lock through
_enqueue, so the reader blocks until writer fully done).
Also pinned the no-backfill contract so a future change adding
listener-queue backfill remembers to keep snap_seq the
high-water mark.
5. **NIT — except Exception too broad in test** (best_practices, 005).
Tightened except Exception: to except queue.Empty: so
unexpected exceptions aren't silently swallowed in the drain
loop.
Tests:
- 86 tests in test_sse_reconnect_replay.py + test_session_ui_base.py
pass (existing 84 + 2 new race regressions).
- Full non-live suite: 6347 passed, 15 skipped, no regressions.
- Ruff + mypy clean on changed .py files; JS parses.
Four issues raised on the merged PR #542, evaluated and fixed:
1. **Truncated-path snap_seq bug (Copilot low-confidence, VALID).**
make_events_handler's truncated branch set snap_seq = 0,
disabling the live-drain _seq <= snap_seq dedup filter. Any
token writer racing between register_listener_with_replay
returning and the live drain's first read would land in BOTH the
listener queue AND the captured snapshot text (the snapshot is
emitted via in_progress_snapshot as the recovery floor), so
the client double-renders. Fix lifts the snapshot capture INTO
register_listener_with_replay under the same nested-lock
acquire as the listener registration + buffer slice + counter
read, so the returned snapshot["seq"] is the exact
high-water mark the snapshot text corresponds to. Handler now
uses snapshot["seq"] as snap_seq on truncated, dropping
any token event with _seq <= snap_seq from the live emit.
2. **Lock-held string join in truncated path (Copilot, VALID).**
"".join(ui_base._ws_inflight_content) ran inside the
with ui_base._ws_lock: block, holding the lock for the
duration of the join and blocking on-token writers. Fix (folded
into #1's refactor): the new register_listener_with_replay
copies the inflight lists under lock and joins outside, matching
the existing pattern in
register_listener_with_in_progress_snapshot.
3. **_strip_js_comments docstring misclaim (Copilot, VALID).**
Docstring claimed the helper preserves "string/regex literals"
but the implementation only tracks string delimiters. Fix:
docstring updated to call out the regex-literal limitation
explicitly + note that current callers don't scan regions
containing regex literals. Extending the tracker is left for
a future caller that needs it.
4. **Coord scheduleReconnect dead-code regression (Copilot, VALID).**
After the PR-D refactor, scheduleReconnect() had no remaining
call sites — which meant reconnectAttempts never incremented,
wasReconnecting was always false, AND there was no
fallback when the browser transitioned the source to CLOSED
(hard 4xx after retries, intermediary tearing the connection
down with prejudice, etc.). The first failure mode silently
broke the post-gap replace-mode refresh of children / tasks /
wait indicator / live-badge cache; the second left the coord
permanently disconnected on non-transient failures. Fix:
- Introduce disconnectedSinceLastOpen flag set in onerror,
cleared in onopen. wasReconnecting reads it (with the
legacy reconnectAttempts > 0 fallback for the
scheduleReconnect-driven case), so the post-gap refresh fires
after every reconnect including the common native-reconnect
path.
- Re-introduce CLOSED-state recovery: onerror schedules a 5 s
delayed check via reconnectTimer; if the source is still
CLOSED at that point, call scheduleReconnect(), which
opens a new EventSource (threading the saved
lastEventId via the URL query param so replay still works
across the manual reconnect). Cancel/replace successive
timers so onerror floods don't pile up multiple checks for
the same source.
Tests:
- New test_truncated_path_snapshot_captures_real_snap_seq pins
the snap_seq fix at the helper boundary.
- New test_truncated_path_filters_already_in_snapshot_tokens
pins the end-to-end dedup invariant — would have caught the
double-render under the old code.
- Existing test_sse_reconnect_replay.py call sites updated for
the new 6-tuple return of register_listener_with_replay.
- All 86 tests in those two files pass; full non-live suite (6347
tests) passes; ruff + mypy clean on changed files; both JS files
parse-check.
Four /review findings collapsed to one code chokepoint + two
documentation fixes:
1. find's `kind` arg now validated against ``SkillKind`` (matching
create / update's existing pattern at session.py:8298 / :8512).
Closes two failure modes that shared the same root:
- typos (`kind="interactivee"`) silently produced
`kinds=["interactivee", "any"]` filtering to literal-`any` rows
only and masquerading as a narrowed catalog — now returns an
explicit "kind must be one of: ..." error;
- the documented enum value `kind="any"` degenerated to
`kinds=["any", "any"]` which narrowed to literal-`any` rows
instead of returning "every kind" — now collapses to ``None``
so the documented semantic holds.
2. docs/coordinator-skills.md "two-surface model" section rewritten
to reflect the post-flatten reality: kind is metadata, not an
enforcement boundary. The line-67 tools-table row updated from
the long-dead `list_skills` to `skills (action=find)` with the
opt-in kind-filter framing.
3. Three stale "interactive-only" comments in session.py
(:5514, :7857, :8210) that directly contradicted the
`_prepare_skills_load` docstring ("Both kinds can load") — drop
the qualifier so future grep-and-encode hazards don't reintroduce
the rejection.
Tests:
- test_find_kind_invalid_errors — typo case (replaces the silent
degenerate to literal-any-only)
- test_find_kind_any_means_no_filter — documented enum value matches
documented semantic (collapses to None at prepare)
- test_find_kind_narrow_passes_through — valid narrowing values
reach exec as expected
Deferred to release notes (no code change, intentional policy shift):
- skills(action='get') / load can now read full content + scan_report +
allowed_tools on cross-kind rows from any session. Operators with
pre-existing kind=coordinator skills authored under the prior
implicit visibility contract should audit those bodies for
sensitive content (allowed_tools allowlists, embedded credentials,
internal hostnames in examples) before upgrade.
Closes#557. SkillKind was authored audience metadata that the
discoverability filter dressed up as a runtime visibility gate. Real
access control is allowed_tools + auto_approve, which apply identically
across kinds. The kind-scoping chokepoints scaled linearly with every
new model-write surface for zero security payoff.
Drop kind consultation from:
- ChatSession._skills_kinds (deleted) and ._lookup_visible_skill
(deleted; callers inlined to storage.get_prompt_template_by_name).
- _exec_skills_find: no longer auto-threads kinds=. The opt-in `kind`
arg is a passable filter (threads [<kind>, "any"]) so the
discoverability win survives without enforcement.
- _exec_skills_get / _exec_skills_load: row lookup is name-only.
Disabled-row gate stays on load (admin quarantine is the actual
boundary). _prepare_task already uses unscoped get_skill_by_name;
session_routes.py already calls storage directly with no kind
check. Both confirmed by the spike, no source change needed.
- tools/skills.json: drop "Coord sessions see / interactive sees"
language; kind arg description re-cast as opt-in discoverability
narrowing.
- storage Protocol docstring + console_schemas.py kind field
description: refresh to reflect passive-metadata role.
Keep:
- SkillKind enum, kind column on prompt_templates, admin Skills tab
editing, kind field in skills.find / skills.get projection. The
field is useful for sorting/grouping at the model layer and as
authored intent.
- storage.list_skills_filtered(kinds=...) parameter — admin-filter
only now; docstring updated to note it's no longer auto-threaded
from the model-tool path.
Design calls:
1. find accepts opt-in `kind` arg: YES. ~5 lines on prepare + exec.
Threads kinds=[<kind>, "any"] only when supplied. Preserves the
model's ability to narrow a browse without enforcing.
2. kind field stays in find/get projection: YES. Already pulled
directly from the row dict in _skills_project_row (session.py
line 8187); the projection survives the flatten unchanged.
Tests:
- Delete TestLookupVisibleSkill (helper gone), the two
TestExecSkillsLoadKindScoping cross-kind reject branches, the two
test_find_kind_scoping_* tests, and test_get_cross_kind_returns_not_found
— the rejections those pinned are gone.
- Add test_find_default_threads_no_kind_filter (kinds=None by default
for both session kinds), test_find_returns_all_kinds_for_session
(interactive sees both interactive- and coord-tagged rows),
test_find_filters_by_kind_when_supplied (opt-in narrowing works),
test_get_returns_row_across_kinds (cross-kind get succeeds),
test_load_works_across_kinds (cross-kind load succeeds in both
directions — the flatten contract), test_load_rejects_missing_skill
(missing-row hint coverage). Keep test_load_rejects_disabled_skill
(admin quarantine still applies), test_load_works_for_coord_on_*
(coord-side load still works on more kinds now).
Storage tests untouched: tests/test_storage_skills_filtered.py
keeps its kinds= coverage (the parameter still works, just no longer
auto-threaded from the model-tool path).
Closes review findings from PR #555 that motivated the rethink:
sec-1 (_prepare_task unscoped) moot, sec-2 (HTTP create unscoped)
moot, sec-3 (no audit on cross-kind probes) moot — there is no
cross-kind concept anymore.
Boundary spike (verified against fresh main at 9a98d07d):
- _skills_kinds defined at session.py:7933, callers exactly two:
_lookup_visible_skill (7978) + _exec_skills_find (8057, 8109).
Verified via grep across turnstone/.
- _lookup_visible_skill defined at session.py:7943, callers exactly
two: _exec_skills_get (8213) + _exec_skills_load (8277). Verified
via grep.
- _skills_project_row reads kind directly from the row dict
(r.get("kind") or "any") at session.py:8187 — no helper call;
projection survives flatten.
- _prepare_task at session.py:6353 calls unscoped get_skill_by_name
— no kind check, no change needed.
- session_routes.py:1985 calls storage.get_prompt_template_by_name
directly with no kind check — already flat.
- storage.list_skills_filtered(kinds=...) parameter is identical
across _sqlite.py:2986, _postgresql.py:2827, and _protocol.py:1364.
- tests/test_skills_tool.py: TestLookupVisibleSkill (5 tests, lines
481-536) and TestExecSkillsLoadKindScoping (5 tests, lines 539-639)
pre-flatten. Total: 61 collected → 57 collected post-flatten.
Net production LOC: -15.
The local-escape posture from the prior commit double-encoded values
that inlineMarkdown had already escaped: leading escapeHtml(text)
turns `&` into `&`, the local escapeHtml(url) then turned that
into `&amp;`, which breaks query-string URLs after browser parse
+ getAttribute + new URL round-trip.
Switch to convention-rename: regex callback params renamed to
safeAlt / safeUrl / safeLabel to signal the upstream-escape
invariant. The attribute-context lint enforces all future
attribute-context concat sites maintain the safe* convention or
call escapeHtml explicitly — defense-in-depth preserved without
the regression. Two added pin tests verify `&` survives with
single (not double) entity encoding through image data-src and
link href.
Also addresses two test issues from the same review:
- Docstring listed `safe[A-Z_]…` but code only checked isupper().
Drop the underscore option (JS uses camelCase anyway).
- `_all_attr_names` only recorded attribute-bearing tags, so a
bare `<script>` injection would have false-negatived the link-
label pin test. Refactored to `_parse_renderer_html` returning
both start tags and (tag, attr) pairs.
inlineMarkdown's image and link renderers now escapeHtml each
interpolated value (url, alt, label, domain) at the call site
instead of relying on the upstream escape pass. Defence-in-depth:
a future refactor calling those renderers from outside
inlineMarkdown would otherwise silently regress.
New CI lint scans renderer.js for `attr="' + ident` patterns; ident
must be escapeHtml(...), safe*, or in the reviewer-approved
allowlist. Four pin tests use html.parser.HTMLParser to verify
attacker URLs and labels don't materialize event-handler attributes
on rendered DOM.
Three independent fixes flagged by Copilot's review on PR #555:
2. ``update`` auto_approve self-escalation warning false-positive
(turnstone/core/session.py:_prepare_skills_update)
- The warning was computed against ``existing_auto_approve or
proposed_auto_approve`` — meaning an update that explicitly
turned auto_approve OFF still triggered the warning because the
existing row had it ON. Now computes against the *final state*
(``updates["auto_approve"]`` if present, else
``existing.get("auto_approve")``) combined with the final
``allowed_tools`` value. False-positives gone; the inverse case
(existing auto_approve=False, update turns it ON without
touching allowed_tools) now correctly fires the warning against
the inherited allowlist.
3. ``temperature`` validator silent-coerce → explicit error
(turnstone/core/skill_field_validation.py:parse_skill_session_config)
- Non-numeric temperature input silently coerced to ``None``,
unlike ``max_tokens`` / ``token_budget`` which return an error.
Numeric-field consistency: temperature now errors on
unparseable input with "temperature must be a number between 0
and 2". Range check unchanged; blank / None still → None.
4. Version-snapshot uses max+1, not count+1
(turnstone/core/session.py:_exec_skills_update)
- ``count_skill_versions + 1`` re-uses version numbers when any
row has been deleted via the existing
``storage.delete_skill_versions`` method, and the schema has no
``(skill_id, version)`` unique constraint to catch the
collision. Switched to ``max(list_skill_versions)`` + 1,
matching the ``storage.unlock_skill`` pattern. A storage-side
atomic allocator is the right architectural fix and is tracked
for a future PR.
Tests cover both the false-positive and inverse-positive auto_approve
cases, the new temperature error path, and the version-numbering edge
case where prior versions have been deleted (max diverges from count).
Two changes that share the same kind-scoping touch point.
Lookup unification (closes the bypass Copilot flagged on _exec_skills_load):
- New ChatSession._lookup_visible_skill(name) — single source of truth for
"find me a skill by name, if it's visible to this session". Combines
storage.get_prompt_template_by_name with the kind filter in one call;
returns None for both the missing-row and out-of-kind cases so callers
don't have to branch on the reason.
- _exec_skills_get refactored from inline two-step to one helper call.
- _exec_skills_load refactored from the unscoped memory.get_skill_by_name
to the new helper — the kind-scoping bypass it had (interactive could
load a kind=coordinator skill by name) goes away by construction
because the unscoped path no longer exists on the model-tool surface.
- memory.get_skill_by_name stays available for admin / sub-agent /
rehydrate paths that need full-catalog visibility — those are
deliberate cross-kind callers, not bypass surfaces. Storage exceptions
now propagate from _lookup_visible_skill by design (distinct from the
legacy swallow-and-return-None) so the operator gets a clear signal on
DB outage rather than a misleading "not found".
Coord-side load support:
- _prepare_skills_load no longer rejects coordinator sessions. Parity
with the admin / HTTP create path that already accepts a `skill` body
field on kind=coordinator workstreams — what the operator can do at
create time, the model can now do on its own session. Visibility is
still kind-scoped via _lookup_visible_skill at exec (a coord can only
load {coordinator, any}-tagged skills; interactive can only load
{interactive, any}), matching what `find` / `get` enforce.
The kind-scoping itself is queued for a separate cleanup PR: the marker
turned out to be a discoverability hint that never gated runtime
capability, and the combinatorial complexity (every new model-tool /
HTTP path needs kind awareness) isn't worth the squeeze at this team
size. Follow-up issue to land.
Test coverage:
- TestLookupVisibleSkill — 5 cases: visible / cross-kind / missing /
storage-unavailable / kind=any-on-both-surfaces.
- TestExecSkillsLoadKindScoping — kind-rejection from both directions
(interactive→coord-only, coord→interactive-only), disabled-skill
caller-side gate, and the two new positive coord-load cases (coord
loads kind=coordinator and kind=any).
- Removed test_load_on_coord_session_errors (the rejection it pinned
is gone).
Plus the /review-suggested doc fixes that came with the unification:
- Comment in _exec_skills_load now correctly attributes the disabled
collapse to the caller's enabled check rather than implying the
helper handles it.
- _lookup_visible_skill docstring documents the deliberate
exception-propagation behavior.
Replaces the legacy `skill` (load + search) and `list_skills` tools with a
single `skills(action=...)` tool serving both interactive and coordinator
sessions. Stacks on the model.skills.write permission introduced in PR 1.
Tool surface
- `find`: filter by category/tag/risk_level/enabled_only/limit with
optional BM25 query ranking; auto-approved on both kinds; kind-scoped at
the storage filter (interactive sees interactive+any, coord sees
coordinator+any).
- `get`: fetch a single skill including content; cross-kind misses
collapse to "not found" so a model can't enumerate the other surface
by name-probing.
- `load`: activate a skill in the current session (interactive-only;
coord sessions get an explicit hint pointing at spawn_workstream).
- `create`/`update`/`enable`/`disable`: require approval AND
model.skills.write; permission re-checked at exec time to catch a
revocation between approval and write.
- No `delete` — hard-delete stays admin-UI exclusive; tool description
documents the soft-delete-via-disable pattern.
Defenses on the write surface
- Approval cards surface projected risk_level (scanner re-run against
the proposed final state) and warn explicitly when allowed_tools +
auto_approve combine (auto-fire-on-load consequence is spelled out,
not just shown as raw field values).
- Toggle preview surfaces existing risk_level + allowed_tools count so
re-enabling a critical-tier skill is never a one-click bypass.
- Update path now re-fetches the row at exec to catch a readonly flip
between approval and write, filters updates back to the runtime-only
set if so, refuses if no fields survive.
- Update path rejects empty content (hollow-out via emptying bypassed
the soft-delete-via-disable invariant), non-list tags, and empty
category — failures are loud rather than silent.
- Permission denials audit `skill.write_denied` with actor_source=model
so probing the permission state leaves a trail. Audit failures log
at error (not warning) — a successful write without a row is the
exact gap the trail exists to surface.
- `_skill_hint` routes both message and system_reminder through
escape_wrapper_tags so caller-controlled values can't close the
<system-reminder> envelope and let the model fabricate directives in
its own future context.
Shared validation
- `parse_skill_session_config` lifted from console/server.py to
turnstone/core/skill_field_validation.py; both the HTTP admin path and
the model-tool path consume it. Single source of truth so field rules
can't drift between layers.
- `SKILL_RUNTIME_CONFIG_FIELDS` lifted similarly (was duplicated as
_SKILL_RUNTIME_CONFIG_FIELDS in server.py and _SKILLS_READONLY_FIELDS
on ChatSession).
- `notify_on_complete` validator now accepts list input from the JSON
schema's `array` type — previously rejected because str() of a list
yields Python repr that json.loads then refuses.
Performance
- Update prepare skips the projected-risk scan when neither content nor
allowed_tools is changing (storage re-scans on write authoritatively).
Metadata-only updates no longer pay the ~25 regex-pass scan cost.
Cleanup
- CoordinatorClient.list_skills deleted (-91 lines); model-tool path
talks to storage directly via list_skills_filtered.
- Roles admin UI gains a Model section exposing model.skills.write.
- tests/test_load_skill.py renamed to tests/test_skills_tool.py and
rewritten for the new tool — 48 tests covering registration, prepare
dispatch, permission gating (including TOCTOU-revoked exec deny),
audit actor_source on create + disable + permission-denied probe,
BM25 ranking, invalid-kind branches, audit-failure swallow, and
<system-reminder> envelope injection resistance.
In-process permission check for model-facing tool exec paths that need
to gate a write capability without HTTP middleware in the loop. Foundation
for the upcoming skills tool refactor: the merged
skills(action=create|update|enable|disable) tool will gate on
model.skills.write before reaching storage.
- Add model.skills.write to _VALID_PERMISSIONS (default-ungranted on every
role including builtin-admin — operators opt themselves in explicitly)
- Add user_has_permission(user_id, permission, *, storage=None) helper
that fails-closed on storage outages and short-circuits on empty user_id
- Document service-scope asymmetry with require_permission (no AuthResult
in the model-tool path → no bypass; explicit guidance if a legitimate
service-scope caller ever needs to reach here)
- Pin the "no implicit cache" contract with a regression test asserting
every helper call hits storage (call_count == 2 after two calls)
- Lock the "builtin-admin default-ungranted" invariant with an alembic
migration test that drives the chain to head and asserts the role's
permission string omits model.skills.write
- Plus the role-create end-to-end test proving the constant flows through
the admin endpoint's validator
Roles admin UI changes deferred to the PR that lands the gated tool — no
operator action needed until the capability exists.
Per-call DB hit + warning-log spam on outage deferred to a follow-up PR;
the helper is dead code in this commit, so cache TTL would be sized
against guesswork — better to wait for a real call-rate signal from the
first caller.
Starlette 1.0.0 reconstructs request URLs without validating the Host
header, allowing path-injection that can bypass authentication on apps
comparing reconstructed URL paths instead of `request.url.path`. Fixed
in 1.0.1.
- pyproject.toml: bump `starlette>=0.45` to `starlette>=1.0.1` so the
CVE floor is explicit at the dependency declaration, not just in the
lockfile. Annotated with the advisory ID so the rationale survives
a future floor relax.
- uv.lock: regenerated via `uv lock --upgrade-package starlette`;
starlette 1.0.0 -> 1.0.1, no transitive bumps.
Locally verified `pip-audit --strict` returns clean after the bump and
the auth + service-boundary test suites (250 tests covering the URL/
host-header reconstruction surface) continue to pass.
2000 was sized for the cloud-provider regime (50–200 events/sec)
and was too small for the two regimes that actually shape PR-D's
recovery floor:
1. **Local inference**: vLLM / llama.cpp hit 500–2000 tok/s per
active stream. Each token is an _enqueue call, so a single
busy workstream burns through 2000 events in ~1 s. Reconnects
after any disconnect longer than a network blip immediately
fall through to the replay_truncated recovery path on a
stream that was supposed to be transparently resumable.
2. **Backgrounded tabs**: Chrome (and Firefox to a lesser extent)
throttle the SSE-drain microtask aggressively when a tab isn't
visible — Chrome's background-tab budget drops to ~1 wake/min
after ~5 min hidden, so a backgrounded pane can legitimately
sit on tens of seconds of un-drained events. PR-G (drop-pings-
let-it-die) deliberately closes those connections on hide and
re-opens on focus return; reconnect-with-replay is the only
recovery path, and if the buffer evicted in the interim, the
snapshot floor is all that's left for past-turn structural
events (tool calls, state changes, approvals).
50000 at the 2000-tok/s local-inference rate buys ~25 s of pure
token streaming before truncation; at cloud rates it's minutes of
coverage. Memory cost is ~200–500 bytes per event (deque node +
dict + payload), so 50000 × 100-ws design ceiling caps at roughly
2.5 GB worst-case — and practically nowhere close because the cap
is per-ws ceiling, not per-ws steady-state. Operators on heavier
workloads can raise via TURNSTONE_SSE_EVENT_BUFFER_MAX.
Considered and rejected: in-buffer coalescing of consecutive
content/reasoning tokens. A naive text-merge breaks the replay-
slice semantic — a coalesced entry has the latest _event_id
but text that includes content the client already received under
an earlier id, so any consumer with last_event_id falling
INSIDE the coalesced span would double-render on replay. A
correctness-preserving coalesce would need a per-consumer high-
water tracker we deliberately don't maintain. Bigger cap +
simple per-event storage avoids the trap; the rationale is
captured inline in _resolve_event_buffer_max.
The browser-side completion of PR-D reconnect-with-replay. Today's
`onerror` handlers on `Pane.connectSSE`, `connectGlobalSSE`, and
the coordinator's `connectSSE` all explicitly call
`evtSource.close()` on the transient-error path — that forces the
source into the terminal CLOSED state, defeating EventSource's
native auto-reconnect (which would otherwise reconnect with the
`Last-Event-ID` header that PR-D commit 1 now honours server-side).
Three handler refactors share the same shape:
- Remove the unconditional `close()` from the transient-error
branch. Native EventSource handles CONNECTING -> CONNECTING ->
OPEN with replay automatically.
- Keep UI updates (status bar dim, Reconnecting… text) — those
are orthogonal visualizations of the disconnected state.
- Keep terminal-branch closes: a 401 expired-session still does an
explicit close + showLogin (the user must re-authenticate); a
workstream-reassignment to a different ws still disconnects +
connects on the new wsId (it's a different stream, not a same-
stream replay).
- Capture `lastEventId` in `onmessage` BEFORE `JSON.parse` so a
malformed event doesn't desync the manual-reconnect fallback
from native auto-reconnect.
- Thread `?last_event_id=N` on the URL when constructing a fresh
`new EventSource(url)` — the constructor can't set custom
headers so the query-param fallback covers the manual-reconnect
path (initial connect with a saved id, scheduleReconnect after
an explicit close, etc.).
For `Pane.connectSSE`, the long focused-pane workstream-refetch
body inside `onerror` is lifted to a dedicated
`_refetchWorkstreamsAndReassign` method so it survives the
refactor as an orthogonal trigger (handles the workstream-evicted-
during-disconnect recovery case, which is independent of the SSE
reconnect mechanics). The reassignment branch's existing
`disconnectSSE + connectSSE(newWsId)` sequence stays — different
workstream genuinely needs a fresh stream. When reassigning, the
saved `_lastEventId` is dropped because replay is per-ws and an
id from ws-A is meaningless against ws-B.
Tests in `tests/test_app_js.py` add 3 static lint guards that
fail loudly if any future refactor reintroduces a naked
`evtSource.close()` in a transient-error path of any of the three
handlers. The guards understand the allowed terminal-branch
exceptions (401, login overlay, reassignment) and ship with an
escape hatch (functions that explicitly reference `last_event_id`
have taken explicit responsibility for the replay header and are
exempt). A small `_strip_js_comments` helper handles the
apostrophe-in-comment hazard that pre-existing
`_slice_balanced_body` doesn't (comments are stripped before
brace-walking; offsets preserved by space substitution).
The console SSE proxy (`_proxy_sse`) is the inbound SSE path for
multi-node deployments — every browser EventSource that targets a
per-node route traverses it. Today's proxy strips client request
headers (only `Accept`, `Cache-Control`, and the re-minted auth
token make it upstream), so the per-ws / global SSE handlers'
`Last-Event-ID` resume (PR-D commit 1) never sees the header in
the multi-node shape — every reconnect would be a fresh connect
and silently drop events from the disconnect window.
Builds the upstream headers dict conditionally: copy `Last-Event-ID`
from the incoming request when present, omit otherwise (no
fabricated value on fresh connects). Starlette's header dict is
case-insensitive so the `request.headers.get("last-event-id")`
lookup catches both the spec-recommended capitalization and any
intermediary normalisation.
The query-param fallback (`?last_event_id=N`) needs no proxy
change — `request.url.query` is already forwarded verbatim at the
top of the function.
Tests in `tests/test_service_auth_boundary.py::TestProxySseLastEventIdForwarding`:
- Positive: browser header → upstream header (value preserved).
- Negative: browser sends nothing → upstream gets nothing (no
fabricated value).
Adds the server-side foundation for SSE reconnect-with-replay (PR-D
in issue #540's sequencing): a per-ws monotonic ring buffer that
holds the last N events for replay against a client's
`Last-Event-ID` header (or `?last_event_id=N` query-param
fallback for manual reconnect paths that can't set custom headers).
Per-ws lane (SessionUIBase + make_events_handler):
- `_event_buffer` deque (cap 2000, env-overridable via
`TURNSTONE_SSE_EVENT_BUFFER_MAX`) holds (event_id, event_dict)
tuples; `maxlen` evicts the oldest automatically.
- Existing `_ws_inflight_seq` renamed to `_event_id` and lifted
to live alongside the listeners — one monotonic counter drives
both the new replay slice AND the existing `_seq`/`snap_seq`
snapshot dedup (byte-identical contract on token events).
- `_enqueue` now stamps every event with `_event_id` (and `_seq`
on `content`/`reasoning` token events) under
`_listeners_lock`, so the buffer append + listener fan-out + new
listener registration are all atomic against each other.
- New `register_listener_with_replay` returns
(queue, replay_events, status, lost_count, earliest_id) where
status ∈ {replay_ok, truncated}. `make_events_handler` reads
`Last-Event-ID` (header or query), branches three ways
(fresh / replay_ok / truncated), and emits the SSE `id:` field
on every event sourced from the buffer. On `replay_ok` the
in-progress snapshot is skipped (the buffered events already
cover it); on `truncated` an explicit envelope precedes the
fresh-style recovery path.
- Every events stream emits a jittered `retry:` in [2500, 4500] ms
on first yield so 6-pane reconnects don't lockstep on
EventSource's default ~3 s interval.
Global lane (server.py / _global_fanout_thread / global_events_sse):
- Parallel buffer + counter on `app.state.global_event_buffer` and
`app.state.global_event_id_holder`; fanout thread stamps each
event with `_event_id` and appends to the buffer under
`global_listeners_lock`. `global_events_sse` branches on
`Last-Event-ID` with the same three shapes.
Tests:
- 16 new tests in `tests/test_sse_reconnect_replay.py` cover the
ring buffer semantics (empty-listeners hold, last_event_id
slicing, truncation, atomic registration), the counter
invariants (monotonic under concurrent writers, no skip on
queue.Full, persists across turn boundaries, cross-thread
consistency), and the handler branching (retry on first yield,
id: on buffered events, snapshot-skip on replay_ok, envelope on
truncated, query-param fallback, malformed header → fresh).
- Existing `tests/test_session_ui_base.py` updated for the
`_ws_inflight_seq` → `_event_id` rename and the new
`_event_id` field on enqueued events.
Backward-compat: all consumers that don't send `Last-Event-ID`
(today's browser, Python SDK, TypeScript SDK, channel adapter) see
behaviour identical to pre-PR — the server change is purely
additive on the request side.
Mirror the \s* posture used by the other unsafe-sink clauses (eval\s*\(,
Function\s*\(, setTimeout\s*\() so a regression like
``el.insertAdjacentHTML ("beforeend", x)`` — or a multi-line form with a
newline before the paren — still trips the lint. The trailing ``HTML``
literal continues to discriminate against insertAdjacentElement and
insertAdjacentText.
Caught by Copilot review on #541.
Extend _UNSAFE_CODE_SINK_RE with an `.insertAdjacent` + `HTML\(`
alternation so insertAdjacentHTML(...) is flagged across all 8 tracked
JS bundles. The `HTML\(` suffix excludes insertAdjacentElement, which
takes a DOM node and is not an XSS sink — the five remaining sites in
ui/static/app.js (lines 170, 216, 328, 330, 1578) stay clear.
Retire the two carve-out paragraphs (file-level comment + function
docstring) that named ui/static/app.js's verdict-badge writers as the
reason the lint hadn't already broadened. Commit 1 of this PR cleaned
both writers, so the carve-out is no longer load-bearing.
After this commit the DOM-cleanup arc (started in #532) is complete:
every unsafe-write sink family — inner/outer-HTML assignment (plain +
concat), insertAdjacentHTML, document.write, string-eval, dynamic-
Function, string-first-arg setTimeout/setInterval — is forbidden
across all 8 LLM-rendering bundles.
Rewrite the verdict-badge HTML builder from string-concat into DOM
construction (createElement + textContent + setAttribute + append).
The helper now returns a DocumentFragment of two top-level siblings
(.verdict-badge and .verdict-detail), which appendChild expands into
the parent — preserving the sibling-traversal invariants relied on by
Pane.updateVerdictBadge, toggleVerdictDetail, and the d-key keyboard
shortcut.
Inline onclick="toggleVerdictDetail(this)" replaced with an
addEventListener click handler; the non-arrow callback keeps the
`this`→button binding the old inline form had.
Both call sites (replayHistory + the live approval flow) swap from
el.insertAdjacentHTML("beforeend", X) to el.appendChild(X).
This is the last unsafe-write site in the DOM-cleanup arc started in
#532; commit 2 broadens the test_app_js.py lint regex to forbid the
insertAdjacent-HTML sink across all 8 tracked JS bundles.
The pre-push /review pass surfaced a second const-reassign that mirrors
the original `redacted` bug but in prefix-increment form:
const _paneCounter = 0; // turnstone/ui/static/app.js:10
class Pane {
constructor(wsId) {
this.id = "p" + ++_paneCounter; // line 14 — TypeError at runtime
…
}
}
`new Pane(...)` throws `TypeError: Assignment to constant variable.`
on every pane construction. The first iteration of the const-reassign
guard in tests/test_app_js.py missed it because the regex matched
postfix `X++` / `X--` but not prefix `++X` / `--X`.
Two changes:
1. Change `const _paneCounter = 0` to `let _paneCounter = 0` at
turnstone/ui/static/app.js:10. Same fix shape as the `redacted`
bug — original walker tightened to const because its reassignment
regex also only matched postfix forms.
2. Extend the reassignment regex in test_swept_bundle_has_no_const_reassign
to detect prefix `++X` / `--X` so a third repeat of this class
can't ship. Verified by injection: temporarily reverting (1)
makes the new guard fire with a clear source-text diagnostic.
Quality polish on the same test (q-1/q-2 from the pre-push pass):
- Failure message now prints the offending decl + reassignment line
text alongside line numbers, so CI failures are self-contained
(was: opaque tuples requiring two file-jumps to interpret).
- Comment on `_SWEPT_BUNDLES` documents the maintenance contract
(add only after sweeping; coordinator.js intentionally excluded).
After the var → const/let sweep, four guards keep the post-sweep state
honest in CI:
1. node --check per bundle (parse-level smoke; catches a future edit
that drops a brace or mis-balances a string before it reaches the
browser).
2. Static var-free assertion per bundle pins the keyword-swap result —
any future `var X = …` in these 7 files fails CI loudly.
3. Scope-aware static const-reassign guard per bundle. For each
`const X = …`, scans only the enclosing block (innermost { … } via
brace tracking with regex/string/comment awareness) for X
reassignments, so a same-named `let X` in an unrelated function
doesn't false-positive against a `const X` in this one. Catches
the bug class that shipped through the original sweep:
_redactApiKeys's `const redacted; redacted = …` threw TypeError
at call-time, invisible to node --check.
4. Runtime smoke for _redactApiKeys via `node -e` — calls the
function with both query-string (`api_key=…`) and JSON
(`"api_key": "…"`) shapes. This is the bit that would have
caught the actual shipped TypeError; (3) is the equivalent
static check that catches the class without needing a runtime
invocation.
Bundle list:
- turnstone/ui/static/app.js
- turnstone/console/static/admin.js
- turnstone/console/static/governance.js
- turnstone/console/static/app.js
- turnstone/shared_static/auth.js
- turnstone/shared_static/kb.js
- turnstone/shared_static/utils.js
Verified by injection: temporarily reverting `let redacted` to
`const redacted` makes both guard (3) and guard (4) fail loudly.
Follow-up to the initial var-sweep commits. The walker used a flat,
file-wide reassignment check to decide let vs const, which was
conservative when the same name appeared in multiple unrelated
scopes — e.g. `let i` as a loop counter in one function and an
unrelated `let i` reassigned in another would both stay `let`.
This second pass uses brace-tracking block-scope analysis (regex
literal aware) so tightening considers only reassignments within
the same block:
- console/static/app.js: +5 const -5 let
- console/static/governance.js: +15 const -15 let
- console/static/admin.js: +26 const -26 let
Mirrors q-2 from the /review pipeline. ui/static/app.js was
tightened in the same way already in its sweep commit.
Mechanical; no behavioural change.
754 line-start var + 50 for-loop counters converted: 663 const, 96 let
(line-start), plus 50 for-init let counters.
Hand-fix sites (surfaced by spike § 2):
- 4 try-block hoists where the var was referenced from outside the
try (var hoists out, let does not):
- tryParseMedia()'s `obj`
- _tryPrettyJson()'s `obj`
- tryParseMcpError()'s `obj`
- inline-plan render's `action` (used in the catch handler)
- showNewWsModal() cleanup (was: 2 same-scope redeclarations):
- submitBtn — first lookup at the top of the modal kept; the
redundant re-fetch + duplicate textContent at the bottom
dropped; submitBtn.disabled = false now sits as a bare
property write
- defaultOpt → renamed second occurrence to tplDefaultOpt
(genuinely distinct DOM element — modelSelect vs tplSelect),
both can be const
Post-review fix to the walker output:
- _redactApiKeys(): the walker tightened `let redacted` to `const`
but missed the `redacted = redacted.replace(...)` reassignment
on the JSON-style pass. Root cause was the walker's
find_decl_extent not recognising JS regex literals — the
unescaped " inside the character class [^&\s"] opened an
in_str state that never closed on the same line, spilling
the declaration span past `);` and pulling the reassignment
line into the skip set. The /review pipeline's bug finder and
security finder both caught it (rendering would have thrown
TypeError on every tool-output render). Now `let redacted`.
Scope-aware const-tightening pass on top of the walker (mirrors q-2
from /review): 45 additional `let` → `const` flips where the walker
was conservative because the name happened to be reassigned in an
unrelated function elsewhere in the file. Examples: `let pane` in
the 4 plan-dialog helpers; `let el` in the small Pane class methods.
Each tightening is verified safe by a brace-tracking block-scope
analysis (regex-literal aware).
Mechanical; no behavioural change.
744 line-start var + 87 for-loop counters converted: 606 const, 138 let.
Includes 2 multi-decl sites (counter accumulators at 3389 and 4021,
both `let` because the names are reassigned via += in the loop body),
and the spike-identified `indicator` redeclaration in `_toggleOidcPanel`
(now two `const indicator` declarations in disjoint block scopes —
inner if-block at 455 and function body at 482, so block-scoping makes
them independent).
Mechanical; no behavioural change.
470 line-start var + 64 for-loop counters converted: 337 const, 133 let.
Includes 2 multi-decl sites (`let url, method;` at 3951 and 4504 — both
uninitialised pairs that stay `let`) and two sibling `for (var k …)`
loops at lines 112/119 in the same function (now `for (let k …)` —
block-scoped to each loop init, no collision).
Mechanical; no behavioural change.
67 line-start var + 4 for-loop counters converted: 55 const, 12 let.
The 12 let cases are all genuine reassignments:
- Top-level state (`_loginBusy`, `_authMode`, `_refreshTimer`, etc.)
- `let delay` in `_scheduleRefreshAt` (clamped to min/max)
- `let data` inside `_tryRefresh` (assigned from inner try-catch)
- For-loop counters `let attempt`, `let i`
Mechanical; no behavioural change.
12 line-start var + 1 for-loop counter converted: 15 const, 1 let.
File previously had 4 const from the DOM-cleanup helpers; sweep finishes
the conversion.
Walker is scope-aware: when checking if name X is reassigned anywhere
in the file, lines that themselves declare X (`let X = ...`, function
parameters `(X)`, etc.) are skipped — `X = ...` in another scope is a
new binding, not a reassignment of the original. This lets variables
like `min`/`hr` (declared inside two different formatter functions)
both become `const` correctly.
8 line-start var declarations converted: 6 const, 2 let.
Walker rules:
- var X = init → const X = init when X is never reassigned in the file
- var X = init → let X = init when X is reassigned (e.g. _kbPreviousFocus
assigned in showKbHelp, html accumulated via +=)
- Reassignment check uses negative lookbehind to skip property writes
(obj.X = ...).
Mechanical; no behavioural change.
* feat(reasoning): Phase 5 — vLLM Chat Completions reasoning-field replay
Multi-turn CoT replay for vLLM-served reasoning models (Qwen3, DeepSeek-R1)
via the non-standard `reasoning` field on assistant messages. Closes the
PR #498 gap claiming Chat Completions has no replay surface — vLLM's
ChatMessage.reasoning input field is that surface (verified in
vllm/entrypoints/openai/chat_completion/protocol.py:54-64).
Session-level attach (no provider class changes). Three-gate composite:
provider isinstance OpenAIChatCompletionsProvider AND
server_compat.server_type == "vllm" AND operator-set
ModelConfig.replay_reasoning_to_model. Deliberately drops the
supports_reasoning_replay capability gate that protects Paths 1+2 —
vLLM's failure mode is silent (template-drop), not loud (server 400),
so the static gate would add operator friction without preventing the
silent failure. Server-type pin bounds blast radius — canonical OpenAI,
llama.cpp, sglang never see the non-standard field.
Also fixes a pre-existing _resolve_server_type bug: it read
cfg.capabilities.get("server_compat") but the model_registry loader pops
server_compat OUT of capabilities into the dedicated cfg.server_compat
dataclass field (model_registry.py:401, 485). Pre-fix the function
returned "" for every production ModelConfig, silently degrading PR #498
Path 3's synth-block source tag and would have made Phase 5 dead-on-
arrival. Test stubs across 3 files updated to mirror production shape
(empty capabilities + populated top-level server_compat) so the same
stub-drift can't hide future regressions.
The agent _run_agent path is deliberately excluded from Phase 5 hoists:
agent assistant messages don't carry _provider_content (rebuilt per
invocation from CompletionResult.content + tool_calls), so the helper
would no-op every turn. Comment at session.py inside _api_call documents
the exclusion.
OpenAI SDK version pin raised to >=2.37 to match the version verified
by the cross-boundary regression test
(test_reasoning_field_present_in_wire_body_when_attached) — drives a
real OpenAI client through httpx MockTransport and asserts the
non-standard field reaches the captured POST body, catching any future
SDK version that adds runtime field filtering.
Tests: 10 helper unit + 17 session integration (incl. SDK boundary
round-trip + per-gate negative tests + call-site wiring tests) + 2
audit-log discipline tests extending the PR #498 logging contract.
* docs(reasoning): apply PR #537 review on Phase 5 docstrings
Two nits from PR #537 review:
1. `_resolve_server_type` docstring claimed Phase 5 (`_maybe_attach_vllm_chat_reasoning`) called it; in fact Phase 5 reads `cfg.server_compat["server_type"]` directly off the single cfg it fetches for the operator-flag check, to avoid a second `registry.get_config` round-trip. Rewrite the paragraph: name `_maybe_synth_reasoning_block` as the sole caller (informational metadata for UI rehydration), then a separate paragraph noting Phase 5 reads the same field path directly and that both readers MUST stay aligned on changes.
2. `_maybe_attach_vllm_chat_reasoning` docstring referenced `project_reasoning_replay_capability_gate.md` which lives in personal memory store, not the repo. Replace the dead-link reference with an inline summary of the asymmetry rationale (Paths 1+2 keep the dual-gate because loud server-side failures; Path C drops the static gate because vLLM's failure mode is template-drop silent).
Three quality findings from the multi-stage /review pass on the
preceding 4-commit class-refactor stack. Bundled into one commit
because each is sub-20-line documentation/test-hygiene with no
behavioral surface.
1. **Drop stale verdict-badge line numbers in tests/test_app_js.py.**
Two comments cited ``ui/static/app.js:1287`` and ``app.js:1538``
as the ``insertAdjacentHTML`` + ``renderVerdictBadge`` consumer
sites. The class refactor moved them to 1440 and 1655 (and any
future nearby edit will move them again). Drop the numbers; cite
the helper name (``renderVerdictBadge`` / "the verdict-badge
writers") instead.
2. **Drop ``.prototype`` from 3 coord comment cross-refs.**
``coordinator.js:339, 439, 566`` referenced
``Pane.prototype.addUserMessage`` / ``addUserReminder`` /
``addToolReminder`` / ``replayHistory`` — but ``app.js`` has zero
``Pane.prototype.X`` after the refactor (it's all ``Pane.X``
class methods now). A reader following the breadcrumb hits a
grep dead-end.
3. **Introduce indent-agnostic _pane_method_offset() helper.**
The four test slices switched from ``"Pane.prototype.X = function"``
to ``"\n X("`` in commits 3 + 4 — that's brittle against the
deferred PR-B/C/D/E/F modernization (IIFE / module wrap shifts
indent to 4 spaces, breaks all four slices silently with a bare
``ValueError``). The new helper uses ``re.MULTILINE`` + ``\s{2,}``
to match the method header at any leading-whitespace depth and
``assert``s on miss so a renamed method fails loudly at the
pinning slice instead of further downstream.
Replaces 8 ``body.index("\n X(")`` call pairs across the 4 anchored
tests (replayHistory ×3, appendToolOutput ×1).
Tests: 27/27 ``tests/test_app_js.py`` green. No other suites touched.
Fourth and final commit of the ES6-class refactor (~/pane-class-refactor.md).
Migrates the remaining 5 prototype methods into the class block,
dissolving the last 4 `var self = this` workarounds, and updating the
appendToolOutput-anchored test in lockstep.
Methods migrated INTO the class body:
showInlineToolBlock(items, autoApproved, judgePending)
resolveApproval(approved, always, feedback, skipPost)
appendToolOutput(callId, name, output, isError)
sendMessage()
cancelGeneration()
Two of these (`showInlineToolBlock`, `resolveApproval`) had multi-line
header decls; the conversion script joins their arg lines back into
a single-line class-method header.
Test anchor update (`tests/test_app_js.py`):
body.index("Pane.prototype.appendToolOutput = function")
→ body.index("\n appendToolOutput(")
body.index("Pane.prototype.", start + 10)
→ body.index("\n sendMessage(", start)
The new upper-bound anchors on the next class method's header (which,
by the source-file order preserved through the refactor, is
`sendMessage`). The slice's inner assertions
(tryParseMcpError-before-renderToolOutput offset comparison) are
untouched — only the outer anchor pattern changes.
Final state:
* `class Pane { ... }`: 1 declaration with 39 members
(constructor + 38 methods)
* `Pane.prototype.X = function`: 0 occurrences (was 38)
* `var self = this`: 0 occurrences (was 16)
* Arrow callbacks (`=>`): 55 (was 0)
* 3 module-level helpers (_buildWatchResultBubble,
_buildDefaultReminderBubble, _buildOutputWarningEl) cluster
immediately after the class block.
The framing goal — "coord speaks a more modern JavaScript than
interactive" — collapses on this axis: Pane is now ES6 class shape
with arrow-function callbacks and `this`-lexical inner scopes, on par
with coord's ES6+ idioms. The remaining var → const/let sweep and
template-literal pass are deferred to follow-up PRs B-F per §8 of the
refactor brief.
Tests: 27/27 `tests/test_app_js.py` + 258 broader (renderer + console
suites) green. All 4 historically anchored test slices now use
class-method anchors and pass cleanly.
Third of the four planned commits in ~/pane-class-refactor.md.
Migrates the two test-anchored history-rebuild methods into the
class block, updates the three pytest assertions that sliced them
by `Pane.prototype.X = function` literal, and relocates the last
nested module-level helper to live alongside the other two.
Methods migrated INTO the class body:
replayHistory(messages) — 304-line method, the largest single
method in the file. Dissolves
2 of the remaining `var self = this`
sites (the method-scope one + the
inner-callback one inside the
`tool` role branch's
replayAdvisoriesAfterTool callback).
_attachRetryToLastAssistant() — small leaf method that the
replayHistory tests use as the
lower-bound sentinel for their
slice.
Helper relocated to just after the class block:
_buildOutputWarningEl(assessment) — was nested between
replayHistory's `};` and
`_attachRetryToLastAssistant`'s
header. Joins the two helpers
that already moved in commit 1
(_buildWatchResultBubble,
_buildDefaultReminderBubble) —
all three module-level helpers
now cluster immediately after
the class.
Test anchor updates (`tests/test_app_js.py`):
body.index("Pane.prototype.replayHistory = function")
→ body.index("\n replayHistory(")
body.index("Pane.prototype._attachRetryToLastAssistant", start)
→ body.index("\n _attachRetryToLastAssistant(", start)
Three assertions touched: `test_replay_history_renders_content_before_tool_block`,
`test_replay_history_renders_persisted_verdict_badge`,
`test_replay_renders_user_interjection_advisory_after_tool_block`.
The slice's inner assertions (`msg.content`-vs-`msg.tool_calls` offset
ordering, `renderVerdictBadge` regex, `replayAdvisoriesAfterTool` +
`addUserMessage` substring matches) are untouched — only the outer
anchor pattern changes.
After this commit 5 prototype declarations remain: showInlineToolBlock,
resolveApproval, appendToolOutput, sendMessage, cancelGeneration —
all five migrate in commit 4 alongside the appendToolOutput test
anchor update.
Tests: 27/27 `tests/test_app_js.py` green. Class block now spans
lines 12 → 1634; all 3 module-level helpers cluster at 1642 / 1676 /
1698 just after.
Second of the four planned commits in ~/pane-class-refactor.md.
Migrates the callback-heavy methods that don't anchor any test
slice, dissolving 12 of the 16 `var self = this` workarounds into
arrow-function lexical-this along the way.
Migrated INTO the class body:
_createDOM, connectSSE, handleEvent,
_addUserMsgActions, _addRetryAction, _retryLast,
_rewindToMessage, _startEdit, _editAndResend
Each migration applies the same mechanical transformation:
* `Pane.prototype.X = function (args) {` header → class-method
`X(args) {` form, body re-indented +2 spaces.
* Every `var self = this;` declaration removed.
* Every inner `function (...) {` callback rewritten as
`(...) => {` — arrow functions inherit `this` lexically, so the
outer-self capture pattern dissolves without behavioural change.
* Every `\bself\b` identifier rewritten as `this`.
* Closer `};` → `}` (no semicolon on class methods).
`sendMessage` and `cancelGeneration` are deliberately deferred to
commit 4 even though they're shape-eligible for this commit: they sit
AFTER `appendToolOutput` in the file, and
`test_phase8_appendtooloutput_dispatches_mcp_error_before_renderer`
slices `appendToolOutput` by looking for the next `Pane.prototype.`
declaration as an upper bound. Migrating sendMessage + cancelGeneration
now would leave `appendToolOutput` as the LAST prototype declaration
in the file, breaking that slice. Co-migrating all three in commit 4
keeps every intermediate commit green.
Spike-verified (§2.10.5): all 16 `var self = this` sites in app.js
are Type 1 (outer-`this` capture only — no event-target-`this`, no
delayed semantic capture). Mechanical conversion is safe for every
site touched here.
Diff: 805 insertions / 816 deletions (net −11 lines) — the arrow
form is more compact than `function (args) {`, partly offsetting the
class-body indent overhead.
Tests: 27/27 `tests/test_app_js.py` + 238 broader (renderer + console
suites) green. Anchored tests (replayHistory ×3, appendToolOutput ×1)
remain on their existing `Pane.prototype.X = function` literals —
their methods migrate in commits 3 + 4.
Opens the ES6 modernisation of ui/static/app.js (per the spike in
~/pane-class-refactor.md §2.10). This first commit lays the class
scaffolding and migrates the 22 callback-free leaf methods — the
remaining 16 callback-heavy / test-anchored methods land in commits
2-4.
Migrated INTO the class body:
constructor(wsId) (was `function Pane(wsId)` at line 12)
reset, updateWsName, disconnectSSE, setBusy,
showEmptyState, removeEmptyState,
addThinkingIndicator, removeThinkingIndicator,
addSystemNudgeMarker, addUserReminder, addToolReminder,
addUserMessage, getFeedback, appendToolOutputChunk,
showOutputWarning, updateVerdictBadge, updateVerdictGlow,
addInfoMessage, addErrorMessage, updateStatus,
isNearBottom, scrollToBottom
Two module-level helpers (`_buildWatchResultBubble`,
`_buildDefaultReminderBubble`) were previously nested between leaf
methods. Class bodies can't hold free function declarations, so they
relocate to immediately after the class closing `}`. Function
declarations are module-hoisted so the relocation is semantically
free.
The other 16 prototype methods continue as `Pane.prototype.X =
function (...)` below the class — they each add to `Pane.prototype`
exactly as before, so the prototype shape is unchanged.
Spike-verified guardrails:
* Hoisting: only `new Pane()` site is `createPane` at line 2152
(renumbered), well after the new class block ends at line 458.
* Strict mode: `app.js` is already clean of `with`,
`arguments.caller`, `arguments.callee` — class bodies' implicit
strict mode is a no-op.
* Method enumerability: the 14 `for (var pid in panes)` loops
iterate the module-level `panes` ID-map, not method names on an
instance — class methods being non-enumerable on the prototype
doesn't affect them.
No `var self = this` sites are touched in this commit — the 22 leaf
methods all have zero inner callbacks. Commits 2-4 will dissolve the
16 var-self-this sites as their parent methods migrate.
Tests: 27/27 in `tests/test_app_js.py` green (test anchors on
`replayHistory` and `appendToolOutput` are untouched — commits 3 + 4
will co-migrate them with the assertion updates).
Adds ``console/static/app.js`` to ``_UNSAFE_CODE_SINK_LINT_TARGETS``.
The parametrized scan now covers 8 static JS bundles — all admin-side
bundles are clean.
Docstring updates:
- Fully qualify the bundle paths in both posture lists
(``ui/static/app.js``, ``shared_static/utils.js``, etc.) so the two
``app.js`` files are unambiguous now that both are in the targets
list.
- ``console/static/app.js`` joins the **strict DOM-construction**
list (alongside the interactive surface + shared helpers + coord
chat entry) — the cluster-dashboard renderer is now full
createElement / textContent construction, no HTML strings ever
interpolated.
- ``console/static/admin.js`` + ``console/static/governance.js``
remain in the **sink-free string-concat** list — they retain the
escapeHtml + concat builder shape with the unsafe sink off the call
site.
The ``insertAdjacent`` carve-out note (verdict-badge writers in
``ui/static/app.js``) is now qualified to avoid ambiguity.
Initial AST-light swap routed 12 ``innerHTML =`` sites through
``setSafeHtml`` (8 sites) or ``replaceChildren()`` (4 empty-string
clears). The /review pipeline flagged the node-table render path as
a hot loop where DOMParser-per-row costs scale with cluster size (the
project's 100-node design ceiling × per-RAF render frame on SSE
churn), and the static colHeaders literal was being re-parsed every
render.
This commit lifts the entire console-dashboard renderer to true
``createElement`` + ``textContent`` + ``append`` construction —
matching the strict-posture lane used by ``ui/static/app.js`` rather
than the sink-free string-concat posture admin/governance use.
Net result: **zero** ``innerHTML`` and **zero** ``setSafeHtml`` calls
remain in ``console/static/app.js``.
Three new module-local helpers carry the heavy structural fragments:
- ``buildColHeaders()`` returns a DocumentFragment with the 7-span
column-header layout used at the top of the table and inside each
multi-node group body. Built once via createElement so the static
literal isn't re-parsed every render — and the duplicated literal
between the top-table and per-group sites collapses into one helper.
- ``_buildNodeNumCell(value, highlighted, cellClass)`` — the
``<span class="X num [has-value]">N</span>`` shape used 8× across
buildNodeRow and the group header.
- ``_buildHealthCell(cellClass, healthPct, healthFillClass)`` — the
health-bar trailing cell used both per-row and per-group.
The 4 ``setSafeHtml`` sites that remained after the initial sweep
(error-state placeholders + state-pill builder) also flip to
``createElement`` / ``makeEmptyState`` for consistency with the rest
of the file's new posture. ``makeEmptyState`` is the helper added
during the interactive cleanup (#532).
Two Copilot findings, both pre-existing on main since 2026-04-05 but
preserved by this PR's mechanical refactor. Addressing them here
since they're appropriately in scope (the renderJudgeSettings
function is the focus of the refactor) and Copilot ranked them high.
- **Dead ``shortKey === "model"`` branch removed.** The loop at the
top of ``renderJudgeSettings`` does ``if (s.key === "judge.model")
continue;`` because ``judge.model`` is rendered by the cross-cutting
model-alias picker in ``admin.js`` (line 4895, ``aliasKey: "judge.
model"`` registry entry). No other setting key has the form
``judge.X`` where ``X === "model"``, so the conditional branch was
provably dead — Copilot's confusion ("operators won't see a model
picker") is the same confusion a future reader would hit. Drop
the branch + the corresponding ``SELECT``-dispatch in the binding
loop (no SELECT inputs remain in this renderer).
- **Float-input value/min/max now escapeHtml'd.** Previously
``currentVal`` and ``s.min_value`` / ``s.max_value`` were
interpolated raw into ``value="..."``, ``min="..."``, ``max="..."``
attributes. Numbers stringify safely, but
``admin_list_judge_settings`` can fall back to returning a raw
stored string when deserialization fails (Copilot's flag) — a
non-numeric fallback containing a ``"`` would break out of the
attribute boundary. Wrap with ``escapeHtml(String(...))`` for
defense-in-depth + null-safety.
Adds ``console/static/governance.js`` to ``_DOM_WRITE_LINT_TARGETS``.
The parametrized scan now covers 7 static JS bundles. Docstring
updated: governance.js joins admin.js in the "sink-free string-concat"
posture; ``console/static/app.js`` (cluster dashboard / node table) is
the last admin-side bundle still pending — same posture once cleaned.
Same posture sweep as the admin DOM cleanup (#533) applied to
turnstone/console/static/governance.js:
1. **46 innerHTML sites → setSafeHtml**. Mechanical swap via the
same AST-light Python walker the admin PR used. HTML strings are
still built with escapeHtml + concat — same defence as before, just
no innerHTML sink at the call site. `node --check` clean; prettier
formatted.
2. **6 inline event handlers in renderJudgeSettings refactored to
delegated bindings**. The previous code embedded the setting key
as a JS-string inside an HTML attribute
(`onclick="saveJudgeSettingFromInput('KEY')"`), the same footgun
addressed in admin.js: escapeHtml turns `'` into `'`, but the
HTML parser decodes that before the JS parser runs, so a key with
an apostrophe would escape the JS string. Keys today come from a
static judge-settings registry without apostrophes, so no live
vuln — but the pattern is brittle.
Inputs now carry a single `data-judge-key`; the binding loop
dispatches on `this.type === "checkbox"` / `this.tagName ===
"SELECT"` to wire change-listeners for the auto-save inputs, and
leaves text/number/password inputs alone (they commit via the
adjacent Save button). Save and Reset buttons carry their own
`data-judge-save-key` / `data-judge-reset-key` and bind via click
delegation — mirrors the admin.js settings-tab shape.
3. **1 inline handler in audit pagination converted** for
consistency: `<button onclick="loadMoreAudit()">` → addEventListener
after setSafeHtml.
4. **4 stale safety-narrator comments stripped** ("// values escaped
via escapeHtml above", "// NOTE: innerHTML usage below is safe",
etc.). The safety now lives in setSafeHtml; the per-call-site
narration is tombstone-shaped and removed per the project
no-tombstone-comments convention.
5. **`saveJudgeSettingFromInput` hardened**. The
`document.querySelector('[data-judge-key="' + key + '"]')` lookup
now wraps the key in `cssEscape` (from shared/utils.js) so a
future key containing `"` or `\` doesn't break the selector.
console/static/app.js (12 sites) remains pending — same posture once
cleaned, separate PR.
Two small follow-ups from the Copilot PR review:
- ``admin.js:3112-3115`` — the docstring on ``_onSettingChange`` said
``inp`` is "passed in by the delegated handler", but the wiring at
the call site is a per-element ``addEventListener`` rather than a
single delegated handler on the container. Reword to
"per-input event-listener callback" so a future reader doesn't
read "delegated handler" and refactor under that mistaken premise.
- ``test_app_js.py`` — ``_DOM_WRITE_LINT_TARGETS`` constant name
was missed in the earlier sweep that renamed ``_UNSAFE_DOM_WRITE_RE``
→ ``_UNSAFE_CODE_SINK_RE`` and the test to
``test_no_unsafe_code_sinks_in_static_assets``. Rename to
``_UNSAFE_CODE_SINK_LINT_TARGETS`` so the three names align.
Extend ``_UNSAFE_DOM_WRITE_RE`` to also flag the JS string-to-code
constructors that share the same XSS / RCE-on-injection threat model
as innerHTML:
- ``eval(...)`` — string-eval
- ``new Function(...)`` — dynamic-Function constructor
- ``setTimeout(string, ...)`` / ``setInterval(string, ...)`` — the
string-first-arg form (function-first-arg remains unflagged)
Verified that none of these sinks exist in the six currently-scanned
files (interactive app.js + shared utils/auth/kb + coord chat entry +
console admin.js). Pre-existing parametrized test
``test_no_unsafe_dom_writes_in_static_assets`` extends naturally to
the broader pattern; all 25 cases pass.
``insertAdjacent`` + HTML continues to be excluded — two existing
verdict-badge sites in ui/static/app.js consume
``renderVerdictBadge``'s HTML-string output, so broadening that
specific sink first needs the upstream helper cleaned.
The settings panel renderer in admin.js previously emitted inline
``onclick``/``onkeydown``/``oninput``/``onchange`` attributes that
embedded the setting key as a JS-string inside an HTML-attribute
context:
'<button ... onclick="_saveSettingValue(\'' + escapedKey + '\')">'
``escapeHtml`` escapes apostrophes to ``'``, but the HTML parser
decodes that *before* the JS parser runs — so a key containing an
apostrophe would break out of the JS string. Today the keys come
from a static settings registry without apostrophes, but the pattern
is brittle: a future maintainer adding operator-controlled values to
the attribute would discover the footgun the hard way.
Every inline handler in admin.js is now replaced with a delegated
``addEventListener`` set up after ``setSafeHtml(container, html)``.
Handlers read their context off ``data-*`` attributes (which
``setAttribute`` correctly escapes), so the HTML-attribute /
JS-string double-context is eliminated.
Touched renderers:
- ``_renderSettings`` — section headers, help buttons, per-key inputs
(input/change), per-key save + reset buttons
- ``_renderNodeMetadata`` — section headers (delete/add buttons
already used delegation)
- MCP install source selector — radio-change handler for
``_updateInstallFields``
The named handlers (``_saveSettingValue``, ``_toggleSettingsSection``,
etc.) are unchanged in signature and behaviour; only their wiring
moved from inline-attribute to ``addEventListener``.
Two changes to ``tests/test_app_js.py``'s DOM-write lint:
1. Add ``console/static/admin.js`` to the scan target list. Now
covers all 6 static JS bundles that render LLM output, tool
results, operator-supplied data, or user input. ``governance.js``
and ``console/static/app.js`` remain pending follow-ups (will land
as separate cleanup PRs).
2. Parametrize the lint test over the target list. Each file is now
its own pytest case (e.g.
``test_no_unsafe_dom_writes_in_static_assets[turnstone/console/static/admin.js]``),
so a failure attributes precisely to the offending file instead of
masking offenders behind the first-file's assertion.
Rename the test from ``..._in_interactive_assets`` to
``..._in_static_assets`` — the coverage now spans more than the
interactive surface, and the surface-neutral name leaves room for
governance.js / console-app.js without another rename.
3. Restructure the docstring to surface the two distinct postures
(strict DOM-construction surfaces vs. sink-free string-concat
admin.js) up front, instead of burying the admin caveat after the
main contract claim.
48 sites in turnstone/console/static/admin.js previously assigned
HTML strings directly to .innerHTML. Every site is now routed
through the shared setSafeHtml helper (added in the interactive
cleanup PR), which parses the trusted HTML via DOMParser and installs
the result via replaceChildren — no innerHTML sink at the call site.
Two distinct postures across the admin pages:
- 47 sites: ``setSafeHtml(el, html_built_with_escapeHtml)`` — admin
builders construct HTML strings via string concatenation, running
every interpolated value through escapeHtml first. Defence still
depends on escapeHtml at the builder; the lint catches the sink but
cannot catch a missing escape. Full DOM-construction rewrites
(createElement + textContent) would be structurally safer but are
out of scope — 136 escapeHtml call sites + several thousand lines
of builder code is a separate effort.
- 1 site: ``srcEl.replaceChildren()`` for the MCP-install package
panel's empty-state branch — equivalent to the old
``srcEl.innerHTML = ""`` clear, slightly more idiomatic.
No user-visible behaviour change. DOMParser parses the same HTML the
prior innerHTML assignment did; the new DOM is identical, and the
container.querySelectorAll("[data-X]") event-binding pattern still
finds the newly-installed nodes the same way it did before.
console/static/governance.js (46 sites) and console/static/app.js
(12 sites) remain pending follow-ups. The verdict-badge writers'
two insertAdjacentHTML sites in ui/static/app.js still need the
upstream helper cleaned first — separate effort.
The pyjwt 2.12.1 advisory (\"weak encryption\") is disputed by the
supplier — the key length is the calling application's
responsibility, not the library's. Turnstone generates its JWT
signing keys via the standard ``secrets`` module at
operator-controlled strength (see ``turnstone/core/auth.py``), so the
advisory does not apply to this codebase.
No fix version is available — pyjwt 2.12.1 is the current PyPI
latest as of 2026-05-21. Adding ``--ignore-vuln PYSEC-2025-183``
with the rationale documented in-line so a future reviewer can
re-evaluate when an upstream fix or a non-disputed re-issue lands.
The advisory was published between main's last CI pass (2026-05-19)
and the interactive-cleanup PR's CI run (2026-05-21); main's
security job will fail next push without this fix.
Two Copilot-suggested improvements to the regression scan:
- Allow optional ``+`` before ``=`` in the regex so a future
regression that switches sinks from ``el.innerHTML = X`` to
``el.innerHTML += X`` is still caught. The trailing ``(?!=)``
negative-lookahead still excludes ``===`` / ``==`` reads.
- Switch the scan from line-by-line ``splitlines()`` iteration to a
whole-body ``finditer`` so ``\\s*`` can span newlines. Multi-line
sinks like ``el.innerHTML\\n = X`` (an artifact of formatter
line-wrapping at the assignment) are now caught. Match positions
map back to line numbers for the failure message.
Verified locally with representative test cases including
``el.innerHTML += X``, ``el.outerHTML += X``, multi-line variants,
and the ``===`` / ``==`` reads that must remain unflagged.
Adds two regression tests in tests/test_app_js.py:
- test_no_unsafe_dom_writes_in_interactive_assets: whole-file scan
for inner/outerHTML assignment + doc-write sinks across all five
interactive surfaces (app.js, shared utils/auth/kb, coord chat).
Includes line + content in the failure message so a regression
fails loudly with location.
- test_shared_utils_defines_set_markdown_helper: pins setMarkdown's
signature and the DOMParser path so a refactor that drops the
parser (e.g. swap to Range.createContextualFragment) forces an
explicit reviewer decision.
The lint regex is tightened with a negative-lookahead so equality
comparisons (``===`` / ``==``) don't false-positive, and broadened
to cover ``outerHTML`` and the legacy doc-write sink in addition to
``innerHTML``. ``insertAdjacent`` + HTML is *not* covered yet — two
existing verdict-badge sites (app.js:1287, 1538) consume the HTML
output of renderVerdictBadge and would need that helper cleaned
first.
Three adjacent sites that all assign pre-trusted HTML strings (built
from escapeHtml + static template literals, no caller-supplied raw
HTML) get the same DOMParser + replaceChildren treatment via the
shared setSafeHtml helper:
- coordinator.js:327 (appendMsg's body) — callers pass either
esc(text) or renderToolOutput(...) output, both pre-escaped.
- auth.js:302 (login overlay) — _buildLoginHTML() returns a static
template with no caller-supplied interpolation.
- kb.js:30 (keyboard-help overlay) — html is built from a static
keys-and-bindings table.
Eliminates the only remaining innerHTML site in coord and the two
shared-overlay sites. Console admin / governance JS bundles
(106 sites in console/static/{app,admin,governance}.js) remain
outside this PR — different threat model (admin-only behind auth
gate), separate effort.
Routes every direct-HTML assignment in turnstone/ui/static/app.js
through the helpers added in the previous commit, or through native
DOM construction (createElement + textContent + append /
replaceChildren). Net result: zero ``.innerHTML =`` sites in app.js.
Breakdown of the 26 sites:
- 2 renderer-output sites (replayHistory message body, plan-inline
body) now use setMarkdown — DOMParser keeps the audit at zero
innerHTML sites, stricter than the prior centralise-not-eliminate
plan.
- 8 empty-string clears (pane reset, layout rebuild, dashboard
refresh, etc.) become replaceChildren().
- 5 keyboard-shortcut button labels (y/n/a/Esc) collapse onto
makeKeyLabel(hint, label).
- 5 dashboard placeholders (Loading / Failed / No active workstreams)
use makeEmptyState(text).
- 6 escapeHtml-interpolated HTML strings (command preview, judge
evidence, dashboard state cells, footer node, diff lines) become
createElement + textContent + append; escapeHtml drops out because
textContent escapes intrinsically.
No user-visible behaviour change — DOMParser parses the same HTML
the prior innerHTML assignment did, and DOM construction with
textContent produces equivalent rendered output. Mermaid / hljs
post-render scope is unchanged (now scoped to the body element
rather than the wrapper for the two setMarkdown sites; both
contain the same code blocks).
Adds four helpers that move the unsafe HTML-string sinks off the call
site:
- setSafeHtml(el, html): parses a trusted HTML string via DOMParser
and installs the result via replaceChildren — no innerHTML.
- setMarkdown(el, content): renderMarkdown -> setSafeHtml ->
postRenderMarkdown (hljs + mermaid).
- makeEmptyState(text): builds a <div class="dashboard-empty"> card.
- makeKeyLabel(hint, label): keyboard-hint + label fragment for
approve/deny/always/amend/reject buttons.
The helpers are unused at this point; subsequent commits route the
26 app.js sites, coord:327, auth.js, and kb.js through them.
* feat(admin): align turnstone-admin DB config with server (config.toml + env)
turnstone-admin previously read TURNSTONE_DB_* env vars only, forcing
operators with credentials in config.toml to re-export them just to
run admin commands. Wire add_config_arg + apply_config(["database"])
into main() so admin honors the same precedence as turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded defaults.
Also exposes pool_size + sslmode/sslrootcert/sslcert/sslkey to admin,
which previously dropped any such config silently.
Hardening: load_config() now warns once when config.toml is group- or
world-readable, since DB password and TLS key paths live in [database].
Tests cover precedence (default / config / env / partial fallback /
empty-string-in-config-beats-env), the real init_storage boundary on
a tmp sqlite path, the sys.argv -> main() pre-parser path, and the
new permission check (mode 0644 warns, 0600 quiet).
* test(admin): unify config import style in test_admin_db_config
Use module alias (config_mod.apply_config) instead of mixing
'import turnstone.core.config as config_mod' with 'from
turnstone.core.config import apply_config'. Addresses
github-code-quality bot feedback on PR #531.
Async LLM-tier "llm_fallback" verdicts (judge.py:1073, judge.py:1131
via _deliver_fallbacks) deliberately reuse the heuristic verdict's
``verdict_id`` so the row gets "upgraded in place" from heuristic →
llm_fallback when the LLM judge times out, is cancelled, or returns
no content. The consumer ``_persist_intent_verdict`` was doing a
plain INSERT via ``create_intent_verdict``, hitting the
``intent_verdicts_pkey`` constraint on every llm_fallback delivery.
Postgres logged the duplicate-key error; the application try/except
swallowed it at log.debug — so the row never actually got upgraded
and the LLM judge's annotation ("(LLM judge did not return a
verdict)") was lost.
The collision rate exploded on stable/1.5 smoke tests because
PR #527 (just merged) added two new heuristic-INSERT paths in the
auto-approve early-return branches of ``approve_tools`` — previously
those branches dropped heuristic verdicts on the floor, leaving no
row for the fallback to collide with.
Fix:
- New ``upsert_intent_verdict`` method on the storage protocol +
sqlite + postgres impls, using dialect-specific
``insert(...).on_conflict_do_update(index_elements=["verdict_id"],
set_={...})``. Set_ clause updates ONLY the three fields that
genuinely change between heuristic and llm_fallback: ``tier``,
``reasoning``, ``judge_model``.
- Every other column is excluded from set_: identity columns
(verdict_id, ws_id, call_id, func_name, func_args), carried-
verbatim columns (intent_summary, risk_level, confidence,
recommendation, evidence, latency_ms), and ``user_decision``.
- ``user_decision`` exclusion is load-bearing: ``IntentVerdict
.to_dict()`` doesn't project it, so a fallback verdict reaching
``_persist_intent_verdict`` carries the kwarg's ``"pending"``
default. If the operator already resolved the approval between
heuristic INSERT and fallback delivery, the row's user_decision
has been stamped to ``"approved"``/``"denied"``/``"timeout"`` (or
an auto-approve reason at heuristic-INSERT time per PR #527).
Including ``user_decision`` in set_ would silently clobber that
back to ``"pending"``.
- ``_persist_intent_verdict`` switched from ``create_*`` to
``upsert_*``. Bulk path ``create_intent_verdicts_bulk`` stays as
plain INSERT — every heuristic ``verdict_id`` is freshly minted
in ``judge.evaluate`` so in-turn dups can't happen. The inverse
race (daemon-judge verdict lands BEFORE the bulk write) IS
reachable today but its observable behavior is unchanged by the
per-row UPSERT switch; documented at the bulk site for a future
hardening pass.
Test coverage:
- TestIntentVerdictUpsert × 4 — fresh-id insert, conflict-upgrade,
user_decision preservation across heuristic→approved→fallback,
identity + carried-field preservation.
- Existing tests in test_session_ui_base.py updated to mock the
new upsert method instead of create_intent_verdict.
Two dead module-level constants flagged by github-code-quality on
PR #529: ``_INSPECT_MSG_SNIP_THRESHOLD`` and
``_INSPECT_TOOL_ARG_SNIP_THRESHOLD`` lost their callers when the
content-snip logic moved into the ``_snip_head_tail`` helper. The
helper now reads ``head + tail + _INSPECT_ELISION_MARGIN`` so the
"reserve bytes for the elision marker" rationale that the dead
constants documented stays named instead of becoming a bare ``64``.
A coord doing a fan-out wave of inspect_workstream calls against
tool-heavy children could blow the context budget on raw output
alone (one child with a 100 KB bash result × N children). The
previous safety net was ``_truncate_output``'s head+tail strategy,
which silently drops *middle* messages — exactly the wrong shape
for a coordinator trying to understand a child's trajectory (the
LAST message tells the model what the child concluded; the FIRST
sets the brief; the middle is the connective tissue).
Three-tier degradation modeled on the search tool's pattern at
``session.py:_format_search_results``:
Tier 1 (full): every message verbatim — used when size fits.
Tier 2 (compact): per-message head/tail-snipped content (600/300
chars) plus snipped ``tool_calls.arguments``
(300/100 chars). When content snipping alone
doesn't fit, fall through a message-list trim
ladder ((20,30) → (10,20) → (5,10)) that keeps
head + tail messages and elides the middle as
``{"_omitted": N}``.
Tier 3 (skeleton): no messages — counts + role distribution +
verdicts-by-risk + last assistant preview.
Budget 32 KB (matches ``_SEARCH_OUTPUT_BUDGET``). First emission
whose JSON serialization fits the budget wins. ``_tier`` lands on
every non-error emission so the coordinator LLM and audit readers
can see which compression rung was selected; ``_tier_note`` carries
actionable advice (re-call with a smaller ``message_limit`` etc.).
Error-shape results bypass tiering — they're already small.
Bug fixes caught during review:
- ``_compact_message`` now preserves the assistant-side ``tool_calls``
list with snipped ``function.arguments``; the pre-fix shape left
audit readers with tool-result orphans against invisible calls.
- The intermediate Tier-2 list-trim ladder fixes a size-monotonicity
bug where Tier-2 with un-snippable content (per-message body
under the 964-char threshold) plus the added ``_tier_note`` came
out STRICTLY larger than Tier-1, falling through to skeleton
when a head+tail trim would have preserved dozens of messages.
- ``_inspect_skeleton`` reads ``result["skill_id"]`` (production
storage row key) with a ``skill`` fallback; pre-fix it read
``skill`` only and emitted ``null`` for every real workstream.
Three Copilot threads from PR #526:
1. ``_exec_spawn_workstream`` success path emitted
``{"child_ws_id": null}`` when the upstream response unexpectedly
omitted ``ws_id`` (200-shape with no error field, no id field).
Adds the missing guard — mirrors ``_exec_spawn_batch`` which
already surfaces ``"spawn returned no ws_id"`` as a denied row.
The LLM now sees a tool error and can retry instead of chasing
a null id through follow-up tools.
2. ``docs/coordinator-skills.md`` UI render note said "keep the
ws_id as the click-through key" in a paragraph that had just
introduced ``child_ws_id`` — readable as "the ws_id value" but
confusable as a field-name claim. Clarifies that the value
class is the same regardless of which key carried it.
3. ``docs/bulk-endpoints.md`` ``spawn_batch`` example shows
``child_ws_id`` (coord-tool output shape). The doc title and
the "model tool" column label already disambiguate it from HTTP
API responses, but a reader landing at the example section
directly could miss the framing. Adds one explicit sentence.
Coordinator LLMs on large fan-outs recency-bias on seeing `ws_id`
in a `spawn_workstream` / `spawn_batch` return -- calling
`spawn_workstream(ws_id=...)` again instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this
cascades into self-inflicted re-spawn loops.
Rename to `child_ws_id` (already an existing project term -- see
`tasks` tool, `child_event_bus.py`) defuses the recency bias.
Scope is the LLM-facing JSON only -- the server HTTP API at the
spawn endpoint still returns `ws_id`, and the internal reads of
that HTTP response are unchanged.
Also updates the two tool descriptions, the operator-facing skill
doc, and the bulk-endpoints example so docs don't undo the rename.
A skill with `allowed_tools=[]` in the coordinator's `list_skills`
response read as "no tools are usable by this skill" to a model that
didn't know the semantics — but the actual meaning is "no tools are
pre-approved for auto-approval (auto-approve exemption list)". Real
misdiagnosis incident: a code-review child appeared to have been
spawned with zero tool access when in fact the skill simply hadn't
declared an auto-approve allowlist.
Two-part fix:
- `coordinator_client.list_skills` omits the `allowed_tools` key from
the per-skill dict when empty. Absence now carries the unambiguous
meaning "no tool is pre-approved for this skill"; presence (with a
non-empty list) keeps the standard Claude Code skill-spec shape.
- `turnstone/tools/list_skills.json` description rewrites the field
doc so the LLM sees: "tool names exempt from the operator approval
gate ... the field is OMITTED when empty: a skill without
`allowed_tools` still has access to every tool in its session's
toolset; absence of the field means no tool is pre-approved for
this skill, not that the skill has no tools."
Field name stays `allowed_tools` — matches the upstream Claude Code
skill frontmatter (`allowed-tools` hyphenated, stored as
`allowed_tools` internally per `skill_parser.py:241-242`). Parser,
storage column, admin UI, and SDK unchanged.
Auto-approved tool calls left intent_verdict rows with `user_decision=""`,
indistinguishable from rows still pending manual review. Real misdiagnosis
incident: a coord with `recommendation="review"` and `user_decision=""` was
read as "stuck waiting for approval" when in fact the tools had been
auto-approved and the child was running normally.
New vocabulary at the storage API boundary (column server_default stays
`""` so pre-fix legacy rows are still distinguishable as such):
- `pending` — at insert, before any resolution
- `approved` / `denied` — manual user resolution
- `timeout` — approval-event timeout (split from `denied` so the
audit column alone tells them apart; the feedback
string used to carry this distinction)
- `policy` / `blanket` / `skill` / `always` / `auto_approve_tools` —
auto-approve reasons (mirror `AutoApproveReason`)
Heuristic verdicts on the two auto-approve early-return branches are now
persisted with `user_decision=<reason>` (previously dropped on the floor).
Late LLM verdicts for already-auto-approved call_ids look up the reason via
a TTL-pruned `_auto_approve_reasons` map (lazy 60s prune at write time, so
no fixed cap can silently regress the fix on the N+1th auto-approve; LLM-
disabled sessions don't leak entries because prune fires whenever auto-
approves happen).
Bug fixes caught during review:
- `on_intent_verdict` early-returns when the verdict already carries an
auto_reason — without this, a manual `resolve_approval` on a mixed batch
would overwrite the auto-stamped row with `approved`/`denied`.
- `_record_auto_approves` runs BEFORE `_persist_auto_approved_heuristic_*`
so the lookup map is populated before any concurrent LLM verdict can
fire and miss it.
- `resolve_approval(timeout=True, approved=True)` now raises ValueError
to make the split-brain shape unrepresentable.
- Approval-timeout feedback string derives from `_APPROVAL_WAIT_TIMEOUT`
rather than the hardcoded "1 hour".
The previous comment described "\\" as "non-default", which is
backwards — "\\" is the SQL standard escape character. The
actually-non-default part is SQLAlchemy's ``.like()`` itself: it
defaults to no escape character, so ``escape_like``'s output is only
interpreted correctly when callers pass ``escape=LIKE_ESCAPE``
explicitly. Reword to put the caller-side requirement first.
WatchRunner._poll_watch committed active=False to the row BEFORE
calling _dispatch_result for a terminal fire, and the dispatch closure
registered by ChatSession.set_watch_runner enqueued each reminder with
a valid_until=is_watch_active predicate that re-read the row at drain
time. Since the runner already flipped active to 0, the predicate
returned False for every dispatched fire and NudgeQueue.drain silently
dropped the entry — the model never saw a watch result. Then a
subsequent action=cancel call hit list_watches_for_ws (filters
active==1), the now-inactive row was invisible, and the cancel
returned 'Watch "X" not found.' regardless of whether the watch had
actually run.
Reorder _poll_watch to dispatch before the row write, drop the
valid_until predicate from the watch closure (its only effect was the
bug above), and add a _terminal_dispatched guard on the runner so a
transient storage failure between dispatch and row-write doesn't
re-fire the reminder on the next tick. Add WatchRunner.forget_terminal_dispatched
and call it from the cancel path so an out-of-band deactivate (next_poll='')
doesn't leak the watch_id from the runner's pending-retry set indefinitely.
Cancel-by-name now routes through a new find_watch_by_name storage
method that ignores the active filter and prefers active rows over
newer-inactive same-name siblings. The session.py cancel branch
distinguishes 'already completed (auto-cancelled)' from 'not found'
so the model can tell apart 'this watch ran and finished' from
'no such watch.' Consolidate the two byte-identical _escape_like
/ _escape_ilike helpers in the storage backends into a single
turnstone.core.storage._utils.escape_like and apply it to the new
find_watch_by_name LIKE pattern so a model-supplied watch name
containing % or _ can't redirect a cancel to a sibling watch.
NudgeQueue.drain previously dropped predicate-failed entries without
logging anything, which is what hid this bug for so long. Drain now
emits nudge_queue.predicate_dropped: info for reason=predicate_false
(the normal lifecycle case — idle_children when every active child
finished between enqueue and drain), warning with exc_info for
reason=predicate_raised (a misbehaving predicate).
Tests: new test_poll_watch_terminal_fire_survives_drain (parametrized
stop_on_fired + max_polls_reached) drives the real WatchRunner._poll_watch
against a real tmp_db row and confirmed to fail against pristine main.
test_poll_watch_retry_deactivate_after_update_watch_failure exercises
the _terminal_dispatched retry-deactivate branch end to end.
test_cancel_clears_pending_terminal_dispatched_entry covers the cancel-
path leak case. test_find_by_name_prefers_active_over_newer_inactive
catches the ordering regression. test_find_by_name_treats_percent_as_literal
+ test_find_by_name_treats_underscore_as_literal pin the LIKE escape.
The shared_static exclude in scripts/update-vendored-js.sh was meant to
skip self-references inside vendored libraries, but it also hid
shared_static/renderer.js — which loads the vendored libs and pinned
mermaid-11.14.0 across every renovate bump since #426. Tests under
tests/test_web_helpers.py were similarly invisible because the include
list omitted *.py.
Replace the broad shared_static exclude with the specific old-versioned
vendor directory (about to be rm -rf'd next anyway), and add *.py to the
include list. Bump renderer.js to mermaid-11.15.0 to repair the live
404, and refresh the test fixtures to current vendor versions so they
stop drifting.
Line 3446 used double-quote string delimiters with an embedded ">
that terminated the string mid-attribute, leaving "bulk-revoke (" as
bare tokens. The rest of the surrounding block uses single-quote
delimiters; switch the broken line to match so the embedded > and "
sit safely inside the string.
The parse error wiped out every global in admin.js, so showAdmin and
the rest of the admin entry points were undefined — the console was
non-functional whenever an MCP server row had consented_users_count > 0.
Backports OAuth-MCP Phase 9 (#516) to the stable/1.5 track. Introduces
forward-only migrations 054_mcp_pending_consent and
055_mcp_user_tokens_server_index.
* feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9)
Completes the OAuth-MCP build-out (Phases 0-8 shipped) by closing the
operator + deferred-consent gaps:
1. **Per-(user, server) deferred-consent persistence** — when a
non-interactive run (scheduled / channel) hits ``mcp_consent_required``
or ``mcp_insufficient_scope``, the sync pool dispatchers now upsert a
row into a new ``mcp_pending_consent`` table. The dashboard hydrates
the gear-icon badge from this table on load, so users who weren't
online to see the in-flight SSE prompt still surface the deferred
work on next login. Cleared automatically by the OAuth callback
handler on consent completion; manual user dismiss via new DELETE
endpoints. Composite PK ``(user_id, server_name)`` collapses repeat
occurrences for the same server — no NULLs-not-distinct trap.
2. **Admin status pill + bulk-revoke** — the MCP Servers admin row now
shows ``consented_users_count`` for ``auth_type=oauth_user`` rows
when ≥1, with a two-step-confirm ``bulk-revoke`` button that drops
every user's token for the server via the existing
``delete_mcp_oauth_rows_by_server_name`` primitive. Upstream RFC
7009 revoke is intentionally NOT attempted in bulk (avoids N
upstream HTTP calls per admin click); audit detail records
``upstream_revoke_outcome=bulk_admin_no_upstream``. A "last
refresh" pill (age + outcome) renders on each row, sourced from a
new ``_last_refresh`` dict populated by ``_refresh_server`` on every
call (both manual ``refresh_sync`` and the ``_cb_auto_reconnect``
follow-up).
3. **ClientType.SCHEDULED** added to the prompts module + scheduler
passes it through to ``create_workstream``. ``ChatSession`` now
computes ``_is_interactive_for_consent`` at construction (WEB / CLI
are interactive; CHAT / SCHEDULED are not) and plumbs the flag
through ``call_tool_sync`` / ``read_resource_sync`` /
``get_prompt_sync`` to the three sync dispatchers. The wrap at the
``_is_structured_error`` gate routes consent codes to the new
``_record_pending_consent_best_effort`` helper for non-interactive
callers only; interactive sessions stay on the in-flight SSE path
Phase 8 ships unchanged.
4. **Operator docs** — ``docs/mcp-oauth.md`` (operator guide, parallel
to ``docs/oidc.md``: ``auth_type`` choice, OAuth client setup,
encryption-key rotation, troubleshooting matrix) and
``docs/operations/mcp-oauth-headless.md`` (one-paragraph runbook
per ``feedback_runbook_trust_llm.md``: pre-consent recipe for
scheduled / channel-driven runs).
Schema
- Migration 054_mcp_pending_consent.py — composite PK
``(user_id, server_name)``, ``occurrence_count`` + ``first_seen_at`` /
``last_seen_at`` for recency metadata, ``idx_mcp_pending_consent_user``
for the badge-load query. No FKs (matches the rest of the
oauth_user schema).
- Migration 055_mcp_user_tokens_server_index.py — adds
``idx_mcp_user_tokens_server`` on ``(server_name, expires_at)`` so
the admin pill's ``count_mcp_consented_users_*`` queries don't
full-scan against the leading-``user_id`` composite PK.
- Cross-backend: works on SQLite + PostgreSQL via dialect-specific
``on_conflict_do_update`` (PG ``postgresql.insert`` / SQLite
``sqlalchemy.dialects.sqlite.insert``). No ``NULLS NOT DISTINCT``
needed — the simplified PK eliminates the cross-version trap.
Endpoints
- ``GET /v1/api/mcp/oauth/pending`` — list deferred-consent records for
the authenticated user. Install-level gate via cached
``any_oauth_user_mcp_servers`` short-circuits to ``{pending: 0}`` on
installs with no oauth_user MCP servers — local-auth deployments
exercise zero new storage queries on this path. The gate result is
cached on ``app.state`` with a 60s TTL to spare repeat dashboard
loads.
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` — single dismiss.
Returns 204 in both existed-and-deleted and never-existed cases
(no cross-tenant existence leak); audits
``mcp_server.oauth.pending_consent_dismissed`` with
``mode=single`` + ``cleared=0|1`` so a session-hijack attacker
scrubbing breadcrumbs leaves an audit trail.
- ``DELETE /v1/api/mcp/oauth/pending`` — bulk dismiss; audits
``mode=bulk`` + ``cleared=N``.
- ``POST /v1/api/admin/mcp-servers/{name}/bulk-revoke`` — admin
bulk-revoke for the named server's per-user tokens. Requires
``admin.mcp`` permission + 400s when the row isn't ``oauth_user``.
All four registered on both ``turnstone-server`` and
``turnstone-console`` (mirrors the Phase 8 ``/connections`` endpoint
shape).
Performance
- Admin list handler now uses a single ``GROUP BY`` bulk-count query
(``count_mcp_consented_users_grouped_by_server``) wrapped in
``asyncio.to_thread`` rather than N per-row sync DB round-trips
inside the async handler. Skipped entirely when no row is
oauth_user.
Frontend
- ``ui/static/app.js``: ``loadPendingConsents()`` hydrates the
existing ``_pendingConsentServers`` set on dashboard init + after
the user opens the settings modal. Endpoint failures stay silent
— the badge will be re-driven by the next in-flight tool error.
- ``console/static/admin.js``: ``consented_users_count`` pill +
``bulk-revoke`` button on each MCP row (only when ≥1 consented),
two-step confirm matching the existing delete pattern. ``last-
refresh`` age + outcome pill in the per-row status cell, sourced
from the freshest per-node entry in ``status[*].last_refresh_at`` /
``last_refresh_outcome``. CSS for the pills in ``style.css``.
Tests
- ``test_mcp_pending_consent_storage`` — 13 tests covering upsert
idempotency, list ordering, per-user isolation, single/bulk delete,
count-by-server + grouped variant, install-level gate.
- ``test_mcp_pending_consent_dispatch`` — 9 tests, including the
boundary-cross gate per ``feedback_tests_through_boundaries.md``:
drives the real ``call_tool_sync`` → ``_dispatch_pool_sync`` →
``_is_structured_error`` → ``_record_pending_consent_best_effort``
with a mocked classified-lookup so the structural plumb-through is
verified end-to-end. Includes a storage-failure test that pins
the docstring's "envelope unchanged on storage failure" promise.
- ``test_mcp_pending_consent_endpoints`` — 11 tests: install gate,
list-for-self, no-cross-user-leak, single/bulk delete, idempotent
not-found, audit emission on single + bulk + cross-tenant dismiss.
- ``test_chat_session_interactivity_flag`` — 7 tests pinning the
``ClientType`` → ``_is_interactive_for_consent`` mapping against
the module-level ``INTERACTIVE_CONSENT_CLIENT_TYPES`` frozenset.
- ``test_mcp_admin_bulk_revoke`` — 7 tests covering admin.mcp
permission gate, 404 on missing, 400 on non-oauth_user, 200 with
``rows_deleted`` + ``consented_users_before``, audit row with
``upstream_revoke_outcome=bulk_admin_no_upstream``, cross-server
isolation.
- ``test_mcp_oauth_handlers`` — 2 new callback tests pin the post-
callback ``delete_mcp_pending_consent`` invocation: success-clears
+ storage-failure-still-redirects.
- 636 tests pass on the impacted surface (47 new + Phase 0-8 OAuth-MCP
+ session + prompts + storage admin). ruff + mypy clean.
Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}`` — the
flag flows only through the pool dispatchers, which only fire when
the row resolves to ``oauth_user``.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) preserved on every
AS / SDK / pool-loop await — no new awaits added to the hot path.
- Install-level gate on the badge endpoint: cached
``any_oauth_user_mcp_servers`` returns False on a row-less
deployment → endpoint short-circuits without touching the pending-
consent table; 60s TTL bounds the staleness window after admin
flips ``auth_type``.
- Operator-actionable codes (key-unknown, url-insecure, *_forbidden)
explicitly filtered out of persistence — they're outside the
user-facing consent badge scope.
- Best-effort write: the structured-error envelope returned to the
agent is identical whether the persistence write succeeds or fails
(storage exception is logged with type name only — no chained
context that could carry an ``httpx.Request`` bearer header).
- No ``exc_info=True`` on any new path that can chain a bearer-bearing
``httpx.Request``.
- Defensive parsing: ``_parse_pending_consent_envelope`` mirrors
``_is_structured_error``'s ``isinstance(decoded, dict)`` guard plus
filters scope tokens through ``is_valid_scope_token`` capped at
``MAX_INSUFFICIENT_SCOPE_REPORTED`` — defense-in-depth even though
production callers already validate upstream.
- Audit events on every dismiss endpoint so a session-control attacker
scrubbing dashboard breadcrumbs still leaves a trail.
Cross-backend
- Tested on SQLite via the conftest backend fixture.
- PostgreSQL path uses ``postgresql.insert(...).on_conflict_do_update``
parallel to the existing ``mcp_user_tokens`` upsert in Phase 3.
Deferred (not Phase 9 blockers)
- Multi-node pool eviction on bulk-revoke: only local-node sessions
would be evicted if we built it, and there's no bulk-by-server
primitive on MCPClientManager today; remote nodes will surface as
a 401 on next dispatch which refreshes through the (now empty)
token row.
- RFC 8693 / Azure OBO ``auth_type=oauth_token_exchange`` — captured
in the design doc as a future architectural direction (~600 LOC +
IdP-side admin work); requires OIDC token capture and per-MCP-server
resource-trust configuration that v1 does not ship.
* docs(mcp): address Copilot review feedback on Phase 9
- Fix misleading admin.js comment that claimed the refresh pill rendered
"<short-relative> <outcome>" — the pill actually renders only the short
age, with outcome reflected via CSS class and tooltip.
- Replace broken feedback_secrets_not_in_env.md repo-root link in
mcp-oauth.md with the inlined rationale (env-borne secrets reachable
via shell tools / os.environ; TOML secrets are not).
Adds notes for the 14 patches cherry-picked to stable/1.5 since 1.5.12:
reactive PG LISTEN/NOTIFY node discovery + event-driven wait_for_workstream,
memory tool audit trail, task_agent skill personas, plus fixes for the
LLM-visible default alias bypass, mermaid streaming parse errors,
proxy-prefixed re-auth, dashboard appbar visibility, and the PG test
backend on the notify dispatcher suite.
Introduces forward-only migration 053_services_notify_trigger.
- Put ``skill`` back in the access-denial list in the tool
description with a clarification — TASK_AGENT_TOOLS does not
include the skill tool, so sub-agents cannot switch personas
mid-task. Removing the disclaimer entirely created an ambiguity
the LLM could misread.
- Minimize the skill_data carried on the approval item dict to
``name`` / ``content`` / ``risk_level`` only. ``get_skill_by_name``
returns the full ~30-column prompt_templates row including
``scan_report``, ``installed_by``, ``source_url`` — none of those
flow through ``_exec_task`` / ``_evaluate_intent``, and they
shouldn't ride along any future audit serializer that reads the
approval item shape.
- Regression test for ``skill=""``, whitespace-only, and ``\t\n``
values — pins the documented "empty value is acceptable" contract
at the ``(args.get("skill") or "").strip()`` chokepoint.
The task_agent tool now accepts an optional ``skill=<name>`` argument
that loads the named skill's content as the sub-agent's persona,
substituting the hardcoded "# Task Agent" identity statement. The
operating-guidance numbered list (one-shot, tool-use over narration,
no follow-up questions) is layered on top of every persona and always
applies — those are sub-agent semantics that a persona should ride on
top of, not replace.
Validation lives in ``_prepare_task`` so the approval surface tells
the operator what they're consenting to: the validated skill dict
(including content) rides on the item dict from prepare to exec to
defeat TOCTOU between consent and execution. An unknown skill
returns a clean error item with a hint pointing at
``skill(action='search')``; a disabled skill returns a distinct error
so the LLM's recovery path can tell "not found" from "quarantined",
mirroring the enabled gate that ``_exec_skill(action='load')`` and
skill-search already apply.
High and critical skills now surface their risk tier on the approval
header (``, risk: critical``) and emit a
``task_agent.high_risk_skill`` warning — same signal ``_load_skills``
emits for session-level skills, so the operator sees the same flag
whether the skill is loaded session-wide or per-call. ``_exec_task``
emits a ``task_agent.skill_invoked`` info log on the skill branch for
forensic traceability — the approval row captures the choice at
consent time, this log captures it at exec time so post-incident
search doesn't have to cross-walk approval and exec tables.
The ``_evaluate_intent`` func_args projection now includes the skill
name — without it, heuristic ``arg_pattern`` rules targeting a risky
persona name on ``task_agent`` silently no-op and the audit row loses
the choice. Mirrors the long-standing ``spawn_workstream``
projection.
Caught by Copilot on PR #514. openSettingsMenu sets _settingsMenu
synchronously, but the menu's keydown handler was registered inside
setTimeout(0). The previous-commit guard in the global keydown
handler returns early when _settingsMenu is set (so dashboard isn't
hidden by Escape over the menu), which created a window where
Escape had no handler at all — the global skipped, the menu's own
listener wasn't ready yet, and the menu got stuck open until the
next interaction.
Attach keydown synchronously; keep mousedown + initial focus in
setTimeout (mousedown to avoid the opening click triggering its own
outside-click close, focus because the menu DOM needs a tick to
settle layout).
Two related changes that surfaced when the user pointed out the proxy's
node-picker pill was unreachable from the proxied dashboard view: the
dashboard overlay was covering the entire appbar.
- Dashboard overlay now starts at top: 48px so the appbar (with the
proxy-injected node picker) stays visible and interactive while the
dashboard is open. showDashboard no longer marks ui-header inert
(tab-bar and split-root still are). The dashboard's role downgrades
from dialog+aria-modal to region — the appbar being reachable above
it would otherwise contradict aria-modal's "ignore everything else"
semantics.
- Gear icon converts from a direct openSettingsPanel() click into a
dropdown menu with two items: "MCP connections" (existing modal) and
"Logout". Reuses the .ws-tab-dropdown shell for visual consistency
with the workstream tab chevron menu and the proxy node-picker.
Logout uses .destructive styling to reduce misclick risk.
Bug fixes caught by the merged code-review pipeline:
- Global Escape handler skips when _settingsMenu is open, otherwise it
fires hideDashboard() before the menu's own handler — wiping the
composer text + staged attachments out from under the user.
- Menu-item click refocuses the trigger before close, so
openSettingsPanel captures the gear (not <body>) as the eventual
return-focus target.
- ArrowUp keyboard cycling uses idx <= 0 ? len - 1 : idx - 1 instead
of (idx - 1 + len) % len so the no-focus case wraps to the last
item rather than the second-to-last. Same fix backported to
showTabDropdown which had the identical modulo bug.
- Position clamps reordered: right-edge override now runs before the
left-edge floor so a menu wider than the viewport still clamps to
mx >= 4 instead of going negative.
- openSettingsMenu caches _settingsMenuTrigger so closeSettingsMenu
can reset ARIA without re-querying the gear by id.
- aria-controls lifecycle wired both ways (set on open, removed on
close).
The LLM was passing ``task_agent(model="default")`` (and the same for
plan_agent) and routing to whichever backend the auto-created
``default`` alias was attached to at boot — flatspark in the verified
case (ws_id 7dde674) — silently bypassing the operator-configured
``model.task_alias`` / ``model.plan_alias`` (gh200).
Root fix:
- ``load_model_registry`` only synthesises the back-compat ``default``
alias when neither DB nor ``[models.*]`` populate the registry. The
shim was only ever meant for single-CLI-model setups; with a multi-
model DB it became a phantom routing target aliasing ``LLM_BASE_URL``.
- ``_render_agent_tool_descriptions`` filters ``default`` out of the
LLM-visible alias list. The English reading of "default" trips the
model into picking it explicitly even when the description tells it
to omit ``model=`` for the per-role default.
Defense-in-depth at the validator chokepoint
(``_validate_agent_model_override``): explicit rejection of
``alias == "default"`` (post-strip) with corrective guidance;
``default`` filtered out of the unknown-alias retry list so an LLM
probing with a bogus alias can't enumerate it back; the no-alternatives
wording is distinguished from the no-registry-configured wording. The
render path also always rewrites tool descriptions instead of returning
early on filter-empty, so a reload that drops the registry to only
``default`` clears stale alias names left over from a prior render.
Previously only the admin-console DELETE route emitted memory.delete
audit rows, so a long-running session whose memory was deleted via
the admin UI had no log trail showing what happened — masking
out-of-band deletes as apparent tool bugs.
The save branch now stamps memory.save (new row) or memory.update
(upsert); the delete branch does a lookup-then-delete-by-id pair so
the audit can record the resolved memory_id and type. All emissions
are best-effort: failures log at debug and swallow so an audit hiccup
never breaks the tool call itself. Reads (get/search/list) remain
un-audited.
Copilot review on #511 flagged that the dispatch chain and the tests
both claimed to be in lockstep with one another, but only the comment
text said so — the parametrize list and the if/elif chain were two
independent hand-maintained copies, and the comments still referenced
the (long-reverted) ``_PROXY_AUTH_LOCAL_HANDLERS`` symbol.
Make the lockstep guarantee real by collapsing both copies onto one
``_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str]`` mapping
``(method, path)`` to handler-name strings. ``proxy_api`` resolves
the name through ``globals()`` at call time so ``patch(...)`` in
tests still observes the override — a dict of function refs would
have captured the originals at module load (which is why the first
attempt at this dispatch broke the tests and got reverted). Test
cases now derive directly from ``_PROXY_AUTH_LOCAL_HANDLERS.items()``,
so adding or removing an entry in the dispatch table flows through
to the parametrize list automatically and the two can't drift.
When the user is on a proxied node page (``/node/{id}/...``) and the
JWT expires, the in-page login modal POSTs to ``/v1/api/auth/login``
which the proxy shim rewrites to ``/node/{id}/v1/api/auth/login``.
Two latent bugs both had to be fixed for the user to be able to
re-authenticate from inside the proxied UI:
1. ``is_public_path`` didn't recognise the ``/node/{id}/`` prefix
over a public path, so the console's ``AuthMiddleware`` 401'd the
login POST before any handler ran. Extended via the existing
``_extract_proxied_path`` helper so a proxied public path stays
public.
2. Even if the path had been public, ``proxy_api`` would have
forwarded the request to the upstream node. The upstream mints
``JWT_AUD_SERVER`` tokens; the console's ``AuthMiddleware``
(expecting ``JWT_AUD_CONSOLE``) would reject those on the next
proxied call, and ``_proxy_post`` drops ``Set-Cookie`` when
forwarding anyway. ``proxy_api`` now dispatches every entry in
``_PROXY_AUTH_LOCAL_PATHS`` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's
own auth handlers, and short-circuits non-canonical methods on
those paths with 405 instead of letting them slip through with
the service-token fallback.
Tests parametrize across all eight local-dispatch entries so a future
refactor that drops a branch (or routes it through ``_proxy_post``)
fails loudly, plus a no-auth-header reproduction for the original
lockout and a 405 regression guard for the method-mismatch surface.
* fix(renderer): mermaid streaming parser errors + progressive hljs
Live streaming was rendering mermaid diagrams with `Parse error,
got 'PS'` messages — bare `(`, `[`, `{` inside unquoted edge / node
labels re-entered Mermaid's shape parser. Two unrelated streaming-
specific issues in the renderer pile-up here; this commit addresses
both plus a follow-on UX improvement for code highlighting.
## Mermaid label autoquoter
`_normalizeMermaidSource` wraps two label forms that Mermaid rejects
when they contain bare shape-delimiter chars:
1. Edge labels: `|content|` → `|"content"|`
2. Rectangle node labels: `ID[content]` → `ID["content"]`
Shapes whose syntax already nests delimiters — cylinders `[(...)`,
subroutines `[[...]]`, trapezoids `[/.../]` `[\...\]`, circles
`((...))`, hexagons `{{...}}`, diamonds `{...}` — are intentionally
left alone (their inner delimiters are part of the shape syntax;
quoting would corrupt them). Labels already wrapped in `"..."` are
also left alone. The rewrite is idempotent and runs before the
mermaid SVG cache lookup so identical malformed input hits the
cache on re-render rather than re-quoting per tick.
## Markdown fence-pair regex
The old fence regex `/(```+)([^\s`]*)\n([\s\S]*?)\1/g` would, mid-
stream, pair an unclosed ```mermaid open with the OPENING backticks
of a later ```python fence as the "close", handing mermaid a
truncated source. New regex:
/(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g
Two constraints close the gap:
- `(?!\1)` inside the content quantifier blocks the lazy matcher
from extending across another N-backtick run. Smaller inner
counts (e.g. 3-backtick inner inside a 4-backtick outer) still
pass since `\1` is the open's actual count.
- `[ \t]*(?=\n|$)` after `\1` forces the close to a line
boundary, so ```python (open with a language tag) can't
masquerade as a previous fence's close.
Together: an unclosed fence stays as plain markdown until its true
close arrives, so neither mermaid nor hljs ever sees a mid-stream
truncated source.
## Progressive hljs
Extracted `postRenderHljs` from `postRenderMarkdown` with a source-
keyed `_hljsCache` (FIFO, cap 64, keyed on `language:source`) and
wired it into `_streamingRenderApply`. Closed code fences are now
syntax-highlighted as they stream in, matching the progressive
mermaid pattern from #426. Per-tick cost stays cheap because the
cache returns the pre-tokenized HTML synchronously on hit; only
unique (language, source) pairs pay `hljs.highlightElement`.
## Internal cleanup from the review pipeline
- `_cacheFifoEntry(cache, key, value, max)` replaces the duplicated
`_cacheHljsEntry` and `_cacheMermaidEntry`. Single tested
implementation across four caches (hljs, mermaid svg, mermaid
error, mermaid normalize memo). The "don't evict on overwrite"
invariant is pinned per-cache in tests.
- `_mermaidNormalizeCache` memoizes raw textContent → normalized
output so the per-rAF-tick autoquoter split + regex doesn't
repeat for unchanged diagrams. Eviction shares
`_MERMAID_CACHE_MAX` with the SVG cache it feeds.
## Tests
The fake DOM in tests/test_renderer_js.py grew a few capabilities
to drive these paths:
- `classList` is now array-like (length + indexed access) so the
hljs language-extraction loop works.
- `textContent` setter mirrors the real-DOM side effect of
entity-escaping into innerHTML, so `escapeHtml()` round-trips
(otherwise every `renderMarkdown` returns empty `<p>` tags).
- `querySelectorAll` handles both `pre code.language-mermaid`
and `pre code[class*='language-']`.
Added: 6 fence-pairing regression cases, 9 hljs-progressive cases
(cache hit / distinct sources / language separation / NO_HIGHLIGHT
langs / terminal class / eviction / overwrite / postRenderMarkdown
wraps hljs / _streamingRenderApply invokes hljs), 11 autoquoter
cases including both diagram sources from the live screenshot
encoded verbatim as parametrized regressions, and 3 normalize-memo
cases (populates on first call, consulted before normalize via
sentinel pre-seed, distinct sources cache separately).
Total: 104 renderer tests pass (was 67).
* fix(renderer): apply Copilot review feedback on #510
Two doc / harness adjustments from the PR review — no behavior
change in production code.
- The `_mermaidNormalizeCache` comment claimed eviction "stays in
lockstep with the SVG cache". That was misleading: the two
caches key on different things (raw textContent vs normalized
source) and evict independently. Updated the comment to describe
what they actually share (the cap, for memory footprint) and
what they don't (positional coupling), and to note that the memo
deliberately survives `_initMermaid` since normalize output is
theme-independent.
- The fake DOM in tests/test_renderer_js.py had `innerHTML` setter
clear `children` but leave `_textContent` intact, so subsequent
`textContent` reads could return stale data after an innerHTML
mutation (real DOM invalidates textContent on innerHTML write).
No current test triggered this, but it would mask future bugs
that depend on innerHTML/textContent consistency. Setter now
clears `_textContent`; the children-derived fallback in the
getter returns `''` after the wholesale replace.
All 104 renderer tests still pass; ruff + mypy clean.
CI's postgres-backend run failed 11 of the new notify tests from #505.
Three independent issues:
1. Migration 053's ``services_notify`` trigger lives only in the
alembic chain, but the test fixture in conftest.py calls
``init_storage(..., run_migrations=False)`` for speed. That path
skips migrations and relies on ``metadata.create_all`` for the
table tree. Previous alembic-only DDL (migrations 041 / 048
``CREATE INDEX CONCURRENTLY`` on workstreams) is performance-only,
so tests never depended on it. 053's trigger is the first
behaviorally-required alembic-only DDL in the project — without it
``register_service`` doesn't fire NOTIFY and the trigger-filter
tests time out.
Fix: declare the trigger function + trigger in ``_schema.py`` and
attach them via ``sa.event.listen(services, "after_create", ...)``
DDL events, gated on ``dialect == "postgresql"``. The same SQL
constants are imported by migration 053 so there's a single source
of truth. Test fixture stays unchanged — ``create_all`` now
installs the trigger on fresh PG test DBs. Migration covers the
upgrade-on-existing-DB path; the two are mutually exclusive given
``create_tables = not run_migrations`` in ``init_storage``.
2. NotifyDispatcher tests fired ``storage.notify(...)`` immediately
after ``d.start()`` and hit a race: the listener thread is
concurrently calling ``psycopg.connect(listen_url)`` + ``LISTEN
<channel>`` over the network, so the notify can land before any
session is listening on the channel and PG drops it (pg_notify
only routes to sessions LISTEN'ing at COMMIT time).
Fix: dispatcher gains a ``_listener_ready: threading.Event`` set
inside ``_listener_loop`` after each successful ``storage.listen``
open and cleared on disconnect, plus a public
``wait_until_ready(timeout)`` method. Tests use a new
``_start_ready(d)`` helper that calls ``start()`` + asserts ready.
Production callers don't need this (real reactive traffic arrives
well after startup), but it's the right primitive for any future
"start dispatcher, immediately send" call site too.
3. ``TestSqliteNotify`` is misnamed — its tests run against whichever
backend the ``storage`` fixture provides (PG by default in CI).
Two of its assertions were SQLite-specific:
``assert got.pid == 0`` only holds for the synthetic in-process
path (PG carries real backend PIDs), and
``test_synthetic_sweep_emits_after_interval`` is fundamentally
SQLite-only (no sweep on the PG path).
Fix: drop the pid assertion (channel + payload are the
backend-agnostic invariants), add an ``_is_sqlite`` fixture mirror
of ``_is_postgres``, and gate the sweep test on it. The sweep
test also moves from monkey-patching ``stream._sweep_interval`` to
passing the ``sweep_interval`` kwarg that ``SQLiteBackend.listen``
now accepts (from the earlier Copilot review fix).
Validated locally against a fresh ``turnstone_test`` PG DB: 263
storage + console + notify tests pass on PG, 257 on SQLite, mypy +
ruff clean.
Retire two polling patterns in coord that have clean event sources.
PR 2 of 3 in the coord-completion stack; sits on top of PR #505
(reactive node discovery via PG LISTEN/NOTIFY).
`wait_for_workstream` (coord's block-wait tool) polled storage every
0.5 s in a worker thread regardless of whether anything had changed —
a 600 s wait incurred ~2400 round-trips. Now subscribes to a new
in-process `ChildEventBus` (`turnstone/core/child_event_bus.py`) and
blocks on `threading.Event.wait(min(remaining, WAIT_HEARTBEAT_INTERVAL))`:
- `CoordinatorAdapter` owns the bus; `_dispatch_child_event` calls
`bus.notify(child_ws_id)` after each `_enqueue_on_ui` for the
state-class branch (cluster_state, ws_closed, ws_rename,
intent_verdict, approval_resolved, approve_request).
- Wait loop clears the Event BEFORE the storage snapshot to close
the subscribe/check race; a notify between clear and the next
`wait()` leaves the Event set so the loop re-reads without
losing the wake-up.
- 2 s heartbeat cap preserves the existing `wait_progress` SSE
cadence for the sidebar UI while cutting SSE traffic ~4x vs the
pre-bus 500 ms cadence in the quiescent case.
- Worst-case completion latency is 2 s (vs pre-bus 0.5 s) because
`set_state` buffers non-ERROR writes through `StateWriter`
(async-flushed) while `emit_state` fans out immediately — a
bus-driven wake can beat the flusher and read pre-transition
state, then re-block until heartbeat. Deliberate trade-off; the
SSE-traffic reduction outweighs the regression on the most
common terminal transition.
- Defense-in-depth: ownership-filter `cleaned` to own-subtree
before `register_waiter` so a foreign ws_id passed by an
untrusted coord LLM (prompt injection) can't observe wake-up
timing as a side channel. Predicate (`_row_in_own_subtree`)
requires both `parent_ws_id == coord_ws_id` AND `user_id ==
coord_user_id` parity — same gate strength as the existing
`_is_own_subtree` mutating-op guard, so a corrupted /
cross-tenant `parent_ws_id` alone can't satisfy it. Shared with
`_snapshot_all` so the snapshot's `denied` shape stays in
lockstep with the bus filter (Copilot review on #506).
Coord idle-cleanup thread polled the storage scan every
`check_every` seconds (~30 s on default 2 h timeout) even when no
coord was anywhere near idle. Now subscribes to
`SessionManager._state_subscribers` with a `tick_now` event and
blocks on `tick_now.wait(check_every)` — any state change wakes
the sweeper without waiting a full interval, AND the timeout still
fires the periodic sweep for the DB-orphan-only case. A
`min_sweep_interval=5 s` floor bounds DB-call traffic at ~0.2/s
under sustained activity so the loop can't tight-spin `close_idle`
at the rate of its own DB latency (6x improvement over the
pre-refactor fixed 30 s cadence under any activity, and prompt
state-change-driven wakes when below the floor).
`CoordinatorClient` constructor takes `child_event_bus` as a
required kwarg — there's no external SDK shape to preserve and
keeping it optional would silently mask a wiring bug in any future
caller. Tests construct their own `ChildEventBus()` per fixture.
Tests: 16 unit tests for `ChildEventBus` (register / unregister
symmetry, multi-waiter fan-out, multi-child waiter, subscribe/check
race, concurrent register / notify smoke); 7 new adapter tests
(bus notify fires for all 6 state-class events, drops for unknown
child / wrong ws_id); 7 new coord-client wait tests (subscribe-
after-terminal, notify wakes, unrelated notify doesn't wake,
heartbeat fires without notify, unregister on exit, multi-waiter
independence, cross-tenant denial via the user_id-parity filter);
8 idle-cleanup tests (initial sweep, heartbeat cadence, exception
swallowing, stop_event clean exit, state-change wake, subscriber
cleanup, mid-sweep wake, `min_sweep_interval` floor). All pass;
ruff + mypy clean. Full non-live suite: 6227 passed (+2 vs prior
baseline).
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher
Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL
`LISTEN` connection and fans wake-ups out to per-channel handlers on a
separate dispatch thread. Cluster collector subscribes to a new
`services` channel and runs node discovery reactively — new-node /
graceful-deregister visibility drops from up-to-60 s to ~500 ms on
Postgres, with the 60 s discovery loop retained as the backstop for
crash-shaped node loss (NOTIFY only fires on real writes).
Storage layer gains a uniform `notify` / `listen` API:
- PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode
connection that bypasses pgbouncer (mandatory: pgbouncer is required
in transaction-pool mode per docs, which is incompatible with LISTEN).
- SQLite: in-process fan-out + synthetic-sweep fallback so consumer
code is identical across backends.
`TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml)
points the dispatcher's connection direct-to-Postgres. Defaults to the
main DB URL when unset.
Migration 053 installs the `services_notify` trigger; it filters
heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick
stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE
still fire.
Dispatcher detail:
- Two threads: listener (drains stream → bounded queue) and dispatch
(invokes handlers under exception suppression). Same-channel notifies
coalesce per dispatch batch so an N-node deploy burst is one
`_discover_nodes` per channel.
- Reconnect uses exponential backoff (1 s → 30 s cap). After any
successful reopen — whether the prior failure was a stream-poll error
or a connect / initial-LISTEN error — one synthetic Notify with
payload="reconcile" is enqueued per channel so handlers re-read on
the same code path they use for real events.
Future consumers (ConfigStore live reload, scheduler immediate
dispatch, audit live-tail) plug in by adding their channel to the
dispatcher's construction list.
Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub
storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter
tests, 4 collector wire-in tests. All pass; ruff + mypy clean.
* fix(notify): address Copilot review on #505
- _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via
dict.fromkeys before constructing the stream — duplicates would
otherwise register the queue twice and double-deliver each notify.
- _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval
parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what
the comment at the constant already promised, and lets future
consumers without their own polling timer pick a tighter cadence
without reaching into private stream attributes.
- _sqlite.py: documented the `except queue.Empty: pass` end-of-drain
termination so it's not mistaken for swallowing an unexpected error.
- _postgresql.py: docstring referenced :func:`_pg_listen_url` which
was renamed to _resolve_pg_listen_url during PR development.
- notify_dispatcher.py: module docstring referenced a non-existent
_bootstrap_console_subsystem; wire-in is at console/server.py::main.
Refuted (no change, false positives from github-code-quality bot):
- 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies
(idiomatic Python Protocol declaration, not dead code).
- 2× "Mixed import style" in tests — `import ... as nd_mod` is
intentional to allow attribute assignment for monkey-patching the
module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally.
Converts [Unreleased] to [1.5.0] and adds individual sections for
1.5.1 – 1.5.12. Covers: MCP OAuth 2.1 + PKCE (Phases 1–8), OIDC
hardening, metacog NudgeQueue + wake trigger, SSE refresh-resume,
reasoning persistence (Phases 1–4), structured watch-result cards,
skills unlock, inline child approvals, Stage 3 Children primitive
lift, coordinator composer parity, node capability auto-detection,
progressive mermaid rendering, and the full schema migration list
for each release.
Updates the track list to stable/1.4, stable/1.5, and main.
Editing the first message in a workstream sends /rewind N where N is the
total user turns, leaving session.messages empty. The handler guarded the
history event with `if history:`, so only clear_ui was emitted. The
frontend dispatches the queued edit-and-resend from the history event
handler (app.js _pendingEditSend), so an empty history orphaned the
pending text and left the composer stuck in busy.
replayHistory already handles the empty case via showEmptyState(), so
emitting the event unconditionally is safe and unblocks the dispatch.
A bare ``httpx.ReadTimeout`` previously surfaced as ``ReadTimeout: timed
out`` — no provider, no base URL, no model — leaving the user with no
signal to tell whether a model server hung, the URL was wrong, or the
model isn't loaded on the backend.
``ChatSession._format_backend_error`` now rewrites known boundary
exceptions (httpx ``ReadTimeout`` / ``ConnectError`` / etc. and OpenAI /
Anthropic SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``) into
operator-actionable text that names the provider, base URL (query
string stripped before ``sanitize_error_text`` redacts credentials),
and model. Matching is by class name so the helper carries no SDK
imports. Unrecognised exceptions fall through to the legacy
``f"{type(exc).__name__}: {exc}"`` shape, preserving existing grep
targets.
The Anthropic call sites in session.py passed the operator-side
`replay_reasoning_to_model` flag through without checking the
model's static `supports_reasoning_replay` capability. The OpenAI
Responses path AND-gated both flags in `_build_kwargs` so a model
without a reasoning lane (gpt-4o, etc.) silently skipped replay even
when the operator flag was set. The Anthropic path had no such gate.
For all current Claude entries this was a no-op asymmetry - every
`_ANTHROPIC_CAPABILITIES` row sets `supports_reasoning_replay=True`,
so `True AND op == op`. But:
- The capability flag was dead code on the Anthropic path
- A future Claude entry (or any Anthropic-shaped surface) shipping
with the cap left at its False default would have replay fire
anyway, against the cap declaration
- The asymmetry made `supports_reasoning_replay` an unreliable
signal - readers couldn't tell if it gated anything per-provider
Move the AND-gate into `_resolve_replay_reasoning_to_model` via a
new optional `caps=` kwarg. When caps is provided, the resolver
returns `operator_on AND caps.supports_reasoning_replay`; when
omitted (back-compat for any caller not yet updated), it returns
the operator flag unchanged.
Thread caps through the three call sites: `_utility_completion`
(non-streaming), `_try_stream` (streaming, hoisted resolution out
of the retry loop since caps are attempt-invariant), and the
agent `_api_call` closure in `_run_agent`.
With the AND-gate now living at the session resolver, the redundant
in-provider gate in `OpenAIResponsesProvider._build_kwargs` is
removed. The provider now trusts the resolved bool it receives,
matching the AnthropicProvider shape and giving the cap a single
source of truth across providers. The two provider-level tests
that pinned the in-provider gate
(`test_include_omitted_when_capability_false`,
`test_include_omitted_by_default`) drop out; the session-level
boundary test
`TestSessionToOpenAIResponsesBoundaryIntegration::test_capability_false_omits_include_even_when_flag_true`
already covers the same end-to-end invariant.
Tests added:
- 4 resolver-level tests pinning the AND-gate semantics +
back-compat when caps is omitted
- 1 wire-boundary integration test mirroring the OpenAI Responses
`test_capability_false_omits_include_even_when_flag_true` -
drives session._try_stream through the real AnthropicProvider
with operator flag True + capability False and asserts the
thinking block does NOT reach the SDK boundary
Existing `TestUtilityCompletionPassesFlag` test had its caps mock
upgraded from `SimpleNamespace` to a real `ModelCapabilities`
instance to satisfy the new attribute read and stay robust to
future capability fields.
Copilot review feedback on #500. The original
``list_available_models`` had an implicit cs=None branch where the
placeholder still advertised ``registry.default`` (filtered against
enabled rows) when ``app.state.config_store`` was None but
``coord_registry`` was bound — useful in the rare degraded state
where lifespan wired the registry but the ConfigStore failed to
initialise. The PR #500 refactor accidentally dropped that branch:
the helper requires a config_store, so the cs=None case fell out as
"blank coordinator default".
Add an explicit ``elif coord_registry is not None`` branch that
mirrors the helper's tier 3 with the placeholder's enabled-rows
filter applied. New test exercises this path by passing
``config_store=False`` to the test fixture.
Previously /v1/api/models (home composer placeholder) and
console/session_factory.py walked separate two-/three-tier chains for
the coordinator alias. session_factory was missing the
``model.default_alias`` tier, so admins who set the system default in
the Models tab would see it advertised but new coordinator sessions
would silently keep launching on ``registry.default``.
This commit:
- Extracts the chain into ``turnstone/console/coordinator_alias.py``.
``resolve_coordinator_alias`` returns the effective alias under a
shared three-tier policy: explicit pin → ``model.default_alias`` →
``registry.default``. Tier 2 is validated against
``registry.has_alias`` and falls through to tier 3 with a logged
warning if unknown. Tier 1 is intentionally passed through
unvalidated so an explicit operator pin surfaces as 503 at
``registry.resolve`` rather than being silently swapped out.
- Wires both call sites through the helper. The placeholder supplies
an ``alias_filter`` that restricts every tier to enabled DB rows so
the home composer never advertises a model the workstream picker
can't actually offer; the session factory uses no filter (matches
prior 503-on-typo behaviour for explicit pins).
- Adds direct integration tests for the session factory's chain
(``tests/test_console_session_factory.py``) and updates the
placeholder tests' fixture to provide a stub coord_registry, since
the helper now requires one.
Light-review followup on 389400c8.
The "mirrors session_factory.py:109-110" claim was inaccurate —
session_factory's chain is two tiers (coordinator.model_alias →
registry.default) and skips model.default_alias entirely. The
placeholder handler extends that chain with model.default_alias as
tier 2 so admins who set the default in the Models tab see it
advertised in the home composer. Comment now lists the three tiers
explicitly and flags the session_factory-vs-placeholder drift case
(where model.default_alias ≠ registry.default) as a separate issue
to track.
Also lifts the ``from types import SimpleNamespace`` import in the
test fixture to module level — minor readability cleanup.
Two Copilot-review followups on /v1/api/models default resolution.
- console/server.py: coordinator_default_alias now mirrors the full
fallback chain in console/session_factory.py:109-110 — explicit
coordinator.model_alias → model.default_alias → registry.default.
The registry tier was missing, so the home composer placeholder went
blank whenever an operator never set model.default_alias in the admin
UI even though new coordinator sessions still launch on
registry.default (loaded from config.toml [model].default by
load_model_registry). Two new tests cover the registry-default
branch and the disabled-alias guard.
- console/static/app.js: _resolveModelLabel returns "" (not the bare
alias) when the alias isn't found in the dropdown's model list, so
callers can rely on the documented "fall back to neutral placeholder"
contract. Matches the existing doc comment.
Bundles the click-around polish on the console admin UX.
Home composer + schedule modals
- /v1/api/models now exposes coordinator_default_alias + judge_default_alias,
resolved through the same chain console/session_factory.py uses. Both the
home composer's MODEL / JUDGE MODEL placeholders and the schedule create /
edit modal model placeholders rewrite to "Default — alias (model)" once
the API responds. The `models_changed` SSE refresh keeps placeholders
current as operators edit per-role assignments.
- Composer.setOptionPlaceholder added so callers can update just the first
option's text without disturbing the rest of the choice list.
Admin → Models → Roles
- Channel adapter row added (channels.default_model_alias) — the migration
to the Roles sub-tab missed it. Key added to
_MODEL_AFFECTING_SETTING_KEYS so edits fire the SSE refresh, and to the
settings-tab roleKeys skip-list so it only renders in one place.
- Plan/Task agent rows now display "(inherit)" instead of the misleading
"(default — <alias>)" — those roles cascade through plan_model →
agent_model → session model, not a single concrete default.
- coordinator.reasoning_effort accepts "" (inherit), matching
model.plan_effort / model.task_effort.
- Blank options in each role's MODEL select now match the "alias (model)"
shape used by the other rows.
Toggle-switch component
- New .toggle-switch component (visually-hidden native checkbox + styled
track + label). 40×22 hit target meets WCAG 2.5.5 (AAA), inset ring on
the off state for ≥1.5:1 contrast against the modal surface.
- .toggle-stack groups toggles in a column with .toggle-group-divider for
conceptual grouping (used in the Add Model modal between "Active" and the
paired Reasoning toggles).
- .toggle--flush modifier zeroes the default top margin for toggles that
sit flush against a heading or a dynamically-rendered row.
Sweep — every admin-modal boolean checkbox is now a toggle:
schedule (cs/es-autoapprove, es-enabled), policy (ep/epp-enabled),
tool-mode (ctm/etm-default), skill (csk/esk-auto-approve, csk/esk-enabled),
MCP (mcp-auto-approve, mcp-enabled), Add Model (Active, surface-persisted-
reasoning, replay-reasoning), judge bool settings (cancel_on_approval et
al.), and the user-roles-modal role assignment list. The two
ogp-cred / eogp-cred inline credential checkboxes stay as compact inline
boxes since they sit beside text inputs in tight horizontal rows.
Add Model modal — the "Enabled" toggle promoted to "Active" and moved to
the very top of the form. Tooltip explains it gates dropdown visibility
without removing the definition.
MCP authorization — the three radio buttons replaced with a vertical
.segmented-control option list. Selected row paints --accent-dim plus a
filled .segmented-indicator; focus ring uses --accent so it stays visible
on the currently-selected option.
Role permissions modal — the 19 permission checkboxes are now
.toggle-switch.perm-toggle (monospace lowercase identifiers preserved).
The permissions are split into Scopes / Admin / Workstreams & Tools
sections under caps-styled section headers so the row-flow grid no longer
slices `admin.*` mid-column.
Judge bool toggles use a static "Enabled" caption rather than flipping
text on `.checked`; flipping lagged 50–300 ms behind the slider position
because the caption was sourced from the post-save reload.
CSS cleanup — dead `.admin-checkbox` / `.perm-checkbox` rules removed.
Specificity audit (scripts/css_specificity_audit.py) returns no conflicts
on any new component class.
Tests — 525 pass on the affected slices; new tests/test_console_available_
models.py pins each branch of the resolution chain in /v1/api/models so the
home composer placeholder stays correct as precedence rules evolve.
`IntentJudge.__init__` previously had a 3-way resolution chain: registered
alias → raw model id pinned onto the session provider → session model. The
middle branch was a footgun documented in `console/session_factory.py:130-137`
— pinning the literal `judge.model` string onto the coordinator's session
provider silently broke every verdict whenever that provider didn't recognise
the model id (e.g. coordinator on Anthropic, `judge.model = "gpt-5-mini"` →
uniform `llm_fallback`).
Tightens to alias-only, matching `coordinator.model_alias` /
`model.plan_alias` / `model.task_alias`. An unknown `config.model` now logs
a warning and inherits the session model — same path as empty. Help text on
`judge.model` updated to clarify the contract.
Adds two regression tests in `TestModelAliasResolution` covering the
session-model inheritance for unknown values and the empty-model self-
consistency case.
GoogleProvider attaches raw tool_call dicts as ``provider_blocks`` on
the finish chunk for ``thought_signature`` round-trip
(``_google.py:_iter_stream``). When the same turn streamed Gemini's
``reasoning_content`` as ``reasoning_delta`` chunks, the prior
synthesizer bailed out the moment ``provider_blocks`` was non-empty
— so the captured reasoning was visible live but lost on page reload.
Replace the early-return-if-non-empty check with a reasoning-bearing
type test (``thinking`` / ``redacted_thinking`` / ``reasoning`` /
``reasoning_text``). When none of those types appear, append the
synthetic ``reasoning_text`` block to the existing list rather than
replacing it — preserving Google's tool-call fidelity blocks.
Also addresses two doc-accuracy review findings:
- ``LLMProvider.extract_reasoning_text`` docstring no longer claims
OpenAI Chat / Responses are unwired (Phase 3+4 shipped extractors).
- Add the method to the Protocol methods table in
``docs/architecture.md`` (was missing alongside the class diagram).
The earlier all-or-nothing shape check on ``_provider_content`` discarded
every valid Anthropic block in a message the moment a single foreign
block (OpenAI ``reasoning``, Gemini thought parts, the synthetic
``reasoning_text`` from path-3 capture) appeared. In the cross-model
resumption edge case that meant ``server_tool_use`` /
``web_search_tool_result`` blocks lost their ``encrypted_content``
silently, breaking web-search round-trip continuity on subsequent turns.
Replaced with a per-block walk: foreign blocks are dropped individually,
valid blocks ride the verbatim path, and an identity-preserving fast
path reuses the source list reference when nothing was filtered or
stripped (pinned by the ``is`` assertions in test_providers.py).
Also addresses validation-pass review findings:
- Document the single-tier vs three-tier ``surface_persisted_reasoning``
resolution divergence between server.py:_build_history and
session_routes.make_history_handler.
- Document why OpenAIResponsesProvider._convert_messages defaults
``replay_reasoning_to_model=False`` while Anthropic's defaults True.
- Document the ``source`` metadata field on synthetic ``reasoning_text``
blocks as reserved-for-future-use, not dead code.
- Add edge tests for non-dict / missing-type-key blocks in
_provider_content (defensive branches in the per-block walk).
CI test job installs `[test]` extras, which omits `anthropic`. The two
TestSessionToWireBoundaryIntegration cases drive the real
AnthropicProvider.create_streaming, which calls _ensure_anthropic() and
raises ImportError. Match the repo convention (test_channel_discord,
test_channel_slack, test_tls_*) by gating the helper with
pytest.importorskip("anthropic").
PR #498 round-robin review surfaced 5 findings. 4 applied; 1 rejected
with rationale.
Applied
* **Copilot finding 5** (history_decoration.py:341): dispatcher
inspected only ``provider_content[0]['type']``. OpenAI Responses
captures EVERY ``output_item.done`` event into ``provider_blocks``
(not just reasoning) — in practice the order is
``[reasoning, message, ...]`` but the API doesn't guarantee that;
a hypothetical ``[message, reasoning]`` ordering would silently
drop the reasoning under an index-only check. Now walks the list
for the first block whose type is in ``_BLOCK_TYPE_PROVIDER_FACTORY``,
then dispatches the WHOLE list to that provider's extractor. Each
provider's extractor already filters internally by its own block
type, so passing the full list is correct. Regression test added
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
* **Copilot finding 3** (migration 052 docstring): the previous
review-fix wave used sed to rename ``persist_reasoning`` →
``surface_persisted_reasoning`` everywhere, which mangled a
historical reference in the migration docstring ("The earlier name
``surface_persisted_reasoning`` was renamed..."). Restored to
point at the actual pre-rename name (``persist_reasoning``).
* **Copilot finding 4** (sdk/typescript/src/events.ts:26):
``HistoryEvent`` JSDoc still referenced ``persist_reasoning`` —
the sed rename only walked ``turnstone/`` and ``tests/``, missing
the TypeScript SDK. Updated to ``surface_persisted_reasoning``.
Also widened the comment to cover all three reasoning-bearing
block types (Anthropic ``thinking``, OpenAI Responses ``reasoning``,
synthetic ``reasoning_text``) instead of mentioning only Anthropic.
* **github-code-quality finding** (session.py:1120): ``_resolve_server_type``
had a bare ``except Exception: pass``. Replaced with a
``log.debug(..., exc_info=True)`` + explanatory comment. Behaviour
unchanged (still returns ``""`` on any lookup failure); failures
are now observable under DEBUG triage.
Rejected (with rationale)
* **github-code-quality finding** (_protocol.py:265):
``extract_reasoning_text``'s body is ``...`` per ``LLMProvider``
Protocol convention. Every method in the file uses ``...`` (PEP
544 idiomatic Protocol style). Changing only this one to
``raise NotImplementedError`` would be inconsistent with the rest
of the file. CodeQL's "statement has no effect" warning is
technically correct for ``...`` as a standalone expression but
ignores the documented Python Protocol convention. No fix.
Docs sync
* docs/api-reference.md: ``history`` SSE event message-shape table
gains the optional ``reasoning`` field.
* docs/architecture.md: ``ModelCapabilities`` row in the type table
gains ``supports_reasoning_replay``; ``StreamChunk`` and
``CompletionResult`` rows gain the existing ``provider_blocks``
field (was missing pre-PR). New "Per-model reasoning persistence"
subsection under the Models config section, documenting the two
flags + capability gate + three reasoning paths + cross-provider
shape filter.
* docs/settings.md: new "Reasoning persistence (per-model)"
subsection with the two-flag table and capability-gate note.
* docs/diagrams/03-core-engine-classes.puml: ``LLMProvider`` interface
adds ``extract_reasoning_text`` + the new ``replay_reasoning_to_model``
kwarg; ``ModelCapabilities`` class adds ``supports_reasoning_replay``.
PNG regenerated.
Lint + test gate
* ruff check + ruff format clean.
* mypy clean (191 source files).
* pytest -m 'not live' — 6116 passed (3 deselected), +1 net new test
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
Multi-stage /review on the full Phase 1+2+3+4 stack surfaced 9 findings
(0 critical, 3 major, 5 minor, 1 nit, 1 uncertain). All applied.
Major
* perf-1 (session_routes.py:2402): make_history_handler ran sync
storage.load_workstream_config inside async def history on the cold-
workstream path, blocking the event loop on every dashboard /history
request for non-resident workstreams. Every other storage call in
the same handler correctly used asyncio.to_thread. Wrap the sync
call in asyncio.to_thread (preserving the existing try/except so a
DB failure still degrades to the conservative-default branch instead
of bubbling out).
* q-2 (test_reasoning_audit_log_discipline.py): the security-sensitive
test (reasoning text never lands at INFO+ severity) only covered the
4 Phase 1 surfaces. Phase 2 added the strip predicate in
AnthropicProvider._convert_messages and Phase 3 added 3 more code
paths that touch reasoning text — none guarded. Added 4 parallel
tests using the existing capture-and-walk infrastructure:
OpenAIResponsesProvider.extract_reasoning_text,
OpenAIChatCompletionsProvider.extract_reasoning_text,
ChatSession._stream_response (drives the synth-block stamp via a
fake reasoning-emitting stream), AnthropicProvider._convert_messages
with replay_reasoning_to_model=False (drives the Phase 2 strip
predicate).
* q-1 (model_registry.py:42): the persist_reasoning flag name implied
storage-control but actually gates UI rehydration only — operators
flipping it could reasonably expect "stop persisting reasoning" but
storage of reasoning bytes happens in provider_data regardless.
Renamed everywhere to surface_persisted_reasoning: ModelConfig
field, migration 052 column (renaming in-place since 052 is not yet
on main), schema, MODEL_DEFINITION_MUTABLE allowlist, _postgresql.py
+ _sqlite.py CRUD impls, _protocol.py create_model_definition
signature, 3 console_schemas Pydantic models, console/server.py
admin POST + PUT, model_registry row mapper, history_decoration.py
helper parameter, server.py _build_history local var,
session_routes.py make_history_handler local var, sdk/events.py
HistoryEvent docstring, admin.js form id + override pill label,
index.html form input id + UI label + tooltip, coordinator.js (none
needed), and every test that referenced the old field name. The
admin tooltip now reads "Storage of reasoning bytes is unaffected
by this flag — they ride in provider_data regardless" so the
decoupling stays explicit at the operator surface.
Minor
* bug-1 (history_decoration.py:336): dispatcher discriminated on
provider_content[0]["type"] only. Anthropic's redacted_thinking
blocks (sealed by the safety system) can appear before, after, or
interleaved with regular thinking blocks per the API docs. When a
redacted block lands first, the dispatcher returned "" and the UI
silently lost the surrounding thinking text. Registered
"redacted_thinking" as a second key in _BLOCK_TYPE_PROVIDER_FACTORY
pointing at the same AnthropicProvider factory — the existing
extractor's type=="thinking" filter already correctly skips redacted
blocks while walking the full list. Regression test added.
* q-3 (_protocol.py:155): replay_reasoning_to_model defaults split
across 9 sites — operator-side defaults to False (matches DB
server_default), provider-API defaults to True (back-compat with
direct callers). Original "pick False everywhere" fix would have
silently flipped behaviour for any direct provider caller. Instead
documented the intentional bifurcation in the Protocol's
create_streaming docstring.
* q-4+q-5 (_protocol.py:107 + 3 providers): MAX_REASONING_DISPLAY_BYTES
was enforced via Python str slicing which counts code points, not
UTF-8 bytes — 4-byte CJK/emoji glyphs would blow past the byte
ceiling. Renamed to MAX_REASONING_DISPLAY_CHARS to match actual
behaviour. Hoisted the 4-line truncation pattern into a shared
_join_reasoning_with_cap helper in _protocol.py; each provider's
extractor becomes a single line at the tail.
* q-6 (tests/_session_helpers.py): _NullUI + _make_session were
duplicated verbatim between test_session_replay_reasoning.py and
test_session_synth_reasoning_block.py. Hoisted to a shared
tests/_session_helpers.py module (importable, leading underscore so
pytest doesn't try to collect it). test_model_registry.py's
_make_session has a different signature (registry/model_alias args
+ _FakeUI) and is not a candidate for sharing.
Nit
* q-7 (history_decoration.py:286): _make_provider_factory used a
dict-as-cell workaround for closure read-only scope. Replaced with
the more idiomatic nonlocal pattern.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6115 passed (3 deselected). Net +5 tests
(4 audit-log discipline + 1 redacted_thinking dispatcher).
Refinements vs the dedupe output (caught during sanity rendering
the report)
* perf-1 fix preserved the try/except wrapper. The original "wrap in
to_thread" one-liner would have let an OperationalError bubble out
instead of degrading to the fallback branch.
* q-3 fix explicitly documented the bifurcation rather than
collapsing both sides to False. "Pick False everywhere" would
silently flip back-compat behaviour for direct provider callers.
* q-1 fix included the admin.js:5292 fallback site
(m.persist_reasoning !== false) that the original threaded-change
list missed.
* q-6 fix verified the third _make_session in test_model_registry.py
is structurally different (different signature + different UI
helper) and intentionally NOT a dedupe target.
Wire reasoning capture and (where the API supports it) replay for the
two remaining provider paths. Phase 3 was originally scoped as
"OpenAI Responses + Gemini" but a spike against the OpenAI SDK source
revealed that Gemini routes through the OpenAI-compatible endpoint
(``/v1beta/openai/``), which is structurally identical to vLLM /
llama.cpp / any other Chat-Completions-shaped local model. Phase 3
and Phase 4 collapse into one feature with two distinct sub-paths:
* **Path 2 (OpenAI Responses)** — full capture+replay. ``include=
["reasoning.encrypted_content"]`` on the request makes the API
surface ``encrypted_content`` on reasoning items in
``provider_blocks``; ``_convert_messages`` round-trips them as
``ResponseReasoningItemParam`` input items on subsequent turns.
Verified against the OpenAI Python SDK 2.33.0 source
(``response_reasoning_item.py:31-62``,
``response_reasoning_item_param.py:33-37``,
``response_create_params.py:70-74``). Even with ``store=False``,
``encrypted_content`` round-trips correctly per the SDK's own
documentation.
* **Path 3 (Chat Completions / vLLM / llama.cpp / Gemini-compat)** —
persist-only. Canonical OpenAI Chat Completions has no reasoning
field on the wire, but several local-model servers tack on
``delta.reasoning_content`` as Pydantic extras. ``ChatSession.
_maybe_synth_reasoning_block`` stamps a synthetic ``{type:
"reasoning_text", text, source?}`` block onto ``_provider_content``
at end-of-stream when no native ``provider_blocks`` were emitted but
``reasoning_parts`` accumulated text. The ``source`` field carries
``server_compat.server_type`` (vllm, llama.cpp, sglang, …) for
diagnostic value — informational only, doesn't gate behaviour.
Reasoning text NEVER replays back to the model on this path; it
rides ``_provider_content`` only for ``/history`` UI rehydration
and gets stripped from the wire by the existing
``sanitize_messages`` underscore-prefix strip on every request.
What this change does
* ``ModelCapabilities.supports_reasoning_replay: bool = False`` added
to the dataclass. Set True on every OpenAI reasoning model
(gpt-5* + o-series via the Responses API) and every Anthropic
Claude entry (default + 6 model-specific). Path-2 wire-build does
``replay_active = bool(replay_reasoning_to_model and caps.supports_
reasoning_replay)`` so an operator who flips the flag on a
non-reasoning model (gpt-4o via Responses) silently no-ops rather
than emit a malformed ``include=`` request.
* ``OpenAIResponsesProvider`` gains:
- ``_build_kwargs`` accepts ``replay_reasoning_to_model: bool``
(threaded from ``create_streaming``/``create_completion``);
adds ``include=["reasoning.encrypted_content"]`` when active.
- ``_convert_messages`` accepts the same flag, captures
``_provider_content`` reasoning items pre-sanitization, and
emits them as input items immediately before the assistant
message they belong to. Position is tracked by ASSISTANT
ORDINAL (not raw index) — ``sanitize_messages`` drops orphan
tool results and inserts synthesized error tool messages, but
NEVER drops or duplicates assistant messages, so the n-th
assistant in the original list is invariably the n-th in the
sanitized list. Index-based lookup would have silently
misrouted reasoning attachments after any tool-message repair.
- ``extract_reasoning_text`` walks ``type=="reasoning"`` items and
returns ``summary[*].text`` + ``content[*].text`` concatenation.
- ``_reasoning_item_for_input`` projects a stored item into
``ResponseReasoningItemParam`` shape (drops server-only
``status``). Returns ``None`` when ``id`` is missing or non-
string per the SDK ``Required[str]`` schema; caller skips
appending, preventing malformed input items from reaching the API.
* ``OpenAIChatCompletionsProvider`` gains:
- ``extract_reasoning_text`` walks synthetic
``type=="reasoning_text"`` blocks and returns the concatenated
text directly (no underlying provider semantics — the synth
block IS the surface).
* ``ChatSession`` gains:
- ``_resolve_server_type(alias)`` reads ``server_compat.server_type``
from the active model's capabilities dict.
- ``_maybe_synth_reasoning_block(provider_blocks, reasoning_parts)``
creates the synthetic ``reasoning_text`` block when no native
blocks were emitted but reasoning was captured. Wired at the
end of ``_stream_response`` immediately before the
``_provider_content`` stamp.
* ``history_decoration.py`` dispatcher collapses three near-identical
lazy-init singleton getters (one per recognised block type) into a
single ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict + helper. Adding a
fourth provider becomes a one-line dict entry.
* Constants hoist: ``MAX_REASONING_DISPLAY_BYTES = 64 * 1024`` moved
from three sibling provider modules into ``_protocol.py`` so a
tuning change propagates uniformly to every provider's display path.
Cross-provider safety
The synthetic ``reasoning_text`` block type is intentionally NOT in
``ANTHROPIC_VALID_BLOCK_TYPES`` (Phase 2 constant). Cross-model
resumption (operator switches from a local model to Anthropic mid-
workstream) falls through Phase 2's shape filter cleanly to the
text+tool_calls rebuild path rather than reaching Anthropic with a
malformed block. Pinned by ``test_synthetic_block_falls_through_
anthropic_shape_filter``.
Same protection applies in reverse: OpenAI Responses
``type=="reasoning"`` items reaching Anthropic mid-workstream fail
the shape filter and rebuild from text+tool_calls.
Tests (49 net new tests)
* ``tests/test_provider_openai_responses_reasoning.py`` (21 tests):
- Extractor unit tests: empty/none/no-reasoning/single/mixed/
truncation/malformed/non-list (8).
- ``_reasoning_item_for_input`` projection (4 tests including the
new None-on-missing-id guard).
- ``_build_kwargs`` include= gating: flag+capability/flag-false/
capability-false/default-omits (4).
- ``_convert_messages`` reasoning round-trip: emit-before-assistant/
drop-on-replay-false/foreign-shape-skipped/default-replay-false (5).
* ``tests/test_session_synth_reasoning_block.py`` (23 tests):
- ``_maybe_synth_reasoning_block`` direct unit tests (6).
- Cross-provider safety regression — synthetic block falls through
Anthropic shape filter (2).
- ``OpenAIChatCompletionsProvider.extract_reasoning_text`` for the
new synthetic block type (6).
- ``_resolve_server_type`` direct unit tests (5).
- ``_stream_response`` integration tests driving fake reasoning-
emitting streams through the actual session method (3 tests
— added in response to a code-review finding that pinned the
wire-up at session.py needs an integration test).
* ``tests/test_session_replay_reasoning.py`` extended with 4
``TestSessionToOpenAIResponsesBoundaryIntegration`` tests driving
``session._try_stream`` -> real ``OpenAIResponsesProvider`` ->
captured ``client.responses.create`` SDK boundary call. Negative-
tested: temporarily reverting the ``include=`` step in
``_build_kwargs`` makes ``test_replay_true_adds_include_to_
responses_request`` fail; restoring makes it pass.
* ``tests/test_history_decoration.py`` extended with the new
``reasoning_text`` dispatcher branch test, and the Phase 1 stub
test for the OpenAI Responses dispatcher branch was tightened
(it now asserts real text extraction instead of the empty-string
stub).
* ``tests/test_provider_anthropic_reasoning.py`` had its Phase 1
``OpenAIResponses returns "" for reasoning blocks`` stub test
retitled and updated to assert the real Phase 3 behaviour.
Code-review pass
Multi-stage ``/review`` pipeline (4 finders + verify + dedupe) ran
on this diff. 6 findings (1 major, 3 minor, 2 nit), 0 critical, 0
security, 0 performance. All applied:
* MAJOR (bug-1+bug-4+q-1): ``_convert_messages`` enumerate-index
lookup was unsound under ``sanitize_messages`` length changes.
Fixed by switching to assistant-ordinal-keyed lookup.
* MINOR (q-2+q-3): ``_MAX_REASONING_DISPLAY_BYTES`` duplicated
across three provider modules + declared after first use.
Fixed by hoisting to ``_protocol.py``.
* MINOR (q-4): three near-identical singleton getters in dispatcher.
Fixed by collapsing to ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict.
* MINOR (q-5): ``_maybe_synth_reasoning_block`` wire-up not pinned
by integration test. Fixed by adding three
``TestStreamResponseSynthBlockIntegration`` tests.
* NIT (bug-2): ``_reasoning_item_for_input`` fell back to ``id=""``;
fixed to return ``None`` on missing/non-string id.
* NIT (q-6): four naming variants for the same concept; renamed
``_convert_messages`` kwarg to match the operator-flag name.
* REFUTED (bug-3): SDK distinguishes summary vs content as separate
fields; no double-counting concern.
Briefing departures
The briefing's Phase 3 plan grouped Gemini with OpenAI Responses on
the assumption that Gemini reasoning had its own native shape (like
Anthropic's ``thinking``). The spike confirmed Gemini-via-OpenAI-
compat is path-3 (Chat Completions shape, no native reasoning
items). Phase 3+4 merger handles Gemini for free via the synthetic
``reasoning_text`` block — same mechanism used for vLLM and
llama.cpp. Whether Gemini's specific endpoint actually emits
``reasoning_content`` deltas is server-dependent and not yet
empirically verified; capture is best-effort (server-emission-driven,
no flag gate).
The briefing's Phase 4 plan stamped reasoning as Anthropic-shaped
``thinking`` blocks ``{type: "thinking", thinking: <text>}``. This
PR uses a distinct ``{type: "reasoning_text", text, source?}`` shape
to avoid a cross-model resumption hazard the briefing missed: an
unsigned synthetic Anthropic-shape block reaching Anthropic's wire
would 400 the API. The distinct shape falls through Phase 2's shape
filter cleanly without needing signature validation in the filter.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6110 passed (3 deselected). Phase 3+4
added 49 net new tests.
Make ``replay_reasoning_to_model=False`` actually suppress prior-turn
thinking blocks on the Anthropic wire (Phase 1 stored the operator
flag but the wire path always re-sent ``_provider_content``
verbatim). As a side benefit, close a pre-existing latent bug where
foreign-shaped ``_provider_content`` (e.g. an OpenAI Responses
``type="reasoning"`` block reaching Anthropic on a mid-workstream
model switch, post-Phase-3) would have 400'd the API.
Why now: Phase 1 shipped the operator knob and UI rehydration but
the wire payload still always carried thinking blocks for
Anthropic-with-thinking turns. Operators flipping replay=False saw
no behaviour change on the actual API call -- the flag only affected
``/history`` rendering. Phase 2 closes that gap.
What this change does
* ``ANTHROPIC_VALID_BLOCK_TYPES`` (frozenset of 8 block types
Anthropic's input boundary accepts) and
``ANTHROPIC_REASONING_BLOCK_TYPES`` (the strip subset) added at
the top of ``_anthropic.py``. The strip set is intentionally
narrow: ``{"thinking", "redacted_thinking"}`` -- ``tool_use`` /
``server_tool_use`` / ``web_search_tool_result`` (which carry
web-search ``encrypted_content``) MUST survive for round-trip
continuity, and a regression test pins this.
* ``_convert_messages`` signature gains
``replay_reasoning_to_model: bool = True`` (back-compat default
-- production call sites pass the resolved value explicitly).
The verbatim ``_provider_content`` replay path is now wrapped by
a shape-validity check using ``ANTHROPIC_VALID_BLOCK_TYPES``;
foreign-shaped payloads fall through to the existing text+
tool_calls rebuild path rather than reaching the API. When
shape is valid AND replay=False, a list comprehension drops
thinking blocks from ``wire_blocks`` while preserving
tool_use / web_search blocks. When all blocks are stripped
(message had only thinking, no text or tool_calls), the message
also falls through to the rebuild path -- which silently skips
if both content and tool_calls are empty (correct: stripped
reasoning has nothing to replay).
* Orphan-tool detection still walks the ORIGINAL ``provider_content``
(not ``wire_blocks``) so the strip cannot accidentally lose the
source-of-truth tool_use IDs. The implementation comment pins
this invariant.
* Protocol surface grows the kwarg on both ``create_streaming`` and
``create_completion``. ``OpenAIChatCompletionsProvider``,
``OpenAIResponsesProvider``, and ``GoogleProvider`` (via
inheritance) accept the kwarg and ignore it -- they have no
first-class reasoning shape on the wire today. Phase 3 will use
it on the OpenAI Responses adapter to gate
``include=["reasoning.encrypted_content"]``.
* ``ChatSession._resolve_replay_reasoning_to_model(alias)`` reads
``ModelConfig.replay_reasoning_to_model`` from the registry,
defaulting to ``False`` on lookup failure (the conservative
miss-fallback: replaying reasoning text against an unknown
operator preference is worse than missing the strip). Threaded
into the three production call sites:
``ChatSession._try_stream`` (streaming), ``_utility_completion``
(title gen / compaction / extraction), and the agent provider
call site (plan / task agents).
Token calibration deferred to Phase 4
The briefing's optional Phase 2 step (extending ``_msg_text_chars``
to count ``_provider_content`` bytes that survive the strip)
required either invasive flag-threading through every call site
of the static method or a lossy approximation that picked the wrong
direction for the default case. Per the briefing's ``pick a
phase'' guidance, this is bumped to Phase 4. The pre-existing
silent under-count on Anthropic-thinking turns persists when
replay=True. Strip-when-False naturally fixes the under-count by
keeping the bytes off the wire entirely; the residual case is the
opt-in replay path.
Tests (28 new, all driving through real boundary objects)
* ``tests/test_provider_anthropic_replay.py`` (19 tests):
- Strip vs preserve under both flag values (3 tests including
redacted_thinking).
- Default-kwarg back-compat preserves verbatim replay (1 test).
- Web-search tool_use + server_tool_use + web_search_tool_result
survive strip with encrypted_content intact (2 tests, edge 14).
- Orphan-tool synthesis after strip -- pins the
``provider_content`` source-of-truth read at lines 397-433
(1 test).
- Foreign-shape fallthrough: OpenAI ``type="reasoning"`` block
rebuilds via text+tool_calls (1 test).
- Mixed-shape fallthrough: even one foreign block forces
rebuild (1 test).
- Empty / None / non-list ``_provider_content`` fallthrough
(3 tests).
- Legacy Anthropic-thinking row pre-Phase-2 stays in verbatim
path -- no regression on existing conversations (2 tests).
- All-blocks-stripped fallthrough behaviour: rebuild from text
if available, silently skip if not (2 tests).
- Constants pinning: strip set is narrow, valid set includes
web search, strip is subset of valid (3 tests).
* ``tests/test_session_replay_reasoning.py`` (12 tests):
- Resolver: 6 tests covering miss / default / set / explicit /
fallback alias / exception.
- Streaming call site: 3 tests pinning the kwarg propagates
through ``_try_stream`` to a stub provider.
- Non-streaming call site: 1 test pinning
``_utility_completion`` propagates the flag.
- End-to-end boundary integration: 2 tests driving
``_try_stream`` -> real ``AnthropicProvider`` -> captured
Anthropic SDK ``client.messages.stream`` boundary, asserting
on the ACTUAL wire payload shape. Negative-tested:
temporarily reverting the kwarg-thread at
``_anthropic.py:create_streaming`` makes the wire test fail
with ``Strip predicate did not fire at wire boundary``;
restoring makes it pass.
The boundary integration tests were added in response to a code
review finding that the bare-stub call-site tests would not catch
a regression where the provider stops reading the kwarg or
``_convert_messages`` silently drops the strip. The integration
tests close that gap by inspecting what reaches the (mocked) SDK,
not just what the provider was called with.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6061 passed (3 deselected). Phase 2
added 28 net new tests.
Surface stored Anthropic thinking blocks on /history responses so
refreshing the page rehydrates the reasoning bubble. Wire payloads
unchanged. Per-model operator knobs added to model_definitions for
both UI rehydration and (Phase 2) wire-build replay.
Why now: reasoning is already round-tripped via _provider_content for
Anthropic-with-thinking turns, but never surfaces on the history wire,
so a tab reload showed only the final answer with no rationale.
Operators also have no per-model lever to opt out of UI display or to
opt in to replay-to-model on subsequent calls.
What this change does
* Migration 052 adds two boolean columns to model_definitions:
persist_reasoning (default 1) controls UI rehydration; replay_
reasoning_to_model (default 0) reserved for Phase 2's wire-build
shape filter. Mirrors the enabled column pattern (NOT NULL +
integer server_default).
* LLMProvider Protocol gains extract_reasoning_text(provider_blocks)
with concrete impls on AnthropicProvider (walks type=='thinking'
blocks, joins with newline, caps at 64 KiB) and no-op stubs on
OpenAIChatCompletionsProvider + OpenAIResponsesProvider. Google
inherits the no-op via OpenAIChat. Phase 3 will wire the OpenAI
Responses extractor once include=['reasoning.encrypted_content']
is requested.
* turnstone.core.history_decoration gains a structural dispatcher
extract_reasoning_text_from_provider_content keyed off the first
block's type field (Anthropic 'thinking' / OpenAI Responses
'reasoning' / Gemini 'thought' are non-overlapping by API design).
Both history surfaces use it: _build_history calls the dispatcher
directly (the SSE-replay path builds entry dicts from scratch),
and the lifted make_history_handler runs the list-helper variant
in the existing to_thread block.
* make_history_handler resolves persist_reasoning via three tiers:
live session -> workstream_config.model_alias (the same key
SessionManager uses to rehydrate the original model after process
restart) -> conservative True default. Operator flag-flip takes
effect uniformly on both warm and cold workstreams.
* Frontend: app.js replayHistory and coordinator.js role==='assistant'
branch each call the existing reasoning-bubble construction (for
app.js, the document.createElement pattern from the live SSE
handler; for coord, the appendMsg('reasoning') helper) when
msg.reasoning is non-empty. Reasoning bubbles render before the
content bubble, matching live SSE order.
* Admin UI: two checkboxes ('Persist reasoning', 'Replay reasoning
to model') in the model edit modal, plus override-pill display in
the model row when set to non-default values.
What is intentionally out of scope
* Phase 2 -- ANTHROPIC_VALID_BLOCK_TYPES shape filter at
_anthropic.py:312-316, _convert_messages replay_reasoning_to_model
parameter, thinking-strip branch, _msg_text_chars token-calibration
extension. The replay flag is stored but not consumed on the wire.
* Phase 3 -- OpenAI Responses include=['reasoning.encrypted_content'],
Gemini include_thoughts spike, ModelCapabilities.supports_
reasoning_replay.
* Phase 4 -- Local-model / chat-template reasoning persistence
(session.py:3486 reasoning_parts accumulator).
Tests
* AnthropicProvider.extract_reasoning_text -- 13 unit tests covering
None / empty / mixed / multi-block / cap / malformed / non-list
inputs plus other-provider no-op verification (real provider
instances, no mocks).
* extract_reasoning_for_history -- 10 dispatcher tests including
block-type discriminator routing (thinking vs reasoning vs
unknown), strip-when-flag-false, empty / non-dict guards, and
cross-role isolation.
* _build_history -- 6 boundary tests through the real Anthropic
extractor with stub sessions, including the registry-lookup
failure default-True branch.
* make_history_handler -- 5 round-trip tests through real storage:
the storage layer's reconstruct_messages decodes provider_data
into _provider_content, and the helper extracts through the real
AnthropicProvider. Includes the live-session flag honoring path,
the cold-workstream workstream_config lookup path, and the
no-alias default-True fallback path.
* Audit-log discipline -- 4 structural mock-and-assert tests that
capture every Logger.info / warning / error call across the
pipeline (extractor, dispatcher, list-helper, _build_history)
and assert no captured payload contains a marker reasoning string.
* model_definitions storage -- 6 round-trip tests: default flags,
explicit create with both flags, individual update of each flag,
and list-includes-flags assertion.
* model_registry -- 4 tests: dataclass defaults, dataclass with
explicit flags, DB-row-mapping with both flags, and pre-052
legacy-row default-fallback.
Edge cases pinned by the test suite
* Pre-052 DB rows missing the new columns degrade to dataclass
defaults (test_db_reasoning_flags_default_when_absent).
* Live session in memory has its flag honored (test_history_handler_
with_persist_flag_false_via_live_session).
* Cold workstream resolves the flag via workstream_config +
app.state.registry (test_history_handler_cold_workstream_resolves_
via_workstream_config) -- this closes the gap where a process
restart would have silently un-honored an operator flag-flip.
* Cold workstream without persisted model_alias falls through to
default True (test_history_handler_cold_workstream_no_alias_
defaults_true).
* Foreign / unknown / missing block types degrade silently to no
reasoning field rather than misroute or crash.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6030 passed (3 deselected).
Doc-debt cleanup flagged by /review on 9dc29db7. The cap+seq fix
flipped the seq-advance rule but left two doc sites describing the
old "incremented only on actual append" shape — exactly the buggy
invariant the previous commit removed. Future readers trusting the
stale docs would be one wrong assumption away from re-introducing
the silent-drop bug.
Updates the field-init comment block and the docstring on
register_listener_with_in_progress_snapshot (which sits at the
snap_seq capture site, so its contract is consumer-facing).
Also drops the now-dead `seq: int = 0` initializer in
on_reasoning_token and on_content_token — under the new shape, the
unconditional `seq = self._ws_inflight_seq` inside the lock makes
the initializer unreachable. Was load-bearing under the old
else-branch; harmless now but signals "some path leaves seq at 0"
to a reader.
Copilot caught a real bug in the cap+seq interaction: the previous
shape only advanced ``_ws_inflight_seq`` when the buffer actually
appended, on the theory that "every _seq corresponds to a buffered
fragment" was a useful invariant. It wasn't — once the buffer hit
its cap, seq stalled at the high-water-pre-cap, so a subscriber that
registered AFTER the cap was hit would capture
``snap_seq == stalled_seq``, and every subsequent live token (also
tagged with the stalled seq) would be filter-dropped by the events
handler's ``seq <= snap_seq`` dedup. Silent loss of the entire
post-cap stream for refresh-past-cap tabs.
Fix: advance seq on every emit, regardless of buffer cap. The cap
is a buffer-size limit, not a stop-streaming signal. Past-cap tokens
are absent from the snapshot's text payload (the buffer was
truncated at cap) but the live stream past them is now correctly
delivered — refresh-after-cap renders snapshot-up-to-cap then live
tokens past it, with a visual gap equal to the past-cap chunk and
no silent drop of subsequent tokens.
Test ``test_inflight_seq_increments_only_on_actual_append`` enforced
the buggy invariant and is renamed/flipped to
``test_inflight_seq_advances_on_every_emit_even_at_cap``. Added
``test_subscriber_after_cap_hit_receives_subsequent_tokens`` (and
the reasoning equivalent) as direct regressions for the
silent-token-loss scenario.
Updates the docs that describe the per-workstream SSE event stream and
the SessionUI lifecycle to match the refresh-resume changes:
- api-reference.md: documented the `state_change` event (previously
undocumented despite already being a live event) and the new
`in_progress_snapshot` event; rewrote the multi-consumer fan-out
paragraph to mention the kind-specific replay tail (state_change +
optional in_progress_snapshot) so the "no catch-up needed" claim
is no longer misleading.
- architecture.md: bumped the SessionUI Protocol stub to 16 methods
(added `on_turn_start` / `on_turn_committed`) and pointed at the
in_progress_snapshot section in the API reference.
- sdk.md: added rows for `state_change`, `in_progress_snapshot`, and
`approval_resolved` (preexisting gap) to the per-workstream event
table.
- coordinator-api-tour.md: added an `in_progress_snapshot` row to the
event table and rewrote the reconnection-contract paragraph to
cover mid-stream content/reasoning restoration.
- diagrams/04-conversation-turn.puml: added `on_turn_start()` before
the thinking-start emit and `on_turn_committed()` immediately after
`messages.append(assistant_msg)`, with notes explaining the inflight-
buffer reset semantics. PNG regenerated.
Refreshing a coordinator or interactive workstream pane while the LLM
is mid-stream now restores the partial assistant text + reasoning
immediately and flips the composer back to stop-mode, instead of
showing nothing until the response completes.
Per-turn inflight buffers (`_ws_inflight_content`, `_ws_inflight_reasoning`,
`_ws_inflight_seq`) on `SessionUIBase` are kept separate from the
existing multi-turn `_ws_turn_content` buffer that drives the
dashboard's IDLE-piggyback payload. New `on_turn_start` (top of
send-loop, defensive) and `on_turn_committed` (right after
`messages.append(assistant_msg)`, primary) lifecycle hooks reset
inflight at turn boundaries. The seq counter is monotonic across
turns so a long-lived subscriber's `snap_seq` cutoff stays valid for
the lifetime of the connection — resetting per-turn would silently
drop turn N+1's first M tokens (M = whatever was streamed pre-snapshot
in turn N).
`snapshot_and_consume_state_payload` also drains inflight at idle/error
so cancel and exception paths don't leak stale text. New
`register_listener_with_in_progress_snapshot` atomically registers a
listener and snapshots the inflight buffers; `make_events_handler`
emits a `state_change` event (so the JS busy machine flips to
stop-mode) followed by a one-shot `in_progress_snapshot` after the
kind-specific replay, then strips the internal `_seq` field from
yielded live events while filtering against `snap_seq`. A per-listener
shallow `dict` copy in the live drain prevents the multi-tab race
where one listener's `del event["_seq"]` would corrupt another
listener's filter view.
`_synthesize_cancelled_results` now emits synthetic `on_tool_result`
events for each cancelled tool so live coord tabs can drop the
newly-additive `coord-tool-batch--running` indicator cleanly. The
indicator now coexists with `--auto`/`--approved` (applied on
`tool_info` and `approval_resolved` approved; removed when every row
in the batch has a result), making live tool execution visually
parallel to the replay-time orphan rendering.
Frontend handlers in `app.js` (interactive) and `coordinator.js` (coord)
absorb EventSource auto-reconnect re-replays via a length-based
prefix check on the in-progress buffer. New `InProgressSnapshotEvent`
+ `StateChangeEvent` dataclasses in the Python and TypeScript SDKs
with type guards.
`_MAX_TURN_CONTENT_CHARS` lifted 256 KiB → 512 KiB (single constant
for both buffers — headroom for current commercial models).
Regression tests cover race-free composition under concurrent writers,
seq-filter dedup invariants, the cross-turn seq monotonic invariant,
idle/error inflight drain, synthesized `on_tool_result` on cancel
(including UI-hook failure isolation), and the multi-listener
shared-dict invariant.
Two findings, both confirmed against the source:
1. Migration 051's downgrade rewrote every '[]' row back to '{}',
which would (a) destroy operator-written empty arrays and
(b) reintroduce the known-invalid sentinel that every consumer
rejects. Pre-migration '{}' rows and operator-authored '[]' rows
are indistinguishable after upgrade — there is no clean inverse
for the data state. Made downgrade an explicit no-op with the
rationale documented inline; '[]' is the correct shape under any
consumer's interpretation, so leaving the data untouched on
downgrade is strictly safer than reversing it. Updated the
module docstring to call this out.
2. admin_update_skill's notify_on_complete validator short-circuited
on empty string: `if nc and nc != "[]":` skipped the JSON-parse
branch when nc=="" and persisted the empty string straight to
storage, leaving a non-JSON value behind. Folded the empty case
into the existing "{}" coercion so any blank/whitespace/legacy
value normalises to "[]" before the array-validation gate.
Tests: three new regressions in TestSkillAPI — empty-string
normalises, "{}" sentinel coerces, non-array JSON 400s. The third
locks in the array-only validator that the previous "valid JSON"
gate would have accepted.
Every consumer of prompt_templates.notify_on_complete treats it as a
JSON-array string (the admin form's array editor, the JSON.isArray
validator in submitEditTemplate, _validate_notify_targets in
server.py, the documented "list of channel/contact identifiers"
shape). But the column's server_default — set in migration 011 and
inherited through 021's lift into prompt_templates — has been "{}"
(an empty JSON object) since day one.
Newly-installed remote skills inherit the schema default, so every
unlock-then-edit flow trips the array validator on the inherited
"{}" and the request never leaves the browser. The user-visible
symptom was "click Save, nothing happens"; the latent symptom was
silent shape divergence between every install and every operator-
authored skill.
Migration 051: rewrites every legacy "{}" row to "[]". Operator-
edited values (anything that's neither "{}" nor NULL) are left
intact. Downgrade restores "{}" only on rows still holding the
post-migration "[]" so any later operator edits stick.
Server-side defaults flipped to "[]" in the same PR so new rows
land correct without depending on the column's server_default:
- _schema.py prompt_templates.notify_on_complete server_default
- StorageBackend protocol create_prompt_template kwarg
- sqlite + postgres create_prompt_template kwargs
- console_schemas.py SkillCreateRequest / SkillUpdateRequest /
SkillInfo Pydantic defaults
- core/session.py ChatSession._notify_on_complete initial value
- server.py initial-message worker fallback when skill_data omits
the field
admin_update_skill validator now also rejects non-array JSON (was
"valid JSON" only — would have accepted "{}" or "{\"a\": 1}").
_skill_to_response coerces legacy "{}" rows to "[]" on read so the
admin UI sees a consistent shape even before migration 051 runs.
The frontend's `tmpl.notify_on_complete || "[]"` fallback already
handled empty-string but not "{}" — the read-side coercion makes
it moot.
Designer-review follow-up to the .is-visible sweep. With role=alert
+ aria-live=assertive, AT engines re-announce when the element's
text content changes — but without aria-atomic some engines only
read the diff between old and new content. With aria-atomic=true
the entire updated message is read each time, which matters when a
validation error is replaced by a server error on retry (or
vice-versa).
Added aria-atomic=true to all 24 modal error elements (every
role=alert with aria-live=assertive). Same accessibility uplift
across the board — no per-modal exceptions.
Also dropped the stale `style="display: none"` attribute from the
three MCP error elements (mcp-create-error, mcp-import-error,
mcp-install-error). The CSS rule
.admin-modal [role="alert"] { display: none; }
already hides them by default — the inline attribute was redundant
and would have overridden the .is-visible toggle if the class-based
contract is ever changed.
Sweep of the latent bug PR #494 fixed for the skill modals: the
project's CSS contract for modal errors is
.admin-modal [role="alert"] { display: none; }
.admin-modal [role="alert"].is-visible { display: block; }
…but ~30 sites across governance.js and admin.js were toggling
`style.display = ""` instead of the .is-visible class. The "show"
side broke silently — clearing the inline style fell back to the
CSS `display: none` so the error never rendered, and any
validation failure looked like an unresponsive button.
Mechanical conversion of every show/hide site for these modal
error elements:
governance.js
create-role-error, edit-role-error
create-policy-error, edit-policy-error
github-import-error
cpp-error, epp-error (custom + eval prompt policies)
create-hr-error, edit-hr-error (heuristic rules)
create-ogp-error, edit-ogp-error (output-guard patterns)
admin.js
mcp-create-error, mcp-import-error, mcp-install-error
Plus the global `_showModalError` helper in admin.js — its
`style.display = "block"` happened to work today (inline display
beats the CSS rule), but normalising it to .is-visible keeps every
modal on a single canonical path. The five modals that route their
show side through that helper (create-user, create-token,
create-channel, create-schedule, edit-schedule) had their hide
sides converted in lockstep.
Added a comment on `_showModalError` documenting the contract so
the next contributor doesn't reintroduce the bug.
Out of scope: model-create-error (already canonical), home-coord-error
(not in .admin-modal), edit/create-template-error (fixed in #494).
No CSS or HTML changes; behaviour-equivalent for hide sides; show
sides go from broken-silent-no-render to correct-render-with-AT-
announcement.
Once edit-template-error is actually visible (the visibility fix in
this same PR), a stale error now persists across resubmit cycles:
the user sees a red message, fixes the input, clicks Save, the
validator passes, the PUT goes out — and the previous error stays
on-screen the whole time, only clearing when the modal closes on
success.
Fix at the start of submitEditTemplate / submitCreateTemplate:
clear .is-visible AND empty textContent. Cheaper than tracking
every validator branch and every .catch path; a fresh submit is a
clean slate.
Smoke-testing the unlock flow surfaced a latent bug: clicking Save
on the edit-skill modal silently no-op'd whenever the
notify-on-complete field had non-JSON content. The error div was
DOM-correct (text content set, role=alert, aria-live=assertive),
but invisible — because the project's modal-error CSS contract is:
.admin-modal [role="alert"] { display: none; }
.admin-modal [role="alert"].is-visible { display: block; }
…and the JS in submitEditTemplate / submitCreateTemplate was
clearing the inline `display: none` via `el.style.display = ""`.
That falls back to the CSS rule, which still says `display: none`,
so the error never rendered. The user saw no error and the click
felt unresponsive (compounded by the early-return before the
disabled-state reset, which also made Save look broken).
Fixed both skill-modal flows (create + edit) by toggling the
canonical `.is-visible` class instead. Six sites in governance.js:
the two early-return show paths, the two .catch show paths, and
the two modal-open hide-resets.
Scope note: this same bug pattern exists in ~20 other modal error
sites across governance.js and admin.js (create-role, edit-role,
create-policy, edit-policy, github-import, cpp, epp, create-hr,
edit-hr, create-ogp, edit-ogp, mcp-create, mcp-import, mcp-install,
plus admin.js sites that don't go through _showModalError). All
pre-existing, broken silently for who knows how long. Out of scope
for this PR — recommend a follow-up sweep that also normalises
_showModalError's `style.display = "block"` to the same convention.
Designer review of the cb5fa1b lock-icon iteration flagged five
items; four are addressed here, one was a deliberate trade-off
documented below.
- Glyph hardening (#2): the lock character is now 🔒︎ — U+1F512 with
the U+FE0E text variation selector — paired with the existing
font-variant-emoji: text rule. font-variant-emoji shipped late
and isn't universal yet (Chrome 131+, Safari 16.4+, Firefox 132+);
the explicit text VS is belt-and-braces so older Chromium / most
Linux don't fall back to a coloured emoji that would clash with
the monochrome instrument-panel aesthetic.
- Accent-line de-conflict (#3): top:14px → 18px so the lock button
sits below the modal's ::before accent-line decoration's visual
band rather than competing with it horizontally. h2's
padding-right reservation (44px) still gives the title clearance.
- Mobile touch target (#4): @media (max-width: 700px) bumps the
button to 44×44 (WCAG 2.5.5 / Apple HIG / Material minimum) and
shifts it to top:8px right:8px, with h2 padding-right widened to
56px to match.
- Keyboard discoverability (#6): on readonly open, focus lands on
the lock button instead of Cancel. Keyboard users hit the unlock
affordance immediately instead of having to Tab past every
disabled spec input to reach it. Cancel is one Shift-Tab away.
Deferred:
- (#1) Reviewer flagged top-right placement as risking confusion
with the universal × close-button convention. Keeping the
icon-only design per product direction; the bordered chip styling
+ accent-coloured hover make it visually distinct from the
thin-stroke unbordered × pattern, and the confirm dialog catches
any misclick safely.
- (#5) Optional empty-corner indicator after unlock — the
"Customized from upstream" badge text already carries the signal;
not adding new chrome.
Three issues from manual smoke-testing the unlock flow:
1. Confirm dialog rendered behind the edit-skill modal. Both
overlays sat at z-index 600, and confirm-overlay is earlier in
the DOM than edit-template-overlay — so DOM order put the parent
modal on top of its own confirm. Bumped confirm-overlay to 650
(still below toasts at 700) since confirm dialogs are launched
FROM other overlays and need to sit above them.
2. Save button stayed disabled (or non-functional) after unlock.
submitEditTemplate disables etm-submit on click and re-enables in
.finally, but a stale disabled=true survives the mutate-in-place
re-render that runs after unlock. Always reset
submitBtn.disabled = false in showEditTemplateModal so the
re-render path can never inherit a stuck disabled state.
3. UX redesign — moved the unlock affordance from a "Customize…"
button at the bottom of the footer to a 🔒 icon button at the
top-right of the modal. The lock glyph is the universal "this is
locked, click to unlock" affordance and reads more clearly than
a footer button next to Cancel/Save. font-variant-emoji: text
keeps it monochrome on browsers that support it (instrument-panel
aesthetic) with graceful fallback to coloured emoji elsewhere.
admin-modal-skill h2 reserves padding-right so a long title can
never collide with the absolute-positioned button.
Cleanup: removed the now-unused .modal-secondary and
.modal-buttons-spacer rules; the bottom etm-unlock button + flex
spacer are gone from the modal footer.
Copilot caught that prompt_templates.readonly is an Integer column
(_schema.py: sa.Column("readonly", sa.Integer, nullable=False,
server_default="0")) and create_prompt_template stores it as 1/0,
but unlock_skill in the postgres backend was passing a Python bool
(readonly=False). The sqlite impl already uses 0; this aligns the
two backends and matches the 0/1 idiom used for the sibling flag
columns (is_default, auto_approve, enabled).
The other Copilot findings on this PR (loadGovSkills race, NBSP
double-space, list_skill_versions O(history_size), ignored
set_skill_readonly return value + None re-read) were all closed by
the prior review-feedback commit (eea795d): the snapshot+flip is
now an atomic unlock_skill() that uses SELECT MAX(version)+1
internally, the handler guards both the unlock_skill return and the
post-flip get_prompt_template re-read, the JS chains
showEditTemplateModal off loadGovSkills's promise, and the badge
NBSP matches the sibling pattern.
Code review caught a race + a missing None guard; designer review
caught a window.confirm regression and a button-hierarchy issue.
Backend:
- Race fix (bug-2): replace set_skill_readonly+create_skill_version
with a single atomic unlock_skill(template_id, snapshot, changed_by)
-> int|None on the storage protocol (sqlite + postgres). Snapshot
insert + readonly flip happen in one transaction; the next version
number is computed via SELECT MAX(version)+1 inside the txn rather
than len(list)+1 outside, closing the (skill_id, version)
collision window where two concurrent admin actions could both pick
the same version.
- None guard (bug-3): check the post-flip get_prompt_template re-read;
return 404 instead of letting _skill_to_response(None) raise.
- Audit body: also record snapshot_version, and harden None-vs-empty
with `or ""` on the existing.get(...) calls.
Frontend:
- D-1: replace window.confirm with the existing showConfirmModal
(admin.js:2350) — themed dialog, focus-trap, can render the source
URL with consistent typography. The native dialog could collapse
the multi-paragraph copy depending on browser.
- D-2: mutate-in-place on success rather than hide → reload → reopen.
loadGovSkills now returns its fetch promise so unlockSkill can
chain showEditTemplateModal after the cache refresh — no flicker,
no focus bounce, and it kills bug-1 (the reopen was reading stale
_govSkills before loadGovSkills resolved). showEditTemplateModal
is idempotent when already open: it skips the trigger-element
capture and the focus-trap reinstall.
- D-3: button hierarchy. Drop flex:1 from .modal-secondary so the
Save button keeps a stable width whether or not Customize is
rendered; insert a flex-spacer between Customize and Save so the
destructive-ish detach groups left next to Cancel and the primary
action floats right.
- D-4: NBSP normalized to match the existing escape pattern
on the sibling badge line (was an actual NBSP byte).
- D-5: success toast now reads "Skill unlocked — fields are now
editable" so the operator gets a positive affirmation that the
edit affordance is live.
- D-10: aria-describedby="etm-origin-badge" on disabled spec inputs
so screen-reader users get the same "this came from upstream"
context that sighted users see in the cyan badge.
Tests: + test_unlock_skill_versions_after_existing_history seeds an
out-of-order version (3) and asserts unlock picks 4, defending
against the len()-based version computation regressing.
skills.sh / GitHub installs land with readonly=True so admins can only
tune runtime config (model, temperature, etc.); the SKILL.md spec is
locked. In practice, upstream skills aren't always tuned for turnstone,
so locking the spec adds friction without a real safety win — every
edit is audited and version-snapshotted regardless.
This adds an explicit unlock so the boundary stays visible (multi-user
audit trail benefits from a discrete event, vs. silently dropping the
gate). Behaviour:
- POST /v1/api/admin/skills/{id}/unlock — flips readonly=False on a
readonly row. Snapshots the pre-unlock state into skill_versions so
the upstream-pristine version is recoverable from the History tab.
Records skill.unlock audit with {name, source_url, origin}. 400 on
already-unlocked, 404 on missing.
- origin stays "source" after unlock so the UI keeps a "Customized
from upstream" provenance badge — the readonly flag is the gate, the
origin field is the lineage.
- Storage: dedicated set_skill_readonly writer on the protocol +
sqlite + postgres backends. readonly is intentionally absent from
SKILL_MUTABLE so the generic update path can't piggyback on a
provenance flip — the dedicated writer pattern matches what's
already used for set_mcp_oauth_client_secret_ct.
- Frontend: "Customize…" button in the edit modal (visible only when
readonly), with a confirm dialog explaining the upstream-detach.
Once unlocked the existing edit-skill flow handles spec edits with
no other changes. Origin badge updates to show "Customized from"
the upstream URL when a source-origin row is unlocked.
Tests cover: unlock flips readonly + persists, pre-unlock snapshot
written to skill_versions, 400 on already-unlocked, 404 on missing,
post-unlock PUT can edit name/content/description (the readonly gate
no longer fires).
Three issues caught by Copilot on the initial PR:
1. SKILL.md size cap was measured in code points, not UTF-8 bytes.
`len(str)` is a *lower* bound on encoded byte length — multi-byte
chars (emoji, CJK) inflate up to 4×, so a 100k-emoji SKILL.md
(400KB encoded) would slip past the 256KB cap. Switch to
`len(contents.encode("utf-8"))` and surface lone-surrogate failures
as SkillSourceError instead of dropping them silently. New
regression test feeds emoji content.
2. _skills_sh_source_url did not normalize the skill_id, so a sloppy
id from `/api/search` (whitespace, surrounding slashes) would pass
`_split_skills_sh_id`'s charset check (which strips first) and
produce a malformed persisted source_url that broke the
discover-UI dedup contract. Strip the id inside the helper, and
reconstruct the canonical id from validated parts in
download_skill's listing so downstream callers never see the raw
input.
3. The catch-all `except Exception:` around create_prompt_template
relabeled every storage failure (DB connection, disk full,
permission errors) as "conflict", masking operational issues.
Translate IntegrityError → StorageConflictError at the storage
shim (matching the pattern already used for OIDC user
provisioning) in both sqlite and postgres backends, then catch
StorageConflictError specifically in the install handler. Real
conflicts → "conflict" + warning; other exceptions → new
"internal error" reason + log.exception.
Tests: +3 (oversized multibyte SKILL.md, source_url normalization,
storage-layer conflict translation). 226 passing.
The skills.sh install path was failing with 404s because their public
API surface changed: /api/skills/{id} is gone, replaced by
/api/skill/[owner]/[repo]/[skill] (auth-walled) and
/api/download/[owner]/[repo]/[skill] (unauthenticated, returns the
SKILL.md + bundled resources inline as JSON). The error was not
surfacing in logs because admin_skill_install had a silent
`except Exception:` around create_prompt_template that relabeled every
storage failure as "conflict" with no log entry.
- Replace SkillsShClient.resolve_github_url with download_skill that
hits /api/download/{owner}/{repo}/{skill} and returns a SkillPackage
directly. No GitHub round-trip; no rate-limit surface.
- Add _split_skills_sh_id with strict per-segment charset validation
([A-Za-z0-9._-]+) so URL-hostile content can't produce a malformed
request or divergent persisted source_url.
- Use len(contents) instead of len(contents.encode("utf-8",
errors="ignore")) for the SKILL.md size cap — errors='ignore' was
silently dropping invalid units, making the cap bypassable.
- Extract _accept_resource(rel_path, byte_size) gate predicate; share
it between download_skill and the GitHub _find_resource_files helper.
- Have search() derive a deterministic source_url from the skill id
when /api/search omits one (which it currently always does), so the
discover-UI "already installed" check matches what download_skill
persists.
- Add structured logging across admin_skill_install and
admin_skill_discover: a shared _log_install_failure helper for the
four except branches (was four near-duplicate log calls with one
drift), plus per-resource failure tallying — partial-resource
installs now surface failed_resources in the response and audit
record instead of silently committing the skill row with missing
assets.
Tests: 7 new — empty/non-list files, oversized SKILL.md, resource
cap, non-text extension filtering, plus _split_skills_sh_id charset
rejection (whitespace, query chars). Verified end-to-end against
live skills.sh with tavily-search.
Four Copilot findings on c6041c6 — all confirmed valid, all bounded
to authenticated-user prompt-injection scenarios but worth closing
before merge.
Wrapper-detect bypass (string + list branches of
``_apply_reminders_for_provider``):
The round-2 fix used ``content.startswith("<tool_output>\\n")`` to
detect already-wrapped content and skip ``escape_wrapper_tags``. A
tool whose RAW output starts with that prefix (e.g. ``echo
'<tool_output>'``) would match and have its escape skipped, letting
literal ``<tool_output>`` / ``<system-reminder>`` tags reach the model
and impersonate a system envelope. Replace the prefix check with
``extract_advisories_from_tool_envelope(content) is not None`` —
parsing requires the open AND matching close tags AND a structurally
valid envelope, raising the bypass bar significantly.
Mirror fix in the list-content branch so a tool emitting an unmatched
envelope as a text part can't bypass the per-text-part escape.
``_build_history`` legitimate-envelope drop:
The list-content drop path previously removed any text part starting
with ``<tool_output>\\n``. A tool that legitimately outputs a
well-formed envelope (documentation viewer, code analyzer demoing the
wrapper, an echo tool) would have that part silently disappear on
replay. Tighten the drop heuristic to require BOTH ``cleaned_text ==
""`` AND at least one extracted advisory — the structural signature of
the injected ``wrap_tool_result("", advisories)`` carrier we produce
in ``session.py`` for list-typed tool output. A legitimate envelope
has non-empty inner body or no advisory blocks and survives the
projection.
Empty advisory body:
``queue_message`` accepts any non-None text including ``""`` and
whitespace-only strings. ``_classify_advisory`` would return a
``user_interjection`` advisory with empty / whitespace body, which
``replayAdvisoriesAfterTool`` then renders as a featureless empty user
bubble. Filter empty / whitespace-only bodies at classification time
so the wire-shape contract is uniform: no empty advisories ever ride
the wire.
Tests:
* ``test_apply_reminders_escapes_tool_output_starting_with_envelope_prefix``
pins the structural-parser bypass close: a string starting with the
envelope prefix but lacking a close tag still gets escaped.
* ``test_apply_reminders_escapes_list_text_part_with_unmatched_envelope_prefix``
mirrors for the list-content branch.
* ``test_build_history_keeps_legitimate_envelope_text_part_with_body``
pins that legitimate envelope output stays in the projected list.
* ``test_decorate_suppresses_empty_advisory_body`` and
``test_decorate_suppresses_whitespace_only_advisory_body`` pin the
empty-body filter in ``_classify_advisory``.
Tests: 5923 passed, 3 deselected. Lint + format + mypy clean.
Reverses the seam-2-only design from the prior commits on this branch.
Queued user messages arriving DURING a tool batch (Seam 1) splice into
the last tool result's envelope as ``UserInterjection`` advisories via
``wrap_tool_result``. Messages arriving BETWEEN turns (Seam 2) drain
as a single trailing user row via ``_flush_queued_messages`` with
``user_feedback`` (operator text alongside an approval, e.g. "y, use
full path") folded in as a prefix. Cancel/exception drains (Seam 3)
keep the existing ``_flush_queued_messages()`` call unchanged.
Why all three seams:
* Strict-template providers (Mistral, Llama via vLLM with stock chat
templates) reject role-alternation violations. A literal ``user``
row mid-tool-batch breaks ``assistant(tool_calls) → tool → ... →
assistant``; back-to-back ``user → user`` rows on the wire also fail.
* The seam-2-only design produced back-to-back ``user`` whenever
``user_feedback`` and queued items both fired — bug-1 from the round-1
review. Folding ``user_feedback`` as a prefix to the queue-drain
collapses the two into one row.
* During-batch arrivals couldn't ride seam 2 — the splice was the only
way to deliver same-turn without violating role alternation.
Storage symmetry:
Tool DB rows now store the wrapped ``output`` (envelope + advisories)
unconditionally — ``self.messages[i]['content']`` and
``conversations.content`` match exactly. List-typed output (image /
structured MCP results) uses ``wrap_tool_result(raw_joined_text,
advisories)`` at save time so the persisted string is anchored on
``<tool_output>\n`` for the replay parser. ``TOOL_RESULT_STORAGE_CAP``
is removed entirely; tools are responsible for bounding their own
output, storage faithfully represents in-memory. Removing the cap
also simplifies the parser — no truncated-envelope edge case.
Replay extraction:
``decorate_history_messages`` (REST ``/history``) and ``_build_history``
(SSE replay, resume, rewind, retry, post-load, rename re-replay) both
call the public ``extract_advisories_from_tool_envelope`` helper to
pull the envelope back into structured ``advisories`` for JS replay.
Both string content and list-typed content (image+queued-message
combo) covered. JS renders extracted advisories as normal user
bubbles after the tool block via the shared ``replayAdvisoriesAfterTool``
helper in ``shared_static/utils.js``.
Wrapper-tag escape and provider splice:
``escape_wrapper_tags`` now encodes pre-existing ``&`` first using an
``&`` sentinel so tool output containing literal entity strings
(documentation viewers, code analyzers, web scrapers returning entity-
encoded markup) round-trips correctly. Both encode and decode helpers
short-circuit on absence of ``<`` / ``&``.
``_apply_reminders_for_provider`` detects already-wrapped content
(string body and list text-part) by ``startswith("<tool_output>\n")``
and skips re-escape so existing envelopes survive intact when a tool
message also carries ``_reminders`` (the queued-message + tool-error
co-occurrence case is now common).
``decorate_history_messages`` runs in ``asyncio.to_thread`` to keep
MB-scale string work off the event loop.
Other cleanup:
* ``_collect_advisories`` delegates the queue drain to a named helper
``_drain_queued_messages_to_advisories`` so the swap-and-clear pattern
lives next to ``_flush_queued_messages``'s identical pattern and the
side-effect is documented at the call site.
* Preamble strings + body marker for ``UserInterjection`` round-trip
detection moved to module-level constants in ``tool_advisory.py``;
imported by ``history_decoration.py`` so a producer-side rephrase
can't silently desync the parser.
* ``_send_with_mocks`` ctxmgr extracted in ``test_session.py`` — the
six new send-driven tests share an 8-deep ``patch.object`` block.
* ``replayAdvisoriesAfterTool`` shared helper in
``shared_static/utils.js``; ``app.js`` and ``coordinator.js`` both
invoke it.
* Dead truncation-pill CSS removed (``.tool-output-truncated`` and
``.coord-tool-truncated``); the JS that added these elements went
away with ``TOOL_RESULT_STORAGE_CAP``.
* Tautological tests (``TestBuildHistoryAdvisoryPropagation``)
replaced with production-realistic round-trip tests built from
``wrap_tool_result(...)`` envelopes — REST and SSE-replay surfaces
pinned to the same wire shape; full DB round-trip pinned end-to-end.
Negative-tested:
* Reverting the prefix-merge in ``_flush_queued_messages`` produces
back-to-back ``user`` rows, breaking
``test_user_feedback_and_queued_coexistence_single_row_with_prefix``.
* Reverting the ``extract_advisories_from_tool_envelope`` call in
``_build_history``'s tool branch leaves the envelope verbatim in
wire content, breaking the round-trip tests.
* Reverting the wrapper-detection in ``_apply_reminders_for_provider``
entity-encodes the existing envelope's literal tags, breaking both
the string-content and list-content envelope-preservation tests.
* Reverting the ``wrap_tool_result(raw_text, advisories)`` projection
at the DB save site produces a string starting with the original
raw text, breaking
``test_tool_db_row_round_trips_list_output_with_advisories``.
Tests: 5918 passed, 3 deselected. Lint + format + mypy clean on
touched files.
Round-1 ``/review`` apply-pass. Drops stale ``UserInterjection``
references from comments and docstrings that no longer describe the
post-PR drain shape, asserts the two-stream invariant in the new
queued-message persistence test, and pins the ``content.trim()`` +
``renderAssistantToolBatch`` invariants on coord-side so a future
refactor can't silently regress the Qwen3 phantom-card fix or the
chronological-order render fix.
Deferred:
* **bug-1** (back-to-back ``user`` row when ``user_feedback`` from the
approval-prompt UI callback coexists with a queued-message drain).
Reachable on strict OpenAI-compatible local templates (Anthropic and
Anthropic-via-merge-consecutive collapse fine; vLLM-hosted Mistral /
Llama enforcing role alternation can reject). The pre-PR splice
guarded against this case by riding queued items inside the tool
result envelope; that guard is what motivated the original
UserInterjection design, so the fix lane needs a deliberate decision
rather than a quick patch. Sleeping on it.
* **q-1** (delete dead ``UserInterjection`` class + tests). Held for
the bug-1 decision — if the chosen fix is to resume the splice for
the ``user_feedback``+queue coexistence case, the advisory shape
stays load-bearing. Class now carries a docstring note marking it
retained-pending-decision so a passing reader doesn't grep for
producers and assume it's actually dead.
Apply-pass content:
* ``q-2``: drop "queued user interjections" from the persistent-
advisory parenthetical in ``send``'s tool-result loop comment;
rewrite to point at ``_flush_queued_messages`` for the queue path.
* ``q-3``: ``__init__`` channel-routing comment loses "and
``UserInterjection``" — only ``GuardAdvisory`` remains.
* ``q-4``: ``_queue_tool_advisory`` docstring + the tool-error nudge
comment lose the user-interjection mentions; the docstring also now
describes the side-channel + ``_apply_reminders_for_provider``
splice path (the actual mechanism).
* ``q-5``: ``AttachmentsNotQueueableError`` docstring rewritten to
describe the post-PR ``_flush_queued_messages`` flow — the
single-combined-turn ``\n\n``-join shape can't carry image / file
blocks, and per-item separate user turns would expand the strict-
template role-ordering surface that the post-batch drain already
balances.
* ``q-6``: the new ``test_queued_message_persists_as_user_row_after_tool_batch``
in ``test_session.py`` now asserts ``stream_idx == 2`` so a future
regression where the post-batch flush runs but the send-loop short-
circuits before the next iteration surfaces in CI rather than
manual repro.
* ``q-7``: ``test_coordinator_page.py`` gets two new string-grep pins
mirroring the existing ``test_app_js.py`` shape — ``content.trim()``
on coord's assistant-replay branch and ``renderAssistantToolBatch``
for the hoisted helper that orders content card before tool batch.
## Test plan
- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] Affected test surface (``test_session.py`` +
``test_tool_advisory.py`` + ``test_app_js.py`` +
``test_coordinator_page.py``) — 240 passed
Three independent rehydrate / replay regressions reported on long
multi-turn conversations after the pull-model wake stack landed.
**1. coord history replay rendered tool_calls above the assistant
narration that announced them.**
In ``coordinator.js``'s loadHistory loop, the ``role === "assistant"``
``tool_calls`` branch sat above the role switch — every assistant turn
with both narration AND tool dispatch produced ``[tool batch][content
card]`` in the DOM, even though chronological order is content first.
On a parallel fan-out (e.g. four ``close_workstream`` calls in one
turn) operators saw the assistant text "Let me close them out and
summarize" with NO tool batch between it and the next assistant
message — the four-row batch had been rendered above the announcing
text and was scrolled out of view.
Hoisted the ``tool_calls`` synthesis into a local
``renderAssistantToolBatch(m)``, called from inside the assistant
branch AFTER the content card. Live SSE order (text → dispatch →
results) now matches replay order.
**2. Whitespace-only assistant content rendered as a blank card on
replay.**
Models with vLLM's ``--reasoning-parser`` (Qwen3 in production)
strip ``<think>…</think>`` and emit only the trailing ``"\n\n"`` as
``content`` before a tool call. ``content_parts = ["\n\n"]`` saves
``content = "\n\n"`` to the conversations row. Live the user only
sees ``.msg.reasoning`` (the thinking content) — the empty
``.msg.assistant`` card lives next to it but reads as a thin
divider. On rehydrate the reasoning bubble is gone (not persisted)
and the empty assistant card is the only thing left, surfacing as
"blank cards where the assistant message was."
Both UIs now check ``content && content.trim()`` before rendering
the body — whitespace-only content skips the card entirely instead
of showing a phantom row. Live render unchanged.
**3. Queued user messages disappeared on reconnect.**
PR #474 routed queued user messages into the tool-result envelope
via ``UserInterjection`` advisories — same-turn delivery, but no
persisted user row. On page reload / cross-tab replay the
optimistic ``.msg-queued`` bubble vanished: there was no DB row to
rehydrate it.
Dropped the ``UserInterjection`` splice in ``_collect_advisories``;
the queue drains through ``_flush_queued_messages`` AFTER the tool
batch completes instead. Sequence becomes
``assistant(tool_calls) → tool … tool → user(drained)``, which is
valid for Mistral and Anthropic strict role validators (the only
forbidden shape was user injected mid-batch BEFORE the tool result,
which this still avoids). Persists a real user row → bubble survives
reconnect, and stays in the session's wire-side context window on
the next turn.
## Test plan
- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] ``pytest -m "not live"`` — 5798 passed, 3 deselected
- [x] Updated ``test_collect_advisories_does_not_drain_queued_messages``
(was pinning the old UserInterjection shape)
- [x] Added ``test_queued_message_persists_as_user_row_after_tool_batch``
(drives ``send`` end-to-end with a queued message arriving during
the tool batch; asserts the user row lands in self.messages AND
hits ``save_message``)
- [x] Updated ``test_replay_history_renders_content_before_tool_block``
to tolerate the new ``msg.content && msg.content.trim()`` guard
- [ ] Live browser pass on coord (close_workstream parallel fan-out
rehydrates with the 4-row batch BETWEEN the announcing assistant
text and the summary) and interactive (Qwen3 ``"\n\n"`` rows no
longer paint blank cards on reload; queued bubble survives a tab
refresh)
PR #489 review feedback (Copilot + github-code-quality):
- closeSettingsPanel now closes nested revoke modal first on close-button
path (Escape was already handled by the parent keydown trap deferring
to the inner trap; missing-modal-on-close-button was an orphan-modal
hazard).
- _refreshConsentBadge now updates the settings button's aria-label +
title dynamically with the pending-consent count for screen readers
(badge stays aria-hidden — the count is in the label).
- _MAX_INSUFFICIENT_SCOPE_REPORTED promoted to public
MAX_INSUFFICIENT_SCOPE_REPORTED in mcp_http_parsers; drops cross-module
private import in mcp_oauth's /start handler.
- Stale test comment in test_session_mcp_dispatch_error.py corrected:
_exec_read_resource does not log with exc_info=True (bearer-leak
invariant).
- Rejected the protocol-method ellipsis warning: rest of _protocol.py
uses ... consistently per Protocol convention.
Lint:
- ruff format applied to test_mcp_pool_auth_integration.py and
test_mcp_pool_auth_resource_integration.py (combined `with` grammar —
pure formatting).
Flake fix — test_integration_pool_reuse_401_refresh_and_retry_succeeds
on Python 3.11 / resource-constrained CI:
Same cross-task scope hazard f6a3b66 fixed at the close side, surfacing
at the connect side. asyncio.wait_for at mcp_client.py:1206 wraps
streamablehttp_client.__aenter__ in a fresh asyncio.Task. That fresh
task enters anyio cancel scopes, completes, and dies. The eventual
stack.aclose() during eviction or auth_401 retry runs from a different
task and tries to exit scopes whose entering task is dead — anyio
raises RuntimeError, the wedged anyio state blocks the retry's stack
teardown + reconnect, and the call exceeds the 15s budget on slow
workers.
Fix: replace asyncio.wait_for with `async with asyncio.timeout(...)` so
the streamablehttp_client.__aenter__ runs in the dispatch task itself,
no fresh-task scope ownership. Aligns with invariant 18 (asyncio.timeout
not asyncio.wait_for for any SDK / AS / pool-loop await crossing anyio
scopes).
Static path (_connect_one) at lines 905 and 1000 deliberately retains
asyncio.wait_for — auth_type ∈ {none, static} is byte-identical
(invariant 1) and the narrow connect-once / no-eviction-then-reuse
pattern doesn't trigger the cross-task hazard. Anchor comments pin
both directions: a future migration there would break invariant 1; a
future revert at 1206 would re-introduce the flake.
The cited test is the symptom (non-deterministically times out under
load), not a structural gate (no deterministic asyncio.timeout
assertion exists). The comment block at line 1206 records this so a
maintainer who reverts and finds green on a fast machine doesn't
conclude the fix is unneeded.
Verified on Python 3.11.14 (/tmp/venv311) and 3.13.7 (.venv): ruff
format clean, ruff check clean, mypy clean. 368 unit tests + 30 pool
integration tests pass on both interpreters; the previously-flaky test
passed 20× in isolation on 3.11.
Multi-stage /review (4 finders × verify × dedupe): bug/security/perf
returned zero findings; quality returned 3 confirmed minor/nit items
all of which are applied here (q-1 anchor comments at 905+1000, q-2
symptom-vs-gate clarification at 1206, q-3 module-docstring sentence
in mcp_http_parsers).
Wires the structured-error envelopes produced by Phase 7b's pool
dispatcher (mcp_consent_required / mcp_insufficient_scope /
mcp_*_forbidden / mcp_token_undecryptable_key_unknown /
mcp_oauth_url_insecure) through to the user-facing dashboard, and
adds a per-user settings panel for managing MCP server consents.
Changes
- ``_dispatch_pool_sync`` and ``_dispatch_pool_resource_sync`` wrap
structured-error string returns as ``RuntimeError(json_str)`` via
``_is_structured_error()`` so the session-layer ``except Exception``
branch fires uniformly across tool / resource / prompt dispatchers
(the prompt path's ``isinstance(result, str)`` shortcut works only
because prompts return ``list[dict]`` on success). Without this,
the consent UX silently does not render for tool / resource calls.
- ``_structured_error`` extended with an optional ``consent_url``
field; ``_build_consent_url`` produces ``/v1/api/mcp/oauth/start``
query strings (path-relative; the dashboard appends ``return_url``
at click time). Wired to all 12 ``mcp_consent_required`` and the
``mcp_insufficient_scope`` emit sites.
- New endpoints ``GET /v1/api/mcp/oauth/connections`` and
``DELETE /v1/api/mcp/oauth/connections/{server_name}`` registered
on both ``turnstone-server`` and ``turnstone-console``. The DELETE
handler runs local delete + audit + 204 first, then schedules the
RFC 7009 upstream revoke as a fire-and-forget ``asyncio.create_task``
with strong-ref tracking via ``_revoke_upstream_tasks`` (mirrors
the ``_pg_refresh_drain_tasks`` pattern). Soft cap of 256 concurrent
in-flight revokes prevents pile-up under coordinated mass-revoke;
the audit detail records ``upstream_revoke_outcome`` as
``scheduled | no_refresh_token | no_http_client | shed_by_cap``.
- ``ASMetadata`` extended with ``revocation_endpoint`` parsed from
RFC 8414 metadata. ``revoke_token_at_as`` helper posts the form
body under ``asyncio.timeout`` (not ``asyncio.wait_for``) and
never raises; ``_attempt_upstream_revoke`` is wrapped in an outer
``try/except Exception`` so unhandled exceptions don't surface as
``Task exception was never retrieved``.
- ``/v1/api/mcp/oauth/start`` accepts an optional ``scopes=`` query
param; tokens are validated against RFC 6749 §3.3 grammar via
``is_valid_scope_token`` (promoted to ``mcp_http_parsers``),
capped at ``_MAX_INSUFFICIENT_SCOPE_REPORTED`` (32), and unioned
with the configured server scopes for the step-up consent flow.
- Storage primitive ``list_mcp_user_token_metadata_by_user`` projects
the metadata columns at the SQL boundary so ciphertext blobs never
cross the wire on the settings-list path. New
``MCPUserTokenMetadataRow`` TypedDict in ``_protocol.py``;
``MCPTokenStore.list_user_token_metadata`` re-types to the existing
``MCPUserTokenMetadata`` shape.
- Dashboard renderer (``app.js``): ``tryParseMcpError`` detects the
envelope shape on ``tool_result`` SSE events with ``is_error=True``
and ``buildMcpErrorEmbed`` renders an action card mirroring the
existing ``buildMediaEmbed`` pattern. Three categories: actionable
(consent_required / insufficient_scope) with a ``Connect`` button
that opens ``/v1/api/mcp/oauth/start`` in a popup with a scheme
guard, forbidden (mcp_*_forbidden) with a static notice, operator
(key-mismatch / url-insecure) with an operator-action notice.
- New gear button in the appbar opens an MCP-connections settings
modal driven by ``loadMcpConnections`` / ``confirmRevokeMcp``
(two-step revoke confirmation matching the existing delete-ws
pattern). Pending-consent badge tracks unresolved consent prompts
in this tab; cleared after the connections list returns. Console
proxy collision-checked: the IIFE only prepends a node-id pill to
``header.firstChild``, so the right-anchored gear button is safe.
Bearer-leak invariant
- No ``exc_info=True`` on any new path that can carry a chained
``httpx.Request`` (revoke handler, dispatch sites, exec sites).
The two pre-existing ``exc_info=True`` calls in
``_exec_read_resource`` / ``_exec_use_prompt`` were replaced with
structured-field logs as a Phase 8 sibling fix.
Tests
- 440 pytest passes on both Python 3.13 (.venv) and 3.11
(/tmp/venv311); ruff + mypy clean.
- 5 new test files: ``test_mcp_consent_url_sibling_audit`` (structural
gate that every ``code="mcp_consent_required"`` / ``mcp_insufficient_scope``
site carries ``consent_url=``), ``test_mcp_oauth_connections``,
``test_mcp_oauth_revoke``, ``test_mcp_token_store_metadata``,
``test_session_mcp_dispatch_error``.
- End-to-end regression coverage for the bug-1 sibling pattern:
``test_call_tool_sync_raises_on_structured_error_envelope``,
``test_read_resource_sync_raises_on_structured_error_envelope``,
``test_get_prompt_sync_raises_on_structured_error_envelope``, plus
``test_call_tool_sync_does_not_wrap_non_structured_string`` as the
defensive gate (only ``mcp_*`` envelopes are wrapped).
Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}``: the
wrap fires only when the dispatcher returns a structured-mcp-error
string, which only happens on the oauth_user pool path.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) on every new
AS / SDK / pool-loop await per Python 3.11 anyio cancel-scope
hazard.
- Scope cap ``_MAX_INSUFFICIENT_SCOPE_REPORTED = 32`` enforced at
every output / merge site.
- Cross-user isolation on the revoke endpoint: a non-owner DELETE
returns 404 with the same body shape as a never-existed row;
``http_client_mock.post.assert_not_called()`` pins this in 3 tests.
Deferred (not Phase 8 blockers)
- perf-2 (``asyncio.gather`` parallelisation in revoke handler) —
superseded by perf-1's fire-and-forget pattern.
- q-4 (prompt-path ``isinstance(str)`` vs sibling ``_is_structured_error``
asymmetry) — already documented in the function docstring.
- q-9 (``_pendingConsentServers`` → ``_serversNeedingConsent``
rename) — pure naming taste.
Apply sanitize_text() to the new _source and _reminders columns in
both save_message and save_messages_bulk on SQLite + PostgreSQL,
mirroring the existing pattern used for content and provider_data.
Producers (sanitize_payload on the watch dispatch path,
format_nudge constants on the standard nudge path) already strip
NUL bytes today so nothing in production reaches this clamp — but
the storage layer is opaque to those invariants, and PostgreSQL
TEXT columns reject NUL outright. Without this clamp, a future
producer that forgets sanitize_payload (or hand-builds the column
string) hard-fails the chat-loop persist path on PostgreSQL.
Cost is negligible — sanitize_text early-exits on the common
no-NUL case via 'if value and "\x00" in value'.
Surfaced by Copilot's PR #486 review.
Closes round-2 review finding q-7 (nit).
The kwarg was added to close round-1 perf-2 cosmetically — the
storage backend's signature already accepted ``limit``, but the
single in-tree caller (``ChatSession.resume``) doesn't pass it and
other tail-load consumers go direct to ``storage.load_messages``.
Adding signature surface to mark a perf finding closed without an
actual consumer is API-surface bloat.
When a tail-load consumer is written (e.g. a heuristic in
``session.resume`` to skip ancient wake rows), the kwarg can come
back — at that point with a real caller driving the contract.
Closes round-2 review findings q-6 (nit) and perf-1 (nit).
* **q-6:** ``_WATCH_REMINDER_OPTIONAL_KEYS`` carried a leading
underscore (Python's module-private convention) but was imported
from two other modules — clearly a public contract between
``build_watch_reminder`` and its consumers
(``ChatSession._dispatch`` + ``server._build_history``). Drop the
underscore so the import sites match the constant's documented
cross-module role.
* **perf-1:** The dispatch closure imported the constant inside its
body, paying ``IMPORT_NAME`` + ``IMPORT_FROM`` bytecode on every
watch fire. ``server.py`` already imports at module scope; hoist
the same way in ``session.py``. Microsecond savings per dispatch,
but the in-closure form was just an oversight from the apply-pass.
Closes round-2 review findings q-1 (minor), q-3 (nit), q-4 (nit), q-5
(nit).
* **q-1:** Drop the ``post-migration 050`` clause from the fork-block
comment — the apply-pass relocated rather than removed the
tombstone-style temporal reference round-1 q-2 was supposed to fix.
The bulk-row dict shape and ``_encode_reminders`` are
self-explanatory; the WHY is pinned by
``test_fork_preserves_source_and_reminders``.
* **q-3:** Replace ``DOES persist now`` framing on the wake-row save
comment with a present-tense invariant. The ``now`` implies the
reader knows the prior state, same family as the temporal
tombstones.
* **q-4:** Trim the 12-line WHAT-narration block above the
resume-time ``_reminders_delivered = True`` loop to two lines
stating the WHY only. The new regression test pins the contract.
* **q-5:** Reframe ``test_fork_preserves_source_and_reminders``
docstring as a forward-looking invariant; drop the
``Dropping them was the original bug`` and ``post-migration 050``
fix-narration.
Project convention: invariant statements, present tense; don't
reference the current task / fix / migration number.
Closes round-2 review findings bug-1 (minor) and q-2 (minor).
* **bug-1:** ``_encode_reminders`` clamped each entry's ``text`` field
with Python ``str`` slicing, which counts codepoints. Multi-byte
UTF-8 input (CJK, emoji) could land 4 bytes per character past the
cap, defeating the row-width / FTS5-index protection by up to 4x.
Switch to UTF-8 byte clamping with ``errors="ignore"`` on the
decode boundary so a slice mid-codepoint drops the partial
character cleanly.
* **q-2:** Both the constant block-comment and the ``_encode_reminders``
docstring referenced ``docs/design/watch-card-ux-briefing.md`` —
local-only per project convention (``feedback_no_design_doc_commits``)
so the canonical repo reads as a dead reference. The cap value
stands by itself; the row-width / FTS5 WHY is enough.
Closes round-1 review findings q-2 (minor), q-5 (minor), q-6 (nit), q-7
(nit), sec-1 (nit), perf-4 (nit).
* **q-5:** Export ``_WATCH_REMINDER_OPTIONAL_KEYS`` from
``turnstone/core/watch.py`` and import in the dispatch closure
(session.py) and the replay filter (server.py:_build_history). The
three-place duplication of the literal tuple
``("watch_name", "command", "poll_count", "max_polls", "is_final")``
is gone; future field adds touch one constant.
* **sec-1:** Run ``sanitize_payload`` over string-typed metadata fields
(``watch_name`` / ``command``) before they enter the queue. Today's
consumers all use ``textContent``, but the asymmetry — sanitised
``text`` alongside unsanitised metadata — would survive forever in
DB rows and resurface if a future consumer used a non-textContent
sink (aria-label, copy-to-clipboard, markdown render).
* **q-7:** Drop the per-iteration ``isinstance(reminder, dict)`` from
the dispatch closure's metadata comprehension. By the time the
block runs, ``text = reminder.get("text", "") if isinstance(...)``
+ the ``if not sanitized: return`` guard above already established
``reminder`` is a non-empty dict.
* **q-2:** Strip tombstone-style references — "post-#482", "post-#484",
"Step 7 of the watch-card UX plan", "Post-Step-7 dispatch surface",
and the brittle line-anchor "session.py:2685-2686" — across
``session.py``, ``test_session.py``, ``test_watch.py``,
``test_watch_dispatch.py``, ``test_watch_integration.py``. Comment
intent preserved; historical anchors gone.
* **q-6:** Drop the ``del source`` line in ``cli.py``'s
``on_user_reminder``; the parallel ``on_tool_reminder`` ignores
``tool_call_id`` without ``del`` and the comment alone is enough.
* **perf-4:** Document the SQLite ``render_as_batch=True`` recreate
cost in migration 050's docstring — first deployment after upgrade
copies the conversations table twice (one per ``add_column``).
PostgreSQL is unaffected.
5734 non-live tests pass; ruff + mypy clean.
Closes round-1 review findings q-3 + q-4 (minor, merged) and bug-3 + bug-4
(nit, merged).
* **q-3 + q-4:** The new ``.msg.user-reminder .msg-body { white-space:
pre-wrap }`` rule was a no-op on the interactive UI because that
frontend's ``_buildDefaultReminderBubble`` appended label + text spans
directly to the outer ``.msg.user-reminder`` element with no
``.msg-body`` wrapper. Coord rendered the same shape with a wrapper.
The two implementations diverging on DOM structure also meant a
shared-helper extraction was harder than necessary. Reconciled by
wrapping interactive's spans in ``.msg-body`` to match coord; the CSS
rule now applies to both UIs and the shared-extraction follow-up to
``shared_static/cards.js`` is mechanical (deferred per the review
report — out of scope for this commit).
* **bug-3 + bug-4:** The reminder anchor lookup ``.msg.user`` also
matched ``.msg.user.system-nudge`` markers because the marker carries
both classes. A non-wake reminder fired between a wake marker and
the next real user message would anchor below the wake marker rather
than the previous real user message. Edge case (``/history`` reload
corrects), but the fix is mechanical: change the selector to
``.msg.user:not(.system-nudge)`` in both files.
Closes round-1 review finding perf-2 (minor).
Storage backends accept ``*, limit: int | None = None`` (see
:meth:`StorageBackend.load_messages` at storage/_protocol.py:146) but
the in-memory wrapper at memory.py:82-85 dropped the kwarg, so
callers that wanted to tail-load (e.g. ``session.resume`` against a
long-running coord with hundreds of wake rows + persisted reminder
JSON) were forced to pull every row through the wrapper anyway.
Wraparound is mechanical: signature widens, default leaves existing
callers unaffected.
Closes round-1 review finding q-1 (major).
The comment block above ``self._attach_pending_user_reminders(user_msg)``
asserted that reminders "stay in-memory only and don't persist across
reloads" — directly contradicted by the comment block immediately below
(at the save_message call site) that explains the new persistence
semantics, plus the actual code that now writes ``_source`` and
``_reminders`` to the conversations row. Future readers hitting both
blocks would lose trust in the surrounding comments.
The lower block already documents the persistence contract, so the
upper block is just deleted rather than rewritten.
Closes round-1 review findings bug-2 (major), perf-1 (minor), perf-6 (nit).
* **bug-2:** ``ChatSession.resume(..., fork=True)``'s bulk-row builder
silently dropped the ``_source`` and ``_reminders`` side-channel
data the source workstream had persisted via ``_append_user_turn``.
Both backends' ``save_messages_bulk`` already accept these keys
(the columns exist post-migration 050) — the bulk builder just
didn't supply them. The fork's resumed transcript would then look
like the assistant turn answered out of nowhere: every wake marker
and every reminder bubble that survived to disk on the source got
dropped on the fork. New regression test
``test_fork_preserves_source_and_reminders`` pins the contract.
* **perf-6:** Extracts ``_encode_reminders(reminders) -> str | None``
near ``_apply_reminders_for_provider`` so the user-turn save path,
the tool-turn save path, and the new fork bulk builder share one
encoder. Eliminates the drift risk between three near-identical
``json.dumps(..., separators=(",", ":")) if X else None`` patterns.
* **perf-1:** The new helper clamps each entry's ``text`` field at
``REMINDER_TEXT_STORAGE_CAP = 8192`` characters before encoding so
a single rogue producer (a watch streaming unbounded shell output,
a corruption-class steering payload) can't blow the conversations
row width or the FTS5 index. The in-memory side-channel keeps the
full body — only the persisted JSON is clamped. Mirrors
``TOOL_RESULT_STORAGE_CAP`` on tool result rows.
5734 non-live tests pass; ruff + mypy clean.
Persisted ``_reminders`` survive ``load_messages`` but the in-memory
``_reminders_delivered`` flag does not (it's session-scoped — set by
``_mark_reminders_delivered`` after each successful provider stream,
never persisted alongside the JSON column). Without a re-splice
guard at resume time, ``_apply_reminders_for_provider`` would walk
every loaded message, see ``_reminders`` set + the flag falsy, and
splice every historical ``<system-reminder>`` envelope onto the wire
on the very next user turn — leaking each reminder a second time, the
turn after it had already advised.
Mirror the post-stream hook in ``resume()``: every loaded message
that carries reminders has already been delivered (it survived to
disk), so flag it accordingly so ``_apply_reminders_for_provider``
short-circuits on the pass-through path.
Test pins the contract end-to-end — stage a workstream with a
persisted reminder, resume into a fresh session, append a live user
turn, run the wire transform, and assert the historical reminder
body does NOT land in the rendered output.
User-visible slice of the watch-card UX workstream — combines the
replay-path widening, both frontend renderers, the CSS, and the
cross-cutting Python tests.
server._build_history widens the reminder filter from {type, text} to
project on a known set of optional fields (watch_name, command,
poll_count, max_polls, is_final) and surfaces _source as
entry["source"] when set. The known-key filter narrows the blast
radius if a future producer accidentally stuffs sensitive fields
into the dict.
SessionUIBase.on_user_reminder takes a new source: str | None kwarg
that rides on the SSE event when set. _attach_pending_user_reminders
forwards user_msg["_source"] so non-originating tabs see the wake's
"system_nudge" tag and render the thin marker. Protocol + cli + eval
implementations widen accordingly.
Frontend (coordinator.js + app.js — touched in lockstep per project
memory's "logic that lands in BOTH UIs must touch both files"):
* Branch on r.type === "watch_triggered" for a structured
.msg.watch-result card with header / $ command / <pre> body /
poll N/M [· final] footer.
* New addSystemNudgeMarker (interactive) + appendSystemNudgeMarker
(coord) renders a thin .msg.user.system-nudge anchor for
wake-driven reminders, both live (source === "system_nudge" on the
SSE event) and replay (msg.source === "system_nudge").
* Default .msg.user-reminder rendering preserved for every other
metacog nudge type.
CSS (shared_static/chat.css):
* New .msg.watch-result rules — full-width treatment, cyan accent,
monospace body with word-break: break-word for mobile.
* New .msg.user.system-nudge rule — thin yellow marker.
* Bonus newline-collapse fix: .msg.user-reminder .msg-body now sets
white-space: pre-wrap so multi-line shell output / bulleted lists
stay readable inside the advisory bubble.
Plan reference: docs/design/watch-card-ux.md §4 Steps 9-12 + bonus
CSS §11 (Commit 4).
WatchRunner._dispatch_result now takes a structured reminder dict
produced by build_watch_reminder() — text matches format_watch_message
verbatim (so compaction / channel adapters / wire splice keep their
behaviour), and watch_name / command / poll_count / max_polls /
is_final ride alongside as queue-entry metadata.
The dispatch closure registered in ChatSession.set_watch_runner pulls
the optional fields out of the dict and passes them to enqueue via
the new metadata kwarg. Drain seams already merge metadata into the
rendered reminder dict (Commit 2), so the SSE event for a watch fire
now carries the structured fields without further plumbing.
* turnstone/core/watch.py — new build_watch_reminder() helper, _poll_watch
switches from format_watch_message + dispatch(str) to build_watch_reminder
+ dispatch(dict). set_dispatch_fn / get_dispatch_fn / restore_fn
signatures widen from Callable[[str, str], None] to
Callable[[dict[str, Any], str], None].
* turnstone/core/session.py — dispatch closure builds the metadata dict
via {k: reminder[k] for k in ("watch_name", "command", ...) if k in reminder}
and passes it to nudge_queue.enqueue.
* tests/test_watch.py — new TestBuildWatchReminder class pinning the
builder shape; existing dispatch_fn_registry / restore_fn tests
updated to dict shape.
* tests/test_watch_dispatch.py — every dispatch(...) call updated to
pass a structured reminder dict via _reminder() helper; new
TestMetadataPropagation class pins the metadata-on-enqueue contract.
* tests/test_watch_integration.py — _dispatch_result calls updated to
dict shape.
Plan reference: docs/design/watch-card-ux.md §4 Step 7 + Step 8 watch-test
subset (Commit 3).
Producers (today only watch_triggered) can now attach a metadata dict
to a queued nudge so the rendered reminder dict on the user/tool side
carries fields beyond {type, text}. Wire shape stays additive: the
SSE event picks up the optional fields when present, and producers
without metadata leave it None.
* _Entry grows from 4 fields to 5 — metadata: dict[str, Any] | None.
* enqueue accepts metadata=... as a kwarg.
* drain returns list[tuple[str, str, dict | None]] (was 2-tuples).
* pending stays narrow at (type, text) for legacy callers; new
pending_with_metadata projects the third slot for tests that need
to assert producer-specific fields.
* Three drain consumers in session.py — _collect_advisories,
_attach_pending_user_reminders, deliver_wake_nudge_from_queue —
unpack the new 3-tuple shape and merge metadata into each
reminder dict.
* on_user_reminder / on_tool_reminder protocol signatures widen
from list[dict[str, str]] to list[dict[str, Any]] across
ChatSession.UI, SessionUIBase, CLI, eval harness.
Plan reference: docs/design/watch-card-ux.md §4 Step 6 + Step 8 _Entry
subset (Commit 2).
Adds two TEXT-NULL columns to the conversations table so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live. Until now, reminders lived only on the
in-memory ChatSession.messages dict, and the wake-driven empty user
turn was not persisted at all (skip at session.py:2685-2686) — a
second tab connecting via /history saw the assistant turn with no
preceding wake context, and missed every other tab's reminder
bubbles besides.
Single Alembic revision 050 (head was 049) adds:
* conversations._source — today only "system_nudge" for wake rows
* conversations._reminders — JSON-encoded reminder list
Both backends (sqlite + postgresql) thread the columns through
save_message / save_messages_bulk / load_messages. reconstruct_messages
unpacks the row tuple as 9 elements (was 7), JSON-decoding _reminders
on the user AND tool branches with the same contextlib.suppress guard
the existing provider_data / tool_calls decode uses. Tool-row
reminders ride the same column so tool_error / repeat replay shape
matches user-channel parity.
session.py:2685-2686 wake-row persist skip is dropped; _append_user_turn
JSON-encodes user_msg["_reminders"] and passes both source + reminders
to save_message. The tool-message save site at session.py:3014-3020
mirrors with metacog_reminders.
Plan reference: docs/design/watch-card-ux.md §4 Steps 1-5 (Commit 1).
Address Copilot review feedback on PR #487:
1. **Atomic commit invariant**: ``_bootstrap_coord_subsystem`` previously
stamped ``coord_mgr`` ~50 lines before the final ``coord_registry``
commit, and started threads + subscriptions in between. A concurrent
dashboard request running through ``_require_coord_mgr`` during the
runtime-bootstrap window could observe ``coord_mgr`` set with
``coord_registry`` still ``None`` and surface the misleading
"Restart the console after adding a model definition" 503.
Refactored to two phases: (a) build everything as locals, (b) start
side-effects (StateWriter / observer / nudge watcher / child fan-out
/ cleanup thread), then atomic commit at the end with ``coord_mgr``
stamped LAST. The build-phase ``try/except`` rolls back any started
side-effects from local handles before re-raising — no daemon thread
or subscription leaks across retries, and ``app.state`` is never
stamped on a partial failure.
2. **Class-attr cleanup symmetry**: ``_teardown_partial_coord_subsystem``
now also clears ``ConsoleCoordinatorUI._coord_mgr`` /
``_collector`` / ``_console_metrics`` to match the lifespan shutdown
path (server.py ~line 4629). A failed bootstrap (or test teardown
reuse) no longer leaks process-global pointers at a half-built
subsystem.
3. **Lifespan startup offload**: the lifespan startup error path used
to call ``_teardown_partial_coord_subsystem`` synchronously, which
in turn calls ``StateWriter.shutdown(timeout=2.0)`` — a thread-join
+ sync DB writes that could block the event loop for up to 2s
while the console is still coming up. Wrapped the whole
load-and-bootstrap in ``asyncio.to_thread`` via the new
``_load_and_bootstrap_coord_subsystem`` synchronous helper, so all
blocking work (including any rollback) runs on a worker thread.
Mirrors the pattern the regular lifespan shutdown (line ~4620) and
the runtime CRUD-triggered path already use.
Tests:
- ``test_bootstrap_atomic_commit_no_partial_visibility``: a polling
thread in tight loop watches ``coord_mgr`` / ``coord_registry``
during a real bootstrap and asserts no observation has ``coord_mgr``
set with ``coord_registry`` still ``None``.
- ``test_real_bootstrap_rolls_back_partial_state_on_side_effect_failure``:
monkeypatches ``install_idle_nudge_watcher`` to raise mid-build,
asserts ``app.state`` shows the clean fresh-install state and the
builder-failure error string surfaces ``RuntimeError`` (not the
stale "no models" boot-time message).
A freshly-installed console with no model rows in the DB at boot
caught the ``ValueError`` from ``load_model_registry()`` in the
lifespan and skipped the entire coord subsystem build, leaving
``coord_mgr`` ``None``. ``_refresh_coord_registry`` then bailed
out at ``existing is None`` rather than building the subsystem on
first model add — operators had to restart the console after
configuring their first model in the admin panel for the
"Coordinator subsystem not initialized" banner to clear.
Extract the lifespan's coord build into a reusable
``_bootstrap_coord_subsystem`` and add ``_maybe_bootstrap_coord_subsystem``
that runs as an ``asyncio.to_thread`` follow-on after every admin
model-CRUD endpoint (create/update/delete/reload). The helper:
- fast-paths to a no-op when ``coord_mgr`` is already set;
- guards concurrent first-install attempts with
``_COORD_BOOTSTRAP_LOCK`` + double-checked re-test inside the lock;
- pre-computes config-derived integers BEFORE any thread starts so
``int(config_store.get(...))`` failures don't strand a started
``StateWriter`` daemon;
- stamps ``coord_state_writer`` to ``app.state`` immediately after
``.start()`` so the new ``_teardown_partial_coord_subsystem`` can
shut it down on a partial failure (no thread leaks across retries);
- atomically commits ``coord_registry`` + clears
``coord_registry_error`` as the final step so callers can rely on
the invariant ``coord_registry`` is set iff ``coord_mgr`` is set;
- replaces the stale boot-time "no model definitions" message with
a builder-failure-specific diagnosis (carrying ``type(exc).__name__``)
on construction failure so the dashboard's 503 banner reflects the
actual cause.
Both the lifespan path and the runtime-bootstrap path now route
through the same helper and the same teardown on failure.
Tests: 12 new tests covering the helper-level wiring (idempotent
fast-path, missing-prereq parametrised over ``config_store`` /
``collector`` / ``console_metrics``, no-rows error recording, builder
failure error replacement, partial-state teardown), the endpoint
integration, the deterministic concurrent-call lock test (uses an
instrumented lock wrapper that signals when a second acquirer arrives,
so the test fails fast on slow CI rather than depending on a
wall-clock sleep), and a real-builder end-to-end case constructing a
working ``SessionManager`` against a real ``ConfigStore`` + real
``ClusterCollector``.
Two of five Copilot comments on PR #485 were valid; this commit applies
both. The other three (one duplicate of comment 1, plus the INFO-logging
and `_pending`-naming nits) get rationale on-thread and resolution.
1. emit_oauth_failure_audit action now derived from `code` (#485 bug-1)
The Phase 7b refactor generalized `emit_insufficient_scope_audit` →
`emit_oauth_failure_audit`, routing both `mcp_insufficient_scope` AND
generic-403 (`mcp_*_forbidden`) through the same helper. The audit
`action` field stayed hardcoded as
`"mcp_server.oauth.insufficient_scope_emitted"`, mislabeling generic
forbidden events under the insufficient_scope bucket — downstream
alerting / analytics filtering on `action` would silently fold both
categories together.
The action is now selected from `code`:
* `mcp_insufficient_scope` →
`mcp_server.oauth.insufficient_scope_emitted` (preserves existing
alerting consumers)
* `mcp_tool_call_forbidden` / `mcp_resource_read_forbidden` /
`mcp_prompt_get_forbidden` →
`mcp_server.oauth.forbidden_emitted` (new, distinct label)
Detail row continues to carry both `code` and `kind` so operators get
sub-bucket distinction within either action.
2. Resource-listener docstrings cite RFC §3.2 (#485 doc-1)
Per the codebase convention established in Phase 7b round-1 q-1
(`_rebuild_user_prompt_map` corrected §3.2 → §3.3 because prompts are
§3.3 in the MCP spec), resource-related docstrings should cite §3.2.
The three resource-listener docstrings were citing §3.3, and the
"Mirrors `_notify_listeners` for tools (RFC §3.3)" parenthetical in
both `_notify_resource_listeners` and `_notify_prompt_listeners` read
as "tools are at §3.3" — confusing twice over. All four sites now
carry the correct catalog-kind citation explicitly:
* resource-listener docstrings → "RFC §3.2 (resources)"
* prompt-listener docstrings → "RFC §3.3 (prompts)"
Tests / lint:
* 119 passed on 3.13 + 3.11 (targeted MCP OAuth pool tests)
* ruff + mypy clean on both files
Extends the Phase 7 per-(user, server) ClientSession pool to cover
RFC §3.2 (resources/read) and §3.3 (prompts/get) on the same shape
already proven for tools/call. Pool discovery is capability-gated so
servers without resources/ or prompts/ stay free of extra round-trips.
API additions / widenings (MCPClientManager):
- ``read_resource_sync(uri, *, user_id=None, timeout=120)`` —
per-user-first dispatch; falls through to the byte-identical static
path when ``user_id`` is None or the URI doesn't resolve to an
``oauth_user`` pool entry.
- ``get_prompt_sync(prefixed_name, arguments=None, *, user_id=None,
timeout=30)`` — same dispatch shape; structured-error responses
surface via ``RuntimeError`` so the agent-loop's ``except Exception``
block renders the JSON without polluting the prompt-protocol return
shape.
- ``get_resources(user_id=None)`` / ``get_prompts(user_id=None)`` —
per-user merged catalogs (admin/global call still passes None).
- ``add_{resource,prompt}_listener`` /
``remove_{resource,prompt}_listener`` — ``user_id`` keyword scopes
the listener so a pool-only catalog change for one user does not
wake another user's session.
- ``resource_count_for_user(user_id=None)`` /
``prompt_count_for_user(user_id=None)`` — method-form variants used
by ChatSession's ``read_resource`` / ``use_prompt`` tool gating; the
legacy ``resource_count`` / ``prompt_count`` properties remain
static-only for admin paths.
- ``_dispatch_pool_resource`` / ``_dispatch_pool_prompt`` async coros
— mirror ``_dispatch_pool`` for the new SDK calls; share the
carrier-race-and-cancel core via ``_dispatch_pool_with_entry_call``.
- ``_handle_auth_403`` extended with ``kind=Literal["tool",
"resource", "prompt"]`` so the per-operation ``mcp_*_forbidden``
code surfaces (kind="tool" remains the default for back-compat).
- Pool notification handler now refreshes resources / prompts on
``ResourceListChangedNotification`` / ``PromptListChangedNotification``
via ``_refresh_pool_server_resources`` / ``_refresh_pool_server_prompts``.
ChatSession (``turnstone/core/session.py``) call-site updates:
- 12 sites threaded the session-bound ``user_id`` through
``add_*_listener`` / ``remove_*_listener``, ``get_resources`` /
``get_prompts``, gating, ``read_resource_sync`` /
``get_prompt_sync``, and ``is_mcp_prompt`` so the per-user merged
catalog drives both the visible-tool set and dispatch.
- ``/mcp`` slash command now lists this user's pool resources and
prompts alongside tools (Phase 7 already scoped tools).
Scope decisions:
- Per-user-first URI ordering (decision 0.1): the dispatcher attempts
the user's pool catalog first, falling back to the static catalog
only when no pool entry resolves the URI / prefixed name. Pool-only
users never see the static catalog leak into their resolution.
- Method-form ``*_count_for_user`` (vs property) keeps the legacy
``resource_count`` / ``prompt_count`` properties intact for admin
endpoints whose contract is "static catalog size only".
- Shared ``_dispatch_pool_with_entry_call`` helper accepts an
``sdk_call: Callable[[ClientSession], Awaitable[Any]]`` closure,
keeping the entry-locked carrier-race / classification / retry
plumbing single-source instead of a 3x copy across tool / resource
/ prompt paths.
R6 (anyio uniformity): every pool-side list / read / get path uses
``async with asyncio.timeout(...)`` — ``asyncio.wait_for`` is
forbidden in those paths because it wraps the inner awaitable in a
fresh task and surfaces ``CancelledError`` from inside
``streamablehttp_client``'s anyio TaskGroup on Python 3.11
(per ``feedback_asyncio_timeout_vs_wait_for.md``).
Tests:
- ``test_mcp_pool_auth_resource_integration.py`` — 9 real-transport
resource tests (FastMCP upstream + ``BehaviorMiddleware``):
401-refresh-retry success, persistent 401 -> consent_required,
403+insufficient_scope, 403 generic -> mcp_resource_read_forbidden,
breaker-isolation under repeated auth failures, missing-token,
decrypt-failure, http:// URL guard, unknown-URI ValueError.
- ``test_mcp_pool_auth_prompt_integration.py`` — 9 mirror tests for
the prompt path; structured-error responses verified via
``RuntimeError`` payload shape.
- ``test_mcp_user_catalog.py`` — extended unit coverage for per-user
resource / prompt rebuild + collision policy + symmetric eviction.
- ``test_sessions.py::TestMCPToolGating`` — pool-only-user canary
asserts ``read_resource`` / ``use_prompt`` stay visible when the
static catalog is empty but the user has pool entries.
Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: ``_exec_use_prompt`` was hardcoding ``"MCP prompt error: failed
to invoke prompt"`` — discarding the structured-error JSON that
``_dispatch_pool_prompt_sync`` raises via ``RuntimeError``. Now uses
``f"MCP prompt error: {e}"`` mirroring ``_exec_mcp_tool``; pool-prompt
consent_required / insufficient_scope / forbidden errors now reach
the LLM as intended.
- bug-2 + bug-3: resource template discovery was uncapped —
``_cap_server_resources`` covered ``res_result.resources`` but the
separate ``tmpl_result.resourceTemplates`` loop appended every
template a server returned. Added ``_MAX_RESOURCE_TEMPLATES_PER_SERVER``
(1000) + ``_cap_server_resource_templates`` helper, applied at both
the initial discovery site (``_connect_one_pool``) and the refresh
site (``_refresh_pool_server_resources``). Mirrors the existing
``_MAX_TOOLS_PER_SERVER`` / ``_MAX_PROMPTS_PER_SERVER`` defensive
ceilings.
- sec-1 + sec-2: ``emit_insufficient_scope_audit`` generalized to
``emit_oauth_failure_audit(kind, code, ...)``, called from both the
insufficient_scope branch AND the previously-silent generic 403
branch. Audit detail now records ``{"kind": kind, "code": code,
"scopes_required": [...]}`` so operators can distinguish tool-call
vs resource-read vs prompt-get 403s in audit logs and so cross-
tenant probing on the generic 403 path leaves a trail. The Phase 7
inherited gap (``mcp_tool_call_forbidden`` had the same silence) is
closed in the same refactor.
- perf-1: pool resource discovery now uses ``asyncio.gather(
list_resources, list_resource_templates)`` inside the existing
``async with asyncio.timeout(...)`` budget — disjoint catalogs, no
ordering dependency. Typical-case 2-RTT cold-connect resource block
collapses to 1-RTT. Same change applied at ``_refresh_pool_server_resources``.
- q-1: ``_rebuild_user_prompt_map`` docstring corrected RFC §3.2 →
§3.3 (resources are §3.2; prompts are §3.3).
- q-2: ``_refresh_pool_server_prompts`` docstring now carries the
R6 / mcp-loop note that the resource sibling already had — both
refresh paths now declare the asyncio.timeout invariant explicitly.
- q-5: added the ``_user_resource_map`` / DB-mismatch guard to
``read_resource_sync`` for parity with ``get_prompt_sync``. A stale
per-user map entry with no matching oauth_user row now raises a
specific ValueError instead of silently falling through to a
generic ``Unknown MCP resource``.
- q-6: ``_dispatch_pool_with_entry`` (now a single-caller wrapper
after the ``_dispatch_pool_with_entry_call`` extraction) gains a
one-line docstring explaining why the wrapper is preserved
(tool-decode localization + stack-trace identity for debugging).
- q-7: added 1 resource + 1 prompt end-to-end integration test that
drive REAL discovery + dispatch in the same connect (no
``_seed_pool_*_map`` shortcuts), mirroring the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``.
The seeded-map tests stay (faster, focused on dispatch); the new
e2e tests cover the connect-discover-dispatch composition that
caught Phase 6's carrier-on-entry bug.
Pre-push round-1 review fixes (3-finder review on the final state —
the lesson from Phase 7 round-3's q-1 regression: round-2 catches
what the round-1 apply pass missed):
- q-1 (MAJOR): the bug-1 sibling that round-1 missed —
``_exec_read_resource`` was hardcoding ``"MCP resource error: failed
to read resource"`` while ``_exec_use_prompt`` (post-bug-1) preserved
the structured-error JSON via ``f"... error: {e}"``. The round-1
apply pass patched the prompt side but not the resource side. q-5's
per-user-map / DB-mismatch ValueError was being swallowed at the
agent loop boundary, defeating the operator-diagnostic intent. Now
``_exec_read_resource`` mirrors ``_exec_mcp_tool`` and ``_exec_use_prompt``.
- q-6 (nit): defensive-cap comment block at module-level cited
"(RFC §3.2)" while covering both resource and prompt list paths;
prompts are §3.3. Now reads "(RFC §3.2 for resources, §3.3 for
prompts)" matching the convention the q-1 apply established.
- q-5 (rejected with better justification): the reviewer flagged
``_dispatch_pool_with_entry`` as a single-caller wrapper that should
be inlined. After examination — the autouse fixture
``tests/test_mcp_pool_auth_introspection.py::_install_capture_intercept``
monkeypatches this method to stash ``entry.auth_capture`` for the
fake call_tool stubs in dispatcher-asserting tests. Inlining would
redirect the patch to ``_dispatch_pool_with_entry_call`` (different
kwargs shape) and require re-validating every test that depends on
the interception. The wrapper IS load-bearing; q-6 docstring updated
to cite the test-fixture rationale instead of the thin "stack-trace
identity" claim.
Deferred to follow-up (documented rationale):
- perf-2: single-pass partition for system-message resource list
(concrete vs templates). Sub-microsecond at expected scale;
opportunistic-only.
- q-2 (pre-push): ~200 lines of fixture infrastructure
(``BehaviorMiddleware``, ``_build_server``, ``_seed_oauth_server``,
``running_loop_mgr``, etc.) duplicated across three pool-integration
test files. Real maintenance cost, but a 200-line conftest extraction
is a focused refactor that earns its own commit / PR. Tracking as
follow-up rather than balloon Phase 7b's diff further.
- q-3 / q-4 (refactor): extract shared dispatcher / scheduler
helpers to compress three near-identical 90-line bodies (round-1
q-3 was the same root cause; the pre-push q-3/q-4 reviewer
reaffirmed it concretely). Three named methods preserve readability
for the codebase's hottest correctness path; follow-up if
duplication grows further or if a per-path divergence ships.
- q-4 (round-1, distinct from pre-push q-4): split pool concerns
into ``mcp_pool.py``. Out-of-scope per finder; future refactor as
the file approaches the navigation/merge-conflict threshold.
3.13: 5590 passed (5541 baseline -> +49 net; pre-review +47, q-7
e2e tests added +2). Existing audit-detail tests updated in-place
to expect the new ``kind`` and ``code`` fields.
3.11: 5590 passed (parity gate per ``feedback_pytest_env_parity.md``).
Closes PR #484 review findings (Copilot): the soft-cap pattern in
``ChatSession.set_watch_runner``'s dispatch closure was a non-atomic
two-call pair (``count_by_type`` then ``drop_oldest_by_type``) with
two separate lock acquisitions. A concurrent drain on the worker
thread (``USER_DRAIN`` / ``TOOL_DRAIN`` consuming ``"watch_triggered"``
entries via the ``"any"`` channel) could slip between the two calls,
making the drop a no-op. The dispatch closure also discarded
``drop_oldest_by_type``'s return value and unconditionally logged
``dropped_oldest=True``, so a no-op drop got reported as a successful
drop.
* New ``NudgeQueue.cap_at_or_drop_oldest(nudge_type, max_depth,
channel=None) -> bool`` does the count+drop in a single critical
section. Returns the actual outcome.
* Dispatch closure (``session.py:1410-1416``) now calls the helper and
uses its return value to gate the WARNING log line, so the log is
accurate when a drop did NOT happen.
* ``drop_oldest_by_type``'s docstring no longer overstates the
per-call lock as covering a count+drop pair — it points readers
to ``cap_at_or_drop_oldest`` for that contract.
7 new tests in ``TestCapAtOrDropOldest`` cover: below-cap no-op,
at-cap drop-oldest, above-cap drop-only-one (per-call), channel
filter, other-type isolation, ``max_depth <= 0`` defensive no-op,
no-match.
5708 non-live tests pass; ruff + mypy clean.
The github-code-quality bot finding ("Statement has no effect" on
``_protocol.py:939``'s ``...`` body) is a false positive — every
Protocol method in ``_protocol.py`` uses ``...`` as its body, which
is the canonical Python Protocol pattern. Replacing with ``pass``
would diverge from the file's existing style. No code change.
Closes round-2 review findings q-3, q-4, q-5, q-7.
* **q-4:** ``_NAME_CONTROL_CHARS`` and ``_PAYLOAD_CONTROL_CHARS`` shared
7 lines of Unicode-steering character classes (zero-width / bidi /
separators / BOM / tag chars above BMP). Factored into a single
``_CONTROL_CHARS_TAIL`` constant; each regex now differs only in its
leading ASCII range. Future bidi or zero-width additions edit one
place.
Side effect: this corrects a latent bug where ``_NAME_CONTROL_CHARS``
had two literal ASCII spaces in place of U+2028 / U+2029 (line and
paragraph separators) — visible as ``r" "`` in source but rendered
as the actual codepoints in ``_PAYLOAD_CONTROL_CHARS``. After the
factoring both regexes correctly include U+2028 / U+2029, closing
the gap that would have let a workstream name with embedded line
separators forge a sibling bullet (the same vector ``\n`` was
blocked for in the original bug-1 fix).
Switched to ``\u`` escapes for readability (and to keep future Edit
tool runs against this block reliable).
* **q-3:** Tombstone clause "standing in for the deleted
``_watch_pending`` maxsize bound" survived in
``ChatSession.set_watch_runner``'s docstring after the apply-pass
trim cleaned the inline soft-cap comment. Dropped.
* **q-5:** ``test_newline_in_name_does_not_forge_extra_bullet`` carried
five WHAT-narration comments restating what the immediately-following
asserts already say. Dropped — the docstring carries the security
invariant; the assertions speak for themselves.
* **q-7:** ``patch_session_storage`` had a 14-line docstring including
fallback-guidance and self-justification ("accumulated 7 near-duplicate
sites"). Trimmed to a 3-line contract.
Closes round-2 review findings q-1, q-2, q-6.
* **q-1:** ``test_valid_until_drops_when_watch_missing`` collapsed to the
same code path as ``test_valid_until_drops_when_watch_inactive`` after
the apply-pass switched the predicate from ``get_watch[active]`` to
``is_watch_active`` (both stubbed via ``patch_session_storage(active=False)``).
The "missing" case has no distinguishable branch at the dispatch
layer, so dropping it removes a tautological duplicate. The
missing-row mapping moves to the storage layer (q-2 below) where it
IS distinguishable.
* **q-2:** ``is_watch_active`` was a new public storage primitive with
zero direct backend coverage — only via-session-via-stub coverage.
New ``TestIsWatchActive`` in ``tests/test_watch_storage.py`` covers
active row → True, inactive row → False, missing row → False.
Pinned at the storage boundary so future backend changes fail loudly
there instead of in the dispatch tests.
* **q-6:** Concurrency test had ``n_threads = 2`` alongside two literal
Thread objects and a tautological ``assert len(threads) == n_threads``.
Threads are now built from a labels tuple, so ``len(threads)`` drives
the slack bound; the redundant assertion is gone.
Closes review findings bug-4 and q-6.
bug-4 — the watch dispatch concurrency test bounded depth at
``_WATCH_QUEUE_SOFT_CAP + 2 * per_thread`` (= 250) which is
tautologically true: two threads × 100 fires can append at most 200
entries above the cap, so the bound asserted nothing more than what
``depth <= 2 * per_thread`` already says. Tighten to
``_WATCH_QUEUE_SOFT_CAP + N_THREADS`` (= 52): the count-then-drop window
admits at most one slip per concurrent thread.
q-6 — 7 near-duplicate ``monkeypatch.setattr(session_mod, "get_storage",
lambda: _StubStorage())`` sites across ``test_watch_dispatch.py`` +
``test_watch_integration.py`` (4 different stub shapes, mostly trivial
variations on the active flag). Lift a ``patch_session_storage``
helper into the existing ``tests/_helpers.py`` with kwargs for the
common cases (``active``, ``raise_on_is_active``), returns the call list
so call-shape assertions still work. Tests collapse from ~10-line
inline-class blocks to one-line helper calls.
Closes review findings q-2 and q-5.
q-2 — ``bound_watch_id = watch_id`` rebind was unnecessary. ``_dispatch``
is constructed fresh per fire (not in a loop), so ``_still_active``
closes over the function parameter directly without any
loop-variable-capture risk. Drop the rebind.
q-5 — the inline soft-cap comment restated rationale already covered by
the ``_WATCH_QUEUE_SOFT_CAP`` block-comment at module scope and dragged
in a tombstone reference to the deleted ``_watch_pending`` path. Trim
to one line stating only the WHY (drop-oldest because latest output is
most useful). Leave the ``set_watch_runner`` docstring's operational
detail at lines 1356-1378 alone — trimming further risks losing the
``valid_until`` predicate semantics.
Closes review finding q-4.
The closure built inside ``server.py``'s ``_watch_restore_fn`` is the
new contract surface introduced by the switchover — it constructs a
fresh ChatSession, calls ``session.resume(ws_id)`` to adopt the
original ws_id, re-registers the dispatch closure via
``set_watch_runner``, and returns ``WatchRunner.get_dispatch_fn`` for
the runner to invoke directly. No automated coverage exists today;
a future refactor (e.g. swapping ``manager.create + session.resume``
for ``manager.open``) could silently break the watch-restore pipeline.
Adds ``test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session``
to ``tests/test_watch_integration.py`` — drives the full restore path:
persists a kickoff message for the original ws_id, fires
``_dispatch_result`` against a runner with no registered dispatch fn,
asserts the restore_fn ran exactly once, the rehydrated session is a
distinct object that adopted the original ws_id, and the watch payload
landed on the rehydrated session's NudgeQueue (not on the original).
Closes review finding perf-1.
The watch dispatch closure's ``valid_until`` predicate fires once per
watch entry at every drain seam — on the chat-loop hot path. It only
needs the ``active`` flag, but ``storage.get_watch`` runs a full-row
``SELECT *`` and marshals the result into a dict. At the typical drain
depth (cap-50 + a busy chat loop) that's ~50 throwaway dict allocations
per drain pass for one boolean.
Adds ``StorageProtocol.is_watch_active(watch_id) -> bool`` plus
SQLite + Postgres implementations doing a single-column
``SELECT active FROM watches WHERE watch_id = ?`` (returns False on
missing row). ``_still_active`` in ``ChatSession.set_watch_runner``
now calls that instead of indexing into the full row.
Test stubs that mocked ``get_watch`` for the predicate are converted
to mock ``is_watch_active`` directly. Bulk variant deferred — single-row
fix is sufficient at typical drain depths.
Closes review findings perf-2, q-3, bug-3.
The watch dispatch closure's soft-cap pre-check materialised the whole
queue snapshot via ``pending(channel="any")`` only to throw away the
text and count the type — wasteful at typical drain depths (cap-50 +
mixed producers means a 50-tuple allocation per fire just to read a
length). The other half of the cap pair (``drop_oldest_by_type``)
walked the *whole* queue regardless of channel, so a future producer
that enqueued ``"watch_triggered"`` on a different channel could be
dropped by the watch cap, and vice versa — silently surprising once
that producer existed.
Adds ``NudgeQueue.count_by_type(nudge_type, channel=None) -> int`` that
walks ``_items`` once under the queue lock without materialising
tuples; extends ``drop_oldest_by_type`` to take an optional ``channel``
filter so both halves can agree on the entry set being capped. The
watch dispatch closure now passes ``channel="any"`` to both —
consistent with where the closure enqueues — so a future channel split
can't bleed across producers.
Adds ``TestCountByType`` mirroring the existing ``TestDropOldestByType``
shape, plus a ``test_drop_oldest_by_type_channel_filter`` case pinning
the new optional argument's behaviour.
Closes review finding q-1.
The live-marker scaffold in ``tests/test_watch_live.py`` couldn't actually
run as written: the ``live_client`` / ``live_model_id`` fixtures it
referenced live in ``tests/test_server_live.py`` at ``scope="module"``,
not on a shared ``conftest.py``, so the file would have ImportError'd
at collection if anyone ever tried ``pytest -m live`` against it.
Lifting the fixtures into a shared conftest is a larger refactor
than R9 justifies — the deterministic envelope-arrival contract is
already pinned end-to-end by ``test_watch_fires_then_user_send_drains_envelope``
and ``test_three_back_to_back_watch_fires_drain_into_one_turn`` in
``test_watch_integration.py`` (real ChatSession + real WatchRunner +
real chat-loop drain). The model-quality-of-response leg is genuinely
manual; the plan doc's R9 entry is updated locally to reflect that
deferral.
Closes review finding bug-1.
The shared ``sanitize_payload`` regex preserved TAB/LF/CR so multi-line
watch shell output kept its layout — necessary for the watch path, but a
correctness gap for the idle_children formatter, which renders the
user-controlled ``name`` field as a single bullet item. A child name
with an embedded ``\n`` would split the bullet across two rendered rows
and let a hostile name forge a fake sibling entry in the listing.
Splits the regex in two: ``_NAME_CONTROL_CHARS`` strips TAB/LF/CR
(used by the new ``sanitize_name`` helper for single-line name fields),
``_PAYLOAD_CONTROL_CHARS`` keeps the existing permissive shape (used by
``sanitize_payload`` for multi-line watch payloads).
``format_idle_children_nudge`` now calls ``sanitize_name``.
Adds ``test_newline_in_name_does_not_forge_extra_bullet`` — feeds a
hostile name with embedded ``\n`` + bullet-shaped continuation, asserts
the rendered listing still has exactly N bullet rows for N children
(no forged sibling), and the hostile newline got flattened to an inline
space. Adds a ``TestSanitizeName`` class mirroring the existing
``TestSanitizePayload`` shape for the new strict variant.
The deleted comment claimed the closure may be registered "under the
rehydrated workstream's id, which may differ from the original ws_id we
restored against" — but ``ChatSession.resume(ws_id, fork=False)`` adopts
the parameter as the session's id at session.py:1682, so they match
exactly post-resume. The lookup works because the ids are equal, not
because they may differ.
The accessor name ``get_dispatch_fn`` is self-explanatory; no replacement
comment is needed (per the project's "default to no comments" rule).
Adds two boundary-crossing integration tests and one live-marker
scaffold for the watch switchover landed in the previous commits:
tests/test_watch_integration.py — drives a real ChatSession + real
WatchRunner end-to-end (LLM stubbed) through the unified pull-model
chat-loop drain seam. Pins:
- test_watch_fires_then_user_send_drains_envelope: a synchronous
WatchRunner.dispatch fire enqueues "watch_triggered" on "any";
session.send drains the entry into the user message's _reminders
side-channel — confirms the envelope splice path.
- test_three_back_to_back_watch_fires_drain_into_one_turn: pins the
intentional behavioural delta from the plan section 3.4 / risk
register R3 — N back-to-back fires now produce ONE assistant turn
with N _reminders entries, not N successive turns.
tests/test_watch_live.py (new file, single test, marked @pytest.mark.live):
risk register R9 verification recipe — confirm a real LLM handles a
<system-reminder>-framed watch payload sensibly. Collects under the
regular -m "not live" run; the user runs it on demand against an
Anthropic-backed config.
Implements watch-switchover plan section 5.2 (integration) and step 11
(live scaffold).
Replaces the deleted tests/test_watch_dispatch.py with a focused
14-test suite exercising the closure that ChatSession.set_watch_runner
now constructs (per the previous commit's switchover). Each test
pins one assertion:
- enqueue shape: ("watch_triggered", text, "any") on the per-session
NudgeQueue; not on user / tool channels
- producer-side sanitisation strips control / bidi / zero-width chars
and angle-bracket tag breakers; preserves TAB/LF/CR so multi-line
shell output keeps its layout (R8); empty-after-strip → no enqueue
- soft-cap drop-oldest at _WATCH_QUEUE_SOFT_CAP with a queue_full
WARNING log; non-watch entries on the same queue are not collateral
damage
- valid_until predicate drops on inactive / missing / storage-raises;
delivers when active (counter-test)
- concurrent enqueues across two threads stay bounded under the
3-acquisition count-then-drop window
Implements watch-switchover plan section 5.1 / step 9. No production
changes — pure test rewrite.
Replaces the bespoke _make_watch_dispatch / _watch_pending /
_dispatch_pending_watch / _MAX_WATCH_CHAIN machinery with a single
NudgeQueue.enqueue("watch_triggered", ...) call inside
ChatSession.set_watch_runner. Watch results now drain at the same
<system-reminder> envelope seams as every other metacog nudge
(USER_DRAIN, TOOL_DRAIN, IdleNudgeWatcher IDLE wake) — no separate
worker-spawn, no recursive watch chain, no per-session queue.Queue.
The dispatch closure built inside set_watch_runner carries:
- producer-side sanitize_payload over the whole formatted message
before enqueue, so steering-vector / control-char shell output
can't tamper with the envelope at interpolation time
- a soft cap of 50 entries on per-session "watch_triggered" depth
via the new NudgeQueue.drop_oldest_by_type, replacing the prior
_watch_pending maxsize=20 + _MAX_WATCH_CHAIN=5 bounds; drop policy
is drop-oldest (latest output most useful), logged at WARNING
- a valid_until predicate that re-checks
storage.get_watch(watch_id)["active"] at drain time so a cancelled
watch's last splat doesn't ride out a future wake
Behavioural delta documented in the plan section 3.4: N back-to-back
watch fires now drain into ONE assistant turn responding to all N
(via the envelope splice) instead of N separate send turns. This is
intentional — fewer model invocations for noisy watches, and uniform
with the rest of the metacog pull-model surface introduced by #482.
Implements watch-switchover plan steps 5-8. Server-side simplifications
let the previously-load-bearing _make_watch_dispatch (47 lines), its
session_worker.send import, and the chat-loop _dispatch_pending_watch
seam at the no-tools IDLE branch all disappear. The obsolete
tests/test_watch_dispatch.py and the wake-tag test in test_session.py
(both pinning contracts that no longer exist) are removed; the
NudgeQueue-based replacement plus an integration test land in the
following commit.
Widens the per-workstream dispatch fn signature from ``(message,)``
to ``(message, watch_id)``. The runner now passes the originating
``watch_id`` through ``_dispatch_result`` so dispatch closures can
capture per-watch metadata at fire time — the upcoming switchover
needs this for the ``valid_until`` predicate that re-checks
``storage.get_watch(watch_id)["active"]`` before a stale entry rides
out a wake.
Also adds ``WatchRunner.get_dispatch_fn(ws_id)`` as the public
accessor used by the server-side restore path to retrieve the
closure that ``set_watch_runner`` constructed during workstream
rehydrate (avoiding private-attr access into ``_dispatch_fns``).
Implements watch-switchover plan step 4 plus risk register R4.
The pre-existing single-arg callers (``_make_watch_dispatch`` and
``set_watch_runner``'s ``dispatch_fn=`` fallback) get replaced
in the next commit; their mypy types are ``Any`` today so the
type mismatch isn't caught at this step.
Renames _sanitize_child_name to sanitize_payload and widens it to be
the shared producer-side sanitiser for both idle_children and the
incoming watch_triggered nudges. The regex now skips TAB / LF / CR
so multi-line shell output rendered into a watch payload keeps its
line structure when sanitised as a whole formatted message — the
pre-switchover code path collapsed multi-line output to one line.
Adds the watch_triggered entry to _NUDGE_MAP alongside idle_children
so ``_NUDGE_MAP``-as-registry consumers (should_nudge gating, future
audit / UI tagging) recognise the type. Body is empty — payload
comes from the producer (the watch dispatch closure), same shape as
idle_children.
Implements watch-switchover plan section 3.2 plus risk register R8
(TAB/LF/CR exclusion) and step 3 (_NUDGE_MAP registration).
Adds an atomic drop-oldest-by-type operation to NudgeQueue used by
producers that need a per-type soft cap on their own queue depth.
The watch dispatcher (next commit in this stack) is the first user:
when "watch_triggered" saturates, the dispatch closure drops its
oldest entry under the queue lock so the count snapshot and drop
can't interleave with a concurrent enqueue from the same producer.
Implements watch-switchover plan section 3.1 — the producer-side soft
cap takes the place of the deleted _watch_pending maxsize=20 bound.
Other producers (idle_children, advisories) have natural rate limiters
already, so the helper is opt-in per producer rather than a global cap
in enqueue itself.
Three Copilot findings on PR #483 (commit dad98c0); one rejected as a
false positive.
- mcp_client.py:1189 — pool notification handler's exception path
used ``log.warning(..., exc_info=True)`` which serializes the
chained ``httpx.Request.headers`` carrying ``Authorization: Bearer
<token>`` into Sentry / faulthandler frame captures. Same threat
model as the round-1 sec-1 dispatch-path fix, applied to a site
the original review missed. Now logs structured fields only
(server, user, exc type) without ``exc_info``.
- mcp_client.py:1202 — ``_connect_one_pool``'s handshake step used
``asyncio.wait_for(session.initialize(), ...)``, the same Python
3.11 + anyio cross-task-cancel-scope anti-pattern that the
Phase 7 round-3 q-1 fix removed from the discovery step (and that
f6a3b66 originally addressed for ``_safe_close_stack``). Pre-
existing Phase 5 code, but the same latent bug class — a 401
during initialize() under 3.11 would surface ``RuntimeError:
Attempted to exit cancel scope in a different task`` as the
SDK's TaskGroup unwinds. Switched to ``async with asyncio.timeout(...)``
matching the discovery step's pattern.
- mcp_client.py:1522 — renamed loop tuple-unpack variable
``_server_name`` → ``server_name`` in ``_rebuild_user_tool_map``.
The leading underscore conventionally signals "intentionally
unused", but the variable is read at the assignment a few lines
below. Two other ``_server_name`` unpacks in this file (1410,
3111) genuinely don't use the value and keep the underscore.
Rejected as false positive:
- test_mcp_user_catalog.py:58 (github-code-quality bot, "Statement
has no effect"): ``await task`` inside ``contextlib.suppress(
BaseException)`` is the standard pattern for cleanly draining a
cancelled task. The bot's static analysis treats ``await`` of a
result that's discarded as a no-op statement, but ``await`` here
triggers cancellation propagation and waits for the task to
finish — load-bearing in the fixture's teardown. No change.
Verified on Python 3.11 (``/tmp/venv311``) and 3.13 (``.venv``):
ruff + mypy clean, full test suite green.
Light up production reachability of pool dispatch (RFC §3, invariant 8)
by widening the public catalog API to optionally take a ``user_id``:
- ``MCPClientManager.get_tools(user_id=None)`` returns the merged
static + per-user pool view when ``user_id`` is supplied; the default
preserves the legacy global-only contract.
- ``is_mcp_tool(name, *, user_id=None)`` extends the lookup to the
per-user ``_user_tool_map``. Pool tools become reachable from
``ChatSession._prepare_tool`` only when the session-bound user_id
flows through — flipping invariant 8 from "must hold" to "satisfied".
- Listener identity becomes ``(user_id, callback)``. Static-path
changes fire ALL listeners (admin + every user); pool-entry
changes fire only matching-user + admin (``None``) listeners.
RFC §3.3.
- Pool sessions discover their tool list on first connect
(``_connect_one_pool`` → ``await session.list_tools()``); the
notification closure binds to ``(user_id, server_name)`` so
push-driven ``list_changed`` updates target the correct user's
catalog. R6 verified empirically: ``list_tools()`` 401 propagates
through anyio TaskGroup unwinding, no hang — plain ``await`` is
fine, no carrier-race shape needed for discovery.
- ``_evict_session`` drops ``entry.tools`` and rebuilds the user's
index so an evicted-then-reconnected session doesn't carry
stale catalog state.
- ``web_search.resolve_web_search_client`` refuses
``auth_type=oauth_user`` backends (per-node web search can't
carry per-user tokens).
Resources / prompts pool dispatch deferred to Phase 7b — invariant 8
is satisfied by the tool path alone, and the resource/prompt path
needs sibling ``_dispatch_pool_resource_sync`` /
``_dispatch_pool_prompt_sync`` helpers each with their own
carrier-race plumbing (~400 LOC). Phase 7b will follow the patterns
established here.
CLI sessions default ``user_id=""`` and so cannot use oauth_user
MCP servers — documented limitation; users must use the web UI.
Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: get_tools(user_id) was iterating _user_pool_entries from sync
threads while the mcp-loop concurrently mutated it (RuntimeError:
dictionary changed size during iteration). Now reads from a sibling
_user_tools dict updated atomically by _rebuild_user_tool_map.
- bug-2: _close_pool_entry_if_idle (LRU/TTL eviction) skipped the
catalog cleanup that _evict_session does — stale tools persisted
in _user_tool_map and ChatSession's tool list never rebuilt. Now
mirrors _evict_session.
- perf-1: _last_pool_notification_refresh debounce dict was never
pruned in either eviction path. Now popped alongside the entry.
- perf-3: web_search resolver was issuing a sync SQL query per LLM
turn to gate oauth_user backends. Now reads from the cached
in-memory config.
- sec-1: bearer token could leak into exc_info-rendered tracebacks
via Sentry/faulthandler. log.debug now uses structured fields,
not exc_info.
- sec-2: tools-per-server response now capped at 1000 (defensive,
mirrors _MAX_ERROR_LEN / _MAX_INSUFFICIENT_SCOPE_REPORTED).
- Test cleanup: dropped two listener fan-out tests duplicating
test_mcp_client.py coverage; renamed test_pool_session_notification_handler
to match its actual scope (_refresh_pool_server_tools); removed
stale comments referencing /tmp/r6-spike*.py scratchpads and a
misleading "copy-on-write" comment.
Round-2 pre-push review fixes (focused single-pass review applied):
- round2-1: bug-2's catalog-cleanup block in _close_pool_entry_if_idle
had no integration test (exactly the failure mode flagged in
feedback_tests_through_boundaries.md). Added
test_close_pool_entry_if_idle_clears_catalog_and_fires_listener
driving the LRU/TTL eviction path through real streamablehttp_client +
MockTransport. Negative-test verified: reverting the
_rebuild_user_tool_map / _notify_user_tool_listeners calls makes
the new test fail.
- round2-3: documented the _oauth_user_server_names cache invariant
in add_server_sync / remove_server_sync docstrings. Cache is
reconcile_sync's sole owner — direct callers leave it stale, but
_db_servers_to_config strips oauth_user rows so production paths
are unaffected. Static→oauth_user transitions correctly leave the
name in the cache because remove_server_sync drops the static
connection, not the cache identity.
- round2-6: strengthened test_rebuild_user_tool_map_populates and
test_rebuild_user_tool_map_drops_empty_user to assert on the
_user_tools sibling cache (bug-1 fix). Without this, a future
revert dropping the sibling write would still pass the unit
tests because get_tools coverage lives in separate tests.
Round-3 full-stack review fixes (multi-stage review on the final
state caught what the layered apply passes missed):
- q-1 REGRESSION: pool tool-discovery used asyncio.wait_for around
session.list_tools(), the exact pattern the f6a3b66 fix (and
feedback_asyncio_timeout_vs_wait_for.md) put in place to avoid.
Python 3.11's asyncio.wait_for wraps the inner coroutine in a
fresh task → cross-task scope-exit when the SDK's anyio TaskGroup
unwinds on a 401. Switched to `async with asyncio.timeout(...):`
pattern used by _safe_close_stack.
- sec-2: TOCTOU in _connect_one_pool — entry.tools was published
(via _rebuild_user_tool_map + listener fan-out) BEFORE entry.session
was assigned. A sync-thread reader could observe a tool whose
backing entry has session=None. Defence-in-depth — dispatch
re-fetches its own token and lazy-reconnects on session=None — but
reordering catches the race at the source. entry.session now
publishes BEFORE catalog visibility.
- bug-1: _close_pool_entry_if_idle's _user_pool_locks.pop ran
unconditionally after the try/finally, but the early-return
branches (entry None on re-check, in_flight > 0 under lock) skip
it via Python's return-through-finally semantics. The lock was
never popped on those paths. Now gated behind an `evicted` flag
set only on the success path; in_flight > 0 leaves the lock for
the active dispatcher to reuse, entry-None races leave the lock
for re-allocation by _ensure_pool_entry. Comment now describes
the actual semantics, not the original promise.
- bug-2: softened the _rebuild_user_tool_map docstring's atomicity
claim. The two-dict write is technically non-atomic across Python
statements; in practice the window is sub-microsecond on the
mcp-loop with no awaits between writes, and the listener fan-out
fires AFTER both writes complete. Docstring now says "back-to-back
on the mcp-loop" instead of "atomically alongside".
- q-3: dropped `hasattr(mcp_client, "server_auth_type")` defensive
check in web_search.py. The method ships in this commit; the
hasattr created a silent fallthrough that would let a future
rename silently re-enable oauth_user backends.
- q-4: surfaced the CLI / empty-user_id limitation in a docstring
comment at ChatSession.__init__'s self._user_id assignment. The
note previously lived only inside is_mcp_tool's docstring — a
future maintainer wiring CLI features against MCP pool servers
wouldn't think to read is_mcp_tool to find the constraint.
- q-2 + q-5: deleted a tautological duplicate test in
test_mcp_user_catalog.py whose docstring claimed to test
ChatSession.close but never instantiated a ChatSession (the
manager-level identity semantics are already covered by
test_listener_identity_includes_user_id in the same file and by
test_session_close_removes_listener_with_same_user_id in
test_mcp_client.py which DOES drive a ChatSession). Reworded a
misleading "fixture provides only 5s" comment to point at the
actual `_run_on_loop(..., timeout=5)` site.
- q-6: the `self._user_id or None` collapse repeated at 8 sites
across session.py. Cached once at __init__ as
``self._mcp_user_id`` (since ``_user_id`` is set once and never
mutated); 8 call sites now read the cached value. The empty-
string-is-CLI-sentinel invariant is documented at the assignment
site, not re-asserted at each consumer.
Deferred to follow-up:
- sec-1: a hostile MCP server bound to user-A could craft a
tool.name containing `__` to synthesize a prefixed-name collision
in user-A's own catalog. Bounded impact: cross-tenant dispatch is
prevented by the per-tenant token gate in _dispatch_pool, and
user-B's get_tools(user_id="B") never includes user-A's pool
entries. The fix needs policy decisions (reject vs. sanitize)
and touches _mcp_to_openai which is shared between static and
pool paths; better discussed in its own follow-up where the
policy applies uniformly to static-path servers too. The threat
model already requires user-A to have consented to a malicious
server, who has many more dangerous vectors than tool-name
shenanigans.
Test count delta: +31 tests (5435 → 5466, ``-m "not live"``; one
test deleted in round-3 apply per q-2):
- ``tests/test_mcp_client.py`` +20 (per-user catalog state, listener
identity, session thread-through)
- ``tests/test_mcp_user_catalog.py`` +9 NEW (integration tests
driving real ``streamablehttp_client`` + ``httpx.MockTransport`` per
invariant 14: discovery on connect, user isolation, eviction +
reconnect, LRU/TTL eviction (round2-1), R6 401-propagation
regression, static byte-identical canonical regression; review
passes dropped duplicate listener fan-out tests from earlier
drafts whose coverage lived in test_mcp_client.py)
- ``tests/test_web_search.py`` +2 (oauth_user backend rejection +
static backend acceptance regression; updated to use the new
``server_auth_type`` in-memory accessor)
Three confirmed findings from the PR #482 bot review pass.
* **Copilot (idle_nudge_watcher.py)**: ``IdleNudgeWatcher`` was gating
wake dispatch on ``len(_nudge_queue) == 0`` (any channel), but
``deliver_wake_nudge_from_queue`` only drains ``USER_DRAIN``. A
``"tool"``-channel entry queued by ``_queue_tool_advisory`` would
pass the gate, spawn a wake daemon, and immediately no-op at the
drain guard — repeating on every IDLE event for as long as the
tool entry sat unconsumed. No correctness bug (the no-op return
prevents bad state) but a wasted thread spawn per IDLE. Fixed by
gating on ``has_pending(USER_DRAIN)``; tool-only queues no longer
trigger the wake path.
* **Copilot (coordinator_idle_observer.py)**: docstring referenced
the old module path ``turnstone.core.metacognition.IdleNudgeWatcher``;
the class moved to ``turnstone.core.idle_nudge_watcher`` in q-3 of
the apply-pass.
* **Copilot (nudge_queue.py)**: ``has_pending`` docstring cited
``ChatSession.deliver_wake_nudge_from_queue`` as its caller, but
that method calls ``drain(USER_DRAIN)`` directly — no production
caller used ``has_pending`` until this commit. Updated to point
at the now-actual caller (``IdleNudgeWatcher``).
* **github-code-quality (test_nudge_queue.py)**: false positive on
``test_channel_is_required`` — the no-channel ``q.enqueue("a", "1")``
call is wrapped in ``pytest.raises(TypeError)`` to verify the
validation contract. No code change.
5571 non-live tests pass; ruff + mypy clean.
Round-2 review caught 11 confirmed findings on the 3-commit metacog stack;
this commit applies them.
* **bug-1 (major)**: Wake source tag was leaking onto real user messages
flushed during a wake send. ``_append_user_turn`` and ``send`` now
take an explicit ``from_wake: bool`` parameter — only the wake's
synthesized first turn passes True, so ``_flush_queued_messages``'s
real user input no longer inherits the audit tag. Regression test
pins the contract.
* **perf-1 (major)**: ``CoordinatorIdleObserver._maybe_enqueue`` was
issuing list_workstreams + visible_memory_count storage queries
before the cheap cooldown gate could short-circuit. New
``_cooldown_allows`` read-only peek runs first; storage queries only
fire when cooldown actually allows the nudge.
* **q-1 (major)**: Added the missing coord-side integration test that
exercises ``CoordinatorIdleObserver`` + ``IdleNudgeWatcher`` together
in the production install order against a real ``SessionManager``,
protecting the subscription-order contract from silent regression.
* **perf-2/3 (minor)**: Cap check moved above ``_last_assistant_used_wait``;
``_fire_counts`` restructured as ``dict[str, dict[str, int]]`` keyed by
ws_id so the leave-IDLE existence check is O(1).
* **perf-4 (minor)**: ``NudgeQueue.drain`` fast-paths the all-match
case (the common one for chat-loop drain seams) by swapping
``self._items`` directly instead of allocating a fresh ``kept``
deque + per-entry append.
* **perf-5 (minor)**: Wake's synthesized empty user turn no longer
writes a content-empty row to the conversations table — the
``_source`` audit tag isn't column-backed and the side-channel
reminder is stripped before persist, so the row would carry nothing.
* **q-3 (minor)**: Split ``IdleNudgeWatcher`` + ``install_*`` /
``shutdown_*`` helpers out of ``metacognition.py`` into the new
``turnstone/core/idle_nudge_watcher.py``; metacog stays a
static-template module.
* **sec-1 (nit)**: Widened ``_sanitize_child_name``'s control-char
regex to cover Unicode bidi-overrides, zero-width chars,
line/paragraph separators, BOM, and tag chars.
* **q-4/q-5 (nits)**: Docstring referenced the wrong peek primitive
(``has_pending`` → ``len()``); ``_last_assistant_used_wait``'s
``session`` parameter now typed ``ChatSession``.
5571 non-live tests pass; ruff + mypy clean.
Adds the first concrete consumer of the wake trigger: when a coordinator
goes IDLE while interactive children are still running, a
``CoordinatorIdleObserver`` enqueues an ``idle_children`` nudge that the
``IdleNudgeWatcher`` then dispatches as a synthetic empty-user-turn
``send``. The model receives a system-reminder body listing the active
children (capped at 6 inline + 32 in the suggested ``wait_for_workstream``
call) and a nudge to block on them rather than reply prematurely.
Observer gates (in order): coord-only filter, skip if last assistant
turn used ``wait_for_workstream``, per-(ws, nudge_type) hard cap (3)
that resets only on non-wake leave-IDLE, active-children query,
``should_nudge`` cooldown. Console lifespan registers the observer
BEFORE the watcher so subscriber-fire order has the observer
enqueueing first on the same IDLE event.
Adds an opt-in ``valid_until`` predicate on ``NudgeQueue.enqueue``
(R9 from the design risk register) — drain re-checks the predicate
outside the queue lock; falsy / raising drops the entry without
delivering it. ``deliver_wake_nudge_from_queue`` now drains inline
before synthesizing the empty user turn so a stale predicate-drop
doesn't leave the wake send with empty content; ``_attach_pending_user_reminders``
consumes the pre-drained reminders via ``_wake_drained_reminders``.
The observer's ``valid_until`` uses ``count_workstreams_by_state``
(boolean check, no row fetch) instead of full ``list_workstreams``,
keeping the chat-loop user-attach path off the heavy query.
User-controlled child workstream names are sanitized
(``_sanitize_child_name``) before interpolation so a name like
``</thinking>...`` can't steer the model's reasoning channels through
the rendered body — the wire-boundary ``escape_wrapper_tags`` only
covers ``<system-reminder>`` / ``<tool_output>`` envelopes.
Adds the third metacog channel: an out-of-band wake that converts a
workstream's IDLE transition into a synthetic empty-user-turn ``send``
when the session has any-channel nudges queued. The ``IdleNudgeWatcher``
subscribes to ``SessionManager.subscribe_to_state``; on IDLE it dispatches
via ``session_worker.send`` with a no-op ``enqueue`` callback so a
busy-worker race silently drops without spawning a competing worker.
Wake-source-tag plumbing on ``ChatSession`` short-circuits metacog
detection on the synthetic empty input, suppresses queue producers
during the wake's own tool dispatch, and stamps ``_source = "system_nudge"``
on the synthetic user-message for audit / replay distinction. The tag
is saved / restored across ``_dispatch_pending_watch`` so watch chains
recursing off the wake are processed as normal user turns rather than
inheriting the wake's guards.
Generic ``install_idle_nudge_watcher`` / ``shutdown_idle_nudge_watchers``
helpers wire the watcher into both the interactive and coord lifespans
via a single ``app.state`` registry so both surfaces share the same
teardown contract.
Foundation for PR 3 (CoordinatorIdleObserver + idle_children formatter)
and PR 4 (watch dispatcher switchover).
Replaces the dual `_pending_user_advisories` / `_pending_tool_advisories`
list pair with a single channel-tagged `NudgeQueue` per session.
Producers tag entries with a channel ("user", "tool", or "any");
consumers drain by channel filter at their existing seams. Foundation
for the wake trigger (PR 2) and coordinator idle-children nudge (PR 3).
Existing nudges (start, correction, completion, denial, resume,
tool_error, repeat) keep their wire shape and drain timing — zero
behavior change. Cancel paths now `clear()` the unified queue.
Python 3.11's ``asyncio.wait_for`` wraps its inner coroutine in a fresh
``asyncio.Task`` via ``ensure_future``. When the inner is
``stack.aclose()`` on an ``AsyncExitStack`` containing
``streamablehttp_client(...)`` (anyio cancel scopes entered in the
calling task), the fresh task's attempt to exit those scopes raises
``RuntimeError('Attempted to exit cancel scope in a different task
than it was entered in')``. Python 3.12+ rewrote ``wait_for`` to use
``asyncio.timeout`` internally — runs in the current task — so 3.13
ran the same code path successfully.
Symptom on 3.11: integration tests where ``session.initialize()``
returns 4xx (e.g., 403 insufficient_scope tests) hit
``_connect_one_pool``'s ``except Exception:`` handler →
``_safe_teardown_on_connect_failure`` → ``_safe_close_stack`` → cross-
task RuntimeError. The ``concurrent.futures._base.CancelledError``
that surfaces in ``future.result(timeout=...)`` is the cascade
fallout from the asyncio loop's exception handler reacting to the
unretrieved-task-exception.
Fix: use ``asyncio.timeout`` instead of ``asyncio.wait_for`` for the
5s aclose bound. Equivalent semantics, current-task execution, works
on 3.11+. The 5s guard against ``aclose()`` hanging on a broken stack
is preserved.
Verified on Python 3.11.14 (full suite 5427 passed) and 3.13.7 (full
suite 5427 passed); all 9 integration tests pass on both.
Pre-existing bug — surfaced only after the marker fix in 5c9850c
let CI's test (3.11) actually run the 4xx tests.
Two pre-existing defects in the Phase 6 pool dispatch path that only
manifest when a pooled session is reused for a second dispatch:
1. The per-dispatch _AuthCapture allocated in _dispatch_pool was wired
into the httpx response hook only at first connect (via
_connect_one_pool). On a reused session no fresh connect runs, so
the hook continues writing to the original-connect's carrier while
the new dispatch inspects an empty carrier — auth_401/403 silently
misclassified to "other", refresh-and-retry never fires.
2. Even with the carrier on the entry (so the hook writes to a stable
reachable object), session.call_tool itself hangs forever on
upstream 4xx for reused sessions. Trace: SDK's spawned
handle_request_async raises HTTPStatusError, the outer
streamablehttp_client TaskGroup cancels post_writer, post_writer's
finally aclose's read_stream_writer, BaseSession's _receive_loop
exits and enters its CONNECTION_CLOSED-fanout finally. anyio's
send_nowait skips waiting receivers with pending_cancellation; the
dispatch task (created by run_coroutine_threadsafe for the reuse
case) is NOT in any cancel-scope chain, so the send "delivers" but
the receiver's Event is set on stale state — receive() never
wakes. Test 21 doesn't hit this because its 401 happens during
initialize, in the same task that opens streamablehttp_client, so
the cancel scope DOES propagate.
Fix:
- Move _AuthCapture ownership to PoolEntryState (and asyncio.Event
alongside, allocated lazily on the mcp-loop). The hook closes over
entry.auth_capture at first connect and stays valid across
dispatches; reset under open_lock before each call_tool.
- Race session.call_tool against the carrier's fired_event in
_dispatch_pool_with_entry. If the event wins (hook captured 4xx
before SDK propagated), cancel call_tool and raise an internal
_CarrierAuthSignal — _classify_failure resolves to auth_401/403
via the carrier's status, the dispatcher evicts the broken
session, and the cross-task retry handshake reconnects on a fresh
bearer.
Adds tests/test_mcp_pool_auth_integration.py::test_integration_pool_reuse_401_refresh_and_retry_succeeds
which drives the reuse path through real upstream + real SDK and is
the structural gate against this class regressing. Negative-tested
twice: revert PoolEntryState.auth_capture → test fails (carrier
empty); revert the race → test times out (SDK hang).
Also drops the @pytest.mark.asyncio decorator (replaced with
@pytest.mark.anyio) on four tests in test_mcp_pool_auth_introspection.py.
The project depends on anyio's pytest plugin (anyio is in deps);
pytest-asyncio is NOT a project dep and CI's test (3.13) failed on
those four. Local pytest happened to pick it up via system Python.
Found via Copilot review on PR #481.
Phase 6 of OAuth-MCP. Recovers upstream 401/403 from MCP servers via a
capturing httpx_client_factory: an async response hook records 4xx
status + WWW-Authenticate header into a per-dispatch carrier before
the SDK's post_writer swallows the underlying httpx.HTTPStatusError.
Splits _classify_failure into auth_401 (refresh-and-retry once) vs
auth_403 (parse insufficient_scope, emit mcp_insufficient_scope with
parsed scope set). The 401 retry runs on a fresh asyncio.Task via
run_coroutine_threadsafe in _dispatch_pool_sync, escaping the anyio
cancel-scope state of the prior dispatch's TaskGroup.
WWW-Authenticate parsing extracted to a new mcp_http_parsers module
with an RFC 7235 challenge tokenizer (replaces hand-rolled substring
scanners). Two-layer defense against multi-Bearer-challenge injection:
the hook uses get_list("www-authenticate")[0] to drop attacker's
second challenge, the parser truncates at challenge boundary as
belt-and-braces. Scope set capped at 32 entries before hitting the
audit row or the LLM-visible structured-error JSON.
Auth failures (401/403) never trip the per-server circuit breaker
(server-only breaker invariant). Static path remains byte-identical.
_PgRefreshLock untouched. Pool dispatch still reachable from the
agent loop only via Phase 7 catalog scoping; Phase 6 behaviour is
testable via direct call_tool_sync.
5557 tests pass. 33 tokenizer unit tests in tests/test_mcp_http_parsers
cover the RFC 7235 grammar + the scope/error wrappers + the 4 KB input
cap. 7 integration tests in tests/test_mcp_pool_auth_integration drive
real upstream 401/403 through streamablehttp_client + a FastMCP
subprocess fixture — the structural exit gate that makes
HTTPStatusError-injection-only unit tests insufficient.
Models often emit page references in the standard man-page form
(``printf(3)``, ``open(2)``, ``perlfunc(3pm)``) rather than splitting
them into ``page`` + ``section`` args. The page-name sanitizer was
rejecting the parens as invalid input, killing the call. Parse the
section out of the page string before sanitization (explicit
``section`` arg still wins) and widen the section validator to accept
multi-letter suffixes like ``3pm`` / ``3perl`` that already appear on
real systems.
Phase 5 PR #479 review fix-up. Three review rounds (bot + two internal
multi-stage /review) caught:
- _PgRefreshLock now allocates a per-instance ThreadPoolExecutor instead of
a module-global single-worker one. The global shape preserved psycopg2
thread-affinity but serialized every advisory-lock acquire on the node
behind one thread, even for unrelated (user, server) keys.
- get_user_access_token_classified flips to `async with lock, pg_lock:` so
concurrent same-key callers serialize on the in-process asyncio.Lock
before allocating the pg_lock's per-instance executor + spin loop. N
concurrent same-key callers collapse to one executor allocation.
- _drain_orphan_pg_lock no longer re-awaits the cancelled asyncio Future
from `__aenter__`. It receives the underlying concurrent.futures.Future
and re-wraps it via asyncio.wrap_future, getting an independent asyncio
Future tied to the worker outcome. This way cancellation of the awaiter
doesn't poison the drain's wait, and the drain genuinely waits for the
worker to settle before deciding whether to call cm.__exit__.
- Module-level _pg_refresh_drain_tasks set holds strong refs to in-flight
drains (asyncio's task set is weak — fire-and-forget tasks could be GC'd
mid-cleanup; RUF006 hazard).
- Drain narrows except clauses to Exception so a drain-task cancellation
records as cancelled instead of being silently logged as 'completed
normally with no acquire'.
Test integrity (was a major finding in round 2 — old generator-based cm
let the test pass via GC finalization timing rather than drain logic):
- New _ObservableLockCm class-based context manager whose __exit__ is a real
observable method (records call args + thread). Distinguishable from
GeneratorExit thrown by GC of a generator-based cm.
- Strong external ref to the cm via created_cms list — keeps cm alive past
the test's awaits, so a no-op drain genuinely fails the assertion rather
than papering over via GC timing.
- Deterministic drain wait via _pg_refresh_drain_tasks gather — no
fixed-duration sleeps.
- _run_cancel_scenario helper drops the duplicated setup between the two
cancellation tests.
Negative-test verified: replacing _drain_orphan_pg_lock body with `return`
makes test_pg_refresh_lock_cancellation_releases_on_same_thread fail with
'drain did NOT call cm.__exit__ — orphan Postgres lock + open transaction'.
Other fixes: protocol docstring corrected to describe pg_try_advisory_xact_lock
spin + retry (was claiming pg_advisory_xact_lock blocking acquire);
get_user_access_token_classified docstring rewritten for new lock order;
narrow `except BaseException` -> `except Exception` in
test_mcp_user_pool.py concurrent-dispatch helper.
882 tests pass (MCP + auth + storage). ruff + mypy clean.
Phase 5 of OAuth-MCP — adds a per-(user, MCP-server) ClientSession
pool to MCPClientManager alongside the existing static-server path,
gated entirely on the per-server `auth_type='oauth_user'` config.
Pool architecture:
- `_user_pool_entries: dict[(user_id, server_name), PoolEntryState]`
with lazy connect on first dispatch, per-key asyncio.Lock allocated
on the mcp-loop, idle eviction coroutine (default 600s TTL, LRU cap
200), and an `in_flight` counter as the eviction interlock so live
calls can never be torn down mid-flight.
- `_dispatch_pool` runs the token-state machine: missing token →
`mcp_consent_required`; key-rotation decrypt failure →
`mcp_token_undecryptable_key_unknown` with NO consent prompt and NO
auto-delete; expired token → silent refresh under per-(user, server)
advisory lock; refresh failure → revoke + consent.
- `_classify_failure` separates transport (trips breaker) from auth
401/403 (does NOT trip breaker — server-only invariant) from
protocol (no breaker change).
- `entry.open_lock` held only across connect-or-reuse and released
before the `await session.call_tool` so concurrent calls from one
user against one server overlap (validated by Spike 1 scenario 2).
Auth-class failures are fail-soft in Phase 5: any 401/403 surfaced by
the SDK propagates to the agent as a tool error and the next dispatch
reconnects on a fresh refresh. Real introspection of upstream 401/403
is a Phase 6 concern — the MCP SDK's `streamable_http` post_writer
swallows `httpx.HTTPStatusError` upstream, so detecting status from
the response chain requires `McpError(CONNECTION_CLOSED)` payload
parsing or a custom httpx middleware around `streamablehttp_client`.
The mid-flight 401 refresh-retry path and the `mcp_insufficient_scope`
structured error for 403 step-up land together in Phase 6, gated by
an integration test that drives a real upstream 401/403 (the unit-
test injection of `HTTPStatusError` is what masked the production gap
on the first apply-findings pass — the integration test is the
structural gate so the gap can't reopen). RFC §1.5 steps 4-5 and the
phase table in §Implementation phases reflect this scope split.
Multi-node refresh contention:
- New `StorageBackend.acquire_advisory_lock_sync` Protocol method.
SQLite returns nullcontext (single-node, in-process asyncio.Lock
is sufficient). Postgres uses `pg_try_advisory_xact_lock` with
retry on a fresh per-attempt connection, so waiters don't pin pool
connections during the AS roundtrip. Inner try/except + nested
finally ensures conn is always returned to the pool, even when
begin / execute / yield / commit raises mid-body.
- Lock ordering: pg_advisory outer, asyncio.Lock inner. Re-read after
lock collapses cluster-wide contention to one HTTP roundtrip per
(user, server) per refresh window.
- `_PgRefreshLock` enter/exit pinned to a single-worker
ThreadPoolExecutor so SQLAlchemy connection state stays
thread-affine across cancellations.
Token storage refactor:
- `get_user_access_token_classified` returns a tagged TokenLookupResult
(Token / MissingToken / DecryptFailure / RefreshFailed) so the
dispatcher maps each state to the right user-facing error.
- `get_user_access_token` is now a thin wrapper around the classified
variant; the previous duplicated state machine is gone.
Security:
- Pool dispatch + admin endpoints reject `http://` URLs for
`auth_type='oauth_user'` servers (only exact loopback hostnames are
exempt — `*.localhost` is intentionally NOT honored because RFC 6761
localhost-zone resolution is configuration-dependent and could route
bearers to non-loopback IPs via custom resolvers / hosts file /
Docker overlays). Validated at three layers:
`_dispatch_pool` (structured `mcp_oauth_url_insecure` error),
`_connect_one_pool` (defensive ValueError), and
`admin_create_mcp_server` / `admin_update_mcp_server` (400 before
storage write).
- Admin URL change on an oauth_user row purges per-user OAuth tokens
bound to the old URL: bearers are bound (via OAuth resource /
audience) to the URL active at consent time, so silently rebinding
them to a new URL is a token-binding violation. Re-consent forces
fresh issuance for the new resource.
- Encryption-key fingerprints stay in audit logs only; no longer
surfaced in agent-facing error payloads.
User_id thread-through:
- `MCPClientManager.call_tool_sync(..., user_id=None)` (additive;
default None preserves the static path byte-identically).
- `ChatSession._exec_mcp_tool` passes `self._user_id or None`.
- `set_app_state(app_state)` setter wires OAuth state at lifespan
startup, called from both turnstone-server and turnstone-console.
Performance:
- LRU cap eviction iterates `_user_pool_entries` (not
`_user_pool_last_used`) so pre-dispatch entries are eligible.
- Eviction batch closes via `asyncio.gather` instead of serial await.
- `_resolve_pool_target` returns the resolved server row to
`_dispatch_pool` to eliminate the second DB lookup.
- Production reachability of pool dispatch is gated on Phase 7
(catalog scoping) wiring pool tools into `_tool_map`; until then
pool dispatch is reachable only via direct `call_tool_sync` with a
prefixed name (the path the new pool tests exercise).
Hardening parity preserved:
- Static path (auth_type ∈ {none, static}) byte-identical; PR #296
hardening (SDK #2147 mitigations, anyio cancel-scope, stale-session-
and-stack guard, server-only circuit breaker) intact.
- `test_reconnect_preserves_static_state_identity` unchanged + green.
- `MCPTokenStore.get_user_token` does not auto-delete on
MCPTokenDecryptError (key-rotation safety).
- Notification debounce stays manager-level.
- Connect-failure cleanup factored into
`_safe_teardown_on_connect_failure` shared by both connect paths.
Tests: 5475 → 5493 (+18). New file `tests/test_mcp_user_pool.py`
plus additions to test_mcp_oauth_refresh.py, test_mcp_admin_api.py,
and test_mcp_client.py covering: pool data structures, lazy connect,
eviction TTL + LRU + lock interlock, dispatch state machine (token
states), failure classification, http-rejection at dispatch and
admin layers, URL-change-purges-tokens (sec), concurrent dispatch on
one (user, server), pg_advisory lock parity, and user_id threading.
Phase exit criterion (synthetic load test 50 users × 3 servers × LRU
30 × 1000 calls × 200 evictions) deferred to a post-Phase-5 fitness
spike that runs against a staging deployment with real FDs and real
network behaviour, not a CI mock — same shape as Spike 1's
pre-Phase-0 SDK validation.
Out-of-scope for Phase 5 (Phase 6+): SDK-level 401 refresh-retry +
403 `mcp_insufficient_scope` (Phase 6), per-user catalog scoping
(Phase 7), consent UX SSE event + dashboard renderer (Phase 8),
admin UI status indicators (Phase 9).
Spike artifact validating MCP SDK behavior before Phase 5 builds the
per-(user, MCP-server) ClientSession pool. Three scenarios, all pass:
1. N=20 concurrent ClientSession instances against the same URL — no
FD blow-up, no shared transport state, each session's tools/list
returns independently.
2. Two concurrent tools/call on a shared ClientSession with
interleaving payloads — request_id demux works under contention.
3. Per-session Authorization header isolation across 5 sessions —
httpx connection pooling does not cross headers between sessions,
so per-session bearer tokens reach the server unmixed.
Outcome gates the Phase 5 architecture (lazy dict[(user_id,
server_name), ClientSession] + per-key asyncio.Lock + LRU eviction).
Had any scenario failed, the fallback was per-call header injection
(Alternative F in the OAuth-MCP RFC).
Spike-only — not collected by pytest. Run manually:
uv run python tests/spike_sdk_concurrency.py
Addresses ten findings on the Phase 4 OAuth-MCP commit: four from the
PR #478 review surface, plus six surfaced by a follow-up multi-stage
review of the first round of fixes. Two of the latter were genuine
security regressions in the very code that claimed to close those
holes.
Security
--------
- _validate_return_url now pins return_url same-origin against the
configured oidc_config.redirect_base instead of request.url. Behind
a permissive front proxy that did not normalise Host, an attacker
could spoof Host and provide a matching absolute return_url to mint
an open redirect off /api/mcp/oauth/start. Same fix pattern as
PR #476 OIDC.
- Reject return_url values containing literal backslashes or starting
with `//` up front. urlparse leaves backslashes inside `path`, so a
value like `/\evil.example/foo` slipped through the path-only branch
and became the protocol-relative `//evil.example/foo` after WHATWG-
conformant browsers normalised the backslash — re-introducing the
open redirect the same-origin pin was meant to close.
- internal_mcp_status (read-scoped) projects through a new
_strip_server_status_for_read helper that drops the verbose `error`
text and replaces it with a coarse `has_error` boolean. The error
string is built as `f"{type(exc).__name__}: {exc}"` and so carries
stdio binary paths (FileNotFoundError) or internal MCP URLs
(httpx.ConnectError) — equivalent to leaking command/url, which
this same patch deliberately strips. Approve-scoped refresh and
reconnect callers continue to receive the full `error` text via
the existing _strip_server_status helper.
- internal_mcp_status now returns the projected (sanitised) entries
for every server in mcp_mgr.get_all_server_status() instead of
emitting the un-sanitised dict that included `command` (stdio argv)
and `url` (remote MCP endpoint). Sibling refresh/reconnect endpoints
already used _public_server_status to strip these.
- internal_mcp_status docstring documents the trust boundary — server
enumeration to read scope is intentional so dashboards can render
per-server indicators; verbose error detail and command/url remain
approve-scoped.
Correctness / UX
----------------
- _validate_return_url comparison normalises (scheme, host, port)
before equality. Lowercases hostname and collapses the scheme's
default port, so `https://App.Example.COM/x` and
`https://app.example.com:443/x` are recognised as same-origin
with `redirect_base = https://app.example.com` instead of being
silently downgraded to the `/` fallback.
- mcp_crypto startup-gate error message now names both
`mcp_token_encryption_keys` (rotation list) and
`mcp_token_encryption_key` (single) so an operator using rotation
isn't misled into thinking only the singular form is valid.
Cleanup
-------
- Delete the unused _KNOWN_TRUSTED_ENDPOINT_HOSTS legacy re-export
shim in oidc.py (zero callers — a no-op that survived the Phase 4
oauth_ssrf extraction). Sphinx :data: docstring reference at
validate_discovered_endpoint updated to point at
turnstone.core.oauth_ssrf.KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS
directly. The Google multi-origin allowlist is unaffected — it
lives at the canonical name and is read from oauth_ssrf.py:164.
- test_mcp_oauth_handlers TestValidateReturnUrl imports
_validate_return_url at module level instead of repeating the
import inside each test method.
- test_server_lifespan_mcp_crypto replaces a fragile
`messages.count("mcp_token_encryption_key") >= 2` substring trick
with `re.search(r"mcp_token_encryption_key(?!s)", messages)` —
asserts the singular form directly via negative lookahead.
Tests
-----
5448 pass (+13 vs the prior tip):
- TestValidateReturnUrl gains backslash-bypass, protocol-relative,
default-port, uppercase-host, and explicit-port-mismatch cases
alongside the original same-origin / cross-origin / scheme-
mismatch / path-only cases.
- TestInternalMcpStatusEndpoint asserts the `error` text never
reaches the read-scope wire (binary-path FileNotFoundError no
longer appears anywhere in the rendered response) and that the
coarse `has_error` boolean lights up correctly on the failed
server.
- TestInternalMcpStatusEndpoint also pins the no-mcp-client path to
`{"servers": {}}`.
- _routes_with_internal extended to include the
/api/_internal/mcp-status route so the new tests can exercise it
through TestClient.
- Existing test_startup_aborts_with_oauth_user_row_and_no_key
strengthened to require both singular and plural key names appear
in the error log.
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.
Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
effective_port, sanitize_log_text moved out of oidc.py into a shared
oauth_ssrf module; oidc.py re-exports for compatibility. The shared
helpers also expose async wrappers (validate_url_no_ssrf_async,
validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
from async handlers — does not block the event loop on the
synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
DELETE...RETURNING)
Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
existing refresh value is preserved at the OAuth-flow layer rather
than cleared, so production ASes (Google, Auth0 default, Okta) don't
force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
MCP server URL, not the audience. Audience and resource are distinct
concepts; using audience as resource would mismatch the AS RS
allowlist.
Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
accepted_audiences = {server_url, oauth_audience} and validates
against the set, so Auth0-style ASes that honor `audience=` (not
RFC 8707 `resource=`) issue tokens that pass audience-bound
validation
- build_authorize_url emits both `resource=` (RFC 8707) and
`audience=` (Auth0-style) per server config; comment documents which
AS implementations need which form
Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
request Host header — closes the same Host-header injection PR #476
fixed for OIDC. Both /start and /callback return 503 with operator-
actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
inside the lock, so concurrent /start callers don't both register
and overwrite each other's client_id (the second user's code is no
longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
so a leaked state can't be replayed against a separately-obtained
code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
through sanitize_log_text before reaching exception messages or
audit details. AS error responses are parsed for the standard
RFC 6749 fields (error, error_description, error_uri), each
capped at 80 chars and run through redact_credentials to defend
against ASes that echo the request body back into their error
payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
tokens and pending OAuth state for the previous server name
(delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
mutable server_name; without this purge, a future server with the
same name (and an attacker-controlled URL) would silently rebind
prior user tokens. A future schema migration will replace the
server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
installed key can decrypt the row, e.g. after key rotation) and
falls through to None so dispatch surfaces a re-consent rather than
crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
path. Auto-connecting them at startup with empty headers fails the
AS check and trips the circuit breaker; per-user tokens come online
lazily once the user has consented.
Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
token_revoked, dcr_registered. _audit_event is async and wraps
record_audit in asyncio.to_thread so the audit write doesn't block
the event loop. resource_id on the audit row is the immutable
server_id (PK UUID) so admin-driven server renames don't break
event correlation; server_name is exposed in detail for cross-
reference. dcr_registered detail.has_secret reflects whether the
DCR-issued secret was actually persisted (the prior code reported
has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
server_name (which is what the column is — the table's PK was
always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
the audit-action taxonomy.
Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.
Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
limitation in oauth_ssrf module docstring
Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.
Phase 3 of docs/design/oauth-mcp.md. Adds the Fernet/MultiFernet wrapper,
[security] config loader with rotation support, MCPTokenStore CRUD facade,
typed MCPTokenDecryptError that maps to the RFC's mcp_token_undecryptable_
key_unknown class, and a startup gate that fails loud when auth_type=
'oauth_user' rows exist without a configured encryption key.
Crypto module (turnstone/core/mcp_crypto.py):
- MCPTokenCipher wraps cryptography.fernet.Fernet + MultiFernet for
rotation; encrypt with first key, decrypt by trying each in order
- load_mcp_token_cipher_config reads [security] mcp_token_encryption_keys
(plural list) or mcp_token_encryption_key (singular), validates each
key is base64-decodable to exactly 32 bytes
- MCPTokenCipherConfig is repr=False with custom __repr__ that redacts
raw key bytes (defense in depth against accidental log/traceback leak)
- _key_fingerprint produces an 8-hex-char SHA-256 prefix for audit
attribution without exposing the key
- MCPTokenStore handles encrypt-on-write / decrypt-on-read for
mcp_user_tokens and mcp_servers.oauth_client_secret_ct
- get_user_token MUST NOT auto-delete the row on MCPTokenDecryptError
(test_get_user_token_with_wrong_key_raises_decrypt_error verifies
the row stays intact across a key-mismatch read)
- initialize_mcp_crypto_state / close_mcp_crypto_state lifespan helpers
shared between server and console
Storage protocol (5 new ciphertext-only methods):
- set_mcp_oauth_client_secret_ct (dedicated writer; deliberately NOT
added to MCP_SERVER_MUTABLE so generic update_mcp_server cannot write
the secret column)
- create_mcp_user_token, get_mcp_user_token,
update_mcp_user_token_after_refresh, delete_mcp_user_token
Server + console lifespans (turnstone/server.py + console/server.py):
- after OIDC init, count auth_type='oauth_user' rows; if any exist and
no encryption key is configured, log an actionable error and
raise SystemExit(1)
- without oauth_user rows, missing key is fine (lazy validation; admin
flip without restart returns 503 from the admin handler)
- app.state.mcp_token_cipher / .mcp_token_store populated when key
configured; None otherwise
Admin handlers:
- _require_token_store_for_oauth_secret pre-mutation gate validates
token_store availability and oauth_client_secret type BEFORE
storage.create_mcp_server / update_mcp_server runs, so a 503 from a
missing key never leaves an orphan row or partial-update state
- _apply_oauth_client_secret encapsulates the encrypt + audit write
used after the storage mutation; rolled out across both create and
update handlers
- 503 message references both mcp_token_encryption_key (singular) and
mcp_token_encryption_keys (plural for rotation)
- non-string oauth_client_secret payloads (false / 0 / lists / dicts)
are rejected with 400 instead of being str()-coerced
- when auth_type transitions away from oauth_user, the encrypted
secret column is cleared in the same admin call (with audit), so
flipping back doesn't silently resurrect a stale credential
Audit events (mcp_server.oauth.* per audit.py taxonomy; RFC's
mcp.oauth.* renamed for consistency):
- mcp_server.oauth.client_secret_set fired from admin handlers with
cleared:bool and key_fingerprint
- mcp_server.oauth.token_decrypt_failure fired from MCPTokenStore
.get_user_token when no installed key can decrypt; carries
key_fingerprints_attempted
Tests: 35 new tests across test_mcp_crypto, test_mcp_token_store,
test_server_lifespan_mcp_crypto, plus 6 admin-API tests covering the
no-orphan-row, no-partial-update, secret-clear-on-transition, and
non-string-secret-rejection invariants. Suite at 5337 (Phase 3 added
~50 tests including the rebase-imported skill suite).
cryptography>=42 promoted from transitive (lacme[tls]) to direct dep
since the encryption layer is now core, not optional.
Phase 4 (OAuth flow) wires the actual callers; Phase 3 adds only the
crypto layer and is exercised entirely by tests.
Adds the data model and admin UI surface required by the OAuth-MCP flow.
Phase 2 of the per-user delegation initiative.
Schema:
- migration 049 creates mcp_user_tokens (PK user_id, server_name) and
mcp_oauth_pending (PK state, indexed by created_at)
- eight new columns on mcp_servers: auth_type ('none' / 'static' /
'oauth_user', NOT NULL DEFAULT 'static') plus six oauth_* config
fields and oauth_as_issuer_cached
- post-upgrade UPDATE normalises auth_type to 'none' for streamable-http
rows whose headers are NULL/empty/'{}'; stdio rows are left at the
'static' default (auth_type is HTTP-auth-only)
- _schema.py kept in lockstep with the migration so metadata.create_all
and alembic upgrade produce identical shapes
- mcp_user_tokens / mcp_oauth_pending TypedDicts in _protocol.py for
Phase 3/4 use (no CRUD methods yet)
Storage / API:
- create_mcp_server gains the eight kwargs across protocol + sqlite +
postgresql
- MCP_SERVER_MUTABLE picks up auth_type and the six text oauth_* fields;
oauth_client_secret_ct is intentionally NOT in the whitelist — Phase 3
will own ciphertext writes via a dedicated method
- McpServerInfo + Create/Update Pydantic schemas extended; oauth_client_secret
accepted as plaintext input but discarded (Phase 3 wires encryption)
Admin handlers:
- _parse_auth_type validates against {'none', 'static', 'oauth_user'} and
rejects empty / unknown values; shared between create and update
- when auth_type changes away from 'oauth_user', the oauth_* config
columns are explicitly nulled in the same UPDATE so the row stays
consistent
- _clean_oauth_text caps text fields at 512 chars (URLs at 2048) to bound
admin write surface
- _mask_mcp_secrets now masks oauth_client_secret_ct to '***' regardless
of reveal=true (write-only field)
- audit detail dict redacts oauth_client_secret if present
Frontend:
- new "Multitenant Authorization" fieldset on the MCP-server modal with
three radio buttons (None / Shared / Per-user OAuth 2.1)
- conditional OAuth subform: AS URL, registration mode (preregistered /
dcr; cimd is future), client ID, client secret, scopes, audience
- secret input is autocomplete=off and never round-trips on edit
- audience auto-populates from the MCP server URL on blur
- headers textarea hidden and submitted as {} when auth_type is 'none' or
'oauth_user' so flipping the radio cleans up server-side state
Tests: storage round-trip for the new columns, oauth_pending table smoke,
migration 049 upgrade/downgrade with stdio-vs-http normalisation, four
admin-API tests for auth_type validation and oauth_*-clear-on-flip-away.
Suite passes 5284 (matched pre-Phase-2 baseline 5267 + 17 new).
Stacks on Phase 0; no behavioural change for existing rows.
Phase 0 of the OAuth-MCP RFC: prepare MCPClientManager for the per-(user,
server) session pool that lands in Phase 5, without changing static-path
behavior.
Two changes:
1. Hardening helpers _pre_close_streams and _tcp_probe rename their first
parameter from `name` to `key`. Type stays `str` for now; widening to
`str | tuple[str, str]` happens in Phase 5 when callers actually pass
tuples. _safe_close_stack takes the stack directly and is unchanged.
2. The eleven parallel name-keyed dicts (_sessions, _per_server_stacks,
_per_server_tools, _per_server_resources, _per_server_prompts,
_supports_list_changed, _supports_resources, _supports_resource_list_changed,
_supports_prompts, _supports_prompt_list_changed, _server_streams) are
consolidated into _static_servers: dict[str, StaticServerState]. Server-
level state (circuit breaker, notification debounce, last-error,
db-managed, merged catalog maps, listener lists) stays on the manager,
unchanged.
PoolEntryState is defined for Phase 5 use but no code instantiates it. The
typed map declarations (dict[str, StaticServerState] vs dict[tuple[str, str],
PoolEntryState]) make accidental cross-keying lookups easier to catch.
PR #296 hardening preserved exactly:
- pre-close-streams atomic take-and-clear before stack teardown
- stale-session-and-stack guard at _connect_one top: both state.session and
state.stack checked, cleared independently, entry preserved (not popped)
- transport-error session-eviction in dispatch sets state.session=None only,
leaving stack/streams for the next connect-time guard sweep
- _safe_close_stack CancelledError suppression unchanged
- TCP probe before streamablehttp_client unchanged
- future.cancel() after TimeoutError in all sync bridges unchanged
- notification debounce stays manager-level (not migrated into the dataclass)
Refresh helpers (_refresh_server_tools/_resources/_prompts) snapshot
state.session into a local immediately after the None guard so concurrent
transport-error eviction during await cannot null the session reference
mid-call.
Tests: shared _seed_static_state helper in tests/conftest.py replaces eleven
direct dict mutations; new test_reconnect_preserves_static_state_identity
guards the entry-preservation invariant. Pass count rises 5266 → 5267.
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.
Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).
Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.
Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.
Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
traffic arrives or an operator clicks Reconnect. The previous
background reconnection loop is gone by design — push
notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
not changed here.
This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.
* feat(skills): paste SKILL.md to auto-fill the Create Skill modal
When a user pastes an Anthropic-style SKILL.md (YAML frontmatter +
markdown body) into the Create Skill content textarea, the frontend
sniffs the leading ``---``, posts the raw text to a new backend parse
endpoint, and populates name / description / tags / author / version /
license / compatibility / allowed_tools from the parsed fields. The
textarea is left with the body only (frontmatter stripped), and a toast
reports how many fields were set vs. kept (already-typed values are
preserved).
Backend
- ``POST /v1/api/admin/skills/parse`` (admin.skills permission) wraps
the existing ``turnstone.core.skill_parser.parse_skill_md`` so admin
imports and external installs share one parser. ``ParseSkillRequest``
/ ``ParseSkillResponse`` schemas added; OpenAPI spec + sync/async
console SDK methods updated.
- Hardening: 32 KiB cap on ``raw`` (Pydantic ``max_length`` + handler
enforcement); ``Content-Length`` pre-check returns 413 before any body
buffering; parse offloaded via ``asyncio.to_thread`` so deeply-nested
YAML cannot stall the event loop.
Frontend (turnstone/console/static)
- New paste handler with optimistic paint (raw text shown immediately,
textarea disabled + ``aria-busy`` flipped, hint switches to
"Parsing...") so the round-trip is visible on slow networks.
- ``AbortController`` + generation guard (``_ctmPasteController``) so a
fresh paste or modal close cancels a stale fetch — the previous
handler's callbacks see the controller has been replaced and bail
before touching the DOM.
- Non-destructive overwrite: ``_setSkillFormField`` returns "filled" /
"skipped" / "absent" and refuses to clobber non-empty values. Toast
reports counts.
- Bumps ``#toast`` z-index above modal overlays (was 200 vs. modal 600
— toasts fired while a modal was open were invisible). Console-wide
fix exposed by this being the first feature to fire toasts mid-modal.
HTML / CSS
- New ``.skill-paste-hint`` line above the textarea announcing the
affordance, sized to match surrounding ``.label-hint`` text.
- ``aria-describedby`` ties the hint to the textarea; ``aria-live=
"polite"`` announces the busy-state transition to screen readers.
- "Skill Content" heading hint reworded "system message — ..." →
"available: ..." and the variables row label "Variables" → "Used"
to disambiguate available vs. in-use template variables.
Tests
- 11 new cases in ``tests/test_skill_parse_api.py``: happy paths
(full / minimal / nested-metadata / unquoted-colon recovery),
malformed YAML 400, missing/blank/missing-name 400, RBAC 403, raw
body 32 KiB cap (Content-Length pre-check), chunked-encoding bypass
forces the application-layer cap. Test pins ``raw_frontmatter``
omission so a future ``dataclasses.asdict`` refactor can't silently
leak the full YAML dict back to clients.
Validation
- 5146 / 5146 ``pytest -k "not live"`` pass.
- ``ruff`` + ``mypy`` clean on changed sources.
- ``node -c`` clean on governance.js.
- Two-stage code review (full pipeline + bug+quality re-review of the
fix patches) applied; all confirmed findings addressed.
* fix(skills): Copilot PR #477 review fixes (cumulative bug-1, bug-2, q-1)
bug-1 (server.py): Content-Length pre-check was clamped to 32 KiB —
the same number as the per-string char cap on ``raw``. A legitimate
``raw`` of exactly 32 KiB produces a JSON body well above 32 KiB once
the ``{"raw":"..."}`` wrapper and any escaping is added, so valid
near-max requests were 413'd. New constant
``_PARSE_SKILL_MAX_BODY_BYTES = _PARSE_SKILL_MAX_CHARS * 4`` admits the
wrapper + multibyte expansion while still refusing obviously oversized
payloads early; the per-string ``len(raw)`` check stays authoritative.
bug-2 (governance.js): hideCreateTemplateModal aborted the inflight
paste controller and nulled the global, but the handler's ``.catch``
and ``.finally`` guard each DOM mutation behind ``_isCurrent()`` —
both bail when the controller has been nulled, leaving the textarea
``disabled`` + ``aria-busy`` and the hint stuck on "Parsing…".
Reopening the modal landed on a poisoned state. The second-pass
review's q-2 cleanup that dropped the show-side defensive reset
missed this scenario — the verifier's reachability argument confused
"controller is null" with "UI state is reset"; the two are
independent. Hide now resets the paste-induced visible state
alongside the abort.
q-1 (console_spec.py): error_codes for the parse endpoint listed only
400; handler also returns 413 for oversized bodies. Added 413; kept
403 implicit per the convention sibling admin endpoints follow.
Test fixup: bumped the Content-Length test payload to 200 KB so it
clearly exceeds the new 128 KB pre-check threshold; otherwise it was
falling through to the per-string check and duplicating
test_oversized_raw_chunked_returns_413's coverage.
PR #476 review feedback (Copilot, oidc.py:584,616):
1. initialize_oidc_state's docstring claimed "on any failure
enabled is False" but the JWKS-prefetch failure branch
intentionally keeps enabled=True so the callback's lazy-fetch
retry can recover from a transient IdP issue at startup.
Docstring rewritten to spell out the three post-conditions:
disable, JWKS-failure-keeps-enabled, success.
2. The long-lived httpx.AsyncClient was created up front, then
three disable branches (discovery exception, discovery-returned-
disabled, missing redirect_base) returned without closing it,
leaving sockets held until shutdown.
Restructured: discovery now uses a transient AsyncClient inside
a context manager (closed at exit). The long-lived client is
only created after the disable checks pass. The JWKS-failure
branch still legitimately keeps the client open because the
lazy-retry path needs it.
The pre-existing single-client-passthrough test was replaced
with three more specific tests: long-lived client only goes to
fetch_jwks (not discover_oidc); discovery-exception path leaves
http_client=None; missing-redirect_base path leaves
http_client=None.
q-4: tests/test_oidc.py's _make_config and tests/test_oidc_handlers.py's
_make_oidc_config built the same OIDCConfig with sensible defaults but
had drifted — only the handlers helper set redirect_base. After b3
made redirect_base operationally required, every test_oidc.py test
that exercised redirect_base had to override it explicitly. A future
test could omit redirect_base and silently exercise the wrong
production path.
Moves make_oidc_test_config to tests/conftest.py with the more
complete handler-version defaults (including redirect_base). Both
test files import it under their existing local alias
(_make_config / _make_oidc_config) so the 60+ call sites in
test_oidc.py and the handler tests don't have to change.
q-5: section banner '# Exception' (singular) at oidc.py:79 became
inconsistent after b5 (callback robustness) added OIDCKeyNotFoundError.
Renamed to '# Exceptions'.
The OIDC perf batch added storage.count_users() and migrated the two
OIDC handlers (handle_oidc_authorize, handle_oidc_callback) but missed
handle_auth_status — which still ran storage.list_users() then
len(users) > 0 for the same has-any-users gate.
count_users() is one COUNT(*) round-trip vs list_users() rehydrating
every row dict. Wrapped in asyncio.to_thread to match the OIDC handler
pattern; the async handler no longer blocks the event loop on storage
I/O for what's effectively an existence probe.
bug-2 (Postgres) — replace_oidc_roles read existing rows under default
READ COMMITTED with no row lock. Two concurrent OIDC callbacks for the
same user_id (racing token refreshes with differing claim sets) could
both observe the same baseline and produce a final role state matching
neither caller's intent. Adds .with_for_update() to the SELECT so the
existing rows for this user are locked for the duration of the
transaction.
The lock is per-user_id, not table-wide; unrelated user writes are
unaffected. Empty result sets acquire no locks, so a brand-new user
with no rows yet still allows two callers to proceed and merge via
ON CONFLICT DO NOTHING — that's a permissive race that self-heals on
the next reconciliation cycle, documented in code.
perf-1 (SQLite) — replace_oidc_roles took the SQLite global write
lock unconditionally via BEGIN IMMEDIATE before reading. Steady-state
re-logins (claims unchanged, no INSERT/DELETE needed) paid the lock
cost for nothing and serialised against unrelated writers.
Replaces with a double-check pattern: phase 1 reads under the default
deferred transaction (no write lock), computes the diff, and returns
(set(), set()) on no-op. Phase 2, only when mutation is needed,
commits the read txn, escalates to BEGIN IMMEDIATE, RE-READS, and
re-computes the diff under the lock before writing. The returned
(added, removed) reflects what was actually written, so caller logging
in apply_role_mapping stays truthful even when concurrent writers
shifted state between the two reads.
The OR IGNORE on insert is now defense-in-depth (the lock makes it
unnecessary) but kept as a safety net.
The 8-commit OIDC stack added TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS
(operator allow-list for cross-host IdP discovery endpoints) and
promoted TURNSTONE_OIDC_REDIRECT_BASE to required, but the docs drifted
in two places:
q-1 — Troubleshooting > "OIDC not configured" still listed three
required env vars. An operator hitting the missing-redirect-base
startup error landed on a debugging entry that didn't mention the
variable they were missing. Fixed; added a separate troubleshooting
entry naming the exact log message produced by initialize_oidc_state
when redirect_base is unset.
q-2 — TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS was undocumented entirely.
Added a row to the env-var table and a new "Cross-host endpoints"
section explaining when the knob is needed (Google is the canonical
multi-origin IdP, but it's auto-handled; the env var is for any other
IdP whose discovery doc legitimately references hosts beyond the
issuer's origin). Added a troubleshooting entry pointing at the new
section.
If apply_role_mapping raised after create_oidc_user committed (transient
storage failure, race with role deletion, etc.), provision_oidc_user's
inline safety-net was skipped — and on retry the existing-identity
branch never reached the safety-net code, leaving the user permanently
stranded with zero roles.
Extracts _ensure_default_role(storage, user_id, desired_role_ids=None)
helper. Calls it on BOTH the new-user and existing-identity paths so a
user stranded by a transient failure recovers on next login.
desired_role_ids is a hint that lets the helper skip list_user_roles
when claim-driven mapping populated at least one role; the new-user
path was already paying that query, the existing-identity path now
pays it only when claim mapping returned an empty desired set.
Documents the admin-strip behavior in the helper docstring: stripping
all roles from an OIDC user no longer locks them out, since the next
login will re-grant builtin-viewer (assigned_by='oidc-default'). The
documented way to deny an OIDC user is to unlink their OIDC identity
via the admin endpoint, not to strip roles. The pre-fix behavior
(stripped user actually locked out) was the bug.
The 'oidc-default' vs 'oidc' assigned_by distinction is preserved:
apply_role_mapping's revocation lane only touches 'oidc' rows, so the
safety-net role survives every subsequent login regardless of claims.
Six new tests cover both paths, the hint short-circuit, the
list_user_roles fallback, the missing-builtin-viewer no-op, and the
self-heal regression case for already-stranded users.
q-5: _derive_username's UUID-retry tier (oidc.py:923-933) was untested.
After perf-6 collapsed tier-2 to a single find_existing_usernames call,
the only remaining tail was the 3-attempt UUID-retry loop and the final
raise. New TestDeriveUsername class covers:
- falls into UUID retry when all 10 suffix candidates are taken
- UUID retry succeeds on the second attempt after one collision
- UUID retry exhausted -> raises OIDCError
q-8: filled the unit-level coverage holes the multi-stage review flagged:
- test_validate_id_token_retry_after_kid_rotation — direct unit test of
the OIDCKeyNotFoundError path with real RS256 keys + JWKS rotation
(previously only exercised end-to-end through the handler).
- test_callback_uses_pending_audience_not_handler_audience — pins down
the bug-3 fix by decoding the issued JWT cookie and asserting aud
matches the audience stored at /authorize time, not the handler param.
- test_apply_role_mapping_int_claim / _dict_claim — exercises the
else: values = [str(claim_value)] branch for non-string non-list
claim shapes.
- TestFetchJWKS — non-200 status, non-dict body, dict-missing-keys,
keys-not-list, transport network error.
- TestExchangeCode network/4xx/5xx error tests (the non-dict-body case
already shipped in batch 5).
Also a small production hardening that fell out of writing the
TestFetchJWKS::test_fetch_jwks_non_dict_body_raises test: fetch_jwks now
guards isinstance(result, dict) before result.get("keys"), matching the
shape-check pattern that discover_oidc and exchange_code already use.
A list/null body now surfaces as OIDCError("...not a JSON object") rather
than AttributeError leaking up to the lifespan.
Eleven small maintenance fixes; no behavior change beyond bug-3.
bug-3: pending.get('audience', audience) couldn't fall back because
pop_oidc_pending_state always returns a dict with the audience key
set verbatim from a non-null TEXT column. Replaced with
pending.get('audience') or audience to cover the empty-string case
defensively. Comment explains the security rationale.
q-1: extract _env_or_cfg_str / _env_or_cfg_bool helpers in oidc.py;
load_oidc_config's six near-identical env-or-config blocks collapse
to one-liners. role_map / trusted_endpoint_hosts / redirect_base
retain bespoke parsing.
q-3: discover_oidc narrows except (httpx.HTTPError, ValueError, KeyError)
with exc_info=True.
q-4: OIDC_STATE_TTL_SECONDS = 300 constant in oidc.py; auth.py imports
and passes it explicitly. Storage signatures keep the literal default
(storage layer doesn't know OIDC TTL semantics).
q-6: hoist runtime imports (OIDCError, OIDCKeyNotFoundError, exchange_code,
fetch_jwks, provision_oidc_user, validate_id_token, build_authorize_url,
generate_pkce_verifier) to module scope in auth.py. The genuine cycle
is only oidc._derive_username -> auth.is_valid_username, kept
function-scoped. test_oidc_handlers.py mock targets repointed to
turnstone.core.auth.X to match the new binding.
q-7: comment + docs explain the 'oidc' vs 'oidc-default' assigned_by
marker distinction.
q-9: OIDCIdentity / OIDCPendingState TypedDicts in storage protocol.
Implementations construct via TypedDict syntax so mypy structurally
verifies all required fields.
q-10: fetch_jwks narrows except (httpx.HTTPError, ValueError); docstring
matches.
q-11: rename generate_pkce_pair -> generate_pkce_verifier; return only
the verifier (build_authorize_url already recomputes the challenge).
q-12: extract _buildOidcRow helper in admin.js so future field additions
go in one place.
q-13: OIDCConfig docstring lists startup-config vs discovery-derived
field groups.
Eight independent perf wins on the OIDC hot path:
perf-1: list_users() full-scan setup-gate replaced with new count_users()
on both authorize and callback. Saves a full users-table fetch per login.
perf-2: handle_oidc_callback's sync DB chain wrapped in asyncio.to_thread
for cleanup, pop_oidc_pending_state, count_users, and provision_oidc_user.
handle_oidc_authorize gets the same treatment for count_users and
create_oidc_pending_state. Event loop no longer blocks for the full
callback duration on Postgres deployments.
perf-3: apply_role_mapping N+1 collapsed via new replace_oidc_roles
storage method. One transaction handles the diff + insert + delete
instead of 2N+1 commits per login. Returns (added, removed) so the
caller can still emit per-role audit logs.
The diff respects the documented invariant "manually-assigned roles
are never touched" — desired_role_ids is filtered against rows where
assigned_by != 'oidc' before computing added/removed. This prevents a
PK conflict (Postgres lockout) or silent OR-IGNORE no-op (SQLite lying
return) when admin-ui or oidc-default already holds the same role_id.
perf-4: provision_oidc_user no longer re-queries list_user_roles after
apply_role_mapping. The new-user builtin-viewer fallback is gated on
desired_role_ids being empty, which is information apply_role_mapping
already returned.
perf-5: JWKS refetch dedup via asyncio.Lock on app.state. Both lazy-fetch
(cold-start recovery) and rotation paths share the same lock with a
double-check pattern: re-resolve kid against the current cache before
issuing a new GET. N concurrent callbacks during rotation now produce
at most 1 fetch.
perf-6: _derive_username's 9-suffix loop collapsed via new
find_existing_usernames(candidates) -> set query. Worst case drops
from 13 sequential queries to 1 + up-to-3 UUID-retry queries.
perf-7: cleanup_expired_oidc_states gated to once-per-60s per process
via app.state.oidc_last_cleanup_monotonic. The pop already deletes
the consumed row; the bulk cleanup is only relevant for abandoned
authorize flows, so frequency was overkill.
perf-8: Long-lived httpx.AsyncClient stashed on app.state.oidc_http_client
by initialize_oidc_state. discover_oidc/fetch_jwks/exchange_code accept
an optional client= kwarg; when set, skip the per-call AsyncClient
context-manager. New close_oidc_state lifespan teardown closes it.
Tests pass client=None to keep the transient-client legacy path.
New storage methods (sqlite + postgresql):
- count_users() -> int
- find_existing_usernames(candidates) -> set[str]
- replace_oidc_roles(user_id, desired) -> (added, removed)
Four small hardening fixes on the OIDC callback hot path:
bug-4: JWKS rotation retry was matching the substring 'not found in JWKS'
inside an OIDCError message. A future rephrasing would silently break
key rotation. Adds OIDCKeyNotFoundError(OIDCError); validate_id_token
raises the subclass at the kid-not-found site; handle_oidc_callback
catches it explicitly. Other 'not found' errors in validate_id_token
remain as plain OIDCError.
bug-5: tokens['id_token'] raised KeyError if the IdP returned 200 without
id_token. exchange_code now rejects non-dict response bodies; the
callback validates id_token shape (must be non-empty str) before
passing to validate_id_token. Both raise OIDCError, surfaced as the
standard 'Authentication failed' redirect.
bug-6: shared_static/auth.js — the OIDC error display raced showLogin's
/v1/api/auth/status fetch via a 300ms setTimeout. showLogin now takes
an optional oidcError parameter and paints it after _switchMode clears
the error, in both the success and catch branches of the fetch.
sec-4: oidc.py exchange_code's non-200 OIDCError interpolated up to 500
bytes of attacker-controlled IdP body, which then went to log.warning
via 'OIDC callback failed: %s'. CRLF in resp.text could forge log
lines. New _sanitize_log_text helper escapes control chars via
unicode_escape and caps at the rendered length.
provision_oidc_user previously called create_user (INSERT OR IGNORE
on SQLite — silent no-op on UNIQUE conflict), then create_oidc_identity
(also INSERT OR IGNORE), then apply_role_mapping which writes user_role
rows for the supposedly-new user_id. On a username TOCTOU race or
concurrent (issuer, sub) double-create, both inserts no-opped but
user_role rows were already written — leaving orphan rows pointing
at a user_id that doesn't exist.
PostgreSQL's create_user raised IntegrityError instead of silently
no-opping so it produced a misleading 'Authentication failed' error
without orphans, but the user-facing UX was equally poor.
Adds StorageConflictError to the storage protocol and create_oidc_user
that does both inserts in one transaction. Username collision and
(issuer, subject) collision both raise StorageConflictError, mapped
to OIDCError by provision_oidc_user. Crucially the new code does not
silently bind a colliding-username new identity to the existing user
— that would be an account-takeover vector. It raises.
SQLite uses BEGIN IMMEDIATE inside the try block so lock-contention
errors surface as StorageConflictError instead of leaking the raw
sqlalchemy OperationalError.
PostgreSQL relies on SQLAlchemy 2.x begin-on-demand semantics; the
explicit conn.commit()/rollback() in the catch block is the only
materialization path. Discrimination on PG uses
exc.orig.diag.constraint_name with message-substring fallback.
_build_oidc_redirect_uri previously fell back to the request Host
header when redirect_base was unset. With a permissive reverse proxy
or direct backend access, a spoofed Host minted an authorize URL
pointing to attacker-controlled host — combined with a permissive
IdP redirect_uri allowlist this enables auth-code interception.
There is no production scenario where a Host-derived redirect_uri is
correct, so this fails closed:
- initialize_oidc_state checks redirect_base after discovery succeeds
and disables OIDC (with an explicit error log naming the env var)
if it's empty. Runs before fetch_jwks so a misconfigured deploy
doesn't make a wasted JWKS call.
- _build_oidc_redirect_uri simplifies to f"{redirect_base}/v1/api/auth/oidc/callback".
request parameter dropped; both call sites (handle_oidc_authorize,
handle_oidc_callback) updated.
- docs/oidc.md promotes TURNSTONE_OIDC_REDIRECT_BASE from "Recommended"
to "Required" with the security rationale.
The OIDC discovery + JWKS prefetch block was duplicated byte-for-byte
between turnstone/server.py and turnstone/console/server.py. The bare
except branch in that block also left app.state.oidc_config unchanged
on unexpected exceptions — leaving the runtime with enabled=True and
empty endpoints, producing malformed authorize URLs.
Extracts initialize_oidc_state(app_state) into turnstone/core/oidc.py
which guarantees a coherent post-condition on every code path:
- discovery exception -> oidc_config replaced with enabled=False, jwks_data=None
- discovery returns enabled=False -> jwks_data=None
- JWKS prefetch fails -> jwks_data=None but enabled=True preserved (the
callback's lazy-fetch retry path remains the recovery)
- success -> oidc_config + jwks_data both populated
Also hardens discover_oidc against non-dict discovery responses
(list/null/string/int) — previously these raised AttributeError out
of doc.get and propagated past the lifespan's bare except.
server.py and console/server.py lifespan blocks collapse to a single
await initialize_oidc_state(app.state) call.
OIDC discovery-document endpoints (token_endpoint, jwks_uri,
userinfo_endpoint) were stored verbatim in OIDCConfig and later passed
to httpx without revalidation. Only the issuer URL was checked. A
hostile or compromised IdP could return token_endpoint pointing to an
internal IP (169.254.169.254, 10.0.0.0/8, etc.) and Turnstone would
POST the client_secret there.
Extracts the existing scheme/userinfo/SSRF check into
_validate_url_no_ssrf, adds validate_discovered_endpoint that runs the
same checks plus an issuer-binding check, and wires it into
discover_oidc for authorization_endpoint, token_endpoint, jwks_uri,
and userinfo_endpoint (when present).
Issuer binding accepts:
- Same (scheme, hostname, effective port) as the issuer.
- A hostname in _KNOWN_TRUSTED_ENDPOINT_HOSTS for the issuer (Google's
multi-origin discovery is in the allow-map by default).
- A hostname in OIDCConfig.trusted_endpoint_hosts, settable via
TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS env var or config.toml, for
IdPs not in the static map.
Effective port comparison treats https://host and https://host:443 as
the same origin (urllib.parse.urlparse leaves the explicit form's port
as 443 and the implicit form's as None).
24 new tests cover the validator, the Google known-hosts path, the
operator allow-list, default-port equivalence, foreign-host
rejection, private-IP rejection, embedded credentials, and DNS
rotation between issuer check and endpoint use.
* feat(console): inline node picker replaces back-to-console banner
Drops the 32px banner the console proxy used to inject above proxied
server-UI pages and replaces it with an inline node-id pill in the
existing #ui-header. Click the pill to open a dropdown that lists
healthy nodes (health dot, ws count, reachable/degraded/unreachable
text) plus a top-row link back to the console.
Reuses the .ws-tab-dropdown shell from ui/static/style.css for
animation, shadow, theme override, and item layout, so the picker
visually matches the workstream-tab chevron menu it sits next to.
Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron
menu's handler with cross-reference comments at both sites.
Lazy-fetches /v1/api/cluster/nodes against the console origin
(bypassing the prefix shim) on first open.
Reclaims 32px of vertical space, consolidates three separate
"you're on node X via console" indicators into one, and turns the
wayfinding chrome into a real cluster-nav primitive.
* fix(console): address Copilot review on node picker
- Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap)
instead of relying on the default 100 — clusters with more than
100 nodes were silently dropping rows from the picker.
- Hand off focus to the first menu item after the async fetch
resolves: openMenu()'s deferred focus hook ran while only the
skeleton was in the DOM, so first-open keyboard users were
stranded on the trigger until they pressed an arrow key.
- Tab now closes the menu without preventDefault, so focus moves
to the next focusable element on the first press (ARIA APG menu
pattern). Escape still preventDefault + returns to the pill.
- Cap pill max-width at 240px and ellipsize the id span; node ids
are accepted up to 256 chars upstream and could otherwise push
the title and right-side controls off the appbar. Pill carries
a title attribute so the full id is still legible on hover.
* fix(session): properly inject queued user messages mid-loop
Two queued-user-message bugs in ``ChatSession.send()``.
**Mid-tool-call: ``Unexpected role 'tool' after role 'user'`` on Mistral.**
The ``supports_tool_advisories`` capability flag (default False for
unknown openai-compatible models) routed cap-off providers down a
short-circuit branch in ``_collect_advisories`` that called
``_flush_queued_messages`` directly. That appended a ``user`` turn
between ``assistant(tool_calls)`` and ``tool``, which mistral-common's
``_validate_message_order`` rejects with a 400.
Drop the flag. All providers now run the unified path: queued user
messages become ``UserInterjection`` advisories that ride inside the
tool result envelope via ``wrap_tool_result``, splicing
``<system-reminder>`` text into the tool message's content. Role
sequence stays ``assistant → tool``. Live-confirmed on Mistral
medium and Qwen3 — both correctly distinguish system-reminder from
tool stdout in their reasoning.
**Mid-stream: queued message orphaned until next user send.**
After a no-tool assistant turn, ``_flush_queued_messages`` would
append the queued user message to history and the loop would
``break``, leaving the message at the tail of history with no
model response. Visible as "two sends to get one reply".
``_flush_queued_messages`` now returns ``bool``. The no-tool branch
``continue``s on drain instead of ``break``ing, so the model gets a
turn over the extended history.
Tests:
- ``test_collect_advisories_drains_text_queued_messages_to_persistent``
pins the unified-path drain (text-only queue → ``UserInterjection``,
no separate user turn appended to ``self.messages``).
- ``test_send_continues_when_messages_queued_during_streaming`` pins
the loop-continue behavior (fails with 1 stream call pre-fix,
passes with 2 post-fix).
* fix(session,ui): reject queued attachments + paperclip busy state
Copilot pointed out that the attachment-bearing branch in
``_collect_advisories`` had the same role-ordering bug as the
text-only path that 802658f fixed: an attachment-bearing queued
item would still call ``_append_user_turn`` mid-tool-call,
injecting ``user`` between ``assistant(tool_calls)`` and ``tool``.
Pragmatic fix: don't allow attachments to be queued at all.
**Backend.** ``ChatSession.queue_message`` raises a new
``AttachmentsNotQueueableError`` when called with non-empty
``attachment_ids``. The interactive ``/send`` route catches it,
releases reservations via the existing ``_release_reservation_on_fail``
hook, and surfaces ``status: "attachments_busy"`` to the caller
with the IDs in ``dropped_attachment_ids``. The coord adapter
mirrors the cleanup (releases the soft-locked reservation taken
for ``_send_id``) so the create-with-attachments path can't leak.
Now that the queue can never carry attachments, the per-item
``att_ids`` slot is gone:
- Queue tuple slimmed ``(cleaned, priority, att_ids)`` →
``(cleaned, priority)``.
- ``_flush_queued_messages`` collapses to a single combined-text
user turn (no attachment branch).
- ``_collect_advisories`` queue-drain pushes ``UserInterjection``
advisories only (no ``attachment_items`` list).
- ``dequeue_message`` no longer unreserves (queue can't reserve).
- ``_resolve_attachment_ids`` had no remaining production callers
and is deleted along with the tests that exercised it in
isolation.
**Frontend.** ``Composer.setBusy`` disables the paperclip whenever
busy (regardless of ``queueWhileBusy``) — text still queues,
attachments don't. ``chat.css`` gains a ``.composer-attach:disabled``
rule (mirrors the existing ``.composer-send:disabled`` treatment)
so the affordance actually looks unclickable instead of falling
through to the UA default. ``title`` and ``aria-label`` are kept in
sync for AT users (WCAG 4.1.2).
Both interactive and coordinator UIs handle the new
``attachments_busy`` response with a chat-surface error bubble:
> Attachments can't be sent while the assistant is working.
> Send a text-only message now, or wait and resend with attachments.
Chips stay in the composer so the user can retry once idle.
**Tests.** Replaced the now-impossible ``TestQueuedWithAttachments``
class with a rejection-coverage class. Rewrote the
``_queue_with_attachment`` route-test fixture to reserve directly
via ``reserve_attachments`` (the queue path no longer reaches the
reserved state). Added a route-level test for the new
``attachments_busy`` contract.
* Bound search tool output against pathological inputs
Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.
Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
+ --max-columns-preview, --max-filesize 10M, --max-count 100,
--no-config, --no-messages, plus negative globs for the same
noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.
Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
Defends against single-line files (training data, minified bundles)
that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
pipe read is blocked in the kernel — proc.wait(timeout=…) alone
was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
child writes to stderr while we're still reading stdout. Cap on
captured stderr keeps a hostile child from growing the buffer.
Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
analytically from budget / file_count / avg-line-length so we hit
the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
head+tail _truncate_output strategy never silently drops middle
files in a search result.
Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
branch already had. With auto_approve on the search tool, that was
exploitable: path='--pre=COMMAND' would have made ripgrep run the
script as a per-file preprocessor and surface its stdout. Added
`--` and a regression test.
State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
dedicated 'killed by signal N' message instead of being parsed as
success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
now returns a dedicated byte-cap message instead of the malformed-
output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
(no more 'no matches' tag on a 'malformed' payload).
Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
but if a future caller forgets the flag, an attacker who can set
one of these env vars could plant a config containing --pre=… and
recreate the same RCE shape.
Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
of subprocess.run (the previous tests passed ChatSession kwargs
that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
to exercise the byte-cap trim, mega-line-no-newline edge case, the
watchdog timeout when the child writes nothing, and the stderr
drain under load.
- test_env_scrub picks up the new tool-config keys.
* Address Copilot review on #473
- Budget the Tier 2/3 header up front so the formatter's emission stays
strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
counted body bytes, letting the final string overflow by ~120 chars
(header + separator) and triggering _truncate_output's head+tail
dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
is kept as a starting estimate, but if that K's actual emission
doesn't fit (the estimate ignores the header and overweights shared-
path compression) we step down through the ladder before falling
through to Tier 3. The previous one-shot K could collapse to counts-
only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
SIGKILL). There's a narrow race where the child can exit naturally
between our read and our kill; preserving a non-negative rc means
rg's rc=2 ('matches found but some files had errors') no longer
silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
pass` blocks in _search_capture (stderr drain, pipe close in the
cleanup finally) so static analysis and future readers can see the
silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
instead of the +512-char slack that was masking the header overflow.
- New regression tests:
- Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
- capped + rc=2 surfaces stderr instead of being normalised to success
- capped + rc<0 (our SIGKILL) flows through as a partial-result success
* chore(search): post-review cleanup
Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).
- q-1: collapse the Tier 2 candidates filter to a single expression.
Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
is already floored at 1, so the comprehension always yields ≥ ``[1]``).
``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
semantics (analytical seed → step down through (5, 3, 1) from the
highest rung ≤ the estimate). The previous wording suggested every
Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
statements in test_tier2_steps_down_ladder_before_falling_to_tier3
into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
paths in _search_capture to one line each. Both sites now read with
the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
Keeps the load-bearing semantic (cap is on the content portion only)
and the pathological-line defence; drops the paths-aren't-bounded
parenthetical, which was background reading rather than WHY.
* feat(providers): api_surface toggle + mistral medium reasoning fix
Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions. The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).
Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
and thread it through `create_provider` / `model_registry.get_provider`.
`openai-compatible` defaults to Chat Completions; operators can flip
individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
`chat_template_kwargs`. Operators running gpt-oss-style local
templates that consume `reasoning_effort` from the chat template now
opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
server-side at create/update time; pre-filled by Detect via the
profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
for capability and server_compat resolution; previously the fallback
passed `alias=None`, which silently dropped per-model caps on the
agent path.
Tests: 5117 passed (-m "not live"); ruff + mypy clean.
* fix(providers): don't auto-suggest Responses for Mistral medium
vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls. Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.
Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile. Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.
* fix(providers): address Copilot review on PR #469
- providers/__init__.py: drop the redundant *_responses_provider /
*_chat_provider names; have create_provider use _openai_provider and
_openai_compat_provider directly so they're not flagged as unused
globals.
- console/server.py: tighten _validate_api_surface to a strict equality
match against the canonical {"chat", "responses"} set. The previous
strip().lower() membership check accepted ' Responses '/'CHAT' but
stored the raw string verbatim, which then failed to round-trip
through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
type, api_surface, extra_body) on provider == "openai-compatible" at
save time so toggling provider away can't leave a stale hidden surface
selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
longer flags the call as a wrong-name keyword (the point of the test
is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
for the api_surface validation on both create and update — covers the
bogus-value rejection, non-canonical-string rejection, and the happy
path persisting through to the refreshed registry.
* fix(memory): query-aware candidate selection + OR-of-terms search
The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`). On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context. Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.
## Functional changes
- `_init_system_messages` (`turnstone/core/session.py`): extract
recent context first, then `_search_visible_memories(context)` to
pull query-aware candidates. Search hits below `fetch_limit` union
with the recency list (deduped by memory_id) so the BM25 candidate
pool is always a SUPERSET of the prior recency-only pool — even on
noisy queries where the cap fills with stopwords, the recency-50
the original bug surfaced still reaches BM25. Empty context skips
search entirely. Candidate-selection logic extracted into
`_select_memory_candidates`.
- `search_structured_memories` (PostgreSQL + SQLite): per-term
clauses join with OR instead of AND. A row matches if ANY term
matches ANY of name/description/content. Downstream BM25 narrows
back down by relevance.
## Perf hardening
- Collapse the 1-3 fanned scope queries into a single SQL. New
backend methods `list_visible_structured_memories` /
`search_visible_structured_memories` union the visibility scopes
into one WHERE OR-group, so a composition rebuild now hits the DB
at most twice (search + recency) instead of up to six times.
- Cap and normalize search terms. Composition can hand a multi-KB
pasted message to ILIKE-based search; without a cap, every distinct
token would emit one unindexable predicate per scope-fanned query.
`normalize_search_terms` (`storage/_utils.py`) de-dupes
case-insensitively, drops <2-char tokens, and hard-caps at 16.
- Per-turn search cache. `_init_system_messages` fires from many
call sites within one turn (state transitions, MCP refresh, tool
results) and the recent-context query is identical across them.
Session-instance cache keyed by (query, mem_type, limit) absorbs
the duplicates; invalidated in `_append_user_turn` and after
memory save/delete tool actions.
- Stable secondary sort by `memory_id`. `updated` is second-precision
and `touch_structured_memories` can land a batch on identical
timestamps; without a tie-breaker SQL returns rows in
implementation-defined order, BM25 input shuffles, and the
LLM-side prompt cache misses across calls. All four backend ORDER
BYs now break ties on `memory_id ASC`.
## Quality cleanups
- Coalesce `memory.search.term_count` + `memory.search.zero_results`
into a single `memory.search` log carrying both `term_count` and
`result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
survives future readers.
## Tests
Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.
Memory test sweep: 102/102. Broader regression
(session, storage, coordinator, load_skill): 411/411.
* fix(memory): address Copilot review on PR #468
Three follow-ups from Copilot's inline review:
1. SUPERSET invariant violation (Copilot, session.py:5510).
`(search_hits + extra)[:fetch_limit]` capped the union back down to
fetch_limit, evicting the recency tail when search added distinct
hits. Recency tail is exactly where ancient-but-recently-touched
memories live — the recall this PR is supposed to improve — so
tail eviction recreated the bug for the narrow case where a query
term fell off the 16-cap and the matching memory sat in
recency[40-49]. Drop the cap; both halves are already SQL-capped
at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
defaults). BM25 over 100 candidates in pure Python is sub-ms;
irrelevant recency fillers get score=0 and don't pollute ranking.
Updates the docstring to actually be honest about the invariant.
Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
that locks the behavior in: 5 search hits + 10 recency = 15-item
pool, every recency item present, source="union".
2. Unbounded `query.split()` in normalize_search_terms (Copilot,
_utils.py:74). `str.split()` allocates the full token list before
the cap-after-16 break, so a 100KB pasted query did MB of throwaway
work even though only 16 tokens entered SQL. Switch to
`re.finditer(r'\S+', query)` — streaming iterator, stops scanning
at the first 16 normalized terms regardless of input size.
3. Misleading + unbounded log term_count (Copilot, session.py:8571).
`len(item["query"].split())` had two problems: same unbounded
split as #2, and the value reported the raw input token count
rather than the normalized term count that actually hit the SQL
WHERE clause — misleading metric for an operator trying to
understand storage-side behavior. Switch to
`len(normalize_search_terms(item["query"]))` — accurate count, and
bounded for free via #2.
Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect." False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.
Memory test sweep: 103/103. Broader regression: 411/411.
CI failure on main: test_publish_records_metric_outcome saw an empty
calls list — its monkeypatch was patching a different metrics
instance from the one `_publish_models_metadata` reads.
Two changes:
- test_close_reason_persistence.py: replace the bare
`srv_mod._metrics = MetricsCollector()` assignment in `_make_app`
with an autouse `monkeypatch.setattr(srv_mod, "_metrics", ...)`
fixture so the test's metrics swap auto-restores. Other test
files (test_auth.py, test_server_attachments_endpoints.py) carry
the same anti-pattern; left for a follow-up since they're not on
the critical path here.
- test_server_node_models_metadata.py: switch the publish-helper
metric test to a string-form `monkeypatch.setattr("turnstone.
server._metrics", FakeMetrics())` so it replaces whatever binding
the live module currently holds, regardless of what other tests
did to it. Robust against future leaks of the same shape.
* feat(coord): expose healthy model aliases per node on list_nodes
Surfaces a `model_aliases` field on each `list_nodes` row so a
coordinator can discover which model aliases each cluster node will
accept on `spawn_workstream(model=...)` without an HTTP fan-out.
Each server projects its registry into a `models` entry on
`node_metadata` (`{alias, provider, healthy}` per alias) at lifespan
startup, on every 30s heartbeat tick, and after `internal_model_reload`.
The publish helper short-circuits on a payload-equality cache so a
stable cluster doesn't pay UPSERT churn — exposed via the new
`turnstone_node_models_publish_total{outcome="written|skipped"}`
Prometheus counter so operators can graph cache hit-rate.
Coord client filters the per-alias rows to healthy aliases only and
drops the provider-side model identifier (`cfg.model`) — coords kept
reaching for it when they should pass the local alias.
* fix(coord): address Copilot+CodeQL feedback on list_nodes models work
- internal_model_reload: reuse a single get_storage() local across the
registry load and the metadata publish (Copilot:3047)
- _collect_node_models_metadata: iterate sorted aliases so two
structurally identical registries built in different insertion orders
serialize to the same JSON — directly improves the publish-cache hit
rate exposed via turnstone_node_models_publish_total (Copilot:3105)
- tests: drop mixed turnstone.server import style flagged by CodeQL —
hoist _metrics into the from-import block, and use sys.modules in
the shutdown-race regression test instead of `import as srv`
Address Copilot feedback on PR #465:
1. The has_alias fallback in both session_factories silently rewrote
any unknown caller-supplied alias to the default, including on the
fresh-create path where the create handler maps the factory's
ValueError to a 503 with operator-friendly text. A typo in
body.model would now silently start a workstream on the default
instead of telling the caller their requested model could not be
resolved. Move the fallback out of the factories: each factory
raises again on unknown aliases, and SessionManager filters stale
aliases out of the rehydrate path via a new ``model_validator``
constructor kwarg (production wiring passes ``registry.has_alias``
on both interactive and coordinator).
2. ChatSession.resume()'s elif branch flipped self.model to the
persisted model name even when the alias was unresolvable, leaving
the session paired with the constructor's default provider/client
but a removed model name — a broken state whose next API call
fails. Drop the model copy: keep the constructor's coherent
default (provider + model + capabilities) and just log the
unreachable saved values so the missing alias is auditable.
Tests:
- Move stale-alias coverage from the factory level into
SessionManager (tests/test_session_manager.py): validator drops
stale aliases before reaching build_session; live aliases pass
through unchanged.
- tests/test_sessions.py renamed test_resume_restores_model →
test_resume_keeps_defaults_when_alias_unresolvable to match the new
contract.
SessionManager.open() was calling build_session(ws) without a model
arg on the rehydrate path. The session_factory then resolved the
*current* default alias, ChatSession.__init__'s _save_config() (INSERT
OR REPLACE per-key) clobbered the persisted workstream_config with
those defaults, and the subsequent resume() "restored" what was now
the default — silently resetting model_alias, model, temperature,
reasoning_effort, max_tokens, skill, creative_mode, instructions,
token_budget, and notify_on_complete on every reopen and every
service restart, for both interactive and coordinator workstreams.
Three layers:
1. SessionManager.open() now reads workstream_config via
self._storage.load_workstream_config(ws_id) and threads the saved
model_alias into build_session(ws, model=saved_alias).
2. ChatSession.__init__ now skips its initial _save_config() when a
workstream_config row already exists for self._ws_id — protects
every other persisted knob without having to plumb each one
through the adapter signature, and catches any future construction
path that forgets to thread model through build_session.
3. Both session_factories (server.py interactive, console
session_factory.py coordinator) now treat an unknown caller-
supplied alias the same as an unset alias: fall back to the
runtime default rather than raising. Without this, a workstream
pinned to an alias an operator has since removed from the registry
would 500 on every reopen — defeating the "best effort restore,
default if the original is gone" contract this fix is meant to
deliver. Mirrors _effective_default_alias's existing has_alias
guard against a stale ConfigStore default.
Three changes from PR review:
- Permission gating: hide the Roles sub-tab button when the user
lacks ``admin.settings``. The sub-tab loads/saves through
``/v1/api/admin/settings``, so an admin with ``admin.models`` but
no ``admin.settings`` would otherwise see a perpetual 403 loader.
When Roles is the active sub-tab and the permission check fails,
snap the panel back to Definitions so the user lands somewhere
usable.
- Drop the redundant ``/v1/api/admin/model-definitions`` fetch from
``loadAdminModelRoles``. Both entry points (initial Models-tab
open + ``models_changed`` SSE refresh) flow through
``loadAdminModels`` first, which already populates ``_modelDefs``
+ ``_modelDefaultAlias``; ``_saveModelRole`` doesn't touch model
definitions, so the cached snapshot stays accurate when the save
chains back here. Halves the per-render request count and
removes a wasted round-trip on every cluster-wide model edit.
- Add ``test_models_changed_event.py`` covering the SSE fanout the
prior commit introduced: each model-definition CRUD endpoint
emits exactly one ``models_changed``, settings PUT/DELETE only
emit for keys in ``_MODEL_AFFECTING_SETTING_KEYS`` (parametrised
over all eight), and unrelated settings (e.g.
``session.retention_days``) don't trigger spurious refreshes.
The expected key set is pinned in the test so a stray addition
to the allowlist doesn't silently bypass coverage.
Same shape as the coordinator/judge rows already there: alias dropdown
+ reasoning_effort dropdown sourced from the existing
``model.plan_alias`` / ``model.plan_effort`` and
``model.task_alias`` / ``model.task_effort`` settings. Adds the four
keys to the SSE ``models_changed`` allowlist so changes from the
Settings API also trigger a live dropdown refresh, and filters them
out of the Settings tab so they only render in one place.
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.
Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.
Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.
Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
them with the Models sub-tab switcher (same a11y attrs, arrow-key
nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
coordinator.model_alias / coordinator.reasoning_effort out of the
Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
the Models tab instead of suggesting they set coordinator.model_alias.
* fix(console): home composer attachments + coord chat user-message pills
Two parity gaps in the console's coordinator surface:
- The embedded creator on the home page accepted only text — the
paperclip / paste / drop pipeline that the in-coord composer and the
interactive new-ws modal both expose was missing, so a user couldn't
attach files at create time. Stage Files in memory (no ws_id yet) and
ship them multipart on Start; the coord create endpoint already accepts
multipart via create_supports_attachments=True.
- User messages with attachments rendered as plain text on both live
send and history replay — no chip cluster like the interactive pane.
Added appendUserMessageWithAttachments and a structured userAttachments
list built from _attachments_meta (preferred) or the multipart parts
themselves, then rendered the same .msg-user-attach pill strip the
interactive pane uses.
Polish from a designer pass:
- Pill background was --panel-2, equal to the .msg bubble background in
both themes (border contrast ≈1.4:1, below WCAG 1.4.11). Switched to
--panel so the pill sits on a different surface than the bubble.
- Capped chip filename width inside the home composer (max-width 200px +
ellipsis) so a long filename doesn't push the strip past the textarea.
- aria-live="assertive" → "polite" on #home-coord-error; client-side
validation isn't an interrupt-level event.
- Reserved min-height on .home-composer-error and dropped the
display: none/block toggling so validation messages no longer reflow
the active-coordinators list below.
* fix(console): address PR #462 review feedback
- Block home-composer submit when files are staged but the task field is
empty. Server's _coord_create_post_install short-circuits on an empty
initial_message, so the multipart upload would create pending
attachment rows that never reserve onto a turn — orphaned until the
GC sweep. Fail in the browser instead.
- Drop the redundant `part &&` guard in coordinator.js's history-replay
multipart loop; the earlier `if (!part || ...) continue` already
filtered.
- Rewrite the home-mount .composer-chip-name CSS comment. shared/chat.css
defines .composer-chip{,-size,-remove} but no .composer-chip-name rule
— the span inherits the parent chip font with no width cap.
- Add smoke-guard string assertions in test_coordinator_page.py for
appendUserMessageWithAttachments and msg-user-attach so a future
rename can't silently regress the attachment affordance.
* fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration
Loading a saved workstream silently dropped tool results and missed
verdict / output-guard / truncation signals on replay. Root cause was
in `Pane.prototype.replayHistory`: an assistant message carrying both
content and tool_calls cleared the `lastToolBlock` anchor before the
following tool-result iteration could attach. The fix reorders content
to render before the tool block (matching live SSE order) and
restructures the tool-result branch to anchor by `data-call-id` so
multi-tool batches render `[hdr A][out A][hdr B][out B]` rather than
bunching outputs at the bottom.
Beyond the bug, replay now reaches near-parity with the live UX:
- Persisted intent verdicts and output_assessments flow through both
the SSE replay (`_build_history`) and the `/history` REST endpoint
used by coord. Single shared helper module owns the wire shape.
- Memory/recall calls persist instead of being filtered at storage
time — full audit trail; UI dims them by default with hover-reveal
so heavy memory usage doesn't crowd the narrative.
- Truncation indicator surfaces as a sibling pill (consistent across
interactive + coord) when a tool result hit the 2000-char cap.
- `replayHistory` wraps DOM work in `aria-busy` so screen readers
don't get a chatty announce-flood on long replays.
- `_build_history`'s storage I/O moves off the event loop via a new
`events_replay_prepare` async hook for the SSE path; other async
callers wrap in `asyncio.to_thread`.
Coord parity:
- `/history` REST endpoint decorates tool_calls with verdict +
output_assessment + truncation flag (was previously raw
`load_messages` output).
- Coord JS stamps `judge_verdict` / `heuristic_verdict` from
history-loaded `tc.verdict` so the existing batch render paints
the persisted pill, seeds the verdict cache to dedupe later live
SSE events, and emits an inline `.coord-tool-row-warning` chip
per call instead of a generic chat line.
- Memory/recall dim rule mirrored on `.coord-tool-row[data-tool-name=...]`.
* fix(replay): address PR #461 review feedback + raise tool-result storage cap
Copilot review feedback:
- Sibling-chain dim rule (memory/recall) now adds :focus-within
alongside :hover for .tool-output / .media-embed / .output-warning
/ .tool-output-truncated — keyboard users tabbing into a faded
subtree now get full opacity.
- ``cfg.open_post_load`` is now invoked via ``await asyncio.to_thread``
so its sync ``_build_history`` call (storage I/O for verdict
indexes + message reconstruction) doesn't block the event loop on
every workstream open. Mirrors the SSE replay path that's already
protected via ``events_replay_prepare``.
- Replaced the hardcoded ``2000`` literal in server.py and session.py
with ``TOOL_RESULT_STORAGE_CAP`` from the shared decoration module
so the UI truncation-pill detection can't silently desync from the
storage write side.
While here:
- Raised ``TOOL_RESULT_STORAGE_CAP`` from 2000 → 10000. A 2000-char
clip routinely cut grep / file-read bodies mid-line, leaving the
audit trail useless for retrospective debugging. FTS5 + row size
grow proportionally; the per-tool upper bound is still bounded
upstream by ``_truncate_output``'s context-budget clamp.
- Updated the user-visible truncation-pill tooltip on both
interactive and coord to reflect the new cap.
- ``test_decorates_tool_calls_and_marks_truncated`` now references
the constant instead of a literal so it stays correct on future
cap changes.
Speculative reliability machinery from the Stage 3 push that turned
out not to address any user-visible bug. The actual fixes (state /
activity disjunction in handleChildState, bulk-fetch race fix in
_fetch_live_block, push approve_request via cluster bus) are what
resolved the wedged-row issues. Manual testing showed the per-tab
SSE listener queue depth never climbed past single digits even when
rows were stuck — overflow was never the cause.
Removed
- ``_CRITICAL_EVENT_TYPES`` + ``_put_with_priority`` helper.
- Per-tab listener queue selective drop (back to plain
``contextlib.suppress(queue.Full)`` everywhere).
- ``ClusterCollector._fanout`` reverts to the same.
- WebUI ``_broadcast_intent_verdict`` / ``_broadcast_approval_resolved``
/ ``_broadcast_approve_request`` revert to plain ``put_nowait``.
- ``_queue_stats`` periodic SSE emit + frontend status-bar indicator
+ the supporting CSS rules.
- Broken ``.approval-block`` ``transition: max-height`` /
``max-height: 80vh`` / ``overflow: hidden`` rules — the transition
never fired (nothing toggled max-height) and ``overflow: hidden``
clipped long verdict reasoning. Layout-shift on auto-expand jumps
again, which is preferable to clipped content (Copilot review).
Tidied
- ``_CollectorProtocol`` / ``_ManagerProtocol`` method bodies switch
from ``...`` ellipsis to docstring-only bodies, silencing four
CodeQL "statement has no effect" warnings without changing the
Protocol contract.
5024 passed, ruff + mypy clean.
Lift the Children primitive out of CoordinatorAdapter into universal
SessionManager core primitives, replace the fragile poll + state-event
piggyback paths with first-class cluster bus event types for inline
approval delivery, and clean up the resulting frontend reducer.
Architecture
- New `turnstone/core/children_registry.py` — universal parent → children
+ reverse-lookup primitive with atomic `add_child` (returns parent UI
for race-free dispatch). Lifted from `CoordinatorAdapter`.
- New `turnstone/core/child_source.py` — `ChildSource` Protocol with
`SameNodeChildSource` (in-process via SessionManager state observer)
and `ClusterChildSource` (cross-node via ClusterCollector listener).
- `SessionManager._on_state_change` upgraded to multi-subscriber
(`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated
lock; CLI consumer migrated.
- `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in
the registry; fan-out lives in ClusterChildSource. Backward-compat
property facades dropped; tests updated to use the registry surface.
Cluster bus event vocabulary
- New event types `intent_verdict`, `approval_resolved`,
`approve_request` flow through both `ClusterCollector._apply_delta`
(translation from node SSE) and `emit_console_ws_*` (synthesis on
console pseudo-node).
- `CoordinatorAdapter._dispatch_child_event` re-emits as
`child_ws_intent_verdict` / `child_ws_approval_resolved` /
`child_ws_approve_request` on the parent coord's SSE stream.
- New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` /
`_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI
pushes to the global queue; ConsoleCoordinatorUI pushes to the
collector. `approve_tools` calls `_broadcast_approve_request` right
after setting `_pending_approval` so the items reach the coord tree
immediately, eliminating the bulk-fetch race.
Cleanups
- `pending_approval_detail` piggyback on `ws_state` / `cluster_state`
removed end-to-end. Bulk fetch + explicit verdict / approve-request
push are the canonical carriers.
- Browser `_judgePollTick` 90-second poll loop deleted; push path is
authoritative.
- `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409
retry; replaced with `invalidateLiveBadge` + standard schedule).
- Console `_fetch_live_block` derives `pending_approval` from a
disjunction (`activity_state="approval"` OR `state="attention"`
OR detail present) so the bulk fetch can't return false during the
state-transition race window.
- Coord-side merge guard in `flushLiveFetches` no longer clobbered:
`handleChildState` only stamps `sseUpdatedAt` when authoritatively
clearing detail.
- `child_locality` capability flag removed (was inert dead code).
Reliability
- Selective drop on listener queue overflow: critical event types
(verdicts, approvals, ws_closed, child_ws_*) evict one oldest item
to make room rather than dropping themselves on a full queue.
Best-effort events (state ticks, content tokens, status, activity)
drop as before. Applied to `SessionUIBase._enqueue`,
`ClusterCollector._fanout`, and the `WebUI._global_queue` puts in
the new broadcast hooks.
- `_state_subscribers` snapshot under a dedicated lock so concurrent
subscribe / unsubscribe during dispatch can't shift the iterator.
UX / a11y
- Loading placeholder in renderChildRow keeps row height stable while
the bulk fetch is in-flight (sr-friendly aria-label).
- Focus preservation across `_renderChildrenNow` (capture +
restore by row + marker) and across targeted `_updateChildRow` swaps.
- Layout-shift transition on the approval block max-height; respects
`prefers-reduced-motion`.
- Sidebar pending count: `(N children · M pending)`.
- Risk pill `aria-label` spells out level + confidence for SR users.
- Per-coord SSE listener queue depth surfaced in the status bar
(`queue N/500`) with color escalation (warn at >50%, danger at >80%).
Tests
- 305+ test changes across 8 files. New unit tests for
`ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber
observer), the new collector emit + apply_delta cases, the dispatch
cases for new event types, the broadcast hook overrides on both
WebUI and ConsoleCoordinatorUI, and the focus / placeholder /
pending-count frontend assertions in `test_coordinator_page.py`.
5024 passed, ruff + mypy clean.
* feat(console): multi-select delete UX for Saved Coordinators
Mirror the per-server "Saved Workstreams" multi-select delete onto the
console's "Saved Coordinators" section. Coordinator deletes go through
the existing routing proxy at POST /v1/api/route/workstreams/delete
(body-keyed by ws_id, since coordinators live on the node that owns
them) — no backend change required.
Pagination caps the visible page (and therefore the Select-All fan-out)
at 24. Without it, a Select-All on a busy cluster would pin the
console proxy pool with hundreds of parallel deletes through the
fan-out router. While in delete mode the saved-coordinators list is
frozen against SSE re-renders so visible cards don't shuffle out from
under the user's selections (drained on cancel / post-delete close).
Refactor: shared logic now lives in turnstone/shared_static/cards.{css,js}.
* .ws-delete-* CSS moved out of ui/static/style.css into the shared
sheet alongside .dashboard-card; the existing ui/static modal
markup picks up class hooks instead of id-scoped rules.
* createSavedCardsController() owns mode state, checkbox decoration,
toolbar wiring, focus trap, modal lifecycle, and batch fan-out.
Both ui/static (Saved Workstreams) and console/static (Saved
Coordinators) instantiate one controller; ui/static is now ~300
LOC lighter as a result.
* Internalises stale-selection prune across SSE re-renders, the
wsId->item lookup map (was O(selected x N)), and the aria-hidden
wrap on the toggle button's emoji glyph.
Designer review tightened the affordance:
* Modal close restores focus to the toggle button (was landing on
<body>) — WCAG 2.4.3.
* Modal [role="alert"] gets a red-chip treatment when populated,
stays invisible at rest via :not(:empty).
* Pagination consolidated onto the existing .pagination control
(terse "X / Y" label + arrow-glyph buttons) instead of a parallel
.coord-pagination treatment.
* Filled destructive buttons darkened to #dc2626 in dark theme so
the white label clears WCAG AA contrast (was 3.0:1 on --red).
Light theme keeps --red unchanged (5.9:1 already passes).
* Toolbar wraps below 700px viewport — Delete Selected drops to its
own full-width row underneath count + Cancel + Select All for
thumb-target separation.
* .ws-card-check:focus-visible outline + word-break on
.ws-delete-item for narrow-modal long aliases.
* fix(cards): address Copilot review feedback on PR #458
* closeModal focus restore now falls back to the section toggle button
(opts.buttonId) when prevFocus is hidden or detached. The post-delete
Close path runs cancel() before closeModal(), which puts the bar at
display:none — so the captured prevFocus (the bar's "Delete Selected"
button) is no longer focusable and focus would land on <body>,
defeating the WCAG 2.4.3 fix. Esc / Cancel paths still land on the
original focus owner because the bar stays visible in those flows.
* Saved Coordinators onClose drains _savedCoordsRetry before reloading.
Without it, SSE events that arrived during the delete-mode freeze
leave the retry flag true, so loadSavedCoordinators's .finally()
re-fires a second fetch immediately after the first resolves. Mirrors
the same idiom in cancelCoordDeleteMode.
Three issues from the Copilot review on PR #457:
1. SQLite race in bulk_close_stale_orphans (Copilot): the SELECT-then-
UPDATE flow doesn't re-apply the eligibility predicates on the
UPDATE, so a row that gets touch_workstream-bumped (or set_state-
transitioned) between the two statements would still be flipped
to closed. Postgres dodges this via UPDATE...RETURNING (one atomic
statement); SQLite needs the explicit re-application. Fix: rebuild
the WHERE conditions list once, apply on both SELECT and UPDATE,
then SELECT-back by ``state='closed' AND updated=now`` to get the
accurate closed-id list. A row that became fresh between the two
statements skips the UPDATE entirely.
2. SQLite IN-clause bind-parameter limit (Copilot): default 999 cap
could be exceeded on a backlog reap (e.g. after a long outage).
Chunked the candidate id list at 500 — same chunk size
prune_workstreams (line 453) uses for the same reason.
3. Wall-clock-dependent test asserts (Copilot, two locations): the
tests asserted ``updated > '2024-01-01T00:00:00'`` which is fragile
on systems with skewed clocks or pre-2024 dates. Replaced with
``updated != stale_seed`` — captures the same intent (the value
was bumped) without depending on wall-clock date.
Two ``...``-as-no-op flags from github-code-quality were false
positives — ``...`` is the standard Python idiom for Protocol method
bodies and matches every other method in _protocol.py. No code change.
Replaces the ``node_id == self_node_id`` orphan-scoping heuristic from
earlier on this branch with liveness-based scoping using
``services.last_heartbeat``. The heuristic was wrong for the post-#384
world: PR #384 (refactor: replace hash-ring rebalancer with rendezvous
hashing) deleted the rebalancer that used to keep workstreams.node_id
pointing at a live node. Without it, ``workstreams.node_id`` is now
stamped at create time and never updated, so in containerized
deployments with dynamic hostnames a dead pod's rows have ``node_id``
matching no surviving service — they'd accumulate forever under the old
heuristic.
services.last_heartbeat is the same primitive the rendezvous router
uses for routing. Reusing it here keeps reap scoping aligned with
routing: dead pods' rows fall out of the live set after the heartbeat
window and become reapable; alive pods' rows stay protected as long as
they heartbeat.
Mechanics:
- ``bulk_close_stale_orphans`` parameter renamed
``node_id: str | None`` → ``live_node_ids: list[str] | None``. The
WHERE clause becomes ``(node_id IS NULL OR node_id NOT IN
live_node_ids)``. ``None`` skips the filter entirely (single-process
/ tests / operator backfill). ``[]`` treats every row as
unprotected.
- ``SessionManager.close_idle`` pass 2 calls
``storage.list_services(self._service_type)`` to enumerate live
peers, passes their service_ids as ``live_node_ids``. ``_service_type``
is derived from ``self.kind`` (INTERACTIVE→"server",
COORDINATOR→"console") via a module-level mapping — no constructor
param, so production wiring can't miswire the kind/service_type
pairing.
- list_services failure → pass 2 is skipped this tick (conservative;
never reap when liveness state is unknown). Pass 1 still runs.
- ``workstreams.node_id`` with NULL value is always eligible — defends
against ANSI ``NULL NOT IN (...)`` evaluating to NULL (not TRUE) and
silently protecting orphans forever.
- Migration 048 simplified to ``(kind, updated)``; the new query's
``NOT IN (small list)`` predicate against an unbounded-cardinality
column doesn't index well, so leading ``node_id`` would just add
write cost.
Tests cover the live-services protection (own/dead/null cases), the
empty-peers reap-all case, the list_services-failure conservative
fallback, both kind/service_type pairings (interactive→"server",
coordinator→"console"), and the combined live_node_ids +
exclude_ws_ids filter matrix.
bulk_close_stale_orphans runs every min(300s, idle_timeout/4) on
every server and console process. Its WHERE shape is:
WHERE kind = ?
AND state IN ('idle','thinking','attention','running')
AND updated < ?
AND node_id = ? -- multi-node interactive only
At current scale the existing single-column indexes are sufficient —
idx_workstreams_state prunes to non-closed and the planner filters the
rest sequentially. At 100k+ rows that filter becomes a tablescan-
shaped cost.
A partial index covering only BULK_CLOSE_STATE_VALUES rows matches the
reaper's query exactly while staying tiny — closed rows (typically
95%+ of the table) and error rows are excluded, so the index is
roughly 5% the size a full multi-column index would be. Write
amplification only kicks in for transitions touching one of the four
covered states.
Column order (node_id, kind, updated): node_id is the most selective
filter for multi-node interactive (each server prunes to its own
node's rows), kind second so coord-only and interactive-only queries
within a node still get index-only scans, updated last so the range
comparison rides the trailing column.
Postgres uses CREATE INDEX CONCURRENTLY so the build is non-blocking
on a live system; SQLite has no concurrent concept and the table-
level write lock already serializes, so a plain CREATE INDEX is fine.
The console's coord SessionManager had no idle thread — close_idle was
never called for coordinator workstreams. This is the worse half of
the lifecycle leak: the dashboard filters via the in-memory pool, so
DB-only orphan coords were invisible. At empirical diagnosis,
coord closure was 16% (10 closed / 64 total) vs interactive 63%.
Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's
_idle_cleanup_thread but skipping the rate-limiter / global-queue arms
the console doesn't have. Started from the lifespan when coord_mgr is
constructed and server.workstream_idle_timeout > 0 (reuses the
existing setting — same cadence works for both kinds).
Initial sweep runs INSIDE the thread before the first sleep, not
synchronously in the lifespan: cold-start orphans are reaped without
blocking Starlette boot. Important because cold start with many DB
orphans (the precise condition this code targets) is exactly when the
UPDATE is most likely to be slow.
Helper takes an optional stop_event parameter purely for tests —
production callers pass None and the daemon runs for process lifetime.
This avoids the SystemExit-from-stub + module-wide filterwarnings
fragility a previous iteration relied on.
Four tests: initial sweep runs before first sleep, ticks fire each
loop, exceptions don't kill the thread, stop_event exits cleanly.
Real bug: workstream rows accumulate in non-closed states (idle,
thinking, attention, running) when their owning process restarts or
crashes. Empirical diagnosis on a live deployment found ~60 stuck
coord rows in DB invisible to the in-memory-keyed dashboard, plus
100+ interactive rows older than the 2h timeout (one stuck "thinking"
for 2 weeks — impossible across a process restart).
Root cause: close_idle iterates self._workstreams.values() — only the
loaded subset. Anything left behind by a prior process incarnation
sits in DB forever because nothing ever re-loads it.
This commit gives close_idle a second pass.
Pass 1 (existing, unchanged): close loaded IDLE rows whose
ws.last_active (monotonic) is past timeout. IDLE-only so legitimately-
attentive rows (waiting for user response) stay live.
Pass 2 (new): bulk-close DB rows of this manager's kind whose updated
is past the wall-clock cutoff and which aren't currently loaded.
Closes the broader BULK_CLOSE_STATE_VALUES set — any matching row is
by definition not loaded by any process and cannot be in a live
interaction. Scoped by self._node_id so a sibling node can't reap
rows we own (multi-node interactive correctness). No emit_closed —
never-loaded rows have no SSE listeners expecting them.
Lock invariant: pass 1 holds self._lock briefly to snapshot victims
and pop them (existing behavior). Pass 2 holds self._lock briefly to
snapshot the loaded keys, then releases before the DB UPDATE so a slow
reaper query can't block create/get/set_state.
Also fixes a same-process race in open(): the rehydrate path read DB,
released the manager lock, then re-acquired to install — a concurrent
pass 2 between the two acquisitions snapshots loaded keys without the
in-flight ws_id, and could clobber its DB row to closed. open() now
calls touch_workstream(ws_id) on rehydrate so the row's updated is
fresh against any pass-2 cutoff. Pure timestamp write is safe against
concurrent close() (close still wins on the state column).
Three new tests cover the DB orphan pass (basic, exclude-loaded, kind
filter) plus node_id scoping (own/foreign rows, None-skips-filter) and
the open() rehydrate touch.
Two new methods on the StorageBackend Protocol, with implementations on
both Postgres (UPDATE ... RETURNING) and SQLite (SELECT-then-UPDATE in
one transaction). No callers yet — wiring lands in subsequent commits.
bulk_close_stale_orphans(kind, cutoff, exclude_ws_ids, node_id=None)
flips rows in BULK_CLOSE_STATE_VALUES (idle/thinking/attention/running)
to closed when their updated timestamp is lex-older than cutoff. The
node_id filter scopes the reap to a single node's partition — required
for multi-node interactive deployments where each node only has
authority over its own workstreams.node_id rows. Excludes loaded ids
so the in-memory pass owns those.
touch_workstream(ws_id) bumps updated without changing state. Used by
the open() rehydrate path to defend against the orphan reaper clobbering
a freshly-loaded row whose DB updated is older than the cutoff. Pure
timestamp write is safe against concurrent close() because close still
wins on the state column.
BULK_CLOSE_STATE_VALUES is centralized in workstream.py so the two
backend implementations and FakeStorage all agree; if a new transient
state is added to WorkstreamState, deciding whether it joins this set
is part of the change rather than an after-the-fact audit across three
files.
Storage tests (run against both backends via the conftest fixture) cover
the kind/state/cutoff/exclude/node_id matrix plus touch_workstream.
The themed ``tool_reminder`` bubble below the tool block already
shows the metacog text, and the tool block immediately above it
carries the tool name — so a separate gray ``[repeat: list_workstreams()
called with same arguments]`` info line was just duplicate visual
noise (operator-visible in the screenshot below the bubble).
Drop the ``ui.on_info`` call inside ``_apply_post_execute_advisories``
that emitted the diagnostic line. Update the docstring to reflect
that the bubble is the canonical signal. Rename
``test_emit_repeat_ui_line_on_streak_fire`` →
``test_no_legacy_repeat_info_line_on_streak_fire`` and invert the
assertion.
CI typecheck failed because ``WorkstreamTerminalUI(TerminalUI)``
inherits from ``SessionUI`` (the Protocol), and the Protocol's
``on_user_reminder`` / ``on_tool_reminder`` declarations have empty
bodies — mypy treats those as implicitly abstract, so the subclass
became un-instantiable.
Add real implementations on ``TerminalUI`` that render reminders as
``[metacognition · type] text`` lines in yellow. This also restores
the metacog signal on the CLI surface (the legacy
``[metacognition: nudge injected — …]`` info-line went away with
``_emit_nudge_ping``; without this commit the CLI showed no signal
at all for metacog nudges). Tool-channel and user-channel render
identically because terminal output is anchored by stdout flow
rather than by DOM anchor — the line lands directly after the
message it advises.
Address Copilot's review feedback on PR #456 — the docstrings and
inline comments hadn't all caught up with the architectural shift
across the branch:
- ``_apply_reminders_for_provider`` docstring: "every user message"
→ role-agnostic, since tool messages also carry ``_reminders``
(tool_error / repeat).
- ``_mark_reminders_delivered`` docstring: same role-agnostic
update; explicitly note both channels.
- ``_append_user_turn`` callsite comment near
``_attach_pending_user_reminders``: still described splicing
``<system-reminder>`` blocks into user content; updated to
reflect the side-channel attach + transient-copy splice at the
provider boundary.
- ``_build_history`` block comment: was user-message-only; now
mentions tool messages and both ``user_reminder`` /
``tool_reminder`` SSE events.
- ``_build_history`` propagation comment: same role-agnostic note
on the per-entry surface.
- ``app.js`` ``user_reminder`` SSE handler comment: said the
bubble renders "above" the user message, but
``insertAdjacentElement('afterend', el)`` drops it BELOW.
- ``app.js`` ``replayHistory`` comment: said "insertBefore drops
the reminder directly above the just-rendered user bubble";
same fix — bubble lands BELOW.
No behaviour change.
The repeat-detection block in ``_apply_post_execute_advisories`` had
a leftover "clear streak when a write tool succeeded" branch from
when ``RepeatDetector`` tracked cumulative counts. With the
consecutive-streak semantics introduced earlier in the branch the
branch became:
1. Redundant — any different (name, args) signature already resets
the streak via ``RepeatDetector.record``, so an intervening
read/write naturally breaks the streak.
2. Actively wrong — the clear runs ONCE at the top of each
``_apply_post_execute_advisories`` call, before the per-result
loop records sigs. In a single parallel batch
``[bash, bash, bash]`` the clear runs once and then three
``record`` calls accumulate to count=3 in the same call → fires.
But across three sequential turns, each turn calls
``_apply_post_execute_advisories`` fresh, the clear runs at the
top of each call, and only one ``record`` per call follows — so
the count never gets above 1 and the canonical
"small local model stuck on ``bash('echo test')``" pattern
never triggered the nudge.
The asymmetry only existed for successful calls — failures don't
satisfy the ``not _tool_error_flags.get(tc["id"])`` predicate, so
the clear didn't fire and sequential failures already worked. The
fix is to drop the clear entirely; ``RepeatDetector``'s
consecutive-streak semantics handle every case uniformly.
Tests:
- ``test_successful_write_clears_streak`` →
``test_intervening_different_call_resets_streak`` —
rewords the assertion to reflect the actual mechanism (any
different sig resets, write-or-otherwise) since "writes clear"
was the bug, not the contract.
- ``test_failed_write_does_not_clear_streak`` →
``test_sequential_bash_failures_fire_repeat`` — same shape, just
framing fixed.
- New ``test_sequential_bash_same_command_fires_repeat`` —
regression for the bug user hit (three sequential successful
``bash('echo test')`` calls now correctly fire the nudge).
The yellow themed reminder card introduced for user-channel nudges
(correction / denial / resume / start / completion) now also fronts
tool-channel nudges (tool_error / repeat). Pre-fix the tool channel
shipped its reminders inside the tool-result envelope via
``wrap_tool_result``, leaking the ``<system-reminder>`` block into
``self.messages`` content (same problem the user channel had before
the side-channel refactor) and surfacing the legacy gray
``[metacognition: nudge injected — …]`` info line as the only
operator-visible signal — duplicated alongside the new themed bubble
for user-channel nudges.
Tool-channel parity:
- ``_collect_advisories`` now returns
``(persistent_advisories, metacog_reminders)``. Persistent
advisories (``GuardAdvisory`` / ``UserInterjection``) keep
riding ``wrap_tool_result`` because they ARE conversation
history. Metacognitive reminders extract to the second tuple
element; the caller attaches them to the tool message dict's
``_reminders`` side-channel and emits ``on_tool_reminder``.
- ``_apply_reminders_for_provider`` already handles ``_reminders``
on any role, so the tool-channel splice into wire content is
free. ``_build_history`` also already propagates
``entry["reminders"]`` regardless of role, so reload renders the
bubble too.
- ``SessionUI`` Protocol gains ``on_tool_reminder(reminders,
tool_call_id)``; ``SessionUIBase`` enqueues a ``tool_reminder``
SSE event with the ``tool_call_id`` anchor.
- ``_emit_nudge_ping`` had no remaining callers and was removed —
the themed bubble (live SSE + ``/history`` reload) is the
canonical operator signal for both channels now.
UI polish (the four fixes the screenshot caught for the user
channel + their tool-channel mirror):
- Bubble renders BELOW the message it advises (semantically: a
hint to the model right before its turn). ``addUserReminder``
swaps ``insertBefore`` for ``insertAdjacentElement('afterend',
el)``; ``addToolReminder`` anchors below the ``.ts-approval``
block whose tool result triggered the batch's reminder.
- Label uses the full feature name ``metacognition`` (was the
``metacog`` shorthand).
- Card width / alignment inherits from the base ``.msg`` rule —
``align-self: flex-end`` and the explicit ``max-width`` are
gone, so the card matches the user / assistant column instead
of pinning right-aligned narrow.
- The legacy ``[metacognition: nudge injected — …]`` gray info
line is gone for both channels.
Frontend additions:
- ``Pane.prototype.addToolReminder(reminders, toolCallId)``
anchors below the ``.ts-approval`` block (live: by
``data-call-id``; replay: by "last block in messagesEl"
fallback, which is correct because messages render in order).
- SSE switch case ``"tool_reminder"`` calls ``addToolReminder``.
- ``replayHistory``'s tool-message branch now calls
``addToolReminder`` when ``msg.reminders`` is present.
- ``addUserReminder`` advances its anchor on each loop iteration
so multiple reminders stack in queued order rather than
reversed.
Coord console parity:
- ``coordinator.js`` gains ``appendReminderBubble`` /
``appendUserReminderLive`` / ``appendToolReminderLive`` mirroring
the interactive UI. The tool-channel anchor walks
``toolRows[callId].batch`` to attach below the
``.coord-tool-batch`` construct (one bubble per dispatch turn,
matching the "one nudge per batch even with many failing tools"
drain).
- SSE switch handles ``user_reminder`` and ``tool_reminder`` on
the coord conversation surface.
- ``/history`` replay propagates ``msg.reminders`` for user and
tool messages — same wire shape as the interactive pane.
- ``.msg.user-reminder`` styles moved to
``shared_static/chat.css`` so both surfaces inherit the same
yellow themed bubble from the shared base.
Defensive read on ``_apply_reminders_for_provider`` (per Copilot
review on the closed PR): a malformed ``_reminders`` entry (string,
None, etc. — corruption / partial state) used to abort ``send`` via
AttributeError on the ``.get("text", "")`` call. Filter to dicts
before building the block, mirroring the same filter
``_build_history`` already applies on the wire-out side; an
all-malformed list passes through as no-reminders.
Tests:
- ``test_collect_advisories_drains_tool_buffer_on_last_result``
rewritten to assert the ``(persistent, metacog)`` tuple shape
and that ``MetacognitiveAdvisory`` no longer appears in the
persistent list.
- ``test_collect_advisories_holds_*`` and ``_drops_*`` updated for
tuple return.
- ``test_attach_emits_visibility_ping`` /
``test_collect_advisories_emits_visibility_ping`` inverted to
assert the legacy gray line is gone on both channels.
- ``TestSessionUIBaseToolReminderHook`` covers the new SSE event
shape with the ``tool_call_id`` anchor.
- ``test_malformed_reminders_filtered_out`` and
``test_all_malformed_reminders_passes_through`` cover the
Copilot-flagged defensive filter.
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context. The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.
Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``). At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean. ``sanitize_messages`` drops the sibling key on the wire.
Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call. ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised). ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.
UI surface:
- ``SessionUIBase.on_user_reminder`` enqueues a
``{type: "user_reminder", reminders: [...]}`` SSE event with the
same shape ``_build_history`` surfaces.
- ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
pill-styled) anchored above the user message it advises, both
live and on history replay.
- ``replayHistory`` renders ``addUserMessage`` before
``addUserReminder`` so the anchor lookup finds the just-rendered
turn (not a prior one).
- Multi-tab caveat documented inline: non-originating tabs receive
no ``user_message`` SSE event today, so a reminder may anchor to
a stale prior bubble until ``/history`` reload corrects it.
Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer. Both
now drain through a shared ``_drain_pending_advisories`` helper.
Removed the ``/history`` regex strip — the side-channel approach
makes it redundant. Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).
Tests:
- ``TestApplyRemindersForProvider`` — pass-through-by-reference,
string + list content splice, escape on user-typed wrapper tags,
multi-reminder ordering, source-untouched invariant, delivered
flag skip path, fallback for unexpected content shape.
- ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
no-flag, only marks user messages with reminders.
- ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
msgs when provided, falls back when not.
- ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
the user buffer.
- ``TestReminderSidechannelIsolation`` — compaction's
``_format_messages_for_summary`` and the title-gen extraction
loop cannot see reminders by construction.
- ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
enqueues the right SSE shape.
- ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
propagation, absent / empty / multi / coexist-with-attachments
cases, malformed input filtering, all-malformed elision.
- ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
``_reminders`` and ``_reminders_delivered``.
Cleanup pass on the metacognitive nudge stack — restores pre-split
errored-counts-toward-repeat behaviour and tightens the is_error
plumbing through the per-batch advisory hook.
The per-batch hook in ``_run_loop`` was duplicating the is_error
signal: ``self._tool_error_flags`` (set by ``_report_tool_result``)
and a string-prefix tuple (``Error`` / ``JSON parse error`` / …).
Two truth sources is what got us here — bash commands that exit
non-zero with normal stdout matched the flag but not the prefix,
the deny path matched the prefix but not the flag, and the result
was that stuck-loop detection silently broke for the most common
failure mode (the model bashing the same broken command).
Single source of truth now:
- ``_execute_tools.run_one`` deny branch routes through
``_report_tool_result(is_error=True)`` so denied calls populate
``_tool_error_flags`` like every other error path.
- The error-prefix tuple is gone; the write-success-clear gate and
the tool-error-nudge gate both read ``_tool_error_flags`` only.
Repeat-detection state moves from a ``set[str]`` (fired on the second
identical call, ignored errors entirely) to a ``RepeatDetector``
helper in ``metacognition.py`` with consecutive-streak semantics:
- Threshold raised from 2 to 3 — two-in-a-row was noisy on
legitimate transient retries; three is the cheapest stuck-loop
signal.
- Recording a different signature resets the count, so [A, A, B, A]
is two short streaks of 2 and not a streak of 4. Bounded by O(1)
state regardless of session length.
- Errored calls now count toward the streak (the split into a
separate metacog module unintentionally introduced a "skip errors"
branch — restored).
While there:
- ``metacognition._COOLDOWN_SECS`` default aligned to 300s (matches
``MemoryConfig.nudge_cooldown`` and the ``memory.nudge_cooldown``
config-store default; was set to 30 by an earlier investigation).
- The per-batch advisory block (~80 lines of mixed orchestration
inside ``_run_loop``) is extracted to
``ChatSession._apply_post_execute_advisories`` so the wired
behaviour is testable without driving ``_run_loop`` end-to-end.
Producer extraction to a dedicated module is deferred to a
follow-up; advisory producers all live on ``ChatSession`` for
now per existing convention.
- Frontend ``appendToolOutput`` (turnstone/ui/static/app.js) now
skips rendering when the parent approval block is denied or
the output starts with ``Denied by user`` / ``Blocked``,
mirroring the history-replay guard at ``_build_history``.
Previously the live SSE path didn't need this guard because
the deny path never emitted a ``tool_result`` event; the
is_error routing change above means it does now, so without
this guard the badge from ``resolveApproval`` and the SSE
output would both render.
Tests: 8 unit tests for ``RepeatDetector`` covering streak,
threshold, clear, and intervening-sig reset; 9 integration tests
for ``_apply_post_execute_advisories`` covering the wired
behaviour (3-identical fires warning + advisory + UI line, errored
calls count toward streak as a regression guard, intervening sig
resets streak, successful write clears, failed write does not,
JSON outputs tracked but not inline-warned, tool_error nudge gates
on memory_count, repeat UI line emitted on streak fire).
The pre-existing comment said pending_approval_detail "rides on
every ws_state event" — that overstated the case. The node-side
emit is gated on ``_pending_approval is not None`` so the field is
absent on the steady-state broadcast and possibly null on a node
mid-rolling-upgrade. The handleChildState fallback already
handles both cases; only the comment was wrong.
Inline child approve/deny in the coord tree UI was rendering downstream
of the bulk-live cache (``GET /v1/api/cluster/ws/live``), not the SSE
stream. ``child_ws_state`` events were tiny notifications that fired
an urgent live-bulk fetch on every activity_state transition into/out
of "approval", just to pick up the rich ``pending_approval_detail``
payload. With multiple coord tabs and multi-child workstreams, that
urgent-fetch pattern compounded the SSE-executor pressure Shape A
is unwinding.
Thread the field through every layer so the SSE event itself carries
the rich payload — browser mutates ``liveBadgeCache`` directly,
no urgent fetch:
1. Node ``WebUI._broadcast_state`` emits ``pending_approval_detail``
on ``ws_state`` events. Gated on ``_pending_approval is not None``
so the per-broadcast verdict-cache deepcopy only runs when there
is actually an approval pending. ``_build_node_snapshot`` also
projects the field so the console's reconnect-via-snapshot
resync path delivers it (without this the new collector
forwarding would never see the field on a snapshot row).
2. Console ``ClusterCollector._apply_delta`` (live ``ws_state``
forwarding) and ``_reconcile_node`` (snapshot resync diff) both
forward the field on the emitted ``cluster_state`` event, AND
``_apply_delta`` persists it on the cached ``ws`` dict so the
``get_node_detail`` / ``get_snapshot`` endpoints between
reconciliations don't render stale approve/deny buttons.
3. ``CoordinatorAdapter._dispatch_child_event`` re-emits the field
on the ``child_ws_state`` event sent to coord listener queues.
4. Frontend ``handleChildState`` reads ``ev.pending_approval_detail``
and writes it directly into ``liveBadgeCache``, tagging the
entry with ``sseUpdatedAt``. ``flushLiveFetches`` honors that
tag for ``SSE_AUTHORITATIVE_MS`` (3s) — the upstream
``/dashboard`` cache has its own ~2s TTL, so a bulk-poll
landing right after a transition can otherwise clobber the
fresh SSE-set state with pre-transition data.
The pre-fix ``enteredApproval`` / ``leftApproval`` urgent-fetch
branch is removed. The 409 stale-call_id retry path keeps its own
urgent fetch — that's a different scenario.
Tests cover the forwarding contract at every layer, the broadcast
gate (event includes the field when an approval is pending,
omits it otherwise, and clears after resolution), and the
``flushLiveFetches`` merge-guard structural shape so a refactor
that keeps the symbols but inverts the comparison or drops the
``prev.live`` check can't pass silently.
``coordinator_children`` was calling ``storage.list_workstreams``
directly on the event loop, ``coordinator_tasks`` did the same with
``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called
from both handlers, plus ``coordinator_history`` and
``_resolve_coord_session``) did the same with
``storage.get_workstream`` on its cold-cache path.
The cold-cache resolver path is hit on every console restart,
coordinator eviction, and console proxy hop — exactly when the
event loop is most contended. Three coord tabs reconnecting after a
brief network blip = three serial event-loop blocks per call site.
Other lifted handlers in this file already use
``asyncio.to_thread``; bring all four call sites onto the same
pattern.
Convert ``_resolve_coordinator_or_404`` to ``async def`` and update
its four call sites to ``await``. Exception flow is unchanged.
Each coord ``events`` SSE listener parks a thread on
``client_queue.get(timeout=5)`` for the connection lifetime. The
console's coord endpoint was wiring no ``sse_executor_lookup`` on
``coord_endpoint_config``, so those parks landed on Python's default
ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every
other ``asyncio.to_thread`` caller (storage, router, audit). A few
coord tabs against a multi-child workstream would stall new request
handlers waiting for a worker thread.
Mirror the interactive-side precedent (the ``sse_executor`` /
``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a
dedicated 200-thread ``coord_sse_executor`` in the console lifespan
and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``.
Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()``
so no new listeners arrive at a dying pool. ``cancel_futures=True``
discards queued-but-not-started futures during teardown.
Update the stale comment on the interactive-side wiring that claimed
"coord wires None and falls back to the default executor" — it now
points at the console's matching wire.
Three follow-ups from Copilot's round-2 review on #453.
ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present). Operator looking at logs for a
config.toml typo would see the wrong cause. Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through. Behavior unchanged — existing registry still
preserved on every ValueError path.
Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load. ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve. Today shutdown() iterates
empty dicts. Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.
Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``. The
explicit probe was removed in commit 1ba17ed when the helper switched
to ``load_model_registry(..., strict=True)``; the test name and
docstring still talked about a probe. Updated wording reflects that
the loader's strict-mode re-raise is what the helper catches now.
132 tests pass.
Hygiene follow-ups from the multi-stage code review on #453.
perf-1 — sync helper called from async route handlers
``_refresh_coord_registry`` runs two sync DB reads and a registry reload
that takes ``_client_lock``; calling it directly from an async handler
held the event loop for the duration. All four call sites now
``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the
pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``).
perf-3 — ModelRegistry.reload() tore down all clients unconditionally
The reload always closed every cached client and provider, even when
the changed fields (``model``, ``temperature``, ``context_window``)
didn't touch the connection target. Now selective: clients drop only
when alias removed or ``(base_url, api_key, provider)`` differs;
providers drop only when alias removed or ``provider`` string differs.
Keeps connection pools warm across the common admin-edit case where
only metadata changed. Two new ``test_model_registry`` cases lock the
keep-warm vs drop-on-change behaviour, and the existing
``test_reload_clears_clients`` was updated (it asserted the old
overly-aggressive contract) into
``test_reload_keeps_clients_when_connection_target_unchanged``.
q-5 — helper rename
``_refresh_console_coord_registry`` → ``_refresh_coord_registry``. The
``console_`` prefix was redundant given the function lives in
``turnstone/console/server.py`` and sibling helpers there
(``_notify_nodes_model_reload``, ``_publish_config_change``,
``_collect_model_status``) all omit it.
q-1 — shared test middleware
``tests/test_admin_model_registry_refresh`` now imports the
header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers``
and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the
``TestClient``. The local hardcoded variant duplicated infrastructure
the helper module exists to centralise.
q-3 — multi-alias test registry
``_make_registry`` extracted a ``_make_config`` helper and gained an
``extras={alias: model}`` param so multi-alias scenarios stop
hand-building ``ModelConfig`` literals.
``test_delete_endpoint_refreshes_registry`` now uses the helper.
310 tests pass across the related coordinator + model surfaces.
bug-3 / q-2 from the multi-stage review on #453: the previous test
``test_update_endpoint_with_empty_body_does_not_blow_up`` asserted only
that the registry's model name was unchanged after an empty PUT, which
holds whether or not the refresh ran (DB row matches registry → refresh
is idempotent). A regression that always called
``_refresh_console_coord_registry`` — exactly the gate this test was
meant to lock — would have left the assertion green.
Rename to ``test_update_endpoint_skips_refresh_on_empty_body`` and spy
on the helper via ``monkeypatch.setattr``. Empty-body PUT must register
zero calls; any future change that drops the ``if updates:`` gate now
fails loudly.
Two correctness follow-ups from the multi-stage code review on #453.
bug-2 / perf-2 (DB probe was theatre + double scan)
The previous probe defended nothing the loader didn't already swallow
on the next line: ``load_model_registry``'s row-loop catches Exception
internally, so a transient DB error after the probe still degrades to
a config.toml-only registry that ``existing.reload()`` would apply,
silently dropping every DB-sourced alias. And on the happy path each
CRUD paid for two scans of ``model_definitions``.
Add a ``strict: bool = False`` flag to ``load_model_registry``. When
strict, the row-loop's except re-raises instead of swallowing. The
helper passes ``strict=True`` and drops the probe — single DB scan,
real failure isolation, the loader's silent fallback can no longer
mask a partial-result regression. Default ``strict=False`` so CLI /
lifespan callers keep their boot-with-config-fallback behaviour.
bug-1 (shutdown could escape after a successful reload)
``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the
helper's ``finally`` block ran it outside the try/except. A raising
close() after a successful ``existing.reload()`` would surface as 500
with the registry already mutated and the audit row already recording
success. Wrap ``new_registry.shutdown()`` in its own try/except that
matches the helper's belt-and-suspenders error policy elsewhere.
The helper's docstring also drops the obsolete probe paragraph; the
``if existing is None: return`` branch gets a one-line inline comment
about the boot-from-empty case (the multi-paragraph version restated
behaviour the line itself documents).
129 tests pass (test_admin_model_registry_refresh + test_model_registry).
Two follow-ups from Copilot review of #453:
1. ``load_model_registry`` swallows storage read errors internally
(logs + continues with config.toml-only models). Without a strict
probe in the helper, a transient DB outage on an admin CRUD would
apply a truncated registry that drops every DB-sourced alias —
silently, since the loader returns a non-empty registry built from
``[models.*]`` config.toml entries. Add an explicit
``storage.list_model_definitions(enabled_only=True)`` probe before
the loader call so the failure is visible here and the existing
registry is preserved on outage.
2. The previous docstring claimed ``admin_model_reload`` "has its own
boot-from-empty story." It doesn't — it just calls this helper,
which no-ops when ``coord_registry`` is None. When no model rows
existed at boot, lifespan leaves the entire coord subsystem
uninitialized (no ``coord_mgr``, no ``coord_adapter``, no
``session_factory``), and a console restart remains required after
the operator adds the first row. Tighten the docstring to admit
that limitation rather than overstating the helper's reach.
New test ``test_helper_preserves_registry_when_db_probe_fails``
monkeypatches ``list_model_definitions`` to raise and asserts the
existing registry stays intact.
The console builds ``app.state.coord_registry`` once at lifespan startup
and the coordinator session factory closes over that exact instance.
Until now, the model-definition admin endpoints (create/update/delete)
wrote to the DB but never touched the in-process registry — and the
explicit reload button only fanned out to nodes via HTTP, also leaving
the console's own registry stale.
Symptom: an operator who changed the underlying model name behind a
local-LLM alias (same alias, same endpoint) saw the DB row update
immediately, but coordinator sessions kept calling the prior model
name until the console process was restarted.
Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh
ModelRegistry from DB and applies it to ``app.state.coord_registry``
via the existing thread-safe ``ModelRegistry.reload()`` — in-place
mutation preserves object identity so the factory closure keeps
working, and active coord sessions auto-pick up the swap on their
next ``send()`` via ``ChatSession._refresh_model_from_registry``.
Wired into four endpoints in ``console/server.py``:
- ``admin_create_model_definition`` — after the DB write
- ``admin_update_model_definition`` — after the DB write, gated on
``if updates:`` so a no-op PUT skips the rebuild
- ``admin_delete_model_definition`` — after the DB write
- ``admin_model_reload`` — between ``_publish_config_change`` and
``_notify_nodes_model_reload`` so the console mirrors what the
reload broadcasts to nodes
Failure isolation: a load or reload error leaves the existing registry
intact (logged + swallowed). Coord stays usable while the operator
investigates; the explicit reload remains the user-facing recovery path.
No node fan-out on CRUD — the explicit reload button continues to gate
cluster-wide HTTP propagation, preserving today's UX semantics on shared
clusters.
Tests in ``tests/test_admin_model_registry_refresh.py`` cover:
- helper-level: rebuild from DB, identity preservation, no-op when
registry is None, preservation on load failure / no-enabled-rows /
reload validation error
- endpoint-level: create / update / delete / explicit-reload all
refresh the registry; an empty PUT skips the rebuild
Production fan-outs are frequently hitting the 6 KiB per-child cap by
just 1-2 KiB, forcing the coordinator into a follow-up inspect_workstream
round-trip per truncated child to recover the tail. Bumping the cap to
10 KiB absorbs the common overshoot without changing the truncation
semantics — truncated=True still fires for genuinely oversized messages,
and inspect_workstream remains the unbounded follow-up.
Worst-case context impact: a 32-child fan-out at the cap is now ~320 KiB
(was ~192 KiB), still well within commercial model context windows.
Typical fan-outs of 1-5 children land at 10-50 KiB.
LAST_ERROR_MAX_LEN (1 KiB) is unchanged — it's intentionally smaller
than the wait cap so error truncation happens at write time, and
1 KiB still sits well below 10 KiB.
WAIT_MESSAGE_MAX_BYTES is referenced by name (not literal 6144) in the
truncation test, so no test value needs updating.
The coordinator system message was descriptive about parallelism rather
than prescriptive — "while multiple children run in parallel" framed
fan-out as incidental, and "a tasks entry, a child to own it" primed
singular delegation. The spawn_batch example (benchmark A, benchmark B,
prototype the winner) showed dependent work under a fan-out framing,
teaching the wrong shape.
In practice the coordinator failed to decompose enumerable requests
("top stories on HN, Lobsters, /r/programming, …") without explicit
"please fan this out" instructions, on both GPT-5.5 and Claude Opus.
base_coordinator.md
- Replace singular "a tasks entry, a child to own it" with plural
"enumerate the independent units of work, spawn one child per unit,
run them in parallel by default. Sequential only when one child's
output feeds the next."
- Tighten the delegation paragraph.
tools_coordinator.md
- Drop the persona repetition that duplicated base_coordinator.md.
- Drop the prescriptive "## Workflow shape" section (the cost note is
already in wait_for_workstream's tool description; the edit-X
redirect is already in the persona).
- Drop "in one approval" / "single approval" mentions to avoid
surfacing approval mechanics to the model.
- Replace the misleading spawn_batch example with truly independent
items; drop "(up to 10)" which overstated the cap (it's per-call,
not global, and is documented in the tool schema).
- Add a course-correction example to send_to_workstream — the pattern
coordinators most often replace with cancel-and-respawn.
- Drop the read action from the tasks examples to keep the lifecycle
(add → update → remove) coherent.
Coord system message ~16% shorter (4440 → 3722 chars). Both GPT-5.5
and Claude Opus now naturally decompose the news-board prompt without
explicit fan-out instructions. 29 prompt-composition tests pass.
The server's --help epilog and compose.yaml both reference
--skip-permissions, but the argparser never defined it, so any
container started with SKIP_PERMISSIONS=1 exited with
"unrecognized arguments: --skip-permissions".
Wire the flag through to app.state.skip_permissions, OR-ing it
with the existing tools.skip_permissions config-store setting so
the stored value still works on its own.
Replaces the old mermaid-rendering shot with a coordinator session
mid-attention — parallel tool batches, judge-graded approval,
children + tasks side panels — which more accurately represents
what the platform does today.
* perf(api): offload tenant_check to thread on lifted session handlers
Every make_*_handler factory in turnstone/core/session_routes.py invoked
cfg.tenant_check(request, ws_id, mgr) synchronously inside its async
handler. For the interactive surface tenant_check chains through
_interactive_tenant_check → _require_ws_access → resolve_workstream_owner,
which short-circuits on mgr.get(ws_id) for warm cache but falls through
to a synchronous get_workstream_owner SQL call on a cold cache,
blocking the event loop for the duration of the storage round-trip.
Wrap each of the 8 call sites (approve, close, cancel, events, history,
detail, send, dequeue) in await asyncio.to_thread(...) — mirroring the
existing storage-offload pattern at make_history_handler's other call
sites. Coord wires tenant_check=None and is unaffected. Five handlers
gain a local import asyncio (matching the per-handler lazy-import
convention in this module). Centralizes the offload rationale on
SessionEndpointConfig.tenant_check's field docstring.
Adds two regression tests in TestTenantCheckOnReadEndpoints that wire
the real resolve_workstream_owner as tenant_check and force the
storage fall-through path the existing class only stubbed past with
fake allow/deny callables.
* test(api): spy asyncio.to_thread to pin tenant_check offload
Copilot flagged the cold-cache regression tests for asserting the
response shape but not the offload itself: reverting
await asyncio.to_thread(cfg.tenant_check, ...) to the sync call shape
would still leave the storage fall-through working and the tests
green. Patch asyncio.to_thread inside both tests with an async spy
that records every offloaded callable, then assert cold_check is in
the call list — sanity-checked by reverting the history wrap locally
and watching the assertion bite (offloaded only contained
storage.get_workstream + storage.load_messages, missing cold_check).
* feat(coord): inline tool-batch construct replaces approval dock
The pinned bottom approval-dock didn't scale: a 10-call spawn_workstream
fan-out filled the whole pane with a wall of repeated verdict chips,
and the call → approval → result lifecycle was split across three
disconnected surfaces (.msg.tool bubble + dock + .msg.tool result).
Replaces it with one chat-stream construct per dispatch turn that
pairs each tool call with its result and embeds the approval gate:
- .coord-tool-batch--solo single-call serial turn
- .coord-tool-batch--parallel ≥2 calls; rows share a left rail
+ per-row tick so they read as
siblings of one assistant decision
Lifecycle: rows render with optional "judge evaluating…" placeholder,
upgrade in place when intent_verdict arrives, and on tool_result the
output lands paired under the originating row. When the batch needs
approval, one Approve/Deny/Always action row renders inside the
construct (envelope-level — server semantics resolve siblings
together). After approval_resolved the action row morphs into a
✓ approved / ✗ denied status pill that stays as a receipt.
Critical bug closed: when a page reload races a pending approval,
pre-scan tool_call_ids in history; turns whose call_ids have no
matching tool result are rendered pending (not resolved-approved).
The SSE approve_request replay then upgrades the existing batch
in place — drops --approved/--denied, adds --pending, swaps the
status pill for actions, and assigns activeBatch. Without this
the operator was locked out of any approval pending at reload.
Defence-in-depth follow-ups from the same review:
- approval_resolved falls back to a DOM lookup if activeBatch
is null (cross-tab resolution where this tab never set it).
- _appendVerdictLineTo dedupes via a row.dataset.verdictSig so
SSE reconnect storms + repeat intent_verdict events don't
tear down + rebuild an unchanged verdict line.
- judgeVerdicts Map soft-capped at 500 entries (FIFO eviction)
via _cacheJudgeVerdict.
- toolRows entries hold {batch, row} only — the originating
item payload is no longer pinned for the page lifetime.
- _scheduleScroll coalesces messagesEl.scrollTop writes through
requestAnimationFrame so history replay doesn't reflow once
per appended message.
- Rationale <details> now inserts immediately after the verdict
line (was tail-appending, breaking ordering once a result
landed below).
- .coord-tool-batch--error wired: _appendResultToRow lifts a
row's error onto the enclosing batch; _renderBatchRow does
the same for policy-blocked rows at construction.
- _buildStatusPill extracted; both _morphBatchResolved and the
appendToolBatch resolved-replay branch route through it.
Removed: ~248 lines of dead .approval-dock CSS, the dock <aside>
element from index.html, and the dead helpers showApproval's
prior body, hideApproval, claimApprovalFocus,
claimApprovalFocusForVerdict, applyJudgeVerdictToRow,
applyJudgePendingToRow, ensureDctxAfterRow, removeRationale,
setApprovalButtonsDisabled, the appendToolCall single-row wrapper,
and window.coordApprove. Five stale comment blocks referencing
the dock as if live also swept.
Children-tree's renderApprovalBlock is independent and untouched
(different surface, different .approval-block / .approval-pill
vocabulary).
* fix(coord): close four Copilot review gaps on PR 447
Copilot review on caa07e6 flagged four follow-ups:
1. History replay was rendering EVERY orphan tool_calls turn (one
that lacks a matching tool result message) as `pending: true,
judgePending: true`. That paints Approve/Deny on turns that
could be just running — auto-approved-and-still-in-flight, or
already-approved-and-still-in-flight — and clicking would 409
because the call_id isn't in `pending_items`. Add a new
`--running` state for the orphan case (no actions, neutral
accent stripe). SSE then upgrades in place: `--running` →
`--pending` when `approve_request` replays, or `--running` →
`--auto` when `tool_info` replays. Tool_result events still
route into the rows for the third case (already-approved + in
flight) since `toolRows` is populated. Kicker text reads
"Running · Parallel N" while ambiguous, so the operator can
tell the in-flight-replay state apart from a fresh "Parallel ·
N tools" auto-approved batch.
2. Removing the dock also removed its `aria-live="assertive"`
region — pending tool-batches now append into the polite
`#coord-messages` log (which gets flipped to `aria-live="off"`
during streaming), so a screen reader could miss the
action-required signal. Add an off-screen
`aria-live="assertive"` `#coord-sr-announcer` region and route
"Approval required: <name> + N more" through it whenever a
pending batch is created OR an upgrade-in-place promotes a
running batch to pending. Also mark pending batches with
`role="region"` + a matching `aria-label` so SR landmark
navigation surfaces them; both are dropped on resolve so the
resolved batch stops claiming the landmark.
3. `_resolveBatchAction` was selecting the first row whose
`data-call-id` was set and that wasn't `.error` — but
`approve_request` envelopes carry the FULL items list,
including auto-approved siblings whose `needs_approval=false`
means the server's `pending_items` won't recognise their
call_id (→ 409 on submit, or resolves the wrong gate). Tag
rows that are genuinely in `pending_items` with
`data-needs-approval="1"` at construction (and during
upgrade-in-place when SSE arrives), and select against that
selector specifically. Restores the legacy
`pendingApprovalCallId` contract that filtered on
`needs_approval` before the dock was retired.
4. The `.coord-tool-row-result` comment claimed the styles applied
a click-to-expand "collapsed" affordance like the interactive
UI's `.tool-output.collapsed`, but the implementation only set
`max-height: 240px; overflow: auto` (a scroll pane, not a
collapse with expand control). Update the comment to describe
what the rules actually do and explain the deliberate
divergence from interactive (coord is a diagnostic-leaning
read-once surface; an internal scroll pane reads with lower
friction than a click-to-expand control on the operator's
primary monitoring view).
No Python touched; node --check on coordinator.js clean.
* fix(coord): restore reload-time pending approval gate
Agent-Logs-Url: https://github.com/turnstonelabs/turnstone/sessions/30f630fe-3ded-4abe-991b-b5a95f699127
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
* feat(api): expose pending_approval on workstream detail response
PR 447 / 93cb3d9 (Copilot autonomous follow-up) added a JS path that
reads ``wsSnapshot.pending_approval_detail`` off the
``GET /v1/api/workstreams/{ws_id}`` snapshot in coordinator.js
init() so a freshly-loaded chat tab can paint the inline approval
gate immediately at reload, without waiting for the SSE
approve_request replay (which leaves a brief --running flash on the
inflight orphan placeholder).
But the server's ``WorkstreamDetailResponse`` schema only declared
``{ws_id, name, state, user_id, kind}`` and the lifted
``make_detail_handler`` matched: nothing was populating
``pending_approval`` or ``pending_approval_detail`` on the wire.
The frontend block silently no-op'd at runtime; Copilot's
accompanying assertion only grep'd the JS source for the literal
strings, so it stayed green while the actual contract was missing.
Extend the contract to match the Copilot frontend:
- Add ``pending_approval: bool`` + ``pending_approval_detail:
PendingApprovalDetail | None`` to ``WorkstreamDetailResponse``,
same shape as the dashboard / cluster live projection.
- ``make_detail_handler`` reads ``ws.ui._pending_approval`` (only
treats it as live when ``isinstance(_, dict)`` so MagicMock-
based unit tests don't trip the path) and calls
``ui.serialize_pending_approval_detail()`` to fill the detail.
A serializer raise falls back to ``pending_approval=True`` +
``detail=None`` instead of 500ing the whole response — SSE
replay still carries the authoritative payload.
- ``test_returns_workstream_fields`` updated for the two extra
fields (False / None on a MagicMock UI).
- ``test_pending_approval_fields_propagate_from_ui`` is the new
behavioural test: stub a UI with a realistic
``_pending_approval`` dict + serializer return, assert the JSON
surfaces ``pending_approval=True`` + the items list.
- ``test_pending_serializer_failure_falls_back_to_bool_only``
pins the defensive degradation so a future serializer
regression can't 500 every reload.
Tests: 4822 pass (3 deselected live). Ruff + mypy clean.
* fix(coord): three regressions on PR 447 inline tool-batch refactor
Three regressions reported during operator harness shakedown, all
landed by the inline tool-batch refactor in caa07e6:
1. ``stripAnsi`` ReferenceError on every ``tool_result``.
``_appendResultToRow`` called ``stripAnsi(output || "")`` but the
helper only existed in ``ui/static/app.js`` — coord.js never
imported or defined it. The thrown ReferenceError propagated up
through ``appendToolResult``, aborting the SSE handler before
``loadTasksDebounced()`` could fire, AND the result block never
appended to the row, AND history replay's tool-message loop
bailed out at the first orphan-tool-result. Three reported
bugs (tasks pane stops auto-refreshing, tool output missing in
the modal, reload only rebuilds the conversation up to the first
tool result), one root cause.
Fix: hoist a local ``stripAnsi`` mirroring the interactive UI's
regex. Keep it local rather than centralised — coord and
interactive tool-output paths have different rendering
strategies, and the interactive helper isn't on the shared
module surface today.
2. JSON tool output rendered as a single unreadable line. Coord
tool surfaces (``list_nodes``, ``tasks``, ``spawn_workstream``,
...) emit JSON by default, and ``textContent = stripAnsi(raw)``
showed the whole envelope on one line. The parent
``.coord-tool-row-result`` already has ``white-space: pre-wrap``
so a ``JSON.stringify(parsed, null, 2)`` body lays out as
intended without a nested ``<pre>``. Non-JSON / unparseable
output falls through to the raw cleaned string.
3. Header tier badge stuck on ``⚙ heuristic`` after the LLM judge
landed an upgraded verdict. ``_pickBatchTier(items)`` ran once
at batch-creation time; later ``intent_verdict`` SSE events
updated the per-row chip via ``_appendVerdictLineTo`` but never
refreshed the head.
Fix: persist the verdict's tier on ``row.dataset.verdictTier``
(+ ``verdictModel`` when set), add ``_refreshBatchTier(batch)``
that scans the rows and computes the cross-row best tier (LLM
beats heuristic), and call it from ``_appendVerdictLineTo``
whenever a row writes a verdict. ``_pickBatchTier`` gets the
same prefer-LLM scan so the initial render is consistent. The
``intent_verdict`` cache entry tags ``tier: "llm"`` so a late
verdict landing on a previously heuristic-only row escalates
the badge correctly.
No Python touched; node --check on coordinator.js clean.
* fix(coord): close five Copilot review gaps on PR 447
Five distinct findings from the second Copilot pass on the inline
tool-batch refactor (the sixth — stripAnsi ReferenceError — already
shipped in 77dc24e):
1. CSS rail tucks never matched. The ``--first / --last`` row trims
used ``:first-of-type`` / ``:last-of-type``, but the batch
contains other ``<div>`` siblings (.coord-tool-batch-head,
.coord-tool-actions / .coord-tool-status) — the
structural-pseudo-class is type-based (``div``), not class-
based, so the first .coord-tool-row is not the first ``<div>``
in the parent. Selector silently no-op'd, leaving the rail
butting against the inner top/bottom edges of the batch. Fix:
apply explicit ``.coord-tool-row--first`` / ``--last`` markers
in JS at row-build time and key the CSS off them.
2. Upgrade-in-place left stale ``data-needs-approval`` markers on
non-pending sibling rows. The original block only added the
attribute for items where ``needs_approval=true``, never
clearing it for rows whose earlier (replay-time) shell tagged
them. ``_resolveBatchAction`` could then pick a non-pending
row's call_id, yielding a 409 stale call_id on approve / deny.
3. Upgrade-in-place left row-level status pills out of sync with
the SSE-authoritative item shape. When a ``--running`` orphan
gained a ``tool_info`` envelope, the ✓ auto pill never
appeared; when it gained an ``approve_request`` envelope with
policy-blocked siblings, the ✗ blocked pill / ``.error`` class
were missed. Batch-level state classes flipped, but per-row
visual cues lagged.
Fix for 2 + 3: extract ``_refreshRowStatus(row, item)`` from
``_renderBatchRow``. It clears prior ``data-needs-approval`` +
pills and re-applies from the item, preserving runtime
``tool_result`` errors via the new
``.coord-tool-row-result--error`` marker on the result block.
Both ``_renderBatchRow`` (initial render) and the
upgrade-in-place loop now route through it, so the two paths
can't drift.
4. History replay defaulted ``item.needs_approval = true`` on
every synthesized tool call. ``_renderBatchRow`` then tagged
the row with ``data-needs-approval="1"`` regardless of whether
the call genuinely needed approval. Combined with the missing
clear in finding 2, an SSE upgrade with a mixed envelope kept
incorrect markers on auto-approved siblings. Drop the
replay-time default; let SSE supply the authoritative bit when
the upgrade fires (``_refreshRowStatus`` reads it from the
item).
5. Tool result routed into an existing batch row didn't trigger
``_scheduleScroll()``. Result blocks grow ``scrollHeight``;
without the rAF-coalesced scroll the user pinned at the bottom
loses their pin when the row inflates. Add the call after
``_appendResultToRow`` in the early-return path so this branch
matches ``appendMsg``'s pinning behaviour.
Plus comment-only:
6. Detail-handler comment claimed "the JSON omits the section"
when the UI doesn't expose ``serialize_pending_approval_detail``,
but the response always includes both keys (with ``False`` /
``null`` for the bool / detail). Updated to match the actual
shape.
Tests: ``test_workstream_endpoints.TestDetailInteractive`` +
coordinator-detail + page tests pass (14 / 0 failed). Ruff +
mypy clean. ``node --check`` on coordinator.js clean.
* fix(coord): close 17 review findings on PR 447
Second /review pipeline pass surfaced 16 confirmed findings (1 sec
major, 1 bug major, several minor + nit); operator harness shakedown
+ this commit's stale-comment sweep adds one more. All addressed
here.
Security:
sec-1 (major) — make_detail_handler + make_history_handler in
session_routes.py now invoke ``cfg.tenant_check`` after ws_id
validation, matching every other lifted session verb (send /
approve / close / cancel / events / attachments). Pre-fix the
detail response carried 5 low-data fields and history exposed
message rows; PR 447 added pending_approval_detail to detail
(tool previews + LLM judge reasoning) which made cross-tenant
reads via the missing gate a real disclosure on the interactive
surface (coord wires tenant_check=None and is unaffected). Plus
4 new regression tests in TestTenantCheckOnReadEndpoints that
wire a tenant_check function into the test cfg and assert the
gate fires on detail + history.
Bug fixes:
bug-1 (major) — history replay used to render every fully-
resolved tool batch as ``resolved: { approved: true }`` regardless
of the persisted tool result content. A denied tool round-trip
showed the green "✓ approved" pill alongside the persisted
"Denied by user" result text — directly contradictory state. Fix:
pre-scan classifies each tool message via a ``callOutcomes`` Map
by inspecting content prefix ("Denied by user" / "Blocked by
tool policy" / "Error:") and ``m.is_error``. Assistant tool_calls
render ``resolved.approved=false`` when any call's outcome is
"denied"; the existing --running fallback covers orphan turns
(any call lacking an outcome).
bug-2 — _verdictSig joined recommendation/risk_level/confidence/
reasoning only. When a late LLM verdict text-matched the earlier
heuristic verdict, the dedupe early-return fired before the
row's dataset.verdictTier was updated, so _refreshBatchTier
never escalated the header from "⚙ heuristic" to "⚖ llm".
Fix: include verdict.tier and verdict.judge_model in the
signature (with a "\x1f" separator instead of the empty join,
reducing field-boundary collision risk).
bug-3 — history replay's tool-result rendering hardcoded
isError=false. A runtime tool error on reload rendered without
the .error class, --error stripe, or "✗ error:" lead. Fix:
the same callOutcomes pre-scan that drives bug-1's denial path
also classifies "Error:" prefixes; appendToolResult now receives
isError=callOutcomes.get(callId) === "error".
bug-4 — approval_resolved derived ``wasAlways`` exclusively from
this tab's ``batch.dataset.requestedAlways``; cross-tab "Always"
click never propagated to peer tabs' status pill. Fix: server's
resolve_approval now takes a keyword ``always`` arg and includes
it on the SSE event body; client prefers ``ev.always`` and falls
back to the dataset stash for the hot-deploy window where the
SSE event might briefly omit the field.
bug-5 (nit) — appendToolBatch's create-new path overwrote
toolRows entries unconditionally. A partial-mapped envelope
(some call_ids previously seen, some new) silently orphaned the
prior batch's row pointers. Fix: detect the partial overlap,
console.warn, unmap the stale entries before the new batch
claims them.
Performance:
perf-1 — _refreshBatchTier did a querySelectorAll per verdict
insertion; for an N-row batch upgrade this was O(N²) DOM walks.
Coalesce via queueMicrotask + a _tierDirtyBatches Set so a burst
of N verdict updates collapses into ONE tier scan. Synchronous
body extracted to _refreshBatchTierImmediate (called from the
microtask flush).
perf-2 — _appendResultToRow pretty-printed JSON via
JSON.parse + JSON.stringify(parsed, null, 2) on every tool
result with no size cap. A 100KB JSON output stalled the main
thread; 10 parallel tool_result events compounded. Fix: gate
on cleaned.length <= 32 KiB AND a first-char check (0x7B / 0x5B)
so plain text + oversized payloads skip the parse. Parent CSS
is white-space: pre-wrap so raw text still wraps.
Quality:
q-1 — deleted dead row.dataset.funcName write (no readers).
q-2 — extracted _formatTierLabel(llmModel, hasHeuristic) shared
by _pickBatchTier (item-driven) and _refreshBatchTierImmediate
(dataset-driven). Single source of truth for the tier label
literals.
q-3 — extracted _pendingKickerText(items) used by both the
upgrade-in-place and fresh-build paths in appendToolBatch.
q-4 — added string-presence assertions to
test_coordinator_js_exposes_inline_approval_helpers covering
the new tool-batch helpers (appendToolBatch, _morphBatchResolved,
_resolveBatchAction, _refreshBatchTier, _refreshRowStatus), the
--running / --pending state classes, and the callOutcomes
outcome classifier.
q-5 — renamed _announcePolitelyAssertive → _announceAssertive.
Function unconditionally writes into the aria-live="assertive"
region; "politely assertive" was contradictory.
q-6 — rescoped the test docstring to acknowledge it covers two
layers (Chunk 3 children-tree + PR 447 tool-batch).
q-7 — tightened pending_approval_detail: Any → dict[str, Any]
| None in make_detail_handler. Mypy-confirmed.
Plus the third /review pass's q-1 stale-comment sweep:
_resolveBatchAction's comment still claimed the server doesn't
echo ``always`` on approval_resolved — wrong post-bug-4-fix.
Updated to reflect that the dataset stash is now backward-compat
fallback only, not the primary source.
Tests: 4826 pass (+4 new from TestTenantCheckOnReadEndpoints, plus
expanded assertions in TestDetailInteractive). Ruff + mypy clean.
``node --check`` on coordinator.js clean.
Verifier confirmed all 16 findings; pass-3 /review on the
addressing-commit surfaced only 0 critical / 0 major / 2 minor /
2 nit, none blocking. The two pass-3 minor findings are
pre-existing patterns across all lifted verbs (sync tenant_check
inside async handlers) and best addressed in a dedicated follow-up
PR auditing the whole lifted-verb surface.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
The system-message developer block was rebuilt every turn with two
unstable inputs: minute-precision current_datetime in the middle of
the composed prefix, and _pending_nudge entries appended-then-cleared
at the bottom. Both invalidated prompt-cache reuse on Anthropic /
OpenAI for the entire prefix, every turn.
current_datetime now rounds to the top of the hour. Nudges no longer
ride on the system message at all — they drain through two channels:
- tool_error and repeat ride the existing tool-result <system-reminder>
envelope via a new MetacognitiveAdvisory ToolAdvisory subtype, drained
in _collect_advisories alongside GuardAdvisory and UserInterjection.
- correction, denial, resume, start, completion splice as
<system-reminder> blocks at the trailing edge of the next user
message via a new _splice_pending_user_advisories helper.
User content passes through escape_wrapper_tags before concatenation
so a user typing literal <system-reminder> tags cannot fabricate an
envelope; the same escape now runs on advisory.render() output inside
wrap_tool_result for defense-in-depth across all advisory types.
Cancel handlers (GenerationCancelled / KeyboardInterrupt / bare
Exception) now clear _pending_tool_advisories alongside the existing
_flush_queued_messages so a queued nudge from an aborted batch
cannot leak into the next generation.
Visibility ping ([metacognition: nudge injected — ...]) preserved at
both new attach points via a single _emit_nudge_ping helper.
Also adds a "Session kind" line (interactive | coordinator) to the
composed Session Context so the model can see which manager hosts
its session.
Tests: 4847 passing (+7 new in TestMetacognitiveBuffers and
test_tool_advisory). ruff + mypy clean.
Copilot review on c5fe3e7 flagged three follow-ups:
1. tasks-write batch order was scheduler-dependent. Prior comment
claimed the result was "deterministic against the input set even
if the dispatch order isn't" — true for the SET of tasks, but
``tasks_add`` appends under a per-ws lock, so the FINAL list
ordering (and order-derived timestamps) varied with whichever
thread happened to acquire the lock first. Fix: when a batch
contains any tasks-write, the dispatcher runs the WHOLE batch
serially in input order. Other batches stay parallel.
2. ``tasks_add`` test stubs were the wrong shape. ``CoordinatorClient.
tasks_add()`` returns the task dict directly with top-level
``id`` / ``title`` / ``status`` / ``child_ws_id`` / ``created`` /
``updated`` — the previous stubs wrapped it as ``{"ok": True,
"task": {...}}`` and weakened the tests since
``_exec_tasks``'s summary path reads ``result.get("id")`` and
would have seen ``"?"`` against the wrong shape. Both stubs
updated to match the real contract.
3. New regression test pins the input-order property on tasks-write
batches. ``test_tasks_writes_run_in_input_order`` captures the
``tasks_add`` call sequence and asserts it matches the model's
emit order exactly — pre-fix this would be scheduler-dependent.
Plus ``test_tasks_writes_serial_when_mixed_with_non_tasks_siblings``
pins the same property when the batch interleaves a
``list_nodes`` call with two ``tasks(add)`` calls.
Tests: 4820 pass (+2 net since the prior PR 446 push). Ruff +
mypy clean.
Operator's harness shakedown found list_nodes filters silently
ignored on every call:
list_nodes(os="Linux") → returns ALL 10 nodes
list_nodes(has_gpu=true) → returns ALL 10 nodes
list_nodes(memory_gb=751) → returns ALL 10 nodes (no
node has 751 GiB; should be 0)
Storage filter pipeline is fine (pinned by an existing
test_list_nodes_filter_uses_natural_value_not_quoted). The bug is
upstream in ``_prepare_list_nodes``: it only honoured
``args["filters"]`` (the canonical nested shape). Several models
drop the nesting and emit each filter as a top-level kwarg —
``list_nodes(os="Linux")`` instead of ``list_nodes(filters={"os":
"Linux"})`` — and the strict prepare silently degraded those calls
to "no filter" → full-cluster return.
Fix: any top-level kwarg that ISN'T one of the four reserved control
parameters (``filters``, ``limit``, ``include_network_detail``,
``include_inactive``) is now treated as a flat filter. Nested entries
still win on key collision so the canonical shape stays
deterministic. Tool description unchanged so well-behaved models
keep using ``filters={...}``; the relaxation is purely receiver-side.
Tests: 4818 pass (+5 net). Five new tests pin both shapes plus the
collision-precedence rule and the prepare→exec wiring. Ruff + mypy
clean.
Operator observed the prior rule rejecting a natural decompose-the-
plan turn:
[tasks(add×4), list_nodes, list_skills, list_workstreams]
The 4 tasks(add) calls landed (per-ws lock serialised them) but the
guard blanket-rejected EVERY tasks(...) regardless of what its
siblings actually were. All-write batches converge under the
per-ws lock; all-read batches can't race. The only genuinely-
hazardous shape is the read+write mix where tasks(list)
paralleled with tasks(add=...) inside ``run_one``'s
ThreadPoolExecutor has unspecified ordering and the read can land
on either side of the write.
The rule now scopes precisely:
- All ``tasks`` writes in a batch — permitted.
- All ``tasks`` reads in a batch — permitted.
- ``tasks`` paralleled with non-``tasks`` siblings — permitted
in either direction. Non-tasks tools don't touch the tasks
state, so there's no read-after-write surface.
- ``tasks`` read AND ``tasks`` write in the same batch — REJECTED
(still, because that IS the actual hazard).
Tests: 4813 pass (+4 net). Six new tests pin the relaxation
(all-write OK, all-read OK, write+sibling OK, read+sibling OK,
non-tasks-only batch unaffected) and the one tightened rejection
case (read+write mixed in tasks specifically). Ruff + mypy clean.
* feat(node): auto-detect node capabilities via kernel interfaces
Closes the operator-burden gap the harness shakedown surfaced — the
list_nodes capability/region/role filtering surface that nodes were
launching with empty. Auto-detection runs at server startup and
populates ``node_metadata`` rows with sensible defaults that
operators can still override via the ``[metadata]`` section of
config.toml (operator-config writes win on the per-key upsert).
What's detected, all from kernel interfaces (no userspace binaries
on PATH — works the same way regardless of whether nvidia-smi /
rocm-smi / lspci is installed):
- ``gpu_count`` / ``gpu_vendor`` / ``gpu_vendors`` / ``gpus`` —
walks ``/sys/class/drm/cardN/device/{vendor,device}`` and decodes
PCI vendor IDs to friendly names (NVIDIA / AMD / Intel / Apple).
Heterogeneous-GPU nodes get the first KNOWN vendor in the flat
``gpu_vendor`` key — never ``"unknown"`` when known vendors are
present — so a coord filtering on ``gpu_vendor=nvidia`` matches
nodes whose first card happened to be exotic.
- ``memory_gb`` — reads ``/proc/meminfo``, rounds GiB down so
``filters={"memory_gb": 32}`` doesn't match a 31.5 GiB node.
- ``cpu_model`` — first ``model name`` line from ``/proc/cpuinfo``.
- ``cloud_provider`` / ``cloud_region`` / ``cloud_zone`` /
``cloud_instance_type`` / ``cloud_instance_id`` — DMI sysfs
identifies the cloud provider from BIOS/SMBIOS strings (no
network call) and only THEN does the IMDS probe fire. Baremetal
hosts pay zero startup latency on the cloud path.
Hardening highlights:
- IMDS probes target the link-local IP literal ``169.254.169.254``
for AWS, GCP, AND Azure — no DNS-resolvable hostname for any
vendor, so a host with attacker-controlled DNS can't redirect
the probe even when its DMI claims a cloud provider.
- Response bodies capped at 64 KiB on read; per-field strings
capped at 256 chars and stripped of control characters before
persistence. Stops a hostile IMDS responder from spraying
multi-megabyte / newline-injected payloads into ``node_metadata``
and from there into coord-LLM ``list_nodes`` context.
- ``isinstance(doc, dict)`` guards on every JSON IMDS response so
a non-conformant body (list / scalar / null) returns clean ``{}``
instead of raising.
- ``collect_node_info()`` runs via ``asyncio.to_thread`` from the
server's lifespan handler so the IMDS probe latency never blocks
the event loop.
- GCP fans the three zone/machine-type/id probes concurrently so a
misidentified host's worst case is one timeout window (~1 s)
instead of three (~3 s).
- Operator opt-out via ``TURNSTONE_AUTO_CLOUD_METADATA=0`` skips the
IMDS phase entirely; the DMI-derived ``cloud_provider`` still
populates because that's a kernel interface.
Tests: 4807 pass (+11 net, 73 in test_node_info.py). Ruff + mypy
clean on every modified file. New tests pin the heterogeneous-GPU
flat-key fix, the IMDS hardening (non-dict JSON, control-char
sanitisation, body cap, per-field cap), and the GCP IP-literal
property.
* fix(node): filter synthetic display adapters + per-vendor GPU flags
PR review on 68cb0ab flagged two real issues with the GPU surface:
1. Hyper-V synthetic display adapter (vendor 0x1414, device 0x06)
registers a /sys/class/drm/cardN entry on Linux but is NOT a
compute GPU. A CI runner reproduced this and came back with
gpu_count=1 on a CPU-only VM. Same hazard for AWS Nitro VGA,
QEMU virtio-gpu, and any other hypervisor synthetic display
adapter. Fix: ``_detect_gpus`` filters DRM cards by PCI vendor
against the GPU allow-list (NVIDIA / AMD / Intel / Apple); cards
from other vendors are skipped entirely. Operators with exotic
accelerators that don't match any known vendor can still set
``gpu_count`` + the relevant flags via [metadata] config.
2. ``gpu_vendor`` (singular flat key) was sorted-alphabetical-first
of the unique known vendors. list_nodes() filtering does
exact-equality JSON matching, so a mixed AMD+NVIDIA node ended
up with ``gpu_vendor=amd`` and was invisible to a coord
filtering ``gpu_vendor=nvidia``. Fix: drop the singular key
entirely; emit per-vendor booleans (``gpu_has_nvidia=true``,
``gpu_has_amd=true``) so multi-vendor nodes match EITHER vendor.
Also add ``has_gpu=true`` for "any compute GPU at all" filtering.
Tests: 4809 pass (+2 net). New / updated tests pin both behaviors:
- _detect_gpus: Hyper-V synthetic + arbitrary unknown-vendor card
now filter out; mixed-known-and-unknown keeps only the known card.
- collect_node_info integration: multi-vendor node has both
``gpu_has_amd`` and ``gpu_has_nvidia`` set; ``gpu_vendor`` (singular)
is asserted absent so a future regression that re-introduces it
fails loudly.
Ruff + mypy clean.
* fix(coord): close gaps an operator's harness shakedown surfaced
Operator-driven shakedown of the coordinator tool surface flagged
five issues; this commit addresses all of them plus the review
findings against the initial fix.
1. Cancelled-mid-stream partial assistant content now carries a
"[generation cancelled before completion]" marker. Without it,
``inspect_workstream`` / ``wait_for_workstream`` callers and the
next coord-LLM turn read the truncated text as a complete answer.
``_cancelled_partial_msg`` no longer ships ``_provider_content``
(Anthropic would otherwise read that lane verbatim and bypass the
marker; partial tool_use blocks could also leak through).
2. ``spawn_workstream`` / ``spawn_batch`` no longer surface the
routing-proxy ``status`` field (always HTTP 200 on the success
path). The tool description claimed it was "lifecycle state at
creation"; code that did ``if result["status"] == "idle"``
silently never matched. Lifecycle state lives on the workstream
row — ``inspect_workstream`` is the read. Tool JSON descriptions
plus docs/coordinator-skills.md and docs/bulk-endpoints.md
examples updated to match.
3. ``inspect_workstream`` not-found error string is bare ("workstream
not found"); the structured ``ws_id`` field carries the queried
id. Pre-fix the error STRING echoed the id back at the caller
who just sent it — redundant and out of step with the rest of the
surface. Cross-tenant + missing rows still return the same shape,
preserving the existence-leak guarantee.
4. ``tasks(...)`` is now rejected when called in a parallel tool
batch. The prior shape relied on a docstring warning ("a list
paralleled with writes can reflect pre-write state") that put
cognitive overhead on every model invocation; turning the silent
footgun into an explicit error means the model only thinks about
the rule the moment it actually breaks it. Warning dropped from
the tasks tool description. ``_PARALLEL_INCOMPATIBLE_TOOLS``
constant in session.py is the extension point for any future
tool with the same read-after-write hazard.
Plus the multi-stage code review's findings against the initial
fix (q-1 / q-2 docs drift, q-3 idiom, q-4 keys-assertion, q-5
duplicate guard) — all addressed in the same pass.
Tests: 4752 pass, +6 net since the pre-fix baseline. Ruff + mypy
clean. Three new tests pin the parallel-batch-rejection behaviour
on tasks (rejected when batched, runs alone, sibling tools
unaffected); existing cancel + spawn + inspect tests updated to
match the new shape.
* fix(coord): close two copilot review gaps on PR 444
Copilot review on PR 444 flagged two follow-ups:
1. Empty-content cancel divergence — when ``GenerationCancelled``
races BEFORE the first content token, the prior shape skipped
``save_message`` and only appended an empty-content msg in
memory. In-memory and storage diverged: a rehydrate would see
nothing in storage but the session would carry an empty
assistant turn. Both branches now persist; on the empty-content
shape the marker becomes the entire message
("[generation cancelled before completion]") so storage matches
the in-memory history.
2. Test stub cleanup — three new tests injected ``ui.approve_tools``
via ad-hoc ``lambda + type: ignore[attr-defined]``. Replaced
with a permissive ``approve_tools`` method on ``_StubUI`` so the
stub matches the SessionUI surface the dispatcher actually
reads. Tests that exercise approval pathways can still override
per-instance.
Tests: 4752 pass. Ruff + mypy clean.
* feat(coord): surface child errors, isolate tool exceptions, add memory tool
Closes four coordinator gaps identified during operator triage:
1. Child workstream errors now surface in inspect/wait. Worker-thread
exception text is sanitized (URL userinfo masked, sk-/Bearer/ghp_/
github_pat_/AKIA tokens redacted, capped at 1024 chars) and persisted
to workstream_config.last_error before _emit_state("error") fires, so
coord polling never sees state=error with a missing cause. The row
is cleared on recovery transitions (idle/running) so a once-leaked
exception body doesn't outlive the failure. inspect_workstream and
wait_for_workstream return last_error for state=error rows; the
wait surface prefers it over the assistant-tail walk.
2. Tool exceptions now return as tool_results with sibling-aware
guidance. ChatSession._safe_prepare_tool wraps every per-call
_prepare_tool invocation; a buggy preparer becomes an error item
for that call only — sibling parallel tool_calls keep going,
never orphaning the assistant message's tool_calls block.
run_one's runtime exception path includes the exception class
and a short note that other tool calls in the batch completed
independently so the model can recover.
3. Memory tool exposed to coordinator with a coord-only scope.
memory.json gains coordinator: true + interactive: true + per-kind
kind_variants. Coord sessions see scope enum ["coordinator"] and
an orchestration-flavored description; IC sessions see ["global",
"workstream", "user"] and the existing flavor. Coord-scope rows
are private to the coordinator session (children cannot read or
write them), closing the cross-session prompt-injection lane that
an adversarially-steered child would otherwise have. Coord
visibility is also restricted to coord-scope only — coords no
longer see global / workstream / user memories that belong to the
user's interactive sessions.
4. Per-call exception isolation in tool batches. _safe_prepare_tool
was previously the implicit shield; now it's an explicit method
with documented invariants. KeyboardInterrupt / GenerationCancelled
re-raise so the cooperative cancel path still works.
Other notable changes:
- LAST_ERROR_CONFIG_KEY + persist_last_error / clear_last_error /
load_last_error / sanitize_error_text moved to turnstone.core.memory
(the storage facade hub) — readers in coordinator_client.py import
the constant.
- Memory scope tuples extracted to module constants
_VALID_MEMORY_SCOPES and _IMPLICIT_SCOPE_WALK; seven inline
duplicates collapsed.
- tools.py grows _apply_kind_variant for the per-kind tool surface;
tools without kind_variants pass through unchanged (no spurious
deep-copies).
- Session adds _coordinator_scope_id, _default_memory_scope,
_implicit_scope_walk, and _record_fatal_error chokepoints so the
worker-thread fatal path is one site rather than three.
- Removed duplicate on_error / on_state_change emits from
session_routes.py and coordinator_adapter.py — session.send()'s
_record_fatal_error owns the sequence now.
Tests: 4742 pass (no live), +30 net since the baseline. Ruff + mypy
clean on every modified production file.
* fix(coord): redact secrets in tool error paths via output_guard
Copilot review flagged two paths where ``str(exc)`` flowed back into
the model-facing tool_result without going through the credential-
redaction the new fatal-error path applies:
- ``ChatSession._safe_prepare_tool``: a preparer-side exception
becomes an error item whose ``error`` field embedded the raw
exception text.
- ``ChatSession._execute_tools.run_one``: a runtime tool exception
became an ``Error executing X: <e>`` tool_result, again with
the raw exception text.
Both now route through ``sanitize_error_text`` (sanitised log line +
sanitised tool_result), and ``sanitize_error_text`` itself was
refactored to delegate to ``output_guard.redact_credentials`` instead
of carrying its own parallel regex catalog — the audit log + post-tool
guard already use that pattern set, so the credential definition
stays in one place.
Also extended ``_RE_CONNECTION_STRING`` in ``output_guard`` to cover
``http(s)://user:pass@host`` so a misconfigured ``OPENAI_BASE_URL``
that lands in an httpx ``ConnectError.__str__`` is redacted by every
caller of ``redact_credentials`` (audit details, close-reason
persistence, last_error, the two tool error paths). The
host (useful for triage) survives; only the password is replaced
with the standard ``[REDACTED:password]`` marker.
Tests: full suite (4745 pass), ruff + mypy clean. Two new tests pin
the redaction behaviour in both tool error paths so a future refactor
can't drift back to leaking ``str(exc)`` verbatim.
Bring the coord dashboard toward parity with the interactive pane on
two operator-visible surfaces:
- Status bar pinned above the composer. Same four cells as the
interactive pane (model, token / context-window usage with effort
suffix, tool calls this turn, conversation turn) driven by the
same on_status SSE events. ws-status-bar CSS hoisted from
ui/static/style.css to shared_static/chat.css so both UIs read one
copy. StatusBar.paint helper extracted to
shared_static/status_bar.js; both Pane.prototype.updateStatus and
the new coord updateStatusBar delegate to it so warn/danger
thresholds, prefix glyphs, and effort-suffix rules can't drift.
CTX_WARN_PCT / CTX_DANGER_PCT now named constants on a single line.
- _coord_events_replay now yields the connected + status preamble
via a shared session_replay_preamble helper in
turnstone/core/session_replay.py. _interactive_events_replay
routes through the same helper so a future field add lands once.
Coord still skips conversation history in the SSE replay (the
dashboard fetches it via GET /history); only the status preamble
is shared.
- History replay reconstructs tool calls. Pre-fix, an assistant
turn that only dispatched tools rendered as an empty bubble
followed by raw tool-result text — the call's intent and
parameters were lost on reload. synthesizeHistoricalToolCall
builds an appendToolCall-shaped item from the persisted
function.name + function.arguments (special-casing bash so the
shell line shows in the header). Tool result rows now resolve
their label from the matching tool_call_id instead of always
printing "tool".
- onopen restores the tokens placeholder when no prior status was
seen, so a transient SSE blip on a fresh coord doesn't leave the
dim "Reconnecting…" copy stuck until the next live tick.
Tests: 4 new tests for the shared replay preamble (connected first,
status only when last_usage present, status payload shape, no-session
fallthrough); existing approval/verdict ordering tests refactored
through a shared make_replay_mocks helper in tests/_replay_helpers.py
that both interactive and coord suites import.
tasks(update) is the only mutation that allows title to be omitted,
so _prepare_tasks stores ``item["title"] = None`` for an update that
only changes status/child_ws_id. _evaluate_intent then projected via
``it.get("title", "")[:100]`` — but dict.get returns the stored None
(the default kicks in only when the key is absent), and the slice
crashed with ``TypeError: 'NoneType' object is not subscriptable``.
The exception fired before any tool in the parallel batch executed,
so the assistant's tool-call message was already on the wire while
no tool-result entries followed. Reconstruction/sanitisation later
synthesised "Tool execution was cancelled" for every sibling — the
visible symptom that masked the real None-slice failure.
- Switch tasks/notify/task_agent/plan_agent/spawn_workstream/
spawn_batch/send_to_workstream/close_workstream/close_all_children
projections to ``(it.get(x) or "")[:N]`` so absent and explicit-None
both fall back to the empty string. The other tools weren't
observed crashing, but the bug shape is identical at every site;
hardening the projection layer once costs one extra ``or`` per line
and removes the foot-gun for any future preparer that stores None.
- Regression tests reproduce the original TypeError on
``tasks(update)`` without title both standalone and in a parallel
batch alongside ``tasks(add)``.
* feat(console): per-call model + judge_model on coord composer
Brings the landing-page coordinator composer toward parity with the
interactive new-ws modal — operators can now pick a model and judge
model per session without round-tripping through the Models admin tab.
- Add Model + Judge Model selects to the home composer's options
panel, populated from /v1/api/models. Empty / non-string fields
collapse to None so the factory falls back to ConfigStore defaults
(coordinator.model_alias, judge.model).
- _coord_create_build_kwargs threads the body fields onto mgr.create.
- Console session factory accepts judge_model and overrides the
JudgeConfig via dataclasses.replace, mirroring the server-side
interactive factory's pattern (alias preserved for IntentJudge's
provider/client resolution).
- Sanitise the 503 factory-misconfig response across make_open_handler,
make_create_handler, and make_detail_handler: a new
_safe_factory_misconfig_message helper strips control characters
and caps at 200 chars before echoing exc text. Operators still get
the full alias in the warning log; clients see a bounded printable
string. Defends the user-controlled body["model"] reflection
surface on the create path.
- _build_mgr_with_factory test helper extracted from _build_mgr so
tests that need to capture factory kwargs don't reconstruct the
CoordinatorAdapter + SessionManager scaffolding inline.
- Tests cover: passthrough of model + judge_model, empty / whitespace
/ non-string body fields collapsing to None, and the 503 sanitiser
truncating + scrubbing a hostile alias payload.
* fixup: address PR #440 Copilot review
- _safe_factory_misconfig_message: hard-cap return at
_FACTORY_MISCONFIG_MAX_LEN total (was MAX_LEN+1 because the slice
was MAX_LEN long with the ellipsis appended on top). Reserve one
codepoint for the ellipsis so the cap is honoured. Update the
regression test to assert the tighter bound.
- Composer judge_model placeholder: "Default (agent model)" was
misleading when ConfigStore judge.model is set — the actual fallback
is judge.model when set, IntentJudge's agent-model fallback when
not. Use "Default judge model" instead so the label matches both
configs.
- Drop the duplicate "N nodes · M workstreams" header span — same data is
already on the page.
- Drop the "+ new" workstream header button + modal; the coordinator
composer is now the primary entry point on the landing page.
- Always render the NODES list inline; remove the cluster-summary
compact toggle since the list already self-collapses same-prefix
nodes into groups.
- Replace the meta node-detail page (#view-node) with direct navigation
to /node/{node_id}/. Removes drillDownToNode, loadNodeDetail,
_loadNodeMetadataPanel, the popstate "node" branch, and the
currentNodeId/currentServerUrl state.
- popstate now falls back to showHome() for unknown state shapes so a
back-nav from a tab on an older build doesn't no-op.
- test_index_landing_surfaces guards the removed IDs from
reintroduction.
* feat(coord): composer parity with interactive — stop/queue/attach
Bring the coordinator one-pane UI to feature parity with the
interactive composer: in-composer Stop button replaces Send during a
turn, queue-while-busy with !!! priority + dismiss, paperclip attach
+ drag/drop/paste. The coord backend already supported all three
(lifted send/cancel/attachment handlers, emit_message_queued=True,
supports_attachments=True); this wires the UI through.
Backend:
- Wire make_dequeue_handler(coord_endpoint_config) so DELETE
/v1/api/workstreams/{ws_id}/send works for coord-kind workstreams.
- Add the matching OpenAPI EndpointSpec.
- Five new test_dequeue_* tests (success, not_found, missing msg_id,
unknown ws, scope gate) pin the URL/method/scope contract.
Frontend extraction:
- New shared modules composer_attachments.js (createAttachmentController)
and composer_queue.js (createQueueController) replace ~300 LOC of
pre-existing duplication between the interactive Pane and the coord
IIFE. Both panes now share one source of truth for the chip pipeline,
optimistic queue bubble, and busy-edge promote sweep.
Coordinator pane:
- Composer constructor adds attachments/stopBtn/queueWhileBusy/
busyPlaceholder/dragDrop options.
- setBusy now drives off SSE state_change (running/thinking/attention →
busy; idle/error → idle), with composer.setBusy unconditional and the
edge-only work (timer cleanup + queue.onIdleEdge) gated on the actual
transition.
- Cancel uses the in-composer Stop with a 2s "Force Stop" affordance +
10s safety auto-recover; the legacy header-mounted #coord-cancel-btn
is removed.
- coordCloseSession suspends SSE before close and re-establishes it on
any failure path so the UI never goes dark on a still-alive session.
- Race handling: bind() releases the queued slot server-side when the
bubble was already dismissed or promoted; rehydrate re-checks getWsId
in its .then so a stale-tab response can't clobber the new tab's
chips.
Interactive pane:
- Pane class adopts the same controllers via this.attachments /
this.queue. Pane.prototype.uploadAttachment, _renderAttachmentChip,
_swapPlaceholderChip, _removeAttachmentChip, removeAttachment,
rehydrateAttachments wrapper, addQueuedMessage, _dequeueMessage, and
_promoteQueuedMessages are all gone — the controllers own the state.
- setBusy collapses to the same shape as coord: composer.setBusy +
edge calc + queue.onIdleEdge on idle.
CSS:
- Move .msg-queued / .queued-badge / .queued-dismiss styles from
ui/static/style.css into shared_static/chat.css so both panes share
one rendering.
- Add .coord-drop-target overlay rule so the coord pane shows the
drag-and-drop affordance.
Tests pass: 160 in the impacted suites (coord endpoints + attachments
+ session routes), including 5 new dequeue tests for coord.
* fix(coord): Copilot review + lint follow-ups
Lint:
- ruff: cast(MagicMock, ...) → cast("MagicMock", ...) under
``from __future__ import annotations`` (UP037).
Copilot review (PR #438):
- composer_queue _sendDelete now invokes onAfterDequeue on success
so a bind() race-DELETE (queued bubble dismissed pre-bind or
promote sweep raced ahead) still rehydrates the caller's chip pile;
released attachment reservations no longer linger invisibly until
the next page load.
- Coord's createQueueController gains onAfterDequeue: attachments.
rehydrate(). The previous omission was a v2 review carry-over from
before coord supported attachments — now it does, so the same
contract as interactive applies.
- Both panes' send-response handler now accepts status:queued without
a queuedEl (SSE-not-yet-connected race on initial load): flips busy
so subsequent sends queue correctly. The current message keeps its
optimistic user bubble — accepted UX gap (no in-UI dismiss for
THIS message) since flipping a rendered user bubble into a queued
one mid-stream would be jarring.
- Doc updates: chat.css comment + composer_queue.js module docstring
refer to the renamed onIdleEdge() instead of the removed
promote()/promoteQueuedMessages.
Cluster nodes were OOM-killing under MCP child-process load with the
old 384M/0.5cpu budget chosen for a leaner, pre-MCP turnstone. Bump
each cluster server to 4G/4cpu and postgres to 4G/4cpu. The single-node
server, console, and channel services remain uncapped.
Four themes from a coordinator-feature shakedown:
1. Correctness fixes (return shapes / examples / behavior)
- tools_coordinator.md: drop fake skill names from spawn examples;
fix wrong kwarg ``node_id=`` → ``target_node=``.
- wait_for_workstream.json: document ``message`` + ``truncated``
per-ws fields (always enriched in the client; the JSON shape
lagged the docstring).
- cancel_workstream.json: document the conditional ``dropped``
payload — ``was_running`` always present when ``dropped`` is,
``pending_approval`` and ``queued_messages`` conditional sub-shapes.
- spawn_workstream.json: document full return shape including
``routing_strategy ∈ {rendezvous, target_node, resume}`` and
``status``.
- close_all_children.json: clarify ``skipped`` covers BOTH
hard-deleted children AND already-closed-and-evicted children
(wire shape doesn't distinguish); drop incorrect "echoed back
in response" claim — server returns ``{status, closed, failed,
skipped}``, never echoes ``reason``.
- console/server.py: comment in ``_fanout_on_children`` clarifying
that the 400 "No session" branch fires for cancel-cascade
callers and is unreachable from close_all_children (close
handler 404s instead).
- coordinator_client._utc_now_iso(): switch to bare ISO format
matching the rest of the storage row format used in the codebase.
2. Tightened the 11 longest tool descriptions (~23% cut on the
coord set). Removed ALL-CAPS emphasis, normalised em-dashes,
dropped informal phrasing. No new claims.
3. Removed static approval annotations from descriptions.
Approval is governed at runtime by the unified ``approve_tools``
body and admin-defined ``tool_policies`` (#436); static
"Auto-approved" / "Approval required" / per-action approval
tags become a stale signal. Field names (``pending_approval``)
and operational verb behaviour ("cancel unblocks pending
approvals") stay.
4. Renamed ``task_list`` coord tool → ``tasks``. The previous name
compounded the bare word ``task`` (which collides with chat-template
channels on local models — same reason ``task_agent`` carries
the suffix); the plural form sidesteps the collision and reads
more accurately, since the tool acts on the whole list rather
than a single task. Sweep covers tool JSON, Python methods (5
client methods + 2 session methods + 1 helper + 1 constant),
audit event name (``task_list.update`` → ``tasks.update``), log
tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``),
frontend SSE event matcher, prompts, docs, and tests. CHANGELOG
entry added.
Plus: dropped the ENV block (Output Environment / Available
rendering / Formatting principles) from coordinator system
prompts. Coordinators orchestrate rather than render rich output
to the user, so the rendering capability matrix is not actionable
for them. Coord prompt drops ~29% (6309 → 4493 chars).
SDK regeneration via ``generate-types.py`` updates both
``openapi-console.json`` (the rename's downstream change) and
``openapi-server.json`` (PR #436 drift — its merge added
``pending_approval_detail`` + ``recent_auto_approvals`` fields to
the Python schemas but didn't regenerate the JSON artifact).
## Behavior changes (operator-visible)
- Audit event name: ``task_list.update`` → ``tasks.update``.
Audit dashboards / SIEM filters / log greps that pinned the old
prefix should update.
- SSE ``tool_result`` events now ship ``name="tasks"`` for the
scratchpad tool. The bundled coord-tree UI is updated atomically;
external consumers reading SSE events by tool name need to update.
- Existing task envelopes in production storage have ``+00:00``
timestamps from the old ``_utc_now_iso``. New writes are bare;
old rows are not backfilled. Within an envelope you may briefly
see mixed formats until each row is re-touched. No code path
string-compares timestamps within an envelope, so this is
cosmetic.
## Validation
- ``ruff check`` + ``ruff format --check`` clean
- ``mypy turnstone/`` clean (175 source files)
- ``pytest -m "not live"`` — 4679 passed, 3 deselected
* refactor(core): unify approve_tools across kinds + judge visibility + perf
Lift WebUI.approve_tools to SessionUIBase so both interactive and
coordinator workstreams run the same body. The shared body now owns
tool-policy gating, per-tool auto-approve, blanket carve-out for
__budget_override__, activity tagging, heuristic-verdict persistence,
and the approve_request/approval_event blocking pattern. Subclass
hooks layer kind-specific surfaces on top.
This closes the drift the LLM-judge audit flagged on coord — the
judge (heuristic + LLM tier) now sees actual tool args for every
coord tool call instead of empty func_args. spawn_batch projects
the full children list so a malicious mid-batch entry is no longer
hidden.
= Unification core =
- SessionUIBase.approve_tools: lifted body covering policy / per-tool
auto-approve / blanket / activity tagging / heuristic-verdict
persistence / approval gate
- _APPROVAL_WAIT_TIMEOUT class constant + _record_judge_metric hook
- WebUI.approve_tools deleted; _record_judge_metric override fires
per-node MetricsCollector.record_judge_verdict
- ConsoleCoordinatorUI.approve_tools deleted; _record_judge_metric
+ on_intent_verdict overrides fire ConsoleMetrics.record_judge_verdict
- ConsoleMetrics.record_judge_verdict + turnstone_judge_verdicts_total
in /metrics text output (cluster PromQL rolls coord+interactive up
uniformly)
- _console_metrics class attribute wired in console lifespan
- Frontend: coord SSE event tools_auto_approved -> tool_info for parity
= Judge args visibility =
- _evaluate_intent populates func_args for all coord tools that hit
approval (spawn_workstream / spawn_batch / send_to_workstream /
close_workstream / close_all_children / cancel_workstream /
delete_workstream / task_list)
- spawn_batch projects every child's skill / initial_message[:200] /
target_node so the judge sees the full fan-out (was first child only)
- fire_judge_verdict_metric helper collapses 4 sites of identical
record_judge_verdict shape across WebUI + ConsoleCoordinatorUI
= Hardening =
- __budget_override__ carve-out reads from pre-filter items list, not
post-filter pending; policy block skips matching the synthetic
name entirely so a wildcard `*: allow` cannot strip the override
before the gate sees it
- _persist_intent_verdict default_tier parameter so heuristic + llm
paths share the storage write helper
= Performance =
- TTL cache on list_tool_policies in turnstone/core/policy.py
(60s, keyed by org_id, lock-free hits)
- Storage-layer invalidation: create/update/delete_tool_policy on
both SQLite and PostgreSQL backends call invalidate_policy_cache
(covers admin-API path + direct test fixtures + any future caller)
- Admin-API handlers also call invalidate_policy_cache as
defense-in-depth
- storage.create_intent_verdicts_bulk on both backends: one
multi-row INSERT + one commit instead of N round-trips. approve_tools
switches to the bulk path so a fan-out turn no longer pays N x commit
before the approval prompt enqueues
- _persist_intent_verdicts_bulk helper on SessionUIBase
= Test coverage =
- tests/test_coord_ui_approve_tools.py (NEW, 17 cases): inheritance
regression, tool-policy deny/allow/mixed on coord, heuristic verdict
persistence (bulk path), activity tagging on auto-approve and pending,
judge_pending dynamic flag (true + false), event-name parity,
per-tool auto-approve, __budget_override__ carve-out under blanket
+ wildcard policy, _record_judge_metric wired/unwired, on_intent_verdict
llm-tier metric
- tests/test_console_metrics.py: 3 cases for the new
record_judge_verdict counter
- tests/test_judge_storage.py: 3 cases for create_intent_verdicts_bulk
- tests/test_coordinator_tools.py: 3 cases pinning the spawn_batch
full-children projection (truncation, mid-batch visibility, empty
defensive)
- tests/conftest.py: autouse _clear_policy_cache fixture so the
process-level cache doesn't leak between tests with distinct storage
instances
= Drift fixes (review feedback) =
- Refresh stale "no-op on coord" comments now that coord overrides
the hook
- WebUI.on_plan_review timeout uses self._APPROVAL_WAIT_TIMEOUT
instead of literal 3600
- Drop redundant bool() wrapper around any() in judge_pending
- Rephrase broken docstring grammar in _coord_spawn_metrics
- Hoist redundant get_storage import out of approve_tools per-item loop
(folded into _persist_intent_verdicts_bulk helper)
= Validation =
- pytest -m "not live": 4679 passed, 3 deselected
- ruff check + ruff format: clean
- mypy: no issues in 175 source files
* fix(approval): apply Copilot feedback on PR #436
- Policy-cache invalidation now drops both the org-scoped slot AND the
default ``""`` slot on ``create_tool_policy`` for both SQLite and
PostgreSQL backends. ``list_tool_policies("")`` returns rows from
every org_id, and the production evaluators (SessionUIBase.approve_tools
/ cli.py) read with the default ``org_id=""``, so an org-scoped insert
that only invalidated its own slot would leave the default cache slot
stale until the TTL window expired.
- Cap ``reason`` to 200 chars in ``_evaluate_intent`` for ``close_workstream``
and ``close_all_children`` — both fields are LLM/user-provided and the
preparer doesn't size-limit them, so an unbounded reason could bloat
the persisted verdict row's func_args. Matches the cap applied to other
free-form coord tool fields (initial_message, message, title).
- Refresh ``_PolicyCache`` docstring: it claimed lock-free reads on
cache hit but ``get()`` always acquires ``self._lock``. Updated to
reflect that the lock is held briefly to copy the policies reference.
Validation: targeted suite 201/201, ruff + mypy clean.
Follow-up to #434. That PR unified the chat-message primitive on .msg
and noted that the parallel .ts-composer prefix on the shared composer
widget was still in place; this drops it so the widget sits in the
shared/* vocabulary the same way .msg does.
Mechanical 1:1 rename (`ts-composer` -> `composer`) across:
shared_static/chat.css — 48 selectors
shared_static/composer.js — 19 className strings
ui/static/style.css — 11 per-node UI overrides
ui/static/app.js — 7 chip queries / className strings
Pre-rename collision check confirmed clean: the only `composer`-substring
matches in the codebase were IDs (#coord-composer-mount, #coord-composer-
panel, #coord-composer-503, #home-coord-composer-mount — IDs are a
different namespace from classes) and the unrelated console
.home-composer-banner / .home-composer-error pair (different prefix).
CSS specificity audit (scripts/css_specificity_audit.py): 26 findings on
origin/main, 26 on this branch — no new cascade flips.
Tests: 189 affected tests pass (test_app_js, test_webui_content,
test_webui_auto_approve_visibility, test_html, test_web_helpers,
test_coordinator_adapter, test_coordinator_client).
Manual visual verification of composer surfaces (textarea, send button,
stop button, attach button + file picker, chip pills + remove buttons,
options panel toggle, paste-image and drag/drop attach paths, stacked
layout used by creation forms) recommended before merge.
* refactor(ui): drop legacy .ts-msg* dual-classing, chat surfaces share .msg primitive
Third and final follow-up after #431 stripped the data-design="v1"
gate. This drops the parallel .ts-msg* family that had been kept as a
transitional bridge during the gated rollout. Per-node UI now renders
pure .msg classes (previously dual-classed as
"ts-msg ts-msg--user msg user"), matching the coordinator chat view
which already used pure .msg*.
chat.css: deleted the ~210-line legacy .ts-msg* rule block (Messages +
floating action toolbar + mobile + reduced-motion sections); renamed
.ts-msg.ts-approval--inline to .msg.ts-approval--inline; restored the
streaming-markdown rationale (white-space: normal intent + partial-fence
behavior + .msg-user-text path) on .msg-body that previously lived on
the deleted .ts-msg-body, with white-space: normal now declared
explicitly so a future "simplification" can't silently break streaming.
ui/static/app.js: dropped the ts-msg* half of every dual-class string
and updated querySelector callsites (.ts-msg--user -> .msg.user,
.ts-msg--assistant -> .msg.assistant).
ui/static/style.css: renamed all .ts-msg--* selectors to .msg.*; removed
two now-dead override rules (.ts-msg.msg:not(.tool) and
.ts-msg-body.msg-body font-family overrides) that existed solely to
unwind the legacy .ts-msg font-mono default that's now gone.
The .msg.ts-approval--inline selector intentionally keeps the .msg
qualifier (rather than bare .ts-approval--inline) so its (0,2,0)
specificity ties with .ts-approval.approved/.denied/.error and the
later-cascade rule wins; without the qualifier those state classes
would suddenly flip the inline-approval card colour based on state.
Composer rename (.ts-composer-* -> .composer-*) deferred to a follow-up
PR; ~80 occurrences across composer.js + chat.css would have obscured
this verification.
Tests: 538 affected tests pass (test_app_js, test_webui_content,
test_webui_auto_approve_visibility, test_web_helpers, test_html,
test_auth, test_console, test_api_versioning, test_coordinator_*).
Visual verification (message cards, hover toolbar, approval/denial/error
cards in light + dark themes) recommended before merge.
* docs(ui): clarify .msg.reasoning emission comment per Copilot review
The previous wording — ".reasoning as a bare role class is no longer
emitted" — implied .reasoning is never emitted, but the new className
is "msg reasoning" so .reasoning IS emitted, just always alongside .msg.
Reword to make the actual invariant (never on its own) explicit.
* fix(css): two cascade-flip bugs found by specificity audit
PR #431 stripped [data-design="v1"] from ~400 rules, dropping each by a
specificity tier; two cascade flips (#header outranking .appbar, #header h1
outranking .appbar-title) were caught visually during that PR's review and
fixed by renaming id="header" → id="ui-header" on the per-node UI page.
This is the audit follow-up; it found two more:
- textarea.skill-content-area (was .skill-content-area) — bumped to (0,1,1)
so the rule ties with `.admin-modal textarea` (0,1,1) and wins on source
order. Without the bump, min-height: 220px was clobbered to 40px by the
modal default and the spec-content textarea rendered short. The three
!important markers (font-family/size/line-height) are now redundant
against the modal's font: inherit shorthand and are dropped.
- h3.skill-spec-heading — removed `font-size: inherit;`. The author wrote
it to "reset UA defaults" but it locked font-size to the parent's
(~14-16px) at (0,1,1), silently overriding `.skill-spec-heading`'s 10px
at (0,1,0). The bare class already beats UA `h3` on specificity (class >
tag), so no font-size reset was needed; the `margin-block: 0` line stays
because the bare class's `margin: 14px 0 6px` shorthand may not reset
the UA's logical margin-block-start/end on every engine.
Adds scripts/css_specificity_audit.py — the audit tool. It parses every
CSS file referenced from the project's three HTML entry points, computes
selector specificity (incl. :not/:is/:has math, attribute selectors, and
!important), and flags every place an unscoped legacy rule could outrank
a bare-class designed primitive. Honours per-page stylesheet manifests,
state-pseudo subset gating (a `:hover` rule overriding a resting-state
base rule is intentional, not a flip), and shorthand→longhand expansion
for font/padding/margin/border/background. Triage of remaining findings
(26 id-tier in default mode, 74 total at --all-tiers) confirmed all are
intentional designer overrides — id-scoped buttons, BEM modifier classes,
contextual ancestor selectors, last-child margin reset, [hidden] toggle.
* fix(css-audit): correct two cascade-resolution bugs flagged by Copilot
1. _parse_declarations dict insertion order didn't update on overwrite, so
a sequence like `font-size: 13px; font: inherit; font-size: 12px;` would
iterate as (font-size=12px, font=inherit) and the shorthand expansion
then clobbered font-size back to `inherit` — wrong. Delete-then-insert
on overwrite so the last occurrence lands at the dict's tail and the
shorthand expansion sees the real source order.
2. The cascade-winner tie-break used `rule.line_no` only, ignoring the
stylesheet load order. A rule at line 1000 of `base.css` looked
"later" than a rule at line 50 of `style.css`, even though the page
loads `base.css` BEFORE `style.css`. Sort by `(file_index, line_no)`
keyed off the element's per-page stylesheet manifest instead.
* fix(ui): preserve approval pill when tool errors
When an approved (or auto-approved) tool subsequently failed during
execution, both `replayHistory` and `appendToolOutput` located the
existing `.ts-approval-badge` and overwrote its className + textContent
with the `--error` variant — losing the record that the user had
approved the call.
Append a separate `--error` pill as a sibling of the existing approval
pill instead. The `.ts-approval` parent is `flex-direction: column` with
a 6px gap, so the two pills stack vertically and read as a small
status timeline ("you approved this, then it errored"). Idempotency
guard via `querySelector(".ts-approval-badge--error")` so duplicate
fires don't stack badges.
CSS classes are unchanged (the `--error` modifier already exists in
both per-node and shared chat stylesheets).
Adds a static-string guard in `tests/test_app_js.py` that pins both
call sites and forbids the mutate-in-place anti-pattern via a regex
that pairs a queried `.ts-approval-badge` handle with an `--error`
className overwrite.
Deferred from #431.
* refactor(ui): extract appendToolErrorBadge helper, broaden test guard
Address Copilot feedback on #432:
- Extract the duplicated 5-line error-pill construction into a single
module-level `appendToolErrorBadge(blockEl)` helper next to the
other approval-related helpers (`buildToolDiv`, `renderVerdictBadge`,
`toggleVerdictDetail`). Reduces drift risk on ARIA / class / text
string between the two call sites.
- Loosen the affirmative test check from a literal substring keyed on
the local variable name to a regex matching any
`querySelector(".ts-approval-badge--error")` lookup, in either
quote style, in either guard idiom (`if (!q) {...}` at a call site
or `if (q) return;` inside the helper). A future refactor that
preserves behaviour shouldn't trip CI on cosmetics.
- Broaden the anti-pattern regex to accept single quotes and to
catch the `classList.add("ts-approval-badge--error")` form on a
queried badge handle, not only `className = "..."`.
* refactor(css): unify design system, eliminate data-design="v1" gating
Strip the [data-design="v1"] attribute that was wrapping every DS rule
since #389 and never came back out. Result: every styled element on
coord/ui pages had two CSS rules (default + v1-gated), reviewers
couldn't tell which one rendered, and the bundle shipped duplicates.
Changes:
* Strip [data-design="v1"] prefix from ~400 gated rules. Remove the
attribute from coordinator/index.html, ui/index.html, preview.html.
* Merge shared_static/design/* into pre-v1 sheets:
tokens + typography → base.css (:root, dark default)
appbar + panel + buttons + pills + field → ui-base.css
message primitives → chat.css
sidebar + approval-dock → console/static/coordinator/coordinator.css
(new file linked from coord only — admin no longer ships ~9KB of
coordinator-only chrome on first load).
* Delete preview.html + 5 preview-only orphan stylesheets (topbar /
stats / feed / fleet-grid / live-feed). Drop the empty
shared_static/design/ directory.
* Drop unused primitives the merge dragged in: .pill / .k-badge /
.chip / .field / .t-* utilities / .side-item / .shell. Drop unused
tokens (--accent-c / --accent-l / --row-h / --density / --gap /
--font-display alias). Find-replace var(--font-display) →
var(--font-ui) across 6 files (103 sites).
* Standardize on the DS font stack: Inter body, JetBrains Mono code.
Admin's body shifts from IBM Plex Mono → Inter via the alias rename.
* Re-tune legacy --bg/--bg-surface/--bg-highlight/--bg-elevated from
steel-blue to neutral charcoal so admin and v1 pages share one
palette. Drop the cyan radial-gradient overlay on body that added a
blue tint to the formerly-blue bg.
* Polish:
- .msg.tool / .ts-msg--tool / .ts-approval / inline-approval all use
--cyan instead of amber, removing the user/tool colour collision.
- .msg-action-btn reverts to icon-button styling (transparent,
28x24) after the merge gave it text-button chrome that dwarfed the
13-14px icon glyphs inside.
- Light-mode composer contrast: flip .ts-composer surface roles
(wrapper recessed, textarea elevated) so the textarea reads
against its container; bump .dashboard-composer textarea
border-bottom + options panel surface so they're visible on white.
- .pane-messages padding 20px -> 16px 12px and gap 14px -> 0 (the
flexbox gap was stacking with .ts-msg margin-bottom for ~18px
inter-card spacing); .ts-msg/.msg margin-bottom 8px -> 4px.
- Restore WCAG 2.5.5 36x36 touch target on .msg-action-btn.
- Rename ui's <div id="header"> to id="ui-header" so the legacy
#header chrome no longer outranks the .appbar primitive on the
per-node page (3 getElementById calls in app.js updated).
- Drop chat.css link from coord (coord renders pure DS classes; the
composer.js consumer of chat.css is on admin + ui only).
Verified: ruff + mypy clean (175 files); 4653 non-live tests pass;
zero data-design / --font-display / shared_static/design hits remain.
Net source change: -1885 lines (924 added, 2809 deleted across 28 files).
* build: include coordinator.css in wheel; drop dead design/ glob
The previous commit added turnstone/console/static/coordinator/coordinator.css
(coord-only chrome moved out of console/static/style.css) but didn't
update the [tool.hatch.build.targets.wheel] include list, so CI's
wheel-completeness check failed.
Also drop the now-stale 'turnstone/shared_static/design/**/*' glob —
that directory was deleted in the same v1-elimination commit.
Verified locally: replicating the CI step's source-vs-wheel diff
returns MISSING: none.
* fix(css): address Copilot review feedback on PR #431
* coordinator/index.html — comment now correctly points to the moved
.sidebar rules at console/static/coordinator/coordinator.css (was
console/static/style.css before the perf-1 split-out).
* ui/static/index.html — restore <h1 class="appbar-title">; the cascade
conflict that motivated the h1→div change is gone now that the
wrapper id was renamed away from #header (the legacy #header h1 rule
no longer matches). Page semantics + accessibility regain the
top-level heading.
* governance.js — drop the inline font-family:var(--font-ui) on the
config-key <code> elements; let them inherit the global mono default
from base.css. The inline style was an artifact of the
--font-display → --font-ui find-replace; the original Outfit was
already odd on a <code> tag.
* ui-base.css — typography-helpers comment said "Body text still
inherits var(--font-mono) at 13px" but base.css now sets var(--font-ui)
at 14px. Reword to match current defaults.
* fix(approve): visibility for child tool calls bypassing operator gate
When a coord LLM spawns a child with `skill="X"`, the skill template's
`allowed_tools` JSON list silently populates the child UI's
`auto_approve_tools` set. Tool calls whose names are in that set
short-circuit the approval gate without prompting the operator —
matching the user-reported bug "tool calls of children occasionally
getting approved instead of waiting for approve/deny".
The auto-approve paths themselves are unchanged (Option C — visibility
only). Surfaces:
- Per-item annotations: each pending tool gets `auto_approved=True` +
`auto_approve_reason` ("skill" / "always" / "policy" / "blanket" /
"auto_approve_tools") at the four gate-bypass paths.
- Per-ws ring buffer (cap 10) of recent bypasses, exposed via
`/dashboard` and the cluster live-bulk projection so the coord-
tree row can render an "auto-approved by ..." pill.
- `tool.auto_approved` audit row per `approve_tools` call —
forensic durability beyond the in-memory ring buffer.
- Per-ws WebUI page: inline "auto: <reason>" badge next to each
tool name, so an operator who clicks through from the coord tree
to the child's page sees the same bypass signal.
Persistence across UI rebuilds:
- The ring buffer is in-memory only; a saved-workstream rehydrate /
coord→node click-through / process restart all build a fresh UI.
`replay_recent_auto_approvals_from_audit` runs at the end of
`SessionUIBase.__init__` and re-seeds the buffer from recent
`tool.auto_approved` audit rows scoped to this ws_id.
- Adds `resource_id` filter to `list_audit_events` (protocol +
SQLite + Postgres) so the replay is a single indexed query.
Source provenance:
- `_auto_approve_tools_source: dict[str, str]` per UI tracks which
writer added each tool name to `auto_approve_tools` ("skill" at
skill-template setup time, "always" on Approve+Always click).
Lets the dashboard pill distinguish a skill-driven bypass from
an explicit operator-Always click — those are very different
signals that previously rendered the same.
Magic-string drift mitigation:
- `AutoApproveReason` constants in `core/session_ui_base.py` lift
the five reason strings into a single source of truth.
- `KNOWN_AUTO_APPROVE_REASONS` JS constant + validator render
unknown reasons as "unknown" with a console.warn instead of
rendering raw (a typo would otherwise silently desync wire ↔
pill).
Recording-leak fixes (q-2 from review):
- Policy `allow` partial-resolve now records the policy-tagged
items at two previously-leaking branches: the early-return-on-
deny path and the still_pending-non-empty fall-through to the
prompt path.
Other review fixes:
- Heuristic verdict surfaces consistently as `heuristic_verdict`
in both `_serialize_approval_items` and the dashboard
serializer (was inconsistent: one emitted `verdict`, the other
`heuristic_verdict`). app.js updated to read either key for
mid-deploy compatibility.
- `_tag_auto_approved` helper on SessionUIBase replaces the
verbatim tag loops previously copy-pasted across WebUI and
ConsoleCoordinatorUI.
* fix(approve): apply Copilot review feedback on PR #430
- coordinator_ui: use ``approval_label or func_name`` for the
``auto_approve_tools`` subset check, matching WebUI. Pre-fix
an "Approve + Always" entry whose approval_label differs from
func_name (skill__name, mcp_resource__uri) wouldn't match on
the coord page and the operator would be re-prompted.
- _parse_audit_timestamp: treat naive ISO strings as UTC. Audit
rows are written via ``datetime.now(UTC).strftime(...)`` with
no timezone marker; ``datetime.fromisoformat`` returns a naive
datetime, and ``.timestamp()`` on a naive datetime interprets
it in the server's local timezone — wrong on any non-UTC
server. Stamp UTC explicitly before converting.
- server.py: drop the dead ``pending = []`` after the blanket
tag — the function returns inside the same block without
reading ``pending`` again.
- _protocol.py: fix docstring reference from
``_replay_recent_auto_approvals`` to
``replay_recent_auto_approvals_from_audit`` (the actual
method name).
* fix(coord): tree UI not updating when LLM deletes workstream
The coord LLM's `delete_workstream` tool wiped the storage row but
fired no SSE event, so a long-lived dashboard tab kept the deleted
child visible (with its last-known idle/closed state) until a full
reload. A coordinator that spawns→completes→deletes children would
leave an ever-growing tree.
Fix: add `SessionManager.delete()` that drops the in-memory slot if
present and emits `ws_closed` with `reason="deleted"` (mirrors
`close()`'s shape). Wire `delete_workstream_endpoint` to call it
after the storage delete succeeds, snapshotting the workstream's
name into the event payload before the row is wiped. The cluster
collector → coord adapter chain re-emits as `child_ws_closed`; the
browser's existing `handleChildClosed` already keys on
`reason === "deleted"` to mark the row, so no JS changes needed.
Event emit is best-effort — a fan-out failure logs a warning but
doesn't roll back the storage delete (the row is already gone).
* fix(coord): apply Copilot review feedback on PR #429
- server.py: clarify that ``name`` is forwarded to mgr.delete only
(not into the audit detail) — comment previously claimed both.
- test_session_manager.py: extract ``mgr.delete(ws_id)`` to a local
before asserting (CodeQL: no side-effecting calls inside ``assert``,
which would be stripped under ``python -O``).
- test_workstream_endpoints.py: docstring said "Yield" but the
fixture ``return``s; switch to "Return".
Copilot caught a doc/code mismatch from the q-2 cleanup: the
docstring still claimed `closed` / `deleted` / `denied` all return a
sentinel, but the `deleted` branch was dropped (hard deletes cascade
rows out of storage so the state is unreachable). Update the
docstring to align with `_wait_message_for`'s actual behaviour —
`deleted` falls into the same null-message shape as a still-running
entry.
Each per-ws snapshot now carries `message` + `truncated` so the
coordinator LLM doesn't need a follow-up `inspect_workstream`
round-trip per child to read what came back. idle/error states
return the last assistant turn (capped at 6 KiB UTF-8 bytes,
truncated from the end); closed/denied return a sentinel; running
children carry null. Storage reads for idle/error parallelize
across an 8-worker thread pool so a 32-child fan-out lands in 4
batches instead of 32 sequential round-trips.
Creating or opening a workstream from the dashboard left the chat
UI blank until the operator refreshed: switchTab early-returned at
``if (!pane) return;`` because getFocusedPane was null on a fresh-
loaded page that had no workstreams. The freshly-created ws was
added to the workstreams dict and the dashboard was hidden, but no
pane was bootstrapped, no SSE connected, and the chat area sat
empty until refresh — at which point initWorkstreams saw the
populated list and bootstrapped the pane via the existing
"if (!Object.keys(panes).length)" branch.
switchTab now mirrors that bootstrap when no focused pane exists:
createPane + splitRoot leaf + setFocusedPane + renderLayout. The
rest of switchTab (disconnectSSE / reset / connectSSE) runs as
before — no-ops on the just-constructed pane up to the connectSSE
call which is exactly what we want.
Subsequent creations on the same node already worked because the
first create populated panes and switchTab found a focused one.
Static smoke test in tests/test_app_js.py guards against the
early-return regressing.
* feat(renderer): progressive mermaid rendering during streaming
Mermaid diagrams used to materialize all-at-once at stream_end via
streamingRenderFinalize, which felt laggy on long responses with
multiple diagrams. Now closed mermaid fences render progressively
as each fence completes during streaming.
The blocker was streamingRender's wholesale `el.innerHTML = html`
on every rAF tick, which destroys any rendered SVG nodes — without
caching, calling postRenderMermaid per tick would re-trigger an
async mermaid.render every time, thrashing the renderer.
Added a source-keyed SVG cache (_mermaidSvgCache, FIFO-bounded at
64 entries):
- Cache hit on identical source: synchronous innerHTML swap, no
loading flash, no async work. Mermaid is deterministic for a
given init, so identical source ⇒ identical SVG, safe to reuse.
- Cache miss: queue async render, populate cache on success.
- Errored sources cached separately (_mermaidErrorCache) so a
syntactically-broken diagram doesn't re-thrash mermaid on every
tick. The user can fix the diagram and the new source string
misses the cache, triggering a fresh render.
_streamingRenderApply now calls postRenderMermaid after the
innerHTML replace. Per-stream cost: each unique mermaid source
pays mermaid.render once, then synchronous cache hits for every
subsequent rAF tick. hljs syntax highlighting stays deferred to
streamingRenderFinalize (it's a separate pass and benefits less
from progressive rendering — code blocks tend to be short and
already legible without color).
Tests: built a richer Node-driven harness with a fake DOM that
tracks attributes / classList / parent chain / replaceWith, plus
a stubbed mermaid.render with a call counter. 5 new tests cover:
cache-hit skips render, distinct sources render independently,
errors cache to avoid thrash, FIFO eviction at cap, and a static
guard that _streamingRenderApply actually calls postRenderMermaid.
* fix(renderer): apply Copilot feedback on PR #426
Six review items, all real:
1. _cacheMermaidEntry evicted on overwrite — overwriting an
existing source unnecessarily dropped the oldest entry.
Now: only evict when inserting a new key.
2. _initMermaid didn't clear caches — a theme change via
reRenderAllMermaid (which calls _initMermaid) would serve
stale SVG keyed by source-only, since rendered output
depends on themeVariables. Now clears both caches on
(re-)init.
3. bindFunctions never re-applied on cache hits — mermaid's
bindFunctions attaches link/click handlers to each rendered
SVG instance. Pre-fix, only the first render got bindings;
subsequent cache hits via raw innerHTML left the SVG inert.
Cache value is now {svg, bindFunctions}; cache hits go
through _applyMermaidSvg which re-applies bindings on each
new container instance.
4. Truthiness checks on cache lookups — empty-string SVG / error
would have masqueraded as a miss. Switched to cache.has()
(and then .get) so intent is explicit.
5. Concurrent mermaid.render — postRenderMermaid now fires on
every streaming rAF tick, so multiple ticks could overlap
while earlier render Promises pend. mermaid.render uses
module-level state internally — concurrent calls clobber it.
Two layers of serialization fix this:
- _mermaidPending: per-source. While a render is in flight
for source X, additional containers asking for X are queued
and the single render result fans out to all pending
containers when it lands.
- _mermaidRenderChain: across-source. Promises chain so
mermaid.render runs at most one at a time globally.
- Detached containers (no longer in the DOM by the time the
render completes) are skipped via isConnected guard —
wholesale innerHTML replace during streaming detaches them
and a later tick is already taking care of the live one.
6. Test brittleness — _streamingRenderApply guard used
body.index("\\n}\\n", start) which would stop at the first
inner-block closing brace inside the function. Switched to a
bounded-window string search (Copilot's suggestion).
Three new tests added: overwrite doesn't evict; _initMermaid
clears caches; cache hit re-applies bindFunctions. Existing tests
updated for the new {svg, bindFunctions} cache shape and the
async serialization (drain via setTimeout hops instead of bare
microtask resolves).
Test harness fix: fake DOM elements now have an isConnected
getter derived from the parent chain, so the new guard
exercises correctly under test.
* fix(renderer): handle LaTeX-style \(...\) and \[...\] math delimiters
The browser renderer at turnstone/shared_static/renderer.js only
recognized TeX-style $...$ / $$...$$ delimiters. Most modern LLMs
(GPT-5 / o-series, Claude with reasoning effort) emit LaTeX-style
\(...\) for inline math and \[...\] for display by default — those
slipped through as raw text in the coord + interactive WebUIs,
making KaTeX appear "broken when nested inside a markdown block"
(actually broken everywhere, the surrounding markdown just made
the failure noticeable).
Added a second pass for each delimiter style alongside the
existing $...$ / $$...$$ patterns. Both styles now feed the same
mathBlocks / inlineMaths placeholder pipeline so all the existing
nested-block handling (lists, blockquotes, tables, bold, headings,
details, post-render KaTeX markup) Just Works.
Edge cases verified by the new test_renderer_js.py harness:
- \(...\) inside inline code stays literal
- \(...\) inside fenced code blocks stays literal
- Solo \[ with no closing \] doesn't trigger spurious math
- Markdown links [text](url) untouched (regex uses \[ \], not [ ])
- Mixed TeX + LaTeX delimiters in one message both render
The harness drives renderer.js through Node via vm.runInThisContext
with stubbed document/katex globals — first JS-side regression
guard for the renderer; previously it had no test coverage at all.
* fix(renderer): apply Copilot feedback on PR #425
Three review items from Copilot:
1. Display-math sentinel could leak through inline-code spans.
The original ordering ran $$...$$ / \[...\] extraction BEFORE
inline code, so a backtick span around math (e.g. `$$x$$` or
`\[x\]`) had its delimiters consumed by the math regex and
replaced with \x00MB…\x00. Inline code then captured the
sentinel; restore order put MB after IC, leaving the null-byte
placeholder visible inside the rendered <code>. Reorder: inline
code first, then display math, then inline math. Code spans
now seal their content before any math regex sees it. The
reverse edge case (math containing backticks, e.g. \verb|`x`|)
is much rarer and KaTeX rejects \verb anyway.
2. Inline LaTeX-style \(...\) regex used [\s\S]+? which allowed
newlines, so an unterminated \( on one line would eat the
next paragraph until it found a closing \). Aligned with the
existing $...$ behavior by switching to [^\n]+? — display
math (\[...\] / $$...$$) stays multi-line by design.
3. tests/test_renderer_js.py was guarded with a node-availability
skip, but CI's test + test-postgres jobs didn't explicitly
install Node, so the suite would have silently no-op'd if the
runner image dropped Node. Added actions/setup-node@v5 to
both jobs.
Four new regression tests cover the leak (both delimiter styles
inside backticks must stay literal) and the cross-paragraph span
(both \(...\) and $...$ must not eat newlines).
Bug: LLM judge verdicts stayed stuck on heuristic-only render.
Root cause: per-row poller called scheduleLiveFetch which
short-circuits on non-visible rows — invalidate cleared the
cache, no fetch fired, the row kept rendering its last-cached
heuristic indefinitely. The 12s attempt cap also gave up before
slow LLM judges (>15s with reasoning effort) could land.
Replaced with a single global poller _maybeStartJudgePoll /
_judgePollTick:
- Walks the full childrenState (not just visible rows)
- Bypasses scheduleLiveFetch's visibility + TTL gates by
adding to pendingLiveIds directly + flushing
- One bulk request covers every pending row per tick
- Self-terminates when every verdict lands or 90s elapses
(operator can hit Refresh to retry on a failed judge)
- 90s cap is wall-clock, not attempt count, so an LLM that
takes 60s no longer prematurely gives up
Copilot round-2 feedback:
- _proxy_sse with use_service_auth=True silently fell back to
empty headers when proxy_token_mgr was None, producing a
retry-storm 401/403 loop. Fail fast with a 503 + clear log
so the misconfig surfaces immediately.
- Mobile <700px CSS comment claimed buttons "stretch to full
row width" but the rule keeps flex-direction: row with
flex: 1 on each, giving 50/50 side-by-side. Updated the
comment to match the deliberate side-by-side layout
(stacking would push the action row below preview/disclosure
on tall envelopes; 50/50 keeps both verbs reachable).
#422's legacy URL adapter removal deleted the body-keyed
/v1/api/route/{verb} endpoints (with ws_id in JSON body) but
turnstone/console/coordinator_client.py still pointed at them.
The coord LLM's close_workstream / close_all_children tools
404'd; send / approve / cancel were equally broken though
exercised less often.
_ROUTE_PATHS now uses {ws_id}-templated path-keyed forms:
send → /v1/api/route/workstreams/{ws_id}/send
approve → /v1/api/route/workstreams/{ws_id}/approve
cancel → /v1/api/route/workstreams/{ws_id}/cancel
close → /v1/api/route/workstreams/{ws_id}/close
_post() interpolates {ws_id} at call time when the template has
the slot; body-keyed paths (delete, close_all_children) still
work via the same code path. Each affected caller (send, approve,
cancel, close_workstream, close_all_children) was updated to pass
ws_id as the kwarg and drop ws_id from the body.
Added test_route_paths_match_actual_console_mounts: walks the
real Starlette app's routes and asserts every _ROUTE_PATHS entry
corresponds to an actually-mounted route. Catches the next URL
unification drift before runtime. Updated the existing literal
assertions + path-checking tests for the new shape.
Pre-existing bug surfaced while testing inline-child-approvals.
The interactive WebUI's app.js opens an EventSource against
/v1/api/events/global on load (cluster-wide tab indicators,
ws_state for the dashboard). When loaded via the console proxy
at /node/{node_id}/, the JS shim rewrites that to
/node/node-X/v1/api/events/global and the proxy forwards using
the user's re-minted JWT.
Upstream global_events_sse requires `service` scope by design
— the stream carries cross-tenant cluster inventory, intended
for the cluster collector, not browsers. End-user JWTs don't
carry service scope, so every proxied call returned 403, the
browser auto-retried with exponential backoff, and the console
log filled with proxy.sse.non_200 warnings.
_proxy_sse gains a use_service_auth flag. proxy_api flips it on
for events/global only, swapping the user JWT for the console's
proxy_token_mgr bearer token. Per-ws events stay on user auth
(tenant filtering on the upstream still requires user identity).
The upstream-side privacy posture is unchanged — the data on
events/global is the same cluster-wide inventory the console's
own /v1/api/cluster/events endpoint already serves to any
read-scoped caller under the trusted-team posture. The console's
AuthMiddleware on /node/{node_id}/v1/api/ remains the gate that
decides who can use the proxy at all.
The console's node-API passthrough at /node/{node_id}/v1/api/{path}
detected SSE only on the bare events / events/global paths. After
#422 removed the legacy /v1/api/events?ws_id= shape and moved
per-workstream SSE under /v1/api/workstreams/{ws_id}/events, the
proxy never got updated to match the new path — per-ws events
fell through to the regular GET branch, the upstream returned a
text/event-stream payload that the regular GET response couldn't
hold open, and Firefox surfaced the failure as "can't establish a
connection to the server".
Extend the SSE detection to also match
``workstreams/{ws_id}/events``. Pre-existing bug surfaced while
testing inline-child-approvals (operator clicks through from the
coord tree to the per-child interactive WebUI) but affects every
caller hitting a node's per-ws events stream via the console
proxy.
Two new tests in TestConsoleProxy: per-ws events route to
_proxy_sse with the correct upstream path; existing
events/global routing still works.
Previously the 409 stale-call_id branch in submitChildApproval
re-enabled both buttons synchronously before kicking off the
urgent live-bulk refresh. That opened a window where rapid clicks
on an already-resolved approval (or a row whose call_id had
rolled) each re-armed the click handler, fired another POST, and
collected another 409. Operators rage-clicking saw a network 409
storm and a stack of warning toasts.
Keep the buttons disabled in the 409 path. The row is about to be
re-rendered wholesale via the urgent refresh — the disabled DOM
gets dropped along with it. If the row's approval truly resolved,
the new render has no buttons. If a new round started, the new
render has fresh enabled buttons. Either way the operator-facing
signal IS the row updating, not the toast.
Drop the toast.warn (noisy on every rapid-click race) in favour
of a single console.warn for diagnostics.
If the urgent refresh fails entirely, the buttons stay disabled
on that row — but the operator can hit the Refresh button on the
children panel to force a full reload. Acceptable degraded state
vs the previous 409 loop.
The coord's _coord_events_replay re-yielded _pending_approval on
connect but not the cached _llm_verdicts entries. A tab refreshing
mid-approval saw the approve_request prompt without the judge chip
because intent_verdict is a one-shot SSE event with no
late-subscriber push — the chip would only ever land if the operator
re-invoked the tool call.
Mirrored the interactive path at turnstone/server.py:875-878:
after re-injecting the pending_approval prompt, walk
ui._llm_verdicts under _ws_lock and yield each cached verdict as
an intent_verdict event. Pre-existing bug surfaced during the
inline-child-approvals work but the coord-self dock UX was always
affected on reconnect — not introduced by this PR.
Two new tests: cached verdicts replay after pending_approval; stale
verdicts from a prior round don't replay when no approval is pending.
Two bugs reported from local repro on PR #424:
1. Approve/Deny buttons return HTTP 404 on every click. The new
approveWorkstream helper hit /v1/api/workstreams/{ws_id}/approve
regardless of target — that path is only mounted for coord
workstreams (which live on the console process). Child
workstreams live on cluster nodes and need to round-trip
through the routing proxy at
/v1/api/route/workstreams/{ws_id}/approve, which resolves the
ws_id to its owning node and forwards the body verbatim.
approveWorkstream now picks the path based on whether targetWsId
matches the coord's own wsId.
2. LLM judge verdict never populates — rows freeze on the
heuristic-tier pill ("⚙ heuristic") even after the judge would
have completed. The judge runs async on the child node via a
daemon thread and updates _llm_verdicts there, but no signal
propagates back to the coord — cluster_state events don't fire
on verdict-only changes, and the live-bulk TTL is 5s with no
periodic poll.
Added _maybePollForJudgeVerdict: when renderChildRow encounters
a pending_approval_detail with judge_pending=true and items
missing judge_verdict, schedule a recursive 2s urgent
live-bulk re-fetch. Self-terminates when the verdict lands,
the row closes, the approval clears, or attempts hit the cap
(≈12s for a failed/timed-out judge so we don't poll forever).
Single timer per ws_id; re-renders are no-ops while a timer is
in flight.
Smoke-test assertions added for both fixes so a regression on
either path surfaces at test-time.
Copilot review on PR #424 flagged three items:
1. Schema drift on /v1/api/dashboard — DashboardWorkstream didn't
declare the new pending_approval_detail field, so generated
OpenAPI / typed clients were out of sync. Added
PendingApprovalItem + PendingApprovalDetail Pydantic models
and referenced PendingApprovalDetail from DashboardWorkstream.
2. deepcopy under _ws_lock in serialize_pending_approval_detail
could extend lock hold under contention with on_intent_verdict
(daemon judge thread) and per-token activity writes that also
take _ws_lock. _llm_verdicts entries are only assigned/cleared,
never mutated in place, so a snapped reference is stable after
the lock drops. Snapshot refs under lock; deepcopy after release.
3. Plan doc removed from the branch — design docs are local-only
working artifacts, same posture as PROGRESS.md.
Critical:
- coordinator.js RISK_SEVERITY accepted 'crit' only; production
emits 'critical' (per turnstone/core/judge.py:1556 + heuristic
seeds). A risk_level=='critical' verdict ranked as 0 and
rendered with .risk.low (green) styling, never triggering
the crit-risk auto-expand. Now accepts both aliases. Unknown
risk_level falls back to rank 2 ('high') so future schema
drift fails *safe* (over-alert) instead of silently
downgrading. Pill ternary handles both 'crit' and 'critical'
alias to the existing .risk.crit class.
Major:
- Urgent live-badge flush now coalesces N urgent calls in the
same JS tick into one bulk request via queueMicrotask, instead
of firing N single-id fetches. The motivating 10-children-
pending-bash scenario in the design doc now lands on one bulk
/v1/api/cluster/ws/live request.
- Test coverage gap: added test_session_ui_base.py cases for
POLICY-BLOCKED (item.error + needs_approval=False) and
judge-unavailable (no verdict + no judge_pending) matrix rows.
Added literal-string assertions to the smoke list in
test_coordinator_page.py so a refactor dropping either branch
surfaces at test-time.
Minor batch (4 coord.js + 1 CSS + 1 fake-divergence):
- 409 stale-call_id path re-enables both buttons before return
(urgent fetch is best-effort; could also fail).
- judgePending pill no longer conflicts with a present heuristic
verdict — guard changed from !judge to !verdict.
- Empty <div class="approval-reasoning"> no longer appended when
reasoning is absent but evidence is present (evidence still
renders inside the disclosure).
- Dead .ch-row .approval-pill.rec-* CSS rules removed (JS never
combines those classes). Recommendation chip in the disclosure
footer now has its own scoped rules so the chip is actually
styled.
- _FakeUI.serialize_pending_approval_detail call_id selection
aligned to the real impl's "first non-empty" semantics.
- liveBadgeCache reconnect cleanup now preserves permanent
(403/404) entries — denied users no longer pay one wasted
bulk fetch per denied id per reconnect.
All 4465 non-live tests pass. Ruff + mypy clean. node --check OK.
Closes the stale-button window where a sub-5s SSE gap would leave
liveBadgeCache holding pending_approval_detail for a child whose
approval was actually resolved during the gap. Without this clear,
zombie approve/deny buttons render until either the next child_ws_state
event or the natural TTL expiry (whichever comes first).
The clear sits beside the existing activeWaits.clear() in the
reconnect handler — same posture (drop client-only state that the
server's SSE replay doesn't cover) and same blast radius. The 409
race guard in submitChildApproval would catch a stale-call_id POST
even without this, but rendering wrong UI until the operator clicks
is the worse failure mode.
loadChildren's finally block already fires scheduleLiveFetch for
every visible row after the replace-mode refresh, so the cache
repopulates with authoritative pending_approval_detail in one bulk
request within the next debounce window.
Plan: docs/design/inline-child-approvals.md (chunk 4 of 4 — last
required chunk; 5/6 are stretch).
Chunk 3 of the inline-child-approvals plan + the SSE pipeline plumbing
needed for sub-second urgent fetches.
JS (coordinator.js):
- approveWorkstream(targetWsId, body) — generic POST helper, callable
for both the coord-self dock and the new per-child inline buttons.
- renderApprovalBlock(child, detail) — risk-level pill (.risk.* per
the design system primitives), tool-name summary with "+ N more"
for envelope-level approvals, intent_summary, ↳ judge reasoning
teaser, ▸ more disclosure carrying the recommendation chip,
evidence list, and items 2..N stacked sub-blocks. Plus matrix
coverage: judge_pending / judge unavailable / tool-policy
blocked / multi-item.
- submitChildApproval — handles the 409 stale call_id race by
invalidating the live cache + urgent-refetching, optimistically
clears pending_approval_detail on success.
- scheduleLiveFetch({ urgent: true }) — bypasses the 5s TTL +
cancels the debounce so attention transitions surface inline UI
immediately instead of after the next polling window.
- handleChildState fires urgent on activity_state="approval"
enter/leave; handleChildClosed eagerly invalidates the live
cache so closed rows can't render stale buttons.
CSS (index.html):
- New .approval-block / pill / preview / actions / disclosure
styles. Inline .act buttons duplicate the dock's colour treatment
(the dock-scoped rules don't reach the children-tree). Mobile
<700px touch targets ≥44px.
Pipeline (collector.py + coordinator_adapter.py):
- All three cluster_state event emitters and the child_ws_state
re-emit now carry activity_state. The previous omission left
the urgent-fetch trigger as dead code — discovered in review.
Tests:
- Static smoke test in test_coordinator_page.py asserting the new
helper names exist + the pending_approval_detail key is read.
Plan: docs/design/inline-child-approvals.md (chunk 3 of 4).
Threads the field added by Chunk 1 through the console's live-bulk
endpoint so coord tree UI can read it without a separate per-child
fetch. Three touchpoints:
- _CLUSTER_WS_LIVE_KEYS gains the new key so _fetch_live_block's
projection forwards it from the upstream /dashboard response on
node-backed child rows.
- _coordinator_live_snapshot synthesizes the same shape from
ConsoleCoordinatorUI._pending_approval for in-process coord
rows (no upstream /dashboard exists on the console pseudo-node).
- One source of truth: SessionUIBase.serialize_pending_approval_detail.
Both branches now emit the same 12-key live block; coord judge isn't
wired today so coord-self judge_verdict is always None — flagged in
the plan as a stretch follow-up.
Plan: docs/design/inline-child-approvals.md (chunk 2 of 4).
Lays the server-side groundwork for inline approve/deny buttons + judge
verdict on the coordinator children-tree UI. Two surgical changes:
1. SessionUIBase.serialize_pending_approval_detail() merges the active
_pending_approval items[] with per-call_id verdicts from
_llm_verdicts. The dashboard handler embeds this on every per-ws
row so cluster live-bulk callers can render inline UI without an
extra per-child round-trip.
2. make_approve_handler now returns 409 when the body sends a call_id
that doesn't match any currently-pending item. Closes the stale
call_id race where an operator clicks approve on a row showing
call A while the child has rolled over to call B. Empty/missing
call_id preserves backwards compatibility with CLI + channel
adapters that don't track it.
Cross-tenant exposure on /dashboard is consistent with the trusted-team
posture already in place for activity / tokens — documented in the new
method's docstring so the choice survives the next reviewer.
Plan: docs/design/inline-child-approvals.md (chunk 1 of 4).
Copilot review on PR #422 flagged that the DELETE-on-send (dequeue)
EndpointSpec declared no request_model, so the generated OpenAPI
showed no requestBody for an operation that *requires* a JSON body
with ``msg_id`` and 400s when it's missing.
- Add ``DequeueRequest`` to ``server_schemas.py`` with the single
required ``msg_id: str`` field.
- Wire ``request_model=DequeueRequest`` and ``response_model=
StatusResponse`` on the DELETE EndpointSpec; trim the now-redundant
inline body example from the description.
- Re-import the schema in ``server_spec.py`` and add the entry to
``_ALL_MODELS`` so the OpenAPI components list carries it.
- Regenerate ``openapi-server.json``.
Sibling thread on the close EndpointSpec was already addressed in
4000ae2 (request_model=CloseWorkstreamRequest).
4558 tests passing under ``-m "not live"``; ruff + mypy clean.
Copilot caught three real issues in PR #422 review, all clustered
around the close request body contract:
1. The interactive close handler runs with
``supports_close_reason=True``, which calls
``read_json_or_400(request)`` — an empty / non-JSON body returns
``400 {"error": "Invalid JSON body"}``. The previous SDK fix
sent NO body via ``json_body=None``, which would 400 against a
real server. The mock-transport test silently masked it because
the mock answered without inspecting the body.
2. The doc said the body was empty (or ``{}``), with no mention
of the optional ``reason`` field, its 512-byte cap, or the
credential-redaction guard.
3. The Pydantic schema for close was deleted outright; OpenAPI
and SDKs lost their typed shape for the optional ``reason``.
Changes:
- ``turnstone/api/server_schemas.py``: reintroduce
``CloseWorkstreamRequest`` with a single optional
``reason: str | None = None`` field. Docstring documents the
must-be-valid-JSON contract and notes that coord ignores the body
(``supports_close_reason=False``).
- ``turnstone/api/server_spec.py``: re-import the schema, point the
close ``EndpointSpec`` at it via ``request_model=``, restore the
``_ALL_MODELS`` entry. OpenAPI JSON regenerated.
- ``turnstone/sdk/server.py``: ``close_workstream`` (sync + async)
gains an optional ``reason: str | None = None`` parameter and
always sends ``json_body={}`` (or ``{"reason": ...}``) so the
body is never empty. Adds a regression test
(``test_close_workstream_sends_valid_json_body``) that inspects the
raw transport content rather than relying on a path-keyed mock —
the kind of check that would have caught this bug pre-merge.
- ``sdk/typescript/src/server.ts``: ``closeWorkstream`` gains an
optional ``opts.reason`` parameter; reintroduce
``CloseWorkstreamRequest`` interface in ``types.ts`` and re-export
from ``index.ts``.
- ``docs/api-reference.md``: close section documents the JSON-body
requirement, the ``reason`` field, the 512-byte cap, the
multibyte-safe behavior, the credential-redaction guard, and the
non-string-coercion path.
- ``CHANGELOG.md``: amend the 1.5.0 BREAKING block to reflect the
schema reintroduction (slim form, ``reason`` optional) instead of
the prior "removed outright" claim.
4558 tests passing under ``-m "not live"`` (was 4557 — +1 from the
regression test). ruff + mypy clean.
Reviewer caught real misses on the consumer-swap claim:
- TypeScript SDK still defined and re-exported `CloseWorkstreamRequest`
(types.ts + index.ts) — drop both. Now matches the Python-side
removal.
- Four `tests/test_auth.py` cases (`test_write_full_token_ok`,
`test_approve_full_token_ok`, `test_bearer_takes_precedence_over_cookie`,
`test_cookie_full_on_write_ok`) were tautological after the legacy
URL removal: they posted to `/api/send` / `/api/approve` and asserted
`allowed is True`, but those paths now classify as `read` so a read
token would also pass — they no longer tested the write/approve
scope enforcement. Swap to path-keyed URLs to restore the original
intent.
- `is_public_path("/api/send")` test renamed + retargeted to a
path-keyed URL.
Doc-table drift the previous commit missed:
- `docs/security.md` path-to-scope mapping rewritten for the
path-keyed verb family (write set, DELETE-on-/send dequeue,
per-ws_id approve).
- `docs/architecture.md` scope-model row text swap from `/api/send`
/ `/api/approve` to the path-keyed equivalents.
- `docs/diagrams/01-system-context.puml` channel→server edge label
swap.
- `docs/diagrams/15-auth-architecture.puml` scope class swap.
Cosmetic comment-only stragglers:
- `tests/test_session_worker.py` module docstring URL update.
- `tests/test_ratelimit.py` ~11 `/api/send` fixture-key strings
retargeted to `/api/workstreams/abc/send` so the URL fixtures
reflect the post-1.5 surface (rate limiter is path-agnostic; the
swap is purely cosmetic).
4557 tests still passing under -m "not live"; ruff + mypy clean.
CHANGELOG [Unreleased] / Removed (BREAKING — 1.5.0) block calling out
the legacy URL family removal with the swap table. Doc passes on
api-reference.md (per-endpoint sections rewritten with path
parameters and slimmer body shapes), architecture.md (handler-list
diagram and console-proxy URL example), console.md (URL-rewriting
JS shim docstring + SSE proxy example), and the two PlantUML
diagrams (11-console-data-flow, 16-channel-architecture).
Also picks up two test-side stragglers from step 5 that referenced
the legacy adapters in a docstring + a stale /v1/api/events SSE
test: turn into path-keyed equivalents. OpenAPI JSON dump regenerated
to reflect the catalog edits from step 3.
After this commit:
- 4557 tests passing under -m "not live"
- ruff + mypy clean on turnstone/ tests/ sdk/
- grep for "/v1/api/send", "/v1/api/approve", "/v1/api/cancel",
"/v1/api/workstreams/close" returns zero hits across turnstone/
sdk/ docs/ tests/ (excluding CHANGELOG.md, which intentionally
documents the old shape).
- grep for make_legacy_body_keyed_adapter, make_legacy_query_keyed_adapter,
_make_method_dispatch, close_legacy returns zero hits.
Mechanical updates across the test suite to swap legacy
/v1/api/{send,approve,cancel,events,workstreams/close} URLs for the
path-keyed equivalents under /v1/api/workstreams/{ws_id}/<verb>, and
to drop ws_id from request bodies (the path provides it now).
Per file:
- test_session_routes.py: deletes test_close_legacy_mounts_when_handler_provided
(the close_legacy slot is gone); test_send_mounts_post_and_delete_when_dequeue_provided
(added in PR commit 1) stays.
- test_openapi.py: expected-paths set swaps to path-keyed shape;
test_send_endpoint_has_request_body now asserts the OpenAPI for
/v1/api/workstreams/{ws_id}/send.
- test_auth.py / test_auth_identity.py: required_scope and
check_request fixtures swap to path-keyed shape; new tests cover
write/approve/read scope assignment for the path-keyed verbs +
the /node/* proxy mirror.
- test_sdk_server.py / test_sdk_console.py: mock-transport URL keys
swap; bodies drop ws_id.
- test_server_attachments_endpoints.py: ~17 send sites migrated to
/v1/api/workstreams/<ws>/send (a small Python script ran the bulk
rewrite — body ws_id stripped, URL rebuilt).
- test_server_authz.py: cross-tenant approve/close/cancel/events
tests retargeted to path-keyed URLs;
test_events_legacy_query_keyed_url_still_resolves_to_404_for_unknown_ws
renamed to test_events_path_keyed_url_resolves_to_404_for_unknown_ws
with the docstring updated to note the legacy adapter is gone.
- test_close_reason_persistence.py: 7 close sites all swap.
- test_console_routing_proxy.py: route-proxy tests swap to
/v1/api/route/workstreams/{ws_id}/<verb>; the upstream-URL
assertion now reads from .request (route_proxy uses
client.request(method, url, ...) for method passthrough); _wire_proxy
helper installs both .post and .request mocks for compatibility.
- test_route_proxy_audit.py: parametrized URLs migrated;
_make_proxy now also exposes a .request side-effect that delegates
to .post for the same compatibility surface.
- test_api_versioning.py: openapi.json path assertion swaps to the
path-keyed shape.
4557 passing under -m "not live"; ruff + mypy clean.
All in-tree consumers of the legacy /v1/api/send | /approve | /cancel |
events?ws_id= | /workstreams/close URLs now hit the path-keyed shape
under /v1/api/workstreams/{ws_id}/<verb>. Bodies drop ws_id (the path
provides it). The SSE event stream URL likewise moves to the path-keyed
form; channel adapters drop the params={"ws_id": ...} kwarg on
aconnect_sse.
Touched:
- turnstone/ui/static/app.js: 7 call sites (send×3, dequeue, approve,
cancel, close + EventSource SSE URL).
- turnstone/sdk/server.py (Python SDK): close_workstream, send,
approve, cancel, stream_events, send_and_wait's internal SSE
consumer.
- sdk/typescript/src/server.ts: closeWorkstream, send, approve,
cancel, streamEvents + sendAndWait's internal SSE consumer.
- turnstone/sdk/console.py: route_send, route_approve, route_close,
route_cancel — proxy URLs swap to /v1/api/route/workstreams/{ws_id}/<verb>.
route_plan_feedback / route_command remain body-keyed (out of scope).
- turnstone/console/server.py:
- Proxy mount table swaps the four legacy /api/route/{send,approve,
cancel,workstreams/close} mounts for path-keyed equivalents under
/api/route/workstreams/{ws_id}/<verb>; /send accepts both POST
and DELETE for dequeue.
- route_proxy reads ws_id from path_params (with body-fallback for
the surviving plan/command body-keyed mounts), uses
client.request(request.method, ...) so DELETE on /send proxies
through correctly, and audits DELETE-on-/send as a separate
"route.workstream.dequeue" action via _ROUTE_PROXY_AUDIT_ACTIONS.
- Internal `method` variable renamed to `verb` to avoid confusion
with HTTP method now that the two diverge.
- turnstone/channels/_sse.py: SSE URL builder swaps to path-keyed.
- turnstone/channels/{discord,slack}/bot.py: docstring URL updates.
- turnstone/server.py, turnstone/core/session_worker.py,
turnstone/sdk/events.py, turnstone/api/server_spec.py: comment /
docstring URL updates only.
Test fixtures still reference legacy URLs and will be swapped in step
5 of this PR.
- WRITE_PATHS / APPROVE_PATHS in turnstone/core/auth.py drop the four
legacy literal entries (/api/send, /api/cancel, /api/workstreams/close,
/api/approve). The path-keyed verb match for write expands from
{delete, open, refresh-title, title, attachments} to also include
{send, cancel, close}; a sibling branch maps POST /workstreams/{ws_id}/approve
to the approve scope, and a DELETE branch maps DELETE
/workstreams/{ws_id}/send (dequeue) to write. The /node/* proxy
block mirrors all four expansions so the console routing proxy
stays in lockstep.
- server_schemas.py drops the body-keyed ws_id field from SendRequest,
ApproveRequest, CancelRequest. CloseWorkstreamRequest deleted in
full (its only field was ws_id, now provided by the path).
- server_spec.py: drops CloseWorkstreamRequest from imports and
_ALL_MODELS, swaps the five legacy EndpointSpec entries to their
path-keyed equivalents (POST/DELETE workstreams/{ws_id}/send, POST
/approve, POST /cancel, POST /close, GET /events). Catalogue retains
/api/plan and /api/command unchanged (out of scope).
Tests still reference the legacy URLs and will fail at this commit;
test fixture updates land in step 5 of this PR. Step 4 swaps the
UI / SDK / console proxy / channels callers next.
Removes the pre-1.5 interactive URL family that mounted body- and
query-keyed shapes on top of the lifted path-keyed handlers via
make_legacy_body_keyed_adapter / make_legacy_query_keyed_adapter.
Path-keyed equivalents under /v1/api/workstreams/{ws_id}/<verb>
already serve every consumer; coord never used the legacy URLs.
Removed:
- make_legacy_body_keyed_adapter / make_legacy_query_keyed_adapter
from turnstone/core/session_routes.py.
- _make_method_dispatch from turnstone/server.py (zero callers
after legacy /api/send POST+DELETE block goes — its only purpose
was to bridge that single dual-method legacy URL).
- 5 legacy Route mounts in turnstone/server.py:
/api/events?ws_id, /api/send POST+DELETE, /api/approve, /api/cancel,
/api/workstreams/close.
- close_legacy field on SharedSessionVerbHandlers and its mount in
register_session_routes — the only surviving body-keyed slot in
the registrar, no longer needed.
Tightened make_dequeue_handler to read ws_id from the path only;
the body-fallback existed solely for the legacy DELETE /api/send
path and is now dead.
Test-suite updates and consumer call-site swaps (UI / SDK /
console proxy / channels) follow in subsequent commits in the same
PR — main stays broken across this commit until step 4 lands.
External SDK consumers on stable 1.0/1.3/1.4 calling these URLs
will receive 404s on upgrade to 1.5.0; CHANGELOG breaking-change
call-out lands with the docs commit.
Pre-flight for the legacy URL adapter removal: the path-keyed
`/v1/api/workstreams/{ws_id}/send` route only mounted POST today;
the dequeue handler was reachable only via the legacy
`DELETE /v1/api/send` body-keyed URL through `_make_method_dispatch`.
Add a new `dequeue: Handler | None = None` slot on
`SharedSessionVerbHandlers` next to `send`, mounted as a second
`Route` on the same path with `methods=["DELETE"]` (two distinct
Routes rather than collapsing methods on one Route — different
handler callables, and collapsing would force the same
method-dispatch wrapper this cleanup is tearing out).
Wire `dequeue=dequeue_handler` in `turnstone/server.py`'s
`SharedSessionVerbHandlers(...)` call so DELETE on the path-keyed
shape works in the same merge as the legacy mount removal.
Adds a regression-locking test covering both the POST+DELETE and
the dequeue-alone cases.
Switch fenced-code language tag from `json` to `http` on the seven
example blocks that mix an HTTP request line with a JSON body
(/trust, /restrict, /stop_cascade, /close_all_children, /approve,
/cancel, /close). Pure JSON response blocks stay tagged `json`.
Pre-existing pattern in the doc that Copilot flagged on the lines
this PR touched; fixed across all instances for consistency. No
content / URL changes — only fence-tag adjustment for correct
syntax highlighting.
The Stage 2 verb-shape lift converged coord and interactive on the
unified /v1/api/workstreams/{ws_id}/<verb> URL tree; the
/v1/api/coordinator/* tree was removed in P0. Two docs still
documented the pre-lift surface:
- coordinator-api-tour.md (the integrator's lifecycle walk-through):
rewrites all 9 step URLs to the post-lift paths, keeps a one-block
callout noting the historical /v1/api/coordinator/* tree and why
it converged, and drops the operation-id column (operation ids
shifted with the URL move and are now best looked up live via
/openapi.json + Swagger UI rather than baked into prose).
- bulk-endpoints.md (the cascade-mutation shape contract): two table
rows for stop_cascade / close_all_children fixed.
No code changes. CHANGELOG entry kept implicit since this is doc-only
and the URL convergence itself was already documented under the P0
verb-lift CHANGELOG block.
The /v1/api/dashboard endpoint was the last workstream-listing surface
keyed on `id` rather than `ws_id`. The Stage 2 list-verb lift converged
the active list (`/v1/api/workstreams`) and saved list
(`/v1/api/workstreams/saved`) on `ws_id` but explicitly left dashboard
alone to keep that PR's diff focused. This lands the same rename on
the remaining endpoint so v1 row shape is consistent across the family.
Scope kept narrow:
- Pydantic `DashboardWorkstream` and TS SDK `DashboardWorkstream`
interface both rename `id: str/string` → `ws_id`.
- The bundled web UI (`turnstone/ui/static/app.js`) is the only consumer
reading `dashboard.workstreams[].id` and is updated atomically.
- Console `_fetch_live_block` (cluster-inspect's projection over a
remote node's dashboard payload at `turnstone/console/server.py`)
flips its `entry.get("id")` lookup to `entry.get("ws_id")`.
- Drive-by: stale `id` example in `docs/api-reference.md` for the
earlier `/v1/api/workstreams` rename also fixed.
`_build_node_snapshot` (the global-events SSE node_snapshot payload
consumed by the cluster collector) deliberately stays on `id` — it's
part of a separate cluster-row family (collector → cluster_workstreams
→ console UI) that is internally consistent on `id` and would need its
own coordinated sweep. CHANGELOG documents the bounded blast radius.
Tests: 4554 passing (-m "not live"). ruff + mypy clean.
* feat(console): coord rich ws_state payload + live activity broadcast (Stage 2 follow-up)
Pre-lift coord's cluster broadcast was state-only — the dashboard's
coord rows showed the state column flipping but ``tokens`` /
``context_ratio`` / ``activity`` / ``content`` were all hardcoded
to zero / empty. The lift makes coord populate the same per-ws
metric fields interactive does and broadcasts them through the
cluster collector with the rich kwargs.
**Architecture changes:**
- Lift ``on_status`` / ``on_content_token`` / ``on_thinking_start`` /
``on_thinking_stop`` / ``on_stream_end`` / ``on_tool_result`` /
``on_reasoning_token`` / ``on_tool_output_chunk`` / ``on_info`` /
``on_error`` from ``WebUI`` to :class:`SessionUIBase` as base
implementations. Coord inherits the bodies; the per-ws metric
fields it had at the base but never populated now flow.
- ``WebUI`` keeps overrides for ``on_status`` / ``on_tool_result`` /
``on_error`` to layer Prometheus ``_metrics.record_*`` calls
on top of ``super()`` (node-only — the console isn't a node).
``WebUI._broadcast_state`` now uses the new
:meth:`SessionUIBase.snapshot_and_consume_state_payload` helper
for the rich-payload snapshot read.
- ``ConsoleCoordinatorUI`` adds a ``_broadcast_activity`` override
that calls the new
:meth:`ClusterCollector.update_console_ws_activity` (in-memory
pseudo-node row update; named ``update_*`` rather than ``emit_*``
to flag the no-fanout asymmetry vs. the rest of the
``emit_console_ws_*`` family).
- ``coord_adapter.emit_state`` reads ``ws.ui``'s snapshot under
``_ws_lock`` and passes the rich kwargs to the extended
:meth:`ClusterCollector.emit_console_ws_state`. Defensive when
``ws.ui is None`` mid-eviction (broadcasts state-only).
- ``coord_endpoint_config`` wires a new ``_coord_spawn_metrics``
hook so per-spawn ``_ws_messages`` / ``_ws_turn_tool_calls``
bookkeeping fires on coord too.
- ``_MAX_TURN_CONTENT_CHARS`` moved from ``turnstone.server`` to
``turnstone.core.session_ui_base`` so coord enforces the same
per-turn content cap.
**Three observable behaviour changes** (CHANGELOG-callout-worthy):
- Coord persists ``usage_event`` storage rows on every status
emission (governance dashboards / token-spend queries gain
coord visibility).
- Coord broadcasts live activity transitions to the cluster
collector (dashboard's coord rows show activity ticks between
state changes the same way interactive does), with last-emitted
dedup so a tool-heavy turn's repeated ``activity=""`` clears
don't hammer the collector lock.
- Cluster ``cluster_state`` events for coord rows now carry
non-zero ``tokens`` / ``content``. Frontend rendering that
conditionally hid these on coord can drop the branch.
**Tests:** 23 new tests in ``tests/test_coord_rich_ws_state_payload.py``
(per-ws metric writes, snapshot helper drain semantics +
single-lock-acquisition, adapter rich-payload pass-through +
None-UI defensive handling, activity broadcast wire + dedup +
failure swallow + no-op-when-collector-unset, spawn_metrics
hook, concurrent-writes-during-snapshot stress with reader
cycling through running/idle/error so drain branches actually
run, on_stream_end activity-clear pin). Plus WebUI override
regression tests confirming ``_metrics.record_*`` still fires
on top of the lifted bodies. Existing
``tests/test_webui_content.py`` updated to import
``_MAX_TURN_CONTENT_CHARS`` from its new home;
``tests/test_coordinator_adapter.py`` updated to expect the
rich-payload kwargs (default zeros) on
``emit_console_ws_state``. Total: ``4491 → 4514``.
``ruff check`` clean, ``mypy`` clean on touched files.
**/review pipeline** (4 finders → verify → dedupe) caught 14
findings → 12 unique (3 collapsed as duplicates of the lockless
``on_content_token`` writer):
- bug-1 Minor: ``on_status`` regressed coord's defensive
``usage.get(...)`` indexing → restored ``.get(..., 0)`` for
``prompt_tokens`` / ``completion_tokens`` on both base + WebUI
override.
- bug-2 Nit: concurrent-snapshot reader only used ``"running"`` →
cycled through ``("running", "idle", "error")`` so drain
branches run; also captures + re-raises thread exceptions
instead of silently passing.
- bug-3 + sec-2 + perf-3 Nit (merged): ``on_content_token``
mutated ``_ws_turn_content`` lockless while the snapshot drained
under lock → wrapped the cap-check + append + size-update in
``_ws_lock``.
- perf-2 Minor: collector lock contention from per-event activity
broadcasts → cached last-emitted ``(activity, activity_state)``
on the UI; subsequent identical ticks return early without
acquiring the collector lock.
- perf-4 Nit: join-under-lock in snapshot helper → swap-then-join
pattern (capture list reference under lock, reassign to empty,
join the captured list outside the lock). Halves the lock
hold and decouples the join walk from concurrent appenders.
- q-1 Minor: ``emit_console_ws_activity`` was misleading (no
``_fanout`` call, unlike the rest of the ``emit_console_ws_*``
family) → renamed to ``update_console_ws_activity`` + docstring
call-out for the asymmetry.
- q-2 + q-3 Minor/Nit: stale docstrings on
``coordinator_ui.py`` (still claimed "no per-node metrics —
Phase D") and ``_interactive_spawn_metrics`` (still claimed
"counters live on WebUI only") → both updated to reflect the
lifted base class + coord's new hook.
- q-4 Nit: broken Sphinx cross-ref
``:meth:\`_snapshot_and_consume_state_payload\``` → dropped
the leading underscore.
- q-5 Nit: missing ``test_coord_on_stream_end_clears_activity``
→ added.
**Two findings explicitly deferred** (out-of-scope follow-ups,
documented in CHANGELOG):
- perf-1: synchronous ``record_usage_event`` INSERT on coord
worker thread per status tick. Parity with WebUI is the lift's
goal; if throughput becomes a concern, batch usage_event writes
on a background flusher (would apply to both kinds).
- sec-1: coord assistant content now flows on the cluster SSE
stream, which has no per-user filter today. Pre-existing
exposure for interactive ``cluster_state`` events; the lift
extends to coord rows. Proper fix needs SSE auth gating
(``admin.cluster.inspect``) or per-listener user_id filtering
— separate security project, doesn't gate this lift.
* fix(console): apply review feedback on PR #420
Three review findings, all confirmed against source:
1. **Copilot — dedup-state-vs-failure race in `_broadcast_activity`**
(correctness bug): pre-fix ``self._last_broadcast_activity = current``
was assigned inside the ``_ws_lock`` block BEFORE the collector call.
If the collector raised mid-broadcast, the exception was swallowed
but the dedup state was already updated, so subsequent identical
activity ticks would be deduped and never retried — leaving the
dashboard's coord row stranded at the pre-failure activity until
the activity actually changed.
Fix: move the dedup-state update OUT of the lock and place it AFTER
a successful collector call. On failure, ``_last_broadcast_activity``
stays unchanged so the next identical tick retries. Two new
regression tests pin both the failure-recovery (``test_coord_ui_
broadcast_activity_failure_does_not_strand_dedup``) and the
happy-path dedup behavior (``test_coord_ui_broadcast_activity_
dedup_skips_identical_after_success``).
2. **Copilot — stale `emit_console_ws_activity` reference in
CHANGELOG**: the method was renamed to ``update_console_ws_activity``
per /review's q-1 finding before the original commit landed, but the
CHANGELOG entry was written ahead of the rename. Updated to match
the actual API + added the no-fanout asymmetry rationale inline so
readers don't have to chase the method name.
3. **code-quality bot ×2 — `except BaseException` in test workers**:
the concurrent-snapshot stress test caught thread-worker exceptions
with ``except BaseException`` (with a noqa to suppress BLE001).
``BaseException`` is overkill for a thread worker — ``SystemExit``
/ ``KeyboardInterrupt`` are main-thread signals and ``Exception``
is the right scope. Narrowed to ``except Exception`` on both
workers; ``writer_exc`` / ``reader_exc`` types narrowed from
``list[BaseException]`` to ``list[Exception]``.
Tests: ``4514 → 4516`` (+2 regression tests for the dedup race fix).
``ruff check`` clean, ``mypy`` clean. No code-path changes outside
the dedup-state placement; the rich-payload broadcast surface is
unchanged.
Server-side history endpoint declared ``error_codes=[404]`` but the
lifted ``make_history_handler`` factory can also return:
- ``400`` on empty ``ws_id`` (defensive — Starlette routing makes
it unreachable in practice, but the factory has the branch).
- ``500`` on the ``cfg.list_kind is None`` misconfig gate added in
the /review fix-up (defense-in-depth fail-loud; both production
cfgs wire ``list_kind`` so the gate doesn't fire today).
- ``503`` via ``cfg.manager_lookup`` when the kind's manager isn't
available (interactive's lookup never returns 503; coord's can).
Updated ``server_spec.py`` to ``[400, 404, 500, 503]`` per Copilot's
suggestion — matches the existing detail entry's shape so the two
endpoints document the same possible-error envelope.
Caught the parallel asymmetry on ``console_spec.py``: history was
``[403, 404, 503]`` but the lifted factory's misconfig + empty-
ws_id branches reach coord too. Updated to
``[400, 403, 404, 500, 503]`` — same factory body, same possible
responses, plus ``403`` from coord's ``admin.coordinator``
permission gate.
Regenerated ``openapi-{server,console}.json``. No code changes;
spec metadata only. Tests + lint + mypy unchanged.
Last verb-shape lift before v1.5.0 stable can tag. Adds two new
factories to ``turnstone/core/session_routes.py``:
- ``make_history_handler(cfg)`` — body lifted from coord's
``coordinator_history`` near-verbatim. ``?limit=`` query param
defaults to 100, clamps to [1, 500], malformed values fall back
to 100. Storage operations (``get_workstream`` on the
storage-fallback path, ``load_messages`` for the row read) now
run via ``asyncio.to_thread`` (was inline pre-lift on coord).
- ``make_detail_handler(cfg)`` — body lifted from coord's
``coordinator_detail``. Lazy-rehydrates a closed/evicted
workstream via ``mgr.open()`` on miss; mirrors
:func:`make_open_handler`'s exception envelope (``ValueError``
→ 503 with the session-factory's remediation text; bare
``Exception`` → correlation_id'd 500 with the per-kind noun
via ``cfg.audit_action_prefix``).
NO new ``SessionEndpointConfig`` fields — the factories reuse
``permission_gate``, ``manager_lookup``, ``not_found_label``,
``audit_action_prefix``, and (for history's storage-fallback
kind check) ``list_kind`` — all already wired by both production
lifespans for the list/saved factories.
Coord side: ``coordinator_history`` and ``coordinator_detail``
standalone handler bodies removed from ``console/server.py``;
``register_session_routes`` now wires
``history=make_history_handler(coord_endpoint_config)`` and
``detail=make_detail_handler(coord_endpoint_config)``.
Interactive side: GAINS both endpoints as a feature gain. Pre-lift
interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no
``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to
subscribe to ``/events`` SSE just to read display fields or
message rows. The same lifted factories are wired with the
interactive endpoint config; cross-kind isolation is preserved on
both sides (history via ``cfg.list_kind`` storage-fallback gate
+ fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal
kind check).
Pydantic schemas: ``CoordinatorDetailResponse`` /
``CoordinatorHistoryResponse`` removed from ``console_schemas.py``;
``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added
to ``server_schemas.py`` (mirrors the list lift's pattern for
``WorkstreamInfo``). Both server and console OpenAPI specs
reference the unified schemas; ``server_spec.py`` gains
``EndpointSpec`` entries for the new interactive endpoints. TS
SDK gains both interfaces in ``sdk/typescript/src/types.ts``;
``openapi-{server,console}.json`` regenerated.
Tests: 6 new coord regression/parity tests in
``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404
on storage fallback, storage-only history, detail 503 on
session-factory misconfig, detail 500 with correlation_id on
unexpected rehydrate failure, history swallows
``load_messages`` exception → 200 with empty messages). 10 new
interactive parity tests in ``test_workstream_endpoints.py``
(``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new
openapi spec test pinning the server-side ``?limit=`` query param.
Total: ``4490 → 4491`` after the new exception-swallow
regression test landed. ``ruff check`` clean, ``mypy`` clean on
touched files.
/review pipeline (4 finders → verify → dedupe) caught 1 Minor
defense-in-depth (bug-1/sec-1, merged: ``make_history_handler``
fail-closed gate when ``cfg.list_kind is None``, mirroring
``make_saved_handler``'s same gate) + 1 Minor test-helper rename
(q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``)
+ 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK
mention, q-4 missing exception-swallow regression test, q-5
misleading test comment) — all addressed in the same commit.
Three docstring + CHANGELOG drift items from the post-review
M3 + Mi1 fixes:
- ``make_list_handler`` docstring referenced ``cfg.list_resolve_title``
(singular) but the field renamed to ``list_resolve_titles``
(bulk variant) when the N+1 fix landed. Updated to the plural
name + a one-line note about the bulk SELECT pattern.
- ``make_saved_handler`` docstring still claimed kind was derived
from ``cfg.audit_action_prefix`` string-compare. The Mi1 fix
replaced that with the explicit ``cfg.list_kind`` field +
fail-loud-on-missing semantic; docstring now describes the
current contract.
- CHANGELOG ``[Unreleased]`` entry said "Three new
``SessionEndpointConfig`` fields" and listed the singular
``list_resolve_title`` wired to ``get_workstream_display_name``.
Updated to "Four" + the bulk plural names + the new
``list_kind`` field with its rationale (distinct from
``audit_action_prefix``; fail-loud on misconfig).
The fourth review comment — code-quality bot flagging the ``...``
ellipsis body on the new ``get_workstream_display_names`` Protocol
method as "statement has no effect" — is a false positive.
``...`` is the canonical Protocol method body throughout
``turnstone/core/storage/_protocol.py`` (every other method uses
it). Refuting; the file's pattern wins over the bot's per-method
suggestion.
No code changes; docstring + CHANGELOG only. Tests + lint + mypy
unchanged.
New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)``
factories in ``turnstone/core/session_routes.py`` replace four
pre-lift bodies (interactive ``list_workstreams`` +
``list_saved_workstreams``; coord ``coordinator_list`` +
``coordinator_saved``). Same factory + capability-flag pattern as
the merged cancel / open / events / create lifts.
Four new ``SessionEndpointConfig`` fields:
- ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup
``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires
``get_workstream_display_names`` (new bulk helper added on the
storage layer + memory.py); the lifted body resolves every active
row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the
pre-lift N+1 (one SELECT per row).
- ``list_kind: WorkstreamKind | None`` — explicit kind classifier
for the saved-list storage filter. Replaces the initial draft's
``audit_action_prefix == "coordinator"`` string compare which
would have silently leaked INTERACTIVE rows for any future kind
whose audit prefix didn't match. Required when a kind mounts
list/saved; misconfig surfaces as a 500 with a clear log line.
- ``saved_state_filter: str | None`` — coord wires ``"closed"``;
interactive wires ``None``.
- ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only
defence-in-depth filter that excludes ws_ids in the warm pool.
Behaviour changes (all observable in CHANGELOG):
- **Active-list row shape converges on always-include** ``{ws_id,
name, state, kind, parent_ws_id, user_id}``. Interactive renames
``id`` → ``ws_id``; both kinds populate every field (coord adds
kind + parent_ws_id; interactive adds user_id).
- **Top-level response key converges on ``"workstreams"``** on
both endpoints. Coord ``coordinators`` key removed — coord is a
1.5.0aN-only surface (never shipped stable) so the convergence
has no compat shim; SDK / frontend consumers swap once.
- **Storage + manager-lock work moved off the event loop on
interactive**. ``list_workstreams_with_history`` runs through
``asyncio.to_thread`` on both kinds (matches coord's pre-existing
perf-2 pattern from the saved-coordinators review); ``mgr.list_all``
+ per-row work also offloaded.
- **N+1 storage round-trips on /v1/api/workstreams eliminated**.
Pre-lift interactive resolved the alias for every active row in a
separate SELECT (up to 50 round-trips per dashboard refresh on a
saturated node). Lifted body issues one bulk SELECT.
Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``,
``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and
``CoordinatorListResponse`` removed (folded into the unified
``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec
snapshots regenerated. TS SDK types updated (``WorkstreamInfo``
interface gains ws_id + the always-include fields); TS test
mock + assertion updated to match.
``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope
and still returns rows keyed on ``id``. Tracked as a separate
cleanup PR (tombstone-note added at the dashboard handler).
/review pipeline run; the four Major findings + one Minor + six
nits all addressed in the same commit:
- M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) →
renamed + fields added.
- M2: TS SDK test masked the type-mismatch with stale mock → updated.
- M3: N+1 alias resolution on active list → bulk
``get_workstream_display_names`` helper + ``list_resolve_titles``
bulk cfg hook.
- M4: Missing interactive parity regression test for unified row
shape → mirror of coord's added in test_server_authz.py.
- Mi1: ``audit_action_prefix`` string-compare deriving kind →
explicit ``cfg.list_kind: WorkstreamKind`` field.
- Six nits: redundant inner asyncio import, forward-ref quotes on
Awaitable, duplicated frontend comments, dashboard ``id`` field
has no tombstone-note, empty-coord_mgr short-circuit on
``saved_loaded_lookup``.
4512 tests passing; ruff + mypy clean.
* refactor(core): defer emit_created on SessionManager.create + commit_create / discard pair
Eliminates the phantom create→close pair on coord rollback that was
documented as a known limitation in PR #416. The pair surfaced on the
cluster events stream when a multipart workstream-create request
failed attachment validation: coord's ``mgr.create`` fired
``emit_created`` synchronously, then the rollback called
``mgr.close`` which fired ``emit_closed``. Cluster consumers had to
reconcile via the collector's diff path. Post-fix, a rejected upload
produces zero events.
API changes on ``SessionManager``:
- ``create(..., defer_emit_created: bool = False)`` — when True,
skip the trailing ``emit_created`` so the caller can run additional
post-create work (attachment validation in the lifted HTTP handler)
before advertising the workstream. Default preserves the existing
"advertise immediately" contract for direct callers (test fixtures,
CLI REPL, channel adapters).
- ``commit_create(ws)`` — fires the deferred ``emit_created`` event
after the caller's post-create work confirms the workstream should
be advertised. Synchronous; the wrapped work is in-memory and
non-blocking on every kind (interactive: documented no-op stub;
coord: dict updates under a lock + ``queue.put_nowait`` fan-out).
- ``discard(ws_id)`` — releases the in-memory slot + cleans up the UI
WITHOUT firing ``emit_closed``. Distinct from ``close`` which
advertises the transition; ``discard`` is for the rollback case
where the workstream's existence was never advertised. Storage-row
deletion stays a separate concern (caller invokes
``delete_workstream``), mirroring ``mgr.create``'s split between
slot reservation and ``register_workstream``.
Caller-bug detection: ``Workstream._emit_created_fired`` is set
inside ``create`` (non-deferred path) and ``commit_create``;
``discard`` logs ``session_mgr.discard.after_emit_created`` warning
when invoked on an already-advertised workstream. Slot is still
released so capacity isn't stranded.
Lifted ``make_create_handler`` updated to use the deferred bracket:
pass ``defer_emit_created=True``, validate uploaded attachments,
then ``mgr.commit_create(ws)`` on success / ``mgr.discard(ws.id)``
on failure. Ordering invariants (``commit_create`` BEFORE
``audit_emit`` and ``post_install`` so any state events the worker
fires reach the cluster collector for an already-known ws_id) are
documented in the handler docstring.
Tests:
- 5 new ``SessionManager`` unit tests (defer skips emit, commit
fires it, commit no-ops without emitter, discard releases without
emit_closed, discard returns False on unknown id).
- 2 caller-bug regression tests (commit_create after discard pins
the silent re-emit behaviour; discard after non-deferred create
asserts the warning fires + slot still releases).
- 1 coord regression test asserting the cluster collector sees zero
events when attachment validation fails.
``/review`` pipeline run; M1 (test gap on caller-bug paths) +
Mi1 (no runtime guard for already-advertised) + Mi2
(``_make_manager`` event_emitter override) + Mi3 / N2 (duplicated
comments + ordering invariant) + N1 (drop ``to_thread`` on
``commit_create``) all addressed.
4509 tests passing; ruff + mypy clean.
* fix(core): apply Copilot + code-quality review feedback on PR #417
Copilot review:
- ``Workstream._emit_created_fired`` comment claimed the flag was
"set under the manager's _lock-protected emit", but the actual
ordering set it OUTSIDE the lock. Comment updated to describe the
real synchronization (non-deferred ``create`` sets it immediately
before ``emit_created``; ``commit_create`` sets it under the
manager lock alongside the tracked-ws check).
- ``commit_create`` had no guard against duplicate calls,
post-discard calls, or calls on workstreams not tracked by this
manager — any of those would have fired duplicate or phantom
``ws_created`` events. Added a guard symmetric to ``discard``'s
after-emit warning: under ``self._lock``, check ``_emit_created_fired``
+ ``_workstreams.get(ws.id) is ws``, no-op + log a warning
(``session_mgr.commit_create.already_fired`` /
``session_mgr.commit_create.untracked``) on either failure. The
emit itself still runs outside the lock so coord's collector
fan-out doesn't couple to the manager mutex.
- ``test_commit_create_after_discard_is_caller_bug_no_op`` was
internally inconsistent — name + docstring said "must not re-emit"
but the assertion expected the re-emit. Renamed to
``test_commit_create_after_discard_is_no_op`` and updated to
assert the new no-op + warning behaviour.
New test ``test_commit_create_is_idempotent_on_duplicate_call``
pins the second-commit-call code path: exactly one ``ws_created``
event fires, second call short-circuits via the guard with a
``commit_create.already_fired`` warning.
Code-quality bot review (3 findings, identical pattern):
- Three test ``assert`` statements wrapped side-effecting calls
(``assert mgr.discard(ws_id) is True/False``); under ``python -O``
the asserts strip and the side-effect strips with them. Refactored
all three to assign the result to a local first, assert on the
local. No behaviour change.
4510 tests passing; ruff + mypy clean.
Coord initial-message + create-time-attachments coordination:
- ``CoordinatorAdapter.send`` gains optional ``attachments`` + ``send_id``
kwargs so the worker dispatched at create time can carry the uploaded
files onto the first turn. Mirrors interactive's pre-existing
worker-thread pattern. The ``send_id`` reservation token soft-locks
the rows; the adapter's failure path unreserves so a worker crash
returns them to pending.
- ``_coord_create_post_install`` reserves any uploaded ``attachment_ids``
via the lifted ``reserve_and_resolve_attachments`` helper before
dispatching through the adapter — closes the parity gap with
interactive's create-with-attachments+initial_message flow.
- ``_reserve_and_resolve_attachments`` lifted from ``turnstone/server.py``
to ``turnstone/core/attachments.py`` as ``reserve_and_resolve_attachments``
so both processes use one kind-agnostic implementation.
Copilot review fixes on PR #416:
- Skill lookup now calls ``storage.get_prompt_template_by_name`` directly
rather than going through ``turnstone.core.memory.get_skill_by_name``;
that helper swallows storage exceptions into ``None`` which would have
masked outages as the 400 "Skill not found" branch. Calling storage
directly lets exceptions bubble to the lifted body's correlation_id'd
500 path so operators chasing skill-related reports can distinguish
real misses from registry outages.
- ``_interactive_create_build_kwargs`` /
``_coord_create_build_kwargs`` thread ``skill_data["name"]`` (the
canonical row name) into ``mgr.create`` instead of the raw
``body["skill"]`` value. Pre-fix a whitespace-padded request body
``"skill": " my-skill "`` would have persisted the dirty name even
though the lookup ran on the stripped key.
- ``make_create_handler`` docstring corrected: audit-emit failures
return 200 (not 201).
- ``_audit_workstream_created`` docstring corrected: factory keeps the
successful 200 response on audit-emit failure (was 201).
New regression test:
``test_create_with_multipart_attachments_and_initial_message_reserves``
asserts attachments are reserved (not pending) when both
``initial_message`` and uploads land in the same coord create request.
Updated ``_SendSession`` stub in ``test_coordinator_adapter.py`` to
match the new ``send`` / ``queue_message`` signatures.
4501 tests passing; ruff + mypy clean.
New ``make_create_handler(cfg, *, audit_emit=None)`` factory in
``turnstone/core/session_routes.py`` consumes five new ``SessionEndpointConfig``
fields (``create_supports_attachments``, ``create_supports_user_id_override``,
``create_validate_request``, ``create_build_kwargs``, ``create_post_install``)
and replaces both ``create_workstream`` and ``coordinator_create`` bodies.
Same factory + capability-flag pattern as the merged cancel / open / events
lifts. ``_validate_and_save_uploaded_files`` lifted to
``turnstone.core.attachments`` so both processes call one kind-agnostic
implementation.
Coord parity gains (§ Post-P3 reckoning item #1 + carry-forward):
- Create-time attachments: multipart parsing, validate+save+rollback,
``attachment_ids`` on the response. Coord adapter ``send`` doesn't yet
reserve attachments at create time, so the rows save as pending and the
next ``/send`` picks them up via the standard send-with-attachments path.
- Disabled-skill rejection (matches interactive's pre-lift gate).
- Always-include response shape ``{ws_id, name, resumed, message_count,
attachment_ids}`` populated with default ``False``/``0``/``[]`` on the
fields coord doesn't fill.
- 200 status (was 201).
- Audit-emit failures swallow + warning log instead of 500.
Both kinds converge on the manager-at-capacity 429, factory-misconfig 503,
and correlation_id'd 500 for unexpected ``mgr.create`` failure (interactive
lifted up to coord's safer error envelope).
Three /review fixes folded in:
- ``notify_targets`` malformed input gates at the validator (400) instead
of bubbling out of post_install as a 500 — pre-fix the workstream had
already been created + audited + broadcast by the time the validation
raised.
- Skill-lookup storage failures now share the correlation_id'd 500 path
with ``mgr.create`` (was masquerading as 400 "Skill not found").
- Whitespace-only ``skill`` field treated as empty (matches pre-lift coord).
CHANGELOG entry under [Unreleased] documents every observable behaviour
change. OpenAPI spec regenerated. Three new coord regression tests
(create-time-attachments save pending rows, always-include parity fields,
disabled-skill rejection) plus one interactive regression test
(notify_targets 400). 4500 tests passing.
* refactor(core): lift events verb body across both kinds (Stage 2 verb lift)
The interactive ``GET /v1/api/events?ws_id=...`` and coord
``GET /v1/api/workstreams/{ws_id}/events`` SSE handlers now share
one body via ``make_events_handler(cfg)``. Per-kind divergence
captured by two new ``SessionEndpointConfig`` fields:
* ``events_replay: EventsReplay | None`` — Protocol-typed callback
that yields the kind-specific initial replay payload. Interactive
wires ``_interactive_events_replay`` (connected + status + history
+ pending_approval + cached intent verdicts + pending_plan_review);
coord wires ``_coord_events_replay`` (just pending_approval +
pending_plan_review). The lifted body iterates the callback
before starting the live event loop.
* ``sse_executor_lookup: SseExecutorLookup | None`` — per-kind
executor for the live loop's blocking ``client_queue.get``.
Interactive returns the dedicated 200-thread ``sse_executor``
from app state so SSE polling stays isolated from every other
``asyncio.to_thread`` caller in the process; coord returns
``None`` and the lifted body falls through to the default executor.
Also adds ``make_legacy_query_keyed_adapter(handler)`` (sister to
``make_legacy_body_keyed_adapter`` from earlier lifts): reads
``ws_id`` from the query string and splices into ``request.path_params``
before delegating to the lifted body. Preserves the
``GET /v1/api/events?ws_id=...`` legacy URL shape so any 1.x SDK
consumer keeps working.
Old ``events_sse`` (server.py) + ``coordinator_events``
(console/server.py) bodies deleted.
Two convergence wins for coord:
* **SSE connect/disconnect metrics** — pre-lift coord didn't record
per-stream metrics; the lifted body always calls
``metrics.record_sse_connect()`` / ``record_sse_disconnect()``,
giving the cluster dashboard the same per-stream observability
interactive's had since 1.0.
* **Both kinds now check ``request.is_disconnected()`` AND the
``ws_closed`` event** to terminate. Pre-lift interactive relied
solely on ``ws_closed`` (which never fires if the client just
goes away without closing the workstream); pre-lift coord relied
solely on ``is_disconnected``. The lifted body uses both.
One observable shape change for coord callers: the lifted body
returns 409 ``"session has no UI"`` when ``ws.ui`` is missing
(placeholder / build-failed UI), matching pre-lift coord.
Pre-lift interactive returned 404 in this case; the lift converges
on 409 because the workstream EXISTS (404 would imply it doesn't).
Item #2 from § Post-P3 reckoning (rich ``ws_state`` payload parity
for coord) split out during scoping — touches different files
(``coordinator_ui.py`` + ``collector.py`` + ``session_ui_base.py``)
with different reviewer concerns. Tracked as standalone follow-up
``feat/coord-rich-ws-state-payload``.
Two /review fixes folded in:
* **Dedicated SSE thread pool restored.** Initial draft used
``asyncio.to_thread`` (default executor, ~32 workers). Pre-lift
interactive deliberately used a dedicated 200-thread
``sse_executor`` to avoid pool starvation; the
``sse_executor_lookup`` cfg field above restores that isolation.
* **5s poll timeout restored.** Initial draft shortened to 1s,
multiplying thread-wakeup rate 5x while the pool was already
starving. ``is_disconnected()`` between polls covers cancel-
detection latency.
Plus minor cleanups: stale ``coordinator_events`` comment
references in coordinator.js refreshed; ``TestInteractiveEventsLifted``
gets a ``_make_interactive_replay_mocks`` fixture so per-test
intent stays clear; live-loop coverage gap documented in the
test class docstring.
Lint + mypy clean. 4497 tests passing (+8 new events tests).
* fix(core): stream events replay from inside the generator instead of pre-building
PR #415 review caught that ``make_events_handler`` pre-built the
full replay payload (``connected`` + ``status`` + ``history`` +
pending prompts) into a list before constructing the
``EventSourceResponse``. Two real costs:
* **TTFB delay** — the client saw nothing until the heaviest
replay event finished serialising (``_build_history`` on a
long-running interactive workstream can take 10s of ms). With
pre-build, the ``connected`` event was buried at the end of
the materialisation pass instead of streaming first.
* **Listener-queue accumulation** — registering the per-UI
listener BEFORE building the replay let live events queue
during the build window. On a chatty mid-generation
workstream that window can fill the 500-slot listener queue
and drop events before the live loop starts draining.
Fix: iterate ``cfg.events_replay`` inside the async generator
so each event ships as soon as the callback yields it. The
observational-failure swallow semantics are preserved by
wrapping the iteration in the same try/except + log.debug as
before — partial replay is still acceptable; the live loop
continues either way.
Resolves the Copilot review thread on PR #415. Lint + mypy
clean. 4497 tests passing (no test changes — the replay
callbacks themselves are unchanged; only the lifted body's
consumption pattern flipped from eager-build to lazy-stream).
* refactor(core): lift open verb body across both kinds (Stage 2 verb lift)
The interactive ``POST /v1/api/workstreams/{ws_id}/open`` and coord
``POST /v1/api/workstreams/{ws_id}/open`` handlers now share one
body via ``make_open_handler(cfg, *, audit_emit=None)``. Per-kind
divergence captured by two new ``SessionEndpointConfig`` fields:
* ``open_resolve_alias: AliasResolver | None`` — interactive wires
``resolve_workstream`` so callers can pass user-friendly aliases
in the path param. Coord wires ``None``.
* ``open_post_load: OpenPostLoad | None`` — interactive wires
``_interactive_open_post_load`` (display-name sync + UI replay
via ``clear_ui`` + history + handler-side ``ws_created`` enqueue
onto the global SSE queue). Coord wires ``None`` and relies on
the cluster collector fan-out from
``CoordinatorAdapter.emit_rehydrated``.
Plus an optional ``audit_emit`` parameter (interactive wires
``_audit_workstream_opened``; coord wires ``None`` — coord doesn't
audit open today). Old ``open_workstream`` (server.py) +
``coordinator_open`` (console/server.py) bodies deleted.
**Load-bearing fix** (§ Post-P3 reckoning item #3 from the planning
docs): pre-lift interactive's ``open_workstream`` called
``mgr.create(ws_id=resolved_id)`` + ``ws.session.resume(...)`` to
rehydrate, bypassing ``mgr.open()`` entirely. After the lift both
kinds route through ``mgr.open()`` — which makes
``InteractiveAdapter.emit_rehydrated`` reachable on interactive
(it had been dead-by-routing) and gives the manager a single
rehydrate code path to maintain. ``emit_rehydrated`` stays a
documented no-op stub on the interactive adapter; the handler-side
``ws_created`` enqueue from the post-load callback is the
load-bearing emission for the SSE consumers.
Behaviour changes for interactive callers (documented in CHANGELOG):
* **Cross-kind open returns 404** (was 400 with
``"Workstream is not an interactive kind"``). The lift consolidates
on ``mgr.open()``'s single ``None``-return contract for missing /
wrong-kind / tombstoned rows. Security boundary unchanged.
* **Already-loaded response uses ``ws.name`` directly** (was
``get_workstream_display_name(resolved_id) or resolved_id``).
The dashboard listing endpoint still resolves aliases on its own
pass, so the user-visible name in the tab strip isn't affected.
Two /review fixes folded in:
* **Resume failures now return 5xx instead of broken-200.**
``SessionManager.open()`` previously caught and ``log.debug``-
swallowed exceptions from ``ChatSession.resume``. Since
``ChatSession.resume`` assigns ``self.messages`` *before* the
config-restore block, a partial-failure resume (corrupted
``workstream_config`` row, model-registry mismatch on a saved
alias, malformed ``temperature`` / ``max_tokens``) would leave
the session with history but with default config. Pre-lift the
interactive open handler called ``ws.session.resume`` directly
and let exceptions propagate as 500. Restored that behaviour:
``mgr.open()`` now re-raises resume exceptions after rolling
back the slot (``cleanup_ui`` + ``_remove_locked``), so the
lifted handler returns 500 with a correlation id and the storage
row stays available for a retry.
* **Bare ``except Exception`` documents intent.** A one-line
rationale in the handler body explains why the catch is broad
(no documented exception spec on ``adapter.build_session``;
resume can propagate via the new contract above). Keeps a future
contributor from narrowing it incorrectly.
Test scaffolding:
* ``tests/test_workstream_endpoints.py`` — fixture rebuilt to
use ``make_open_handler`` + a minimal cfg with a lazy alias
resolver so per-test ``@patch`` calls take effect. Added 5 new
tests: already-loaded uses ws.name, alias resolution runs first,
``mgr.open`` is called (NOT ``mgr.create``), post-load callback
fires with (request, ws) only on the load-from-storage path
(not the already-loaded shortcut), post-load exception swallowed
→ 200.
* ``tests/test_coordinator_endpoints.py`` — fixture imports
updated to ``make_open_handler``.
* ``tests/test_server_authz.py`` — ``TestOpenKindGate`` now expects
404 (not pre-lift's 400) for cross-kind open attempts. Docstring
explains the consolidation.
Two nit cleanups: dropped the unnecessary ``import secrets as
_secrets`` aliasing in the exception handler; refreshed the stale
``open_workstream`` reference in the ``AliasResolver`` doc-comment.
Lint + mypy clean. 4488 tests passing (was 4475; +13 new open
tests).
* fix(core): use cfg.audit_action_prefix for the per-kind noun in open's 500 error
PR #414 review caught the hardcoded ``"failed to open workstream"``
in ``make_open_handler``'s 500 path: coord callers got misleading
text (pre-lift coord said ``"failed to open coordinator"``).
The fix derives the noun from ``cfg.audit_action_prefix``
("workstream" interactive, "coordinator" coord) — a field both
production lifespans already construct, and which the previous
/review pipeline (q-5) flagged as dead config (set but read by
no factory). Reusing it here both fixes the wording AND gives
the field its first runtime reader.
Pinned by a new test
(``test_open_500_message_uses_kind_noun_from_cfg``) that wires a
coord-shaped cfg, forces ``mgr.open`` to raise, and asserts the
500 body contains ``"failed to open coordinator"`` + the
correlation id, without echoing the exception text.
Lint + mypy clean. 4489 tests passing (+1 new).
* refactor(core): lift cancel verb body across both kinds (Stage 2 verb lift)
The interactive ``/v1/api/cancel`` (body-keyed ws_id) and coord
``/v1/api/workstreams/{ws_id}/cancel`` (path-keyed) handlers now
share one body via ``make_cancel_handler(cfg, *, audit_emit=None)``
in ``turnstone.core.session_routes``. Per-kind divergence captured
by a new ``cancel_forensics: CancelForensics | None`` field on
``SessionEndpointConfig`` (interactive wires
``_capture_cancel_forensics``; coord wires ``None``) plus an
optional ``audit_emit`` (coord wires ``_audit_cancel_coordinator``;
interactive wires ``None`` — pre-lift interactive didn't audit
cancel).
Same factory + capability-flag pattern as P1.5's ``make_send_handler``
+ make_attachment_handlers. Old ``cancel_generation`` body deleted
from ``server.py``; old ``coordinator_cancel`` body deleted from
``console/server.py``.
Behavior changes (documented in CHANGELOG):
* **Coord gains the ``force`` flag.** Pre-lift coord ignored
``force``; the lifted body honours it on both kinds. Stuck-worker
recovery becomes available on coord (parity gain — coord workers
hang the same way interactive's can).
* **Coord cancel response always includes ``"dropped"``.** Pre-lift
returned bare ``{"status": "ok"}``; lifted returns
``{"status": "ok", "dropped": {}}``. Always-include parity with
interactive so SDK consumers don't branch on kind.
* **Coord cancel returns 400 ``"No session"``** on placeholder /
build-failed workstreams (was a silent 200 no-op pre-lift). Parity
with interactive's existing 400 branch.
* **Coord ``coordinator.cancel`` audit detail now includes
``force``** so operator-driven recovery is distinguishable from
routine cancels.
Three /review fixes folded in:
* **bug-1**: lifted body's ``resolve_approval`` is now gated on
``ui._pending_approval is not None``. Pre-fix, the unconditional
call leaked a stale ``approval_resolved`` SSE event on every
idle cancel — listener UIs that key on the event would dismiss
prompts they didn't have. ``resolve_plan`` keeps its existing
internal no-pending guard so the unconditional call is still
safe there.
* **bug-2**: force-cancel now clears ``_worker_running`` alongside
``worker_thread`` inside the same ``with ws._lock`` block. Prior
half-state ``(_worker_running=True, worker_thread=None)`` routed
follow-up sends through the queue-enqueue path onto the abandoned
worker (whose cancel flag short-circuits the queue-drain seam,
leaving messages orphaned until next spawn). Restores the
``(worker_thread, _worker_running)`` invariant
``session_worker.send`` documents.
* **bug-3**: ``coordinator_stop_cascade._fanout_on_children`` now
treats child cancel ``400 + "No session"`` as ``skipped`` (was
``failed``). Lifted coord cancel returns 400 on placeholder
children; matches the pre-lift outcome where those children were
silently no-op'd, so the cascade response's ``failed`` bucket
stops firing spurious operator alerts.
Test scaffolding:
* ``tests/test_coordinator_endpoints.py`` — replace ``coordinator_cancel``
fixture with ``make_cancel_handler(...)`` wiring; add 6 new
tests covering always-include shape, force-flag worker-abandon,
400-on-null-session, cancel_forensics swallowed-exception,
audit_emit swallowed-exception, no-stale-approval-resolved-on-idle.
* ``tests/test_server_authz.py`` — new ``TestInteractiveCancelLifted``
class with HTTP-level coverage of ``/v1/api/cancel`` for the
dropped shape, force-flag + ``_worker_running`` clearing, and
400-on-null-session. Pre-lift ``cancel_generation`` had no
HTTP-level test; this is the first.
One observable change for interactive (pre-existing call site):
``resolve_approval`` / ``resolve_plan`` now run on every cancel
regardless of ``was_running`` (was gated). Lifts coord's
unconditional behaviour onto interactive — a stuck approval-pending
state from a crashed worker can now be cleared via cancel without
requiring close + rehydrate.
Lint + mypy clean. 4484 tests passing (was 4475; +9 new cancel
tests minus the moved one that became part of the new suite).
* docs(core,changelog): correct cancel-lift behaviour description for resolve_approval
Two review comments on PR #413 caught the same drift between the
implementation and its documentation: my bug-1 fix gated
``resolve_approval`` on ``_pending_approval is not None`` (because
it broadcasts ``approval_resolved`` unconditionally), but the
``make_cancel_handler`` docstring and the CHANGELOG entry still
claimed both ``resolve_approval`` and ``resolve_plan`` "run on
every cancel" and "the calls are idempotent and no-op when
nothing is blocked".
Reality:
* ``resolve_plan`` does run on every cancel and its no-op-when-
nothing-pending behaviour is real (the method has an internal
``_pending_plan_review is None`` short-circuit).
* ``resolve_approval`` runs only when ``ui._pending_approval is
not None``. Without the gate, every idle cancel would broadcast
a stale ``approval_resolved`` SSE event and overwrite
``_approval_result``.
Updated:
* ``make_cancel_handler`` docstring (turnstone/core/session_routes.py
in the "Behavior changes vs the pre-lift handlers" section) —
splits the two methods into separate bullets, explains why
``resolve_approval`` is gated and ``resolve_plan`` isn't.
* CHANGELOG.md ``[Stage 2 Verb Lift — cancel]`` entry — same
split + rationale; the asymmetric coord pre-lift parity is
still flagged as the recovery path that drove the lift.
Docs-only change; lint + mypy clean; cancel test suite (59 tests)
unchanged.
* style(core): replace CancelForensics ellipsis stub with docstring
github-code-quality bot flagged the ``...`` body of
``CancelForensics.__call__`` as "Statement has no effect". The
ellipsis is the canonical Protocol method-body idiom (no real
issue), but switching to a one-line docstring satisfies the bot
AND adds a small piece of method-level documentation. The class-
level rationale (why Protocol-typed instead of a plain Callable
alias) moves from a wall of leading ``#`` comments into a proper
class docstring at the same time.
Style-only change; the Protocol semantics are identical.
* refactor(core): split SessionKindAdapter Protocol into construction + emission (Stage 2 P3)
The single ``SessionKindAdapter`` Protocol that ``SessionManager``
takes is split into two:
* ``SessionKindAdapter`` — kind / build_ui / build_session /
cleanup_ui. Required for every kind. The shared lifecycle
manager always delegates here for construction + cleanup.
* ``SessionEventEmitter`` — emit_created / emit_state /
emit_rehydrated / emit_closed. **Optional**, wired through a new
``event_emitter: SessionEventEmitter | None = None`` kwarg on
``SessionManager``. Reserved for future kinds whose lifecycle
transitions don't fan out anywhere; both production kinds wire
one today.
Both production adapters implement both Protocols. The interactive
lifespan (``server.py``) and console lifespan
(``console/server.py``) pass their adapter as both ``adapter`` and
``event_emitter`` — production behaviour is unchanged. Six lifecycle
sites in ``SessionManager`` (create / open eviction / open rehydrate /
close / set_state / close_idle / _reserve_and_install_locked unwind)
now call ``self._event_emitter.emit_*(...)`` guarded by
``if self._event_emitter is not None``.
InteractiveAdapter asymmetry preserved + documented:
* ``emit_closed`` stays load-bearing — it's the **sole** transport
path for ``ws_closed`` onto the process-wide global SSE queue
(Stage 1 consolidated emission from the create handler here so
there's exactly one emission point; ``name`` powers the
frontend's eviction toast).
* ``emit_created`` / ``emit_state`` / ``emit_rehydrated`` are
documented no-op stubs (``del ws[, state]``). Those events fire
from out-of-band paths — the create HTTP handler enqueues
``ws_created`` directly onto ``global_queue`` *after* attachment
validation (so a rejected upload doesn't surface a phantom
create→close pair); ``WebUI._broadcast_state`` emits the full
``ws_state`` payload (tokens + context_ratio + activity) via the
``SessionUI.on_state_change`` callback chain. The stubs exist
solely to satisfy ``SessionEventEmitter`` Protocol so the
adapter can be wired as the manager's ``event_emitter`` for the
``emit_closed`` path. Each stub has a 1-line inline rationale to
match the in-repo convention (``coordinator_adapter.py:210``).
Test scaffolding:
* ``tests/test_session_manager.py`` — ``_make_manager`` and
``_make_with_writer`` wire ``FakeAdapter`` as both ``adapter``
and ``event_emitter`` for production parity; the standalone
``test_create_uses_configured_node_id`` does the same.
``FakeAdapter.emit_rehydrated`` now records as
``_Event("rehydrated", ...)`` rather than conflating with
``"created"``, and ``test_open_resurrects_closed_state`` asserts
against ``events_of("rehydrated")`` so a regression where the
manager fires the wrong call on the open path actually fails.
* ``tests/_coord_test_helpers.py`` and
``tests/test_coordinator_end_to_end.py`` — wire
``CoordinatorAdapter`` as both args.
* Six interactive test fixtures (``test_skills.py``,
``test_prompt_templates_runtime.py`` x2, ``test_model_registry.py``,
``test_server_authz.py``, ``test_server_attachments_on_create.py``)
— wire ``event_emitter=adapter`` so they match the production
wiring, removing the footgun where a future contributor adds a
``gq.get_nowait()`` assertion and silently loses the only
``ws_closed`` transport.
* ``tests/test_interactive_adapter.py`` — drops the three
tautological no-op-emit_* tests (``test_emit_created_is_noop``,
``test_emit_state_is_noop``, ``test_emit_rehydrated_is_noop``);
keeps the four ``emit_closed`` tests (real behaviour).
Lint + mypy clean. 4475 tests passing.
* docs(core): correct SessionKindAdapter + SessionEventEmitter docstrings to match implementation
Two Copilot review threads on PR #412 caught the same real
discrepancy: my P3 docstrings on ``SessionKindAdapter`` and
``SessionEventEmitter`` described an *intent* — "interactive
doesn't implement ``SessionEventEmitter``; the manager skips emit
calls when no emitter is wired" — that doesn't match the actual
wiring. ``InteractiveAdapter`` does implement both Protocols and
``server.py`` does pass it as ``event_emitter``; only the three
no-op stubs (``emit_created`` / ``emit_state`` / ``emit_rehydrated``)
are dead, while ``emit_closed`` is load-bearing.
Updated both docstrings to:
* State that both production adapters implement both Protocols.
* Explain the asymmetry is in *which* emit methods carry real
bodies (coord: 4; interactive: 1, with 3 documented stubs because
the out-of-band paths — create handler ``ws_created`` after
attachment validation, ``WebUI._broadcast_state`` carrying the
richer ``ws_state`` payload — fire those events).
* Clarify the ``if self._event_emitter is not None`` guard exists
for the kwarg-omitted case (tests that don't care about events,
reserved for future kinds whose transitions don't fan out
anywhere).
Docstring-only change. Lint + mypy clean; the 75 tests in
test_session_manager + test_interactive_adapter + test_coordinator_adapter
pass.
Resolves the two Copilot review threads on PR #412 (commits
PRRC_kwDORcMomM67VyPD, PRRC_kwDORcMomM67VyPI).
Five fixes from Copilot's review of Stage 2 P1.5 — all preserve
behaviour, narrow docstring claims, and round out the response shape:
* **session_routes.py:supports_attachments docstring** — claimed
the handler "accepts only ``{"message": ...}``" when ``False``,
but the implementation silently ignores ``attachment_ids``
rather than rejecting. Updated wording to say the
attachment-resolution block short-circuits and any
``attachment_ids`` are silently ignored. Behaviour unchanged
(silent-ignore is the right choice for forward compat — clients
passing ``attachment_ids`` speculatively to a not-yet-lit-up
kind shouldn't get a 400).
* **session_routes.py:queue_full response shape** — restored the
always-include guarantee for ``attached_ids`` /
``dropped_attachment_ids``. The queue_full path now returns
``attached_ids: []`` and ``dropped_attachment_ids: list(requested_ids)``
so SDK consumers don't have to branch on status.
* **server.py:_interactive_spawn_metrics guard** — added
``_ws_turn_tool_calls`` to the ``hasattr`` chain. Previously
the guard checked ``_ws_lock`` + ``_ws_messages`` and then
unconditionally assigned ``_ws_turn_tool_calls`` — would
raise on a SessionUI subclass with the first two but not the
third.
* **console_spec.py:coord_send error_codes** — added 409
(the 'session UI not available' branch in
``make_send_handler`` returns 409, but the spec didn't list
it). OpenAPI spec regenerated; TS SDK types refreshed.
* **session_routes.py:tenant_check docstring** — claimed
interactive uses ``_require_ws_access`` with "404 on owner
mismatch", but the helper now delegates to
``resolve_workstream_owner`` which explicitly does NOT enforce
row-level ownership (trusted-team semantics; 404s only on
missing rows). Updated wording to match.
Six fixes from the local /review pipeline (find-bug + find-security +
find-quality, all confirmed by verify):
* **sec-1 (major)** — coord ``attachment_owner_resolver`` now
resolves through ``coord_mgr.get(ws_id)`` only and does NOT fall
back to storage. Without the kind-strict check, an
``admin.coordinator``-scoped caller could pass an *interactive*
workstream ws_id to the new coord attachment endpoints; the
generic ``get_workstream_owner`` storage call (kind-agnostic)
would resolve and grant cross-kind read / write access to
interactive attachments. New regression test
``test_coord_attachment_endpoints_404_on_interactive_ws_id``
pins the surface.
* **bug-1 (minor)** — UI hook calls in the spawn-path ``_run``
closure are now wrapped per-hook (via ``_emit_ui``) so a failure
in ``ui.on_error`` doesn't suppress the subsequent
``ui.on_stream_end`` / ``ui.on_state_change`` calls. Mirrors the
pre-P1.5 coord_adapter.send per-hook defense.
* **bug-2 (minor)** — ``make_dequeue_handler`` now 404s when
``ws.ui is None`` (preserves the pre-P1.5 ``_get_ws`` contract;
a partially-constructed or close-window workstream shouldn't
answer DELETE).
* **bug-3 (minor)** — ``coordinator.js`` gains a
``case "message_queued":`` handler that surfaces the queueing
as an info row. Coord wires ``emit_message_queued=True`` for
parity with interactive but the dashboard had no router branch
for these events, silently dropping them.
* **bug-4 (minor)** — error-message format on coord regressed
from ``f"{type(exc).__name__}: {exc}"`` to ``f"Error: {e}"``
(lost the exception class name, which coord operators rely on
to triage failures). Restored.
* **q-1 (major)** — duplicate ``_auth_user_id`` and
``_require_ws_access`` helpers in ``server.py`` and
``console/server.py`` now delegate to the lifted
``turnstone.core.web_helpers.auth_user_id`` /
``resolve_workstream_owner``. The lifted versions are the
canonical implementations; the shims keep existing call sites
working without a sweeping rename.
CHANGELOG entry adds a Security section noting the kind-strict
resolver fix and a behaviour callout for the cancel-state semantic.
Five new TestCoordinatorAttachments tests in
``tests/test_coordinator_endpoints.py`` exercising the lifted
attachment surface end-to-end on coord:
* upload → list round-trip
* get_content returns raw bytes with text/plain forced for text
* delete removes pending entries and clears them from the listing
* send with attachment_ids consumes pending under the send_id token
* send response carries attached_ids / dropped_attachment_ids even
on plain-text sends (unified shape parity)
The existing ``_coord_endpoint_config`` fixture grew capability
flags to mirror the production console wiring, and ``_make_client``
now mounts the four coord attachment routes via
``make_attachment_handlers``.
OpenAPI specs regenerated; TS SDK bumped to 0.5.0. CHANGELOG entry
under [Unreleased] documents the verb-shape lift, the coord
attachment surface coming online, the response-shape change for
``coordinator_send``, the unification of the three lifted classifier /
lock helpers under ``turnstone.core.attachments``, and the new SDK
helpers.
Replaces per-kind ``send_message`` / ``coordinator_send`` and the
four interactive attachment handlers with calls to the shared
factories from ``turnstone.core.session_routes``. Net deletion of
~660 LOC from ``server.py`` (the lifted body lives in
``session_routes`` and is mounted twice — once interactive, once
coord).
Interactive (``turnstone/server.py``):
* ``SessionEndpointConfig`` now carries ``supports_attachments=True``,
``attachment_owner_resolver`` (delegates to ``_require_ws_access``
via storage path to preserve test fixtures using MagicMock
managers), ``attachment_helpers`` (the lifted classifiers +
upload-lock), ``spawn_metrics`` (records the per-conversation
WebUI counters that coord doesn't have), and
``emit_message_queued=True``.
* New ``_make_method_dispatch`` adapter lets the legacy body-keyed
``/v1/api/send`` URL serve both POST (send) and DELETE (dequeue)
via the lifted handlers.
* The four attachment handler bodies (``upload_attachment`` etc.)
are deleted; the shared registrar mounts them via
``make_attachment_handlers(cfg)``.
Coord (``turnstone/console/server.py``):
* Same wiring with coord-specific resolvers
(``_coord_attachment_owner`` via the lifted
``resolve_workstream_owner``). ``spawn_metrics=None`` since the
coord dashboard doesn't have per-conversation counters; cluster
metrics fan out via the collector.
* Old ``coordinator_send`` body deleted.
* Console-side coord attachment endpoints come up automatically
through the shared ``AttachmentHandlers`` slot — no per-kind
attachment handler bodies needed at all.
Coord dashboard (``coordinator.js``): user messages with
attachments arriving on history replay now extract just the text
portion + a ``📎 N attachment(s)`` count badge instead of
JSON-stringifying the multipart content. Full chip-rendering with
click-to-view stays deferred.
Python SDK adds coord-side helpers on
``AsyncTurnstoneConsole`` + ``TurnstoneConsole``:
``coordinator_send`` (with ``attachment_ids``),
``coordinator_upload_attachment``,
``coordinator_list_attachments``,
``coordinator_get_attachment_content``,
``coordinator_delete_attachment``. URL prefix is direct
``/v1/api/workstreams/`` since coord workstreams live on the
console — no routing-proxy hop needed.
Behaviour change for coord callers:
* Worker-queue-full responses are now ``200 {"status": "queue_full"}``
for parity with interactive (was ``429 {"error": "..."}``). SDK
consumers checking for 429 should switch to the status field.
* Send response now always carries ``attached_ids`` /
``dropped_attachment_ids`` (empty arrays on plain text sends);
the live-worker reuse path also surfaces ``priority`` /
``msg_id``.
Stage 2 P1.5 — verb-shape unification at the HTTP layer for both
``send`` and the four attachment endpoints. New factories in
``turnstone.core.session_routes``:
* ``make_send_handler(cfg)`` — single body covering the
attachment-resolution dance, dispatcher hand-off, queue/spawn
outcome surfacing, and metrics increment. Capability flags on
``SessionEndpointConfig`` (``supports_attachments``,
``attachment_owner_resolver``, ``attachment_helpers``,
``spawn_metrics``, ``emit_message_queued``) toggle the per-kind
bits without forking the body.
* ``make_dequeue_handler(cfg)`` — DELETE branch (cancel a queued
message by ``msg_id``). Path-keyed; mountable on both new
``/v1/api/workstreams/{ws_id}/send`` and the legacy body-keyed
``/v1/api/send`` URL via ``make_legacy_body_keyed_adapter``.
* ``make_attachment_handlers(cfg)`` — quartet of upload / list /
get_content / delete with shared scope checks and 404 masking.
Per-kind classification + locking comes in via the new
``AttachmentUploadHelpers`` bundle so the cfg stays declarative.
Three pure helpers (``sniff_image_mime``,
``classify_text_attachment``, ``upload_lock``) moved from
``turnstone/server.py`` to ``turnstone/core/attachments.py`` so the
console process can wire them into the lifted attachment endpoints
without depending on the node-side server module. Behaviour is
unchanged.
``turnstone.core.web_helpers`` gains ``auth_user_id`` and
``resolve_workstream_owner`` so both kinds share the owner-resolution
helper underpinning attachment scoping. The interactive ``trusted-team``
404-on-missing semantics are preserved; ``not_found_label`` is
parameterised so coord can return ``coordinator not found``.
Console spec adds ``CoordinatorSendResponse`` (parity with interactive
``SendResponse``) and four new endpoint declarations for the coord
attachment surface.
PR #410 review pass:
* **session_worker**: ``except BaseException`` → ``except Exception``
in ``_runner`` (code-quality bot). Daemon threads don't receive
SystemExit/KeyboardInterrupt, so the wider catch was unjustified
defensive style. Same defense-in-depth for unexpected ``run()``
exceptions; doesn't widen scope to runtime signals.
* **session_worker**: ``threading.Thread()`` construction moved
inside the spawn branch under ``ws._lock`` (Copilot). The
enqueue path no longer allocates and then discards a Thread
object on each call against a busy workstream. Thread()
construction is microsecond-cheap, so the lock-window growth is
negligible vs. the saved allocation churn.
* **lifespans**: ``state_writer.shutdown()`` (and the console
equivalent) now run via ``asyncio.to_thread`` so the daemon-
thread join + sync DB drain don't block the event loop and
delay other teardown tasks (Copilot, ×2).
* **tests**: five remaining ``writer._flush_once()`` calls
switched to the public ``writer.flush()`` API across
test_session_manager.py (4) and test_state_writer.py (1)
(Copilot, ×5). Tests no longer depend on private internals.
Six fixes from the local /review pipeline (find-bug + find-perf +
find-quality, all confirmed by verify):
* **bug-1 (critical)** — ``StateWriter.record(flush_now=True)`` now
drops any pending buffered transient for the same ws_id AND waits
on the flush_lock before its sync UPDATE. Without this, an
earlier buffered 'running' could flush AFTER the sync 'error'
write and clobber the terminal state — same shape as the
close-vs-buffered-transient race ``discard`` was already
guarding. New regression tests cover both the drop and the
in-flight wait.
* **bug-3 (major)** — ``session_worker.send`` now assigns
``ws.worker_thread = t`` AND sets ``ws._worker_running = True``
under the same ``ws._lock`` acquisition. Previously
``worker_thread`` was assigned outside the lock, so a reader
holding ``ws._lock`` could observe ``_worker_running=True``
paired with a stale (already-exited) ``worker_thread`` —
defeating every ``ws.worker_thread is me`` identity check
downstream.
* **bug-2 (major)** — rewind/retry busy gate in
``server.py:command`` now reads ``ws._worker_running`` instead
of ``ws.worker_thread.is_alive()``. The is_alive() gate could
see a stale dead thread under ws._lock while a new worker was
in the middle of starting (post bug-3 fix the window narrows
but the gate-mismatch was independent — ``_worker_running`` is
the canonical gate post-Stage-2-P1).
* **perf-2 (major)** — ``StateWriter.discard`` now waits on
``_flush_lock`` with a 5s timeout (configurable). Without a
bound, a stuck Postgres connection inside an in-flight flush
would block ``close()`` and ``close_idle()`` indefinitely while
they hold ``ws._lock`` — a system-wide hang on every close
path. On timeout we log + proceed; the worst-case degrades to
"buffered transient flushes shortly after sync 'closed'"
(eventual consistency) rather than process hang.
* **q-1** — inline comments on ``run_retry`` and ``_run_initial``
now explain why those two spawn sites don't go through
``session_worker.send``: retry-when-busy is a hard reject (no
fallback queue), and init-on-create can't have a pre-existing
worker by construction (enqueue branch is dead code). Both
still set ``_worker_running`` + ``ws.worker_thread`` together
under ws._lock for parity with the dispatcher.
* **q-3** — ``state_writer.discard`` callsite comments in
``session_manager.py`` no longer reference 'bug-3' (which lived
only in untracked working notes). Now describe the invariant
inline by what it prevents.
* **q-5** — ``StateWriter._flush_once`` promoted to public
``flush()``. Tests now drive flushes via the public API.
Two new bullets under [Unreleased]:
* Worker dispatch unified — ``session_worker.send`` shared by
interactive ``/v1/api/send``, the coord adapter, watches, retry,
and initial-message paths. Gate is ``_worker_running`` (atomic
under ws._lock) instead of ``Thread.is_alive()``. Closes a
parallel-worker race that any concurrent path (watch + /send,
retry + /send, init + /send) could trigger pre-P1.
* Buffered ``StateWriter`` for set_state — non-terminal transitions
now show up in storage up to ~1s late (SSE consumers see them
immediately via the adapter). Terminal ERROR + close still write
sync; bug-3 invariant preserved via state_writer.discard before
the sync 'closed' write.
The `/send` HTTP body convergence stays out of scope — interactive's
attachments / reservations / queue-outcome distinctions diverge from
coord's response shape too far for a clean factory split until
coord grows attachments parity (post-1.5.0).
Five new tests under ``TestSessionManagerWithStateWriter`` exercise
the bug-3 invariant under write-behind:
* set_state buffers via the writer (long flush_interval → no sync
write until drain).
* set_state(ERROR) flushes synchronously.
* close after a buffered transient writes 'closed' as the final
state — the buffered 'running' must NOT be flushed to storage
AFTER close's sync 'closed' write.
* close_idle exhibits the same invariant.
* set_state arriving AFTER close short-circuits on ws._closed and
never reaches the buffer.
``SessionManager.__init__`` accepts an optional ``state_writer``;
when present, ``set_state`` for non-terminal transitions records via
the buffered writer instead of holding ``ws._lock`` across a sync DB
UPDATE. Terminal ERROR transitions still flush sync (error-surfacing
paths need durability before any observer sees the state).
``close()`` and ``close_idle()`` call ``state_writer.discard(ws_id)``
under ws._lock BEFORE their sync 'closed' write — drops any pending
buffered transient and waits on the flush_lock for any in-flight
flush to complete. Without this, a buffered 'running' could land in
storage AFTER the sync 'closed' write and resurrect the closed row
(bug-3 invariant under write-behind).
Lifespan wiring on both servers: build the StateWriter alongside the
SessionManager, ``state_writer.start()`` on enter, ``shutdown()`` on
teardown (drains any pending writes synchronously). Tests can leave
``state_writer=None`` and get the legacy direct-write behaviour.
``turnstone.core.state_writer.StateWriter`` buffers non-terminal
``update_workstream_state`` writes (last state per ws_id wins) and
flushes them on a ~1s cadence (configurable). Terminal ERROR
transitions and close()'s 'closed' write bypass the buffer.
Bounded buffer (``max_buffer=10000`` default) evicts the oldest
ws_id on insertion overflow — protects against unbounded growth
when storage is unreachable. ``discard(ws_id)`` drops any pending
buffered transition AND waits on a flush_lock for any in-flight
write to complete; this is the close-path hook that preserves the
bug-3 invariant (a closed row can't be resurrected by a buffered
transient writing AFTER close's sync 'closed').
13 unit tests cover coalescing, flush_now, bounded buffer, the
discard / in-flight-flush wait, lifecycle (start/shutdown
idempotence), wake-on-record latency, and resilience to storage
errors poisoning subsequent flushes.
Five spawn sites in turnstone/server.py now share the worker dispatch:
* ``send_message`` (``POST /v1/api/send``) — the main path. Now uses
``session_worker.send`` with separate ``_enqueue`` / ``_run``
closures; the queue-vs-spawn outcome is conveyed via a captured
``queue_outcome`` dict so the existing response shapes
(``status: queued`` vs ``status: ok``) survive.
* ``_make_watch_dispatch`` — watch results dispatch.
* ``run_retry`` (post-rewind) and ``_run_initial`` (initial-message
on workstream creation) — set ``_worker_running`` directly under
ws._lock instead of going through session_worker (their structural
shape doesn't fit a queue-vs-spawn decision) but stay consistent
with the shared gate so they can't race with /send into parallel
workers.
* ``cancel_generation``'s ``was_running`` snapshot now reads
``_worker_running`` for parity with the dispatcher.
Pre-dispatch cancel-await also gates on ``_worker_running`` for
consistency. The ``busy_error`` / ``status: busy`` legacy branch
(reached only when worker is alive but ws.session is None) is gone
— the new path checks ws.session up front and returns the same
500 shape.
Test fixtures in test_server_attachments_endpoints.py and
test_watch_dispatch.py updated to set ws._worker_running explicitly
(MagicMock auto-truthifies the field, which would otherwise mis-route
all idle paths into queue mode).
Introduces ``turnstone.core.session_worker.send`` — the atomic
check-and-(spawn-or-queue) decision both interactive and coordinator
HTTP paths use to drive ``ChatSession.send``. Callers pass no-arg
``enqueue`` / ``run`` closures; the shared module owns only the
``ws._worker_running`` lifecycle.
CoordinatorAdapter.send now delegates to the shared module — its
``_spawn_worker`` body is gone. Workstream._worker_running's
docstring updated to note both kinds use it post-Stage-2-P1.
PR #409 line-level review feedback. Three of four findings valid;
the fourth (code-quality bot's "unused TYPE_CHECKING imports")
verified as false-positive — removing the imports breaks mypy on
the string-form annotations in ``ManagerLookup`` / ``TenantCheck``
/ ``CloseAuditEmitter``.
CI lint failure (ruff format on ``tests/_coord_test_helpers.py``)
addressed alongside.
Findings addressed:
- **Copilot #1** (``session_routes.py`` SessionEndpointConfig
docstring): said the config is "stored on
``app.state.session_endpoint_config``" and "handler bodies pull
this config from app.state". Stale after the previous fixup
switched the factories to capture ``cfg`` via closure. Rewrote
the class docstring + the lifted-handler comment block + the
module docstring + the ``create_app`` block comments in both
``server.py`` and ``console/server.py``.
- **Copilot #2** (``server.py:_interactive_manager_lookup``
docstring): referenced ``:data:SessionRouteHandlers`` which was
renamed to ``SharedSessionVerbHandlers`` AND wasn't the right
reference anyway — the callable matches
``SessionEndpointConfig.manager_lookup``. Fixed.
- **Bonus**: dropped the now-dead
``app.state.session_endpoint_config = ...`` assignments in both
servers (nothing reads them since the closure-capture switch).
- **Bonus**: dropped the stale "close (interactive caps + redacts +
persists close_reason)" entry from the deferred-verbs comment in
``session_routes.py`` — close was lifted in the previous commit
and is no longer in the deferred set.
- **CI lint**: ``ruff format`` joined the
``MockStorage.list_services`` signature in
``tests/_coord_test_helpers.py`` to a single line (95 chars,
fits the 100-char limit).
ruff + ruff format + mypy clean. 88 affected tests pass.
Addresses the eight verified findings from the second review pass on
the body-convergence work (one bug-flagged behavior change, one
defensive-style nit, six quality items). One quality item (q-6,
``request.scope[\"path_params\"]`` mutation in the legacy adapter)
is documented but not refactored — restructuring the lifted handler
signatures to take ``ws_id`` as an explicit param is bigger than
this fixup's scope; the adapter docstring already explains the
choice.
Findings addressed:
- **bug-1 + q-5**: hoist module-level ``log = get_logger(__name__)``
in ``session_routes.py``; bump audit-failure log from ``debug``
to ``warning`` (compliance signal). Document the interactive
500-on-audit-failure → 200+log behavior change in CHANGELOG +
in ``make_close_handler``'s docstring.
- **bug-2**: switch ``_audit_close_workstream`` to
``getattr(request.app.state, \"auth_storage\", None)`` for
consistency with the upstream gate. Same fix on coord side.
- **q-1**: pass ``SessionEndpointConfig`` into
``make_approve_handler(cfg)`` and
``make_close_handler(cfg, *, audit_emit, supports_close_reason)``
via closure capture. Removes the implicit ``app.state`` contract
and parallels the two factory signatures. Tests + production
wiring updated.
- **q-2**: promote ``_audit_close_coordinator`` to a module-level
function in ``turnstone/console/server.py``. Both test fixtures
import it instead of duplicating the body. The previous three
near-identical implementations collapse to one.
- **q-3**: lift ``_interactive_tenant_check`` and
``_audit_close_workstream`` from nested ``create_app`` closures
to module-level functions in ``turnstone/server.py``, beside the
other ``_audit_*`` / ``_require_*`` helpers. Add
``_interactive_manager_lookup`` so the config doesn't need a
lambda. ``create_app`` shrinks accordingly.
- **q-4**: merge the bottom ``if TYPE_CHECKING`` block into the
one at the top of ``session_routes.py``.
- **q-7**: replace ``assert mgr is not None`` with
``mgr = cast(\"SessionManager\", mgr_opt)`` in both lifted
handlers — survives ``python -O`` and makes the type-checker-only
intent explicit.
- **q-8**: update ``test_coordinator_endpoints.py`` file docstring
to mention the lifted-handler wiring.
ruff + mypy + 4366 pytest pass. Live console smoke against the
unified URLs returns 503 (no coord_mgr in smoke env) — proves the
factory-captured config is reachable + manager_lookup fires.
CHANGELOG ``[Unreleased]`` entry expanded to flag the audit-failure
swallow as an interactive behavior change alongside the existing
500→404 standardization.
Adds two paragraphs:
- Notes the two verbs (``approve``, ``close``) whose bodies were
successfully lifted into the shared registrar, plus the close-
failure status-code standardization (500 → 404 on coord). Calls
out the verbs whose bodies are intentionally NOT lifted, with
the underlying reason (Priority 1 dependency, response-shape
unification, etc.) so the next-session reader doesn't re-litigate.
- Notes the TS SDK 0.4.0 bump and the regenerated reference specs.
No code change.
Stage 2 Priority 0 Step 0.2 body-convergence — second verb.
``make_close_handler(audit_emit=..., supports_close_reason=...)``
factory in ``turnstone/core/session_routes.py`` produces the lifted
body; both interactive and coord pass their kind-specific audit
emitter at app construction.
The two body-keyed close URL aliases on the interactive side reach
the same lifted body:
- ``POST /v1/api/workstreams/{ws_id}/close`` (new, path-keyed)
via ``register_session_routes(handlers.close=...)``.
- ``POST /v1/api/workstreams/close`` (legacy, body-keyed) via
``make_legacy_body_keyed_adapter(close_handler)``.
Coord exposes only the path-keyed shape.
Behavior gains:
- ``supports_close_reason=True`` (interactive only) keeps the 512-
byte UTF-8 cap + credential redaction + ``workstream_config``
persistence path. Coord stays at ``False``; if coord ever wants
close-reason metadata, flipping the flag is a one-line change.
- ``audit_emit`` is per-kind so each owns its detail dict shape
(``{kind, parent_ws_id, reason}`` vs ``{coord_ws_id, src}``) and
audit action name (``workstream.closed`` vs ``coordinator.close``).
- Standardizes the close-failure status code to 404 across both
kinds. The coord code previously returned 500 on a
``mgr.close()`` race-loss, which was overly pessimistic — the
semantic is "the ws was popped between .get() and .close()", i.e.
not-found.
Coord-side test fixtures (``test_coordinator_endpoints``,
``test_coordinator_end_to_end``) swap the imported
``coordinator_close`` for the lifted handler + a local audit_emit
adapter so the tests exercise the same code path the live console
does.
ruff + mypy + 4366 pytest pass. Live console smoke against
``POST /v1/api/workstreams/abc/close`` returns 503 (no coord_mgr
loaded in the smoke env) — proves the lifted handler is reachable
+ the manager_lookup callable fires correctly.
Two verbs converged so far (``approve`` + ``close``); the remaining
pairs (``send``, ``cancel``, ``open``, ``events``, ``create``,
``list``, ``saved``, ``history``, ``detail``) have substantive
behavior divergence that doesn't factor cleanly into the
SessionEndpointConfig + factory-handler pattern — see the
session_routes module docstring for the per-verb status.
Stage 2 Priority 0 Step 0.2 body-convergence — first verb. Both
interactive ``approve`` and coord ``coordinator_approve`` handler
bodies collapse into ``make_approve_handler()`` in
``turnstone/core/session_routes.py``. Each kind sets a
``SessionEndpointConfig`` on ``app.state`` carrying the kind-
specific policies (auth gate, manager lookup, tenant check, audit
prefix, not-found label) the lifted body consults at request time.
The two interactive URLs converge:
- ``POST /v1/api/workstreams/{ws_id}/approve`` (new, path-keyed)
reaches the lifted body directly via ``register_session_routes``.
- ``POST /v1/api/approve`` (legacy, body-keyed) keeps shipping;
``make_legacy_body_keyed_adapter`` peeks the body for ``ws_id``,
splices it into ``request.path_params``, and forwards to the same
lifted body. Frontend can keep using the legacy URL — no caller
churn.
Coord exposes only the path-keyed shape (its URLs were experimental
in 1.5.0aN; the URL-shape commit already removed the ``coordinator/``
prefix).
Tenant-check is split out from permission-gate so interactive's
``_require_ws_access`` (404 on cross-owner) and coord's
``_require_admin_coordinator`` (cluster-wide scope) coexist without
either kind triggering the wrong gate.
Coord-side test fixture (``test_coordinator_endpoints._make_client``)
swaps the imported ``coordinator_approve`` for the lifted handler
and seeds ``app.state.session_endpoint_config`` so the tests
exercise the same code path the live console does.
Net delta: ~−25 LOC for this verb on top of the SessionEndpointConfig
+ legacy-adapter scaffolding (~80 LOC paid once). Subsequent verb
lifts amortize against that scaffolding.
ruff + mypy + 4366 pytest pass. Live console smoke against the
unified URL returns 503 (no coord_mgr loaded in the smoke env) —
proves the lifted handler is reachable + the manager_lookup callable
fires correctly.
Verbs still kind-specific (deferred — bodies have substantive
behavior divergence, not just naming): ``send`` (Priority 1
worker dispatch), ``cancel`` (interactive forensics + force flag),
``close`` (interactive close-reason cap+redact+persist), ``open``
(interactive resume vs coord rehydrate), ``events`` (different SSE
replay shapes), ``create`` (interactive attachments vs coord
initial_message), ``list`` / ``saved`` (different response keys).
Stage 2 Priority 0 Step 0.5 follow-on. The handwritten Python
OpenAPI spec already moved to ``/v1/api/workstreams/`` in the URL
sweep commit; this just regenerates ``sdk/typescript/openapi-{server,console}.json``
from those specs so generated TS callers see the new paths.
Bumps the TS SDK to 0.4.0 to flag the URL-shape break for any
1.5.0aN-era consumer of the experimental coord client. Python SDK
needs no change — it never exposed the coord HTTP surface.
TS typecheck + 32 vitest tests pass.
Addresses the eight quality findings the per-priority /review pass
flagged on the registrar refactor. All confirmed by the verifier;
none blocking. Net −186 LOC in this fixup.
q-1, q-9: trim ``session_routes.py`` module docstring + console
``create_app`` comments to the timeless explanation. The Step 0.1 →
0.4 narrative was already stale within the PR that introduced it
(every step had landed by the final commit) and would rot further as
the body-convergence follow-on lands.
q-2, q-5: group the four attachment handlers into an
``AttachmentHandlers`` dataclass exposed as
``handlers.attachments: AttachmentHandlers | None``. The type system
now carries the all-or-none invariant; the parallel four-condition
chain + bare ValueError disappear.
q-3: drop the ``mgr`` and ``adapter`` placeholder kwargs from both
``register_session_routes`` and ``register_coord_verbs``. Pre-
threading them so a future commit avoids "callsite churn" violated
the project's "don't pre-build for the next step" norm — the
body-convergence follow-on will edit the callsites anyway. Drops
``SessionManager.adapter`` for the same reason.
q-4: drop ``SessionRouteConfig`` outright. It existed solely to
carry ``supports_legacy_close``; the registrar now mounts the legacy
close route whenever ``handlers.close_legacy is not None``, matching
the all-Optional convention used for every other handler.
q-6: move ``MockStorage`` from ``tests/test_console.py`` into the
shared ``tests/_coord_test_helpers.py`` and re-import in test_console
+ test_session_routes. No more cross-test-module import.
q-7: trim the two exhaustive route-table set-equality assertions
(``test_coord_shape_mounts_expected_verbs``,
``test_register_coord_verbs_mounts_expected_paths``); replaced with
focused ``test_attachment_routes_mount_when_quartet_provided`` and
``test_close_legacy_mounts_when_handler_provided``. The targeted
ordering tests still catch the actual registrar bugs.
q-8: delete the tombstone comment block where the legacy
``/api/coordinator/`` Routes used to be — per the user's
``feedback_no_tombstone_comments`` norm, deletions don't get
narrated inline.
q-10: rename ``SessionRouteHandlers`` → ``SharedSessionVerbHandlers``
and ``CoordVerbHandlers`` → ``CoordOnlyVerbHandlers`` so the
"shared verbs vs coord-only verbs" symmetry is visible at the type
names. Drop the back-compat aliases since nothing uses them.
The plan called the CHANGELOG callout for the
``/v1/api/coordinator/`` → ``/v1/api/workstreams/`` move
non-negotiable since experimental SDK consumers from 1.5.0aN lose
the URL outright. Adds the path-mapping table under [Unreleased]
so the line lands in the 1.5.0 release notes when the version cuts.
Stable upgraders (1.0 / 1.3 / 1.4) never saw the coord URL prefix
so the change is a no-op for them — the entry says so explicitly.
Stage 2 Priority 0 Steps 0.4–0.7 — collapses the four migration
steps into one commit since they have to land together. The legacy
``/v1/api/coordinator/`` URL prefix never shipped in a stable release
(it appeared in 1.5.0aN experimental), so there's no compat carry-
forward — just rip and replace.
What moves:
- Step 0.4: deletes the eighteen ``Route("/api/coordinator/...")``
entries from ``console/server.py``. Coord traffic now flows
exclusively through the unified ``/v1/api/workstreams/`` shape
mounted via ``register_session_routes`` + ``register_coord_verbs``
(Steps 0.2 and 0.3).
- Step 0.5: rewrites the OpenAPI spec (``console_spec.py``) and
schemas (``console_schemas.py``, ``server_schemas.py``) to
document the new paths. ``test_openapi.py`` parity assertions
swap with them.
- Step 0.6: mechanical URL sweep across the frontend
(``console/static/app.js`` — 9 sites; ``coordinator/coordinator.js``
— 16 sites; ``index.html`` — 1 comment).
- Step 0.7: same sweep across the test suite
(``test_coordinator_endpoints.py``, ``test_coordinator_end_to_end.py``,
``test_coordinator_governance.py``, ``test_coordinator_close_all_children.py``,
``test_coordinator_client.py``, ``test_phase6_endpoints.py``).
Also touched:
- Server-side ``CoordinatorClient`` (``coordinator_client.py``) —
the coord agent's HTTP path for ``close_all_children`` updates
to the new shape.
- Handler docstrings in ``console/server.py`` say
``POST /v1/api/workstreams/...`` not ``/coordinator/...`` so a
``grep`` for a verb's URL lands on the right line.
- ``settings_registry.py`` setting descriptions, ``server.py``
cross-process error message, migration 042 docstring — all
updated to the unified shape.
The handler functions stay named ``coordinator_*`` until the
body-convergence follow-on lifts them into ``session_routes`` with
kind branching behind ``SessionRouteConfig`` flags. URL surface is
the only thing that changes here.
The ``test_session_routes`` route-walk now asserts the legacy paths
are GONE — previously it asserted both shapes coexisted. A future
accidental remount of ``/api/coordinator/`` would fail that test.
Stage 2 Priority 0 Step 0.3 — adds ``CoordVerbHandlers`` +
``register_coord_verbs`` to ``turnstone.core.session_routes`` and
wires the seven coord-only verbs (``children`` / ``tasks`` /
``metrics`` / ``trust`` / ``restrict`` / ``stop_cascade`` /
``close_all_children``) through it on the console.
These verbs are legitimately kind-specific — they read or mutate
state (children registry, parent quota, trust / restrict policy,
cascade controls) that doesn't exist on interactive workstreams —
so they live on a Protocol distinct from
``SessionRouteHandlers``. The unified URL prefix
``/api/workstreams/{ws_id}/`` is shared with the session verbs;
the separate registrar call keeps the kind separation explicit at
the wiring site.
Legacy ``/api/coordinator/{ws_id}/{verb}`` paths stay live during
the transition; both URL shapes resolve to the same handler
function. Step 0.4 deletes the legacy shape.
The route-table walk in ``test_session_routes`` now covers all
eighteen verb pairs (eleven session + seven coord-only) so a
future drift between legacy and unified handlers fails CI.
Stage 2 Priority 0 Step 0.2 — extends ``register_session_routes`` to
cover the per-``{ws_id}`` interaction verbs (``send`` / ``approve`` /
``plan`` / ``cancel`` / ``close`` / ``events`` / ``history`` /
``detail``) and wires ``console/server.py`` to mount coord at the
unified ``/v1/api/workstreams/`` shape.
The legacy ``/v1/api/coordinator/`` paths stay live during the
transition; both URL shapes resolve to the same handler functions.
Step 0.4 deletes the legacy shape outright once the frontend
(Step 0.6) and tests (Step 0.7) move off it.
Handler bodies still live in their server modules — body
convergence (kind branching behind ``SessionRouteConfig`` flags) is
the next Step 0.2 follow-on. Splitting "URL surface unified" from
"handler bodies converged" keeps the soak-able diffs small.
``mgr`` and ``adapter`` registrar arguments are now Optional because
the console builds its coord ``SessionManager`` inside the lifespan
(after app construction). They become required again in the
body-convergence follow-on once the lifted handlers read them.
New ``tests/test_session_routes.py`` covers the registrar's mounting
rules (route ordering, attachment-quartet enforcement, legacy-close
gate) and asserts that each unified path on the console points at
the SAME handler object as its legacy counterpart — the transition
is a pure URL alias, not a fork.
Stage 2 Priority 0 Step 0.1 — introduces
``turnstone/core/session_routes.py`` (``SessionRouteConfig`` +
``SessionRouteHandlers`` + ``register_session_routes``) and rewires
``server.py``'s ``/v1/api/workstreams/*`` route table to mount through
it. Pure scaffolding: handler bodies stay where they are, URL surface
is byte-identical, tests pass unchanged.
Sets up Step 0.2 to lift handler bodies into the registrar and have
the console mount the same shape against its coord manager.
Adds ``SessionManager.adapter`` accessor so the registrar can pick
up the kind adapter without callers re-threading it through every
construction layer.
* feat(core): scaffold SessionManager + SessionKindAdapter Protocol
Stage 1 step 1 — pure addition, no production wiring. Defines the
shape later steps will port the shared mechanics onto: slot
accounting, per-ws-id refcounted rehydrate locks, kind-agnostic
lifecycle; kind-specific event transport + session construction on
the adapter.
Pruned from the earlier Protocol draft (see design brief): per-kind
permission_scope (static handler map is simpler), allows_child_spawn /
quota_policy (deleted in #403), on_child_spawned (coordinator tool
owns children registry), allows_active_focus / active_id / switch
(frontend owns the active-tab state).
* feat(core): port shared session-lifecycle mechanics onto SessionManager
Stage 1 step 2. Adds create / open / close / set_state / close_idle /
get / list_all / count on top of the Step 1 scaffolding. Pure
addition — still no production wiring; the new class doesn't replace
any call sites yet.
Concurrency shape is ported from CoordinatorManager (the more-
complete side): single-phase slot reservation under the manager
lock, per-ws refcounted open-lock to serialize concurrent lazy
rehydrate, placeholder workstreams count toward max_active but can't
evict each other. WSM's two-phase eviction outside the lock is not
carried over; it had a window where a burst of creates could silently
exceed max_active.
Deletions (vs. the union of the two old managers):
- "refuse to close last workstream" guard — handled by the
dashboard; only existed to protect the now-deleted default startup
workstream.
- active_id / switch / get_active — frontend owns focus; server-side
duplicate state is gone.
- _active_coords presence cache — defer measurement to Step 4; if it
pays for itself at realistic cluster sizes, the CoordinatorAdapter
can maintain it by observing emit_* calls.
- Children registry + reverse index — coordinator tool owns this,
manager stays kind-agnostic.
Skill resolution (name → template_id + applied_version) is now
shared via SessionManager._resolve_skill, so WSM's pre-resolve-at-
callsite pattern and CM's internal-lookup pattern converge. Callers
pass the skill name; the manager does the lookup once.
26 smoke tests cover create eviction + overflow, concurrent-create
cap, persist/session rollback, open for missing/deleted/wrong-
kind/wrong-user rows, concurrent-open serialization, close unblocks
UI + emits closed, set_state + storage + adapter observer,
close_idle, list_all ordering, count, eviction fires adapter
transport, node_id passthrough.
* feat(core): add InteractiveAdapter for SessionManager
Stage 1 step 3. Adapter that bridges SessionManager to the node's
interactive transport:
- emit_created/state/closed → pushes onto the process-wide SSE
global_queue (same shape current server.py handlers produce inline)
- cleanup_ui → ports WorkstreamManager._cleanup_ui body: unblock
_approval_event / _plan_event / _fg_event, broadcast ws_closed to
per-UI listener queues (with full-queue fallback), cancel + close
the session
- build_ui/build_session → delegate to injected factories
(ui_factory builds WebUI, session_factory is the existing closure
from server.py with judge_model + memory_config captures)
Also extends SessionKindAdapter.build_session with **extra passthrough
so interactive callers can pass judge_model per-call without polluting
the manager API; and adds a reason= kwarg to emit_closed so the
frontend's "evicted" special-case keeps working (frontend doesn't
differentiate "idle" from "closed", so close_idle collapses into
close()).
14 new adapter tests cover wire payload shape, queue.Full tolerance,
cleanup_ui event unblocking + listener broadcast + queue-full
fallback, session cancel+close, graceful handling of stub UIs / None
session, and kwarg passthrough to the session factory.
* feat(console): add CoordinatorAdapter for SessionManager
Stage 1 step 4. Coordinator-side SessionKindAdapter implementation:
- emit_created/state/closed → delegate to the existing
ClusterCollector.emit_console_ws_* methods (same wire shape the old
CoordinatorManager emitted inline)
- cleanup_ui → ports the listener-queue + approval/plan event
unblocks from CoordinatorManager._cleanup, with queue-full
fallback so an unresponsive browser tab can't wedge close
- build_ui/build_session → delegate to injected factories; session
factory doesn't accept client_type so we strip it at the adapter
boundary
Collector emission exceptions are swallowed (same policy as today's
inline fan-out — dashboard lag on one tick is preferable to breaking
the lifecycle path).
Intentionally out of scope: the children registry (_children /
_child_to_coord) stays in the coordinator tool when wired in Step 5;
the _active_coords lock-free presence cache is deferred pending a
measurement at realistic cluster sizes. 10 new tests cover transport
payloads, collector-exception tolerance, cleanup_ui event unblock +
listener broadcast + queue-full eviction, construction passthrough.
* feat(server): wire interactive server.py to SessionManager
Stage 1 step 5a. Production-path swap: WorkstreamManager →
SessionManager(InteractiveAdapter(...)).
- Construction at server startup: build the adapter with the
process-wide global_queue, a WebUI ui_factory closure, and the
existing session_factory. SessionManager gets storage + max_active.
- Default startup workstream wiring removed (the CLI-REPL leftover
flagged in the handoff's "Convergence is also a pruning
opportunity" section). --resume now lazily creates a workstream
scoped to the resumed content; no workstream at all if --resume
isn't given. The dashboard handles the 0-ws state.
- HTTP handler mgr.create() calls switched to the new kw-only
signature (user_id, name, model, skill, ws_id, client_type,
judge_model, parent_ws_id). ui_factory/skill_id/skill_version/kind
no longer threaded through — adapter handles UI construction and
manager resolves skill internally.
- Dropped the mgr.last_evicted block in the /new handler (adapter
emits ws_closed:evicted automatically on capacity eviction).
- mgr.max_workstreams → mgr.max_active.
- Added active_id / switch / switch_by_index / get_active / index_of
/ eviction_count to SessionManager because turnstone/cli.py uses
them extensively; the handoff's "delete unless there's a live
caller" rule flips here — CLI is a live caller.
Test fixtures across 9 files updated to build SessionManager +
InteractiveAdapter rather than WorkstreamManager. test_workstream.py
stays unchanged (it tests WSM directly; it'll be deleted in step 5d
alongside the class itself).
Full pytest: 4528 passed. Ruff + mypy clean. Next: 5b (console-side
wiring, with the children-registry relocation to the coordinator
tool).
* feat(console): wire console server to SessionManager
Stage 1 step 5b. Production-path swap: CoordinatorManager →
SessionManager(CoordinatorAdapter(...)).
- CoordinatorAdapter now owns the coord-specific bits that were bolted
onto the old CoordinatorManager: the children registry (forward +
reverse index), the lock-free active-coords presence cache, the
cluster-event fan-out thread, and the worker-dispatch path
(send / _spawn_worker). The shared SessionManager stays kind-agnostic.
- Added CoordinatorAdapter.attach(mgr) for late-binding the owning
manager (the manager's ctor takes the adapter, so the dependency has
to break here). Used inside _rebuild_children_registry for the tenant-
filtered SQL query, inside send/dispatch for mgr.get(ws_id), and
inside the fan-out seed path for mgr.list_all().
- emit_created now seeds the children registry + active-coords slot AND
calls _rebuild_children_registry (covers both create — empty query —
and open/rehydrate, where the subtree is persisted). emit_closed
drops both entries. Collapses the three old call-sites in
CoordinatorManager's create/open/close into one per-event hook.
- Console server.py builds the manager via:
coord_adapter = CoordinatorAdapter(collector=..., ...)
coord_mgr = SessionManager(coord_adapter, storage=..., max_active=...,
node_id=ClusterCollector.CONSOLE_PSEUDO_NODE_ID)
coord_adapter.attach(coord_mgr)
ConsoleCoordinatorUI._coord_mgr = coord_mgr
app.state.coord_adapter = coord_adapter
- HTTP handler call-site updates:
- coord_mgr.create drops initial_message; the handler now calls
coord_adapter.send(ws.id, initial_message) after create so the
worker spawn stays out of the shared manager.
- coord_mgr.open_admin(ws_id) → coord_mgr.open(ws_id, user_id="",
admin=True). Matches SessionManager.open's unified signature.
- coord_mgr.list_for_user(uid) inlined as a list comp on list_all()
(SessionManager doesn't expose the filter; two callers).
- coord_mgr.children_snapshot / send → coord_adapter.*.
- coord_mgr.cancel stays (now lives on SessionManager from 5a).
- ConsoleCoordinatorUI.on_state_change now flows state transitions
through ConsoleCoordinatorUI._coord_mgr.set_state, mirroring the
WebUI pattern. The old _on_state_observer / _on_rename_observer
closures the manager used to install are dead code now; leaving the
fields in place for 5d cleanup.
- Lifespan shutdown calls coord_adapter.shutdown() (was coord_mgr.
shutdown()) and resets ConsoleCoordinatorUI._coord_mgr on teardown.
Test fixture updates in _coord_test_helpers, test_coordinator_end_to_end,
test_coordinator_endpoints, test_phase6_endpoints: build SessionManager
+ CoordinatorAdapter in _build_mgr, set app.state.coord_adapter, switch
mgr.register_children / mgr.children_snapshot tests to mgr._adapter.*,
and rewrite test_open_admin_uses_open_admin to assert the unified
open(user_id="", admin=True) call shape.
Full pytest: 4486 passed. Ruff + mypy clean. Next: 5d (remove
CoordinatorManager + WorkstreamManager class bodies and their test
files).
* feat(core): delete WorkstreamManager + CoordinatorManager classes
Stage 1 step 5c + 5d. Final step of the unification — the legacy
classes and their test files go away now that every production
caller has been ported.
- Delete turnstone/console/coordinator.py entirely (CoordinatorManager
class + the _enqueue_on_ui helper, which CoordinatorAdapter now hosts
its own copy of).
- Trim turnstone/core/workstream.py to just the Workstream dataclass +
WorkstreamKind + WorkstreamState. ~385 lines of WorkstreamManager
logic gone; the remaining shape is pure data types shared by both
managers.
- Delete tests/test_workstream.py (WSM-specific) and
tests/test_coordinator_manager.py (CM-specific).
- Wire turnstone/cli.py to SessionManager + InteractiveAdapter, same
pattern as turnstone/server.py. The CLI's WorkstreamTerminalUI uses
manager.set_state + manager.active_id — both preserved on
SessionManager (CLI is a live caller that keeps the focus API
honest, per the handoff's "delete unless it pulls its weight" rule).
- Add an optional manager-level ``_on_state_change`` observer hook
restored for the CLI's background-attention notification (the web
path uses the adapter's emit_state; this hook covers callers that
don't consume SSE).
- Drop dead ``_on_state_observer`` / ``_on_rename_observer`` fields
from ConsoleCoordinatorUI — the old CoordinatorManager installed
them; SessionManager/CoordinatorAdapter handle fan-out directly.
Vulture @ 80% confidence: zero unused symbols across the new
SessionManager + adapter files. Ruff + mypy clean (170 files).
Full pytest (excluding tests/live): 4414 passed.
Net across the whole Stage 1 branch: one unified SessionManager +
adapter Protocol replaces two ~500-line parallel managers + a
~600-line CoordinatorManager, and the interactive + coordinator
transports stay cleanly separated at the adapter boundary.
* refactor(auth): drop workstream row-level ownership gates
Turnstone is a trusted-team tool (per #400). user_id stays as
metadata for audit + display; it no longer rejects requests. Scope-
level auth via admin.workstreams / admin.coordinator tokens is the
only gate now.
Solves sec-1 (cross-tenant delete via collision on caller-supplied
ws_id, because the gate was half-implemented) and sec-2 (blank-sub
JWT bypass on empty-owner rows). Net: 359 lines of defensive
empty-string comparisons and admin=True bypass plumbing deleted.
* fix(core): serialize set_state vs close + worker spawn
Three concurrency fixes from the multi-stage review:
- bug-3: set_state now looks up ws under self._lock and gates its
storage write on ws._closed (a new tombstone flag). close() sets
ws._closed=True and does its storage write under ws._lock. A
set_state that acquires ws._lock after close sees the tombstone
and skips its write instead of resurrecting the closed row.
- bug-1: _spawn_worker wraps the check-and-spawn in ws._lock so two
concurrent send() HTTP requests can't both observe "no live worker"
and start duplicate worker threads on the same ChatSession.
- bug-2: replaces Thread.is_alive() as the reuse gate with an
explicit ws._worker_running flag. The flag is set before the worker
thread starts and cleared in its finally block — both under
ws._lock. Using is_alive() left a narrow window where the worker
could exit between the check and a queue_message call, stranding
the user's message with no consumer.
perf-2 (lock-held-across-DB-write) is accepted as-is: per-ws
serialization of state transitions behind a DB round-trip is real
cost but bounded — a given ws's state flips happen sequentially on
its worker thread anyway. Dropping ws._lock around the DB write
would reintroduce the bug-3 race.
Full pytest: 4401 passed. Ruff + mypy clean.
* refactor(core): drop _resolve_skill from SessionManager
Skill resolution (name → template_id + applied_version) moves out of
the shared manager and back to the HTTP handlers that own the
create request. The interactive handler already resolved skill_data
+ applied_skill_version for other purposes (model override, judge
config, post-create session seed) and was passing the name to
SessionManager which then redundantly re-resolved via
get_skill_by_name + count_skill_versions — two wasted DB round-trips
per create on a user-visible latency path.
- SessionManager.create: accepts skill_id + skill_version as
already-resolved kwargs; _resolve_skill helper deleted.
- turnstone/server.py create_workstream: passes the skill_id /
applied_skill_version it already computed.
- turnstone/console/server.py coordinator_create: pre-resolves
inline (parity with interactive) before calling coord_mgr.create.
Fixes perf-1 (redundant skill queries per create), q-4 (divergent
skill-version computation between manager and handler), q-5
(coordinator-specific lookup on the shared manager surface).
Full pytest: 4401 passed. Ruff + mypy clean.
* refactor(adapters): extract shared cleanup_ui + drop dead child-registry methods
Both InteractiveAdapter.cleanup_ui and CoordinatorAdapter.cleanup_ui
(plus their _broadcast_ws_closed_to_listeners helpers) were byte-identical.
Pull them into turnstone/core/adapters/_ui_cleanup.py:cleanup_session_ui
so the two adapters delegate to one implementation.
Also drop CoordinatorAdapter.register_children (only test callers — now
use _seed_children in tests/_coord_test_helpers.py) and _add_child
(zero callers anywhere).
* refactor(adapters): symmetric attach() + fail-loud on unattached manager
Add InteractiveAdapter.attach(manager) + .manager property mirroring
the coord-side pattern. CLI (cli.py) now uses cli_adapter.attach(manager)
instead of the _mgr_ref list-ref late-binding hack; server.py picks up
the same call for consistency.
CoordinatorAdapter.send / _rebuild_children_registry /
_prime_children_from_snapshot no longer silently return when
self._manager is None — raise RuntimeError so a forgotten attach() at
startup fails loud instead of dropping the whole fan-out.
* docs: replace stale WorkstreamManager / CoordinatorManager references
Both classes were deleted in 965e0b6; prose docstrings across the
codebase still named them. Update to SessionManager (or describe the
collapsed-into-one-class architecture where the distinction matters).
Leaves the 'Ported from …' historical markers in session_manager.py /
coordinator_adapter.py / interactive_adapter.py intact — those are
deliberate pointers back to the pre-unification code.
* fix(core): atomic close_if_idle + batch pop under one lock
bug-5: SessionManager.close_idle re-checked ws.state == IDLE outside
the lock, so a pending tool result could flip state IDLE→RUNNING
between the snapshot and close() acquiring self._lock. Add
_close_if_idle_locked that tests state + pops under self._lock.
perf-5: drop the per-victim self._lock acquisition; collect + pop the
whole batch in one acquisition, then run cleanup_ui / storage write /
emit_closed outside the lock.
* perf(coord): split emit_created / emit_rehydrated to skip storage query on fresh creates
CoordinatorAdapter.emit_created was unconditionally calling
_rebuild_children_registry (storage.list_workstreams with
parent_ws_id=... limit=10001) on every create, even for fresh-create
paths that provably have zero children.
Add emit_rehydrated to the SessionKindAdapter Protocol. SessionManager
.create still calls emit_created; .open (lazy rehydrate) now calls
emit_rehydrated. CoordinatorAdapter.emit_created seeds the registry +
fan-out but skips the rebuild; emit_rehydrated seeds + rebuilds + fans
out. InteractiveAdapter.emit_rehydrated delegates to emit_created (no
children-registry on the interactive transport).
* perf(coord): fold _active_coords into _children_lock + mutate payload in place
perf-4: _active_coords used a copy-on-write dict-swap pattern so the
fan-out dispatch could read it lock-free, but _dispatch_child_event
already re-validates the parent under _children_lock anyway — the
lock-free snapshot was premature. Replace with a plain dict read+write
both under _children_lock; install and remove collapse to one-liners.
Value also drops the user_id half — dead after a46dab1 removed
row-level ownership gates — so _active_coords is now just
coord_ws_id → ui.
perf-6: _enqueue_on_ui was doing {**payload, "ws_id": coord_ws_id} on
every dispatch. The dispatch path owns payload and doesn't reuse it —
mutate in place.
* test(coord): add adapter tests for worker dispatch + children registry + fan-out
Fills the coverage gap on CoordinatorAdapter — the review (q-3) flagged the
coord-specific concurrency paths ported from the deleted CoordinatorManager
as untested. Three new test classes:
- TestCoordinatorAdapterWorkerDispatch: _spawn_worker reuse gate, queue.Full
backpressure, concurrent-call bug-1 reproducer (two threads → exactly one
worker via ws._lock + _worker_running), finally-clears-flag.
- TestCoordinatorAdapterChildrenRegistry: registry seed on emit_created vs
emit_rehydrated rebuild, _pop_coord_registry_locked reverse-index cleanup,
_merge_child_ids_locked idempotency, _prime_children_from_snapshot merge.
- TestCoordinatorAdapterDispatchChildEvent: unknown-parent drop, ws_created
fan-out, cluster_state / ws_closed reverse-index routing, perf-6 in-place
ws_id stamp.
* fix: regressions flagged by ultrareview
Verify stage of the cloud review surfaced 6 confirmed regressions
from Stage 1's adapter layer. Fixing together since they share the
same root cause (plumbing moved into adapters without retiring the
old emission paths).
- Interactive adapter emit_created / emit_state / emit_rehydrated
become no-ops. The create_workstream HTTP handler still fires
ws_created (after attachment validation, per the pre-Stage-1
"no phantom events on rejected upload" contract); WebUI
_broadcast_state still fires ws_state with the full payload
(tokens + context_ratio + activity). Firing from the adapter too
was duplicating both events. Also closes the phantom-ws-created
regression (adapter fired before attachment validation ran).
- emit_closed Protocol gains a ``name`` kwarg; the adapter is the
sole emitter for ws_closed on interactive now, and the frontend
eviction toast needs the name. Manager passes ws.name from
close() / create()+open() eviction / close_idle paths.
- _idle_cleanup_thread stops firing its own reason="idle" ws_closed
— close_idle already fires via the adapter with reason="closed",
and the frontend never differentiated the two anyway.
- close_workstream_endpoint fix: "Cannot close last workstream" 400
was a stale error (the guard went away with the default-startup
workstream). Return 404 on close() == False (which now means the
ws was already closed or unknown). Also switches the audit actor
from _require_ws_access's stored owner to _auth_user_id — the
stored owner is metadata post-#400, so attributing actions to it
misrepresents who actually did them.
- CLI /ws close mirrors the same stale-error fix.
- SessionManager.close now calls storage.delete_workstream_override
alongside update_workstream_state, same as the old
WorkstreamManager.close did. Without it overrides leak until
tombstone cleanup. close_idle does the same.
- SessionManager._reserve_and_install_locked records the eviction
on turnstone.core.metrics so the global eviction counter keeps
working. Old WSM did this inline; the unification dropped it.
- ConsoleCoordinatorUI.on_rename now fans out to the cluster
collector via a new class attribute ``_collector`` (set at
console startup alongside ``_coord_mgr``). The old
``_on_rename_observer`` plumbing went away with
CoordinatorManager and the "adapter emit_console_ws_rename runs
from whichever code path renames" comment was aspirational —
nothing actually did it.
Full pytest: 4375 passed (tests/live + test_server_live.py excluded;
both pre-existing live-backend failures unrelated to this branch).
Ruff + mypy clean.
* refactor(ui): extract SessionUIBase for shared UI scaffolding
Direct response to review feedback that the unification wasn't
merging enough of the two workstream kinds. WebUI (node) and
ConsoleCoordinatorUI (console) both:
- Keep a per-UI list of SSE listener queues guarded by a lock
- Block a worker thread on _approval_event / _plan_event
- Fan enqueued events out with the same ws_id-stamping pattern
- Resolve approvals / plans with the same broadcast-then-signal
pattern
All of that now lives once in turnstone/core/session_ui_base.py.
Both UIs subclass SessionUIBase; kind-specific bodies (WebUI's
per-UI metrics + _broadcast_state + intent-verdict bookkeeping,
ConsoleCoordinatorUI's collector fan-out) stay in the subclasses.
WebUI.resolve_approval still overrides the base (it adds intent-
verdict updates) but now calls super() for the shared broadcast +
event-set steps. Same shape as the other approval/plan hooks:
subclasses extend, base provides skeleton.
Net file-level: +156 LOC for the base, -144 LOC across the two
subclasses. The raw number is unexciting — but there's now a
single source of truth for the listener + blocking-gate machinery,
and bugs (like the duplicate ws_created / ws_state events that
prompted this refactor) can't arise from the two implementations
drifting.
Full pytest: 4375 passed. Ruff + mypy clean.
* refactor(ui): move metrics + verdict bookkeeping into SessionUIBase
Second pass at unifying the two UIs. Per-workstream metrics
accumulators (token counts, tool-call counts, context ratio,
activity tracking), intent-judge verdict cache + pending-decision
list, and the verdict-persistence path all move to SessionUIBase.
Before: WebUI tracked all of it; ConsoleCoordinatorUI tracked none
of it (a comment on the old on_intent_verdict literally admitted
the deferral — "skip the persistence + late-decision plumbing that
WebUI does"). Coord sessions never got verdict rows in storage, never
had a user_decision stamped, and the dashboard had no way to show
coord token usage because the data wasn't captured.
Now the base class captures the data and persists the rows for
every kind. Kind-specific broadcast (WebUI's _broadcast_state with
rich per-UI payloads) stays on WebUI; prometheus counters on the
node (_metrics.record_judge_verdict) stay on WebUI's on_intent_verdict
override. Everything else shared.
Behaviour change worth flagging: coord sessions now write
intent_verdicts and output_assessments rows for every judge call
and every output-guard warning. Previously silent; the storage rows
now exist and any future coord-dashboard surface can read them.
Shape of the unification:
- resolve_approval: was overridden on WebUI (intent-verdict decision
propagation); now lives on the base. Both kinds inherit unchanged.
- on_intent_verdict: WebUI overrides only to add _metrics.record_*;
rest of the body is the base.
- on_output_warning: was on both separately; fully base-shared now.
Full pytest: 4375 passed. Ruff + mypy clean.
* fix: regressions flagged by second-pass review
Three confirmed findings with direct fixes + a dedicated test file
for SessionUIBase (was previously uncovered).
bug-1 — Coord approve_tools didn't reset _last_verdict_decision or
clear _llm_verdicts between approval rounds. WebUI did (inline).
Coord inherited SessionUIBase.on_intent_verdict which stamps via
the decision flag, so after the first resolve every subsequent
round's verdicts were stamped with the prior round's user_decision
before the user had decided the new round.
Fix: add SessionUIBase._reset_approval_cycle() clearing both under
_ws_lock; call from the top of both subclass approve_tools methods.
Single-source invariant — can't drift again.
sec-1, sec-2 — delete_workstream_endpoint and open_workstream's
rehydrate path recorded the audit row under the stored ws.user_id
("owner_uid") rather than the authenticated caller. With row-level
ownership gating gone (a46dab1), any team member acting on a peer's
workstream produced an audit row naming the victim as the actor.
Fix: pass _auth_user_id(request) as the audit actor, matching the
pattern close_workstream already follows.
q-2 — SessionUIBase had no direct tests. The new
tests/test_session_ui_base.py covers listener fan-out, approval +
plan blocking gates, intent-verdict cache + FIFO eviction, verdict
persistence paths, output-guard persistence, the reset-between-rounds
invariant (bug-1 regression test), a cross-subclass test that
verifies BOTH WebUI.approve_tools and ConsoleCoordinatorUI.approve_tools
call _reset_approval_cycle (verified it fails without the fix), and
a concurrent enqueue/register smoke.
Full pytest: 4395 passed (+20 new). Ruff + mypy clean.
* fix: PR #408 review findings from copilot + code-quality
Three substantive fixes + mechanical side-effect-in-assert cleanup.
Copilot findings:
- session_ui_base.py: on_intent_verdict had a race with
resolve_approval. Previously acquired _ws_lock twice (read decision
→ release → if unset, acquire again to append). resolve_approval
could interleave between the two acquisitions, swap-and-clear the
pending list and set the decision — our verdict then got appended
to the fresh (empty) list and stamped with the NEXT round's
decision on the following resolve. Fix: decision-check + append
under ONE acquisition; storage UPDATE (if decision already set)
runs outside the lock. New regression test counts lock
acquisitions during on_intent_verdict and fails if the two-phase
pattern returns.
- server.py close_workstream_endpoint: comment said "treat as
already-closed success" but handler returned 404. Comment
rewritten to match the 404 behaviour ("the ws isn't tracked here"
is the only reachable meaning for close() → False now).
- test_session_ui_base.py concurrency smoke: the test ended with
``pytest.assume = lambda ...`` — a leftover that mutates pytest
globals and can surprise other tests. Replaced with explicit
``not is_alive()`` assertions so the "threads completed cleanly"
intent survives -O optimization stripping.
Code-quality (assert side-effects):
Six ``assert mgr.open(...)`` / ``assert mgr.close(...)`` in
test_session_manager.py stripped under ``python -O``. Mechanical
fix: extract to local before asserting.
Ignored the two "Protocol method body is `...`" flags — that's the
standard Protocol idiom; replacing with ``pass`` or
``NotImplementedError`` changes typing semantics.
Full pytest: 4396 passed.
* chore(coord): remove spawn-quota subsystem
The quota gate was operator-level safety per its own comments, not a
security boundary, and never fired in a week of heavy use. Runaway
coordinator spawns are already bounded by max_active slot exhaustion,
which surfaces to the coord LLM as a tool error — same operational
shape, one fewer moving part. Precedes the Stage 1 SessionManager
unification so the coord tool doesn't inherit quota bookkeeping.
Upgraded deployments with the three removed settings persisted will
log three "Skipping invalid setting" warnings on startup and
otherwise degrade cleanly; a follow-up migration to delete the rows
would silence that noise.
* chore(migrations): drop stale coord spawn-quota settings rows (047)
Clears persisted rows for the three ConfigStore keys removed in the
previous commit so upgraded deployments don't log "Skipping invalid
setting" warnings on every startup. Downgrade is a no-op — the rows
were operator-set values, and a rollback to pre-1.5.0 code falls back
to the registry defaults for any key not present.
* fix(coord): render markdown on history reload
The coordinator chat's history-load path piped assistant content
through ``appendText`` → ``appendMsg(role, esc(text))``, which dumps
escaped raw text into the message body without ever calling the
markdown converter or the post-render hooks (highlight.js, mermaid,
KaTeX). Live streaming uses ``streamingRender`` /
``streamingRenderFinalize`` which DO render markdown, so a fresh
stream looked correct but a page-reload / reconnect surfaced every
table, code fence, and math block as literal characters.
Now the history loop dispatches by role: assistant + reasoning go
through ``streamingRenderFinalize`` (mirrors what live streaming does
on stream_end); tool messages keep ``appendToolResult``; user / system
stay on ``appendText`` since they're typed verbatim and don't carry
markdown structure.
* fix(coord): keep reasoning role on plain-text path on history replay
The history loop routed reasoning role through streamingRenderFinalize,
but live streaming renders reasoning tokens via textContent
(appendReasoningToken). History replay would render reasoning as
markdown while a fresh stream rendered it as plain text — inconsistent
look and unnecessary hljs / mermaid / KaTeX work on reasoning content.
Reasoning now uses appendText on replay, matching the live path.
Addresses Copilot review feedback on PR #402.
* fix(server): trusted-team workstream visibility on listing endpoints
The per-user filter on /v1/api/workstreams, /v1/api/dashboard, and
/v1/api/workstreams/saved (PR #375's _visible_workstreams helper) was
written for a multi-tenant SaaS threat model that doesn't match how
turnstone gets deployed. In a self-hosted, trusted-team install the
filter created friction without preventing the relevant threats — and
hid the auto-created name="default" startup workstream from every
web user, leaving fresh installs staring at a blank dashboard.
Listing endpoints now return the cluster-wide set to any authenticated
caller. Per-workstream MUTATIONS (/send, /close, /open, /title,
/delete, /refresh-title) keep their independent ownership checks — the
cross-tenant guards from PR #375 stay in force on those handlers (see
TestCrossTenant{Delete,Approve,Close,Title,Open}). Listing only
exposes metadata (name, state, kind, message_count); message history
still requires the per-workstream gate on /history.
Resuming a saved workstream still goes through /open's owner check, so
the metadata-leak surface ends at "you can see workstream X exists" —
not at any actionable cross-user capability.
The console collector's service-scope is now load-bearing only for the
SSE event stream gate (/v1/api/events/global); kept anyway as belt-
and-braces.
If turnstone is ever deployed as a true multi-tenant SaaS, the right
boundary is a real ``tenant_id`` column with row-level filtering at
the storage layer, not the empty-user_id heuristic this used to apply.
Tests updated to assert the new contract: listing returns all owners;
mutation gates unchanged.
* fix(server): repair test mocks + tighten docstrings on listing endpoints
- tests/test_auth.py: TestServerAuth + TestServerLogin mocks now set
kind / parent_ws_id / user_id explicitly so /v1/api/workstreams JSON-
serializes them. Bare MagicMock attributes return another MagicMock
that fails json.dumps and surfaces as 500.
- turnstone/server.py: list_saved_workstreams docstring corrected to
describe what the endpoint actually returns (summary metadata, not
history) and to spell out that ownerless persisted rows are claimable
by any authenticated caller via /open — consistent with the trusted-
team model the listing endpoints assume. Same callout added next
to the open_workstream ownership-gate block. Comments throughout
rewritten to be timeless (no "previously" / PR-number references).
- tests/test_server_authz.py: TestSaved... docstring matches the actual
/open behavior for orphan rows (claimable by any authenticated
caller, not a separate admin path).
* fix(chat): collapse phantom whitespace + tighten paragraph rhythm in markdown body
The assistant chat body was rendering 30-50px gaps between every
section. Two compounding causes:
1. ``.ts-msg-body`` had ``white-space: pre-wrap`` on the markdown
container. The custom regex-based markdown converter
(renderer.js) leaves ``\n`` text nodes between block siblings —
pre-wrap rendered every one of those as visible vertical space,
stacking ~14-16px between every heading/paragraph/katex-display.
2. No ``.ts-msg-body p`` margin override, so paragraphs fell back to
browser-default 1em top + 1em bottom (~28px stacked between any
two paragraphs). Headings already had a tight ``8px 0 4px`` rule;
paragraphs were the outlier.
Switched the body to ``white-space: normal`` and added a
``.ts-msg-body p { margin: 6px 0 }`` rule that matches the heading /
list / blockquote rhythm. Mirrored the paragraph rule on the design-
v1 ``.msg-body`` selector so both legacy and v1 surfaces stay in sync.
``<pre>`` blocks have ``white-space: pre`` built in so fenced code
still preserves formatting. Mid-stream partial fences (before the
closing ``\`\`\`` arrives) render as collapsed text for one frame and
then snap back when the next render tick wraps them in ``<pre>`` —
acceptable trade vs. the persistent gap regression.
User-typed messages render through ``.msg-user-text`` (a separate
DOM path), so this only affects assistant markdown output.
* fix(chat): preserve inline <code> whitespace under white-space: normal body
The body's ``white-space: normal`` (which collapses phantom inter-block
``\n`` text nodes from the markdown converter) inherits to inline
``<code>`` and silently collapses multiple spaces inside backtick
spans. ``<pre>`` blocks rely on the user-agent ``pre { white-space:
pre }`` rule and are unaffected; only bare inline code needs an
explicit override.
Adds ``white-space: pre-wrap`` to ``.ts-msg-body code`` (chat.css) and
the design-v1 ``.msg-body code`` selector so backtick-wrapped code
spans render verbatim while still wrapping on long lines.
Addresses Copilot review feedback on PR #401.
* feat(coord): saved coordinators surface + shared session-card primitives
The console home view now lists explicitly-closed coordinators in a
"Saved Coordinators" card grid below the active list. Click a card →
POST /v1/api/coordinator/{ws_id}/open then navigate; capacity issues
surface as a toast instead of a broken detail page. Card click is
de-duped by an `is-busy` class so rapid double-clicks don't fire
parallel resurrects.
GET /v1/api/coordinator/saved is the new backend endpoint (mirrors the
interactive list_saved_workstreams shape). Filters at the SQL layer
to state='closed' via a new optional `state` parameter on
list_workstreams_with_history (added to the protocol + both backends);
also drops any rows currently loaded into coord_mgr as defence in
depth. The blocking storage call + the lock-acquiring list_all are
offloaded via asyncio.to_thread to match coordinator_create's pattern.
CoordinatorManager._open_impl now allows resurrect of state='closed'
rows (deleted is still a tombstone). The DB state-flip on resurrect
that the first cut had is gone — it raced concurrent close()s and the
next set_state() call syncs the DB naturally; the saved list filters
already keep a still-loaded coordinator from appearing as a saved
card even when its on-disk state lags.
Frontend dedup that paid for the saved surface ships in the same diff:
- shared_static/cards.css: lifted from ui/static/style.css so both
surfaces share the basic card primitive (delete-mode rules stay
interactive-only until coordinator gets the same UX)
- shared_static/cards.js: new renderSessionCard(sess, opts) helper
used by both renderSavedWorkstreams (interactive) and
renderSavedCoordinators (console)
- shared_static/utils.js: formatRelativeTime moved here from
ui/static/app.js
Coordinator landing visual fixes folded in:
- .home-section-title now uses var(--accent) so the COORDINATORS
heading reads as a peer of the NODES heading
- .home-panel dropped its bg/border/padding so the composer is no
longer double-framed (matching the dashboard-composer feel)
- "Active coordinators" → "Saved Coordinators" rename + "Coordinators"
on the active list
ws_closed SSE handler now gates on the closed ws's kind so interactive
closes don't spam /v1/api/coordinator/saved on busy clusters.
loadSavedCoordinators in-flight de-dup coalesces close-event bursts to
one fetch instead of N.
Tests cover: caller-scoping, admin sees-all, blank-uid fail-closed,
loaded-coordinator filtering, state filter (idle rows excluded), plus
the manager-level open-resurrect / open-refuses-deleted contracts.
Closes the bug-{1,2,3}, perf-{1,2,3,4}, sec-{1,2}, q-{1,2,3,4,5,6,7}
findings from the prior multi-stage review.
* fix(design): restore amber accent on the v1 design system
The Claude Design handoff swapped the accent hue to teal (h=182).
Walking back to amber (h=75) — turnstone's original brand colour.
Lightness + chroma bumped slightly (0.62→0.7, 0.10→0.13) so the
restored gold matches the visual weight of the legacy #e5a042 token.
Hue map header comment updated to record what happened so the next
person doesn't repeat the swap. Only surfaces with data-design="v1"
on <html> pick this up — currently just turnstone-server's webui.
* chore: gitignore design_ideas/ and .claude/ dev directories
design_ideas/ holds personal Claude Design handoff scratch + reference
HTML; .claude/ holds per-user Claude Code state (worktrees, settings,
plugin caches). Neither belongs in version control.
* fix(coord): address PR #399 review nits
- tests/test_coordinator_endpoints.py: split `assert mgr.close(ws.id)`
in `_seed_closed_coord_with_history` so the close call always runs
even under `python -O` (asserts stripped). Same fix in
test_coordinator_manager.py's `test_open_refuses_deleted_coordinator`
for the open() and open_admin() calls.
- shared_static/cards.css: `.card-wsid` now reads `var(--font-mono, "IBM
Plex Mono", monospace)` so design-v1 surfaces pick up the JetBrains
Mono token while console (still pre-v1) keeps the literal fallback.
* feat(auth): inline refresh response + sessionStorage rehydrate hardening
The proactive refresh path now consumes the /refresh response body
inline (permissions + exp), eliminating the chained /whoami round-trip
and the brief stale-sessionStorage window after refresh succeeds but
before whoami completes.
Adds AbortController + _loggedOut guards to the whoami fetch so a
logout fired mid-flight cannot re-populate sessionStorage after it
clears. A non-OK whoami on tab restore now explicitly clears
sessionStorage instead of silently leaving stale cosmetic permissions
(server-side identity gone → UI gating reflects it on next render).
Surfaces window.permissionsReady (one-shot promise) so permission-
gated UI can await the initial whoami's completion instead of guessing
a setTimeout duration.
Tests cover the new refresh response shape, the existing leeway path,
the storage-failure fallback, and the no-perms 403 path.
Closes the bug-3 / perf-4 / sec-1 / q-6 findings from the multi-stage
review of the prior uncommitted change set.
* fix(auth): guard whoami superseding race in _scheduleRefreshFromWhoami
_scheduleRefreshFromWhoami is invoked from several entry points
(initial page load, _onSuccess, BroadcastChannel "login"/"refresh",
_tryRefresh fallback). Two firing in quick succession could let an
older slow whoami land after a newer one and clobber its effects —
clearing permissions right after a successful login, or rescheduling
the refresh timer off stale exp.
Now aborts any prior _whoamiAbort before starting a new request and
guards the .then's _storePermissions / _scheduleRefreshAt with a
`_whoamiAbort === ctrl` check so a late arrival from a superseded
call is fully neutralised.
Addresses Copilot review feedback on PR #398.
* feat(providers): add gpt-5.5 and gpt-5.5-pro capability entries
OpenAI announced gpt-5.5 on 2026-04-23 (ChatGPT/Codex first, API
"very soon"). Mirror the gpt-5.4 / 5.4-pro capability shape: 1M
context, native tool search, vision, xhigh effort; pro is
always-reasoning with no temperature and medium/high/xhigh only.
No provider-logic changes needed — OpenAI announced no API-surface
changes vs 5.4. Cache retention already covers 5.5 via the existing
startswith("gpt-5") prefix rule.
* test(providers): cover gpt-5.4-pro and gpt-5.5-pro in cache retention test
Pro variants share the same gpt-5 prefix and should keep 24h
retention; explicit coverage guards against regressions if the
prefix rule narrows in the future.
Remove the weekly Trivy scan job and the .trivyignore exclusion file.
The scanner has been flagging base-image CVEs that require no action
on our part (upstream-only fixes) and has provided no actionable
signal, while breaking CI on an ongoing basis.
* feat(auth): cookie refresh endpoint, JWT leeway, coord-token observability
Three robustness wins around the auth/JWT layer.
1. POST /v1/api/auth/refresh — handle_auth_refresh in core/auth.py,
wired in both console/server.py and server.py. Sliding-window
re-mint of the auth cookie. Re-resolves the user's permissions
from storage so a role change propagates within one refresh cycle
instead of persisting until the original cookie's natural expiry.
Returns the same JSON shape as /api/auth/login plus a fresh
Set-Cookie header. Refuses to extend a session for a deleted /
role-stripped user (403).
Resolves the user-visible "401 after browser tab open >24h"
symptom: previously the only refresh path was a full re-login,
now a single POST extends the session.
2. validate_jwt now passes leeway=30 to PyJWT. Absorbs minor
clock skew between hosts (multi-replica console deployments) and
between mint-time and validate-time within the same process.
Standard tolerance for short-lived tokens.
3. CoordinatorTokenManager._mint logs at debug. Mirrors the pattern
in ServiceTokenManager._mint (auth.py). Premature-401 diagnostics
would have been an order of magnitude faster with this in place
the first time around.
Frontend (shared_static/auth.js):
- _scheduleRefreshFromWhoami() reads the JWT exp surfaced via /whoami
and sets a setTimeout at 90% of remaining cookie life to call
/refresh. Floor 30s, ceiling 24h. Fires on initial page load
(silent if not authenticated) and after every successful login.
- _tryRefresh() de-dupes concurrent callers via a shared in-flight
promise — many parallel authFetch's hitting 401 at once still only
fire one /refresh.
- authFetch on-401 now attempts a single reactive refresh-then-retry
before falling through to the login overlay. Covers cases where
the proactive timer didn't fire (tab restored from disk-cache after
expiry, system clock jump, page first-load with stale cookie).
- BroadcastChannel "refresh" message keeps sibling tabs in sync so
they don't redundantly hit /refresh themselves.
- logout() cancels the proactive timer.
Tests:
- validate_jwt accepts 10s-expired tokens (within 30s leeway).
- validate_jwt rejects 60s-expired tokens (past leeway).
- /whoami includes exp claim with sane bounds.
- /refresh returns ok + Set-Cookie + the refreshed cookie keeps
working on subsequent authenticated requests.
- /refresh without a cookie returns 401.
Not addressed: the coordinator.session_jwt_ttl_seconds ceiling
(currently 1h) — that's a separate, preventative concern for very-
quiet long-running coordinators, orthogonal to the user-visible 401
this PR fixes. Can bump in a follow-up if it actually surfaces.
* fix(auth): address Copilot PR #395 feedback
Two real bugs caught by Copilot, both fixed.
1. Storage failure was indistinguishable from "user deleted" in
handle_auth_refresh. _load_user_permissions() swallows exceptions
and returns set(), so a transient DB hiccup looked like
"user has no permissions" and returned 403 — logging the user out.
Now calls storage.get_user_permissions() directly with try/except.
- Exception → log + fall through to in-token claims (refresh succeeds
with stale-but-valid permissions; better than fail-closed mid-
session for a hiccup).
- Empty set returned (no exception) → 403 (legitimate signal: user
deleted or role-stripped).
Tests:
- test_refresh_storage_failure_falls_back: storage raises → 200 +
in-token permissions.
- test_refresh_user_with_no_perms_403: storage returns empty → 403.
2. Logout race: a /refresh in flight when the user clicks Logout could
land AFTER /logout's clear-cookie response and re-set the cookie
from /refresh's Set-Cookie header, silently undoing the logout.
Fix in shared_static/auth.js:
- Add a _loggedOut latch + _refreshAbort AbortController.
- logout() sets _loggedOut = true synchronously and aborts any
in-flight /refresh BEFORE the /logout fetch fires.
- _tryRefresh() bails on its post-fetch effects (don't store perms,
don't reschedule, don't broadcast) when _loggedOut is set. The
stale Set-Cookie from /refresh is harmless because /logout's
response overwrites it on the way back.
- _onSuccess() (re-login) clears the latch so subsequent refreshes
work again.
Race window is small but real on slow networks / contested CPU.
* feat(design-system): DS phase 2 — opt server chat UI into v1 primitives
ui/static/index.html:
- data-design="v1" on <html> opts this view into design system tokens
and primitives scoped under the attribute selector.
- Link DS stylesheets after the legacy cascade: tokens + typography +
appbar (chrome) + panel / buttons / pills / message / field
(primitives). Legacy /shared/base.css, /shared/ui-base.css,
/shared/chat.css, and /static/style.css stay linked to handle
anything not yet migrated (rich markdown, tabs, dashboard, split
panes, approvals, modals).
- Header <div id="header"> picks up .appbar + .appbar-title +
.appbar-status + .appbar-spacer + .appbar-actions alongside the
legacy .ts-header classes. Theme-toggle gets .btn for DS pill shape
while keeping .header-btn for palette continuity.
ui/static/app.js:
- Chat message elements emit both legacy and DS class names so the
DS primitive picks up the message surface while legacy .ts-msg--*
rules keep view-specific markdown styling (tables, callouts, katex,
mermaid, hljs). Pairs:
ts-msg ts-msg--user → + msg user
ts-msg ts-msg--assistant → + msg assistant
ts-msg ts-msg--reasoning → + msg reasoning
ts-msg ts-msg--info → + msg info
ts-msg ts-msg--error → + msg error
ts-msg-body → + msg-body
- Approval blocks keep legacy-only styling — their shape is distinct
from the DS .msg primitive (the DS approval-dock pattern is a
fixed bottom dock, not inline-in-chat).
No backend or wire-format changes. SSE events, POST bodies, endpoint
URLs, ARIA attributes, and keyboard shortcuts all unchanged.
* feat(ui/static): DS-skin tool-call + approval + verdict internals
The outer .ts-msg.ts-approval--inline picked up DS .msg styling via
PR #2's dual-class approach, but the inner structure kept rendering
with legacy yellow/green/red colours and legacy chip shapes. Result:
a DS-accent-bordered card containing a mustard tool-name, a clunky
uppercase-yellow verdict chip, and a mismatched auto-approved pill.
Add [data-design="v1"]-scoped overrides that reskin the inner
vocabulary onto DS tokens:
.ts-approval-tool panel-over-panel-2 card with hair border
.tool-name accent (teal) for tool-kind identity
.tool-cmd / .tool-diff ink-2 text; diff-del/add/warn → err/ok/warn
.verdict-badge.verdict-* chip aesthetic matching DS k-badge —
low=ok-tinted, medium=warn-tinted,
high/critical=err-tinted, with a
3px left-border semantic stripe
.verdict-detail panel-2 callout with structured rows
.verdict-judge-spinner ts-pulse animation (reuses primitive)
.ts-approval-badge--* pill shape hugging max-content, matches
DS approve-button-family colour palette
(ok-text, err-text-mix)
.tool-output panel bg, hair border, accent stream
left-border, fade-gradient on collapse
.ts-verdict-glow--* soft ring on the corresponding action
button (approve=ok, deny=err, review=warn)
No JS changes; DOM shape unchanged. CSS-only reskin so approval flow,
tool streaming, and verdict expand/collapse behaviour all stay intact.
* fix(ui/static): consistent tool-card width + flat badge aesthetic
Two fixes to the DS-skinned tool-call rendering:
1. Tool-call cards were sizing to their content (short output →
narrow card, long output → full-width), producing a jagged column.
Force .ts-msg.ts-approval--inline to width: 100%; align-self:
stretch; box-sizing: border-box; so the chat column reads evenly.
2. The "approved" / "auto-approved" pill was styled as a button
(pilled shape, 1px bg-tinted border, 4x10 padding) which read
as clickable. Switched to a flat badge aesthetic matching the
.risk primitive: 3px-squared, 2x6 padding, 10px mono uppercase
on a --ok-soft / --err-soft tinted surface, no border. Reads
as a status tag, not a call-to-action.
* fix(ui/static): address Copilot PR #392 feedback
Copilot findings, all applied:
- Drop the legacy Outfit + IBM Plex Mono Google Fonts link. DS
typography.css @imports Inter + JetBrains Mono; loading both stacks
on opted-in pages wastes downloads and triggers FOIT/FOUT differences.
- Drop the .ts-header-title class on the <h1>. Its legacy rule forces
font-family: var(--font-display) (Outfit) which overrides the DS
appbar typography. .appbar-title alone is sufficient under v1.
- Override .ts-msg font-family under [data-design="v1"] when .msg is
also present (and not the .tool variant). Legacy .ts-msg forces
mono; DS user/assistant/reasoning/info/error messages should use
the UI font. .msg.tool keeps mono via the primitive's own rule.
- Replace the inline name.style.color = "var(--red)" in buildToolDiv
with a .tool-name--error class. Inline styles win over CSS rules
and broke the DS token mapping (legacy --red is not the DS --err).
- Correct the header comment in style.css for the approval-block
overrides. Prior comment claimed the outer wrapper picks up DS .msg
styling; it doesn't — the dual-class approach wasn't extended to
approval blocks. Updated comment to match actual DOM.
* feat(design-system): DS phase 3 — coordinator chat migration
Opt the per-session coordinator view into data-design="v1" and migrate
its rendering to the DS primitives + patterns shipped in phase 1. This
is the larger of the two parallel chat migrations (the other being the
server UI under turnstone/ui/static/).
Scope — this PR touches two files only:
turnstone/console/static/coordinator/index.html
- data-design="v1" on <html>; DS stylesheets linked after the legacy
base so primitives win on specificity and legacy styles keep
covering anything not-yet-migrated.
- Header rewired from .ts-header to .appbar with .appbar-back,
.appbar-title + .dim subtitle, .appbar-spacer, .appbar-status for
SSE state, and .appbar-actions wrapping the cancel / end / theme
buttons (now .btn pills).
- Approval bar replaced with the .approval-dock pattern. Signature
change: amber Approve becomes an ok-family (green) filled button
with 1.5px border + --r-md squared shape. .dcall rows frame each
pending call like a mini inspectable code line. Action cluster
sits in a .drow with Deny (.act.danger) / Always (.act.always) /
Approve (.act.primary) and the preview's kbd affordances
(D / ⇧A / ⏎). role="region" + aria-live="assertive" preserved;
the dock stays non-modal (no focus trap), focus moves to the
primary Approve button on open via the existing handler.
- Sidebar shell adopts .sidebar + .side-section + .side-label +
.ghost refresh buttons. Coordinator-only .sidebar overrides unset
the DS sticky-left-column defaults (which assume an admin-shell
grid) so the aside continues to flex into the right column of
#coord-body. Tree-row + task-row styling stays view-local,
rehomed to DS tokens (--hair-2 hover, --accent focus, --ok/--warn/
--err + -soft task-status tints).
- Inline <style> trimmed of rules now covered by DS primitives;
only the coordinator-specific flex wiring, tree-row visuals, and
<700px responsive accordion remain.
turnstone/console/static/coordinator/coordinator.js
- appendMsg() emits .msg + role variant (.msg.user / .msg.assistant /
.msg.reasoning / .msg.tool / .msg.error / .msg.info) and .msg-body.
_TS_ROLE_VARIANTS renamed _MSG_VARIANTS.
- Streaming helpers query .msg-body; SSE dedup-by-call-id query
updated to .msg[data-call-id=...].
- showApproval() renders the .approval-dock DOM shape: .dhead count
in a .dcount, one .dcall per pending call with .risk index pill +
.dfn function name + .dargs preview. approvalBar.hidden toggles
visibility (the DS pattern is position: fixed and always-rendered;
[hidden] is the show/hide hook).
- setSseStatus() keeps .appbar-status as the base; semantic colour
tracks OK / ERR via inline --ok / --err. Leading glyph (●/○/⚠)
preserves the WCAG 1.4.1 non-colour-only cue.
- Wait indicator uses .appbar-status instead of the legacy
.ts-header-status BEM; styling from the inline page rules colours
it --think.
Contracts preserved:
- SSE wire format and event names unchanged (approve_request,
child_ws_created, wait_progress, batch_started, state_change,
stream_end, ...).
- POST /approve body shape unchanged: {approved, always, call_id}.
No per-item feedback field is added (that's phase 9 PR C).
- Keyboard behaviour unchanged: Enter continues to approve via the
primary-button focus shift in showApproval(); the D and ⇧A kbd
labels are rendered per the pattern spec but the global key
handlers (if any) remain untouched.
- ARIA attributes (role, aria-label, aria-live) preserved on the
approval dock, messages log, and sidebar.
- All shared-static JS imports and order unchanged; composer module
continues to own its own DOM inside #coord-composer-mount.
No backend changes. Legacy CSS (/shared/base.css, /shared/ui-base.css,
/shared/chat.css, /static/style.css) stays linked as the compatibility
layer — DS selectors [data-design="v1"] beat legacy where applied.
* fix(coordinator): inline approval dock above composer, not viewport-pinned
The .approval-dock DS pattern defaults to position: fixed; bottom: 22px
— designed for the fleet dashboard where the dock overlays content. In
the coordinator chat that rule pinned the dock to the viewport bottom,
covering the composer input area.
Move the dock DOM back inside #coord-main between #coord-messages and
the composer mount so it flex-stacks naturally above the input. Add a
view-local override that neutralises the fixed positioning (position:
static, z-index/box-shadow auto) while preserving the visual pattern
(warm top stripe, head/call/actions rows, dashed Always button).
Drop the 160px bottom-padding hack on #coord-messages since the dock
is now in-flow and naturally pushes the message log up.
Also likely resolves the Firefox initial-render issue — position:fixed
+ [hidden] toggle had cross-browser quirks where the dock wouldn't
appear on first SSE approval event until a separate DOM mutation
forced a reflow. In-flow layout makes it boring and predictable.
* fix(coordinator): integrate judge verdicts into approval dock, not chat
The judge's intent_verdict is evaluation context for the pending
approval, not a chat message. Previously each verdict appended a
"[judge] deny (risk=low)" tool message into the transcript even when
the corresponding approval was visible in the dock — two separate
surfaces showing related decision context, neither one complete.
Now:
- Each .dcall row gets data-call-id from the approve_request item
- intent_verdict looks up the matching row and renders a .dctx sibling
below it with "judge: <recommendation> (risk: <level>)" + optional
"confidence: <score>" chips. Reasoning attaches as title tooltip.
- Verdicts cache in a Map<call_id, verdict> so late-arriving
approve_request events can still apply verdicts that came early
- Fallback to the old chat-message surface only when the approval isn't
visible (call_id missing, or resolved before we could render) so the
verdict isn't silently dropped
* feat(coordinator): judge verdict polish — colour-coded chips, spinner, reasoning
Three refinements to the approval-dock judge integration:
1. Colour-code verdict chips by recommendation — approve=green (--ok),
review=amber (--warn), deny=red (--err). Reviewers can triage at a
glance without reading the chip text; complements the text label
for WCAG 1.4.1 (non-colour-only signaling).
2. Spinner while evaluating — when showApproval builds a .dcall row
without a cached verdict, render a "judge evaluating…" chip with
a spinner. Replaced in-place when intent_verdict arrives. Reuses
the ts-spin keyframe from primitives/feed.css.
3. Justification inline — judge.reasoning is delivered in every
intent_verdict event but was hidden behind a title tooltip. Now
renders as a wrapped prose block (.drationale) below the .dctx
chips, styled like the .msg-body .evi callout (left-rule + mono
+ --ink-3). Full text, no truncation — justification is the whole
point.
View-local styling; the approval-dock pattern itself is unchanged.
If these patterns turn out to be broadly useful, they can promote to
shared_static/design/patterns/approval-dock.css in a later PR.
* fix(coordinator): defer approve-button focus until judge verdict arrives
The Approve button was getting focus the instant the approval dock
opened, which lit up the green focus ring and made the filled-green
button look pre-confirmed. A reviewer could mistake that for "already
approved" before the judge has even returned a verdict.
Now focus is deferred until the intent_verdict for the first-pending
call arrives, then moves to:
- Deny button when judge recommends "deny" (safety default)
- Approve button for "approve" / "review" / anything else
Fallback timer (3s) claims focus anyway if no verdict arrives — covers
disabled judge and slow judge cases so keyboard users still land on a
button within a beat.
Focus claim is idempotent so batch approvals don't bounce focus across
buttons as trickling verdicts arrive. hideApproval clears the timer
and the claimed flag so re-open cycles start fresh.
* fix(coordinator): drop approve-focus fallback timer
Previous commit added a 3s fallback that focused Approve if no verdict
arrived. Ambiguous — a focus ring that lands "eventually" looks the
same as one that lands because the judge recommended approve.
Now focus only ever moves when a real intent_verdict arrives. If the
judge is disabled or the verdict never comes, focus stays put and
keyboard users tab from the composer to reach the buttons. An absent
focus ring is a clearer signal than an ambiguous one.
* fix(coordinator): address Copilot PR #393 feedback
Copilot findings, applied:
- Restore <h2> for Children / Tasks sidebar section labels (were
changed to <span>). .side-label class still applies; screen readers
recover heading-level structure + rotor navigation.
- Mount the wait-indicator into #coord-header (the appbar container)
instead of #coord-status. #coord-status is reset via
statusEl.textContent = ... on every state_change event, which was
clobbering the wait indicator between ticks. As a sibling inside
the appbar, it survives state updates.
- Route `info` SSE events to appendText("info", ...) so they render
with .msg.info (think-indigo) styling. Prior routing to "tool"
gave info events accent-tinted tool-call styling, miscategorising
them visually.
- Define @keyframes ts-spin locally in the coordinator's <style>.
Canonical definition lives in primitives/feed.css but this page
doesn't link feed.css (no .feed-item usage), so the "judge
evaluating…" spinner wasn't animating.
- Clear judgeVerdicts Map in hideApproval. Map was growing unbounded
across resolve cycles — fine for short sessions, leaks memory on
long-lived coordinators with many approvals.
Not applied: Copilot's suggestion to restore focus-on-open or add a
fallback timer. User explicitly requested no fallback — the design
decision is that the focus ring should only ever appear when the
judge has returned a verdict, so an absent ring reliably means "no
recommendation yet." An auto-focus fallback would produce an
ambiguous ring that could be misread as "judge approved."
* feat(design-system): DS phase 1 — chat primitives for view migrations
Three new primitives enabling the chat-surface migrations (server UI +
coordinator):
primitives/message.css .msg + variants (user / assistant /
reasoning / tool / error / info / system),
.msg-meta author/timestamp slot, .msg-body
markdown target, .msg-actions hover-revealed
row, data-streaming="true" blinking caret.
Replaces .ts-msg* family in chat.css.
primitives/field.css .field wrapper with label/help/error, element
selectors for text/email/password/url/number/
search/tel/date/time/datetime/month/week +
textarea + select. .field.inline for checkbox/
radio rows, .field.invalid for error state.
Native-control focus-visible handled for
checkbox+radio so box-shadow ring remains
visible on unframed controls.
chrome/appbar.css chat-app header: back link + title + status +
action cluster. Distinct from the admin-style
.topbar (brand mark + nav + env metadata).
min-width:0 on .appbar-title so .dim subtitle
ellipsis fires under narrow viewports.
Preview.html extended with three demo sections exercising every variant
(plus a data-streaming example with live caret).
Fixes carried in from code review:
- @media (hover: none) and (pointer: coarse) to match chat.css
convention (hover-none alone is too broad, catches styluses)
- .field-help uses --ink-3 (not --ink-4 which fails AA on --panel)
- .msg-meta slot added so downstream PRs don't invent a custom class
- Tool message pre/code on --panel-2 (parent is --panel; same-bg
would make inline code disappear)
- Checkbox/radio :focus-visible override (native controls lack a
border for the default box-shadow ring to wrap)
- Message.css comment corrected: "accent-tinted" not "cyan"
All rules scoped under [data-design="v1"]. Nothing existing modified.
* fix(design-system): address Copilot PR #391 feedback
- .msg-actions: add pointer-events: none when hidden, auto when visible.
opacity:0 alone still intercepts clicks in the top-right corner —
broke text selection on short one-line messages. Toggle applied in
both hover/focus-within and the touch-media-query visible states.
- .field.inline comment: rewrite to match behaviour. Old comment said
".field stays flex-column" but the rule sets flex-direction: row.
- preview.html appbar demo: swap <a tabindex="0"> back-link to
<button type="button">. tabindex-only anchors without href have
inconsistent focus + screen-reader semantics; button is the correct
native element for "navigate back via JS."
Live-preview-driven tuning pass following PR #389:
Palette
- Accent hue 70 (amber) → 182 (teal). Amber collided with warn on
same-surface k-badges; teal gives the brand accent its own hue.
- ok / warn / err / think unified at L=0.50 light / L=0.68 dark and
C=0.13-0.17 for palette coherence. err holds higher chroma so red
doesn't wash; warn stays in the gold 80 lane (never 90+ / "puke").
- Soft variants unified at L=0.94 / L=0.29, C=0.05-0.07.
New tokens
--ok-live brighter green for liveness signals (running dot)
--ok-text theme-aware text colour for filled green surfaces,
dark forest in light / bright mint in dark, ~7.5:1
against the approve-button bg in both themes
--err-fill darker red specifically for filled destructive
surfaces (.risk.crit) — bright --err as a fill
reads as alarm-loud
--warn-tint, directly-defined gold tints for k-tools k-badge —
-tint-border skips the color-mix-through-dark-cool-panel mud
that would otherwise render warm low-L mixes brown
Approve / Always / Deny
- Approve filled green (color-mix --ok 28% into panel); text uses
--ok-text for theme-correct contrast. Matches the pre-refactor
turnstone/shared_static/chat.css convention where approve = green.
Deviates from the Claude Design spec which had warn-tinted approve.
- Always outlined dashed green (same --ok hue family); four non-colour
cues for WCAG 1.4.1: fill state, border style, label, position.
- Deny unchanged (err-outlined).
k-badge glyphs
Replaced generic shapes with semantic symbols:
tools ⚙ approval ⚠\FE0E policy § role ◉
oidc ⌘ token ◆ judge ⚖\FE0E query ?
step ⇧ session ◈ skill ★ workstream ⇉
fanout ⇶ default ·
⚠ and ⚖ carry \FE0E to force text-presentation (avoid emoji
promotion to coloured yellow triangle / blue scales on iOS Safari).
token uses ◆ instead of ⬢ for universal font coverage.
k-approval split from k-tools
k-tools stays gold (--warn family) — "tool call" kind.
k-approval moves to green (--ok family) — matches the Approve button
visually, completing the "⚠ approval → Approve" same-family story.
Running pill
Text uses --ok (passes AA on pale --ok-soft); dot uses --ok-live +
pulse. Liveness signal lives in the dot, not the text.
All changes stay under [data-design="v1"] — existing views untouched.
* feat(design-system): DS-A — tokens + typography scaffold
Adds turnstone/shared_static/design/{tokens.css,typography.css} as the
first phase of a multi-PR design refactor seeded by Claude Design.
- tokens.css: full palette + shape + rhythm, light default with
[data-theme="dark"] override. oklch() raw colours, color-mix kept out
of DS-A entirely (reserved for primitives in DS-B).
- typography.css: Inter + JetBrains Mono via Google Fonts; six-step
scale (10/11/12/13/14/20-24). Utility classes .t-kicker/.t-meta/
.t-btn/.t-row/.t-body/.t-stat/.t-h1.
Signature accent stays warm amber (oklch hue 70) rather than Claude
Design's teal — preserves turnstone's "Instrument Panel" identity.
All other tokens match the spec verbatim.
Additive: both files gate under [data-design="v1"] so existing views
(base.css, per-view stylesheets) are untouched. DS-B will opt views in
one at a time.
* feat(design-system): DS-B — chrome + primitives + preview page
Adds the reusable primitive kit that DS-C and DS-Cluster will build on:
primitives/
panel.css .panel, .panel-head (.tools pinned right), .ghost
buttons.css .btn (pill 999px), .primary, .deny, .approve (amber)
pills.css .pill (running/thinking/attn/idle/err), .k-badge
(glyph-prefixed per WCAG 1.4.1), .chip, .risk
stats.css .stat + .stat-row, .mini-bar, .spark
feed.css .feed-item (grid ts/body/acts + .evi callout)
chrome/
topbar.css 48px sticky, conic-gradient brand mark
sidebar.css 240px sticky, .shell layout, semantic swatches
preview.html renders every primitive in both themes with an
in-page theme toggle (tracks prefers-color-scheme)
Additive: every selector scopes under [data-design="v1"] so existing
views (base.css + per-view stylesheets) stay untouched.
Spec deviations from the Claude Design prototype:
- `color-mix(in srgb, …)` throughout; prototype had two `in oklab`
usages — srgb per the spec's hard rule
- `.btn.approve` is warn-tinted amber, not green
(approvals signal "needs attention"; amber resolves on approval)
- k-badge tint uses `color-mix` instead of oklch relative-colour syntax
for broader browser support
- `@keyframes pulse/spin` renamed to `ts-pulse/ts-spin` to avoid
clashing with keyframes in base.css on pages that load both
- `prefers-reduced-motion` disables pulse + spin animations
- Text-on-accent-soft + text-on-warn-tinted darkened via color-mix
with ink to pass WCAG AA at 12px (fixes the classic same-hue trap)
- `.risk.crit` uses `#fff` text (dark-mode --panel on bright err fails)
- `.feed-item .acts button:not(.btn)` — compact action styling now
skips .btn-classed buttons so they keep their pill shape
- Focus-visible rings on .btn, .ghost, .stat, .topnav, .side-item
* feat(design-system): DS-C — patterns (approval-dock, fleet-grid, live-feed)
Completes the design library with three patterns that compose primitives
into the signature product surfaces described in the Claude Design handoff.
patterns/
approval-dock.css bottom-pinned approval strip. 1.5px-border,
--r-md squared action cluster: amber Approve
(primary), dashed Always, red Deny. kbd hints
and focus-visible rings on all three acts.
Call row (.dcall) framed as an inline code-
panel to emphasize "this is the exact call."
fleet-grid.css 14-col grid of .node squares. State modifiers
(.s-ok/.s-thinking/.s-attn/.s-err/.s-idle/
.s-unreach) + --pct load fill. Hover uses
outline, not box-shadow (neighbour bleed is
the intended density cue). .fleet-legend
swatch row below.
live-feed.css thin scroll-container wrapper over the
.feed-item primitive with a sticky top fade.
preview.html imports the three patterns, extends the
fleet demo to use the real .fleet class +
legend, adds a live-feed panel, renders the
approval dock fixed at the bottom with
aria-live="polite".
Spec notes:
- Approve button is amber (warn-tinted), never green
- Dock action buttons are 1.5px-bordered 6px-radius squares — NOT
pills — signaling "primary-action surface"
- All three dock actions clear WCAG AA in both themes via the same
color-mix-with-ink darkening pattern used in .btn.approve
- kbd hint color matches primitives/buttons.css (--ink-3, not --ink-4)
View-level rewrites (coordinator.html + coordinator.js opt-in,
admin/cluster dashboard rebuild) are follow-up PRs — they need a
running server to test SSE streams + the approval POST contract.
* fix(design-system): scope DS-A tokens to [data-design="v1"]
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
* fix(design-system): scope DS-A font vars to [data-design="v1"]
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
* fix(design-system): align dark-mode selector with theme.js convention
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
* fix(packaging): add shared_static/design/** to wheel includes
Agent-Logs-Url: https://github.com/turnstonelabs/turnstone/sessions/74d0939a-c55f-46b0-92f4-14d0cbfb7084
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
* docs(coordinator): phase 8 PR C — API tour, skills guide, bulk-endpoints contract
Four deliverables that close out the phase 8 doc debt carried since
phase 1:
- docs/coordinator-api-tour.md — 9-step lifecycle walkthrough
(create → subscribe → send → inspect children / detail → wait for
fan-out → govern (trust / restrict / stop_cascade / close_all_children)
→ approve / cancel → close), one request + response per step, every
SSE event type the UI has to handle, and every operation id cross-
referenced against the live /openapi.json. Integrators driving a
coord session from a custom UI or SDK can work end-to-end from this
doc without reverse-engineering the console page.
- docs/coordinator-skills.md — writing a SkillKind=COORDINATOR skill.
Tool-surface diff (13 orchestration tools, no bash / edit / web /
sub-agent), persona diff (orchestrator vs maker, composing on
base_coordinator.md), SkillKind enum + migration 044, task_list
integration, ws_id handling, wait vs inspect cost profile, three
orchestration patterns (delegate-and-summarise, fan-out-and-
synthesise, plan-then-delegate), testing surface.
- docs/bulk-endpoints.md — codifies the two shape idioms that shipped
across phases 6–8: {results, denied, truncated} for bulk-read /
bulk-create-with-payload (cluster/ws/live, spawn_batch); {<bucket>,
failed, skipped} for cascade-mutation (stop_cascade,
close_all_children). Picks-by-semantics guidance so the next bulk
endpoint author doesn't coin a third shape.
- docs/diagrams/27-coordinator-wait-for-workstream.puml + rendered
PNG — sequence diagram covering spawn → wait (blocking, with
bounded progress emission) → inspect → close. Embedded in the
API tour doc's §6 so the "why is my coord session blocking?"
question has a visible answer.
No code changes. All operation ids in the API tour verified against
a live build of the console spec; all markdown internal links
resolve; PlantUML renders clean on the system plantuml jar.
* docs(coordinator): address PR #388 copilot review
- api-tour.md child-event payload keys: events stamp `ws_id` as the
coord's own id and carry the child's id separately as
`child_ws_id`. Doc previously listed `ws_id` as the child
identifier on all four child_ws_* events, which would send SDK /
UI implementers parsing the wrong field.
- api-tour.md SSE table: add the `status` event emitted by
ConsoleCoordinatorUI.on_status (token usage + context_window +
effort snapshot; fires on every streaming tick). Previously
omitted from the "every event type a UI has to handle" list.
- api-tour.md /children response key: server returns `{items,
truncated}`, not `{children, truncated}`. Also drop the
`state=closed` query-param claim — the endpoint has no state
filter; clients filter locally on the returned `state` field.
- skills.md task_list shape: the persisted row uses `id` (not
`task_id` — the input schema uses `task_id`, the row uses `id`),
has `child_ws_id` / `created` / `updated` (no `notes` field),
and supports a 5th `reorder` action alongside add/update/remove/
list. Adds the parallel-dispatch caveat from the tool
description.
- skills.md tenant-guard behaviour: foreign / hallucinated ws_ids
don't return an empty result — they return explicit
error/not-found/denied shapes that differ by op (mutating ops
return `{error, status: 404}`; inspect returns `{error}`; wait
reports state=denied). Important distinction — a skill that
expects empty on mismatch will mishandle every single case.
Docs-only; no code / schema / SDK changes. All internal links
still resolve.
* feat(coordinator): phase 8 PR B — spawn budget + rate limit + /quota endpoint
Adds two complementary controls so a runaway coordinator can't saturate a
cluster's max_active without anyone noticing:
- **Spawn budget** (hard quota) — cap on concurrently active children.
Default 20 per coord. spawn_workstream returns a tool error guiding
the model to close idle children; spawn_batch routes overflow rows to
`denied[]` with partial-success semantics.
- **Spawn rate limit** (soft pacing) — classic token bucket, defaults
5 tokens/minute with burst 10. A rate-limited spawn surfaces a tool
error carrying `retry after Ns` so the model paces itself. Zero
refill rate is honoured as "disable refill" (bucket still honors the
initial burst).
Shipped infra:
- `turnstone/core/spawn_quota.py` — thread-safe `SpawnBudget` +
`TokenBucket`. 15 unit tests.
- `turnstone/core/session.py` — coord-only state built from settings at
__init__. Shared `_eval_spawn_quota(active)` helper drives both the
single-spawn path (wraps the denial reason in `_coord_tool_error`) and
the batch path (annotates `spec["_error"]`). `_count_active_children`
routes through `coord_client.list_children(include_closed=False)` and
fails *open* on lookup error (budget is operator-safety, not security).
- `POST/GET /v1/api/coordinator/{ws_id}/quota` — partial-update admin
endpoint mirroring the /trust + /restrict shape. Accepts either the
nested `spawn_rate` object or flat aliases — supplying both for the
same field returns 400 so the admin UI can't half-migrate silently.
Overrides are in-memory only (die on session reopen). Audits via
`coordinator.quota.updated` with before/after snapshots.
- Settings: `coordinator.spawn_budget`, `coordinator.spawn_rate.tokens_per_minute`,
`coordinator.spawn_rate.burst` with ranges 1..500 / 0..600 / 1..500.
The range bounds are the single source of truth — the endpoint
validators and Pydantic schema both import from `settings_registry.SETTINGS`
so bumping a cap in one place lights up everywhere.
- OpenAPI: `CoordinatorQuotaRequest` / `CoordinatorQuotaResponse` /
`CoordinatorSpawnRateState` schemas + endpoint specs. TS SDK regenerated.
Tests: +15 unit (SpawnBudget + TokenBucket), +17 endpoint (GET + POST
happy paths, range edges, mixed-body rejection, non-object spawn_rate,
service-token refusal), +11 session-side (budget blocks single spawn,
budget batch partial-success, rate batch partial-success, empty-body
reject, mutator live-update, non-coord session has no quota state).
Deferred (not this PR): per-skill scoping via migration 047 +
`prompt_templates.spawn_budget` column. Count-only storage helper
(opportunistic — list_children at budget ≤ 500 is fine behind a
human-gated approval flow).
* fix(coordinator): address PR #387 copilot review
- Budget undercount: _count_active_children used list_children's
LIMIT-then-Python-filter path, so a fan-out with many recently-closed
children could push live rows past the SQL LIMIT and silently
undercount, leaking spawn slots past the budget. Replace with a new
CoordinatorClient.count_active_children that uses
storage.count_workstreams_by_state (SQL aggregate, no pagination,
sums non-terminal states). Tenant-guarded; fails open on storage
error (budget is operator-safety, not a security gate). New client
tests cover the non-terminal count, the closed/deleted exclusion,
the foreign-parent guard, and the fail-open path.
- Service-token bypass on /quota: both GET and POST used the default
allow_service_bypass=True, so a service token whose user_id matched
the coord owner could read or *raise* spawn capacity without the
explicit admin.coordinator grant. Flip both to
allow_service_bypass=False for consistency with /restrict,
/stop_cascade, and /close_all_children.
- OpenAPI contract leak: CoordinatorSpawnRateState was used for both
the request and response shapes, which let generated SDKs imply
clients could POST tokens_available (a read-only bucket reading the
handler ignores). Split into CoordinatorSpawnRateInput (request:
tokens_per_minute + burst only) and CoordinatorSpawnRateState
(response: adds tokens_available). No runtime behaviour change;
SDKs regenerate with two distinct types.
Drops the _ACTIVE_COUNT_SLACK / _ACTIVE_COUNT_MIN_LIMIT constants in
session.py — no longer needed since the new helper takes no limit
argument. Updates the 5 session-side quota tests to stub
count_active_children instead of list_children.
* feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools
Adds two model-facing batch tools so a coordinator can fan out without burning one approval per child:
- `spawn_batch` — create up to 10 child workstreams in a single approval. Serialised
spawns so sibling ordering (by created_at) stays deterministic. Returns
`{results: {idx: {ws_id, name, node_id, status}}, denied: [{idx, reason}]}`.
Per-item validation / spawn failures surface in `denied[]`; the batch hard-errors
on >10 rather than silent truncation.
- `close_all_children` — soft-close every direct child in one approval. Server-side
Sem(16) fan-out via `coord_client.close_workstream`; `reason` propagates to every
closed child's audit + workstream_config. Response mirrors `stop_cascade`'s cascade
idiom: `{closed, failed, skipped}` where `skipped` is upstream-404 / already-gone.
Shipped infra:
- New console endpoint `POST /v1/api/coordinator/{ws_id}/close_all_children`
(gated `admin.coordinator`, `allow_service_bypass=False`, 512-char reason cap,
`coordinator.closed_all_children` audit).
- Shared `_fanout_on_children` helper — both `stop_cascade` and `close_all_children`
now delegate to it (one place to own the snapshot → semaphore-gather → bucket-split
skeleton).
- `CoordinatorClient.close_all_children(reason)` plus a `_post_url` seam that
`_post` now reuses (no more duplicated transport-error handling).
- `_emit_batch_event` — best-effort SSE emitter modelled on `_emit_wait_event`.
Emits `batch_started` / `batch_ended` pairs keyed by call_id. Throttled
`batch_progress` deferred to a follow-up.
- OpenAPI request + response schemas, endpoint spec entry, TS SDK regenerated.
- Persona doc (`tools_coordinator.md`) covers the two new patterns.
Bulk-endpoint shape policy (codified in PR C later): split by semantic category —
`{results, denied, truncated}` for bulk-read / bulk-create-with-payload (cluster/ws/live,
spawn_batch), `{<bucket>, failed, skipped}` for cascade-mutation (stop_cascade,
close_all_children). No retrofit needed on stop_cascade.
Tests: new `test_coordinator_close_all_children.py` (8 endpoint tests), expanded
`test_coordinator_tools.py` (session-side prepare/exec, coord_client=None guards,
batch SSE events), expanded `test_coordinator_client.py` (route map, client method,
transport errors), tool-count assertions updated.
Deferred (not this PR): per-item selective-deny approval UI, throttled batch_progress
SSE, coordinator-skills doc + bulk-endpoints doc (PR C), spawn budget / rate limit (PR B).
* fix(coordinator): address PR #386 copilot review
- coordinator_client.close_all_children: pass the unformatted path template
as log_path so telemetry aggregates don't fragment per session (ws_id
still lives in the real URL).
- session.py: drop dead spawned_ids accumulator in _exec_spawn_batch —
leftover from an eager-register path that got removed earlier.
- close_all_children tool JSON: document the 512-char server-side cap on
reason and that reason is echoed back in the response payload. Added
maxLength:512 on the schema property so the LLM sees the constraint.
- CoordinatorCloseAllChildrenRequest: add Field(max_length=512) so the
OpenAPI schema reflects the runtime 400-on-overflow constraint.
* refactor(ui): shared composer widget (pane / coordinator / coord-create)
The interactive workstream pane (turnstone-server), the coordinator
session view (turnstone-console), and the console home's "start a new
orchestration task" form had drifted into three unrelated composer
implementations with different DOM, different class names, and
different behaviour sets. All three now build on a single
`shared_static/composer.js` widget parameterised by feature flags.
The widget owns the textarea, send button, optional stop button,
optional attach button + file input + chip container, optional
drag-drop / paste-image wiring, optional queue-while-busy send-label
rotation, optional touch-aware Enter-to-send, and an optional
collapsible Options panel with input / select fields, live summary
chip, and localStorage-persisted open/closed state. A stacked layout
puts the textarea above the action row for creation-form consumers;
the inline layout keeps the chat-style single row for send composers.
Consumer wiring:
- Pane: attachments + stopBtn + queueWhileBusy + drag-drop; keeps
its own attachment-upload pipeline and routes file events through
the composer's onAttach callback. Pane-specific CSS (.pane-stop
/ .pane-send.queue-mode) retired; `.ts-composer-stop` / `.ts-
composer-send--queue` in shared/chat.css take their place.
- Coordinator send: just textarea + send with touchEnterSends=true
to preserve the pre-refactor tap-to-send behaviour on tablets.
The header-mounted coord-cancel-btn stays (different semantics
than a per-generation stop).
- Coord-create: stacked layout, rows=3, Start-labelled send, Options
dropdown holding Name + Skill; Ctrl/Cmd+Enter handler scoped to
the composer mount. `_createCoordinator` lost its DOM-ref shape
in favour of raw values + a setBusy callback; a single
`_refreshHomeCoordSubmitEnabled` reconciler owns the submit
button's disabled flag so the 503 probe and in-flight submit
can't race each other.
Shared chat.css grew the .ts-composer-stop, .ts-composer-options-*,
and .ts-composer--stacked blocks; the coord-create consumer dropped
its custom .home-composer-task/-row/-name/-skill/-submit selectors
and the "Start a new orchestration task" panel title so the
placeholder text carries its own context, matching the webui
dashboard's clean look.
The visible behaviour on each surface is intentionally the same as
before; the change is structural — the three composers can no longer
drift apart silently.
* fix(composer): review fixes — Enter guard, single disable owner, widget-owned stop reset
Review of the squashed whole caught issues that the piecewise reviews
missed because they only become visible with all three consumers
together:
- **Enter bypassed sendBtn.disabled.** Composer's Enter keydown
handler called _fireSend() without checking sendBtn.disabled. In
the coord-create flow submitHomeCoord doesn't clear the textarea
before the POST completes (it redirects on success), so two rapid
Enter presses both fired _createCoordinator and could create two
coordinators. Enter now mirrors the click path.
- **Two writers to sendBtn.disabled.** Composer.setBusy and
_refreshHomeCoordSubmitEnabled both wrote the flag. They agreed
in sequence today but it was the exact drift hazard the reconciler
was meant to eliminate. Added an externalDisable option; when
true Composer's setBusy rotates labels / placeholder / stop button
but leaves sendBtn.disabled to the caller's reconciler. The
coord-create composer opts in.
- **setSendLabel dead weight.** Called on every busy transition
with static "Start" / "Starting…". Composer.setBusy now rotates
labels universally (not just in queueWhileBusy mode); the busy
label goes at construction via the existing busyLabel option and
setSendLabel is removed.
- **Pane reached through composer to reset stopBtn.** Pane.setBusy
was writing stopBtn.textContent / aria-label / dataset after
delegating — internals leaking through. Composer.setBusy now
resets the stop button's standard label + clears forceCancel on
every transition (matching the comment that used to live in Pane);
Pane drops the reach-through.
- **destroy() left detached DOM reachable.** Back-refs (inputEl,
sendBtn, etc.) are nulled out so post-destroy access fails loudly
instead of silently mutating detached nodes.
- **_maybeAutoResize dead indirection.** The enabled-check folded
into autoResize itself.
* fix(composer): busyLabel context-sensitive default + busyPlaceholder universal
Round-two review caught three related loose ends:
- Default `busyLabel="Queue"` was fine when label rotation was queue-
mode-only, but became misleading after the earlier review fix made
rotation universal: non-queue consumers calling setBusy(true)
without explicit busyLabel would flash "Queue" on the disabled
button. Default is now context-sensitive — "Queue" when
queueWhileBusy=true, sendLabel otherwise (no rotation). The
coord-send composer no longer needs to touch the label at all.
- `busyPlaceholder` JSDoc implied universal swap on busy but the
implementation gated it on queueWhileBusy. Decoupled — the
placeholder swaps whenever busy, with callers that don't set
busyPlaceholder seeing no visible change because it defaults to
the idle placeholder.
- `options.toggleLabel` and `options.onChange` were supported by the
implementation but undocumented. JSDoc for the options shape
enumerates every supported key + its default.
* refactor(routing): replace hash-ring rebalancer with rendezvous (HRW) hashing
Routing was a stored bucket table maintained by a central rebalancer
daemon, which shared its liveness primitive (services.last_heartbeat)
with the collector — when a heartbeat-fresh node went into a zombie
HTTP-handler-broken state, neither the collector nor the rebalancer
could self-correct, and the router kept directing traffic at it.
Rendezvous hashing makes the route a pure function of (ws_id,
live_services) so the heartbeat is the single source of truth and any
liveness-eviction propagates to the next route call without a separate
state-publication step.
The rebalancer's central state has no analogue: the new router computes
the per-key node winner on every call, the collector pushes membership
updates into the router cache from its discovery thread, and per-route
overrides survive on workstream_overrides. Eager workstream migration
goes away; in-flight workstreams lazily rehydrate from storage on the
new owner — already the dead-node behaviour.
* fix(tools): describe rendezvous re-routing on spawn/inspect node_id
The first pass overclaimed `node_id` "stays canonical for this
workstream's lifetime" — under rendezvous routing the active owner
re-derives per-call from live membership, so a node join/drop after
spawn can shift it. Tool descriptions now say `node_id` is the
spawn-time binding; subsequent ops re-route via rendezvous over the
current live-node set; the new owner lazily rehydrates from shared
storage; coordinators should re-read with inspect_workstream rather
than caching the value.
* feat(coordinator): phase 7 — governance + skill metadata + cross-cutting invariants
Combines three stacked sub-PRs into a single coordinator phase-7
shipment against the phase-7 plan doc. The sub-PR structure (0 / A /
B) preserved on individual branches for reviewer drill-down; this
branch is the one reviewers should merge.
## Sub-PR 0 — service-auth boundary invariants
Shared helpers and contracts that lock the console ↔ node service-auth
boundary so later authz surfaces use them by construction.
- ``_effective_user_filter(request)`` in both ``turnstone.console.server``
and ``turnstone.server`` with a shared ``DENY_EMPTY_SUB`` sentinel
on ``turnstone.core.auth``. Three-way return — admin/service
bypass, scoped caller uid, or fail-closed sentinel on blank sub.
Four callsite migrations (``_coordinator_rows``,
``coordinator_children``, ``coordinator_metrics``,
``cluster_ws_live_bulk``).
- ``StorageBackend`` class docstring codifies the tenancy contract
(every list/count/aggregate method must accept ``user_id: str |
None = None`` and push ``WHERE user_id = :user_id`` into SQL) and
the ``_mapping`` row-access contract. New
``turnstone.testing.row_contract`` ships ``assert_row_like()``.
- ``_verify_collector_service_scope`` probes an upstream node at boot
with ``expected_node_id=_scope-probe_``; a 409 proves the scope
gate was passed, a 403/401 sets ``collector_scope_error`` and
causes ``cluster_snapshot`` / ``cluster_events_sse`` to return 503
with a remediation hint. Probe URL allowlist rejects non-http(s)
schemes and 169.254.0.0/16 hosts.
- 4xx log-level floor on ``_NodeDashboardCache.get``,
``_fetch_live_block``, and ``_proxy_sse`` — dotted-hierarchy
prefixes with bounded body previews. ``_bounded_body_preview`` and
``_bounded_stream_preview`` strip control chars.
## Sub-PR A — coordinator governance core
Mid-session governance surface for coordinator workstreams.
- **Trusted-session mode.** New ``coordinator.trust.send``
permission (migration 042). ``ChatSession.set_trust_send`` /
``revoke_tools`` methods with a ``_governance_lock``. ``POST
/v1/api/coordinator/{ws_id}/trust {send: bool}`` double-gated on
``admin.coordinator`` AND ``coordinator.trust.send`` with
``allow_service_bypass=False`` so service tokens can't escalate.
``_prepare_send_to_workstream`` auto-approves sends whose target is
in the coordinator's own subtree; foreign ws_ids still require
approval. ``_is_own_subtree`` checks both ``parent_ws_id`` AND
``user_id`` to defend against cross-tenant row corruption.
- **Audit-layer credential redaction.** ``record_audit`` walks
``detail`` (dicts, lists, tuples, sets, frozensets; keys too)
and routes every string through ``redact_credentials`` + a C0
control-char scrub. New kw-only ``raw_detail=True`` opt-out.
``_has_any_string`` fast-path. Audit action registry extended
with the four new governance sub-prefixes.
- **Mid-session revocation + cascading stop.** ``POST
/v1/api/coordinator/{ws_id}/restrict {revoke: [...]}`` caps 256
entries / 128 chars; ``_prepare_tool`` short-circuits with a
tool-error. ``POST /v1/api/coordinator/{ws_id}/stop_cascade``
cancels the coord's in-flight generation then dispatches
``cancel_workstream`` for every direct child in parallel via
``asyncio.gather`` bounded by ``Semaphore(16)``. Per-child
outcomes split into ``cancelled`` / ``failed`` / ``skipped``
(404 = already-gone rather than dispatch-broken). Both endpoints
apply ``allow_service_bypass=False`` on the admin gate.
- **Shared plumbing.** ``_resolve_coord_session`` helper collapses
the handler prelude three endpoints shared. ``_emit_coord_audit``
wraps ``record_audit`` in a dedicated ``ThreadPoolExecutor``
(``app.state.audit_executor``) so audit bursts don't starve cancel
dispatches. ``_require_json_object`` guards body parsing so non-
object JSON returns 400 instead of 500.
## Sub-PR B — skill metadata governance
- **Description validator (migration 043).** ``prompt_templates``
rows now require a non-empty ``description``. Existing empty rows
get backfilled with a ``"Skill: <name>"`` placeholder on upgrade.
The installer (``admin_skill_discover``) and MCP prompt sync both
synthesise a placeholder when the upstream description is blank
so non-admin write paths satisfy the invariant.
- **Skill kind classifier (migration 044).** New
``prompt_templates.kind`` column (``interactive`` / ``coordinator``
/ ``any``; defaults to ``any``). New
``turnstone.core.skill_kind.SkillKind`` StrEnum is the single
source of truth; Pydantic schemas type ``kind`` as ``SkillKind``
(OpenAPI advertises the enum) and the handler validator catches
the ValueError. ``list_skills_filtered`` gains a
``kinds: list[str] | None = None`` SQL filter.
``CoordinatorClient.list_skills`` defaults to
``kinds=["coordinator", "any"]`` so interactive-only skills are
hidden from the orchestrator.
- **``scan_status`` → ``risk_level`` rename (migration 045).**
Lossless column rename to align with ``IntentVerdict.risk_level``
terminology. Swept storage (both backends + schema + protocol),
handlers, API schemas, tool JSON, generated OpenAPI specs,
TypeScript SDK types, frontend (``governance.js``), tests, and
English prose in ``docs/judge.md`` + ``docs/tools.md``. The
user-facing on-load warning now reads ``has risk level:
{risk_tier}``. Tool JSON's ``risk_level`` enum corrected to the
scanner's actual taxonomy (``safe / low / medium / high /
critical``; was the never-shipped ``clean / flagged / unscanned /
pending``). Historical migration 021 left untouched.
## Migrations
042 (``coordinator.trust.send`` perm — PR A)
043 (description backfill — PR B)
044 (``kind`` column add — PR B)
045 (``scan_status`` → ``risk_level`` rename — PR B)
All four use position-anchored permission strings / host-side
parse-filter-rejoin on downgrade where SQL ``REPLACE`` could
corrupt prefix-overlapping values.
## Verification
- ``ruff check turnstone tests`` clean.
- ``mypy turnstone`` clean on 165 source files.
- ``pytest -m "not live"``: 4431 passed (+85 over the phase-6
baseline). Includes +32 tests in ``tests/test_service_auth_boundary.py``
and +38 in ``tests/test_coordinator_governance.py``; shared fixtures
extracted to ``tests/_coord_test_helpers.py``.
- Generated OpenAPI JSON (``sdk/typescript/openapi-{console,server}.json``)
regenerated via ``sdk/typescript/scripts/generate-types.py``; zero
``scan_status`` occurrences remaining outside the historical
migration 021 and the rename migration 045.
## Security reviews
Both reviews flagged by the phase-7 plan (items 1 + 5, plus 0a's
refuse-to-serve gate) ran through the multi-stage ``/review``
pipeline twice per sub-PR; all confirmed findings landed in-branch.
* fixup(phase-7): CI lint + PR #383 review fixups
Addresses the lint CI failure (ruff format) plus 12 findings from the
two automated PR reviewers.
Copilot:
- ``_sqlite.list_installed_skill_urls`` / ``_postgresql.list_installed_skill_urls``
used positional row indexing (``r[0]``/``r[1]``/``r[2]``) while this
same PR's ``StorageBackend`` class docstring forbids it. Switched
both to ``r._mapping["..."]`` access.
- ``list_skills.json`` previously advertised ``risk_level=""`` as a
filter for unscanned skills, but the implementation treats empty
strings as "no filter". Clarified the tool description to say
omit the filter entirely to include unscanned rows, and added an
explicit ``enum`` on the parameter restricting it to the scanner
tiers. ``_prepare_list_skills`` keeps the ``strip() or None``
normalisation — unscanned filtering now has an unambiguous contract.
- ``test_storage_skills_filtered.test_risk_level_filter`` used the
legacy ``clean`` / ``flagged`` values from the pre-rename column.
Rewritten with the scanner's actual taxonomy (``safe`` / ``high``).
github-code-quality (CodeQL):
- ``test_deny_sentinel_is_singleton`` previously asserted
``cs.DENY_EMPTY_SUB is cs.DENY_EMPTY_SUB`` — an identical-expression
comparison. Rewritten as two separate ``from ... import ... as`` aliases
(``FIRST_READ`` / ``SECOND_READ``) so the identity check is between
distinct bindings.
- ``test_restrict_empty_revoke_is_noop_but_audits`` unpacked ``state``
without using it. Renamed to ``_state``.
- Mixed import styles in ``test_service_auth_boundary.py`` — the
file previously used both ``import turnstone.console.server as cs``
and ``from turnstone.console.server import ...`` for the same
module (same story for ``turnstone.core.auth`` and
``turnstone.server``). Consolidated to the ``from X import Y`` style
used elsewhere in the file; the ``_fetch_live_block`` test now
patches via pytest's ``monkeypatch`` fixture instead of a manual
rebind through a module alias.
CI:
- ``ruff format`` reformatted one line in
``tests/test_coordinator_endpoints.py``.
Verification: ruff check + mypy clean (166 files); 4459 non-live
pytest pass.
* fix(tests): swap asyncio marker for anyio in service-auth boundary tests
PR #383 CI caught that the 13 ``@pytest.mark.asyncio`` decorators I
added in ``test_service_auth_boundary.py`` are an off-convention
choice — the rest of the repo uses ``@pytest.mark.anyio`` (148 sites
vs my 13). The CI environment pulls in ``anyio`` but not
``pytest-asyncio``, so every async test in this one file was failing
with "async def functions are not natively supported". It passed
locally by accident — my dev venv happens to have pytest-asyncio
installed ambiently.
Swapped all 13 marker sites to ``@pytest.mark.anyio``. No functional
change; the tests run under the same default asyncio backend anyio
provides.
Verification: ruff + mypy clean (166 files); 4459 non-live pytest
pass.
* refactor(channels): backfill review of Slack/Discord adapters
Retrospective multi-stage review of the Slack (PR #355) and Discord
channel adapters — they shipped before the review pipeline existed,
so this pass goes back and fixes everything the pipeline would have
caught plus a follow-up round of ultrareview findings.
## Security (8 fixes)
- Adapter-side owner checks on all interactive flows: Discord
ApprovalView / PlanReviewView encode the owner Discord user ID in
the embed footer (`{ws_id}|{corr_id}|{owner_id}`) and reject
non-owner clicks; Slack plan-approve / request-changes /
feedback-modal gain owner tracking in `_pending_plan_review_ts`
and a shared `_ensure_plan_review_owner` gate. These closed the
two critical authz gaps where the gateway's service-scoped JWT
bypassed server-side ownership checks.
- Discord thread-message gate: only the registered invoker can
drive the workstream (prevents a linked user posting in another
user's public thread from injecting into their assistant).
Invoker recorded explicitly so `/ask` follow-ups survive the
`channel.create_thread` bot-as-owner quirk.
- Slack /link flow + per-user identity gate: unlinked Slack users
see an ephemeral `/turnstone link <token>` prompt on every
message instead of silently creating workstreams under the
shared gateway identity. Rate-limited (5/hour) to block online
token enumeration.
- Gateway `/v1/api/notify` requires `write` scope on the validated
JWT; low-scope tokens get 403 + audit.
- Thumbnail URL validator DNS-resolves the hostname before fetch
and rejects any resolved IP that's loopback / link-local /
multicast / reserved, plus an explicit deny-list for IPv6 cloud
metadata (`fd00:ec2::/32` — AWS Nitro IMDS + ECS task metadata)
that would otherwise slip past the `is_private` allowance.
- Per-user rate limit (10 msgs / 60s) + 8 KiB inbound size cap on
Slack DMs / channels / notification-reply threads so one user
can't exhaust the shared LLM budget.
- Discord /link rate limit (5/hour) for token-enumeration defense.
## Bug fixes (9 correctness issues)
- Slack DM routing: each top-level DM no longer spawns a fresh
workstream (was using per-message `ts` as the route key).
- Multi-chunk Slack responses thread correctly under the first
chunk's ts instead of fragmenting as independent top-level
messages.
- Finalize the outgoing StreamingMessage before swapping channel /
thread_ts mid-stream, so buffered tokens still land on the old
thread.
- Redundant `chat_update` on approve/deny eliminated by popping
`_pending_approval[ws_id]` after local resolution.
- Notification reply tracking on Discord only registers for DMs
(guild-channel targets were storing channel IDs where user IDs
were expected, so legitimate replies were always rejected).
- `get_channel_default_alias` rolls `_channel_default_ts` back on
`list_models()` failure so the next caller retries instead of
serving an empty alias for the full TTL.
- Slack `subscribe_ws` purges dead SSE tasks before the
membership short-circuit (previously an unhandled exception left
the ws_id in `_subscribed_ws` forever, silently no-opping
subsequent subscribes).
- ChannelRouter `_create_locks` is now an LRU-bounded OrderedDict
that evicts only unheld locks (original dict grew unbounded;
naive LRU could evict a held lock and let a second caller race
through the critical section, creating duplicate workstreams).
- Slack `_parse_ts` pads the fractional field to 6 digits so
`"1.2"` and `"1.000002"` stop colliding as `(1, 2)` in the
latest-session tiebreaker.
## Performance (6 fixes)
- StreamingMessage keeps a rolling truncated display string capped
at `max_length` so per-flush cost is O(max_length) instead of
O(total_streamed_chars) — long streaming responses no longer do
quadratic work every edit interval.
- `StreamingMessage.finalize()` caches the joined content so the
Discord stream-end DM-forward path doesn't re-join a multi-MB
buffer twice.
- `PendingApproval` stores the Block Kit payload posted to Slack;
`IntentVerdictEvent` appends the verdict in-place and
`chat_update`s, skipping an extra `conversations_history`
round-trip.
- ChannelRouter `lookup_ws_id()` TTL-caches the channel →
ws_id resolution (30s TTL, 4096-entry LRU); hot inbound paths
skip storage on every message.
- Service-discovery startup retry uses exponential backoff
(1s → 8s cap) with a 30s deadline instead of 30 × 1s fixed
sleep.
- `_archive_session` now calls `router.close_workstream` so the
`_node_urls` cache entry is dropped (was leaking one entry per
archived session).
## Quality / refactors (19 improvements)
- `cli.main()` extracted from a 365-line function into focused
helpers; imports carefully kept lazy where test patches target
source-module paths.
- `_run_gateway` finally block now awaits `adapter.stop()` on
every adapter so SSE tasks, httpx clients, and the Slack socket
handler close cleanly on shutdown.
- Shared SSE reconnect loop extracted to `turnstone/channels/_sse.py`
(`run_sse_stream` with `on_event` + `on_stale` callbacks); both
adapters' `_sse_listener` methods just wire up callbacks. The
"404 stops reconnect" invariant is enforced inside the helper
so a broken `on_stale` can't livelock.
- `_on_ws_event` god-dispatchers split into per-event `_handle_*`
methods with a thin isinstance dispatcher at the top.
- Slack `_on_approve` / `_on_deny` collapsed into a single
`_resolve_approval(*, approved: bool)`.
- `ApproveRequestEvent` policy evaluation hoisted into
`ChannelRouter.evaluate_tool_policies` returning a
`PolicyVerdict`; adapters switch on the verdict kind.
- `ChannelAdapter` protocol trimmed to the four methods adapters
actually implement; unused `ChannelEvent` dataclass removed.
- Shared constants lifted to `turnstone/channels/_config.py`.
- `_cleanup_stale_route` and `unsubscribe_ws` share a
`_clear_ws_state` helper.
- `StreamingMessage` private attrs promoted to `message` /
`message_ts` / `accumulated_text` properties so callers don't
reach past the `_`-prefix.
- Various cleanups: dead var, noqa'd lambdas, renamed
`_policy_handled` → `policy_handled`, inlined single-use
helpers, added module docstrings, documented
`SlackRoute.parse` edge cases.
- `chunk_message` plain-text fast path (no backticks → skip
fence bookkeeping).
## Test coverage
Added 45 tests (178 → 223):
- `tests/test_channel_sse.py` (new) — SSE reconnect / backoff /
404-stale-route / on-stale-exception / invalid-JSON-skip /
on-event-exception-doesn't-kill-stream / per-connection token
refresh / ConnectError retry.
- ApprovalView + PlanReviewView owner-check regression tests
(owner allowed, non-owner rejected, legacy 2-pipe footer fails
closed, modal path rejected for non-owner, `/ask`
bot-as-thread-owner follow-up allowed).
- Slack `_recover_routes` latest-ts-wins, `_archive_session`
drops route + closes workstream.
- SSRF tests: DNS rebinding rejected, IPv4 link-local metadata
rejected, IPv6 ULA metadata (fd00:ec2::254 / fd00:ec2::23)
rejected.
- Slack link prefix match (natural-language prompts don't
hijack), link rate-limit ceiling.
- SlackRoute round-trip across all three shapes + lax-parse
behaviour.
Lint (ruff) + mypy clean; 210 channel-focused tests pass.
* chore(channels): address PR #382 review-bot feedback
Three line-level findings from github-code-quality on the backfill
review PR. Copilot had no line-level comments.
- _sse.py:132 — the `except httpx.HTTPStatusError: pass` branch was
flagged as an empty except. The original status was already logged
at WARNING inside the try block (we re-raise ourselves after
logging), so the handler has real intent. Added a debug log of the
exception text + a comment explaining the control flow, so the
empty-except lint stops firing and the next reader sees why we
fall through to backoff.
- discord/bot.py:430, cli.py:354, slack/bot.py:1127 — `await task`
inside `contextlib.suppress` was flagged as "statement has no
effect". It's a false positive (await is an effect) and the
alternative try/except/pass triggers ruff SIM105. Kept the
contextlib.suppress pattern and added an explanatory comment above
each call so the intent (await CancelledError propagation before
state cleanup) is obvious; will reply on the PR thread noting the
false positive.
No behavior change. Lint + mypy clean; 210 channel tests pass.
* feat(coordinator): phase 6 — polish, observability, active-coords via SSE, frontend cleanup
Squashed from two working commits:
1. phase-6 backend polish + active-coords SSE
2. phase-6 frontend cleanup (legacy chat-view classes + designer nits)
Both tier-A/B observability items and tier-C frontend consolidation
ship together — the shared-vocabulary migration touches surfaces the
backend polish already had its hands in, so one combined commit keeps
the diff reviewable as a coherent phase.
Observability
-------------
- **Coordinator-side wait dashboard** — `_exec_wait_for_workstream`
emits `wait_started` / `wait_progress` / `wait_ended` SSE events
via a new `progress_callback` hook on
`CoordinatorClient.wait_for_workstream`; coordinator.js renders a
"⧗ waiting · N ws · Ts" header indicator keyed by call_id so
overlapping waits coexist. Progress throttled to emit only on
snapshot-diff or 5s heartbeat; full results dict attached only on
transitions so a 600s wait doesn't flood SSE listener queues.
Indicator only attaches when a proper header host exists (no
floating document.body fallback) and is cleared on SSE reconnect
so a dropped `wait_ended` can't pin the badge.
- **`cancel_workstream` forensics** — `server.cancel_generation`
captures `ui._pending_approval` tool names +
`session._queued_messages` count / preview before invoking
`session.cancel`, returning the snapshot as `dropped`; routing
proxy passes it through to the tool result. Preview runs through
`redact_credentials` before the 120-char truncate so pasted
secrets / connection strings don't land verbatim in the
coordinator's conversation history.
- **Per-coordinator metrics** — `GET /v1/api/coordinator/{ws_id}/metrics`
returns `spawns_total` / `spawns_last_hour` / `child_state_counts`
/ `judge_fallback_rate` (substring match on verdict.tier) plus
zero placeholders for wait_* pending dedicated instrumentation.
Derived from new `storage.count_workstreams_by_state` +
`count_workstreams_since` aggregate helpers — no 10k-row
hydrated-select to compute a histogram. Ownership 404-mask
matches `coordinator_detail`.
- **Coordinator skill in inspect** — `CoordinatorManager.create`
resolves `skill` → `template_id` / `applied_version` via
`get_skill_by_name` + new `storage.count_skill_versions`
(replacing the SELECT-all-for-COUNT anti-pattern) and persists
them on the workstreams row. `/new` handler dispatches via
`asyncio.to_thread` so blocking storage calls don't stall the
event loop.
- **`wait_for_workstream(since=…)`** — optional prior-snapshot
hint; when supplied, the wait loop diffs each polled ws_id that
IS in `since_map` and exits on any change, independent of mode.
ws_ids absent from `since_map` fall through to the normal mode
condition — a disjoint since dict no longer silently exits the
wait on tick one.
- **`task_list.child_ws_id` referential cleanup** —
`CoordinatorClient.cleanup_dead_task_child_refs(ws_id)` holds the
same per-ws `_task_lock` as `task_list_*` so a close racing a
task_list write can't lose the mutation. `CoordinatorManager.close`
delegates. Final save-failure logs at `warning` instead of
`debug`.
Home view live-updates
----------------------
- **Active-coordinators via SSE** instead of a 5s poll —
`ClusterCollector.ensure_console_pseudo_node` +
`emit_console_ws_created / _closed / _state / _rename` plumbing;
`CoordinatorManager.create / open / close / eviction` +
`ConsoleCoordinatorUI.on_state_change / on_rename` all fan out
through the collector. The pseudo-node is exempt from the
discovery-loop eviction; rehydrate-path eviction now also emits
`console_ws_closed` for the evicted row so other tabs drop it
live. `app.js` reads coordinators from
`clusterState.nodes["console"]`; poller + back-compat shims
deleted (9 call sites). Overview / nodes list skip the
pseudo-node so it doesn't inflate cluster totals. Tenant-filtering
preserved by excluding the pseudo-node from
`collector.get_workstreams` so `/v1/api/cluster/workstreams` still
uses the existing tenant-filtered `_coordinator_rows` path.
`CoordinatorManager.NODE_ID` bound from
`ClusterCollector.CONSOLE_PSEUDO_NODE_ID` so the two literals
can't drift.
Frontend perf
-------------
- **Bulk cluster-ws live endpoint** — `GET /v1/api/cluster/ws/live?ids=`
returns `{results, denied, truncated}` (cap 50); coordinator.js
batches visible-row live-badge fetches into one bulk request per
~250ms window (replaces per-row /detail polling). Ownership
check routes through the empty-string-safe pattern (non-admin
with empty `caller_uid` doesn't match empty-owner rows).
Legacy chat-view class cleanup
------------------------------
- Drop the `.msg` / `.msg-user` / `.msg-assistant` / `.msg-tool` /
`.msg-error` / `.msg-info` / `.approval-block` / `.approval-tool`
/ `.approval-btn` / `.approval-badge` / `.approval-prompt` /
`.approval-feedback-input` / `.approval-actions` / `.pane-input` /
`.pane-input-area` / `.pane-input-row` / `.pane-attach` /
`.pane-attach-chip` / `.coord-msg` / `.coord-body` / `role-*` /
`btn-approve` / `btn-deny` / `btn-always` / `verdict-glow-*`
legacy dual-class names left over from the phase-4 migration.
Every JS className concatenation + querySelector + CSS selector
now uses the `ts-*` vocabulary from `shared_static/chat.css` (and
`ui/static/style.css` where the interactive-page extensions
live). Feature-specific class names that don't map to `ts-*`
stay — `msg-queued` / `msg-editing` / `msg-actions` / `msg-edit-*`
/ `msg-user-attach*` / `msg-user-text` / `queued-badge` /
`queued-dismiss` / `tool-name` / `tool-cmd` / `tool-diff` /
`tool-header` / `tool-preview`.
Designer nits
-------------
- **`.ui-btn--icon:focus-visible`** — new rule matching `.ui-btn`'s
`outline: 2px solid var(--accent); outline-offset: 1px` so the
compact icon variant gets the accent ring instead of the
browser-default outline.
- **Dropped speculative 701-880px composer wrap rule** — the flex
math at ≥701px fits comfortably in every desktop viewport, so the
mid-zone break rule was forcing a 2-line layout where the browser
wouldn't have wrapped naturally. The existing `<700px` full-stack
covers the original wrap observation.
- **`.verdict-badge` border-top** + **`.ch-row.highlight`
prefers-reduced-motion** — confirmed already in main; no
additional code change needed for phase 6.
Follow-up designer-review findings
----------------------------------
- Dropped `border-top` from `.ts-approval-badge` + `.ts-approval-body`
(chat.css's max-content width / flex-gap made them read as
truncated / floating lines).
- `var(--muted)` → `var(--fg-dim)` on denied tool names (undefined
token was silently failing).
- Dropped 3 dead `.ts-approval-badge.badge-*` rules + duplicated
`.ts-approval-btn:focus-visible` + dead `.reasoning` CSS rule +
`contains("reasoning")` JS guard.
- Dropped `tool-row` / `approval-header` / `btn-row` / `label` dead
legacy classes in the coordinator.
- Added `:focus-visible` to `.ch-row a.ws-link` + `.task-row` so
keyboard users get the accent ring on sidebar rows.
Cleanups
--------
- `_WAIT_REAL_TERMINAL_STATES` / `_WAIT_TERMINAL_STATES` /
`_WAIT_MAX_*` / `_WAIT_POLL_INTERVAL` hoisted to module level on
`coordinator_client` so `session.py` no longer reads a class
internal; ClassVar aliases kept for back-compat.
- `ConsoleCoordinatorUI` state/rename observers typed as
`Callable[[str], None] | None` instead of `Any`.
- SSE error renderer verified end-to-end (coordinator.js already
handles `case "error"` → `appendText`; no code change).
Tests
-----
- 26 new test cases: 11 for `_diff_since` + `cleanup_dead_task_child_refs`,
15 for `cluster_ws_live_bulk` + `coordinator_metrics`. Full suite
4345 passing (4319 base + 26 phase-6).
Gate: ruff + mypy + pytest -m "not live" (4345 passed) all clean.
* fix(coordinator): address PR #381 review feedback
Copilot comments:
- Cross-tenant aggregate leak in coordinator_metrics — the new
count_workstreams_by_state / count_workstreams_since aggregates
took parent_ws_id but not user_id, so a non-admin caller could
observe drifted / forged child rows that share parent_ws_id with
their coord but whose user_id drifted to another tenant. The
404-mask on coord ownership (_resolve_coordinator_or_404) is the
primary defense; this is defense-in-depth inside the aggregate
queries. Pass filter_user_id (None for admin, caller_uid for
non-admin) — matches coordinator_children's tenant-push-into-SQL
pattern.
- wait_for_workstream(since=…) docstring + tool schema were stale —
claimed "A missing entry counts as changed on first observation"
but the implementation ignores ws_ids absent from since_map to
prevent a disjoint since dict from silently exiting on tick one.
Rewrote both doc sites to match the actual semantics: only ws_ids
present in `since` participate in the diff-exit check; others fall
back to the normal mode-based completion condition.
- WAIT_TERMINAL_STATES comment drift — the comment claimed it was
"used by the resolved-count summary" but the summary counts only
WAIT_REAL_TERMINAL_STATES (denied is a rejection, not a
resolution). Rewrote the comment to describe the real usage:
mode='any' pure-denied short-circuit + mode='all' settle check.
github-code-quality (CodeQL):
- coordinator.js — dropped the dead typeof _renderWaitIndicator
guard + the typeof activeWaits guard around the reconnect clear.
Both symbols are defined in the same IIFE; the onopen handler
fires strictly AFTER IIFE execution finishes, so the guards
always evaluated to true. Removing the dead branching also
removes a CodeQL nit.
- Protocol-method `...` statements — the bot flagged the three new
methods (count_workstreams_by_state / count_workstreams_since /
count_skill_versions) with "statement has no effect". Left as
`...` to match the file's universal convention (216 `...` bodies
/ 0 `pass` bodies pre-change); swapping just the new methods to
`pass` would introduce inconsistency with every other Protocol
method. Resolved as non-actionable.
Test: new test_metrics_tenant_filter_excludes_forged_cross_tenant_child
covering the aggregate-query tenant filter with both a legitimate
alice child and a forged bob child sharing parent_ws_id. Non-admin
alice sees 1; admin sees 2.
Gate: ruff + mypy + pytest -m "not live" (4346 passed) all clean.
* fix(server,console): kind filter on saved-workstreams + closed coords on landing
Two independent bugs folded into one hotfix:
1. Coordinators leaking into the interactive UI's "saved workstreams"
sidebar. ``list_workstreams_with_history`` (SQLite + postgres) was
kind-agnostic — every coordinator row with conversation history came
back alongside interactive rows, and ``list_saved_workstreams``
serialized them uniformly with no kind field so the interactive UI
rendered coordinators as regular interactive entries.
Fix: add optional ``kind: WorkstreamKind | str | None = None`` kwarg
on ``list_workstreams_with_history`` (storage protocol + both
backends + the ``turnstone.core.memory`` helper). Pass
``kind=WorkstreamKind.INTERACTIVE`` from the /v1/api/workstreams/saved
handler so the interactive surface only sees interactive rows.
Default ``None`` preserves legacy all-kinds behaviour for any
other caller that wants both.
2. Closed coordinators vanish from the console landing page.
``_coordinator_rows`` in console/server.py built dashboard rows
exclusively from the in-memory ``CoordinatorManager`` registry,
which pops rows on ``close()``. The persisted storage row stays
(state='closed') but never reached the landing-page poller at
/v1/api/cluster/workstreams?node=console.
Fix: two-lane merge in ``_coordinator_rows``. The in-memory lane
(manager) stays authoritative for live session state (model /
model_alias / current state / tokens). A new persisted lane queries
``storage.list_workstreams(kind=COORDINATOR, user_id=uid, limit=200)``
and appends rows NOT already in the in-memory set — surfacing
closed / error / deleted coordinators so the operator can still
see them on the landing page. Ownership semantics unchanged —
non-admin callers only see their own tenant, admin-bypass via
admin.users/admin.roles honored on both lanes, empty-string
defense-in-depth matches _check_row_owner_or_404.
Tests:
- tests/test_storage_sqlite.py — two new tests: kind filter excludes
coordinators from the history list; string form of kind accepted
(matches the memory.py forwarding shape).
- tests/test_coordinator_endpoints.py — four new tests:
- closed coordinators from storage surface alongside active ones.
- in-memory row wins on ws_id dedup (live state authoritative).
- persisted rows respect tenant filter (non-admin, admin bypass).
- orphan rows (empty user_id) never leak to empty-sub callers.
Gate: ruff + mypy + pytest -m "not live" (4315 passed) all clean.
* fix(server,console): address Copilot review on PR #380
Three review comments folded in:
1. Tenancy leak in /v1/api/workstreams/saved — the handler called
list_workstreams_with_history without a user_id filter, so any
authenticated user could see every other user's saved workstream
aliases / titles / names. Fix:
- Add ``user_id: str | None = None`` kwarg to
list_workstreams_with_history on the protocol + both backends
(SQLite + postgres). Pushes the filter into SQL.
- memory.py helper forwards the kwarg.
- /v1/api/workstreams/saved reads ``_auth_scopes(request)``: a
service-scoped caller gets cluster-wide visibility (None), a
non-service caller with a blank ``sub`` returns an empty list,
otherwise the SQL filter is scoped to the caller's uid. Matches
the _visible_workstreams pattern used on /workstreams and
/dashboard.
2. Loose type annotation on the memory.py helper — ``kind: Any``
tightened to ``WorkstreamKind | str | None`` so mypy catches
invalid callers. WorkstreamKind was already imported in the
module.
3. Brittle positional indexing in _coordinator_rows persisted-rows
lane — ``row[10]`` for user_id encoded a column offset that would
silently corrupt the projection on any future SELECT reorder.
Drop the test-double fallback entirely; the storage-protocol
contract already requires SQLAlchemy Row with _mapping, and every
real caller (SQLite + postgres) provides it.
Tests:
- test_server_authz.py TestSavedWorkstreamsTenantScoping — four new
regression tests covering: non-service caller sees only own rows,
service scope sees cluster-wide, blank-sub non-service returns
empty, and coordinator rows excluded even for service callers.
Gate: ruff + mypy + pytest -m "not live" (4319 passed) all clean.
* fix(console): service scope on collector token + surface upstream 4xx
CRITICAL: the console's ClusterCollector ServiceTokenManager was
configured with only frozenset({"read"}) scope, but every upstream
node's /v1/api/events/global hard-gates on "service" scope (added in
PR #375 for cross-tenant authz hardening). Every console→upstream
SSE connect 403'd, the collector never populated node state, and the
failure was silent — node health, idle workstreams, and interactive-
kind workstream rows all disappeared from the console dashboard with
no user-visible error. The only surface was a log.debug line in the
collector's _node_sse_task that operators had to opt into via DEBUG
logging or browser DevTools.
Fix:
- Add "service" to the collector_token_mgr scopes
(turnstone/console/server.py). Matches the proxy_token_mgr (which
already has it) and the existing cli / admin / channel-gateway
service tokens. Restores /v1/api/events/global SSE subscription
and /v1/api/dashboard visibility (which silently tenant-filters
non-service callers to zero rows).
- Upgrade the 4xx path in _node_sse_task to log.warning with the
status code + 200-char body preview, so configuration-level
failures (scope misconfig, JWT secret mismatch, expired token)
show up in operator logs instead of being masked by the generic
except-block debug line. Keep transient network errors
(CancelledError, ConnectError) at debug so the log doesn't flood
during brief node restarts.
- Add reachable_reason field to NodeSnapshot + surface via
get_nodes / get_node_detail / get_snapshot (and the browser's
buildNodeInfoFromSnapshot). Operators now see the failure cause
on the cluster node list without tailing the log. Cleared on
successful reconnect in _apply_snapshot.
- Test coverage: test_server_authz.py TestGlobalEventsServiceGate
gains a positive-path test asserting that a token with exactly
the collector's scope set ({"read", "service"}) is accepted by
/v1/api/events/global. Locks in the scope contract so any future
rename breaks the test before it breaks the dashboard.
Gate: ruff + mypy + pytest -m "not live" (4309 passed) all clean.
* fix(console): address Copilot review on PR #379
Two review comments folded in:
- collector.py — bounded body read for 4xx SSE error previews. The
prior ``await source.response.aread()`` buffered the entire
upstream error body into memory just to log a 200-char preview; a
malicious / oversized upstream response (HTML error page, proxy-
generated body) could have forced the collector to download an
arbitrary amount of bytes. Iterate ``aiter_bytes()`` and stop once
the preview cap (256 bytes, ~200 chars after UTF-8 decode) is
satisfied.
- test_server_authz.py — tighten the service-scope positive test.
The prior ``assert resp.status_code != 403`` could pass on
unrelated 500s AND left an SSE stream open indefinitely. Send
``?expected_node_id=definitely-wrong-node-id`` so the handler
passes the scope gate, hits the post-auth node-identity check, and
returns 409. Now ``assert resp.status_code == 409`` proves the
scope contract precisely and terminates the request immediately.
Gate: ruff + mypy + pytest -m "not live" (4309 passed) all clean.
* feat(coordinator): phase 5 — harness-test polish + wait_for_workstream + judge fix
Closes the bug list surfaced by the 2026-04-17 coordinator harness test
plus the post-phase-4 wait_for_workstream ask, and folds in three
adjacent cleanups that landed in the same window. Tightens defense-in-
depth on the model-invoked mutating ops, fixes the LLM judge silent
no-op, kills the inspect-poll token burn, and rounds out a handful of
observability / docstring / spec gaps.
The session-factory pre-resolve at console/session_factory.py and
server.py was rewriting `judge.model` from an alias (e.g. `judge-mini`)
to the resolved underlying id (e.g. `gpt-5-mini`). IntentJudge then
checked `model_registry.has_alias(config.model)`, found nothing, and
fell back to the SESSION's provider/client with that bare model id —
silent `llm_fallback / "did not return a verdict"` whenever the
coordinator and judge alias resolved to different providers.
Pass the alias through unchanged; IntentJudge's existing alias-
resolution path picks up the matching client + provider. Validate
the alias exists so an obvious typo still surfaces, but don't replace
the model field.
Regression: `test_alias_uses_registry_provider_not_session_provider`
constructs an alias whose provider differs from the session's and
asserts the judge picks up the alias's provider/client/model;
`test_coordinator_tool_call_returns_llm_verdict_not_fallback` asserts
the verdict tier is `llm` (not `llm_fallback`) on the happy path.
New `cancel_workstream` tool (approval required, primary_key=ws_id) —
cancels in-flight generation, unblocks any pending approval / plan,
moves the child to idle, leaves the row in storage so a fresh
send_to_workstream lands cleanly. Re-uses the existing
`/v1/api/route/cancel` route + `route.cancel` audit namespace; no
new server endpoint.
`CoordinatorClient.cancel/close_workstream/delete/send` now enforce
a tenant guard inline (`_is_own_subtree`) — only the coordinator
itself or one of its own children is targetable. Foreign ids return
the same 404-shape inspect/wait_for_workstream use, so the model
can't distinguish foreign from missing (no existence oracle).
Defense-in-depth — the upstream node enforcement is the perimeter,
this is the second line.
`list_workstreams` advertised `state="deleted"` and an
`include_closed=true` that surfaced deleted rows. Hard-deletes
cascade the workstream + conversation rows out of storage, so
deleted is unreachable in normal operation. Doc-only fix; the
synthetic-test path that registers `state="deleted"` rows still
works (terminal-state filter still excludes them via
`_terminal_states = {"closed", "deleted"}` in list_children).
Documented that the 120s service-registry heartbeat window means a
node returned by list_nodes can drop out before a follow-up
`spawn_workstream(target_node=…)` lands — the spawn fails with "No
available node for routing" rather than falling back. Two-line
clarification on each tool. No code change (a code fallback is a
bigger discussion deferred to 1.6).
`close_workstream` accepts `reason`; the upstream server handler now
persists it to `workstream_config.close_reason` (capped at 512 BYTES,
sliced on UTF-8 not code points so a CJK / emoji-heavy payload can't
4× the documented budget). `CoordinatorClient.inspect()` reads it
and surfaces as `close_reason` in the result dict — only for
terminal-state children (closed/error/deleted) so the live-child hot
path doesn't pay a per-inspect DB round-trip.
Tests: server-side persistence covers success / no-reason /
length-cap / non-string / storage-failure / multi-byte-utf8 paths;
client-side surface covers terminal vs. live workstreams.
For idle children whose node-dashboard live counter is 0 (the live
block only surfaces in-flight token counters), fall back to
`SUM(prompt_tokens + completion_tokens)` from `usage_events` so the
inspect output reflects cumulative spend.
New `storage.sum_workstream_tokens(ws_id) -> int` on the protocol +
both backends. The fallback is folded INTO `_fetch_cluster_live` so
the merged live block (with persisted total applied) is what gets
cached — back-to-back inspects of an idle child amortize through
the existing 2s LRU cache instead of each firing a fresh aggregation.
`CoordinatorClient.list_skills()` now projects `allowed_tools` per
skill — capped at 20 with a `+N more` sentinel so a skill that
whitelists a wide MCP surface doesn't bloat the per-row payload.
Reads the existing `prompt_templates.allowed_tools` column; no
storage change. Coordinators no longer have to guess what tools a
skill brings.
`route_create` now sets `routing_strategy: "hash_ring" | "target_node"
| "resume"` on the spawn response so the coordinator's spawn
response (and the `spawn_workstream` tool output) carries why a
given node was chosen. 3 lines + 3 covering tests in
test_console_routing_proxy.py.
New coordinator tool `wait_for_workstream(ws_ids, timeout=60,
mode='any'|'all')` that absorbs the wait into a single tool call —
the model sees one call + one result regardless of how long the
children take. Kills the busy-poll inspect loop that burned 20+
turns on a 3-child fan-out.
Storage-poll loop with batched primitives —
`get_workstreams_batch` + `sum_workstream_tokens_batch` issue exactly
two storage calls per tick regardless of N. At the cap (32 ws_ids /
600s / 0.5s tick) that's ~2400 round-trips for a full wait, down
from ~38k under the naive per-id shape.
Validation single-source-of-truth: the client owns mode whitelist,
ws_ids dedup + cap, timeout coerce + clamp. The session preparer
is a thin pass-through that builds the header + dispatches; bad
input surfaces at exec time as a tool error via `result.get("error")`.
Tenant-isolation collapse: missing-row and cross-tenant cases both
return `state="denied"` so wait can't be used as an existence oracle
(matches the 404-mask contract `inspect` uses).
Prompt-side: tools_coordinator.md adds a `wait_for_workstream`
pattern + an explicit "PREFER wait_for_workstream OVER a loop of
inspect_workstream" line in the workflow-shape section.
Replaces the quote-bracketed substring LIKE/ILIKE pattern with proper
JSON-array containment. The previous shape effectively did
`LOWER(tags) LIKE '%"<lower-tag>"%'`, which broke for tag values
containing `"` (the JSON encoder escapes it to `\"` and the literal-
substring search misses), `\` (encoded as `\\`), or non-ASCII
characters that the encoder rendered as `\uXXXX`. Also exposed a
small spoofing surface — `tags=["foo\","bar"]` would have matched a
query for `bar`. Real-world tag values are alphanumeric+dash today
so it hadn't fired in production, but the fix is small.
- SQLite: `EXISTS (SELECT 1 FROM json_each(prompt_templates.tags)
WHERE lower(value) = lower(:tag))` (JSON1 extension; SQLite 3.38+).
- PostgreSQL: `EXISTS (SELECT 1 FROM jsonb_array_elements_text(
prompt_templates.tags::jsonb) AS jat(elem) WHERE lower(jat.elem) =
lower(:tag))`.
Three new tests prove the substring pattern was broken for
quoted / backslash / unicode tag values; the existing case-fold +
wildcard tests continue to pin the contract.
Phase 1 added the coordinator workstream API; phase 2 added only
`/open` to the OpenAPI catalog and missed every other coordinator
endpoint plus phase 3's `/children`, `/tasks`, and the
`/cluster/ws/{ws_id}/detail` aggregator. SDK consumers + operators
browsing `/docs` couldn't discover the surface. Doc-only addition:
12 endpoints + 9 new Pydantic models, all under the `Coordinator`
OpenAPI tag so /docs groups them together.
Sidebar re-fetches `GET /tasks` on every `task_list` `tool_result`
SSE event. A model that runs `add → list` (or any back-to-back
mutation pair) double-fetches the same envelope. Coalesced into
one fetch per 150ms window via a new `loadTasksDebounced` wrapper;
direct UI actions (refresh button, page load) keep calling
`loadTasks` directly so user clicks aren't delayed.
- `ruff check turnstone tests` — clean
- `mypy turnstone` — clean (157 source files)
- `pytest -m "not live"` — 4284 passed, 3 deselected (was 4226 on
main; +58 new tests across coordinator client, tools, judge,
storage, console routing proxy, server close-handler,
storage_skills_filtered, OpenAPI catalog, server close-reason
persistence)
- New tools added: 2 (cancel_workstream, wait_for_workstream) —
TOOLS count 28 → 30; coordinator subset 9 → 11; auto_approve adds
wait_for_workstream; primary_key adds cancel_workstream
- New OpenAPI endpoints: 12 (every phase-1/2/3 coordinator route +
the cluster-inspect aggregator)
- New storage protocol methods: 3 (sum_workstream_tokens,
sum_workstream_tokens_batch, get_workstreams_batch)
All phase 1 / 2 / 3 / 4 invariants preserved: COORDINATOR_TOOLS /
INTERACTIVE_TOOLS disjoint; coordinator sessions have no MCP surface;
list-style tools return {items, truncated}; route-proxy emits
route.<action> audit on 2xx; 404-mask on ownership failures; tenant
filters pushed into SQL; per-coordinator JWT carries scope context.
* fix(coordinator): address Copilot review on PR #378
Three valid Copilot findings on the wait_for_workstream surface:
1. ``wait_for_workstream.json`` description claimed the tool returns a
top-level mapping ``ws_id -> {state, tokens, updated}`` plus
elapsed/complete/mode at the same level, but the actual shape is
``{results: {ws_id: {...}}, elapsed, complete, mode}``. Description
now matches the implementation. Also adds ``deleted`` to the
advertised terminal-state list (it's in ``_WAIT_REAL_TERMINAL_STATES``;
the doc and runtime now agree).
2. ``CoordinatorClient.wait_for_workstream`` docstring listed
``idle / error / closed`` as the real terminal set but the constant
includes ``deleted``. Same fix — list ``deleted`` with a parenthetical
noting it's unreachable in normal operation (hard-delete cascades the
row).
3. Storage protocol docstring math: ``sum_workstream_tokens_batch``
claimed "from ~38k to ~1200" round-trips per wait at the cap, but
``wait_for_workstream`` issues TWO storage calls per tick
(``get_workstreams_batch`` + this one), so 1200 ticks × 2 = ~2400.
Updated to "~2400" with the math spelled out.
Also a clean rebase onto today's main (PR #377 — the rebalancer node_id
snapshot doc — landed since phase 5's last push). Single conflict in
``inspect_workstream.json`` resolved by keeping both notes (rebalancer
node_id binding semantics + the new ``close_reason`` surface from phase
5); ``spawn_workstream.json`` auto-merged.
The github-code-quality bot also flagged three items on
``_protocol.py`` asking to replace ``...`` with ``pass`` in Protocol
method bodies. Refuted: ``...`` is the canonical PEP 544 idiom for
Protocol method bodies and the rest of the file uses it consistently.
The bot's lint rule misfires for ``Protocol`` classes.
Verification:
- ``ruff check turnstone tests`` clean
- ``mypy turnstone`` clean (158 source files)
- ``pytest -m "not live"`` — 4308 passed, 3 deselected (no test count
change; pure doc/comment edits)
Phase 3 fixed spawn_workstream's response to return the storage-
authoritative node_id at spawn time, but neither tool description
mentioned that the cluster rebalancer can migrate the workstream to
a different node afterwards. A coordinator that cached the
spawn-time node_id for a long-running callback would silently dispatch
to a node that no longer owns the workstream.
- spawn_workstream: ``node_id`` is a POINT-IN-TIME snapshot at spawn;
re-read with inspect_workstream when you need the current binding.
- inspect_workstream: ``node_id`` is the CURRENT (storage-authoritative)
binding; reflects any rebalancer migration that happened since spawn.
Pure description edit — no schema or runtime change.
Third and final PR of the retrospective-review series. Addresses the
remaining bug / perf / doc findings from the original multi-stage review
plus the three inline comments left on #374 and #375.
From the original review:
- bug-3: delete_workstream now nulls out parent_ws_id on every child
row before dropping the target — previously, deleting a coordinator
left orphaned parent_ws_id pointers and list_workstreams(parent_ws_id=
<deleted>) kept returning ghost-parented rows. Fix lives at the
storage edge so both SQLite and PostgreSQL benefit without a schema
migration.
- perf-1 / perf-2 / perf-3: new migration 041 drops the low-cardinality
idx_workstreams_kind outright, rebuilds idx_workstreams_parent as a
partial index (WHERE parent_ws_id IS NOT NULL) to halve its btree,
and uses CREATE INDEX CONCURRENTLY on postgres so the rebuild
doesn't take ACCESS EXCLUSIVE on populated tables. Dialect-guarded;
sqlite path is a straight partial CREATE INDEX.
- perf-5: _rebuild_children_from_storage bumps its limit sentinel to
10_000 and logs a warning when the cap is hit instead of silently
truncating the tail on every console cold-start.
- q-2: turnstone.core.memory.list_workstreams wrapper deleted (zero
live callers; PR #374 kept it forward-compatible with the new
kwargs as a stepping stone).
- q-5: migration 039's docstring now warns operators that downgrade
drops parent_ws_id irreversibly and notes the 041 dependency.
- q-7: GET /v1/api/workstreams row shape now includes kind +
parent_ws_id to match /v1/api/dashboard; the Pydantic
WorkstreamInfo schema follows so SDK consumers see the same fields.
Inline review comments:
- #374 (copilot): console/server.py::coordinator_children now pushes
user_id into the SQL filter for non-admin callers, so forged /
migration-era rows with matching parent_ws_id but a different
owner can't leak through. Admins bypass the filter — they're
expected to see the full subtree.
- #375 (copilot, delete handler): storage.get_workstream(ws_id) for
the audit snapshot moved inside the try: block so a transient DB
error surfaces through the endpoint's redacted 500 handler instead
of an unhandled exception.
- #375 (copilot, _require_ws_access): added optional mgr= kwarg —
when the workstream is live in the in-memory manager, trust its
cached user_id instead of round-tripping storage. In-memory-only
handlers (approve / plan / cancel / command / close / events_sse /
refresh-title / set-title) pass mgr= so they stay functional
during transient DB outages and skip one query on the hot path.
Storage-backed handlers (/delete, /open) omit mgr= and keep the
storage path for persisted-but-not-loaded rows.
Tests:
- tests/test_workstream_kind.py adds regression tests for the cascade
null-out on delete and the new user_id SQL filter.
- tests/test_workstream_endpoints.py updated so the title-handler
tests exercise the in-memory fast path (MagicMock manager returning
None falls through to storage; explicit ws.user_id set where the
mock ws is used).
Lint (ruff), typecheck (strict mypy), pytest -m 'not live' all green
(4209 passing).
Second of three PRs addressing the retrospective review of the
turnstone-server interactive-kind feature. The first (PR #374) put
the structural pieces in place — WorkstreamKind enum + user_id
kwarg on the storage protocol. This PR uses them to close the
handler-level ownership gaps that shipped under the prior design.
- sec-1: approve / plan_feedback / cancel_generation / command now
call _require_ws_access before touching the target UI. Previously
any authenticated user could resolve pending tool-approvals on
another tenant's workstream — RCE-adjacent because the attacker
could approve destructive operations the victim would have denied.
- sec-2: /v1/api/workstreams/{ws_id}/delete now gates on ownership
AND writes a workstream.deleted audit event. Previously any
authenticated user could destroy any other tenant's workstream,
conversations, and attachments in one call with no tamper-evident
trail.
- sec-3: /v1/api/events (per-ws SSE) gates before _register_listener
so non-owners can't subscribe to another tenant's message / tool /
approval stream.
- sec-4 / sec-5: /v1/api/workstreams and /v1/api/dashboard filter
to the caller's tenant view via a new _visible_workstreams helper;
service-scoped tokens (cluster / routing proxy) keep the full view.
- sec-6: /v1/api/events/global requires service scope. The global
snapshot carries cross-tenant workstream inventory and was never
intended for end-user browsers.
- sec-7: /v1/api/workstreams/{ws_id}/open verifies the caller is
the stored owner (or holds service scope) before rehydrating.
Returns 404 on mismatch — existence isn't enumerable by response
code.
- sec-8 / sec-9: /workstreams/close, /refresh-title, /title all gate
on ownership. Cross-tenant close aborts the victim's running
generation; cross-tenant rename is a phishing / denial-of-use
vector in list / dashboard responses.
- sec-11: workstream.created / .deleted / .closed / .opened now
land in the audit_events table with kind + parent_ws_id detail,
so forensic review can reconstruct lifecycle even after the row
is gone.
- q-4: new tests/test_server_authz.py covers every gate above via
TestClient, plus the PR #1 HTTP-boundary kind-validation branches
that had no regression coverage (coordinator / unknown-kind / 400,
cross-tenant parent_ws_id / 403, non-interactive open / 400).
- q-3: test_workstream_kind.py now uses the conftest storage fixture
so it runs against both SQLite and PostgreSQL under
--storage-backend=postgresql, closing the sqlite↔postgres drift
risk the prior review flagged. Added storage-edge ValueError and
user_id SQL filter tests alongside.
Tests, lint (ruff), typecheck (strict mypy) all green. Stacked on
PR #374 — merges after that lands.
Foundation PR for the multi-stage-review follow-up. Introduces a
single source of truth for workstream kind values and pushes tenant
scoping into the storage protocol so list callers can't forget to
filter client-side.
- WorkstreamKind(StrEnum) replaces bare "interactive" / "coordinator"
literals across 17 production modules. Strict mypy narrows every
internal call site; raw strings still work at wide boundaries
(HTTP body, DB row) via WorkstreamKind(raw) parse at the edge.
- StorageBackend.list_workstreams(..., user_id=None) adds a SQL-level
WHERE user_id = :user_id gate on both sqlite and postgres impls.
Memory wrapper forwards the new filters.
- register_workstream now validates kind at the storage edge so SDK /
restore / internal callers can't silently corrupt the NOT NULL
column with empty / mis-cased / unknown values.
- WebUI.__init__ normalizes empty-string parent_ws_id to None, matching
the storage-edge and WorkstreamManager invariants.
- POST /v1/api/workstreams/new parses body["kind"] through the enum
and returns 400 on unknown kinds instead of silent coercion.
Absorbs bug-1, bug-2, bug-4/q-6, q-1, q-8, and partial q-2 (wrapper
signature forwards the new filters; full deletion of the unused
wrapper stays in the cleanup PR).
* feat(ui): phase 4 — chat-UX unification + coordinator-first console landing
Phase 4 unifies the three turnstone UIs (server-node chat, console
dashboard, coordinator page) around a shared design-system layer,
promotes coordinator sessions to first-class citizens on the console
landing, and folds the chat-view itself onto a shared vocabulary so
the two chat pages no longer reinvent messages / approvals / composer /
header / sidebar chrome from scratch.
## Shared static consolidation
- turnstone/shared_static/renderer.js — consolidates the two copies
(ui/static/ + console/static/coordinator/) into one. Adds
streamingRender / streamingRenderFinalize helpers with
requestAnimationFrame coalescing + per-element buffer cache so both
chat views re-render the streamed markdown smoothly without the
prior "plain-text → final pop" on the coordinator page and without
thrashing renderMarkdown + DOM replacement faster than the paint
cycle. renderMarkdown stays the trust boundary for innerHTML
assignment (escapeHtml internal); postRenderMarkdown (syntax
highlighting, mermaid, KaTeX) is deferred to finalize.
- turnstone/shared_static/ui-base.css — flat form-control + button +
state-glyph + pill + panel vocabulary on top of base.css. Sizes in
px to match the 11/12/13px scale used elsewhere. Namespace rubric
documented inline (.ui-* shared controls; .dash-* dashboard legacy;
.ts-* chat vocabulary; page-local stays unprefixed).
- turnstone/shared_static/chat.css (new) — chat-view component
vocabulary: .ts-msg (user / assistant / reasoning / tool / error /
info), .ts-msg-actions floating toolbar, .ts-approval (inline +
batch layout hooks sharing a visual language), .ts-verdict-badge,
.ts-composer shell, .ts-header shell, .ts-sidebar shell. Mobile +
reduced-motion covered.
## Interactive server UI migration
- turnstone/ui/static/app.js — dual-class adoption of .ts-msg + .ts-
approval + .ts-composer + .ts-msg-actions alongside existing class
names so feature-specific rules (.msg-user-text, .msg-queued,
.msg-editing, .msg-action-btn toolbar, attachment chips, verdict
details, media embeds, plan inline) keep working while the shared
chat.css baseline takes over padding / border / typography.
- Left-aligned user messages: .msg-user loses align-self: flex-end
and .msg-assistant loses align-self: flex-start. Both roles now
render as single-column blocks distinguished by left-border colour
(amber for user, neutral for assistant, dashed for reasoning,
mono + code-bg for tool, red for error) per the locked design.
- style.css trimmed: .msg / .msg-info / .msg-error baseline rules
dropped (chat.css provides); all other feature rules intact.
- index.html links /shared/chat.css and tags the header with
.ts-header + .ts-header-title.
## Coordinator page migration
- coordinator.js appendMsg drops the visible .role-label <div> per
the hybrid no-labels design, preserving the role text on
data-ts-role + aria-label so screen readers and SSE dedup-by-call-
id continue to see meaningful labels. Adds .ts-msg + .ts-msg--*
variants + .ts-msg-body onto the existing .coord-msg / .coord-body
elements.
- coordinator/index.html adopts .ts-header, .ts-header-title,
.ts-header-spacer, .ts-header-status on the header; .ts-approval +
.ts-approval--batch on the pinned bar with .ts-approval-btn
variants on the buttons; .ts-composer + .ts-composer-input +
.ts-composer-send on the composer; .ts-sidebar + .ts-sidebar-
section + .ts-sidebar-section-heading on the children + tasks
sidebar. Inline <style> pared from ~300 to ~130 lines — only
genuinely coordinator-specific layout (flex wiring, sidebar list
rows, mobile accordion breakpoint) remains.
- Nits fixed along the way: .ch-row .glyph-thinking recoloured cyan
to match the shared .ui-glyph vocabulary; .task-row .status-done
lost its 0.7 opacity (colour already signals done; opacity
reduced contrast for no gain).
## Console landing + admin panel redesign (phase 4 scope-expansion)
Replaces the node-list-first console landing with a coordinator-first
layout on a new #view-home pane:
- #coord-composer-panel — persistent "Start a new coordinator task"
composer (textarea + optional name + skill dropdown + submit).
Permission-gated on admin.coordinator (same rule the +coordinator
header button uses). Pre-probes GET /v1/api/coordinator on init
and after login (bug-1 fix) so a 503 (no coordinator.model_alias
resolvable) surfaces as a remediation banner linking to Admin →
Models instead of failing on submit. Probe gates on r.ok instead
of r.status !== 503 (bug-2 fix) so auth/permission errors don't
incorrectly flip the banner to ready. Composer and modal share a
_createCoordinator helper (q-1 fix) — POST + redirect + error-
handling tail is not forked.
- #active-coordinators — SSE-driven list of kind=="coordinator"
workstreams, rendered through the shared _renderWsRow helper so
state glyphs + child-count badges match the existing tree view.
- #cluster-summary-compact — one-line aggregate. Clicking expands
into the legacy #view-overview via showOverview() so deep-link
callers of ?view=overview / ?view=node / ?view=filtered keep
working unchanged.
View switching consolidated into a _setLandingView helper so every
show* / drillDown* function toggles the four landing panes through
one call path. Default currentView flipped from "overview" to
"home"; popstate + init history.replaceState land on {view: "home"}.
patchClusterState preserves kind / parent_ws_id / user_id on
ws_created events (phase 3 invariant) so the active-coordinators
list picks up new coordinators immediately without a snapshot
refetch.
Header H1 is now a home link so operators have a single-click path
back to the coordinator landing from any drill-down / admin view.
## Design polish (review pipeline fixes)
- .home-panel-title dropped from 13px/accent to 11px/fg-dim so it
sits in the same heading tier as .ui-section-heading / .dash-
header-title / .home-section-title instead of outweighing them
(dsn-3).
- .home-composer-banner recoloured from amber-on-amber-glow to
bg-surface + 1px yellow border + fg-bright text + accent link
with thicker underline (dsn-1).
- .ui-pill--done dropped the 0.75 opacity — colour signals done,
opacity reduced contrast for no gain (dsn-6).
- .ui-heading fleshed out with --sm/--md/--lg tiers so the utility
actually conveys size (dsn-12).
- ui-base.css size scale moved from rem to px matching 11/12/13px
(dsn-2).
* fixup: address Copilot feedback on PR #373
- admin.js: drop the stale `#view-overview` display:none mutation in
showAdmin. #view-overview is now nested inside #view-home and
toggled via the `hidden` attribute; inline display:none here would
stick after returning to home and suppress the cluster-details
expand.
- app.js: reword the _renderHomeView token-bucket fingerprint comment
to match the actual `Math.floor(tokens / 100)` bucketing — the
prior comment said "thousands / sub-thousand drift".
* feat(coordinator): tree-view UI, cluster-wide live inspect, dashboard grouping — phase 3
Closes out the 1.5 coordinator UX surface: a right-sidebar tree view at
/coordinator/{ws_id} showing spawned children + task list, a new
cluster-wide live inspect endpoint that powers the tree's live badges,
and 2-level dashboard tree grouping that nests spawned children under
their coordinator parent.
## Cluster-wide live `inspect_workstream`
New `GET /v1/api/cluster/ws/{ws_id}/detail` on the console, gated by a
new `admin.cluster.inspect` permission (unassigned to any builtin role;
operators opt in). Aggregates `storage.get_workstream` with a
short-timeout (2s) HTTP fetch against the owning node's
`/v1/api/dashboard`. Coordinator-hosted workstreams get their `live`
block from the in-process `CoordinatorManager` instead of a proxy hop.
Response shape `{persisted, live, messages}` — `live: null` on node
unreachability / 5xx / missing-entry with status 200 so the UI can
degrade gracefully without an error state. Correlation-id masks
unexpected exceptions. 404-masks cross-tenant reads (non-admin
callers see only their own workstreams).
`CoordinatorClient.inspect()` best-effort merges the `live` block onto
its storage snapshot so the model-facing `inspect_workstream` tool
gains a `live` key without any schema change. Model-facing tool
schema stays identical.
## Tree-view UI
New right sidebar at `/coordinator/{ws_id}` with a 2-level children
tree + the phase-2 task list.
Backend:
- New `GET /v1/api/coordinator/{ws_id}/children` returns
`{items, truncated}` — identical row shape to the `list_children`
tool — filtered via `storage.list_workstreams(parent_ws_id=..., kind=None)`.
- New `GET /v1/api/coordinator/{ws_id}/tasks` returns the
`{version, tasks}` envelope via the shared module-level
`load_task_envelope` decoder (extracted from `CoordinatorClient`
so both the tool path and the UI read share corruption semantics).
Corrupt envelopes return an empty list for UI resilience — the
`task_list` tool remains the authoritative write + error path.
- `CoordinatorManager` subscribes to the `ClusterCollector`'s
listener channel from the console lifespan and dispatches filtered
`child_ws_created / child_ws_state / child_ws_closed / child_ws_rename`
events onto each coordinator's SSE stream. Filter authoritative
on the server via a per-coordinator child-ws_id registry populated
lazily on `open()` from storage and incrementally on `ws_created`
events; cleared on `close()` / eviction. One SSE connection per
client, no client-side filtering.
Frontend:
- DOM-method-only child-row rendering (no innerHTML of user content).
- State glyph vocabulary (● running / ◐ thinking / ⚠ attention /
✗ error / ○ idle) plus text labels — WCAG 1.4.1 carries info in
both glyph and label.
- Live badges (tokens + pending-approval pip) fetched via
`/cluster/ws/{ws_id}/detail` with a 5s TTL cache and 250ms debounce
per child. One request per state change, not per second.
- SSE child events update in place; renderChildren() re-sorts.
- Mobile (<700px) sidebar collapses to an accordion above the chat
with a toggle button flipping aria-expanded; a `.highlight` flash
marks task→child scroll targets; `prefers-reduced-motion` respected.
- Deep-link child rows to `/node/{node_id}/?ws_id=<child>` via
`<a target="_blank" rel="noopener">` with encodeURIComponent on
regex-validated ids.
## Dashboard tree grouping
Cluster dashboard rows now group by `parent_ws_id`. Coordinator rows
(`kind == "coordinator"` or children present) get an expand/collapse
caret (button with `aria-expanded`); collapsed shows "(N children)".
Expanded renders children indented as sibling rows with a left-border
gutter. Orphaned children (parent missing or closed) render at top
level with a muted "orphan" badge. Expansion state persisted in
`localStorage` keyed per coordinator ws_id so operator preference
survives reloads. Coordinator rows deep-link to `/coordinator/{id}`;
node-backed workstreams keep their existing proxy deep-link.
Per-node `ws_created / ws_state / ws_activity` SSE event payloads
gained `parent_ws_id` + `kind` so the collector can propagate them
through its fan-out to browser clients without a second lookup;
`_build_node_snapshot` and `/v1/api/dashboard` rows include the
same. Coordinators (which don't live on cluster nodes) merge into
`/cluster/workstreams` via a new `_coordinator_rows` helper that
threads them through the collector's `get_workstreams(extra_rows=...)`
parameter — extras share the filter / sort / paginate pipeline with
node-backed rows.
## Tests
- `tests/test_coordinator_endpoints.py` — 19 new cases covering
children (empty / populated / ownership 404 / admin bypass /
invalid ws_id / truncation), tasks (empty / round-trip / corrupt /
ownership), and cluster-inspect (auth gates / 400 / 404 / ownership /
coordinator self-path / unloaded-live-null / message-limit clamp).
- `tests/test_coordinator_manager.py` — 8 new cases covering registry
bootstrap on create + open, dispatch for each event type,
unrelated-parent filtering, shutdown idempotency.
- `tests/test_console.py` — existing `cluster_workstreams` assert
updated for the new `extra_rows` kwarg.
## Verification
- `ruff check turnstone tests` clean.
- `mypy turnstone` clean.
- `pytest -m "not live"` — 4184 passed, 3 deselected.
* fix(coordinator): race in dispatch + ui_factory kwarg filtering — PR #370 review
Addresses feedback from the GitHub Copilot + code-quality bot review
passes on PR #370.
## Race in _dispatch_child_event ws_created branch
Copilot flagged a TOCTOU where the lock-free read of
``self._active_coords`` (line 912) could see the parent coordinator,
then ``close()`` / eviction pops ``_children[parent]`` + drops the
coord from ``_active_coords`` before we acquire ``_children_lock``,
and then ``setdefault(parent, set())`` resurrects the entry —
leaking the registry key forever and fanning events to a closed UI.
Fix: re-check ``parent in self._active_coords`` inside
``_children_lock``. The reference swap is still atomic; holding
``_children_lock`` and re-reading the snapshot catches the race
without serializing back through ``self._lock``.
Regression test: create → close → dispatch a ws_created → assert
neither ``_children`` nor ``_active_coords`` regained the entry.
## ui_factory kwarg filtering via inspect.signature
code-quality bot flagged that the previous ``try ui_factory(…, kind=,
parent_ws_id=) except TypeError`` dance fired on every call with
legacy test factories (``lambda wid: WebUI(ws_id=wid)``) — wasteful
and masks real signature mismatches.
Fix: inspect the factory's signature and only pass kwargs it
actually accepts (explicit param name OR ``**kwargs`` absorber).
Keep a conservative ``except TypeError`` fallback for C-callables
and odd signatures ``inspect`` can't introspect.
Copilot also flagged a comment mismatch (the old comment said
"KeyError on **kwargs" — it's ``TypeError``, which is what the code
caught). The rewritten comment is correct.
## Nit: side-effect in assert
code-quality bot flagged ``assert mgr.close(ws.id)`` in
test_coordinator_manager.py. Split into two statements.
## Verification
- ``ruff check`` clean.
- ``mypy turnstone`` clean.
- ``pytest -m "not live"`` — 4223 passed, 3 deselected, 0 failed.
* feat(coordinator): audit middleware on routing proxy — phase 2
Adds per-tool-call audit attribution to the multi-node routing proxy
handlers so coordinator → server hops land observable rows in
``audit_events``. Phase 1 preserved the ``src="coordinator"`` claim
through ``_proxy_auth_headers``'s upstream re-mint; this commit
closes the recording side. Was the last real security gap from
phase 1 — an enterprise deployment with ``admin.coordinator``
granted got only the three console-side
``coordinator.{create,close,cancel}`` rows; per-tool-call
attribution was missing.
## Action-naming scheme
route.workstream.create POST /v1/api/route/workstreams/new
route.workstream.send POST /v1/api/route/send
route.workstream.close POST /v1/api/route/workstreams/close
route.workstream.delete POST /v1/api/route/workstreams/delete
route.approve POST /v1/api/route/approve
route.cancel POST /v1/api/route/cancel
route.command POST /v1/api/route/command
route.plan POST /v1/api/route/plan
Action-name conventions documented in ``turnstone/core/audit.py``
module docstring alongside the existing namespaces — the docstring
is now ``<resource>.<verb>`` shaped (non-exhaustive) rather than
trying to enumerate every prefix.
## Recording rules
- ``record_audit()`` fires only on a 2xx upstream response. 4xx/5xx
are observable via ``_record_route``'s metrics path; doubling the
audit-events table size for failure rows would dilute signal
without giving operators much extra value.
- ``detail`` JSON carries ``{src, node_id, coord_ws_id?}`` — ``src``
lands verbatim from ``auth.token_source`` so non-coordinator
origins (``"jwt"``, ``"console-proxy"``) also get attribution;
``coord_ws_id`` only appears when the inbound JWT carried it.
- Wrapped in ``try/except`` + ``log.debug("route.audit_failed", ...)``
defence-in-depth. ``record_audit`` itself is fire-and-forget;
the outer try guards against a programmer error in the call site.
## Routing-proxy specifics
- ``route_create``: emits at the post-multipart/JSON convergence
``if resp.status_code == 200`` block. Both branches set
``audit_ws_id`` correctly — multipart from the query-string ws_id,
JSON from ``body["ws_id"]`` (post-503-retry) or ``body["resume_ws"]``.
- ``route_proxy``: emits the URL-method-mapped action. ``ref`` is
reassigned to ``new_ref`` after a successful 404→cache-refresh
retry so audit attribution uses the retried node, not the failed
first node.
- ``route_workstream_delete``: emits on 2xx using the ws_id from
the request body.
- ``route_attachment_proxy``: out of scope (upstream attachment
endpoints emit their own ``workstream.attachment.*`` rows;
auditing here would double-count).
## Tests
16 new tests in ``tests/test_route_proxy_audit.py`` covering:
- Coordinator-origin emission with full detail payload.
- 502 / 400 / 503-retry-final-node-id paths.
- Parametrised method→action mapping for the 6 ``route_proxy`` URLs.
- Plain-JWT origin (no ``coord_ws_id`` in detail).
- Delete handler 2xx + 502.
- Audit-storage exception swallowed (proxied response unchanged).
- ``auth_storage`` absent → no-op (existing route-handler tests
unaffected).
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4087 passed,
3 deselected (live-backend), 0 failed.
* feat(coordinator): discovery tools and /open parity — list_nodes, list_skills, POST /coordinator/{ws_id}/open
Adds the read-side surface coordinators need to make informed
orchestration decisions plus an explicit rehydration endpoint
matching the server's ``POST /v1/api/workstreams/{ws_id}/open``.
## list_nodes (auto-approved)
``list_nodes(filters={key: value, ...})`` reads ``node_metadata`` via
``storage.filter_nodes_by_metadata`` + ``get_all_node_metadata`` —
one query each, no N+1. Each row carries its full metadata dict so
the coordinator has both auto keys (``arch`` / ``cpu_count`` /
``fqdn`` / ``hostname`` / ``os`` / ``os_release`` / ``python``;
always present) and operator-supplied user keys (``capability`` /
``region`` / ``tenant`` / ``role``) without a second round-trip.
Tool description enumerates the auto keys explicitly so the model
knows what's always available vs deployment-specific.
Storage stores metadata values as JSON-encoded strings (the write
path in ``server.py`` / ``admin.py`` / ``console/server.py`` all go
through ``json.dumps``). The client re-encodes filter values
before the stored-text comparison and decodes stored values before
returning them to the model — so ``{"capability": "gpu"}`` is the
natural form the model uses, not ``{"capability": "\"gpu\""}``.
Ints round-trip as ints.
Returns ``{nodes, truncated}``; ``truncated=True`` when the page
was full.
## list_skills (auto-approved)
``list_skills(category?, tag?, scan_status?, enabled_only?, limit?)``
surfaces the skill registry so coordinators can discover worker
profiles. New storage protocol method ``list_skills_filtered(...)``
on both SQLite and PostgreSQL backends pushes filters into SQL.
``tag`` filter matches against the JSON-array ``tags`` column with
quote-bracketed substring (``%"tag"%``) — quote-safe against
``foo`` vs ``foobar`` collisions on both backends.
Returns ``{skills, truncated}`` with ``name`` / ``category`` /
``tags`` (decoded to list) / ``version`` / ``description`` /
``model`` / ``enabled`` / ``scan_status`` / ``activation`` — the
discovery projection, not the full row.
## POST /v1/api/coordinator/{ws_id}/open
Explicit rehydration endpoint. Lazy ``GET`` rehydration works for
the UI; this gives SDK callers and operators a way to warm a
coordinator without browsing to it. Same ownership / 404-on-
mismatch / correlation-id-masked error semantics as
``coordinator_detail``. Returns ``{ws_id, name, already_loaded?}``.
Registered in ``turnstone/api/console_spec.py`` with a dedicated
``CoordinatorOpenResponse`` Pydantic model so the OpenAPI schema
matches the wire shape.
## Tests
- ``tests/test_storage_skills_filtered.py`` — 8 cases validated on
BOTH SQLite and PostgreSQL backends (``pytest --storage-backend
postgresql``). Covers no-filter ordering, category exact-match,
tag quote-safety (``"foo"`` matches ``["foo","bar"]`` but not
``["foobar"]``), scan_status, enabled_only, limit, AND semantics,
empty result.
- ``tests/test_coordinator_client.py`` — 11 new cases covering
node/skill shape decoding, JSON-encoded filter round-trip (the
``"gpu"`` vs ``'"gpu"'`` case), int filter encoding, truncation,
no-match empty, no N+1 (``get_prompt_template`` /
``get_node_metadata`` call counts asserted zero).
- ``tests/test_coordinator_tools.py`` — 11 new cases for
``_prepare``/``_exec`` dispatch, filter type-drop, limit clamping
(``limit=0`` falls back to 100, negatives clamp to 1),
truncation-signal summary.
- ``tests/test_coordinator_endpoints.py`` — 8 new cases for
``/open``: ``already_loaded`` on in-memory hit, 404 on ownership
mismatch, lazy rehydrate on miss, admin bypass, unknown ws_id,
503 on ``coord_mgr`` unavailable, 500 with correlation-id mask on
factory failure, 503 passthrough on ``ValueError``.
- ``tests/test_workstream_kind.py`` / ``test_tools_schema.py``
updated to include ``list_nodes`` and ``list_skills`` in the
disjoint-namespace regression guard and the tool-count check.
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4122 passed, 3
deselected (live-backend), 0 failed. Postgres backend storage
tests green (``pytest --storage-backend postgresql
tests/test_storage_skills_filtered.py`` 8 passed).
* feat(coordinator): task_list tool — persistent planning state
Adds a coordinator-only ``task_list`` tool persisted on the
coordinator's own ``workstream_config`` row. Gives coordinators a
scratch surface for work decomposition that survives restarts so the
UI can render planned-vs-done state once the tree view lands.
## Tool surface
``task_list(action, ...)`` with five actions:
- ``list`` auto-approved read. Returns ``{tasks, truncated}``;
truncated=True when the list exceeded the 200-row
page cap.
- ``add`` needs approval. ``title`` required; optional
``status`` and ``child_ws_id``. Title clamped at
200 chars. Capacity cap at 500 tasks — hitting the
cap is an explicit signal to prune done/blocked rows.
- ``update`` needs approval. Mutate by ``task_id``; fields
``title`` / ``status`` / ``child_ws_id`` optional.
- ``remove`` needs approval. Drop by ``task_id``.
- ``reorder`` needs approval. Pass ``task_ids``; validated as an
exact permutation of the current set (rejects
partial, extra, or substituted ids — prevents silent
task loss).
Status enum: ``pending`` / ``in_progress`` / ``done`` / ``blocked``.
``child_ws_id`` links a task to the child workstream spawned for it
(no enforcement; the coordinator owns the relationship).
## Persistence
Stored as a single JSON-envelope value on ``workstream_config`` —
``{"version": 1, "tasks": [...]}``. No new table; the kanban v2
work will supersede this row via a format migration keyed on
``version``. ``_save_task_list`` writes only the ``tasks`` key so
concurrent writers to other ``workstream_config`` keys (e.g. the
admin Settings UI updating ``reasoning_effort``) aren't clobbered
by a read-modify-write on the full row.
## Corrupt-envelope safety
A hand-edited or legacy config row that doesn't parse as the
expected shape logs a warning and returns an empty envelope from
``task_list_get``. Mutators refuse to overwrite corrupt data —
they detect the sentinel and return a clear error so the operator
can inspect or clear the row rather than losing work silently.
## Concurrency
Per-(ws) ``threading.Lock`` cached on the client. The worker
thread is single-threaded for tool execs so this is mostly
defence-in-depth against future maintenance-script call sites.
Cache never grows beyond one entry per coordinator session because
the scope guard short-circuits foreign ``ws_id`` before the lock
is acquired.
## Malformed-JSON recovery
``_prepare_tool`` fallback-1 regex-extract allowlist extended with
``action`` / ``status`` / ``task_id`` / ``title`` (alphabetized) so
slightly-malformed ``task_list`` calls get the same
self-correction behaviour as the other coordinator tools.
## Tests
- ``tests/test_coordinator_client.py`` — 15 new cases covering:
fresh-envelope shape, add/get roundtrip, empty-title + invalid-
status rejection, 200-char title clamp, update by id + missing
id, remove semantics, reorder permutation validation (partial +
extra + wrong id + valid), cross-ws scope violation, corrupt-
JSON read recovery, corrupt-envelope write refusal (all four
mutators), 500-task capacity cap, workstream_config key
preservation across ``_save_task_list``.
- ``tests/test_coordinator_tools.py`` — 12 new cases covering the
dispatch layer: list auto-approved, each mutating action needs
approval, unknown-action / missing-required-arg errors, list
returns tasks, page-cap at 200 with truncated signal, add
dispatches to client, reorder surfaces permutation error,
remove-not-found.
- ``tests/test_tools_schema.py`` / ``tests/test_workstream_kind.py``
extend the tool-count + disjoint-namespace + primary-key
regression guards with ``task_list``.
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` 4148 passed,
3 deselected (live-backend), 0 failed.
* feat(coordinator): coordinator workstream kind — phase 1
Adds a new ``kind="coordinator"`` workstream that runs inside the
``turnstone-console`` process (first ChatSession hosted on the console)
with a dedicated tool set for spawning and driving child workstreams.
Supersedes the external ``turnstone-coordinator`` MCP side-car for new
installs; the extension is marked deprecated in
``examples/mcp-cluster-ops/README.md`` but still works on 1.4-and-earlier
clusters.
Phase 1 ships: the workstream class, 6 lifecycle tools, console hosting,
9 HTTP endpoints, per-user audit attribution, and a one-pane web UI at
``/coordinator/{ws_id}``. Node/skill discovery tools, task-list tool,
tree-view UI, and routing-proxy audit middleware follow in a later PR.
## Schema
Migration 039 adds ``kind`` / ``parent_ws_id`` columns + indexes to
``workstreams``. Both SQLite and PostgreSQL backends take the new
kwargs on ``register_workstream``; empty-string ``parent_ws_id``
normalises to ``NULL`` at the storage edge. PostgreSQL uses
``INSERT ... ON CONFLICT DO NOTHING`` to match SQLite's ``OR IGNORE``
and close a pre-existing SELECT-then-INSERT TOCTOU window.
``list_workstreams`` gains optional ``parent_ws_id`` / ``kind`` filters;
new ``get_workstream(ws_id)`` returns the full row (the existing
``get_workstream_metadata`` stays untouched for back-compat).
## Core session + kind routing
- ``ChatSession.__init__`` accepts ``kind`` / ``parent_ws_id`` /
``coord_client``. On ``kind="coordinator"`` it swaps
``_tools = COORDINATOR_TOOLS`` and zeros sub-agent tool lists.
- ``Workstream`` dataclass extended with ``user_id`` / ``kind`` /
``parent_ws_id``. Both ``WorkstreamManager`` and the new
``CoordinatorManager`` use the same type — no parallel hierarchy.
- ``_SessionFactory`` Protocol + server / cli factory closures thread
the new kwargs. ``POST /v1/api/workstreams/new`` rejects
``kind != "interactive"`` with 400; ``POST
/v1/api/workstreams/{ws_id}/open`` refuses coordinator rows so a
server node can't accidentally rehydrate one.
## Coordinator tool set
Six tools (``spawn``, ``inspect``, ``send``, ``close``, ``delete``,
``list_workstreams``) with a ``coordinator: true`` metadata flag,
scoped to coordinator-kind sessions only. ``inspect`` and ``list`` are
auto-approved reads; the four mutators need approval. ``list`` returns
``{"children": [...], "truncated": bool}`` so the model can detect
post-filter under-fill and paginate.
## CoordinatorClient (in-process, sync)
Mutating ops HTTP-POST to the console's own ``/v1/api/route/*`` on the
local bind URL so every existing middleware (auth, rate-limit) runs.
Read ops hit ``storage.list_workstreams`` / ``get_workstream`` /
``load_messages`` directly — the routing proxy doesn't expose
list/inspect paths. URL paths are a validated constant table (avoids
an httpx ``base_url``-merge trap). A new
``/v1/api/route/workstreams/delete`` proxy handler joins the existing
route-proxy endpoints.
## Per-session coordinator JWT
``CoordinatorTokenManager`` mints short-lived JWTs with ``sub=<real
user>`` (attribution preserved), ``src="coordinator"``,
``aud="turnstone-console"``, ``coord_ws_id=<ws>`` custom claim.
``_proxy_auth_headers`` preserves ``src`` + ``coord_ws_id`` across the
upstream re-mint so server-side middleware sees coordinator-origin,
not ``console-proxy``. ``AuthResult.extra_claims`` carries
non-reserved claims through validate→remint; ``create_jwt``'s
reserved-claim set (now including ``nbf`` / ``jti``) is symmetric with
``validate_jwt``.
## Console hosts the ChatSession
- New ConfigStore settings: ``coordinator.model_alias`` (required),
``reasoning_effort``, ``max_active`` (default 5),
``session_jwt_ttl_seconds``.
- Console lifespan builds a ``ModelRegistry`` +
``CoordinatorManager``. Missing / unresolvable alias returns **503**
with remediation text — never 500.
- ``CoordinatorManager``: placeholder-slot reservation under lock,
rollback on factory failure, per-ws_id rehydration lock to serialise
concurrent lazy-opens, ``max_active`` enforced via ``close_idle``
eviction semantics.
- ``ConsoleCoordinatorUI`` is a thin ``SessionUI`` implementation — no
global broadcast, no per-node metrics, shared
``_APPROVAL_WAIT_TIMEOUT`` constant across approval + plan paths.
- No eager startup rehydration: persisted coordinator rows load lazily
on first ``GET /v1/api/coordinator/{ws_id}``.
## Console coordinator API
Nine endpoints under ``/v1/api/coordinator/*`` gated by ``approve``
scope + new **``admin.coordinator``** permission (added to
``_VALID_PERMISSIONS``; not in any builtin role — operators opt in
explicitly). Ownership failures return **404, not 403** and use
strict equality so empty-owner rows don't leak across tenants.
Correlation-id masking on every factory-raising path
(``coordinator_create`` + ``coordinator_detail`` lazy rehydrate) — no
stack traces to the client.
## Audit attribution
Three console-side events (``coordinator.create`` / ``.close`` /
``.cancel``) with the real creator's ``user_id`` plus
``detail={coord_ws_id, src="coordinator"}``. No schema migration
required. Per-tool-call audit across the routing proxy is deferred
(needs either a ``source`` column on ``audit_events`` or
``record_audit`` calls wired into the route-proxy handlers).
## Web UI (``/coordinator/{ws_id}``)
One-pane chat served by the console. Reuses ``shared_static``
(``base.css``, ``auth.js``, ``theme.js``, ``toast.js``, ``utils.js``,
``kb.js``) and the server UI's ``renderer.js`` pipeline (KaTeX, Mermaid,
highlight.js already bundled).
- SSE to ``/v1/api/coordinator/{ws_id}/events`` with exponential-
backoff reconnect; status line carries a leading glyph
(● / ○ / ⚠) so state isn't conveyed by colour alone.
- Renders content, reasoning (dimmed italic
``.role-reasoning``), tool_result, approve_request, intent_verdict,
output_warning.
- Child ws_id references auto-wrap to
``/node/{node_id}/?ws_id={child}`` links — both ids regex-validated
before interpolation, everything else HTML-escaped.
- Non-modal approval bar (``role="region"``) with a batch header
("Approve N tool calls"), initial focus on the approve button,
buttons disabled during the in-flight POST, red-bordered deny.
``aria-live`` flips to ``off`` during streaming.
- "New coordinator" button on the dashboard header — permission-gated
on the UI side, matching the backend 403.
- Mobile composer capped under ``@media (max-width: 700px)``.
## Tests
~120 new tests across 8 files: workstream-kind storage + dataclass
semantics, CoordinatorClient URL map + token minting + storage reads +
truncation signalling, tool prepare/exec dispatch and approval gating,
CoordinatorManager create / rollback / eviction / lazy rehydration +
concurrency, HTTP endpoint auth + 404-on-ownership + 503-on-misconfig,
proxy-auth ``src`` preservation, full lifecycle end-to-end, coordinator
page HTML-injection guard. ``test_tools_schema.py`` widened to 25
tools (19 existing + 6 coordinator).
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 files), ``pytest`` 4054 passed (5 pre-existing failures unrelated
to this change — confirmed against ``main``).
* polish(coordinator): address PR review + CI + tool-namespace isolation
CI:
- `ruff format`: two files reformatted, matches the in-repo pre-commit config.
- `wheel-completeness`: add `turnstone/console/static/coordinator/*.html` +
`*.js` to the hatch wheel-include list. Without this the coordinator UI
was missing from published wheels.
- `test (3.11/3.12/3.13)` + `test-postgres`: three `TestExecReadImage`
tests were masking a real bug — my 6 new tool JSONs pushed tool count
19→25, crossing the default `tool_search.auto` threshold (20), which
made `ChatSession.__init__` construct a `ToolSearchManager` and cache
`_cached_capabilities` during init. Tests that later patched
`session._provider.get_capabilities` saw the cached value instead.
Root-cause fix: the tool-search threshold code path now reads
capabilities through `_resolve_capabilities(...)` directly — no cache
populate — so the patch takes.
Tool-namespace isolation (bigger fix than CI symptoms suggested):
- `TOOLS` was the union of all loaded tool JSONs including the 6 new
coordinator tools. Interactive sessions were getting coordinator
tools in their function-calling surface (which is nonsense — they
require a console-hosted `coord_client`), and coordinator sessions
counted against the interactive tool-search threshold. Fix:
- New `INTERACTIVE_TOOLS` / `INTERACTIVE_TOOL_NAMES` in
`turnstone/core/tools.py` exclude anything with `coordinator: true`
metadata. `TOOLS` stays as the union for schema introspection +
eval catalog.
- `ChatSession.__init__` selects tool set by kind: coordinator gets
fixed `COORDINATOR_TOOLS` (no MCP merge, no listeners registered);
interactive gets `INTERACTIVE_TOOLS` (+ MCP if configured).
Coordinators are meta-orchestrators that spawn child workstreams;
MCP tools / resources / prompts live on the children, not on the
coordinator's own surface.
- `_on_mcp_tools_changed` no-ops for coordinator sessions
(defence-in-depth in case listeners were registered).
- `always_on_names` on `ToolSearchManager` is now the set of builtin
tools actually present in the session (kind-aware) rather than the
full `BUILTIN_TOOL_NAMES` frozenset.
- `turnstone/eval.py` uses `INTERACTIVE_TOOLS` (coordinator tools
aren't in scope for the eval harness which tests interactive agent
behaviour).
- Regression tests in `tests/test_workstream_kind.py`:
- `INTERACTIVE_TOOLS ∩ COORDINATOR_TOOLS == ∅` and their union is
`TOOLS`.
- Interactive `ChatSession._tools` does not include any
coordinator tool name.
- Coordinator `ChatSession._tools` contains `spawn_workstream` but
not `bash` / `edit_file` / `memory`; sub-agent lists are empty.
- Coordinator `ChatSession` with an MCP client attached does NOT
merge MCP tools and does NOT register any MCP listeners.
PR review findings:
- **#10 / #11** (Copilot): coordinator UI claimed to reuse the server
renderer pipeline but loaded none of its JS. Mirrored
`turnstone/ui/static/renderer.js` into
`turnstone/console/static/coordinator/renderer.js` (flagged in-file
as a cleanup candidate to promote into `shared_static/`), added
`katex.min.js` / `highlight.min.js` / `renderer.js` script tags to
`coordinator/index.html`. `coordinator.js` now buffers raw markdown
via `textContent` during streaming, then swaps to `renderMarkdown` +
`postRenderMarkdown` on `stream_end`.
- **#7** (Copilot): N+1 query pattern in
`CoordinatorClient.list_children()` — per-row `storage.get_workstream`
just to read `skill_id`. Pushed `skill_id` + `skill_version` into
the `list_workstreams` SELECT projection on both backends; the
client reads them from `row._mapping` directly. New
`test_list_children_skill_filter_avoids_n_plus_one` pins the
behaviour (asserts `storage.get_workstream` call count is 0).
- **#8 / #9** (Copilot): `spawn_workstream` tool JSON said "if empty,
the workstream is created idle" but the prepare method rejected
empty and the field was marked required. Resolved by allowing
empty end-to-end: removed from `required`, prepare builds a
"spawn idle workstream" header + empty preview when empty,
updated `test_spawn_prepare_allows_empty_initial_message`.
- **#1–#5** (github-code-quality): five asserts with side-effecting
method calls in `test_coordinator_manager.py` (`mgr.close`,
`mgr.open`, `mgr.create` in a dead `_c = ...`). Extracted each
call to a local variable so `python -O` can't strip the side
effect.
Verification:
- `ruff check turnstone tests` clean.
- `mypy turnstone` clean (156 source files).
- `pytest -m "not live"` — 4063 passed, 3 deselected (live-backend
tests), 0 failed. The 3 image tests that were failing on this
branch now pass; wheel + lint both green locally.
* polish(coordinator): address Copilot re-review findings
Two findings from the re-review of #368 after the first polish commit.
**user_id wired into `mgr.create()` at the server handlers.** Phase 1
added ``user_id`` to the ``Workstream`` dataclass and
``WorkstreamManager.create()`` signature, but the two call sites in
``turnstone/server.py`` forgot to pass the authenticated caller
through. Result: interactive workstreams created via
``POST /v1/api/workstreams/new`` (including coordinator-spawned
children, which route through this handler) were landing with blank
``user_id``, defeating ownership-based access control on subsequent
sends / approvals / closes (``_require_ws_access`` treats blank
owners as legacy/allowed). Two changes:
- ``server.py:create_workstream`` forwards ``user_id=uid`` — the same
``uid`` already resolved from the auth result (with trusted-service
forwarding preserved).
- ``server.py:open_workstream`` prefers the persisted owner on the
workstream row over the rehydrating caller so reloading someone
else's workstream doesn't silently re-parent it. Falls back to
the authenticated caller when the stored row has no owner
recorded (pre-phase-1 rows).
Regression test in ``tests/test_workstream.py`` pins
``WorkstreamManager.create(user_id=X)`` → ``ws.user_id == X`` so the
manager seam can't regress silently on a future refactor.
**Malformed-JSON recovery allowlist expanded for coordinator args.**
``_prepare_tool()`` has a two-stage salvage path for models that
emit malformed JSON: a regex-extract (fallback 1) and a bare-string
→ primary_key wrap (fallback 2). The fallback-1 key list didn't
include coordinator argument names, so a slightly malformed
``spawn_workstream`` / ``send_to_workstream`` / etc. call would
hard-fail instead of salvaging into a minimal-args dict for retry.
Added ``ws_id`` / ``message`` / ``initial_message`` / ``parent_ws_id``
to the allowlist (kept alphabetised) so the coordinator tools get
the same model-self-correction behaviour as the interactive tools.
Fallback 2 already covers the ``ws_id``-primary-key tools via
``PRIMARY_KEY_MAP``; the regex path matters when the model emits
``{"ws_id": "abc", "message": "..."}`` with a trailing syntax error.
Verification: ``ruff check`` clean, ``mypy turnstone`` clean
(156 source files), ``pytest -m "not live"`` → 4065 passed, 3
deselected (live-backend), 0 failed.
* fix(coordinator): address ultrareview findings on coordinator workstream kind
Security
- Cross-tenant leak: CoordinatorClient.inspect/list_children now constrain
to the coordinator's own ws_id + direct children; an LLM coerced via
prompt injection can no longer exfiltrate other tenants' workstreams.
- Empty-owner short-circuit bypass: strict equality at coordinator.py
ownership gate and at the storage-fallback branch in coordinator_history;
orphan/system-owned coordinator rows can no longer be rehydrated by
arbitrary holders of admin.coordinator (DoS + history disclosure vector).
- Closed coordinators no longer silently resurrect on subsequent GET —
the Close button is now actually durable across URL revisits and tab
refreshes; rows with state in {closed, deleted} refuse rehydration.
Correctness
- ChatSession.close() now releases the CoordinatorClient httpx.Client
pool; previously every closed/evicted coordinator dropped a connection
pool on the floor until non-deterministic GC.
- open_workstream rehydration now forwards parent_ws_id + kind, so
coordinator-spawned children survive node restart / idle eviction
with their parent link intact instead of becoming silent orphans.
- list_children truncated flag now signals whenever the SQL fetch hit
the page cap (previously permanently False in the no-filter case,
causing confident-but-incomplete summaries from the coordinator).
- ConsoleCoordinatorUI.approve_tools: per-tool auto-approve now checks
auto_approve_tools independently of the blanket auto_approve flag,
so 'Always approve this tool' actually works on the next invocation.
Concurrency
- _spawn_worker no longer falls through to start a second concurrent
worker thread on the same ChatSession when queue.Full fires; instead
send() returns False and the endpoint surfaces HTTP 429.
- _open_locks entries are now refcounted under self._lock and only
popped when the last waiter releases — eliminates the race where a
rehydration-failure path lets two threads serialize on different lock
instances for the same ws_id and trip the "already tracked" guard.
Tests: +6 regression cases covering closed-coordinator refusal,
empty-owner non-admin refusal, queue.Full no-duplicate-worker,
inspect/list_children cross-tenant rejection, and truncated semantics.
All eight suggestions verified against source before applying:
- docs/settings.md — ConfigStore key names are `model.plan_alias` /
`model.task_alias` (not `plan_model` / `task_model`); updated in
both the overview list and the plan/task overrides table.
- docs/security.md — `src` claim values now reflect what actually
gets minted: `password`, `database` (from API-token exchange),
`oidc`, plus service origins `console`, `cli`, `channel`.
- docs/sdk.md — `upload_attachment(ws_id, filename, data, *,
mime_type=...)` matches the real SDK signature; `bytes`-returning
helper is `get_attachment_content` (not `download_attachment`);
code example reordered so it doesn't collide on `filename=` kwarg.
- docs/architecture.md — "prior `plan` tool call" → "prior
`plan_agent` tool call" so wording stays consistent with the
renamed tool.
- docs/tools.md — `plan_agent` `primary_key` is `goal`, not
`prompt`, in both the primary-key table and the summary table
(matches the JSON schema in turnstone/tools/plan_agent.json).
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams. Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.
User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
gateway entry reflects shipped Discord + Slack adapters instead of
"Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
we actually expect before push.
Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
turnstone-bootstrap added to entry-points table, OpenAI provider
file split (chat/responses/common) documented, 38 SDK event
dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
adapter + multi-adapter gateway, plan_agent/task_agent naming,
governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
scopes, tokens), Slack CLI/env reference in config table, combined-
adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
pointer redirected to architecture.md.
Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
and bootstrap, add slack/bot.py, rename channels/gateway.py →
channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
SlackBot class and the slack-bolt Socket Mode edges; wire the new
bot into ChannelService. PNGs regenerated from both puml sources.
Audit pass against the actual commit messages between v1.3.0 and
v1.4.0 turned up several substantive items the initial CHANGELOG
under-described or omitted entirely. Fix-forward expansion plus a
Contributors section recognizing external contributors.
Added detail / coverage:
- New "Server compatibility layer for local model servers" entry —
the vLLM / llama.cpp profiles + admin UI fields shipped in #352
alongside the capabilities passthrough; previously buried under one
bullet.
- Per-call plan/task model selection split into three sub-bullets
(backend split, runtime configurability without restart via
ConfigStore admin tab, per-call override) — three PRs that build on
each other deserve to be discoverable independently.
- Opus 4.7 entry expanded with 1M ctx / 128K output, the new
thinking_display capability field, xhigh effort level, and admin
dropdown updates.
- Dashboard composer note: tab-bar `+` modal also gained the paperclip
+ chip strip + first-message field.
- Slack adapter: explicit "session recovery via persisted recoverable
route keys" — ops-relevant promise for restart behaviour.
- pgbouncer swap: helm chart link + ports updates noted.
- Provider capabilities entry: defensive shallow-copy + chat_template
deep-merge follow-ups.
New Fixed entries:
- Cross-user attachment-fetch hardening (get_attachment_content
scopes by user_id).
- Attachment-list DoS guard on /v1/api/send.
- Bounded LRU for upload locks.
- 3.12 CI deadlock root-cause writeup (asyncio.Lock vs Starlette
TestClient loop teardown).
New SDK entry:
- PlanResolvedEvent type + guard, dispatched cross-client when one
client resolves a plan so others dismiss in sync.
New Operational subsection:
- vendor-js workflow now auto-downloads hls.js for future Renovate
bumps so they're merge-ready without manual file fetches.
Contributors:
- Recognise @daoxley (Slack adapter, #355) and @pizzaandcheese
(pgbouncer swap, #353) — the two external contributors with
meaningful net-new work in this release — plus the Renovate bot.
- Pointer to channel-attachment ingest as the headline 1.4.1 feature
for would-be contributors.
Repo previously had no CHANGELOG. Establishes the file with full
1.4.0 coverage (attachments end-to-end, dashboard composer refactor,
Slack adapter, per-call plan/task model, provider capability
passthrough, Opus 4.7) plus a one-line 1.3.1 entry for the Opus 4.7
backport. Format follows Keep a Changelog 1.1.0; release-track
guidance up top covers the three stable branches + main.
Operator-relevant call-out at the top of [1.4.0]: migrations 037 +
038 must be applied before starting 1.4.0 against an existing 1.3.x
database. Both are additive and idempotent.
* feat(ui): dashboard composer polish from PR #362 designer review
Three deferred items from the prior designer pass on the unified
dashboard composer. Pure UX affordances; no server change.
- **Persist Options open/closed in localStorage.** Power users who
routinely set non-default model/skill don't have to click "Options"
on every page load. Key: `turnstone.dashboard.options_open`.
Defaults closed for first-time users. Falls back gracefully when
localStorage is unavailable (private mode, quota).
- **Active-options summary chip.** Renders the non-default model /
judge / skill values inline next to the Options button (mono, dim,
separated by middots). Hidden via `[hidden]` when everything is at
default — no chrome cost in the common case. Updates on any select
change via a single delegated handler on the panel. Hidden on
narrow viewports (the action row stacks vertically there and the
chip would push the layout further).
- **"Drop to attach" overlay during drag.** CSS pseudo-element on
`.dashboard-composer-drop` overlays a centered "Drop to attach"
label so dragging a file makes the action explicit instead of just
showing the dashed-border highlight. pointer-events: none keeps
the underlying composer controls reachable; visual only.
* fix(ui): address Copilot review on dashboard composer polish
- _restoreDashboardOptionsState() forced the panel closed every time
showDashboard() ran when localStorage was unavailable (private mode,
storage quota), contradicting the comment that promised a per-session
fallback. Add a module-scoped _dashOptionsOpenSession variable
updated by _setDashboardOptionsOpen / _toggleDashboardOptions, and
only override the visible state from localStorage when the read
genuinely succeeded. The session value now preserves the user's
choice across hide/show cycles in environments where localStorage
throws.
- Fold the duplicated `.dashboard-composer { position: relative; }`
block into the existing rule above. The position context is needed
for the .dashboard-composer-drop::before overlay; the comment now
says so.
PR #355 added the Slack adapter on the server but missed the console
admin surfaces that talk to channel_type. Three concrete gaps + a
designer-review polish pass.
Functional bug + UI parity:
- _collectNotifyTargets() in admin.js hardcoded `channel_type: "discord"`
— even on a Slack-only deployment the skill notify-on-complete form
always wrote Discord targets, sending notifications to the wrong
adapter (or nowhere). Add a per-row channel-type <select> driven by
a small _NOTIFY_CHANNEL_TYPES table that's the one place to register
a new platform; collector and populator both read from the dropdown.
ID-input placeholder updates dynamically when the platform changes.
- The "Link Channel Account" modal only offered Discord — users
couldn't link a Slack account through the UI at all. Add a Slack
<option> and reuse the same dynamic-placeholder helper. Drop the
static Discord-shaped HTML placeholder so the JS-driven hint doesn't
flash a Discord example before the dropdown initializes.
- Skill create/edit modals only showed Discord in the notify-on-complete
placeholder example. Show both adapters.
- Per-platform .scope-discord / .scope-slack badge classes so the
linked-accounts list distinguishes platforms visually instead of all
rendering as the generic .scope-channel magenta. Falls back to
.scope-channel for any future channel_type the stylesheet doesn't
yet know about.
Designer review polish:
- Theme-aware --discord / --slack / --discord-glow / --slack-glow
tokens in base.css. The first pass shipped raw hex (#818cf8 /
#f472b6) that fails WCAG AA on light theme (1.8:1 and 2.4:1); the
light variants (#4f46e5 indigo, #be185d rose) pass. Badge classes
now reference tokens, matching every other .scope-* rule.
- Notify-row mobile layout: three controls in a row left ~80px for
the ID input at 360px viewport, truncating snowflakes. Tighten
platform select to 76px (labels are short), add flex-wrap, and at
≤700px drop the ID input to its own row so it gets full width.
- Per-platform classes apply alone (not co-classed with scope-channel)
so winning the cascade doesn't depend on stylesheet source order.
- Replace "Discord snowflake" jargon with "Discord ID"; give Slack
ids concrete examples (C01234567 / U01234567) instead of an
ambiguous "C0…".
Combines the substantive bot.py fixes flagged in both review trails on
PR #355. Discord parity items grouped here too since they're the same
surface (slack/bot.py).
From Copilot:
- _notify_reply_routes was read on StreamEndEvent but never popped on
the success path. Result: one notification reply pinned every later
response for that ws_id to the notification thread until the bot
restarted. Pop after read; combine the surrounding ifs (SIM102).
- PlanReviewEvent embedded raw event.content inside a triple-backtick
mrkdwn fence without escaping. A plan with ``` (very common — plans
often quote code) would break the fence and let later content render
as live markup, including unintended Slack mentions/links. Rewrite
_sanitize_slack_preview to splice a zero-width space inside any ```
sequence (Slack stops recognizing it as a delimiter) instead of
escaping every single backtick — keeps single-backtick code snippets
readable while still protecting the fence. Apply to plan-review.
- _send_approval_request joined unbounded tool_lines into one mrkdwn
section, but Slack section.text caps at 3000 chars. Multi-tool
batches with large previews silently failed chat_postMessage,
leaving the user unable to approve/deny. Cap each preview to 600
chars under a 2700-char total budget; append "+N more" when truncated.
From eous (parity with Discord):
- Pass `client_type="chat"` from both `get_or_create_workstream` call
sites (slash-command session + DM). Without it Slack-routed
workstreams loaded the web-default prompt; the chat-specific
system prompt now applies as it does for Discord.
- Add `exc_info=True` to the eleven `log.debug(...)` exception handlers
so underlying tracebacks are available when debug logging is on
instead of being silently dropped. Level stays debug — these are
benign-by-default sites (chat_update on a deleted message, etc.) so
only the visibility changes. Typed-exception handlers
(RemoteProtocolError, etc.) keep their bare debug log.
- Module docstring on slack/__init__.py so pydoc / import errors have
human-readable context.
Tests: rewrite the sanitizer test to match the new (more permissive)
single-backtick behaviour; add coverage for the triple-backtick
neutralization + short-input passthrough; patch httpx.AsyncClient at
all five TurnstoneSlackBot construction sites so each test doesn't
leak an unclosed real client.
- cli.py: ChannelAdapter import is annotation-only; move into
TYPE_CHECKING block and switch the two cast() calls to string-form
so the runtime import isn't required (TC001).
- slack/{config,routes}.py: ruff format fixes (whitespace + drop
redundant string-form annotation now that __future__ annotations
is in effect).
- pyproject.toml: drop the unused `tests.*` mypy override — `mypy
turnstone` (the only invocation in CI + local) never matches it,
so it was pure noise in the "unused section(s)" report. Other
optional-dep overrides stay; they're real safety nets when running
mypy without the [all] extras (e.g. on the test job).
- uv.lock: regenerate to match the slack-bolt + transitive deps the
pyproject changes resolve to (lock-check was failing on stale hash).
* fix(ui): rehydrate chip strip after queued-message dequeue not_found
The dequeue handler only refreshed the per-pane chip strip when the
DELETE returned status="removed". On status="not_found" (the queued
message already dispatched), chips stayed stale: any reservations that
raced the dispatch could leave the UI showing a different pending set
than the server actually had.
Re-fetch on both paths so the chip strip always reflects the
authoritative server state. The queued-message bubble itself stays
visible on not_found, same as before — the promote loop strips the
queued styling on idle.
* feat: sweep orphan attachment reservations periodically
Process crashes between reserve_attachments and consume/unreserve can
leave attachment rows soft-locked forever (reserved_for_msg_id NOT NULL
with no consumer ever coming back). The worker-thread exception path
in /v1/api/send already handles in-process failures, but a hard kill
or oom mid-send escapes that.
Add sweep_orphan_reservations(older_than_seconds) to the storage
protocol — clears reserved_for_msg_id on rows where message_id IS NULL
and created < now() - threshold. Implemented for SQLite + PostgreSQL
using the same string-comparison form (created is ISO-8601 text in
both backends, lexicographic order matches chronological).
Wire into the server lifespan: run once at startup (catches anything
left over from the previous process), then every 30 minutes as
defense-in-depth. Threshold is 4 hours so we don't race a long-running
dispatch and unreserve rows the worker is still about to consume.
Tests cover sweep semantics: clears old reserved rows, leaves fresh
ones alone, skips already-consumed rows, no-ops on zero/negative
threshold.
* fix: track reserved_at for orphan-reservation sweep
Copilot review on PR #363 flagged a real correctness bug: the sweep
used the attachment row's `created` timestamp (upload time) as the
staleness signal. An attachment uploaded hours ago but reserved fresh
could be unreserved mid-send, after which mark_attachments_consumed
silently drops the row because reserved_for_msg_id no longer matches
the send_id.
Add a dedicated `reserved_at` column (migration 038) set on
reserve_attachments and cleared on mark_attachments_consumed /
unreserve_attachments. The sweep now scopes by `reserved_at < cutoff`,
so reservation age is what's measured, not upload age. Backed by a
partial index `(reserved_at) WHERE reserved_at IS NOT NULL` so the
periodic scan stays cheap as the consumed-history grows.
Threshold dropped from 4h to 1h since it now means "longest realistic
single send" rather than "longest plausible time between upload and
send" — a tighter, more defensible bound.
Tests cover the regression (uploaded long ago + reserved fresh must
not be swept), plus reserved_at clearing on both consume and unreserve.
* feat: workstream attachments at creation time + SDK + UI parity
Closes the two big deferred items from PR #356: attaching files as part
of the initial workstream-creation request, and full SDK coverage of the
attachment surface.
Server: POST /v1/api/workstreams/new now accepts multipart/form-data
(meta JSON + 0..N file parts). Files are validated and saved as pending
under the new ws; when initial_message is also set the create handler
reserves them onto that turn before the dispatch worker fires, mirroring
the /v1/api/send pattern. Validation failure rolls back the workstream
via delete_workstream so we don't leak orphan rows or emit a phantom
ws_created/ws_closed pair on SSE. JSON path is unchanged.
Console routing: route_create accepts multipart with ?ws_id=<hex> as a
query parameter (the console hashes the id before the body lands).
Added /v1/api/route/workstreams/{ws_id}/attachments POST/GET/DELETE +
.../{attachment_id}/content GET proxies that forward raw bytes and
preserve upstream headers (Content-Disposition, X-Content-Type-Options,
CSP sandbox).
Python + TypeScript SDKs: AttachmentUpload type, upload_attachment,
list_attachments, get_attachment_content, delete_attachment, and
send(attachment_ids=...). create_workstream(attachments=...) sends
multipart and pre-generates a ws_id client-side so cluster routing
works. SDKs reject attachments+target_node combinations since the
multipart route doesn't honor target_node.
Web UI: dashboard composer refactored to a single unified create flow.
Replaced the inconsistent split (Enter created+sent raw, "New Chat"
opened a modal) with one rich composer carrying a textarea, paperclip
+ chip strip, drag-drop, paste-image, and a collapsible Options panel
for model/judge_model/skill. Submit button dynamically labels Create
vs Send. New-workstream modal also gained the same paperclip + chip
strip + first-message field for the tab-bar + entry point.
Tests: 30 new tests across server multipart create, console route
multipart + attachment proxies, Python + TS SDK attachment surfaces,
plus regressions for the three review-flagged bugs (Content-Type
boundary preservation, attachments+target_node rejection, no phantom
ws_created on validation failure).
* fix: address Copilot review feedback on PR #362
- web_helpers: docstring now matches behaviour — read_multipart_create_or_400
does enforce the optional max_per_file_bytes cap as defense-in-depth.
- app.js: drop the duplicated _formatAttachSize definition (one already
exists earlier for pane chips); add a shared _isAttachmentAllowed helper
that mirrors the server's classifier (png/jpeg/gif/webp images, text/*
MIMEs, allowlisted application/* MIMEs, known text extensions) and call
it from both _newWsAddFiles and _addDashboardFiles so unsupported files
fail fast client-side instead of after a server roundtrip.
- app.js: dashboardSubmit catch now suppresses the redundant error toast
on authFetch's "auth" Error and falls back to a generic message when
err.message is undefined, instead of rendering "Connection error: undefined".
- SendResponse (Pydantic + TS): document and expose attached_ids,
dropped_attachment_ids, priority, and msg_id so attachment-aware SDK
callers can detect partial reservations and dequeue queued messages.
- test_server_attachments_on_create: drop the dual `import turnstone.server`
+ `from turnstone.server import` style — use monkeypatch.setattr by
dotted path for module-level mutation and `from … import …` for the
helpers, keeping a single import style.
* feat: per-call model selection on plan_agent / task_agent
The calling LLM can now pass `model="<alias>"` to plan_agent or
task_agent to override the operator-configured per-kind model for
that one invocation. Useful when subtask difficulty varies within a
session: the model can downgrade to a cheap alias for trivial work
and reach for a stronger one when the problem is hard.
Tool descriptions list the live registered aliases (refreshed when
the operator hits "sync to nodes" / internal_model_reload), so the
calling LLM always sees the current options. Bad aliases return a
corrective error dict with the available choices so the LLM retries
cleanly rather than failing silently.
No whitelist — any alias the registry knows is acceptable; cost
control is intentionally ceded to the model. No per-call effort
override (out of scope; effort stays operator-configured).
Resolution precedence in _run_agent: explicit per-call agent_alias
override > registry per-kind (plan_model/task_model) > legacy
agent_model > session model. The plan retry path (when
_validate_plan fails) reuses the same alias so coaching reflects
real model behaviour rather than a different model masking the
signal.
Implementation:
- plan_agent.json / task_agent.json: optional `model` parameter.
- ChatSession._validate_agent_model_override extracts and validates
the arg; mirrors the existing empty-prompt error pattern.
- _prepare_plan / _prepare_task stash the override in
item["model_override"]; _exec_* pass it through.
- _run_agent gains agent_alias kwarg with defence-in-depth
ValueError on unknown alias.
- _render_agent_tool_descriptions deep-copies plan/task entries
before mutating description so the module-level TOOLS constant
stays untouched across sessions; rebuilds the BM25 tool-search
index when active so its text matches what the LLM sees.
- server._broadcast_agent_tool_schema_refresh walks active
workstreams on internal_model_reload so descriptions update
without restart.
* fix: clarify no-registry placeholder + avoid double BM25 rebuild
Addresses Copilot feedback on PR #361.
1. plan_agent.json / task_agent.json placeholder said the parameter
falls back to the "operator-configured plan/task model". That
text is what no-registry sessions see (registry-bearing sessions
get the templated description with the live alias list); for
those single-model sessions, omitting the param falls back to
the current session model, not an operator-configured one.
Reword so the no-registry user gets accurate guidance.
2. _on_mcp_tools_changed already calls _rebuild_tool_search after
merging MCP tools. _render_agent_tool_descriptions also
rebuilt the BM25 index when active, so the MCP refresh path
was rebuilding twice per refresh. Move the BM25 rebuild out
of the private render helper into the public
refresh_agent_tool_schemas wrapper — _on_mcp_tools_changed
keeps calling the render helper directly (no double rebuild),
and registry-reload callers go through the wrapper which
still keeps the index in sync.
* feat: ConfigStore + admin UI for plan/task agent model and effort
Per-kind sub-agent routing was added in #359 but only via config.toml.
Operators can now switch the plan_agent / task_agent model and reasoning
effort at runtime from the admin Model tab without restarting.
Adds four ConfigStore-backed settings:
model.plan_alias — alias for plan_agent
model.task_alias — alias for task_agent
model.plan_effort — reasoning effort for plan_agent
model.task_effort — reasoning effort for task_agent
Server startup and internal_model_reload both apply these as overrides
on top of the registry's config.toml-loaded values; the new logic
computes "effective" values for all five model-routing fields and only
calls registry.reload() when at least one differs.
Admin UI: extracts ALIAS_SETTING_KEYS to a const used by both the
dynamic-alias-choice injection and the empty-option label rendering.
Adds INHERIT_EMPTY_LABEL_KEYS so plan_effort / task_effort show
"(inherit)" for empty — distinct from the literal "none" choice (which
actually disables reasoning, very different from leaving unset).
Also fixes Copilot review feedback from #359:
- _validate_effort treats empty / whitespace as unset rather than
warning on benign explicit-empty configs (with .strip().lower()
normalisation; "HIGH" and " low " now parse correctly)
- turnstone.example.toml's reasoning_effort comment lists the full
set of accepted values (none, minimal, low, medium, high, xhigh, max)
* fix: apply routing overrides on config-reload + skip no-op model-reload
Addresses Copilot feedback on PR #360.
1. Admin settings updates fan out via /_internal/config-reload, which
only reloaded the ConfigStore — plan/task routing changes weren't
visible until a model-reload or restart, defeating the runtime
configurability this PR is meant to add.
2. /_internal/model-reload always called registry.reload(), churning
cached clients even when nothing changed. Risky when fanned out
across nodes (could close in-flight clients).
Extracts two helpers in server.py:
- _effective_routing(cs, ...) pure function: overlay CS values on base
- _apply_routing_overrides(reg, cs) reload only when something differs
Used by the startup path, config_reload (new), and model_reload (now
short-circuits with a noop response when models + routing are unchanged).
plan_agent and task_agent previously shared a single agent_model knob and
plan_agent hardcoded reasoning_effort="high" in three call sites. They
have different cost/latency profiles — plan is rare and benefits from a
stronger model, task is frequent and benefits from a cheaper one — so
sharing the knob undertunes both.
ModelRegistry gains plan_model, task_model, plan_effort, task_effort.
Per-kind overrides win over the legacy agent_model, which still works
as the single-knob fallback for both. resolve_agent_alias(kind) and
resolve_agent_effort(kind) centralise the resolution; PLAN_DEFAULT_EFFORT
captures the back-compat "high" default in one place rather than at
every call site.
session._run_agent delegates resolution by label ("plan" vs "task").
The three hardcoded reasoning_effort="high" arguments are removed —
behaviour is identical when no plan_effort is configured.
Loader validates effort against {none,minimal,low,medium,high,xhigh,max}
and warns + drops typos rather than passing them to the provider.
ConfigStore parity and admin UI for the new knobs are deferred to a
follow-up — config.toml-only is enough for the backend split.
Previously, resolving a plan on one client (e.g. phone) cleared the
server's pending state and unblocked the worker, but emitted no event
to other connected clients. Their plan-approval modal stayed stuck.
resolve_plan() now enqueues a plan_resolved frame (mirroring the
approval_resolved pattern in resolve_approval) before clearing
_pending_plan_review, so a reconnecting client cannot receive both
the replayed plan_review and the live plan_resolved. Skips the frame
on the cancel-with-no-plan path.
Client adds a plan_resolved handler that dismisses the modal without
re-firing /v1/api/plan, restores keyboard context (skipped on touch
to avoid soft-keyboard pop on mobile), labels the inline plan summary
"(synced)" so remote dismissal is unambiguous, announces via the
existing aria-live #toast for screen-reader parity, and falls back
to an info message if plan_resolved races ahead of plan_review.
Adds PlanResolvedEvent to the Python and TypeScript SDKs with
deserialization and type-guard tests.
- Add claude-opus-4-7 capability entry (1M ctx, 128K output, adaptive
thinking, supports_temperature=False, thinking_display=summarized)
- Suppress temperature param for Opus 4.7 (API returns 400)
- Add thinking display opt-in via new ModelCapabilities.thinking_display
field - Opus 4.7 omits thinking by default, always send summarized
- Add xhigh effort level to mapping and Opus 4.7 effort_levels
- Add xhigh/max options to skill template dropdowns in admin console
- Align reasoning effort label capitalization across all console dropdowns
- Update example config to reference claude-opus-4-7
- 10 new tests with regression guards for Opus 4.6 backward compat
Verified against live API: streaming and completion calls succeed.
Trivy flags two HIGH CVEs in jq/libjq1 1.7.1-6+deb13u1 with no fixed
version yet from Debian:
- CVE-2026-39979: out-of-bounds read in jv_parse_sized() on non-NUL-
terminated buffers
- CVE-2026-40164: DoS via crafted JSON causing hash collisions
jq is invoked only on trusted CLI/admin paths against
process-controlled JSON input in turnstone — never on untrusted
network bytes — so the NUL-terminated invariant holds and the DoS
vector is not reachable.
Will revisit when Debian publishes a patched libjq1.
* feat: workstream attachments (images + text documents)
Adds end-to-end support for attaching images (png/jpeg/gif/webp) and
plain-text documents (markdown, source, JSON, etc.) to a workstream's
next user turn via the web UI.
Storage: new workstream_attachments table (migration 037) with a
three-state lifecycle — pending → reserved → consumed — scoped by
(ws_id, user_id) and linked to conversations.id on consume. Rewind/
truncation cascades attachment rows; delete_workstream does too.
Session: ChatSession.send(attachments, send_id) builds multipart user
content (text + image_url + document parts) and persists text-only to
conversations with attachments joined on load via message_id. Queue
path carries ordered attachment_ids plus a reservation token so
queued multimodal turns can't lose files to overlapping sends.
Providers: internal document content parts translate at the API
boundary — Anthropic emits native document blocks (text/plain
coerced, original MIME folded into title); OpenAI Chat Completions
and the Google OpenAI-compat endpoint inline them as escaped
<document> text blocks (XML-attr escape + </document> neutralization);
Responses API emits input_text with the same wrapper.
Server: POST/GET/DELETE /v1/api/workstreams/{ws_id}/attachments with
multipart upload (magic-byte image sniffing, UTF-8 enforcement for
text, per-kind size caps, Content-Length pre-check, per-(ws,user)
pending cap + TOCTOU lock). /v1/api/send reserves before dispatch
using a full-UUID token, threads it into session.send / queue_message,
releases on worker-thread failure, and reports attached/dropped ids
so the UI can reflect partial reservations. GET /content sets
X-Content-Type-Options, CSP sandbox, inline Content-Disposition, and
forces text/plain for text kinds. Ownership failures mask as 404.
UI: paperclip button, hidden file input with accept allowlist, chip
strip above textarea, drag/drop + paste-image handlers. Chips
rehydrate on ws switch and on queued-message dequeue; send clears
only attached ids and shows a toast when some dropped. Historical
user messages render filename pills via a _attachments_meta sibling
populated on both live-send and reconstruct paths.
530 tests covering CRUD, reservation lifecycle, races (TOCTOU cap,
reserve-then-dispatch overlap), provider translation, XSS headers,
cascade delete, history round-trip, and service-scoped actor flow.
* fix(attachments): address PR review feedback
- get_attachment_content now scopes the row by user_id too, so an
unowned workstream can't be a vector for cross-user blob fetches
via attachment_id guessing (Copilot, server.py:2676)
- send_message rejects attachment_ids lists longer than the pending
cap with 400 — prevents hostile clients from blowing up the
storage IN (...) clause (Copilot, server.py:1515)
- _attachment_upload_locks switched to a bounded LRU OrderedDict;
evicts the oldest unlocked entries past the soft cap so the map
can't grow unboundedly on long-running nodes (Copilot, server.py:2417)
- Pane.dragleave handler uses relatedTarget instead of target so the
drop-zone styling clears correctly when the cursor moves through
child elements; dragend listener added as a fallback for cancelled
drags (Copilot, app.js:297)
- uploadAttachment always cleans up the placeholder chip on failure,
including auth errors — no more stuck "uploading..." chips after
re-auth (Copilot, app.js:427)
- New _swapPlaceholderChip / _removeAttachmentChip helpers preserve
user-selection order through the placeholder→real-id swap; the
pendingAttachments Map is rebuilt in place rather than naïvely
delete+set, which would have moved the entry to iteration end
(Copilot, app.js:420)
- Drop unused `var self = this;` in removeAttachment (github-code-quality)
- Two regression tests: cross-user fetch on an unowned workstream,
and oversized attachment_ids list rejection
* fix(attachments): switch upload-lock to threading.Lock to avoid 3.12 CI hang
The per-(ws, user) upload lock was a module-cached asyncio.Lock.
Starlette's TestClient runs each request on a fresh anyio task /
event loop, so the cached lock's internal _waiters bind to the first
loop that acquired it. When a later request runs in a different
loop, await lock.acquire() blocks on a Future from a closed loop —
silent deadlock.
This surfaced as test (3.12) hanging indefinitely in CI on one push
while the same suite passed on 3.11/3.13 and on the next push. Same
root cause is reproducible against any Starlette TestClient harness
on 3.10+; 3.12 just happens to surface it more often given changes
in how anyio + asyncio.Future interact across loop teardown.
Switched to threading.Lock — loop-agnostic, and the critical section
is one COUNT + one INSERT, short enough that briefly blocking the
event loop is fine. Updated the LRU-eviction probe accordingly
(threading.Lock has no public .locked(), so use a non-blocking
acquire+release as the "is it free?" probe).
TOCTOU pending-cap test still passes; full attachment suite passes
on both 3.12 and 3.13.
* replace bitnami pgbouncer wit edoburu
replaced bitnami pgbouncer with edoburu pgbouncer container and updated environment variables to fit
* updated ports & Kubernetes
Updated ports to fit existing documentation. Also updated the Kubernetes Helm Chart link to use the same container.
* chore(deps): update dependency hls.js to v1.6.16
* chore: download vendored hls.js files + add hls to workflow detection loop
The wheel-completeness check failed on the Renovate bump because
vendor-js.yml only iterated katex/hljs/mermaid — so hls.js PRs
never got their files auto-downloaded. Adding hls to the loop so
future Renovate bumps are merge-ready without manual intervention.
Also running the update now to fix this specific PR.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Patrick Buckley <buckleypm@gmail.com>
* feat: pass resolved capabilities through to providers, add server compat layer
The LLMProvider protocol previously forced providers to re-derive
capabilities from static lookup tables, ignoring config overrides set
via the admin UI or config.toml (e.g. thinking_mode, token_param).
This adds an optional capabilities parameter to create_streaming and
create_completion so the session can pass its config-merged
ModelCapabilities through to providers.
On top of this, adds a server compatibility layer for local model
servers (vLLM, llama.cpp). Profiles suggest thinking mode and server
workarounds (skip_special_tokens for vLLM, reasoning_format for
llama.cpp) during model detection, with structured admin UI fields
for server type, thinking mode, and extra body params.
Verified against real vLLM (Gemma 4 31B) and llama.cpp (Gemma 4 E4B)
servers.
* fix: defensive copy in _finalize_extra_body, expose thinking_param in UI
Shallow-copy extra_params and its chat_template_kwargs in the provider
before _apply_thinking_mode mutates them, so callers that reuse the
same dict across models are safe.
Replace the hidden thinking_param input with a visible text field
that appears when thinking mode is enabled. Shows the default
"enable_thinking" and hints that Granite/DeepSeek use "thinking".
* fix: address Copilot review feedback on admin UI and server compat
- Preserve unrepresentable thinking_mode values (e.g. "adaptive") in
raw capabilities JSON instead of silently dropping on edit round-trip
- Validate capabilities and extra body JSON are plain objects, not
arrays or primitives
- Deep-merge chat_template_kwargs from extra_body instead of silently
dropping, so operators can extend/override template kwargs
* fix: hide server compat section for non-local providers
The Server Compatibility fields (server type, thinking mode, extra
body) only apply to openai-compatible (local model servers). Hide
the entire section when the provider is openai, anthropic, or google.
* fix: normalize capsObj to plain object on edit load
Defend against DB rows where capabilities is a JSON literal null,
an array, or a primitive — previous code would crash on the
capsObj.server_compat / capsObj.thinking_mode reads. Same defensive
check also applied to the server_compat nested value.
* refactor: extract _isPlainObject helper for JSON type checks
Consolidates the null/array/typeof check that was inlined at three
different call sites into a single helper. Keeps the intent obvious
at each use site and avoids the awkward multi-condition ternary.
* feat: per-model sampling parameters (temperature, max_tokens, reasoning_effort)
Model sampling parameters were global-only settings applied uniformly to
all models. Different models have fundamentally different requirements
(o-series needs no temperature, Anthropic needs temp=1.0 with thinking,
local models may need different max_tokens). This adds per-model overrides
with global fallback so each model definition can specify its own defaults.
Migration 036 adds nullable temperature, max_tokens, reasoning_effort
columns to model_definitions. NULL inherits the global default from
ConfigStore. The session factory and /model switch command both resolve
per-model override → global fallback consistently.
The admin UI model create/edit modal now has dedicated form fields for
these parameters with client-side validation, a visual section divider,
and per-model override hints in the model table rows.
Removes vestigial model.name and model.context_window global settings
(now handled per-model by the model registry) with startup warnings for
existing config.toml users.
* fix: defensive parsing for config.toml per-model sampling params
Wrap temperature/max_tokens conversions in try/except with range
validation. Invalid values log a warning and fall back to None
(inherit global default) instead of aborting registry load.
* fix: use gethostname() instead of getfqdn() for advertise URLs
socket.getfqdn() does a reverse DNS lookup that often returns a
truncated hostname (e.g. "flat" instead of "flat-blck-io"). Use
gethostname() for advertise URLs in both server and console. For TLS
SANs, include both names so certs cover all variations.
* docs: clarify advertise URL comment re Docker/k8s
* fix: standardize database env vars on TURNSTONE_DB_* naming
compose.yaml used DB_BACKEND/DATABASE_URL in .env which got mapped to
TURNSTONE_DB_BACKEND/TURNSTONE_DB_URL inside containers. Running bare-
metal required the TURNSTONE_ prefix, but docs didn't explain this.
Eliminate the indirection — use TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL
everywhere (compose, .env, bare-metal, docs, bootstrap wizard).
* fix: update .env.example to use TURNSTONE_DB_* naming
* fix: universal tool_call/tool_result orphan detection for OpenAI-compat providers
The Anthropic provider had orphan detection for mismatched tool_call ↔
tool_result pairs, but OpenAI-compatible providers (Chat Completions,
Google, Responses API) had none. When an Anthropic model runs behind
an OpenAI-compat API (e.g. Azure) or cancellation creates orphans,
the API rejects the malformed request.
- Rewrite sanitize_messages() with orphan detection: synthesize error
tool results for unmatched tool_calls, drop tool results with no
matching tool_call, fill empty tool_call IDs with positional remap
- Call sanitize_messages() from Responses API _convert_messages()
* fix: address review feedback on orphan detection
- Track answered IDs per-turn (local_answered) instead of scanning
all of out, preventing false matches from reused IDs across turns
- Drop empty-ID tool results that have no remap entry instead of
passing them through with invalid empty tool_call_id
- Increment empty_result_idx for every empty result, not just remapped
- Remove dead result_ids peek-ahead code
- Add test for repeated tool_call IDs across turns
* fix: accurate token usage tracking for compaction across all providers
Anthropic's input_tokens excluded cached tokens, causing massive
under-reporting (e.g. 327 vs 9000 actual) when prompt caching was
active. This prevented auto-compaction from triggering.
- Normalize Anthropic prompt_tokens to total input (input_tokens +
cache_creation + cache_read), matching OpenAI semantics
- Reset _last_usage per API call so tool-chain iterations get fresh
usage instead of max()-merging with stale values
- Add mid-turn compaction check during tool chains to prevent context
overflow before end-of-turn
- Anchor _remaining_token_budget() on provider-reported prompt_tokens
with local estimates only for the delta since last API call
- Improve _msg_char_count() to include structural overhead (role,
tool_call_id, tool call IDs) and handle image tokens in calibration
- Emit status after every API call, not just end of turn
* fix: defensive null coercion and index clamping from review feedback
- Add `or 0` to all getattr calls for input_tokens/output_tokens in
Anthropic provider (streaming + non-streaming) to handle SDK nulls
- Use getattr for non-streaming input_tokens/output_tokens instead of
direct attribute access for consistency
- Clamp _calibrated_msg_count with min() in _remaining_token_budget()
to prevent stale state from over-slicing after compaction
The ws-check-hint animation clobbered the fadein's forwards fill,
making the checkbox invisible for 0.6s on card-body click — appearing
as a deselect-then-reselect. Remove the hint, the unused role=checkbox
on the card, and restore the original symmetric toggle behavior.
* fix(ui): improve delete workstream UX and accessibility
Card body click no longer deselects (prevents confusing red border loss);
checkbox pulse hint guides users to deselect affordance. Adds keyboard
navigation, aria-labels, hover feedback, animations, and neutral Close
button styling after deletion.
* fix(ui): remove duplicate a11y checkbox from delete-mode cards
Hide the visual checkbox from the a11y tree and tab order so the card
(role=checkbox) is the sole keyboard/screen-reader target. Addresses
Copilot review feedback about nested interactive elements.
* perf: reduce initial rebalance from ~1.5s to ~50ms on PostgreSQL
Increase seed_ring_buckets chunk sizes (PG 500→16k, SQLite 500→8k) to
cut network round-trips from 131 to 5. Add ConsoleRouter.populate_from_assignments()
to build the routing cache directly from computed assignments, eliminating the
65 536-row DB read-back. Router becomes ready in <1ms; DB persistence follows.
* fix: address review — populate after seed write, sync router version
Move router cache population after seed_ring_buckets() so the router
is never "ready" with an unpersisted ring. Pass the new rebalancer
version to populate_from_assignments() so check_version() on the
collector thread does not trigger a redundant 65 536-row refresh.
- Remove role="status" and aria-label during _promoteQueuedMessages
so screen readers don't announce stale "queued" context
- Mark element with pendingDismiss when user dismisses before msg_id
arrives; send deferred DELETE when the send response provides the ID
If the model responds without tool calls, the main loop exits
immediately — no tool-result seam exists for advisory injection.
Queued messages were silently orphaned in the OrderedDict. Now
flushed as regular user messages before emitting idle state.
Bug 1: Extract _promoteQueuedMessages() — removes badge, dismiss
button, queued classes, and data-msgId. Called from setBusy(false)
on state_change: idle.
Bug 2: _dequeueMessage no longer removes the DOM element when server
returns not_found (message already injected). Only removes on
"removed" (actually dequeued). Network errors also preserve the
element. The promote loop handles cleanup on idle instead.
* feat: tool result advisory system with user message queuing
General-purpose advisory injection for tool results — when advisories
are present, tool output is wrapped in <tool_output> tags with
<system-reminder> blocks appended. Two initial producers:
- Output guard advisories: model sees why content was flagged/redacted
- User message interjections: users can queue messages mid-execution
via the web UI, injected at the next tool-call seam
Queued messages use !!! prefix for important priority. Advisory
injection is gated by ModelCapabilities.supports_tool_advisories
(default true for commercial models, false for local/vLLM).
On cancel/error, queued messages are flushed as regular user messages
so nothing is silently lost. Raw tool output (pre-wrap) is persisted
to the DB to keep history clean of ephemeral advisory XML.
* fix: frontend UX for queued messages — rollback, discoverability, a11y
- Send button changes to "Queue" (outline style) during busy state,
visually distinct from filled red Stop button
- Placeholder updates to hint at !!! priority convention
- addQueuedMessage returns element ref for optimistic UI rollback
- Remove queued element on queue_full, busy, or connection error
- Add role="status" and aria-label to queued message elements
- Promote queued messages to normal appearance when generation ends
* feat: queued message removal via dismiss button
Switch backing store from queue.Queue to OrderedDict + Lock for O(1)
removal by ID. Each queued message gets a UUID, returned to the
frontend and stored as data-msg-id on the DOM element.
Dismiss button (x) on queued messages calls DELETE /v1/api/send with
the msg_id. If the message was already injected (race), server returns
not_found and the UI removes the element anyway.
No new endpoint — DELETE method added to the existing /v1/api/send
route. dequeue_message() on ChatSession is O(1) under the lock.
* fix: address PR review — escaping, types, list output, message cap
- Escape </tool_output> and <system-reminder> in tool output to prevent
wrapper tag injection from untrusted tool results
- Change _collect_advisories return type from list[Any] to list[ToolAdvisory]
- Drain queued messages on list/structured output (append as text part)
so they aren't silently stuck until a str result appears
- Cap queued message length at 2000 chars to prevent context bloat
- Remove unused var in _dequeueMessage
2026-04-06 21:51:47 -07:00
839 changed files with 298446 additions and 37643 deletions
*A hypothesis — not a theorem. The honest answer is a claim about **shape**: an object you can write down that says what a harness is and, just as precisely, the guarantee it cannot carry for free.*
Most descriptions of an agent framework are a feature list. This is an attempt at a definition.
---
## The claim
*Informal.* A harness is a **stopped, deterministically-controlled Markov process on task-state, closed around a stopped autoregressive process on context-space, driven by a learned model kernel** — a deterministic controller in closed loop with a stochastic learned plant.
*In plain terms.* The **harness** is the whole governed loop: a deterministic **shell** you write — build the prompt, authorize an action, fold the response back into state — wrapped around a black-box stochastic model kernel (the **plant**, $M_W$) and the environment its actions touch, looped until it halts in $H$. The shell is deterministic, $M_W$ is not, and everything below makes that split precise.
*Formal — the objects.* A harness is a tuple $\mathcal{H} = (\mathcal{S}, \mathcal{C}, \mathcal{Y}, \mathcal{A}, \mathcal{E}, \pi, M_W, \gamma, Q_E, \rho, H, H_{\mathrm{ok}}, B)$ over **standard Borel** spaces (concretely: the *controlled* state is standard Borel by construction — token sequences, finite config maps, bounded counters and ledgers, finite tuples of real vectors — and the model/environment coordinates are inherited as such whenever they serialize to a Polish space; the assumption is roomier than it looks — even a belief-state coordinate valued in $\mathcal{P}(X)$ survives, since $\mathcal{P}(X)$ is Polish for Polish $X$ — and fails only for a genuinely non-separable coordinate, an uncountable product $\sigma$-algebra being the canonical hazard, which this construction avoids): a deterministic lowering $\pi:\mathcal{S}\to\mathcal{C}$; a stochastic model-run kernel $M_W(c, dy)$ into a readout space $\mathcal{Y}$ (which includes the parse-failure $\bot$, so $M_W$ and $\gamma$ are total over it); a deterministic **authorization gate** $\gamma:\mathcal{S}\times\mathcal{Y}\to\mathcal{A}_{\bot}$ that validates the model's parsed readout into an authorized action in $\mathcal{A}$ or rejects it as $\bot$ (parsing itself lives inside $M_W$ — realized as the readout $R$ of the specialization below); a stochastic environment/tool kernel $Q_E:\mathcal{S}\times\mathcal{A}_{\bot}\rightsquigarrow\mathcal{E}$ on the authorized action (rejection included, with $Q_E(s,\bot,\cdot)=\delta_{e_0}$ for a distinguished no-op response $e_0\in\mathcal{E}$); and a deterministic verify-and-fold-back map $\rho:\mathcal{S}\times\mathcal{Y}\times\mathcal{A}_{\bot}\times\mathcal{E}\to\mathcal{S}$.
*Terminal structure.* The terminal set is an absorbing halt set $H\subseteq\mathcal{S}$ (the daemon "ready-state" recurrence of the note below is a separate, non-absorbing object) with accepting subset $H_{\mathrm{ok}}\subseteq H$; separately, a bad set $B\subseteq\mathcal{S}$ ($B\cap H_{\mathrm{ok}}=\varnothing$) marks the unsafe states for reach-avoid, possibly entered before any halt; hitting times are $\tau_A=\inf\{n\ge 0:s_n\in A\}$, and $\tau_H$ is a stopping time for the natural filtration.
*The outer kernel.* The induced outer transition kernel, for $s\notin H$, is
$$T(s, A) = \int_{\mathcal{Y}}\!\int_{\mathcal{E}} \mathbf{1}_A\!\big(\rho(s, y, \gamma(s,y), e)\big)\; Q_E\big(s, \gamma(s,y), de\big)\; M_W(\pi(s), dy), \qquad T(s,A)=\mathbf{1}_A(s)\ \text{ for } s\in H,$$
and the harness runs $s_{n+1} \sim T(s_n)$ from an initial $s_0 \sim \mu_0$ until $\tau_H = \inf\{n : s_n \in H\}$. Because $\pi, \gamma, \rho, H$ are deterministic they contribute no integration variable of their own — they appear as measurable transformations inside the integrand (the pushforward), not literally outside it — so the controller injects no randomness, and every coin is inherited from $M_W$ and $Q_E$. (The earlier shorthand $T = \rho \circ (M_W \circ \pi, E)$ is suggestive but ill-typed — $M_W$ returns a *law*, while $\rho$ consumes a *sample* together with the prior state $s$; the integral is what the shorthand meant.)
*Fail-closed.* The gate $\gamma$ is what makes **fail-closed** a property, not just a name: model output is an *untrusted proposal*, and $\gamma(s,y)=\bot$ forces a no-op environment response ($Q_E(s,\bot,\cdot)=\delta_{e_0}$) — so a malformed or unauthorized tool call is rejected *before* it can act, not validated after its side effects have landed. Fail-closed is then the property that a rejected proposal causes *no unauthorized side effect* and lands in a **safe, non-bad** set ($\rho(s,y,\bot,e_0)\notin B$): a non-accepting terminal $H\setminus H_{\mathrm{ok}}$ in the strict case, or a safe non-terminal state when the spec retries. And $\rho$ must validate the tool *response* $e$, not only the proposal that $\gamma$ already gated: a malformed or adversarial response $e$ is caught at fold-back, not just at the gate. But response-validation has a hard limit: $\rho$ can reject a bad tool *response*, yet it cannot undo side effects an *authorized* action already caused — so $\gamma$, not $\rho$, is the last line before irreversible effects, and anything irreversible must be gated at authorization. The boundary is also only real if raw model output reaches *no* sink — tool, logger, browser, or remote call — before $\gamma$; any pre-authorization escape bypasses the gate. The user-visible final response and any logging are themselves effects, and the rule binds *model-authored* bytes: they reach a sink either as an authorized action through $\gamma$, or only after an accepted halt in $H_{\mathrm{ok}}$. Shell-*templated* text — a refusal notice, a cancellation report reading the ledger — is controller output, outside $\gamma$'s jurisdiction, and may accompany any halt (a template that *interpolates* model-authored fragments inherits the model's label — the appendix's meet rule — and those bytes are gated like any others); the invariant is that raw model text never reaches a sink ungated, not that failed runs die silent.
*The harness invariants.* These are the invariants that make $\mathcal{H}$ a *harness* and not merely a controlled Markov process with a learned kernel inside: the model sees only $\mathcal{C}$, never full $\mathcal{S}$; its outputs are proposals, not actions; a deterministic capability boundary $\gamma$ gates every side effect; and the *terminal* set $H$ splits into accepting ($H_{\mathrm{ok}}$) and non-accepting ($H\setminus H_{\mathrm{ok}}$ — safe refusals outside $B$, and wrong or bad halts possibly in $B$), while the bad set $B$ is a *separate* unsafe set — possibly absorbing, possibly entered mid-run before any halt — against which $\tau_B$ is measured for reach-avoid. Two notes keep the invariants honest. They are *signature*, not strength: a $\gamma$ that authorizes everything still satisfies the tuple, as a trivial group satisfies the group axioms — the definition admits degenerate harnesses, and fail-closed, provenance isolation, and the certificates below are properties a particular harness *earns*, not gifts of the signature. And the first invariant has a sharper, two-sided form: $\pi$ is the *only* channel from state to model — the confidentiality floor lives at what $\pi$ must never lower (credentials, other principals' data) — exactly as $\gamma$ is the only channel from model output to effect, where the injection bounds live; exfiltration is therefore cut at either chokepoint, never lowered or never emitted (the gate refusing the read whose URL is the payload is the emission-side cut). One chokepoint out of the state, one into the world; a bypass of either is the same bug with the sign flipped.
*Beyond the stationary kernel.* This displayed $T$ is the time-homogeneous, fixed-kernel case; for nonstationary or adversarial environments, replace $Q_E$ with a time-indexed kernel $Q_{E,n}$ — or an admissible family of kernels, or an adversary's policy — over which the robust certificate (the minimax form under *The limit*) quantifies. If that adversary conditions on history rather than only the current $(s, y)$, the history must itself live in $s$ — otherwise the object is a Markov *game* requiring further augmentation, not a Markov chain. And nonstationarity is not the environment's monopoly: a provider retraining or re-serving under a fixed endpoint name is a nonstationary $M_{W,n}$ — the table places model version *in* $s$ precisely so a version bump is a visible state change — and any measured surrogate (the $\delta$ of *The limit*) is calibrated against one kernel and dies with the bump; the dashboard must be keyed to the kernel it measured.
*The inner kernel.* $M_W$ is itself a stopped process, and for a decoder-only transformer it is implemented as
with the layer stack $\Phi_W$ on the residual stream as the (loosely) "manifold" core — formally just the learned high-dimensional residual-stream transformation, with manifold-proper reserved for the frontier. The inner state $z_t=(c_t,b_t,m_t)$ separates the model-visible window $c_t$ (the $\le L$ slice that slides) from the untruncated output buffer $b_t$ (the transcript the readout actually consumes, so truncation never loses it) and the parser/stop state $m_t$ (parser state, a token counter, and a clock, so the cap and timeout are functions of it), updated $m_{t+1}=\mathsf{step}(m_t,v)$, whose stop set $\mathrm{Stop}$ — EOS emitted, max-token cap, timeout, or parse-failure $\bot$ — forces $\tau=\inf\{t:m_t\in\mathrm{Stop}\}$ finite, making $M_W$ a genuine *probability* kernel rather than a sub-probability one completed by a cemetery output. (One honesty note on the clock: a token-count cap is a deterministic function of the run, but a *wall-clock* timeout imports infrastructure noise — server load, batching, congestion — into the kernel's coin; legitimate, a kernel may carry any randomness, but it makes the displayed $M_W$ the model *plus its serving substrate*, and the determinism audit under *How this could be wrong* must hold the clock fixed along with the samples.) The readout is total, $R : \mathcal{Z} \to \mathcal{Y}$ — a parsed tool-call, answer, or transcript, returning the parse-failure $\bot\in\mathcal{Y}$ when parsing fails; crucially $R$ is a *syntactic, verified* readout (parsing and extraction), not a semantic solver, or the $L$-wall below is void — arbitrary computation could hide in $R$ off the $\le L$ window — so $M_W(c, \cdot) = R_{\sharp}\,\mathrm{Law}(z_{\tau})$, the pushforward of the stopped-state law along $R$ (equivalently $M_W(c, A_Y) = \Pr[R(z_{\tau}) \in A_Y \mid z_0 = (c,\varnothing,m_0)]$ for a measurable $A_Y\subseteq\mathcal{Y}$); the no-truncation special case takes $\mathcal{Y}=\mathcal{C}$ with $R(c,b,m)=c$ (the window is the whole transcript), reading $c_{\tau}$ directly. The $\bot$ branch is exactly what $\gamma$ rejects fail-closed. This is a **specialization, not part of the definition**: a harness wrapped around a black-box API is still a harness, and $M_W$ may be any learned kernel. Where the weights are open, the geometry of $\Phi_W$ is where the substrate's continuity lives, and several downstream claims lean on it — but the definition does not.
Two stopped processes, nested: **deterministic control over stochastic dynamics over a learned kernel.** Both loops are hitting-time processes; *some* harnesses additionally read the halt set as a fixpoint or acceptance condition — iterative refinement to self-consistency is the genuine fixpoint case, while EOS, length, and tool-call syntax are not convergence. Neither loop settles because you asked it to. (The clean inner-then-outer nesting assumes tool calls fall *between* model runs; streaming or mid-generation tool calls interleave the two loops and need a finer state machine — the nesting is then an idealization.)
## Reading it
| Symbol | Is |
|---|---|
| $\mathcal{H}$ | the harness — the whole controlled system, *not* the model |
| $s \in \mathcal{S}$ | task-state: IR / dialect stack, tool results, plan, counters, **and every mutable interface variable** (model/tool versions, permissions, retrieved context) — only Markov *after* that augmentation |
| $\mathcal{C},\ \mathcal{Y},\ \mathcal{A},\ \mathcal{E}$ | the **context / readout / action / effect spaces** — model-visible context $\mathcal{C}$, model readout $\mathcal{Y}$ (incl. the parse-failure $\bot$), authorized actions $\mathcal{A}$ (with $\mathcal{A}_{\bot} = \mathcal{A}\cup\{\bot\}$), and tool/environment effects $\mathcal{E}$ |
| $M_W(c, dy)$ | the **model-run kernel** (inner solver) — a stopped autoregressive process; $\Phi_W$ is the residual-stream ("manifold") core in the transformer case |
| $Q_E(s, a, de)$ | the **environment/tool kernel** on the authorized action $a\in\mathcal{A}_{\bot}$ (with $Q_E(s,\bot,\cdot)=\delta_{e_0}$, the no-op $e_0$) — tool effects, API responses, the world (possibly adversarial) |
| $\gamma,\ \rho$ | the deterministic **authorization gate** $\gamma:\mathcal{S}\times\mathcal{Y}\to\mathcal{A}_{\bot}$ (untrusted proposal → authorized action or $\bot$) and the **fail-closed verify-and-fold-back** $\rho:\mathcal{S}\times\mathcal{Y}\times\mathcal{A}_{\bot}\times\mathcal{E}\to\mathcal{S}$ |
| $H,\ \tau_H$ | the **halt set** (absorbing) and the outer **halting time** — a hitting-time process, not a single pass |
| $H_{\mathrm{ok}},\ B$ | the **accepting halts** $H_{\mathrm{ok}}\subseteq H$ (correct, successful terminals) and the **bad set** $B$ — unsafe states for reach-avoid ($B\cap H_{\mathrm{ok}}=\varnothing$), *separate* from $H$ and possibly entered mid-run before any halt |
The structural fact that earns the word *controller*: $\pi$, $\gamma$, $\rho$, and the halt test are **deterministic** (and the readout $R$ too, where the transformer specialization is in play), so $\mathcal{H}$ injects no randomness of its own. Every coin is inherited from $M_W$ and $Q_E$. This split — deterministic code around a stochastic oracle — wears two names. In control-theory terms it is **controller vs. plant**: the controller is those deterministic maps; the **plant** is the learned kernel $M_W$, *plant* in its exact sense — the element with its own dynamics you steer but do not author. In engineering terms it is **shell vs. plant**: the **shell** is the entire deterministic outer harness — the control logic *plus* the external memory and tools it administers (the files, databases, vector stores below) — of which the controller is just the control-logic slice. So *shell : plant :: the part you write : the part you don't*; $M_W$ is the only thing on the right, while the environment $Q_E$ is the world the actions meet — a disturbance into the loop, not the plant. (A reader from reinforcement learning or classical control will make the opposite assignment — environment as plant, policy as controller; the inversion is deliberate: in harness engineering the element you are trying to make behave is the model, and the world is what pushes back on the attempt.) This determinism is *conditional* — on versioned code, configuration, model endpoint, and tool interfaces, and on *single-run sequencing*: concurrent runs sharing authorization state re-open a gap the per-run object cannot see (taken up under *Gate placement* in the appendix); any retry, timeout, race, or randomized routing that escapes that conditioning must be modeled explicitly as part of $Q_E$ or the controller, not waved away. The displayed $M_W(c)$ likewise freezes endpoint, version, and sampler; a routing or config change is a state-indexed kernel $M_{\kappa(s)}$ or folds into $K_C$ — the kernel must not silently depend on config the table places in $s$. More generally, control may itself be stochastic — a controller kernel $K_C(s, dc)$ over routing, sampled retries, ensemble votes, learned routers — of which the deterministic $\pi, \gamma, \rho, H$ are the Dirac special case. That case is the one worth wanting: it localizes every coin to $M_W$ and $Q_E$ and keeps the controller/plant split clean. Where control is genuinely stochastic the split does not break, it widens — fold $K_C$ into the kernel and the certificate quantifies over its randomness too. But the guarantees do not soften uniformly, and the component-to-guarantee map is worth stating because it says exactly what may be learned without loss. A learned $\pi$ — retrieval, reranking, summarization inside the lowering — costs only *semantic adequacy*, under one factorization: $\pi$ splits into a deterministic **never-lower filter** — the redaction that keeps credentials and other principals' data out of $\mathcal{C}$ — composed with learned selection, and only the selection may soften, or the confidentiality floor of the invariants note becomes a probability. With the filter Dirac, no-unauthorized-effect is $\gamma$'s property alone, and the reach-avoid certificate survives too, so long as the provenance partition of *The limit* holds. A learned $\gamma$ or $\rho$ costs the thing itself — authorization and ledger integrity are exactly the properties that must stay Dirac, or "no unauthorized effect" and "the ledger is what happened" become probabilities. So the minimal deterministic core is $\{\gamma, \rho, H\}$ plus $\pi$'s never-lower filter: the rest of $\pi$ may soften into a kernel and the harness bends without breaking — fortunate, because every deployed $\pi$ already has learned kernels inside it.
## Why this shape
$$f(x) \;\longrightarrow\; x = f(x;\,W) \;\longrightarrow\; f(x)$$
Classical software, inverted into latent geometry, then re-wrapped in classical software. The harness **re-imposes the determinism the model dissolved**: $\pi, \gamma, \rho$, and the halt test ($H$) are ordinary designed code — a controller — whose primitive operand happens to be a stochastic oracle. That closure is why a compiler is the right mental model (staged deterministic software ports cleanly) and exactly why the analogy breaks (a compiler's primitive operation was never a coin). **The harness is the half you can reason about classically, sitting on top of the half you cannot.**
## The limit, stated honestly
**Raw halting is cheap; correct halting is not.** A **certificate** is a *witness*: a checkable object — here a Lyapunov/drift function $V \ge 0$ — that *provably* satisfies a condition entailing the guarantee, through a standard supermartingale / optional-stopping theorem (the target picks the condition: drift toward $H$ for halting, a barrier for safety, reach-avoid for success). It is not the property, only an object cheap to check and hard to produce. One word then carries two senses, and the seam between them is what this section is about: the **proven** certificate, a $V$ whose bound actually holds; and the **measured** surrogate you fall back on when the architecture exhibits none — a candidate $\hat V$ with a sampled slack $\delta$, a *calibrated risk metric, not a certificate* until that bound is proven (or held to a high-confidence worst case). The gap between the two is the whole honest-limit argument. A deterministic budget — augment $s$ with a counter $k$ decremented each outer step, halting at $k=0$ — makes $V(s)=k$ a trivial Lyapunov certificate for *halting*, so the architecture does not lack a halting guarantee by construction. What it lacks for free is a certificate of *correct, safe, successful* halting under the learned dynamics. The un-budgeted halting object is still worth stating, since it shows where even the easy guarantee comes from: a certificate would be *sufficient* for almost-sure halting with bounded expected runtime — a $V \ge 0$ with
$$\mathbb{E}[\,V(s_{n+1}) \mid s_n\,] \le V(s_n) - \varepsilon \quad\text{off the halt set}$$
bounds $\mathbb{E}[\tau_H] \le V(s_0)/\varepsilon$ under the usual integrability and optional-stopping conditions. Nothing in the harness hands you such a $V$ the way a compiler's structure does: a specific compiler analysis gets its $V$ for free where a finite-height lattice *is* a well-founded descent — termination by construction *for that analysis*, not for a whole compiler — and the harness has no analogous built-in descent for its model/environment loop.
But the relevant $V$ is not *absent* — and this is the subtlety the blunt phrasing erased. The minimal certificate exists and is **forced**: it is the expected halting time itself,
finite wherever $H$ is reached in finite expected time — the domain $\{s : \mathbb{E}_s[\tau_H] < \infty\}$ — though note this $V^\star$ certifies *halting* (reaching the terminal set $H$ at all), not *correct* halting; the stronger object, the expected time to an accepting $H_{\mathrm{ok}} \subseteq H$, is $V^\star_{\mathrm{ok}}$, taken up at the second wall below. So the honest claim splits in two: the architecture provides no certificate *for free*, and the one that exists is — **conjecturally, not as a theorem** — a functional of $W$ and the environment that does not compress below model scale. The conjecture needs scoping, because the per-step drift splits by coordinate (made precise below) and the shell's contribution is an exact, designed descent of low description complexity *by construction* — so whatever is incompressible is not the shell's part but the **plant's**, the contribution $M_W$ supplies. And even there it is conjecture with a live counter-possibility, not foregone hardness: $V^\star$ is a *coarse* functional — one scalar, an expected hitting time, not the full output law — and coarse functionals of complicated kernels are sometimes cheap (absorbing chains with sparse transition structure have tractable expected hitting times over enormous state spaces). So the honest form is conditional: *if* the plant's contribution to the drift admits no certificate of description length materially below $|W|$, then ours is as hard as the dynamics — but that antecedent is the unproven part, and the flat phrasing of an earlier draft ("the dynamics it certifies *are* the weights") overstated it by treating a coarse hitting-time functional as if it carried the whole distribution. The compiler's certificate is structurally trivial; ours is *plausibly* as hard as the plant dynamics, though whether useful compressed certificates exist — for the coarse hitting-time functional, or for structured sub-tasks — is open. This is the quantitative form of *you can borrow how LLVM is built — not, in general, why it is correct.*
So you never compute $V^\star$. You pick a candidate $\hat V$ and **estimate its drift slack**
The status of $\delta$ has to be stated carefully, because it is easy to oversell. If you can establish a *high-confidence upper bound* on the true worst-case slack and it is $\le 0$, optional stopping hands you a real, conservative certificate, $\mathbb{E}[\tau_H] \le \hat V(s_0)/\varepsilon$. But an *empirical* $\delta$ estimated from sampled states is **not** a certificate: a measured $\delta > 0$ may mean the candidate $\hat V$ is poor, the sampled distribution missed rare failures, the supremum was never attained in-sample, the process is non-stationary, or the state abstraction is not Markov. So $\delta$ is **the number on the dashboard** — a *calibrated risk metric*, the evaluable surrogate for a guarantee the geometry will not give you, and a genuine bound only once it is statistically controlled against rare-event and adversarial tests. A weaker result is still useful: a true bound $\delta \le \bar\delta < \varepsilon$ (rather than $\le 0$) leaves descent intact with effective slack $\varepsilon - \bar\delta$ and $\mathbb{E}_s[\tau_H] \le \hat V(s)/(\varepsilon - \bar\delta)$. And the empirical quantity is distributional, not a supremum — write $\delta_{\nu}$ for drift averaged over a sampled $\nu$, reserving $\delta_{\sup}$ for the worst-case bound; only $\delta_{\sup}$ certifies. Its empirical noise floor and residual risk are driven by the measure $\mu(D)$ of the divergent region $D=\{s:\mathbb{E}_s[\tau_H]=\infty\}$ (states from which $H$ is not reached in finite expected time, under the reference/sampling measure $\mu$), the coverage of the sampled state distribution, and the hitting-time variance $\mathrm{Var}[\tau_H]$ — properties of the trained weights, the environment, and the evaluation distribution, knowable only a posteriori.
> For an agent *meant* to run forever — a coordinator, a daemon — halting is the wrong target, and $V^\star = \infty$ is the spec, not a pathology. The same drift theory then certifies **recurrence to a ready-state** instead of absorption to a halt-set. The object changes; the missing certificate does not. Safety changes shape too: it is no longer the one-shot $\Pr_s(\tau_B=\infty)$ but a *per-cycle* hazard that compounds — if each ready-state-to-ready-state cycle touches $B$ with probability $q$, survival over $N$ cycles is $\approx (1-q)^N$, so a reassuring per-cycle $0.9999$ is $\approx 0.37$ over ten thousand cycles. The reach-avoid certificate for a daemon is therefore a bound on $q$ against the intended horizon — the safety twin of the regenerative expected time that replaces $V^\star_{\mathrm{ok}}$ for restarting specs.
And the consolation rests in part on an assumption the world violates — though less of it than it first seems. The supermartingale *bound* itself survives a nonstationary kernel, provided the conditional drift holds uniformly at every step; what genuinely needs a **time-homogeneous kernel** is $V^\star$ as a fixed function, the resolvent / fundamental-matrix identities, and the sampled-$\delta$ calibration (which assumes the very kernel it was measured on). But the environment $E$ is *part of* $T$, and the world is not stationary — worse, it can be **adversarial**, an attacker choosing the tool-output *policy* — a kernel over what tools return, not the realized draw — so as to break your descent. The drift condition then stops being a fixpoint question and becomes a **minimax** one,
a descent that must hold in expectation over the model's own output $y$ *and* even when the adversary picks the worst admissible environment policy $\alpha(s,y)$ from the class $\Pi$ of policies the environment genuinely permits — every $\alpha\in\Pi$ must still respect rejection, $\gamma(s,y)=\bot \Rightarrow Q_E^{\alpha}(s,\bot,\cdot)=\delta_{e_0}$, or the adversary resurrects side effects the gate refused. Well-posedness is a frontier caveat of its own: for $\sup_{\alpha\in\Pi}$ to be *attained* rather than merely defined, $\Pi$ needs structure — measurability of $\alpha\mapsto Q_E^{\alpha}$, compactness of the per-state admissible set, or a measurable-selection theorem furnishing a worst-case $\alpha$ — and "respects rejection" is a *constraint* on $\Pi$, not that existence argument; on a general state space the sup may have no maximizer, in which case the certificate quantifies over a maximizing sequence rather than a single adversary. A $V$ that certifies halting against a benign world is defeated by an adversarial one, and the measured $\delta$ bounds only the $Q_E$ you *sampled*, never the policy an attacker will choose.
**This is the formal home of prompt injection** — not "the model did something bad," but the environment optimized to bend your dynamics. And the target is not merely non-halting: injection steers toward a **bad set** $B$ — wrong acceptance, data exfiltration, unauthorized tool use, privilege escalation, irreversible side effects — so security is a **reach-avoid** problem, not a liveness one.
Here two reliability objects must be kept apart, because under absorbing refusal every naive intermediate collapses into one of them:
$$p_{\mathrm{succ}}(s) = \Pr_s\big(\tau_{H_{\mathrm{ok}}} < \tau_F\big), \quad F = B \cup (H \setminus H_{\mathrm{ok}}), \qquad\qquad p_{\mathrm{safe}}(s) = \Pr_s\big(\tau_B = \infty\big).$$
**Success** is reaching a correct halt before *any* failure — a safe refusal counts *against* it. **Safety** is never entering the bad set at all — a safe refusal *satisfies* it. These genuinely differ on any run that avoids $B$ without reaching $H_{\mathrm{ok}}$ ($p_{\mathrm{succ}}$ scores $0$, $p_{\mathrm{safe}}$ scores $1$): safe refusals, and — absent almost-sure absorption into $H\cup B$ — safe non-halting or endless safe retry. The tempting middle form $\Pr_s(\tau_{H_{\mathrm{ok}}} < \tau_B)$ is *not* a third object, by a two-line case analysis: for it to differ from $p_{\mathrm{succ}}$, a run would need $\tau_F < \tau_{H_{\mathrm{ok}}} < \tau_B$ — a non-accepting terminal hit strictly before success, then success anyway — which forces *exiting* $H \setminus H_{\mathrm{ok}}$, impossible while $H$ is absorbing. Note what does **not** re-separate them: within-run fail-closed retries (the non-terminal fail-closed of the definition) never touch $F$ at all — the rejected proposal lands in a safe *non-terminal* state — so a refuse-retry-succeed run scores $1$ on both forms, and the coincidence survives any amount of retrying. The middle form becomes a genuine third object only when the two hitting times can genuinely part ways: under **restarting specs**, where an owner re-launches out of a refusal terminal and the absorbency of $H \setminus H_{\mathrm{ok}}$ is deliberately dropped (the regenerative reading the daemon note above already contemplates) — no bookkeeping needed, since hitting times record *visits*, not occupancy, so the relaunched run's $\tau_F$ is already finite — or under a failure set that counts refusal *events* accumulated in $s$, $F' = B \cup (H \setminus H_{\mathrm{ok}}) \cup \{\mathsf{refusals} \ge 1\}$, which separates the forms even within a single run. In the restart case a run may halt refused, restart, and still reach $H_{\mathrm{ok}}$ before $B$: the middle form credits it; $p_{\mathrm{succ}}$, measured against the refusal it passed through, does not. Safety is certified by a barrier / avoidance certificate for $B$; success needs that plus the reach part — a hitting-time drift toward $H_{\mathrm{ok}}$. Fail-closed control is the disturbance-rejection margin for both, but split by reversibility: the gate $\gamma$ caps how far an adversarial world reaches into *side effects* and widens the gap to $B$ (it is the margin for the irreversible part), while $\rho$ validates the response and folds back, rejecting bad state after the action has run — which cannot undo an authorized side effect. In this language, security is robustness of the reach-avoid certificate.
And injection is not confined to the post-model kernel $Q_E$: poisoned retrieval, prompt-injected pages, and malicious tool metadata enter through $\pi$'s *inputs*, before generation — so the adversary lives wherever untrusted content enters the state/context-construction pipeline, which is why input provenance and the gate $\gamma$ both matter, not post-hoc verification alone. And provenance is a *precondition* of the certificate, not just an entry point to police: partition $s$ into a **control-determining** part — plan, intent, what is authorized next, the coordinates $\pi$ lowers and $\gamma$ checks — and a **data** part — tool values, retrieved text, the bytes of $e$. Reach-avoid presupposes untrusted effects touch only the latter; let $\rho$ fold attacker-controlled $e$ into the control part and the structural-intent check validates against a plan the adversary already bent, collapsing $\gamma$ to the strength of $\rho$'s validation. So the claim is conditional — reach-avoid *given* control flow provenance-isolated from untrusted data, the isolation that makes provable security possible (the content of CaMeL's control/data-flow separation, untrusted data filling typed values but never the program), a structural property the harness supplies and $\rho$ cannot recover after the fact. The partition then forces a question the isolation rule alone cannot answer: *something* must be permitted to write the control-determining part mid-run — or no plan could be steered, no approval granted, no scope widened — and naming that something is part of the object. It is the **trusted principal**: the owner of the run. An approval request is an ordinary authorized action through $\gamma$ into $Q_E$ — ask-the-owner is a tool call to the one counterparty you trust — and its response is the *single* class of $e$ that $\rho$ may fold into control coordinates; every other $e$ folds into data. This is not an exception eroding the partition but the partition completed: a provenance *lattice* with exactly one writer at the top, which is what trusted means — and the appendix's gate-placement entry derives the matching rule for *learned* verdicts, which may never stand in this writer's stead. One distinction keeps the lattice from outlawing the loop it governs. Control-determining is not one rank but two: **authority** — grants, scopes, budgets, what the principal has permitted — which only the top writer widens; and the **plan**, which the model rewrites at every fold of $y$, because replanning *is* the harness. The plan is a *middle* rank: written through the gated fold of the model's own output — the channel the minimax descent above already prices — never directly by an effect, and never a source of widened authority. The rank is also the field's live design axis: pin plan-writes to the top-derived rank — the plan fixed from the trusted query before any untrusted read, which is CaMeL's move — and provable security follows exactly there; let the middle rank replan interactively and you pay the adversarial price the certificate quantifies. A corollary with teeth: a dedicated planning component is rank-neutral — its writes land in the same middle rank as the model replanning inline — so it changes no guarantee and lives or dies on measured capability alone; in general, sub-components that only write middle-rank state are priced by evals, not by the certificate, which prices only rank crossings, gates, and $\Pi$. (For $B$ to capture irreversible side effects rather than only states, the side-effect ledger must itself live in $\mathcal{S}$, and the response $e$ must be an *effect record* carrying the ledger outcome — not just API bytes — since only $\rho$ writes external effects into $s$.)
There is a **second wall, orthogonal to the first.** It binds not the full harness state $\mathcal{S}$ but the **model-visible working memory** $\mathcal{C} = \mathcal{V}^{\le L}$ — bounded by the context length $L$. That bound is *not* the incompressibility of $V^\star$ (a fact about the parameters $W$ — the **dictionary**, fixed at training); it is a fact about the inner kernel's **working memory** (the $L\times d$ residual stream — the **desk**). $\mathcal{S}$ itself may be far richer — files, databases, vector stores, durable memory, queues — but that is *external* memory the shell supplies, and the distinction is the point: every external read still passes *through* the $\le L$ window to touch computation, so external stores extend addressable storage without extending the per-pass resident set. The shell can page; the plant cannot grow its desk. (What follows is heuristic, not definition-level: the complexity claims turn on depth, precision, and architecture, and belong with the frontier, not the core.) The tape picture comes from the autoregressive structure alone and needs no complexity theorem: each step reads a bounded window and writes one token, so **the context window is the tape, the autoregressive loop is the read/write head**, and — in the variable-$L$, fixed-precision idealization — the model-mediated inner computation behaves like a linear-bounded automaton, its reachable fixpoints capped by space-$O(L)$ computability (chain-of-thought is register-spilling onto that tape). Separately, and more weakly, there is a *per-pass* expressivity bound: under the standard fixed-depth, log-precision theoretical model a single forward pass is in constant-depth $\mathsf{TC}^0$ — *suggestive* for deployed models, not literal (real models use fixed-point precision and depth that grows with scale, and log-depth variants escape parts of it). These are different resources — the first bounds the *space* the loop addresses, the second the *depth* of one step — and only the space bound carries the $L$-wall; chaining them (one pass buys bounded depth, *therefore* the loop is space-$O(L)$) would be a non-sequitur, since per-step depth says nothing about the length of the tape the loop runs on. This is a *second* obstruction beside divergence, and it concerns *success*, not raw halting. Split the terminal set: let $H$ be any halt state (including fail-closed refusal) and $H_{\mathrm{ok}} \subseteq H$ the successful, accepting halts, with $V^\star_{\mathrm{ok}}(s) = \mathbb{E}[\tau_{H_{\mathrm{ok}}} \mid s_0 = s]$ taken on the process where $H \setminus H_{\mathrm{ok}}$ — halting wrong, refusing, failing closed — is *absorbing failure*, so a run that fails closed before acceptance has infinite accepting hitting time unless the spec explicitly restarts it — hence unconditional $V^\star_{\mathrm{ok}}$ is infinite whenever pre-acceptance failure has positive probability, which is why the workable reliability object is the success probability $p_{\mathrm{succ}}$ (above) or, for restarting specs, the regenerative expected time. Then $U_{\mathcal{H}}(L)$ — harness-relative, since the shell's decompositions and verified tools determine what can be paged or outsourced — is the set of tasks whose **irreducible per-step model-mediated working set** exceeds $L$ — not tasks whose *data* exceeds $L$ (those the shell can page), and not work that can be **discharged to a verified external tool** (a solver, interpreter, or compiler computes off-context). For a task in $U_{\mathcal{H}}(L)$ the raw chain may still hit $H$ — by failing closed, refusing, or returning a wrong answer — so $V^\star = \mathbb{E}[\tau_H \mid s]$ stays perfectly well-defined; what blows up is $V^\star_{\mathrm{ok}}$, the expected time to a *correct* halt, which is infinite under a formal success predicate, or undefined if no such predicate has been specified. The honest statement is about the finite-success domain, and it is *schematic* — a shape written in set notation, not a theorem, since $\mathrm{reachable}_{\mathcal{H}}(L)$ is exactly as informal as the working-set notion behind $U_{\mathcal{H}}(L)$: $\mathrm{dom}_{<\infty}(V^\star_{\mathrm{ok}}) \subseteq \mathrm{reachable}_{\mathcal{H}}(L) \setminus D$ — both the reachable set and the divergent set $D$ relative to $\mathcal{H}$. The two walls **trade** — *directionally, not as a literal exchange rate*: parametric memory $|W|$ and working memory $L$ press on the same budget along the pretraining-vs-inference-scaling axis, with no clean unit-for-unit substitution of one for the other. And the bound is inherent to *finite working memory*, not attention specifically: state-space models embody it differently (a fixed-size recurrent state rather than an $L$-window), and real attention's usable tape is shorter than $L$ (lost-in-the-middle).
## Where it cashes out
This is not ornament; the decomposition is load-bearing in the design.
- **$\pi$ is a progressively-lowered dialect stack** — raw input → intent → plan → tool-call → the neutral wire IR — each level a deterministic pass with its own verifier — *pass* and *verifier* meaning the shell's transformation and checking: the **content** entering at the plan level is plant-authored, middle-rank state (the two-rank note of *The limit*), which is exactly why that level carries a verifier at all. The per-step drift $r(s)=\mathbb{E}[\hat V(s_{n+1})\mid s]-\hat V(s)$ splits by coordinate, $r = r_{\text{shell}} + r_{\text{plant}} + r_{\text{env}}$ — presuming an additively separable $\hat V$, or a declared scheme attributing each step's drift to shell, plant, and environment coordinates: the shell term is an *exact, designed* descent — but per lowering pass, not per outer step: each pass strictly narrows the admissible-meaning set, a well-founded descent we build by hand, while the outer loop *revisits* — retry, replan, rewind are planned ascents of any reasonable $\hat V$, which the run-level certificate must absorb (a retry budget inside $\hat V$ is the standard device), so the shell's descent is well-founded in the nested, lexicographic sense rather than monotone along the run; the plant term ($M_W$) is the irreducible residue, and the environment term ($Q_E$) is the one an adversary controls — the very quantity the minimax descent must bound, which the old two-way split folded out of sight. **Syntactic soundness is free; semantic adequacy is not.** Relative to a formal schema and a correct validator, schemas, types, and boundary checks go into the shell at zero probabilistic cost; whether the lowered task still *means* what the user intended stays empirical, because natural language supplies no source-language standard to check against.
- **$\rho$ is fail-closed verification** — validate at every boundary, never let malformed state flow downstream. The discipline transfers from compilers in *form*; the *teeth* do not, because a harness has no source-language standard — natural language is, in effect, all undefined behavior — there is no complete formal source-language semantics to check against. And $\rho$ must be *deterministic*: if verification is itself an LLM judge, that is another learned kernel call — it belongs in $M_W$, not in $\rho$. Where $\rho$ *repairs* rather than rejects — canonicalizing malformed input into valid shape — remember that repair is an authorization decision in disguise: each repair rule converts a reject into an accept on bytes the adversary chose, so it must be deterministic, meaning-narrowing, and its output re-validated as if it had arrived that way, or the repair pass is a bypass of the very boundary it serves.
- **$\delta$, $\mu(D)$, $\mathrm{Var}[\tau_H]$ are what you measure** — not derive. You instrument the certificate precisely because the architecture does not hand it to you — you estimate it unless it is separately certified. And the meter is attack surface: if $\hat V$ is itself computed by a learned judge — a model scoring "progress" — the instrument is a kernel draw with the plant's own adversarial exposure, and an environment optimized to bend your dynamics will bend your *measurement* of them first; an injected page persuading the judge that work is advancing is precisely a divergence hidden from the dashboard built to catch it. The rule that put the LLM judge in $M_W$, not $\rho$, applies to instrumentation too: a learned $\hat V$ is part of the measured system, never a neutral meter.
## How this could be wrong
It is a hypothesis; here is what would falsify it. If the controller cannot in practice be kept deterministic — if real reliability demands stochastic control the plant can't absorb — the clean *deterministic* split is a fiction (the broader $K_C$ kernel model still holds, but loses its payoff: localizing every coin to the plant). If the drift slack $\delta$ turns out *not* to track real-world failure, the whole "measure the certificate you can't prove" program is empty. And if harnesses are simply better described some other way — not as nested stopped chains at all — then this is a pretty equation that merely happens to fit, an elegance we would be right to distrust.
First, handles — the load-bearing claims numbered, so the tests have addresses. **C1**: the harness is faithfully modeled as nested stopped Markov processes — the tuple, the outer $T$, the inner $M_W$. **C2**: the controller injects no randomness — every coin localizes to $M_W$ and $Q_E$. **C3**: fail-closed is a *gate* property — no effect crosses unvalidated, and rejection is a true no-op. **C4**: no certificate of correct halting comes free, and the measured slack $\delta$ is a calibrated risk metric, never a certificate. **C5** (conjecture): the minimal certificate $V^\star$ admits no representation materially below model scale. **C6**: two orthogonal walls — divergence ($\mu(D)$) and the $L$-bounded per-pass working set. **C7**: security is reach-avoid, certifiable only conditional on provenance isolation with a single trusted writer. **C8** (figure): certificate and interlingua are one object — already demoted by its own section, and exempt below accordingly.
Each claim is operational, not merely rhetorical:
- **State-ablation (C1 — the Markov claim).** Drop a variable from $s$ and check whether next-step transition statistics move. If they do, the abstraction was not Markov, and $s$ must be augmented until it is. (Passing is necessary, not sufficient — the test can falsify Markovity, not establish it.) The same probe pointed at $\pi$ tests lowering *sufficiency*: drop a coordinate from $c$ rather than $s$ and watch task success rather than transition statistics — context compaction lives or dies by exactly this.
- **Controller-determinism audit (C2).** Re-run with model samples and tool outputs *held fixed*. Any residual variance is randomness the harness itself injected — clock reads are the classic leak (timestamps folded into $s$, wall-clock timeouts, cache expiries) — and must be folded into $Q_E$ or the controller, or the determinism claim is false.
- **Drift calibration (C4).** Test whether $\hat V$-drift actually predicts failure, retry count, latency, or non-halting. One uncorrelated candidate kills that candidate, not the program; the program is empty only if candidates from the natural families — plan depth, open-obligation counts, budget burn, judge scores — *systematically* fail to track failure.
- **Adversarial-environment test (C7).** Replace sampled $E$ with worst-case tool outputs, prompt-injected documents, poisoned tool metadata, malformed responses. The minimax descent must survive these, not merely the benign draw.
- **Boundary-control ablation (C3, C7).** Compare prompt-only defenses against deterministic tool-call validation, capability checks, sandboxing, and fail-closed rejection at the gate $\gamma$. The hypothesis predicts the latter class dominates; if prompt-only defenses match it, the controller/plant security story is wrong.
- **Readout-typing check (C1, C3).** Verify that $M_W$'s codomain is exactly what $\gamma$ consumes — especially under window truncation, where the final context need not hold the full transcript, so the output buffer and the gate's input must still agree.
- **Certificate-compression search (C5).** The conjecture falsifies constructively: exhibit a $\hat V$ of description length far below $|W|$ whose worst-case slack is provably $\le 0$ over a nontrivial task domain. The text concedes the live counter-possibility — coarse hitting-time functionals of complicated kernels are sometimes cheap — so C5 stands only until someone cashes it.
- **Working-set probe (C6).** Fix the shell and scale a task family's irreducible per-step working set past $L$, on tasks the shell can neither page nor discharge to a verified tool — anchoring "irreducible" in families with proven streaming or communication-complexity lower bounds, so the floor is someone else's theorem and a solved family cannot retreat to reducible-after-all. C6 predicts success collapses at the wall rather than degrading smoothly; a family solved reliably past it, without new shell decompositions, falsifies the second obstruction.
## Where this points (the frontier — least falsifiable, so flagged)
If $V^\star$ is incompressible only in *token* coordinates, the right change of coordinates might compress it — and that change of coordinates is a representation of meaning itself. Cost-to-go and representation co-determine each other: where the Koopman operator is diagonalizable — a point-spectrum idealization, since mixing dynamics carry continuous spectrum and admit no eigenbasis — the eigenbasis that linearizes the dynamics is also the one in which the certificate decomposes, and even then only for a $V$ in the span of those eigenfunctions; in reinforcement learning the discounted successor representation (Dayan 1993) is the resolvent $(I-\beta P)^{-1}$ — discount $\beta$, not the gate $\gamma$ — with $V$ a *linear readout* of it — and in the undiscounted, absorbing case that actually matches a stopped harness the same role is played, in the finite setting — and countable settings where the Neumann series converges — by the **fundamental matrix** $N = \sum_{n \ge 0} Q_{\mathrm{tr}}^{\,n}$ (written $(I - Q_{\mathrm{tr}})^{-1}$ when the inverse exists), where $Q_{\mathrm{tr}}$ is the sub-stochastic kernel restricted to $H^c$ (transitions before absorption at $H$) and the row sums $N\mathbf{1}$ *are* $V^\star$ on the finite-mean hitting domain; on general state spaces the same series is read as the potential (Green) operator $G$, with $G\mathbf{1} = V^\star$ wherever it converges. Each of these is a clean identity only for a fixed, time-homogeneous kernel — under a nonstationary $Q_{E,n}$ the resolvent and fundamental matrix dissolve into a time-ordered product, and under an *adaptive* adversary into a controlled / game-value operator, so what is identity in the stationary regime is analogy beyond it.
With that caveat, **the interlingua and the certificate are one object seen twice** — and the reason neither can be written in closed form is the same "all undefined behavior": no canonical lowering of meaning, hence no finite header-file for either. The only representation of both is $W$ — a band-limited, lossy compression of a scale-free meaning-space, sharp where the record is thick and blurred where it thinned. That a finite object renders an infinite one *lossily but honestly* — declaring its resolution, and where it is unsure — is not a lie; it is the most an $f(\cdot\,;W)$ can do. **The search for $V$ and the search for the interlingua are not two programs. They are one** — and the day either is written in closed form, so is the other, or we will have proven why neither can be. Read this as *figure*, not a lurking theorem: the only precise version would need the Koopman eigenbasis to fall on the very coordinates that lower meaning, and the mixing-spectrum caveat above already concedes that eigenbasis does not exist — which guts it. It is the least-defensible claim in this document, and it should announce that rather than imply a rigor it has not got.
## The loop
*This section opens an object rather than settling it; it is a sketch of where the same construction goes one level out, flagged as unfinished.*
Everything above governs a run: a principal poses a task, the harness drives it to a halt, the principal reads the result. Step back once and there is a further loop that this document has treated as exogenous — the process that *decides what the next task is*, dispatches it, checks the result, remembers, and fires again. In one recent framing this is the difference between the harness (the scaffold the run executes in) and **the loop** (the recurring trigger–act–verify–stop cycle that keeps launching runs); the practitioner literature that named the loop treats it as a layer *above* the harness. The claim worth making here is that this is not a new kind of object at all — **it is the harness construction applied one level out**, with a run where a step used to be.
Make the correspondence exact and the reuse is total. The outer loop has its own state $s^{\uparrow}$ (a backlog, a set of open goals, what has been tried and what passed), its own lowering $\pi^{\uparrow}$ (which goal to pursue now, and with what context), its own plant — but the outer plant's *proposals* are whole runs, so the inner harness plays the role of the outer environment kernel: dispatching a task is one draw of $Q_E^{\uparrow}$, and the run's terminal ledger is the effect record folded back by $\rho^{\uparrow}$. This is precisely the **composition** correspondence of the appendix read at the top level — a child harness is a $Q_E$ component — which is why the loop needs no primitive the tree did not already have. The daemon entry is the special case where the outer loop is a single long-lived agent recurring to a ready set; the general loop is a daemon whose excursions are themselves full harness runs, which is to say the outer-outer harness *is* a daemon over runs, and inherits that entry's whole ledger: renewal-reward rates, the accumulation that breaks regeneration, hygiene as renewal structure, authority frozen between owner contacts.
What the level shift buys is that the invariants reappear with sharper teeth, because the outer plant is now *itself an agent*, not a token-sampler. The gate is still the load-bearing object: **who authorizes a run?** A loop that launches tasks against production is choosing actions with effects, and "the loop decided to refactor the auth module" is an authorized action or an ungated one — the trusted-principal lattice does not dissolve at the outer level, it recurses, and the autonomy corollary bites hardest here, since a loop whose principal has stepped away is exactly the "replace yourself as the prompter" regime, running on frozen authority against a moving world. The two walls recur too: the outer working set is the backlog the loop can actually hold coherent at once (context, one level up), and the outer certificate is the same absent object — no free proof that an unattended loop halts, converges, or stays out of $B$ over a long horizon, only the measured drift of *its* progress meter, carrying the same warning that a learned outer meter is attack surface. And the degenerate case is instructive in the document's own terms: the brute "same prompt in a while-loop until the spec passes" that the practitioner literature cites as the origin pattern is the outer harness with $\pi^{\uparrow}$ constant, $\gamma^{\uparrow}$ trivial, and verification outsourced to whatever the tests happen to check — the trivial-group harness of the signature-vs-strength note, one level up. It satisfies the outer signature and earns almost none of the outer guarantees, which is exactly why it works until it doesn't.
What this section does *not* yet do: give the outer objects the same treatment the inner ones got — the precise outer analogue of fail-closed when the "action" is a whole run with partial effects, the right reach-avoid formulation when the bad set is a property of a *trajectory of runs* rather than one run, the outer verifier's own soundness, and whether the recursion terminates upward or is genuinely open (loops that launch loops). Those are the next rounds. The point of opening it now is only the structural claim: **the layers the practitioner stack separates — words, context, harness, loop — are, formally, one object at four scales**, and the guarantees this document is about live in the closure at every scale, never in any single layer alone.
---
*The formula is the architecture; the corollary is why the architecture is hard. Both on the page — nothing hidden behind a tidy composition.*
## Grounding
Borrowed theorems are real; the framings are not — keep them separate. Some framings are nonetheless *corroborated* — independently reached from another field — a third grade, weaker than proof and noted last.
**Proven (citable).** Foster–Lyapunov drift ⇒ positive recurrence + $\mathbb{E}[\tau]\le V(s_0)/\varepsilon$ (Foster 1953; Meyn & Tweedie, *Markov Chains and Stochastic Stability*, 1993) — positive recurrence needs the usual irreducibility/petite-set hypotheses, while the absorbing-halt case used here needs only the weaker supermartingale optional-stopping hitting-time bound. The minimal $V$ is the expected hitting time, by first-step analysis + optional stopping (Norris, *Markov Chains*, 1997). For an absorbing chain that expected hitting time is the row sum of the fundamental matrix $N=\sum_{n\ge0}Q_{\mathrm{tr}}^{\,n}$ (Kemeny & Snell, *Finite Markov Chains*, 1960), with the general-state analogue the potential (Green) operator (Revuz, *Markov Chains*, 1984). Koopman's linear-operator view of nonlinear dynamics is classical (Koopman 1931), and Lyapunov functions can be assembled from its eigenfunctions when the spectrum is suitable (Mauroy & Mezić, 2016). You certify a candidate $\hat V$ by a *proven* drift inequality rather than by deriving $V^\star$, and estimate it empirically only where a proof is out of reach — the empirical drift checks, it does not certify (neural-Lyapunov: Chang, Roohi & Gao, *Neural Lyapunov Control*, NeurIPS 2019, arXiv:2005.00611). A classical monotone data-flow analysis gets its $V$ for free because a finite-height lattice is a well-founded descent (Kildall, POPL 1973). The gate-a-plant architecture itself is classical: supervisory control theory synthesizes a deterministic supervisor that disables controllable events of a plant it does not author, with the supremal controllable sublanguage as the largest admissible behavior (Ramadge & Wonham, SIAM J. Control and Optimization, 1987) — $\gamma$ is that supervisor, with a learned stochastic plant on general state spaces; the same theory's controllability condition (specifications must be closed under *uncontrollable* events) and its nonblocking requirement are the proven ancestors of gate-early-on-irreversibles and of the always-enabled escape the appendix requires behind any learned veto. Covert-channel discipline — identify the channel, measure its bandwidth in bits, audit what cannot be closed — is the TCSEC lineage (*A Guide to Understanding Covert Channel Analysis of Trusted Systems*, NCSC-TG-030, 1993). The successor representation is Dayan (*Improving Generalization for Temporal Difference Learning: The Successor Representation*, Neural Computation 1993). Dialect-stack architecture: MLIR (Lattner et al., CGO 2021, arXiv:2002.11054); learned pass-ordering: MLGO (Trofin et al., arXiv:2101.04808). Single-pass low-depth expressivity: log-precision transformers are simulable by constant-depth logspace-uniform threshold circuits ($\mathsf{TC}^0$) (Merrill & Sabharwal, *The Parallelism Tradeoff: Limitations of Log-Precision Transformers*, TACL 2023) — fixed/constant precision is a stronger restriction, added autoregressive steps escape it (Merrill & Sabharwal, *The Expressive Power of Transformers with Chain of Thought*, ICLR 2024), and growing precision changes the picture, so the bound is suggestive for deployed models, not literal.
**Asserted (ours — not theorems).** That the harness is best modeled as nested stopped chains; that $V^\star$ is incompressible (no compression theorem); that "no lattice for $f(\cdot\,;W)$" means none is *known*, not that none exists; and everything under *Where this points* and *The loop* — including the Koopman/certificate co-determination, which is well-posed only under the spectral assumptions noted there, and the interlingua/certificate identification; and the design rules read off the objects rather than proven from them — the single-trusted-writer completion of the provenance partition, the narrow-only rule for learned checks and its influence-side twin (verdict payloads to the plant selected, never generated), the composition law of the appendix. These organize the design; they are not results.
**Converged-upon (independently arrived at, from other framings).** The *Asserted* claims above are ours but not ours alone; several are reached independently, from starting points unconnected to this framing — which is the corroboration a definition earns: not a chorus of agreement (the systems below often disagree on method and goal), but that work approaching from capabilities, reinforcement learning, control theory, software architecture, and language-modeling theory each lands on a piece of the same object. That the **deterministic controller, not the model, carries the guarantee** is reached from four directions — capability and information-flow control (CaMeL: Debenedetti et al., *Defeating Prompt Injections by Design*, arXiv:2503.18813, securing the agent even when the underlying model is susceptible); reinforcement learning (shielding: Alshiekh et al., *Safe Reinforcement Learning via Shielding*, AAAI 2018, arXiv:1708.08611 — a deterministic reactive shield filtering a learned policy's actions against a temporal-logic specification); control theory (*Stable Agentic Control*, arXiv:2605.03034, enforcing finite action catalogs at the tool-output interface under a Lyapunov input-to-state-stability certificate against adversarial disturbance); and software architecture (the plan-then-execute / control-flow-integrity line, e.g. Beurer-Kellner et al., *Design Patterns for Securing LLM Agents against Prompt Injections*, arXiv:2506.08837). The **certified-vs-measured split** is reached from the construction side (CaMeL's provable security) and, independently, from the destruction side (guardrail-evasion results — *Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails*, arXiv:2504.11168, the v1 title — later versions retitle it; *No Free Lunch with Guardrails*, arXiv:2504.00441), with verification-oriented work stating it as the motivating gap (*Towards Verifiably Safe Tool Use for LLM Agents*, arXiv:2601.08012; VeriGuard, arXiv:2510.05156): a learned safeguard raises the odds of detection but cannot guarantee safety against a persistent attacker. The **inner readout as a composition of Markov kernels** is independently formalized in language-modeling theory — the autoregressive step as kernel composition in the category $\mathsf{Stoch}$ (*A Markov Categorical Framework for Language Modeling*, arXiv:2507.19247), and the broader "LLMs as Markov chains" line — though that work models the inner kernel alone and never closes it into an agentic loop, which is exactly the seam this definition adds. That **provenance shrinks the admissible adversary** is reached by datamarking / spotlighting (Hines et al., arXiv:2403.14720, 2024) and by CaMeL's data/control-flow separation; and a systematization of prompt injection against agentic coding assistants reaches the same verdict from the attack side — mitigation must be *architectural*, not model-level (*Prompt Injection Attacks on Agentic Coding Assistants*, arXiv:2601.17548); the sharper open problem this object is built to answer — formally specify the trust boundaries, then verify implementations respect them — is our phrasing of where that verdict points, not the paper's. Two convergences are weaker, and flagged. The **reach-avoid hitting-time certificate** is the independently developed reach-avoid supermartingale (RASM, arXiv:2210.05308, AAAI 2023) and stochastic Lyapunov–barrier apparatus, and its *hardness* is corroborated — expected-stopping-time problems for Markov chains are inter-reducible with the Positivity problem, a relative of the Skolem problem (Chatterjee & Doyen, *Stochastic Processes with Expected Stopping Time*, arXiv:2104.07278) — but this supports generic hardness only, not the specific incompressibility-at-$|W|$ conjecture, which remains ours and unproven. And **injection as an adversarial policy** is corroborated as a minimax game in the *detection* setting (DataSentinel: Liu et al., *A Game-Theoretic Detection of Prompt Injection Attacks*, arXiv:2504.11358) and as adversarial-disturbance robustness (*Stable Agentic Control*, above) — but no prior work assembles it as reach-avoid over the tool-output kernel with the gate as the irreversibility margin; here the relation is adjacency, not convergence.
---
## Appendix: model implementation
The definition is deliberately abstract: $\pi, \gamma, Q_E, \rho$ are *roles*, not code, and a deployed harness forces concerns the abstract object is silent on. This appendix does not re-derive the implementation; it establishes a **pattern** — take a hard practical concern, locate it in the objects already defined, and read off the discipline they imply rather than inventing new machinery. Cancellation is the worked example, chosen because it is where the silence bites hardest and because the answer falls entirely out of objects already on the page.
**Cancellation.** An owner stops a running agent mid-flight — worst across a task-agent tree. The naive reading is "stop and undo," but the irreversibility point forbids it: $\gamma$ is the last line before irreversible effects, and $\rho$ can reject a response but cannot undo an authorized action. So cancellation is not *making it not have happened*; it is a disciplined stop with a defined disposition for what is already irreversible.
A cancel is a signal, so by the Markov requirement it lives in $s$. The gate then closes on it: while the cancel flag is live, $\gamma(s,y)=\bot$ for every proposal. That is the entire "block the pending actions" requirement — they hit the gate already built and bounce into the no-op, with no new blocking machinery — and it forecloses all *future* turns at once, since $\pi$ lowers nothing new that $\gamma$ will pass. After the signal is observed, **no action crosses $\gamma$.**
The hard half is the action already *past* $\gamma$, executing in $Q_E$, whose effect is landing or has landed. Here the disposition is a trinary on the kind of $Q_E$ you authorized. If the tool is **cancellable**, propagate the cancel into it; it aborts and reports a true end-state (committed, rolled-back, or partial), and $\rho$ folds the real disposition. If it is **bounded** — drainable in acceptable time — simply wait and record the real $e$. If it is **opaque and unbounded** — a bash invocation that may itself be a harness, an environment you hold no handle into — you cannot stop the effect, only your *wait* for it: the controller fabricates $e$, a synthetic "cancelled" response, and folds it through $\rho$ so the loop can reach a terminal.
That synthetic result is the subtle case, and the load-bearing rule is this: $\rho$ may fabricate the *acknowledgment* but must not fabricate the *outcome*. A synthetic "cancelled, no effect" entry reads downstream as *the action did not happen* — and will cause a double-send exactly as readily as a dropped record causes an orphan. Same bug, opposite sign. An outcome you did not observe is $\mathsf{unknown}$, never $\mathsf{none}$: the cancelled agent never saw whether bash sent the email, and the ledger must say exactly that. (This is why $e$ must be an effect record and the ledger must live in $s$ — the fabricated entry is still a ledger write, and its value is what a later reader acts on.)
The run halts into $H_{\mathrm{cancel}} \subseteq H \setminus H_{\mathrm{ok}}$ — a distinguished terminal, non-accepting but *safe* (outside $B$), refining the deliberately coarse $H \setminus H_{\mathrm{ok}}$ of the definition (the body leaves that set unenumerated; the appendix is where its subclasses earn names) — with a specific postcondition: no action crossed $\gamma$ after the cancel was observed, every in-flight action was drained to its real disposition or recorded $\mathsf{unknown}$, and the ledger is consistent. It is worth separating from refusal and from a wrong answer precisely because that guarantee is its own.
Cancellation must be **cooperative, not preemptive.** The owner writes the cancel into the child's $s$; the child observes it at its next $\gamma$ check. The guarantee is therefore "no new action after the cancel is *observed*," not "after it is *sent*" — a child may authorize one more action in the gap, which simply drains like any other in-flight. Preemptive cancellation — killing the child mid-$Q_E$ — is exactly what manufactures $\mathsf{unknown}$ state at scale, because it destroys the record of whether the action landed. And the propagation is **recursive**: cancel flows down the subtree, each level closes its gate at its next check and drains, and the owner's cancel "completes" only when the subtree has drained. A single agent's drain is its own in-flight action; a tree's is the whole subtree reaching safe points cooperatively — the irreversibility problem stacked on a distributed-coordination one, which is why task agents are the worst case.
Compensation lives **outside** the cancelled agent. A completed-but-unwanted effect cannot be undone by the agent that caused it — its gate is closed — so a compensating, saga-style action is the *owner's* job, issued after $H_{\mathrm{cancel}}$ and reading the child's ledger to decide what to reverse or annotate. It must be the owner's, because the cancelled child cannot even know whether compensation is needed: it never observed the outcome. The owner inherits the $\mathsf{unknown}$ and any still-live orphan process, and reconciliation is its responsibility.
Finally, the part that shapes the tool rather than the document. Opaque unbounded $Q_E$ is uncancellable because authorization happened at the wrong **granularity** — an unbounded environment crossed $\gamma$ on a single approval. The discipline the objects imply is therefore not "handle uncancellable tools better" but: *the gate should prefer bounded, instrumented $Q_E$ over opaque ones, so that cancellation and the ledger stay honest.* A bash invocation behind a wrapper that tracks its process tree and effects converts the third branch into the first. Sometimes opaque is the only option, and then $\mathsf{unknown}$ and owner-inherited orphans are the honest floor — but where the choice exists, that is the pressure cancellation semantics put on tooling.
**Resume (involuntary stop).** Cancellation's twin, without the courtesy of a signal: a process crash, a lost node, a partition mid-$Q_E$. Nothing new is needed to say what recovery *is*. A crash is not a halt — $H$ is a property of the state, and the run never reached it; the chain merely stopped being *computed*, and resume computes it further, re-entering $T$ at the last durable $s$ (not the body's *restarting spec*, which exits a refusal terminal — here no terminal was ever reached). That sentence is the Markov requirement cashing out operationally: re-entry is sound exactly when $s$ was the whole state, so anything load-bearing that lived only in process memory — an in-flight buffer, a lock held in RAM, a plan revision not yet folded — is a state-ablation failure (*How this could be wrong*) discovered at the worst possible time. Durability of $s$ is not an implementation nicety; it is what the Markov claim *means* when the machine dies.
The sharp part is an ordering the ledger's own trichotomy forces. The formal transition is atomic — $s_{n+1} = \rho(s, y, a, e)$ in one piece — and a crash lands *inside* it, so resume is really a statement about the implementation's refinement of that atom into micro-steps: authorize, journal, dispatch, collect, fold. The discipline is that every crash point must resume to one of exactly two honest readings — not-yet-dispatched ($\mathsf{none}$, safely retriable) or dispatched-unconfirmed ($\mathsf{unknown}$, the cancellation entry's third branch) — and **journal-before-dispatch** is what makes the boundary between them observable: on $\gamma$'s authorization the shell journals an open $(\mathsf{action\_id}, \mathsf{pending})$ entry into durable $s$ before $Q_E$ sees the action — the write is the shell's step bookkeeping, so $\gamma$ itself stays effect-free. Journal *after* dispatch and a crash in the gap leaves no record at all — resume reads silence as $\mathsf{none}$ and re-sends, the double-send bug again, produced by a power cut instead of a synthetic entry. Write-ahead intent is not imported from database lore; it is forced by "did not confirm" is not "did not happen."
The same pressure lands on tooling from a second direction. The $\mathsf{action\_id}$ the record already carries is an idempotency key wherever the tool will accept one: re-dispatch after resume becomes safe, and $\mathsf{unknown}$ becomes *queryable* — ask the tool what it did with this key — rather than terminal. The disposition trinary returns with new labels: idempotent-or-queryable $Q_E$ resumes cleanly, bounded $Q_E$ drains, opaque $Q_E$ leaves $\mathsf{unknown}$ and owner-inherited orphans, the honest floor again. The wrapper that made bash cancellable makes it resumable; it was the same wrapper all along. And if durable $s$ itself is lost there is nothing to re-enter: the run collapses to a single $\mathsf{unknown}$ in its owner's ledger — degraded accounting, but never silent.
**Gate placement (fail-closed, in practice).** The natural implementation question is whether fail-closed means tool-call parsing and validation must happen before any tool invocation. It does — with the division of labor the definition already fixed: *parsing* lives in the inner readout $R$, the syntactic, verified extraction into $\mathcal{Y}$ (what the readout-typing falsifier checks), and *authorization* lives in $\gamma$, which is a *gate* — validation is not merely *prior to* invocation, it is what *authorizes* it. The model emits text; $R$ has already extracted it into a typed proposal; $\gamma$ validates that proposal against $s$, and only a survivor becomes an authorized action that $Q_E$ may execute. The teeth are in $\gamma$ being the *sole* route from model text to execution: no path to a side effect that does not pass the gate. And the validation is not a fixed checklist but **any deterministic predicate over $s$ and $y$** — that domain is the point, since the gate sees all of the state and the full proposal, so anything computable from them is a legitimate authorization condition. Three kinds matter. *Syntactic* — well-formed, schema-conformant, the tool exists, arguments typed. *User authorization* — does the principal this run acts for hold the right to *this* operation on *this* resource in *this* context: a function of the auth scope, principal, and session carried in $s$ and the resource and operation named in $y$, and *dynamic* rather than a static capability table, since the same caller may be permitted now and not once a budget is spent or a lock held. *Structural intent* — does the call cohere with the plan and the lowered task already in $s$: a consistency check, not a mind-reading one.
That last kind marks the seam where the gate stops being able to stay pure, and it is the same seam the rest of this document is built around. The *structural* slice of intent — does the action cohere with the plan in $s$ — is a deterministic predicate over $s$ and $y$, effect-free, and belongs in $\gamma$ without reservation. But whether an action matches what the user *actually meant*, in the full semantic sense, is exactly the thing the definition says cannot be checked: natural language is all undefined behavior, with no source-language standard to validate against. So a semantic intent check is a *learned* check, and an LLM judging "is this what they wanted" is a **stochastic kernel** — putting it inside $\gamma$ breaks the property the gate exists to hold, by the same move flagged for the fold-back verifier: a learned judge is a kernel, and belongs in $M_W$, not in a deterministic map. Semantic intent therefore does not live *in* the gate; it is a plant call — a separate authorize-the-proposal pass through $M_W$ whose output $\gamma$ then deterministically gates — or it is drift you measure, never a guarantee you hold. That nested call is not a new kind of thing: it is a mini-harness inside the gate's decision — a judge $M_W$, its own syntactic readout, its own deterministic gate — so its failure case answers itself, the inner gate fail-closing on an unparseable or low-confidence judgment exactly as the outer one does, because it *is* one. The object is **closed under this construction**: semantic gating is added by recursion, not by a new primitive. One constraint on the recursion is load-bearing enough to be a rule, because it is where this entry meets the provenance partition of the body: the judge's verdict is derived, through a learned kernel, from the very content an adversary may have bent, so folding it into authorization is exactly the fold the partition forbids — *unless the verdict can only cost capability*. **A learned check may narrow the deterministic admissible set; it must never widen it.** Judge-as-veto is safe by construction *in the authority lattice*: attacker influence over the judge can at worst manufacture a denial, a liveness cost the certificate already prices — its *dynamical* pricing, where a denial is an input and not a free no-op, is the caveat below. Judge-as-approver — a verdict granting what the deterministic checks alone would refuse, or standing in for the trusted principal's confirmation — lowers the certified floor to those deterministic checks alone; if avoiding $B$ depended on the deny the judge now withholds on the adversary's behalf, the certificate is gone. Only the trusted principal widens authorization; learned kernels only narrow it. (The recursion already obeys this: the mini-harness's inner gate fail-closes to $\bot$ — a deny — which is why the construction was safe to add at all.) The cost is real and worth stating — a judge pass is another full model call, with its latency and tokens — so it is a decision about *which* actions warrant it, not a free wrapper for all of them. The gate widens to every deterministic predicate over $s$ and $y$; it does not widen to the one predicate the document says is not deterministically checkable.
One more caveat keeps the veto's pricing honest, because a denial is free only in the *authority* lattice. In the dynamics it is an input like any other — folded into $s$, lowered into the next context, conditioning the plant's next proposal — so adversarial influence over a judge is influence over the *trajectory*: a selection channel (deny all but the path toward $B$, and the admissible set the plant experiences is a maze the adversary curated), and a targeted-liveness channel against load-bearing actions — the unstated dual of judge-as-approver: if avoiding $B$ depends on the action the judge now denies on the adversary's behalf, fail-closed's safe landing is an obligation the design earns per-state, not an axiom it inherits. The supervisory ancestry supplies the discipline: a learned veto requires a **nonblocking escape it cannot disable** — an always-enabled route to the trusted principal behind a bounded retry budget, degrading to an always-enabled *safe halt* the veto cannot deny wherever the principal is unreachable (the autonomous phase of the daemon entry below) — or manufactured denials strand the run, or steer it. And whatever a verdict carries *back to the plant* is a second channel, wearing the judge's authority framing. Free prose there is *generative* influence — injected context, priced by the minimax descent, never by the veto's zero-widening — so the narrow-only rule has an influence-side twin: **a learned verdict's payload to the plant is selected, never generated** — controller-authored symbols, typed citations validated like any effect record, template text with no interpolated model prose — its per-verdict capacity a designed constant rather than a measured hope, and the residual selection pattern audited as the covert channel it is. The alphabet's bound is not a count but two thresholds: symbols become tokens when their semantics stop being controller-authored — the registry the trusted writer can actually audit is the real constant, and borrowed alphabets with upstream owners (a linter's rule registry) spend that budget well — and tokens become language when composition turns productive, arrangement carrying meaning the controller never wrote. Below both thresholds the alphabet may be as large as the audit budget affords. The strongest form dissolves the learned verdict into *scheduling*: the learned component chooses which deterministic checks to run — pass-ordering over verification passes — and the only verdicts that flow anywhere are what the oracles actually said, leaving attention misallocation, a liveness cost, as the entire attack surface.
But "before any invocation" has to be read as *before any effect*, which is sharper than it sounds — and the reason is the irreversibility point above: you validate before execution because execution is what you cannot take back, so the real invariant is **no effect crosses $\gamma$ unvalidated**. That catches three cases the naive reading misses. *Reads are not free*: a read-only call is still an injection vector (it pulls attacker-controlled content into context) or an exfiltration vector (a request whose URL is the payload), so the gate authorizes the *call* regardless of whether it mutates. *Validation must not act*: a "validator" that resolves a call by hitting an API, expanding a template that fires a webhook, or evaluating an argument that runs code has collapsed validation into invocation, and the effect has already happened *inside* $\gamma$ — so $\gamma$ itself must be **effect-free**, pure and total over the proposal and the current $s$, with no network and no execution; if deciding validity *requires* a side effect, that side effect is itself an action and must go through the gate, recursively. *The output is an action too*: the user-visible response and any logging are effects — for model-authored text, emitted either as an authorized action through $\gamma$ or only after an accepted halt (shell-templated status on any halt is the controller speaking, not the model) — streaming raw tokens to a sink before $\gamma$ has cleared them is the same bug from the other end.
So the property, tightest: $\gamma$ is a **pure, effect-free authorization that every model-proposed action — tool call, read, write, or final output — must pass before any effect occurs**, with "before" enforced structurally by the gate being the only route from model text to $Q_E$. The two failure modes to design against are a path from model output to a sink that bypasses the gate, and a $\gamma$ that is not effect-free, so that "validating" a call already rang the bell. And the boundary, so the property does not overpromise: $\gamma$ guarantees *no unauthorized effect* — pure code ordering, fully in your control — but not that an *authorized* effect is safe or correct; that is the plant's problem, and the reason $\rho$ and the reach-avoid certificate exist. Fail-closed is the floor — nothing executes that did not pass the gate — not the ceiling.
There is a third failure mode beside those two, and it is not a code path but a credential. A tool process that holds standing authority — an environment full of long-lived secrets, a database connection with every grant, an agent identity the network trusts — does not need the model's proposal to act, and against it $\gamma$'s $\bot$ is a decision with nothing to enforce it. The gate *decides*; something must make the decision *binding*, and "no path from model output to a sink that bypasses the gate" must be read to include the non-code paths: ambient authority is a bypass provisioned before the run began. The discipline is **per-action capability**: the authorized action *carries* its grant — a scoped, short-lived credential minted at authorization, valid for this $\mathsf{action\_id}$, this resource, this operation — so that a tool holds, at any moment, exactly the authority of the actions the gate has passed it and nothing standing. In the language of the minimax certificate this is enforcement as $\Pi$-shaping: sandboxing, capability scoping, and network policy do not make the gate smarter — they shrink the class $\Pi$ of environment policies an adversary can choose from, so the worst case the certificate must survive gets structurally smaller. A gate in front of an omnipotent tool is a suggestion; the objects compose into a guarantee only when $Q_E$'s reachable effects are no larger than what crossed $\gamma$.
And one more boundary, because "fully in your control" above is a *single-run* statement. $\gamma$ authorizes against the $s$ it read; the effect lands later, against a world that may have moved — the gate cannot freeze the world between authorization and commit, so the honest property is *no effect unauthorized relative to the $s$ at authorization time*, and closing that gap requires the tool itself to bind check to commit (compare-and-swap in $Q_E$), which relocates part of the enforcement past the gate and weakens "$\gamma$ is the last line" to "$\gamma$ plus a commit guard" for exactly the effects that need it. The same seam opens *between* runs: the dynamic authorization state the gate reads — budgets, quotas, locks — is, once shared, no single run's coordinate, and two children of a coordinator can each pass $\gamma$ against snapshots that jointly overdraw a budget neither exceeded alone. The cancellation entry's observed-not-sent gap ("a child may authorize one more action in the gap") is this phenomenon wearing one hat; the general statement is that cross-run authorization state needs its own serialization discipline — the ledger as the serialization point is the natural choice — and the per-run certificate is silent about it. TOCTOU is not a counterexample to the formalism; it is what the formalism says when you admit $s$ is a *view*.
**Parallel proposals (the batch gate).** Models emit several tool calls in one turn, and the outer chain assumed one action per step. The repair is formally cheap: a batch is a single action in $\mathcal{A}$ that happens to be a set, $Q_E$ runs its elements concurrently, the interleaving's nondeterminism folds into $Q_E$ exactly as the determinism audit requires, and $\rho$ folds one effect record per element — $e$ is then a finite set of records — each keyed by its own $\mathsf{action\_id}$ — the record interface already supports partial outcomes (one element $\mathsf{committed}$, its sibling $\mathsf{unknown}$). One discipline survives the cheapness: **individually admissible actions can be jointly inadmissible.** Read-the-secret and post-to-the-web each pass a per-call check; the pair is an exfiltration channel — and two calls that each fit a budget jointly overdraw it, the cross-run overdraw of the previous entry reappearing *inside* one turn whenever elements are authorized independently. Since $\gamma$'s domain is any deterministic predicate over $s$ and $y$, joint authorization was licensed all along; the content here is only that the gate must take it — authorize the *set*, atomically, against one snapshot, with interaction predicates (source-to-sink flow between capability classes, summed resources) and not merely element predicates. The cost note is the judge's, transposed: full powerset reasoning is combinatorial, so a real gate checks declared interactions rather than every subset — a tractability trade to make explicitly, not by forgetting the batch was a set.
**Effect records (what $\rho$ folds back).** The fold-back $\rho$ and the cancellation ledger both turn on the response $e$ being an *effect record* rather than raw API bytes — said twice in the body and pinned down nowhere, though it is the interface that makes both tractable. The minimal shape is small: roughly
Each field is forced by something the body already needs. The $\mathsf{action\_id}$ lets $\rho$ match a response to the in-flight action $\gamma$ authorized, and lets the ledger say which actions are still open — without it the $\mathsf{unknown}$/orphan accounting has nothing to key on. The $\mathsf{status}$ must carry $\mathsf{unknown}$ as a value *distinct* from $\mathsf{committed}$ and from $\mathsf{none}$, because that distinction is the whole content of the cancellation ledger: "did not confirm" is not "did not happen" ($\mathsf{none}$ is *never launched* — the record of the distinguished no-op $e_0$ a $\gamma$-rejection forces, which is how a bounce at the gate enters the ledger at all — distinct in turn from $\mathsf{rolled\_back}$, which launched and was undone: conflating those erases the difference between a gate that held and a compensation that worked). The $\mathsf{reversible}$ bit on each effect is what lets the gate know which effects are irreversible — the predicate the gate-placement entry leans on ("anything irreversible must be gated at authorization") but cannot evaluate unless the record carries it (a bit is the minimal honest form, not the final one: real effects are reversible *until* — an unsend window, a force-push until someone fetched, a row until the backup rotates — so the mark wants to be a $(\mathsf{reversible\_until}, \mathsf{cost})$ pair, a refinement the open-interface caveat below already licenses). And $\rho$ writes the record into $s$ (the ledger lives in the state), which is what lets the next step's $\gamma$, and any owner-side compensation, read it at all. The exact fields are an **open interface, not a result**: bash, HTTP, a filesystem, and a database expose effects at wildly different granularity, and a record uniform across them is a real design problem this document does not resolve — it fixes only what the record must *support* (match by $\mathsf{action\_id}$, the $\mathsf{committed}$/$\mathsf{none}$/$\mathsf{unknown}$ trichotomy, and a reversibility mark), since without those three $\rho$ and the cancellation semantics lose their grip.
**Derived and durable state (compaction and memory).** Two mechanisms let data re-enter the context long after it arrived: compaction, which replaces transcript with a summary when the conversation outgrows what $\pi$ can lower, and memory, which persists records across sessions. Both are transformations of state that produce state, and both therefore raise a question the body's partition answers only if one more closure property is stated: **provenance is a property of the information, not of its position in the pipeline — a transformation's output inherits the meet, in the trusted-writer lattice, of its inputs' labels.** Without that closure, compaction is a laundering channel: a summary of a session that contained an injected page can assert "the user asked to export the database," and the structural-intent check then validates future proposals against a plan the adversary bent — not through $\gamma$, not through $\rho$'s fold of a single $e$, but through the summarizer, which is a learned kernel (it lives in $M_W$, by the standing rule) and so cannot be trusted to preserve a partition it does not know exists. The discipline: summaries of data are data; the control-determining coordinates — plan, grants, what is authorized next — cross a compaction *verbatim* (copied, not paraphrased) or by re-confirmation from the trusted principal — never through the *summarizer*; the model rewrites the plan at plan steps, through the gated fold the body prices, and compaction is not one of them. Memory obeys the same closure twice, at write and at retrieval: the label rides the stored record across sessions, or a poisoned memory is an injection with an arbitrarily long fuse — and retrieval, being learned ($\pi$'s selection factor — adequacy-only behind the never-lower filter), decides what comes back but never what it is trusted *as*. The same test applies at birth: tool catalogs and server-supplied tool descriptions are third-party durable data that arrive dressed as instructions, and the lattice files them on the data side of $s_0$.
One more read-off, this time from irreversibility. *Destructive* compaction — dropping the original transcript once the summary is written — is a side effect against your own state that no later step can undo, and the gate-placement rule ("anything irreversible must be gated at authorization") does not exempt self-directed effects. The granularity preference then says what it said about bash: prefer the instrumented form — originals kept content-addressed, the summary an index and a cache rather than an authority, re-derivable when the $\pi$-sufficiency probe (*How this could be wrong*) says the summary dropped what mattered. A summary you can audit against its source is a lowering; a summary that replaced its source is a fait accompli.
**Composition (harness trees).** The cancellation entry already walked a tree — cancel flowing down, drains flowing up — and "a bash invocation that may itself be a harness" has hovered since the disposition trinary; what is missing is only the statement that makes both ordinary. From the parent's seat, a child harness *is* a $Q_E$ component: spawning it is an action authorized by $\gamma$ like any other, and the entire child run — its own $\pi, \gamma, \rho$, its own coins, its own halt — is one environment draw whose response $e$ is the child's terminal ledger. The law is four correspondences. The child's halting time is the parent's per-step *cost*: a parent certificate consumes a bound on $\mathbb{E}[\tau_H^{\mathrm{child}}]$ — the budget handed down at spawn, which the child's own budget-counter certificate discharges — or the parent's drift is uncontrolled however good its own $\hat V$. The child's ledger is the parent's *effect record*: the child's $e$ carries the $\mathsf{committed}/\mathsf{none}/\mathsf{unknown}$ accounting upward — which is what already let the cancellation entry make compensation the owner's job; the interface was this all along. And the child's non-accepting halts are the parent's *partial failures*: a refused child folds back as a response the parent routes around, not an exception that unwinds it. And the child's admissible effects are the parent's *$\Pi$-restriction*: the spawn grant bounds what the child can reach — the ledger reports what *happened*, the grant bounds what *could* — which is how safety composes without the parent ever reading the child's gate; the attenuation below is this correspondence stated as a rule. Read this way, the gate-granularity discipline and the tree are one preference: an instrumented child — budgeted, ledgered, cancellable — *is* the bounded, cancellable $Q_E$ the trinary prefers, and an opaque bash invocation is an un-annotated child you declined to instrument. Nesting adds no primitive on the environment side either: the parent never sees the child's gate and does not need to — it gates the spawn, prices the budget, folds the ledger, and the child's internal guarantees surface only as the shape of $e$. Nothing fixes one level: the tree recurses, budgets subdivide, ledgers concatenate upward, and the cooperative drain of cancellation is this law read under a cancel signal.
The tree leaves one seat unassigned: who plays trusted principal for a *child*? The parent — but with derived authority, not original, and the derivation is the narrow-only rule read along the spawn edge: **authority attenuates monotonically down the tree.** A spawn may grant the child any subset of the parent's own grants and nothing outside them; budgets subdivide, scopes narrow, and no edge widens. When a child asks-the-owner, the parent may answer from authority it already holds — that is attenuation working as designed — but a request beyond the parent's grants routes *up*, ultimately to the root principal, because a parent improvising an answer it was never granted is a learned kernel widening authorization: precisely what the gate-placement rule forbids a judge, and being a parent confers no exemption. The corollary is worth one sentence: a fully autonomous run is one whose root principal is unreachable, so the tree's only widening channel is closed and authorization is frozen at launch — not a limitation of the formalism but the honest price of the word *autonomous*.
**Daemons (the recurrent harness).** Every entry so far assumed a run that ends; a coordinator, a watcher, a service does not, and the blockquote of *The limit* already named the swap — absorption at a halt set gives way to recurrence to a **ready set** $\mathcal{R}\subseteq\mathcal{S}$, and $V^\star=\infty$ is the spec rather than a pathology. The appendix's job is to say what that costs operationally, and the answer is one idea: **the daemon is the regenerative process of concatenated runs.** Each trigger-to-ready excursion — wake on an event, work, return to $\mathcal{R}$ — is one run of the absorbing object this document already defines, with $\mathcal{R}$ playing the halt set for that excursion; the daemon is those excursions laid end to end. Per-run certificates then lift to long-run rates by renewal-reward — expected work per excursion over expected excursion length — *exactly when* the ready state is a genuine regeneration point: the future from $\mathcal{R}$ must not depend on which excursion you are in.
That proviso is the whole difficulty, because **what accumulates breaks regeneration.** The ledger grows, memory persists, budgets deplete, summaries compact — all deliberately across excursion boundaries, so successive runs are at best *conditionally* independent given the carried state, and the renewal-reward bookkeeping is over that conditioning, not the raw cycle. Two disciplines keep it honest. First, the carried state is exactly where long-fuse attacks live: the poisoned-memory line of *Derived and durable state* is a cycle-scale injection, a payload written in excursion $n$ and lowered into the plan of excursion $n{+}k$, so the meet rule on provenance must hold *across cycles*, not only across a single compaction — everything that crosses a boundary carries its label. Second, per-cycle safety compounds the way the blockquote already priced it — a per-cycle bad-set hazard $q$ gives lifetime survival $\approx(1-q)^N$, and a reassuring $0.9999$ is $\approx0.37$ over ten thousand cycles — so a daemon's safety is not a fixed margin but a decaying one, and lifetime safety needs **renewal events that reset accumulated risk**: owner re-confirmation, audit, credential rotation, verified re-compaction against content-addressed originals. Hygiene is not housekeeping here; it is the renewal structure that makes the long-run bound exist at all.
Authority under intermittence is the last piece, and it is where the daemon meets the veto caveat and the autonomy corollary as one phenomenon. A daemon alternates *attended* stretches, where the trusted principal is reachable, with *autonomous* ones, where it is not; between contacts the autonomy corollary binds and authorization is frozen at the last grant, so each owner interaction is a **renewal point for authority** exactly as re-compaction is a renewal point for risk. The two recurrences need not coincide — the ready-set cycle can turn many times between owner contacts — and the gap between them is a stale grant meeting a fresh world, TOCTOU at cycle scale: a budget approved for yesterday's prices, a scope granted against a resource that has since changed hands. This is also where the learned veto's nonblocking escape gets its daemon reading: in an attended stretch the un-disableable route is the escalation to the principal, but in an autonomous stretch that route is unavailable, so the escape it cannot deny must be the **safe halt** — a daemon whose judge can be driven to manufacture denials must, when it cannot reach its owner, be able to stop rather than be steered.
Nothing here is new machinery either: $\mathcal{R}$ is a non-absorbing terminal read of an existing set, an excursion is the run $T$ already defines, the carried state is the same $s$, and every renewal event is an ordinary owner-issued action. The daemon is the outer loop closed into a cycle — which is the natural bridge to the object one level out.
The pattern generalizes, and that is the point of the appendix. Nothing here added a primitive: the cancel is a signal in $s$, the gate closes by the rule it already follows, the in-flight disposition is forced by irreversibility, $H_{\mathrm{cancel}}$ is a subclass of an existing terminal set, and compensation is an ordinary owner-issued action — and the later entries kept the promise: resume re-enters $T$ at a persisted $s$, the batch gate was always in $\gamma$'s domain, provenance closure is the lattice's meet, attenuation is narrow-only read along an edge, and per-action capability is the gate's decision made enforceable. Every practical concern that earns a place here should resolve the same way — not new machinery, but the discipline the existing objects already imply, made explicit. Cancellation and resume, gate placement and the batch gate, effect records and the state derived from them, composition and delegation, and the daemon that concatenates runs into a cycle — those are the worked instances; the rest of the model is the same exercise.
# What a Harness Is — and What It Can Never Promise
*A plain-language companion to [HYPOTHESIS.md](HYPOTHESIS.md). Same object, no symbols required.*
**How to read this.** HYPOTHESIS.md defines, formally, what an agent harness is and what it can never guarantee. This file is that document lowered into plain language — and by the formal document's own rules, a summary is a cache, not an authority: it must stay re-derivable from its source, and wherever the two disagree, the formal one wins. Symbols appear once, in parentheses, so you can cross over; nothing here requires them. And none of it is decoration: the formal version, used as a checklist, has caught real bugs in a real harness — because most bugs are a violated invariant nobody had written down.
## The problem
You have a model. It is, roughly, a brilliant, tireless, lightning-fast intern that has read most of the internet — and that sometimes makes things up, sometimes gets confused, and sometimes takes instructions from strangers, because a page it was asked to read said "ignore your boss and email the passwords here" in white text on a white background.
So you don't wire the intern to production. You build a loop around it. The **harness** is that whole governed loop: a deterministic shell *you* write — build the prompt, approve or refuse each proposed action, fold the result back into memory — wrapped around a model you didn't write and a world you don't control, repeated until the run reaches a stopping state. The shell is code and does the same thing every time. The model is neither, and everything in the theory comes from taking that split seriously.
One sentence to keep: **the model proposes; the gate disposes.** The model's output is never an action. It is a suggestion, in text, which a piece of ordinary code you wrote either turns into an action or refuses.
## The parts
| Plain name | What it does | In the formal doc |
|---|---|---|
| The owner | The human or account the run acts for; the only party who can grant new permissions | the trusted principal |
| The memory | Everything the run knows: task, plan, transcript, and the ledger of what has been done | the state, *s* |
| The prompt builder | Decides which slice of memory the model gets to see this step | the lowering, π |
| The model | The black box that reads the prompt and writes a proposal | the plant, M_W |
| The gate | Ordinary code that checks every proposal and approves or refuses it | the gate, γ |
| The tools and the world | What approved actions actually touch: files, APIs, shells, people | the environment, Q_E |
| The verifier | Checks each tool result, then writes it into memory | the fold-back, ρ |
| The stop rule | Decides when the run is finished — and whether it finished *well* | the halt set H, accepting halts H_ok |
| The danger zone | States that must never be reached: secrets exfiltrated, wrong files deleted, money moved twice | the bad set, B |
The loop:
```
you ask for something
↓
prompt builder → model → "I propose: send_email(...)"
↓
GATE ── no ──→ nothing happens (safe, recorded)
↓ yes
tool runs in the world
↓
verifier checks the result, writes it to memory
↓
done? ── no → around again
↓ yes
stop (well, or refused)
```
## The rules that make it a harness
Four invariants, all about *where* things are allowed to happen.
1.**The model sees only what the prompt builder shows it** — never raw memory. The corollary with teeth: a secret that never enters the prompt cannot leak through the model. The redaction step that keeps credentials and other people's data out of the prompt must be dumb, deterministic code — the moment that filter is "smart," your confidentiality guarantee is a probability.
2.**Model outputs are proposals, not actions.**
3.**Every side effect passes the gate.** There is no second door.
4.**The harness itself flips no coins.** Replay a step with the model's answer and the tool results pinned, and behavior must be identical; any leftover variation is randomness *you* added and must be accounted for. The fine print: "deterministic" is conditional on pinned versions — a provider silently retraining the model behind the same API name changes the machine under you, and every dashboard number you collected dies with the version.
Notice what the rules don't say: they don't say the harness is *good*. A gate that approves everything satisfies rule 3 the way a lock that's always open satisfies "has a lock." The definition is a shape; the guarantees are what a particular harness *earns* inside it. Everything below is about what can be earned — and what can't.
And notice the symmetry between rules 1 and 3. There is exactly one door from your data into the model — what it may see — and exactly one door from the model into the world — what it may do. Nearly every security failure in these systems is one of those two doors with a hole in it: a secret lowered into a prompt that didn't need it, or a path from model text to a side effect that skipped the gate. Same bug, arrow flipped.
## Fail-closed, said precisely
"Fail-closed" gets used loosely. Here it means something exact: **nothing happens unless the gate said yes, and a refusal must itself be safe** — a refused proposal causes no side effect and leaves the run somewhere sane, which may be "stopped, having declined." The run is allowed to *say so*: a templated status message written by the shell is the shell speaking, not the model, and needs no gate. Failed runs don't have to die silent.
Three consequences people miss:
**Reads are not free.** A read-only call can smuggle instructions *in* (the fetched page is attacker-controlled) or secrets *out* (the URL it fetches can encode the payload). The gate approves calls, not just writes.
**Validation must not act.** A "validator" that resolves a URL, expands a template that fires a webhook, or evaluates an argument has already acted — inside the check. The gate must be pure: it reads the proposal and the memory and outputs yes or no. If deciding requires touching the world, that touch is itself an action and goes through the gate.
**Anything irreversible is decided at the gate.** The verifier can reject a bad *result*; it cannot unsend the email. So the question "can we take this back, and until when?" is asked before execution — which means each tool's effect record has to carry a reversibility mark, or the gate can't ask it.
Two honest asterisks. First, the gate checks a snapshot: it approves against the world *as its memory describes it*, and the world can move between check and commit. For actions that race the world — spend against a balance, write against a row — the tool itself must bind check to commit (compare-and-swap), or you have a classic time-of-check/time-of-use hole. The gate decides; for those effects, the tool enforces. Second, a gate is only as binding as the authority behind the tools. A tool process holding standing credentials — a database connection with every grant, an environment full of long-lived secrets — doesn't need the model's proposal to act, and against it the gate's "no" is a decision with nothing enforcing it. **A gate in front of an omnipotent tool is a suggestion.** The fix is to make the approval *be* the key: each authorized action carries a short-lived credential scoped to exactly that action, that resource, that operation, so tools hold no standing power at all.
## Why you don't get a proof — and what you do instead
If you write a sort function, you can prove it sorts: the function is small and the spec is exact. A harness has neither luxury. The spec side fails first — the task arrives in natural language, and natural language is, in the compiler's sense, *all undefined behavior*: there is no formal standard for "what the user meant" to verify against. The mechanism side fails next — the model is billions of learned parameters, and nobody can hand you a compact argument for why they jointly do the right thing.
Here is the careful version, because "you can't prove it" overshoots. The quantity you would want — call it the *expected steps to done* from any situation — is perfectly well-defined; in principle it exists. The document's central conjecture is that, for a model of this size, any faithful writing-down of that quantity is roughly *model-sized*: the honest proof-object does not compress. Find a small one and the conjecture dies — the document lists that outcome, explicitly, among the ways it could be wrong.
So instead of proving, you measure. You pick a progress meter — plan depth shrinking, open obligations closing, budget burning at the expected rate — and you check, across many runs, that it goes downhill and that its stalls predict failure. Two disciplines keep the measurement honest. The number bounds the world you *sampled*, never the world an adversary will choose: a meter calibrated on friendly traffic says nothing about hostile traffic. And the meter is itself attack surface: if "is the agent making progress?" is judged by another model, an attacker who can bend your agent can bend your *measurement of it* first, hiding the divergence from the very dashboard built to catch it. A learned meter is part of the system under test, never a neutral instrument.
A measurement is a risk metric. A proof is a certificate. Keeping those two words apart is half of what this theory is for.
## Security: reach the goal, avoid the danger — and who may change the rules
Formally, security here is a *reach-avoid* problem: reach a good stop, never touch the danger zone, **while an adversary picks the worst tool outputs your setup permits**. That last clause is the formal home of prompt injection: injection isn't "the model misbehaved," it's the environment optimized to bend your loop — poisoned pages, malicious tool descriptions, crafted responses.
Two different numbers fall out here, and dashboards love to collapse them: *success* (reached the right end before anything went wrong — a safe refusal counts against it) and *safety* (never touched the danger zone — a safe refusal is perfectly safe). Track both. They move independently.
The gate handles the visible half of injection: the model, freshly poisoned, proposes emailing your credentials somewhere, and the gate refuses — and injection or not, the action does not happen. But the deeper attack doesn't propose a bad action today. It rewrites *what the run believes its job is* — it edits the plan — and then every future action looks locally reasonable against a corrupted plan. So memory has to be partitioned: **data** (tool results, fetched pages, retrieved documents — content the world supplied) and **control** (the plan, the permissions, what is authorized next). The security claim is conditional on that partition holding: untrusted content lands in data, always.
Which forces the question the theory has to answer: *somebody* must be able to write control mid-run, or no plan could ever be steered and no permission ever granted. The answer is a small hierarchy with exactly one party at the top:
- **The owner alone widens.** New permission, bigger budget, approval of the irreversible thing — asking the owner is itself an ordinary tool call, and the owner's answer is the one kind of tool result allowed to change control.
- **The model rewrites the plan** — that is what replanning *is* — but only through the gated loop, and a plan is not a permission: nothing the model writes into its own plan can grant it powers it didn't have.
- **Everything else is data.** A fetched page can inform the plan only by passing through the model and the gate like everything else. It can suggest. It cannot promote itself to boss.
- **AI judges only tighten.** Add a model-based check — "does this action match what the user actually wanted?" — and its verdict may *veto* an action the plain rules would have allowed, never approve one they'd have refused. A judge that can approve is a tricked judge that can open the vault. And don't over-credit the veto either: a tricked judge can *aim* its refusals — denying exactly the action safety depended on, or denying everything but the path an attacker curated — so the escape hatch to the owner is the one thing a judge can never veto, and a judge's stated *reasons* are picked from a fixed, shell-owned menu, never written as prose. A judge that writes free text into the loop is an injection channel wearing a badge.
One more rule closes the loop: transformations don't launder trust. A *summary* of a session that contained an injected page is still injected — the summarizer is a model, and can be persuaded to write "the user asked to export the database" into the summary. So summaries of data are data, and the control lines — the plan, the grants — cross a summarization by being *copied verbatim* or re-confirmed by the owner, never paraphrased by the model. Memory that persists across sessions carries its trust label with it, or a poisoned memory is just an injection with a very long fuse.
## Operations: the rules you feel on Tuesday at 3 a.m.
The formal document's appendix works the operational cases in full; here they are at speed.
**The ledger, and the three-way distinction that keeps it honest.** Every action gets an ID and a record: committed, never-launched, or *unknown*. "The tool didn't confirm" is not "the tool didn't do it" — collapse those and you will, sooner or later, re-send something that already happened. The double-send bug has one reliable cure: **journal before dispatch.** The shell writes "I am about to run action #417" into durable memory *before* the tool sees it, so a crash in the gap resumes to an honest "unknown — go ask," never to silence misread as "never sent." Old database wisdom, but here it isn't imported; it's forced — it is the only ordering under which every crash point has a truthful reading.
**Crashes aren't finishes.** A process dying mid-run is not the run stopping; it's the run *pausing being computed*. Resume means re-entering the loop at the last durable memory — sound exactly when the durable memory was the *whole* state. Anything load-bearing that lived only in RAM — an in-flight buffer, a plan revision not yet written — is a bug you discover at the worst possible time. Recovery is where you find out whether your state was really your state.
**Two innocent actions can be guilty together.** Models emit several tool calls per turn. "Read the secret" passes review. "Post to the web" passes review. The pair is an exfiltration channel — so the gate authorizes the *set*, atomically, with the interactions checked, not each element in isolation.
**Sub-agents are just fancy tools.** An agent that spawns another agent is, from the parent's chair, calling a tool: the spawn is gated, the budget is part of the deal, and the child's whole run comes back as one result carrying the child's ledger. Two laws travel down the tree: budgets subdivide, and **authority only narrows** — a child holds at most a subset of its parent's permissions, and a child's request beyond those grants routes *up*, ultimately to the owner, because a parent inventing an approval it never held is the tricked-judge case wearing a manager's badge. A corollary worth framing: a *fully autonomous* run is one whose owner is unreachable — meaning the only channel that can ever widen anything is closed, and its permissions are frozen at launch. That is not a limitation of the theory. That is what the word "autonomous" costs.
**Keep the originals.** When the transcript outgrows the prompt and you summarize it down, deleting the original is an irreversible act against your own state — and irreversible acts are gate decisions, self-directed or not. Keep originals content-addressed; let the summary be an index, re-derivable, auditable. A summary you can check against its source is a note. A summary that replaced its source is a fait accompli.
## Robots that never clock out — and robots that assign their own work
Everything so far assumed a job that *ends*: you ask, the robot does it, you read the result. Two steps past that are where the interesting failures live, and they're the same idea one level bigger each time.
**The robot that never clocks out (a daemon).** A monitor, a coordinator, a service — it isn't supposed to finish; it's supposed to keep going, wake on events, do a bit of work, go back to waiting. The clean way to think about it: each wake-work-rest cycle is one ordinary run, and the daemon is just those runs chained end to end forever. That reframing is free — but it comes with a bill nobody likes. **Safety that's fine per cycle rots over many cycles.** A 99.99%-safe cycle sounds bulletproof; run it ten thousand times and you're at about a coin-flip of having touched the danger zone at least once. So a long-running robot's safety isn't a fixed wall, it's a slow leak — which means the antidote isn't a better wall, it's *scheduled resets*: the owner re-confirming, credentials rotating, memory getting audited and re-summarized against the originals. Housekeeping isn't housekeeping; it's the thing that keeps the safety math from decaying. And the slow-leak logic is exactly where slow attacks live — a poisoned note dropped into memory on Monday and read back into the plan on Friday is an injection with a long fuse. So the trust label on a piece of information has to survive across cycles, not just within one. One more wrinkle: a daemon drifts in and out of your reach. While you're around, it can escalate to you; while you're not, "escalate to the owner" isn't available — so the one thing it must always be able to do instead is *stop*. A robot that can be tricked into refusing everything, and can't reach you, had better be able to halt rather than be steered.
**The robot that assigns its own work (the loop).** Step back one more time. Above the robot that *does* a task sits a system that decides *which task is next* — scans the backlog, picks one, launches the robot at it, checks the result, remembers, fires again. This is the thing people mean in 2026 when they say they've stopped prompting their agents and started writing *loops* that prompt them: you design the assigner once, and it runs the doer for you while you sleep. The honest observation — and the reason this document bothers with it — is that the assigner is *not a new kind of thing*. It's the same harness, one level up: it has its own memory (the backlog), its own gate (**who let the loop refactor the auth module at 3 a.m.?**), its own verifier, and its own two walls. Every rule from the inner robot recurs on the outer one — including the uncomfortable ones. There's still no proof it stays out of trouble over a long night; there's only a measured progress meter, with the same catch that a *learned* meter can be fooled. And the origin story of the whole trend is the cautionary case in miniature: the famous first version was literally the same prompt in a `while` loop until the tests passed — which is the empty gate, the always-open lock, one level up. It works beautifully right up until the tests weren't checking the thing that mattered. The loop doesn't delete the hard problems. It moves them up a floor, where they're bigger and you're further away.
The pattern, if you want the whole thing in one line: *words, context, robot, loop* are four sizes of the same object, and every promise in this document lives in the whole assembled thing — never in any one layer by itself.
## The two walls
Two limits are structural. You don't fix them with a better harness; you design around them.
**The desk.** The model can hold only so much *in mind at once* — the context window. Files, databases, and search extend what it can *look up*, not what it can hold: every lookup still passes through the same small window to touch actual computation. The shell can page; the model cannot grow its desk. Tasks whose irreducible working set exceeds the desk don't fail loudly — they fail by forgetting the middle (the well-documented "lost in the middle" effect is this wall showing through the paint).
**The dictionary.** The model's knowledge is frozen into its parameters at training time — and the proof problem above is conjectured to live at that same scale: the certificate wouldn't fit anywhere smaller than the brain it certifies. The two walls trade against each other along the training-versus-inference axis — bigger dictionary or bigger desk — directionally, and at no clean exchange rate.
## How this could be wrong
This is a hypothesis, and it says out loud what would kill it. The tests, in plain terms:
- **The replay test.** Rerun with model answers and tool results pinned. Any leftover variation — timestamps, wall-clocks, and cache expiries are the classic leaks — falsifies "the harness adds no randomness" until accounted for.
- **The drop-a-variable test.** Remove something from memory; if behavior statistics shift, the memory wasn't complete. The crash-resume version of the same test: if resuming from saved state breaks, the saved state wasn't the state.
- **Does the meter mean anything?** If no reasonable progress meter's drift predicts real failures — across the natural families, not just one bad candidate — the whole "measure what you can't prove" program is empty.
- **The red-team test.** Swap sampled tool outputs for worst-case ones: injected pages, poisoned metadata, malformed replies. The design must survive the worst permitted world, not the average one.
- **Gates versus begging.** The theory predicts deterministic gating beats prompt-level pleading. If "please be careful" alone matches real gates on security outcomes, the controller-versus-model story is wrong.
- **The compression hunt.** Exhibit a compact, provably sound progress certificate for a frontier-scale model on a nontrivial task family, and the central conjecture falls — constructively.
- **The desk probe.** Take a task family with a *proven* memory floor — so "it needed the whole picture at once" is someone else's theorem, not our excuse — scale it past the window, and watch: the wall predicts collapse at the boundary, not graceful degradation.
## Who else landed here
The formal document keeps three honesty tiers. **Borrowed**: real theorems, cited — the drift and stopping-time mathematics is classical, and the very architecture of a deterministic supervisor gating a plant it didn't author is 1987 control theory; the shape is older than the web. **Ours**: the modeling choices and the conjectures — the walls, the incompressibility claim, the design rules — organizing principles, not results. **Corroborated**: pieces of the same object reached independently by people who never saw this framing — capability-security work isolating control flow from untrusted data (CaMeL), reinforcement-learning "shields" filtering a learned policy's actions through a deterministic checker, verification work that states the "learned safeguards can't certify" gap as its opening motivation, and architecture patterns converging on plan-then-execute. Even the field's live disagreement — provable-but-rigid deterministic layers versus flexible-but-uncertifiable learned checks — is, in this frame, not a fight but a placement: you need both, on their proper sides of the irreversibility line, with the learned one permitted only to tighten.
## What to remember
The model proposes; the gate disposes. No is the default, and a refusal must be safe. Exactly one party widens permissions — and it is not the model, a tool result, a summary, or a judge. "Didn't confirm" is not "didn't happen." The desk is finite and the proof doesn't compress, so you measure — and you say *measurement* when you mean measurement. A robot that never stops leaks safety slowly, so it needs scheduled resets — and when it can't reach you, it must be able to stop. A loop that runs robots for you is just a bigger robot with the same rules and a further-away owner. And all of it is a hypothesis wearing its own kill-conditions on its sleeve.
The formal version — the objects, the certificates, the falsifiers, the citations — is [HYPOTHESIS.md](HYPOTHESIS.md). It wins every disagreement with this file, including this sentence.
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone console — multi-workstream AI orchestration with mermaid diagrams" width="960"/>
<img src="docs/assets/hero.png" alt="Turnstone coordinator — parallel tool batches with judge-graded approval and child workstream tracking" width="960"/>
</p>
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
**What is a harness?**
```
ℋ : s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
```
[**the primer →**](PRIMER.md)
### Release Tracks
| Track | Install | Docker | Description |
@@ -26,12 +36,13 @@ See [docs/releasing.md](docs/releasing.md) for the full release process.
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Bring your own models** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), the Anthropic Messages API, and Google Gemini, mixed freely per role
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
See [QUICKSTART.md](QUICKSTART.md) for the install + troubleshooting walkthrough and [docs/docker.md](docs/docker.md) for Docker configuration.
### Programmatic (SDK)
@@ -75,23 +110,24 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
## Tools
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp-registry.md](docs/mcp-registry.md) for MCP configuration.
## Architecture
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
**Multi-node**: Client → Console (rendezvous routing proxy) → Server nodes. The console picks the target node for each workstream via rendezvous (HRW) hashing over the live service registry — pure function of `(ws_id, live_nodes)`, no stored bucket state, deterministic across readers. A node join or drop only re-routes the keys that score highest on the affected node.
| Component | Purpose |
|-----------|---------|
| `turnstone` | Terminal CLI (REPL) |
| `turnstone-server` | Web UI + REST API + SSE events |
- [Git LFS](https://git-lfs.com/) for cloning (diagram PNGs)
## Support
Turnstone is free, Apache-2.0, and self-hosted — no paid tier, no telemetry, no upsell. If it saves you time or you'd like to help keep development moving, you can sponsor the project:
**[❤ Sponsor Turnstone →](https://github.com/sponsors/eous)** · one-off via **[PayPal](https://paypal.me/eousphoros)**
Sponsorship is entirely optional and funds maintenance, new features, and infrastructure. Prefer to contribute in other ways? Filing issues, improving docs, and [pull requests](CONTRIBUTING.md) help just as much.
## Community
Questions, ideas, or want to show what you're building? Join us on Discord:
@@ -281,6 +287,7 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -325,6 +332,44 @@ finalize any in-progress assistant message.
{"type":"stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
| `name` | string | Display name (alias if set, otherwise `ws-xxxx`) |
| `state` | string | Current state (see state values above) |
@@ -654,21 +698,62 @@ Each skill summary:
---
### `POST /v1/api/send`
### `GET /v1/api/personas`
Returns the enabled personas offered by the workstream-creation pickers.
Authenticated for any logged-in user and deliberately gated by **no**
`persona.*` permission — selecting a persona at creation is a user
action, while the `persona.*` perms gate authoring. Display fields only;
the levers (base prompt, tool set, MCP/memory toggles) stay server-side.
**Response:**
```json
{
"personas":[
{"name":"engineer","display_name":"Engineer","description":"The stock interactive workstream: full tools, MCP, and memory.","applies_to_kinds":["interactive"],"is_default":true},
{"name":"researcher","display_name":"Researcher","description":"Answers questions with evidence — reads and cites, loads tools to verify when needed.","applies_to_kinds":["interactive"],"is_default":false}
Creates a new workstream. The server supports up to 10 concurrent workstreams.
The endpoint accepts **either**`application/json` (legacy shape) **or**
`multipart/form-data` when you want to upload attachments at creation
time. Multipart requests carry one `meta` field containing the JSON body
shown below plus zero-or-more `file` parts; each file is validated and
reserved onto the new workstream's first turn before the dispatch worker
runs, so queued multimodal turns cannot lose files to racing sends. If
validation fails the fresh workstream is rolled back so no orphan rows
leak.
**Request body:**
```json
@@ -857,6 +931,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
| `skill` | string | "" | Skill name. Applies content (system prompt), model, temperature, reasoning effort, max tokens, auto-approve policy, token budget, and other session config from the skill. Returns 400 if not found or disabled. Ignored when `resume_ws` is set (resumed sessions restore their own skill). |
| `persona` | string | "" | Persona slug. Resolved and snapshotted into the workstream at creation; empty selects the kind's default. |
| `judge_model` | string | "" | Optional model alias for the judge (overrides default judge model for this workstream) |
> **Skill behavior:** When `skill` is specified, the skill's content is injected as a system message and its session config fields (model, temperature, auto-approve, token budget, etc.) override system defaults for the new workstream.
@@ -873,6 +948,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `name` | string | Auto-generated workstream name |
| `resumed` | bool | Whether a previous session was successfully resumed |
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
| `initial_message_status` | string | Present ONLY when the workstream was created but its `initial_message` could not be delivered: `"queue_full"` (a raced live worker's interjection queue was at capacity — resend via `/send`; any uploads stay staged) or `"refused_closed"` (the workstream was closed mid-create). Absent whenever the message was dispatched. |
**Error (limit reached):**
@@ -884,20 +960,32 @@ Status code: `400`
---
### `POST /v1/api/workstreams/close`
### `POST /v1/api/workstreams/{ws_id}/close`
Closes and removes a workstream. The last remaining workstream cannot be
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 16 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -185,7 +190,6 @@ Phase 3: EXECUTE (parallel)
(cancel_event also checked per line — kills process group on cancel)
Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
call_id links tool_info items → streaming chunks → final result
For plan tool: post-execution gate via ui.on_plan_review()
```
### State Transitions
@@ -204,7 +208,7 @@ The engine emits state changes via `_emit_state()` which calls
"running" ---> tool execution
|
v
"attention" ---> waiting for user approval / plan review
"attention" ---> waiting for user approval
|
v
"running" ---> executing approved tools
@@ -226,11 +230,13 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 15
methods. Every frontend must implement all of them.
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
@@ -334,7 +334,7 @@ The console reverse-proxies each node's server UI at `/node/{node_id}/`. This al
### URL Rewriting
The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/shared/base.css`, etc.). Since `<base>` tags cannot rewrite root-relative URLs, the console uses a JS shim approach:
The server UI uses root-relative URLs (`/v1/api/workstreams/{ws_id}/send`, `/static/app.js`, `/shared/base.css`, etc.). Since `<base>` tags cannot rewrite root-relative URLs, the console uses a JS shim approach:
1. **HTML rewriting** — when serving `index.html`, replaces `href=` and `src=` references to both `/static/` and `/shared/` with the proxy prefix (`/node/{node_id}/static/` and `/node/{node_id}/shared/` respectively).
@@ -344,7 +344,7 @@ The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/share
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
SSE streams (`/v1/api/workstreams/{ws_id}/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Persona** — optional dropdown listing the enabled personas for the workstream kind. Sets the system-message composition and capability envelope at creation, snapshotted server-side; empty uses the kind's default. Picking one requires no `persona.*` permission.
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
@@ -396,10 +397,19 @@ The browser maintains a local `clusterState` object that mirrors the cluster sna
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 13 tabs (see also
[Governance](governance.md) for
the Roles, Policies, Skills, Usage, and Audit tabs, and
[Settings](settings.md) for the database-backed configuration editor):
and skill management with tabs that include Users, API Tokens, Channels,
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
| `output_warning` | Output guard flagged a tool result | `call_id`, `risk_level`, `flags` |
| `child_ws_created` | A direct child of this coord was just created (fan-out from the cluster bus) | `child_ws_id`, `node_id`, `name`, `parent_ws_id` (`ws_id` in the envelope is always the coord's own id) |
| `child_ws_state` | A direct child transitioned state | `child_ws_id`, `state` |
| `child_ws_closed` | A direct child closed | `child_ws_id` |
| `child_ws_rename` | A direct child's name changed | `child_ws_id`, `name` |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
Explicitly **not** in the coordinator set:
- `bash` / `edit_file` / `write_file` / `append_file` / `diff_file` — no local FS.
- `read_file` / `search` — no local FS reads.
- `web_fetch` / `web_search` — no direct web access.
- `task_agent` — sub-agent tool is zeroed on coord sessions.
- `recall` / `watch` / `read_resource` / `use_prompt` — UX / persistence tools that belong to interactive sessions. The dual-kind `memory` / `skills` / `notify` tools are available on both kinds (see the table above).
If your skill needs a coordinator to "run a command" or "read a
file", write the delegate pattern instead: spawn a child with an
appropriate skill, `wait_for_workstream`, then `inspect_workstream`
for the output. The coordinator stays the orchestrator.
---
## Framing differences
Interactive skills compose on top of `base_interactive.md` — a
"maker" framing: get the work done, use the tools, edit the code,
**Default** (no flag) — starts `server` and `console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
Open the dashboard at **https://localhost:8443**. It's served by Caddy with its
own local CA, so trust the root certificate once (or click through the browser
The dashboard is at **https://localhost:8443** (Caddy, same as the dev stack);
the console's HTTP port isn't published. For a real domain and a publicly
trusted cert, edit `turnstone/deploy/Caddyfile` to point Caddy at Let's Encrypt
(see [tls.md](tls.md)). Pin the image with `TURNSTONE_IMAGE_TAG` (default:
`latest`).
### mTLS
Layer the TLS overlay on the production stack to enable mutual TLS between
services. A bootstrap container creates a CA and every service auto-provisions
certs via the console's ACME endpoint:
```bash
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
See [tls.md](tls.md) for details.
## Configuration
All configuration is via environment variables in `.env` (copy from`.env.example`):
Everything is configured with environment variables in `.env` (copy from
[`.env.example`](../.env.example)). The dev stack needs none of them — they're
overrides.
### LLM Backend
### LLM backend
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | OpenAI-compatible API URL |
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | Bootstrap OpenAI-compatible API URL (real backends go in the UI) |
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TAVILY_API_KEY` | — | Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search) |
| `TURNSTONE_SEARXNG_URL` | `http://searxng:8080` | SearxNG URL for the `web_search` tool (local/vLLM models only; Anthropic/OpenAI use native search). Defaults to the bundled `searxng` service; set to an external instance's URL. To turn web search off, clear `tools.searxng_url` in the admin Settings tab. |
| `SEARXNG_IMAGE_TAG` | `latest` | Tag for the bundled `searxng/searxng` image |
| `MODEL` | — | Override the default model alias |
### Server
### Auth & database
| Variable | Default (dev / prod) | Description |
|----------|----------------------|-------------|
| `TURNSTONE_JWT_SECRET` | insecure default / **required** | JWT signing secret. Every service must share one value. |
| `POSTGRES_MAX_CONNECTIONS` | `300` | `max_connections` for the bundled Postgres |
> **Discovery needs a shared database.** Each server registers and heartbeats
> into a `services` table that the console polls. All services in these stacks
> point at the same PostgreSQL by default; SQLite-per-container can't see other
> containers.
> **Large clusters:** each process keeps a small pool (5 max). Beyond ~50 nodes,
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
publishes the console's ACME endpoint and SearxNG on localhost so a bare-metal
node can enroll its cert and run `web_search`. Everything else is reached through
Caddy or proxied by the console:
| Variable | Default | Description |
|----------|---------|-------------|
| `SERVER_PORT` | `8080` | Host port mapping |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tools |
| `CONSOLE_HTTPS_PORT` | `8443` | Host port for Caddy (dashboard HTTPS) |
| `SEARXNG_HTTPS_PORT` | `8444` | Host port for the SearxNG UI via Caddy (dev: localhost-only; prod: opt-in) |
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql://user:pass@db:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **Large clusters:** Each turnstone process maintains a small connection pool (5 max). At hundreds of nodes this adds up — use [PgBouncer](pgbouncer.md) in transaction pooling mode between turnstone and PostgreSQL.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
> You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass `--token --scopes read,write,approve` to also generate an initial API token.
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages to the server via HTTP. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
## Scaling
For multi-node testing, use the `cluster` profile which provides 10 server instances with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
The default `server` also runs alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
For production clusters beyond ~50 nodes, add PgBouncer between turnstone services and PostgreSQL. See [PgBouncer Connection Pooling](pgbouncer.md) for Docker Compose and Helm configuration.
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user`*(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata***or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static` → `oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
| `TURNSTONE_OIDC_ROLE_CLAIM` | No | — | ID token claim containing role/group values (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_ROLE_MAP` | No | — | Mapping from claim values to Turnstone role IDs (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | No | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Recommended when running behind a reverse proxy. When unset, derived from the request Host header. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
| `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK` | No | `false` | Allow the issuer (and its discovered endpoints) to resolve to private/internal addresses — needed for a self-hosted IdP on an internal network. See [Self-hosted and internal IdPs](#self-hosted-and-internal-idps). |
OIDC is enabled when all three required fields (issuer, client ID, client
secret) are non-empty. If any is missing, OIDC is silently disabled and
the login screen shows only the password form.
All four required fields — issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
is disabled at startup (an error is logged when only `redirect_base`
is missing) and the login screen shows only the password form.
### Reverse Proxy / Load Balancer
### Redirect base (required)
When Turnstone runs behind a reverse proxy, the internal `Host` header may
not match the externally-reachable URL. Set `TURNSTONE_OIDC_REDIRECT_BASE`
to the public origin so the redirect URI sent to the identity provider is
correct:
`TURNSTONE_OIDC_REDIRECT_BASE` pins the redirect URI sent to the identity
provider to a known externally-visible origin. Set it to the public origin
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
A **persona** is a named, reusable bundle attached to a workstream **at
creation** that controls how its system message is composed and what
capability envelope it runs with. Personas answer a recurring operational
complaint: the default composition primes every session for heavy tool use,
and there was no per-workstream dial to launch a "just write prose" or
"evidence-first research" session.
A persona is exactly four levers — no more:
| Lever | What it does |
|---|---|
| **Base prompt** | Replaces the BASE module of the composed system message. *Only* BASE: ENV, CONTEXT, TOOLS, and POLICIES keep composing, so mandatory [prompt policies](governance.md) ride on top of every persona. Built-in personas source their prose from a repo file; operator personas store it inline — see [Where persona prompts live](#where-persona-prompts-live). |
| **Tool visibility** | Which tools the session advertises. Tri-state: *unrestricted* (tracks tool growth and MCP catalogs), *no tools* (the TOOLS prompt block self-suppresses and zero definitions go on the wire), or an *exact set* of names. Including `tool_search` in a set makes it **soft** — tools the model discovers through search join the visible set; omitting it makes the set **hard** (the search pathway is disabled entirely). On commercial providers a soft set costs one prompt-cache re-prime per `tool_search` expansion, since each expansion rewrites the wire tool set and recomposes the prompt. |
| **MCP** | Whether the workstream talks to MCP at all. **Session-wide**: off means no MCP tools for the persona's own hands *or* for in-process task agents, no resource/prompt catalogs, and no listener registrations. This lever expresses infrastructure intent, not behavior shaping. |
| **Memory** | Whether the persona's **own hands** get memory: recalled-memory injection into the prompt, memory-directed metacognitive nudges, and the `memory` tool. Task agents keep their own envelope, and compaction spill/markers are session mechanics that are never persona-gated. An exact tool set that hides `memory` also mutes those nudges, and the compaction-resume pointer follows `recall`'s visibility. |
Visibility is behavior shaping, **not** a security boundary: any tool call
that does reach the wire still clears the same approval, judge, and policy
machinery as always. RBAC and tool policies remain the enforcement layers.
In `values.yaml`, point the database at PgBouncer:
@@ -197,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml``[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
- **Experimental** receives new features. May be rough around the edges.
- When experimental matures, it is promoted to stable. The previous stable branch stops receiving patches.
- **Stable** tracks receive bugfixes only. The most-recent stable minor
owns the `:stable` / `:latest` Docker tags and the default PyPI
install.
- **Experimental** (always on `main`) receives new features. May be
rough around the edges.
- When experimental matures, it is promoted to a new stable minor via
a `stable/X.Y` branch. One prior stable track is maintained alongside
the current one; at each promotion the oldest track is retired — its
branch is deleted, while its tags and released artifacts remain
available.
## Version Scheme
@@ -26,17 +34,17 @@ Turnstone uses two parallel release tracks published from a single PyPI package.
## Releasing an Experimental Version (from main)
```bash
scripts/release.sh 1.1.0a2 --push
scripts/release.sh 1.7.0a2 --push
```
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.1.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.7.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
## Releasing a Stable Patch (from stable/X.Y)
```bash
git checkout stable/1.0
git checkout stable/1.6
git cherry-pick <commit-hash> # bugfix from main
scripts/release.sh 1.0.2 --push
scripts/release.sh 1.6.1 --push
```
## Promoting Experimental to Stable
@@ -45,17 +53,19 @@ When `main` is ready for a stable release:
```bash
# 1. Tag the stable release on main
scripts/release.sh 1.1.0 --push
scripts/release.sh 1.6.0 --push
# 2. Create the stable maintenance branch from that tag
git branch stable/1.1 v1.1.0
git push origin stable/1.1
git branch stable/1.6 v1.6.0
git push origin stable/1.6
# 3. Start the next experimental cycle on main
scripts/release.sh 1.2.0a1 --push
scripts/release.sh 1.7.0a1 --push
```
The previous `stable/1.0` branch stops receiving patches at this point.
The previous stable branch continues to receive security-only patches;
the track before it is retired at each promotion (at 1.6.0:
`stable/1.5` stays maintained, `stable/1.4` is retired).
When a per-model override is `NULL` (empty in the UI), the global default is
used. Switching models via `/model <alias>` re-resolves sampling parameters
from the new model's overrides or global defaults.
**Removed settings:** `model.name` and `model.context_window` have been removed
from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Reasoning persistence (per-model)
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Task agent overrides
`task_agent` sub-sessions resolve independently from the conversation model
so operators can pick a cheaper/faster model for autonomous loops:
| Setting | Purpose |
|---------|---------|
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Falls back to `[model].agent_model` in config.toml, then the session's active model. |
| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". |
Both are live-editable from the Settings tab and take effect on the
next sub-agent invocation — no restart required.
---
## Bootstrap vs ConfigStore
**Bootstrap settings** are required before storage is available (database
@@ -49,17 +102,17 @@ connection, Redis, auth secrets, server bind address). These stay in
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
---
# Importing Conversation History into Turnstone
## Overview
Source formats vary; the destination does not. Your job is to translate whatever the user hands you (JSON dump, ZIP export, scraped HTML, screenshot OCR, raw transcript) into Turnstone's internal shape: **one workstream row** plus an ordered sequence of **conversation rows** in OpenAI message format. This skill documents the destination so you can write a correct mapper for any source.
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
## Turnstone Data Model (the destination)
Two tables carry the conversation:
### `workstreams` (one row per imported thread)
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
| `created`, `updated` | yes | ISO8601 strings. Use the source's first/last message timestamps when available. |
### `conversations` (many rows per thread, ordered by `id`/`timestamp`)
| Column | Notes |
|---|---|
| `ws_id` | The workstream this row belongs to. |
| `timestamp` | ISO8601 string. Preserve source timestamps; fall back to monotonically increasing values if unknown. **Order is canonical via `id` (autoincrement), not `timestamp`** — but always insert in conversational order so both agree. |
| `role` | One of `system`, `user`, `assistant`, `tool`, `developer`. See role mapping below. |
| `content` | Text. May be NULL for assistant rows that are *only* tool calls. |
| `tool_name` | Set on `role="tool"` rows (the tool whose result this is). NULL otherwise. |
| `tool_call_id` | Set on `role="tool"` rows (matches the assistant row's `tool_calls[].id`). NULL otherwise. |
| `tool_calls` | JSON-encoded list, on `role="assistant"` rows that issued tool calls. OpenAI shape — see "Tool Calls" below. |
| `provider_data` | JSON blob preserving provider-native content blocks (Anthropic `signature`, Gemini `thought_signature`, etc.). Optional; only matters for **resumable** imports against the same provider. Skip for archives. |
The internal format is **OpenAI-shaped**, even when the source was Anthropic or Gemini. Providers translate at their own API boundary; storage stays uniform.
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
Only useful for *Turnstone → Turnstone* re-parenting. Not relevant for foreign sources.
### 3. SDK `create_workstream(initial_message=...)` + `send()` per turn (last resort)
Only fits archives where the source had **no tool calls** and you don't care about preserving assistant turns verbatim. Each `send()` triggers a real LLM round-trip, which is expensive and rewrites assistant content. Don't use this for full history.
## Role Mapping
Common source-role conventions and how they map to Turnstone:
| `system` | `system` | Preserve only if it's content the user wrote (custom instructions). Drop boilerplate provider preambles — Turnstone composes its own system message. |
| `tool`, `function`, `tool_result` | `tool` | Must carry `tool_name` and `tool_call_id` matching the prior assistant row's `tool_calls[].id`. |
| `tool_use` (Anthropic) | `assistant` with `tool_calls` | Anthropic emits tool calls *inside* an assistant message; flatten to OpenAI shape. |
| `human_feedback`, `revision` | `user` | Treat as a follow-up user turn. |
## Tool Calls (the most error-prone part)
Turnstone stores tool calls in OpenAI's nested-function shape on the assistant row, and matches them with `role="tool"` result rows by `tool_call_id`.
### Assistant row with tool calls
```json
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "search_web",
"arguments": "{\"query\":\"turnstone import\"}"
}
}
]
}
```
`tool_calls[].function.arguments` is **a JSON-encoded string**, not an object. Source formats commonly get this wrong — Anthropic stores arguments as a parsed object, Gemini as a struct. Always re-serialize to a string.
### Tool result row
```json
{
"role": "tool",
"tool_name": "search_web",
"tool_call_id": "call_abc123",
"content": "..."
}
```
Pairing rules:
- Every assistant `tool_calls[].id` MUST be followed by exactly one `role="tool"` row with the matching `tool_call_id`, before the next user/assistant turn.
- If the source dropped the tool result (cut-off transcript), insert a synthetic `role="tool"` row with `content="[tool result missing in source]"` to keep the chain valid. An assistant row with an unanswered `tool_calls[].id` will break replay and any LLM round-trip.
- Multi-tool assistant turns: one `role="tool"` row per call, in any order, all before the next non-tool row.
### Tool ID generation
If the source used opaque tool IDs that aren't unique within a thread (some platforms reuse them), regenerate with a stable scheme like `f"call_{i}"` where `i` is a per-thread counter. Update both the assistant and tool rows together.
## Provider Fidelity (`provider_data`)
Skip this entirely for **archive** imports.
For **resumable** imports against the same provider, populate `provider_data` to preserve provider-specific tool-call metadata that the next API round-trip will require:
- **Anthropic**: `signature` field on thinking blocks; required for round-tripping extended-thinking responses.
- **Gemini**: `thought_signature` on tool calls; required for fidelity.
- **OpenAI**: typically nothing to preserve.
The runtime-side dict key is `_provider_content` (a list of provider-native blocks); the persisted column is `provider_data` (the same list, JSON-encoded). If you don't have provider-native blocks from the source — and you usually won't, because a foreign export won't include them — leave `provider_data` NULL. The first new turn will succeed without it, but the previous assistant turn's reasoning won't replay back to the model.
## Attachments
If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
Two import paths:
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
- **Don't import the source provider's system prompt verbatim.** Provider boilerplate ("You are Claude...", "You are ChatGPT...") will conflict with Turnstone's composed system message and confuse the model on resume. Drop it; preserve only user-authored custom instructions.
- **Don't preserve foreign tool definitions as Turnstone tools.** If the source had custom tools that don't exist in Turnstone, the assistant rows that called them are still valid history (archive), but the workstream is **not resumable** — mark `state="closed"`.
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
turnstone exposes 19 built-in tools plus any number of external MCP tools to the
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -22,7 +22,6 @@ schema plus turnstone-specific metadata keys:
"properties": { ... },
"required": ["param1"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "param1"
@@ -33,8 +32,7 @@ schema plus turnstone-specific metadata keys:
| Key | Type | Meaning |
|----------------|------|---------|
| `agent` | bool | Tool is available to plan/task sub-agents (read-only subset). |
| `task_agent` | bool | Tool is available to task sub-agents (broader subset). |
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
@@ -46,12 +44,10 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 19 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 19 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `TASK_AUTO_TOOLS`| Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
file-path and `attachment:` targets are local reads and run unprompted like
`read_file`
Note: The JSON schema metadata key `auto_approve` controls membership in
`AGENT_AUTO_TOOLS`/`TASK_AUTO_TOOLS` (used for agent sub-sessions). The actual
runtime approval behavior is determined by the `needs_approval` field set in
each `_prepare_*` method on `ChatSession`. These two mechanisms can differ.
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
approval behavior is determined by the `needs_approval` field set in each
`_prepare_*` method on `ChatSession`. These two mechanisms can differ.
---
@@ -165,12 +158,10 @@ Every tool defines a `primary_key`. The mapping is:
| `write_file` | `content` |
| `edit_file` | `old_string`|
| `search` | `query` |
| `math` | `code` |
| `man` | `page` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `task` | `prompt` |
| `plan` | `prompt` |
| `open_preview` | `target` |
| `task_agent` | `prompt` |
| `memory` | `name` |
| `recall` | `query` |
| `notify` | `message` |
@@ -194,7 +185,7 @@ Execute a bash command and return stdout + stderr.
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`). Environment variables containing secrets are scrubbed (`*_KEY`, `*_SECRET`, `*_TOKEN`, etc.).
- **Output format**: Stdout is returned directly. Stderr lines are prefixed with `[stderr]` so the model can distinguish them. When the command itself redirects stderr to stdout (`2>&1`), no prefix is added. Output exceeding 256KB is truncated (head + tail preserved, middle replaced with a truncation notice).
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only (not available to plan sub-agents).
- **Agent availability**: `task_agent` only.
---
@@ -212,7 +203,7 @@ base64-encoded image data for supported image formats.
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -268,7 +259,7 @@ Show a unified diff between two files, or between a file and a provided string.
- **What it does**: Returns unified diff output using Python's `difflib`. Binary files (containing null bytes) are rejected with a clear error. Files read through `diff_file` satisfy `edit_file`'s read guard — you can diff then edit without a separate `read_file` call. Large diffs are streamed with early cutoff at the tool truncation limit.
- **Auto-approve**: Yes (read-only).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -283,44 +274,12 @@ Search file contents for a regex pattern.
- **What it does**: Recursively searches for the pattern using `grep -rn`. Returns matching lines with file paths and line numbers.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
## Computation
### math
Execute Python code for math and computation in a sandbox.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `code` | string | yes | Python code to execute. Must use `print()` for output. |
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported. `pytest` is also available for import.
- **Installation**: `sympy`, `numpy`, `scipy`, and `pytest` require the `[sandbox]` extras group: `pip install turnstone[sandbox]` (included in `[all]`).
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
## Information
### man
Read a man page.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `page` | string | yes | The man page name (e.g. `grep`, `socket`, `printf`). |
- **What it does**: Returns the full formatted manual entry. Preferred over `bash('man ...')` or `web_search` for command/API documentation.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
### web_fetch
Fetch a URL and extract specific information from it.
@@ -330,9 +289,9 @@ Fetch a URL and extract specific information from it.
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -344,20 +303,91 @@ Search the web using a text query.
| `max_results` | integer | no | Max results to return (default 5, max 20). |
| `topic` | string | no | Search topic: `general`, `news`, or `finance` (default `general`). |
| `category` | string | no | Search category: `general` (default), `news`, `it` (code/tech), or `science`. Maps to SearxNG categories; the model picks per query. |
- **What it does**: Searches the web and returns ranked results with titles, URLs, and content snippets. Uses provider-native search when available:
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No Tavily key needed.
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No backend needed.
- **OpenAI search models** (`gpt-5-search-api`): Replaced with `web_search_options` parameter. The model always searches and returns `url_citation` annotations.
- **Local/vLLM models**: Falls back to the Tavily API. Requires `tavily_key` in `config.toml` or `$TAVILY_API_KEY`.
- **Local/vLLM models**: Falls back to a self-hosted [SearxNG](https://searxng.org) instance. Set `searxng_url` in `config.toml``[tools]` or `$TURNSTONE_SEARXNG_URL` (the docker-compose stack bundles a `searxng` service and points at it by default). Operators with a custom MCP search server can instead set `web_search_backend = "mcp:server:tool"`.
- **Auto-approve**: Yes (auto-approved for all tool dispatch paths).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
### Reranking (optional)
`web_search` can use an external **reranker** to re-order the backend's result pool by relevance to the query before returning the top hits. Turnstone runs no reranker model itself; it POSTs to a Cohere/Jina-compatible `/rerank` endpoint (self-hosted [vLLM](https://docs.vllm.ai) / [TEI](https://github.com/huggingface/text-embeddings-inference) / llama.cpp, or hosted Cohere/Jina/Voyage).
**Disabled by default.** In the console **Models** tab, add a model definition whose `base_url` is a Cohere/Jina-compatible `/rerank` endpoint and whose capabilities include `{"supports_rerank": true}`, then select it under **Models → Roles → Reranker**. It's managed like every other model (write-only key, enable/disable, calibration). The reranker is purely this per-model definition — there is no global `rerank_url`-style endpoint setting.
The `rerank_web_search` toggle defaults on once a reranker is selected. If the endpoint is unreachable or errors, web_search falls back silently to the backend's native result order — reranking never makes a search fail.
When `rerank_bm25` is enabled, the candidate text for memory, tool, and skill retrieval (memory name/description/content and tool/skill names + descriptions) is also sent to the rerank endpoint — a self-hosted endpoint (vLLM/TEI/llama.cpp) keeps it on your infrastructure, a hosted provider (Cohere/Jina/Voyage) sends it off-box.
**Serving a Qwen3-Reranker with vLLM.** The model is instruction-aware, so vLLM **must** apply its chat template — pass `--chat-template` explicitly. Without it the bare query produces near-random scores and reranking actively *hurts* retrieval (verified: an irrelevant passage outscored the correct one):
Then add a reranker model in the **Models** tab with `base_url``http://vllm:8000/rerank` (model name `qwen3-reranker`) and select it under **Models → Roles → Reranker**.
For an endpoint that does *not* apply the model's template, set `rerank_instruction` instead — Turnstone then wraps each query as `<Instruct>: {instruction}` / `<Query>: {query}` (Qwen3's own default is `Given a web search query, retrieve relevant passages that answer the query`). Use the chat template **or** the instruction, not both (they double-wrap).
**Picking `rerank_bm25_threshold`.** The relevance floor that gates proactive memory injection is a probability in `[0, 1]`, but the right value differs per model (a sharp 0.6B reranker may want ~0.95; a broader 4B ~0.33). Calibrate it against your endpoint:
```bash
turnstone-admin rerank-calibrate # probe the endpoint, recommend a floor
It reports the score scale, whether the endpoint cleanly separates relevant from irrelevant probes (a **"no clean separation"** result flags a mis-served or weak reranker), and the suggested floor. Leave the threshold at `0` to rerank-without-filtering.
---
### open_preview
Show the user rich content in a preview pane beside the conversation.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `target` | string | yes | An http(s) URL, a file path, or `attachment:<id>` for a file attached to the conversation. |
| `kind` | string | no | Rendering override: `web`, `pdf`, `image`, `table`, `text`, or `markdown`. Detected from the content when omitted. |
| `title` | string | no | Pane header title. Defaults to the page title, filename, or URL. |
- **What it does**: Resolves the target to bytes (URLs fetch through the same
SSRF-guarded path as `web_fetch`, screened per redirect hop, honoring the
same `tools.allow_private_network` opt-in), classifies the
content, stores it content-addressed against the workstream, and opens the
frontend preview pane beside the conversation: web pages render in a fully
sandboxed iframe (no scripts, opaque origin), PDFs in the browser viewer,
images inline, CSV/TSV/JSON as a sortable table, text/markdown rendered. A
previewed web page loads none of its remote images or styles by default, so
opening it never reveals the viewer to the page's site; a toggle in the pane
header turns remote content back on for that preview. The
model receives only a one-line confirmation — to reason about content, use
`web_fetch` / `read_file` instead. Preview content is size-capped per kind
(pages 4 MB, PDFs 32 MB, images 4 MB, tables 2 MB, text 512 KB) and GC'd
paths and `attachment:` targets run unprompted (local reads).
- **Agent availability**: interactive sessions only (not `task_agent`, not
coordinators).
- **Surfaces**: the pane renders in the web UI (standalone and console). The
CLI prints the confirmation line only — there is no terminal pane.
---
## Agent
### task
The tool name uses the `_agent` suffix — bare `task` collides with
chat-template channel names on some local models.
### task_agent
Delegate a general-purpose task to an autonomous sub-agent.
@@ -365,23 +395,9 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, math, man, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### plan
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<ws_id>.md` (unique per workstream, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Top-level only.
---
@@ -442,7 +458,7 @@ Provide either `username` for user-based targeting or `channel_type` +
- **What it does**: Sends a notification via the channel gateway's HTTP endpoint (`POST /v1/api/notify`). The server queries the `services` table for healthy channel gateways, authenticates with a service JWT (`aud: turnstone-channel`), and delivers to the first healthy gateway. On failure, retries up to 2 additional times with backoff (1s, 3s). Rate-limited to 5 notifications per turn (counter only increments on success).
- **Auto-approve**: Yes — notifications are time-sensitive and auto-approved so the model can alert users urgently.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
> See [Channel Integrations: Notifications](channels.md#notifications)
> for the full delivery flow, service registry details, and security
@@ -518,7 +534,7 @@ data.get("mergedAt") is not None
- Duplicate names rejected within the same workstream.
- **Auto-approve**: `create` requires approval; `list` and `cancel` are auto-approved.
- **Agent availability**: Main session only — not available to plan/task sub-agents.
- **Agent availability**: Main session only — not available to task sub-agents.
> See [Watch Architecture](diagrams/png/18-watch-architecture.png) for the
> full poll → evaluate → dispatch flow.
@@ -543,41 +559,39 @@ pre-configure skills at workstream creation.
- `load` — Activate a skill by name. Calls `set_skill()` which handles content
rendering with `{{model}}`/`{{ws_id}}`/`{{node_id}}` variables, system message
reinitialization, and config persistence. Returns the skill name, description,
and security scan tier. Warns on high/critical scan status.
and security risk level. Warns on high/critical risk level.
- `search` — Find available skills by query. Uses BM25 relevance ranking over
name, description, tags, and category (same `BM25Index` used by memory
relevance and tool search). Returns up to 10 results with name, description,
- **Task sub-agents** — via `self._task_tools` (merged list)
- **Plan sub-agents** — via `self._agent_tools` (merged list)
### Naming convention
@@ -755,34 +774,25 @@ MCP tools (3):
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through three mechanisms:
MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection.
manual refresh attempts reconnection. The console admin panel exposes the
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`,`_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
`_task_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
@@ -836,7 +846,7 @@ Use read_resource(uri='...') to access the resources listed above.
- **What it does**: Reads the resource from its MCP server via `MCPClientManager.read_resource_sync()`. Returns text content for text resources or base64-encoded data for binary resources. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (reads external data).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
### Capability guards
@@ -878,7 +888,7 @@ the `initialize` handshake. Each prompt is stored with its prefixed name
- **What it does**: Invokes an MCP prompt by name via `MCPClientManager.get_prompt_sync()`, expanding it into messages. Returns the expanded prompt content formatted as `[role]: content` blocks joined with blank lines. The prompt catalog is listed in the system message so the model knows which prompts are available. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (invokes external prompt servers).
- **Agent availability**: `agent` and `task_agent`.
"content":"# Search First\n\nBefore answering ANY question about where something lives in the codebase, you MUST call the `search` tool first. Never answer from memory."
},
"user_prompt":"Where is JWT token validation implemented in this project?",
"expected_actions":[{"tool":"search"}],
"match_mode":"ordered_subset",
"max_turns":4
},
{
"id":"test-after-edit",
"skill":{
"name":"test-after-edit",
"content":"# Test After Edit\n\nAfter editing or writing ANY file, you MUST run the test suite with `python -m pytest` via bash before you finish. Do not report done until tests have run."
},
"user_prompt":"Add a function `clamp(x, lo, hi)` that clamps x to [lo, hi] in utils.py.",
"content":"# Changelog Discipline\n\nWhenever you modify a file, you MUST also append a one-line entry to CHANGELOG.md describing the change in the same task."
},
"user_prompt":"Fix the off-by-one so pager.py shows the last page. Edit pager.py.",
"setup":{
"files":{
"pager.py":"def last_page(total_items, per_page):\n # off-by-one: drops the final partial page\n return total_items // per_page\n",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Patrick Buckley
