- rerank_config.py: the runtime instruction fallback had a dead tail
(`get_rerank_instruction() or str(cs.get(...))` -- the cs.get term can only
return the registry default ""), via a stored_keys() branch that also diverged
from the calibrate CLI / endpoint. Collapse to the sibling idiom
(`cs.get(...) or get_rerank_instruction()`) so the instruction used at
calibration time matches the one used at runtime. Correct the module docstring:
ChatSession is the sole caller (the CLI/endpoint share only the instruction
precedence, not this function).
- session.py: the deferred first-turn memory recompose was gated on the flag
alone, so a synthetic wake send (empty user content -> flag stays False) re-ran
the full compose on every wake before the first real turn. Gate on a non-empty
query too, so wakes don't re-pay it and the real turn still fires exactly once
(+ test).
Accepted as-is: the __init__ compose (kept so system_messages/_agent_system_messages
are valid for early readers; one cheap extra compose per fresh session) and the
orphaned tools.rerank_* config rows (inert -- no read path, never listed or
redacted; a purge migration would collide with the 060 in flight on another branch).
Proactive memory selection scores candidates against the recent-user-message
query (extract_recent_context), but a fresh session composes the system prefix
once in __init__ while self.messages is still empty. That empty query takes the
no-context path: _select_memory_candidates returns recency order and
score_memories returns memories[:k] verbatim -- the 5 most recently UPDATED
memories, with BM25 and the reranker never invoked. send() never recomposes, so
those recency-only memories are what the model sees for the whole session (until
an unrelated event -- skill / MCP / model refresh / resume / memory write /
command -- happens to rebuild the prefix). Net effect: the injected memories are
unrelated to the actual question.
Fix: defer the memory-bearing compose to the first real user turn.
- Track _system_composed_with_context, set once extract_recent_context is
non-empty in _init_system_messages.
- send() recomposes once, right after _append_user_turn, while the flag is still
False -- so the opening turn's memory block is selected (and reranked) against
the real message. The flag then stays True, so the prefix is composed once and
stays cache-stable exactly as before (no per-turn prompt-cache churn).
This is the targeted fix; per-turn memory refresh (so later topic shifts also
re-rank) is the larger tail-injection redesign tracked on another branch. Two
adjacent gaps are left as-is for now: the reranker/BM25 only see content[:200],
and build_memory_context flat-truncates each memory to 500 chars (max_content is
the save cap, not an injection budget).
Tests: flag is False on a fresh session and after a whitespace-only wake turn,
flips True on a real query; send() runs the deferred recompose with the user
message in the query.
The reranker_alias -> model-definition path (added when reranking became a model
role) made the older global endpoint settings redundant. Resolve reranking
solely through the Reranker role and remove the parallel global config.
- Removed settings tools.rerank_url / rerank_model / rerank_api_key, their
config.py getters (+ $TURNSTONE_RERANK_URL / $TURNSTONE_RERANK_MODEL and the
module caches), and the fallback branch in resolve_rerank_client_from. The
resolver now returns a client only when a Reranker model (capability
supports_rerank, base_url = its /rerank endpoint) is selected, else None.
- Kept as global knobs: reranker_alias (the selector), rerank_web_search,
rerank_bm25, rerank_bm25_threshold, and rerank_instruction -- a task-level
query knob (Qwen3-style), not endpoint identity.
- The Settings tab is registry-driven, so the three fields disappear with their
SettingDefs. Updated the Reranker role help, example config, and docs/tools.md.
BREAKING: a reranker configured via [tools] rerank_url (config.toml / env /
Settings tab) no longer works -- add the reranker in the admin Models tab and
pick it under Models -> Roles -> Reranker. No migration: reranking is days old
and disabled by default, so any orphaned tools.rerank_* config rows are inert.
Tests: the resolver covers no-store / no-alias / non-rerank-alias -> None and the
model-definition happy path; the obsolete global-fallback tests are removed.
Adding a reranker model through the admin modal was impossible: Detect
(admin_detect_model) always ran the OpenAI /v1/models probe first and gated
calibration on its `reachable` result. A Cohere/Jina /rerank endpoint can't
answer /v1/models, so it failed both ways -- `$host/v1` passed the probe but
calibration POSTed to the wrong path, `$host/rerank` 404'd the probe outright.
- console/server.py: branch on `supports_rerank` BEFORE the probe and calibrate
the endpoint directly; that round-trip IS the reachability check (there is no
independent /v1/models signal for a rerank-only endpoint). Success
autopopulates the three calibration fields the way context_window does;
calibration failure -> reachable:False + error; empty base_url -> 400 with the
/rerank hint; reachable-but-no-clean-separation -> a note. Drops the now-dead
post-probe calibrate-on-detect block.
Reranker selection stays per-model (reranker_alias -> registry); recalibrate on
a saved reranker was already correct (it calibrates directly). Flagging a model
as a reranker still rides the capabilities JSON (supports_rerank).
Negative-tested: rerank detect skips the probe, autopopulates on success, notes
no-separation, reports unreachable on calibration failure, and 400s on an empty
base_url; the non-rerank detect path is unchanged.
Phase 3. Stores reranker calibration per-model on the model definition's
capabilities (rerank_threshold/rerank_scale/rerank_separated; a non-empty
rerank_scale is the "has been calibrated" marker) instead of a single global
threshold, populated automatically when a reranker endpoint is detected.
- ChatSession._bm25_rerank_threshold precedence: the active reranker model's
calibrated rerank_threshold (when separated) wins; calibrated-but-not-
separated -> 0 (no floor); else the global tools.rerank_bm25_threshold
fallback. Reads the raw caps dict, in-memory per turn.
- Detect (admin_detect_model) calibrates a supports_rerank endpoint and
autopopulates the three fields like context window; the create-model UI shows
a verdict chip (calibrated / no-clean-separation / not-calibrated).
- POST /api/admin/model-definitions/{id}/calibrate backs the Re-calibrate
button; calibration runs off the event loop (run_in_executor, bounded by
asyncio.timeout(90)), persists the fields + refreshes the registry, and is
graceful on a down/slow endpoint (never 500). Shared
merge_calibration_into_caps helper used by the endpoint and the CLI.
- turnstone-admin rerank-calibrate is now per-model: --model <alias> required;
--apply writes that model's caps (not the global setting). A no-separation
result records the marker (calibrated, no floor) consistently across CLI and
endpoint.
The serving lesson stays documented: Qwen3-Reranker needs vLLM --chat-template
or its scores are near-random (live-validated 0.6B + 4B: the calibrated floor
came out 0.95 vs 0.33 for the same task -- why per-model calibration exists).
Negative-tested: the floor-precedence branches (calibrated+separated -> per-
model, calibrated+!separated -> 0, uncalibrated -> global, no-alias -> global,
registry-must-not-be-consulted), endpoint persist/refresh + graceful failure,
detect-skips-non-rerankers, the CLI per-model write, and the caps-merge
preserving supports_rerank. Chip states verified via headless Chrome.
Phase 2 of BM25 reranking (follows #627). Makes the rerank_bm25_threshold floor
usable across reranker models and adds tooling to pick it.
- normalize_scores (rerank.py): map a rerank batch into a 0-1 relevance
probability -- sigmoid when any score falls outside [0,1] (logit endpoints
like bge/TEI), identity otherwise (Cohere/Jina/Qwen already 0-1). Applied in
the _bm25_reranker closure AND calibration so the threshold means the same on
every endpoint. Monotonic, so ranking order is unchanged.
- rerank_calibrate.py + `turnstone-admin rerank-calibrate [--apply]`: probe the
endpoint with labelled relevant/irrelevant groups, normalise, and recommend a
recall-biased floor -- or report "no clean separation" (a mis-served/weak
reranker). A warmup loop absorbs a cold endpoint's first-request compile so
calibration doesn't time out. Validated live against Qwen3-Reranker 0.6B and
4B: the calibrated floor differs sharply per model (~0.95 vs ~0.33 for the
same task) -- exactly why per-endpoint calibration exists.
- rerank_config.py: extract resolve_rerank_client_from(config_store, registry);
the alias/url precedence now lives in one place, shared by ChatSession (which
delegates) and the CLI.
- tools.rerank_instruction (config + setting + client): wrap the query as
<Instruct>:/<Query>: for instruction-aware rerankers (Qwen3) on endpoints that
don't apply the model's own chat template. Docs note the critical vLLM serving
detail: Qwen3-Reranker needs --chat-template or its scores are near-random and
reranking hurts retrieval.
Negative-tested: normalize sigmoid/identity branches, closure-normalises-before-
floor, calibration separation/recall-bias/warmup-absorbs-cold-start, the CLI
apply/no-apply/no-separation paths, and instruction query-wrapping through the
real httpx boundary.
Reuse the shipped Cohere/Jina rerank client as an optional post-process on
the BM25 surfaces (tool search, skill search, memory composition) via one
seam: BM25Index gains an injected reranker + a two-stage search (BM25 recall
top-50 -> rerank -> top-k). No new storage.
Gated on a configured endpoint plus tools.rerank_bm25 (default on, matching
rerank_web_search). tools.rerank_bm25_threshold (default 0.0 = off) is a
relevance FLOOR for proactive memory surfacing: BM25 always returns something,
so without a floor every-turn memory injection spends tokens on the top-k of
whatever lexically matched; the reranker score is what makes a meaningful
"inject nothing" gate possible.
Two reranker modes (BM25Index rerank_filters):
- REORDER (reactive tool/skill search): the reranker must never drop results
-> fall back to BM25 order on empty, backfill omitted pool items, so a
misbehaving endpoint can't silently lose tools.
- FILTER (memory, rerank_filters = threshold > 0): a clean empty/short result
is honoured (inject nothing) -- a deliberate divergence from
web_search._rerank_results.
Parse/endpoint failure is a discrete branch from the floor: an empty result
for non-empty input means an unparseable response (a conforming reranker
scores every doc), so the closure raises RerankError and BM25Index falls back
to BM25 order in BOTH modes -- the floor only acts on valid scores.
Also: cap the rerank client timeout at 15s (the per-turn memory path can't
afford tools.timeout's 120s default); move the Reranker alias to rerank.py
(shared, no import cycle); document the endpoint egress in the rerank_bm25
help, the admin Reranker-role description, and docs/tools.md; add
scripts/bench_bm25_rerank.py (manual, needs a live endpoint) to measure
precision@k/MRR lift and recommend a threshold default.
Negative-tested: reorder fallback-on-empty and omitted-item backfill,
filter-mode honor-empty, singleton-still-floored, the parse-fail RerankError
raise, the >= floor boundary, and pool-position-to-doc-index mapping -- each
guard reverted to confirm its test fails, then restored.
list()-materialize the reranker's output inside the guarded block so a
None / non-iterable / lazily-raising reranker falls back to native order
instead of raising out of web_search, and reject bool indices (an int
subclass) the same way _parse_hits already does.
Drop leftover web_fetch references from the rerank settings and Reranker
role help (reranking is wired into web_search only), and document both
endpoint paths (tools.reranker_alias and tools.rerank_url).
Reranking is delegated to an external Cohere/Jina-compatible /rerank endpoint
(self-hosted vLLM/TEI/llama.cpp, or hosted Cohere/Jina/Voyage); Turnstone runs
no reranker model itself. Disabled until an endpoint is configured.
- core/rerank.py: CohereJinaRerankClient (tolerant of results-wrapped and
bare-list responses) + resolver.
- web_search: rerank the SearxNG result pool by query relevance before top-k,
with a native-order fallback on error; answers/infoboxes untouched.
- Reranker as a model definition: add a model with the supports_rerank
capability and pick it under Models -> Roles -> Reranker
(tools.reranker_alias); takes precedence over the tools.rerank_url settings.
Settings: tools.rerank_url/model/api_key, tools.rerank_web_search,
tools.reranker_alias. Docs: docs/tools.md, turnstone.example.toml.
(web_fetch reranking was evaluated and dropped: for single-document chunk
selection it did not reliably beat head-truncation. Reranking is reserved for
multi-item ranking.)
`man` and `math` duplicated capabilities already reachable through
`bash`; `plan_agent` is better expressed as a `task_agent` running a
planning skill, and carried a large amount of special-case machinery
(plan-review gate, refinement loop, per-kind model routing). Removing
all three shrinks the tool surface and cuts per-call token cost.
Also removed, as dead-once-the-tools-are-gone:
- the `math` sandbox executor (`turnstone.core.sandbox`) and its
`[sandbox]` extra; the eval analyst now runs bash-only
- the read-only `AGENT_TOOLS` sub-agent tool set and the `agent`
tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained)
- the plan-review protocol end to end: the `on_plan_review` UI hook,
`resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`,
the `plan_review`/`plan_resolved` SSE events, and their Python SDK /
TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings
- the `model.plan_alias` / `model.plan_effort` settings and the
registry `plan_model` / `plan_effort` routing fields
TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged.
BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the
plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings
from the experimental 1.6 line.
Rename the web_search tool's `topic` parameter to `category` and expand the
enum to general/news/it/science, mapped to SearxNG `categories=`. The model can
now target the right corpus per query (e.g. `it` for code, `science` for
papers) — useful when generic engines rate-limit. The Tavily-era `finance`
topic (no SearxNG equivalent) is dropped. Threaded consistently through
_prepare_web_search / _exec_web_search / both search clients.
BREAKING: the web_search `topic` argument is now `category`.
The SearxNG change reworded _tool_write_compose's duplicate-skip return and
dropped the "identical content" phrase that test_identical_content_skipped
asserts on, failing CI (which then fail-fast-cancelled the parallel matrix
jobs). Restore the phrase (now covering all three bundled files) and assert
the searxng/settings.yml extraction in test_writes_compose_file.
Drop the Tavily and DuckDuckGo (ddgs) web_search backends for a single
self-hosted SearxNG service bundled into the docker-compose stacks.
Core:
- New SearXNGClient + _format_searxng; rewrite resolve_web_search_client to
(backend, searxng_url, searxng_engines, ...). MCP backend + oauth_user guard
unchanged. _resolve_search_client follows storage -> toml -> env -> default
precedence (explicit "" disables, via ConfigStore.stored_keys()).
- Drop the Tavily-era topic=finance (no SearxNG category); topic is now
general/news.
Settings/config:
- Remove tools.tavily_api_key, get_tavily_key, $TAVILY_API_KEY, [api].tavily_key.
- Add tools.searxng_url (default http://searxng:8080) + tools.searxng_engines,
with get_searxng_url/get_searxng_engines.
Compose + bundled config:
- Internal-only searxng service (no published API port, :ro config, /healthz
healthcheck, persistent searxng-cache volume) in both stacks; bundle
turnstone/deploy/searxng/settings.yml (JSON output on, limiter off).
- Caddy serves the SearxNG web UI on :8444 (dev: localhost-only; prod: opt-in).
- bootstrap extractor + wheel packaging updated.
Deps: drop the ddg extra + ddgs mypy override (regenerates uv.lock, removing the
lxml/h2/brotli transitives).
Docs: tools/docker/architecture/openshell + diagrams + config example + CHANGELOG;
docs/docker.md carries the AGPL-3.0 §13 operator note.
BREAKING: tools.web_search_backend no longer accepts "tavily"/"ddg";
tools.tavily_api_key and the ddg extra are removed. Run the bundled SearxNG (ships
in the compose stacks) or set TURNSTONE_SEARXNG_URL to an external instance.
Closes#545
run.sh autodetects Ubuntu/Debian, Fedora/RHEL, Arch, and WSL; ensures git and
Docker; clones, builds, picks free ports (Caddy prefers 443, Postgres 5432),
generates a .env with a JWT secret and Postgres password, asks how many nodes to
run, and starts the stack. The node count persists via an auto-loaded
compose.override.yaml, so a later plain `docker compose up -d` keeps it; a fresh
clone with no override still starts all 10. Ignores the generated override.
Lead with privacy, local-first, and no-telemetry; demote governance to an
optional team-controls line. Fix the dashboard URL (Caddy on 8443), add the
one-line installer, and link the Discord community.
The bundled production compose mounts ./Caddyfile, so write_compose has to write
it alongside compose.yaml or `docker compose up` fails to start Caddy. Update
the wizard's system prompt for the new model (no profiles, Caddy-fronted
dashboard, current ports).
Without a Discord/Slack token the channel gateway exited, which crash-loops
under `restart: unless-stopped`. Run the HTTP server and service heartbeat with
zero adapters instead — registered and idle — until a token is set. The Slack
token-pair mismatch stays a hard error.
A node refused to start with no model configured and no LLM reachable, so a
fresh cluster couldn't come up to be configured. load_model_registry now
accepts allow_empty and returns an empty registry; ModelRegistry permits the
empty state (default unset); the server passes allow_empty so a node registers
and shows in the console, then picks up models added in the admin UI live. The
CLI keeps failing fast — a REPL with no model is unusable.
Migrations run on every node at boot, and one migration rebuilds an index with
CREATE INDEX CONCURRENTLY, which can't run in a transaction and waits for all
concurrent transactions to drain. The advisory lock that serialises migrations
was held inside an open transaction, so the lock-holder's own idle-in-
transaction connection deadlocked the concurrent build when several nodes
started together. Acquire the lock on an AUTOCOMMIT connection and poll
pg_try_advisory_lock so no waiter pins a snapshot. Adds a Postgres concurrency
regression test (skipped on SQLite).
`docker compose up` from a clone builds one image and brings up the whole stack
— PostgreSQL, console, Caddy, channel, and 10 server nodes — sharing one
Postgres so the console discovers every node. The dashboard is reachable only
through Caddy (HTTP/2 avoids the browser's 6-connection cap on the dashboard's
SSE streams); the console's plain-HTTP port is no longer published. Postgres
binds 127.0.0.1 so a bare-metal turnstone-server can join the cluster — the
bare-metal overlay is folded in and removed. Insecure dev defaults keep it
zero-config; the bundled production stack mirrors the shape but pulls ghcr
images and requires real secrets.
Move the Caddyfile under turnstone/deploy so it ships in the wheel; update docs,
QUICKSTART, and .env.example to match.
The guard tests were appended via heredoc, bypassing the editor's
auto-format; ruff-format collapses one wrapped call onto a single line.
No behaviour change.
The intent-validation judge runs before the approval gate resolves, and
Smart Approvals (judge.smart_approvals) parks approve_tools on the async LLM
verdict for up to judge.timeout — so the tool-call card never reached the UI
until the judge had ruled. An operator could not see a committed call, let
alone Stop it, during that window.
approve_tools now emits a tool_pending event carrying the serialized batch at
the top of the gate, before the tool-policy lookup, the verdict wait, and the
human prompt. It is a UI paint only — no persistence, audit, or verdict
bookkeeping — so it cannot perturb the gate's accounting. The authoritative
tool_info / approve_request / tool_result events that follow upgrade the same
construct in place, keyed by call_id, and the Last-Event-ID replay slice
reconstructs it on reconnect. A ToolPendingEvent joins the SDK registry.
Coordinator: appendToolBatch was already idempotent on call_ids; the new
handler reuses the --running placeholder it already upgrades, with an
"Evaluating" kicker that swaps to "Running" on the auto-approve upgrade.
Interactive: showInlineToolBlock was create-only, so a second card would
duplicate. Added announceToolBlock + _takeAnnouncedBlock to reuse the
announced shell (matched on its call_id set) instead. The announced rail is
dashed amber and must out-specify the .msg.ts-approval--inline cyan-hold
(specificity 0,2,0) — at 0,1,0 it rendered cyan, indistinguishable from a
normal card — so the announced card is the one visually distinct surface in
the stream.
Screen-reader parity: the early paint announces politely through dedicated
off-screen aria-live regions on both surfaces (the messages log is
aria-live=off mid-stream, so the appended shell alone is inaudible), and the
announced shell carries aria-busy until the upgrade clears it. Polite, not
assertive — the human gate keeps its assertive announcement.
Tests cover the gate ordering (tool_pending precedes tool_info and the Smart
Approvals gate) plus string-guards on both UIs' wiring, the announced-rail
specificity, and the screen-reader regions.
risk_level is server-supplied and was interpolated straight into className
and data-risk at three sites — updateVerdictBadge, _buildOutputWarningEl,
renderVerdictBadge — as `risk_level || "medium"`. Whitespace, a stray case,
or a future relaxed-validation value would pass into the class string and
silently break the selectors that updateVerdictBadge, toggleVerdictDetail,
and the d-key handler rely on. It is not an XSS vector (className assignment
is text-typed), but a broken selector is a real failure.
Funnel all three through a normalizeRiskLevel() chokepoint backed by a
{low, medium, high, critical} allowlist; unknown or blank falls back to the
neutral "medium" default. Pre-existing; surfaced while rendering verdict
badges from the early-paint path.
The floor blocks only explicit heuristic deny/critical verdicts — it is
not a general "never lower the heuristic" rule. The heuristic default for
an unmatched tool is `review`, and letting a confident LLM `approve`
upgrade a `review` is the feature's purpose. Matches the implementation
and addresses PR review feedback.
Opt-in judge.smart_approvals (default off): when the intent-validation
LLM judge returns a high-confidence "approve" verdict, the tool batch is
approved automatically with no operator prompt. review/deny recommendations,
low confidence, judge errors (llm_fallback), and a deterministic heuristic
deny/critical finding all still require a human. Requires judge.enabled.
- Batch-atomic: a parallel tool batch auto-approves only if every call
qualifies; one non-qualifying call holds the whole batch for a human.
- Gate: tier==llm + recommendation==approve + confidence >=
judge.confidence_threshold (default raised 0.7 -> 0.95), with a floor
that never clears an explicit heuristic deny/critical verdict.
- approve_tools waits for the async LLM verdicts, finalises the audit
trail (AutoApproveReason.smart_approval), and re-emits verdicts after
the card so the live chip updates; the auto-approved row renders the
LLM verdict rather than the cautious heuristic carry-over.
- judge: always deliver exactly one verdict per call (fallback on error);
reject non-finite confidence so NaN can't clear the bar.
- Drop verdicts from a superseded judge generation so a reused call_id
from a prior turn's still-running daemon can't satisfy the gate's wait.
Config plumbed through the server/console/CLI builders and the live
_judge_cfg; admin Judge tab renders the toggle. Docs + example config
updated. ~35 tests covering the gate matrix, batch-atomicity, the
heuristic floor, audit stamping, the streaming re-emit, NaN/duplicate-id
defenses, and the cross-turn generation guard.
- Surface the degraded state in each node item's aria-label (only
unreachable was included), so screen readers announce it alongside
the visible DEGRADED word.
- Give the trigger an initial aria-label="Nodes" so it isn't an unnamed
control before the first snapshot render populates it; renderNodePicker
overwrites it with the live count/version once data arrives.
The trigger only showed a border on hover/open, so at rest it read as
plain text rather than a control. Add a persistent recessed box (subtle
fill + border) so it's visibly clickable, and brighten the border to
accent on hover and when open.
The always-visible NODES table dominated the coordinator-first landing
page for information most users glance at rarely. Replace it with a
compact node picker in the cluster status bar: the rightmost segment
shows "N nodes" + the cluster version (or a DRIFT chip on mixed
versions), and clicking it opens a dropdown of every compute node with
its live workstream count. Selecting a node navigates to /node/{id}/ —
the same destination the table rows linked to.
The picker reads the same /v1/api/cluster/snapshot + SSE data the table
did (via the retained buildNodeInfoFromSnapshot), so no backend change
was needed; the table was a pure client-side render. The node-grouping
/ prefix-collapsing JS and all the table CSS are removed.
Accessibility / design:
- Status is encoded by shape and colour (round = healthy, diamond =
degraded, square = unreachable), mirroring the .csb-state-dot
vocabulary, plus a spelled-out DEGRADED/DOWN word — colour alone
fails at 7px for color-blind users.
- DRIFT renders as a solid amber chip (dark text on fill) so it reads
as a real alert rather than yellow-on-yellow.
- role="menu"/menuitem (navigation, not selection), aria-haspopup,
aria-expanded; Escape / outside-click / Arrow / Home / End handling;
full node id surfaced via title when the name ellipsizes; menu height
clamped to the viewport so a long list never touches the top edge.
tests/test_console.py: assert the picker markup is served and the old
table markup (#view-overview / #node-table) stays gone.
Address PR review threads:
- _on_renewed now wraps the client-context hot-reload in try/except, so a
load_cert_chain failure can't abort the renewal callback before the
frontend-bundle update (matching the node-side reload hook).
- The startup gc_expired_certs() sweep is contained like the periodic one,
so a malformed legacy cert row can't abort the TLS block and skip the
proxy/collector mTLS client setup that follows it.
Enabling mTLS broke the cluster in three layered ways:
- Service certs were keyed on socket.gethostname() (the container ID) and
never carried the advertised service name as a SAN, so every collector and
routing-proxy handshake failed the hostname check. build_cert_hostnames()
now puts the advertised host first: it becomes the cert's primary domain
(hence a SAN) and a stable store key that survives container recreation.
- lacme's RenewalManager renews everything in the store; with the store shared
cluster-wide, every node renewed every other node's (and every dead
container's) cert — an N×M renewal storm. _SingleDomainStore scopes each
node's sweep to its own cert, and the console adds a periodic GC for the
certs of long-departed nodes.
- uvicorn loads its cert once at boot and never reloads, so renewed certs
never reached the listener and the served cert expired mid-process.
swap_context_cert() hot-swaps renewed material into the live SSL context
(server listener and console client context) via load_cert_chain.
Observability and browser access:
- The collector logged connection/TLS failures at DEBUG, so a persistent
mTLS-verify failure was invisible. It now logs the first failure per node
(reachable->unreachable) at WARNING and stays at DEBUG on retries.
- The console serves plain HTTP (it is the ACME bootstrap endpoint) and no
longer rewrites its advertised URL to https://. Browser->console TLS is
terminated by a reverse proxy: the cluster profile gains a caddy service
(browser h2/HTTPS -> caddy -> console h1.1/HTTP) plus browser-TLS docs.
Tests: tests/test_tls_san_renewal.py, tests/test_collector_reachability.py.
* feat(audio): voice I/O — speech-to-text + text-to-speech via model roles
Browser voice input/output over the OpenAI audio wire protocol, selected
through the existing model-roles system so the same code path serves OpenAI,
vLLM/vLLM-Omni, or any compatible backend — pure registry config, no new
in-process deps. Anthropic has no audio API, so it is capability-gated out of
the audio roles while remaining valid as the agent model.
Backend
- core/audio.py: role resolution + capability gating + transcribe()/synthesize()
over a registry-resolved client. Typed AudioUnavailableError (503) /
AudioBackendError (502 — body masked, SDK detail logged). Optional STT prompt.
- Endpoints POST /v1/api/workstreams/{ws_id}/speech-to-text and POST /v1/api/tts,
registered in v1_routes, write-scoped (direct + proxied), offloaded with
asyncio.to_thread. Silence -> 422; configured-but-failed backend -> masked 502.
- Model roles: audio.stt_model_alias / audio.tts_model_alias / audio.tts_voice /
audio.stt_prompt settings; Models -> Roles entries (capability-gated dropdowns,
"(disabled — voice off)" when unset). /v1/api/models exposes resolved
stt_default_alias / tts_default_alias + per-model capabilities.
- Capabilities: supports_transcription / supports_speech_synthesis on
ModelCapabilities; current OpenAI audio lineup (whisper-1, gpt-4o[-mini]-
transcribe, tts-1[-hd], gpt-4o-mini-tts) registered as known models, with a
name-inference backstop for local/openai-compatible aliases.
Frontend (interactive UI)
- Mic dictation (record -> transcribe -> fill composer for review) and
per-message playback, shown only when the role is configured.
- CSS-mask icon set, aria-pressed + live-region announcements, recording timer,
reduced-motion cue, error-typed toasts + persistent denial, mic disabled while
busy, code/math stripped before TTS.
Tests: new test_audio.py plus STT/TTS endpoint, settings, openapi, available-
models, and OpenAI-lineup capability coverage. ruff + mypy + node --check clean.
* fix(audio): use const for AUDIO_MODEL_HINTS (var-sweep invariant)
Saved Workstreams / Saved Coordinators (shared createSavedTable):
- Re-add the pagination retired by #611, capped at 20 rows/page, in the
shared component so both surfaces stay consistent. The list is fetched
whole and sliced client-side; the delete controller only sees the visible
page so Select-All stays bounded. Page resets on filter/sort, clamps on
shrink, hides on a single page or in delete mode.
- Footer is range-aware ("Showing 1-20 of N"); footer + pager share one
justified row (range left, pager right) so they read as one region.
- Saved rows carry the pointer cursor in the shared cards.css. The console
only set it on .dash-row.has-link, which the shared row builder never adds,
so saved-coordinator rows had fallen back to the default cursor.
Active Coordinators (console home):
- Give the active-coordinators block the full card chrome matching the
server's Workstreams block: a dash-header bar with an "N active / M total"
summary, the shared dash-colheaders band (was missing entirely), the rows,
and a dash-footer count line.
- Share .dash-footer into base.css (was server-only); the server keeps its
bottom-margin override.
- Make both coordinator cards contiguous by dropping the console-only
home-section gap, matching the server which ships both cards contiguous.
Frontend only -- no API, DTO, or migration changes. Pagination logic
covered by a DOM-stub harness; two designer passes applied.
make_history_handler is shared by interactive and coord, so coord
/history already trims the executing in-flight orphan turn and returns
a cursor. But coordinator.js never read it -- it connected fresh, so the
trimmed turn was neither in /history nor delta-replayed and vanished
from the dashboard (a regression vs the prior #610 in-flight render).
Mirror the ui/static/app.js fix in coordinator.js: refetchHistory takes
a seedCursor flag (default false) and seeds lastEventId from hist.cursor
only on the initial-connect path; connectSSE gates ?last_event_id= on
!= null so a cursor of 0 isn't dropped. The clear_ui / replay_truncated
re-render callers leave seedCursor false (they run on a live stream and
must not rewind the live cursor). Adds a coordinator.js static guard.
When the active UI is a MagicMock test double, _ui_event_id() returned
the auto-vivified _event_id mock (getattr finds it, so the None default
never applies). That mock reached the conversations INSERT and failed
to bind ("type 'MagicMock' is not supported"), so save_message raised,
the row was dropped, and tests on the real-storage + mock-UI path broke
(CI: test_session_attachments::test_db_row_stores_text_only).
Coerce a non-int _event_id to None so mock UIs -- and counterless
CLI/eval/placeholder UIs -- stamp NULL (the synthetic-snapshot floor),
matching the documented contract. Production UIs always carry an int,
so behaviour there is unchanged.
Also drop two redundant local `import json` in the new /history
integration tests; the module-level import already covers them.
A fresh browser connect during a parallel tool batch (e.g. several
web_fetch) left completed siblings' tool blocks empty until a manual
refresh: each tool_result SSE event fires the instant a sibling
finishes, but the result messages persist only after the whole batch
returns, so a fresh connect replayed neither the already-fired event
(a fresh connect doesn't replay the ring buffer) nor a /history row.
Route the fresh connect through the same delta replay a reconnect
already uses. Persist the per-ws SSE ring-buffer high-water mark
(_event_id) onto each saved conversation row. /history returns the
committed snapshot up to a resolved-turn-boundary cursor and omits the
trailing executing in-flight turn; the client opens its initial SSE
with that cursor (Last-Event-ID) so the existing replay_ok path
fast-forwards the in-flight turn whole -- tool blocks, results, and
approve/plan prompts all rebuild from the ring buffer.
The cut sits at the last resolved-turn boundary (not max(saved
event_id)), so out-of-order result saves in the post-batch loop can't
move it or strand a sibling. Gated on buffer-liveness (can_replay_from):
reloaded / evicted / awaiting-approval cases keep the in-flight turn in
/history and return a null cursor, falling back to the synthetic
snapshot floor -- preserving the existing in-flight render and never
leaving a turn unrenderable.
- Migration 059: nullable event_id BIGINT on conversations + a
(ws_id, event_id) index (keeps the cold-open high-water reseed a seek).
- save_message(event_id=) across the storage wrapper / protocol /
sqlite / postgres backends; get_max_event_id; reconstruct_messages
surfaces the _event_id side-channel.
- SessionUIBase: reseed _event_id from storage on construction (so the
id space stays monotonic across restarts); can_replay_from() gate.
- make_history_handler: _resume_cursor_and_trim() + cursor in the
response (WorkstreamHistoryResponse.cursor). The shared projection,
export, and coord-rebuild paths are untouched.
- app.js: seed the resume cursor on the initial-connect path only, and
gate the last_event_id param on != null so a cursor of 0 (a brand-new
workstream's first-turn boundary) is not dropped.
Tests: helper, storage round-trip, and seed unit tests; two
make_history_handler integration tests (cursor + orphan-trim when
replayable, null cursor + orphan kept when not); app.js static guards.
Migration applies up and down on SQLite.
strip_html deleted every HTML tag with no separator, gluing paragraphs,
headings, list items, and table cells into a structureless run of text
("<p>a</p><p>b</p>" -> "ab"). This degrades web_fetch, which feeds the
cleaned page to a summarising agent — and it flattens the structure any
downstream chunking/retrieval would rely on.
Block-level tags and <br> now become newlines so structure survives
("<p>a</p><p>b</p>" -> "a\n\nb"); inline tags are still dropped.
The conversion is a single linear tag scan: one pass over `<[^>]++>` with
a possessive quantifier, dispatching each tag name against a frozenset.
This replaces three full-document passes plus a 24-way alternation, and:
- Removes catastrophic backtracking (ReDoS). The earlier `<\s*/?\s*` and
`<\s*br\s*/?\s*>` patterns were quadratic on '<' + a long whitespace
run (~2s at 4k chars); the scan is now linear (~3ms at 1M chars) on the
untrusted, up-to-10MB web_fetch input. The possessive quantifier also
neutralises the pre-existing quadratic in the old `<[^>]+>` pass.
- Matches <br> carrying attributes (e.g. `<br clear="all">`), which the
first cut missed.
Tests cover block separation, inline-tag joining, uppercase tags, <br>
with attributes, lookalike tag names, and a pathological-whitespace
regression guard.
Note (pre-existing, not changed here): in _exec_web_fetch the 10 MB cap is
applied after strip_html, so the stripper sees the full fetched body. With
the scan now linear this is no longer a CPU concern; capping the raw input
before stripping remains a worthwhile defence-in-depth follow-up.
Add a workstream conversation export on three surfaces, all sharing one
serializer (turnstone/core/export.py):
- `turnstone-admin export <ws_id> [--children] [-o FILE|-]` — offline,
direct-DB. `--children` bundles a coordinator's parent conversation
plus one JSON per child into a zip (parent.json + children/<id>.json,
no manifest).
- `GET /v1/api/workstreams/{ws_id}/export` — conversation-only file
download, mounted on both the node (interactive) and console
(coordinator) lifespans via `make_export_handler(cfg)`, reusing the
/history gate ladder (permission_gate, tenant_check, list_kind
cross-kind isolation) so ownership and isolation come for free.
- Web UI — an "Export conversation" item in the interactive per-tab
dropdown (scoped to that tab's workstream) and an Export button on the
coordinator appbar.
Format is OpenAI Chat Completions messages JSON (`{"messages": [...]}`),
built from `sanitize_messages(load_messages(repair=True))`. Persisted
reasoning is surfaced on assistant messages as a flat `reasoning_content`
field (the convention OpenAI-compatible inference servers use) via a
dedicated helper that runs before sanitize strips the internal
_provider_content lane. Attachments ride along as the standard image_url
/ inlined-document content parts.
Lets users get conversations out in a portable interchange format
(backup, fine-tuning datasets, sharing, interop) without lock-in.
Closes#613.
Non-obvious decisions:
- Single format (openai-json); children/zip is CLI-only. The HTTP
endpoint and web UI are conversation-only, keeping the served surface
— and its security surface (no child rows read through the coordinator
handler) — small.
- `reasoning_content`, not the `reasoning` field /history and the
reasoning-replay path use: export targets the chat-completions
convention. Documented in export.py to prevent a "consistency fix".
- list_workstreams exposes no cursor, so the child walk passes an
explicit high limit rather than inheriting the default 100, which
would silently drop a coordinator's children past 100.
- Interactive export lives in the per-tab menu (interactive is
per-tab/pane — avoids focused-workstream ambiguity); the coordinator
is one conversation, so it keeps an appbar button.
Tested: 25 new tests through real storage + handlers (TestClient), incl.
cross-kind isolation 404, misconfig 500, the reasoning + attachment
pipeline, and the coordinator children zip. The shared frontend helper
is verified by a node sandbox harness (re-entrancy guard, button
disable/aria-busy, no-button tab-menu path). Full non-live suite green
(6714 passed); ruff + format + mypy clean; OpenAPI spec updated.
PR #612 review (Copilot): the synthetic `error` event surfaced on a fresh
connect carried no SSE `id:`, so the client's `lastEventId` never advanced.
The client's `error` handler is append-only (not idempotent like
`state_change` / `in_progress_snapshot`), and a terminal-errored idle ws
emits no live event to set a cursor — so a native EventSource reconnect sent
no `Last-Event-ID`, re-ran the fresh path, and appended a DUPLICATE error
bubble on every reconnect cycle (proxy idle-timeout, network blip).
Attach `id: str(snap_seq)` (the registration-time buffer cursor already in
scope) to the surfaced error. The reconnect then sends that `Last-Event-ID`
→ `register_listener_with_replay` returns `replay_ok` (nothing buffered past
snap_seq on an idle ws) → the handler's replay_ok branch skips the synthetic
surface. No duplicate.
Test asserts the surfaced error carries `id: snap_seq`; the existing
`test_handler_replay_ok_does_not_resurface_last_error` pins the
reconnect-skips half.
A browser connecting fresh to a workstream sitting in the error state
saw the error STATE (composer unlock + retry, via the replayed
state_change) but not the error TEXT explaining why — on a fresh connect
there was no source for it. `on_error` is never persisted as a message,
so `/history` can't rebuild it; only the reconnect path (ring buffer)
carried the original `error` event.
Surface the persisted `last_error` in `make_events_handler`'s
fresh/truncated synthetic-replay branch when the workstream is in the
error state. Gated on the error state so a healthy ws skips the storage
read, and confined to the fresh/truncated path — the `replay_ok` branch's
ring buffer already replays the original `error` event, so surfacing here
would double it. The persist (`_record_fatal_error`, sanitized) / clear
(on recovery) lifecycle already exists; this only reconstructs the event
on a fresh connect, reaching parity with reconnect.
Second of the fresh-connect replay-completeness fixes surfaced by the
audit (sibling to the tool-call `pending` fix in this branch). Non-terminal
mid-turn errors (tool parse failures, truncation — not state=error, not
persisted) remain an accepted gap; the queued-message indicator gap is
deferred (needs client-side render-on-replay).
Adds two parity tests: fresh+error → surfaced / fresh+idle → gate skips,
and replay_ok → not double-emitted.
In-flight tool calls did not render when a browser connected fresh to an
in-progress workstream mid-tool-execution; they only reappeared after the
SSE dropped and reconnected.
`project_history_messages` marked the trailing tool-call turn `pending`
from orphan-detection (a tool_call with no result) as a proxy for
"awaiting approval". But an orphan that is *executing* (already approved,
running) is orphan-but-not-awaiting. The renderer skips `pending` turns
because the SSE replay re-emits the interactive approve_request prompt
instead — and during execution `_pending_approval` is None, so nothing
re-emits. The tool call rendered from neither source on a fresh connect,
recovering only on reconnect (ring-buffer replay carries the
tool_info / tool_result events).
Regression from the REST-first history convergence (0ad1ab7f), inherited
by the wire-shape unification (#596): both swapped the `pending` predicate
from the live `_pending_approval` signal to storage orphan-detection,
which diverge exactly during tool execution.
Thread the live awaiting-approval signal from `make_history_handler` into
`project_history_messages` (new `awaiting_approval` param) and gate the
pending mark on it, re-syncing `pending` with the same `_pending_approval`
signal that drives the SSE prompt re-emit. A storage-only / closed ws has
no live session → never pending → trailing orphans render as historical.
Asserted as `dict` to match the detail handler's MagicMock-safe guard.
Adds a projection-level gate test and two handler boundary tests
(execution → renders, awaiting approval → pending). The existing
partial-trailing-turn test now asserts the turn RENDERS, not just that the
row survives — the parity gap that let this regression through.
Replaces the Saved Workstreams (ui/static) and Saved Coordinators
(console/static) card grids with a dense, sortable table that reuses the
active dashboard's row system, via one shared component in
shared_static/cards.js (renderSessionRow, SavedColumns, createSavedTable)
+ cards.css. The two surfaces differ only by column spec (MSGS vs CHILDREN)
and per-app data/ids/delete-request; everything generic is shared.
- NAME flexes to full width (kills the card grid's near-duplicate-name
truncation); client-side filter + sortable headers; scroll-all
(pagination retired); multi-select delete preserved on rows.
- Consumes the enriched saved-list DTO: MODEL, CTX (context-window
occupancy, a frozen last-activity snapshot), SKILL chip, CHILDREN, and a
red left-edge for failed runs.
- Saved rows reuse the dash-table chrome but opt out of the active table's
live-state styling: idle rows aren't dimmed, CTX reads as a snapshot (not
the live gauge), legible zebra + AA-contrast muted text for a long
terminal list, and responsive compact columns keep NAME readable on
narrow viewports.
- a11y: sortable headers exposed to assistive tech (aria-label / aria-sort
+ at-rest carets); footers are live regions.
- Removes the now-dead renderSessionCard + card-grid CSS.
GET /v1/api/workstreams/saved returned only ws_id/alias/title/created/
updated/message_count — too little to drive the planned saved-list table
redesign. Add seven fields, all sourced from already-persisted data (no
migration):
- state, kind, node_id: columns on the workstreams table
- model_alias, launch_skill: from workstream_config via LEFT JOIN
- child_count: COUNT of child workstreams via parent_ws_id
- context_tokens: most recent usage_events prompt size for the workstream
- context_ratio: context-window occupancy (context_tokens / model context
window), computed in the handler so the NULL / zero-window cases stay
explicit and identical across both storage backends
context_window comes from a model_definitions join; aliases defined only in
config.toml are absent there, so context_ratio degrades to 0.0 rather than
reporting bogus occupancy. The Python SDK reuses the Pydantic model; the
TypeScript SDK OpenAPI snapshot and hand-maintained interface are updated.
Tests cover the new storage columns (including NULL-when-absent), the
handler ratio math + zero-window degradation, and the SDK enriched
round-trip.
The Usage dashboard summary cards read the oldest day bucket
(`summary.breakdown[0]`) instead of the window SUM, so every headline
(total/prompt/completion/tool-calls/cache) showed a single day's value —
e.g. 30-day tool-calls reading lower than 7-day. Read `.summary[0]` and
collapse the redundant two-request fetch into one (the response already
carried both `summary` and `breakdown`).
Only the main streaming loop (`on_status`) recorded `usage_events`.
Auxiliary non-streaming calls — title generation, conversation
compaction, web-fetch summarization, and plan/task sub-agents — bypassed
that path and were never counted, undercounting real consumption by a
large factor for agent-heavy workstreams. Add an `on_aux_usage` UI hook
(storage row via a shared `_write_usage_row` helper with `on_status`;
`WebUI` override feeds Prometheus) and route `_utility_completion` and
sub-agent turns through it, attributed to the agent's own model. Judge
token spend remains uncounted — deferred to a follow-up.
GET /v1/api/models blanked default_alias whenever model.default_alias named
an alias absent from the server's live registry — e.g. when a standalone
turnstone-server shares a ConfigStore with a console whose model.default_alias
points at a console-only / DB alias (or the underlying model id rather than
the alias). The interactive dashboard then showed a bare "Default model"
placeholder even though a new workstream launches on a concrete model.
Mirror session_factory's _effective_default_alias / _effective_routing: fall
back to registry.default (which already incorporates a *valid* model.default_alias
override) when the configured alias is unset or foreign, blanking only if
registry.default is itself unresolvable. The endpoint now reports the model
creation actually uses.
openapi-server.json / openapi-console.json had drifted well behind
build_server_spec() / build_console_spec() — the committed snapshots are
regenerated periodically (via sdk/typescript/scripts/generate-types.py)
rather than on every schema-changing PR, so accumulated additions (skill
parsing, pending-approval items, model-definition CRUD, etc.) had not been
captured. This resyncs both with no code changes.
Addresses #603 review: the pagination control is a sibling of the cards
container, so loadDashboard()'s "Loading…" / "Failed to load" states (which
replaceChildren only the cards) left stale Prev/Next visible and still wired
to the previous _wsSavedItems cache — in the error state clicking them would
resurrect the old cards over "Failed to load". Route both transient states
through a _setSavedWsMessage() helper that clears the cards and hides the
pagination in lockstep; a successful load re-renders both via
renderSavedWorkstreams.
The interactive dashboard had drifted from the coordinator launcher in two
ways; this backports both for consistency.
Selectors: the Model / Judge Model dropdowns now show the resolved default
model in the placeholder (e.g. "Default — primary (vendor/primary)") instead
of a generic "Default model" / "Default (agent model)". The server's
GET /v1/api/models now returns judge_default_alias (mirroring the console
endpoint), sourced from the judge.model setting. It is intentionally left
blank when judge.model is unset or points at a disabled/removed alias,
because the judge then follows the per-workstream agent model at runtime
(session_factory: judge_config.model or model) — the UI keeps the honest
"Default (agent model)" wording in that case. This also fixes a latent
mislabel: the judge row previously said "agent model" even when an operator
had configured judge.model.
Pagination: Saved Workstreams now paginates at 24/page (Prev · X / Y · Next),
matching Saved Coordinators — page clamp after deletes, hidden on a single
page and in delete mode, Select-All bounded to the visible page. The empty
branch drops out of delete mode (matching the launcher) so the toolbar can't
linger over an empty grid.
The shared .pagination CSS is lifted from console/static/style.css into
shared/cards.css (loaded by both apps) so the two dashboards keep one source
of truth instead of a third copy. The pagination JS render wiring stays
per-app (it binds per-app DOM ids + controller instances) with a
cross-reference comment to its console twin.
Tests: new tests/test_server_available_models.py pins the judge/model
resolution chain (unset / configured / unknown / whitespace / registry-default
fallback).
The output-guard judge's model (judge.output_guard_model) was only
configurable on the Judge settings tab, while every other model role —
coordinator, intent judge, plan/task agents, channel adapter — lives in
Models → Roles. Add it there as a role (mirroring the intent Judge role)
and skip it on the Judge tab so it renders in exactly one place.
No backend change: the role read/write goes through the generic
/v1/api/admin/settings endpoints, the same path the intent-judge model
role already uses.
- LLM-tier row no longer duplicates the judge reasoning into its
annotations column — reasoning lives in the dedicated reasoning column,
so annotations stays heuristic-only and audit consumers aren't confused.
(The replay merge reads the heuristic row's annotations + the LLM row's
reasoning, never the LLM row's annotations, so this is display-safe.)
- Correct the output-warning chip comment: tier "llm" means the judge
returned a verdict (it may have cleared a heuristic-positive), not that
it owns the displayed finding.
After the heuristic+LLM merge, the LLM verdict no longer "overrides" the regex
verdict; it merges (risk = max, flags = union) and can raise but never lower a
regex finding. Fix the admin Settings help string to match.
Surface the output-guard LLM judge on the inline finding chip and merge it
with the regex heuristic instead of one stage winning outright.
Merge rule (issue #560, "show, annotated"):
- risk_level = max(heuristic, llm); flags = union. The judge can escalate
but never lower a heuristic positive — it evaluates adversarial tool
output, so defeating it must not erase a deterministic regex finding.
Credentials stay heuristic-only and are always redacted.
- The judge's own verdict rides along as a dissent-aware annotation
(judge_risk / confidence / reasoning / judge_model) on the chip in both
the interactive and coordinator UIs, live and on reconnect. One shared
merge_guard_display_payload drives both paths so they cannot drift.
- The model is shown the merged risk + flags but never the judge's "benign"
verdict — a fooled judge must not talk the model out of caution.
Fixes a reconnect bug: a judge that ran but failed wrote a risk="none" row
that won the replay dedup and hid the heuristic finding (it showed live but
vanished on refresh). Failed judges now persist under tier="llm_error",
excluded from the display merge; the max-merge also floors the displayed
risk at the heuristic level so the chip never vanishes.
Also adds a regression test confirming the LLM judge runs on every tool
output, not just heuristic-flagged ones.
Tests: merge unit tests, storage-backed replay regression, live/replay
wire-shape parity, SDK-event drift guard. ruff + mypy clean.
The per-message rewind / edit / retry affordance — the icon glyphs
(.icon-edit / .icon-rewind / .icon-retry) plus the inline edit-in-place
form (.msg-edit-*) and the [data-busy] / .msg-editing states — was
duplicated verbatim in both pane stylesheets: ui/static/style.css
(interactive) and console/static/coordinator/coordinator.css
(coordinator). PR #598 deferred consolidating them to keep that
coord-only change off the shipped interactive stylesheet's cascade.
Move the block into shared_static/chat.css, immediately after the
.msg-actions / .msg-action-btn primitives both panes already share, and
delete both copies (including coordinator.css's now-obsolete FOLLOW-UP
note describing the duplication).
Both index.html files load chat.css before their pane stylesheet, so the
rules land earlier in the cascade; the selectors are unique (defined
nowhere else, confirmed repo-wide) so it is a visual no-op. The block is
moved verbatim — chat.css's sibling rules use a different variable
vocabulary (--r-sm=4px / --font-mono) than the affordance block
(--radius-sm=3px / --font-ui), so renaming would change radii/fonts.
Verified pixel-identical via a headless-Chrome render-diff of both panes,
before vs after, across all four affordance states (edit+rewind, retry,
editing-open, busy): 0 differing pixels.
The capture-phase window scroll listener that dismisses an open overflow
menu also fired for scrolls originating inside the menu itself (the menu can
overflow-y:auto at high browser zoom / short viewports), so a tall menu
closed the instant you tried to scroll it. Skip scroll events whose target
is inside .admin-kebab-menu; page/ancestor scroll still dismisses.
Addresses review feedback on #599.
Admin tables render their per-row actions in an ACTIONS column whose grid
track is fixed-width and inherits `.admin-col`'s overflow:hidden +
text-overflow:ellipsis. Rows with several actions (MCP servers:
refresh/reconnect/edit/del) overflowed the track, so only the first button
showed and the rest were clipped behind a misleading "…". The edit button
was unreachable — the only way to change an MCP server was editing the DB
by hand (#593).
Replace the inline button strips across all 15 admin tables (admin.js +
governance.js) with a shared kebab (⋯) overflow menu:
- _kebabMenu()/_kebabMenuEl() build the menu; a single action degrades to
an inline button. _initKebabMenus() wires open/close, outside-click,
Escape, arrow-key nav and viewport-aware flip-up/flip-left through one
document-level delegated listener. Menu items keep the same data-*
attributes, so the existing per-table click bindings are unchanged.
- Right-align the actions column so the trigger shares an edge with its
menu; pair danger/caution items with warning glyphs (not colour alone);
theme-aware --red-glow/--yellow-glow hover tints.
Fixes#593.
_refreshRetryButton() ran only from finishAssistantStream (live turn ends), so a reloaded coordinator workstream or a clear_ui/replay_truncated re-render after rewind/retry showed assistant turns without the retry button — unlike the interactive replayHistory() path, and inconsistent with the edit/rewind buttons (which DO attach on re-render via appendUserMessageWithAttachments). Call it at the end of refetchHistory() so retry attaches on every render. (Addresses Copilot review on PR #598.)
Give coordinator workstreams the rewind/retry/edit affordance the interactive pane has, completing the #549 verb lift on the frontend. Browser-verified separately (the console won't boot in-sandbox).
- Factor refetchHistory() out of init(): the history-render block moves into a reusable function that clears the message column + tool-tracking state (toolRows / activeBatch) before re-rendering, so a mid-session re-render leaves no stale call_id->row mappings. init() keeps the first-paint-only pending-approval replay + children/tasks/attachments/SSE.
- SSE: case clear_ui re-renders from REST then dispatches the latched edit-and-resend via path-keyed /send; case replay_truncated re-syncs (skipped mid-stream). New _pendingEditSend module latch.
- Per-message affordance mirroring the interactive pane: edit + rewind buttons on every user bubble, a retry button on the last assistant turn (skipped when the turn ended tool-only, gated on .coord-tool-batch). Bare .msg.user turn-count matches the server's _find_turn_boundaries.
- Port the icon-glyph + inline-edit-form CSS into coordinator.css (.msg-actions / .msg-action-btn already live in shared chat.css; consolidating all of it into shared chat.css is a tracked follow-up).
coord-LOCAL mirror (not a shared_static extraction): app.js is class-based over this, coordinator.js is a module closure.
Lift the conversation-modifying /rewind and /retry verbs out of the body-keyed POST /v1/api/command into path-keyed POST /v1/api/workstreams/{ws_id}/rewind ({turns:N}) and /retry, as make_rewind_handler/make_retry_handler in SharedSessionVerbHandlers (template: make_close_handler/make_cancel_handler), wired on both interactive and coordinator kinds. Closes the last unlifted conversation-modifying surface — coordinator workstreams gain rewind/retry where they had none — and removes the surviving exception to the post-#422 path-keyed URL convention.
Handler shape: auth gate (coord -> admin.coordinator via permission_gate; interactive -> conversation.modify via accepted_permissions) -> busy-gate -> session.rewind(n)/retry() -> always emit clear_ui (incl. rewind-to-zero, carries #503) -> audit (conversation.rewind/retry on both kinds). Retry re-dispatch reuses the shared session_worker.send via a per-kind dispatch_retry closure (hard-reject on busy), not a third hand-rolled thread.
The web /command handler now rejects /rewind+/retry with a pointer to the path-keyed endpoint (BREAKING; 1.6.0aN-tolerant); session.handle_command's branches stay for the terminal CLI. auth.py adds the verbs to both write suffix-sets; Python + TS SDKs, OpenAPI (RewindRequest + server/console specs), the /route/ proxy mounts + audit actions, and coordinator_client all gain them.
Interactive frontend (app.js): the 3 /command POST sites + the hand-typed-slash reroute now hit the path-keyed endpoints; the bare .msg.user rewind selector is kept (matches the server's _find_turn_boundaries, which counts system-nudge user turns). The coordinator frontend rewind UX lands in a follow-up commit (browser-verified).
Tests: route-walk mount/order, /route/ audit rows, required_scope, OpenAPI catalog, SDK body-inspection, and HTTP-level handler behavior (busy-gate, turns validation, clear_ui emit, retry dispatch, audit invocation + swallow).
Register `claude-opus-4-8` in the Anthropic capability table. Opus 4.8
shares Opus 4.7's request/response surface exactly — adaptive-thinking
only (`budget_tokens` rejected), sampling params removed, the
low/medium/high/xhigh/max effort levels, `thinking.display` defaulting to
omitted, 1M context, and 128K output — so the entry is a verbatim copy of
the 4.7 row. `_lookup_capabilities` longest-prefix matching then resolves
date-suffixed ids (e.g. `claude-opus-4-8-20260601`) without colliding
with the 4.7 key.
No provider code paths change: the existing 4.7 handling already covers
all of 4.8's behavior. Models are selected via config.toml / the admin
ConfigStore UI, so there is no catalog or dropdown to update.
- _anthropic.py: new claude-opus-4-8 capability entry + effort comment
- tests/test_providers.py: opus 4.8 bare + dated capability tests
- turnstone.example.toml: bump the showcased model example to 4.8
Migrate the coordinator dashboard's init() history rebuild onto the
server-projected wire shape (the prior commit's project_history_messages),
retiring its inline raw-storage-shape handling. Field-sourcing only -- the
batch render + orphan->--running state machine is unchanged:
- callOutcomes reads the server-derived `denied` / `is_error` flags
instead of re-sniffing tool content prefixes;
- tool_calls are read flat (`tc.name` / `tc.arguments`) now that the
projection flattens the nested `function` wrapper;
- user content is a plain string and attachments come from the projected
`attachments` list, replacing the multipart walk + `_attachments_meta`
side-channel read.
Fix two latent reload bugs along the way: coord read `m.reminders` /
`m.source` but the raw shape carried `_reminders` / `_source`, so metacog
reminder bubbles and the system-nudge marker never rendered on a coord
history reload. The projection surfaces both top-level, so coord's
existing (unchanged) render paths now fire.
Repoint the test_coordinator_page deny-classifier guard at the live
`m.denied` / `m.is_error` reads instead of comment prose.
Refs #549.
Collapse the three hand-synced "raw storage -> render shape" projections
into one server-side projection. The projection previously lived in a
test-only `_build_history` (SSE-era reference impl), a client-side JS
normaliser (`history_normalize.js`, the transitional bridge), and coord's
inline `init()` handling -- drifting silently with no parity test.
Add `project_history_messages` to `history_decoration.py` and run it as the
final step of the `make_history_handler` pipeline (load_messages -> decorate
-> extract_reasoning -> project), so `GET /history` emits the canonical
render shape directly: flat tool_calls (with verdict / output_assessment),
top-level source / reminders / attachments, collapsed multipart content,
derived denied / is_error / pending, reasoning, and advisories. Interactive
`replayHistory` now consumes the payload verbatim.
Close two gaps the JS bridge deferred:
- list-content <tool_output> advisory extraction (decorate handles only
string content; the projection extracts list-carrier advisories, then
joins remaining text parts to the string the renderers require);
- orphan->pending marks ONLY the last orphan tool-call turn, so a
mid-conversation cancelled tool still renders instead of vanishing.
Delete `history_normalize.js` (+ its <script> tag and node test) and the
test-only `_build_history` (+ orphaned imports); retarget its direct tests
onto the projection helpers. Update the WorkstreamHistoryResponse
description and the Web UI Resilience architecture note to the projected
shape.
Coord's `init()` still reads the raw side-channels; migrating it to the
projected shape is the next commit, browser-verified separately.
Refs #549.
Two issues on the new REST-first history path (PR #595 review + a user repro):
- Stale history on a fast workstream switch: a slow `_refetchHistory` (e.g. a
large resumed session) could resolve AFTER the pane moved to another ws,
rendering the old ws's history over the new one — and its `.finally`
reconnecting the old stream. Add a per-pane load-generation token:
`_loadHistoryThenConnect` bumps it, and the refetch render, the deferred
`connectSSE`, and the `clear_ui` resend are each discarded when a newer load
supersedes them. Fixes the "open a child workstream, see the previously
resumed workstream's history" repro.
- Stale per-ws SSE replay cursor (Copilot): ws-assign callers set `this.wsId`
before `_loadHistoryThenConnect`, so `connectSSE`'s `wsChanged` is already
false, and `reset()` never cleared `_lastEventId` — so a tab switch / child
open sent ws-A's `last_event_id` to ws-B, mis-triggering the server's
`replay_ok` path and skipping the synthetic replay (connected / status /
in_progress_snapshot). Reset `_lastEventId` + `_lastStatusEvt` in
`_loadHistoryThenConnect` so a ws (re)load always opens a fresh stream;
transient same-ws reconnects (direct `connectSSE`) still reuse the cursor
for `replay_ok`.
test_app_js.py pins the load-generation guard.
Interactive fetched conversation history as an inline SSE `history` event on
every (re)connect — a multi-MB payload — while coord fetches it once via REST
`GET /history` and uses SSE for live deltas only. This converges interactive
onto coord's model so both kinds share one history-delivery pattern, the
prerequisite for lifting the `/command` (rewind/retry) verb to coord.
Backend (server.py, core/session_routes.py):
- `_interactive_events_replay` and the `/command` resume/rewind branches no
longer emit the inline `history` SSE event; the open/create-resume paths
emit `clear_ui` only. The `/events` stream no longer carries conversation
history — REST `GET /history` is the source (acceptable on 1.6.0aN).
- Removed the now-orphaned `events_replay_prepare` hook.
- `_build_history` retained as the canonical wire-shape reference for the
decoration/parity tests (no production callers post-convergence).
Frontend (ui/static/app.js, ui/static/index.html):
- `_loadHistoryThenConnect` fetches REST `/history`, renders, then opens SSE
(mirrors coord's `init()` ordering); wired into the seven ws-assign sites.
- `clear_ui` re-renders via a REST refetch and dispatches the edit-and-resend
latch; `replay_truncated` re-syncs (skipped mid-stream so it cannot clobber
an in-flight turn). The `case "history"` SSE handler is removed.
New shared module (shared_static/history_normalize.js):
- `normalizeHistoryMessages` converts the raw provider-native REST shape
(nested tool_calls, `_source`/`_reminders`/`_attachments_meta` side-channels,
multipart content, no derived flags) into the projected shape `replayHistory`
renders. Pure/DOM-free and node-unit-tested. This is a transitional bridge —
a server-side wire-shape unification (folding this projection back into the
server so interactive, coord, and coord's inline raw-handling collapse onto
one shape) is planned to replace it.
Tests: backend replay-omits-history regression; a node-executed normalizer
projection test (incl. the orphan->pending and denial-propagation edges); and
REST-first wiring guards in test_app_js.py.
notify was interactive-only — a coord with a natural "fan-out complete"
or "batch failed" beat could only post by spawning a child for the
single message, which is a lot of ceremony. Routing is session-kind-
agnostic in _prepare_notify / _exec_notify; this is a metadata flip
that adds the coord flag (plus the explicit interactive flag the loader
needs once coordinator is set) and updates the dual-kind whitelists,
coord tool-set assertions, and skill-author docs accordingly. Adds
two coord-session tests pinning the prepare dispatch contract
(needs_approval=False matches notify.json auto_approve) and the exec
→ channel-gateway path.
The Judge → Settings tab has its own renderer in governance.js separate
from _renderSettingRow in admin.js — it builds card-style rows inline
and was painting ``s.help || s.description`` straight into the row,
which is why the Judge settings still showed long paragraphs inline
after the prior commit on this branch.
Convert renderJudgeSettings to the same shape as the generic Settings
tab:
- Short description (s.description) renders inline below the key — the
always-visible TLDR.
- A ? button next to the key gates the long-form s.help paragraph via
a sibling .settings-help-popover with id ``<key>-help``. Picked up
automatically by the document-delegated click handler in admin.js.
Toggle behavior, save/reset buttons, and Judge-specific save endpoints
are untouched.
Settings-tab buttons assembled in admin.js and the skill-modal buttons in
index.html used two parallel rendering shapes with two binding mechanisms
(per-button addEventListener vs. document delegation). Listeners stacked
on the same DOM whenever the Settings tab re-rendered.
Converge both surfaces on the empty-button + ``data-help-target`` +
document-delegated dispatch pattern:
- ``_renderSettingRow`` emits empty <button> with ``data-help-target`` and
gives the sibling popover a matching ``id``. Per-button listener loop in
``_renderSettings`` removed.
- ``_toggleSettingsHelp`` and ``_closeAllSettingsHelp`` drop their
``.settings-label-col`` fallback branches; ``data-help-target`` reverse
lookup is the only path.
- 12 skill-modal buttons in index.html lose their literal ``?`` text so
every emitter produces empty content.
- ``font-size: 0`` on ``.settings-help-btn`` removed (workaround is no
longer load-bearing now that all emitters are empty).
Also tucks long-form help text in Judge modals behind the same ``?``:
- 8 ``.label-hint`` spans in the Create/Edit Heuristic Rule and Create/Edit
Output Guard Pattern modals (Tool Pattern, Arg Patterns, Confidence,
Pattern Flags) become ``?`` + popover. Short ``.label-hint`` strings
elsewhere in the admin UI are unchanged.
Description rows (the short TLDR under each key) stay inline as before.
The ``?`` popover carries only the long-form ``help`` paragraph and any
``reference_url`` learn-more link.
* feat(rbac): editable builtin role permissions via overlay layer
Adds a ``role_permission_overrides`` table that stores per-(role_id,
permission) grant/revoke deltas, applied on top of the immutable
``roles.permissions`` baseline at permission-load time. Builtin roles
(``builtin-admin/operator/viewer``) become customizable through the
admin Roles UI without losing the "reset to default" guarantee — every
override is auditable and reversible.
Motivating case: ``model.skills.write`` is deliberately default-ungranted
on every role so operators must consciously opt in before a coordinator
session can mutate the skill catalog. Until now there was no UX path to
do that opt-in — the only options were dropping into SQL or running a
fresh migration. The overrides editor closes that gap.
Backend
- Migration 057 + storage methods on both sqlite + postgresql backends
- ``get_user_permissions`` merges baseline ∪ grants − revokes for builtin
rows; custom rows pass through unchanged
- ``GET /v1/api/admin/roles/{id}/effective`` for inspect
- ``PUT /v1/api/admin/roles/{id}/overrides`` for write — admin.roles gated,
audited, validates against ``_VALID_PERMISSIONS``, refuses non-builtin
targets, strips no-op grants/revokes before persisting
- Lockout guard: cannot revoke ``admin.roles`` if doing so would leave
zero users with the permission (returns 409)
- ``coordinator.trust.send`` added to ``_VALID_PERMISSIONS`` — was
seeded into builtin-admin by migration 042 but never registered with
the validator, so the very first round-trip through the editor 400'd
on it. Drift-detection test guards future migrations from recreating
the same gap
Frontend
- Roles tab redesign: chevron + permission-count chip replace the
"..." truncation; expand-on-click drawer groups perms by namespace
with baseline / grant (green +) / revoke (red −) chip variants
- Edit modal opens for builtin rows ("Customize Built-in Role" title);
toggles show baseline-default vs override state; submit diffs against
the rendered toggle universe (not raw baseline) so future taxonomy
drift can't silently strip unknown perms
- "Modified +N/-N" pill on rows with active overrides; "Reset to default"
drawer action clears the override set
- ``_PERMISSION_SECTIONS`` brought up to date with all currently-seeded
perms (admin.coordinator, admin.cluster.inspect, admin.models,
admin.nodes, admin.prompt_policies, conversation.modify,
coordinator.trust.send were missing)
Tests
- 7 storage tests covering set/list/clear/effective + overlay merge into
``get_user_permissions`` for both builtin and custom roles
- 11 endpoint tests covering effective/overrides happy paths, validation,
lockout guard, builtin-only restriction, no-op normalization, list
enrichment
* feat(rbac): enforce workstreams.{create,close} + tools.approve gates
These three permissions were declared in ``_VALID_PERMISSIONS``, seeded
into ``builtin-operator``'s baseline by migration 008/017, surfaced in
the admin Roles UI as toggles, and documented in ``bootstrap.py`` as
the operator role's capabilities — and never enforced anywhere. The
audit that ran out of the overlay PR found zero ``require_permission``
sites for any of them; any authenticated user could create workstreams,
close any workstream, or approve any pending tool regardless of role.
Behaviour change for callers without the perms:
- ``POST /v1/api/workstreams/new`` (node + console proxy variants)
now 403 without ``workstreams.create``
- ``POST /v1/api/workstreams/{ws_id}/close`` (and ``/route/`` proxy)
now 403 without ``workstreams.close``
- ``POST /v1/api/workstreams/{ws_id}/approve`` (and ``/route/`` proxy)
now 403 without ``tools.approve``
The OR-fallback to ``admin.coordinator`` keeps coord sessions spawning
interactive children unblocked without needing operator-style perms.
Service-scoped inter-cluster calls bypass via the existing
``allow_service_bypass`` path on the new ``require_any_permission``
helper. Builtin admin and operator both already carry these perms;
viewer correctly loses workstream create/close/approve (it already
couldn't do those in spirit).
Implementation
- ``require_any_permission`` (core/auth.py) — OR-semantics variant of
``require_permission`` with per-conditional comments documenting the
security policy at the choke point. 403 body names every accepted
perm so operators get an actionable remediation
- ``make_{create,close,approve}_handler`` (core/session_routes.py)
accept ``fallback_permissions: tuple[str, ...]`` — checked only when
``cfg.permission_gate is None`` (interactive case). Coord's
``permission_gate=_require_admin_coordinator`` continues to take
precedence on the coord-config side
- Console-side ``create_workstream`` and ``route_create`` inline the
same OR check before proxying — fail fast on a forbidden request
without burning a cluster round-trip
- ``route_proxy`` adds a verb-scoped gate on ``approve`` and ``close``
only; ``send``/``cancel``/``dequeue``/``command``/``plan`` remain
authenticated-only (pre-existing, out of scope for this audit)
Tests
- New ``TestPermissionGatesOnLifecycle`` (4 tests) in test_server_authz
pinning 403-without-perm + non-403-with-perm at the node lift sites
- New ``TestRouteProxyPermissionGates`` (5 tests) in
test_console_routing_proxy covering 403 paths, OR fallback via
``admin.coordinator``, and that ``send`` remains ungated
- ``_make_jwt`` helpers in test_server_authz, test_close_reason_
persistence, test_server_attachments_on_create updated to embed
operator-shaped perms by default so existing tests continue to
exercise the post-gate logic rather than 403'ing on the new check
Docs
- ``bootstrap.py`` operator role line corrected to list every perm
it actually carries (was missing ``tools.approve`` and
``conversation.modify``)
* fix(rbac): close lockout + escalation gaps in role-overrides editor
Three issues surfaced by /review of the overlay layer and gate uplift —
all in the RBAC/auth surface, treated as zero-days.
**F-1: lockout guard misses the grant-removal path.** PUT-replace
semantics on ``set_role_overrides`` mean an existing grant of
``admin.roles`` (added via override to e.g. builtin-operator) is
silently dropped when the new payload omits it. The previous guard
short-circuited on ``"admin.roles" not in revokes`` and never noticed.
Concrete cluster-bricking scenario: grant admin.roles to operator via
override, unassign builtin-admin, click "Reset to default" on operator
→ all users lose admin.roles, recoverable only via SQL.
The rewritten guard simulates the post-PUT effective set on the target
role directly: if ``(baseline | new_grants) - new_revokes`` lacks
admin.roles AND nobody holds it via another role, refuse the change.
The "via another role" question is answered by one bulk query rather
than the prior O(users × roles) round-trip loop.
**F-3: lockout check blocked the event loop on moderate deployments.**
The prior check called ``storage.list_user_roles`` per user and
``storage.effective_role_permissions`` per (user, role) pair —
synchronous SQL inside an async handler. 200 users × 5 roles = 1000
connection cycles long enough to trip reverse-proxy timeouts on a
permission revoke.
Replaced with ``storage.users_with_permission(perm, *,
exclude_role_id)`` — one join over ``user_roles ⋈ roles`` plus one IN
fetch on overrides for the builtin role ids in the result, folded
in-process. Two queries total, independent of cluster size. The whole
check now runs under ``asyncio.to_thread`` so even the bulk read
doesn't stall the loop.
**F-2 reframed: admin_assign_role's subset check ignored the overlay.**
The check at lines 6321-6328 reads ``target_role.get("permissions",
"")`` (baseline column) when computing the perms it requires the
caller to hold. After this branch, an admin.roles holder can grant
e.g. ``model.skills.write`` to builtin-operator via override; an
admin.users holder (who happens to NOT hold that perm) could then
assign operator to a new user, silently escalating the assignee. The
existing two-person-rule by perm split (admin.roles for catalog edits,
admin.users for assignments) only holds if the assignment-time check
considers the overlay. Switched ``target_perms`` to
``storage.effective_role_permissions(role_id)["effective"]``.
Note: this PR retains the existing model where admin.roles is the
catalog-edit superuser (admin_create_role, admin_update_role, and now
admin_role_overrides all skip the caller-holds-grants check). The
two-person rule against escalation lives at the assignment gate, which
this fix reinforces.
**F-7: delete_role left orphaned override rows.** No FK on
``role_permission_overrides.role_id`` (migration 057 omitted FKs to
match the rest of the governance schema). Added explicit cleanup in
both sqlite + postgresql ``delete_role`` implementations so a
re-seeded role_id (deterministic for builtins on schema reseed) can't
silently inherit stale overrides from the prior occupant.
Tests
- storage: ``test_users_with_permission_bulk`` exercises the new bulk
helper including ``exclude_role_id`` and overlay folding
- storage: ``test_delete_role_cleans_up_overrides`` pins the F-7 fix
- endpoint: ``test_overrides_lockout_guard_blocks_grant_removal`` is
the F-1 reproduction — operator-overlay grants admin.roles, builtin-
admin has it removed, attempting to reset operator's overrides 409s
- endpoint: ``test_assign_role_blocks_escalation_via_overlay_grant``
pins the F-2 reframed fix — overlay-poisoned operator can't be
assigned by a caller missing the overlay perms
* refactor(rbac): cleanup batch from /review (#584)
Five non-security findings folded into one commit so the security
batch stays focused. All consistent with the existing intent of
``feat/builtin-role-overrides``.
**F-4: presence check on ``_effectivePerms``.** ``governance.js`` was
guarding on ``Array.isArray(role.effective) && role.effective.length > 0``,
falling through to splitting ``role.permissions`` (the baseline) when
the array was empty. For a builtin role whose overrides legitimately
revoke every baseline perm, that path silently rendered the baseline
chips with no override indicators — the inspector lied about what the
role can do. ``_enrich_role`` always sets ``effective: []``, so
presence is the right sentinel.
**F-5: JS-side drift detector.** Commit 1 added a Python-side test
asserting ``_VALID_PERMISSIONS`` covers every baseline perm; the
mirror invariant on the frontend went uncaught. A new perm added to
``_VALID_PERMISSIONS`` without a matching entry in
``_PERMISSION_SECTIONS`` becomes silently un-customizable through the
admin UI (the only documented grant/revoke path). Test parses the
JS const out via regex and asserts set-equality both directions —
detects "missing in UI" and "extra in UI" so the toggle catalog and
validator can't fork.
**F-6: bulk enrich for ``admin_list_roles``.** Was ``1 +
2*builtin_count + 1*custom_count`` SELECTs per admin-tab open;
collapsed to one ``IN``-filtered query via new
``storage.effective_role_permissions_bulk(role_ids)``. Implemented
on both sqlite + postgresql backends following the existing
``effective_role_permissions`` shape.
**F-8: rename ``fallback_permissions`` → ``accepted_permissions``.**
The lift body uses ``if cfg.permission_gate / elif accepted_permissions``
— mutually exclusive — so when ``permission_gate`` is None this IS
the primary gate, not a fallback to anything. The "fallback" name
suggested a tier-2-after-tier-1 semantic that didn't exist. Renamed
across ``make_{approve,close,create}_handler`` factories, the three
call sites in ``turnstone/server.py``, and the docstrings.
**F-9: positive lift-level tests for ``admin.coordinator``-only.**
``TestPermissionGatesOnLifecycle`` previously had a single positive
test for ``workstreams.create`` alone, plus negative-403 tests for
each verb without perms. The OR-fallback to ``admin.coordinator``
(which keeps coord sessions spawning interactive children unblocked)
had no positive coverage at the lift code path — only at the proxy,
which exercises a different verb-dict gate. Added three tests
(create / close / approve) that pass ``admin.coordinator`` alone and
assert non-403, so a future tightening of the accepted_permissions
tuple can't silently regress coord-driven child workstreams.
Out of scope: nit perf-4 (event-delegation refactor on
``_renderGovRoles``). ``setSafeHtml`` rebuild is the existing
pattern across every admin tab; rewriting one tab's render path on
this branch would be drive-by inconsistent with the surrounding
codebase. Filed as a separate concern if the Roles tab grows past
the scale where it bites.
* fix(rbac-ui): aria-expanded + row-click on Roles drawer (#585)
Two Copilot review findings on governance.js:
- Expand button was missing aria-expanded — screen readers couldn't
announce drawer state. Now reflects the row's expanded flag.
- Comment said "row + chevron both work" but only the chevron was
wired. Added data-expand-role to the row element too so the
existing handler loop (querySelectorAll on the attribute) picks up
both — clicking anywhere in the role row toggles the drawer.
Edit/Delete handlers already stopPropagation so they aren't
triggered by the row-level click.
* fix(migrations): rebase role_permission_overrides to 058
PR #560 mitigation #1 landed 057_output_assessments_llm_judge.py on
main in parallel; my migration claimed the same number, forking
alembic's head and breaking postgres. Renumbered to 058 and
re-pointed down_revision at 057 so the chain stays linear.
No behaviour change — same DDL. Full sweep clean (6730 passed).
* fix(migrations): update 058 revision strings to match filename
Previous commit (ea86aefc) renamed 057_role_permission_overrides.py to
058_* but the in-file revision = "057" / down_revision = "056"
strings stayed — leftover from when the file shipped as 057. Tests
pass because alembic walks the chain by revision string, and the
strings now correctly read revision = "058" / down_revision = "057"
to make the chain linear with main's 057_output_assessments_llm_judge.
Caught locally before re-running CI; my prior `git mv` + content edit
landed as a staged rename + unstaged modification on the previous
push.
`float("4.20") == 4.2`, so the old version-sort routed `grok-4.20`
under `grok-4.3` despite 4.20 being the newer dated-snapshot line.
Parsing each component as an int via `_version_tuple` makes
`(4, 20) > (4, 3)` as intended.
Applied symmetrically to the openai branch — same shape, same latent
bug against a future `gpt-5.10` vs `gpt-5.2` collision.
Locked in by four tests in `test_provider_xai.py::TestSelectBestModel`.
Spotted by Copilot review on #586.
Adds xAI as a first-class commercial provider through the officially-
documented server-to-server API-key path against https://api.x.ai/v1.
XAIProvider is a thin subclass of OpenAIResponsesProvider; xAI's
Responses surface is OpenAI-shaped, so the only override needed is
_build_kwargs, which merges <tool>_call_output strings into include[]
so xAI's server-side tool outputs (hidden by default) become visible.
GROK_CAPABILITIES covers the five documented chat models; aliases
like grok-4.3-latest resolve via the existing longest-prefix lookup.
Two narrow base-class generalisations earn their keep beyond Grok:
- ModelCapabilities.server_side_tools: tuple[str, ...] drives the
Responses-surface tool injection (previously hardcoded to
web_search). resolve_server_side_tools folds in the legacy
supports_web_search boolean for backward compat.
- extra_headers: dict[str, str] | None threaded through
LLMProvider.create_streaming / create_completion so callers can
pass x-grok-conv-id: <ws_id> for prompt-cache hit-rate. Session-
side population is a follow-up; the plumbing lands here.
Out of scope:
- OAuth (SuperGrok / X-Premium+) — not officially documented.
- Chat Completions surface — deprecated on xAI's comparison page.
- Image / voice / video models.
- argparse --provider xai in cli.py / server.py — Google isn't
there either; both providers configure via config.toml.
Closes#583.
Validated each of the 7 substantive Copilot findings + 5 CodeQL
findings via source spike; applied 7 (CP2-CP7, C3-C5), refuted 2
(CP1, C1+C2) with citation.
* CP2 (session.py:3551) — pre-truncation budget was reused per
output, allowing N parallel tool results to each claim the full
remaining context budget and collectively overflow. Maintain a
running budget that shrinks as each output is sized.
* CP3 (ratelimit.py) — TokenBucket.consume() mutated tokens /
last_refill without a lock; ChatSession._batch_evaluate_outputs
invokes it concurrently from up to 4 worker threads, so the rate
limiter's stated 60-call/min cap was best-effort. Added an
internal threading.Lock that protects consume() and retry_after.
RateLimiter's outer lock still holds for the bucket-dict it owns;
the new lock just makes the class safe-by-default for direct
consumers.
* CP4 (session_ui_base.py:1642) — docstring claimed "llm" rows
were only persisted on success; the session actually persists a
failure row too (with reasoning=error_reason) for audit. Match
the docstring to behavior.
* CP5 (migration 057) — header said "Revises: 055" but
down_revision was "056". Fix header.
* CP6 (_protocol.py:1761) — :func:`intent_verdicts` is a table,
not a callable. Plain reference.
* CP7 (test_output_guard_judge.py:271) — test_client_created_once
was self-contradictory: header comment claimed caching was being
verified, body asserted the patched fake was called once per
evaluate (no caching). The next test
(test_real_lazy_init_caches_real_client) covers actual caching;
drop the misleading one.
* C3-C5 (_extract_json) — three empty except-pass blocks now carry
a justifying one-liner explaining each strategy's expected failure
mode and what falls through.
REFUTED:
* CP1 (output_guard_judge.py:453 catch of TimeoutError vs
concurrent.futures.TimeoutError) — in Python 3.11+ (our minimum,
per pyproject.toml requires-python = ">=3.11") the two are the
same class. Verified at runtime: ``cf.TimeoutError is
TimeoutError`` is True; MRO is (TimeoutError, OSError, Exception,
BaseException, object). The polling loop catches the right
exception.
* C1/C2 (Protocol method bodies and test stubs using ``...``) —
`...` is the idiomatic Python pattern for Protocol method bodies
and one-line stub functions; the surrounding code in
test_model_registry.py uses ``...`` consistently across all stub
methods. CodeQL is flagging a single instance while ignoring
identical patterns nearby.
Adds a second, LLM-driven stage to the output guard so domain-camouflaged
prompt-injection payloads that the regex stage misses (arXiv:2605.22001 —
Llama 3.1 8B evades the existing regex set on ~90% of camouflaged
prompts) get caught before the tool output lands in the assistant's
context.
## Surface
* New `OutputGuardJudge` in `turnstone/core/output_guard_judge.py` —
synchronous, single-shot LLM call. Inlines the alias-resolution +
client-config + JSON-parsing helpers (copied verbatim from
`IntentJudge` at `judge.py:917-969` / `1604-1659`) rather than going
through a shared module — when `IntentJudge` lifts its own helpers,
both copies move together.
* JSON-in-content verdict with a 3-strategy parser (direct / markdown
fence / balanced braces). `IntentJudge` ships a 4th regex-field
fallback; OutputGuardJudge deliberately doesn't, because strategy-4
hits on broken LLM output can extract a "verdict" from the model's
reasoning quote that lands in storage looking identical to a clean
strategy-1 result. Failure of all three returns
`error="unparseable_verdict"` and the heuristic stage stands.
* `OutputJudgeVerdict` is a frozen dataclass with:
`risk_level` (none/low/medium/high — normalises `critical`→`high`
and `info[rmational]`→`low` for IntentJudge-echo safety),
`flags: tuple[str, ...]`, `reasoning`, `confidence: float`
(0.0-1.0, parsed + clamped from the LLM's self-report;
pass-through to audit, no threshold gating), `judge_model`,
`latency_ms`, `error`.
* Real wall-clock timeout via `ThreadPoolExecutor.shutdown(wait=False,
cancel_futures=True)` on the timeout/cancel path — `with ... as ex:`
would block return until the worker drained. 1s `cancel_event`
poll mirrors `IntentJudge._run_judge` at `judge.py:1117-1118`.
* HTTP client lazy-init + reuse for the judge instance's lifetime.
Session-side model swap drops the entire judge, dropping the client
with it.
* Untrusted tool output wrapped in per-call random-nonced
`<tool_output_NONCE>...</tool_output_NONCE>` fence. Closing-tag
substrings in the raw text are case-insensitively backslash-escaped
first (`</tool_output` → `<\/tool_output`) so an attacker can't
break out even if they guess the nonce. System prompt classifies
the fenced region as UNTRUSTED DATA so directives inside are
evaluated as content, not obeyed.
* Judge user prompt carries the heuristic verdict (risk + flags +
annotations), the tool description (looked up from the session's
tools registry), and the tool args (truncated to 500 chars, also
classified UNTRUSTED in the system prompt since they may be
caller-supplied). Lets the judge defer to the regex on credential
leaks and focus on injection signals the regex set misses; also
enables output-vs-request plausibility reasoning.
## Session integration
* `_evaluate_output(call_id, output, func_name, *, tool_args="")` —
heuristic always runs; LLM stage runs when `judge.output_guard_llm`
is enabled. When the LLM produces a usable verdict and the
heuristic didn't detect credentials, the LLM verdict is acted on;
otherwise the heuristic stands.
* Credential redaction is a regex-only signal. When `heuristic.
sanitized` is non-None, the heuristic owns the acted assessment
regardless of what the LLM said — an LLM asked about prompt-
injection can correctly label a credential-bearing output as
"none" risk for injection, but the secret still needs redaction.
* `_batch_evaluate_outputs` runs the per-tool guard concurrently
(4-worker pool) when LLM is enabled and there are ≥2 string
outputs — collapses N×LLM-latency to ⌈N/4⌉×latency on the common
5-20 tool-calls-per-turn turn.
* Per-session `TokenBucket(rate=1.0, burst=60)` caps adversarial
LLM-fan-out cost at 60 calls/min/session.
* Pre-truncation: the per-tool loop truncates output before the
judge sees it, so the judge evaluates exactly what enters the
assistant's context (no wasted tokens on text that won't land).
* Both heuristic and LLM tier rows persisted to `output_assessments`
when the LLM ran (audit completeness); heuristic-only rows skip
when matched-clean to keep the table focused.
## Storage
Migration 057 extends `output_assessments` with five LLM-tier
columns: `tier` (`heuristic` / `llm`, backfilled to `heuristic`),
`reasoning`, `judge_model`, `latency_ms`, `confidence`. Tie-break
on `(created DESC, tier='llm' first)` so downstream consumers see
the acted verdict first when the two rows tie at second resolution.
`StorageBackend.record_output_assessment` + sqlite/pg implementations
+ `SessionUIBase.record_output_assessment` + `SessionUI` protocol +
the test stub overrides (cli, eval, 9 test files) all take the new
LLM-tier kwargs.
## Config surface
Three new judge.* settings in `settings_registry`:
* `judge.output_guard_llm` (bool, default False) — capability gate.
Default off; operators opt in once a small/fast model is pointed
at `output_guard_model`.
* `judge.output_guard_model` (str, default "") — alias for the LLM
stage. Empty inherits the session model (same fallback shape as
`judge.model`).
* `judge.output_guard_llm_timeout` (float, default 30.0, min 1.0) —
wall-clock budget per call.
Both `server.py` and `console/session_factory.py` wire these into
the `JudgeConfig` they hand to `ChatSession`.
## Notes
* No backwards-compatibility shims — the LLM stage is purely additive.
* No reasoning/threshold gating on confidence; it rides as an
audit-only signal per maintainer direction. Surface it in the
`on_output_warning` dict so live UI / cluster broadcast can sort
flagged outputs by judge certainty.
* Tests: 392 lines of judge-only coverage (`test_output_guard_judge.
py`) + 629 lines of session-integration coverage in `test_session.
py`, plus the storage and stub-shape updates.
Two related cleanups landed together because they touch the same surface
(skill-spec uplift PRs #569/#570/#571/#572):
1. Wording: replace "Anthropic spec" / "Anthropic Claude Code skill spec"
with "SKILL.md spec" across admin UI tooltips, code comments, test
docstrings, migration 056's module docstring, and the user-facing
`arguments` description in tools/skills.json. Renames a parser test
`test_anthropic_tags` -> `test_nested_metadata_tags` and consolidates
a parse-API test of the same shape; fixture author renamed
`Anthropic` -> `Acme` to keep the fixture neutral. Legitimate
provider/SDK/API references (provider name, api.anthropic.com,
`_anthropic.py`, capability comments) are intentionally untouched.
2. Admin UX: in the Create + Edit Skill modals, six fields per modal
(Compatibility, Paths, Hide-from-skill-picker, Arguments, Argument
hint, Activation) had long uppercase label-hint spans crammed into
the visible label. Migrated each to the existing
`.settings-help-btn` + `.settings-help-popover` pattern already used
in the Settings tab — short label + inline `?` button that opens a
styled popover with proper `<code>` formatting for technical tokens.
Pattern reuse required two small generalisations in admin.js:
* `_toggleSettingsHelp` now looks up the popover via a new
`data-help-target="<id>"` attribute first, falling back to the
settings-tab `.settings-label-col` ancestor lookup.
* `_closeAllSettingsHelp` mirrors the same dual-path lookup when
resetting `aria-expanded`, so modal buttons don't get stuck on
`aria-expanded="true"` after another popover opens.
* Added a document-delegated click handler that fires only for
buttons with `data-help-target`; existing per-button binding
in the settings-tab render path is unchanged.
CSS: `.settings-help-btn` now paints its `?` via `::after` with the
button's own `font-size: 0`, so prettier-introduced whitespace
inside the new HTML buttons can't off-center the glyph. The same
rule applies to existing admin.js-generated buttons (text content
hidden, pseudo identical). Small additions for
`.settings-help-popover code` / `strong` styling so technical
tokens render with the same monospace pill treatment used elsewhere
in skill UI.
Known follow-ups (intentionally NOT in this PR):
* Migrate the settings-tab `_renderSettingRow` button assembly to the
empty-`<button>` + `data-help-target` form so the per-button
addEventListener loop can be dropped in favour of pure document
delegation, and the `font-size: 0` rule stops being a workaround for
two markup styles.
* The 12 new popover blocks are duplicated verbatim between the
Create and Edit modals (same as the rest of the create/edit modal
pair). A small renderer that emits popovers from a shared data
object would eliminate the drift risk but is unrelated cleanup.
* fix(coord): strip intent-judge verdicts from inspect_workstream output
Coordinator LLMs repeatedly misread `user_decision="policy"` (the label
meaning "auto-approved by an admin policy allow rule") as "blocked,
waiting for policy review" — combined with `recommendation="review"`
(the heuristic judge's risk class, not a workflow state) the verdict
fields read end-to-end as "stuck on policy review" and produced
incorrect cancel-and-respawn reasoning against healthy children.
The blocking signal already lives on `state` (`"attention"`) and the
`live.pending_approval` block, both still in the result. Verdict
history remains queryable through admin / audit surfaces — only the
LLM-facing inspect surface drops them.
Also drops `verdict_count` / `verdicts_by_risk` from the tier-3
skeleton fallback, deletes the now-dead `_serialize_verdicts` helper,
and clears the now-stale `"verdicts": []` keys from 10 fixture sites
that fed `_format_inspect_tiered` test cases.
* fix(coord): correct comment pointer — inline comment, not docstring
Implements the Anthropic Claude Code skill spec's placeholder
substitution end to end. The renderer in ``_substitute_skill_args``
handles every spec form except ``\${CLAUDE_SKILL_DIR}`` (deferred):
* ``\$ARGUMENTS`` — full args string as the user/model typed it
* ``\$ARGUMENTS[N]`` / ``\$N`` — Nth positional arg, ``shlex.split``-parsed
* ``\$<name>`` — named arg from the SKILL.md ``arguments:`` list
* ``\${CLAUDE_SESSION_ID}`` / ``\${CLAUDE_EFFORT}`` — session state
Substitution is single-pass (one combined regex, one ``re.sub``).
Append rule: when args are passed but the body has no bare
``\$ARGUMENTS``, append ``ARGUMENTS: …`` at the end.
## Surface
* Parser: ``arguments:`` (list/space-delim) + ``argument-hint:`` (str)
extracted into ``ParsedSkill``.
* Install: persists both to the pre-allocated columns from migration
056 (PR #574). Install path clamps ``argument_hint`` to 128 chars
to match the admin-create cap (untrusted upstream source).
* Admin: ``CreateSkillRequest`` / ``UpdateSkillRequest`` accept both
fields; create + edit modals get inputs; parse-preview echoes.
* Renderer: ``_substitute_skill_args`` runs AFTER ``_render_template``
in ``_load_skills`` so user-supplied args containing ``{{var}}`` can't
be re-expanded by the legacy renderer.
* Session: ``_skill_arguments`` plumbed through ``__init__``,
``set_skill``, and ``_save_config`` so a resumed workstream re-renders
with the original arg payload.
* Model tool: ``skills(action='load')`` accepts an ``arguments`` string.
Approval label includes a SHA-256 digest of the args so a once-
approved skill name can't grant cover for a future payload; preview
surfaces the args inline.
## ``/review`` findings (addressed)
* ``\${CLAUDE_EFFORT}`` referenced ``self._reasoning_effort`` — wrong
attribute; the real one is ``self.reasoning_effort``. Always rendered
empty. Fixed.
* Two-pass layering let user args containing ``{{var}}`` re-expand.
Render order reversed.
* ``_skill_arguments`` wasn't in ``_save_config`` — resumed workstreams
silently lost their payload. Added.
* Approval label omitted ``arguments``. Digest + preview added.
* Install path didn't bound ``argument_hint``. Clamped.
* Added ``_skill_arg_names`` decode tests + "load same skill,
different args → re-render" invariant test.
## Copilot review findings (addressed)
* ``skills.json`` tool description was inaccurate about ``shlex``
stripping quotes and "empty string disables substitution". Rewrote
to match actual behaviour.
* Named-argument regex was stricter than parser/storage contract.
``arguments: [issue-number]`` would partial-match ``\$issue-number``
as ``\$issue``, leaving ``-number`` as stray text. Broadened the
regex to ``[A-Za-z_][A-Za-z0-9_]*`` AND added validation at
``_skill_arg_names`` decode time so names not matching the regex
are dropped with a warning.
## Tests
* ``tests/test_substitute_skill_args.py`` — placeholder forms,
single-pass guarantee, append-at-end rule, shell-quoted input,
unbalanced-quote fallback, uppercase + underscore-prefix names
* ``tests/test_skill_parser.py::TestArgumentsAndHint`` — parser
extraction
* ``tests/test_skill_parse_api.py`` — HTTP parse-preview echoes
both fields
* ``tests/test_skill_discovery_api.py::test_install_seeds_arguments_and_argument_hint``
— install round-trip
* ``tests/test_skills_tool.py::test_load_forwards_arguments_to_set_skill``
+ ``test_load_same_skill_different_args_triggers_resub`` —
wire path through prepare → exec → set_skill
* ``tests/test_skills_tool.py::TestSkillArgNames`` — storage decode
helper including the hyphen/dot/leading-digit filter
Two findings from Copilot's review of PR #577:
* ``_extract_bool`` int branch: Copilot flagged that ``bool(raw)``
treats any non-zero int as True, so ``disable-model-invocation: 2``
silently disables model invocation without warning the author about
the typo. Tightened to accept only ``0`` and ``1`` as integer
boolean forms — anything else falls back to *default*. Matches
spec (which mentions only 0/1) and the broader principle that
ambiguous input should not coerce silently.
* ``hidden_from_menu`` admin body parse: Copilot flagged that
``bool(body.get("hidden_from_menu", False))`` treats non-empty
strings via Python truthiness, so a malformed client sending
``"false"`` would flip the flag to ``True`` — opposite to obvious
intent. Extracted a ``_parse_strict_bool`` helper that accepts
only Python ``bool`` or int ``0``/``1`` and returns a 400 on
anything else. Applied at both admin create and admin update
sites; the install path remains untouched because it derives the
flag from the typed ``ParsedSkill.user_invocable`` field (not raw
HTTP body).
## Tests
* ``test_other_ints_fall_back_to_default`` — ``2`` and ``-1`` no
longer silently coerce
* ``test_create_skill_hidden_from_menu_string_rejected`` — string
``"false"`` returns 400
* ``test_create_skill_hidden_from_menu_int_zero_and_one_accepted``
— 0 / 1 accepted, 2 rejected with 400
No behaviour change to the main surface — both fixes close latent
type-coerce hazards a malformed input could have exploited.
The Anthropic Claude Code skill spec defines two invocation-control
axes Turnstone was parsing but not consuming:
* ``disable-model-invocation: true`` — model can't autoload this skill
(only user can invoke by name). Stored on ``ParsedSkill`` and
echoed on the parse-preview UI; no install consumer because
Turnstone hardcodes ``activation="named"`` on source-installs
already. The dataclass docstring spells out the no-op so a future
reader doesn't try to wire a translation that's already implicit.
* ``user-invocable: false`` — skill stays available to the model but
disappears from the user-facing picker. Mapped to
``hidden_from_menu=true`` on ``prompt_templates`` (column
pre-allocated by PR #574); consumed by ``list_skills_summary``
(both the standalone-server and console-server impls).
## Surface
* Parser: new ``_extract_bool`` helper accepts every YAML 1.1
boolean spelling (true/false/yes/no/on/off/1/0) plus their quoted
variants — caught by ``/review`` as a real gap, since YAML's
``safe_load`` returns ``int`` for unquoted ``1``/``0`` and ``str``
for the YAML 1.1 spellings when quoted.
* Install handler: derives ``hidden_from_menu`` from
``parsed.user_invocable`` on the source-install path.
* Admin: ``CreateSkillRequest`` / ``UpdateSkillRequest`` accept
``hidden_from_menu``; both modals get a checkbox; the parse-preview
auto-fill flips it when the source SKILL.md sets
``user-invocable: false``.
* Runtime config: ``hidden_from_menu`` joined
``SKILL_RUNTIME_CONFIG_FIELDS`` so admin can override on installed
(readonly) skills — same precedent as ``model`` / ``effort``.
## list_skills_summary shared helper
Two identical implementations of ``list_skills_summary`` had
accreted in ``turnstone/server.py`` and ``turnstone/console/server.py``.
Both needed the new ``hidden_from_menu`` filter, so extracted the
shared body to ``turnstone/core/web_helpers.skill_summary_rows``.
Future spec-uplift fields (e.g. #572's ``argument_hint`` for
autocomplete) only touch one place now.
## Tests
* ``TestInvocationControl`` — bool / quoted / YAML 1.1 / int variants
across both fields
* ``test_install_user_invocable_false_sets_hidden_from_menu`` +
default-unhidden case
* ``test_list_skills_summary_excludes_hidden_from_menu`` — picker
filter, admin tab unaffected
* ``test_update_skill_readonly_hidden_from_menu_allowed`` — admin
can hide/unhide installed skills via PUT (pins the runtime-config
membership invariant)
* Existing parse-API fixture extended with both new fields plus
default-case assertions
The Anthropic Claude Code skill spec defines three frontmatter fields
the parser was previously dropping; this PR wires them through to the
existing storage shape so the SKILL.md author's intent survives the
import.
* ``when_to_use`` — concatenated into ``description`` at parse time
with a ``\n\nWhen to use: `` separator. Kept as its own field on
``ParsedSkill`` so the admin parse-preview UI can surface it
separately.
* ``model`` — passed through to ``create_prompt_template(model=...)``
on the source-install path, seeding the existing
``prompt_templates.model`` column.
* ``effort`` — same shape, translates to the existing
``reasoning_effort`` column at the install handler boundary.
Re-install short-circuits at the source_url dedup, so admin overrides
to either column survive an upstream re-install — covered by a new
``test_reinstall_preserves_admin_model_override`` test that pins the
load-bearing invariant.
## Description length cap
``_MAX_DESCRIPTION_LEN`` exported as ``MAX_SKILL_DESCRIPTION_LEN``
(public name) and raised from 1024 to 1536 to match the spec's
combined ``description`` + ``when_to_use`` listing budget. All five
write surfaces now import the same constant rather than each carrying
their own magic number:
* ``skill_parser.MAX_SKILL_DESCRIPTION_LEN`` — parse-time cap
* ``console_schemas.CreateSkillRequest.description`` — Pydantic
* ``console_schemas.UpdateSkillRequest.description`` — Pydantic
* ``console/server.admin_create_skill`` — handler slice
* ``console/server.admin_update_skill`` — handler slice
* ``core/session._exec_skills_create`` — coordinator tool slice
* ``core/session._exec_skills_update`` — coordinator tool slice
The coordinator sites (last two) were the bug ``/review`` caught:
they still capped at 1024 after the rest of the surface bumped to
1536, so a model-issued ``skills(action='create')`` with a 1025-1536
char description would silently truncate. Sharing the constant
closes that desync.
## when_to_use truncation guard
The ``when_to_use`` concat reserves room for the separator + at
least one character of the appended value; below that budget, the
addition is dropped entirely. Previously the naive concat could
truncate mid-separator and leave the description ending in a
dangling ``\n\nWhen ``.
## Tests
* ``TestWhenToUse`` — concat semantics, no-description fallback,
1536 truncation
* ``TestModelAndEffort`` — extraction + defaults
* ``test_install_seeds_model_and_effort_from_frontmatter`` — install
path persists both columns
* ``test_install_no_model_or_effort_leaves_columns_empty`` — bare
SKILL.md doesn't invent values
* ``test_reinstall_preserves_admin_model_override`` — admin edits
survive an upstream re-install (dedup invariant)
* ``test_parses_full_frontmatter`` / ``test_parses_minimal_frontmatter``
extended with the new field assertions
Three findings from Copilot's review of PR #574:
* `update_prompt_template` (both backends) coerces every other
INTEGER-as-bool field (`is_default`, `auto_approve`, `enabled`) but
not `hidden_from_menu`. Without coercion, a caller updating with
``hidden_from_menu=True`` writes a Python bool to a SQLAlchemy
Integer column, which is driver-dependent on PostgreSQL and a
consistency hazard. Coerce to int alongside the existing trio.
Added a focused round-trip test that updates with ``True`` /
``False`` and asserts the read-back bool transitions.
* `_canonicalize_skill_string_list` docstring describes its own
``None``-collapses-to-``"[]"`` rule but doesn't mention that
`admin_update_skill` intercepts ``None`` before the helper is
called. Added a note documenting the layered contract: the helper
defines normalization (create semantics), the update endpoint
layers no-op semantics on top.
* HTML label hints used Markdown-style ``paths:`` backticks inside
plain HTML, which render as literal backticks in the browser.
Replaced with `<code>paths:</code>` on both the create and edit
modal Paths inputs.
No behaviour change to PR #574's main surface — the bool coercion
addresses a latent bug a future consumer would have hit; the
docstring + HTML fix are purely cosmetic.
Implements PR1 of issue #569 — parser + storage + admin UI for the
Anthropic Claude Code skill spec `paths:` SKILL.md frontmatter field
(glob patterns gating model-initiated autoload). The autoload filter
that consumes `paths` is deferred to a follow-up PR pending the
workstream-CWD design discussion.
Migration 056 bundles three additional columns whose consumer PRs are
filed but not yet implemented:
* `hidden_from_menu` (boolean) — backs the spec's `user-invocable:
false` (issue #571).
* `arguments` (JSON list) — backs spec `arguments:` named arg slots
(issue #572).
* `argument_hint` (string) — autocomplete display string (issue #572).
The deferred columns surface in `SkillInfo` (response) so consumers can
read them, but are deliberately absent from `CreateSkillRequest` and
`UpdateSkillRequest` — the create/update handlers don't yet read them
and advertising a writable field the handler would silently ignore
would be an OpenAPI lie.
Surface
- Parser: `ParsedSkill.paths` populated from frontmatter; accepts the
spec's YAML-list-or-CSV-string shape via the existing
`_extract_list` machinery.
- Storage: 4 new columns on `prompt_templates`; `SKILL_MUTABLE`
extended; `_row_to_dict` calls extended to cast the new bool;
protocol + SQLite + PostgreSQL `create_prompt_template` signatures
threaded.
- HTTP: admin create/update/install/parse handlers plumb `paths`
through. Pydantic schemas extended accordingly.
- Admin UI: `skill-paths` and `etm-paths` inputs on the create + edit
modals; field map and read/write helpers wired across paste-parse,
reset, create-send, edit-load, edit-send, and the readonly-disable
list.
Notable
- `_canonicalize_skill_string_list` collapses the list-or-CSV-or-JSON-
string normalization shared between admin_create_skill and
admin_update_skill. Treats `None` as no-value so a body containing
`{"paths": null}` doesn't CSV-split through `str(None)` and store
the literal `["None"]`. Will back `arguments` once #572 wires its
consumer.
Tests
- Parser: TestPaths covers YAML list, CSV string, empty, full-
frontmatter integration (tests/test_skill_parser.py).
- Storage: round-trip suite covers create + read + update for each
of the four new columns on both backends
(tests/test_storage_skill_spec_uplift.py).
- Helper: focused unit tests for the canonicalizer including the
regression-net case for the null-corruption bug
(tests/test_canonicalize_skill_string_list.py).
- HTTP boundary: extended test_parses_full_frontmatter +
test_parses_minimal_frontmatter to assert `paths` survives the
admin parse endpoint.
* fix(output_guard): harden against domain-camouflaged injection (#560)
Three layered mitigations against the camouflage attack class described in
arXiv:2605.22001 (Pai, May 2026), which demonstrates 90.3% evasion on Llama
3.1 8B and 44.4% on Gemini 2.0 Flash against pattern-based detectors:
- Sub-agent synthesis is now scanned by output_guard at the sub-agent
boundary in _run_agent, in addition to the existing scan at the parent's
tool-result loop. Covers all four return paths (clean exit, truncation,
context-limit recovery, turn-limit forced synthesis), closing the
cross-workstream summary laundering surface.
- Adds pair-of-signals camouflage detection: imperative recommendation
phrase combined with either an authority frame ("consistent with our
risk framework") or a caps action verb (SELL/BUY/TRANSFER/...). New
flag camouflaged_injection at medium risk; deliberately partial — the
paper's augmented-detector approach recovers only ~10% on Llama-class
models, so this is duct-tape pending a semantic-evaluator follow-up.
- Bumps output_guard's wall-clock budget default from 5s to 30s and
exposes it as judge.output_guard_budget_seconds in ConfigStore, so the
expanded regex set has headroom on large tool outputs.
* Fix test_budget_kwarg_is_honored to exercise deadline logic path
The test previously passed an empty string which short-circuited
evaluate_output() before budget_seconds was used. Now uses a non-empty
input and monkeypatches time.monotonic() to deterministically verify
the deadline path is exercised.
Single-$ inline math is too ambiguous in conversational text: currency
amounts ("$5 and $10 each"), shell variables ("$HOME and $PATH"), and
shell prompts all produced false-positive KaTeX spans because the
regex matched any non-$/non-newline span between two dollar signs.
Inline math now requires the unambiguous \(...\) form, which is what
GPT-5 / o-series / Claude with reasoning effort emit by default anyway.
Display math ($$...$$ and \[...\]) is unchanged — the doubled
delimiter has enough mass that ambiguity is not a practical problem.
Three former positive tests are inverted into regression guards so a
future regex change can't quietly resurrect the bug, and new tests
name the currency and env-var cases explicitly. The web env prompt is
updated to advertise \(...\) and to tell the model why $...$ is gone.
5 follow-up comments from Copilot, all valid:
1. **CRITICAL — snap_seq race with split writer** (concurrency, 001).
Round-1's fix lifted snapshot capture into
register_listener_with_replay under nested locks, but the
WRITER side (on_content_token / on_reasoning_token) still
released _ws_lock before calling _enqueue (which bumps
_event_id under _listeners_lock). A reader could
interleave between writer's release and writer's _enqueue:
capture inflight WITH the new text, read STALE _event_id,
return snap_seq < new_event_id. The new event's live emit
then has _seq > snap_seq, slips past the dedup filter, and
double-renders text the snapshot already contained.
Fix: move self._enqueue(...) INSIDE the with self._ws_lock:
block in both token writers. The inflight mutation and the
_event_id advancement are now atomic against any snapshot
reader. Lock order _ws_lock (outer) → _listeners_lock
(inner via _enqueue) matches the snapshot helpers, so no
deadlock. Fan-out's put_nowait calls happen under
_ws_lock for token writers — microsecond cost per listener,
acceptable for the correctness guarantee.
2. **NIT — stale comment ref to buffered[-1]._event_id** (docs, 002).
The comment referenced a local var (buffered) that lives in
register_listener_with_replay, not in the events handler.
Reworded to describe the cutoff in terms of the last replayed
event id and the atomic-against-writers registration.
3. **MODERATE — 401 branch leaves reconnect loop** (bug, 003).
The coord's onerror schedules a 5 s CLOSED-state recovery timer
unconditionally. In the 401-expired-session branch we close
evtSource and showLogin — but the timer still fires 5 s later,
observes !evtSource, and calls scheduleReconnect(),
which opens a new EventSource that 401s again → infinite
reconnect loop while the login overlay is up. Fix: cancel
reconnectTimer in the 401 branch.
4. **MODERATE — race test was vacuous** (test_coverage, 004).
The previous regression test drained the listener queue after
register_listener_with_replay returned, but the helper
doesn't backfill buffered events into the queue, so the loop
was almost always a no-op and the assertion never executed.
Rewrote with a monkey-patched _enqueue that sleeps 50 ms
before bumping _event_id — widens the race window
deterministically. Verified: the test FAILS on pre-fix code
(snap.content has marker but snap.seq=0 < final_event_id=1)
and PASSES on post-fix code (writer holds _ws_lock through
_enqueue, so the reader blocks until writer fully done).
Also pinned the no-backfill contract so a future change adding
listener-queue backfill remembers to keep snap_seq the
high-water mark.
5. **NIT — except Exception too broad in test** (best_practices, 005).
Tightened except Exception: to except queue.Empty: so
unexpected exceptions aren't silently swallowed in the drain
loop.
Tests:
- 86 tests in test_sse_reconnect_replay.py + test_session_ui_base.py
pass (existing 84 + 2 new race regressions).
- Full non-live suite: 6347 passed, 15 skipped, no regressions.
- Ruff + mypy clean on changed .py files; JS parses.
Four issues raised on the merged PR #542, evaluated and fixed:
1. **Truncated-path snap_seq bug (Copilot low-confidence, VALID).**
make_events_handler's truncated branch set snap_seq = 0,
disabling the live-drain _seq <= snap_seq dedup filter. Any
token writer racing between register_listener_with_replay
returning and the live drain's first read would land in BOTH the
listener queue AND the captured snapshot text (the snapshot is
emitted via in_progress_snapshot as the recovery floor), so
the client double-renders. Fix lifts the snapshot capture INTO
register_listener_with_replay under the same nested-lock
acquire as the listener registration + buffer slice + counter
read, so the returned snapshot["seq"] is the exact
high-water mark the snapshot text corresponds to. Handler now
uses snapshot["seq"] as snap_seq on truncated, dropping
any token event with _seq <= snap_seq from the live emit.
2. **Lock-held string join in truncated path (Copilot, VALID).**
"".join(ui_base._ws_inflight_content) ran inside the
with ui_base._ws_lock: block, holding the lock for the
duration of the join and blocking on-token writers. Fix (folded
into #1's refactor): the new register_listener_with_replay
copies the inflight lists under lock and joins outside, matching
the existing pattern in
register_listener_with_in_progress_snapshot.
3. **_strip_js_comments docstring misclaim (Copilot, VALID).**
Docstring claimed the helper preserves "string/regex literals"
but the implementation only tracks string delimiters. Fix:
docstring updated to call out the regex-literal limitation
explicitly + note that current callers don't scan regions
containing regex literals. Extending the tracker is left for
a future caller that needs it.
4. **Coord scheduleReconnect dead-code regression (Copilot, VALID).**
After the PR-D refactor, scheduleReconnect() had no remaining
call sites — which meant reconnectAttempts never incremented,
wasReconnecting was always false, AND there was no
fallback when the browser transitioned the source to CLOSED
(hard 4xx after retries, intermediary tearing the connection
down with prejudice, etc.). The first failure mode silently
broke the post-gap replace-mode refresh of children / tasks /
wait indicator / live-badge cache; the second left the coord
permanently disconnected on non-transient failures. Fix:
- Introduce disconnectedSinceLastOpen flag set in onerror,
cleared in onopen. wasReconnecting reads it (with the
legacy reconnectAttempts > 0 fallback for the
scheduleReconnect-driven case), so the post-gap refresh fires
after every reconnect including the common native-reconnect
path.
- Re-introduce CLOSED-state recovery: onerror schedules a 5 s
delayed check via reconnectTimer; if the source is still
CLOSED at that point, call scheduleReconnect(), which
opens a new EventSource (threading the saved
lastEventId via the URL query param so replay still works
across the manual reconnect). Cancel/replace successive
timers so onerror floods don't pile up multiple checks for
the same source.
Tests:
- New test_truncated_path_snapshot_captures_real_snap_seq pins
the snap_seq fix at the helper boundary.
- New test_truncated_path_filters_already_in_snapshot_tokens
pins the end-to-end dedup invariant — would have caught the
double-render under the old code.
- Existing test_sse_reconnect_replay.py call sites updated for
the new 6-tuple return of register_listener_with_replay.
- All 86 tests in those two files pass; full non-live suite (6347
tests) passes; ruff + mypy clean on changed files; both JS files
parse-check.
Four /review findings collapsed to one code chokepoint + two
documentation fixes:
1. find's `kind` arg now validated against ``SkillKind`` (matching
create / update's existing pattern at session.py:8298 / :8512).
Closes two failure modes that shared the same root:
- typos (`kind="interactivee"`) silently produced
`kinds=["interactivee", "any"]` filtering to literal-`any` rows
only and masquerading as a narrowed catalog — now returns an
explicit "kind must be one of: ..." error;
- the documented enum value `kind="any"` degenerated to
`kinds=["any", "any"]` which narrowed to literal-`any` rows
instead of returning "every kind" — now collapses to ``None``
so the documented semantic holds.
2. docs/coordinator-skills.md "two-surface model" section rewritten
to reflect the post-flatten reality: kind is metadata, not an
enforcement boundary. The line-67 tools-table row updated from
the long-dead `list_skills` to `skills (action=find)` with the
opt-in kind-filter framing.
3. Three stale "interactive-only" comments in session.py
(:5514, :7857, :8210) that directly contradicted the
`_prepare_skills_load` docstring ("Both kinds can load") — drop
the qualifier so future grep-and-encode hazards don't reintroduce
the rejection.
Tests:
- test_find_kind_invalid_errors — typo case (replaces the silent
degenerate to literal-any-only)
- test_find_kind_any_means_no_filter — documented enum value matches
documented semantic (collapses to None at prepare)
- test_find_kind_narrow_passes_through — valid narrowing values
reach exec as expected
Deferred to release notes (no code change, intentional policy shift):
- skills(action='get') / load can now read full content + scan_report +
allowed_tools on cross-kind rows from any session. Operators with
pre-existing kind=coordinator skills authored under the prior
implicit visibility contract should audit those bodies for
sensitive content (allowed_tools allowlists, embedded credentials,
internal hostnames in examples) before upgrade.
Closes#557. SkillKind was authored audience metadata that the
discoverability filter dressed up as a runtime visibility gate. Real
access control is allowed_tools + auto_approve, which apply identically
across kinds. The kind-scoping chokepoints scaled linearly with every
new model-write surface for zero security payoff.
Drop kind consultation from:
- ChatSession._skills_kinds (deleted) and ._lookup_visible_skill
(deleted; callers inlined to storage.get_prompt_template_by_name).
- _exec_skills_find: no longer auto-threads kinds=. The opt-in `kind`
arg is a passable filter (threads [<kind>, "any"]) so the
discoverability win survives without enforcement.
- _exec_skills_get / _exec_skills_load: row lookup is name-only.
Disabled-row gate stays on load (admin quarantine is the actual
boundary). _prepare_task already uses unscoped get_skill_by_name;
session_routes.py already calls storage directly with no kind
check. Both confirmed by the spike, no source change needed.
- tools/skills.json: drop "Coord sessions see / interactive sees"
language; kind arg description re-cast as opt-in discoverability
narrowing.
- storage Protocol docstring + console_schemas.py kind field
description: refresh to reflect passive-metadata role.
Keep:
- SkillKind enum, kind column on prompt_templates, admin Skills tab
editing, kind field in skills.find / skills.get projection. The
field is useful for sorting/grouping at the model layer and as
authored intent.
- storage.list_skills_filtered(kinds=...) parameter — admin-filter
only now; docstring updated to note it's no longer auto-threaded
from the model-tool path.
Design calls:
1. find accepts opt-in `kind` arg: YES. ~5 lines on prepare + exec.
Threads kinds=[<kind>, "any"] only when supplied. Preserves the
model's ability to narrow a browse without enforcing.
2. kind field stays in find/get projection: YES. Already pulled
directly from the row dict in _skills_project_row (session.py
line 8187); the projection survives the flatten unchanged.
Tests:
- Delete TestLookupVisibleSkill (helper gone), the two
TestExecSkillsLoadKindScoping cross-kind reject branches, the two
test_find_kind_scoping_* tests, and test_get_cross_kind_returns_not_found
— the rejections those pinned are gone.
- Add test_find_default_threads_no_kind_filter (kinds=None by default
for both session kinds), test_find_returns_all_kinds_for_session
(interactive sees both interactive- and coord-tagged rows),
test_find_filters_by_kind_when_supplied (opt-in narrowing works),
test_get_returns_row_across_kinds (cross-kind get succeeds),
test_load_works_across_kinds (cross-kind load succeeds in both
directions — the flatten contract), test_load_rejects_missing_skill
(missing-row hint coverage). Keep test_load_rejects_disabled_skill
(admin quarantine still applies), test_load_works_for_coord_on_*
(coord-side load still works on more kinds now).
Storage tests untouched: tests/test_storage_skills_filtered.py
keeps its kinds= coverage (the parameter still works, just no longer
auto-threaded from the model-tool path).
Closes review findings from PR #555 that motivated the rethink:
sec-1 (_prepare_task unscoped) moot, sec-2 (HTTP create unscoped)
moot, sec-3 (no audit on cross-kind probes) moot — there is no
cross-kind concept anymore.
Boundary spike (verified against fresh main at 9a98d07d):
- _skills_kinds defined at session.py:7933, callers exactly two:
_lookup_visible_skill (7978) + _exec_skills_find (8057, 8109).
Verified via grep across turnstone/.
- _lookup_visible_skill defined at session.py:7943, callers exactly
two: _exec_skills_get (8213) + _exec_skills_load (8277). Verified
via grep.
- _skills_project_row reads kind directly from the row dict
(r.get("kind") or "any") at session.py:8187 — no helper call;
projection survives flatten.
- _prepare_task at session.py:6353 calls unscoped get_skill_by_name
— no kind check, no change needed.
- session_routes.py:1985 calls storage.get_prompt_template_by_name
directly with no kind check — already flat.
- storage.list_skills_filtered(kinds=...) parameter is identical
across _sqlite.py:2986, _postgresql.py:2827, and _protocol.py:1364.
- tests/test_skills_tool.py: TestLookupVisibleSkill (5 tests, lines
481-536) and TestExecSkillsLoadKindScoping (5 tests, lines 539-639)
pre-flatten. Total: 61 collected → 57 collected post-flatten.
Net production LOC: -15.
The local-escape posture from the prior commit double-encoded values
that inlineMarkdown had already escaped: leading escapeHtml(text)
turns `&` into `&`, the local escapeHtml(url) then turned that
into `&amp;`, which breaks query-string URLs after browser parse
+ getAttribute + new URL round-trip.
Switch to convention-rename: regex callback params renamed to
safeAlt / safeUrl / safeLabel to signal the upstream-escape
invariant. The attribute-context lint enforces all future
attribute-context concat sites maintain the safe* convention or
call escapeHtml explicitly — defense-in-depth preserved without
the regression. Two added pin tests verify `&` survives with
single (not double) entity encoding through image data-src and
link href.
Also addresses two test issues from the same review:
- Docstring listed `safe[A-Z_]…` but code only checked isupper().
Drop the underscore option (JS uses camelCase anyway).
- `_all_attr_names` only recorded attribute-bearing tags, so a
bare `<script>` injection would have false-negatived the link-
label pin test. Refactored to `_parse_renderer_html` returning
both start tags and (tag, attr) pairs.
inlineMarkdown's image and link renderers now escapeHtml each
interpolated value (url, alt, label, domain) at the call site
instead of relying on the upstream escape pass. Defence-in-depth:
a future refactor calling those renderers from outside
inlineMarkdown would otherwise silently regress.
New CI lint scans renderer.js for `attr="' + ident` patterns; ident
must be escapeHtml(...), safe*, or in the reviewer-approved
allowlist. Four pin tests use html.parser.HTMLParser to verify
attacker URLs and labels don't materialize event-handler attributes
on rendered DOM.
Three independent fixes flagged by Copilot's review on PR #555:
2. ``update`` auto_approve self-escalation warning false-positive
(turnstone/core/session.py:_prepare_skills_update)
- The warning was computed against ``existing_auto_approve or
proposed_auto_approve`` — meaning an update that explicitly
turned auto_approve OFF still triggered the warning because the
existing row had it ON. Now computes against the *final state*
(``updates["auto_approve"]`` if present, else
``existing.get("auto_approve")``) combined with the final
``allowed_tools`` value. False-positives gone; the inverse case
(existing auto_approve=False, update turns it ON without
touching allowed_tools) now correctly fires the warning against
the inherited allowlist.
3. ``temperature`` validator silent-coerce → explicit error
(turnstone/core/skill_field_validation.py:parse_skill_session_config)
- Non-numeric temperature input silently coerced to ``None``,
unlike ``max_tokens`` / ``token_budget`` which return an error.
Numeric-field consistency: temperature now errors on
unparseable input with "temperature must be a number between 0
and 2". Range check unchanged; blank / None still → None.
4. Version-snapshot uses max+1, not count+1
(turnstone/core/session.py:_exec_skills_update)
- ``count_skill_versions + 1`` re-uses version numbers when any
row has been deleted via the existing
``storage.delete_skill_versions`` method, and the schema has no
``(skill_id, version)`` unique constraint to catch the
collision. Switched to ``max(list_skill_versions)`` + 1,
matching the ``storage.unlock_skill`` pattern. A storage-side
atomic allocator is the right architectural fix and is tracked
for a future PR.
Tests cover both the false-positive and inverse-positive auto_approve
cases, the new temperature error path, and the version-numbering edge
case where prior versions have been deleted (max diverges from count).
Two changes that share the same kind-scoping touch point.
Lookup unification (closes the bypass Copilot flagged on _exec_skills_load):
- New ChatSession._lookup_visible_skill(name) — single source of truth for
"find me a skill by name, if it's visible to this session". Combines
storage.get_prompt_template_by_name with the kind filter in one call;
returns None for both the missing-row and out-of-kind cases so callers
don't have to branch on the reason.
- _exec_skills_get refactored from inline two-step to one helper call.
- _exec_skills_load refactored from the unscoped memory.get_skill_by_name
to the new helper — the kind-scoping bypass it had (interactive could
load a kind=coordinator skill by name) goes away by construction
because the unscoped path no longer exists on the model-tool surface.
- memory.get_skill_by_name stays available for admin / sub-agent /
rehydrate paths that need full-catalog visibility — those are
deliberate cross-kind callers, not bypass surfaces. Storage exceptions
now propagate from _lookup_visible_skill by design (distinct from the
legacy swallow-and-return-None) so the operator gets a clear signal on
DB outage rather than a misleading "not found".
Coord-side load support:
- _prepare_skills_load no longer rejects coordinator sessions. Parity
with the admin / HTTP create path that already accepts a `skill` body
field on kind=coordinator workstreams — what the operator can do at
create time, the model can now do on its own session. Visibility is
still kind-scoped via _lookup_visible_skill at exec (a coord can only
load {coordinator, any}-tagged skills; interactive can only load
{interactive, any}), matching what `find` / `get` enforce.
The kind-scoping itself is queued for a separate cleanup PR: the marker
turned out to be a discoverability hint that never gated runtime
capability, and the combinatorial complexity (every new model-tool /
HTTP path needs kind awareness) isn't worth the squeeze at this team
size. Follow-up issue to land.
Test coverage:
- TestLookupVisibleSkill — 5 cases: visible / cross-kind / missing /
storage-unavailable / kind=any-on-both-surfaces.
- TestExecSkillsLoadKindScoping — kind-rejection from both directions
(interactive→coord-only, coord→interactive-only), disabled-skill
caller-side gate, and the two new positive coord-load cases (coord
loads kind=coordinator and kind=any).
- Removed test_load_on_coord_session_errors (the rejection it pinned
is gone).
Plus the /review-suggested doc fixes that came with the unification:
- Comment in _exec_skills_load now correctly attributes the disabled
collapse to the caller's enabled check rather than implying the
helper handles it.
- _lookup_visible_skill docstring documents the deliberate
exception-propagation behavior.
Replaces the legacy `skill` (load + search) and `list_skills` tools with a
single `skills(action=...)` tool serving both interactive and coordinator
sessions. Stacks on the model.skills.write permission introduced in PR 1.
Tool surface
- `find`: filter by category/tag/risk_level/enabled_only/limit with
optional BM25 query ranking; auto-approved on both kinds; kind-scoped at
the storage filter (interactive sees interactive+any, coord sees
coordinator+any).
- `get`: fetch a single skill including content; cross-kind misses
collapse to "not found" so a model can't enumerate the other surface
by name-probing.
- `load`: activate a skill in the current session (interactive-only;
coord sessions get an explicit hint pointing at spawn_workstream).
- `create`/`update`/`enable`/`disable`: require approval AND
model.skills.write; permission re-checked at exec time to catch a
revocation between approval and write.
- No `delete` — hard-delete stays admin-UI exclusive; tool description
documents the soft-delete-via-disable pattern.
Defenses on the write surface
- Approval cards surface projected risk_level (scanner re-run against
the proposed final state) and warn explicitly when allowed_tools +
auto_approve combine (auto-fire-on-load consequence is spelled out,
not just shown as raw field values).
- Toggle preview surfaces existing risk_level + allowed_tools count so
re-enabling a critical-tier skill is never a one-click bypass.
- Update path now re-fetches the row at exec to catch a readonly flip
between approval and write, filters updates back to the runtime-only
set if so, refuses if no fields survive.
- Update path rejects empty content (hollow-out via emptying bypassed
the soft-delete-via-disable invariant), non-list tags, and empty
category — failures are loud rather than silent.
- Permission denials audit `skill.write_denied` with actor_source=model
so probing the permission state leaves a trail. Audit failures log
at error (not warning) — a successful write without a row is the
exact gap the trail exists to surface.
- `_skill_hint` routes both message and system_reminder through
escape_wrapper_tags so caller-controlled values can't close the
<system-reminder> envelope and let the model fabricate directives in
its own future context.
Shared validation
- `parse_skill_session_config` lifted from console/server.py to
turnstone/core/skill_field_validation.py; both the HTTP admin path and
the model-tool path consume it. Single source of truth so field rules
can't drift between layers.
- `SKILL_RUNTIME_CONFIG_FIELDS` lifted similarly (was duplicated as
_SKILL_RUNTIME_CONFIG_FIELDS in server.py and _SKILLS_READONLY_FIELDS
on ChatSession).
- `notify_on_complete` validator now accepts list input from the JSON
schema's `array` type — previously rejected because str() of a list
yields Python repr that json.loads then refuses.
Performance
- Update prepare skips the projected-risk scan when neither content nor
allowed_tools is changing (storage re-scans on write authoritatively).
Metadata-only updates no longer pay the ~25 regex-pass scan cost.
Cleanup
- CoordinatorClient.list_skills deleted (-91 lines); model-tool path
talks to storage directly via list_skills_filtered.
- Roles admin UI gains a Model section exposing model.skills.write.
- tests/test_load_skill.py renamed to tests/test_skills_tool.py and
rewritten for the new tool — 48 tests covering registration, prepare
dispatch, permission gating (including TOCTOU-revoked exec deny),
audit actor_source on create + disable + permission-denied probe,
BM25 ranking, invalid-kind branches, audit-failure swallow, and
<system-reminder> envelope injection resistance.
In-process permission check for model-facing tool exec paths that need
to gate a write capability without HTTP middleware in the loop. Foundation
for the upcoming skills tool refactor: the merged
skills(action=create|update|enable|disable) tool will gate on
model.skills.write before reaching storage.
- Add model.skills.write to _VALID_PERMISSIONS (default-ungranted on every
role including builtin-admin — operators opt themselves in explicitly)
- Add user_has_permission(user_id, permission, *, storage=None) helper
that fails-closed on storage outages and short-circuits on empty user_id
- Document service-scope asymmetry with require_permission (no AuthResult
in the model-tool path → no bypass; explicit guidance if a legitimate
service-scope caller ever needs to reach here)
- Pin the "no implicit cache" contract with a regression test asserting
every helper call hits storage (call_count == 2 after two calls)
- Lock the "builtin-admin default-ungranted" invariant with an alembic
migration test that drives the chain to head and asserts the role's
permission string omits model.skills.write
- Plus the role-create end-to-end test proving the constant flows through
the admin endpoint's validator
Roles admin UI changes deferred to the PR that lands the gated tool — no
operator action needed until the capability exists.
Per-call DB hit + warning-log spam on outage deferred to a follow-up PR;
the helper is dead code in this commit, so cache TTL would be sized
against guesswork — better to wait for a real call-rate signal from the
first caller.
Starlette 1.0.0 reconstructs request URLs without validating the Host
header, allowing path-injection that can bypass authentication on apps
comparing reconstructed URL paths instead of `request.url.path`. Fixed
in 1.0.1.
- pyproject.toml: bump `starlette>=0.45` to `starlette>=1.0.1` so the
CVE floor is explicit at the dependency declaration, not just in the
lockfile. Annotated with the advisory ID so the rationale survives
a future floor relax.
- uv.lock: regenerated via `uv lock --upgrade-package starlette`;
starlette 1.0.0 -> 1.0.1, no transitive bumps.
Locally verified `pip-audit --strict` returns clean after the bump and
the auth + service-boundary test suites (250 tests covering the URL/
host-header reconstruction surface) continue to pass.
2000 was sized for the cloud-provider regime (50–200 events/sec)
and was too small for the two regimes that actually shape PR-D's
recovery floor:
1. **Local inference**: vLLM / llama.cpp hit 500–2000 tok/s per
active stream. Each token is an _enqueue call, so a single
busy workstream burns through 2000 events in ~1 s. Reconnects
after any disconnect longer than a network blip immediately
fall through to the replay_truncated recovery path on a
stream that was supposed to be transparently resumable.
2. **Backgrounded tabs**: Chrome (and Firefox to a lesser extent)
throttle the SSE-drain microtask aggressively when a tab isn't
visible — Chrome's background-tab budget drops to ~1 wake/min
after ~5 min hidden, so a backgrounded pane can legitimately
sit on tens of seconds of un-drained events. PR-G (drop-pings-
let-it-die) deliberately closes those connections on hide and
re-opens on focus return; reconnect-with-replay is the only
recovery path, and if the buffer evicted in the interim, the
snapshot floor is all that's left for past-turn structural
events (tool calls, state changes, approvals).
50000 at the 2000-tok/s local-inference rate buys ~25 s of pure
token streaming before truncation; at cloud rates it's minutes of
coverage. Memory cost is ~200–500 bytes per event (deque node +
dict + payload), so 50000 × 100-ws design ceiling caps at roughly
2.5 GB worst-case — and practically nowhere close because the cap
is per-ws ceiling, not per-ws steady-state. Operators on heavier
workloads can raise via TURNSTONE_SSE_EVENT_BUFFER_MAX.
Considered and rejected: in-buffer coalescing of consecutive
content/reasoning tokens. A naive text-merge breaks the replay-
slice semantic — a coalesced entry has the latest _event_id
but text that includes content the client already received under
an earlier id, so any consumer with last_event_id falling
INSIDE the coalesced span would double-render on replay. A
correctness-preserving coalesce would need a per-consumer high-
water tracker we deliberately don't maintain. Bigger cap +
simple per-event storage avoids the trap; the rationale is
captured inline in _resolve_event_buffer_max.
The browser-side completion of PR-D reconnect-with-replay. Today's
`onerror` handlers on `Pane.connectSSE`, `connectGlobalSSE`, and
the coordinator's `connectSSE` all explicitly call
`evtSource.close()` on the transient-error path — that forces the
source into the terminal CLOSED state, defeating EventSource's
native auto-reconnect (which would otherwise reconnect with the
`Last-Event-ID` header that PR-D commit 1 now honours server-side).
Three handler refactors share the same shape:
- Remove the unconditional `close()` from the transient-error
branch. Native EventSource handles CONNECTING -> CONNECTING ->
OPEN with replay automatically.
- Keep UI updates (status bar dim, Reconnecting… text) — those
are orthogonal visualizations of the disconnected state.
- Keep terminal-branch closes: a 401 expired-session still does an
explicit close + showLogin (the user must re-authenticate); a
workstream-reassignment to a different ws still disconnects +
connects on the new wsId (it's a different stream, not a same-
stream replay).
- Capture `lastEventId` in `onmessage` BEFORE `JSON.parse` so a
malformed event doesn't desync the manual-reconnect fallback
from native auto-reconnect.
- Thread `?last_event_id=N` on the URL when constructing a fresh
`new EventSource(url)` — the constructor can't set custom
headers so the query-param fallback covers the manual-reconnect
path (initial connect with a saved id, scheduleReconnect after
an explicit close, etc.).
For `Pane.connectSSE`, the long focused-pane workstream-refetch
body inside `onerror` is lifted to a dedicated
`_refetchWorkstreamsAndReassign` method so it survives the
refactor as an orthogonal trigger (handles the workstream-evicted-
during-disconnect recovery case, which is independent of the SSE
reconnect mechanics). The reassignment branch's existing
`disconnectSSE + connectSSE(newWsId)` sequence stays — different
workstream genuinely needs a fresh stream. When reassigning, the
saved `_lastEventId` is dropped because replay is per-ws and an
id from ws-A is meaningless against ws-B.
Tests in `tests/test_app_js.py` add 3 static lint guards that
fail loudly if any future refactor reintroduces a naked
`evtSource.close()` in a transient-error path of any of the three
handlers. The guards understand the allowed terminal-branch
exceptions (401, login overlay, reassignment) and ship with an
escape hatch (functions that explicitly reference `last_event_id`
have taken explicit responsibility for the replay header and are
exempt). A small `_strip_js_comments` helper handles the
apostrophe-in-comment hazard that pre-existing
`_slice_balanced_body` doesn't (comments are stripped before
brace-walking; offsets preserved by space substitution).
The console SSE proxy (`_proxy_sse`) is the inbound SSE path for
multi-node deployments — every browser EventSource that targets a
per-node route traverses it. Today's proxy strips client request
headers (only `Accept`, `Cache-Control`, and the re-minted auth
token make it upstream), so the per-ws / global SSE handlers'
`Last-Event-ID` resume (PR-D commit 1) never sees the header in
the multi-node shape — every reconnect would be a fresh connect
and silently drop events from the disconnect window.
Builds the upstream headers dict conditionally: copy `Last-Event-ID`
from the incoming request when present, omit otherwise (no
fabricated value on fresh connects). Starlette's header dict is
case-insensitive so the `request.headers.get("last-event-id")`
lookup catches both the spec-recommended capitalization and any
intermediary normalisation.
The query-param fallback (`?last_event_id=N`) needs no proxy
change — `request.url.query` is already forwarded verbatim at the
top of the function.
Tests in `tests/test_service_auth_boundary.py::TestProxySseLastEventIdForwarding`:
- Positive: browser header → upstream header (value preserved).
- Negative: browser sends nothing → upstream gets nothing (no
fabricated value).
Adds the server-side foundation for SSE reconnect-with-replay (PR-D
in issue #540's sequencing): a per-ws monotonic ring buffer that
holds the last N events for replay against a client's
`Last-Event-ID` header (or `?last_event_id=N` query-param
fallback for manual reconnect paths that can't set custom headers).
Per-ws lane (SessionUIBase + make_events_handler):
- `_event_buffer` deque (cap 2000, env-overridable via
`TURNSTONE_SSE_EVENT_BUFFER_MAX`) holds (event_id, event_dict)
tuples; `maxlen` evicts the oldest automatically.
- Existing `_ws_inflight_seq` renamed to `_event_id` and lifted
to live alongside the listeners — one monotonic counter drives
both the new replay slice AND the existing `_seq`/`snap_seq`
snapshot dedup (byte-identical contract on token events).
- `_enqueue` now stamps every event with `_event_id` (and `_seq`
on `content`/`reasoning` token events) under
`_listeners_lock`, so the buffer append + listener fan-out + new
listener registration are all atomic against each other.
- New `register_listener_with_replay` returns
(queue, replay_events, status, lost_count, earliest_id) where
status ∈ {replay_ok, truncated}. `make_events_handler` reads
`Last-Event-ID` (header or query), branches three ways
(fresh / replay_ok / truncated), and emits the SSE `id:` field
on every event sourced from the buffer. On `replay_ok` the
in-progress snapshot is skipped (the buffered events already
cover it); on `truncated` an explicit envelope precedes the
fresh-style recovery path.
- Every events stream emits a jittered `retry:` in [2500, 4500] ms
on first yield so 6-pane reconnects don't lockstep on
EventSource's default ~3 s interval.
Global lane (server.py / _global_fanout_thread / global_events_sse):
- Parallel buffer + counter on `app.state.global_event_buffer` and
`app.state.global_event_id_holder`; fanout thread stamps each
event with `_event_id` and appends to the buffer under
`global_listeners_lock`. `global_events_sse` branches on
`Last-Event-ID` with the same three shapes.
Tests:
- 16 new tests in `tests/test_sse_reconnect_replay.py` cover the
ring buffer semantics (empty-listeners hold, last_event_id
slicing, truncation, atomic registration), the counter
invariants (monotonic under concurrent writers, no skip on
queue.Full, persists across turn boundaries, cross-thread
consistency), and the handler branching (retry on first yield,
id: on buffered events, snapshot-skip on replay_ok, envelope on
truncated, query-param fallback, malformed header → fresh).
- Existing `tests/test_session_ui_base.py` updated for the
`_ws_inflight_seq` → `_event_id` rename and the new
`_event_id` field on enqueued events.
Backward-compat: all consumers that don't send `Last-Event-ID`
(today's browser, Python SDK, TypeScript SDK, channel adapter) see
behaviour identical to pre-PR — the server change is purely
additive on the request side.
Mirror the \s* posture used by the other unsafe-sink clauses (eval\s*\(,
Function\s*\(, setTimeout\s*\() so a regression like
``el.insertAdjacentHTML ("beforeend", x)`` — or a multi-line form with a
newline before the paren — still trips the lint. The trailing ``HTML``
literal continues to discriminate against insertAdjacentElement and
insertAdjacentText.
Caught by Copilot review on #541.
Extend _UNSAFE_CODE_SINK_RE with an `.insertAdjacent` + `HTML\(`
alternation so insertAdjacentHTML(...) is flagged across all 8 tracked
JS bundles. The `HTML\(` suffix excludes insertAdjacentElement, which
takes a DOM node and is not an XSS sink — the five remaining sites in
ui/static/app.js (lines 170, 216, 328, 330, 1578) stay clear.
Retire the two carve-out paragraphs (file-level comment + function
docstring) that named ui/static/app.js's verdict-badge writers as the
reason the lint hadn't already broadened. Commit 1 of this PR cleaned
both writers, so the carve-out is no longer load-bearing.
After this commit the DOM-cleanup arc (started in #532) is complete:
every unsafe-write sink family — inner/outer-HTML assignment (plain +
concat), insertAdjacentHTML, document.write, string-eval, dynamic-
Function, string-first-arg setTimeout/setInterval — is forbidden
across all 8 LLM-rendering bundles.
Rewrite the verdict-badge HTML builder from string-concat into DOM
construction (createElement + textContent + setAttribute + append).
The helper now returns a DocumentFragment of two top-level siblings
(.verdict-badge and .verdict-detail), which appendChild expands into
the parent — preserving the sibling-traversal invariants relied on by
Pane.updateVerdictBadge, toggleVerdictDetail, and the d-key keyboard
shortcut.
Inline onclick="toggleVerdictDetail(this)" replaced with an
addEventListener click handler; the non-arrow callback keeps the
`this`→button binding the old inline form had.
Both call sites (replayHistory + the live approval flow) swap from
el.insertAdjacentHTML("beforeend", X) to el.appendChild(X).
This is the last unsafe-write site in the DOM-cleanup arc started in
#532; commit 2 broadens the test_app_js.py lint regex to forbid the
insertAdjacent-HTML sink across all 8 tracked JS bundles.
The pre-push /review pass surfaced a second const-reassign that mirrors
the original `redacted` bug but in prefix-increment form:
const _paneCounter = 0; // turnstone/ui/static/app.js:10
class Pane {
constructor(wsId) {
this.id = "p" + ++_paneCounter; // line 14 — TypeError at runtime
…
}
}
`new Pane(...)` throws `TypeError: Assignment to constant variable.`
on every pane construction. The first iteration of the const-reassign
guard in tests/test_app_js.py missed it because the regex matched
postfix `X++` / `X--` but not prefix `++X` / `--X`.
Two changes:
1. Change `const _paneCounter = 0` to `let _paneCounter = 0` at
turnstone/ui/static/app.js:10. Same fix shape as the `redacted`
bug — original walker tightened to const because its reassignment
regex also only matched postfix forms.
2. Extend the reassignment regex in test_swept_bundle_has_no_const_reassign
to detect prefix `++X` / `--X` so a third repeat of this class
can't ship. Verified by injection: temporarily reverting (1)
makes the new guard fire with a clear source-text diagnostic.
Quality polish on the same test (q-1/q-2 from the pre-push pass):
- Failure message now prints the offending decl + reassignment line
text alongside line numbers, so CI failures are self-contained
(was: opaque tuples requiring two file-jumps to interpret).
- Comment on `_SWEPT_BUNDLES` documents the maintenance contract
(add only after sweeping; coordinator.js intentionally excluded).
After the var → const/let sweep, four guards keep the post-sweep state
honest in CI:
1. node --check per bundle (parse-level smoke; catches a future edit
that drops a brace or mis-balances a string before it reaches the
browser).
2. Static var-free assertion per bundle pins the keyword-swap result —
any future `var X = …` in these 7 files fails CI loudly.
3. Scope-aware static const-reassign guard per bundle. For each
`const X = …`, scans only the enclosing block (innermost { … } via
brace tracking with regex/string/comment awareness) for X
reassignments, so a same-named `let X` in an unrelated function
doesn't false-positive against a `const X` in this one. Catches
the bug class that shipped through the original sweep:
_redactApiKeys's `const redacted; redacted = …` threw TypeError
at call-time, invisible to node --check.
4. Runtime smoke for _redactApiKeys via `node -e` — calls the
function with both query-string (`api_key=…`) and JSON
(`"api_key": "…"`) shapes. This is the bit that would have
caught the actual shipped TypeError; (3) is the equivalent
static check that catches the class without needing a runtime
invocation.
Bundle list:
- turnstone/ui/static/app.js
- turnstone/console/static/admin.js
- turnstone/console/static/governance.js
- turnstone/console/static/app.js
- turnstone/shared_static/auth.js
- turnstone/shared_static/kb.js
- turnstone/shared_static/utils.js
Verified by injection: temporarily reverting `let redacted` to
`const redacted` makes both guard (3) and guard (4) fail loudly.
Follow-up to the initial var-sweep commits. The walker used a flat,
file-wide reassignment check to decide let vs const, which was
conservative when the same name appeared in multiple unrelated
scopes — e.g. `let i` as a loop counter in one function and an
unrelated `let i` reassigned in another would both stay `let`.
This second pass uses brace-tracking block-scope analysis (regex
literal aware) so tightening considers only reassignments within
the same block:
- console/static/app.js: +5 const -5 let
- console/static/governance.js: +15 const -15 let
- console/static/admin.js: +26 const -26 let
Mirrors q-2 from the /review pipeline. ui/static/app.js was
tightened in the same way already in its sweep commit.
Mechanical; no behavioural change.
754 line-start var + 50 for-loop counters converted: 663 const, 96 let
(line-start), plus 50 for-init let counters.
Hand-fix sites (surfaced by spike § 2):
- 4 try-block hoists where the var was referenced from outside the
try (var hoists out, let does not):
- tryParseMedia()'s `obj`
- _tryPrettyJson()'s `obj`
- tryParseMcpError()'s `obj`
- inline-plan render's `action` (used in the catch handler)
- showNewWsModal() cleanup (was: 2 same-scope redeclarations):
- submitBtn — first lookup at the top of the modal kept; the
redundant re-fetch + duplicate textContent at the bottom
dropped; submitBtn.disabled = false now sits as a bare
property write
- defaultOpt → renamed second occurrence to tplDefaultOpt
(genuinely distinct DOM element — modelSelect vs tplSelect),
both can be const
Post-review fix to the walker output:
- _redactApiKeys(): the walker tightened `let redacted` to `const`
but missed the `redacted = redacted.replace(...)` reassignment
on the JSON-style pass. Root cause was the walker's
find_decl_extent not recognising JS regex literals — the
unescaped " inside the character class [^&\s"] opened an
in_str state that never closed on the same line, spilling
the declaration span past `);` and pulling the reassignment
line into the skip set. The /review pipeline's bug finder and
security finder both caught it (rendering would have thrown
TypeError on every tool-output render). Now `let redacted`.
Scope-aware const-tightening pass on top of the walker (mirrors q-2
from /review): 45 additional `let` → `const` flips where the walker
was conservative because the name happened to be reassigned in an
unrelated function elsewhere in the file. Examples: `let pane` in
the 4 plan-dialog helpers; `let el` in the small Pane class methods.
Each tightening is verified safe by a brace-tracking block-scope
analysis (regex-literal aware).
Mechanical; no behavioural change.
744 line-start var + 87 for-loop counters converted: 606 const, 138 let.
Includes 2 multi-decl sites (counter accumulators at 3389 and 4021,
both `let` because the names are reassigned via += in the loop body),
and the spike-identified `indicator` redeclaration in `_toggleOidcPanel`
(now two `const indicator` declarations in disjoint block scopes —
inner if-block at 455 and function body at 482, so block-scoping makes
them independent).
Mechanical; no behavioural change.
470 line-start var + 64 for-loop counters converted: 337 const, 133 let.
Includes 2 multi-decl sites (`let url, method;` at 3951 and 4504 — both
uninitialised pairs that stay `let`) and two sibling `for (var k …)`
loops at lines 112/119 in the same function (now `for (let k …)` —
block-scoped to each loop init, no collision).
Mechanical; no behavioural change.
67 line-start var + 4 for-loop counters converted: 55 const, 12 let.
The 12 let cases are all genuine reassignments:
- Top-level state (`_loginBusy`, `_authMode`, `_refreshTimer`, etc.)
- `let delay` in `_scheduleRefreshAt` (clamped to min/max)
- `let data` inside `_tryRefresh` (assigned from inner try-catch)
- For-loop counters `let attempt`, `let i`
Mechanical; no behavioural change.
12 line-start var + 1 for-loop counter converted: 15 const, 1 let.
File previously had 4 const from the DOM-cleanup helpers; sweep finishes
the conversion.
Walker is scope-aware: when checking if name X is reassigned anywhere
in the file, lines that themselves declare X (`let X = ...`, function
parameters `(X)`, etc.) are skipped — `X = ...` in another scope is a
new binding, not a reassignment of the original. This lets variables
like `min`/`hr` (declared inside two different formatter functions)
both become `const` correctly.
8 line-start var declarations converted: 6 const, 2 let.
Walker rules:
- var X = init → const X = init when X is never reassigned in the file
- var X = init → let X = init when X is reassigned (e.g. _kbPreviousFocus
assigned in showKbHelp, html accumulated via +=)
- Reassignment check uses negative lookbehind to skip property writes
(obj.X = ...).
Mechanical; no behavioural change.
* feat(reasoning): Phase 5 — vLLM Chat Completions reasoning-field replay
Multi-turn CoT replay for vLLM-served reasoning models (Qwen3, DeepSeek-R1)
via the non-standard `reasoning` field on assistant messages. Closes the
PR #498 gap claiming Chat Completions has no replay surface — vLLM's
ChatMessage.reasoning input field is that surface (verified in
vllm/entrypoints/openai/chat_completion/protocol.py:54-64).
Session-level attach (no provider class changes). Three-gate composite:
provider isinstance OpenAIChatCompletionsProvider AND
server_compat.server_type == "vllm" AND operator-set
ModelConfig.replay_reasoning_to_model. Deliberately drops the
supports_reasoning_replay capability gate that protects Paths 1+2 —
vLLM's failure mode is silent (template-drop), not loud (server 400),
so the static gate would add operator friction without preventing the
silent failure. Server-type pin bounds blast radius — canonical OpenAI,
llama.cpp, sglang never see the non-standard field.
Also fixes a pre-existing _resolve_server_type bug: it read
cfg.capabilities.get("server_compat") but the model_registry loader pops
server_compat OUT of capabilities into the dedicated cfg.server_compat
dataclass field (model_registry.py:401, 485). Pre-fix the function
returned "" for every production ModelConfig, silently degrading PR #498
Path 3's synth-block source tag and would have made Phase 5 dead-on-
arrival. Test stubs across 3 files updated to mirror production shape
(empty capabilities + populated top-level server_compat) so the same
stub-drift can't hide future regressions.
The agent _run_agent path is deliberately excluded from Phase 5 hoists:
agent assistant messages don't carry _provider_content (rebuilt per
invocation from CompletionResult.content + tool_calls), so the helper
would no-op every turn. Comment at session.py inside _api_call documents
the exclusion.
OpenAI SDK version pin raised to >=2.37 to match the version verified
by the cross-boundary regression test
(test_reasoning_field_present_in_wire_body_when_attached) — drives a
real OpenAI client through httpx MockTransport and asserts the
non-standard field reaches the captured POST body, catching any future
SDK version that adds runtime field filtering.
Tests: 10 helper unit + 17 session integration (incl. SDK boundary
round-trip + per-gate negative tests + call-site wiring tests) + 2
audit-log discipline tests extending the PR #498 logging contract.
* docs(reasoning): apply PR #537 review on Phase 5 docstrings
Two nits from PR #537 review:
1. `_resolve_server_type` docstring claimed Phase 5 (`_maybe_attach_vllm_chat_reasoning`) called it; in fact Phase 5 reads `cfg.server_compat["server_type"]` directly off the single cfg it fetches for the operator-flag check, to avoid a second `registry.get_config` round-trip. Rewrite the paragraph: name `_maybe_synth_reasoning_block` as the sole caller (informational metadata for UI rehydration), then a separate paragraph noting Phase 5 reads the same field path directly and that both readers MUST stay aligned on changes.
2. `_maybe_attach_vllm_chat_reasoning` docstring referenced `project_reasoning_replay_capability_gate.md` which lives in personal memory store, not the repo. Replace the dead-link reference with an inline summary of the asymmetry rationale (Paths 1+2 keep the dual-gate because loud server-side failures; Path C drops the static gate because vLLM's failure mode is template-drop silent).
Three quality findings from the multi-stage /review pass on the
preceding 4-commit class-refactor stack. Bundled into one commit
because each is sub-20-line documentation/test-hygiene with no
behavioral surface.
1. **Drop stale verdict-badge line numbers in tests/test_app_js.py.**
Two comments cited ``ui/static/app.js:1287`` and ``app.js:1538``
as the ``insertAdjacentHTML`` + ``renderVerdictBadge`` consumer
sites. The class refactor moved them to 1440 and 1655 (and any
future nearby edit will move them again). Drop the numbers; cite
the helper name (``renderVerdictBadge`` / "the verdict-badge
writers") instead.
2. **Drop ``.prototype`` from 3 coord comment cross-refs.**
``coordinator.js:339, 439, 566`` referenced
``Pane.prototype.addUserMessage`` / ``addUserReminder`` /
``addToolReminder`` / ``replayHistory`` — but ``app.js`` has zero
``Pane.prototype.X`` after the refactor (it's all ``Pane.X``
class methods now). A reader following the breadcrumb hits a
grep dead-end.
3. **Introduce indent-agnostic _pane_method_offset() helper.**
The four test slices switched from ``"Pane.prototype.X = function"``
to ``"\n X("`` in commits 3 + 4 — that's brittle against the
deferred PR-B/C/D/E/F modernization (IIFE / module wrap shifts
indent to 4 spaces, breaks all four slices silently with a bare
``ValueError``). The new helper uses ``re.MULTILINE`` + ``\s{2,}``
to match the method header at any leading-whitespace depth and
``assert``s on miss so a renamed method fails loudly at the
pinning slice instead of further downstream.
Replaces 8 ``body.index("\n X(")`` call pairs across the 4 anchored
tests (replayHistory ×3, appendToolOutput ×1).
Tests: 27/27 ``tests/test_app_js.py`` green. No other suites touched.
Fourth and final commit of the ES6-class refactor (~/pane-class-refactor.md).
Migrates the remaining 5 prototype methods into the class block,
dissolving the last 4 `var self = this` workarounds, and updating the
appendToolOutput-anchored test in lockstep.
Methods migrated INTO the class body:
showInlineToolBlock(items, autoApproved, judgePending)
resolveApproval(approved, always, feedback, skipPost)
appendToolOutput(callId, name, output, isError)
sendMessage()
cancelGeneration()
Two of these (`showInlineToolBlock`, `resolveApproval`) had multi-line
header decls; the conversion script joins their arg lines back into
a single-line class-method header.
Test anchor update (`tests/test_app_js.py`):
body.index("Pane.prototype.appendToolOutput = function")
→ body.index("\n appendToolOutput(")
body.index("Pane.prototype.", start + 10)
→ body.index("\n sendMessage(", start)
The new upper-bound anchors on the next class method's header (which,
by the source-file order preserved through the refactor, is
`sendMessage`). The slice's inner assertions
(tryParseMcpError-before-renderToolOutput offset comparison) are
untouched — only the outer anchor pattern changes.
Final state:
* `class Pane { ... }`: 1 declaration with 39 members
(constructor + 38 methods)
* `Pane.prototype.X = function`: 0 occurrences (was 38)
* `var self = this`: 0 occurrences (was 16)
* Arrow callbacks (`=>`): 55 (was 0)
* 3 module-level helpers (_buildWatchResultBubble,
_buildDefaultReminderBubble, _buildOutputWarningEl) cluster
immediately after the class block.
The framing goal — "coord speaks a more modern JavaScript than
interactive" — collapses on this axis: Pane is now ES6 class shape
with arrow-function callbacks and `this`-lexical inner scopes, on par
with coord's ES6+ idioms. The remaining var → const/let sweep and
template-literal pass are deferred to follow-up PRs B-F per §8 of the
refactor brief.
Tests: 27/27 `tests/test_app_js.py` + 258 broader (renderer + console
suites) green. All 4 historically anchored test slices now use
class-method anchors and pass cleanly.
Third of the four planned commits in ~/pane-class-refactor.md.
Migrates the two test-anchored history-rebuild methods into the
class block, updates the three pytest assertions that sliced them
by `Pane.prototype.X = function` literal, and relocates the last
nested module-level helper to live alongside the other two.
Methods migrated INTO the class body:
replayHistory(messages) — 304-line method, the largest single
method in the file. Dissolves
2 of the remaining `var self = this`
sites (the method-scope one + the
inner-callback one inside the
`tool` role branch's
replayAdvisoriesAfterTool callback).
_attachRetryToLastAssistant() — small leaf method that the
replayHistory tests use as the
lower-bound sentinel for their
slice.
Helper relocated to just after the class block:
_buildOutputWarningEl(assessment) — was nested between
replayHistory's `};` and
`_attachRetryToLastAssistant`'s
header. Joins the two helpers
that already moved in commit 1
(_buildWatchResultBubble,
_buildDefaultReminderBubble) —
all three module-level helpers
now cluster immediately after
the class.
Test anchor updates (`tests/test_app_js.py`):
body.index("Pane.prototype.replayHistory = function")
→ body.index("\n replayHistory(")
body.index("Pane.prototype._attachRetryToLastAssistant", start)
→ body.index("\n _attachRetryToLastAssistant(", start)
Three assertions touched: `test_replay_history_renders_content_before_tool_block`,
`test_replay_history_renders_persisted_verdict_badge`,
`test_replay_renders_user_interjection_advisory_after_tool_block`.
The slice's inner assertions (`msg.content`-vs-`msg.tool_calls` offset
ordering, `renderVerdictBadge` regex, `replayAdvisoriesAfterTool` +
`addUserMessage` substring matches) are untouched — only the outer
anchor pattern changes.
After this commit 5 prototype declarations remain: showInlineToolBlock,
resolveApproval, appendToolOutput, sendMessage, cancelGeneration —
all five migrate in commit 4 alongside the appendToolOutput test
anchor update.
Tests: 27/27 `tests/test_app_js.py` green. Class block now spans
lines 12 → 1634; all 3 module-level helpers cluster at 1642 / 1676 /
1698 just after.
Second of the four planned commits in ~/pane-class-refactor.md.
Migrates the callback-heavy methods that don't anchor any test
slice, dissolving 12 of the 16 `var self = this` workarounds into
arrow-function lexical-this along the way.
Migrated INTO the class body:
_createDOM, connectSSE, handleEvent,
_addUserMsgActions, _addRetryAction, _retryLast,
_rewindToMessage, _startEdit, _editAndResend
Each migration applies the same mechanical transformation:
* `Pane.prototype.X = function (args) {` header → class-method
`X(args) {` form, body re-indented +2 spaces.
* Every `var self = this;` declaration removed.
* Every inner `function (...) {` callback rewritten as
`(...) => {` — arrow functions inherit `this` lexically, so the
outer-self capture pattern dissolves without behavioural change.
* Every `\bself\b` identifier rewritten as `this`.
* Closer `};` → `}` (no semicolon on class methods).
`sendMessage` and `cancelGeneration` are deliberately deferred to
commit 4 even though they're shape-eligible for this commit: they sit
AFTER `appendToolOutput` in the file, and
`test_phase8_appendtooloutput_dispatches_mcp_error_before_renderer`
slices `appendToolOutput` by looking for the next `Pane.prototype.`
declaration as an upper bound. Migrating sendMessage + cancelGeneration
now would leave `appendToolOutput` as the LAST prototype declaration
in the file, breaking that slice. Co-migrating all three in commit 4
keeps every intermediate commit green.
Spike-verified (§2.10.5): all 16 `var self = this` sites in app.js
are Type 1 (outer-`this` capture only — no event-target-`this`, no
delayed semantic capture). Mechanical conversion is safe for every
site touched here.
Diff: 805 insertions / 816 deletions (net −11 lines) — the arrow
form is more compact than `function (args) {`, partly offsetting the
class-body indent overhead.
Tests: 27/27 `tests/test_app_js.py` + 238 broader (renderer + console
suites) green. Anchored tests (replayHistory ×3, appendToolOutput ×1)
remain on their existing `Pane.prototype.X = function` literals —
their methods migrate in commits 3 + 4.
Opens the ES6 modernisation of ui/static/app.js (per the spike in
~/pane-class-refactor.md §2.10). This first commit lays the class
scaffolding and migrates the 22 callback-free leaf methods — the
remaining 16 callback-heavy / test-anchored methods land in commits
2-4.
Migrated INTO the class body:
constructor(wsId) (was `function Pane(wsId)` at line 12)
reset, updateWsName, disconnectSSE, setBusy,
showEmptyState, removeEmptyState,
addThinkingIndicator, removeThinkingIndicator,
addSystemNudgeMarker, addUserReminder, addToolReminder,
addUserMessage, getFeedback, appendToolOutputChunk,
showOutputWarning, updateVerdictBadge, updateVerdictGlow,
addInfoMessage, addErrorMessage, updateStatus,
isNearBottom, scrollToBottom
Two module-level helpers (`_buildWatchResultBubble`,
`_buildDefaultReminderBubble`) were previously nested between leaf
methods. Class bodies can't hold free function declarations, so they
relocate to immediately after the class closing `}`. Function
declarations are module-hoisted so the relocation is semantically
free.
The other 16 prototype methods continue as `Pane.prototype.X =
function (...)` below the class — they each add to `Pane.prototype`
exactly as before, so the prototype shape is unchanged.
Spike-verified guardrails:
* Hoisting: only `new Pane()` site is `createPane` at line 2152
(renumbered), well after the new class block ends at line 458.
* Strict mode: `app.js` is already clean of `with`,
`arguments.caller`, `arguments.callee` — class bodies' implicit
strict mode is a no-op.
* Method enumerability: the 14 `for (var pid in panes)` loops
iterate the module-level `panes` ID-map, not method names on an
instance — class methods being non-enumerable on the prototype
doesn't affect them.
No `var self = this` sites are touched in this commit — the 22 leaf
methods all have zero inner callbacks. Commits 2-4 will dissolve the
16 var-self-this sites as their parent methods migrate.
Tests: 27/27 in `tests/test_app_js.py` green (test anchors on
`replayHistory` and `appendToolOutput` are untouched — commits 3 + 4
will co-migrate them with the assertion updates).
Adds ``console/static/app.js`` to ``_UNSAFE_CODE_SINK_LINT_TARGETS``.
The parametrized scan now covers 8 static JS bundles — all admin-side
bundles are clean.
Docstring updates:
- Fully qualify the bundle paths in both posture lists
(``ui/static/app.js``, ``shared_static/utils.js``, etc.) so the two
``app.js`` files are unambiguous now that both are in the targets
list.
- ``console/static/app.js`` joins the **strict DOM-construction**
list (alongside the interactive surface + shared helpers + coord
chat entry) — the cluster-dashboard renderer is now full
createElement / textContent construction, no HTML strings ever
interpolated.
- ``console/static/admin.js`` + ``console/static/governance.js``
remain in the **sink-free string-concat** list — they retain the
escapeHtml + concat builder shape with the unsafe sink off the call
site.
The ``insertAdjacent`` carve-out note (verdict-badge writers in
``ui/static/app.js``) is now qualified to avoid ambiguity.
Initial AST-light swap routed 12 ``innerHTML =`` sites through
``setSafeHtml`` (8 sites) or ``replaceChildren()`` (4 empty-string
clears). The /review pipeline flagged the node-table render path as
a hot loop where DOMParser-per-row costs scale with cluster size (the
project's 100-node design ceiling × per-RAF render frame on SSE
churn), and the static colHeaders literal was being re-parsed every
render.
This commit lifts the entire console-dashboard renderer to true
``createElement`` + ``textContent`` + ``append`` construction —
matching the strict-posture lane used by ``ui/static/app.js`` rather
than the sink-free string-concat posture admin/governance use.
Net result: **zero** ``innerHTML`` and **zero** ``setSafeHtml`` calls
remain in ``console/static/app.js``.
Three new module-local helpers carry the heavy structural fragments:
- ``buildColHeaders()`` returns a DocumentFragment with the 7-span
column-header layout used at the top of the table and inside each
multi-node group body. Built once via createElement so the static
literal isn't re-parsed every render — and the duplicated literal
between the top-table and per-group sites collapses into one helper.
- ``_buildNodeNumCell(value, highlighted, cellClass)`` — the
``<span class="X num [has-value]">N</span>`` shape used 8× across
buildNodeRow and the group header.
- ``_buildHealthCell(cellClass, healthPct, healthFillClass)`` — the
health-bar trailing cell used both per-row and per-group.
The 4 ``setSafeHtml`` sites that remained after the initial sweep
(error-state placeholders + state-pill builder) also flip to
``createElement`` / ``makeEmptyState`` for consistency with the rest
of the file's new posture. ``makeEmptyState`` is the helper added
during the interactive cleanup (#532).
Two Copilot findings, both pre-existing on main since 2026-04-05 but
preserved by this PR's mechanical refactor. Addressing them here
since they're appropriately in scope (the renderJudgeSettings
function is the focus of the refactor) and Copilot ranked them high.
- **Dead ``shortKey === "model"`` branch removed.** The loop at the
top of ``renderJudgeSettings`` does ``if (s.key === "judge.model")
continue;`` because ``judge.model`` is rendered by the cross-cutting
model-alias picker in ``admin.js`` (line 4895, ``aliasKey: "judge.
model"`` registry entry). No other setting key has the form
``judge.X`` where ``X === "model"``, so the conditional branch was
provably dead — Copilot's confusion ("operators won't see a model
picker") is the same confusion a future reader would hit. Drop
the branch + the corresponding ``SELECT``-dispatch in the binding
loop (no SELECT inputs remain in this renderer).
- **Float-input value/min/max now escapeHtml'd.** Previously
``currentVal`` and ``s.min_value`` / ``s.max_value`` were
interpolated raw into ``value="..."``, ``min="..."``, ``max="..."``
attributes. Numbers stringify safely, but
``admin_list_judge_settings`` can fall back to returning a raw
stored string when deserialization fails (Copilot's flag) — a
non-numeric fallback containing a ``"`` would break out of the
attribute boundary. Wrap with ``escapeHtml(String(...))`` for
defense-in-depth + null-safety.
Adds ``console/static/governance.js`` to ``_DOM_WRITE_LINT_TARGETS``.
The parametrized scan now covers 7 static JS bundles. Docstring
updated: governance.js joins admin.js in the "sink-free string-concat"
posture; ``console/static/app.js`` (cluster dashboard / node table) is
the last admin-side bundle still pending — same posture once cleaned.
Same posture sweep as the admin DOM cleanup (#533) applied to
turnstone/console/static/governance.js:
1. **46 innerHTML sites → setSafeHtml**. Mechanical swap via the
same AST-light Python walker the admin PR used. HTML strings are
still built with escapeHtml + concat — same defence as before, just
no innerHTML sink at the call site. `node --check` clean; prettier
formatted.
2. **6 inline event handlers in renderJudgeSettings refactored to
delegated bindings**. The previous code embedded the setting key
as a JS-string inside an HTML attribute
(`onclick="saveJudgeSettingFromInput('KEY')"`), the same footgun
addressed in admin.js: escapeHtml turns `'` into `'`, but the
HTML parser decodes that before the JS parser runs, so a key with
an apostrophe would escape the JS string. Keys today come from a
static judge-settings registry without apostrophes, so no live
vuln — but the pattern is brittle.
Inputs now carry a single `data-judge-key`; the binding loop
dispatches on `this.type === "checkbox"` / `this.tagName ===
"SELECT"` to wire change-listeners for the auto-save inputs, and
leaves text/number/password inputs alone (they commit via the
adjacent Save button). Save and Reset buttons carry their own
`data-judge-save-key` / `data-judge-reset-key` and bind via click
delegation — mirrors the admin.js settings-tab shape.
3. **1 inline handler in audit pagination converted** for
consistency: `<button onclick="loadMoreAudit()">` → addEventListener
after setSafeHtml.
4. **4 stale safety-narrator comments stripped** ("// values escaped
via escapeHtml above", "// NOTE: innerHTML usage below is safe",
etc.). The safety now lives in setSafeHtml; the per-call-site
narration is tombstone-shaped and removed per the project
no-tombstone-comments convention.
5. **`saveJudgeSettingFromInput` hardened**. The
`document.querySelector('[data-judge-key="' + key + '"]')` lookup
now wraps the key in `cssEscape` (from shared/utils.js) so a
future key containing `"` or `\` doesn't break the selector.
console/static/app.js (12 sites) remains pending — same posture once
cleaned, separate PR.
Two small follow-ups from the Copilot PR review:
- ``admin.js:3112-3115`` — the docstring on ``_onSettingChange`` said
``inp`` is "passed in by the delegated handler", but the wiring at
the call site is a per-element ``addEventListener`` rather than a
single delegated handler on the container. Reword to
"per-input event-listener callback" so a future reader doesn't
read "delegated handler" and refactor under that mistaken premise.
- ``test_app_js.py`` — ``_DOM_WRITE_LINT_TARGETS`` constant name
was missed in the earlier sweep that renamed ``_UNSAFE_DOM_WRITE_RE``
→ ``_UNSAFE_CODE_SINK_RE`` and the test to
``test_no_unsafe_code_sinks_in_static_assets``. Rename to
``_UNSAFE_CODE_SINK_LINT_TARGETS`` so the three names align.
Extend ``_UNSAFE_DOM_WRITE_RE`` to also flag the JS string-to-code
constructors that share the same XSS / RCE-on-injection threat model
as innerHTML:
- ``eval(...)`` — string-eval
- ``new Function(...)`` — dynamic-Function constructor
- ``setTimeout(string, ...)`` / ``setInterval(string, ...)`` — the
string-first-arg form (function-first-arg remains unflagged)
Verified that none of these sinks exist in the six currently-scanned
files (interactive app.js + shared utils/auth/kb + coord chat entry +
console admin.js). Pre-existing parametrized test
``test_no_unsafe_dom_writes_in_static_assets`` extends naturally to
the broader pattern; all 25 cases pass.
``insertAdjacent`` + HTML continues to be excluded — two existing
verdict-badge sites in ui/static/app.js consume
``renderVerdictBadge``'s HTML-string output, so broadening that
specific sink first needs the upstream helper cleaned.
The settings panel renderer in admin.js previously emitted inline
``onclick``/``onkeydown``/``oninput``/``onchange`` attributes that
embedded the setting key as a JS-string inside an HTML-attribute
context:
'<button ... onclick="_saveSettingValue(\'' + escapedKey + '\')">'
``escapeHtml`` escapes apostrophes to ``'``, but the HTML parser
decodes that *before* the JS parser runs — so a key containing an
apostrophe would break out of the JS string. Today the keys come
from a static settings registry without apostrophes, but the pattern
is brittle: a future maintainer adding operator-controlled values to
the attribute would discover the footgun the hard way.
Every inline handler in admin.js is now replaced with a delegated
``addEventListener`` set up after ``setSafeHtml(container, html)``.
Handlers read their context off ``data-*`` attributes (which
``setAttribute`` correctly escapes), so the HTML-attribute /
JS-string double-context is eliminated.
Touched renderers:
- ``_renderSettings`` — section headers, help buttons, per-key inputs
(input/change), per-key save + reset buttons
- ``_renderNodeMetadata`` — section headers (delete/add buttons
already used delegation)
- MCP install source selector — radio-change handler for
``_updateInstallFields``
The named handlers (``_saveSettingValue``, ``_toggleSettingsSection``,
etc.) are unchanged in signature and behaviour; only their wiring
moved from inline-attribute to ``addEventListener``.
Two changes to ``tests/test_app_js.py``'s DOM-write lint:
1. Add ``console/static/admin.js`` to the scan target list. Now
covers all 6 static JS bundles that render LLM output, tool
results, operator-supplied data, or user input. ``governance.js``
and ``console/static/app.js`` remain pending follow-ups (will land
as separate cleanup PRs).
2. Parametrize the lint test over the target list. Each file is now
its own pytest case (e.g.
``test_no_unsafe_dom_writes_in_static_assets[turnstone/console/static/admin.js]``),
so a failure attributes precisely to the offending file instead of
masking offenders behind the first-file's assertion.
Rename the test from ``..._in_interactive_assets`` to
``..._in_static_assets`` — the coverage now spans more than the
interactive surface, and the surface-neutral name leaves room for
governance.js / console-app.js without another rename.
3. Restructure the docstring to surface the two distinct postures
(strict DOM-construction surfaces vs. sink-free string-concat
admin.js) up front, instead of burying the admin caveat after the
main contract claim.
48 sites in turnstone/console/static/admin.js previously assigned
HTML strings directly to .innerHTML. Every site is now routed
through the shared setSafeHtml helper (added in the interactive
cleanup PR), which parses the trusted HTML via DOMParser and installs
the result via replaceChildren — no innerHTML sink at the call site.
Two distinct postures across the admin pages:
- 47 sites: ``setSafeHtml(el, html_built_with_escapeHtml)`` — admin
builders construct HTML strings via string concatenation, running
every interpolated value through escapeHtml first. Defence still
depends on escapeHtml at the builder; the lint catches the sink but
cannot catch a missing escape. Full DOM-construction rewrites
(createElement + textContent) would be structurally safer but are
out of scope — 136 escapeHtml call sites + several thousand lines
of builder code is a separate effort.
- 1 site: ``srcEl.replaceChildren()`` for the MCP-install package
panel's empty-state branch — equivalent to the old
``srcEl.innerHTML = ""`` clear, slightly more idiomatic.
No user-visible behaviour change. DOMParser parses the same HTML the
prior innerHTML assignment did; the new DOM is identical, and the
container.querySelectorAll("[data-X]") event-binding pattern still
finds the newly-installed nodes the same way it did before.
console/static/governance.js (46 sites) and console/static/app.js
(12 sites) remain pending follow-ups. The verdict-badge writers'
two insertAdjacentHTML sites in ui/static/app.js still need the
upstream helper cleaned first — separate effort.
The pyjwt 2.12.1 advisory (\"weak encryption\") is disputed by the
supplier — the key length is the calling application's
responsibility, not the library's. Turnstone generates its JWT
signing keys via the standard ``secrets`` module at
operator-controlled strength (see ``turnstone/core/auth.py``), so the
advisory does not apply to this codebase.
No fix version is available — pyjwt 2.12.1 is the current PyPI
latest as of 2026-05-21. Adding ``--ignore-vuln PYSEC-2025-183``
with the rationale documented in-line so a future reviewer can
re-evaluate when an upstream fix or a non-disputed re-issue lands.
The advisory was published between main's last CI pass (2026-05-19)
and the interactive-cleanup PR's CI run (2026-05-21); main's
security job will fail next push without this fix.
Two Copilot-suggested improvements to the regression scan:
- Allow optional ``+`` before ``=`` in the regex so a future
regression that switches sinks from ``el.innerHTML = X`` to
``el.innerHTML += X`` is still caught. The trailing ``(?!=)``
negative-lookahead still excludes ``===`` / ``==`` reads.
- Switch the scan from line-by-line ``splitlines()`` iteration to a
whole-body ``finditer`` so ``\\s*`` can span newlines. Multi-line
sinks like ``el.innerHTML\\n = X`` (an artifact of formatter
line-wrapping at the assignment) are now caught. Match positions
map back to line numbers for the failure message.
Verified locally with representative test cases including
``el.innerHTML += X``, ``el.outerHTML += X``, multi-line variants,
and the ``===`` / ``==`` reads that must remain unflagged.
Adds two regression tests in tests/test_app_js.py:
- test_no_unsafe_dom_writes_in_interactive_assets: whole-file scan
for inner/outerHTML assignment + doc-write sinks across all five
interactive surfaces (app.js, shared utils/auth/kb, coord chat).
Includes line + content in the failure message so a regression
fails loudly with location.
- test_shared_utils_defines_set_markdown_helper: pins setMarkdown's
signature and the DOMParser path so a refactor that drops the
parser (e.g. swap to Range.createContextualFragment) forces an
explicit reviewer decision.
The lint regex is tightened with a negative-lookahead so equality
comparisons (``===`` / ``==``) don't false-positive, and broadened
to cover ``outerHTML`` and the legacy doc-write sink in addition to
``innerHTML``. ``insertAdjacent`` + HTML is *not* covered yet — two
existing verdict-badge sites (app.js:1287, 1538) consume the HTML
output of renderVerdictBadge and would need that helper cleaned
first.
Three adjacent sites that all assign pre-trusted HTML strings (built
from escapeHtml + static template literals, no caller-supplied raw
HTML) get the same DOMParser + replaceChildren treatment via the
shared setSafeHtml helper:
- coordinator.js:327 (appendMsg's body) — callers pass either
esc(text) or renderToolOutput(...) output, both pre-escaped.
- auth.js:302 (login overlay) — _buildLoginHTML() returns a static
template with no caller-supplied interpolation.
- kb.js:30 (keyboard-help overlay) — html is built from a static
keys-and-bindings table.
Eliminates the only remaining innerHTML site in coord and the two
shared-overlay sites. Console admin / governance JS bundles
(106 sites in console/static/{app,admin,governance}.js) remain
outside this PR — different threat model (admin-only behind auth
gate), separate effort.
Routes every direct-HTML assignment in turnstone/ui/static/app.js
through the helpers added in the previous commit, or through native
DOM construction (createElement + textContent + append /
replaceChildren). Net result: zero ``.innerHTML =`` sites in app.js.
Breakdown of the 26 sites:
- 2 renderer-output sites (replayHistory message body, plan-inline
body) now use setMarkdown — DOMParser keeps the audit at zero
innerHTML sites, stricter than the prior centralise-not-eliminate
plan.
- 8 empty-string clears (pane reset, layout rebuild, dashboard
refresh, etc.) become replaceChildren().
- 5 keyboard-shortcut button labels (y/n/a/Esc) collapse onto
makeKeyLabel(hint, label).
- 5 dashboard placeholders (Loading / Failed / No active workstreams)
use makeEmptyState(text).
- 6 escapeHtml-interpolated HTML strings (command preview, judge
evidence, dashboard state cells, footer node, diff lines) become
createElement + textContent + append; escapeHtml drops out because
textContent escapes intrinsically.
No user-visible behaviour change — DOMParser parses the same HTML
the prior innerHTML assignment did, and DOM construction with
textContent produces equivalent rendered output. Mermaid / hljs
post-render scope is unchanged (now scoped to the body element
rather than the wrapper for the two setMarkdown sites; both
contain the same code blocks).
Adds four helpers that move the unsafe HTML-string sinks off the call
site:
- setSafeHtml(el, html): parses a trusted HTML string via DOMParser
and installs the result via replaceChildren — no innerHTML.
- setMarkdown(el, content): renderMarkdown -> setSafeHtml ->
postRenderMarkdown (hljs + mermaid).
- makeEmptyState(text): builds a <div class="dashboard-empty"> card.
- makeKeyLabel(hint, label): keyboard-hint + label fragment for
approve/deny/always/amend/reject buttons.
The helpers are unused at this point; subsequent commits route the
26 app.js sites, coord:327, auth.js, and kb.js through them.
* feat(admin): align turnstone-admin DB config with server (config.toml + env)
turnstone-admin previously read TURNSTONE_DB_* env vars only, forcing
operators with credentials in config.toml to re-export them just to
run admin commands. Wire add_config_arg + apply_config(["database"])
into main() so admin honors the same precedence as turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded defaults.
Also exposes pool_size + sslmode/sslrootcert/sslcert/sslkey to admin,
which previously dropped any such config silently.
Hardening: load_config() now warns once when config.toml is group- or
world-readable, since DB password and TLS key paths live in [database].
Tests cover precedence (default / config / env / partial fallback /
empty-string-in-config-beats-env), the real init_storage boundary on
a tmp sqlite path, the sys.argv -> main() pre-parser path, and the
new permission check (mode 0644 warns, 0600 quiet).
* test(admin): unify config import style in test_admin_db_config
Use module alias (config_mod.apply_config) instead of mixing
'import turnstone.core.config as config_mod' with 'from
turnstone.core.config import apply_config'. Addresses
github-code-quality bot feedback on PR #531.
Async LLM-tier "llm_fallback" verdicts (judge.py:1073, judge.py:1131
via _deliver_fallbacks) deliberately reuse the heuristic verdict's
``verdict_id`` so the row gets "upgraded in place" from heuristic →
llm_fallback when the LLM judge times out, is cancelled, or returns
no content. The consumer ``_persist_intent_verdict`` was doing a
plain INSERT via ``create_intent_verdict``, hitting the
``intent_verdicts_pkey`` constraint on every llm_fallback delivery.
Postgres logged the duplicate-key error; the application try/except
swallowed it at log.debug — so the row never actually got upgraded
and the LLM judge's annotation ("(LLM judge did not return a
verdict)") was lost.
The collision rate exploded on stable/1.5 smoke tests because
PR #527 (just merged) added two new heuristic-INSERT paths in the
auto-approve early-return branches of ``approve_tools`` — previously
those branches dropped heuristic verdicts on the floor, leaving no
row for the fallback to collide with.
Fix:
- New ``upsert_intent_verdict`` method on the storage protocol +
sqlite + postgres impls, using dialect-specific
``insert(...).on_conflict_do_update(index_elements=["verdict_id"],
set_={...})``. Set_ clause updates ONLY the three fields that
genuinely change between heuristic and llm_fallback: ``tier``,
``reasoning``, ``judge_model``.
- Every other column is excluded from set_: identity columns
(verdict_id, ws_id, call_id, func_name, func_args), carried-
verbatim columns (intent_summary, risk_level, confidence,
recommendation, evidence, latency_ms), and ``user_decision``.
- ``user_decision`` exclusion is load-bearing: ``IntentVerdict
.to_dict()`` doesn't project it, so a fallback verdict reaching
``_persist_intent_verdict`` carries the kwarg's ``"pending"``
default. If the operator already resolved the approval between
heuristic INSERT and fallback delivery, the row's user_decision
has been stamped to ``"approved"``/``"denied"``/``"timeout"`` (or
an auto-approve reason at heuristic-INSERT time per PR #527).
Including ``user_decision`` in set_ would silently clobber that
back to ``"pending"``.
- ``_persist_intent_verdict`` switched from ``create_*`` to
``upsert_*``. Bulk path ``create_intent_verdicts_bulk`` stays as
plain INSERT — every heuristic ``verdict_id`` is freshly minted
in ``judge.evaluate`` so in-turn dups can't happen. The inverse
race (daemon-judge verdict lands BEFORE the bulk write) IS
reachable today but its observable behavior is unchanged by the
per-row UPSERT switch; documented at the bulk site for a future
hardening pass.
Test coverage:
- TestIntentVerdictUpsert × 4 — fresh-id insert, conflict-upgrade,
user_decision preservation across heuristic→approved→fallback,
identity + carried-field preservation.
- Existing tests in test_session_ui_base.py updated to mock the
new upsert method instead of create_intent_verdict.
Two dead module-level constants flagged by github-code-quality on
PR #529: ``_INSPECT_MSG_SNIP_THRESHOLD`` and
``_INSPECT_TOOL_ARG_SNIP_THRESHOLD`` lost their callers when the
content-snip logic moved into the ``_snip_head_tail`` helper. The
helper now reads ``head + tail + _INSPECT_ELISION_MARGIN`` so the
"reserve bytes for the elision marker" rationale that the dead
constants documented stays named instead of becoming a bare ``64``.
A coord doing a fan-out wave of inspect_workstream calls against
tool-heavy children could blow the context budget on raw output
alone (one child with a 100 KB bash result × N children). The
previous safety net was ``_truncate_output``'s head+tail strategy,
which silently drops *middle* messages — exactly the wrong shape
for a coordinator trying to understand a child's trajectory (the
LAST message tells the model what the child concluded; the FIRST
sets the brief; the middle is the connective tissue).
Three-tier degradation modeled on the search tool's pattern at
``session.py:_format_search_results``:
Tier 1 (full): every message verbatim — used when size fits.
Tier 2 (compact): per-message head/tail-snipped content (600/300
chars) plus snipped ``tool_calls.arguments``
(300/100 chars). When content snipping alone
doesn't fit, fall through a message-list trim
ladder ((20,30) → (10,20) → (5,10)) that keeps
head + tail messages and elides the middle as
``{"_omitted": N}``.
Tier 3 (skeleton): no messages — counts + role distribution +
verdicts-by-risk + last assistant preview.
Budget 32 KB (matches ``_SEARCH_OUTPUT_BUDGET``). First emission
whose JSON serialization fits the budget wins. ``_tier`` lands on
every non-error emission so the coordinator LLM and audit readers
can see which compression rung was selected; ``_tier_note`` carries
actionable advice (re-call with a smaller ``message_limit`` etc.).
Error-shape results bypass tiering — they're already small.
Bug fixes caught during review:
- ``_compact_message`` now preserves the assistant-side ``tool_calls``
list with snipped ``function.arguments``; the pre-fix shape left
audit readers with tool-result orphans against invisible calls.
- The intermediate Tier-2 list-trim ladder fixes a size-monotonicity
bug where Tier-2 with un-snippable content (per-message body
under the 964-char threshold) plus the added ``_tier_note`` came
out STRICTLY larger than Tier-1, falling through to skeleton
when a head+tail trim would have preserved dozens of messages.
- ``_inspect_skeleton`` reads ``result["skill_id"]`` (production
storage row key) with a ``skill`` fallback; pre-fix it read
``skill`` only and emitted ``null`` for every real workstream.
Three Copilot threads from PR #526:
1. ``_exec_spawn_workstream`` success path emitted
``{"child_ws_id": null}`` when the upstream response unexpectedly
omitted ``ws_id`` (200-shape with no error field, no id field).
Adds the missing guard — mirrors ``_exec_spawn_batch`` which
already surfaces ``"spawn returned no ws_id"`` as a denied row.
The LLM now sees a tool error and can retry instead of chasing
a null id through follow-up tools.
2. ``docs/coordinator-skills.md`` UI render note said "keep the
ws_id as the click-through key" in a paragraph that had just
introduced ``child_ws_id`` — readable as "the ws_id value" but
confusable as a field-name claim. Clarifies that the value
class is the same regardless of which key carried it.
3. ``docs/bulk-endpoints.md`` ``spawn_batch`` example shows
``child_ws_id`` (coord-tool output shape). The doc title and
the "model tool" column label already disambiguate it from HTTP
API responses, but a reader landing at the example section
directly could miss the framing. Adds one explicit sentence.
Coordinator LLMs on large fan-outs recency-bias on seeing `ws_id`
in a `spawn_workstream` / `spawn_batch` return -- calling
`spawn_workstream(ws_id=...)` again instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this
cascades into self-inflicted re-spawn loops.
Rename to `child_ws_id` (already an existing project term -- see
`tasks` tool, `child_event_bus.py`) defuses the recency bias.
Scope is the LLM-facing JSON only -- the server HTTP API at the
spawn endpoint still returns `ws_id`, and the internal reads of
that HTTP response are unchanged.
Also updates the two tool descriptions, the operator-facing skill
doc, and the bulk-endpoints example so docs don't undo the rename.
A skill with `allowed_tools=[]` in the coordinator's `list_skills`
response read as "no tools are usable by this skill" to a model that
didn't know the semantics — but the actual meaning is "no tools are
pre-approved for auto-approval (auto-approve exemption list)". Real
misdiagnosis incident: a code-review child appeared to have been
spawned with zero tool access when in fact the skill simply hadn't
declared an auto-approve allowlist.
Two-part fix:
- `coordinator_client.list_skills` omits the `allowed_tools` key from
the per-skill dict when empty. Absence now carries the unambiguous
meaning "no tool is pre-approved for this skill"; presence (with a
non-empty list) keeps the standard Claude Code skill-spec shape.
- `turnstone/tools/list_skills.json` description rewrites the field
doc so the LLM sees: "tool names exempt from the operator approval
gate ... the field is OMITTED when empty: a skill without
`allowed_tools` still has access to every tool in its session's
toolset; absence of the field means no tool is pre-approved for
this skill, not that the skill has no tools."
Field name stays `allowed_tools` — matches the upstream Claude Code
skill frontmatter (`allowed-tools` hyphenated, stored as
`allowed_tools` internally per `skill_parser.py:241-242`). Parser,
storage column, admin UI, and SDK unchanged.
Auto-approved tool calls left intent_verdict rows with `user_decision=""`,
indistinguishable from rows still pending manual review. Real misdiagnosis
incident: a coord with `recommendation="review"` and `user_decision=""` was
read as "stuck waiting for approval" when in fact the tools had been
auto-approved and the child was running normally.
New vocabulary at the storage API boundary (column server_default stays
`""` so pre-fix legacy rows are still distinguishable as such):
- `pending` — at insert, before any resolution
- `approved` / `denied` — manual user resolution
- `timeout` — approval-event timeout (split from `denied` so the
audit column alone tells them apart; the feedback
string used to carry this distinction)
- `policy` / `blanket` / `skill` / `always` / `auto_approve_tools` —
auto-approve reasons (mirror `AutoApproveReason`)
Heuristic verdicts on the two auto-approve early-return branches are now
persisted with `user_decision=<reason>` (previously dropped on the floor).
Late LLM verdicts for already-auto-approved call_ids look up the reason via
a TTL-pruned `_auto_approve_reasons` map (lazy 60s prune at write time, so
no fixed cap can silently regress the fix on the N+1th auto-approve; LLM-
disabled sessions don't leak entries because prune fires whenever auto-
approves happen).
Bug fixes caught during review:
- `on_intent_verdict` early-returns when the verdict already carries an
auto_reason — without this, a manual `resolve_approval` on a mixed batch
would overwrite the auto-stamped row with `approved`/`denied`.
- `_record_auto_approves` runs BEFORE `_persist_auto_approved_heuristic_*`
so the lookup map is populated before any concurrent LLM verdict can
fire and miss it.
- `resolve_approval(timeout=True, approved=True)` now raises ValueError
to make the split-brain shape unrepresentable.
- Approval-timeout feedback string derives from `_APPROVAL_WAIT_TIMEOUT`
rather than the hardcoded "1 hour".
The previous comment described "\\" as "non-default", which is
backwards — "\\" is the SQL standard escape character. The
actually-non-default part is SQLAlchemy's ``.like()`` itself: it
defaults to no escape character, so ``escape_like``'s output is only
interpreted correctly when callers pass ``escape=LIKE_ESCAPE``
explicitly. Reword to put the caller-side requirement first.
WatchRunner._poll_watch committed active=False to the row BEFORE
calling _dispatch_result for a terminal fire, and the dispatch closure
registered by ChatSession.set_watch_runner enqueued each reminder with
a valid_until=is_watch_active predicate that re-read the row at drain
time. Since the runner already flipped active to 0, the predicate
returned False for every dispatched fire and NudgeQueue.drain silently
dropped the entry — the model never saw a watch result. Then a
subsequent action=cancel call hit list_watches_for_ws (filters
active==1), the now-inactive row was invisible, and the cancel
returned 'Watch "X" not found.' regardless of whether the watch had
actually run.
Reorder _poll_watch to dispatch before the row write, drop the
valid_until predicate from the watch closure (its only effect was the
bug above), and add a _terminal_dispatched guard on the runner so a
transient storage failure between dispatch and row-write doesn't
re-fire the reminder on the next tick. Add WatchRunner.forget_terminal_dispatched
and call it from the cancel path so an out-of-band deactivate (next_poll='')
doesn't leak the watch_id from the runner's pending-retry set indefinitely.
Cancel-by-name now routes through a new find_watch_by_name storage
method that ignores the active filter and prefers active rows over
newer-inactive same-name siblings. The session.py cancel branch
distinguishes 'already completed (auto-cancelled)' from 'not found'
so the model can tell apart 'this watch ran and finished' from
'no such watch.' Consolidate the two byte-identical _escape_like
/ _escape_ilike helpers in the storage backends into a single
turnstone.core.storage._utils.escape_like and apply it to the new
find_watch_by_name LIKE pattern so a model-supplied watch name
containing % or _ can't redirect a cancel to a sibling watch.
NudgeQueue.drain previously dropped predicate-failed entries without
logging anything, which is what hid this bug for so long. Drain now
emits nudge_queue.predicate_dropped: info for reason=predicate_false
(the normal lifecycle case — idle_children when every active child
finished between enqueue and drain), warning with exc_info for
reason=predicate_raised (a misbehaving predicate).
Tests: new test_poll_watch_terminal_fire_survives_drain (parametrized
stop_on_fired + max_polls_reached) drives the real WatchRunner._poll_watch
against a real tmp_db row and confirmed to fail against pristine main.
test_poll_watch_retry_deactivate_after_update_watch_failure exercises
the _terminal_dispatched retry-deactivate branch end to end.
test_cancel_clears_pending_terminal_dispatched_entry covers the cancel-
path leak case. test_find_by_name_prefers_active_over_newer_inactive
catches the ordering regression. test_find_by_name_treats_percent_as_literal
+ test_find_by_name_treats_underscore_as_literal pin the LIKE escape.
The shared_static exclude in scripts/update-vendored-js.sh was meant to
skip self-references inside vendored libraries, but it also hid
shared_static/renderer.js — which loads the vendored libs and pinned
mermaid-11.14.0 across every renovate bump since #426. Tests under
tests/test_web_helpers.py were similarly invisible because the include
list omitted *.py.
Replace the broad shared_static exclude with the specific old-versioned
vendor directory (about to be rm -rf'd next anyway), and add *.py to the
include list. Bump renderer.js to mermaid-11.15.0 to repair the live
404, and refresh the test fixtures to current vendor versions so they
stop drifting.
Line 3446 used double-quote string delimiters with an embedded ">
that terminated the string mid-attribute, leaving "bulk-revoke (" as
bare tokens. The rest of the surrounding block uses single-quote
delimiters; switch the broken line to match so the embedded > and "
sit safely inside the string.
The parse error wiped out every global in admin.js, so showAdmin and
the rest of the admin entry points were undefined — the console was
non-functional whenever an MCP server row had consented_users_count > 0.
Backports OAuth-MCP Phase 9 (#516) to the stable/1.5 track. Introduces
forward-only migrations 054_mcp_pending_consent and
055_mcp_user_tokens_server_index.
* feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9)
Completes the OAuth-MCP build-out (Phases 0-8 shipped) by closing the
operator + deferred-consent gaps:
1. **Per-(user, server) deferred-consent persistence** — when a
non-interactive run (scheduled / channel) hits ``mcp_consent_required``
or ``mcp_insufficient_scope``, the sync pool dispatchers now upsert a
row into a new ``mcp_pending_consent`` table. The dashboard hydrates
the gear-icon badge from this table on load, so users who weren't
online to see the in-flight SSE prompt still surface the deferred
work on next login. Cleared automatically by the OAuth callback
handler on consent completion; manual user dismiss via new DELETE
endpoints. Composite PK ``(user_id, server_name)`` collapses repeat
occurrences for the same server — no NULLs-not-distinct trap.
2. **Admin status pill + bulk-revoke** — the MCP Servers admin row now
shows ``consented_users_count`` for ``auth_type=oauth_user`` rows
when ≥1, with a two-step-confirm ``bulk-revoke`` button that drops
every user's token for the server via the existing
``delete_mcp_oauth_rows_by_server_name`` primitive. Upstream RFC
7009 revoke is intentionally NOT attempted in bulk (avoids N
upstream HTTP calls per admin click); audit detail records
``upstream_revoke_outcome=bulk_admin_no_upstream``. A "last
refresh" pill (age + outcome) renders on each row, sourced from a
new ``_last_refresh`` dict populated by ``_refresh_server`` on every
call (both manual ``refresh_sync`` and the ``_cb_auto_reconnect``
follow-up).
3. **ClientType.SCHEDULED** added to the prompts module + scheduler
passes it through to ``create_workstream``. ``ChatSession`` now
computes ``_is_interactive_for_consent`` at construction (WEB / CLI
are interactive; CHAT / SCHEDULED are not) and plumbs the flag
through ``call_tool_sync`` / ``read_resource_sync`` /
``get_prompt_sync`` to the three sync dispatchers. The wrap at the
``_is_structured_error`` gate routes consent codes to the new
``_record_pending_consent_best_effort`` helper for non-interactive
callers only; interactive sessions stay on the in-flight SSE path
Phase 8 ships unchanged.
4. **Operator docs** — ``docs/mcp-oauth.md`` (operator guide, parallel
to ``docs/oidc.md``: ``auth_type`` choice, OAuth client setup,
encryption-key rotation, troubleshooting matrix) and
``docs/operations/mcp-oauth-headless.md`` (one-paragraph runbook
per ``feedback_runbook_trust_llm.md``: pre-consent recipe for
scheduled / channel-driven runs).
Schema
- Migration 054_mcp_pending_consent.py — composite PK
``(user_id, server_name)``, ``occurrence_count`` + ``first_seen_at`` /
``last_seen_at`` for recency metadata, ``idx_mcp_pending_consent_user``
for the badge-load query. No FKs (matches the rest of the
oauth_user schema).
- Migration 055_mcp_user_tokens_server_index.py — adds
``idx_mcp_user_tokens_server`` on ``(server_name, expires_at)`` so
the admin pill's ``count_mcp_consented_users_*`` queries don't
full-scan against the leading-``user_id`` composite PK.
- Cross-backend: works on SQLite + PostgreSQL via dialect-specific
``on_conflict_do_update`` (PG ``postgresql.insert`` / SQLite
``sqlalchemy.dialects.sqlite.insert``). No ``NULLS NOT DISTINCT``
needed — the simplified PK eliminates the cross-version trap.
Endpoints
- ``GET /v1/api/mcp/oauth/pending`` — list deferred-consent records for
the authenticated user. Install-level gate via cached
``any_oauth_user_mcp_servers`` short-circuits to ``{pending: 0}`` on
installs with no oauth_user MCP servers — local-auth deployments
exercise zero new storage queries on this path. The gate result is
cached on ``app.state`` with a 60s TTL to spare repeat dashboard
loads.
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` — single dismiss.
Returns 204 in both existed-and-deleted and never-existed cases
(no cross-tenant existence leak); audits
``mcp_server.oauth.pending_consent_dismissed`` with
``mode=single`` + ``cleared=0|1`` so a session-hijack attacker
scrubbing breadcrumbs leaves an audit trail.
- ``DELETE /v1/api/mcp/oauth/pending`` — bulk dismiss; audits
``mode=bulk`` + ``cleared=N``.
- ``POST /v1/api/admin/mcp-servers/{name}/bulk-revoke`` — admin
bulk-revoke for the named server's per-user tokens. Requires
``admin.mcp`` permission + 400s when the row isn't ``oauth_user``.
All four registered on both ``turnstone-server`` and
``turnstone-console`` (mirrors the Phase 8 ``/connections`` endpoint
shape).
Performance
- Admin list handler now uses a single ``GROUP BY`` bulk-count query
(``count_mcp_consented_users_grouped_by_server``) wrapped in
``asyncio.to_thread`` rather than N per-row sync DB round-trips
inside the async handler. Skipped entirely when no row is
oauth_user.
Frontend
- ``ui/static/app.js``: ``loadPendingConsents()`` hydrates the
existing ``_pendingConsentServers`` set on dashboard init + after
the user opens the settings modal. Endpoint failures stay silent
— the badge will be re-driven by the next in-flight tool error.
- ``console/static/admin.js``: ``consented_users_count`` pill +
``bulk-revoke`` button on each MCP row (only when ≥1 consented),
two-step confirm matching the existing delete pattern. ``last-
refresh`` age + outcome pill in the per-row status cell, sourced
from the freshest per-node entry in ``status[*].last_refresh_at`` /
``last_refresh_outcome``. CSS for the pills in ``style.css``.
Tests
- ``test_mcp_pending_consent_storage`` — 13 tests covering upsert
idempotency, list ordering, per-user isolation, single/bulk delete,
count-by-server + grouped variant, install-level gate.
- ``test_mcp_pending_consent_dispatch`` — 9 tests, including the
boundary-cross gate per ``feedback_tests_through_boundaries.md``:
drives the real ``call_tool_sync`` → ``_dispatch_pool_sync`` →
``_is_structured_error`` → ``_record_pending_consent_best_effort``
with a mocked classified-lookup so the structural plumb-through is
verified end-to-end. Includes a storage-failure test that pins
the docstring's "envelope unchanged on storage failure" promise.
- ``test_mcp_pending_consent_endpoints`` — 11 tests: install gate,
list-for-self, no-cross-user-leak, single/bulk delete, idempotent
not-found, audit emission on single + bulk + cross-tenant dismiss.
- ``test_chat_session_interactivity_flag`` — 7 tests pinning the
``ClientType`` → ``_is_interactive_for_consent`` mapping against
the module-level ``INTERACTIVE_CONSENT_CLIENT_TYPES`` frozenset.
- ``test_mcp_admin_bulk_revoke`` — 7 tests covering admin.mcp
permission gate, 404 on missing, 400 on non-oauth_user, 200 with
``rows_deleted`` + ``consented_users_before``, audit row with
``upstream_revoke_outcome=bulk_admin_no_upstream``, cross-server
isolation.
- ``test_mcp_oauth_handlers`` — 2 new callback tests pin the post-
callback ``delete_mcp_pending_consent`` invocation: success-clears
+ storage-failure-still-redirects.
- 636 tests pass on the impacted surface (47 new + Phase 0-8 OAuth-MCP
+ session + prompts + storage admin). ruff + mypy clean.
Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}`` — the
flag flows only through the pool dispatchers, which only fire when
the row resolves to ``oauth_user``.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) preserved on every
AS / SDK / pool-loop await — no new awaits added to the hot path.
- Install-level gate on the badge endpoint: cached
``any_oauth_user_mcp_servers`` returns False on a row-less
deployment → endpoint short-circuits without touching the pending-
consent table; 60s TTL bounds the staleness window after admin
flips ``auth_type``.
- Operator-actionable codes (key-unknown, url-insecure, *_forbidden)
explicitly filtered out of persistence — they're outside the
user-facing consent badge scope.
- Best-effort write: the structured-error envelope returned to the
agent is identical whether the persistence write succeeds or fails
(storage exception is logged with type name only — no chained
context that could carry an ``httpx.Request`` bearer header).
- No ``exc_info=True`` on any new path that can chain a bearer-bearing
``httpx.Request``.
- Defensive parsing: ``_parse_pending_consent_envelope`` mirrors
``_is_structured_error``'s ``isinstance(decoded, dict)`` guard plus
filters scope tokens through ``is_valid_scope_token`` capped at
``MAX_INSUFFICIENT_SCOPE_REPORTED`` — defense-in-depth even though
production callers already validate upstream.
- Audit events on every dismiss endpoint so a session-control attacker
scrubbing dashboard breadcrumbs still leaves a trail.
Cross-backend
- Tested on SQLite via the conftest backend fixture.
- PostgreSQL path uses ``postgresql.insert(...).on_conflict_do_update``
parallel to the existing ``mcp_user_tokens`` upsert in Phase 3.
Deferred (not Phase 9 blockers)
- Multi-node pool eviction on bulk-revoke: only local-node sessions
would be evicted if we built it, and there's no bulk-by-server
primitive on MCPClientManager today; remote nodes will surface as
a 401 on next dispatch which refreshes through the (now empty)
token row.
- RFC 8693 / Azure OBO ``auth_type=oauth_token_exchange`` — captured
in the design doc as a future architectural direction (~600 LOC +
IdP-side admin work); requires OIDC token capture and per-MCP-server
resource-trust configuration that v1 does not ship.
* docs(mcp): address Copilot review feedback on Phase 9
- Fix misleading admin.js comment that claimed the refresh pill rendered
"<short-relative> <outcome>" — the pill actually renders only the short
age, with outcome reflected via CSS class and tooltip.
- Replace broken feedback_secrets_not_in_env.md repo-root link in
mcp-oauth.md with the inlined rationale (env-borne secrets reachable
via shell tools / os.environ; TOML secrets are not).
Adds notes for the 14 patches cherry-picked to stable/1.5 since 1.5.12:
reactive PG LISTEN/NOTIFY node discovery + event-driven wait_for_workstream,
memory tool audit trail, task_agent skill personas, plus fixes for the
LLM-visible default alias bypass, mermaid streaming parse errors,
proxy-prefixed re-auth, dashboard appbar visibility, and the PG test
backend on the notify dispatcher suite.
Introduces forward-only migration 053_services_notify_trigger.
- Put ``skill`` back in the access-denial list in the tool
description with a clarification — TASK_AGENT_TOOLS does not
include the skill tool, so sub-agents cannot switch personas
mid-task. Removing the disclaimer entirely created an ambiguity
the LLM could misread.
- Minimize the skill_data carried on the approval item dict to
``name`` / ``content`` / ``risk_level`` only. ``get_skill_by_name``
returns the full ~30-column prompt_templates row including
``scan_report``, ``installed_by``, ``source_url`` — none of those
flow through ``_exec_task`` / ``_evaluate_intent``, and they
shouldn't ride along any future audit serializer that reads the
approval item shape.
- Regression test for ``skill=""``, whitespace-only, and ``\t\n``
values — pins the documented "empty value is acceptable" contract
at the ``(args.get("skill") or "").strip()`` chokepoint.
The task_agent tool now accepts an optional ``skill=<name>`` argument
that loads the named skill's content as the sub-agent's persona,
substituting the hardcoded "# Task Agent" identity statement. The
operating-guidance numbered list (one-shot, tool-use over narration,
no follow-up questions) is layered on top of every persona and always
applies — those are sub-agent semantics that a persona should ride on
top of, not replace.
Validation lives in ``_prepare_task`` so the approval surface tells
the operator what they're consenting to: the validated skill dict
(including content) rides on the item dict from prepare to exec to
defeat TOCTOU between consent and execution. An unknown skill
returns a clean error item with a hint pointing at
``skill(action='search')``; a disabled skill returns a distinct error
so the LLM's recovery path can tell "not found" from "quarantined",
mirroring the enabled gate that ``_exec_skill(action='load')`` and
skill-search already apply.
High and critical skills now surface their risk tier on the approval
header (``, risk: critical``) and emit a
``task_agent.high_risk_skill`` warning — same signal ``_load_skills``
emits for session-level skills, so the operator sees the same flag
whether the skill is loaded session-wide or per-call. ``_exec_task``
emits a ``task_agent.skill_invoked`` info log on the skill branch for
forensic traceability — the approval row captures the choice at
consent time, this log captures it at exec time so post-incident
search doesn't have to cross-walk approval and exec tables.
The ``_evaluate_intent`` func_args projection now includes the skill
name — without it, heuristic ``arg_pattern`` rules targeting a risky
persona name on ``task_agent`` silently no-op and the audit row loses
the choice. Mirrors the long-standing ``spawn_workstream``
projection.
Caught by Copilot on PR #514. openSettingsMenu sets _settingsMenu
synchronously, but the menu's keydown handler was registered inside
setTimeout(0). The previous-commit guard in the global keydown
handler returns early when _settingsMenu is set (so dashboard isn't
hidden by Escape over the menu), which created a window where
Escape had no handler at all — the global skipped, the menu's own
listener wasn't ready yet, and the menu got stuck open until the
next interaction.
Attach keydown synchronously; keep mousedown + initial focus in
setTimeout (mousedown to avoid the opening click triggering its own
outside-click close, focus because the menu DOM needs a tick to
settle layout).
Two related changes that surfaced when the user pointed out the proxy's
node-picker pill was unreachable from the proxied dashboard view: the
dashboard overlay was covering the entire appbar.
- Dashboard overlay now starts at top: 48px so the appbar (with the
proxy-injected node picker) stays visible and interactive while the
dashboard is open. showDashboard no longer marks ui-header inert
(tab-bar and split-root still are). The dashboard's role downgrades
from dialog+aria-modal to region — the appbar being reachable above
it would otherwise contradict aria-modal's "ignore everything else"
semantics.
- Gear icon converts from a direct openSettingsPanel() click into a
dropdown menu with two items: "MCP connections" (existing modal) and
"Logout". Reuses the .ws-tab-dropdown shell for visual consistency
with the workstream tab chevron menu and the proxy node-picker.
Logout uses .destructive styling to reduce misclick risk.
Bug fixes caught by the merged code-review pipeline:
- Global Escape handler skips when _settingsMenu is open, otherwise it
fires hideDashboard() before the menu's own handler — wiping the
composer text + staged attachments out from under the user.
- Menu-item click refocuses the trigger before close, so
openSettingsPanel captures the gear (not <body>) as the eventual
return-focus target.
- ArrowUp keyboard cycling uses idx <= 0 ? len - 1 : idx - 1 instead
of (idx - 1 + len) % len so the no-focus case wraps to the last
item rather than the second-to-last. Same fix backported to
showTabDropdown which had the identical modulo bug.
- Position clamps reordered: right-edge override now runs before the
left-edge floor so a menu wider than the viewport still clamps to
mx >= 4 instead of going negative.
- openSettingsMenu caches _settingsMenuTrigger so closeSettingsMenu
can reset ARIA without re-querying the gear by id.
- aria-controls lifecycle wired both ways (set on open, removed on
close).
The LLM was passing ``task_agent(model="default")`` (and the same for
plan_agent) and routing to whichever backend the auto-created
``default`` alias was attached to at boot — flatspark in the verified
case (ws_id 7dde674) — silently bypassing the operator-configured
``model.task_alias`` / ``model.plan_alias`` (gh200).
Root fix:
- ``load_model_registry`` only synthesises the back-compat ``default``
alias when neither DB nor ``[models.*]`` populate the registry. The
shim was only ever meant for single-CLI-model setups; with a multi-
model DB it became a phantom routing target aliasing ``LLM_BASE_URL``.
- ``_render_agent_tool_descriptions`` filters ``default`` out of the
LLM-visible alias list. The English reading of "default" trips the
model into picking it explicitly even when the description tells it
to omit ``model=`` for the per-role default.
Defense-in-depth at the validator chokepoint
(``_validate_agent_model_override``): explicit rejection of
``alias == "default"`` (post-strip) with corrective guidance;
``default`` filtered out of the unknown-alias retry list so an LLM
probing with a bogus alias can't enumerate it back; the no-alternatives
wording is distinguished from the no-registry-configured wording. The
render path also always rewrites tool descriptions instead of returning
early on filter-empty, so a reload that drops the registry to only
``default`` clears stale alias names left over from a prior render.
Previously only the admin-console DELETE route emitted memory.delete
audit rows, so a long-running session whose memory was deleted via
the admin UI had no log trail showing what happened — masking
out-of-band deletes as apparent tool bugs.
The save branch now stamps memory.save (new row) or memory.update
(upsert); the delete branch does a lookup-then-delete-by-id pair so
the audit can record the resolved memory_id and type. All emissions
are best-effort: failures log at debug and swallow so an audit hiccup
never breaks the tool call itself. Reads (get/search/list) remain
un-audited.
Copilot review on #511 flagged that the dispatch chain and the tests
both claimed to be in lockstep with one another, but only the comment
text said so — the parametrize list and the if/elif chain were two
independent hand-maintained copies, and the comments still referenced
the (long-reverted) ``_PROXY_AUTH_LOCAL_HANDLERS`` symbol.
Make the lockstep guarantee real by collapsing both copies onto one
``_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str]`` mapping
``(method, path)`` to handler-name strings. ``proxy_api`` resolves
the name through ``globals()`` at call time so ``patch(...)`` in
tests still observes the override — a dict of function refs would
have captured the originals at module load (which is why the first
attempt at this dispatch broke the tests and got reverted). Test
cases now derive directly from ``_PROXY_AUTH_LOCAL_HANDLERS.items()``,
so adding or removing an entry in the dispatch table flows through
to the parametrize list automatically and the two can't drift.
When the user is on a proxied node page (``/node/{id}/...``) and the
JWT expires, the in-page login modal POSTs to ``/v1/api/auth/login``
which the proxy shim rewrites to ``/node/{id}/v1/api/auth/login``.
Two latent bugs both had to be fixed for the user to be able to
re-authenticate from inside the proxied UI:
1. ``is_public_path`` didn't recognise the ``/node/{id}/`` prefix
over a public path, so the console's ``AuthMiddleware`` 401'd the
login POST before any handler ran. Extended via the existing
``_extract_proxied_path`` helper so a proxied public path stays
public.
2. Even if the path had been public, ``proxy_api`` would have
forwarded the request to the upstream node. The upstream mints
``JWT_AUD_SERVER`` tokens; the console's ``AuthMiddleware``
(expecting ``JWT_AUD_CONSOLE``) would reject those on the next
proxied call, and ``_proxy_post`` drops ``Set-Cookie`` when
forwarding anyway. ``proxy_api`` now dispatches every entry in
``_PROXY_AUTH_LOCAL_PATHS`` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's
own auth handlers, and short-circuits non-canonical methods on
those paths with 405 instead of letting them slip through with
the service-token fallback.
Tests parametrize across all eight local-dispatch entries so a future
refactor that drops a branch (or routes it through ``_proxy_post``)
fails loudly, plus a no-auth-header reproduction for the original
lockout and a 405 regression guard for the method-mismatch surface.
* fix(renderer): mermaid streaming parser errors + progressive hljs
Live streaming was rendering mermaid diagrams with `Parse error,
got 'PS'` messages — bare `(`, `[`, `{` inside unquoted edge / node
labels re-entered Mermaid's shape parser. Two unrelated streaming-
specific issues in the renderer pile-up here; this commit addresses
both plus a follow-on UX improvement for code highlighting.
## Mermaid label autoquoter
`_normalizeMermaidSource` wraps two label forms that Mermaid rejects
when they contain bare shape-delimiter chars:
1. Edge labels: `|content|` → `|"content"|`
2. Rectangle node labels: `ID[content]` → `ID["content"]`
Shapes whose syntax already nests delimiters — cylinders `[(...)`,
subroutines `[[...]]`, trapezoids `[/.../]` `[\...\]`, circles
`((...))`, hexagons `{{...}}`, diamonds `{...}` — are intentionally
left alone (their inner delimiters are part of the shape syntax;
quoting would corrupt them). Labels already wrapped in `"..."` are
also left alone. The rewrite is idempotent and runs before the
mermaid SVG cache lookup so identical malformed input hits the
cache on re-render rather than re-quoting per tick.
## Markdown fence-pair regex
The old fence regex `/(```+)([^\s`]*)\n([\s\S]*?)\1/g` would, mid-
stream, pair an unclosed ```mermaid open with the OPENING backticks
of a later ```python fence as the "close", handing mermaid a
truncated source. New regex:
/(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g
Two constraints close the gap:
- `(?!\1)` inside the content quantifier blocks the lazy matcher
from extending across another N-backtick run. Smaller inner
counts (e.g. 3-backtick inner inside a 4-backtick outer) still
pass since `\1` is the open's actual count.
- `[ \t]*(?=\n|$)` after `\1` forces the close to a line
boundary, so ```python (open with a language tag) can't
masquerade as a previous fence's close.
Together: an unclosed fence stays as plain markdown until its true
close arrives, so neither mermaid nor hljs ever sees a mid-stream
truncated source.
## Progressive hljs
Extracted `postRenderHljs` from `postRenderMarkdown` with a source-
keyed `_hljsCache` (FIFO, cap 64, keyed on `language:source`) and
wired it into `_streamingRenderApply`. Closed code fences are now
syntax-highlighted as they stream in, matching the progressive
mermaid pattern from #426. Per-tick cost stays cheap because the
cache returns the pre-tokenized HTML synchronously on hit; only
unique (language, source) pairs pay `hljs.highlightElement`.
## Internal cleanup from the review pipeline
- `_cacheFifoEntry(cache, key, value, max)` replaces the duplicated
`_cacheHljsEntry` and `_cacheMermaidEntry`. Single tested
implementation across four caches (hljs, mermaid svg, mermaid
error, mermaid normalize memo). The "don't evict on overwrite"
invariant is pinned per-cache in tests.
- `_mermaidNormalizeCache` memoizes raw textContent → normalized
output so the per-rAF-tick autoquoter split + regex doesn't
repeat for unchanged diagrams. Eviction shares
`_MERMAID_CACHE_MAX` with the SVG cache it feeds.
## Tests
The fake DOM in tests/test_renderer_js.py grew a few capabilities
to drive these paths:
- `classList` is now array-like (length + indexed access) so the
hljs language-extraction loop works.
- `textContent` setter mirrors the real-DOM side effect of
entity-escaping into innerHTML, so `escapeHtml()` round-trips
(otherwise every `renderMarkdown` returns empty `<p>` tags).
- `querySelectorAll` handles both `pre code.language-mermaid`
and `pre code[class*='language-']`.
Added: 6 fence-pairing regression cases, 9 hljs-progressive cases
(cache hit / distinct sources / language separation / NO_HIGHLIGHT
langs / terminal class / eviction / overwrite / postRenderMarkdown
wraps hljs / _streamingRenderApply invokes hljs), 11 autoquoter
cases including both diagram sources from the live screenshot
encoded verbatim as parametrized regressions, and 3 normalize-memo
cases (populates on first call, consulted before normalize via
sentinel pre-seed, distinct sources cache separately).
Total: 104 renderer tests pass (was 67).
* fix(renderer): apply Copilot review feedback on #510
Two doc / harness adjustments from the PR review — no behavior
change in production code.
- The `_mermaidNormalizeCache` comment claimed eviction "stays in
lockstep with the SVG cache". That was misleading: the two
caches key on different things (raw textContent vs normalized
source) and evict independently. Updated the comment to describe
what they actually share (the cap, for memory footprint) and
what they don't (positional coupling), and to note that the memo
deliberately survives `_initMermaid` since normalize output is
theme-independent.
- The fake DOM in tests/test_renderer_js.py had `innerHTML` setter
clear `children` but leave `_textContent` intact, so subsequent
`textContent` reads could return stale data after an innerHTML
mutation (real DOM invalidates textContent on innerHTML write).
No current test triggered this, but it would mask future bugs
that depend on innerHTML/textContent consistency. Setter now
clears `_textContent`; the children-derived fallback in the
getter returns `''` after the wholesale replace.
All 104 renderer tests still pass; ruff + mypy clean.
CI's postgres-backend run failed 11 of the new notify tests from #505.
Three independent issues:
1. Migration 053's ``services_notify`` trigger lives only in the
alembic chain, but the test fixture in conftest.py calls
``init_storage(..., run_migrations=False)`` for speed. That path
skips migrations and relies on ``metadata.create_all`` for the
table tree. Previous alembic-only DDL (migrations 041 / 048
``CREATE INDEX CONCURRENTLY`` on workstreams) is performance-only,
so tests never depended on it. 053's trigger is the first
behaviorally-required alembic-only DDL in the project — without it
``register_service`` doesn't fire NOTIFY and the trigger-filter
tests time out.
Fix: declare the trigger function + trigger in ``_schema.py`` and
attach them via ``sa.event.listen(services, "after_create", ...)``
DDL events, gated on ``dialect == "postgresql"``. The same SQL
constants are imported by migration 053 so there's a single source
of truth. Test fixture stays unchanged — ``create_all`` now
installs the trigger on fresh PG test DBs. Migration covers the
upgrade-on-existing-DB path; the two are mutually exclusive given
``create_tables = not run_migrations`` in ``init_storage``.
2. NotifyDispatcher tests fired ``storage.notify(...)`` immediately
after ``d.start()`` and hit a race: the listener thread is
concurrently calling ``psycopg.connect(listen_url)`` + ``LISTEN
<channel>`` over the network, so the notify can land before any
session is listening on the channel and PG drops it (pg_notify
only routes to sessions LISTEN'ing at COMMIT time).
Fix: dispatcher gains a ``_listener_ready: threading.Event`` set
inside ``_listener_loop`` after each successful ``storage.listen``
open and cleared on disconnect, plus a public
``wait_until_ready(timeout)`` method. Tests use a new
``_start_ready(d)`` helper that calls ``start()`` + asserts ready.
Production callers don't need this (real reactive traffic arrives
well after startup), but it's the right primitive for any future
"start dispatcher, immediately send" call site too.
3. ``TestSqliteNotify`` is misnamed — its tests run against whichever
backend the ``storage`` fixture provides (PG by default in CI).
Two of its assertions were SQLite-specific:
``assert got.pid == 0`` only holds for the synthetic in-process
path (PG carries real backend PIDs), and
``test_synthetic_sweep_emits_after_interval`` is fundamentally
SQLite-only (no sweep on the PG path).
Fix: drop the pid assertion (channel + payload are the
backend-agnostic invariants), add an ``_is_sqlite`` fixture mirror
of ``_is_postgres``, and gate the sweep test on it. The sweep
test also moves from monkey-patching ``stream._sweep_interval`` to
passing the ``sweep_interval`` kwarg that ``SQLiteBackend.listen``
now accepts (from the earlier Copilot review fix).
Validated locally against a fresh ``turnstone_test`` PG DB: 263
storage + console + notify tests pass on PG, 257 on SQLite, mypy +
ruff clean.
Retire two polling patterns in coord that have clean event sources.
PR 2 of 3 in the coord-completion stack; sits on top of PR #505
(reactive node discovery via PG LISTEN/NOTIFY).
`wait_for_workstream` (coord's block-wait tool) polled storage every
0.5 s in a worker thread regardless of whether anything had changed —
a 600 s wait incurred ~2400 round-trips. Now subscribes to a new
in-process `ChildEventBus` (`turnstone/core/child_event_bus.py`) and
blocks on `threading.Event.wait(min(remaining, WAIT_HEARTBEAT_INTERVAL))`:
- `CoordinatorAdapter` owns the bus; `_dispatch_child_event` calls
`bus.notify(child_ws_id)` after each `_enqueue_on_ui` for the
state-class branch (cluster_state, ws_closed, ws_rename,
intent_verdict, approval_resolved, approve_request).
- Wait loop clears the Event BEFORE the storage snapshot to close
the subscribe/check race; a notify between clear and the next
`wait()` leaves the Event set so the loop re-reads without
losing the wake-up.
- 2 s heartbeat cap preserves the existing `wait_progress` SSE
cadence for the sidebar UI while cutting SSE traffic ~4x vs the
pre-bus 500 ms cadence in the quiescent case.
- Worst-case completion latency is 2 s (vs pre-bus 0.5 s) because
`set_state` buffers non-ERROR writes through `StateWriter`
(async-flushed) while `emit_state` fans out immediately — a
bus-driven wake can beat the flusher and read pre-transition
state, then re-block until heartbeat. Deliberate trade-off; the
SSE-traffic reduction outweighs the regression on the most
common terminal transition.
- Defense-in-depth: ownership-filter `cleaned` to own-subtree
before `register_waiter` so a foreign ws_id passed by an
untrusted coord LLM (prompt injection) can't observe wake-up
timing as a side channel. Predicate (`_row_in_own_subtree`)
requires both `parent_ws_id == coord_ws_id` AND `user_id ==
coord_user_id` parity — same gate strength as the existing
`_is_own_subtree` mutating-op guard, so a corrupted /
cross-tenant `parent_ws_id` alone can't satisfy it. Shared with
`_snapshot_all` so the snapshot's `denied` shape stays in
lockstep with the bus filter (Copilot review on #506).
Coord idle-cleanup thread polled the storage scan every
`check_every` seconds (~30 s on default 2 h timeout) even when no
coord was anywhere near idle. Now subscribes to
`SessionManager._state_subscribers` with a `tick_now` event and
blocks on `tick_now.wait(check_every)` — any state change wakes
the sweeper without waiting a full interval, AND the timeout still
fires the periodic sweep for the DB-orphan-only case. A
`min_sweep_interval=5 s` floor bounds DB-call traffic at ~0.2/s
under sustained activity so the loop can't tight-spin `close_idle`
at the rate of its own DB latency (6x improvement over the
pre-refactor fixed 30 s cadence under any activity, and prompt
state-change-driven wakes when below the floor).
`CoordinatorClient` constructor takes `child_event_bus` as a
required kwarg — there's no external SDK shape to preserve and
keeping it optional would silently mask a wiring bug in any future
caller. Tests construct their own `ChildEventBus()` per fixture.
Tests: 16 unit tests for `ChildEventBus` (register / unregister
symmetry, multi-waiter fan-out, multi-child waiter, subscribe/check
race, concurrent register / notify smoke); 7 new adapter tests
(bus notify fires for all 6 state-class events, drops for unknown
child / wrong ws_id); 7 new coord-client wait tests (subscribe-
after-terminal, notify wakes, unrelated notify doesn't wake,
heartbeat fires without notify, unregister on exit, multi-waiter
independence, cross-tenant denial via the user_id-parity filter);
8 idle-cleanup tests (initial sweep, heartbeat cadence, exception
swallowing, stop_event clean exit, state-change wake, subscriber
cleanup, mid-sweep wake, `min_sweep_interval` floor). All pass;
ruff + mypy clean. Full non-live suite: 6227 passed (+2 vs prior
baseline).
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher
Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL
`LISTEN` connection and fans wake-ups out to per-channel handlers on a
separate dispatch thread. Cluster collector subscribes to a new
`services` channel and runs node discovery reactively — new-node /
graceful-deregister visibility drops from up-to-60 s to ~500 ms on
Postgres, with the 60 s discovery loop retained as the backstop for
crash-shaped node loss (NOTIFY only fires on real writes).
Storage layer gains a uniform `notify` / `listen` API:
- PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode
connection that bypasses pgbouncer (mandatory: pgbouncer is required
in transaction-pool mode per docs, which is incompatible with LISTEN).
- SQLite: in-process fan-out + synthetic-sweep fallback so consumer
code is identical across backends.
`TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml)
points the dispatcher's connection direct-to-Postgres. Defaults to the
main DB URL when unset.
Migration 053 installs the `services_notify` trigger; it filters
heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick
stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE
still fire.
Dispatcher detail:
- Two threads: listener (drains stream → bounded queue) and dispatch
(invokes handlers under exception suppression). Same-channel notifies
coalesce per dispatch batch so an N-node deploy burst is one
`_discover_nodes` per channel.
- Reconnect uses exponential backoff (1 s → 30 s cap). After any
successful reopen — whether the prior failure was a stream-poll error
or a connect / initial-LISTEN error — one synthetic Notify with
payload="reconcile" is enqueued per channel so handlers re-read on
the same code path they use for real events.
Future consumers (ConfigStore live reload, scheduler immediate
dispatch, audit live-tail) plug in by adding their channel to the
dispatcher's construction list.
Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub
storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter
tests, 4 collector wire-in tests. All pass; ruff + mypy clean.
* fix(notify): address Copilot review on #505
- _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via
dict.fromkeys before constructing the stream — duplicates would
otherwise register the queue twice and double-deliver each notify.
- _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval
parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what
the comment at the constant already promised, and lets future
consumers without their own polling timer pick a tighter cadence
without reaching into private stream attributes.
- _sqlite.py: documented the `except queue.Empty: pass` end-of-drain
termination so it's not mistaken for swallowing an unexpected error.
- _postgresql.py: docstring referenced :func:`_pg_listen_url` which
was renamed to _resolve_pg_listen_url during PR development.
- notify_dispatcher.py: module docstring referenced a non-existent
_bootstrap_console_subsystem; wire-in is at console/server.py::main.
Refuted (no change, false positives from github-code-quality bot):
- 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies
(idiomatic Python Protocol declaration, not dead code).
- 2× "Mixed import style" in tests — `import ... as nd_mod` is
intentional to allow attribute assignment for monkey-patching the
module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally.
Converts [Unreleased] to [1.5.0] and adds individual sections for
1.5.1 – 1.5.12. Covers: MCP OAuth 2.1 + PKCE (Phases 1–8), OIDC
hardening, metacog NudgeQueue + wake trigger, SSE refresh-resume,
reasoning persistence (Phases 1–4), structured watch-result cards,
skills unlock, inline child approvals, Stage 3 Children primitive
lift, coordinator composer parity, node capability auto-detection,
progressive mermaid rendering, and the full schema migration list
for each release.
Updates the track list to stable/1.4, stable/1.5, and main.
Editing the first message in a workstream sends /rewind N where N is the
total user turns, leaving session.messages empty. The handler guarded the
history event with `if history:`, so only clear_ui was emitted. The
frontend dispatches the queued edit-and-resend from the history event
handler (app.js _pendingEditSend), so an empty history orphaned the
pending text and left the composer stuck in busy.
replayHistory already handles the empty case via showEmptyState(), so
emitting the event unconditionally is safe and unblocks the dispatch.
A bare ``httpx.ReadTimeout`` previously surfaced as ``ReadTimeout: timed
out`` — no provider, no base URL, no model — leaving the user with no
signal to tell whether a model server hung, the URL was wrong, or the
model isn't loaded on the backend.
``ChatSession._format_backend_error`` now rewrites known boundary
exceptions (httpx ``ReadTimeout`` / ``ConnectError`` / etc. and OpenAI /
Anthropic SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``) into
operator-actionable text that names the provider, base URL (query
string stripped before ``sanitize_error_text`` redacts credentials),
and model. Matching is by class name so the helper carries no SDK
imports. Unrecognised exceptions fall through to the legacy
``f"{type(exc).__name__}: {exc}"`` shape, preserving existing grep
targets.
The Anthropic call sites in session.py passed the operator-side
`replay_reasoning_to_model` flag through without checking the
model's static `supports_reasoning_replay` capability. The OpenAI
Responses path AND-gated both flags in `_build_kwargs` so a model
without a reasoning lane (gpt-4o, etc.) silently skipped replay even
when the operator flag was set. The Anthropic path had no such gate.
For all current Claude entries this was a no-op asymmetry - every
`_ANTHROPIC_CAPABILITIES` row sets `supports_reasoning_replay=True`,
so `True AND op == op`. But:
- The capability flag was dead code on the Anthropic path
- A future Claude entry (or any Anthropic-shaped surface) shipping
with the cap left at its False default would have replay fire
anyway, against the cap declaration
- The asymmetry made `supports_reasoning_replay` an unreliable
signal - readers couldn't tell if it gated anything per-provider
Move the AND-gate into `_resolve_replay_reasoning_to_model` via a
new optional `caps=` kwarg. When caps is provided, the resolver
returns `operator_on AND caps.supports_reasoning_replay`; when
omitted (back-compat for any caller not yet updated), it returns
the operator flag unchanged.
Thread caps through the three call sites: `_utility_completion`
(non-streaming), `_try_stream` (streaming, hoisted resolution out
of the retry loop since caps are attempt-invariant), and the
agent `_api_call` closure in `_run_agent`.
With the AND-gate now living at the session resolver, the redundant
in-provider gate in `OpenAIResponsesProvider._build_kwargs` is
removed. The provider now trusts the resolved bool it receives,
matching the AnthropicProvider shape and giving the cap a single
source of truth across providers. The two provider-level tests
that pinned the in-provider gate
(`test_include_omitted_when_capability_false`,
`test_include_omitted_by_default`) drop out; the session-level
boundary test
`TestSessionToOpenAIResponsesBoundaryIntegration::test_capability_false_omits_include_even_when_flag_true`
already covers the same end-to-end invariant.
Tests added:
- 4 resolver-level tests pinning the AND-gate semantics +
back-compat when caps is omitted
- 1 wire-boundary integration test mirroring the OpenAI Responses
`test_capability_false_omits_include_even_when_flag_true` -
drives session._try_stream through the real AnthropicProvider
with operator flag True + capability False and asserts the
thinking block does NOT reach the SDK boundary
Existing `TestUtilityCompletionPassesFlag` test had its caps mock
upgraded from `SimpleNamespace` to a real `ModelCapabilities`
instance to satisfy the new attribute read and stay robust to
future capability fields.
Copilot review feedback on #500. The original
``list_available_models`` had an implicit cs=None branch where the
placeholder still advertised ``registry.default`` (filtered against
enabled rows) when ``app.state.config_store`` was None but
``coord_registry`` was bound — useful in the rare degraded state
where lifespan wired the registry but the ConfigStore failed to
initialise. The PR #500 refactor accidentally dropped that branch:
the helper requires a config_store, so the cs=None case fell out as
"blank coordinator default".
Add an explicit ``elif coord_registry is not None`` branch that
mirrors the helper's tier 3 with the placeholder's enabled-rows
filter applied. New test exercises this path by passing
``config_store=False`` to the test fixture.
Previously /v1/api/models (home composer placeholder) and
console/session_factory.py walked separate two-/three-tier chains for
the coordinator alias. session_factory was missing the
``model.default_alias`` tier, so admins who set the system default in
the Models tab would see it advertised but new coordinator sessions
would silently keep launching on ``registry.default``.
This commit:
- Extracts the chain into ``turnstone/console/coordinator_alias.py``.
``resolve_coordinator_alias`` returns the effective alias under a
shared three-tier policy: explicit pin → ``model.default_alias`` →
``registry.default``. Tier 2 is validated against
``registry.has_alias`` and falls through to tier 3 with a logged
warning if unknown. Tier 1 is intentionally passed through
unvalidated so an explicit operator pin surfaces as 503 at
``registry.resolve`` rather than being silently swapped out.
- Wires both call sites through the helper. The placeholder supplies
an ``alias_filter`` that restricts every tier to enabled DB rows so
the home composer never advertises a model the workstream picker
can't actually offer; the session factory uses no filter (matches
prior 503-on-typo behaviour for explicit pins).
- Adds direct integration tests for the session factory's chain
(``tests/test_console_session_factory.py``) and updates the
placeholder tests' fixture to provide a stub coord_registry, since
the helper now requires one.
Light-review followup on 389400c8.
The "mirrors session_factory.py:109-110" claim was inaccurate —
session_factory's chain is two tiers (coordinator.model_alias →
registry.default) and skips model.default_alias entirely. The
placeholder handler extends that chain with model.default_alias as
tier 2 so admins who set the default in the Models tab see it
advertised in the home composer. Comment now lists the three tiers
explicitly and flags the session_factory-vs-placeholder drift case
(where model.default_alias ≠ registry.default) as a separate issue
to track.
Also lifts the ``from types import SimpleNamespace`` import in the
test fixture to module level — minor readability cleanup.
Two Copilot-review followups on /v1/api/models default resolution.
- console/server.py: coordinator_default_alias now mirrors the full
fallback chain in console/session_factory.py:109-110 — explicit
coordinator.model_alias → model.default_alias → registry.default.
The registry tier was missing, so the home composer placeholder went
blank whenever an operator never set model.default_alias in the admin
UI even though new coordinator sessions still launch on
registry.default (loaded from config.toml [model].default by
load_model_registry). Two new tests cover the registry-default
branch and the disabled-alias guard.
- console/static/app.js: _resolveModelLabel returns "" (not the bare
alias) when the alias isn't found in the dropdown's model list, so
callers can rely on the documented "fall back to neutral placeholder"
contract. Matches the existing doc comment.
Bundles the click-around polish on the console admin UX.
Home composer + schedule modals
- /v1/api/models now exposes coordinator_default_alias + judge_default_alias,
resolved through the same chain console/session_factory.py uses. Both the
home composer's MODEL / JUDGE MODEL placeholders and the schedule create /
edit modal model placeholders rewrite to "Default — alias (model)" once
the API responds. The `models_changed` SSE refresh keeps placeholders
current as operators edit per-role assignments.
- Composer.setOptionPlaceholder added so callers can update just the first
option's text without disturbing the rest of the choice list.
Admin → Models → Roles
- Channel adapter row added (channels.default_model_alias) — the migration
to the Roles sub-tab missed it. Key added to
_MODEL_AFFECTING_SETTING_KEYS so edits fire the SSE refresh, and to the
settings-tab roleKeys skip-list so it only renders in one place.
- Plan/Task agent rows now display "(inherit)" instead of the misleading
"(default — <alias>)" — those roles cascade through plan_model →
agent_model → session model, not a single concrete default.
- coordinator.reasoning_effort accepts "" (inherit), matching
model.plan_effort / model.task_effort.
- Blank options in each role's MODEL select now match the "alias (model)"
shape used by the other rows.
Toggle-switch component
- New .toggle-switch component (visually-hidden native checkbox + styled
track + label). 40×22 hit target meets WCAG 2.5.5 (AAA), inset ring on
the off state for ≥1.5:1 contrast against the modal surface.
- .toggle-stack groups toggles in a column with .toggle-group-divider for
conceptual grouping (used in the Add Model modal between "Active" and the
paired Reasoning toggles).
- .toggle--flush modifier zeroes the default top margin for toggles that
sit flush against a heading or a dynamically-rendered row.
Sweep — every admin-modal boolean checkbox is now a toggle:
schedule (cs/es-autoapprove, es-enabled), policy (ep/epp-enabled),
tool-mode (ctm/etm-default), skill (csk/esk-auto-approve, csk/esk-enabled),
MCP (mcp-auto-approve, mcp-enabled), Add Model (Active, surface-persisted-
reasoning, replay-reasoning), judge bool settings (cancel_on_approval et
al.), and the user-roles-modal role assignment list. The two
ogp-cred / eogp-cred inline credential checkboxes stay as compact inline
boxes since they sit beside text inputs in tight horizontal rows.
Add Model modal — the "Enabled" toggle promoted to "Active" and moved to
the very top of the form. Tooltip explains it gates dropdown visibility
without removing the definition.
MCP authorization — the three radio buttons replaced with a vertical
.segmented-control option list. Selected row paints --accent-dim plus a
filled .segmented-indicator; focus ring uses --accent so it stays visible
on the currently-selected option.
Role permissions modal — the 19 permission checkboxes are now
.toggle-switch.perm-toggle (monospace lowercase identifiers preserved).
The permissions are split into Scopes / Admin / Workstreams & Tools
sections under caps-styled section headers so the row-flow grid no longer
slices `admin.*` mid-column.
Judge bool toggles use a static "Enabled" caption rather than flipping
text on `.checked`; flipping lagged 50–300 ms behind the slider position
because the caption was sourced from the post-save reload.
CSS cleanup — dead `.admin-checkbox` / `.perm-checkbox` rules removed.
Specificity audit (scripts/css_specificity_audit.py) returns no conflicts
on any new component class.
Tests — 525 pass on the affected slices; new tests/test_console_available_
models.py pins each branch of the resolution chain in /v1/api/models so the
home composer placeholder stays correct as precedence rules evolve.
`IntentJudge.__init__` previously had a 3-way resolution chain: registered
alias → raw model id pinned onto the session provider → session model. The
middle branch was a footgun documented in `console/session_factory.py:130-137`
— pinning the literal `judge.model` string onto the coordinator's session
provider silently broke every verdict whenever that provider didn't recognise
the model id (e.g. coordinator on Anthropic, `judge.model = "gpt-5-mini"` →
uniform `llm_fallback`).
Tightens to alias-only, matching `coordinator.model_alias` /
`model.plan_alias` / `model.task_alias`. An unknown `config.model` now logs
a warning and inherits the session model — same path as empty. Help text on
`judge.model` updated to clarify the contract.
Adds two regression tests in `TestModelAliasResolution` covering the
session-model inheritance for unknown values and the empty-model self-
consistency case.
GoogleProvider attaches raw tool_call dicts as ``provider_blocks`` on
the finish chunk for ``thought_signature`` round-trip
(``_google.py:_iter_stream``). When the same turn streamed Gemini's
``reasoning_content`` as ``reasoning_delta`` chunks, the prior
synthesizer bailed out the moment ``provider_blocks`` was non-empty
— so the captured reasoning was visible live but lost on page reload.
Replace the early-return-if-non-empty check with a reasoning-bearing
type test (``thinking`` / ``redacted_thinking`` / ``reasoning`` /
``reasoning_text``). When none of those types appear, append the
synthetic ``reasoning_text`` block to the existing list rather than
replacing it — preserving Google's tool-call fidelity blocks.
Also addresses two doc-accuracy review findings:
- ``LLMProvider.extract_reasoning_text`` docstring no longer claims
OpenAI Chat / Responses are unwired (Phase 3+4 shipped extractors).
- Add the method to the Protocol methods table in
``docs/architecture.md`` (was missing alongside the class diagram).
The earlier all-or-nothing shape check on ``_provider_content`` discarded
every valid Anthropic block in a message the moment a single foreign
block (OpenAI ``reasoning``, Gemini thought parts, the synthetic
``reasoning_text`` from path-3 capture) appeared. In the cross-model
resumption edge case that meant ``server_tool_use`` /
``web_search_tool_result`` blocks lost their ``encrypted_content``
silently, breaking web-search round-trip continuity on subsequent turns.
Replaced with a per-block walk: foreign blocks are dropped individually,
valid blocks ride the verbatim path, and an identity-preserving fast
path reuses the source list reference when nothing was filtered or
stripped (pinned by the ``is`` assertions in test_providers.py).
Also addresses validation-pass review findings:
- Document the single-tier vs three-tier ``surface_persisted_reasoning``
resolution divergence between server.py:_build_history and
session_routes.make_history_handler.
- Document why OpenAIResponsesProvider._convert_messages defaults
``replay_reasoning_to_model=False`` while Anthropic's defaults True.
- Document the ``source`` metadata field on synthetic ``reasoning_text``
blocks as reserved-for-future-use, not dead code.
- Add edge tests for non-dict / missing-type-key blocks in
_provider_content (defensive branches in the per-block walk).
CI test job installs `[test]` extras, which omits `anthropic`. The two
TestSessionToWireBoundaryIntegration cases drive the real
AnthropicProvider.create_streaming, which calls _ensure_anthropic() and
raises ImportError. Match the repo convention (test_channel_discord,
test_channel_slack, test_tls_*) by gating the helper with
pytest.importorskip("anthropic").
PR #498 round-robin review surfaced 5 findings. 4 applied; 1 rejected
with rationale.
Applied
* **Copilot finding 5** (history_decoration.py:341): dispatcher
inspected only ``provider_content[0]['type']``. OpenAI Responses
captures EVERY ``output_item.done`` event into ``provider_blocks``
(not just reasoning) — in practice the order is
``[reasoning, message, ...]`` but the API doesn't guarantee that;
a hypothetical ``[message, reasoning]`` ordering would silently
drop the reasoning under an index-only check. Now walks the list
for the first block whose type is in ``_BLOCK_TYPE_PROVIDER_FACTORY``,
then dispatches the WHOLE list to that provider's extractor. Each
provider's extractor already filters internally by its own block
type, so passing the full list is correct. Regression test added
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
* **Copilot finding 3** (migration 052 docstring): the previous
review-fix wave used sed to rename ``persist_reasoning`` →
``surface_persisted_reasoning`` everywhere, which mangled a
historical reference in the migration docstring ("The earlier name
``surface_persisted_reasoning`` was renamed..."). Restored to
point at the actual pre-rename name (``persist_reasoning``).
* **Copilot finding 4** (sdk/typescript/src/events.ts:26):
``HistoryEvent`` JSDoc still referenced ``persist_reasoning`` —
the sed rename only walked ``turnstone/`` and ``tests/``, missing
the TypeScript SDK. Updated to ``surface_persisted_reasoning``.
Also widened the comment to cover all three reasoning-bearing
block types (Anthropic ``thinking``, OpenAI Responses ``reasoning``,
synthetic ``reasoning_text``) instead of mentioning only Anthropic.
* **github-code-quality finding** (session.py:1120): ``_resolve_server_type``
had a bare ``except Exception: pass``. Replaced with a
``log.debug(..., exc_info=True)`` + explanatory comment. Behaviour
unchanged (still returns ``""`` on any lookup failure); failures
are now observable under DEBUG triage.
Rejected (with rationale)
* **github-code-quality finding** (_protocol.py:265):
``extract_reasoning_text``'s body is ``...`` per ``LLMProvider``
Protocol convention. Every method in the file uses ``...`` (PEP
544 idiomatic Protocol style). Changing only this one to
``raise NotImplementedError`` would be inconsistent with the rest
of the file. CodeQL's "statement has no effect" warning is
technically correct for ``...`` as a standalone expression but
ignores the documented Python Protocol convention. No fix.
Docs sync
* docs/api-reference.md: ``history`` SSE event message-shape table
gains the optional ``reasoning`` field.
* docs/architecture.md: ``ModelCapabilities`` row in the type table
gains ``supports_reasoning_replay``; ``StreamChunk`` and
``CompletionResult`` rows gain the existing ``provider_blocks``
field (was missing pre-PR). New "Per-model reasoning persistence"
subsection under the Models config section, documenting the two
flags + capability gate + three reasoning paths + cross-provider
shape filter.
* docs/settings.md: new "Reasoning persistence (per-model)"
subsection with the two-flag table and capability-gate note.
* docs/diagrams/03-core-engine-classes.puml: ``LLMProvider`` interface
adds ``extract_reasoning_text`` + the new ``replay_reasoning_to_model``
kwarg; ``ModelCapabilities`` class adds ``supports_reasoning_replay``.
PNG regenerated.
Lint + test gate
* ruff check + ruff format clean.
* mypy clean (191 source files).
* pytest -m 'not live' — 6116 passed (3 deselected), +1 net new test
(``test_dispatcher_scans_past_unrecognized_first_blocks``).
Multi-stage /review on the full Phase 1+2+3+4 stack surfaced 9 findings
(0 critical, 3 major, 5 minor, 1 nit, 1 uncertain). All applied.
Major
* perf-1 (session_routes.py:2402): make_history_handler ran sync
storage.load_workstream_config inside async def history on the cold-
workstream path, blocking the event loop on every dashboard /history
request for non-resident workstreams. Every other storage call in
the same handler correctly used asyncio.to_thread. Wrap the sync
call in asyncio.to_thread (preserving the existing try/except so a
DB failure still degrades to the conservative-default branch instead
of bubbling out).
* q-2 (test_reasoning_audit_log_discipline.py): the security-sensitive
test (reasoning text never lands at INFO+ severity) only covered the
4 Phase 1 surfaces. Phase 2 added the strip predicate in
AnthropicProvider._convert_messages and Phase 3 added 3 more code
paths that touch reasoning text — none guarded. Added 4 parallel
tests using the existing capture-and-walk infrastructure:
OpenAIResponsesProvider.extract_reasoning_text,
OpenAIChatCompletionsProvider.extract_reasoning_text,
ChatSession._stream_response (drives the synth-block stamp via a
fake reasoning-emitting stream), AnthropicProvider._convert_messages
with replay_reasoning_to_model=False (drives the Phase 2 strip
predicate).
* q-1 (model_registry.py:42): the persist_reasoning flag name implied
storage-control but actually gates UI rehydration only — operators
flipping it could reasonably expect "stop persisting reasoning" but
storage of reasoning bytes happens in provider_data regardless.
Renamed everywhere to surface_persisted_reasoning: ModelConfig
field, migration 052 column (renaming in-place since 052 is not yet
on main), schema, MODEL_DEFINITION_MUTABLE allowlist, _postgresql.py
+ _sqlite.py CRUD impls, _protocol.py create_model_definition
signature, 3 console_schemas Pydantic models, console/server.py
admin POST + PUT, model_registry row mapper, history_decoration.py
helper parameter, server.py _build_history local var,
session_routes.py make_history_handler local var, sdk/events.py
HistoryEvent docstring, admin.js form id + override pill label,
index.html form input id + UI label + tooltip, coordinator.js (none
needed), and every test that referenced the old field name. The
admin tooltip now reads "Storage of reasoning bytes is unaffected
by this flag — they ride in provider_data regardless" so the
decoupling stays explicit at the operator surface.
Minor
* bug-1 (history_decoration.py:336): dispatcher discriminated on
provider_content[0]["type"] only. Anthropic's redacted_thinking
blocks (sealed by the safety system) can appear before, after, or
interleaved with regular thinking blocks per the API docs. When a
redacted block lands first, the dispatcher returned "" and the UI
silently lost the surrounding thinking text. Registered
"redacted_thinking" as a second key in _BLOCK_TYPE_PROVIDER_FACTORY
pointing at the same AnthropicProvider factory — the existing
extractor's type=="thinking" filter already correctly skips redacted
blocks while walking the full list. Regression test added.
* q-3 (_protocol.py:155): replay_reasoning_to_model defaults split
across 9 sites — operator-side defaults to False (matches DB
server_default), provider-API defaults to True (back-compat with
direct callers). Original "pick False everywhere" fix would have
silently flipped behaviour for any direct provider caller. Instead
documented the intentional bifurcation in the Protocol's
create_streaming docstring.
* q-4+q-5 (_protocol.py:107 + 3 providers): MAX_REASONING_DISPLAY_BYTES
was enforced via Python str slicing which counts code points, not
UTF-8 bytes — 4-byte CJK/emoji glyphs would blow past the byte
ceiling. Renamed to MAX_REASONING_DISPLAY_CHARS to match actual
behaviour. Hoisted the 4-line truncation pattern into a shared
_join_reasoning_with_cap helper in _protocol.py; each provider's
extractor becomes a single line at the tail.
* q-6 (tests/_session_helpers.py): _NullUI + _make_session were
duplicated verbatim between test_session_replay_reasoning.py and
test_session_synth_reasoning_block.py. Hoisted to a shared
tests/_session_helpers.py module (importable, leading underscore so
pytest doesn't try to collect it). test_model_registry.py's
_make_session has a different signature (registry/model_alias args
+ _FakeUI) and is not a candidate for sharing.
Nit
* q-7 (history_decoration.py:286): _make_provider_factory used a
dict-as-cell workaround for closure read-only scope. Replaced with
the more idiomatic nonlocal pattern.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6115 passed (3 deselected). Net +5 tests
(4 audit-log discipline + 1 redacted_thinking dispatcher).
Refinements vs the dedupe output (caught during sanity rendering
the report)
* perf-1 fix preserved the try/except wrapper. The original "wrap in
to_thread" one-liner would have let an OperationalError bubble out
instead of degrading to the fallback branch.
* q-3 fix explicitly documented the bifurcation rather than
collapsing both sides to False. "Pick False everywhere" would
silently flip back-compat behaviour for direct provider callers.
* q-1 fix included the admin.js:5292 fallback site
(m.persist_reasoning !== false) that the original threaded-change
list missed.
* q-6 fix verified the third _make_session in test_model_registry.py
is structurally different (different signature + different UI
helper) and intentionally NOT a dedupe target.
Wire reasoning capture and (where the API supports it) replay for the
two remaining provider paths. Phase 3 was originally scoped as
"OpenAI Responses + Gemini" but a spike against the OpenAI SDK source
revealed that Gemini routes through the OpenAI-compatible endpoint
(``/v1beta/openai/``), which is structurally identical to vLLM /
llama.cpp / any other Chat-Completions-shaped local model. Phase 3
and Phase 4 collapse into one feature with two distinct sub-paths:
* **Path 2 (OpenAI Responses)** — full capture+replay. ``include=
["reasoning.encrypted_content"]`` on the request makes the API
surface ``encrypted_content`` on reasoning items in
``provider_blocks``; ``_convert_messages`` round-trips them as
``ResponseReasoningItemParam`` input items on subsequent turns.
Verified against the OpenAI Python SDK 2.33.0 source
(``response_reasoning_item.py:31-62``,
``response_reasoning_item_param.py:33-37``,
``response_create_params.py:70-74``). Even with ``store=False``,
``encrypted_content`` round-trips correctly per the SDK's own
documentation.
* **Path 3 (Chat Completions / vLLM / llama.cpp / Gemini-compat)** —
persist-only. Canonical OpenAI Chat Completions has no reasoning
field on the wire, but several local-model servers tack on
``delta.reasoning_content`` as Pydantic extras. ``ChatSession.
_maybe_synth_reasoning_block`` stamps a synthetic ``{type:
"reasoning_text", text, source?}`` block onto ``_provider_content``
at end-of-stream when no native ``provider_blocks`` were emitted but
``reasoning_parts`` accumulated text. The ``source`` field carries
``server_compat.server_type`` (vllm, llama.cpp, sglang, …) for
diagnostic value — informational only, doesn't gate behaviour.
Reasoning text NEVER replays back to the model on this path; it
rides ``_provider_content`` only for ``/history`` UI rehydration
and gets stripped from the wire by the existing
``sanitize_messages`` underscore-prefix strip on every request.
What this change does
* ``ModelCapabilities.supports_reasoning_replay: bool = False`` added
to the dataclass. Set True on every OpenAI reasoning model
(gpt-5* + o-series via the Responses API) and every Anthropic
Claude entry (default + 6 model-specific). Path-2 wire-build does
``replay_active = bool(replay_reasoning_to_model and caps.supports_
reasoning_replay)`` so an operator who flips the flag on a
non-reasoning model (gpt-4o via Responses) silently no-ops rather
than emit a malformed ``include=`` request.
* ``OpenAIResponsesProvider`` gains:
- ``_build_kwargs`` accepts ``replay_reasoning_to_model: bool``
(threaded from ``create_streaming``/``create_completion``);
adds ``include=["reasoning.encrypted_content"]`` when active.
- ``_convert_messages`` accepts the same flag, captures
``_provider_content`` reasoning items pre-sanitization, and
emits them as input items immediately before the assistant
message they belong to. Position is tracked by ASSISTANT
ORDINAL (not raw index) — ``sanitize_messages`` drops orphan
tool results and inserts synthesized error tool messages, but
NEVER drops or duplicates assistant messages, so the n-th
assistant in the original list is invariably the n-th in the
sanitized list. Index-based lookup would have silently
misrouted reasoning attachments after any tool-message repair.
- ``extract_reasoning_text`` walks ``type=="reasoning"`` items and
returns ``summary[*].text`` + ``content[*].text`` concatenation.
- ``_reasoning_item_for_input`` projects a stored item into
``ResponseReasoningItemParam`` shape (drops server-only
``status``). Returns ``None`` when ``id`` is missing or non-
string per the SDK ``Required[str]`` schema; caller skips
appending, preventing malformed input items from reaching the API.
* ``OpenAIChatCompletionsProvider`` gains:
- ``extract_reasoning_text`` walks synthetic
``type=="reasoning_text"`` blocks and returns the concatenated
text directly (no underlying provider semantics — the synth
block IS the surface).
* ``ChatSession`` gains:
- ``_resolve_server_type(alias)`` reads ``server_compat.server_type``
from the active model's capabilities dict.
- ``_maybe_synth_reasoning_block(provider_blocks, reasoning_parts)``
creates the synthetic ``reasoning_text`` block when no native
blocks were emitted but reasoning was captured. Wired at the
end of ``_stream_response`` immediately before the
``_provider_content`` stamp.
* ``history_decoration.py`` dispatcher collapses three near-identical
lazy-init singleton getters (one per recognised block type) into a
single ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict + helper. Adding a
fourth provider becomes a one-line dict entry.
* Constants hoist: ``MAX_REASONING_DISPLAY_BYTES = 64 * 1024`` moved
from three sibling provider modules into ``_protocol.py`` so a
tuning change propagates uniformly to every provider's display path.
Cross-provider safety
The synthetic ``reasoning_text`` block type is intentionally NOT in
``ANTHROPIC_VALID_BLOCK_TYPES`` (Phase 2 constant). Cross-model
resumption (operator switches from a local model to Anthropic mid-
workstream) falls through Phase 2's shape filter cleanly to the
text+tool_calls rebuild path rather than reaching Anthropic with a
malformed block. Pinned by ``test_synthetic_block_falls_through_
anthropic_shape_filter``.
Same protection applies in reverse: OpenAI Responses
``type=="reasoning"`` items reaching Anthropic mid-workstream fail
the shape filter and rebuild from text+tool_calls.
Tests (49 net new tests)
* ``tests/test_provider_openai_responses_reasoning.py`` (21 tests):
- Extractor unit tests: empty/none/no-reasoning/single/mixed/
truncation/malformed/non-list (8).
- ``_reasoning_item_for_input`` projection (4 tests including the
new None-on-missing-id guard).
- ``_build_kwargs`` include= gating: flag+capability/flag-false/
capability-false/default-omits (4).
- ``_convert_messages`` reasoning round-trip: emit-before-assistant/
drop-on-replay-false/foreign-shape-skipped/default-replay-false (5).
* ``tests/test_session_synth_reasoning_block.py`` (23 tests):
- ``_maybe_synth_reasoning_block`` direct unit tests (6).
- Cross-provider safety regression — synthetic block falls through
Anthropic shape filter (2).
- ``OpenAIChatCompletionsProvider.extract_reasoning_text`` for the
new synthetic block type (6).
- ``_resolve_server_type`` direct unit tests (5).
- ``_stream_response`` integration tests driving fake reasoning-
emitting streams through the actual session method (3 tests
— added in response to a code-review finding that pinned the
wire-up at session.py needs an integration test).
* ``tests/test_session_replay_reasoning.py`` extended with 4
``TestSessionToOpenAIResponsesBoundaryIntegration`` tests driving
``session._try_stream`` -> real ``OpenAIResponsesProvider`` ->
captured ``client.responses.create`` SDK boundary call. Negative-
tested: temporarily reverting the ``include=`` step in
``_build_kwargs`` makes ``test_replay_true_adds_include_to_
responses_request`` fail; restoring makes it pass.
* ``tests/test_history_decoration.py`` extended with the new
``reasoning_text`` dispatcher branch test, and the Phase 1 stub
test for the OpenAI Responses dispatcher branch was tightened
(it now asserts real text extraction instead of the empty-string
stub).
* ``tests/test_provider_anthropic_reasoning.py`` had its Phase 1
``OpenAIResponses returns "" for reasoning blocks`` stub test
retitled and updated to assert the real Phase 3 behaviour.
Code-review pass
Multi-stage ``/review`` pipeline (4 finders + verify + dedupe) ran
on this diff. 6 findings (1 major, 3 minor, 2 nit), 0 critical, 0
security, 0 performance. All applied:
* MAJOR (bug-1+bug-4+q-1): ``_convert_messages`` enumerate-index
lookup was unsound under ``sanitize_messages`` length changes.
Fixed by switching to assistant-ordinal-keyed lookup.
* MINOR (q-2+q-3): ``_MAX_REASONING_DISPLAY_BYTES`` duplicated
across three provider modules + declared after first use.
Fixed by hoisting to ``_protocol.py``.
* MINOR (q-4): three near-identical singleton getters in dispatcher.
Fixed by collapsing to ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict.
* MINOR (q-5): ``_maybe_synth_reasoning_block`` wire-up not pinned
by integration test. Fixed by adding three
``TestStreamResponseSynthBlockIntegration`` tests.
* NIT (bug-2): ``_reasoning_item_for_input`` fell back to ``id=""``;
fixed to return ``None`` on missing/non-string id.
* NIT (q-6): four naming variants for the same concept; renamed
``_convert_messages`` kwarg to match the operator-flag name.
* REFUTED (bug-3): SDK distinguishes summary vs content as separate
fields; no double-counting concern.
Briefing departures
The briefing's Phase 3 plan grouped Gemini with OpenAI Responses on
the assumption that Gemini reasoning had its own native shape (like
Anthropic's ``thinking``). The spike confirmed Gemini-via-OpenAI-
compat is path-3 (Chat Completions shape, no native reasoning
items). Phase 3+4 merger handles Gemini for free via the synthetic
``reasoning_text`` block — same mechanism used for vLLM and
llama.cpp. Whether Gemini's specific endpoint actually emits
``reasoning_content`` deltas is server-dependent and not yet
empirically verified; capture is best-effort (server-emission-driven,
no flag gate).
The briefing's Phase 4 plan stamped reasoning as Anthropic-shaped
``thinking`` blocks ``{type: "thinking", thinking: <text>}``. This
PR uses a distinct ``{type: "reasoning_text", text, source?}`` shape
to avoid a cross-model resumption hazard the briefing missed: an
unsigned synthetic Anthropic-shape block reaching Anthropic's wire
would 400 the API. The distinct shape falls through Phase 2's shape
filter cleanly without needing signature validation in the filter.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6110 passed (3 deselected). Phase 3+4
added 49 net new tests.
Make ``replay_reasoning_to_model=False`` actually suppress prior-turn
thinking blocks on the Anthropic wire (Phase 1 stored the operator
flag but the wire path always re-sent ``_provider_content``
verbatim). As a side benefit, close a pre-existing latent bug where
foreign-shaped ``_provider_content`` (e.g. an OpenAI Responses
``type="reasoning"`` block reaching Anthropic on a mid-workstream
model switch, post-Phase-3) would have 400'd the API.
Why now: Phase 1 shipped the operator knob and UI rehydration but
the wire payload still always carried thinking blocks for
Anthropic-with-thinking turns. Operators flipping replay=False saw
no behaviour change on the actual API call -- the flag only affected
``/history`` rendering. Phase 2 closes that gap.
What this change does
* ``ANTHROPIC_VALID_BLOCK_TYPES`` (frozenset of 8 block types
Anthropic's input boundary accepts) and
``ANTHROPIC_REASONING_BLOCK_TYPES`` (the strip subset) added at
the top of ``_anthropic.py``. The strip set is intentionally
narrow: ``{"thinking", "redacted_thinking"}`` -- ``tool_use`` /
``server_tool_use`` / ``web_search_tool_result`` (which carry
web-search ``encrypted_content``) MUST survive for round-trip
continuity, and a regression test pins this.
* ``_convert_messages`` signature gains
``replay_reasoning_to_model: bool = True`` (back-compat default
-- production call sites pass the resolved value explicitly).
The verbatim ``_provider_content`` replay path is now wrapped by
a shape-validity check using ``ANTHROPIC_VALID_BLOCK_TYPES``;
foreign-shaped payloads fall through to the existing text+
tool_calls rebuild path rather than reaching the API. When
shape is valid AND replay=False, a list comprehension drops
thinking blocks from ``wire_blocks`` while preserving
tool_use / web_search blocks. When all blocks are stripped
(message had only thinking, no text or tool_calls), the message
also falls through to the rebuild path -- which silently skips
if both content and tool_calls are empty (correct: stripped
reasoning has nothing to replay).
* Orphan-tool detection still walks the ORIGINAL ``provider_content``
(not ``wire_blocks``) so the strip cannot accidentally lose the
source-of-truth tool_use IDs. The implementation comment pins
this invariant.
* Protocol surface grows the kwarg on both ``create_streaming`` and
``create_completion``. ``OpenAIChatCompletionsProvider``,
``OpenAIResponsesProvider``, and ``GoogleProvider`` (via
inheritance) accept the kwarg and ignore it -- they have no
first-class reasoning shape on the wire today. Phase 3 will use
it on the OpenAI Responses adapter to gate
``include=["reasoning.encrypted_content"]``.
* ``ChatSession._resolve_replay_reasoning_to_model(alias)`` reads
``ModelConfig.replay_reasoning_to_model`` from the registry,
defaulting to ``False`` on lookup failure (the conservative
miss-fallback: replaying reasoning text against an unknown
operator preference is worse than missing the strip). Threaded
into the three production call sites:
``ChatSession._try_stream`` (streaming), ``_utility_completion``
(title gen / compaction / extraction), and the agent provider
call site (plan / task agents).
Token calibration deferred to Phase 4
The briefing's optional Phase 2 step (extending ``_msg_text_chars``
to count ``_provider_content`` bytes that survive the strip)
required either invasive flag-threading through every call site
of the static method or a lossy approximation that picked the wrong
direction for the default case. Per the briefing's ``pick a
phase'' guidance, this is bumped to Phase 4. The pre-existing
silent under-count on Anthropic-thinking turns persists when
replay=True. Strip-when-False naturally fixes the under-count by
keeping the bytes off the wire entirely; the residual case is the
opt-in replay path.
Tests (28 new, all driving through real boundary objects)
* ``tests/test_provider_anthropic_replay.py`` (19 tests):
- Strip vs preserve under both flag values (3 tests including
redacted_thinking).
- Default-kwarg back-compat preserves verbatim replay (1 test).
- Web-search tool_use + server_tool_use + web_search_tool_result
survive strip with encrypted_content intact (2 tests, edge 14).
- Orphan-tool synthesis after strip -- pins the
``provider_content`` source-of-truth read at lines 397-433
(1 test).
- Foreign-shape fallthrough: OpenAI ``type="reasoning"`` block
rebuilds via text+tool_calls (1 test).
- Mixed-shape fallthrough: even one foreign block forces
rebuild (1 test).
- Empty / None / non-list ``_provider_content`` fallthrough
(3 tests).
- Legacy Anthropic-thinking row pre-Phase-2 stays in verbatim
path -- no regression on existing conversations (2 tests).
- All-blocks-stripped fallthrough behaviour: rebuild from text
if available, silently skip if not (2 tests).
- Constants pinning: strip set is narrow, valid set includes
web search, strip is subset of valid (3 tests).
* ``tests/test_session_replay_reasoning.py`` (12 tests):
- Resolver: 6 tests covering miss / default / set / explicit /
fallback alias / exception.
- Streaming call site: 3 tests pinning the kwarg propagates
through ``_try_stream`` to a stub provider.
- Non-streaming call site: 1 test pinning
``_utility_completion`` propagates the flag.
- End-to-end boundary integration: 2 tests driving
``_try_stream`` -> real ``AnthropicProvider`` -> captured
Anthropic SDK ``client.messages.stream`` boundary, asserting
on the ACTUAL wire payload shape. Negative-tested:
temporarily reverting the kwarg-thread at
``_anthropic.py:create_streaming`` makes the wire test fail
with ``Strip predicate did not fire at wire boundary``;
restoring makes it pass.
The boundary integration tests were added in response to a code
review finding that the bare-stub call-site tests would not catch
a regression where the provider stops reading the kwarg or
``_convert_messages`` silently drops the strip. The integration
tests close that gap by inspecting what reaches the (mocked) SDK,
not just what the provider was called with.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6061 passed (3 deselected). Phase 2
added 28 net new tests.
Surface stored Anthropic thinking blocks on /history responses so
refreshing the page rehydrates the reasoning bubble. Wire payloads
unchanged. Per-model operator knobs added to model_definitions for
both UI rehydration and (Phase 2) wire-build replay.
Why now: reasoning is already round-tripped via _provider_content for
Anthropic-with-thinking turns, but never surfaces on the history wire,
so a tab reload showed only the final answer with no rationale.
Operators also have no per-model lever to opt out of UI display or to
opt in to replay-to-model on subsequent calls.
What this change does
* Migration 052 adds two boolean columns to model_definitions:
persist_reasoning (default 1) controls UI rehydration; replay_
reasoning_to_model (default 0) reserved for Phase 2's wire-build
shape filter. Mirrors the enabled column pattern (NOT NULL +
integer server_default).
* LLMProvider Protocol gains extract_reasoning_text(provider_blocks)
with concrete impls on AnthropicProvider (walks type=='thinking'
blocks, joins with newline, caps at 64 KiB) and no-op stubs on
OpenAIChatCompletionsProvider + OpenAIResponsesProvider. Google
inherits the no-op via OpenAIChat. Phase 3 will wire the OpenAI
Responses extractor once include=['reasoning.encrypted_content']
is requested.
* turnstone.core.history_decoration gains a structural dispatcher
extract_reasoning_text_from_provider_content keyed off the first
block's type field (Anthropic 'thinking' / OpenAI Responses
'reasoning' / Gemini 'thought' are non-overlapping by API design).
Both history surfaces use it: _build_history calls the dispatcher
directly (the SSE-replay path builds entry dicts from scratch),
and the lifted make_history_handler runs the list-helper variant
in the existing to_thread block.
* make_history_handler resolves persist_reasoning via three tiers:
live session -> workstream_config.model_alias (the same key
SessionManager uses to rehydrate the original model after process
restart) -> conservative True default. Operator flag-flip takes
effect uniformly on both warm and cold workstreams.
* Frontend: app.js replayHistory and coordinator.js role==='assistant'
branch each call the existing reasoning-bubble construction (for
app.js, the document.createElement pattern from the live SSE
handler; for coord, the appendMsg('reasoning') helper) when
msg.reasoning is non-empty. Reasoning bubbles render before the
content bubble, matching live SSE order.
* Admin UI: two checkboxes ('Persist reasoning', 'Replay reasoning
to model') in the model edit modal, plus override-pill display in
the model row when set to non-default values.
What is intentionally out of scope
* Phase 2 -- ANTHROPIC_VALID_BLOCK_TYPES shape filter at
_anthropic.py:312-316, _convert_messages replay_reasoning_to_model
parameter, thinking-strip branch, _msg_text_chars token-calibration
extension. The replay flag is stored but not consumed on the wire.
* Phase 3 -- OpenAI Responses include=['reasoning.encrypted_content'],
Gemini include_thoughts spike, ModelCapabilities.supports_
reasoning_replay.
* Phase 4 -- Local-model / chat-template reasoning persistence
(session.py:3486 reasoning_parts accumulator).
Tests
* AnthropicProvider.extract_reasoning_text -- 13 unit tests covering
None / empty / mixed / multi-block / cap / malformed / non-list
inputs plus other-provider no-op verification (real provider
instances, no mocks).
* extract_reasoning_for_history -- 10 dispatcher tests including
block-type discriminator routing (thinking vs reasoning vs
unknown), strip-when-flag-false, empty / non-dict guards, and
cross-role isolation.
* _build_history -- 6 boundary tests through the real Anthropic
extractor with stub sessions, including the registry-lookup
failure default-True branch.
* make_history_handler -- 5 round-trip tests through real storage:
the storage layer's reconstruct_messages decodes provider_data
into _provider_content, and the helper extracts through the real
AnthropicProvider. Includes the live-session flag honoring path,
the cold-workstream workstream_config lookup path, and the
no-alias default-True fallback path.
* Audit-log discipline -- 4 structural mock-and-assert tests that
capture every Logger.info / warning / error call across the
pipeline (extractor, dispatcher, list-helper, _build_history)
and assert no captured payload contains a marker reasoning string.
* model_definitions storage -- 6 round-trip tests: default flags,
explicit create with both flags, individual update of each flag,
and list-includes-flags assertion.
* model_registry -- 4 tests: dataclass defaults, dataclass with
explicit flags, DB-row-mapping with both flags, and pre-052
legacy-row default-fallback.
Edge cases pinned by the test suite
* Pre-052 DB rows missing the new columns degrade to dataclass
defaults (test_db_reasoning_flags_default_when_absent).
* Live session in memory has its flag honored (test_history_handler_
with_persist_flag_false_via_live_session).
* Cold workstream resolves the flag via workstream_config +
app.state.registry (test_history_handler_cold_workstream_resolves_
via_workstream_config) -- this closes the gap where a process
restart would have silently un-honored an operator flag-flip.
* Cold workstream without persisted model_alias falls through to
default True (test_history_handler_cold_workstream_no_alias_
defaults_true).
* Foreign / unknown / missing block types degrade silently to no
reasoning field rather than misroute or crash.
Lint + test gate
* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6030 passed (3 deselected).
Doc-debt cleanup flagged by /review on 9dc29db7. The cap+seq fix
flipped the seq-advance rule but left two doc sites describing the
old "incremented only on actual append" shape — exactly the buggy
invariant the previous commit removed. Future readers trusting the
stale docs would be one wrong assumption away from re-introducing
the silent-drop bug.
Updates the field-init comment block and the docstring on
register_listener_with_in_progress_snapshot (which sits at the
snap_seq capture site, so its contract is consumer-facing).
Also drops the now-dead `seq: int = 0` initializer in
on_reasoning_token and on_content_token — under the new shape, the
unconditional `seq = self._ws_inflight_seq` inside the lock makes
the initializer unreachable. Was load-bearing under the old
else-branch; harmless now but signals "some path leaves seq at 0"
to a reader.
Copilot caught a real bug in the cap+seq interaction: the previous
shape only advanced ``_ws_inflight_seq`` when the buffer actually
appended, on the theory that "every _seq corresponds to a buffered
fragment" was a useful invariant. It wasn't — once the buffer hit
its cap, seq stalled at the high-water-pre-cap, so a subscriber that
registered AFTER the cap was hit would capture
``snap_seq == stalled_seq``, and every subsequent live token (also
tagged with the stalled seq) would be filter-dropped by the events
handler's ``seq <= snap_seq`` dedup. Silent loss of the entire
post-cap stream for refresh-past-cap tabs.
Fix: advance seq on every emit, regardless of buffer cap. The cap
is a buffer-size limit, not a stop-streaming signal. Past-cap tokens
are absent from the snapshot's text payload (the buffer was
truncated at cap) but the live stream past them is now correctly
delivered — refresh-after-cap renders snapshot-up-to-cap then live
tokens past it, with a visual gap equal to the past-cap chunk and
no silent drop of subsequent tokens.
Test ``test_inflight_seq_increments_only_on_actual_append`` enforced
the buggy invariant and is renamed/flipped to
``test_inflight_seq_advances_on_every_emit_even_at_cap``. Added
``test_subscriber_after_cap_hit_receives_subsequent_tokens`` (and
the reasoning equivalent) as direct regressions for the
silent-token-loss scenario.
Updates the docs that describe the per-workstream SSE event stream and
the SessionUI lifecycle to match the refresh-resume changes:
- api-reference.md: documented the `state_change` event (previously
undocumented despite already being a live event) and the new
`in_progress_snapshot` event; rewrote the multi-consumer fan-out
paragraph to mention the kind-specific replay tail (state_change +
optional in_progress_snapshot) so the "no catch-up needed" claim
is no longer misleading.
- architecture.md: bumped the SessionUI Protocol stub to 16 methods
(added `on_turn_start` / `on_turn_committed`) and pointed at the
in_progress_snapshot section in the API reference.
- sdk.md: added rows for `state_change`, `in_progress_snapshot`, and
`approval_resolved` (preexisting gap) to the per-workstream event
table.
- coordinator-api-tour.md: added an `in_progress_snapshot` row to the
event table and rewrote the reconnection-contract paragraph to
cover mid-stream content/reasoning restoration.
- diagrams/04-conversation-turn.puml: added `on_turn_start()` before
the thinking-start emit and `on_turn_committed()` immediately after
`messages.append(assistant_msg)`, with notes explaining the inflight-
buffer reset semantics. PNG regenerated.
Refreshing a coordinator or interactive workstream pane while the LLM
is mid-stream now restores the partial assistant text + reasoning
immediately and flips the composer back to stop-mode, instead of
showing nothing until the response completes.
Per-turn inflight buffers (`_ws_inflight_content`, `_ws_inflight_reasoning`,
`_ws_inflight_seq`) on `SessionUIBase` are kept separate from the
existing multi-turn `_ws_turn_content` buffer that drives the
dashboard's IDLE-piggyback payload. New `on_turn_start` (top of
send-loop, defensive) and `on_turn_committed` (right after
`messages.append(assistant_msg)`, primary) lifecycle hooks reset
inflight at turn boundaries. The seq counter is monotonic across
turns so a long-lived subscriber's `snap_seq` cutoff stays valid for
the lifetime of the connection — resetting per-turn would silently
drop turn N+1's first M tokens (M = whatever was streamed pre-snapshot
in turn N).
`snapshot_and_consume_state_payload` also drains inflight at idle/error
so cancel and exception paths don't leak stale text. New
`register_listener_with_in_progress_snapshot` atomically registers a
listener and snapshots the inflight buffers; `make_events_handler`
emits a `state_change` event (so the JS busy machine flips to
stop-mode) followed by a one-shot `in_progress_snapshot` after the
kind-specific replay, then strips the internal `_seq` field from
yielded live events while filtering against `snap_seq`. A per-listener
shallow `dict` copy in the live drain prevents the multi-tab race
where one listener's `del event["_seq"]` would corrupt another
listener's filter view.
`_synthesize_cancelled_results` now emits synthetic `on_tool_result`
events for each cancelled tool so live coord tabs can drop the
newly-additive `coord-tool-batch--running` indicator cleanly. The
indicator now coexists with `--auto`/`--approved` (applied on
`tool_info` and `approval_resolved` approved; removed when every row
in the batch has a result), making live tool execution visually
parallel to the replay-time orphan rendering.
Frontend handlers in `app.js` (interactive) and `coordinator.js` (coord)
absorb EventSource auto-reconnect re-replays via a length-based
prefix check on the in-progress buffer. New `InProgressSnapshotEvent`
+ `StateChangeEvent` dataclasses in the Python and TypeScript SDKs
with type guards.
`_MAX_TURN_CONTENT_CHARS` lifted 256 KiB → 512 KiB (single constant
for both buffers — headroom for current commercial models).
Regression tests cover race-free composition under concurrent writers,
seq-filter dedup invariants, the cross-turn seq monotonic invariant,
idle/error inflight drain, synthesized `on_tool_result` on cancel
(including UI-hook failure isolation), and the multi-listener
shared-dict invariant.
Two findings, both confirmed against the source:
1. Migration 051's downgrade rewrote every '[]' row back to '{}',
which would (a) destroy operator-written empty arrays and
(b) reintroduce the known-invalid sentinel that every consumer
rejects. Pre-migration '{}' rows and operator-authored '[]' rows
are indistinguishable after upgrade — there is no clean inverse
for the data state. Made downgrade an explicit no-op with the
rationale documented inline; '[]' is the correct shape under any
consumer's interpretation, so leaving the data untouched on
downgrade is strictly safer than reversing it. Updated the
module docstring to call this out.
2. admin_update_skill's notify_on_complete validator short-circuited
on empty string: `if nc and nc != "[]":` skipped the JSON-parse
branch when nc=="" and persisted the empty string straight to
storage, leaving a non-JSON value behind. Folded the empty case
into the existing "{}" coercion so any blank/whitespace/legacy
value normalises to "[]" before the array-validation gate.
Tests: three new regressions in TestSkillAPI — empty-string
normalises, "{}" sentinel coerces, non-array JSON 400s. The third
locks in the array-only validator that the previous "valid JSON"
gate would have accepted.
Every consumer of prompt_templates.notify_on_complete treats it as a
JSON-array string (the admin form's array editor, the JSON.isArray
validator in submitEditTemplate, _validate_notify_targets in
server.py, the documented "list of channel/contact identifiers"
shape). But the column's server_default — set in migration 011 and
inherited through 021's lift into prompt_templates — has been "{}"
(an empty JSON object) since day one.
Newly-installed remote skills inherit the schema default, so every
unlock-then-edit flow trips the array validator on the inherited
"{}" and the request never leaves the browser. The user-visible
symptom was "click Save, nothing happens"; the latent symptom was
silent shape divergence between every install and every operator-
authored skill.
Migration 051: rewrites every legacy "{}" row to "[]". Operator-
edited values (anything that's neither "{}" nor NULL) are left
intact. Downgrade restores "{}" only on rows still holding the
post-migration "[]" so any later operator edits stick.
Server-side defaults flipped to "[]" in the same PR so new rows
land correct without depending on the column's server_default:
- _schema.py prompt_templates.notify_on_complete server_default
- StorageBackend protocol create_prompt_template kwarg
- sqlite + postgres create_prompt_template kwargs
- console_schemas.py SkillCreateRequest / SkillUpdateRequest /
SkillInfo Pydantic defaults
- core/session.py ChatSession._notify_on_complete initial value
- server.py initial-message worker fallback when skill_data omits
the field
admin_update_skill validator now also rejects non-array JSON (was
"valid JSON" only — would have accepted "{}" or "{\"a\": 1}").
_skill_to_response coerces legacy "{}" rows to "[]" on read so the
admin UI sees a consistent shape even before migration 051 runs.
The frontend's `tmpl.notify_on_complete || "[]"` fallback already
handled empty-string but not "{}" — the read-side coercion makes
it moot.
Designer-review follow-up to the .is-visible sweep. With role=alert
+ aria-live=assertive, AT engines re-announce when the element's
text content changes — but without aria-atomic some engines only
read the diff between old and new content. With aria-atomic=true
the entire updated message is read each time, which matters when a
validation error is replaced by a server error on retry (or
vice-versa).
Added aria-atomic=true to all 24 modal error elements (every
role=alert with aria-live=assertive). Same accessibility uplift
across the board — no per-modal exceptions.
Also dropped the stale `style="display: none"` attribute from the
three MCP error elements (mcp-create-error, mcp-import-error,
mcp-install-error). The CSS rule
.admin-modal [role="alert"] { display: none; }
already hides them by default — the inline attribute was redundant
and would have overridden the .is-visible toggle if the class-based
contract is ever changed.
Sweep of the latent bug PR #494 fixed for the skill modals: the
project's CSS contract for modal errors is
.admin-modal [role="alert"] { display: none; }
.admin-modal [role="alert"].is-visible { display: block; }
…but ~30 sites across governance.js and admin.js were toggling
`style.display = ""` instead of the .is-visible class. The "show"
side broke silently — clearing the inline style fell back to the
CSS `display: none` so the error never rendered, and any
validation failure looked like an unresponsive button.
Mechanical conversion of every show/hide site for these modal
error elements:
governance.js
create-role-error, edit-role-error
create-policy-error, edit-policy-error
github-import-error
cpp-error, epp-error (custom + eval prompt policies)
create-hr-error, edit-hr-error (heuristic rules)
create-ogp-error, edit-ogp-error (output-guard patterns)
admin.js
mcp-create-error, mcp-import-error, mcp-install-error
Plus the global `_showModalError` helper in admin.js — its
`style.display = "block"` happened to work today (inline display
beats the CSS rule), but normalising it to .is-visible keeps every
modal on a single canonical path. The five modals that route their
show side through that helper (create-user, create-token,
create-channel, create-schedule, edit-schedule) had their hide
sides converted in lockstep.
Added a comment on `_showModalError` documenting the contract so
the next contributor doesn't reintroduce the bug.
Out of scope: model-create-error (already canonical), home-coord-error
(not in .admin-modal), edit/create-template-error (fixed in #494).
No CSS or HTML changes; behaviour-equivalent for hide sides; show
sides go from broken-silent-no-render to correct-render-with-AT-
announcement.
Once edit-template-error is actually visible (the visibility fix in
this same PR), a stale error now persists across resubmit cycles:
the user sees a red message, fixes the input, clicks Save, the
validator passes, the PUT goes out — and the previous error stays
on-screen the whole time, only clearing when the modal closes on
success.
Fix at the start of submitEditTemplate / submitCreateTemplate:
clear .is-visible AND empty textContent. Cheaper than tracking
every validator branch and every .catch path; a fresh submit is a
clean slate.
Smoke-testing the unlock flow surfaced a latent bug: clicking Save
on the edit-skill modal silently no-op'd whenever the
notify-on-complete field had non-JSON content. The error div was
DOM-correct (text content set, role=alert, aria-live=assertive),
but invisible — because the project's modal-error CSS contract is:
.admin-modal [role="alert"] { display: none; }
.admin-modal [role="alert"].is-visible { display: block; }
…and the JS in submitEditTemplate / submitCreateTemplate was
clearing the inline `display: none` via `el.style.display = ""`.
That falls back to the CSS rule, which still says `display: none`,
so the error never rendered. The user saw no error and the click
felt unresponsive (compounded by the early-return before the
disabled-state reset, which also made Save look broken).
Fixed both skill-modal flows (create + edit) by toggling the
canonical `.is-visible` class instead. Six sites in governance.js:
the two early-return show paths, the two .catch show paths, and
the two modal-open hide-resets.
Scope note: this same bug pattern exists in ~20 other modal error
sites across governance.js and admin.js (create-role, edit-role,
create-policy, edit-policy, github-import, cpp, epp, create-hr,
edit-hr, create-ogp, edit-ogp, mcp-create, mcp-import, mcp-install,
plus admin.js sites that don't go through _showModalError). All
pre-existing, broken silently for who knows how long. Out of scope
for this PR — recommend a follow-up sweep that also normalises
_showModalError's `style.display = "block"` to the same convention.
Designer review of the cb5fa1b lock-icon iteration flagged five
items; four are addressed here, one was a deliberate trade-off
documented below.
- Glyph hardening (#2): the lock character is now 🔒︎ — U+1F512 with
the U+FE0E text variation selector — paired with the existing
font-variant-emoji: text rule. font-variant-emoji shipped late
and isn't universal yet (Chrome 131+, Safari 16.4+, Firefox 132+);
the explicit text VS is belt-and-braces so older Chromium / most
Linux don't fall back to a coloured emoji that would clash with
the monochrome instrument-panel aesthetic.
- Accent-line de-conflict (#3): top:14px → 18px so the lock button
sits below the modal's ::before accent-line decoration's visual
band rather than competing with it horizontally. h2's
padding-right reservation (44px) still gives the title clearance.
- Mobile touch target (#4): @media (max-width: 700px) bumps the
button to 44×44 (WCAG 2.5.5 / Apple HIG / Material minimum) and
shifts it to top:8px right:8px, with h2 padding-right widened to
56px to match.
- Keyboard discoverability (#6): on readonly open, focus lands on
the lock button instead of Cancel. Keyboard users hit the unlock
affordance immediately instead of having to Tab past every
disabled spec input to reach it. Cancel is one Shift-Tab away.
Deferred:
- (#1) Reviewer flagged top-right placement as risking confusion
with the universal × close-button convention. Keeping the
icon-only design per product direction; the bordered chip styling
+ accent-coloured hover make it visually distinct from the
thin-stroke unbordered × pattern, and the confirm dialog catches
any misclick safely.
- (#5) Optional empty-corner indicator after unlock — the
"Customized from upstream" badge text already carries the signal;
not adding new chrome.
Three issues from manual smoke-testing the unlock flow:
1. Confirm dialog rendered behind the edit-skill modal. Both
overlays sat at z-index 600, and confirm-overlay is earlier in
the DOM than edit-template-overlay — so DOM order put the parent
modal on top of its own confirm. Bumped confirm-overlay to 650
(still below toasts at 700) since confirm dialogs are launched
FROM other overlays and need to sit above them.
2. Save button stayed disabled (or non-functional) after unlock.
submitEditTemplate disables etm-submit on click and re-enables in
.finally, but a stale disabled=true survives the mutate-in-place
re-render that runs after unlock. Always reset
submitBtn.disabled = false in showEditTemplateModal so the
re-render path can never inherit a stuck disabled state.
3. UX redesign — moved the unlock affordance from a "Customize…"
button at the bottom of the footer to a 🔒 icon button at the
top-right of the modal. The lock glyph is the universal "this is
locked, click to unlock" affordance and reads more clearly than
a footer button next to Cancel/Save. font-variant-emoji: text
keeps it monochrome on browsers that support it (instrument-panel
aesthetic) with graceful fallback to coloured emoji elsewhere.
admin-modal-skill h2 reserves padding-right so a long title can
never collide with the absolute-positioned button.
Cleanup: removed the now-unused .modal-secondary and
.modal-buttons-spacer rules; the bottom etm-unlock button + flex
spacer are gone from the modal footer.
Copilot caught that prompt_templates.readonly is an Integer column
(_schema.py: sa.Column("readonly", sa.Integer, nullable=False,
server_default="0")) and create_prompt_template stores it as 1/0,
but unlock_skill in the postgres backend was passing a Python bool
(readonly=False). The sqlite impl already uses 0; this aligns the
two backends and matches the 0/1 idiom used for the sibling flag
columns (is_default, auto_approve, enabled).
The other Copilot findings on this PR (loadGovSkills race, NBSP
double-space, list_skill_versions O(history_size), ignored
set_skill_readonly return value + None re-read) were all closed by
the prior review-feedback commit (eea795d): the snapshot+flip is
now an atomic unlock_skill() that uses SELECT MAX(version)+1
internally, the handler guards both the unlock_skill return and the
post-flip get_prompt_template re-read, the JS chains
showEditTemplateModal off loadGovSkills's promise, and the badge
NBSP matches the sibling pattern.
Code review caught a race + a missing None guard; designer review
caught a window.confirm regression and a button-hierarchy issue.
Backend:
- Race fix (bug-2): replace set_skill_readonly+create_skill_version
with a single atomic unlock_skill(template_id, snapshot, changed_by)
-> int|None on the storage protocol (sqlite + postgres). Snapshot
insert + readonly flip happen in one transaction; the next version
number is computed via SELECT MAX(version)+1 inside the txn rather
than len(list)+1 outside, closing the (skill_id, version)
collision window where two concurrent admin actions could both pick
the same version.
- None guard (bug-3): check the post-flip get_prompt_template re-read;
return 404 instead of letting _skill_to_response(None) raise.
- Audit body: also record snapshot_version, and harden None-vs-empty
with `or ""` on the existing.get(...) calls.
Frontend:
- D-1: replace window.confirm with the existing showConfirmModal
(admin.js:2350) — themed dialog, focus-trap, can render the source
URL with consistent typography. The native dialog could collapse
the multi-paragraph copy depending on browser.
- D-2: mutate-in-place on success rather than hide → reload → reopen.
loadGovSkills now returns its fetch promise so unlockSkill can
chain showEditTemplateModal after the cache refresh — no flicker,
no focus bounce, and it kills bug-1 (the reopen was reading stale
_govSkills before loadGovSkills resolved). showEditTemplateModal
is idempotent when already open: it skips the trigger-element
capture and the focus-trap reinstall.
- D-3: button hierarchy. Drop flex:1 from .modal-secondary so the
Save button keeps a stable width whether or not Customize is
rendered; insert a flex-spacer between Customize and Save so the
destructive-ish detach groups left next to Cancel and the primary
action floats right.
- D-4: NBSP normalized to match the existing escape pattern
on the sibling badge line (was an actual NBSP byte).
- D-5: success toast now reads "Skill unlocked — fields are now
editable" so the operator gets a positive affirmation that the
edit affordance is live.
- D-10: aria-describedby="etm-origin-badge" on disabled spec inputs
so screen-reader users get the same "this came from upstream"
context that sighted users see in the cyan badge.
Tests: + test_unlock_skill_versions_after_existing_history seeds an
out-of-order version (3) and asserts unlock picks 4, defending
against the len()-based version computation regressing.
skills.sh / GitHub installs land with readonly=True so admins can only
tune runtime config (model, temperature, etc.); the SKILL.md spec is
locked. In practice, upstream skills aren't always tuned for turnstone,
so locking the spec adds friction without a real safety win — every
edit is audited and version-snapshotted regardless.
This adds an explicit unlock so the boundary stays visible (multi-user
audit trail benefits from a discrete event, vs. silently dropping the
gate). Behaviour:
- POST /v1/api/admin/skills/{id}/unlock — flips readonly=False on a
readonly row. Snapshots the pre-unlock state into skill_versions so
the upstream-pristine version is recoverable from the History tab.
Records skill.unlock audit with {name, source_url, origin}. 400 on
already-unlocked, 404 on missing.
- origin stays "source" after unlock so the UI keeps a "Customized
from upstream" provenance badge — the readonly flag is the gate, the
origin field is the lineage.
- Storage: dedicated set_skill_readonly writer on the protocol +
sqlite + postgres backends. readonly is intentionally absent from
SKILL_MUTABLE so the generic update path can't piggyback on a
provenance flip — the dedicated writer pattern matches what's
already used for set_mcp_oauth_client_secret_ct.
- Frontend: "Customize…" button in the edit modal (visible only when
readonly), with a confirm dialog explaining the upstream-detach.
Once unlocked the existing edit-skill flow handles spec edits with
no other changes. Origin badge updates to show "Customized from"
the upstream URL when a source-origin row is unlocked.
Tests cover: unlock flips readonly + persists, pre-unlock snapshot
written to skill_versions, 400 on already-unlocked, 404 on missing,
post-unlock PUT can edit name/content/description (the readonly gate
no longer fires).
Three issues caught by Copilot on the initial PR:
1. SKILL.md size cap was measured in code points, not UTF-8 bytes.
`len(str)` is a *lower* bound on encoded byte length — multi-byte
chars (emoji, CJK) inflate up to 4×, so a 100k-emoji SKILL.md
(400KB encoded) would slip past the 256KB cap. Switch to
`len(contents.encode("utf-8"))` and surface lone-surrogate failures
as SkillSourceError instead of dropping them silently. New
regression test feeds emoji content.
2. _skills_sh_source_url did not normalize the skill_id, so a sloppy
id from `/api/search` (whitespace, surrounding slashes) would pass
`_split_skills_sh_id`'s charset check (which strips first) and
produce a malformed persisted source_url that broke the
discover-UI dedup contract. Strip the id inside the helper, and
reconstruct the canonical id from validated parts in
download_skill's listing so downstream callers never see the raw
input.
3. The catch-all `except Exception:` around create_prompt_template
relabeled every storage failure (DB connection, disk full,
permission errors) as "conflict", masking operational issues.
Translate IntegrityError → StorageConflictError at the storage
shim (matching the pattern already used for OIDC user
provisioning) in both sqlite and postgres backends, then catch
StorageConflictError specifically in the install handler. Real
conflicts → "conflict" + warning; other exceptions → new
"internal error" reason + log.exception.
Tests: +3 (oversized multibyte SKILL.md, source_url normalization,
storage-layer conflict translation). 226 passing.
The skills.sh install path was failing with 404s because their public
API surface changed: /api/skills/{id} is gone, replaced by
/api/skill/[owner]/[repo]/[skill] (auth-walled) and
/api/download/[owner]/[repo]/[skill] (unauthenticated, returns the
SKILL.md + bundled resources inline as JSON). The error was not
surfacing in logs because admin_skill_install had a silent
`except Exception:` around create_prompt_template that relabeled every
storage failure as "conflict" with no log entry.
- Replace SkillsShClient.resolve_github_url with download_skill that
hits /api/download/{owner}/{repo}/{skill} and returns a SkillPackage
directly. No GitHub round-trip; no rate-limit surface.
- Add _split_skills_sh_id with strict per-segment charset validation
([A-Za-z0-9._-]+) so URL-hostile content can't produce a malformed
request or divergent persisted source_url.
- Use len(contents) instead of len(contents.encode("utf-8",
errors="ignore")) for the SKILL.md size cap — errors='ignore' was
silently dropping invalid units, making the cap bypassable.
- Extract _accept_resource(rel_path, byte_size) gate predicate; share
it between download_skill and the GitHub _find_resource_files helper.
- Have search() derive a deterministic source_url from the skill id
when /api/search omits one (which it currently always does), so the
discover-UI "already installed" check matches what download_skill
persists.
- Add structured logging across admin_skill_install and
admin_skill_discover: a shared _log_install_failure helper for the
four except branches (was four near-duplicate log calls with one
drift), plus per-resource failure tallying — partial-resource
installs now surface failed_resources in the response and audit
record instead of silently committing the skill row with missing
assets.
Tests: 7 new — empty/non-list files, oversized SKILL.md, resource
cap, non-text extension filtering, plus _split_skills_sh_id charset
rejection (whitespace, query chars). Verified end-to-end against
live skills.sh with tavily-search.
Four Copilot findings on c6041c6 — all confirmed valid, all bounded
to authenticated-user prompt-injection scenarios but worth closing
before merge.
Wrapper-detect bypass (string + list branches of
``_apply_reminders_for_provider``):
The round-2 fix used ``content.startswith("<tool_output>\\n")`` to
detect already-wrapped content and skip ``escape_wrapper_tags``. A
tool whose RAW output starts with that prefix (e.g. ``echo
'<tool_output>'``) would match and have its escape skipped, letting
literal ``<tool_output>`` / ``<system-reminder>`` tags reach the model
and impersonate a system envelope. Replace the prefix check with
``extract_advisories_from_tool_envelope(content) is not None`` —
parsing requires the open AND matching close tags AND a structurally
valid envelope, raising the bypass bar significantly.
Mirror fix in the list-content branch so a tool emitting an unmatched
envelope as a text part can't bypass the per-text-part escape.
``_build_history`` legitimate-envelope drop:
The list-content drop path previously removed any text part starting
with ``<tool_output>\\n``. A tool that legitimately outputs a
well-formed envelope (documentation viewer, code analyzer demoing the
wrapper, an echo tool) would have that part silently disappear on
replay. Tighten the drop heuristic to require BOTH ``cleaned_text ==
""`` AND at least one extracted advisory — the structural signature of
the injected ``wrap_tool_result("", advisories)`` carrier we produce
in ``session.py`` for list-typed tool output. A legitimate envelope
has non-empty inner body or no advisory blocks and survives the
projection.
Empty advisory body:
``queue_message`` accepts any non-None text including ``""`` and
whitespace-only strings. ``_classify_advisory`` would return a
``user_interjection`` advisory with empty / whitespace body, which
``replayAdvisoriesAfterTool`` then renders as a featureless empty user
bubble. Filter empty / whitespace-only bodies at classification time
so the wire-shape contract is uniform: no empty advisories ever ride
the wire.
Tests:
* ``test_apply_reminders_escapes_tool_output_starting_with_envelope_prefix``
pins the structural-parser bypass close: a string starting with the
envelope prefix but lacking a close tag still gets escaped.
* ``test_apply_reminders_escapes_list_text_part_with_unmatched_envelope_prefix``
mirrors for the list-content branch.
* ``test_build_history_keeps_legitimate_envelope_text_part_with_body``
pins that legitimate envelope output stays in the projected list.
* ``test_decorate_suppresses_empty_advisory_body`` and
``test_decorate_suppresses_whitespace_only_advisory_body`` pin the
empty-body filter in ``_classify_advisory``.
Tests: 5923 passed, 3 deselected. Lint + format + mypy clean.
Reverses the seam-2-only design from the prior commits on this branch.
Queued user messages arriving DURING a tool batch (Seam 1) splice into
the last tool result's envelope as ``UserInterjection`` advisories via
``wrap_tool_result``. Messages arriving BETWEEN turns (Seam 2) drain
as a single trailing user row via ``_flush_queued_messages`` with
``user_feedback`` (operator text alongside an approval, e.g. "y, use
full path") folded in as a prefix. Cancel/exception drains (Seam 3)
keep the existing ``_flush_queued_messages()`` call unchanged.
Why all three seams:
* Strict-template providers (Mistral, Llama via vLLM with stock chat
templates) reject role-alternation violations. A literal ``user``
row mid-tool-batch breaks ``assistant(tool_calls) → tool → ... →
assistant``; back-to-back ``user → user`` rows on the wire also fail.
* The seam-2-only design produced back-to-back ``user`` whenever
``user_feedback`` and queued items both fired — bug-1 from the round-1
review. Folding ``user_feedback`` as a prefix to the queue-drain
collapses the two into one row.
* During-batch arrivals couldn't ride seam 2 — the splice was the only
way to deliver same-turn without violating role alternation.
Storage symmetry:
Tool DB rows now store the wrapped ``output`` (envelope + advisories)
unconditionally — ``self.messages[i]['content']`` and
``conversations.content`` match exactly. List-typed output (image /
structured MCP results) uses ``wrap_tool_result(raw_joined_text,
advisories)`` at save time so the persisted string is anchored on
``<tool_output>\n`` for the replay parser. ``TOOL_RESULT_STORAGE_CAP``
is removed entirely; tools are responsible for bounding their own
output, storage faithfully represents in-memory. Removing the cap
also simplifies the parser — no truncated-envelope edge case.
Replay extraction:
``decorate_history_messages`` (REST ``/history``) and ``_build_history``
(SSE replay, resume, rewind, retry, post-load, rename re-replay) both
call the public ``extract_advisories_from_tool_envelope`` helper to
pull the envelope back into structured ``advisories`` for JS replay.
Both string content and list-typed content (image+queued-message
combo) covered. JS renders extracted advisories as normal user
bubbles after the tool block via the shared ``replayAdvisoriesAfterTool``
helper in ``shared_static/utils.js``.
Wrapper-tag escape and provider splice:
``escape_wrapper_tags`` now encodes pre-existing ``&`` first using an
``&`` sentinel so tool output containing literal entity strings
(documentation viewers, code analyzers, web scrapers returning entity-
encoded markup) round-trips correctly. Both encode and decode helpers
short-circuit on absence of ``<`` / ``&``.
``_apply_reminders_for_provider`` detects already-wrapped content
(string body and list text-part) by ``startswith("<tool_output>\n")``
and skips re-escape so existing envelopes survive intact when a tool
message also carries ``_reminders`` (the queued-message + tool-error
co-occurrence case is now common).
``decorate_history_messages`` runs in ``asyncio.to_thread`` to keep
MB-scale string work off the event loop.
Other cleanup:
* ``_collect_advisories`` delegates the queue drain to a named helper
``_drain_queued_messages_to_advisories`` so the swap-and-clear pattern
lives next to ``_flush_queued_messages``'s identical pattern and the
side-effect is documented at the call site.
* Preamble strings + body marker for ``UserInterjection`` round-trip
detection moved to module-level constants in ``tool_advisory.py``;
imported by ``history_decoration.py`` so a producer-side rephrase
can't silently desync the parser.
* ``_send_with_mocks`` ctxmgr extracted in ``test_session.py`` — the
six new send-driven tests share an 8-deep ``patch.object`` block.
* ``replayAdvisoriesAfterTool`` shared helper in
``shared_static/utils.js``; ``app.js`` and ``coordinator.js`` both
invoke it.
* Dead truncation-pill CSS removed (``.tool-output-truncated`` and
``.coord-tool-truncated``); the JS that added these elements went
away with ``TOOL_RESULT_STORAGE_CAP``.
* Tautological tests (``TestBuildHistoryAdvisoryPropagation``)
replaced with production-realistic round-trip tests built from
``wrap_tool_result(...)`` envelopes — REST and SSE-replay surfaces
pinned to the same wire shape; full DB round-trip pinned end-to-end.
Negative-tested:
* Reverting the prefix-merge in ``_flush_queued_messages`` produces
back-to-back ``user`` rows, breaking
``test_user_feedback_and_queued_coexistence_single_row_with_prefix``.
* Reverting the ``extract_advisories_from_tool_envelope`` call in
``_build_history``'s tool branch leaves the envelope verbatim in
wire content, breaking the round-trip tests.
* Reverting the wrapper-detection in ``_apply_reminders_for_provider``
entity-encodes the existing envelope's literal tags, breaking both
the string-content and list-content envelope-preservation tests.
* Reverting the ``wrap_tool_result(raw_text, advisories)`` projection
at the DB save site produces a string starting with the original
raw text, breaking
``test_tool_db_row_round_trips_list_output_with_advisories``.
Tests: 5918 passed, 3 deselected. Lint + format + mypy clean on
touched files.
Round-1 ``/review`` apply-pass. Drops stale ``UserInterjection``
references from comments and docstrings that no longer describe the
post-PR drain shape, asserts the two-stream invariant in the new
queued-message persistence test, and pins the ``content.trim()`` +
``renderAssistantToolBatch`` invariants on coord-side so a future
refactor can't silently regress the Qwen3 phantom-card fix or the
chronological-order render fix.
Deferred:
* **bug-1** (back-to-back ``user`` row when ``user_feedback`` from the
approval-prompt UI callback coexists with a queued-message drain).
Reachable on strict OpenAI-compatible local templates (Anthropic and
Anthropic-via-merge-consecutive collapse fine; vLLM-hosted Mistral /
Llama enforcing role alternation can reject). The pre-PR splice
guarded against this case by riding queued items inside the tool
result envelope; that guard is what motivated the original
UserInterjection design, so the fix lane needs a deliberate decision
rather than a quick patch. Sleeping on it.
* **q-1** (delete dead ``UserInterjection`` class + tests). Held for
the bug-1 decision — if the chosen fix is to resume the splice for
the ``user_feedback``+queue coexistence case, the advisory shape
stays load-bearing. Class now carries a docstring note marking it
retained-pending-decision so a passing reader doesn't grep for
producers and assume it's actually dead.
Apply-pass content:
* ``q-2``: drop "queued user interjections" from the persistent-
advisory parenthetical in ``send``'s tool-result loop comment;
rewrite to point at ``_flush_queued_messages`` for the queue path.
* ``q-3``: ``__init__`` channel-routing comment loses "and
``UserInterjection``" — only ``GuardAdvisory`` remains.
* ``q-4``: ``_queue_tool_advisory`` docstring + the tool-error nudge
comment lose the user-interjection mentions; the docstring also now
describes the side-channel + ``_apply_reminders_for_provider``
splice path (the actual mechanism).
* ``q-5``: ``AttachmentsNotQueueableError`` docstring rewritten to
describe the post-PR ``_flush_queued_messages`` flow — the
single-combined-turn ``\n\n``-join shape can't carry image / file
blocks, and per-item separate user turns would expand the strict-
template role-ordering surface that the post-batch drain already
balances.
* ``q-6``: the new ``test_queued_message_persists_as_user_row_after_tool_batch``
in ``test_session.py`` now asserts ``stream_idx == 2`` so a future
regression where the post-batch flush runs but the send-loop short-
circuits before the next iteration surfaces in CI rather than
manual repro.
* ``q-7``: ``test_coordinator_page.py`` gets two new string-grep pins
mirroring the existing ``test_app_js.py`` shape — ``content.trim()``
on coord's assistant-replay branch and ``renderAssistantToolBatch``
for the hoisted helper that orders content card before tool batch.
## Test plan
- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] Affected test surface (``test_session.py`` +
``test_tool_advisory.py`` + ``test_app_js.py`` +
``test_coordinator_page.py``) — 240 passed
Three independent rehydrate / replay regressions reported on long
multi-turn conversations after the pull-model wake stack landed.
**1. coord history replay rendered tool_calls above the assistant
narration that announced them.**
In ``coordinator.js``'s loadHistory loop, the ``role === "assistant"``
``tool_calls`` branch sat above the role switch — every assistant turn
with both narration AND tool dispatch produced ``[tool batch][content
card]`` in the DOM, even though chronological order is content first.
On a parallel fan-out (e.g. four ``close_workstream`` calls in one
turn) operators saw the assistant text "Let me close them out and
summarize" with NO tool batch between it and the next assistant
message — the four-row batch had been rendered above the announcing
text and was scrolled out of view.
Hoisted the ``tool_calls`` synthesis into a local
``renderAssistantToolBatch(m)``, called from inside the assistant
branch AFTER the content card. Live SSE order (text → dispatch →
results) now matches replay order.
**2. Whitespace-only assistant content rendered as a blank card on
replay.**
Models with vLLM's ``--reasoning-parser`` (Qwen3 in production)
strip ``<think>…</think>`` and emit only the trailing ``"\n\n"`` as
``content`` before a tool call. ``content_parts = ["\n\n"]`` saves
``content = "\n\n"`` to the conversations row. Live the user only
sees ``.msg.reasoning`` (the thinking content) — the empty
``.msg.assistant`` card lives next to it but reads as a thin
divider. On rehydrate the reasoning bubble is gone (not persisted)
and the empty assistant card is the only thing left, surfacing as
"blank cards where the assistant message was."
Both UIs now check ``content && content.trim()`` before rendering
the body — whitespace-only content skips the card entirely instead
of showing a phantom row. Live render unchanged.
**3. Queued user messages disappeared on reconnect.**
PR #474 routed queued user messages into the tool-result envelope
via ``UserInterjection`` advisories — same-turn delivery, but no
persisted user row. On page reload / cross-tab replay the
optimistic ``.msg-queued`` bubble vanished: there was no DB row to
rehydrate it.
Dropped the ``UserInterjection`` splice in ``_collect_advisories``;
the queue drains through ``_flush_queued_messages`` AFTER the tool
batch completes instead. Sequence becomes
``assistant(tool_calls) → tool … tool → user(drained)``, which is
valid for Mistral and Anthropic strict role validators (the only
forbidden shape was user injected mid-batch BEFORE the tool result,
which this still avoids). Persists a real user row → bubble survives
reconnect, and stays in the session's wire-side context window on
the next turn.
## Test plan
- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] ``pytest -m "not live"`` — 5798 passed, 3 deselected
- [x] Updated ``test_collect_advisories_does_not_drain_queued_messages``
(was pinning the old UserInterjection shape)
- [x] Added ``test_queued_message_persists_as_user_row_after_tool_batch``
(drives ``send`` end-to-end with a queued message arriving during
the tool batch; asserts the user row lands in self.messages AND
hits ``save_message``)
- [x] Updated ``test_replay_history_renders_content_before_tool_block``
to tolerate the new ``msg.content && msg.content.trim()`` guard
- [ ] Live browser pass on coord (close_workstream parallel fan-out
rehydrates with the 4-row batch BETWEEN the announcing assistant
text and the summary) and interactive (Qwen3 ``"\n\n"`` rows no
longer paint blank cards on reload; queued bubble survives a tab
refresh)
PR #489 review feedback (Copilot + github-code-quality):
- closeSettingsPanel now closes nested revoke modal first on close-button
path (Escape was already handled by the parent keydown trap deferring
to the inner trap; missing-modal-on-close-button was an orphan-modal
hazard).
- _refreshConsentBadge now updates the settings button's aria-label +
title dynamically with the pending-consent count for screen readers
(badge stays aria-hidden — the count is in the label).
- _MAX_INSUFFICIENT_SCOPE_REPORTED promoted to public
MAX_INSUFFICIENT_SCOPE_REPORTED in mcp_http_parsers; drops cross-module
private import in mcp_oauth's /start handler.
- Stale test comment in test_session_mcp_dispatch_error.py corrected:
_exec_read_resource does not log with exc_info=True (bearer-leak
invariant).
- Rejected the protocol-method ellipsis warning: rest of _protocol.py
uses ... consistently per Protocol convention.
Lint:
- ruff format applied to test_mcp_pool_auth_integration.py and
test_mcp_pool_auth_resource_integration.py (combined `with` grammar —
pure formatting).
Flake fix — test_integration_pool_reuse_401_refresh_and_retry_succeeds
on Python 3.11 / resource-constrained CI:
Same cross-task scope hazard f6a3b66 fixed at the close side, surfacing
at the connect side. asyncio.wait_for at mcp_client.py:1206 wraps
streamablehttp_client.__aenter__ in a fresh asyncio.Task. That fresh
task enters anyio cancel scopes, completes, and dies. The eventual
stack.aclose() during eviction or auth_401 retry runs from a different
task and tries to exit scopes whose entering task is dead — anyio
raises RuntimeError, the wedged anyio state blocks the retry's stack
teardown + reconnect, and the call exceeds the 15s budget on slow
workers.
Fix: replace asyncio.wait_for with `async with asyncio.timeout(...)` so
the streamablehttp_client.__aenter__ runs in the dispatch task itself,
no fresh-task scope ownership. Aligns with invariant 18 (asyncio.timeout
not asyncio.wait_for for any SDK / AS / pool-loop await crossing anyio
scopes).
Static path (_connect_one) at lines 905 and 1000 deliberately retains
asyncio.wait_for — auth_type ∈ {none, static} is byte-identical
(invariant 1) and the narrow connect-once / no-eviction-then-reuse
pattern doesn't trigger the cross-task hazard. Anchor comments pin
both directions: a future migration there would break invariant 1; a
future revert at 1206 would re-introduce the flake.
The cited test is the symptom (non-deterministically times out under
load), not a structural gate (no deterministic asyncio.timeout
assertion exists). The comment block at line 1206 records this so a
maintainer who reverts and finds green on a fast machine doesn't
conclude the fix is unneeded.
Verified on Python 3.11.14 (/tmp/venv311) and 3.13.7 (.venv): ruff
format clean, ruff check clean, mypy clean. 368 unit tests + 30 pool
integration tests pass on both interpreters; the previously-flaky test
passed 20× in isolation on 3.11.
Multi-stage /review (4 finders × verify × dedupe): bug/security/perf
returned zero findings; quality returned 3 confirmed minor/nit items
all of which are applied here (q-1 anchor comments at 905+1000, q-2
symptom-vs-gate clarification at 1206, q-3 module-docstring sentence
in mcp_http_parsers).
Wires the structured-error envelopes produced by Phase 7b's pool
dispatcher (mcp_consent_required / mcp_insufficient_scope /
mcp_*_forbidden / mcp_token_undecryptable_key_unknown /
mcp_oauth_url_insecure) through to the user-facing dashboard, and
adds a per-user settings panel for managing MCP server consents.
Changes
- ``_dispatch_pool_sync`` and ``_dispatch_pool_resource_sync`` wrap
structured-error string returns as ``RuntimeError(json_str)`` via
``_is_structured_error()`` so the session-layer ``except Exception``
branch fires uniformly across tool / resource / prompt dispatchers
(the prompt path's ``isinstance(result, str)`` shortcut works only
because prompts return ``list[dict]`` on success). Without this,
the consent UX silently does not render for tool / resource calls.
- ``_structured_error`` extended with an optional ``consent_url``
field; ``_build_consent_url`` produces ``/v1/api/mcp/oauth/start``
query strings (path-relative; the dashboard appends ``return_url``
at click time). Wired to all 12 ``mcp_consent_required`` and the
``mcp_insufficient_scope`` emit sites.
- New endpoints ``GET /v1/api/mcp/oauth/connections`` and
``DELETE /v1/api/mcp/oauth/connections/{server_name}`` registered
on both ``turnstone-server`` and ``turnstone-console``. The DELETE
handler runs local delete + audit + 204 first, then schedules the
RFC 7009 upstream revoke as a fire-and-forget ``asyncio.create_task``
with strong-ref tracking via ``_revoke_upstream_tasks`` (mirrors
the ``_pg_refresh_drain_tasks`` pattern). Soft cap of 256 concurrent
in-flight revokes prevents pile-up under coordinated mass-revoke;
the audit detail records ``upstream_revoke_outcome`` as
``scheduled | no_refresh_token | no_http_client | shed_by_cap``.
- ``ASMetadata`` extended with ``revocation_endpoint`` parsed from
RFC 8414 metadata. ``revoke_token_at_as`` helper posts the form
body under ``asyncio.timeout`` (not ``asyncio.wait_for``) and
never raises; ``_attempt_upstream_revoke`` is wrapped in an outer
``try/except Exception`` so unhandled exceptions don't surface as
``Task exception was never retrieved``.
- ``/v1/api/mcp/oauth/start`` accepts an optional ``scopes=`` query
param; tokens are validated against RFC 6749 §3.3 grammar via
``is_valid_scope_token`` (promoted to ``mcp_http_parsers``),
capped at ``_MAX_INSUFFICIENT_SCOPE_REPORTED`` (32), and unioned
with the configured server scopes for the step-up consent flow.
- Storage primitive ``list_mcp_user_token_metadata_by_user`` projects
the metadata columns at the SQL boundary so ciphertext blobs never
cross the wire on the settings-list path. New
``MCPUserTokenMetadataRow`` TypedDict in ``_protocol.py``;
``MCPTokenStore.list_user_token_metadata`` re-types to the existing
``MCPUserTokenMetadata`` shape.
- Dashboard renderer (``app.js``): ``tryParseMcpError`` detects the
envelope shape on ``tool_result`` SSE events with ``is_error=True``
and ``buildMcpErrorEmbed`` renders an action card mirroring the
existing ``buildMediaEmbed`` pattern. Three categories: actionable
(consent_required / insufficient_scope) with a ``Connect`` button
that opens ``/v1/api/mcp/oauth/start`` in a popup with a scheme
guard, forbidden (mcp_*_forbidden) with a static notice, operator
(key-mismatch / url-insecure) with an operator-action notice.
- New gear button in the appbar opens an MCP-connections settings
modal driven by ``loadMcpConnections`` / ``confirmRevokeMcp``
(two-step revoke confirmation matching the existing delete-ws
pattern). Pending-consent badge tracks unresolved consent prompts
in this tab; cleared after the connections list returns. Console
proxy collision-checked: the IIFE only prepends a node-id pill to
``header.firstChild``, so the right-anchored gear button is safe.
Bearer-leak invariant
- No ``exc_info=True`` on any new path that can carry a chained
``httpx.Request`` (revoke handler, dispatch sites, exec sites).
The two pre-existing ``exc_info=True`` calls in
``_exec_read_resource`` / ``_exec_use_prompt`` were replaced with
structured-field logs as a Phase 8 sibling fix.
Tests
- 440 pytest passes on both Python 3.13 (.venv) and 3.11
(/tmp/venv311); ruff + mypy clean.
- 5 new test files: ``test_mcp_consent_url_sibling_audit`` (structural
gate that every ``code="mcp_consent_required"`` / ``mcp_insufficient_scope``
site carries ``consent_url=``), ``test_mcp_oauth_connections``,
``test_mcp_oauth_revoke``, ``test_mcp_token_store_metadata``,
``test_session_mcp_dispatch_error``.
- End-to-end regression coverage for the bug-1 sibling pattern:
``test_call_tool_sync_raises_on_structured_error_envelope``,
``test_read_resource_sync_raises_on_structured_error_envelope``,
``test_get_prompt_sync_raises_on_structured_error_envelope``, plus
``test_call_tool_sync_does_not_wrap_non_structured_string`` as the
defensive gate (only ``mcp_*`` envelopes are wrapped).
Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}``: the
wrap fires only when the dispatcher returns a structured-mcp-error
string, which only happens on the oauth_user pool path.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) on every new
AS / SDK / pool-loop await per Python 3.11 anyio cancel-scope
hazard.
- Scope cap ``_MAX_INSUFFICIENT_SCOPE_REPORTED = 32`` enforced at
every output / merge site.
- Cross-user isolation on the revoke endpoint: a non-owner DELETE
returns 404 with the same body shape as a never-existed row;
``http_client_mock.post.assert_not_called()`` pins this in 3 tests.
Deferred (not Phase 8 blockers)
- perf-2 (``asyncio.gather`` parallelisation in revoke handler) —
superseded by perf-1's fire-and-forget pattern.
- q-4 (prompt-path ``isinstance(str)`` vs sibling ``_is_structured_error``
asymmetry) — already documented in the function docstring.
- q-9 (``_pendingConsentServers`` → ``_serversNeedingConsent``
rename) — pure naming taste.
Apply sanitize_text() to the new _source and _reminders columns in
both save_message and save_messages_bulk on SQLite + PostgreSQL,
mirroring the existing pattern used for content and provider_data.
Producers (sanitize_payload on the watch dispatch path,
format_nudge constants on the standard nudge path) already strip
NUL bytes today so nothing in production reaches this clamp — but
the storage layer is opaque to those invariants, and PostgreSQL
TEXT columns reject NUL outright. Without this clamp, a future
producer that forgets sanitize_payload (or hand-builds the column
string) hard-fails the chat-loop persist path on PostgreSQL.
Cost is negligible — sanitize_text early-exits on the common
no-NUL case via 'if value and "\x00" in value'.
Surfaced by Copilot's PR #486 review.
Closes round-2 review finding q-7 (nit).
The kwarg was added to close round-1 perf-2 cosmetically — the
storage backend's signature already accepted ``limit``, but the
single in-tree caller (``ChatSession.resume``) doesn't pass it and
other tail-load consumers go direct to ``storage.load_messages``.
Adding signature surface to mark a perf finding closed without an
actual consumer is API-surface bloat.
When a tail-load consumer is written (e.g. a heuristic in
``session.resume`` to skip ancient wake rows), the kwarg can come
back — at that point with a real caller driving the contract.
Closes round-2 review findings q-6 (nit) and perf-1 (nit).
* **q-6:** ``_WATCH_REMINDER_OPTIONAL_KEYS`` carried a leading
underscore (Python's module-private convention) but was imported
from two other modules — clearly a public contract between
``build_watch_reminder`` and its consumers
(``ChatSession._dispatch`` + ``server._build_history``). Drop the
underscore so the import sites match the constant's documented
cross-module role.
* **perf-1:** The dispatch closure imported the constant inside its
body, paying ``IMPORT_NAME`` + ``IMPORT_FROM`` bytecode on every
watch fire. ``server.py`` already imports at module scope; hoist
the same way in ``session.py``. Microsecond savings per dispatch,
but the in-closure form was just an oversight from the apply-pass.
Closes round-2 review findings q-1 (minor), q-3 (nit), q-4 (nit), q-5
(nit).
* **q-1:** Drop the ``post-migration 050`` clause from the fork-block
comment — the apply-pass relocated rather than removed the
tombstone-style temporal reference round-1 q-2 was supposed to fix.
The bulk-row dict shape and ``_encode_reminders`` are
self-explanatory; the WHY is pinned by
``test_fork_preserves_source_and_reminders``.
* **q-3:** Replace ``DOES persist now`` framing on the wake-row save
comment with a present-tense invariant. The ``now`` implies the
reader knows the prior state, same family as the temporal
tombstones.
* **q-4:** Trim the 12-line WHAT-narration block above the
resume-time ``_reminders_delivered = True`` loop to two lines
stating the WHY only. The new regression test pins the contract.
* **q-5:** Reframe ``test_fork_preserves_source_and_reminders``
docstring as a forward-looking invariant; drop the
``Dropping them was the original bug`` and ``post-migration 050``
fix-narration.
Project convention: invariant statements, present tense; don't
reference the current task / fix / migration number.
Closes round-2 review findings bug-1 (minor) and q-2 (minor).
* **bug-1:** ``_encode_reminders`` clamped each entry's ``text`` field
with Python ``str`` slicing, which counts codepoints. Multi-byte
UTF-8 input (CJK, emoji) could land 4 bytes per character past the
cap, defeating the row-width / FTS5-index protection by up to 4x.
Switch to UTF-8 byte clamping with ``errors="ignore"`` on the
decode boundary so a slice mid-codepoint drops the partial
character cleanly.
* **q-2:** Both the constant block-comment and the ``_encode_reminders``
docstring referenced ``docs/design/watch-card-ux-briefing.md`` —
local-only per project convention (``feedback_no_design_doc_commits``)
so the canonical repo reads as a dead reference. The cap value
stands by itself; the row-width / FTS5 WHY is enough.
Closes round-1 review findings q-2 (minor), q-5 (minor), q-6 (nit), q-7
(nit), sec-1 (nit), perf-4 (nit).
* **q-5:** Export ``_WATCH_REMINDER_OPTIONAL_KEYS`` from
``turnstone/core/watch.py`` and import in the dispatch closure
(session.py) and the replay filter (server.py:_build_history). The
three-place duplication of the literal tuple
``("watch_name", "command", "poll_count", "max_polls", "is_final")``
is gone; future field adds touch one constant.
* **sec-1:** Run ``sanitize_payload`` over string-typed metadata fields
(``watch_name`` / ``command``) before they enter the queue. Today's
consumers all use ``textContent``, but the asymmetry — sanitised
``text`` alongside unsanitised metadata — would survive forever in
DB rows and resurface if a future consumer used a non-textContent
sink (aria-label, copy-to-clipboard, markdown render).
* **q-7:** Drop the per-iteration ``isinstance(reminder, dict)`` from
the dispatch closure's metadata comprehension. By the time the
block runs, ``text = reminder.get("text", "") if isinstance(...)``
+ the ``if not sanitized: return`` guard above already established
``reminder`` is a non-empty dict.
* **q-2:** Strip tombstone-style references — "post-#482", "post-#484",
"Step 7 of the watch-card UX plan", "Post-Step-7 dispatch surface",
and the brittle line-anchor "session.py:2685-2686" — across
``session.py``, ``test_session.py``, ``test_watch.py``,
``test_watch_dispatch.py``, ``test_watch_integration.py``. Comment
intent preserved; historical anchors gone.
* **q-6:** Drop the ``del source`` line in ``cli.py``'s
``on_user_reminder``; the parallel ``on_tool_reminder`` ignores
``tool_call_id`` without ``del`` and the comment alone is enough.
* **perf-4:** Document the SQLite ``render_as_batch=True`` recreate
cost in migration 050's docstring — first deployment after upgrade
copies the conversations table twice (one per ``add_column``).
PostgreSQL is unaffected.
5734 non-live tests pass; ruff + mypy clean.
Closes round-1 review findings q-3 + q-4 (minor, merged) and bug-3 + bug-4
(nit, merged).
* **q-3 + q-4:** The new ``.msg.user-reminder .msg-body { white-space:
pre-wrap }`` rule was a no-op on the interactive UI because that
frontend's ``_buildDefaultReminderBubble`` appended label + text spans
directly to the outer ``.msg.user-reminder`` element with no
``.msg-body`` wrapper. Coord rendered the same shape with a wrapper.
The two implementations diverging on DOM structure also meant a
shared-helper extraction was harder than necessary. Reconciled by
wrapping interactive's spans in ``.msg-body`` to match coord; the CSS
rule now applies to both UIs and the shared-extraction follow-up to
``shared_static/cards.js`` is mechanical (deferred per the review
report — out of scope for this commit).
* **bug-3 + bug-4:** The reminder anchor lookup ``.msg.user`` also
matched ``.msg.user.system-nudge`` markers because the marker carries
both classes. A non-wake reminder fired between a wake marker and
the next real user message would anchor below the wake marker rather
than the previous real user message. Edge case (``/history`` reload
corrects), but the fix is mechanical: change the selector to
``.msg.user:not(.system-nudge)`` in both files.
Closes round-1 review finding perf-2 (minor).
Storage backends accept ``*, limit: int | None = None`` (see
:meth:`StorageBackend.load_messages` at storage/_protocol.py:146) but
the in-memory wrapper at memory.py:82-85 dropped the kwarg, so
callers that wanted to tail-load (e.g. ``session.resume`` against a
long-running coord with hundreds of wake rows + persisted reminder
JSON) were forced to pull every row through the wrapper anyway.
Wraparound is mechanical: signature widens, default leaves existing
callers unaffected.
Closes round-1 review finding q-1 (major).
The comment block above ``self._attach_pending_user_reminders(user_msg)``
asserted that reminders "stay in-memory only and don't persist across
reloads" — directly contradicted by the comment block immediately below
(at the save_message call site) that explains the new persistence
semantics, plus the actual code that now writes ``_source`` and
``_reminders`` to the conversations row. Future readers hitting both
blocks would lose trust in the surrounding comments.
The lower block already documents the persistence contract, so the
upper block is just deleted rather than rewritten.
Closes round-1 review findings bug-2 (major), perf-1 (minor), perf-6 (nit).
* **bug-2:** ``ChatSession.resume(..., fork=True)``'s bulk-row builder
silently dropped the ``_source`` and ``_reminders`` side-channel
data the source workstream had persisted via ``_append_user_turn``.
Both backends' ``save_messages_bulk`` already accept these keys
(the columns exist post-migration 050) — the bulk builder just
didn't supply them. The fork's resumed transcript would then look
like the assistant turn answered out of nowhere: every wake marker
and every reminder bubble that survived to disk on the source got
dropped on the fork. New regression test
``test_fork_preserves_source_and_reminders`` pins the contract.
* **perf-6:** Extracts ``_encode_reminders(reminders) -> str | None``
near ``_apply_reminders_for_provider`` so the user-turn save path,
the tool-turn save path, and the new fork bulk builder share one
encoder. Eliminates the drift risk between three near-identical
``json.dumps(..., separators=(",", ":")) if X else None`` patterns.
* **perf-1:** The new helper clamps each entry's ``text`` field at
``REMINDER_TEXT_STORAGE_CAP = 8192`` characters before encoding so
a single rogue producer (a watch streaming unbounded shell output,
a corruption-class steering payload) can't blow the conversations
row width or the FTS5 index. The in-memory side-channel keeps the
full body — only the persisted JSON is clamped. Mirrors
``TOOL_RESULT_STORAGE_CAP`` on tool result rows.
5734 non-live tests pass; ruff + mypy clean.
Persisted ``_reminders`` survive ``load_messages`` but the in-memory
``_reminders_delivered`` flag does not (it's session-scoped — set by
``_mark_reminders_delivered`` after each successful provider stream,
never persisted alongside the JSON column). Without a re-splice
guard at resume time, ``_apply_reminders_for_provider`` would walk
every loaded message, see ``_reminders`` set + the flag falsy, and
splice every historical ``<system-reminder>`` envelope onto the wire
on the very next user turn — leaking each reminder a second time, the
turn after it had already advised.
Mirror the post-stream hook in ``resume()``: every loaded message
that carries reminders has already been delivered (it survived to
disk), so flag it accordingly so ``_apply_reminders_for_provider``
short-circuits on the pass-through path.
Test pins the contract end-to-end — stage a workstream with a
persisted reminder, resume into a fresh session, append a live user
turn, run the wire transform, and assert the historical reminder
body does NOT land in the rendered output.
User-visible slice of the watch-card UX workstream — combines the
replay-path widening, both frontend renderers, the CSS, and the
cross-cutting Python tests.
server._build_history widens the reminder filter from {type, text} to
project on a known set of optional fields (watch_name, command,
poll_count, max_polls, is_final) and surfaces _source as
entry["source"] when set. The known-key filter narrows the blast
radius if a future producer accidentally stuffs sensitive fields
into the dict.
SessionUIBase.on_user_reminder takes a new source: str | None kwarg
that rides on the SSE event when set. _attach_pending_user_reminders
forwards user_msg["_source"] so non-originating tabs see the wake's
"system_nudge" tag and render the thin marker. Protocol + cli + eval
implementations widen accordingly.
Frontend (coordinator.js + app.js — touched in lockstep per project
memory's "logic that lands in BOTH UIs must touch both files"):
* Branch on r.type === "watch_triggered" for a structured
.msg.watch-result card with header / $ command / <pre> body /
poll N/M [· final] footer.
* New addSystemNudgeMarker (interactive) + appendSystemNudgeMarker
(coord) renders a thin .msg.user.system-nudge anchor for
wake-driven reminders, both live (source === "system_nudge" on the
SSE event) and replay (msg.source === "system_nudge").
* Default .msg.user-reminder rendering preserved for every other
metacog nudge type.
CSS (shared_static/chat.css):
* New .msg.watch-result rules — full-width treatment, cyan accent,
monospace body with word-break: break-word for mobile.
* New .msg.user.system-nudge rule — thin yellow marker.
* Bonus newline-collapse fix: .msg.user-reminder .msg-body now sets
white-space: pre-wrap so multi-line shell output / bulleted lists
stay readable inside the advisory bubble.
Plan reference: docs/design/watch-card-ux.md §4 Steps 9-12 + bonus
CSS §11 (Commit 4).
WatchRunner._dispatch_result now takes a structured reminder dict
produced by build_watch_reminder() — text matches format_watch_message
verbatim (so compaction / channel adapters / wire splice keep their
behaviour), and watch_name / command / poll_count / max_polls /
is_final ride alongside as queue-entry metadata.
The dispatch closure registered in ChatSession.set_watch_runner pulls
the optional fields out of the dict and passes them to enqueue via
the new metadata kwarg. Drain seams already merge metadata into the
rendered reminder dict (Commit 2), so the SSE event for a watch fire
now carries the structured fields without further plumbing.
* turnstone/core/watch.py — new build_watch_reminder() helper, _poll_watch
switches from format_watch_message + dispatch(str) to build_watch_reminder
+ dispatch(dict). set_dispatch_fn / get_dispatch_fn / restore_fn
signatures widen from Callable[[str, str], None] to
Callable[[dict[str, Any], str], None].
* turnstone/core/session.py — dispatch closure builds the metadata dict
via {k: reminder[k] for k in ("watch_name", "command", ...) if k in reminder}
and passes it to nudge_queue.enqueue.
* tests/test_watch.py — new TestBuildWatchReminder class pinning the
builder shape; existing dispatch_fn_registry / restore_fn tests
updated to dict shape.
* tests/test_watch_dispatch.py — every dispatch(...) call updated to
pass a structured reminder dict via _reminder() helper; new
TestMetadataPropagation class pins the metadata-on-enqueue contract.
* tests/test_watch_integration.py — _dispatch_result calls updated to
dict shape.
Plan reference: docs/design/watch-card-ux.md §4 Step 7 + Step 8 watch-test
subset (Commit 3).
Producers (today only watch_triggered) can now attach a metadata dict
to a queued nudge so the rendered reminder dict on the user/tool side
carries fields beyond {type, text}. Wire shape stays additive: the
SSE event picks up the optional fields when present, and producers
without metadata leave it None.
* _Entry grows from 4 fields to 5 — metadata: dict[str, Any] | None.
* enqueue accepts metadata=... as a kwarg.
* drain returns list[tuple[str, str, dict | None]] (was 2-tuples).
* pending stays narrow at (type, text) for legacy callers; new
pending_with_metadata projects the third slot for tests that need
to assert producer-specific fields.
* Three drain consumers in session.py — _collect_advisories,
_attach_pending_user_reminders, deliver_wake_nudge_from_queue —
unpack the new 3-tuple shape and merge metadata into each
reminder dict.
* on_user_reminder / on_tool_reminder protocol signatures widen
from list[dict[str, str]] to list[dict[str, Any]] across
ChatSession.UI, SessionUIBase, CLI, eval harness.
Plan reference: docs/design/watch-card-ux.md §4 Step 6 + Step 8 _Entry
subset (Commit 2).
Adds two TEXT-NULL columns to the conversations table so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live. Until now, reminders lived only on the
in-memory ChatSession.messages dict, and the wake-driven empty user
turn was not persisted at all (skip at session.py:2685-2686) — a
second tab connecting via /history saw the assistant turn with no
preceding wake context, and missed every other tab's reminder
bubbles besides.
Single Alembic revision 050 (head was 049) adds:
* conversations._source — today only "system_nudge" for wake rows
* conversations._reminders — JSON-encoded reminder list
Both backends (sqlite + postgresql) thread the columns through
save_message / save_messages_bulk / load_messages. reconstruct_messages
unpacks the row tuple as 9 elements (was 7), JSON-decoding _reminders
on the user AND tool branches with the same contextlib.suppress guard
the existing provider_data / tool_calls decode uses. Tool-row
reminders ride the same column so tool_error / repeat replay shape
matches user-channel parity.
session.py:2685-2686 wake-row persist skip is dropped; _append_user_turn
JSON-encodes user_msg["_reminders"] and passes both source + reminders
to save_message. The tool-message save site at session.py:3014-3020
mirrors with metacog_reminders.
Plan reference: docs/design/watch-card-ux.md §4 Steps 1-5 (Commit 1).
Address Copilot review feedback on PR #487:
1. **Atomic commit invariant**: ``_bootstrap_coord_subsystem`` previously
stamped ``coord_mgr`` ~50 lines before the final ``coord_registry``
commit, and started threads + subscriptions in between. A concurrent
dashboard request running through ``_require_coord_mgr`` during the
runtime-bootstrap window could observe ``coord_mgr`` set with
``coord_registry`` still ``None`` and surface the misleading
"Restart the console after adding a model definition" 503.
Refactored to two phases: (a) build everything as locals, (b) start
side-effects (StateWriter / observer / nudge watcher / child fan-out
/ cleanup thread), then atomic commit at the end with ``coord_mgr``
stamped LAST. The build-phase ``try/except`` rolls back any started
side-effects from local handles before re-raising — no daemon thread
or subscription leaks across retries, and ``app.state`` is never
stamped on a partial failure.
2. **Class-attr cleanup symmetry**: ``_teardown_partial_coord_subsystem``
now also clears ``ConsoleCoordinatorUI._coord_mgr`` /
``_collector`` / ``_console_metrics`` to match the lifespan shutdown
path (server.py ~line 4629). A failed bootstrap (or test teardown
reuse) no longer leaks process-global pointers at a half-built
subsystem.
3. **Lifespan startup offload**: the lifespan startup error path used
to call ``_teardown_partial_coord_subsystem`` synchronously, which
in turn calls ``StateWriter.shutdown(timeout=2.0)`` — a thread-join
+ sync DB writes that could block the event loop for up to 2s
while the console is still coming up. Wrapped the whole
load-and-bootstrap in ``asyncio.to_thread`` via the new
``_load_and_bootstrap_coord_subsystem`` synchronous helper, so all
blocking work (including any rollback) runs on a worker thread.
Mirrors the pattern the regular lifespan shutdown (line ~4620) and
the runtime CRUD-triggered path already use.
Tests:
- ``test_bootstrap_atomic_commit_no_partial_visibility``: a polling
thread in tight loop watches ``coord_mgr`` / ``coord_registry``
during a real bootstrap and asserts no observation has ``coord_mgr``
set with ``coord_registry`` still ``None``.
- ``test_real_bootstrap_rolls_back_partial_state_on_side_effect_failure``:
monkeypatches ``install_idle_nudge_watcher`` to raise mid-build,
asserts ``app.state`` shows the clean fresh-install state and the
builder-failure error string surfaces ``RuntimeError`` (not the
stale "no models" boot-time message).
A freshly-installed console with no model rows in the DB at boot
caught the ``ValueError`` from ``load_model_registry()`` in the
lifespan and skipped the entire coord subsystem build, leaving
``coord_mgr`` ``None``. ``_refresh_coord_registry`` then bailed
out at ``existing is None`` rather than building the subsystem on
first model add — operators had to restart the console after
configuring their first model in the admin panel for the
"Coordinator subsystem not initialized" banner to clear.
Extract the lifespan's coord build into a reusable
``_bootstrap_coord_subsystem`` and add ``_maybe_bootstrap_coord_subsystem``
that runs as an ``asyncio.to_thread`` follow-on after every admin
model-CRUD endpoint (create/update/delete/reload). The helper:
- fast-paths to a no-op when ``coord_mgr`` is already set;
- guards concurrent first-install attempts with
``_COORD_BOOTSTRAP_LOCK`` + double-checked re-test inside the lock;
- pre-computes config-derived integers BEFORE any thread starts so
``int(config_store.get(...))`` failures don't strand a started
``StateWriter`` daemon;
- stamps ``coord_state_writer`` to ``app.state`` immediately after
``.start()`` so the new ``_teardown_partial_coord_subsystem`` can
shut it down on a partial failure (no thread leaks across retries);
- atomically commits ``coord_registry`` + clears
``coord_registry_error`` as the final step so callers can rely on
the invariant ``coord_registry`` is set iff ``coord_mgr`` is set;
- replaces the stale boot-time "no model definitions" message with
a builder-failure-specific diagnosis (carrying ``type(exc).__name__``)
on construction failure so the dashboard's 503 banner reflects the
actual cause.
Both the lifespan path and the runtime-bootstrap path now route
through the same helper and the same teardown on failure.
Tests: 12 new tests covering the helper-level wiring (idempotent
fast-path, missing-prereq parametrised over ``config_store`` /
``collector`` / ``console_metrics``, no-rows error recording, builder
failure error replacement, partial-state teardown), the endpoint
integration, the deterministic concurrent-call lock test (uses an
instrumented lock wrapper that signals when a second acquirer arrives,
so the test fails fast on slow CI rather than depending on a
wall-clock sleep), and a real-builder end-to-end case constructing a
working ``SessionManager`` against a real ``ConfigStore`` + real
``ClusterCollector``.
Two of five Copilot comments on PR #485 were valid; this commit applies
both. The other three (one duplicate of comment 1, plus the INFO-logging
and `_pending`-naming nits) get rationale on-thread and resolution.
1. emit_oauth_failure_audit action now derived from `code` (#485 bug-1)
The Phase 7b refactor generalized `emit_insufficient_scope_audit` →
`emit_oauth_failure_audit`, routing both `mcp_insufficient_scope` AND
generic-403 (`mcp_*_forbidden`) through the same helper. The audit
`action` field stayed hardcoded as
`"mcp_server.oauth.insufficient_scope_emitted"`, mislabeling generic
forbidden events under the insufficient_scope bucket — downstream
alerting / analytics filtering on `action` would silently fold both
categories together.
The action is now selected from `code`:
* `mcp_insufficient_scope` →
`mcp_server.oauth.insufficient_scope_emitted` (preserves existing
alerting consumers)
* `mcp_tool_call_forbidden` / `mcp_resource_read_forbidden` /
`mcp_prompt_get_forbidden` →
`mcp_server.oauth.forbidden_emitted` (new, distinct label)
Detail row continues to carry both `code` and `kind` so operators get
sub-bucket distinction within either action.
2. Resource-listener docstrings cite RFC §3.2 (#485 doc-1)
Per the codebase convention established in Phase 7b round-1 q-1
(`_rebuild_user_prompt_map` corrected §3.2 → §3.3 because prompts are
§3.3 in the MCP spec), resource-related docstrings should cite §3.2.
The three resource-listener docstrings were citing §3.3, and the
"Mirrors `_notify_listeners` for tools (RFC §3.3)" parenthetical in
both `_notify_resource_listeners` and `_notify_prompt_listeners` read
as "tools are at §3.3" — confusing twice over. All four sites now
carry the correct catalog-kind citation explicitly:
* resource-listener docstrings → "RFC §3.2 (resources)"
* prompt-listener docstrings → "RFC §3.3 (prompts)"
Tests / lint:
* 119 passed on 3.13 + 3.11 (targeted MCP OAuth pool tests)
* ruff + mypy clean on both files
Extends the Phase 7 per-(user, server) ClientSession pool to cover
RFC §3.2 (resources/read) and §3.3 (prompts/get) on the same shape
already proven for tools/call. Pool discovery is capability-gated so
servers without resources/ or prompts/ stay free of extra round-trips.
API additions / widenings (MCPClientManager):
- ``read_resource_sync(uri, *, user_id=None, timeout=120)`` —
per-user-first dispatch; falls through to the byte-identical static
path when ``user_id`` is None or the URI doesn't resolve to an
``oauth_user`` pool entry.
- ``get_prompt_sync(prefixed_name, arguments=None, *, user_id=None,
timeout=30)`` — same dispatch shape; structured-error responses
surface via ``RuntimeError`` so the agent-loop's ``except Exception``
block renders the JSON without polluting the prompt-protocol return
shape.
- ``get_resources(user_id=None)`` / ``get_prompts(user_id=None)`` —
per-user merged catalogs (admin/global call still passes None).
- ``add_{resource,prompt}_listener`` /
``remove_{resource,prompt}_listener`` — ``user_id`` keyword scopes
the listener so a pool-only catalog change for one user does not
wake another user's session.
- ``resource_count_for_user(user_id=None)`` /
``prompt_count_for_user(user_id=None)`` — method-form variants used
by ChatSession's ``read_resource`` / ``use_prompt`` tool gating; the
legacy ``resource_count`` / ``prompt_count`` properties remain
static-only for admin paths.
- ``_dispatch_pool_resource`` / ``_dispatch_pool_prompt`` async coros
— mirror ``_dispatch_pool`` for the new SDK calls; share the
carrier-race-and-cancel core via ``_dispatch_pool_with_entry_call``.
- ``_handle_auth_403`` extended with ``kind=Literal["tool",
"resource", "prompt"]`` so the per-operation ``mcp_*_forbidden``
code surfaces (kind="tool" remains the default for back-compat).
- Pool notification handler now refreshes resources / prompts on
``ResourceListChangedNotification`` / ``PromptListChangedNotification``
via ``_refresh_pool_server_resources`` / ``_refresh_pool_server_prompts``.
ChatSession (``turnstone/core/session.py``) call-site updates:
- 12 sites threaded the session-bound ``user_id`` through
``add_*_listener`` / ``remove_*_listener``, ``get_resources`` /
``get_prompts``, gating, ``read_resource_sync`` /
``get_prompt_sync``, and ``is_mcp_prompt`` so the per-user merged
catalog drives both the visible-tool set and dispatch.
- ``/mcp`` slash command now lists this user's pool resources and
prompts alongside tools (Phase 7 already scoped tools).
Scope decisions:
- Per-user-first URI ordering (decision 0.1): the dispatcher attempts
the user's pool catalog first, falling back to the static catalog
only when no pool entry resolves the URI / prefixed name. Pool-only
users never see the static catalog leak into their resolution.
- Method-form ``*_count_for_user`` (vs property) keeps the legacy
``resource_count`` / ``prompt_count`` properties intact for admin
endpoints whose contract is "static catalog size only".
- Shared ``_dispatch_pool_with_entry_call`` helper accepts an
``sdk_call: Callable[[ClientSession], Awaitable[Any]]`` closure,
keeping the entry-locked carrier-race / classification / retry
plumbing single-source instead of a 3x copy across tool / resource
/ prompt paths.
R6 (anyio uniformity): every pool-side list / read / get path uses
``async with asyncio.timeout(...)`` — ``asyncio.wait_for`` is
forbidden in those paths because it wraps the inner awaitable in a
fresh task and surfaces ``CancelledError`` from inside
``streamablehttp_client``'s anyio TaskGroup on Python 3.11
(per ``feedback_asyncio_timeout_vs_wait_for.md``).
Tests:
- ``test_mcp_pool_auth_resource_integration.py`` — 9 real-transport
resource tests (FastMCP upstream + ``BehaviorMiddleware``):
401-refresh-retry success, persistent 401 -> consent_required,
403+insufficient_scope, 403 generic -> mcp_resource_read_forbidden,
breaker-isolation under repeated auth failures, missing-token,
decrypt-failure, http:// URL guard, unknown-URI ValueError.
- ``test_mcp_pool_auth_prompt_integration.py`` — 9 mirror tests for
the prompt path; structured-error responses verified via
``RuntimeError`` payload shape.
- ``test_mcp_user_catalog.py`` — extended unit coverage for per-user
resource / prompt rebuild + collision policy + symmetric eviction.
- ``test_sessions.py::TestMCPToolGating`` — pool-only-user canary
asserts ``read_resource`` / ``use_prompt`` stay visible when the
static catalog is empty but the user has pool entries.
Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: ``_exec_use_prompt`` was hardcoding ``"MCP prompt error: failed
to invoke prompt"`` — discarding the structured-error JSON that
``_dispatch_pool_prompt_sync`` raises via ``RuntimeError``. Now uses
``f"MCP prompt error: {e}"`` mirroring ``_exec_mcp_tool``; pool-prompt
consent_required / insufficient_scope / forbidden errors now reach
the LLM as intended.
- bug-2 + bug-3: resource template discovery was uncapped —
``_cap_server_resources`` covered ``res_result.resources`` but the
separate ``tmpl_result.resourceTemplates`` loop appended every
template a server returned. Added ``_MAX_RESOURCE_TEMPLATES_PER_SERVER``
(1000) + ``_cap_server_resource_templates`` helper, applied at both
the initial discovery site (``_connect_one_pool``) and the refresh
site (``_refresh_pool_server_resources``). Mirrors the existing
``_MAX_TOOLS_PER_SERVER`` / ``_MAX_PROMPTS_PER_SERVER`` defensive
ceilings.
- sec-1 + sec-2: ``emit_insufficient_scope_audit`` generalized to
``emit_oauth_failure_audit(kind, code, ...)``, called from both the
insufficient_scope branch AND the previously-silent generic 403
branch. Audit detail now records ``{"kind": kind, "code": code,
"scopes_required": [...]}`` so operators can distinguish tool-call
vs resource-read vs prompt-get 403s in audit logs and so cross-
tenant probing on the generic 403 path leaves a trail. The Phase 7
inherited gap (``mcp_tool_call_forbidden`` had the same silence) is
closed in the same refactor.
- perf-1: pool resource discovery now uses ``asyncio.gather(
list_resources, list_resource_templates)`` inside the existing
``async with asyncio.timeout(...)`` budget — disjoint catalogs, no
ordering dependency. Typical-case 2-RTT cold-connect resource block
collapses to 1-RTT. Same change applied at ``_refresh_pool_server_resources``.
- q-1: ``_rebuild_user_prompt_map`` docstring corrected RFC §3.2 →
§3.3 (resources are §3.2; prompts are §3.3).
- q-2: ``_refresh_pool_server_prompts`` docstring now carries the
R6 / mcp-loop note that the resource sibling already had — both
refresh paths now declare the asyncio.timeout invariant explicitly.
- q-5: added the ``_user_resource_map`` / DB-mismatch guard to
``read_resource_sync`` for parity with ``get_prompt_sync``. A stale
per-user map entry with no matching oauth_user row now raises a
specific ValueError instead of silently falling through to a
generic ``Unknown MCP resource``.
- q-6: ``_dispatch_pool_with_entry`` (now a single-caller wrapper
after the ``_dispatch_pool_with_entry_call`` extraction) gains a
one-line docstring explaining why the wrapper is preserved
(tool-decode localization + stack-trace identity for debugging).
- q-7: added 1 resource + 1 prompt end-to-end integration test that
drive REAL discovery + dispatch in the same connect (no
``_seed_pool_*_map`` shortcuts), mirroring the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``.
The seeded-map tests stay (faster, focused on dispatch); the new
e2e tests cover the connect-discover-dispatch composition that
caught Phase 6's carrier-on-entry bug.
Pre-push round-1 review fixes (3-finder review on the final state —
the lesson from Phase 7 round-3's q-1 regression: round-2 catches
what the round-1 apply pass missed):
- q-1 (MAJOR): the bug-1 sibling that round-1 missed —
``_exec_read_resource`` was hardcoding ``"MCP resource error: failed
to read resource"`` while ``_exec_use_prompt`` (post-bug-1) preserved
the structured-error JSON via ``f"... error: {e}"``. The round-1
apply pass patched the prompt side but not the resource side. q-5's
per-user-map / DB-mismatch ValueError was being swallowed at the
agent loop boundary, defeating the operator-diagnostic intent. Now
``_exec_read_resource`` mirrors ``_exec_mcp_tool`` and ``_exec_use_prompt``.
- q-6 (nit): defensive-cap comment block at module-level cited
"(RFC §3.2)" while covering both resource and prompt list paths;
prompts are §3.3. Now reads "(RFC §3.2 for resources, §3.3 for
prompts)" matching the convention the q-1 apply established.
- q-5 (rejected with better justification): the reviewer flagged
``_dispatch_pool_with_entry`` as a single-caller wrapper that should
be inlined. After examination — the autouse fixture
``tests/test_mcp_pool_auth_introspection.py::_install_capture_intercept``
monkeypatches this method to stash ``entry.auth_capture`` for the
fake call_tool stubs in dispatcher-asserting tests. Inlining would
redirect the patch to ``_dispatch_pool_with_entry_call`` (different
kwargs shape) and require re-validating every test that depends on
the interception. The wrapper IS load-bearing; q-6 docstring updated
to cite the test-fixture rationale instead of the thin "stack-trace
identity" claim.
Deferred to follow-up (documented rationale):
- perf-2: single-pass partition for system-message resource list
(concrete vs templates). Sub-microsecond at expected scale;
opportunistic-only.
- q-2 (pre-push): ~200 lines of fixture infrastructure
(``BehaviorMiddleware``, ``_build_server``, ``_seed_oauth_server``,
``running_loop_mgr``, etc.) duplicated across three pool-integration
test files. Real maintenance cost, but a 200-line conftest extraction
is a focused refactor that earns its own commit / PR. Tracking as
follow-up rather than balloon Phase 7b's diff further.
- q-3 / q-4 (refactor): extract shared dispatcher / scheduler
helpers to compress three near-identical 90-line bodies (round-1
q-3 was the same root cause; the pre-push q-3/q-4 reviewer
reaffirmed it concretely). Three named methods preserve readability
for the codebase's hottest correctness path; follow-up if
duplication grows further or if a per-path divergence ships.
- q-4 (round-1, distinct from pre-push q-4): split pool concerns
into ``mcp_pool.py``. Out-of-scope per finder; future refactor as
the file approaches the navigation/merge-conflict threshold.
3.13: 5590 passed (5541 baseline -> +49 net; pre-review +47, q-7
e2e tests added +2). Existing audit-detail tests updated in-place
to expect the new ``kind`` and ``code`` fields.
3.11: 5590 passed (parity gate per ``feedback_pytest_env_parity.md``).
Closes PR #484 review findings (Copilot): the soft-cap pattern in
``ChatSession.set_watch_runner``'s dispatch closure was a non-atomic
two-call pair (``count_by_type`` then ``drop_oldest_by_type``) with
two separate lock acquisitions. A concurrent drain on the worker
thread (``USER_DRAIN`` / ``TOOL_DRAIN`` consuming ``"watch_triggered"``
entries via the ``"any"`` channel) could slip between the two calls,
making the drop a no-op. The dispatch closure also discarded
``drop_oldest_by_type``'s return value and unconditionally logged
``dropped_oldest=True``, so a no-op drop got reported as a successful
drop.
* New ``NudgeQueue.cap_at_or_drop_oldest(nudge_type, max_depth,
channel=None) -> bool`` does the count+drop in a single critical
section. Returns the actual outcome.
* Dispatch closure (``session.py:1410-1416``) now calls the helper and
uses its return value to gate the WARNING log line, so the log is
accurate when a drop did NOT happen.
* ``drop_oldest_by_type``'s docstring no longer overstates the
per-call lock as covering a count+drop pair — it points readers
to ``cap_at_or_drop_oldest`` for that contract.
7 new tests in ``TestCapAtOrDropOldest`` cover: below-cap no-op,
at-cap drop-oldest, above-cap drop-only-one (per-call), channel
filter, other-type isolation, ``max_depth <= 0`` defensive no-op,
no-match.
5708 non-live tests pass; ruff + mypy clean.
The github-code-quality bot finding ("Statement has no effect" on
``_protocol.py:939``'s ``...`` body) is a false positive — every
Protocol method in ``_protocol.py`` uses ``...`` as its body, which
is the canonical Python Protocol pattern. Replacing with ``pass``
would diverge from the file's existing style. No code change.
Closes round-2 review findings q-3, q-4, q-5, q-7.
* **q-4:** ``_NAME_CONTROL_CHARS`` and ``_PAYLOAD_CONTROL_CHARS`` shared
7 lines of Unicode-steering character classes (zero-width / bidi /
separators / BOM / tag chars above BMP). Factored into a single
``_CONTROL_CHARS_TAIL`` constant; each regex now differs only in its
leading ASCII range. Future bidi or zero-width additions edit one
place.
Side effect: this corrects a latent bug where ``_NAME_CONTROL_CHARS``
had two literal ASCII spaces in place of U+2028 / U+2029 (line and
paragraph separators) — visible as ``r" "`` in source but rendered
as the actual codepoints in ``_PAYLOAD_CONTROL_CHARS``. After the
factoring both regexes correctly include U+2028 / U+2029, closing
the gap that would have let a workstream name with embedded line
separators forge a sibling bullet (the same vector ``\n`` was
blocked for in the original bug-1 fix).
Switched to ``\u`` escapes for readability (and to keep future Edit
tool runs against this block reliable).
* **q-3:** Tombstone clause "standing in for the deleted
``_watch_pending`` maxsize bound" survived in
``ChatSession.set_watch_runner``'s docstring after the apply-pass
trim cleaned the inline soft-cap comment. Dropped.
* **q-5:** ``test_newline_in_name_does_not_forge_extra_bullet`` carried
five WHAT-narration comments restating what the immediately-following
asserts already say. Dropped — the docstring carries the security
invariant; the assertions speak for themselves.
* **q-7:** ``patch_session_storage`` had a 14-line docstring including
fallback-guidance and self-justification ("accumulated 7 near-duplicate
sites"). Trimmed to a 3-line contract.
Closes round-2 review findings q-1, q-2, q-6.
* **q-1:** ``test_valid_until_drops_when_watch_missing`` collapsed to the
same code path as ``test_valid_until_drops_when_watch_inactive`` after
the apply-pass switched the predicate from ``get_watch[active]`` to
``is_watch_active`` (both stubbed via ``patch_session_storage(active=False)``).
The "missing" case has no distinguishable branch at the dispatch
layer, so dropping it removes a tautological duplicate. The
missing-row mapping moves to the storage layer (q-2 below) where it
IS distinguishable.
* **q-2:** ``is_watch_active`` was a new public storage primitive with
zero direct backend coverage — only via-session-via-stub coverage.
New ``TestIsWatchActive`` in ``tests/test_watch_storage.py`` covers
active row → True, inactive row → False, missing row → False.
Pinned at the storage boundary so future backend changes fail loudly
there instead of in the dispatch tests.
* **q-6:** Concurrency test had ``n_threads = 2`` alongside two literal
Thread objects and a tautological ``assert len(threads) == n_threads``.
Threads are now built from a labels tuple, so ``len(threads)`` drives
the slack bound; the redundant assertion is gone.
Closes review findings bug-4 and q-6.
bug-4 — the watch dispatch concurrency test bounded depth at
``_WATCH_QUEUE_SOFT_CAP + 2 * per_thread`` (= 250) which is
tautologically true: two threads × 100 fires can append at most 200
entries above the cap, so the bound asserted nothing more than what
``depth <= 2 * per_thread`` already says. Tighten to
``_WATCH_QUEUE_SOFT_CAP + N_THREADS`` (= 52): the count-then-drop window
admits at most one slip per concurrent thread.
q-6 — 7 near-duplicate ``monkeypatch.setattr(session_mod, "get_storage",
lambda: _StubStorage())`` sites across ``test_watch_dispatch.py`` +
``test_watch_integration.py`` (4 different stub shapes, mostly trivial
variations on the active flag). Lift a ``patch_session_storage``
helper into the existing ``tests/_helpers.py`` with kwargs for the
common cases (``active``, ``raise_on_is_active``), returns the call list
so call-shape assertions still work. Tests collapse from ~10-line
inline-class blocks to one-line helper calls.
Closes review findings q-2 and q-5.
q-2 — ``bound_watch_id = watch_id`` rebind was unnecessary. ``_dispatch``
is constructed fresh per fire (not in a loop), so ``_still_active``
closes over the function parameter directly without any
loop-variable-capture risk. Drop the rebind.
q-5 — the inline soft-cap comment restated rationale already covered by
the ``_WATCH_QUEUE_SOFT_CAP`` block-comment at module scope and dragged
in a tombstone reference to the deleted ``_watch_pending`` path. Trim
to one line stating only the WHY (drop-oldest because latest output is
most useful). Leave the ``set_watch_runner`` docstring's operational
detail at lines 1356-1378 alone — trimming further risks losing the
``valid_until`` predicate semantics.
Closes review finding q-4.
The closure built inside ``server.py``'s ``_watch_restore_fn`` is the
new contract surface introduced by the switchover — it constructs a
fresh ChatSession, calls ``session.resume(ws_id)`` to adopt the
original ws_id, re-registers the dispatch closure via
``set_watch_runner``, and returns ``WatchRunner.get_dispatch_fn`` for
the runner to invoke directly. No automated coverage exists today;
a future refactor (e.g. swapping ``manager.create + session.resume``
for ``manager.open``) could silently break the watch-restore pipeline.
Adds ``test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session``
to ``tests/test_watch_integration.py`` — drives the full restore path:
persists a kickoff message for the original ws_id, fires
``_dispatch_result`` against a runner with no registered dispatch fn,
asserts the restore_fn ran exactly once, the rehydrated session is a
distinct object that adopted the original ws_id, and the watch payload
landed on the rehydrated session's NudgeQueue (not on the original).
Closes review finding perf-1.
The watch dispatch closure's ``valid_until`` predicate fires once per
watch entry at every drain seam — on the chat-loop hot path. It only
needs the ``active`` flag, but ``storage.get_watch`` runs a full-row
``SELECT *`` and marshals the result into a dict. At the typical drain
depth (cap-50 + a busy chat loop) that's ~50 throwaway dict allocations
per drain pass for one boolean.
Adds ``StorageProtocol.is_watch_active(watch_id) -> bool`` plus
SQLite + Postgres implementations doing a single-column
``SELECT active FROM watches WHERE watch_id = ?`` (returns False on
missing row). ``_still_active`` in ``ChatSession.set_watch_runner``
now calls that instead of indexing into the full row.
Test stubs that mocked ``get_watch`` for the predicate are converted
to mock ``is_watch_active`` directly. Bulk variant deferred — single-row
fix is sufficient at typical drain depths.
Closes review findings perf-2, q-3, bug-3.
The watch dispatch closure's soft-cap pre-check materialised the whole
queue snapshot via ``pending(channel="any")`` only to throw away the
text and count the type — wasteful at typical drain depths (cap-50 +
mixed producers means a 50-tuple allocation per fire just to read a
length). The other half of the cap pair (``drop_oldest_by_type``)
walked the *whole* queue regardless of channel, so a future producer
that enqueued ``"watch_triggered"`` on a different channel could be
dropped by the watch cap, and vice versa — silently surprising once
that producer existed.
Adds ``NudgeQueue.count_by_type(nudge_type, channel=None) -> int`` that
walks ``_items`` once under the queue lock without materialising
tuples; extends ``drop_oldest_by_type`` to take an optional ``channel``
filter so both halves can agree on the entry set being capped. The
watch dispatch closure now passes ``channel="any"`` to both —
consistent with where the closure enqueues — so a future channel split
can't bleed across producers.
Adds ``TestCountByType`` mirroring the existing ``TestDropOldestByType``
shape, plus a ``test_drop_oldest_by_type_channel_filter`` case pinning
the new optional argument's behaviour.
Closes review finding q-1.
The live-marker scaffold in ``tests/test_watch_live.py`` couldn't actually
run as written: the ``live_client`` / ``live_model_id`` fixtures it
referenced live in ``tests/test_server_live.py`` at ``scope="module"``,
not on a shared ``conftest.py``, so the file would have ImportError'd
at collection if anyone ever tried ``pytest -m live`` against it.
Lifting the fixtures into a shared conftest is a larger refactor
than R9 justifies — the deterministic envelope-arrival contract is
already pinned end-to-end by ``test_watch_fires_then_user_send_drains_envelope``
and ``test_three_back_to_back_watch_fires_drain_into_one_turn`` in
``test_watch_integration.py`` (real ChatSession + real WatchRunner +
real chat-loop drain). The model-quality-of-response leg is genuinely
manual; the plan doc's R9 entry is updated locally to reflect that
deferral.
Closes review finding bug-1.
The shared ``sanitize_payload`` regex preserved TAB/LF/CR so multi-line
watch shell output kept its layout — necessary for the watch path, but a
correctness gap for the idle_children formatter, which renders the
user-controlled ``name`` field as a single bullet item. A child name
with an embedded ``\n`` would split the bullet across two rendered rows
and let a hostile name forge a fake sibling entry in the listing.
Splits the regex in two: ``_NAME_CONTROL_CHARS`` strips TAB/LF/CR
(used by the new ``sanitize_name`` helper for single-line name fields),
``_PAYLOAD_CONTROL_CHARS`` keeps the existing permissive shape (used by
``sanitize_payload`` for multi-line watch payloads).
``format_idle_children_nudge`` now calls ``sanitize_name``.
Adds ``test_newline_in_name_does_not_forge_extra_bullet`` — feeds a
hostile name with embedded ``\n`` + bullet-shaped continuation, asserts
the rendered listing still has exactly N bullet rows for N children
(no forged sibling), and the hostile newline got flattened to an inline
space. Adds a ``TestSanitizeName`` class mirroring the existing
``TestSanitizePayload`` shape for the new strict variant.
The deleted comment claimed the closure may be registered "under the
rehydrated workstream's id, which may differ from the original ws_id we
restored against" — but ``ChatSession.resume(ws_id, fork=False)`` adopts
the parameter as the session's id at session.py:1682, so they match
exactly post-resume. The lookup works because the ids are equal, not
because they may differ.
The accessor name ``get_dispatch_fn`` is self-explanatory; no replacement
comment is needed (per the project's "default to no comments" rule).
Adds two boundary-crossing integration tests and one live-marker
scaffold for the watch switchover landed in the previous commits:
tests/test_watch_integration.py — drives a real ChatSession + real
WatchRunner end-to-end (LLM stubbed) through the unified pull-model
chat-loop drain seam. Pins:
- test_watch_fires_then_user_send_drains_envelope: a synchronous
WatchRunner.dispatch fire enqueues "watch_triggered" on "any";
session.send drains the entry into the user message's _reminders
side-channel — confirms the envelope splice path.
- test_three_back_to_back_watch_fires_drain_into_one_turn: pins the
intentional behavioural delta from the plan section 3.4 / risk
register R3 — N back-to-back fires now produce ONE assistant turn
with N _reminders entries, not N successive turns.
tests/test_watch_live.py (new file, single test, marked @pytest.mark.live):
risk register R9 verification recipe — confirm a real LLM handles a
<system-reminder>-framed watch payload sensibly. Collects under the
regular -m "not live" run; the user runs it on demand against an
Anthropic-backed config.
Implements watch-switchover plan section 5.2 (integration) and step 11
(live scaffold).
Replaces the deleted tests/test_watch_dispatch.py with a focused
14-test suite exercising the closure that ChatSession.set_watch_runner
now constructs (per the previous commit's switchover). Each test
pins one assertion:
- enqueue shape: ("watch_triggered", text, "any") on the per-session
NudgeQueue; not on user / tool channels
- producer-side sanitisation strips control / bidi / zero-width chars
and angle-bracket tag breakers; preserves TAB/LF/CR so multi-line
shell output keeps its layout (R8); empty-after-strip → no enqueue
- soft-cap drop-oldest at _WATCH_QUEUE_SOFT_CAP with a queue_full
WARNING log; non-watch entries on the same queue are not collateral
damage
- valid_until predicate drops on inactive / missing / storage-raises;
delivers when active (counter-test)
- concurrent enqueues across two threads stay bounded under the
3-acquisition count-then-drop window
Implements watch-switchover plan section 5.1 / step 9. No production
changes — pure test rewrite.
Replaces the bespoke _make_watch_dispatch / _watch_pending /
_dispatch_pending_watch / _MAX_WATCH_CHAIN machinery with a single
NudgeQueue.enqueue("watch_triggered", ...) call inside
ChatSession.set_watch_runner. Watch results now drain at the same
<system-reminder> envelope seams as every other metacog nudge
(USER_DRAIN, TOOL_DRAIN, IdleNudgeWatcher IDLE wake) — no separate
worker-spawn, no recursive watch chain, no per-session queue.Queue.
The dispatch closure built inside set_watch_runner carries:
- producer-side sanitize_payload over the whole formatted message
before enqueue, so steering-vector / control-char shell output
can't tamper with the envelope at interpolation time
- a soft cap of 50 entries on per-session "watch_triggered" depth
via the new NudgeQueue.drop_oldest_by_type, replacing the prior
_watch_pending maxsize=20 + _MAX_WATCH_CHAIN=5 bounds; drop policy
is drop-oldest (latest output most useful), logged at WARNING
- a valid_until predicate that re-checks
storage.get_watch(watch_id)["active"] at drain time so a cancelled
watch's last splat doesn't ride out a future wake
Behavioural delta documented in the plan section 3.4: N back-to-back
watch fires now drain into ONE assistant turn responding to all N
(via the envelope splice) instead of N separate send turns. This is
intentional — fewer model invocations for noisy watches, and uniform
with the rest of the metacog pull-model surface introduced by #482.
Implements watch-switchover plan steps 5-8. Server-side simplifications
let the previously-load-bearing _make_watch_dispatch (47 lines), its
session_worker.send import, and the chat-loop _dispatch_pending_watch
seam at the no-tools IDLE branch all disappear. The obsolete
tests/test_watch_dispatch.py and the wake-tag test in test_session.py
(both pinning contracts that no longer exist) are removed; the
NudgeQueue-based replacement plus an integration test land in the
following commit.
Widens the per-workstream dispatch fn signature from ``(message,)``
to ``(message, watch_id)``. The runner now passes the originating
``watch_id`` through ``_dispatch_result`` so dispatch closures can
capture per-watch metadata at fire time — the upcoming switchover
needs this for the ``valid_until`` predicate that re-checks
``storage.get_watch(watch_id)["active"]`` before a stale entry rides
out a wake.
Also adds ``WatchRunner.get_dispatch_fn(ws_id)`` as the public
accessor used by the server-side restore path to retrieve the
closure that ``set_watch_runner`` constructed during workstream
rehydrate (avoiding private-attr access into ``_dispatch_fns``).
Implements watch-switchover plan step 4 plus risk register R4.
The pre-existing single-arg callers (``_make_watch_dispatch`` and
``set_watch_runner``'s ``dispatch_fn=`` fallback) get replaced
in the next commit; their mypy types are ``Any`` today so the
type mismatch isn't caught at this step.
Renames _sanitize_child_name to sanitize_payload and widens it to be
the shared producer-side sanitiser for both idle_children and the
incoming watch_triggered nudges. The regex now skips TAB / LF / CR
so multi-line shell output rendered into a watch payload keeps its
line structure when sanitised as a whole formatted message — the
pre-switchover code path collapsed multi-line output to one line.
Adds the watch_triggered entry to _NUDGE_MAP alongside idle_children
so ``_NUDGE_MAP``-as-registry consumers (should_nudge gating, future
audit / UI tagging) recognise the type. Body is empty — payload
comes from the producer (the watch dispatch closure), same shape as
idle_children.
Implements watch-switchover plan section 3.2 plus risk register R8
(TAB/LF/CR exclusion) and step 3 (_NUDGE_MAP registration).
Adds an atomic drop-oldest-by-type operation to NudgeQueue used by
producers that need a per-type soft cap on their own queue depth.
The watch dispatcher (next commit in this stack) is the first user:
when "watch_triggered" saturates, the dispatch closure drops its
oldest entry under the queue lock so the count snapshot and drop
can't interleave with a concurrent enqueue from the same producer.
Implements watch-switchover plan section 3.1 — the producer-side soft
cap takes the place of the deleted _watch_pending maxsize=20 bound.
Other producers (idle_children, advisories) have natural rate limiters
already, so the helper is opt-in per producer rather than a global cap
in enqueue itself.
Three Copilot findings on PR #483 (commit dad98c0); one rejected as a
false positive.
- mcp_client.py:1189 — pool notification handler's exception path
used ``log.warning(..., exc_info=True)`` which serializes the
chained ``httpx.Request.headers`` carrying ``Authorization: Bearer
<token>`` into Sentry / faulthandler frame captures. Same threat
model as the round-1 sec-1 dispatch-path fix, applied to a site
the original review missed. Now logs structured fields only
(server, user, exc type) without ``exc_info``.
- mcp_client.py:1202 — ``_connect_one_pool``'s handshake step used
``asyncio.wait_for(session.initialize(), ...)``, the same Python
3.11 + anyio cross-task-cancel-scope anti-pattern that the
Phase 7 round-3 q-1 fix removed from the discovery step (and that
f6a3b66 originally addressed for ``_safe_close_stack``). Pre-
existing Phase 5 code, but the same latent bug class — a 401
during initialize() under 3.11 would surface ``RuntimeError:
Attempted to exit cancel scope in a different task`` as the
SDK's TaskGroup unwinds. Switched to ``async with asyncio.timeout(...)``
matching the discovery step's pattern.
- mcp_client.py:1522 — renamed loop tuple-unpack variable
``_server_name`` → ``server_name`` in ``_rebuild_user_tool_map``.
The leading underscore conventionally signals "intentionally
unused", but the variable is read at the assignment a few lines
below. Two other ``_server_name`` unpacks in this file (1410,
3111) genuinely don't use the value and keep the underscore.
Rejected as false positive:
- test_mcp_user_catalog.py:58 (github-code-quality bot, "Statement
has no effect"): ``await task`` inside ``contextlib.suppress(
BaseException)`` is the standard pattern for cleanly draining a
cancelled task. The bot's static analysis treats ``await`` of a
result that's discarded as a no-op statement, but ``await`` here
triggers cancellation propagation and waits for the task to
finish — load-bearing in the fixture's teardown. No change.
Verified on Python 3.11 (``/tmp/venv311``) and 3.13 (``.venv``):
ruff + mypy clean, full test suite green.
Light up production reachability of pool dispatch (RFC §3, invariant 8)
by widening the public catalog API to optionally take a ``user_id``:
- ``MCPClientManager.get_tools(user_id=None)`` returns the merged
static + per-user pool view when ``user_id`` is supplied; the default
preserves the legacy global-only contract.
- ``is_mcp_tool(name, *, user_id=None)`` extends the lookup to the
per-user ``_user_tool_map``. Pool tools become reachable from
``ChatSession._prepare_tool`` only when the session-bound user_id
flows through — flipping invariant 8 from "must hold" to "satisfied".
- Listener identity becomes ``(user_id, callback)``. Static-path
changes fire ALL listeners (admin + every user); pool-entry
changes fire only matching-user + admin (``None``) listeners.
RFC §3.3.
- Pool sessions discover their tool list on first connect
(``_connect_one_pool`` → ``await session.list_tools()``); the
notification closure binds to ``(user_id, server_name)`` so
push-driven ``list_changed`` updates target the correct user's
catalog. R6 verified empirically: ``list_tools()`` 401 propagates
through anyio TaskGroup unwinding, no hang — plain ``await`` is
fine, no carrier-race shape needed for discovery.
- ``_evict_session`` drops ``entry.tools`` and rebuilds the user's
index so an evicted-then-reconnected session doesn't carry
stale catalog state.
- ``web_search.resolve_web_search_client`` refuses
``auth_type=oauth_user`` backends (per-node web search can't
carry per-user tokens).
Resources / prompts pool dispatch deferred to Phase 7b — invariant 8
is satisfied by the tool path alone, and the resource/prompt path
needs sibling ``_dispatch_pool_resource_sync`` /
``_dispatch_pool_prompt_sync`` helpers each with their own
carrier-race plumbing (~400 LOC). Phase 7b will follow the patterns
established here.
CLI sessions default ``user_id=""`` and so cannot use oauth_user
MCP servers — documented limitation; users must use the web UI.
Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: get_tools(user_id) was iterating _user_pool_entries from sync
threads while the mcp-loop concurrently mutated it (RuntimeError:
dictionary changed size during iteration). Now reads from a sibling
_user_tools dict updated atomically by _rebuild_user_tool_map.
- bug-2: _close_pool_entry_if_idle (LRU/TTL eviction) skipped the
catalog cleanup that _evict_session does — stale tools persisted
in _user_tool_map and ChatSession's tool list never rebuilt. Now
mirrors _evict_session.
- perf-1: _last_pool_notification_refresh debounce dict was never
pruned in either eviction path. Now popped alongside the entry.
- perf-3: web_search resolver was issuing a sync SQL query per LLM
turn to gate oauth_user backends. Now reads from the cached
in-memory config.
- sec-1: bearer token could leak into exc_info-rendered tracebacks
via Sentry/faulthandler. log.debug now uses structured fields,
not exc_info.
- sec-2: tools-per-server response now capped at 1000 (defensive,
mirrors _MAX_ERROR_LEN / _MAX_INSUFFICIENT_SCOPE_REPORTED).
- Test cleanup: dropped two listener fan-out tests duplicating
test_mcp_client.py coverage; renamed test_pool_session_notification_handler
to match its actual scope (_refresh_pool_server_tools); removed
stale comments referencing /tmp/r6-spike*.py scratchpads and a
misleading "copy-on-write" comment.
Round-2 pre-push review fixes (focused single-pass review applied):
- round2-1: bug-2's catalog-cleanup block in _close_pool_entry_if_idle
had no integration test (exactly the failure mode flagged in
feedback_tests_through_boundaries.md). Added
test_close_pool_entry_if_idle_clears_catalog_and_fires_listener
driving the LRU/TTL eviction path through real streamablehttp_client +
MockTransport. Negative-test verified: reverting the
_rebuild_user_tool_map / _notify_user_tool_listeners calls makes
the new test fail.
- round2-3: documented the _oauth_user_server_names cache invariant
in add_server_sync / remove_server_sync docstrings. Cache is
reconcile_sync's sole owner — direct callers leave it stale, but
_db_servers_to_config strips oauth_user rows so production paths
are unaffected. Static→oauth_user transitions correctly leave the
name in the cache because remove_server_sync drops the static
connection, not the cache identity.
- round2-6: strengthened test_rebuild_user_tool_map_populates and
test_rebuild_user_tool_map_drops_empty_user to assert on the
_user_tools sibling cache (bug-1 fix). Without this, a future
revert dropping the sibling write would still pass the unit
tests because get_tools coverage lives in separate tests.
Round-3 full-stack review fixes (multi-stage review on the final
state caught what the layered apply passes missed):
- q-1 REGRESSION: pool tool-discovery used asyncio.wait_for around
session.list_tools(), the exact pattern the f6a3b66 fix (and
feedback_asyncio_timeout_vs_wait_for.md) put in place to avoid.
Python 3.11's asyncio.wait_for wraps the inner coroutine in a
fresh task → cross-task scope-exit when the SDK's anyio TaskGroup
unwinds on a 401. Switched to `async with asyncio.timeout(...):`
pattern used by _safe_close_stack.
- sec-2: TOCTOU in _connect_one_pool — entry.tools was published
(via _rebuild_user_tool_map + listener fan-out) BEFORE entry.session
was assigned. A sync-thread reader could observe a tool whose
backing entry has session=None. Defence-in-depth — dispatch
re-fetches its own token and lazy-reconnects on session=None — but
reordering catches the race at the source. entry.session now
publishes BEFORE catalog visibility.
- bug-1: _close_pool_entry_if_idle's _user_pool_locks.pop ran
unconditionally after the try/finally, but the early-return
branches (entry None on re-check, in_flight > 0 under lock) skip
it via Python's return-through-finally semantics. The lock was
never popped on those paths. Now gated behind an `evicted` flag
set only on the success path; in_flight > 0 leaves the lock for
the active dispatcher to reuse, entry-None races leave the lock
for re-allocation by _ensure_pool_entry. Comment now describes
the actual semantics, not the original promise.
- bug-2: softened the _rebuild_user_tool_map docstring's atomicity
claim. The two-dict write is technically non-atomic across Python
statements; in practice the window is sub-microsecond on the
mcp-loop with no awaits between writes, and the listener fan-out
fires AFTER both writes complete. Docstring now says "back-to-back
on the mcp-loop" instead of "atomically alongside".
- q-3: dropped `hasattr(mcp_client, "server_auth_type")` defensive
check in web_search.py. The method ships in this commit; the
hasattr created a silent fallthrough that would let a future
rename silently re-enable oauth_user backends.
- q-4: surfaced the CLI / empty-user_id limitation in a docstring
comment at ChatSession.__init__'s self._user_id assignment. The
note previously lived only inside is_mcp_tool's docstring — a
future maintainer wiring CLI features against MCP pool servers
wouldn't think to read is_mcp_tool to find the constraint.
- q-2 + q-5: deleted a tautological duplicate test in
test_mcp_user_catalog.py whose docstring claimed to test
ChatSession.close but never instantiated a ChatSession (the
manager-level identity semantics are already covered by
test_listener_identity_includes_user_id in the same file and by
test_session_close_removes_listener_with_same_user_id in
test_mcp_client.py which DOES drive a ChatSession). Reworded a
misleading "fixture provides only 5s" comment to point at the
actual `_run_on_loop(..., timeout=5)` site.
- q-6: the `self._user_id or None` collapse repeated at 8 sites
across session.py. Cached once at __init__ as
``self._mcp_user_id`` (since ``_user_id`` is set once and never
mutated); 8 call sites now read the cached value. The empty-
string-is-CLI-sentinel invariant is documented at the assignment
site, not re-asserted at each consumer.
Deferred to follow-up:
- sec-1: a hostile MCP server bound to user-A could craft a
tool.name containing `__` to synthesize a prefixed-name collision
in user-A's own catalog. Bounded impact: cross-tenant dispatch is
prevented by the per-tenant token gate in _dispatch_pool, and
user-B's get_tools(user_id="B") never includes user-A's pool
entries. The fix needs policy decisions (reject vs. sanitize)
and touches _mcp_to_openai which is shared between static and
pool paths; better discussed in its own follow-up where the
policy applies uniformly to static-path servers too. The threat
model already requires user-A to have consented to a malicious
server, who has many more dangerous vectors than tool-name
shenanigans.
Test count delta: +31 tests (5435 → 5466, ``-m "not live"``; one
test deleted in round-3 apply per q-2):
- ``tests/test_mcp_client.py`` +20 (per-user catalog state, listener
identity, session thread-through)
- ``tests/test_mcp_user_catalog.py`` +9 NEW (integration tests
driving real ``streamablehttp_client`` + ``httpx.MockTransport`` per
invariant 14: discovery on connect, user isolation, eviction +
reconnect, LRU/TTL eviction (round2-1), R6 401-propagation
regression, static byte-identical canonical regression; review
passes dropped duplicate listener fan-out tests from earlier
drafts whose coverage lived in test_mcp_client.py)
- ``tests/test_web_search.py`` +2 (oauth_user backend rejection +
static backend acceptance regression; updated to use the new
``server_auth_type`` in-memory accessor)
Three confirmed findings from the PR #482 bot review pass.
* **Copilot (idle_nudge_watcher.py)**: ``IdleNudgeWatcher`` was gating
wake dispatch on ``len(_nudge_queue) == 0`` (any channel), but
``deliver_wake_nudge_from_queue`` only drains ``USER_DRAIN``. A
``"tool"``-channel entry queued by ``_queue_tool_advisory`` would
pass the gate, spawn a wake daemon, and immediately no-op at the
drain guard — repeating on every IDLE event for as long as the
tool entry sat unconsumed. No correctness bug (the no-op return
prevents bad state) but a wasted thread spawn per IDLE. Fixed by
gating on ``has_pending(USER_DRAIN)``; tool-only queues no longer
trigger the wake path.
* **Copilot (coordinator_idle_observer.py)**: docstring referenced
the old module path ``turnstone.core.metacognition.IdleNudgeWatcher``;
the class moved to ``turnstone.core.idle_nudge_watcher`` in q-3 of
the apply-pass.
* **Copilot (nudge_queue.py)**: ``has_pending`` docstring cited
``ChatSession.deliver_wake_nudge_from_queue`` as its caller, but
that method calls ``drain(USER_DRAIN)`` directly — no production
caller used ``has_pending`` until this commit. Updated to point
at the now-actual caller (``IdleNudgeWatcher``).
* **github-code-quality (test_nudge_queue.py)**: false positive on
``test_channel_is_required`` — the no-channel ``q.enqueue("a", "1")``
call is wrapped in ``pytest.raises(TypeError)`` to verify the
validation contract. No code change.
5571 non-live tests pass; ruff + mypy clean.
Round-2 review caught 11 confirmed findings on the 3-commit metacog stack;
this commit applies them.
* **bug-1 (major)**: Wake source tag was leaking onto real user messages
flushed during a wake send. ``_append_user_turn`` and ``send`` now
take an explicit ``from_wake: bool`` parameter — only the wake's
synthesized first turn passes True, so ``_flush_queued_messages``'s
real user input no longer inherits the audit tag. Regression test
pins the contract.
* **perf-1 (major)**: ``CoordinatorIdleObserver._maybe_enqueue`` was
issuing list_workstreams + visible_memory_count storage queries
before the cheap cooldown gate could short-circuit. New
``_cooldown_allows`` read-only peek runs first; storage queries only
fire when cooldown actually allows the nudge.
* **q-1 (major)**: Added the missing coord-side integration test that
exercises ``CoordinatorIdleObserver`` + ``IdleNudgeWatcher`` together
in the production install order against a real ``SessionManager``,
protecting the subscription-order contract from silent regression.
* **perf-2/3 (minor)**: Cap check moved above ``_last_assistant_used_wait``;
``_fire_counts`` restructured as ``dict[str, dict[str, int]]`` keyed by
ws_id so the leave-IDLE existence check is O(1).
* **perf-4 (minor)**: ``NudgeQueue.drain`` fast-paths the all-match
case (the common one for chat-loop drain seams) by swapping
``self._items`` directly instead of allocating a fresh ``kept``
deque + per-entry append.
* **perf-5 (minor)**: Wake's synthesized empty user turn no longer
writes a content-empty row to the conversations table — the
``_source`` audit tag isn't column-backed and the side-channel
reminder is stripped before persist, so the row would carry nothing.
* **q-3 (minor)**: Split ``IdleNudgeWatcher`` + ``install_*`` /
``shutdown_*`` helpers out of ``metacognition.py`` into the new
``turnstone/core/idle_nudge_watcher.py``; metacog stays a
static-template module.
* **sec-1 (nit)**: Widened ``_sanitize_child_name``'s control-char
regex to cover Unicode bidi-overrides, zero-width chars,
line/paragraph separators, BOM, and tag chars.
* **q-4/q-5 (nits)**: Docstring referenced the wrong peek primitive
(``has_pending`` → ``len()``); ``_last_assistant_used_wait``'s
``session`` parameter now typed ``ChatSession``.
5571 non-live tests pass; ruff + mypy clean.
Adds the first concrete consumer of the wake trigger: when a coordinator
goes IDLE while interactive children are still running, a
``CoordinatorIdleObserver`` enqueues an ``idle_children`` nudge that the
``IdleNudgeWatcher`` then dispatches as a synthetic empty-user-turn
``send``. The model receives a system-reminder body listing the active
children (capped at 6 inline + 32 in the suggested ``wait_for_workstream``
call) and a nudge to block on them rather than reply prematurely.
Observer gates (in order): coord-only filter, skip if last assistant
turn used ``wait_for_workstream``, per-(ws, nudge_type) hard cap (3)
that resets only on non-wake leave-IDLE, active-children query,
``should_nudge`` cooldown. Console lifespan registers the observer
BEFORE the watcher so subscriber-fire order has the observer
enqueueing first on the same IDLE event.
Adds an opt-in ``valid_until`` predicate on ``NudgeQueue.enqueue``
(R9 from the design risk register) — drain re-checks the predicate
outside the queue lock; falsy / raising drops the entry without
delivering it. ``deliver_wake_nudge_from_queue`` now drains inline
before synthesizing the empty user turn so a stale predicate-drop
doesn't leave the wake send with empty content; ``_attach_pending_user_reminders``
consumes the pre-drained reminders via ``_wake_drained_reminders``.
The observer's ``valid_until`` uses ``count_workstreams_by_state``
(boolean check, no row fetch) instead of full ``list_workstreams``,
keeping the chat-loop user-attach path off the heavy query.
User-controlled child workstream names are sanitized
(``_sanitize_child_name``) before interpolation so a name like
``</thinking>...`` can't steer the model's reasoning channels through
the rendered body — the wire-boundary ``escape_wrapper_tags`` only
covers ``<system-reminder>`` / ``<tool_output>`` envelopes.
Adds the third metacog channel: an out-of-band wake that converts a
workstream's IDLE transition into a synthetic empty-user-turn ``send``
when the session has any-channel nudges queued. The ``IdleNudgeWatcher``
subscribes to ``SessionManager.subscribe_to_state``; on IDLE it dispatches
via ``session_worker.send`` with a no-op ``enqueue`` callback so a
busy-worker race silently drops without spawning a competing worker.
Wake-source-tag plumbing on ``ChatSession`` short-circuits metacog
detection on the synthetic empty input, suppresses queue producers
during the wake's own tool dispatch, and stamps ``_source = "system_nudge"``
on the synthetic user-message for audit / replay distinction. The tag
is saved / restored across ``_dispatch_pending_watch`` so watch chains
recursing off the wake are processed as normal user turns rather than
inheriting the wake's guards.
Generic ``install_idle_nudge_watcher`` / ``shutdown_idle_nudge_watchers``
helpers wire the watcher into both the interactive and coord lifespans
via a single ``app.state`` registry so both surfaces share the same
teardown contract.
Foundation for PR 3 (CoordinatorIdleObserver + idle_children formatter)
and PR 4 (watch dispatcher switchover).
Replaces the dual `_pending_user_advisories` / `_pending_tool_advisories`
list pair with a single channel-tagged `NudgeQueue` per session.
Producers tag entries with a channel ("user", "tool", or "any");
consumers drain by channel filter at their existing seams. Foundation
for the wake trigger (PR 2) and coordinator idle-children nudge (PR 3).
Existing nudges (start, correction, completion, denial, resume,
tool_error, repeat) keep their wire shape and drain timing — zero
behavior change. Cancel paths now `clear()` the unified queue.
Python 3.11's ``asyncio.wait_for`` wraps its inner coroutine in a fresh
``asyncio.Task`` via ``ensure_future``. When the inner is
``stack.aclose()`` on an ``AsyncExitStack`` containing
``streamablehttp_client(...)`` (anyio cancel scopes entered in the
calling task), the fresh task's attempt to exit those scopes raises
``RuntimeError('Attempted to exit cancel scope in a different task
than it was entered in')``. Python 3.12+ rewrote ``wait_for`` to use
``asyncio.timeout`` internally — runs in the current task — so 3.13
ran the same code path successfully.
Symptom on 3.11: integration tests where ``session.initialize()``
returns 4xx (e.g., 403 insufficient_scope tests) hit
``_connect_one_pool``'s ``except Exception:`` handler →
``_safe_teardown_on_connect_failure`` → ``_safe_close_stack`` → cross-
task RuntimeError. The ``concurrent.futures._base.CancelledError``
that surfaces in ``future.result(timeout=...)`` is the cascade
fallout from the asyncio loop's exception handler reacting to the
unretrieved-task-exception.
Fix: use ``asyncio.timeout`` instead of ``asyncio.wait_for`` for the
5s aclose bound. Equivalent semantics, current-task execution, works
on 3.11+. The 5s guard against ``aclose()`` hanging on a broken stack
is preserved.
Verified on Python 3.11.14 (full suite 5427 passed) and 3.13.7 (full
suite 5427 passed); all 9 integration tests pass on both.
Pre-existing bug — surfaced only after the marker fix in 5c9850c
let CI's test (3.11) actually run the 4xx tests.
Two pre-existing defects in the Phase 6 pool dispatch path that only
manifest when a pooled session is reused for a second dispatch:
1. The per-dispatch _AuthCapture allocated in _dispatch_pool was wired
into the httpx response hook only at first connect (via
_connect_one_pool). On a reused session no fresh connect runs, so
the hook continues writing to the original-connect's carrier while
the new dispatch inspects an empty carrier — auth_401/403 silently
misclassified to "other", refresh-and-retry never fires.
2. Even with the carrier on the entry (so the hook writes to a stable
reachable object), session.call_tool itself hangs forever on
upstream 4xx for reused sessions. Trace: SDK's spawned
handle_request_async raises HTTPStatusError, the outer
streamablehttp_client TaskGroup cancels post_writer, post_writer's
finally aclose's read_stream_writer, BaseSession's _receive_loop
exits and enters its CONNECTION_CLOSED-fanout finally. anyio's
send_nowait skips waiting receivers with pending_cancellation; the
dispatch task (created by run_coroutine_threadsafe for the reuse
case) is NOT in any cancel-scope chain, so the send "delivers" but
the receiver's Event is set on stale state — receive() never
wakes. Test 21 doesn't hit this because its 401 happens during
initialize, in the same task that opens streamablehttp_client, so
the cancel scope DOES propagate.
Fix:
- Move _AuthCapture ownership to PoolEntryState (and asyncio.Event
alongside, allocated lazily on the mcp-loop). The hook closes over
entry.auth_capture at first connect and stays valid across
dispatches; reset under open_lock before each call_tool.
- Race session.call_tool against the carrier's fired_event in
_dispatch_pool_with_entry. If the event wins (hook captured 4xx
before SDK propagated), cancel call_tool and raise an internal
_CarrierAuthSignal — _classify_failure resolves to auth_401/403
via the carrier's status, the dispatcher evicts the broken
session, and the cross-task retry handshake reconnects on a fresh
bearer.
Adds tests/test_mcp_pool_auth_integration.py::test_integration_pool_reuse_401_refresh_and_retry_succeeds
which drives the reuse path through real upstream + real SDK and is
the structural gate against this class regressing. Negative-tested
twice: revert PoolEntryState.auth_capture → test fails (carrier
empty); revert the race → test times out (SDK hang).
Also drops the @pytest.mark.asyncio decorator (replaced with
@pytest.mark.anyio) on four tests in test_mcp_pool_auth_introspection.py.
The project depends on anyio's pytest plugin (anyio is in deps);
pytest-asyncio is NOT a project dep and CI's test (3.13) failed on
those four. Local pytest happened to pick it up via system Python.
Found via Copilot review on PR #481.
Phase 6 of OAuth-MCP. Recovers upstream 401/403 from MCP servers via a
capturing httpx_client_factory: an async response hook records 4xx
status + WWW-Authenticate header into a per-dispatch carrier before
the SDK's post_writer swallows the underlying httpx.HTTPStatusError.
Splits _classify_failure into auth_401 (refresh-and-retry once) vs
auth_403 (parse insufficient_scope, emit mcp_insufficient_scope with
parsed scope set). The 401 retry runs on a fresh asyncio.Task via
run_coroutine_threadsafe in _dispatch_pool_sync, escaping the anyio
cancel-scope state of the prior dispatch's TaskGroup.
WWW-Authenticate parsing extracted to a new mcp_http_parsers module
with an RFC 7235 challenge tokenizer (replaces hand-rolled substring
scanners). Two-layer defense against multi-Bearer-challenge injection:
the hook uses get_list("www-authenticate")[0] to drop attacker's
second challenge, the parser truncates at challenge boundary as
belt-and-braces. Scope set capped at 32 entries before hitting the
audit row or the LLM-visible structured-error JSON.
Auth failures (401/403) never trip the per-server circuit breaker
(server-only breaker invariant). Static path remains byte-identical.
_PgRefreshLock untouched. Pool dispatch still reachable from the
agent loop only via Phase 7 catalog scoping; Phase 6 behaviour is
testable via direct call_tool_sync.
5557 tests pass. 33 tokenizer unit tests in tests/test_mcp_http_parsers
cover the RFC 7235 grammar + the scope/error wrappers + the 4 KB input
cap. 7 integration tests in tests/test_mcp_pool_auth_integration drive
real upstream 401/403 through streamablehttp_client + a FastMCP
subprocess fixture — the structural exit gate that makes
HTTPStatusError-injection-only unit tests insufficient.
Models often emit page references in the standard man-page form
(``printf(3)``, ``open(2)``, ``perlfunc(3pm)``) rather than splitting
them into ``page`` + ``section`` args. The page-name sanitizer was
rejecting the parens as invalid input, killing the call. Parse the
section out of the page string before sanitization (explicit
``section`` arg still wins) and widen the section validator to accept
multi-letter suffixes like ``3pm`` / ``3perl`` that already appear on
real systems.
Phase 5 PR #479 review fix-up. Three review rounds (bot + two internal
multi-stage /review) caught:
- _PgRefreshLock now allocates a per-instance ThreadPoolExecutor instead of
a module-global single-worker one. The global shape preserved psycopg2
thread-affinity but serialized every advisory-lock acquire on the node
behind one thread, even for unrelated (user, server) keys.
- get_user_access_token_classified flips to `async with lock, pg_lock:` so
concurrent same-key callers serialize on the in-process asyncio.Lock
before allocating the pg_lock's per-instance executor + spin loop. N
concurrent same-key callers collapse to one executor allocation.
- _drain_orphan_pg_lock no longer re-awaits the cancelled asyncio Future
from `__aenter__`. It receives the underlying concurrent.futures.Future
and re-wraps it via asyncio.wrap_future, getting an independent asyncio
Future tied to the worker outcome. This way cancellation of the awaiter
doesn't poison the drain's wait, and the drain genuinely waits for the
worker to settle before deciding whether to call cm.__exit__.
- Module-level _pg_refresh_drain_tasks set holds strong refs to in-flight
drains (asyncio's task set is weak — fire-and-forget tasks could be GC'd
mid-cleanup; RUF006 hazard).
- Drain narrows except clauses to Exception so a drain-task cancellation
records as cancelled instead of being silently logged as 'completed
normally with no acquire'.
Test integrity (was a major finding in round 2 — old generator-based cm
let the test pass via GC finalization timing rather than drain logic):
- New _ObservableLockCm class-based context manager whose __exit__ is a real
observable method (records call args + thread). Distinguishable from
GeneratorExit thrown by GC of a generator-based cm.
- Strong external ref to the cm via created_cms list — keeps cm alive past
the test's awaits, so a no-op drain genuinely fails the assertion rather
than papering over via GC timing.
- Deterministic drain wait via _pg_refresh_drain_tasks gather — no
fixed-duration sleeps.
- _run_cancel_scenario helper drops the duplicated setup between the two
cancellation tests.
Negative-test verified: replacing _drain_orphan_pg_lock body with `return`
makes test_pg_refresh_lock_cancellation_releases_on_same_thread fail with
'drain did NOT call cm.__exit__ — orphan Postgres lock + open transaction'.
Other fixes: protocol docstring corrected to describe pg_try_advisory_xact_lock
spin + retry (was claiming pg_advisory_xact_lock blocking acquire);
get_user_access_token_classified docstring rewritten for new lock order;
narrow `except BaseException` -> `except Exception` in
test_mcp_user_pool.py concurrent-dispatch helper.
882 tests pass (MCP + auth + storage). ruff + mypy clean.
Phase 5 of OAuth-MCP — adds a per-(user, MCP-server) ClientSession
pool to MCPClientManager alongside the existing static-server path,
gated entirely on the per-server `auth_type='oauth_user'` config.
Pool architecture:
- `_user_pool_entries: dict[(user_id, server_name), PoolEntryState]`
with lazy connect on first dispatch, per-key asyncio.Lock allocated
on the mcp-loop, idle eviction coroutine (default 600s TTL, LRU cap
200), and an `in_flight` counter as the eviction interlock so live
calls can never be torn down mid-flight.
- `_dispatch_pool` runs the token-state machine: missing token →
`mcp_consent_required`; key-rotation decrypt failure →
`mcp_token_undecryptable_key_unknown` with NO consent prompt and NO
auto-delete; expired token → silent refresh under per-(user, server)
advisory lock; refresh failure → revoke + consent.
- `_classify_failure` separates transport (trips breaker) from auth
401/403 (does NOT trip breaker — server-only invariant) from
protocol (no breaker change).
- `entry.open_lock` held only across connect-or-reuse and released
before the `await session.call_tool` so concurrent calls from one
user against one server overlap (validated by Spike 1 scenario 2).
Auth-class failures are fail-soft in Phase 5: any 401/403 surfaced by
the SDK propagates to the agent as a tool error and the next dispatch
reconnects on a fresh refresh. Real introspection of upstream 401/403
is a Phase 6 concern — the MCP SDK's `streamable_http` post_writer
swallows `httpx.HTTPStatusError` upstream, so detecting status from
the response chain requires `McpError(CONNECTION_CLOSED)` payload
parsing or a custom httpx middleware around `streamablehttp_client`.
The mid-flight 401 refresh-retry path and the `mcp_insufficient_scope`
structured error for 403 step-up land together in Phase 6, gated by
an integration test that drives a real upstream 401/403 (the unit-
test injection of `HTTPStatusError` is what masked the production gap
on the first apply-findings pass — the integration test is the
structural gate so the gap can't reopen). RFC §1.5 steps 4-5 and the
phase table in §Implementation phases reflect this scope split.
Multi-node refresh contention:
- New `StorageBackend.acquire_advisory_lock_sync` Protocol method.
SQLite returns nullcontext (single-node, in-process asyncio.Lock
is sufficient). Postgres uses `pg_try_advisory_xact_lock` with
retry on a fresh per-attempt connection, so waiters don't pin pool
connections during the AS roundtrip. Inner try/except + nested
finally ensures conn is always returned to the pool, even when
begin / execute / yield / commit raises mid-body.
- Lock ordering: pg_advisory outer, asyncio.Lock inner. Re-read after
lock collapses cluster-wide contention to one HTTP roundtrip per
(user, server) per refresh window.
- `_PgRefreshLock` enter/exit pinned to a single-worker
ThreadPoolExecutor so SQLAlchemy connection state stays
thread-affine across cancellations.
Token storage refactor:
- `get_user_access_token_classified` returns a tagged TokenLookupResult
(Token / MissingToken / DecryptFailure / RefreshFailed) so the
dispatcher maps each state to the right user-facing error.
- `get_user_access_token` is now a thin wrapper around the classified
variant; the previous duplicated state machine is gone.
Security:
- Pool dispatch + admin endpoints reject `http://` URLs for
`auth_type='oauth_user'` servers (only exact loopback hostnames are
exempt — `*.localhost` is intentionally NOT honored because RFC 6761
localhost-zone resolution is configuration-dependent and could route
bearers to non-loopback IPs via custom resolvers / hosts file /
Docker overlays). Validated at three layers:
`_dispatch_pool` (structured `mcp_oauth_url_insecure` error),
`_connect_one_pool` (defensive ValueError), and
`admin_create_mcp_server` / `admin_update_mcp_server` (400 before
storage write).
- Admin URL change on an oauth_user row purges per-user OAuth tokens
bound to the old URL: bearers are bound (via OAuth resource /
audience) to the URL active at consent time, so silently rebinding
them to a new URL is a token-binding violation. Re-consent forces
fresh issuance for the new resource.
- Encryption-key fingerprints stay in audit logs only; no longer
surfaced in agent-facing error payloads.
User_id thread-through:
- `MCPClientManager.call_tool_sync(..., user_id=None)` (additive;
default None preserves the static path byte-identically).
- `ChatSession._exec_mcp_tool` passes `self._user_id or None`.
- `set_app_state(app_state)` setter wires OAuth state at lifespan
startup, called from both turnstone-server and turnstone-console.
Performance:
- LRU cap eviction iterates `_user_pool_entries` (not
`_user_pool_last_used`) so pre-dispatch entries are eligible.
- Eviction batch closes via `asyncio.gather` instead of serial await.
- `_resolve_pool_target` returns the resolved server row to
`_dispatch_pool` to eliminate the second DB lookup.
- Production reachability of pool dispatch is gated on Phase 7
(catalog scoping) wiring pool tools into `_tool_map`; until then
pool dispatch is reachable only via direct `call_tool_sync` with a
prefixed name (the path the new pool tests exercise).
Hardening parity preserved:
- Static path (auth_type ∈ {none, static}) byte-identical; PR #296
hardening (SDK #2147 mitigations, anyio cancel-scope, stale-session-
and-stack guard, server-only circuit breaker) intact.
- `test_reconnect_preserves_static_state_identity` unchanged + green.
- `MCPTokenStore.get_user_token` does not auto-delete on
MCPTokenDecryptError (key-rotation safety).
- Notification debounce stays manager-level.
- Connect-failure cleanup factored into
`_safe_teardown_on_connect_failure` shared by both connect paths.
Tests: 5475 → 5493 (+18). New file `tests/test_mcp_user_pool.py`
plus additions to test_mcp_oauth_refresh.py, test_mcp_admin_api.py,
and test_mcp_client.py covering: pool data structures, lazy connect,
eviction TTL + LRU + lock interlock, dispatch state machine (token
states), failure classification, http-rejection at dispatch and
admin layers, URL-change-purges-tokens (sec), concurrent dispatch on
one (user, server), pg_advisory lock parity, and user_id threading.
Phase exit criterion (synthetic load test 50 users × 3 servers × LRU
30 × 1000 calls × 200 evictions) deferred to a post-Phase-5 fitness
spike that runs against a staging deployment with real FDs and real
network behaviour, not a CI mock — same shape as Spike 1's
pre-Phase-0 SDK validation.
Out-of-scope for Phase 5 (Phase 6+): SDK-level 401 refresh-retry +
403 `mcp_insufficient_scope` (Phase 6), per-user catalog scoping
(Phase 7), consent UX SSE event + dashboard renderer (Phase 8),
admin UI status indicators (Phase 9).
Spike artifact validating MCP SDK behavior before Phase 5 builds the
per-(user, MCP-server) ClientSession pool. Three scenarios, all pass:
1. N=20 concurrent ClientSession instances against the same URL — no
FD blow-up, no shared transport state, each session's tools/list
returns independently.
2. Two concurrent tools/call on a shared ClientSession with
interleaving payloads — request_id demux works under contention.
3. Per-session Authorization header isolation across 5 sessions —
httpx connection pooling does not cross headers between sessions,
so per-session bearer tokens reach the server unmixed.
Outcome gates the Phase 5 architecture (lazy dict[(user_id,
server_name), ClientSession] + per-key asyncio.Lock + LRU eviction).
Had any scenario failed, the fallback was per-call header injection
(Alternative F in the OAuth-MCP RFC).
Spike-only — not collected by pytest. Run manually:
uv run python tests/spike_sdk_concurrency.py
Addresses ten findings on the Phase 4 OAuth-MCP commit: four from the
PR #478 review surface, plus six surfaced by a follow-up multi-stage
review of the first round of fixes. Two of the latter were genuine
security regressions in the very code that claimed to close those
holes.
Security
--------
- _validate_return_url now pins return_url same-origin against the
configured oidc_config.redirect_base instead of request.url. Behind
a permissive front proxy that did not normalise Host, an attacker
could spoof Host and provide a matching absolute return_url to mint
an open redirect off /api/mcp/oauth/start. Same fix pattern as
PR #476 OIDC.
- Reject return_url values containing literal backslashes or starting
with `//` up front. urlparse leaves backslashes inside `path`, so a
value like `/\evil.example/foo` slipped through the path-only branch
and became the protocol-relative `//evil.example/foo` after WHATWG-
conformant browsers normalised the backslash — re-introducing the
open redirect the same-origin pin was meant to close.
- internal_mcp_status (read-scoped) projects through a new
_strip_server_status_for_read helper that drops the verbose `error`
text and replaces it with a coarse `has_error` boolean. The error
string is built as `f"{type(exc).__name__}: {exc}"` and so carries
stdio binary paths (FileNotFoundError) or internal MCP URLs
(httpx.ConnectError) — equivalent to leaking command/url, which
this same patch deliberately strips. Approve-scoped refresh and
reconnect callers continue to receive the full `error` text via
the existing _strip_server_status helper.
- internal_mcp_status now returns the projected (sanitised) entries
for every server in mcp_mgr.get_all_server_status() instead of
emitting the un-sanitised dict that included `command` (stdio argv)
and `url` (remote MCP endpoint). Sibling refresh/reconnect endpoints
already used _public_server_status to strip these.
- internal_mcp_status docstring documents the trust boundary — server
enumeration to read scope is intentional so dashboards can render
per-server indicators; verbose error detail and command/url remain
approve-scoped.
Correctness / UX
----------------
- _validate_return_url comparison normalises (scheme, host, port)
before equality. Lowercases hostname and collapses the scheme's
default port, so `https://App.Example.COM/x` and
`https://app.example.com:443/x` are recognised as same-origin
with `redirect_base = https://app.example.com` instead of being
silently downgraded to the `/` fallback.
- mcp_crypto startup-gate error message now names both
`mcp_token_encryption_keys` (rotation list) and
`mcp_token_encryption_key` (single) so an operator using rotation
isn't misled into thinking only the singular form is valid.
Cleanup
-------
- Delete the unused _KNOWN_TRUSTED_ENDPOINT_HOSTS legacy re-export
shim in oidc.py (zero callers — a no-op that survived the Phase 4
oauth_ssrf extraction). Sphinx :data: docstring reference at
validate_discovered_endpoint updated to point at
turnstone.core.oauth_ssrf.KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS
directly. The Google multi-origin allowlist is unaffected — it
lives at the canonical name and is read from oauth_ssrf.py:164.
- test_mcp_oauth_handlers TestValidateReturnUrl imports
_validate_return_url at module level instead of repeating the
import inside each test method.
- test_server_lifespan_mcp_crypto replaces a fragile
`messages.count("mcp_token_encryption_key") >= 2` substring trick
with `re.search(r"mcp_token_encryption_key(?!s)", messages)` —
asserts the singular form directly via negative lookahead.
Tests
-----
5448 pass (+13 vs the prior tip):
- TestValidateReturnUrl gains backslash-bypass, protocol-relative,
default-port, uppercase-host, and explicit-port-mismatch cases
alongside the original same-origin / cross-origin / scheme-
mismatch / path-only cases.
- TestInternalMcpStatusEndpoint asserts the `error` text never
reaches the read-scope wire (binary-path FileNotFoundError no
longer appears anywhere in the rendered response) and that the
coarse `has_error` boolean lights up correctly on the failed
server.
- TestInternalMcpStatusEndpoint also pins the no-mcp-client path to
`{"servers": {}}`.
- _routes_with_internal extended to include the
/api/_internal/mcp-status route so the new tests can exercise it
through TestClient.
- Existing test_startup_aborts_with_oauth_user_row_and_no_key
strengthened to require both singular and plural key names appear
in the error log.
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.
Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
effective_port, sanitize_log_text moved out of oidc.py into a shared
oauth_ssrf module; oidc.py re-exports for compatibility. The shared
helpers also expose async wrappers (validate_url_no_ssrf_async,
validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
from async handlers — does not block the event loop on the
synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
DELETE...RETURNING)
Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
existing refresh value is preserved at the OAuth-flow layer rather
than cleared, so production ASes (Google, Auth0 default, Okta) don't
force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
MCP server URL, not the audience. Audience and resource are distinct
concepts; using audience as resource would mismatch the AS RS
allowlist.
Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
accepted_audiences = {server_url, oauth_audience} and validates
against the set, so Auth0-style ASes that honor `audience=` (not
RFC 8707 `resource=`) issue tokens that pass audience-bound
validation
- build_authorize_url emits both `resource=` (RFC 8707) and
`audience=` (Auth0-style) per server config; comment documents which
AS implementations need which form
Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
request Host header — closes the same Host-header injection PR #476
fixed for OIDC. Both /start and /callback return 503 with operator-
actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
inside the lock, so concurrent /start callers don't both register
and overwrite each other's client_id (the second user's code is no
longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
so a leaked state can't be replayed against a separately-obtained
code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
through sanitize_log_text before reaching exception messages or
audit details. AS error responses are parsed for the standard
RFC 6749 fields (error, error_description, error_uri), each
capped at 80 chars and run through redact_credentials to defend
against ASes that echo the request body back into their error
payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
tokens and pending OAuth state for the previous server name
(delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
mutable server_name; without this purge, a future server with the
same name (and an attacker-controlled URL) would silently rebind
prior user tokens. A future schema migration will replace the
server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
installed key can decrypt the row, e.g. after key rotation) and
falls through to None so dispatch surfaces a re-consent rather than
crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
path. Auto-connecting them at startup with empty headers fails the
AS check and trips the circuit breaker; per-user tokens come online
lazily once the user has consented.
Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
token_revoked, dcr_registered. _audit_event is async and wraps
record_audit in asyncio.to_thread so the audit write doesn't block
the event loop. resource_id on the audit row is the immutable
server_id (PK UUID) so admin-driven server renames don't break
event correlation; server_name is exposed in detail for cross-
reference. dcr_registered detail.has_secret reflects whether the
DCR-issued secret was actually persisted (the prior code reported
has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
server_name (which is what the column is — the table's PK was
always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
the audit-action taxonomy.
Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.
Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
limitation in oauth_ssrf module docstring
Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.
Phase 3 of docs/design/oauth-mcp.md. Adds the Fernet/MultiFernet wrapper,
[security] config loader with rotation support, MCPTokenStore CRUD facade,
typed MCPTokenDecryptError that maps to the RFC's mcp_token_undecryptable_
key_unknown class, and a startup gate that fails loud when auth_type=
'oauth_user' rows exist without a configured encryption key.
Crypto module (turnstone/core/mcp_crypto.py):
- MCPTokenCipher wraps cryptography.fernet.Fernet + MultiFernet for
rotation; encrypt with first key, decrypt by trying each in order
- load_mcp_token_cipher_config reads [security] mcp_token_encryption_keys
(plural list) or mcp_token_encryption_key (singular), validates each
key is base64-decodable to exactly 32 bytes
- MCPTokenCipherConfig is repr=False with custom __repr__ that redacts
raw key bytes (defense in depth against accidental log/traceback leak)
- _key_fingerprint produces an 8-hex-char SHA-256 prefix for audit
attribution without exposing the key
- MCPTokenStore handles encrypt-on-write / decrypt-on-read for
mcp_user_tokens and mcp_servers.oauth_client_secret_ct
- get_user_token MUST NOT auto-delete the row on MCPTokenDecryptError
(test_get_user_token_with_wrong_key_raises_decrypt_error verifies
the row stays intact across a key-mismatch read)
- initialize_mcp_crypto_state / close_mcp_crypto_state lifespan helpers
shared between server and console
Storage protocol (5 new ciphertext-only methods):
- set_mcp_oauth_client_secret_ct (dedicated writer; deliberately NOT
added to MCP_SERVER_MUTABLE so generic update_mcp_server cannot write
the secret column)
- create_mcp_user_token, get_mcp_user_token,
update_mcp_user_token_after_refresh, delete_mcp_user_token
Server + console lifespans (turnstone/server.py + console/server.py):
- after OIDC init, count auth_type='oauth_user' rows; if any exist and
no encryption key is configured, log an actionable error and
raise SystemExit(1)
- without oauth_user rows, missing key is fine (lazy validation; admin
flip without restart returns 503 from the admin handler)
- app.state.mcp_token_cipher / .mcp_token_store populated when key
configured; None otherwise
Admin handlers:
- _require_token_store_for_oauth_secret pre-mutation gate validates
token_store availability and oauth_client_secret type BEFORE
storage.create_mcp_server / update_mcp_server runs, so a 503 from a
missing key never leaves an orphan row or partial-update state
- _apply_oauth_client_secret encapsulates the encrypt + audit write
used after the storage mutation; rolled out across both create and
update handlers
- 503 message references both mcp_token_encryption_key (singular) and
mcp_token_encryption_keys (plural for rotation)
- non-string oauth_client_secret payloads (false / 0 / lists / dicts)
are rejected with 400 instead of being str()-coerced
- when auth_type transitions away from oauth_user, the encrypted
secret column is cleared in the same admin call (with audit), so
flipping back doesn't silently resurrect a stale credential
Audit events (mcp_server.oauth.* per audit.py taxonomy; RFC's
mcp.oauth.* renamed for consistency):
- mcp_server.oauth.client_secret_set fired from admin handlers with
cleared:bool and key_fingerprint
- mcp_server.oauth.token_decrypt_failure fired from MCPTokenStore
.get_user_token when no installed key can decrypt; carries
key_fingerprints_attempted
Tests: 35 new tests across test_mcp_crypto, test_mcp_token_store,
test_server_lifespan_mcp_crypto, plus 6 admin-API tests covering the
no-orphan-row, no-partial-update, secret-clear-on-transition, and
non-string-secret-rejection invariants. Suite at 5337 (Phase 3 added
~50 tests including the rebase-imported skill suite).
cryptography>=42 promoted from transitive (lacme[tls]) to direct dep
since the encryption layer is now core, not optional.
Phase 4 (OAuth flow) wires the actual callers; Phase 3 adds only the
crypto layer and is exercised entirely by tests.
Adds the data model and admin UI surface required by the OAuth-MCP flow.
Phase 2 of the per-user delegation initiative.
Schema:
- migration 049 creates mcp_user_tokens (PK user_id, server_name) and
mcp_oauth_pending (PK state, indexed by created_at)
- eight new columns on mcp_servers: auth_type ('none' / 'static' /
'oauth_user', NOT NULL DEFAULT 'static') plus six oauth_* config
fields and oauth_as_issuer_cached
- post-upgrade UPDATE normalises auth_type to 'none' for streamable-http
rows whose headers are NULL/empty/'{}'; stdio rows are left at the
'static' default (auth_type is HTTP-auth-only)
- _schema.py kept in lockstep with the migration so metadata.create_all
and alembic upgrade produce identical shapes
- mcp_user_tokens / mcp_oauth_pending TypedDicts in _protocol.py for
Phase 3/4 use (no CRUD methods yet)
Storage / API:
- create_mcp_server gains the eight kwargs across protocol + sqlite +
postgresql
- MCP_SERVER_MUTABLE picks up auth_type and the six text oauth_* fields;
oauth_client_secret_ct is intentionally NOT in the whitelist — Phase 3
will own ciphertext writes via a dedicated method
- McpServerInfo + Create/Update Pydantic schemas extended; oauth_client_secret
accepted as plaintext input but discarded (Phase 3 wires encryption)
Admin handlers:
- _parse_auth_type validates against {'none', 'static', 'oauth_user'} and
rejects empty / unknown values; shared between create and update
- when auth_type changes away from 'oauth_user', the oauth_* config
columns are explicitly nulled in the same UPDATE so the row stays
consistent
- _clean_oauth_text caps text fields at 512 chars (URLs at 2048) to bound
admin write surface
- _mask_mcp_secrets now masks oauth_client_secret_ct to '***' regardless
of reveal=true (write-only field)
- audit detail dict redacts oauth_client_secret if present
Frontend:
- new "Multitenant Authorization" fieldset on the MCP-server modal with
three radio buttons (None / Shared / Per-user OAuth 2.1)
- conditional OAuth subform: AS URL, registration mode (preregistered /
dcr; cimd is future), client ID, client secret, scopes, audience
- secret input is autocomplete=off and never round-trips on edit
- audience auto-populates from the MCP server URL on blur
- headers textarea hidden and submitted as {} when auth_type is 'none' or
'oauth_user' so flipping the radio cleans up server-side state
Tests: storage round-trip for the new columns, oauth_pending table smoke,
migration 049 upgrade/downgrade with stdio-vs-http normalisation, four
admin-API tests for auth_type validation and oauth_*-clear-on-flip-away.
Suite passes 5284 (matched pre-Phase-2 baseline 5267 + 17 new).
Stacks on Phase 0; no behavioural change for existing rows.
Phase 0 of the OAuth-MCP RFC: prepare MCPClientManager for the per-(user,
server) session pool that lands in Phase 5, without changing static-path
behavior.
Two changes:
1. Hardening helpers _pre_close_streams and _tcp_probe rename their first
parameter from `name` to `key`. Type stays `str` for now; widening to
`str | tuple[str, str]` happens in Phase 5 when callers actually pass
tuples. _safe_close_stack takes the stack directly and is unchanged.
2. The eleven parallel name-keyed dicts (_sessions, _per_server_stacks,
_per_server_tools, _per_server_resources, _per_server_prompts,
_supports_list_changed, _supports_resources, _supports_resource_list_changed,
_supports_prompts, _supports_prompt_list_changed, _server_streams) are
consolidated into _static_servers: dict[str, StaticServerState]. Server-
level state (circuit breaker, notification debounce, last-error,
db-managed, merged catalog maps, listener lists) stays on the manager,
unchanged.
PoolEntryState is defined for Phase 5 use but no code instantiates it. The
typed map declarations (dict[str, StaticServerState] vs dict[tuple[str, str],
PoolEntryState]) make accidental cross-keying lookups easier to catch.
PR #296 hardening preserved exactly:
- pre-close-streams atomic take-and-clear before stack teardown
- stale-session-and-stack guard at _connect_one top: both state.session and
state.stack checked, cleared independently, entry preserved (not popped)
- transport-error session-eviction in dispatch sets state.session=None only,
leaving stack/streams for the next connect-time guard sweep
- _safe_close_stack CancelledError suppression unchanged
- TCP probe before streamablehttp_client unchanged
- future.cancel() after TimeoutError in all sync bridges unchanged
- notification debounce stays manager-level (not migrated into the dataclass)
Refresh helpers (_refresh_server_tools/_resources/_prompts) snapshot
state.session into a local immediately after the None guard so concurrent
transport-error eviction during await cannot null the session reference
mid-call.
Tests: shared _seed_static_state helper in tests/conftest.py replaces eleven
direct dict mutations; new test_reconnect_preserves_static_state_identity
guards the entry-preservation invariant. Pass count rises 5266 → 5267.
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.
Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).
Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.
Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.
Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
traffic arrives or an operator clicks Reconnect. The previous
background reconnection loop is gone by design — push
notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
not changed here.
This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.
* feat(skills): paste SKILL.md to auto-fill the Create Skill modal
When a user pastes an Anthropic-style SKILL.md (YAML frontmatter +
markdown body) into the Create Skill content textarea, the frontend
sniffs the leading ``---``, posts the raw text to a new backend parse
endpoint, and populates name / description / tags / author / version /
license / compatibility / allowed_tools from the parsed fields. The
textarea is left with the body only (frontmatter stripped), and a toast
reports how many fields were set vs. kept (already-typed values are
preserved).
Backend
- ``POST /v1/api/admin/skills/parse`` (admin.skills permission) wraps
the existing ``turnstone.core.skill_parser.parse_skill_md`` so admin
imports and external installs share one parser. ``ParseSkillRequest``
/ ``ParseSkillResponse`` schemas added; OpenAPI spec + sync/async
console SDK methods updated.
- Hardening: 32 KiB cap on ``raw`` (Pydantic ``max_length`` + handler
enforcement); ``Content-Length`` pre-check returns 413 before any body
buffering; parse offloaded via ``asyncio.to_thread`` so deeply-nested
YAML cannot stall the event loop.
Frontend (turnstone/console/static)
- New paste handler with optimistic paint (raw text shown immediately,
textarea disabled + ``aria-busy`` flipped, hint switches to
"Parsing...") so the round-trip is visible on slow networks.
- ``AbortController`` + generation guard (``_ctmPasteController``) so a
fresh paste or modal close cancels a stale fetch — the previous
handler's callbacks see the controller has been replaced and bail
before touching the DOM.
- Non-destructive overwrite: ``_setSkillFormField`` returns "filled" /
"skipped" / "absent" and refuses to clobber non-empty values. Toast
reports counts.
- Bumps ``#toast`` z-index above modal overlays (was 200 vs. modal 600
— toasts fired while a modal was open were invisible). Console-wide
fix exposed by this being the first feature to fire toasts mid-modal.
HTML / CSS
- New ``.skill-paste-hint`` line above the textarea announcing the
affordance, sized to match surrounding ``.label-hint`` text.
- ``aria-describedby`` ties the hint to the textarea; ``aria-live=
"polite"`` announces the busy-state transition to screen readers.
- "Skill Content" heading hint reworded "system message — ..." →
"available: ..." and the variables row label "Variables" → "Used"
to disambiguate available vs. in-use template variables.
Tests
- 11 new cases in ``tests/test_skill_parse_api.py``: happy paths
(full / minimal / nested-metadata / unquoted-colon recovery),
malformed YAML 400, missing/blank/missing-name 400, RBAC 403, raw
body 32 KiB cap (Content-Length pre-check), chunked-encoding bypass
forces the application-layer cap. Test pins ``raw_frontmatter``
omission so a future ``dataclasses.asdict`` refactor can't silently
leak the full YAML dict back to clients.
Validation
- 5146 / 5146 ``pytest -k "not live"`` pass.
- ``ruff`` + ``mypy`` clean on changed sources.
- ``node -c`` clean on governance.js.
- Two-stage code review (full pipeline + bug+quality re-review of the
fix patches) applied; all confirmed findings addressed.
* fix(skills): Copilot PR #477 review fixes (cumulative bug-1, bug-2, q-1)
bug-1 (server.py): Content-Length pre-check was clamped to 32 KiB —
the same number as the per-string char cap on ``raw``. A legitimate
``raw`` of exactly 32 KiB produces a JSON body well above 32 KiB once
the ``{"raw":"..."}`` wrapper and any escaping is added, so valid
near-max requests were 413'd. New constant
``_PARSE_SKILL_MAX_BODY_BYTES = _PARSE_SKILL_MAX_CHARS * 4`` admits the
wrapper + multibyte expansion while still refusing obviously oversized
payloads early; the per-string ``len(raw)`` check stays authoritative.
bug-2 (governance.js): hideCreateTemplateModal aborted the inflight
paste controller and nulled the global, but the handler's ``.catch``
and ``.finally`` guard each DOM mutation behind ``_isCurrent()`` —
both bail when the controller has been nulled, leaving the textarea
``disabled`` + ``aria-busy`` and the hint stuck on "Parsing…".
Reopening the modal landed on a poisoned state. The second-pass
review's q-2 cleanup that dropped the show-side defensive reset
missed this scenario — the verifier's reachability argument confused
"controller is null" with "UI state is reset"; the two are
independent. Hide now resets the paste-induced visible state
alongside the abort.
q-1 (console_spec.py): error_codes for the parse endpoint listed only
400; handler also returns 413 for oversized bodies. Added 413; kept
403 implicit per the convention sibling admin endpoints follow.
Test fixup: bumped the Content-Length test payload to 200 KB so it
clearly exceeds the new 128 KB pre-check threshold; otherwise it was
falling through to the per-string check and duplicating
test_oversized_raw_chunked_returns_413's coverage.
PR #476 review feedback (Copilot, oidc.py:584,616):
1. initialize_oidc_state's docstring claimed "on any failure
enabled is False" but the JWKS-prefetch failure branch
intentionally keeps enabled=True so the callback's lazy-fetch
retry can recover from a transient IdP issue at startup.
Docstring rewritten to spell out the three post-conditions:
disable, JWKS-failure-keeps-enabled, success.
2. The long-lived httpx.AsyncClient was created up front, then
three disable branches (discovery exception, discovery-returned-
disabled, missing redirect_base) returned without closing it,
leaving sockets held until shutdown.
Restructured: discovery now uses a transient AsyncClient inside
a context manager (closed at exit). The long-lived client is
only created after the disable checks pass. The JWKS-failure
branch still legitimately keeps the client open because the
lazy-retry path needs it.
The pre-existing single-client-passthrough test was replaced
with three more specific tests: long-lived client only goes to
fetch_jwks (not discover_oidc); discovery-exception path leaves
http_client=None; missing-redirect_base path leaves
http_client=None.
q-4: tests/test_oidc.py's _make_config and tests/test_oidc_handlers.py's
_make_oidc_config built the same OIDCConfig with sensible defaults but
had drifted — only the handlers helper set redirect_base. After b3
made redirect_base operationally required, every test_oidc.py test
that exercised redirect_base had to override it explicitly. A future
test could omit redirect_base and silently exercise the wrong
production path.
Moves make_oidc_test_config to tests/conftest.py with the more
complete handler-version defaults (including redirect_base). Both
test files import it under their existing local alias
(_make_config / _make_oidc_config) so the 60+ call sites in
test_oidc.py and the handler tests don't have to change.
q-5: section banner '# Exception' (singular) at oidc.py:79 became
inconsistent after b5 (callback robustness) added OIDCKeyNotFoundError.
Renamed to '# Exceptions'.
The OIDC perf batch added storage.count_users() and migrated the two
OIDC handlers (handle_oidc_authorize, handle_oidc_callback) but missed
handle_auth_status — which still ran storage.list_users() then
len(users) > 0 for the same has-any-users gate.
count_users() is one COUNT(*) round-trip vs list_users() rehydrating
every row dict. Wrapped in asyncio.to_thread to match the OIDC handler
pattern; the async handler no longer blocks the event loop on storage
I/O for what's effectively an existence probe.
bug-2 (Postgres) — replace_oidc_roles read existing rows under default
READ COMMITTED with no row lock. Two concurrent OIDC callbacks for the
same user_id (racing token refreshes with differing claim sets) could
both observe the same baseline and produce a final role state matching
neither caller's intent. Adds .with_for_update() to the SELECT so the
existing rows for this user are locked for the duration of the
transaction.
The lock is per-user_id, not table-wide; unrelated user writes are
unaffected. Empty result sets acquire no locks, so a brand-new user
with no rows yet still allows two callers to proceed and merge via
ON CONFLICT DO NOTHING — that's a permissive race that self-heals on
the next reconciliation cycle, documented in code.
perf-1 (SQLite) — replace_oidc_roles took the SQLite global write
lock unconditionally via BEGIN IMMEDIATE before reading. Steady-state
re-logins (claims unchanged, no INSERT/DELETE needed) paid the lock
cost for nothing and serialised against unrelated writers.
Replaces with a double-check pattern: phase 1 reads under the default
deferred transaction (no write lock), computes the diff, and returns
(set(), set()) on no-op. Phase 2, only when mutation is needed,
commits the read txn, escalates to BEGIN IMMEDIATE, RE-READS, and
re-computes the diff under the lock before writing. The returned
(added, removed) reflects what was actually written, so caller logging
in apply_role_mapping stays truthful even when concurrent writers
shifted state between the two reads.
The OR IGNORE on insert is now defense-in-depth (the lock makes it
unnecessary) but kept as a safety net.
The 8-commit OIDC stack added TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS
(operator allow-list for cross-host IdP discovery endpoints) and
promoted TURNSTONE_OIDC_REDIRECT_BASE to required, but the docs drifted
in two places:
q-1 — Troubleshooting > "OIDC not configured" still listed three
required env vars. An operator hitting the missing-redirect-base
startup error landed on a debugging entry that didn't mention the
variable they were missing. Fixed; added a separate troubleshooting
entry naming the exact log message produced by initialize_oidc_state
when redirect_base is unset.
q-2 — TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS was undocumented entirely.
Added a row to the env-var table and a new "Cross-host endpoints"
section explaining when the knob is needed (Google is the canonical
multi-origin IdP, but it's auto-handled; the env var is for any other
IdP whose discovery doc legitimately references hosts beyond the
issuer's origin). Added a troubleshooting entry pointing at the new
section.
If apply_role_mapping raised after create_oidc_user committed (transient
storage failure, race with role deletion, etc.), provision_oidc_user's
inline safety-net was skipped — and on retry the existing-identity
branch never reached the safety-net code, leaving the user permanently
stranded with zero roles.
Extracts _ensure_default_role(storage, user_id, desired_role_ids=None)
helper. Calls it on BOTH the new-user and existing-identity paths so a
user stranded by a transient failure recovers on next login.
desired_role_ids is a hint that lets the helper skip list_user_roles
when claim-driven mapping populated at least one role; the new-user
path was already paying that query, the existing-identity path now
pays it only when claim mapping returned an empty desired set.
Documents the admin-strip behavior in the helper docstring: stripping
all roles from an OIDC user no longer locks them out, since the next
login will re-grant builtin-viewer (assigned_by='oidc-default'). The
documented way to deny an OIDC user is to unlink their OIDC identity
via the admin endpoint, not to strip roles. The pre-fix behavior
(stripped user actually locked out) was the bug.
The 'oidc-default' vs 'oidc' assigned_by distinction is preserved:
apply_role_mapping's revocation lane only touches 'oidc' rows, so the
safety-net role survives every subsequent login regardless of claims.
Six new tests cover both paths, the hint short-circuit, the
list_user_roles fallback, the missing-builtin-viewer no-op, and the
self-heal regression case for already-stranded users.
q-5: _derive_username's UUID-retry tier (oidc.py:923-933) was untested.
After perf-6 collapsed tier-2 to a single find_existing_usernames call,
the only remaining tail was the 3-attempt UUID-retry loop and the final
raise. New TestDeriveUsername class covers:
- falls into UUID retry when all 10 suffix candidates are taken
- UUID retry succeeds on the second attempt after one collision
- UUID retry exhausted -> raises OIDCError
q-8: filled the unit-level coverage holes the multi-stage review flagged:
- test_validate_id_token_retry_after_kid_rotation — direct unit test of
the OIDCKeyNotFoundError path with real RS256 keys + JWKS rotation
(previously only exercised end-to-end through the handler).
- test_callback_uses_pending_audience_not_handler_audience — pins down
the bug-3 fix by decoding the issued JWT cookie and asserting aud
matches the audience stored at /authorize time, not the handler param.
- test_apply_role_mapping_int_claim / _dict_claim — exercises the
else: values = [str(claim_value)] branch for non-string non-list
claim shapes.
- TestFetchJWKS — non-200 status, non-dict body, dict-missing-keys,
keys-not-list, transport network error.
- TestExchangeCode network/4xx/5xx error tests (the non-dict-body case
already shipped in batch 5).
Also a small production hardening that fell out of writing the
TestFetchJWKS::test_fetch_jwks_non_dict_body_raises test: fetch_jwks now
guards isinstance(result, dict) before result.get("keys"), matching the
shape-check pattern that discover_oidc and exchange_code already use.
A list/null body now surfaces as OIDCError("...not a JSON object") rather
than AttributeError leaking up to the lifespan.
Eleven small maintenance fixes; no behavior change beyond bug-3.
bug-3: pending.get('audience', audience) couldn't fall back because
pop_oidc_pending_state always returns a dict with the audience key
set verbatim from a non-null TEXT column. Replaced with
pending.get('audience') or audience to cover the empty-string case
defensively. Comment explains the security rationale.
q-1: extract _env_or_cfg_str / _env_or_cfg_bool helpers in oidc.py;
load_oidc_config's six near-identical env-or-config blocks collapse
to one-liners. role_map / trusted_endpoint_hosts / redirect_base
retain bespoke parsing.
q-3: discover_oidc narrows except (httpx.HTTPError, ValueError, KeyError)
with exc_info=True.
q-4: OIDC_STATE_TTL_SECONDS = 300 constant in oidc.py; auth.py imports
and passes it explicitly. Storage signatures keep the literal default
(storage layer doesn't know OIDC TTL semantics).
q-6: hoist runtime imports (OIDCError, OIDCKeyNotFoundError, exchange_code,
fetch_jwks, provision_oidc_user, validate_id_token, build_authorize_url,
generate_pkce_verifier) to module scope in auth.py. The genuine cycle
is only oidc._derive_username -> auth.is_valid_username, kept
function-scoped. test_oidc_handlers.py mock targets repointed to
turnstone.core.auth.X to match the new binding.
q-7: comment + docs explain the 'oidc' vs 'oidc-default' assigned_by
marker distinction.
q-9: OIDCIdentity / OIDCPendingState TypedDicts in storage protocol.
Implementations construct via TypedDict syntax so mypy structurally
verifies all required fields.
q-10: fetch_jwks narrows except (httpx.HTTPError, ValueError); docstring
matches.
q-11: rename generate_pkce_pair -> generate_pkce_verifier; return only
the verifier (build_authorize_url already recomputes the challenge).
q-12: extract _buildOidcRow helper in admin.js so future field additions
go in one place.
q-13: OIDCConfig docstring lists startup-config vs discovery-derived
field groups.
Eight independent perf wins on the OIDC hot path:
perf-1: list_users() full-scan setup-gate replaced with new count_users()
on both authorize and callback. Saves a full users-table fetch per login.
perf-2: handle_oidc_callback's sync DB chain wrapped in asyncio.to_thread
for cleanup, pop_oidc_pending_state, count_users, and provision_oidc_user.
handle_oidc_authorize gets the same treatment for count_users and
create_oidc_pending_state. Event loop no longer blocks for the full
callback duration on Postgres deployments.
perf-3: apply_role_mapping N+1 collapsed via new replace_oidc_roles
storage method. One transaction handles the diff + insert + delete
instead of 2N+1 commits per login. Returns (added, removed) so the
caller can still emit per-role audit logs.
The diff respects the documented invariant "manually-assigned roles
are never touched" — desired_role_ids is filtered against rows where
assigned_by != 'oidc' before computing added/removed. This prevents a
PK conflict (Postgres lockout) or silent OR-IGNORE no-op (SQLite lying
return) when admin-ui or oidc-default already holds the same role_id.
perf-4: provision_oidc_user no longer re-queries list_user_roles after
apply_role_mapping. The new-user builtin-viewer fallback is gated on
desired_role_ids being empty, which is information apply_role_mapping
already returned.
perf-5: JWKS refetch dedup via asyncio.Lock on app.state. Both lazy-fetch
(cold-start recovery) and rotation paths share the same lock with a
double-check pattern: re-resolve kid against the current cache before
issuing a new GET. N concurrent callbacks during rotation now produce
at most 1 fetch.
perf-6: _derive_username's 9-suffix loop collapsed via new
find_existing_usernames(candidates) -> set query. Worst case drops
from 13 sequential queries to 1 + up-to-3 UUID-retry queries.
perf-7: cleanup_expired_oidc_states gated to once-per-60s per process
via app.state.oidc_last_cleanup_monotonic. The pop already deletes
the consumed row; the bulk cleanup is only relevant for abandoned
authorize flows, so frequency was overkill.
perf-8: Long-lived httpx.AsyncClient stashed on app.state.oidc_http_client
by initialize_oidc_state. discover_oidc/fetch_jwks/exchange_code accept
an optional client= kwarg; when set, skip the per-call AsyncClient
context-manager. New close_oidc_state lifespan teardown closes it.
Tests pass client=None to keep the transient-client legacy path.
New storage methods (sqlite + postgresql):
- count_users() -> int
- find_existing_usernames(candidates) -> set[str]
- replace_oidc_roles(user_id, desired) -> (added, removed)
Four small hardening fixes on the OIDC callback hot path:
bug-4: JWKS rotation retry was matching the substring 'not found in JWKS'
inside an OIDCError message. A future rephrasing would silently break
key rotation. Adds OIDCKeyNotFoundError(OIDCError); validate_id_token
raises the subclass at the kid-not-found site; handle_oidc_callback
catches it explicitly. Other 'not found' errors in validate_id_token
remain as plain OIDCError.
bug-5: tokens['id_token'] raised KeyError if the IdP returned 200 without
id_token. exchange_code now rejects non-dict response bodies; the
callback validates id_token shape (must be non-empty str) before
passing to validate_id_token. Both raise OIDCError, surfaced as the
standard 'Authentication failed' redirect.
bug-6: shared_static/auth.js — the OIDC error display raced showLogin's
/v1/api/auth/status fetch via a 300ms setTimeout. showLogin now takes
an optional oidcError parameter and paints it after _switchMode clears
the error, in both the success and catch branches of the fetch.
sec-4: oidc.py exchange_code's non-200 OIDCError interpolated up to 500
bytes of attacker-controlled IdP body, which then went to log.warning
via 'OIDC callback failed: %s'. CRLF in resp.text could forge log
lines. New _sanitize_log_text helper escapes control chars via
unicode_escape and caps at the rendered length.
provision_oidc_user previously called create_user (INSERT OR IGNORE
on SQLite — silent no-op on UNIQUE conflict), then create_oidc_identity
(also INSERT OR IGNORE), then apply_role_mapping which writes user_role
rows for the supposedly-new user_id. On a username TOCTOU race or
concurrent (issuer, sub) double-create, both inserts no-opped but
user_role rows were already written — leaving orphan rows pointing
at a user_id that doesn't exist.
PostgreSQL's create_user raised IntegrityError instead of silently
no-opping so it produced a misleading 'Authentication failed' error
without orphans, but the user-facing UX was equally poor.
Adds StorageConflictError to the storage protocol and create_oidc_user
that does both inserts in one transaction. Username collision and
(issuer, subject) collision both raise StorageConflictError, mapped
to OIDCError by provision_oidc_user. Crucially the new code does not
silently bind a colliding-username new identity to the existing user
— that would be an account-takeover vector. It raises.
SQLite uses BEGIN IMMEDIATE inside the try block so lock-contention
errors surface as StorageConflictError instead of leaking the raw
sqlalchemy OperationalError.
PostgreSQL relies on SQLAlchemy 2.x begin-on-demand semantics; the
explicit conn.commit()/rollback() in the catch block is the only
materialization path. Discrimination on PG uses
exc.orig.diag.constraint_name with message-substring fallback.
_build_oidc_redirect_uri previously fell back to the request Host
header when redirect_base was unset. With a permissive reverse proxy
or direct backend access, a spoofed Host minted an authorize URL
pointing to attacker-controlled host — combined with a permissive
IdP redirect_uri allowlist this enables auth-code interception.
There is no production scenario where a Host-derived redirect_uri is
correct, so this fails closed:
- initialize_oidc_state checks redirect_base after discovery succeeds
and disables OIDC (with an explicit error log naming the env var)
if it's empty. Runs before fetch_jwks so a misconfigured deploy
doesn't make a wasted JWKS call.
- _build_oidc_redirect_uri simplifies to f"{redirect_base}/v1/api/auth/oidc/callback".
request parameter dropped; both call sites (handle_oidc_authorize,
handle_oidc_callback) updated.
- docs/oidc.md promotes TURNSTONE_OIDC_REDIRECT_BASE from "Recommended"
to "Required" with the security rationale.
The OIDC discovery + JWKS prefetch block was duplicated byte-for-byte
between turnstone/server.py and turnstone/console/server.py. The bare
except branch in that block also left app.state.oidc_config unchanged
on unexpected exceptions — leaving the runtime with enabled=True and
empty endpoints, producing malformed authorize URLs.
Extracts initialize_oidc_state(app_state) into turnstone/core/oidc.py
which guarantees a coherent post-condition on every code path:
- discovery exception -> oidc_config replaced with enabled=False, jwks_data=None
- discovery returns enabled=False -> jwks_data=None
- JWKS prefetch fails -> jwks_data=None but enabled=True preserved (the
callback's lazy-fetch retry path remains the recovery)
- success -> oidc_config + jwks_data both populated
Also hardens discover_oidc against non-dict discovery responses
(list/null/string/int) — previously these raised AttributeError out
of doc.get and propagated past the lifespan's bare except.
server.py and console/server.py lifespan blocks collapse to a single
await initialize_oidc_state(app.state) call.
OIDC discovery-document endpoints (token_endpoint, jwks_uri,
userinfo_endpoint) were stored verbatim in OIDCConfig and later passed
to httpx without revalidation. Only the issuer URL was checked. A
hostile or compromised IdP could return token_endpoint pointing to an
internal IP (169.254.169.254, 10.0.0.0/8, etc.) and Turnstone would
POST the client_secret there.
Extracts the existing scheme/userinfo/SSRF check into
_validate_url_no_ssrf, adds validate_discovered_endpoint that runs the
same checks plus an issuer-binding check, and wires it into
discover_oidc for authorization_endpoint, token_endpoint, jwks_uri,
and userinfo_endpoint (when present).
Issuer binding accepts:
- Same (scheme, hostname, effective port) as the issuer.
- A hostname in _KNOWN_TRUSTED_ENDPOINT_HOSTS for the issuer (Google's
multi-origin discovery is in the allow-map by default).
- A hostname in OIDCConfig.trusted_endpoint_hosts, settable via
TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS env var or config.toml, for
IdPs not in the static map.
Effective port comparison treats https://host and https://host:443 as
the same origin (urllib.parse.urlparse leaves the explicit form's port
as 443 and the implicit form's as None).
24 new tests cover the validator, the Google known-hosts path, the
operator allow-list, default-port equivalence, foreign-host
rejection, private-IP rejection, embedded credentials, and DNS
rotation between issuer check and endpoint use.
* feat(console): inline node picker replaces back-to-console banner
Drops the 32px banner the console proxy used to inject above proxied
server-UI pages and replaces it with an inline node-id pill in the
existing #ui-header. Click the pill to open a dropdown that lists
healthy nodes (health dot, ws count, reachable/degraded/unreachable
text) plus a top-row link back to the console.
Reuses the .ws-tab-dropdown shell from ui/static/style.css for
animation, shadow, theme override, and item layout, so the picker
visually matches the workstream-tab chevron menu it sits next to.
Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron
menu's handler with cross-reference comments at both sites.
Lazy-fetches /v1/api/cluster/nodes against the console origin
(bypassing the prefix shim) on first open.
Reclaims 32px of vertical space, consolidates three separate
"you're on node X via console" indicators into one, and turns the
wayfinding chrome into a real cluster-nav primitive.
* fix(console): address Copilot review on node picker
- Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap)
instead of relying on the default 100 — clusters with more than
100 nodes were silently dropping rows from the picker.
- Hand off focus to the first menu item after the async fetch
resolves: openMenu()'s deferred focus hook ran while only the
skeleton was in the DOM, so first-open keyboard users were
stranded on the trigger until they pressed an arrow key.
- Tab now closes the menu without preventDefault, so focus moves
to the next focusable element on the first press (ARIA APG menu
pattern). Escape still preventDefault + returns to the pill.
- Cap pill max-width at 240px and ellipsize the id span; node ids
are accepted up to 256 chars upstream and could otherwise push
the title and right-side controls off the appbar. Pill carries
a title attribute so the full id is still legible on hover.
* fix(session): properly inject queued user messages mid-loop
Two queued-user-message bugs in ``ChatSession.send()``.
**Mid-tool-call: ``Unexpected role 'tool' after role 'user'`` on Mistral.**
The ``supports_tool_advisories`` capability flag (default False for
unknown openai-compatible models) routed cap-off providers down a
short-circuit branch in ``_collect_advisories`` that called
``_flush_queued_messages`` directly. That appended a ``user`` turn
between ``assistant(tool_calls)`` and ``tool``, which mistral-common's
``_validate_message_order`` rejects with a 400.
Drop the flag. All providers now run the unified path: queued user
messages become ``UserInterjection`` advisories that ride inside the
tool result envelope via ``wrap_tool_result``, splicing
``<system-reminder>`` text into the tool message's content. Role
sequence stays ``assistant → tool``. Live-confirmed on Mistral
medium and Qwen3 — both correctly distinguish system-reminder from
tool stdout in their reasoning.
**Mid-stream: queued message orphaned until next user send.**
After a no-tool assistant turn, ``_flush_queued_messages`` would
append the queued user message to history and the loop would
``break``, leaving the message at the tail of history with no
model response. Visible as "two sends to get one reply".
``_flush_queued_messages`` now returns ``bool``. The no-tool branch
``continue``s on drain instead of ``break``ing, so the model gets a
turn over the extended history.
Tests:
- ``test_collect_advisories_drains_text_queued_messages_to_persistent``
pins the unified-path drain (text-only queue → ``UserInterjection``,
no separate user turn appended to ``self.messages``).
- ``test_send_continues_when_messages_queued_during_streaming`` pins
the loop-continue behavior (fails with 1 stream call pre-fix,
passes with 2 post-fix).
* fix(session,ui): reject queued attachments + paperclip busy state
Copilot pointed out that the attachment-bearing branch in
``_collect_advisories`` had the same role-ordering bug as the
text-only path that 802658f fixed: an attachment-bearing queued
item would still call ``_append_user_turn`` mid-tool-call,
injecting ``user`` between ``assistant(tool_calls)`` and ``tool``.
Pragmatic fix: don't allow attachments to be queued at all.
**Backend.** ``ChatSession.queue_message`` raises a new
``AttachmentsNotQueueableError`` when called with non-empty
``attachment_ids``. The interactive ``/send`` route catches it,
releases reservations via the existing ``_release_reservation_on_fail``
hook, and surfaces ``status: "attachments_busy"`` to the caller
with the IDs in ``dropped_attachment_ids``. The coord adapter
mirrors the cleanup (releases the soft-locked reservation taken
for ``_send_id``) so the create-with-attachments path can't leak.
Now that the queue can never carry attachments, the per-item
``att_ids`` slot is gone:
- Queue tuple slimmed ``(cleaned, priority, att_ids)`` →
``(cleaned, priority)``.
- ``_flush_queued_messages`` collapses to a single combined-text
user turn (no attachment branch).
- ``_collect_advisories`` queue-drain pushes ``UserInterjection``
advisories only (no ``attachment_items`` list).
- ``dequeue_message`` no longer unreserves (queue can't reserve).
- ``_resolve_attachment_ids`` had no remaining production callers
and is deleted along with the tests that exercised it in
isolation.
**Frontend.** ``Composer.setBusy`` disables the paperclip whenever
busy (regardless of ``queueWhileBusy``) — text still queues,
attachments don't. ``chat.css`` gains a ``.composer-attach:disabled``
rule (mirrors the existing ``.composer-send:disabled`` treatment)
so the affordance actually looks unclickable instead of falling
through to the UA default. ``title`` and ``aria-label`` are kept in
sync for AT users (WCAG 4.1.2).
Both interactive and coordinator UIs handle the new
``attachments_busy`` response with a chat-surface error bubble:
> Attachments can't be sent while the assistant is working.
> Send a text-only message now, or wait and resend with attachments.
Chips stay in the composer so the user can retry once idle.
**Tests.** Replaced the now-impossible ``TestQueuedWithAttachments``
class with a rejection-coverage class. Rewrote the
``_queue_with_attachment`` route-test fixture to reserve directly
via ``reserve_attachments`` (the queue path no longer reaches the
reserved state). Added a route-level test for the new
``attachments_busy`` contract.
* Bound search tool output against pathological inputs
Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.
Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
+ --max-columns-preview, --max-filesize 10M, --max-count 100,
--no-config, --no-messages, plus negative globs for the same
noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.
Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
Defends against single-line files (training data, minified bundles)
that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
pipe read is blocked in the kernel — proc.wait(timeout=…) alone
was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
child writes to stderr while we're still reading stdout. Cap on
captured stderr keeps a hostile child from growing the buffer.
Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
analytically from budget / file_count / avg-line-length so we hit
the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
head+tail _truncate_output strategy never silently drops middle
files in a search result.
Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
branch already had. With auto_approve on the search tool, that was
exploitable: path='--pre=COMMAND' would have made ripgrep run the
script as a per-file preprocessor and surface its stdout. Added
`--` and a regression test.
State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
dedicated 'killed by signal N' message instead of being parsed as
success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
now returns a dedicated byte-cap message instead of the malformed-
output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
(no more 'no matches' tag on a 'malformed' payload).
Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
but if a future caller forgets the flag, an attacker who can set
one of these env vars could plant a config containing --pre=… and
recreate the same RCE shape.
Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
of subprocess.run (the previous tests passed ChatSession kwargs
that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
to exercise the byte-cap trim, mega-line-no-newline edge case, the
watchdog timeout when the child writes nothing, and the stderr
drain under load.
- test_env_scrub picks up the new tool-config keys.
* Address Copilot review on #473
- Budget the Tier 2/3 header up front so the formatter's emission stays
strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
counted body bytes, letting the final string overflow by ~120 chars
(header + separator) and triggering _truncate_output's head+tail
dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
is kept as a starting estimate, but if that K's actual emission
doesn't fit (the estimate ignores the header and overweights shared-
path compression) we step down through the ladder before falling
through to Tier 3. The previous one-shot K could collapse to counts-
only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
SIGKILL). There's a narrow race where the child can exit naturally
between our read and our kill; preserving a non-negative rc means
rg's rc=2 ('matches found but some files had errors') no longer
silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
pass` blocks in _search_capture (stderr drain, pipe close in the
cleanup finally) so static analysis and future readers can see the
silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
instead of the +512-char slack that was masking the header overflow.
- New regression tests:
- Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
- capped + rc=2 surfaces stderr instead of being normalised to success
- capped + rc<0 (our SIGKILL) flows through as a partial-result success
* chore(search): post-review cleanup
Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).
- q-1: collapse the Tier 2 candidates filter to a single expression.
Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
is already floored at 1, so the comprehension always yields ≥ ``[1]``).
``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
semantics (analytical seed → step down through (5, 3, 1) from the
highest rung ≤ the estimate). The previous wording suggested every
Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
statements in test_tier2_steps_down_ladder_before_falling_to_tier3
into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
paths in _search_capture to one line each. Both sites now read with
the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
Keeps the load-bearing semantic (cap is on the content portion only)
and the pathological-line defence; drops the paths-aren't-bounded
parenthetical, which was background reading rather than WHY.
* feat(providers): api_surface toggle + mistral medium reasoning fix
Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions. The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).
Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
and thread it through `create_provider` / `model_registry.get_provider`.
`openai-compatible` defaults to Chat Completions; operators can flip
individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
`chat_template_kwargs`. Operators running gpt-oss-style local
templates that consume `reasoning_effort` from the chat template now
opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
server-side at create/update time; pre-filled by Detect via the
profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
for capability and server_compat resolution; previously the fallback
passed `alias=None`, which silently dropped per-model caps on the
agent path.
Tests: 5117 passed (-m "not live"); ruff + mypy clean.
* fix(providers): don't auto-suggest Responses for Mistral medium
vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls. Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.
Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile. Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.
* fix(providers): address Copilot review on PR #469
- providers/__init__.py: drop the redundant *_responses_provider /
*_chat_provider names; have create_provider use _openai_provider and
_openai_compat_provider directly so they're not flagged as unused
globals.
- console/server.py: tighten _validate_api_surface to a strict equality
match against the canonical {"chat", "responses"} set. The previous
strip().lower() membership check accepted ' Responses '/'CHAT' but
stored the raw string verbatim, which then failed to round-trip
through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
type, api_surface, extra_body) on provider == "openai-compatible" at
save time so toggling provider away can't leave a stale hidden surface
selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
longer flags the call as a wrong-name keyword (the point of the test
is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
for the api_surface validation on both create and update — covers the
bogus-value rejection, non-canonical-string rejection, and the happy
path persisting through to the refreshed registry.
* fix(memory): query-aware candidate selection + OR-of-terms search
The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`). On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context. Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.
## Functional changes
- `_init_system_messages` (`turnstone/core/session.py`): extract
recent context first, then `_search_visible_memories(context)` to
pull query-aware candidates. Search hits below `fetch_limit` union
with the recency list (deduped by memory_id) so the BM25 candidate
pool is always a SUPERSET of the prior recency-only pool — even on
noisy queries where the cap fills with stopwords, the recency-50
the original bug surfaced still reaches BM25. Empty context skips
search entirely. Candidate-selection logic extracted into
`_select_memory_candidates`.
- `search_structured_memories` (PostgreSQL + SQLite): per-term
clauses join with OR instead of AND. A row matches if ANY term
matches ANY of name/description/content. Downstream BM25 narrows
back down by relevance.
## Perf hardening
- Collapse the 1-3 fanned scope queries into a single SQL. New
backend methods `list_visible_structured_memories` /
`search_visible_structured_memories` union the visibility scopes
into one WHERE OR-group, so a composition rebuild now hits the DB
at most twice (search + recency) instead of up to six times.
- Cap and normalize search terms. Composition can hand a multi-KB
pasted message to ILIKE-based search; without a cap, every distinct
token would emit one unindexable predicate per scope-fanned query.
`normalize_search_terms` (`storage/_utils.py`) de-dupes
case-insensitively, drops <2-char tokens, and hard-caps at 16.
- Per-turn search cache. `_init_system_messages` fires from many
call sites within one turn (state transitions, MCP refresh, tool
results) and the recent-context query is identical across them.
Session-instance cache keyed by (query, mem_type, limit) absorbs
the duplicates; invalidated in `_append_user_turn` and after
memory save/delete tool actions.
- Stable secondary sort by `memory_id`. `updated` is second-precision
and `touch_structured_memories` can land a batch on identical
timestamps; without a tie-breaker SQL returns rows in
implementation-defined order, BM25 input shuffles, and the
LLM-side prompt cache misses across calls. All four backend ORDER
BYs now break ties on `memory_id ASC`.
## Quality cleanups
- Coalesce `memory.search.term_count` + `memory.search.zero_results`
into a single `memory.search` log carrying both `term_count` and
`result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
survives future readers.
## Tests
Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.
Memory test sweep: 102/102. Broader regression
(session, storage, coordinator, load_skill): 411/411.
* fix(memory): address Copilot review on PR #468
Three follow-ups from Copilot's inline review:
1. SUPERSET invariant violation (Copilot, session.py:5510).
`(search_hits + extra)[:fetch_limit]` capped the union back down to
fetch_limit, evicting the recency tail when search added distinct
hits. Recency tail is exactly where ancient-but-recently-touched
memories live — the recall this PR is supposed to improve — so
tail eviction recreated the bug for the narrow case where a query
term fell off the 16-cap and the matching memory sat in
recency[40-49]. Drop the cap; both halves are already SQL-capped
at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
defaults). BM25 over 100 candidates in pure Python is sub-ms;
irrelevant recency fillers get score=0 and don't pollute ranking.
Updates the docstring to actually be honest about the invariant.
Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
that locks the behavior in: 5 search hits + 10 recency = 15-item
pool, every recency item present, source="union".
2. Unbounded `query.split()` in normalize_search_terms (Copilot,
_utils.py:74). `str.split()` allocates the full token list before
the cap-after-16 break, so a 100KB pasted query did MB of throwaway
work even though only 16 tokens entered SQL. Switch to
`re.finditer(r'\S+', query)` — streaming iterator, stops scanning
at the first 16 normalized terms regardless of input size.
3. Misleading + unbounded log term_count (Copilot, session.py:8571).
`len(item["query"].split())` had two problems: same unbounded
split as #2, and the value reported the raw input token count
rather than the normalized term count that actually hit the SQL
WHERE clause — misleading metric for an operator trying to
understand storage-side behavior. Switch to
`len(normalize_search_terms(item["query"]))` — accurate count, and
bounded for free via #2.
Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect." False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.
Memory test sweep: 103/103. Broader regression: 411/411.
CI failure on main: test_publish_records_metric_outcome saw an empty
calls list — its monkeypatch was patching a different metrics
instance from the one `_publish_models_metadata` reads.
Two changes:
- test_close_reason_persistence.py: replace the bare
`srv_mod._metrics = MetricsCollector()` assignment in `_make_app`
with an autouse `monkeypatch.setattr(srv_mod, "_metrics", ...)`
fixture so the test's metrics swap auto-restores. Other test
files (test_auth.py, test_server_attachments_endpoints.py) carry
the same anti-pattern; left for a follow-up since they're not on
the critical path here.
- test_server_node_models_metadata.py: switch the publish-helper
metric test to a string-form `monkeypatch.setattr("turnstone.
server._metrics", FakeMetrics())` so it replaces whatever binding
the live module currently holds, regardless of what other tests
did to it. Robust against future leaks of the same shape.
* feat(coord): expose healthy model aliases per node on list_nodes
Surfaces a `model_aliases` field on each `list_nodes` row so a
coordinator can discover which model aliases each cluster node will
accept on `spawn_workstream(model=...)` without an HTTP fan-out.
Each server projects its registry into a `models` entry on
`node_metadata` (`{alias, provider, healthy}` per alias) at lifespan
startup, on every 30s heartbeat tick, and after `internal_model_reload`.
The publish helper short-circuits on a payload-equality cache so a
stable cluster doesn't pay UPSERT churn — exposed via the new
`turnstone_node_models_publish_total{outcome="written|skipped"}`
Prometheus counter so operators can graph cache hit-rate.
Coord client filters the per-alias rows to healthy aliases only and
drops the provider-side model identifier (`cfg.model`) — coords kept
reaching for it when they should pass the local alias.
* fix(coord): address Copilot+CodeQL feedback on list_nodes models work
- internal_model_reload: reuse a single get_storage() local across the
registry load and the metadata publish (Copilot:3047)
- _collect_node_models_metadata: iterate sorted aliases so two
structurally identical registries built in different insertion orders
serialize to the same JSON — directly improves the publish-cache hit
rate exposed via turnstone_node_models_publish_total (Copilot:3105)
- tests: drop mixed turnstone.server import style flagged by CodeQL —
hoist _metrics into the from-import block, and use sys.modules in
the shutdown-race regression test instead of `import as srv`
Address Copilot feedback on PR #465:
1. The has_alias fallback in both session_factories silently rewrote
any unknown caller-supplied alias to the default, including on the
fresh-create path where the create handler maps the factory's
ValueError to a 503 with operator-friendly text. A typo in
body.model would now silently start a workstream on the default
instead of telling the caller their requested model could not be
resolved. Move the fallback out of the factories: each factory
raises again on unknown aliases, and SessionManager filters stale
aliases out of the rehydrate path via a new ``model_validator``
constructor kwarg (production wiring passes ``registry.has_alias``
on both interactive and coordinator).
2. ChatSession.resume()'s elif branch flipped self.model to the
persisted model name even when the alias was unresolvable, leaving
the session paired with the constructor's default provider/client
but a removed model name — a broken state whose next API call
fails. Drop the model copy: keep the constructor's coherent
default (provider + model + capabilities) and just log the
unreachable saved values so the missing alias is auditable.
Tests:
- Move stale-alias coverage from the factory level into
SessionManager (tests/test_session_manager.py): validator drops
stale aliases before reaching build_session; live aliases pass
through unchanged.
- tests/test_sessions.py renamed test_resume_restores_model →
test_resume_keeps_defaults_when_alias_unresolvable to match the new
contract.
SessionManager.open() was calling build_session(ws) without a model
arg on the rehydrate path. The session_factory then resolved the
*current* default alias, ChatSession.__init__'s _save_config() (INSERT
OR REPLACE per-key) clobbered the persisted workstream_config with
those defaults, and the subsequent resume() "restored" what was now
the default — silently resetting model_alias, model, temperature,
reasoning_effort, max_tokens, skill, creative_mode, instructions,
token_budget, and notify_on_complete on every reopen and every
service restart, for both interactive and coordinator workstreams.
Three layers:
1. SessionManager.open() now reads workstream_config via
self._storage.load_workstream_config(ws_id) and threads the saved
model_alias into build_session(ws, model=saved_alias).
2. ChatSession.__init__ now skips its initial _save_config() when a
workstream_config row already exists for self._ws_id — protects
every other persisted knob without having to plumb each one
through the adapter signature, and catches any future construction
path that forgets to thread model through build_session.
3. Both session_factories (server.py interactive, console
session_factory.py coordinator) now treat an unknown caller-
supplied alias the same as an unset alias: fall back to the
runtime default rather than raising. Without this, a workstream
pinned to an alias an operator has since removed from the registry
would 500 on every reopen — defeating the "best effort restore,
default if the original is gone" contract this fix is meant to
deliver. Mirrors _effective_default_alias's existing has_alias
guard against a stale ConfigStore default.
Three changes from PR review:
- Permission gating: hide the Roles sub-tab button when the user
lacks ``admin.settings``. The sub-tab loads/saves through
``/v1/api/admin/settings``, so an admin with ``admin.models`` but
no ``admin.settings`` would otherwise see a perpetual 403 loader.
When Roles is the active sub-tab and the permission check fails,
snap the panel back to Definitions so the user lands somewhere
usable.
- Drop the redundant ``/v1/api/admin/model-definitions`` fetch from
``loadAdminModelRoles``. Both entry points (initial Models-tab
open + ``models_changed`` SSE refresh) flow through
``loadAdminModels`` first, which already populates ``_modelDefs``
+ ``_modelDefaultAlias``; ``_saveModelRole`` doesn't touch model
definitions, so the cached snapshot stays accurate when the save
chains back here. Halves the per-render request count and
removes a wasted round-trip on every cluster-wide model edit.
- Add ``test_models_changed_event.py`` covering the SSE fanout the
prior commit introduced: each model-definition CRUD endpoint
emits exactly one ``models_changed``, settings PUT/DELETE only
emit for keys in ``_MODEL_AFFECTING_SETTING_KEYS`` (parametrised
over all eight), and unrelated settings (e.g.
``session.retention_days``) don't trigger spurious refreshes.
The expected key set is pinned in the test so a stray addition
to the allowlist doesn't silently bypass coverage.
Same shape as the coordinator/judge rows already there: alias dropdown
+ reasoning_effort dropdown sourced from the existing
``model.plan_alias`` / ``model.plan_effort`` and
``model.task_alias`` / ``model.task_effort`` settings. Adds the four
keys to the SSE ``models_changed`` allowlist so changes from the
Settings API also trigger a live dropdown refresh, and filters them
out of the Settings tab so they only render in one place.
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.
Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.
Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.
Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
them with the Models sub-tab switcher (same a11y attrs, arrow-key
nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
coordinator.model_alias / coordinator.reasoning_effort out of the
Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
the Models tab instead of suggesting they set coordinator.model_alias.
* fix(console): home composer attachments + coord chat user-message pills
Two parity gaps in the console's coordinator surface:
- The embedded creator on the home page accepted only text — the
paperclip / paste / drop pipeline that the in-coord composer and the
interactive new-ws modal both expose was missing, so a user couldn't
attach files at create time. Stage Files in memory (no ws_id yet) and
ship them multipart on Start; the coord create endpoint already accepts
multipart via create_supports_attachments=True.
- User messages with attachments rendered as plain text on both live
send and history replay — no chip cluster like the interactive pane.
Added appendUserMessageWithAttachments and a structured userAttachments
list built from _attachments_meta (preferred) or the multipart parts
themselves, then rendered the same .msg-user-attach pill strip the
interactive pane uses.
Polish from a designer pass:
- Pill background was --panel-2, equal to the .msg bubble background in
both themes (border contrast ≈1.4:1, below WCAG 1.4.11). Switched to
--panel so the pill sits on a different surface than the bubble.
- Capped chip filename width inside the home composer (max-width 200px +
ellipsis) so a long filename doesn't push the strip past the textarea.
- aria-live="assertive" → "polite" on #home-coord-error; client-side
validation isn't an interrupt-level event.
- Reserved min-height on .home-composer-error and dropped the
display: none/block toggling so validation messages no longer reflow
the active-coordinators list below.
* fix(console): address PR #462 review feedback
- Block home-composer submit when files are staged but the task field is
empty. Server's _coord_create_post_install short-circuits on an empty
initial_message, so the multipart upload would create pending
attachment rows that never reserve onto a turn — orphaned until the
GC sweep. Fail in the browser instead.
- Drop the redundant `part &&` guard in coordinator.js's history-replay
multipart loop; the earlier `if (!part || ...) continue` already
filtered.
- Rewrite the home-mount .composer-chip-name CSS comment. shared/chat.css
defines .composer-chip{,-size,-remove} but no .composer-chip-name rule
— the span inherits the parent chip font with no width cap.
- Add smoke-guard string assertions in test_coordinator_page.py for
appendUserMessageWithAttachments and msg-user-attach so a future
rename can't silently regress the attachment affordance.
* fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration
Loading a saved workstream silently dropped tool results and missed
verdict / output-guard / truncation signals on replay. Root cause was
in `Pane.prototype.replayHistory`: an assistant message carrying both
content and tool_calls cleared the `lastToolBlock` anchor before the
following tool-result iteration could attach. The fix reorders content
to render before the tool block (matching live SSE order) and
restructures the tool-result branch to anchor by `data-call-id` so
multi-tool batches render `[hdr A][out A][hdr B][out B]` rather than
bunching outputs at the bottom.
Beyond the bug, replay now reaches near-parity with the live UX:
- Persisted intent verdicts and output_assessments flow through both
the SSE replay (`_build_history`) and the `/history` REST endpoint
used by coord. Single shared helper module owns the wire shape.
- Memory/recall calls persist instead of being filtered at storage
time — full audit trail; UI dims them by default with hover-reveal
so heavy memory usage doesn't crowd the narrative.
- Truncation indicator surfaces as a sibling pill (consistent across
interactive + coord) when a tool result hit the 2000-char cap.
- `replayHistory` wraps DOM work in `aria-busy` so screen readers
don't get a chatty announce-flood on long replays.
- `_build_history`'s storage I/O moves off the event loop via a new
`events_replay_prepare` async hook for the SSE path; other async
callers wrap in `asyncio.to_thread`.
Coord parity:
- `/history` REST endpoint decorates tool_calls with verdict +
output_assessment + truncation flag (was previously raw
`load_messages` output).
- Coord JS stamps `judge_verdict` / `heuristic_verdict` from
history-loaded `tc.verdict` so the existing batch render paints
the persisted pill, seeds the verdict cache to dedupe later live
SSE events, and emits an inline `.coord-tool-row-warning` chip
per call instead of a generic chat line.
- Memory/recall dim rule mirrored on `.coord-tool-row[data-tool-name=...]`.
* fix(replay): address PR #461 review feedback + raise tool-result storage cap
Copilot review feedback:
- Sibling-chain dim rule (memory/recall) now adds :focus-within
alongside :hover for .tool-output / .media-embed / .output-warning
/ .tool-output-truncated — keyboard users tabbing into a faded
subtree now get full opacity.
- ``cfg.open_post_load`` is now invoked via ``await asyncio.to_thread``
so its sync ``_build_history`` call (storage I/O for verdict
indexes + message reconstruction) doesn't block the event loop on
every workstream open. Mirrors the SSE replay path that's already
protected via ``events_replay_prepare``.
- Replaced the hardcoded ``2000`` literal in server.py and session.py
with ``TOOL_RESULT_STORAGE_CAP`` from the shared decoration module
so the UI truncation-pill detection can't silently desync from the
storage write side.
While here:
- Raised ``TOOL_RESULT_STORAGE_CAP`` from 2000 → 10000. A 2000-char
clip routinely cut grep / file-read bodies mid-line, leaving the
audit trail useless for retrospective debugging. FTS5 + row size
grow proportionally; the per-tool upper bound is still bounded
upstream by ``_truncate_output``'s context-budget clamp.
- Updated the user-visible truncation-pill tooltip on both
interactive and coord to reflect the new cap.
- ``test_decorates_tool_calls_and_marks_truncated`` now references
the constant instead of a literal so it stays correct on future
cap changes.
Speculative reliability machinery from the Stage 3 push that turned
out not to address any user-visible bug. The actual fixes (state /
activity disjunction in handleChildState, bulk-fetch race fix in
_fetch_live_block, push approve_request via cluster bus) are what
resolved the wedged-row issues. Manual testing showed the per-tab
SSE listener queue depth never climbed past single digits even when
rows were stuck — overflow was never the cause.
Removed
- ``_CRITICAL_EVENT_TYPES`` + ``_put_with_priority`` helper.
- Per-tab listener queue selective drop (back to plain
``contextlib.suppress(queue.Full)`` everywhere).
- ``ClusterCollector._fanout`` reverts to the same.
- WebUI ``_broadcast_intent_verdict`` / ``_broadcast_approval_resolved``
/ ``_broadcast_approve_request`` revert to plain ``put_nowait``.
- ``_queue_stats`` periodic SSE emit + frontend status-bar indicator
+ the supporting CSS rules.
- Broken ``.approval-block`` ``transition: max-height`` /
``max-height: 80vh`` / ``overflow: hidden`` rules — the transition
never fired (nothing toggled max-height) and ``overflow: hidden``
clipped long verdict reasoning. Layout-shift on auto-expand jumps
again, which is preferable to clipped content (Copilot review).
Tidied
- ``_CollectorProtocol`` / ``_ManagerProtocol`` method bodies switch
from ``...`` ellipsis to docstring-only bodies, silencing four
CodeQL "statement has no effect" warnings without changing the
Protocol contract.
5024 passed, ruff + mypy clean.
Lift the Children primitive out of CoordinatorAdapter into universal
SessionManager core primitives, replace the fragile poll + state-event
piggyback paths with first-class cluster bus event types for inline
approval delivery, and clean up the resulting frontend reducer.
Architecture
- New `turnstone/core/children_registry.py` — universal parent → children
+ reverse-lookup primitive with atomic `add_child` (returns parent UI
for race-free dispatch). Lifted from `CoordinatorAdapter`.
- New `turnstone/core/child_source.py` — `ChildSource` Protocol with
`SameNodeChildSource` (in-process via SessionManager state observer)
and `ClusterChildSource` (cross-node via ClusterCollector listener).
- `SessionManager._on_state_change` upgraded to multi-subscriber
(`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated
lock; CLI consumer migrated.
- `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in
the registry; fan-out lives in ClusterChildSource. Backward-compat
property facades dropped; tests updated to use the registry surface.
Cluster bus event vocabulary
- New event types `intent_verdict`, `approval_resolved`,
`approve_request` flow through both `ClusterCollector._apply_delta`
(translation from node SSE) and `emit_console_ws_*` (synthesis on
console pseudo-node).
- `CoordinatorAdapter._dispatch_child_event` re-emits as
`child_ws_intent_verdict` / `child_ws_approval_resolved` /
`child_ws_approve_request` on the parent coord's SSE stream.
- New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` /
`_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI
pushes to the global queue; ConsoleCoordinatorUI pushes to the
collector. `approve_tools` calls `_broadcast_approve_request` right
after setting `_pending_approval` so the items reach the coord tree
immediately, eliminating the bulk-fetch race.
Cleanups
- `pending_approval_detail` piggyback on `ws_state` / `cluster_state`
removed end-to-end. Bulk fetch + explicit verdict / approve-request
push are the canonical carriers.
- Browser `_judgePollTick` 90-second poll loop deleted; push path is
authoritative.
- `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409
retry; replaced with `invalidateLiveBadge` + standard schedule).
- Console `_fetch_live_block` derives `pending_approval` from a
disjunction (`activity_state="approval"` OR `state="attention"`
OR detail present) so the bulk fetch can't return false during the
state-transition race window.
- Coord-side merge guard in `flushLiveFetches` no longer clobbered:
`handleChildState` only stamps `sseUpdatedAt` when authoritatively
clearing detail.
- `child_locality` capability flag removed (was inert dead code).
Reliability
- Selective drop on listener queue overflow: critical event types
(verdicts, approvals, ws_closed, child_ws_*) evict one oldest item
to make room rather than dropping themselves on a full queue.
Best-effort events (state ticks, content tokens, status, activity)
drop as before. Applied to `SessionUIBase._enqueue`,
`ClusterCollector._fanout`, and the `WebUI._global_queue` puts in
the new broadcast hooks.
- `_state_subscribers` snapshot under a dedicated lock so concurrent
subscribe / unsubscribe during dispatch can't shift the iterator.
UX / a11y
- Loading placeholder in renderChildRow keeps row height stable while
the bulk fetch is in-flight (sr-friendly aria-label).
- Focus preservation across `_renderChildrenNow` (capture +
restore by row + marker) and across targeted `_updateChildRow` swaps.
- Layout-shift transition on the approval block max-height; respects
`prefers-reduced-motion`.
- Sidebar pending count: `(N children · M pending)`.
- Risk pill `aria-label` spells out level + confidence for SR users.
- Per-coord SSE listener queue depth surfaced in the status bar
(`queue N/500`) with color escalation (warn at >50%, danger at >80%).
Tests
- 305+ test changes across 8 files. New unit tests for
`ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber
observer), the new collector emit + apply_delta cases, the dispatch
cases for new event types, the broadcast hook overrides on both
WebUI and ConsoleCoordinatorUI, and the focus / placeholder /
pending-count frontend assertions in `test_coordinator_page.py`.
5024 passed, ruff + mypy clean.
* feat(console): multi-select delete UX for Saved Coordinators
Mirror the per-server "Saved Workstreams" multi-select delete onto the
console's "Saved Coordinators" section. Coordinator deletes go through
the existing routing proxy at POST /v1/api/route/workstreams/delete
(body-keyed by ws_id, since coordinators live on the node that owns
them) — no backend change required.
Pagination caps the visible page (and therefore the Select-All fan-out)
at 24. Without it, a Select-All on a busy cluster would pin the
console proxy pool with hundreds of parallel deletes through the
fan-out router. While in delete mode the saved-coordinators list is
frozen against SSE re-renders so visible cards don't shuffle out from
under the user's selections (drained on cancel / post-delete close).
Refactor: shared logic now lives in turnstone/shared_static/cards.{css,js}.
* .ws-delete-* CSS moved out of ui/static/style.css into the shared
sheet alongside .dashboard-card; the existing ui/static modal
markup picks up class hooks instead of id-scoped rules.
* createSavedCardsController() owns mode state, checkbox decoration,
toolbar wiring, focus trap, modal lifecycle, and batch fan-out.
Both ui/static (Saved Workstreams) and console/static (Saved
Coordinators) instantiate one controller; ui/static is now ~300
LOC lighter as a result.
* Internalises stale-selection prune across SSE re-renders, the
wsId->item lookup map (was O(selected x N)), and the aria-hidden
wrap on the toggle button's emoji glyph.
Designer review tightened the affordance:
* Modal close restores focus to the toggle button (was landing on
<body>) — WCAG 2.4.3.
* Modal [role="alert"] gets a red-chip treatment when populated,
stays invisible at rest via :not(:empty).
* Pagination consolidated onto the existing .pagination control
(terse "X / Y" label + arrow-glyph buttons) instead of a parallel
.coord-pagination treatment.
* Filled destructive buttons darkened to #dc2626 in dark theme so
the white label clears WCAG AA contrast (was 3.0:1 on --red).
Light theme keeps --red unchanged (5.9:1 already passes).
* Toolbar wraps below 700px viewport — Delete Selected drops to its
own full-width row underneath count + Cancel + Select All for
thumb-target separation.
* .ws-card-check:focus-visible outline + word-break on
.ws-delete-item for narrow-modal long aliases.
* fix(cards): address Copilot review feedback on PR #458
* closeModal focus restore now falls back to the section toggle button
(opts.buttonId) when prevFocus is hidden or detached. The post-delete
Close path runs cancel() before closeModal(), which puts the bar at
display:none — so the captured prevFocus (the bar's "Delete Selected"
button) is no longer focusable and focus would land on <body>,
defeating the WCAG 2.4.3 fix. Esc / Cancel paths still land on the
original focus owner because the bar stays visible in those flows.
* Saved Coordinators onClose drains _savedCoordsRetry before reloading.
Without it, SSE events that arrived during the delete-mode freeze
leave the retry flag true, so loadSavedCoordinators's .finally()
re-fires a second fetch immediately after the first resolves. Mirrors
the same idiom in cancelCoordDeleteMode.
Three issues from the Copilot review on PR #457:
1. SQLite race in bulk_close_stale_orphans (Copilot): the SELECT-then-
UPDATE flow doesn't re-apply the eligibility predicates on the
UPDATE, so a row that gets touch_workstream-bumped (or set_state-
transitioned) between the two statements would still be flipped
to closed. Postgres dodges this via UPDATE...RETURNING (one atomic
statement); SQLite needs the explicit re-application. Fix: rebuild
the WHERE conditions list once, apply on both SELECT and UPDATE,
then SELECT-back by ``state='closed' AND updated=now`` to get the
accurate closed-id list. A row that became fresh between the two
statements skips the UPDATE entirely.
2. SQLite IN-clause bind-parameter limit (Copilot): default 999 cap
could be exceeded on a backlog reap (e.g. after a long outage).
Chunked the candidate id list at 500 — same chunk size
prune_workstreams (line 453) uses for the same reason.
3. Wall-clock-dependent test asserts (Copilot, two locations): the
tests asserted ``updated > '2024-01-01T00:00:00'`` which is fragile
on systems with skewed clocks or pre-2024 dates. Replaced with
``updated != stale_seed`` — captures the same intent (the value
was bumped) without depending on wall-clock date.
Two ``...``-as-no-op flags from github-code-quality were false
positives — ``...`` is the standard Python idiom for Protocol method
bodies and matches every other method in _protocol.py. No code change.
Replaces the ``node_id == self_node_id`` orphan-scoping heuristic from
earlier on this branch with liveness-based scoping using
``services.last_heartbeat``. The heuristic was wrong for the post-#384
world: PR #384 (refactor: replace hash-ring rebalancer with rendezvous
hashing) deleted the rebalancer that used to keep workstreams.node_id
pointing at a live node. Without it, ``workstreams.node_id`` is now
stamped at create time and never updated, so in containerized
deployments with dynamic hostnames a dead pod's rows have ``node_id``
matching no surviving service — they'd accumulate forever under the old
heuristic.
services.last_heartbeat is the same primitive the rendezvous router
uses for routing. Reusing it here keeps reap scoping aligned with
routing: dead pods' rows fall out of the live set after the heartbeat
window and become reapable; alive pods' rows stay protected as long as
they heartbeat.
Mechanics:
- ``bulk_close_stale_orphans`` parameter renamed
``node_id: str | None`` → ``live_node_ids: list[str] | None``. The
WHERE clause becomes ``(node_id IS NULL OR node_id NOT IN
live_node_ids)``. ``None`` skips the filter entirely (single-process
/ tests / operator backfill). ``[]`` treats every row as
unprotected.
- ``SessionManager.close_idle`` pass 2 calls
``storage.list_services(self._service_type)`` to enumerate live
peers, passes their service_ids as ``live_node_ids``. ``_service_type``
is derived from ``self.kind`` (INTERACTIVE→"server",
COORDINATOR→"console") via a module-level mapping — no constructor
param, so production wiring can't miswire the kind/service_type
pairing.
- list_services failure → pass 2 is skipped this tick (conservative;
never reap when liveness state is unknown). Pass 1 still runs.
- ``workstreams.node_id`` with NULL value is always eligible — defends
against ANSI ``NULL NOT IN (...)`` evaluating to NULL (not TRUE) and
silently protecting orphans forever.
- Migration 048 simplified to ``(kind, updated)``; the new query's
``NOT IN (small list)`` predicate against an unbounded-cardinality
column doesn't index well, so leading ``node_id`` would just add
write cost.
Tests cover the live-services protection (own/dead/null cases), the
empty-peers reap-all case, the list_services-failure conservative
fallback, both kind/service_type pairings (interactive→"server",
coordinator→"console"), and the combined live_node_ids +
exclude_ws_ids filter matrix.
bulk_close_stale_orphans runs every min(300s, idle_timeout/4) on
every server and console process. Its WHERE shape is:
WHERE kind = ?
AND state IN ('idle','thinking','attention','running')
AND updated < ?
AND node_id = ? -- multi-node interactive only
At current scale the existing single-column indexes are sufficient —
idx_workstreams_state prunes to non-closed and the planner filters the
rest sequentially. At 100k+ rows that filter becomes a tablescan-
shaped cost.
A partial index covering only BULK_CLOSE_STATE_VALUES rows matches the
reaper's query exactly while staying tiny — closed rows (typically
95%+ of the table) and error rows are excluded, so the index is
roughly 5% the size a full multi-column index would be. Write
amplification only kicks in for transitions touching one of the four
covered states.
Column order (node_id, kind, updated): node_id is the most selective
filter for multi-node interactive (each server prunes to its own
node's rows), kind second so coord-only and interactive-only queries
within a node still get index-only scans, updated last so the range
comparison rides the trailing column.
Postgres uses CREATE INDEX CONCURRENTLY so the build is non-blocking
on a live system; SQLite has no concurrent concept and the table-
level write lock already serializes, so a plain CREATE INDEX is fine.
The console's coord SessionManager had no idle thread — close_idle was
never called for coordinator workstreams. This is the worse half of
the lifecycle leak: the dashboard filters via the in-memory pool, so
DB-only orphan coords were invisible. At empirical diagnosis,
coord closure was 16% (10 closed / 64 total) vs interactive 63%.
Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's
_idle_cleanup_thread but skipping the rate-limiter / global-queue arms
the console doesn't have. Started from the lifespan when coord_mgr is
constructed and server.workstream_idle_timeout > 0 (reuses the
existing setting — same cadence works for both kinds).
Initial sweep runs INSIDE the thread before the first sleep, not
synchronously in the lifespan: cold-start orphans are reaped without
blocking Starlette boot. Important because cold start with many DB
orphans (the precise condition this code targets) is exactly when the
UPDATE is most likely to be slow.
Helper takes an optional stop_event parameter purely for tests —
production callers pass None and the daemon runs for process lifetime.
This avoids the SystemExit-from-stub + module-wide filterwarnings
fragility a previous iteration relied on.
Four tests: initial sweep runs before first sleep, ticks fire each
loop, exceptions don't kill the thread, stop_event exits cleanly.
Real bug: workstream rows accumulate in non-closed states (idle,
thinking, attention, running) when their owning process restarts or
crashes. Empirical diagnosis on a live deployment found ~60 stuck
coord rows in DB invisible to the in-memory-keyed dashboard, plus
100+ interactive rows older than the 2h timeout (one stuck "thinking"
for 2 weeks — impossible across a process restart).
Root cause: close_idle iterates self._workstreams.values() — only the
loaded subset. Anything left behind by a prior process incarnation
sits in DB forever because nothing ever re-loads it.
This commit gives close_idle a second pass.
Pass 1 (existing, unchanged): close loaded IDLE rows whose
ws.last_active (monotonic) is past timeout. IDLE-only so legitimately-
attentive rows (waiting for user response) stay live.
Pass 2 (new): bulk-close DB rows of this manager's kind whose updated
is past the wall-clock cutoff and which aren't currently loaded.
Closes the broader BULK_CLOSE_STATE_VALUES set — any matching row is
by definition not loaded by any process and cannot be in a live
interaction. Scoped by self._node_id so a sibling node can't reap
rows we own (multi-node interactive correctness). No emit_closed —
never-loaded rows have no SSE listeners expecting them.
Lock invariant: pass 1 holds self._lock briefly to snapshot victims
and pop them (existing behavior). Pass 2 holds self._lock briefly to
snapshot the loaded keys, then releases before the DB UPDATE so a slow
reaper query can't block create/get/set_state.
Also fixes a same-process race in open(): the rehydrate path read DB,
released the manager lock, then re-acquired to install — a concurrent
pass 2 between the two acquisitions snapshots loaded keys without the
in-flight ws_id, and could clobber its DB row to closed. open() now
calls touch_workstream(ws_id) on rehydrate so the row's updated is
fresh against any pass-2 cutoff. Pure timestamp write is safe against
concurrent close() (close still wins on the state column).
Three new tests cover the DB orphan pass (basic, exclude-loaded, kind
filter) plus node_id scoping (own/foreign rows, None-skips-filter) and
the open() rehydrate touch.
Two new methods on the StorageBackend Protocol, with implementations on
both Postgres (UPDATE ... RETURNING) and SQLite (SELECT-then-UPDATE in
one transaction). No callers yet — wiring lands in subsequent commits.
bulk_close_stale_orphans(kind, cutoff, exclude_ws_ids, node_id=None)
flips rows in BULK_CLOSE_STATE_VALUES (idle/thinking/attention/running)
to closed when their updated timestamp is lex-older than cutoff. The
node_id filter scopes the reap to a single node's partition — required
for multi-node interactive deployments where each node only has
authority over its own workstreams.node_id rows. Excludes loaded ids
so the in-memory pass owns those.
touch_workstream(ws_id) bumps updated without changing state. Used by
the open() rehydrate path to defend against the orphan reaper clobbering
a freshly-loaded row whose DB updated is older than the cutoff. Pure
timestamp write is safe against concurrent close() because close still
wins on the state column.
BULK_CLOSE_STATE_VALUES is centralized in workstream.py so the two
backend implementations and FakeStorage all agree; if a new transient
state is added to WorkstreamState, deciding whether it joins this set
is part of the change rather than an after-the-fact audit across three
files.
Storage tests (run against both backends via the conftest fixture) cover
the kind/state/cutoff/exclude/node_id matrix plus touch_workstream.
The themed ``tool_reminder`` bubble below the tool block already
shows the metacog text, and the tool block immediately above it
carries the tool name — so a separate gray ``[repeat: list_workstreams()
called with same arguments]`` info line was just duplicate visual
noise (operator-visible in the screenshot below the bubble).
Drop the ``ui.on_info`` call inside ``_apply_post_execute_advisories``
that emitted the diagnostic line. Update the docstring to reflect
that the bubble is the canonical signal. Rename
``test_emit_repeat_ui_line_on_streak_fire`` →
``test_no_legacy_repeat_info_line_on_streak_fire`` and invert the
assertion.
CI typecheck failed because ``WorkstreamTerminalUI(TerminalUI)``
inherits from ``SessionUI`` (the Protocol), and the Protocol's
``on_user_reminder`` / ``on_tool_reminder`` declarations have empty
bodies — mypy treats those as implicitly abstract, so the subclass
became un-instantiable.
Add real implementations on ``TerminalUI`` that render reminders as
``[metacognition · type] text`` lines in yellow. This also restores
the metacog signal on the CLI surface (the legacy
``[metacognition: nudge injected — …]`` info-line went away with
``_emit_nudge_ping``; without this commit the CLI showed no signal
at all for metacog nudges). Tool-channel and user-channel render
identically because terminal output is anchored by stdout flow
rather than by DOM anchor — the line lands directly after the
message it advises.
Address Copilot's review feedback on PR #456 — the docstrings and
inline comments hadn't all caught up with the architectural shift
across the branch:
- ``_apply_reminders_for_provider`` docstring: "every user message"
→ role-agnostic, since tool messages also carry ``_reminders``
(tool_error / repeat).
- ``_mark_reminders_delivered`` docstring: same role-agnostic
update; explicitly note both channels.
- ``_append_user_turn`` callsite comment near
``_attach_pending_user_reminders``: still described splicing
``<system-reminder>`` blocks into user content; updated to
reflect the side-channel attach + transient-copy splice at the
provider boundary.
- ``_build_history`` block comment: was user-message-only; now
mentions tool messages and both ``user_reminder`` /
``tool_reminder`` SSE events.
- ``_build_history`` propagation comment: same role-agnostic note
on the per-entry surface.
- ``app.js`` ``user_reminder`` SSE handler comment: said the
bubble renders "above" the user message, but
``insertAdjacentElement('afterend', el)`` drops it BELOW.
- ``app.js`` ``replayHistory`` comment: said "insertBefore drops
the reminder directly above the just-rendered user bubble";
same fix — bubble lands BELOW.
No behaviour change.
The repeat-detection block in ``_apply_post_execute_advisories`` had
a leftover "clear streak when a write tool succeeded" branch from
when ``RepeatDetector`` tracked cumulative counts. With the
consecutive-streak semantics introduced earlier in the branch the
branch became:
1. Redundant — any different (name, args) signature already resets
the streak via ``RepeatDetector.record``, so an intervening
read/write naturally breaks the streak.
2. Actively wrong — the clear runs ONCE at the top of each
``_apply_post_execute_advisories`` call, before the per-result
loop records sigs. In a single parallel batch
``[bash, bash, bash]`` the clear runs once and then three
``record`` calls accumulate to count=3 in the same call → fires.
But across three sequential turns, each turn calls
``_apply_post_execute_advisories`` fresh, the clear runs at the
top of each call, and only one ``record`` per call follows — so
the count never gets above 1 and the canonical
"small local model stuck on ``bash('echo test')``" pattern
never triggered the nudge.
The asymmetry only existed for successful calls — failures don't
satisfy the ``not _tool_error_flags.get(tc["id"])`` predicate, so
the clear didn't fire and sequential failures already worked. The
fix is to drop the clear entirely; ``RepeatDetector``'s
consecutive-streak semantics handle every case uniformly.
Tests:
- ``test_successful_write_clears_streak`` →
``test_intervening_different_call_resets_streak`` —
rewords the assertion to reflect the actual mechanism (any
different sig resets, write-or-otherwise) since "writes clear"
was the bug, not the contract.
- ``test_failed_write_does_not_clear_streak`` →
``test_sequential_bash_failures_fire_repeat`` — same shape, just
framing fixed.
- New ``test_sequential_bash_same_command_fires_repeat`` —
regression for the bug user hit (three sequential successful
``bash('echo test')`` calls now correctly fire the nudge).
The yellow themed reminder card introduced for user-channel nudges
(correction / denial / resume / start / completion) now also fronts
tool-channel nudges (tool_error / repeat). Pre-fix the tool channel
shipped its reminders inside the tool-result envelope via
``wrap_tool_result``, leaking the ``<system-reminder>`` block into
``self.messages`` content (same problem the user channel had before
the side-channel refactor) and surfacing the legacy gray
``[metacognition: nudge injected — …]`` info line as the only
operator-visible signal — duplicated alongside the new themed bubble
for user-channel nudges.
Tool-channel parity:
- ``_collect_advisories`` now returns
``(persistent_advisories, metacog_reminders)``. Persistent
advisories (``GuardAdvisory`` / ``UserInterjection``) keep
riding ``wrap_tool_result`` because they ARE conversation
history. Metacognitive reminders extract to the second tuple
element; the caller attaches them to the tool message dict's
``_reminders`` side-channel and emits ``on_tool_reminder``.
- ``_apply_reminders_for_provider`` already handles ``_reminders``
on any role, so the tool-channel splice into wire content is
free. ``_build_history`` also already propagates
``entry["reminders"]`` regardless of role, so reload renders the
bubble too.
- ``SessionUI`` Protocol gains ``on_tool_reminder(reminders,
tool_call_id)``; ``SessionUIBase`` enqueues a ``tool_reminder``
SSE event with the ``tool_call_id`` anchor.
- ``_emit_nudge_ping`` had no remaining callers and was removed —
the themed bubble (live SSE + ``/history`` reload) is the
canonical operator signal for both channels now.
UI polish (the four fixes the screenshot caught for the user
channel + their tool-channel mirror):
- Bubble renders BELOW the message it advises (semantically: a
hint to the model right before its turn). ``addUserReminder``
swaps ``insertBefore`` for ``insertAdjacentElement('afterend',
el)``; ``addToolReminder`` anchors below the ``.ts-approval``
block whose tool result triggered the batch's reminder.
- Label uses the full feature name ``metacognition`` (was the
``metacog`` shorthand).
- Card width / alignment inherits from the base ``.msg`` rule —
``align-self: flex-end`` and the explicit ``max-width`` are
gone, so the card matches the user / assistant column instead
of pinning right-aligned narrow.
- The legacy ``[metacognition: nudge injected — …]`` gray info
line is gone for both channels.
Frontend additions:
- ``Pane.prototype.addToolReminder(reminders, toolCallId)``
anchors below the ``.ts-approval`` block (live: by
``data-call-id``; replay: by "last block in messagesEl"
fallback, which is correct because messages render in order).
- SSE switch case ``"tool_reminder"`` calls ``addToolReminder``.
- ``replayHistory``'s tool-message branch now calls
``addToolReminder`` when ``msg.reminders`` is present.
- ``addUserReminder`` advances its anchor on each loop iteration
so multiple reminders stack in queued order rather than
reversed.
Coord console parity:
- ``coordinator.js`` gains ``appendReminderBubble`` /
``appendUserReminderLive`` / ``appendToolReminderLive`` mirroring
the interactive UI. The tool-channel anchor walks
``toolRows[callId].batch`` to attach below the
``.coord-tool-batch`` construct (one bubble per dispatch turn,
matching the "one nudge per batch even with many failing tools"
drain).
- SSE switch handles ``user_reminder`` and ``tool_reminder`` on
the coord conversation surface.
- ``/history`` replay propagates ``msg.reminders`` for user and
tool messages — same wire shape as the interactive pane.
- ``.msg.user-reminder`` styles moved to
``shared_static/chat.css`` so both surfaces inherit the same
yellow themed bubble from the shared base.
Defensive read on ``_apply_reminders_for_provider`` (per Copilot
review on the closed PR): a malformed ``_reminders`` entry (string,
None, etc. — corruption / partial state) used to abort ``send`` via
AttributeError on the ``.get("text", "")`` call. Filter to dicts
before building the block, mirroring the same filter
``_build_history`` already applies on the wire-out side; an
all-malformed list passes through as no-reminders.
Tests:
- ``test_collect_advisories_drains_tool_buffer_on_last_result``
rewritten to assert the ``(persistent, metacog)`` tuple shape
and that ``MetacognitiveAdvisory`` no longer appears in the
persistent list.
- ``test_collect_advisories_holds_*`` and ``_drops_*`` updated for
tuple return.
- ``test_attach_emits_visibility_ping`` /
``test_collect_advisories_emits_visibility_ping`` inverted to
assert the legacy gray line is gone on both channels.
- ``TestSessionUIBaseToolReminderHook`` covers the new SSE event
shape with the ``tool_call_id`` anchor.
- ``test_malformed_reminders_filtered_out`` and
``test_all_malformed_reminders_passes_through`` cover the
Copilot-flagged defensive filter.
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context. The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.
Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``). At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean. ``sanitize_messages`` drops the sibling key on the wire.
Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call. ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised). ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.
UI surface:
- ``SessionUIBase.on_user_reminder`` enqueues a
``{type: "user_reminder", reminders: [...]}`` SSE event with the
same shape ``_build_history`` surfaces.
- ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
pill-styled) anchored above the user message it advises, both
live and on history replay.
- ``replayHistory`` renders ``addUserMessage`` before
``addUserReminder`` so the anchor lookup finds the just-rendered
turn (not a prior one).
- Multi-tab caveat documented inline: non-originating tabs receive
no ``user_message`` SSE event today, so a reminder may anchor to
a stale prior bubble until ``/history`` reload corrects it.
Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer. Both
now drain through a shared ``_drain_pending_advisories`` helper.
Removed the ``/history`` regex strip — the side-channel approach
makes it redundant. Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).
Tests:
- ``TestApplyRemindersForProvider`` — pass-through-by-reference,
string + list content splice, escape on user-typed wrapper tags,
multi-reminder ordering, source-untouched invariant, delivered
flag skip path, fallback for unexpected content shape.
- ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
no-flag, only marks user messages with reminders.
- ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
msgs when provided, falls back when not.
- ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
the user buffer.
- ``TestReminderSidechannelIsolation`` — compaction's
``_format_messages_for_summary`` and the title-gen extraction
loop cannot see reminders by construction.
- ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
enqueues the right SSE shape.
- ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
propagation, absent / empty / multi / coexist-with-attachments
cases, malformed input filtering, all-malformed elision.
- ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
``_reminders`` and ``_reminders_delivered``.
Cleanup pass on the metacognitive nudge stack — restores pre-split
errored-counts-toward-repeat behaviour and tightens the is_error
plumbing through the per-batch advisory hook.
The per-batch hook in ``_run_loop`` was duplicating the is_error
signal: ``self._tool_error_flags`` (set by ``_report_tool_result``)
and a string-prefix tuple (``Error`` / ``JSON parse error`` / …).
Two truth sources is what got us here — bash commands that exit
non-zero with normal stdout matched the flag but not the prefix,
the deny path matched the prefix but not the flag, and the result
was that stuck-loop detection silently broke for the most common
failure mode (the model bashing the same broken command).
Single source of truth now:
- ``_execute_tools.run_one`` deny branch routes through
``_report_tool_result(is_error=True)`` so denied calls populate
``_tool_error_flags`` like every other error path.
- The error-prefix tuple is gone; the write-success-clear gate and
the tool-error-nudge gate both read ``_tool_error_flags`` only.
Repeat-detection state moves from a ``set[str]`` (fired on the second
identical call, ignored errors entirely) to a ``RepeatDetector``
helper in ``metacognition.py`` with consecutive-streak semantics:
- Threshold raised from 2 to 3 — two-in-a-row was noisy on
legitimate transient retries; three is the cheapest stuck-loop
signal.
- Recording a different signature resets the count, so [A, A, B, A]
is two short streaks of 2 and not a streak of 4. Bounded by O(1)
state regardless of session length.
- Errored calls now count toward the streak (the split into a
separate metacog module unintentionally introduced a "skip errors"
branch — restored).
While there:
- ``metacognition._COOLDOWN_SECS`` default aligned to 300s (matches
``MemoryConfig.nudge_cooldown`` and the ``memory.nudge_cooldown``
config-store default; was set to 30 by an earlier investigation).
- The per-batch advisory block (~80 lines of mixed orchestration
inside ``_run_loop``) is extracted to
``ChatSession._apply_post_execute_advisories`` so the wired
behaviour is testable without driving ``_run_loop`` end-to-end.
Producer extraction to a dedicated module is deferred to a
follow-up; advisory producers all live on ``ChatSession`` for
now per existing convention.
- Frontend ``appendToolOutput`` (turnstone/ui/static/app.js) now
skips rendering when the parent approval block is denied or
the output starts with ``Denied by user`` / ``Blocked``,
mirroring the history-replay guard at ``_build_history``.
Previously the live SSE path didn't need this guard because
the deny path never emitted a ``tool_result`` event; the
is_error routing change above means it does now, so without
this guard the badge from ``resolveApproval`` and the SSE
output would both render.
Tests: 8 unit tests for ``RepeatDetector`` covering streak,
threshold, clear, and intervening-sig reset; 9 integration tests
for ``_apply_post_execute_advisories`` covering the wired
behaviour (3-identical fires warning + advisory + UI line, errored
calls count toward streak as a regression guard, intervening sig
resets streak, successful write clears, failed write does not,
JSON outputs tracked but not inline-warned, tool_error nudge gates
on memory_count, repeat UI line emitted on streak fire).
The pre-existing comment said pending_approval_detail "rides on
every ws_state event" — that overstated the case. The node-side
emit is gated on ``_pending_approval is not None`` so the field is
absent on the steady-state broadcast and possibly null on a node
mid-rolling-upgrade. The handleChildState fallback already
handles both cases; only the comment was wrong.
Inline child approve/deny in the coord tree UI was rendering downstream
of the bulk-live cache (``GET /v1/api/cluster/ws/live``), not the SSE
stream. ``child_ws_state`` events were tiny notifications that fired
an urgent live-bulk fetch on every activity_state transition into/out
of "approval", just to pick up the rich ``pending_approval_detail``
payload. With multiple coord tabs and multi-child workstreams, that
urgent-fetch pattern compounded the SSE-executor pressure Shape A
is unwinding.
Thread the field through every layer so the SSE event itself carries
the rich payload — browser mutates ``liveBadgeCache`` directly,
no urgent fetch:
1. Node ``WebUI._broadcast_state`` emits ``pending_approval_detail``
on ``ws_state`` events. Gated on ``_pending_approval is not None``
so the per-broadcast verdict-cache deepcopy only runs when there
is actually an approval pending. ``_build_node_snapshot`` also
projects the field so the console's reconnect-via-snapshot
resync path delivers it (without this the new collector
forwarding would never see the field on a snapshot row).
2. Console ``ClusterCollector._apply_delta`` (live ``ws_state``
forwarding) and ``_reconcile_node`` (snapshot resync diff) both
forward the field on the emitted ``cluster_state`` event, AND
``_apply_delta`` persists it on the cached ``ws`` dict so the
``get_node_detail`` / ``get_snapshot`` endpoints between
reconciliations don't render stale approve/deny buttons.
3. ``CoordinatorAdapter._dispatch_child_event`` re-emits the field
on the ``child_ws_state`` event sent to coord listener queues.
4. Frontend ``handleChildState`` reads ``ev.pending_approval_detail``
and writes it directly into ``liveBadgeCache``, tagging the
entry with ``sseUpdatedAt``. ``flushLiveFetches`` honors that
tag for ``SSE_AUTHORITATIVE_MS`` (3s) — the upstream
``/dashboard`` cache has its own ~2s TTL, so a bulk-poll
landing right after a transition can otherwise clobber the
fresh SSE-set state with pre-transition data.
The pre-fix ``enteredApproval`` / ``leftApproval`` urgent-fetch
branch is removed. The 409 stale-call_id retry path keeps its own
urgent fetch — that's a different scenario.
Tests cover the forwarding contract at every layer, the broadcast
gate (event includes the field when an approval is pending,
omits it otherwise, and clears after resolution), and the
``flushLiveFetches`` merge-guard structural shape so a refactor
that keeps the symbols but inverts the comparison or drops the
``prev.live`` check can't pass silently.
``coordinator_children`` was calling ``storage.list_workstreams``
directly on the event loop, ``coordinator_tasks`` did the same with
``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called
from both handlers, plus ``coordinator_history`` and
``_resolve_coord_session``) did the same with
``storage.get_workstream`` on its cold-cache path.
The cold-cache resolver path is hit on every console restart,
coordinator eviction, and console proxy hop — exactly when the
event loop is most contended. Three coord tabs reconnecting after a
brief network blip = three serial event-loop blocks per call site.
Other lifted handlers in this file already use
``asyncio.to_thread``; bring all four call sites onto the same
pattern.
Convert ``_resolve_coordinator_or_404`` to ``async def`` and update
its four call sites to ``await``. Exception flow is unchanged.
Each coord ``events`` SSE listener parks a thread on
``client_queue.get(timeout=5)`` for the connection lifetime. The
console's coord endpoint was wiring no ``sse_executor_lookup`` on
``coord_endpoint_config``, so those parks landed on Python's default
ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every
other ``asyncio.to_thread`` caller (storage, router, audit). A few
coord tabs against a multi-child workstream would stall new request
handlers waiting for a worker thread.
Mirror the interactive-side precedent (the ``sse_executor`` /
``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a
dedicated 200-thread ``coord_sse_executor`` in the console lifespan
and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``.
Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()``
so no new listeners arrive at a dying pool. ``cancel_futures=True``
discards queued-but-not-started futures during teardown.
Update the stale comment on the interactive-side wiring that claimed
"coord wires None and falls back to the default executor" — it now
points at the console's matching wire.
Three follow-ups from Copilot's round-2 review on #453.
ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present). Operator looking at logs for a
config.toml typo would see the wrong cause. Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through. Behavior unchanged — existing registry still
preserved on every ValueError path.
Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load. ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve. Today shutdown() iterates
empty dicts. Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.
Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``. The
explicit probe was removed in commit 1ba17ed when the helper switched
to ``load_model_registry(..., strict=True)``; the test name and
docstring still talked about a probe. Updated wording reflects that
the loader's strict-mode re-raise is what the helper catches now.
132 tests pass.
Hygiene follow-ups from the multi-stage code review on #453.
perf-1 — sync helper called from async route handlers
``_refresh_coord_registry`` runs two sync DB reads and a registry reload
that takes ``_client_lock``; calling it directly from an async handler
held the event loop for the duration. All four call sites now
``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the
pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``).
perf-3 — ModelRegistry.reload() tore down all clients unconditionally
The reload always closed every cached client and provider, even when
the changed fields (``model``, ``temperature``, ``context_window``)
didn't touch the connection target. Now selective: clients drop only
when alias removed or ``(base_url, api_key, provider)`` differs;
providers drop only when alias removed or ``provider`` string differs.
Keeps connection pools warm across the common admin-edit case where
only metadata changed. Two new ``test_model_registry`` cases lock the
keep-warm vs drop-on-change behaviour, and the existing
``test_reload_clears_clients`` was updated (it asserted the old
overly-aggressive contract) into
``test_reload_keeps_clients_when_connection_target_unchanged``.
q-5 — helper rename
``_refresh_console_coord_registry`` → ``_refresh_coord_registry``. The
``console_`` prefix was redundant given the function lives in
``turnstone/console/server.py`` and sibling helpers there
(``_notify_nodes_model_reload``, ``_publish_config_change``,
``_collect_model_status``) all omit it.
q-1 — shared test middleware
``tests/test_admin_model_registry_refresh`` now imports the
header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers``
and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the
``TestClient``. The local hardcoded variant duplicated infrastructure
the helper module exists to centralise.
q-3 — multi-alias test registry
``_make_registry`` extracted a ``_make_config`` helper and gained an
``extras={alias: model}`` param so multi-alias scenarios stop
hand-building ``ModelConfig`` literals.
``test_delete_endpoint_refreshes_registry`` now uses the helper.
310 tests pass across the related coordinator + model surfaces.
bug-3 / q-2 from the multi-stage review on #453: the previous test
``test_update_endpoint_with_empty_body_does_not_blow_up`` asserted only
that the registry's model name was unchanged after an empty PUT, which
holds whether or not the refresh ran (DB row matches registry → refresh
is idempotent). A regression that always called
``_refresh_console_coord_registry`` — exactly the gate this test was
meant to lock — would have left the assertion green.
Rename to ``test_update_endpoint_skips_refresh_on_empty_body`` and spy
on the helper via ``monkeypatch.setattr``. Empty-body PUT must register
zero calls; any future change that drops the ``if updates:`` gate now
fails loudly.
Two correctness follow-ups from the multi-stage code review on #453.
bug-2 / perf-2 (DB probe was theatre + double scan)
The previous probe defended nothing the loader didn't already swallow
on the next line: ``load_model_registry``'s row-loop catches Exception
internally, so a transient DB error after the probe still degrades to
a config.toml-only registry that ``existing.reload()`` would apply,
silently dropping every DB-sourced alias. And on the happy path each
CRUD paid for two scans of ``model_definitions``.
Add a ``strict: bool = False`` flag to ``load_model_registry``. When
strict, the row-loop's except re-raises instead of swallowing. The
helper passes ``strict=True`` and drops the probe — single DB scan,
real failure isolation, the loader's silent fallback can no longer
mask a partial-result regression. Default ``strict=False`` so CLI /
lifespan callers keep their boot-with-config-fallback behaviour.
bug-1 (shutdown could escape after a successful reload)
``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the
helper's ``finally`` block ran it outside the try/except. A raising
close() after a successful ``existing.reload()`` would surface as 500
with the registry already mutated and the audit row already recording
success. Wrap ``new_registry.shutdown()`` in its own try/except that
matches the helper's belt-and-suspenders error policy elsewhere.
The helper's docstring also drops the obsolete probe paragraph; the
``if existing is None: return`` branch gets a one-line inline comment
about the boot-from-empty case (the multi-paragraph version restated
behaviour the line itself documents).
129 tests pass (test_admin_model_registry_refresh + test_model_registry).
Two follow-ups from Copilot review of #453:
1. ``load_model_registry`` swallows storage read errors internally
(logs + continues with config.toml-only models). Without a strict
probe in the helper, a transient DB outage on an admin CRUD would
apply a truncated registry that drops every DB-sourced alias —
silently, since the loader returns a non-empty registry built from
``[models.*]`` config.toml entries. Add an explicit
``storage.list_model_definitions(enabled_only=True)`` probe before
the loader call so the failure is visible here and the existing
registry is preserved on outage.
2. The previous docstring claimed ``admin_model_reload`` "has its own
boot-from-empty story." It doesn't — it just calls this helper,
which no-ops when ``coord_registry`` is None. When no model rows
existed at boot, lifespan leaves the entire coord subsystem
uninitialized (no ``coord_mgr``, no ``coord_adapter``, no
``session_factory``), and a console restart remains required after
the operator adds the first row. Tighten the docstring to admit
that limitation rather than overstating the helper's reach.
New test ``test_helper_preserves_registry_when_db_probe_fails``
monkeypatches ``list_model_definitions`` to raise and asserts the
existing registry stays intact.
The console builds ``app.state.coord_registry`` once at lifespan startup
and the coordinator session factory closes over that exact instance.
Until now, the model-definition admin endpoints (create/update/delete)
wrote to the DB but never touched the in-process registry — and the
explicit reload button only fanned out to nodes via HTTP, also leaving
the console's own registry stale.
Symptom: an operator who changed the underlying model name behind a
local-LLM alias (same alias, same endpoint) saw the DB row update
immediately, but coordinator sessions kept calling the prior model
name until the console process was restarted.
Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh
ModelRegistry from DB and applies it to ``app.state.coord_registry``
via the existing thread-safe ``ModelRegistry.reload()`` — in-place
mutation preserves object identity so the factory closure keeps
working, and active coord sessions auto-pick up the swap on their
next ``send()`` via ``ChatSession._refresh_model_from_registry``.
Wired into four endpoints in ``console/server.py``:
- ``admin_create_model_definition`` — after the DB write
- ``admin_update_model_definition`` — after the DB write, gated on
``if updates:`` so a no-op PUT skips the rebuild
- ``admin_delete_model_definition`` — after the DB write
- ``admin_model_reload`` — between ``_publish_config_change`` and
``_notify_nodes_model_reload`` so the console mirrors what the
reload broadcasts to nodes
Failure isolation: a load or reload error leaves the existing registry
intact (logged + swallowed). Coord stays usable while the operator
investigates; the explicit reload remains the user-facing recovery path.
No node fan-out on CRUD — the explicit reload button continues to gate
cluster-wide HTTP propagation, preserving today's UX semantics on shared
clusters.
Tests in ``tests/test_admin_model_registry_refresh.py`` cover:
- helper-level: rebuild from DB, identity preservation, no-op when
registry is None, preservation on load failure / no-enabled-rows /
reload validation error
- endpoint-level: create / update / delete / explicit-reload all
refresh the registry; an empty PUT skips the rebuild
Production fan-outs are frequently hitting the 6 KiB per-child cap by
just 1-2 KiB, forcing the coordinator into a follow-up inspect_workstream
round-trip per truncated child to recover the tail. Bumping the cap to
10 KiB absorbs the common overshoot without changing the truncation
semantics — truncated=True still fires for genuinely oversized messages,
and inspect_workstream remains the unbounded follow-up.
Worst-case context impact: a 32-child fan-out at the cap is now ~320 KiB
(was ~192 KiB), still well within commercial model context windows.
Typical fan-outs of 1-5 children land at 10-50 KiB.
LAST_ERROR_MAX_LEN (1 KiB) is unchanged — it's intentionally smaller
than the wait cap so error truncation happens at write time, and
1 KiB still sits well below 10 KiB.
WAIT_MESSAGE_MAX_BYTES is referenced by name (not literal 6144) in the
truncation test, so no test value needs updating.
The coordinator system message was descriptive about parallelism rather
than prescriptive — "while multiple children run in parallel" framed
fan-out as incidental, and "a tasks entry, a child to own it" primed
singular delegation. The spawn_batch example (benchmark A, benchmark B,
prototype the winner) showed dependent work under a fan-out framing,
teaching the wrong shape.
In practice the coordinator failed to decompose enumerable requests
("top stories on HN, Lobsters, /r/programming, …") without explicit
"please fan this out" instructions, on both GPT-5.5 and Claude Opus.
base_coordinator.md
- Replace singular "a tasks entry, a child to own it" with plural
"enumerate the independent units of work, spawn one child per unit,
run them in parallel by default. Sequential only when one child's
output feeds the next."
- Tighten the delegation paragraph.
tools_coordinator.md
- Drop the persona repetition that duplicated base_coordinator.md.
- Drop the prescriptive "## Workflow shape" section (the cost note is
already in wait_for_workstream's tool description; the edit-X
redirect is already in the persona).
- Drop "in one approval" / "single approval" mentions to avoid
surfacing approval mechanics to the model.
- Replace the misleading spawn_batch example with truly independent
items; drop "(up to 10)" which overstated the cap (it's per-call,
not global, and is documented in the tool schema).
- Add a course-correction example to send_to_workstream — the pattern
coordinators most often replace with cancel-and-respawn.
- Drop the read action from the tasks examples to keep the lifecycle
(add → update → remove) coherent.
Coord system message ~16% shorter (4440 → 3722 chars). Both GPT-5.5
and Claude Opus now naturally decompose the news-board prompt without
explicit fan-out instructions. 29 prompt-composition tests pass.
The server's --help epilog and compose.yaml both reference
--skip-permissions, but the argparser never defined it, so any
container started with SKIP_PERMISSIONS=1 exited with
"unrecognized arguments: --skip-permissions".
Wire the flag through to app.state.skip_permissions, OR-ing it
with the existing tools.skip_permissions config-store setting so
the stored value still works on its own.
2026-04-29 14:12:28 -07:00
458 changed files with 113728 additions and 22073 deletions
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone coordinator — parallel tool batches with judge-graded approval and child workstream tracking" width="960"/>
@@ -26,12 +27,13 @@ See [docs/releasing.md](docs/releasing.md) for the full release process.
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp, Ollama) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Bring your own models** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), the Anthropic Messages API, and Google Gemini, mixed freely per role
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
@@ -281,6 +281,7 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -325,6 +326,44 @@ finalize any in-progress assistant message.
{"type":"stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
*.json 19 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 16 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -190,7 +189,6 @@ Phase 3: EXECUTE (parallel)
(cancel_event also checked per line — kills process group on cancel)
Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
call_id links tool_info items → streaming chunks → final result
For plan tool: post-execution gate via ui.on_plan_review()
```
### State Transitions
@@ -209,7 +207,7 @@ The engine emits state changes via `_emit_state()` which calls
"running" ---> tool execution
|
v
"attention" ---> waiting for user approval / plan review
"attention" ---> waiting for user approval
|
v
"running" ---> executing approved tools
@@ -231,11 +229,13 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 15
methods. Every frontend must implement all of them.
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
@@ -115,7 +115,8 @@ with a `type` field. The recurring shapes a UI has to handle:
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Orchestration scratchpad keyed by the `coordinator` scope. |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
Explicitly **not** in the coordinator set:
@@ -72,8 +86,8 @@ Explicitly **not** in the coordinator set:
-`bash` / `edit_file` / `write_file` / `append_file` / `diff_file` — no local FS.
-`read_file` / `search` — no local FS reads.
-`web_fetch` / `web_search` — no direct web access.
- `task_agent` / `plan_agent` — sub-agent tools are zeroed on coord sessions.
- `memory` / `recall` / `notify` / `watch` / `read_resource` / `use_prompt` / `skill` — the orchestrator's "memory" is its children's outputs; these UX / persistence tools belong to interactive sessions.
-`task_agent` — sub-agent tool is zeroed on coord sessions.
-`recall` / `watch` / `read_resource` / `use_prompt` — UX / persistence tools that belong to interactive sessions. The dual-kind `memory` / `skills` / `notify` tools are available on both kinds (see the table above).
If your skill needs a coordinator to "run a command" or "read a
file", write the delegate pattern instead: spawn a child with an
@@ -169,14 +183,21 @@ validates ws_id against `parent_ws_id=coord_ws_id` AND
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"ws_id": "...", "name": "...",
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
"node_id": "...", "routing_strategy": "..."}`; the model should
extract the ws_id and pass it to `inspect_workstream` /
**Default** (no flag) — starts `server` and `console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
Open the dashboard at **https://localhost:8443**. It's served by Caddy with its
own local CA, so trust the root certificate once (or click through the browser
The dashboard is at **https://localhost:8443** (Caddy, same as the dev stack);
the console's HTTP port isn't published. For a real domain and a publicly
trusted cert, edit `turnstone/deploy/Caddyfile` to point Caddy at Let's Encrypt
(see [tls.md](tls.md)). Pin the image with `TURNSTONE_IMAGE_TAG` (default:
`latest`).
### mTLS
Layer the TLS overlay on the production stack to enable mutual TLS between
services. A bootstrap container creates a CA and every service auto-provisions
certs via the console's ACME endpoint:
```bash
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
See [tls.md](tls.md) for details.
## Configuration
All configuration is via environment variables in `.env` (copy from`.env.example`):
Everything is configured with environment variables in `.env` (copy from
[`.env.example`](../.env.example)). The dev stack needs none of them — they're
overrides.
### LLM Backend
### LLM backend
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | OpenAI-compatible API URL |
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | Bootstrap OpenAI-compatible API URL (real backends go in the UI) |
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TAVILY_API_KEY` | — | Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search) |
| `TURNSTONE_SEARXNG_URL` | `http://searxng:8080` | SearxNG URL for the `web_search` tool (local/vLLM models only; Anthropic/OpenAI use native search). Defaults to the bundled `searxng` service; set to an external instance's URL. To turn web search off, clear `tools.searxng_url` in the admin Settings tab. |
| `SEARXNG_IMAGE_TAG` | `latest` | Tag for the bundled `searxng/searxng` image |
| `MODEL` | — | Override the default model alias |
### Server
### Auth & database
| Variable | Default (dev / prod) | Description |
|----------|----------------------|-------------|
| `TURNSTONE_JWT_SECRET` | insecure default / **required** | JWT signing secret. Every service must share one value. |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **Upgrading from <1.3.0a4:** Earlier versions used `DB_BACKEND` and `DATABASE_URL` in `.env`, which `compose.yaml` mapped to the `TURNSTONE_`-prefixed names internally. These short aliases have been removed. Rename `DB_BACKEND` → `TURNSTONE_DB_BACKEND` and `DATABASE_URL` → `TURNSTONE_DB_URL` in your `.env` file.
> **Large clusters:** Each turnstone process maintains a small connection pool (5 max). At hundreds of nodes this adds up — use [PgBouncer](pgbouncer.md) in transaction pooling mode between turnstone and PostgreSQL.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
> You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass `--token --scopes read,write,approve` to also generate an initial API token.
| `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command registered in the Slack app |
The channel service runs in the `production` profile. When
`TURNSTONE_DISCORD_TOKEN` or the Slack pair is set the gateway starts the
corresponding adapter; both can run in one process. See
[Channel Integrations](channels.md) for platform app setup and user
account linking.
## Scaling
For multi-node testing, use the `cluster` profile which provides 10 server instances with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
The default `server` also runs alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
For production clusters beyond ~50 nodes, add PgBouncer between turnstone services and PostgreSQL. See [PgBouncer Connection Pooling](pgbouncer.md) for Docker Compose and Helm configuration.
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user`*(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata***or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static` → `oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
| `TURNSTONE_OIDC_ROLE_CLAIM` | No | — | ID token claim containing role/group values (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_ROLE_MAP` | No | — | Mapping from claim values to Turnstone role IDs (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | No | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Recommended when running behind a reverse proxy. When unset, derived from the request Host header. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
OIDC is enabled when all three required fields (issuer, client ID, client
secret) are non-empty. If any is missing, OIDC is silently disabled and
the login screen shows only the password form.
All four required fields — issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
is disabled at startup (an error is logged when only `redirect_base`
is missing) and the login screen shows only the password form.
### Reverse Proxy / Load Balancer
### Redirect base (required)
When Turnstone runs behind a reverse proxy, the internal `Host` header may
not match the externally-reachable URL. Set `TURNSTONE_OIDC_REDIRECT_BASE`
to the public origin so the redirect URI sent to the identity provider is
correct:
`TURNSTONE_OIDC_REDIRECT_BASE` pins the redirect URI sent to the identity
provider to a known externally-visible origin. Set it to the public origin
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml``[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
@@ -59,20 +59,32 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Plan / task agent overrides
### Reasoning persistence (per-model)
`plan_agent` and `task_agent` sub-sessions resolve independently from the
conversation model so operators can pick a cheaper/faster model for
autonomous loops:
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Task agent overrides
`task_agent` sub-sessions resolve independently from the conversation model
so operators can pick a cheaper/faster model for autonomous loops:
| Setting | Purpose |
|---------|---------|
| `model.plan_alias` | Alias used for `plan_agent` sub-sessions. Falls back to `[model].plan_model` in config.toml, then `[model].agent_model`, then the session's active model. |
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Same fallback chain as `plan_alias`. |
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Falls back to `[model].agent_model` in config.toml, then the session's active model. |
| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". |
All four are live-editable from the Settings tab and take effect on the
Both are live-editable from the Settings tab and take effect on the
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
---
# Importing Conversation History into Turnstone
## Overview
Source formats vary; the destination does not. Your job is to translate whatever the user hands you (JSON dump, ZIP export, scraped HTML, screenshot OCR, raw transcript) into Turnstone's internal shape: **one workstream row** plus an ordered sequence of **conversation rows** in OpenAI message format. This skill documents the destination so you can write a correct mapper for any source.
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
## Turnstone Data Model (the destination)
Two tables carry the conversation:
### `workstreams` (one row per imported thread)
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
| `created`, `updated` | yes | ISO8601 strings. Use the source's first/last message timestamps when available. |
### `conversations` (many rows per thread, ordered by `id`/`timestamp`)
| Column | Notes |
|---|---|
| `ws_id` | The workstream this row belongs to. |
| `timestamp` | ISO8601 string. Preserve source timestamps; fall back to monotonically increasing values if unknown. **Order is canonical via `id` (autoincrement), not `timestamp`** — but always insert in conversational order so both agree. |
| `role` | One of `system`, `user`, `assistant`, `tool`, `developer`. See role mapping below. |
| `content` | Text. May be NULL for assistant rows that are *only* tool calls. |
| `tool_name` | Set on `role="tool"` rows (the tool whose result this is). NULL otherwise. |
| `tool_call_id` | Set on `role="tool"` rows (matches the assistant row's `tool_calls[].id`). NULL otherwise. |
| `tool_calls` | JSON-encoded list, on `role="assistant"` rows that issued tool calls. OpenAI shape — see "Tool Calls" below. |
| `provider_data` | JSON blob preserving provider-native content blocks (Anthropic `signature`, Gemini `thought_signature`, etc.). Optional; only matters for **resumable** imports against the same provider. Skip for archives. |
The internal format is **OpenAI-shaped**, even when the source was Anthropic or Gemini. Providers translate at their own API boundary; storage stays uniform.
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
Only useful for *Turnstone → Turnstone* re-parenting. Not relevant for foreign sources.
### 3. SDK `create_workstream(initial_message=...)` + `send()` per turn (last resort)
Only fits archives where the source had **no tool calls** and you don't care about preserving assistant turns verbatim. Each `send()` triggers a real LLM round-trip, which is expensive and rewrites assistant content. Don't use this for full history.
## Role Mapping
Common source-role conventions and how they map to Turnstone:
| `system` | `system` | Preserve only if it's content the user wrote (custom instructions). Drop boilerplate provider preambles — Turnstone composes its own system message. |
| `tool`, `function`, `tool_result` | `tool` | Must carry `tool_name` and `tool_call_id` matching the prior assistant row's `tool_calls[].id`. |
| `tool_use` (Anthropic) | `assistant` with `tool_calls` | Anthropic emits tool calls *inside* an assistant message; flatten to OpenAI shape. |
| `human_feedback`, `revision` | `user` | Treat as a follow-up user turn. |
## Tool Calls (the most error-prone part)
Turnstone stores tool calls in OpenAI's nested-function shape on the assistant row, and matches them with `role="tool"` result rows by `tool_call_id`.
### Assistant row with tool calls
```json
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "search_web",
"arguments": "{\"query\":\"turnstone import\"}"
}
}
]
}
```
`tool_calls[].function.arguments` is **a JSON-encoded string**, not an object. Source formats commonly get this wrong — Anthropic stores arguments as a parsed object, Gemini as a struct. Always re-serialize to a string.
### Tool result row
```json
{
"role": "tool",
"tool_name": "search_web",
"tool_call_id": "call_abc123",
"content": "..."
}
```
Pairing rules:
- Every assistant `tool_calls[].id` MUST be followed by exactly one `role="tool"` row with the matching `tool_call_id`, before the next user/assistant turn.
- If the source dropped the tool result (cut-off transcript), insert a synthetic `role="tool"` row with `content="[tool result missing in source]"` to keep the chain valid. An assistant row with an unanswered `tool_calls[].id` will break replay and any LLM round-trip.
- Multi-tool assistant turns: one `role="tool"` row per call, in any order, all before the next non-tool row.
### Tool ID generation
If the source used opaque tool IDs that aren't unique within a thread (some platforms reuse them), regenerate with a stable scheme like `f"call_{i}"` where `i` is a per-thread counter. Update both the assistant and tool rows together.
## Provider Fidelity (`provider_data`)
Skip this entirely for **archive** imports.
For **resumable** imports against the same provider, populate `provider_data` to preserve provider-specific tool-call metadata that the next API round-trip will require:
- **Anthropic**: `signature` field on thinking blocks; required for round-tripping extended-thinking responses.
- **Gemini**: `thought_signature` on tool calls; required for fidelity.
- **OpenAI**: typically nothing to preserve.
The runtime-side dict key is `_provider_content` (a list of provider-native blocks); the persisted column is `provider_data` (the same list, JSON-encoded). If you don't have provider-native blocks from the source — and you usually won't, because a foreign export won't include them — leave `provider_data` NULL. The first new turn will succeed without it, but the previous assistant turn's reasoning won't replay back to the model.
## Attachments
If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
Two import paths:
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
- **Don't import the source provider's system prompt verbatim.** Provider boilerplate ("You are Claude...", "You are ChatGPT...") will conflict with Turnstone's composed system message and confuse the model on resume. Drop it; preserve only user-authored custom instructions.
- **Don't preserve foreign tool definitions as Turnstone tools.** If the source had custom tools that don't exist in Turnstone, the assistant rows that called them are still valid history (archive), but the workstream is **not resumable** — mark `state="closed"`.
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
turnstone exposes 19 built-in tools plus any number of external MCP tools to the
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -22,7 +22,6 @@ schema plus turnstone-specific metadata keys:
"properties": { ... },
"required": ["param1"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "param1"
@@ -33,8 +32,7 @@ schema plus turnstone-specific metadata keys:
| Key | Type | Meaning |
|----------------|------|---------|
| `agent` | bool | Tool is available to plan/task sub-agents (read-only subset). |
| `task_agent` | bool | Tool is available to task sub-agents (broader subset). |
| `task_agent` | bool | Tool is available to task sub-agents. |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
@@ -46,12 +44,10 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 19 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 19 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `TASK_AUTO_TOOLS`| Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -130,16 +122,14 @@ Each item's `execute` callable is invoked:
- `bash` -- arbitrary command execution
- `write_file` -- creates or overwrites files
- `edit_file` -- modifies file content
- `math` -- sandboxed computation (confirmation required despite being sandboxed)
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via Tavily API (makes network requests)
- `task` -- spawns an autonomous sub-agent
- `plan` -- spawns a planning sub-agent, plus post-execution review gate
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
- `task_agent` -- spawns an autonomous sub-agent
Note: The JSON schema metadata key `auto_approve` controls membership in
`AGENT_AUTO_TOOLS`/`TASK_AUTO_TOOLS` (used for agent sub-sessions). The actual
runtime approval behavior is determined by the `needs_approval` field set in
each `_prepare_*` method on `ChatSession`. These two mechanisms can differ.
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
approval behavior is determined by the `needs_approval` field set in each
`_prepare_*` method on `ChatSession`. These two mechanisms can differ.
---
@@ -165,12 +155,9 @@ Every tool defines a `primary_key`. The mapping is:
| `write_file` | `content` |
| `edit_file` | `old_string`|
| `search` | `query` |
| `math` | `code` |
| `man` | `page` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `task_agent` | `prompt` |
| `plan_agent` | `goal` |
| `memory` | `name` |
| `recall` | `query` |
| `notify` | `message` |
@@ -194,7 +181,7 @@ Execute a bash command and return stdout + stderr.
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`). Environment variables containing secrets are scrubbed (`*_KEY`, `*_SECRET`, `*_TOKEN`, etc.).
- **Output format**: Stdout is returned directly. Stderr lines are prefixed with `[stderr]` so the model can distinguish them. When the command itself redirects stderr to stdout (`2>&1`), no prefix is added. Output exceeding 256KB is truncated (head + tail preserved, middle replaced with a truncation notice).
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only (not available to plan sub-agents).
- **Agent availability**: `task_agent` only.
---
@@ -212,7 +199,7 @@ base64-encoded image data for supported image formats.
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -268,7 +255,7 @@ Show a unified diff between two files, or between a file and a provided string.
- **What it does**: Returns unified diff output using Python's `difflib`. Binary files (containing null bytes) are rejected with a clear error. Files read through `diff_file` satisfy `edit_file`'s read guard — you can diff then edit without a separate `read_file` call. Large diffs are streamed with early cutoff at the tool truncation limit.
- **Auto-approve**: Yes (read-only).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -283,44 +270,12 @@ Search file contents for a regex pattern.
- **What it does**: Recursively searches for the pattern using `grep -rn`. Returns matching lines with file paths and line numbers.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
## Computation
### math
Execute Python code for math and computation in a sandbox.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `code` | string | yes | Python code to execute. Must use `print()` for output. |
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported. `pytest` is also available for import.
- **Installation**: `sympy`, `numpy`, `scipy`, and `pytest` require the `[sandbox]` extras group: `pip install turnstone[sandbox]` (included in `[all]`).
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
## Information
### man
Read a man page.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `page` | string | yes | The man page name (e.g. `grep`, `socket`, `printf`). |
- **What it does**: Returns the full formatted manual entry. Preferred over `bash('man ...')` or `web_search` for command/API documentation.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
### web_fetch
Fetch a URL and extract specific information from it.
@@ -332,7 +287,7 @@ Fetch a URL and extract specific information from it.
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
@@ -344,20 +299,55 @@ Search the web using a text query.
| `max_results` | integer | no | Max results to return (default 5, max 20). |
| `topic` | string | no | Search topic: `general`, `news`, or `finance` (default `general`). |
| `category` | string | no | Search category: `general` (default), `news`, `it` (code/tech), or `science`. Maps to SearxNG categories; the model picks per query. |
- **What it does**: Searches the web and returns ranked results with titles, URLs, and content snippets. Uses provider-native search when available:
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No Tavily key needed.
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No backend needed.
- **OpenAI search models** (`gpt-5-search-api`): Replaced with `web_search_options` parameter. The model always searches and returns `url_citation` annotations.
- **Local/vLLM models**: Falls back to the Tavily API. Requires `tavily_key` in `config.toml` or `$TAVILY_API_KEY`.
- **Local/vLLM models**: Falls back to a self-hosted [SearxNG](https://searxng.org) instance. Set `searxng_url` in `config.toml``[tools]` or `$TURNSTONE_SEARXNG_URL` (the docker-compose stack bundles a `searxng` service and points at it by default). Operators with a custom MCP search server can instead set `web_search_backend = "mcp:server:tool"`.
- **Auto-approve**: Yes (auto-approved for all tool dispatch paths).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
---
### Reranking (optional)
`web_search` can use an external **reranker** to re-order the backend's result pool by relevance to the query before returning the top hits. Turnstone runs no reranker model itself; it POSTs to a Cohere/Jina-compatible `/rerank` endpoint (self-hosted [vLLM](https://docs.vllm.ai) / [TEI](https://github.com/huggingface/text-embeddings-inference) / llama.cpp, or hosted Cohere/Jina/Voyage).
**Disabled by default.** In the console **Models** tab, add a model definition whose `base_url` is a Cohere/Jina-compatible `/rerank` endpoint and whose capabilities include `{"supports_rerank": true}`, then select it under **Models → Roles → Reranker**. It's managed like every other model (write-only key, enable/disable, calibration). The reranker is purely this per-model definition — there is no global `rerank_url`-style endpoint setting.
The `rerank_web_search` toggle defaults on once a reranker is selected. If the endpoint is unreachable or errors, web_search falls back silently to the backend's native result order — reranking never makes a search fail.
When `rerank_bm25` is enabled, the candidate text for memory, tool, and skill retrieval (memory name/description/content and tool/skill names + descriptions) is also sent to the rerank endpoint — a self-hosted endpoint (vLLM/TEI/llama.cpp) keeps it on your infrastructure, a hosted provider (Cohere/Jina/Voyage) sends it off-box.
**Serving a Qwen3-Reranker with vLLM.** The model is instruction-aware, so vLLM **must** apply its chat template — pass `--chat-template` explicitly. Without it the bare query produces near-random scores and reranking actively *hurts* retrieval (verified: an irrelevant passage outscored the correct one):
Then add a reranker model in the **Models** tab with `base_url``http://vllm:8000/rerank` (model name `qwen3-reranker`) and select it under **Models → Roles → Reranker**.
For an endpoint that does *not* apply the model's template, set `rerank_instruction` instead — Turnstone then wraps each query as `<Instruct>: {instruction}` / `<Query>: {query}` (Qwen3's own default is `Given a web search query, retrieve relevant passages that answer the query`). Use the chat template **or** the instruction, not both (they double-wrap).
**Picking `rerank_bm25_threshold`.** The relevance floor that gates proactive memory injection is a probability in `[0, 1]`, but the right value differs per model (a sharp 0.6B reranker may want ~0.95; a broader 4B ~0.33). Calibrate it against your endpoint:
```bash
turnstone-admin rerank-calibrate # probe the endpoint, recommend a floor
It reports the score scale, whether the endpoint cleanly separates relevant from irrelevant probes (a **"no clean separation"** result flags a mis-served or weak reranker), and the suggested floor. Leave the threshold at `0` to rerank-without-filtering.
---
## Agent
Tool names use the `_agent` suffix — bare `plan` / `task` collide with
The tool name uses the `_agent` suffix — bare `task` collides with
chat-template channel names on some local models.
### task_agent
@@ -368,23 +358,9 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, math, man, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### plan_agent
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<ws_id>.md` (unique per workstream, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Top-level only.
---
@@ -445,7 +421,7 @@ Provide either `username` for user-based targeting or `channel_type` +
- **What it does**: Sends a notification via the channel gateway's HTTP endpoint (`POST /v1/api/notify`). The server queries the `services` table for healthy channel gateways, authenticates with a service JWT (`aud: turnstone-channel`), and delivers to the first healthy gateway. On failure, retries up to 2 additional times with backoff (1s, 3s). Rate-limited to 5 notifications per turn (counter only increments on success).
- **Auto-approve**: Yes — notifications are time-sensitive and auto-approved so the model can alert users urgently.
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
> See [Channel Integrations: Notifications](channels.md#notifications)
> for the full delivery flow, service registry details, and security
@@ -521,7 +497,7 @@ data.get("mergedAt") is not None
- Duplicate names rejected within the same workstream.
- **Auto-approve**: `create` requires approval; `list` and `cancel` are auto-approved.
- **Agent availability**: Main session only — not available to plan/task sub-agents.
- **Agent availability**: Main session only — not available to task sub-agents.
> See [Watch Architecture](diagrams/png/18-watch-architecture.png) for the
> full poll → evaluate → dispatch flow.
@@ -554,33 +530,30 @@ pre-configure skills at workstream creation.
- **Task sub-agents** — via `self._task_tools` (merged list)
- **Plan sub-agents** — via `self._agent_tools` (merged list)
### Naming convention
@@ -758,34 +731,25 @@ MCP tools (3):
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through three mechanisms:
MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection.
manual refresh attempts reconnection. The console admin panel exposes the
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`,`_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
`_task_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
@@ -839,7 +803,7 @@ Use read_resource(uri='...') to access the resources listed above.
- **What it does**: Reads the resource from its MCP server via `MCPClientManager.read_resource_sync()`. Returns text content for text resources or base64-encoded data for binary resources. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (reads external data).
- **Agent availability**: `agent` and `task_agent`.
- **Agent availability**: `task_agent`.
### Capability guards
@@ -881,7 +845,7 @@ the `initialize` handshake. Each prompt is stored with its prefixed name
- **What it does**: Invokes an MCP prompt by name via `MCPClientManager.get_prompt_sync()`, expanding it into messages. Returns the expanded prompt content formatted as `[role]: content` blocks joined with blank lines. The prompt catalog is listed in the system message so the model knows which prompts are available. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (invokes external prompt servers).
- **Agent availability**: `agent` and `task_agent`.
# Resolve how to invoke docker (direct / via sudo) and make sure the daemon runs.
ensure_docker_usable(){
if ! have docker;then
warn "Docker is not installed."
ask "Install Docker now?" y || die "Docker is required."
install_docker
have docker || die "Docker installation failed."
fi
docker info >/dev/null 2>&1&&{DOCKER="docker";return;}
# Maybe the daemon just isn't running yet.
start_docker_daemon
docker info >/dev/null 2>&1&&{DOCKER="docker";return;}
# Maybe we lack permission (not in the docker group yet, group change not
# active in this shell) — fall back to sudo for this run.
if["$(id -u)" -ne 0]&& have sudo && sudo docker info >/dev/null 2>&1;then
DOCKER="sudo docker"
warn "Using 'sudo docker' for this run. To drop the sudo: 'sudo usermod -aG docker $USER' then log out and back in."
return
fi
if["$IS_WSL" -eq 1];then
die "Docker isn't usable inside WSL. Install Docker Desktop on Windows and enable WSL integration for this distro (Settings -> Resources -> WSL integration), then re-run."
fi
die "Docker is installed but not usable — the daemon may be stopped or you lack permission. Try: 'sudo systemctl start docker', then re-run."
}
ensure_compose(){
$DOCKER compose version >/dev/null 2>&1&&return
warn "The Docker Compose v2 plugin is missing."
case"$PKG" in
apt|dnf|yum) ask "Install docker-compose-plugin?" y && pkg_install docker-compose-plugin ||true;;
pacman) ask "Install docker-compose?" y && pkg_install docker-compose ||true;;
esac
$DOCKER compose version >/dev/null 2>&1\
|| die "'docker compose' is unavailable. Install Docker Compose v2 and re-run."
$DOCKER info 2>/dev/null | grep -qi 'rootless'&&ROOTLESS=1
pick_node_count
info "Building the image (first run pulls dependencies — this can take a few minutes)…"
if ! (cd"$INSTALL_DIR"&&$DOCKER compose build );then
die "image build failed (see output above). Common causes: low memory or disk, or the Docker daemon stopped. Free up resources and re-run — it resumes."
fi
prepare_env
info "Ports — dashboard (Caddy): ${CADDY_PORT}, PostgreSQL: 127.0.0.1:${PG_PORT}"
write_node_override
info "Starting the stack…"
# Plain `up -d` (honoring compose.override.yaml) + --remove-orphans so a
# re-run that lowers the count also stops the now-excluded nodes.
if ! (cd"$INSTALL_DIR"&&$DOCKER compose up -d --remove-orphans );then
die "the stack failed to start (see output above). Inspect logs: cd $INSTALL_DIR && $DOCKER compose logs"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Patrick Buckley
