mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-13 07:22:24 -06:00
Compare commits
72 Commits
v1.5.10
...
stable/1.5
| Author | SHA1 | Date | |
|---|---|---|---|
| 4a38b835f5 | |||
| 415be00149 | |||
| 346ad2a6aa | |||
| 8003fcbebe | |||
| d4f3711d63 | |||
| cea229206b | |||
| b7b4dcc0df | |||
| 97080e1df9 | |||
| af0cbaaec3 | |||
| b9ce0d388e | |||
| d07d2242aa | |||
| 70514cc406 | |||
| 687a3367c0 | |||
| bfd99c6a81 | |||
| 1a813c8130 | |||
| d6aa85db6d | |||
| c0c7fda7f9 | |||
| 7fae2698d7 | |||
| 4bbe64755e | |||
| c1281b9721 | |||
| 3124dbe52f | |||
| a8a1e738ca | |||
| 27768abecc | |||
| 1e5017293a | |||
| b2c688f3b1 | |||
| 09b1f07e18 | |||
| 1de8f6b4c7 | |||
| 4f89c1c3f3 | |||
| 6898860f18 | |||
| 3d07cad272 | |||
| 3737ddf89f | |||
| 464450b9e2 | |||
| 1fba80a9b5 | |||
| 28a5914be0 | |||
| bb2515eddf | |||
| 19978bfd89 | |||
| 8eec44d809 | |||
| f1cf516eb6 | |||
| 1df1e739ef | |||
| fbbb21012a | |||
| 617de2488f | |||
| 1d5189bb8d | |||
| 89a282f86b | |||
| 59c943b83a | |||
| 14b7516b3f | |||
| 5d0ec99449 | |||
| 6dfd1b5c18 | |||
| 658c65aee8 | |||
| 641ce8e7f6 | |||
| d76c57e687 | |||
| 58bf811a0e | |||
| df942e375e | |||
| 84fd5dc859 | |||
| ba8b1d9126 | |||
| 685b1e3d9b | |||
| 91300060fa | |||
| 688f047ce1 | |||
| 87893aa4ab | |||
| 0988142303 | |||
| 40ecebf012 | |||
| c91869c7e5 | |||
| 53b52092f9 | |||
| 0d1a009a4c | |||
| e8352bd8e5 | |||
| de4cc568c4 | |||
| b477c85ddc | |||
| 00bd80a658 | |||
| 47df9d23c5 | |||
| 16fc7efce2 | |||
| e8eca2ec9b | |||
| 57563b0c12 | |||
| 43e622840b |
+649
-3
@@ -8,13 +8,659 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
|
||||
|
||||
Three release tracks are maintained:
|
||||
|
||||
- **`stable/1.0`** — patch-only (`v1.0.x`)
|
||||
- **`stable/1.3`** — patch-only (`v1.3.x`)
|
||||
- **`stable/1.4`** — patch-only (`v1.4.x`)
|
||||
- **`main`** — experimental (`v1.5.0aN`)
|
||||
- **`stable/1.5`** — patch-only (`v1.5.x`)
|
||||
- **`main`** — experimental (next major)
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.5.18]
|
||||
|
||||
Backports the `turnstone-admin` config-loading alignment from `main`
|
||||
plus the accompanying `load_config` permission-warning hardening. No
|
||||
schema changes.
|
||||
|
||||
### Added
|
||||
|
||||
- **`turnstone-admin` reads `config.toml`** — the admin CLI now honors
|
||||
the same `[database]` section that `turnstone-server` does, with the
|
||||
same precedence (`CLI / config.toml > TURNSTONE_DB_* env > defaults`).
|
||||
Operators with DB credentials in `config.toml` no longer need to
|
||||
re-export `TURNSTONE_DB_URL` before every admin invocation. Newly
|
||||
plumbed through to `init_storage`: `pool_size`, `sslmode`,
|
||||
`sslrootcert`, `sslcert`, `sslkey` — previously the admin CLI
|
||||
silently dropped these. A new `--config PATH` flag mirrors the
|
||||
one already on `turnstone-server`.
|
||||
|
||||
### Security
|
||||
|
||||
- **Permissive `config.toml` now warns** — `turnstone.core.config.load_config`
|
||||
logs a single warning when the resolved config file is group- or
|
||||
world-readable (any bit in `0o077`). DB password and TLS key paths
|
||||
live in `[database]`; operators usually want the file at `0600`.
|
||||
|
||||
## [1.5.17]
|
||||
|
||||
Backports a clutch of coordinator-tool clarity fixes plus a watch-delivery
|
||||
correctness fix from `main` to the `stable/1.5` track, plus a previously-
|
||||
latent intent-verdicts persistence bug exposed by the new heuristic-verdict
|
||||
INSERT paths. No schema changes.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`intent_verdicts` PK collisions on every llm_fallback delivery** —
|
||||
async LLM-tier "llm_fallback" verdicts (`turnstone/core/judge.py` —
|
||||
`_deliver_fallbacks` and the in-loop fallback path) deliberately
|
||||
reuse the heuristic verdict's `verdict_id` so the row gets
|
||||
"upgraded in place" from `tier="heuristic"` → `tier="llm_fallback"`
|
||||
when the LLM judge times out, is cancelled, or returns no content.
|
||||
The consumer `_persist_intent_verdict` was doing a plain INSERT,
|
||||
hitting the `intent_verdicts_pkey` constraint on every fallback
|
||||
delivery; Postgres logged the duplicate-key error, the application
|
||||
try/except swallowed it at `log.debug`, and the row never actually
|
||||
got upgraded — the LLM judge's annotation
|
||||
(`"(LLM judge did not return a verdict)"`) was lost. The collision
|
||||
rate exploded on this release because the new heuristic-INSERT
|
||||
paths in the auto-approve early-return branches of `approve_tools`
|
||||
(introduced below) leave no gap for the fallback to land cleanly
|
||||
into. Fix: new `upsert_intent_verdict` storage method using
|
||||
`ON CONFLICT (verdict_id) DO UPDATE` that updates only `tier`,
|
||||
`reasoning`, `judge_model` — the three fields that genuinely
|
||||
change between heuristic and llm_fallback. Every other column
|
||||
(identity, carried-verbatim, and `user_decision`) is excluded;
|
||||
`user_decision` in particular would otherwise be clobbered back
|
||||
to `"pending"` when a fallback arrives after the operator has
|
||||
already resolved the approval. The bulk-INSERT path stays as
|
||||
plain INSERT — fresh UUIDs in `judge.evaluate` make in-turn dups
|
||||
impossible; the inverse race (fallback wins before bulk lands) is
|
||||
reachable but unchanged in observable behavior by this fix,
|
||||
documented at the bulk site for a future hardening pass.
|
||||
- **Coordinator LLM re-spawn loops on large fan-outs** — the spawn-tool
|
||||
return JSON used `ws_id` as its key, which primed the model's recency
|
||||
bias to feed the spawn result straight back into another
|
||||
`spawn_workstream(ws_id=...)` call instead of progressing to
|
||||
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this cascaded
|
||||
into self-inflicted re-spawn loops. The LLM-facing tool result now emits
|
||||
`child_ws_id` (the storage column / HTTP API contract is unchanged); the
|
||||
field name is already an existing project term so the rename aligns
|
||||
rather than introduces new vocabulary. Also handles the silent
|
||||
upstream-omits-ws_id success-shape edge that previously emitted
|
||||
`{"child_ws_id": null}` to the LLM — now surfaces a tool error so the
|
||||
model retries rather than chasing a null id.
|
||||
- **`inspect_workstream` blowing the coordinator context budget** — a
|
||||
coord doing a fan-out wave against tool-heavy children could land
|
||||
>100 KB of raw output per inspect call, and the previous safety net
|
||||
(`_truncate_output`'s head+tail strategy) silently dropped *middle*
|
||||
messages — exactly the wrong shape for understanding a child's
|
||||
trajectory (the FIRST sets the brief, the LAST shows the conclusion,
|
||||
the middle is the connective tissue). Output now goes through a
|
||||
three-tier degradation ladder mirroring the search tool's
|
||||
`_format_search_results`: `_tier="full"` (every message verbatim) →
|
||||
`_tier="compact"` (per-message head/tail-snipped content + snipped
|
||||
`tool_calls.arguments`, falling through a `(20,30)` / `(10,20)` /
|
||||
`(5,10)` message-list trim ladder) → `_tier="skeleton"` (counts, role
|
||||
distribution, last-assistant preview). Budget 32 KiB matches the
|
||||
search tool's; the chosen tier is annotated on the response so the
|
||||
model can recall with a tighter `message_limit` if signal was lost.
|
||||
- **Auto-approved verdicts indistinguishable from pending review** —
|
||||
`intent_verdict` rows for auto-approved tool calls landed with
|
||||
`user_decision=""`, which read identically to "still waiting for the
|
||||
operator" in the audit trail and led to a real misdiagnosis incident.
|
||||
The column now carries an explicit vocabulary at insert: `pending` /
|
||||
`approved` / `denied` / `timeout` / `policy` / `blanket` / `skill` /
|
||||
`always` / `auto_approve_tools`. The auto-approve early-return
|
||||
branches in `approve_tools` now persist heuristic verdicts stamped
|
||||
with their reason (previously dropped on the floor), and late LLM-tier
|
||||
verdicts that arrive for an already-auto-approved call_id are stamped
|
||||
via a TTL-pruned lookup map — so the audit row carries the
|
||||
auto-approve reason even when the LLM judge daemon completes after
|
||||
the synchronous approval cycle finished. `resolve_approval` gains a
|
||||
`timeout` kwarg writing `"timeout"` (the previous shape collapsed
|
||||
passive timeouts and active denials into the same column).
|
||||
- **`list_skills` empty `allowed_tools` misread as "no tool access"** —
|
||||
the response previously emitted `"allowed_tools": []` for every skill
|
||||
that hadn't declared an auto-approve allowlist, which a coordinator
|
||||
model read as "this skill can't use any tools" (real misdiagnosis: a
|
||||
code-review child appeared to have been spawned with zero tool
|
||||
access). The field is now omitted entirely when empty — absence
|
||||
carries the unambiguous meaning "no tool is pre-approved for this
|
||||
skill", presence (non-empty list) keeps the standard Claude Code
|
||||
skill-spec shape. The tool description rewrite makes the
|
||||
auto-approve-allowlist semantics explicit so a future reader doesn't
|
||||
re-derive the gating misread.
|
||||
- **Watch terminal-fires silently dropped on backpressure** —
|
||||
delivery now routes terminal events through the same path as
|
||||
normal fires instead of being filtered out when the consumer was
|
||||
saturated.
|
||||
|
||||
### Documentation
|
||||
|
||||
- **Storage `LIKE_ESCAPE` contract** — clarify that callers passing
|
||||
`.like(escape=...)` must use the same escape character that the
|
||||
storage helper assumes; previous wording let a reader pass a
|
||||
different escape and silently produce no matches.
|
||||
|
||||
## [1.5.15]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Admin console blank-page on MCP server rows with consented users** — a
|
||||
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
|
||||
delimiters on the bulk-revoke button HTML literal, but the literal embeds
|
||||
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
|
||||
into bare tokens, and the resulting `SyntaxError` wiped out every global
|
||||
in `admin.js` — `showAdmin` and all other admin entry points became
|
||||
undefined, so the console UI was non-functional whenever the rendered MCP
|
||||
server list contained at least one row with `consented_users_count > 0`.
|
||||
Switch the literal to single-quote delimiters to match the surrounding
|
||||
block.
|
||||
|
||||
## [1.5.14]
|
||||
|
||||
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
|
||||
|
||||
### Added
|
||||
|
||||
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
|
||||
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
|
||||
dispatchers now upsert into a new `mcp_pending_consent` table on
|
||||
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
|
||||
(scheduled / channel) that hits an unconsented server surfaces the deferred
|
||||
prompt to the user on their next dashboard load via the gear-icon badge —
|
||||
rows are cleared automatically by the OAuth callback handler on consent
|
||||
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
|
||||
admin row gains a `consented_users_count` pill and a two-step-confirm
|
||||
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
|
||||
revoke is intentionally not attempted in bulk to avoid N synchronous
|
||||
round-trips against the provider). Operator-facing docs land at
|
||||
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
|
||||
|
||||
Introduces forward-only migrations `054_mcp_pending_consent` and
|
||||
`055_mcp_user_tokens_server_index`.
|
||||
|
||||
## [1.5.13]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
|
||||
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
|
||||
the dispatcher uses in-process fan-out).
|
||||
|
||||
### Added
|
||||
|
||||
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
|
||||
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
|
||||
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
|
||||
per-channel handlers on a separate dispatch thread. The cluster collector
|
||||
subscribes to a new `services` channel and reacts to node register /
|
||||
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
|
||||
loop; the 60 s loop is retained as the backstop for crash-shaped loss
|
||||
(NOTIFY only fires on real writes). The storage layer also gains a uniform
|
||||
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
|
||||
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
|
||||
`[database] listen_url` in `config.toml`) points the dispatcher at a
|
||||
direct-to-Postgres URL; defaults to the main DB URL when unset.
|
||||
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
|
||||
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
|
||||
whenever a child state change is dispatched to the UI, and the wait loop
|
||||
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
|
||||
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
|
||||
storage ~2400 times now wakes only on real state transitions, with ~4× lower
|
||||
SSE traffic in the quiescent case.
|
||||
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
|
||||
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
|
||||
route previously emitted only `memory.delete`, so tool-initiated mutations
|
||||
had no audit footprint). All emissions are best-effort and never break the
|
||||
tool call itself.
|
||||
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
|
||||
an optional `skill=<name>` argument that loads the named skill's content as
|
||||
the sub-agent's persona in place of the hardcoded identity statement. The
|
||||
fixed operating-guidance block (one-shot, tool-use over narration,
|
||||
no follow-up questions) is still layered on top of every persona. High- and
|
||||
critical-risk skills surface their risk tier in the approval header and
|
||||
emit a `task_agent.high_risk_skill` warning, matching the existing
|
||||
session-load gate.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
|
||||
back-compat `default` alias auto-synthesised by `load_model_registry`
|
||||
remained visible to the model even when an operator had configured
|
||||
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
|
||||
routed to whichever backend the synthesised alias was attached to at boot
|
||||
instead of the configured per-role default. The synthesised alias is now
|
||||
only added when neither the DB nor `[models.*]` populates the registry,
|
||||
filtered out of the LLM-visible alias list, and explicitly rejected at the
|
||||
validator chokepoint as defense-in-depth.
|
||||
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
|
||||
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
|
||||
node labels were re-entering the shape parser and producing
|
||||
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
|
||||
affected label forms (`|content|` and `ID[content]`) before the SVG cache
|
||||
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
|
||||
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
|
||||
highlights code blocks progressively as they stream rather than only after
|
||||
completion.
|
||||
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
|
||||
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
|
||||
POST went to `/v1/api/auth/login` and was rewritten to
|
||||
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
|
||||
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
|
||||
public path, and `proxy_api` would have forwarded the login request to the
|
||||
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
|
||||
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
|
||||
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
|
||||
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
|
||||
handlers. The dispatch table is a single `(method, path) → handler` mapping
|
||||
so the test parametrize list can't drift from the implementation.
|
||||
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
|
||||
overlay was covering the entire appbar, hiding the proxy-injected node
|
||||
picker. The overlay now starts at `top: 48px` and the dashboard's role
|
||||
downgrades from `dialog+aria-modal` to `region` so the appbar above it
|
||||
remains reachable. The gear icon converts from a direct settings-panel
|
||||
click into a dropdown with "MCP connections" and "Logout" (the latter with
|
||||
`.destructive` styling). The settings-menu keydown handler is now attached
|
||||
synchronously so `Escape` can't fall through the brief window between the
|
||||
menu opening and its listeners being installed.
|
||||
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
|
||||
`services_notify` trigger lives only in the alembic chain, but the test
|
||||
fixture creates tables via `metadata.create_all`. The trigger function +
|
||||
trigger are now declared in `_schema.py` and attached via
|
||||
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
|
||||
PostgreSQL dialect, with the same SQL constants imported by migration 053
|
||||
so there's a single source of truth.
|
||||
|
||||
## [1.5.12]
|
||||
|
||||
### Added
|
||||
|
||||
- **Enriched backend error messages** — provider name and attempted URL are now
|
||||
included in session error responses, so operators can triage connectivity
|
||||
failures without enabling debug logging.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
|
||||
had no messages remaining after a rewind the history event was skipped,
|
||||
leaving connected UIs with stale content and blocking edit-and-resend flows.
|
||||
|
||||
## [1.5.11]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`052_model_reasoning_persistence` — `surface_persisted_reasoning` and
|
||||
`replay_reasoning_to_model` flag columns on `model_definitions`.
|
||||
|
||||
### Added
|
||||
|
||||
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
|
||||
restore) now receive an `in_progress_snapshot` event carrying the buffered
|
||||
partial response, so the UI can resume rendering the in-flight turn without
|
||||
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
|
||||
counter so a reconnecting client can skip events it already saw.
|
||||
- **Reasoning persistence** (Phases 1–4) — model reasoning text can now be
|
||||
persisted to conversation history and optionally replayed to the model on
|
||||
subsequent turns. Phase 1 persists reasoning text on the history payload.
|
||||
Phase 2 wires a build-time shape filter and a per-model
|
||||
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
|
||||
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
|
||||
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
|
||||
new per-model capability flags (`surface_persisted_reasoning`,
|
||||
`replay_reasoning_to_model`) both default `False` on unknown and
|
||||
local-server models.
|
||||
- **Console home composer: placeholders + toggle** — the console landing-page
|
||||
composer now shows context-aware placeholder text and a toggle component for
|
||||
advanced options; an admin polish pass tightened spacing and focus behaviour
|
||||
across the form.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`judge.model` now requires a named alias** — raw provider model IDs on
|
||||
`judge.model` in config are no longer accepted; the judge must reference an
|
||||
alias registered in the model registry. The session-provider raw-model
|
||||
fallback is removed. Existing configs using an unregistered model ID need a
|
||||
corresponding alias entry.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
|
||||
flag for a model that does not declare reasoning-replay support now silently
|
||||
no-ops instead of forwarding reasoning blocks and triggering a provider error.
|
||||
- **Coordinator alias resolution unified across placeholder + factory** — a
|
||||
placeholder coordinator and the real coordinator factory could previously
|
||||
resolve to different model aliases, producing a visible mismatch in the model
|
||||
display. Both paths now share the same resolution logic.
|
||||
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
|
||||
under-initialised coordinator state no longer 500s when the models endpoint
|
||||
is hit before the coordinator subsystem is fully bootstrapped.
|
||||
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
|
||||
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
|
||||
counter that broke `state_change` / `in_progress_snapshot` ordering on
|
||||
reconnect.
|
||||
- **Reasoning persistence shape + replay fixes** — per-block
|
||||
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
|
||||
synthesised alongside non-reasoning `provider_blocks` so both appear
|
||||
together in the history payload.
|
||||
|
||||
## [1.5.10]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`051_skill_notify_on_complete_array_default` — backfills
|
||||
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Skills unlock action** — operators can unlock an installed skill to allow
|
||||
local customisation. Once unlocked, the skill's resource content, system
|
||||
prompt additions, and notify configuration are editable through the admin UI.
|
||||
Skills shipped as part of a bundle remain locked (read-only) until explicitly
|
||||
unlocked; the unlock is logged to the audit trail. A lock icon in the
|
||||
top-right of the Skills detail pane doubles as the unlock trigger.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`skills.sh` install endpoint** — the install script was targeting an
|
||||
endpoint removed in an earlier refactor; switched to `/api/download`.
|
||||
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
|
||||
(object) instead of `[]` (array), causing notify configurations to be
|
||||
rejected at schema validation.
|
||||
- **Skills admin UI modal errors** — `.is-visible` class used consistently
|
||||
instead of inline `style.display`; stale error text is cleared on submit;
|
||||
designer-review lock-icon UX applied.
|
||||
|
||||
## [1.5.9]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`repair=False` on all display-read `load_messages` call sites** —
|
||||
passing `repair=True` on display paths was silently mutating the stored
|
||||
message list, causing divergence between what the UI showed and what the
|
||||
model received on the next turn.
|
||||
|
||||
## [1.5.8]
|
||||
|
||||
This release introduces two forward-only schema migrations:
|
||||
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
|
||||
`050_conversations_source_and_reminders` — `_source` and `_reminders` columns
|
||||
on `conversations`.
|
||||
|
||||
### Added
|
||||
|
||||
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
|
||||
configured with a client ID and secret through the admin UI. The full token
|
||||
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
|
||||
stored encrypted at rest using a key derived from the JWT secret. The consent
|
||||
flow runs in-browser via a provider redirect. Rolled out in phases:
|
||||
|
||||
- Minimum admin form and OAuth schema (`21663d15`).
|
||||
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
|
||||
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
|
||||
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
|
||||
- SDK 401/403 introspection via httpx response hook (`bde09134`).
|
||||
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
|
||||
their OAuth token is permitted to call (`cfc8a6c8`).
|
||||
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
|
||||
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
|
||||
use of an OAuth-gated server and can revoke consent from their profile;
|
||||
admins see per-server consent counts in the MCP Servers tab (`61051339`).
|
||||
|
||||
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
|
||||
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
|
||||
that delivers at most one nudge per turn, preventing multi-channel pile-ups
|
||||
that inflate context. Observable changes:
|
||||
|
||||
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
|
||||
through to the system message so the model can reason about recency.
|
||||
- Coordinator idle-children observer: a coordinator with no in-flight
|
||||
children for longer than the configured idle threshold receives a nudge.
|
||||
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
|
||||
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
|
||||
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
|
||||
rather than the previous `_watch_pending` list, giving them the same
|
||||
delivery guarantees and priority handling as other advisories.
|
||||
|
||||
- **Structured watch-result card** — the UI renders watch results as a styled
|
||||
card with a system-nudge marker, distinct from the assistant message body.
|
||||
On history replay, system-nudge turns are visually distinguished from normal
|
||||
assistant turns.
|
||||
- **Side-channel persistence** — `_source` and `_reminders` side-channel
|
||||
fields are persisted to the `conversations` storage table and restored on
|
||||
session resume, so metacognitive context survives process restarts. A
|
||||
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Replay consistency** — queued user messages captured mid-loop are now
|
||||
persisted and replayed in the correct order on a subsequent `events`
|
||||
subscription. Coordinator history replay fixed: blank assistant cards and
|
||||
out-of-order tool results on the coordinator tree no longer occur when the
|
||||
coordinator has mixed queued + delivered messages.
|
||||
- **Session reminder preservation on fork + resume** — `_source` and
|
||||
`_reminders` are carried through workstream fork and restored from storage
|
||||
on resume.
|
||||
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
|
||||
columns; `_source` and `_reminders` now strip NUL bytes on write.
|
||||
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
|
||||
now committed atomically on first model add; startup teardown is offloaded
|
||||
to avoid blocking the event loop.
|
||||
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
|
||||
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
|
||||
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
|
||||
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
|
||||
now replaces the pool entry with a fresh session; the carrier token is
|
||||
owned by the pool entry to prevent a race between the 401 handler and a
|
||||
concurrent request.
|
||||
- **OIDC hardening** — multiple security and correctness fixes:
|
||||
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
|
||||
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
|
||||
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
|
||||
callback robustness — typed exceptions, shape checks, log sanitization, JS
|
||||
race (bug-4–6, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
|
||||
stranded-user self-heal on role-mapping failure (cumulative bug-1).
|
||||
|
||||
## [1.5.7]
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline node picker** — a compact node-switcher dropdown in the console
|
||||
header replaces the "← Back to console" banner, so operators can switch
|
||||
between nodes without a full navigation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Queued user messages injected mid-loop** — messages queued while a
|
||||
generation was in progress were not being delivered at the correct seam and
|
||||
could be dropped or reordered when the worker consumed the queue.
|
||||
- **Search tool output bounded** — pathological inputs (very long lines with
|
||||
no whitespace) could produce search results exceeding the context budget.
|
||||
Output is now clamped before reaching the message.
|
||||
|
||||
## [1.5.6]
|
||||
|
||||
### Added
|
||||
|
||||
- **`api_surface` toggle** — model definitions gain an `api_surface` field
|
||||
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
|
||||
the provider client uses. Enables Mistral Medium reasoning via the Responses
|
||||
surface; Chat Completions remains the default for all other models.
|
||||
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
|
||||
includes a `healthy_aliases` list per node, so the coordinator and operators
|
||||
can see which model aliases are currently reachable without a separate
|
||||
per-model health probe.
|
||||
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
|
||||
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
|
||||
configure per-kind reasoning effort and alias overrides from the UI rather
|
||||
than editing `config.toml`. Live-refresh dropdowns update in place when
|
||||
model definitions change.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
|
||||
query-aware candidate-set selection, dramatically improving recall for
|
||||
queries whose terms span multiple stored entries.
|
||||
- **Workstream model + config preserved on rehydrate** — reopening a closed
|
||||
workstream no longer overwrites the model alias and per-workstream config
|
||||
with session defaults.
|
||||
- **Console home composer: attachments + user-message pills** — multipart
|
||||
attachments in the home composer were not forwarded correctly; user-message
|
||||
pills in the coordinator chat pane were missing.
|
||||
|
||||
## [1.5.5]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Saved-workstream tool result rendering** — tool results in closed
|
||||
workstreams were not rendering on history replay. Audit-trail decoration for
|
||||
tool calls is now applied on the replay path.
|
||||
|
||||
## [1.5.4]
|
||||
|
||||
### Added
|
||||
|
||||
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
|
||||
first-class citizens in the cluster event bus. `child_ws_state` events are
|
||||
pushed through the cluster SSE stream so the console tree view updates in
|
||||
real time without polling. `list_children` and `get_child` primitives on
|
||||
`SessionManager` provide a consistent cross-node view of the coordinator's
|
||||
spawn tree.
|
||||
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
|
||||
in the console admin panel now supports checkbox multi-select with a
|
||||
bulk-delete action.
|
||||
|
||||
## [1.5.3]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`048_workstream_reaper_index` — partial composite index on `workstreams` for
|
||||
the orphan-reaper query.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
|
||||
`close_idle` pass now scopes the DB-orphan reaper by
|
||||
`services.last_heartbeat` so workstreams belonging to a live node are not
|
||||
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
|
||||
storage primitives added; a partial composite index keeps the reaper scan
|
||||
cheap.
|
||||
- **Coordinator pool idle cleanup** — a periodic task on the console now
|
||||
closes coordinator pool entries whose session has gone idle past the
|
||||
configurable threshold, preventing pool exhaustion on long-running consoles.
|
||||
|
||||
## [1.5.2]
|
||||
|
||||
### Added
|
||||
|
||||
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
|
||||
nudges are rendered as a distinct styled bubble rather than being injected
|
||||
inline into the assistant message, making it easier to distinguish model
|
||||
output from metacognitive annotations. The CLI REPL gains matching
|
||||
`on_user_reminder` / `on_tool_reminder` callbacks.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
|
||||
now fires correctly on the third repetition; a write-success-clear that
|
||||
reset the counter after a successful tool call (preventing streaks across
|
||||
mixed-outcome sequences) was removed.
|
||||
- **Metacog reminders isolated to side-channel** — reminder text no longer
|
||||
appears in the user content turn; it flows through a dedicated side-channel
|
||||
the session injects into the system context, preventing the model from
|
||||
attributing it to the user.
|
||||
|
||||
## [1.5.1]
|
||||
|
||||
### Added
|
||||
|
||||
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
|
||||
now receive the child's pending approval detail in `child_ws_state` events,
|
||||
enabling the coordinator to surface approval prompts without a separate poll.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator registry auto-refresh** — the console coordinator registry now
|
||||
refreshes when model definitions change, so a newly added alias is visible
|
||||
to coordinators without restarting.
|
||||
- **Coordinator fan-out default** — coordinators now fan out to independent
|
||||
child workstreams by default instead of serialising them, matching the
|
||||
documented contract for parallel-work patterns.
|
||||
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
|
||||
summaries and tool results from child workstreams were silently truncated at
|
||||
the previous 4 KiB cap.
|
||||
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
|
||||
polling now runs on a dedicated 200-thread executor, matching interactive's
|
||||
`sse_executor`, so coordinator long-poll blocking no longer contends with
|
||||
storage and routing workers on the default pool.
|
||||
|
||||
## [1.5.0]
|
||||
|
||||
User-visible additions: a unified workstream HTTP surface (interactive and
|
||||
coordinator under one URL family), inline child approvals, coordinator
|
||||
composer parity, progressive rendering, OIDC authentication, MCP OAuth
|
||||
foundations, and a redesigned UI built on the Design System v1 token layer.
|
||||
|
||||
This release removes the pre-1.5 body-keyed and query-keyed URL family.
|
||||
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
|
||||
|
||||
This release introduces the following forward-only schema migrations that the
|
||||
server applies automatically on first startup. All are additive; no data loss.
|
||||
|
||||
- `039_workstream_kind` — `kind` + `parent_ws_id` columns on `workstreams`.
|
||||
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
|
||||
`admin.cluster.inspect` to the builtin-admin role.
|
||||
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
|
||||
introduced by 039.
|
||||
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
|
||||
builtin-admin.
|
||||
- `043_skill_description_required` — backfills empty `description` rows in
|
||||
`prompt_templates`.
|
||||
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
|
||||
(`interactive` / `coordinator` / `any`).
|
||||
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status` →
|
||||
`risk_level`.
|
||||
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
|
||||
by rendezvous routing in 1.4.
|
||||
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
|
||||
rows removed from the coordinator in 1.5.0a4.
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline child approvals** — pending tool approvals on coordinator child
|
||||
workstreams surface directly in the coordinator tree view. A risk pill shows
|
||||
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
|
||||
buttons appear inline so operators do not need to navigate to the child's
|
||||
workstream. `pending_approval_detail` is exposed on
|
||||
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
|
||||
so all connected clients render approval prompts simultaneously. LLM judge
|
||||
verdicts are cached client-side and replayed on SSE reconnect.
|
||||
- **Coordinator composer parity** — the coordinator composer now supports
|
||||
Stop, Send-to-queue, and Attach (file upload), matching the interactive
|
||||
workstream composer feature set.
|
||||
- **Per-call model and judge override on coordinator composer** — operators
|
||||
can override the model alias and judge model for a single coordinator send
|
||||
from the composer, without changing the node-wide or role-wide defaults. Bad
|
||||
aliases return a corrective error listing available choices.
|
||||
- **Coordinator status bar + richer history replay** — each coordinator
|
||||
workstream gains a per-coordinator status bar showing active children, token
|
||||
spend, and generation state. History replay in the coordinator panel is
|
||||
extended to include tool results and thinking blocks.
|
||||
- **Coordinator child error surfacing + memory tool** — child workstream
|
||||
errors are surfaced as distinct error rows in the coordinator tree view
|
||||
rather than disappearing silently. The coordinator gains access to a
|
||||
`memory` tool (same interface as interactive) for retrieving stored facts.
|
||||
- **Coordinator inline tool-batch construct** — the coordinator tool approval
|
||||
UI replaces the separate approval dock with an inline batch construct that
|
||||
groups all pending tool calls for a given turn into a single review card.
|
||||
- **Node capability auto-detection** — nodes report kernel-level capabilities
|
||||
(available memory, CPU count, accelerator presence) via
|
||||
`/v1/api/node/capabilities` at startup, enabling the console to filter model
|
||||
aliases offered to coordinators routing to that node.
|
||||
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
|
||||
a `SKILL.md` file's content into the modal auto-populates the name,
|
||||
description, and configuration fields.
|
||||
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
|
||||
soon as a complete diagram block is detected in the stream rather than
|
||||
waiting for the full response; the diagram re-renders in place as the model
|
||||
extends it.
|
||||
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
|
||||
math delimiters are now recognised alongside the existing `$$` fences.
|
||||
|
||||
### Removed (BREAKING — 1.5.0)
|
||||
|
||||
- **Legacy body-keyed and query-keyed URL family for the workstream
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ FROM python:3.14-slim
|
||||
LABEL org.opencontainers.image.title="turnstone" \
|
||||
org.opencontainers.image.description="Multi-node AI orchestration platform"
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.11.14 /uv /usr/local/bin/uv
|
||||
|
||||
# Remove the slim image's man page exclusion so man-db has actual content
|
||||
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
|
||||
|
||||
+46
-1
@@ -281,6 +281,7 @@ Each message in the `messages` array has:
|
||||
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
|
||||
| `content` | string or null | Text content of the message |
|
||||
| `tool_calls` | array or null | Present only on assistant messages with calls |
|
||||
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
|
||||
|
||||
Each entry in `tool_calls`:
|
||||
|
||||
@@ -325,6 +326,44 @@ finalize any in-progress assistant message.
|
||||
{"type": "stream_end"}
|
||||
```
|
||||
|
||||
**`state_change`** -- the worker thread transitioned to a new state. Drives
|
||||
the client's busy-mode (composer in send vs. stop, spinner indicators,
|
||||
auto-focus on idle). Sent live during normal operation AND on every fresh
|
||||
SSE subscribe (so a mid-stream page refresh restores the correct composer
|
||||
state without waiting for the next live transition).
|
||||
|
||||
```json
|
||||
{"type": "state_change", "state": "running"}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|----------|--------|----------------------------------------------------------------------|
|
||||
| `state` | string | One of `"running"`, `"thinking"`, `"attention"`, `"idle"`, `"error"` |
|
||||
|
||||
**`in_progress_snapshot`** -- one-shot replay of the in-progress turn's
|
||||
content + reasoning text-so-far when this client connects mid-stream.
|
||||
Lets a refreshing browser tab restore partial assistant text immediately
|
||||
instead of waiting for the response to complete. Yielded once after the
|
||||
kind-specific replay phase (history + pending), only when at least one
|
||||
of `content` / `reasoning` is non-empty. Both halves render into the same
|
||||
assistant bubble the live `content` / `reasoning` events would target;
|
||||
clients should treat the snapshot as idempotent (skip overwrite if the
|
||||
current local buffer is already a superset prefix — covers EventSource
|
||||
auto-reconnect re-replays).
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "in_progress_snapshot",
|
||||
"content": "Here is the answer so far: it depends on ",
|
||||
"reasoning": "The user is asking about a comparison; let me think about..."
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Description |
|
||||
|--------------|--------|------------------------------------------------------------|
|
||||
| `content` | string | Joined assistant content text accumulated this turn |
|
||||
| `reasoning` | string | Joined reasoning / chain-of-thought text accumulated |
|
||||
|
||||
**`tool_info`** -- one or more tool calls that were auto-approved (no user
|
||||
action required).
|
||||
|
||||
@@ -522,7 +561,13 @@ Each SSE connection to a workstream receives its own delivery queue. Events
|
||||
produced by the worker thread are fanned out to all registered listener queues,
|
||||
so multiple consumers (browser, console proxy, SDK) can connect
|
||||
simultaneously and each receives every event. On reconnect the client receives
|
||||
a full history replay, so no catch-up mechanism is needed.
|
||||
the kind-specific replay (`connected` + `status` + `history` + pending
|
||||
approval / plan for interactive; `connected` + `status` + pending for coord)
|
||||
followed by a `state_change` carrying the current worker state and an
|
||||
optional `in_progress_snapshot` carrying any partial content / reasoning
|
||||
buffered for the in-progress turn — so a mid-stream refresh restores both
|
||||
the busy-mode UI and the partial assistant text without waiting for the
|
||||
response to complete.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+45
-5
@@ -91,7 +91,7 @@ turnstone/
|
||||
discord/ Discord adapter (bot, cog, views, streaming, config)
|
||||
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
|
||||
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
|
||||
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
katex-0.16.47/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
ui/
|
||||
colors.py ANSI color constants with NO_COLOR support
|
||||
markdown.py Streaming terminal markdown renderer (line-buffered)
|
||||
@@ -231,11 +231,13 @@ The engine emits state changes via `_emit_state()` which calls
|
||||
|
||||
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
|
||||
|
||||
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
|
||||
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 16
|
||||
methods. Every frontend must implement all of them.
|
||||
|
||||
```python
|
||||
class SessionUI(Protocol):
|
||||
def on_turn_start(self) -> None: ...
|
||||
def on_turn_committed(self) -> None: ...
|
||||
def on_thinking_start(self) -> None: ...
|
||||
def on_thinking_stop(self) -> None: ...
|
||||
def on_reasoning_token(self, text: str) -> None: ...
|
||||
@@ -252,6 +254,14 @@ class SessionUI(Protocol):
|
||||
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
|
||||
```
|
||||
|
||||
`on_turn_start` fires at the top of each iteration of the send-loop;
|
||||
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
|
||||
`SessionUIBase` uses both to reset the per-turn inflight buffers
|
||||
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
|
||||
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
|
||||
the per-workstream events stream in
|
||||
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).
|
||||
|
||||
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
|
||||
|
||||
### Three Implementations
|
||||
@@ -619,14 +629,15 @@ LLMProvider (protocol)
|
||||
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
|
||||
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
|
||||
| `retryable_error_names` | Exception class names that trigger retry |
|
||||
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |
|
||||
|
||||
**Normalized data types:**
|
||||
|
||||
| Type | Fields |
|
||||
|------|--------|
|
||||
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
|
||||
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
|
||||
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
|
||||
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
|
||||
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
|
||||
|
||||
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
|
||||
@@ -714,6 +725,35 @@ and `"openai-compatible"`.
|
||||
`max_tokens`, and `reasoning_effort` to override the global defaults from
|
||||
ConfigStore. When unset (`NULL`), the global default is used.
|
||||
|
||||
**Per-model reasoning persistence:** Two booleans on `model_definitions`
|
||||
(migration 052) control how reasoning text round-trips:
|
||||
|
||||
* `surface_persisted_reasoning` (default `True`) — gates whether stored
|
||||
reasoning text is surfaced on `/history` payloads for UI rehydration.
|
||||
**Storage of reasoning bytes happens regardless of this flag** — they
|
||||
ride in `provider_data` independently. Phase-1 admin UI label "Surface
|
||||
persisted reasoning."
|
||||
* `replay_reasoning_to_model` (default `False`) — gates whether stored
|
||||
reasoning blocks are sent back to the provider on subsequent turns.
|
||||
Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
|
||||
also be `True` for the wire path to actually replay (canonical OpenAI
|
||||
gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
|
||||
server models default to `False`).
|
||||
|
||||
Three reasoning paths are recognised:
|
||||
|
||||
| Path | Provider | Capture | Persist | Replay |
|
||||
|------|----------|---------|---------|--------|
|
||||
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
|
||||
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
|
||||
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |
|
||||
|
||||
Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
|
||||
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
|
||||
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
|
||||
text+tool_calls rebuild path rather than reaching Anthropic's input
|
||||
boundary as malformed content.
|
||||
|
||||
```toml
|
||||
[models.local]
|
||||
base_url = "http://localhost:8000/v1"
|
||||
|
||||
@@ -110,11 +110,18 @@ owns it; the node is just currently unreachable.
|
||||
|
||||
### Example — `spawn_batch`
|
||||
|
||||
This is the coordinator-tool result shape (the JSON the LLM receives),
|
||||
not an HTTP API response — the table above keys it under "model tool"
|
||||
to distinguish it from the `/v1/api/...` endpoints in the same table.
|
||||
The underlying HTTP spawn endpoint still returns `ws_id`; the tool
|
||||
result re-keys it to `child_ws_id` to defuse a coordinator-LLM recency
|
||||
bias (see `docs/coordinator-skills.md`).
|
||||
|
||||
```json
|
||||
{
|
||||
"results": {
|
||||
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
|
||||
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
|
||||
"0": {"child_ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
|
||||
"2": {"child_ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
|
||||
},
|
||||
"denied": [
|
||||
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
|
||||
|
||||
@@ -115,7 +115,8 @@ with a `type` field. The recurring shapes a UI has to handle:
|
||||
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
|
||||
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
|
||||
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
|
||||
| `state_change` | Worker-thread state transition | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
|
||||
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state` ∈ `running`, `thinking`, `attention`, `idle`, `error` |
|
||||
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
|
||||
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
|
||||
| `rename` | Session's display name changed | `name` |
|
||||
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
|
||||
@@ -130,9 +131,13 @@ with a `type` field. The recurring shapes a UI has to handle:
|
||||
|
||||
**Reconnection contract:** a freshly-opened SSE connection receives
|
||||
the current snapshot of any pending tool approval (`approve_request`
|
||||
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
|
||||
indicator — so a tab refresh mid-approval doesn't strand the
|
||||
operator.
|
||||
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
|
||||
indicator, the worker's current `state_change`, and an
|
||||
`in_progress_snapshot` carrying any partial content / reasoning the
|
||||
model has produced for the in-progress turn — so a tab refresh
|
||||
mid-approval, mid-tool-execution, or mid-stream restores both the
|
||||
correct composer mode and the partial assistant text without waiting
|
||||
for the response to complete.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -169,14 +169,21 @@ validates ws_id against `parent_ws_id=coord_ws_id` AND
|
||||
the wait into reporting "complete".
|
||||
|
||||
Pattern: capture each spawn result in the next tool call's input.
|
||||
The JSON tool-result carries `{"ws_id": "...", "name": "...",
|
||||
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
|
||||
"node_id": "...", "routing_strategy": "..."}`; the model should
|
||||
extract the ws_id and pass it to `inspect_workstream` /
|
||||
`wait_for_workstream` / `send_to_workstream` / `close_workstream`
|
||||
verbatim.
|
||||
extract the `child_ws_id` and pass it as `ws_id` (or in the `ws_ids`
|
||||
list) to `inspect_workstream` / `wait_for_workstream` /
|
||||
`send_to_workstream` / `close_workstream` verbatim. The asymmetry
|
||||
— spawn returns `child_ws_id` but the other tools accept `ws_id` /
|
||||
`ws_ids` — is intentional: it defuses a coordinator-LLM recency
|
||||
bias where seeing `ws_id` in a spawn return primed re-spawn loops
|
||||
instead of progression to the wait phase.
|
||||
|
||||
A UI that wants human-readable identifiers should render the `name`
|
||||
field and keep the ws_id as the click-through key.
|
||||
field and keep the workstream id as the click-through key — note
|
||||
that the id *value* is the same regardless of whether it arrived
|
||||
under the `child_ws_id` key (spawn return) or the `ws_id` key
|
||||
(every other tool's input/output); only the field name differs.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -69,9 +69,10 @@ class "NullUI" as NullUI {
|
||||
interface "LLMProvider" as LLMProvider <<Protocol>> {
|
||||
+ provider_name: str {property}
|
||||
+ get_capabilities(model) → ModelCapabilities
|
||||
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
|
||||
+ create_completion(client, model, messages, ...) → CompletionResult
|
||||
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
|
||||
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
|
||||
+ convert_tools(tools) → list[dict]
|
||||
+ extract_reasoning_text(provider_blocks) → str
|
||||
+ retryable_error_names: frozenset[str] {property}
|
||||
--
|
||||
core/providers/_protocol.py
|
||||
@@ -126,6 +127,7 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
|
||||
+ supports_web_search: bool
|
||||
+ supports_tool_search: bool
|
||||
+ supports_vision: bool
|
||||
+ supports_reasoning_replay: bool
|
||||
}
|
||||
|
||||
' ChatSession
|
||||
|
||||
@@ -24,6 +24,14 @@ CS -> DB : save_message(ws_id, "user", input)
|
||||
|
||||
group loop [while tool_calls present]
|
||||
|
||||
CS -> UI : on_turn_start()
|
||||
note right of UI
|
||||
SessionUIBase resets the per-turn inflight
|
||||
buffers (_ws_inflight_content / reasoning /
|
||||
seq) that fuel the SSE in_progress_snapshot
|
||||
event for mid-stream refresh resume.
|
||||
end note
|
||||
|
||||
CS -> UI : on_state_change("thinking")
|
||||
CS -> UI : on_thinking_start()
|
||||
|
||||
@@ -73,6 +81,14 @@ group loop [while tool_calls present]
|
||||
|
||||
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
|
||||
CS -> CS : messages.append(assistant_msg)
|
||||
CS -> UI : on_turn_committed()
|
||||
note right of UI
|
||||
Drops the per-turn inflight buffers — the
|
||||
assistant message is now in the history
|
||||
list, so the in_progress_snapshot must
|
||||
not re-render it during the next tool-
|
||||
execution window or the next streaming turn.
|
||||
end note
|
||||
CS -> DB : save_message(ws_id, "assistant", content)
|
||||
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:25b5448bbb7da8ddafe4f65c6c5e6cbcaa9cb9f31746ca46d3a2241bc47b1956
|
||||
size 259687
|
||||
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
|
||||
size 620214
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3aa8d972bba40d78152f9f0c762b9f5ec616d8052c45fa52b7dd1c679ed81d61
|
||||
size 325245
|
||||
oid sha256:c14dfbb2db8dcb22cd332b2cf0e53ba75141adb213dae47dd9dbfd389ed482fe
|
||||
size 354799
|
||||
|
||||
@@ -84,6 +84,7 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|
||||
|----------|---------|-------------|
|
||||
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
|
||||
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
|
||||
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
|
||||
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
|
||||
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
|
||||
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# MCP OAuth — per-user authorization for MCP servers
|
||||
|
||||
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
|
||||
|
||||
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
|
||||
|
||||
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
|
||||
|
||||
---
|
||||
|
||||
## When to use which `auth_type`
|
||||
|
||||
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
|
||||
|
||||
| `auth_type` | What it means | When to use |
|
||||
|---|---|---|
|
||||
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
|
||||
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
|
||||
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
|
||||
|
||||
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites for `auth_type=oauth_user`
|
||||
|
||||
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
|
||||
|
||||
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
|
||||
|
||||
3. **OAuth client registration**. Two paths:
|
||||
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
|
||||
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
|
||||
|
||||
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Per-server fields (admin UI)
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
|
||||
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
|
||||
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
|
||||
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
|
||||
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
|
||||
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
|
||||
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
|
||||
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
|
||||
|
||||
### Encryption key
|
||||
|
||||
```toml
|
||||
[security]
|
||||
mcp_token_encryption_key = "base64-fernet-key"
|
||||
# For rotation, list the keys in priority order — first is used for new
|
||||
# writes, all are tried for reads.
|
||||
# mcp_token_encryption_keys = ["new-key", "old-key"]
|
||||
```
|
||||
|
||||
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
|
||||
|
||||
---
|
||||
|
||||
## Lifecycle
|
||||
|
||||
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
|
||||
|
||||
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
|
||||
|
||||
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
|
||||
|
||||
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
|
||||
|
||||
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
|
||||
|
||||
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
|
||||
|
||||
---
|
||||
|
||||
## Admin status indicators
|
||||
|
||||
The MCP Servers admin tab shows per-server status pills (Phase 9):
|
||||
|
||||
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
|
||||
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
|
||||
|
||||
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
|
||||
|
||||
---
|
||||
|
||||
## Auth-type transitions
|
||||
|
||||
| From | To | What happens |
|
||||
|---|---|---|
|
||||
| `none` / `static` → `oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
|
||||
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
|
||||
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
|
||||
|
||||
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---|---|---|
|
||||
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
|
||||
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
|
||||
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
|
||||
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
|
||||
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
|
||||
|
||||
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
|
||||
@@ -0,0 +1,29 @@
|
||||
# MCP OAuth in headless / scheduled / channel-driven runs
|
||||
|
||||
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
|
||||
|
||||
**Affected surfaces**:
|
||||
|
||||
- Scheduled workstreams (`turnstone-console` task scheduler).
|
||||
- Discord adapter runs.
|
||||
- Slack adapter runs.
|
||||
- Any future channel adapter without an interactive browser session.
|
||||
|
||||
**What happens when consent is missing**:
|
||||
|
||||
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
|
||||
|
||||
**Recovery**:
|
||||
|
||||
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
|
||||
|
||||
**Pre-consent recipe**:
|
||||
|
||||
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
|
||||
|
||||
1. Open the dashboard.
|
||||
2. Open the settings modal (gear icon).
|
||||
3. Click Connect on each MCP server the schedule will use.
|
||||
4. Confirm consent in the popup.
|
||||
|
||||
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
|
||||
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
|
||||
not use server-side prepared statements by default, so this is not an
|
||||
issue.
|
||||
|
||||
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
|
||||
transaction pooling assigns a real server connection only for the
|
||||
duration of each transaction, then returns it to the pool. PostgreSQL
|
||||
`LISTEN` is session state — a transaction-pooled client can't hold the
|
||||
multi-statement session a long-lived `LISTEN` needs. The console's
|
||||
`NotifyDispatcher` (reactive node discovery via the `services` channel)
|
||||
therefore opens a **dedicated, direct-to-Postgres** connection that
|
||||
bypasses PgBouncer.
|
||||
|
||||
Configure via `config.toml` `[database] listen_url` (preferred —
|
||||
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
|
||||
(config.toml wins when both are set). Defaults to the main DB URL when
|
||||
unset.
|
||||
|
||||
| Setting | Behaviour |
|
||||
|---|---|
|
||||
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
|
||||
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
|
||||
|
||||
Set this whenever PgBouncer is in transaction mode (the recommended
|
||||
setting per this doc). The override only adds one long-lived PG
|
||||
connection per console process — sized into the cluster's
|
||||
`max_connections` budget alongside the pool.
|
||||
|
||||
See also: [Docker deployment](docker.md) · [Security](security.md)
|
||||
|
||||
@@ -138,6 +138,9 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
|
||||
| `error` | `ErrorEvent` | `message` |
|
||||
| `info` | `InfoEvent` | `message` |
|
||||
| `stream_end` | `StreamEndEvent` | — |
|
||||
| `state_change` | `StateChangeEvent` | `state` ∈ `running`/`thinking`/`attention`/`idle`/`error` |
|
||||
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
|
||||
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
|
||||
| `cancelled` | `CancelledEvent` | — |
|
||||
|
||||
**Global events** (from `stream_global_events()`):
|
||||
|
||||
@@ -59,6 +59,21 @@ from ConfigStore. Model names and context windows are now configured per-model
|
||||
in the Models tab. A startup warning is logged if these keys appear in
|
||||
`config.toml`.
|
||||
|
||||
### Reasoning persistence (per-model)
|
||||
|
||||
Two boolean flags on `model_definitions` (migration 052) control how
|
||||
reasoning text round-trips per model:
|
||||
|
||||
| Flag | Default | Effect |
|
||||
|------|---------|--------|
|
||||
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
|
||||
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
|
||||
|
||||
Edit both via the admin Models tab. See the architecture doc for the
|
||||
provider-side mechanics (Anthropic `thinking`, OpenAI Responses
|
||||
`reasoning` + `include=["reasoning.encrypted_content"]`, synthetic
|
||||
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
|
||||
|
||||
### Plan / task agent overrides
|
||||
|
||||
`plan_agent` and `task_agent` sub-sessions resolve independently from the
|
||||
|
||||
+3
-3
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "1.5.10"
|
||||
version = "1.5.18"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
@@ -82,9 +82,9 @@ include = [
|
||||
"turnstone/console/static/coordinator/*.js",
|
||||
"turnstone/shared_static/*.css",
|
||||
"turnstone/shared_static/*.js",
|
||||
"turnstone/shared_static/katex-0.16.45/**/*",
|
||||
"turnstone/shared_static/katex-0.16.47/**/*",
|
||||
"turnstone/shared_static/hljs-11.11.1/**/*",
|
||||
"turnstone/shared_static/mermaid-11.14.0/**/*",
|
||||
"turnstone/shared_static/mermaid-11.15.0/**/*",
|
||||
"turnstone/shared_static/hls-1.6.16/**/*",
|
||||
"turnstone/sdk/py.typed",
|
||||
"turnstone/deploy/*.yaml",
|
||||
|
||||
@@ -50,11 +50,14 @@ update_refs() {
|
||||
local old_pattern="$1" # e.g. katex-0.16.38
|
||||
local new_pattern="$2" # e.g. katex-0.16.39
|
||||
|
||||
# Find all files with version references (excludes vendored JS and worktrees)
|
||||
# Find all files with version references. Excludes the old versioned vendor
|
||||
# directory itself (about to be rm -rf'd anyway) so we don't bother rewriting
|
||||
# self-references inside it — but does NOT exclude all of shared_static/,
|
||||
# because shared_static/renderer.js loads the vendored libs and needs the bump.
|
||||
local files
|
||||
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' \
|
||||
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' --include='*.py' \
|
||||
-F "$old_pattern" . \
|
||||
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir='shared_static' \
|
||||
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir="$old_pattern" \
|
||||
2>/dev/null || true)
|
||||
for f in $files; do
|
||||
sed -i "s|${old_pattern}|${new_pattern}|g" "$f"
|
||||
|
||||
Generated
+127
-127
@@ -74,9 +74,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-project/types": {
|
||||
"version": "0.127.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
|
||||
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
|
||||
"version": "0.130.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.130.0.tgz",
|
||||
"integrity": "sha512-ibD2usx9JRu7f5pu2tMKMI4cpA4NgXJQoYRP4pQ7Pxmn1l6k/53qWtQWZayhYy3X4QZkt90Ot+mJEaeXouio6Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -84,9 +84,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-android-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-fJI3I0r3C3Oj/zdBCpaCmBRZYf07xpaq4yCfDDoSFm+beWNzbIl26puW8RraUdugoJw/95zerNOn6jasAhzSmg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -101,9 +101,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-cKnAhWEsV7TPcA/5EAteDp6KcJZBQ2G+BqE7zayMMi7kMvwRsbv7WT9aOnn0WNl4SKEIf43vjS31iUPu80nzXg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -118,9 +118,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.1.tgz",
|
||||
"integrity": "sha512-YKrVwQjIRBPo+5G/u03wGjbdy4q7pyzCe93DK9VJ7zkVmeg8LJ7GbgsiHWdR4xSoe4CAXRD7Bcjgbtr64bkXNg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -135,9 +135,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-freebsd-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.1.tgz",
|
||||
"integrity": "sha512-z/oBsREo46SsFqBwYtFe0kpJeBijAT48O/WXLI4suiCLBkr03RTtTJMCzSdDd2znlh8VJizL09XVkQgk8IZonw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -152,9 +152,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.1.tgz",
|
||||
"integrity": "sha512-ik8q7GM11zxvYxFc2PeDcT6TBvhCQMaUxfph/M5l9sKuTs/Sjg3L+Byw0F7w0ZVLBZmx30P+gG0ECzzN+MFcmQ==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
@@ -169,9 +169,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.1.tgz",
|
||||
"integrity": "sha512-QoSx2EkyrrdZ6kcyE8stqZ62t0Yra8Fs5ia9lOxJrh6TMQJK7gQKmscdTHf7pOXKREKrVwOtJcQG3qVSfc866A==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -189,9 +189,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.1.tgz",
|
||||
"integrity": "sha512-uwNwFpwKeNiZawfAWBgg0VIztPTV3ihhh1vV334h9ivnNLorxnQMU6Fz8wG1Zb4Qh9LC1/MkcyT3YlDXG3Rsgg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -209,9 +209,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.1.tgz",
|
||||
"integrity": "sha512-zY1bul7OWr7DFBiJ++wofXvnr8B45ce3QsQUhKrIhXsygAh7bTkwyeM1bi1a2g5C/yC/N8TZyGDEoMfm/l9mpg==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
@@ -229,9 +229,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-s390x-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.1.tgz",
|
||||
"integrity": "sha512-0frlsT/f4Ft6I7SMESTKnF3cZsdicQn1dCMkF/jT9wDLE+gGoiQfv1nmT9e+s7s/fekvvy6tZM2jHvI2tkbJDQ==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
@@ -249,9 +249,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.1.tgz",
|
||||
"integrity": "sha512-XABVmGp9Tg0WspTVvwduTc4fpqy6JnAUrSQe6OuyqD/03nI7r0O9OWUkMIwFrjKAIqolvqoA4ZrJppgwE0Gxmw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -269,9 +269,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.1.tgz",
|
||||
"integrity": "sha512-bV4fzswuzVcKD90o/VM6QqKxnxlDq0g2BISDLNVmxrnhpv1DDbyPhCIjYfvzYLV+MvkKKnQt2Q6AO86SEBULUQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -289,9 +289,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-openharmony-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.1.tgz",
|
||||
"integrity": "sha512-/Mh0Zhq3OP7fVs0kcQHZP6lZEthMGTaSf8UBQYSFEZDWGXXlEC+nJ6EqenaK2t4LBXMe3A+K/G2BVXXdtOr4PQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -306,9 +306,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-wasm32-wasi": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.1.tgz",
|
||||
"integrity": "sha512-+1xc9X45l8ufsBAm6Gjvx2qDRIY9lTVt0cgWNcJ+1gdhXvkbxePA60yRTwSTuXL09CMhyJmjpV7E3NoyxbqFQQ==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
@@ -325,9 +325,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-arm64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.1.tgz",
|
||||
"integrity": "sha512-1D+UqZdfnuR+Jy1GgMJwi85bD40H21uNmOPRWQhw4oRSuolZ/B5rixZ45DK2KXOTCvmVCecauWgEhbw8bI7tOw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -342,9 +342,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-x64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.1.tgz",
|
||||
"integrity": "sha512-INAycaWuhlOK3wk4mRHGsdgwYWmd9cChdPdE9bwWmy6rn9VqVNYNFGhOdXrofXUxwHIncSiPNb8tNm8knDVIeQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -359,9 +359,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/pluginutils": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz",
|
||||
"integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -402,23 +402,23 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
|
||||
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
|
||||
"version": "1.0.9",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
|
||||
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@vitest/expect": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz",
|
||||
"integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.6.tgz",
|
||||
"integrity": "sha512-7EHDquPthALSV0jhhjgEW8FXaviMx7rSqu8W6oqCoAuOhKov814P99QDV1pxMA3QPv21YudvJngIhjrNI4opLg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@standard-schema/spec": "^1.1.0",
|
||||
"@types/chai": "^5.2.2",
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"chai": "^6.2.2",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
@@ -427,13 +427,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/mocker": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz",
|
||||
"integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.6.tgz",
|
||||
"integrity": "sha512-MCFc63czMjEInOlcY2cpQCvCN+KgbAn+60xu9cMgP4sKaLC5JNAKw7JH8QdAnoAC88hW1IiSNZ+GgVXlN1UcMQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"estree-walker": "^3.0.3",
|
||||
"magic-string": "^0.30.21"
|
||||
},
|
||||
@@ -454,9 +454,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/pretty-format": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz",
|
||||
"integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.6.tgz",
|
||||
"integrity": "sha512-h5SxD/IzNhZYnrSZRsUZQIC+vD0GY8cUvq0iwsmkFKixRCKLLWqCXa/FIQ4S1R+sI+PGoojkHsdNrbZiM9Qpgw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -467,13 +467,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/runner": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz",
|
||||
"integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.6.tgz",
|
||||
"integrity": "sha512-nOPCmn2+yD0ZNmKdsXGv/UxMMWbMuKeD6GyYncNwdkYDxpQvrPSKYj2rWuDjC2Y4b6w6hjip5dBKFzEUuZe3vA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"pathe": "^2.0.3"
|
||||
},
|
||||
"funding": {
|
||||
@@ -481,14 +481,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/snapshot": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz",
|
||||
"integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.6.tgz",
|
||||
"integrity": "sha512-YhsdE6xAVfTDmzjxL2ZDUvjj+ZsgyOKe+TdQzqkD72wIOmHka8NuGQ6NpTNZv9D2Z63fbwWKJPeVpEw4EQgYxw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"magic-string": "^0.30.21",
|
||||
"pathe": "^2.0.3"
|
||||
},
|
||||
@@ -497,9 +497,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/spy": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz",
|
||||
"integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.6.tgz",
|
||||
"integrity": "sha512-JFKxMx6udhwKh/Ldo270e17QX710vgunMkuPAvXjHSvC6oqLWAHhVhjg/I71q0u0CBSErIODV1Kjv0FQNSWjdg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -507,13 +507,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/utils": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz",
|
||||
"integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.6.tgz",
|
||||
"integrity": "sha512-FxIY+U81R3LGKCxaHHFRQ5+g6/iRgGLmeHWdp2Amj4ljQRrEIWHmZyDfDYBRZlpyqA7qKxtS9DD1dhk8RnRIVQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"convert-source-map": "^2.0.0",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
@@ -959,9 +959,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.5.13",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.13.tgz",
|
||||
"integrity": "sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==",
|
||||
"version": "8.5.14",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
|
||||
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -988,14 +988,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/rolldown": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz",
|
||||
"integrity": "sha512-X0KQHljNnEkWNqqiz9zJrGunh1B0HgOxLXvnFpCOcadzcy5qohZ3tqMEUg00vncoRovXuK3ZqCT9KnnKzoInFQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@oxc-project/types": "=0.127.0",
|
||||
"@rolldown/pluginutils": "1.0.0-rc.17"
|
||||
"@oxc-project/types": "=0.130.0",
|
||||
"@rolldown/pluginutils": "^1.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"rolldown": "bin/cli.mjs"
|
||||
@@ -1004,21 +1004,21 @@
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
|
||||
"@rolldown/binding-android-arm64": "1.0.1",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.1",
|
||||
"@rolldown/binding-darwin-x64": "1.0.1",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.1",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.1",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.1",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.1",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.1",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.1",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.1",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.1",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.1",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.1",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.1",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.1"
|
||||
}
|
||||
},
|
||||
"node_modules/siginfo": {
|
||||
@@ -1119,16 +1119,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "8.0.10",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
|
||||
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
|
||||
"version": "8.0.13",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.13.tgz",
|
||||
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.4",
|
||||
"postcss": "^8.5.10",
|
||||
"rolldown": "1.0.0-rc.17",
|
||||
"postcss": "^8.5.14",
|
||||
"rolldown": "1.0.1",
|
||||
"tinyglobby": "^0.2.16"
|
||||
},
|
||||
"bin": {
|
||||
@@ -1145,7 +1145,7 @@
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@types/node": "^20.19.0 || >=22.12.0",
|
||||
"@vitejs/devtools": "^0.1.0",
|
||||
"@vitejs/devtools": "^0.1.18",
|
||||
"esbuild": "^0.27.0 || ^0.28.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "^4.0.0",
|
||||
@@ -1197,19 +1197,19 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vitest": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz",
|
||||
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.6.tgz",
|
||||
"integrity": "sha512-6lvjbS3p9b4CrdCmguzbh2/4uoXhGE2q71R4OX5sqF9R1bo9Xd6fGrMAfvp5wnCzlBnFVdCOp6onuTQVbo8iUQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/expect": "4.1.5",
|
||||
"@vitest/mocker": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/runner": "4.1.5",
|
||||
"@vitest/snapshot": "4.1.5",
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/expect": "4.1.6",
|
||||
"@vitest/mocker": "4.1.6",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"@vitest/runner": "4.1.6",
|
||||
"@vitest/snapshot": "4.1.6",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"es-module-lexer": "^2.0.0",
|
||||
"expect-type": "^1.3.0",
|
||||
"magic-string": "^0.30.21",
|
||||
@@ -1237,12 +1237,12 @@
|
||||
"@edge-runtime/vm": "*",
|
||||
"@opentelemetry/api": "^1.9.0",
|
||||
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
|
||||
"@vitest/browser-playwright": "4.1.5",
|
||||
"@vitest/browser-preview": "4.1.5",
|
||||
"@vitest/browser-webdriverio": "4.1.5",
|
||||
"@vitest/coverage-istanbul": "4.1.5",
|
||||
"@vitest/coverage-v8": "4.1.5",
|
||||
"@vitest/ui": "4.1.5",
|
||||
"@vitest/browser-playwright": "4.1.6",
|
||||
"@vitest/browser-preview": "4.1.6",
|
||||
"@vitest/browser-webdriverio": "4.1.6",
|
||||
"@vitest/coverage-istanbul": "4.1.6",
|
||||
"@vitest/coverage-v8": "4.1.6",
|
||||
"@vitest/ui": "4.1.6",
|
||||
"happy-dom": "*",
|
||||
"jsdom": "*",
|
||||
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
|
||||
|
||||
@@ -13,6 +13,20 @@ export interface ConnectedEvent {
|
||||
|
||||
export interface HistoryEvent {
|
||||
type: "history";
|
||||
/**
|
||||
* Per-message dicts the frontend consumes directly. Common optional keys:
|
||||
* - `role`: "user" | "assistant" | "tool"
|
||||
* - `content`: string or list (image/document parts)
|
||||
* - `tool_calls`: assistant turns — list of `{id, name, arguments, verdict?, output_assessment?}`
|
||||
* - `tool_call_id`: tool turns — id of the originating call
|
||||
* - `reminders`: metacognitive nudge bubbles (user/tool channels)
|
||||
* - `advisories`: extracted `UserInterjection` payloads on tool turns
|
||||
* - `reasoning`: concatenated reasoning text for assistant turns whose
|
||||
* `provider_data` carried reasoning-bearing blocks (Anthropic
|
||||
* `thinking`, OpenAI Responses `reasoning`, or synthetic
|
||||
* `reasoning_text` from path-3 servers). Present only when the
|
||||
* active model's `surface_persisted_reasoning` flag is true.
|
||||
*/
|
||||
messages: Array<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
@@ -38,6 +52,14 @@ export interface StreamEndEvent {
|
||||
type: "stream_end";
|
||||
}
|
||||
|
||||
/** One-shot replay of the in-progress turn's content + reasoning emitted
|
||||
* by the events SSE handler when a fresh subscriber connects mid-stream. */
|
||||
export interface InProgressSnapshotEvent {
|
||||
type: "in_progress_snapshot";
|
||||
content: string;
|
||||
reasoning: string;
|
||||
}
|
||||
|
||||
export interface StateChangeEvent {
|
||||
type: "state_change";
|
||||
state: "idle" | "thinking" | "running" | "attention" | "error";
|
||||
@@ -162,6 +184,7 @@ export type ServerEvent =
|
||||
| ContentEvent
|
||||
| ReasoningEvent
|
||||
| StreamEndEvent
|
||||
| InProgressSnapshotEvent
|
||||
| StateChangeEvent
|
||||
| ToolInfoEvent
|
||||
| ApproveRequestEvent
|
||||
@@ -261,6 +284,12 @@ export function isStreamEndEvent(e: ServerEvent): e is StreamEndEvent {
|
||||
return e.type === "stream_end";
|
||||
}
|
||||
|
||||
export function isInProgressSnapshotEvent(
|
||||
e: ServerEvent,
|
||||
): e is InProgressSnapshotEvent {
|
||||
return e.type === "in_progress_snapshot";
|
||||
}
|
||||
|
||||
export function isStateChangeEvent(e: ServerEvent): e is StateChangeEvent {
|
||||
return e.type === "state_change";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Shared session-test helpers.
|
||||
|
||||
Two reasoning-test modules (``test_session_replay_reasoning.py`` and
|
||||
``test_session_synth_reasoning_block.py``) need the same minimal
|
||||
``ChatSession`` factory + a ``SessionUIBase`` no-op subclass. Hoisting
|
||||
keeps a future third caller from drifting on the defaults — the third
|
||||
existing ``_make_session`` (``test_model_registry.py``) deliberately
|
||||
takes a different signature (registry / model_alias / reasoning_effort
|
||||
+ ``_FakeUI``) and is NOT a candidate for sharing this helper.
|
||||
|
||||
Module is named with a leading underscore so pytest doesn't try to
|
||||
collect it as a test file — it's an importable utility, not a test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.session_ui_base import SessionUIBase
|
||||
|
||||
|
||||
class NullUI(SessionUIBase):
|
||||
"""Bare-bones UI satisfying the SessionUIBase contract for tests
|
||||
that don't care about UI side effects."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
|
||||
|
||||
def make_session(**kwargs: Any) -> ChatSession:
|
||||
"""Build a ChatSession with minimal defaults; tests override
|
||||
individual fields via kwargs."""
|
||||
defaults: dict[str, Any] = {
|
||||
"client": MagicMock(),
|
||||
"model": "test-model",
|
||||
"ui": NullUI(),
|
||||
"instructions": None,
|
||||
"temperature": 0.5,
|
||||
"max_tokens": 4096,
|
||||
"tool_timeout": 30,
|
||||
}
|
||||
defaults.update(kwargs)
|
||||
return ChatSession(**defaults)
|
||||
@@ -0,0 +1,220 @@
|
||||
"""Tests for turnstone-admin DB configuration precedence.
|
||||
|
||||
Locks in the alignment with turnstone-server:
|
||||
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded default
|
||||
|
||||
The motivation is to keep DB secrets in config.toml (see
|
||||
feedback_secrets_not_in_env) rather than forcing operators to export
|
||||
TURNSTONE_DB_URL before every admin invocation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import turnstone.core.config as config_mod
|
||||
from turnstone.admin import _get_storage
|
||||
|
||||
|
||||
def _reset_cache() -> None:
|
||||
config_mod._cache = None
|
||||
config_mod._config_path = None
|
||||
|
||||
|
||||
def _build_args(config_path: str | None) -> argparse.Namespace:
|
||||
"""Build an args namespace the way admin.main() does.
|
||||
|
||||
Skips ``add_config_arg`` (which reads ``sys.argv``) — the test
|
||||
constructs the args programmatically instead.
|
||||
"""
|
||||
config_mod.set_config_path(config_path or "/nonexistent/turnstone-admin-test.toml")
|
||||
parser = argparse.ArgumentParser()
|
||||
config_mod.apply_config(parser, ["database"])
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
sub.add_parser("list-users")
|
||||
return parser.parse_args(["list-users"])
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clear_db_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
|
||||
"""Clean slate: no TURNSTONE_DB_* env vars unless a test sets them."""
|
||||
for var in (
|
||||
"TURNSTONE_DB_BACKEND",
|
||||
"TURNSTONE_DB_URL",
|
||||
"TURNSTONE_DB_PATH",
|
||||
"TURNSTONE_DB_POOL_SIZE",
|
||||
"TURNSTONE_DB_SSLMODE",
|
||||
"TURNSTONE_DB_SSLROOTCERT",
|
||||
"TURNSTONE_DB_SSLCERT",
|
||||
"TURNSTONE_DB_SSLKEY",
|
||||
"TURNSTONE_CONFIG",
|
||||
):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
_reset_cache()
|
||||
yield
|
||||
_reset_cache()
|
||||
|
||||
|
||||
def test_defaults_to_sqlite_when_neither_config_nor_env_set() -> None:
|
||||
args = _build_args(None)
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
assert init.call_args.args == ("sqlite",)
|
||||
assert init.call_args.kwargs["url"] == ""
|
||||
assert init.call_args.kwargs["path"] == ""
|
||||
assert init.call_args.kwargs["pool_size"] == 2
|
||||
|
||||
|
||||
def test_config_toml_database_section_drives_init_storage(tmp_path: Path) -> None:
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text(
|
||||
"[database]\n"
|
||||
'backend = "postgresql"\n'
|
||||
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
|
||||
"pool_size = 5\n"
|
||||
'sslmode = "verify-full"\n'
|
||||
'sslrootcert = "/etc/ssl/ca.pem"\n'
|
||||
'sslcert = "/etc/ssl/client.pem"\n'
|
||||
'sslkey = "/etc/ssl/client.key"\n'
|
||||
)
|
||||
args = _build_args(str(cfg))
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
assert init.call_args.args == ("postgresql",)
|
||||
kw = init.call_args.kwargs
|
||||
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
|
||||
assert kw["pool_size"] == 5
|
||||
assert kw["sslmode"] == "verify-full"
|
||||
assert kw["sslrootcert"] == "/etc/ssl/ca.pem"
|
||||
assert kw["sslcert"] == "/etc/ssl/client.pem"
|
||||
assert kw["sslkey"] == "/etc/ssl/client.key"
|
||||
|
||||
|
||||
def test_env_used_as_fallback_when_config_absent(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "postgresql")
|
||||
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
|
||||
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "7")
|
||||
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
|
||||
|
||||
args = _build_args(None)
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
assert init.call_args.args == ("postgresql",)
|
||||
kw = init.call_args.kwargs
|
||||
assert kw["url"] == "postgresql+psycopg://fromenv:x@host/db"
|
||||
assert kw["pool_size"] == 7
|
||||
assert kw["sslmode"] == "require"
|
||||
|
||||
|
||||
def test_config_toml_wins_over_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""config.toml beats env — operators should put secrets in TOML."""
|
||||
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
|
||||
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
|
||||
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text(
|
||||
"[database]\n"
|
||||
'backend = "postgresql"\n'
|
||||
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
|
||||
'sslmode = "verify-full"\n'
|
||||
)
|
||||
args = _build_args(str(cfg))
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
assert init.call_args.args == ("postgresql",)
|
||||
kw = init.call_args.kwargs
|
||||
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
|
||||
assert kw["sslmode"] == "verify-full"
|
||||
|
||||
|
||||
def test_partial_config_falls_through_to_env_per_key(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""A key missing from [database] should fall back to its env var."""
|
||||
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
|
||||
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "9")
|
||||
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text(
|
||||
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromconfig:x@host/db"\n'
|
||||
)
|
||||
args = _build_args(str(cfg))
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
kw = init.call_args.kwargs
|
||||
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
|
||||
assert kw["sslmode"] == "require"
|
||||
assert kw["pool_size"] == 9
|
||||
|
||||
|
||||
def test_empty_string_in_config_beats_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
"""`url = ""` in config.toml beats an env var.
|
||||
|
||||
Locks in the `is not None` guard — a falsy-but-present TOML value
|
||||
should NOT silently fall through to the env fallback.
|
||||
"""
|
||||
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text('[database]\nbackend = "sqlite"\nurl = ""\n')
|
||||
args = _build_args(str(cfg))
|
||||
with patch("turnstone.core.storage.init_storage") as init:
|
||||
_get_storage(args)
|
||||
assert init.call_args.kwargs["url"] == ""
|
||||
|
||||
|
||||
def test_main_threads_config_toml_through_real_argv(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""End-to-end: ``turnstone-admin --config <toml> list-users`` honors TOML.
|
||||
|
||||
Covers the ``add_config_arg`` -> ``apply_config`` -> ``_get_storage``
|
||||
chain that the programmatic ``_build_args`` helper skips.
|
||||
"""
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text(
|
||||
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromcli:x@host/db"\n'
|
||||
)
|
||||
monkeypatch.setattr("sys.argv", ["turnstone-admin", "--config", str(cfg), "list-users"])
|
||||
|
||||
fake_storage = patch("turnstone.core.storage.init_storage").start()
|
||||
fake_storage.return_value.list_users.return_value = []
|
||||
try:
|
||||
from turnstone.admin import main
|
||||
|
||||
main()
|
||||
finally:
|
||||
patch.stopall()
|
||||
|
||||
assert fake_storage.call_args.args == ("postgresql",)
|
||||
assert fake_storage.call_args.kwargs["url"] == "postgresql+psycopg://fromcli:x@host/db"
|
||||
|
||||
|
||||
def test_get_storage_initializes_real_sqlite_backend(tmp_path: Path) -> None:
|
||||
"""Drives the real ``init_storage`` boundary on a fresh sqlite file.
|
||||
|
||||
Mock-only tests would miss a kwarg-name typo (sslmode -> ssl_mode).
|
||||
This test trips on any such drift because Alembic + the backend
|
||||
actually run.
|
||||
"""
|
||||
from turnstone.core.storage import reset_storage
|
||||
|
||||
db_file = tmp_path / "admin.db"
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text(f'[database]\nbackend = "sqlite"\npath = "{db_file}"\n')
|
||||
args = _build_args(str(cfg))
|
||||
|
||||
reset_storage()
|
||||
try:
|
||||
storage = _get_storage(args)
|
||||
assert storage.list_users() == []
|
||||
finally:
|
||||
reset_storage()
|
||||
+75
-4
@@ -340,7 +340,7 @@ def test_phase8_no_unsafe_dom_write_in_settings_panel() -> None:
|
||||
|
||||
|
||||
def test_phase8_settings_button_in_index_html() -> None:
|
||||
"""The gear-icon entry-point for the settings panel must remain
|
||||
"""The gear-icon entry-point for the settings menu must remain
|
||||
in the appbar's actions span. The console proxy IIFE prepends a
|
||||
node pill to ``header.firstChild`` (turnstone/console/server.py:
|
||||
202); our button is appended inside ``<span class='appbar-actions'>``
|
||||
@@ -351,9 +351,10 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
"index.html must keep the #settings-btn — onclick handlers "
|
||||
"and the consent badge target it by id."
|
||||
)
|
||||
assert 'onclick="openSettingsPanel()"' in body, (
|
||||
"settings-btn must wire onclick=openSettingsPanel() — losing "
|
||||
"the binding leaves the panel unreachable."
|
||||
assert 'onclick="toggleSettingsMenu(this)"' in body, (
|
||||
"settings-btn must wire onclick=toggleSettingsMenu(this) — "
|
||||
"the gear opens a dropdown with MCP connections + Logout; "
|
||||
"losing the binding leaves the menu unreachable."
|
||||
)
|
||||
# The button must live inside <span class="appbar-actions"> so the
|
||||
# console proxy's header.insertBefore(pill, header.firstChild)
|
||||
@@ -366,6 +367,76 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_settings_menu_handlers_defined() -> None:
|
||||
"""The gear-icon dropdown exposes a toggle/open/close trio that the
|
||||
inline ``onclick="toggleSettingsMenu(this)"`` in index.html depends
|
||||
on, plus the menu items themselves must wire to existing entry
|
||||
points (``openSettingsPanel`` for MCP connections, ``logout`` for
|
||||
sign-out). Pin all four so a rename or deletion fails loudly here
|
||||
instead of silently leaving the gear's menu broken or wired to a
|
||||
stale function."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
for name in [
|
||||
"function toggleSettingsMenu",
|
||||
"function openSettingsMenu",
|
||||
"function closeSettingsMenu",
|
||||
]:
|
||||
assert name in body, f"Missing required handler: {name}"
|
||||
# Bound to the settings-menu region so we don't accidentally match
|
||||
# an unrelated openSettingsPanel/logout call elsewhere in the file.
|
||||
start = body.index("function openSettingsMenu(")
|
||||
end = body.index("function closeSettingsMenu(", start)
|
||||
section = body[start:end]
|
||||
assert "openSettingsPanel()" in section, (
|
||||
"Settings menu's MCP-connections item must call openSettingsPanel() "
|
||||
"— otherwise the existing settings overlay is unreachable from the "
|
||||
"new dropdown."
|
||||
)
|
||||
assert "logout()" in section, (
|
||||
"Settings menu's Logout item must call logout() — that's the "
|
||||
"shared auth.js entry point that clears the cookie + session state."
|
||||
)
|
||||
|
||||
|
||||
def test_dashboard_overlay_is_region_not_dialog() -> None:
|
||||
"""The dashboard overlay must be role='region' (not role='dialog' +
|
||||
aria-modal='true'). The role downgrade is what allows ui-header to
|
||||
stay interactive while the dashboard is open — see the comment at
|
||||
showDashboard() in app.js. A revert to role='dialog' + aria-modal
|
||||
would re-trap focus and break the gear/theme buttons + the console
|
||||
proxy's node-picker pill while the dashboard is open."""
|
||||
body = _INDEX_HTML.read_text(encoding="utf-8")
|
||||
idx = body.index('id="dashboard"')
|
||||
# Bound to ~600 chars after the tag so we only check this element's
|
||||
# attributes — same shape as test_phase8_settings_modal_in_index_html.
|
||||
chunk = body[idx : idx + 600]
|
||||
assert 'role="region"' in chunk, (
|
||||
"dashboard must be role='region' — see showDashboard() comment."
|
||||
)
|
||||
assert "aria-modal" not in chunk, (
|
||||
"dashboard must NOT be aria-modal — re-trapping focus breaks "
|
||||
"the appbar's interactive controls (theme toggle, settings menu, "
|
||||
"proxy node-picker pill) while the dashboard is open."
|
||||
)
|
||||
|
||||
|
||||
def test_close_settings_menu_resets_aria() -> None:
|
||||
"""closeSettingsMenu must reset aria-expanded='false' AND remove
|
||||
aria-controls from the gear trigger. Without the reset the gear
|
||||
keeps reporting 'expanded' to assistive tech after the menu closes;
|
||||
without the removal aria-controls points at a dead DOM id."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function closeSettingsMenu(")
|
||||
# Bound to ~600 chars so we don't catch unrelated handlers.
|
||||
section = body[start : start + 600]
|
||||
assert 'setAttribute("aria-expanded", "false")' in section, (
|
||||
"closeSettingsMenu must set aria-expanded='false' on the gear."
|
||||
)
|
||||
assert 'removeAttribute("aria-controls")' in section, (
|
||||
"closeSettingsMenu must remove aria-controls from the gear."
|
||||
)
|
||||
|
||||
|
||||
def test_phase8_settings_modal_in_index_html() -> None:
|
||||
"""Both the settings overlay and the revoke-confirmation overlay
|
||||
must remain in the modal area. The Escape-key deferral list in
|
||||
|
||||
@@ -83,6 +83,39 @@ class TestIsPublicPath:
|
||||
def test_shared_static_public(self):
|
||||
assert is_public_path("/shared/base.css") is True
|
||||
|
||||
# Console proxy: a public proxied path must still be public, otherwise
|
||||
# the login modal can never re-authenticate from inside a ``/node/{id}/``
|
||||
# proxied page once the cookie expires.
|
||||
def test_proxy_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/login") is True
|
||||
|
||||
def test_proxy_no_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/api/auth/login") is True
|
||||
|
||||
def test_proxy_v1_status_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/status") is True
|
||||
|
||||
def test_proxy_v1_setup_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
|
||||
|
||||
def test_proxy_v1_logout_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
|
||||
|
||||
def test_proxy_v1_oidc_authorize_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
|
||||
|
||||
def test_proxy_v1_oidc_callback_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
|
||||
|
||||
def test_proxy_v1_workstreams_still_not_public(self):
|
||||
"""Proxy prefix must not turn protected paths into public ones."""
|
||||
assert is_public_path("/node/node-a/v1/api/workstreams") is False
|
||||
|
||||
def test_proxy_v1_refresh_still_requires_auth(self):
|
||||
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
|
||||
a valid cookie. Proxy-prefix shouldn't change that."""
|
||||
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestRequiredRole
|
||||
|
||||
@@ -141,3 +141,126 @@ class TestRemindersWidening:
|
||||
}
|
||||
history = _build([msg])
|
||||
assert history[0]["reminders"] == [{"type": "denial", "text": "ok"}]
|
||||
|
||||
|
||||
class _StubRegistry:
|
||||
"""Minimal model registry — only ``get_config`` is read by
|
||||
``_build_history``."""
|
||||
|
||||
def __init__(self, surface_persisted_reasoning: bool = True) -> None:
|
||||
self._cfg = SimpleNamespace(surface_persisted_reasoning=surface_persisted_reasoning)
|
||||
|
||||
def get_config(self, alias: str) -> Any:
|
||||
return self._cfg
|
||||
|
||||
|
||||
def _build_with_registry(
|
||||
messages: list[dict[str, Any]],
|
||||
surface_persisted_reasoning: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
session = SimpleNamespace(
|
||||
messages=messages,
|
||||
_ws_id="ws-test",
|
||||
_registry=_StubRegistry(surface_persisted_reasoning=surface_persisted_reasoning),
|
||||
_model_alias="claude-opus-4-7",
|
||||
)
|
||||
with patch(
|
||||
"turnstone.server._load_verdict_indexes",
|
||||
return_value=({}, {}),
|
||||
):
|
||||
return _build_history(session)
|
||||
|
||||
|
||||
class TestReasoningSurfacing:
|
||||
"""Phase 1 — surface stored Anthropic thinking blocks on the
|
||||
history payload so refresh-the-page rehydrates the reasoning bubble.
|
||||
Drives through the real ``AnthropicProvider`` extractor (no mock-of-
|
||||
extractor) — only the model registry is stubbed.
|
||||
"""
|
||||
|
||||
def test_reasoning_surfaces_for_anthropic_thinking_msg(self) -> None:
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "let me think", "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
}
|
||||
history = _build_with_registry([msg], surface_persisted_reasoning=True)
|
||||
assert len(history) == 1
|
||||
assert history[0]["reasoning"] == "let me think"
|
||||
|
||||
def test_reasoning_empty_when_persist_flag_false(self) -> None:
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "hidden", "signature": "s"},
|
||||
],
|
||||
}
|
||||
history = _build_with_registry([msg], surface_persisted_reasoning=False)
|
||||
assert "reasoning" not in history[0]
|
||||
|
||||
def test_provider_content_never_in_wire_entry(self) -> None:
|
||||
# The build path does not copy ``_provider_content`` into the
|
||||
# entry dict regardless of flag — wire payload stays tight.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "x", "signature": "s"},
|
||||
],
|
||||
}
|
||||
history = _build_with_registry([msg], surface_persisted_reasoning=True)
|
||||
assert "_provider_content" not in history[0]
|
||||
|
||||
def test_no_reasoning_field_when_provider_content_missing(self) -> None:
|
||||
msg = {"role": "assistant", "content": "plain answer"}
|
||||
history = _build_with_registry([msg], surface_persisted_reasoning=True)
|
||||
assert "reasoning" not in history[0]
|
||||
|
||||
def test_no_reasoning_field_for_non_assistant_messages(self) -> None:
|
||||
# Defensive — user/tool messages with a stray _provider_content
|
||||
# do not get the reasoning field stamped.
|
||||
msgs: list[dict[str, Any]] = [
|
||||
{"role": "user", "content": "hi"},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "c1",
|
||||
"content": "out",
|
||||
"_provider_content": [{"type": "thinking", "thinking": "leak", "signature": "s"}],
|
||||
},
|
||||
]
|
||||
history = _build_with_registry(msgs, surface_persisted_reasoning=True)
|
||||
assert "reasoning" not in history[0]
|
||||
assert "reasoning" not in history[1]
|
||||
|
||||
def test_default_true_when_registry_lookup_raises(self) -> None:
|
||||
# Conservative default — Phase 1 spec mandates rehydration on
|
||||
# refresh. A registry/alias mismatch must not silently kill the
|
||||
# bubble.
|
||||
class BrokenRegistry:
|
||||
def get_config(self, alias: str) -> Any:
|
||||
raise KeyError(alias)
|
||||
|
||||
session = SimpleNamespace(
|
||||
messages=[
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "still works", "signature": "s"}
|
||||
],
|
||||
}
|
||||
],
|
||||
_ws_id="ws-test",
|
||||
_registry=BrokenRegistry(),
|
||||
_model_alias="missing-alias",
|
||||
)
|
||||
with patch(
|
||||
"turnstone.server._load_verdict_indexes",
|
||||
return_value=({}, {}),
|
||||
):
|
||||
history = _build_history(session)
|
||||
assert history[0]["reasoning"] == "still works"
|
||||
|
||||
@@ -19,6 +19,12 @@ class NullUI:
|
||||
self.infos = []
|
||||
self.stream_ends = 0
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
@@ -820,3 +826,109 @@ class TestForceCancelThreaded:
|
||||
assert "idle" in ui.states
|
||||
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
|
||||
assert any("Fresh response" in m.get("content", "") for m in assistant_msgs)
|
||||
|
||||
|
||||
class TestSynthesizeCancelledResults:
|
||||
"""Regression coverage for ``_synthesize_cancelled_results`` — must
|
||||
fire ``on_tool_result`` for each synthesized cancellation so live
|
||||
SSE listeners (e.g. coord's ``--running`` indicator added by
|
||||
tool_info) can complete the in-DOM tool batch. Without this, the
|
||||
coord JS would spin the running indicator forever on cancelled
|
||||
batches because ``state_change`` doesn't strip ``--running`` from
|
||||
individual batches."""
|
||||
|
||||
def _ui_with_tool_result_tracking(self):
|
||||
class _TrackingUI(NullUI):
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.tool_results: list[tuple[str, str, str, bool]] = []
|
||||
|
||||
def on_tool_result(self, call_id, name, output, **kwargs):
|
||||
self.tool_results.append(
|
||||
(call_id, name, output, bool(kwargs.get("is_error", False))),
|
||||
)
|
||||
|
||||
return _TrackingUI()
|
||||
|
||||
def test_synthesizes_tool_result_for_unanswered_calls(self, tmp_db):
|
||||
ui = self._ui_with_tool_result_tracking()
|
||||
session = _make_session(ui=ui)
|
||||
session.messages.append(
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "calling tools",
|
||||
"tool_calls": [
|
||||
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
|
||||
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
|
||||
],
|
||||
},
|
||||
)
|
||||
session._msg_tokens.append(1)
|
||||
|
||||
session._synthesize_cancelled_results("Cancelled by user.")
|
||||
|
||||
# Both unanswered calls fired ``on_tool_result``.
|
||||
assert len(ui.tool_results) == 2
|
||||
ids = {tr[0] for tr in ui.tool_results}
|
||||
assert ids == {"call_a", "call_b"}
|
||||
# All emitted as errors so the live UI renders them as
|
||||
# ``coord-tool-row-result--error``.
|
||||
assert all(tr[3] is True for tr in ui.tool_results)
|
||||
# Reason text propagates as the synthetic tool output.
|
||||
assert all(tr[2] == "Cancelled by user." for tr in ui.tool_results)
|
||||
# And the message list has the synthesized tool entries
|
||||
# (preserves the prior contract).
|
||||
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
|
||||
assert len(tool_msgs) == 2
|
||||
|
||||
def test_skips_calls_already_answered(self, tmp_db):
|
||||
ui = self._ui_with_tool_result_tracking()
|
||||
session = _make_session(ui=ui)
|
||||
session.messages.append(
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
|
||||
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
|
||||
],
|
||||
},
|
||||
)
|
||||
session._msg_tokens.append(1)
|
||||
# call_a already answered.
|
||||
session.messages.append(
|
||||
{"role": "tool", "tool_call_id": "call_a", "content": "result"},
|
||||
)
|
||||
session._msg_tokens.append(1)
|
||||
|
||||
session._synthesize_cancelled_results("Cancelled by user.")
|
||||
|
||||
# Only call_b synthesized.
|
||||
assert len(ui.tool_results) == 1
|
||||
assert ui.tool_results[0][0] == "call_b"
|
||||
|
||||
def test_ui_emit_failure_does_not_break_synthesis(self, tmp_db):
|
||||
"""The UI hook is wrapped in try/except — a hook failure
|
||||
during cancel must NOT compound the problem. Synthesis still
|
||||
appends to messages + storage."""
|
||||
|
||||
class _ExplodingUI(NullUI):
|
||||
def on_tool_result(self, call_id, name, output, **kwargs):
|
||||
raise RuntimeError("ui hook blew up")
|
||||
|
||||
ui = _ExplodingUI()
|
||||
session = _make_session(ui=ui)
|
||||
session.messages.append(
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
|
||||
],
|
||||
},
|
||||
)
|
||||
session._msg_tokens.append(1)
|
||||
|
||||
# Must not raise.
|
||||
session._synthesize_cancelled_results("Cancelled by user.")
|
||||
|
||||
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
|
||||
assert len(tool_msgs) == 1
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""ChatSession interactivity flag tests (Phase 9).
|
||||
|
||||
Validates that ``ChatSession._is_interactive_for_consent`` is computed
|
||||
correctly from ``client_type`` on construction. This is the front of
|
||||
the Phase 9 plumb-through: the flag flows from here to
|
||||
``_dispatch_pool_sync`` to the structured-error → pending-consent
|
||||
write path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from tests._session_helpers import make_session
|
||||
from turnstone.prompts import INTERACTIVE_CONSENT_CLIENT_TYPES, ClientType
|
||||
|
||||
|
||||
def test_web_is_interactive() -> None:
|
||||
s = make_session(client_type=ClientType.WEB)
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_cli_is_interactive() -> None:
|
||||
s = make_session(client_type=ClientType.CLI)
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_chat_is_not_interactive() -> None:
|
||||
# Discord / Slack adapters cannot drive a browser redirect from
|
||||
# inside the channel — consent prompts must be deferred to the
|
||||
# dashboard badge.
|
||||
s = make_session(client_type=ClientType.CHAT)
|
||||
assert s._is_interactive_for_consent is False
|
||||
|
||||
|
||||
def test_scheduled_is_not_interactive() -> None:
|
||||
# The scheduler runs autonomously; the user isn't online to
|
||||
# complete the OAuth redirect.
|
||||
s = make_session(client_type=ClientType.SCHEDULED)
|
||||
assert s._is_interactive_for_consent is False
|
||||
|
||||
|
||||
def test_interactive_set_matches_module_constant() -> None:
|
||||
# Pin the module-level frozenset against the flag computation —
|
||||
# a future reorganisation that drifts the set vs the per-session
|
||||
# logic would silently break the gating.
|
||||
for ct in ClientType:
|
||||
s = make_session(client_type=ct)
|
||||
assert s._is_interactive_for_consent == (ct in INTERACTIVE_CONSENT_CLIENT_TYPES), ct
|
||||
|
||||
|
||||
def test_default_client_type_is_cli_interactive() -> None:
|
||||
# Defaults preserved — make_session uses ChatSession's default
|
||||
# which is CLI. Sanity check that the default user experience
|
||||
# stays interactive-for-consent.
|
||||
s = make_session()
|
||||
assert s._client_type == ClientType.CLI
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_scheduled_env_file_exists() -> None:
|
||||
"""The SCHEDULED env module must exist; otherwise
|
||||
``compose_system_message`` for a scheduled session would 500."""
|
||||
from turnstone.prompts import _load
|
||||
|
||||
text = _load("env/scheduled.md")
|
||||
assert "Output Environment" in text
|
||||
assert "consent" in text.lower()
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
|
||||
|
||||
The bus is the in-process wakeup primitive for ``wait_for_workstream``
|
||||
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
|
||||
of ws_id → set[threading.Event] under a lock — focused tests for
|
||||
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
|
||||
multi-child waiter, and concurrent register/notify (smoke). End-to-end
|
||||
integration with the dispatch sink lives in
|
||||
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
|
||||
def test_register_returns_event_that_starts_unset() -> None:
|
||||
"""A waiter must not see leftover state from before it registered —
|
||||
a fresh wait should always block until the first notify."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert isinstance(event, threading.Event)
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
|
||||
"""Different ws_ids must keep independent waiter sets — a notify on
|
||||
a stranger ws can't wake the wait or the bus stops being keyed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-other")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_with_no_subscribers_is_noop() -> None:
|
||||
"""The dispatch sink calls notify on every translated event; the
|
||||
steady state has no wait tool active. Must not raise."""
|
||||
bus = ChildEventBus()
|
||||
bus.notify("ws-nobody-cares") # no exception
|
||||
|
||||
|
||||
def test_multi_waiter_each_gets_independent_event() -> None:
|
||||
"""Two waits on the same ws_id must wake independently — clearing
|
||||
one Event must not silence the other."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
assert e1 is not e2
|
||||
bus.notify("ws-1")
|
||||
assert e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
|
||||
"""A wait on [A, B, C] returns a single Event registered against
|
||||
all three. Notify on ANY of A/B/C must wake the wait — the
|
||||
caller's snapshot re-read disambiguates which one changed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
|
||||
bus.notify("ws-b")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
|
||||
"""After unregister, notify on any of the previously-watched ws_ids
|
||||
must NOT wake the Event — leaks would mean every future notify on
|
||||
that ws_id wakes a long-dead wait."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b"])
|
||||
bus.unregister_waiter(["ws-a", "ws-b"], event)
|
||||
bus.notify("ws-a")
|
||||
bus.notify("ws-b")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_unregister_is_idempotent() -> None:
|
||||
"""A double-unregister must silently no-op — finally blocks may
|
||||
run twice in odd shutdown paths, the bus must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
bus.unregister_waiter(["ws-1"], event) # no exception
|
||||
|
||||
|
||||
def test_unregister_pops_empty_buckets() -> None:
|
||||
"""Empty per-ws_id buckets must be popped so a long-lived bus
|
||||
doesn't accumulate dead keys after many waits have churned through.
|
||||
Reaches into the private state — the property is structural, not
|
||||
behavioral, so the assertion is also."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert "ws-1" in bus._waiters
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
assert "ws-1" not in bus._waiters
|
||||
|
||||
|
||||
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
|
||||
"""Removing one waiter from a multi-waiter bucket must not drop
|
||||
the others — popping the bucket would silently disable notifies
|
||||
for every concurrent wait on the same ws_id."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], e1)
|
||||
bus.notify("ws-1")
|
||||
assert not e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
|
||||
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
|
||||
is reachable from other callers in future use; falsy ids should be
|
||||
silently dropped, not registered against an empty-string key."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["", "ws-1", ""])
|
||||
# Only the real ws_id should bucket the waiter.
|
||||
assert list(bus._waiters.keys()) == ["ws-1"]
|
||||
bus.notify("") # no crash, no spurious wake
|
||||
assert not event.is_set()
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
|
||||
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
|
||||
must return promptly after a notify on a watched ws_id. This is
|
||||
the property that retires the 0.5s polling cadence."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
woken_at = [0.0]
|
||||
|
||||
def _waiter() -> None:
|
||||
event.wait(timeout=2.0)
|
||||
woken_at[0] = time.monotonic()
|
||||
|
||||
t = threading.Thread(target=_waiter, daemon=True)
|
||||
t.start()
|
||||
# Give the waiter a beat to enter Event.wait, then notify.
|
||||
time.sleep(0.05)
|
||||
notified_at = time.monotonic()
|
||||
bus.notify("ws-1")
|
||||
t.join(timeout=1.0)
|
||||
assert not t.is_alive(), "waiter did not wake within 1s of notify"
|
||||
# Latency budget is generous; the contract is "well under the legacy
|
||||
# 0.5s poll cadence", not microsecond timing.
|
||||
assert woken_at[0] - notified_at < 0.2
|
||||
|
||||
|
||||
def test_clear_before_check_race_does_not_lose_wake() -> None:
|
||||
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
|
||||
A notify between clear and wait must leave the Event set, so the
|
||||
next wait returns immediately and the loop re-snapshots. Same
|
||||
standard subscribe/check race the wait loop guards against."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
# Simulate wait-loop ordering: clear, then notify "between" clear
|
||||
# and the next wait.
|
||||
event.clear()
|
||||
bus.notify("ws-1")
|
||||
# The next wait must return True immediately (set is sticky until
|
||||
# the next clear).
|
||||
assert event.wait(timeout=0.1) is True
|
||||
|
||||
|
||||
def test_concurrent_register_and_notify_is_safe() -> None:
|
||||
"""Smoke test: many threads registering / notifying / unregistering
|
||||
in parallel must not raise or deadlock. Doesn't assert specific
|
||||
interleavings — only structural safety of the lock discipline."""
|
||||
bus = ChildEventBus()
|
||||
stop = threading.Event()
|
||||
errors: list[BaseException] = []
|
||||
|
||||
def _worker(ws_id: str) -> None:
|
||||
try:
|
||||
for _ in range(200):
|
||||
if stop.is_set():
|
||||
return
|
||||
ev = bus.register_waiter([ws_id])
|
||||
bus.notify(ws_id)
|
||||
bus.unregister_waiter([ws_id], ev)
|
||||
except BaseException as e: # noqa: BLE001
|
||||
errors.append(e)
|
||||
|
||||
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
stop.set()
|
||||
assert not errors, f"worker threads raised: {errors!r}"
|
||||
# All buckets should have been popped (every register paired with
|
||||
# unregister).
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ws_id", ["", None])
|
||||
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
|
||||
"""Defensive: the dispatch sink already guards against empty
|
||||
ws_ids, but a falsy slip-through must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify(ws_id) # type: ignore[arg-type]
|
||||
assert not event.is_set()
|
||||
@@ -50,6 +50,41 @@ def test_load_config_invalid_toml(tmp_path):
|
||||
assert load_config() == {}
|
||||
|
||||
|
||||
def test_load_config_warns_when_world_readable(tmp_path, caplog):
|
||||
"""Secrets in config.toml — warn if anyone but the owner can read it."""
|
||||
import logging
|
||||
import os
|
||||
|
||||
_reset_cache()
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
|
||||
os.chmod(cfg, 0o644)
|
||||
set_config_path(str(cfg))
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
|
||||
load_config()
|
||||
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any("group/world-readable" in m for m in messages)
|
||||
|
||||
|
||||
def test_load_config_quiet_when_mode_0600(tmp_path, caplog):
|
||||
import logging
|
||||
import os
|
||||
|
||||
_reset_cache()
|
||||
cfg = tmp_path / "config.toml"
|
||||
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
|
||||
os.chmod(cfg, 0o600)
|
||||
set_config_path(str(cfg))
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
|
||||
load_config()
|
||||
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert not any("group/world-readable" in m for m in messages)
|
||||
|
||||
|
||||
def test_load_config_caches(tmp_path):
|
||||
_reset_cache()
|
||||
cfg = tmp_path / "config.toml"
|
||||
|
||||
@@ -3,11 +3,13 @@
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.collector import ClusterCollector, NodeSnapshot
|
||||
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
|
||||
|
||||
# Shared test auth — JWT-based
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
@@ -149,6 +151,78 @@ class TestCollectorDiscovery:
|
||||
assert c._nodes["node-a"].started == 1234567890.0
|
||||
|
||||
|
||||
class TestCollectorNotifyWireIn:
|
||||
"""NotifyDispatcher-driven discovery — reactive node visibility."""
|
||||
|
||||
def test_start_subscribes_to_services_channel(self):
|
||||
# Stub dispatcher records subscriptions without spawning threads.
|
||||
class _StubDispatcher:
|
||||
def __init__(self):
|
||||
self.subscriptions: list[tuple[str, Any]] = []
|
||||
|
||||
def subscribe(self, channel, handler):
|
||||
self.subscriptions.append((channel, handler))
|
||||
return lambda: None
|
||||
|
||||
stub = _StubDispatcher()
|
||||
storage = MockStorage()
|
||||
c = ClusterCollector(
|
||||
storage=storage,
|
||||
discovery_interval=999,
|
||||
notify_dispatcher=stub,
|
||||
)
|
||||
try:
|
||||
c.start()
|
||||
assert len(stub.subscriptions) == 1
|
||||
channel, handler = stub.subscriptions[0]
|
||||
assert channel == "services"
|
||||
assert handler == c._on_services_notify
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_no_dispatcher_means_no_subscribe(self):
|
||||
# Collector without a dispatcher (single-node / SQLite dev) just
|
||||
# falls back to the 60 s discovery-loop polling — no error.
|
||||
c = _make_collector(MockStorage())
|
||||
try:
|
||||
c.start()
|
||||
assert c._notify_unsubscribe is None
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_on_notify_runs_discovery(self):
|
||||
# Construct a synthetic Notify and invoke the handler directly —
|
||||
# asserts the wire-in delegates back to ``_discover_nodes``.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
c = _make_collector(storage)
|
||||
c._running = True # bypass start() so we don't spawn threads
|
||||
q: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
storage.services = [
|
||||
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
|
||||
]
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
|
||||
event = q.get_nowait()
|
||||
assert event["type"] == "node_joined"
|
||||
assert event["node_id"] == "node-z"
|
||||
|
||||
def test_on_notify_when_not_running_is_noop(self):
|
||||
# If a stray notify arrives after stop, the handler doesn't run
|
||||
# discovery on a half-torn-down collector.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
|
||||
c = _make_collector(storage)
|
||||
# _running stays False (never called start()).
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
assert c.get_overview()["nodes"] == 0
|
||||
|
||||
|
||||
class TestCollectorSnapshot:
|
||||
"""Applying node_snapshot SSE events."""
|
||||
|
||||
@@ -1489,6 +1563,142 @@ class TestConsoleProxy:
|
||||
assert sse_mock.await_count == 1
|
||||
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Proxied auth endpoints — handled locally by the console, not
|
||||
# forwarded to the upstream node. Cases derive directly from
|
||||
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
|
||||
# added without a matching test (or vice versa). See proxy_api's
|
||||
# docstring for the JWT-audience reasoning.
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "handler_name"),
|
||||
[
|
||||
(method, path, handler_name)
|
||||
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
|
||||
],
|
||||
)
|
||||
def test_proxy_auth_endpoint_dispatches_to_local_handler(
|
||||
self, client, method, path, handler_name
|
||||
):
|
||||
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
|
||||
local console handler and never reach the upstream proxy. The
|
||||
lockout class of bug this dispatch was added to fix is exactly
|
||||
what a regression here would reintroduce silently — covering all
|
||||
eight branches keeps each path tied to its handler."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
f"turnstone.console.server.{handler_name}",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
resp = client.request(method, f"/node/node-a/v1/api/{path}")
|
||||
assert resp.status_code == 200
|
||||
assert local_mock.await_count == 1
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
|
||||
"""Without this fix the AuthMiddleware 401s before any handler
|
||||
runs — the user is locked out of the proxied UI once the cookie
|
||||
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import _load_static, create_app
|
||||
|
||||
_load_static()
|
||||
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
|
||||
unauth_client = TestClient(app, raise_server_exceptions=False)
|
||||
try:
|
||||
with patch(
|
||||
"turnstone.console.server.auth_login",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock:
|
||||
resp = unauth_client.post(
|
||||
"/node/node-a/v1/api/auth/login",
|
||||
json={"username": "x", "password": "y"},
|
||||
)
|
||||
# AuthMiddleware must classify the proxied login path as
|
||||
# public (is_public_path change) AND proxy_api must
|
||||
# dispatch to the local handler (proxy_api change).
|
||||
assert resp.status_code == 200, (
|
||||
f"login locked out: got {resp.status_code}, body={resp.text}"
|
||||
)
|
||||
assert local_mock.await_count == 1
|
||||
finally:
|
||||
unauth_client.close()
|
||||
|
||||
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
|
||||
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
|
||||
must short-circuit with 405 instead of falling through to the
|
||||
upstream proxy — falling through would forward the request
|
||||
authenticated as the console's service token (``_proxy_auth_headers``
|
||||
fallback)."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
# PUT on a POST-only auth path → 405
|
||||
put_resp = client.put("/node/node-a/v1/api/auth/login")
|
||||
assert put_resp.status_code == 405
|
||||
# POST on a GET-only auth path → 405
|
||||
post_resp = client.post("/node/node-a/v1/api/auth/status")
|
||||
assert post_resp.status_code == 405
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
|
||||
"""Sanity: only auth/* paths intercept. Other API paths still
|
||||
forward to the upstream node."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
mock_collector.get_node_detail.return_value = {
|
||||
"node_id": "node-a",
|
||||
"server_url": "http://a:8080",
|
||||
"reachable": True,
|
||||
}
|
||||
with patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"ok": True}),
|
||||
) as proxy_mock:
|
||||
resp = client.get("/node/node-a/v1/api/workstreams")
|
||||
assert resp.status_code == 200
|
||||
assert proxy_mock.await_count == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Proxy URL rewriting unit tests (no HTTP needed)
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
"""``GET /v1/api/models`` resolution-chain coverage.
|
||||
|
||||
The console handler resolves four defaults from settings + the enabled
|
||||
model list:
|
||||
|
||||
* ``default_alias`` ← ``model.default_alias``
|
||||
* ``channel_default_alias`` ← ``channels.default_model_alias``
|
||||
* ``coordinator_default_alias`` ← ``coordinator.model_alias``, falling
|
||||
back to ``default_alias`` when empty *or* pointing at a disabled /
|
||||
removed alias (mirrors :mod:`turnstone.console.session_factory`).
|
||||
* ``judge_default_alias`` ← ``judge.model``, falling back to the
|
||||
resolved coordinator alias when empty *or* pointing at a value that
|
||||
isn't an enabled alias. ``judge.model`` is alias-only — same
|
||||
contract as the other model roles — and
|
||||
:class:`turnstone.core.judge.IntentJudge` silently inherits the
|
||||
session model when an unknown value is configured, so the API
|
||||
surfaces the resolved coordinator alias rather than echoing the
|
||||
misconfigured string.
|
||||
|
||||
These tests pin each branch so the home composer's resolved-alias
|
||||
placeholder stays correct as the precedence rules evolve.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.routing import Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from tests._coord_test_helpers import _AuthMiddleware, _FakeConfigStore
|
||||
from turnstone.console.server import list_available_models
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path: Any) -> SQLiteBackend:
|
||||
return SQLiteBackend(str(tmp_path / "available_models.db"))
|
||||
|
||||
|
||||
def _seed_model(
|
||||
storage: SQLiteBackend,
|
||||
*,
|
||||
definition_id: str,
|
||||
alias: str,
|
||||
model: str = "model-x",
|
||||
enabled: bool = True,
|
||||
) -> None:
|
||||
storage.create_model_definition(
|
||||
definition_id=definition_id,
|
||||
alias=alias,
|
||||
model=model,
|
||||
provider="openai-compatible",
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="sk-test",
|
||||
context_window=8192,
|
||||
capabilities="{}",
|
||||
enabled=enabled,
|
||||
created_by="admin",
|
||||
)
|
||||
|
||||
|
||||
class _StubRegistry:
|
||||
"""Mimics the surface ``resolve_coordinator_alias`` reads from
|
||||
``coord_registry``: ``.default`` and ``.has_alias()``.
|
||||
|
||||
Production wires this through ``ModelRegistry``, which in turn
|
||||
pulls aliases from both DB rows and config.toml. The fixture
|
||||
mirrors the storage's enabled-row set so ``has_alias()`` agrees
|
||||
with what the placeholder's enabled-row filter would accept —
|
||||
without that alignment the helper rejects every tier-2 candidate
|
||||
and the placeholder goes blank in cases that production handles
|
||||
fine."""
|
||||
|
||||
def __init__(self, *, default: str, known: set[str]) -> None:
|
||||
self.default = default
|
||||
self._known = known
|
||||
|
||||
def has_alias(self, alias: str) -> bool:
|
||||
return alias in self._known
|
||||
|
||||
|
||||
def _make_client(
|
||||
storage: SQLiteBackend,
|
||||
*,
|
||||
settings: dict[str, str] | None = None,
|
||||
registry_default: str = "",
|
||||
config_store: bool = True,
|
||||
) -> TestClient:
|
||||
app = Starlette(
|
||||
routes=[Route("/v1/api/models", list_available_models)],
|
||||
middleware=[Middleware(_AuthMiddleware)],
|
||||
)
|
||||
app.state.auth_storage = storage
|
||||
if config_store:
|
||||
app.state.config_store = _FakeConfigStore(dict(settings or {}))
|
||||
# ``coord_registry`` is always set in production after lifespan
|
||||
# startup; mirror that here. ``has_alias`` answers from the same
|
||||
# enabled-rows set the handler filters against.
|
||||
enabled = {r["alias"] for r in storage.list_model_definitions(enabled_only=True)}
|
||||
app.state.coord_registry = _StubRegistry(default=registry_default, known=enabled)
|
||||
client = TestClient(app)
|
||||
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": ""})
|
||||
return client
|
||||
|
||||
|
||||
def _get_models(client: TestClient) -> dict[str, Any]:
|
||||
resp = client.get("/v1/api/models")
|
||||
assert resp.status_code == 200, resp.text
|
||||
return resp.json()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Coordinator resolution
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_settings_leaves_all_defaults_blank(storage: SQLiteBackend) -> None:
|
||||
"""No model.default_alias, no per-role overrides → every default
|
||||
field is empty and ``models`` is an empty list."""
|
||||
body = _get_models(_make_client(storage))
|
||||
assert body == {
|
||||
"models": [],
|
||||
"default_alias": "",
|
||||
"channel_default_alias": "",
|
||||
"coordinator_default_alias": "",
|
||||
"judge_default_alias": "",
|
||||
}
|
||||
|
||||
|
||||
def test_coordinator_inherits_default_alias_when_unset(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(_make_client(storage, settings={"model.default_alias": "primary"}))
|
||||
assert body["default_alias"] == "primary"
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
|
||||
|
||||
def test_coordinator_explicit_enabled_alias_passes_through(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
_seed_model(storage, definition_id="m2", alias="fast")
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"coordinator.model_alias": "fast",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["coordinator_default_alias"] == "fast"
|
||||
|
||||
|
||||
def test_coordinator_set_to_disabled_alias_falls_back_to_default(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""Operator disabled the alias the coordinator was pinned to —
|
||||
fall back to the registry default rather than advertising a model
|
||||
that workstream creation would refuse to use."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
_seed_model(storage, definition_id="m2", alias="legacy", enabled=False)
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"coordinator.model_alias": "legacy",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
|
||||
|
||||
def test_coordinator_set_to_unknown_alias_falls_back_to_default(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"coordinator.model_alias": "ghost",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
|
||||
|
||||
def test_coordinator_falls_back_to_registry_default_when_config_store_empty(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""Match ``console/session_factory.py:109-110``: when both
|
||||
``coordinator.model_alias`` and ``model.default_alias`` are unset, new
|
||||
coordinator sessions run on ``registry.default`` (loaded from
|
||||
config.toml ``[model].default``). The placeholder must report the
|
||||
same alias rather than going blank — otherwise the home composer
|
||||
advertises "Default model" while sessions actually launch on a
|
||||
concrete alias."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(_make_client(storage, registry_default="primary"))
|
||||
assert body["default_alias"] == ""
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
assert body["judge_default_alias"] == "primary"
|
||||
|
||||
|
||||
def test_coordinator_skips_registry_default_when_alias_disabled(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""Registry default points at an alias that's been disabled in the DB
|
||||
— the placeholder stays blank rather than advertising a model that
|
||||
workstream creation would refuse to use."""
|
||||
_seed_model(storage, definition_id="m1", alias="legacy", enabled=False)
|
||||
body = _get_models(_make_client(storage, registry_default="legacy"))
|
||||
assert body["coordinator_default_alias"] == ""
|
||||
|
||||
|
||||
def test_coordinator_falls_back_to_registry_default_when_config_store_missing(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""Edge case from PR #500 review: lifespan can leave
|
||||
``app.state.config_store`` as None (e.g. a startup exception) while
|
||||
``coord_registry`` still binds successfully. The placeholder must
|
||||
still advertise ``registry.default`` (filtered against enabled rows)
|
||||
rather than going blank — otherwise the home composer is uselessly
|
||||
empty in a degraded-but-recoverable state."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(_make_client(storage, registry_default="primary", config_store=False))
|
||||
assert body["default_alias"] == ""
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
assert body["judge_default_alias"] == "primary"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Judge resolution
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_judge_empty_inherits_resolved_coordinator_alias(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
_seed_model(storage, definition_id="m2", alias="fast")
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"coordinator.model_alias": "fast",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["coordinator_default_alias"] == "fast"
|
||||
assert body["judge_default_alias"] == "fast"
|
||||
|
||||
|
||||
def test_judge_explicit_enabled_alias_passes_through(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
_seed_model(storage, definition_id="m2", alias="judge-fast")
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"judge.model": "judge-fast",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["judge_default_alias"] == "judge-fast"
|
||||
|
||||
|
||||
def test_judge_set_to_unknown_value_inherits_coordinator(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""``judge.model`` is alias-only — same contract as the other model
|
||||
roles. An unknown value silently inherits the session model in
|
||||
:class:`IntentJudge`, so the API surfaces the resolved coordinator
|
||||
alias rather than echoing the misconfigured string."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"judge.model": "anthropic/claude-haiku-4-5", # raw, not an alias
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["coordinator_default_alias"] == "primary"
|
||||
assert body["judge_default_alias"] == "primary"
|
||||
|
||||
|
||||
def test_judge_set_to_disabled_alias_inherits_coordinator(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
"""Disabled-alias case is handled identically to the unknown-value
|
||||
case — both trip the alias-not-resolved path."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
_seed_model(storage, definition_id="m2", alias="judge-old", enabled=False)
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={
|
||||
"model.default_alias": "primary",
|
||||
"judge.model": "judge-old",
|
||||
},
|
||||
)
|
||||
)
|
||||
assert body["judge_default_alias"] == "primary"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-existing fields stay correct under the new resolution code
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_channel_default_alias_blanked_when_disabled(
|
||||
storage: SQLiteBackend,
|
||||
) -> None:
|
||||
_seed_model(storage, definition_id="m1", alias="primary", enabled=False)
|
||||
body = _get_models(
|
||||
_make_client(
|
||||
storage,
|
||||
settings={"channels.default_model_alias": "primary"},
|
||||
)
|
||||
)
|
||||
assert body["channel_default_alias"] == ""
|
||||
|
||||
|
||||
def test_models_payload_strips_secret_fields(storage: SQLiteBackend) -> None:
|
||||
"""Regression guard: only alias/model/provider land in the response,
|
||||
never api_key / base_url / context_window / capabilities."""
|
||||
_seed_model(storage, definition_id="m1", alias="primary")
|
||||
body = _get_models(_make_client(storage))
|
||||
assert body["models"] == [
|
||||
{"alias": "primary", "model": "model-x", "provider": "openai-compatible"}
|
||||
]
|
||||
@@ -5,11 +5,15 @@ lifting is in ``SessionManager.close_idle`` (covered in
|
||||
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
|
||||
in ``test_storage_sqlite.py``). These tests verify the glue:
|
||||
|
||||
- the helper runs an initial sweep BEFORE its first sleep (cold-start
|
||||
- the helper runs an initial sweep BEFORE its first wait (cold-start
|
||||
cleanup without blocking the lifespan),
|
||||
- the helper swallows exceptions so a transient DB blip can't kill the
|
||||
daemon thread,
|
||||
- the helper exits cleanly when ``stop_event`` is set.
|
||||
- the helper exits cleanly when ``stop_event`` is set,
|
||||
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
|
||||
event wakes the next sweep early (event-driven, not polling),
|
||||
- the helper unsubscribes when the thread exits so the subscriber
|
||||
doesn't leak past one cleanup-thread lifetime.
|
||||
|
||||
The ``stop_event`` parameter is exclusively for tests — production
|
||||
callers pass ``None`` and the daemon runs for process lifetime.
|
||||
@@ -17,28 +21,36 @@ callers pass ``None`` and the daemon runs for process lifetime.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import threading
|
||||
from unittest.mock import patch
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.console.server import _coord_idle_cleanup_thread
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
|
||||
class _StubMgr:
|
||||
"""Minimal SessionManager substitute exposing only what the cleanup
|
||||
thread touches: ``close_idle``, ``subscribe_to_state``,
|
||||
``unsubscribe_from_state``. Records call ordering for assertions
|
||||
and lets the test fire state-change events manually via
|
||||
:meth:`fire_state_change`.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
|
||||
) -> None:
|
||||
self.calls: list[float] = []
|
||||
self.sleep_calls_at_each_close: list[int] = []
|
||||
self._stop_event = stop_event
|
||||
self._expected = expected_calls
|
||||
self._raise_after = raise_after
|
||||
self._sleep_count = 0
|
||||
self._subscribers: list[Callable[[str, object], None]] = []
|
||||
self._sub_lock = threading.Lock()
|
||||
|
||||
def close_idle(self, timeout_sec: float) -> list[str]:
|
||||
# Snapshot how many sleeps preceded this close — lets the
|
||||
# "initial sweep" test verify the first close_idle ran with
|
||||
# zero preceding sleeps.
|
||||
self.sleep_calls_at_each_close.append(self._sleep_count)
|
||||
self.calls.append(timeout_sec)
|
||||
try:
|
||||
if 0 <= self._raise_after < len(self.calls):
|
||||
@@ -50,39 +62,78 @@ class _StubMgr:
|
||||
self._stop_event.set()
|
||||
return []
|
||||
|
||||
def record_sleep(self, _seconds: float) -> None:
|
||||
self._sleep_count += 1
|
||||
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock:
|
||||
self._subscribers.append(callback)
|
||||
|
||||
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock, contextlib.suppress(ValueError):
|
||||
self._subscribers.remove(callback)
|
||||
|
||||
@property
|
||||
def subscribers_count(self) -> int:
|
||||
with self._sub_lock:
|
||||
return len(self._subscribers)
|
||||
|
||||
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
|
||||
with self._sub_lock:
|
||||
snapshot = list(self._subscribers)
|
||||
for cb in snapshot:
|
||||
cb(ws_id, state)
|
||||
|
||||
|
||||
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
|
||||
with patch("turnstone.console.server.time.sleep", mgr.record_sleep):
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
# ``min_sweep_interval=0.0`` disables the production cadence floor
|
||||
# (default 5 s) so tests can fire many close_idle calls back-to-back
|
||||
# without waiting real time between them. The floor is exercised
|
||||
# in its own dedicated test below.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_sleep() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first time.sleep —
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first wait —
|
||||
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
|
||||
on default 2h timeout) for the first reap. Crucial because the
|
||||
lifespan no longer does a synchronous initial sweep."""
|
||||
lifespan no longer does a synchronous initial sweep.
|
||||
|
||||
Verified structurally: a single ``expected_calls=1`` run completes
|
||||
in well under one ``check_every`` (here 0.04 s timeout → 0.01 s
|
||||
check_every), so the initial sweep must have happened before any
|
||||
real wait could have blocked it.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
assert mgr.sleep_calls_at_each_close == [0], "first close_idle should run before any sleep"
|
||||
started = time.monotonic()
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
elapsed = time.monotonic() - started
|
||||
assert len(mgr.calls) == 1
|
||||
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
|
||||
# gated behind one full wait would have taken ~0.01+ s anyway, so
|
||||
# the upper bound here is "much less than one check_every plus
|
||||
# process noise" — the explicit 1.0 s gives generous CI headroom
|
||||
# while still asserting the test is testing the right thing.
|
||||
assert elapsed < 1.0
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
|
||||
"""Heartbeat path: with no state-change events, close_idle fires
|
||||
each ``check_every`` interval. Test uses a tiny timeout so the
|
||||
test runs fast — the contract under test is "the loop iterates",
|
||||
not the production cadence.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert len(mgr.calls) == 3
|
||||
assert all(t == 120.0 for t in mgr.calls)
|
||||
assert all(t == 0.04 for t in mgr.calls)
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
@@ -91,7 +142,7 @@ def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
blip would silently leak orphans forever."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
# All four calls must have fired despite calls 2-4 raising.
|
||||
assert len(mgr.calls) == 4
|
||||
|
||||
@@ -102,5 +153,154 @@ def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
|
||||
daemon-process termination."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert stop_event.is_set()
|
||||
|
||||
|
||||
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
|
||||
"""The event-driven path is the whole point of the refactor: a
|
||||
workstream state-change must wake the cleanup sweep without
|
||||
waiting one ``check_every`` interval. Tested with a long
|
||||
timeout_sec so the heartbeat would NOT have fired in the test
|
||||
window — the close_idle call past the initial sweep must come
|
||||
from a state-change wake.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
|
||||
# window. Any close_idle call past the initial sweep must come
|
||||
# from a fire_state_change-driven wake-up.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
# Wait for the initial sweep to complete AND the thread to enter
|
||||
# its first ``tick_now.wait`` (signalled here by the subscriber
|
||||
# being registered + calls advancing to 1).
|
||||
deadline = time.monotonic() + 1.0
|
||||
while time.monotonic() < deadline:
|
||||
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
|
||||
assert len(mgr.calls) == 1, "initial sweep didn't fire"
|
||||
# One state-change fire wakes the first ``wait`` → close_idle runs
|
||||
# again → stop_event is set (expected_calls=2) → thread exits.
|
||||
mgr.fire_state_change()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
|
||||
# 2 = initial + state-change-driven. If the state change weren't
|
||||
# being honoured, close_idle would have stalled on the 30 s wait
|
||||
# and the thread.join would have timed out.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_subscriber_unregisters_when_thread_exits() -> None:
|
||||
"""The cleanup thread's state-change subscriber must be removed
|
||||
when the thread exits — otherwise long-running processes that
|
||||
restart their cleanup threads (admin model-CRUD path, tests) leak
|
||||
subscribers and every state change fires N stale callbacks.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
|
||||
|
||||
|
||||
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
|
||||
"""A state-change fired during the initial sweep (e.g. close_idle's
|
||||
own ``close()`` calls firing subscribers) must wake the next
|
||||
``tick_now.wait`` rather than being lost to the clear-before-sweep
|
||||
ordering. The clear runs INSIDE the loop just before close_idle,
|
||||
so a fire during the initial sweep — which precedes the loop —
|
||||
arrives at an already-set event that the first wait sees set and
|
||||
returns on immediately.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
|
||||
# One-shot fire during the initial sweep, mirroring what
|
||||
# close_idle's own close() calls do in production (set_state →
|
||||
# state-change subscribers).
|
||||
fired = [False]
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
result = real_close_idle(timeout_sec)
|
||||
if not fired[0]:
|
||||
fired[0] = True
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
|
||||
# 2 = initial sweep + state-change-driven follow-up. Without the
|
||||
# event surviving the clear-before-sweep ordering, the thread
|
||||
# would have blocked on the 30 s ``wait`` and the test would have
|
||||
# timed out at thread.join.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
|
||||
"""Cadence floor: even when state-change events keep firing
|
||||
``tick_now.set()``, ``close_idle`` must not run more often than
|
||||
``min_sweep_interval`` — otherwise the loop tight-spins close_idle
|
||||
at the rate of its own DB latency, doing 600-1500x more DB work
|
||||
than the pre-refactor fixed-30 s cadence.
|
||||
|
||||
Wires a state-change subscriber that fires another state change
|
||||
from inside close_idle, so the bus would tick forever if not
|
||||
floored. Asserts the elapsed-between-sweeps is at least
|
||||
``min_sweep_interval`` modulo small wall-clock noise.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
sweep_times: list[float] = []
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
sweep_times.append(time.monotonic())
|
||||
result = real_close_idle(timeout_sec)
|
||||
# Always fire another state-change to simulate sustained
|
||||
# activity (each turn fires thinking/running/attention/idle).
|
||||
# If the floor were absent, the next wake would race the next
|
||||
# close_idle immediately and ``sweep_times`` deltas would be
|
||||
# bounded by close_idle latency (microseconds), not the floor.
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
# 0.15 s floor keeps the test fast (~0.3 s total) while still
|
||||
# representing a meaningful gap relative to close_idle's
|
||||
# near-zero stub latency.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.15},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=3.0)
|
||||
assert not thread.is_alive(), "thread didn't exit"
|
||||
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
|
||||
# Gap between sweep 1 (post-initial) and sweep 2 must respect
|
||||
# the floor. Initial sweep at sweep_times[0] is unfloored
|
||||
# (no prior sweep to compare against), so the meaningful
|
||||
# assertion is on sweep_times[1] - sweep_times[0].
|
||||
gap = sweep_times[1] - sweep_times[0]
|
||||
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
"""``console/session_factory.py`` alias-resolution coverage.
|
||||
|
||||
The console session factory resolves the coordinator alias through a
|
||||
three-tier chain that must stay in lockstep with the placeholder logic
|
||||
in ``console/server.py:list_available_models`` — otherwise the home
|
||||
composer advertises one alias while sessions launch on another.
|
||||
|
||||
Tier order (highest priority first):
|
||||
|
||||
1. Per-call ``model_alias`` arg, or the ``coordinator.model_alias``
|
||||
ConfigStore setting (admin-pinned coordinator-specific override).
|
||||
2. ``model.default_alias`` ConfigStore setting (admin-managed system
|
||||
default surfaced in the Models tab).
|
||||
3. ``registry.default`` (config.toml ``[model].default``, the boot-time
|
||||
fallback).
|
||||
|
||||
These tests pin each branch by intercepting ``registry.resolve`` —
|
||||
they short-circuit before ChatSession construction so the test never
|
||||
has to satisfy ChatSession's full kwarg contract.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._coord_test_helpers import _FakeConfigStore
|
||||
from turnstone.console.session_factory import build_console_session_factory
|
||||
|
||||
|
||||
class _StopBeforeChatSessionError(Exception):
|
||||
"""Sentinel raised by the capturing registry to short-circuit
|
||||
factory execution after alias resolution but before ChatSession is
|
||||
built. The factory's outer code path is irrelevant to alias
|
||||
resolution and would force the test to satisfy a long kwarg
|
||||
contract for no extra coverage."""
|
||||
|
||||
|
||||
class _CapturingRegistry:
|
||||
"""Records the alias passed to ``resolve()`` and short-circuits.
|
||||
|
||||
``has_alias`` answers from the configured known set so the
|
||||
``model.default_alias`` validation tier behaves realistically.
|
||||
Mirrors the public surface ``ModelRegistry`` exposes to
|
||||
session_factory: ``has_alias``, ``resolve``, and ``default``.
|
||||
"""
|
||||
|
||||
def __init__(self, *, default: str, known: set[str]) -> None:
|
||||
self.default = default
|
||||
self._known = known
|
||||
self.captured_alias: str | None = None
|
||||
|
||||
def has_alias(self, alias: str) -> bool:
|
||||
return alias in self._known
|
||||
|
||||
def resolve(self, alias: str) -> Any:
|
||||
self.captured_alias = alias
|
||||
raise _StopBeforeChatSessionError()
|
||||
|
||||
|
||||
def _build_factory(
|
||||
*,
|
||||
registry_default: str = "registry-default",
|
||||
known_aliases: set[str] | None = None,
|
||||
settings: dict[str, Any] | None = None,
|
||||
) -> tuple[Any, _CapturingRegistry]:
|
||||
"""Construct the factory with stub deps. Returns ``(factory_callable,
|
||||
registry)`` so tests can read back ``registry.captured_alias``."""
|
||||
|
||||
registry = _CapturingRegistry(
|
||||
default=registry_default,
|
||||
known=known_aliases if known_aliases is not None else {registry_default},
|
||||
)
|
||||
config_store = _FakeConfigStore(dict(settings or {}))
|
||||
factory = build_console_session_factory(
|
||||
registry=registry, # type: ignore[arg-type]
|
||||
config_store=config_store, # type: ignore[arg-type]
|
||||
node_id="console",
|
||||
coord_client_factory=lambda ws_id, uid: MagicMock(),
|
||||
)
|
||||
return factory, registry
|
||||
|
||||
|
||||
def _invoke(factory: Any, **factory_kwargs: Any) -> None:
|
||||
"""Call the factory with a stub UI and absorb the sentinel.
|
||||
|
||||
Forwards ``factory_kwargs`` to the factory so per-call overrides
|
||||
(e.g. ``model_alias``) can flow through. Raises if any other
|
||||
exception comes out — the test should fail loudly when alias
|
||||
resolution itself errors rather than swallowing it.
|
||||
"""
|
||||
ui = MagicMock()
|
||||
ui._user_id = "" # skip storage-backed username lookup branch
|
||||
with pytest.raises(_StopBeforeChatSessionError):
|
||||
factory(ui, **factory_kwargs)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tier 1 — explicit pin (per-call arg or coordinator.model_alias)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_per_call_model_alias_arg_wins_over_everything() -> None:
|
||||
"""The ``model_alias`` kwarg on the factory call (e.g. body field on
|
||||
POST /workstreams/new) wins over both ConfigStore tiers and the
|
||||
registry default."""
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"per-call", "coord-pin", "admin-default", "registry-default"},
|
||||
settings={
|
||||
"coordinator.model_alias": "coord-pin",
|
||||
"model.default_alias": "admin-default",
|
||||
},
|
||||
)
|
||||
_invoke(factory, model_alias="per-call")
|
||||
assert registry.captured_alias == "per-call"
|
||||
|
||||
|
||||
def test_coordinator_model_alias_wins_when_no_per_call_override() -> None:
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"coord-pin", "admin-default", "registry-default"},
|
||||
settings={
|
||||
"coordinator.model_alias": "coord-pin",
|
||||
"model.default_alias": "admin-default",
|
||||
},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "coord-pin"
|
||||
|
||||
|
||||
def test_coordinator_model_alias_passed_through_unvalidated() -> None:
|
||||
"""Tier 1 is an *explicit* operator pin — when it's stale or typoed
|
||||
we deliberately pass it through to ``registry.resolve`` so the
|
||||
request layer turns it into a 503 with the alias surfaced in the
|
||||
error. Falling through silently would mask the misconfiguration."""
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"admin-default", "registry-default"},
|
||||
settings={
|
||||
"coordinator.model_alias": "ghost", # unknown
|
||||
"model.default_alias": "admin-default",
|
||||
},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "ghost"
|
||||
|
||||
|
||||
def test_per_call_model_alias_arg_passed_through_unvalidated() -> None:
|
||||
"""The per-call ``model_alias`` kwarg (POST body field — the more
|
||||
common production trigger) is the same kind of explicit pin as the
|
||||
ConfigStore setting, so a stale value passes through to
|
||||
``registry.resolve`` rather than silently falling through to the
|
||||
system default."""
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"registry-default"},
|
||||
settings={"model.default_alias": "registry-default"},
|
||||
)
|
||||
_invoke(factory, model_alias="ghost")
|
||||
assert registry.captured_alias == "ghost"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tier 2 — model.default_alias (admin-managed system default)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_model_default_alias_used_when_coordinator_unset() -> None:
|
||||
"""Regression for the historical drift: admin sets the system
|
||||
default in the Models tab, the home composer advertises it, and new
|
||||
coordinator sessions must launch on the same alias rather than
|
||||
silently falling through to ``registry.default``."""
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"admin-default", "registry-default"},
|
||||
settings={"model.default_alias": "admin-default"},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "admin-default"
|
||||
|
||||
|
||||
def test_unknown_model_default_alias_falls_through_to_registry_default() -> None:
|
||||
"""Tier 2 is *not* an explicit pin — operators set
|
||||
``model.default_alias`` once in the UI and forget about it; an alias
|
||||
that's later disabled or typo'd should not 503 the coordinator,
|
||||
since tier 3 (``registry.default``) is guaranteed to resolve."""
|
||||
factory, registry = _build_factory(
|
||||
known_aliases={"registry-default"}, # admin-default got removed
|
||||
settings={"model.default_alias": "admin-default"},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "registry-default"
|
||||
|
||||
|
||||
def test_blank_model_default_alias_falls_through_to_registry_default() -> None:
|
||||
factory, registry = _build_factory(
|
||||
settings={"model.default_alias": ""},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "registry-default"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tier 3 — registry.default (config.toml [model].default)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_no_settings_uses_registry_default() -> None:
|
||||
factory, registry = _build_factory()
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "registry-default"
|
||||
|
||||
|
||||
def test_whitespace_only_coord_alias_falls_through() -> None:
|
||||
"""``" "`` is not an explicit pin — ``.strip()`` reduces it to
|
||||
"", which the chain should treat as unset."""
|
||||
factory, registry = _build_factory(
|
||||
settings={"coordinator.model_alias": " "},
|
||||
)
|
||||
_invoke(factory)
|
||||
assert registry.captured_alias == "registry-default"
|
||||
@@ -716,3 +716,82 @@ class TestCoordinatorAdapterDispatchChildEvent:
|
||||
},
|
||||
)
|
||||
assert recorder.enqueued == []
|
||||
|
||||
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
|
||||
"""Every translated state-class event must call
|
||||
``ChildEventBus.notify(ws_id)`` so a registered
|
||||
``wait_for_workstream`` waiter wakes promptly. Notify fires
|
||||
AFTER the UI enqueue so the SSE fan-out keeps priority — the
|
||||
order assertion here is structural (one notify call, matching
|
||||
ws_id) since the bus side-effect lookup is what guards against
|
||||
regressions, not the relative event ordering.
|
||||
"""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
|
||||
|
||||
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
|
||||
"""The dispatch sink translates six event types into the
|
||||
``child_ws_*`` SSE shape; all six must also fire the bus so
|
||||
a wait on any of them wakes. ``ws_created`` is intentionally
|
||||
NOT in this set — waiters register against ws_ids they already
|
||||
know exist (the wait tool takes a pre-known list)."""
|
||||
for etype, extra in [
|
||||
("cluster_state", {"state": "running"}),
|
||||
("ws_closed", {"reason": "evicted"}),
|
||||
("ws_rename", {"name": "renamed"}),
|
||||
("intent_verdict", {"verdict": {"call_id": "c1"}}),
|
||||
("approval_resolved", {"approved": True}),
|
||||
("approve_request", {"detail": {}}),
|
||||
]:
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{"type": etype, "ws_id": "child-a1", **extra},
|
||||
)
|
||||
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
|
||||
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
|
||||
waiter registered against ws Y, or every state change anywhere
|
||||
in the system would shake every concurrent wait."""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-other"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
|
||||
"""Events whose ws_id isn't in any coord's registry are dropped
|
||||
BEFORE the bus notify (early return at ``coord_id is None``).
|
||||
Notify only fires for events the dispatch sink fully translated,
|
||||
keeping the bus side-effect aligned with the UI enqueue."""
|
||||
adapter, _, _ = self._setup()
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["ws-orphan"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "ws-orphan",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
|
||||
|
||||
@@ -9,6 +9,7 @@ storage-call path.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import httpx
|
||||
@@ -20,6 +21,7 @@ from turnstone.console.coordinator_client import (
|
||||
CoordinatorTokenManager,
|
||||
)
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -145,6 +147,7 @@ def _mock_client(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
return client, captured
|
||||
|
||||
@@ -470,6 +473,7 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -663,6 +667,7 @@ def _make_client_with_cluster_response(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -1223,6 +1228,49 @@ def test_list_skills_hides_interactive_only_skills(tmp_path):
|
||||
assert skill["kind"] in {"coordinator", "any"}
|
||||
|
||||
|
||||
def test_list_skills_omits_allowed_tools_when_empty(tmp_path):
|
||||
"""``allowed_tools`` is the auto-approve allowlist (tools exempt
|
||||
from the operator approval gate), NOT the set of tools the skill
|
||||
can use. An empty list reads as "no tool access" to a model
|
||||
that doesn't know the semantics — real misdiagnosis source: a
|
||||
code-review skill with no auto-approve allowlist looked like it
|
||||
had been spawned with zero tools. Dropping the key when empty
|
||||
removes the ambiguity at the source; absence of the field carries
|
||||
the unambiguous meaning "no tool is pre-approved for this skill"
|
||||
while a tool list reads as "these specific tools bypass the prompt".
|
||||
"""
|
||||
st = SQLiteBackend(str(tmp_path / "skills_empty.db"))
|
||||
st.create_prompt_template(
|
||||
template_id="s-empty",
|
||||
name="empty-skill",
|
||||
category="ops",
|
||||
content="",
|
||||
variables="[]",
|
||||
is_default=False,
|
||||
org_id="",
|
||||
created_by="test",
|
||||
tags="[]",
|
||||
allowed_tools="[]",
|
||||
)
|
||||
st.create_prompt_template(
|
||||
template_id="s-nonempty",
|
||||
name="nonempty-skill",
|
||||
category="ops",
|
||||
content="",
|
||||
variables="[]",
|
||||
is_default=False,
|
||||
org_id="",
|
||||
created_by="test",
|
||||
tags="[]",
|
||||
allowed_tools='["read_file"]',
|
||||
)
|
||||
client = _make_read_client(st)
|
||||
result = client.list_skills()
|
||||
by_name = {s["name"]: s for s in result["skills"]}
|
||||
assert "allowed_tools" not in by_name["empty-skill"]
|
||||
assert by_name["nonempty-skill"]["allowed_tools"] == ["read_file"]
|
||||
|
||||
|
||||
def test_list_skills_projects_allowed_tools_capped_with_sentinel(tmp_path):
|
||||
"""Each row carries the skill's allowed_tools (capped at the projection
|
||||
cap with a +N more sentinel) so coordinators can pick a skill without
|
||||
@@ -1443,6 +1491,23 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
|
||||
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
|
||||
matches the coordinator but whose ``user_id`` belongs to a
|
||||
different tenant must collapse to ``denied`` — otherwise a
|
||||
forged / migration-era / pre-tenant-gate row would let a
|
||||
coordinator's LLM observe foreign-tenant state through
|
||||
``wait_for_workstream``. The ``populated_storage`` fixture's
|
||||
``cross-tenant-child`` row has exactly this shape
|
||||
(parent_ws_id="coord-1", user_id="user-2").
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
|
||||
assert result["results"]["cross-tenant-child"]["state"] == "denied"
|
||||
assert result["complete"] is False
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
|
||||
"""A ws_id that doesn't exist collapses into the same 'denied'
|
||||
shape as a foreign ws_id so wait can't be used as an existence
|
||||
@@ -1531,10 +1596,22 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
|
||||
assert list(result["results"].keys()) == ["child-a"]
|
||||
|
||||
|
||||
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
|
||||
"""Per-tick polling must issue batched storage calls — at the
|
||||
documented cap (32 ws_ids over a 600s wait) the naive per-id
|
||||
shape produced ~38k row reads. Guard against regression."""
|
||||
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
|
||||
populated_storage, monkeypatch
|
||||
):
|
||||
"""All storage reads issued by ``wait_for_workstream`` must go
|
||||
through the batched paths. At the documented cap (32 ws_ids over
|
||||
a 600 s wait) the naive per-id shape produced ~38k row reads, so
|
||||
a regression to per-id is the meaningful failure mode this test
|
||||
guards against.
|
||||
|
||||
The primary safety net is the ``pytest.fail`` mock on the per-id
|
||||
``get_workstream`` / ``sum_workstream_tokens`` paths — any call
|
||||
there blows up loudly with the regression message. The
|
||||
additional ``batch_calls`` / ``sum_calls`` assertions cover the
|
||||
subtler regression where the call IS batched but only covers a
|
||||
subset of ws_ids (e.g. one ws_id per call in a loop).
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
batch_calls: list[list[str]] = []
|
||||
sum_calls: list[list[str]] = []
|
||||
@@ -1565,11 +1642,16 @@ def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monke
|
||||
|
||||
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
# One tick is enough since child-a is already idle (terminal).
|
||||
assert len(batch_calls) == 1
|
||||
assert len(sum_calls) == 1
|
||||
assert set(batch_calls[0]) == {"child-a", "child-b"}
|
||||
assert set(sum_calls[0]) == {"child-a", "child-b"}
|
||||
# Every batched call carried the full ws_id set. The exact count
|
||||
# (currently 2: one pre-loop ownership filter + one snapshot tick)
|
||||
# is incidental; if either gains another batched read it stays
|
||||
# batched, which is the property under test.
|
||||
assert batch_calls, "no batched get_workstreams_batch call observed"
|
||||
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
|
||||
first_batch = set(batch_calls[0])
|
||||
first_sum = set(sum_calls[0])
|
||||
assert first_batch == {"child-a", "child-b"}
|
||||
assert first_sum == {"child-a", "child-b"}
|
||||
|
||||
|
||||
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
@@ -1581,6 +1663,205 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
assert "invalid mode" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — event-driven (ChildEventBus wired in)
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# When the coord adapter wires its ``child_event_bus`` into the client,
|
||||
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
|
||||
# and only re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. The legacy ``time.sleep`` poll path remains intact for tests
|
||||
# that don't wire the bus (above), so this section adds focused
|
||||
# coverage of the bus-driven behaviour without re-running the full
|
||||
# matrix of mode / since / cross-tenant cases.
|
||||
|
||||
|
||||
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
|
||||
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
|
||||
|
||||
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
|
||||
simulate the dispatch-sink wake-up.
|
||||
"""
|
||||
transport = httpx.MockTransport(lambda r: httpx.Response(200))
|
||||
http = httpx.Client(transport=transport)
|
||||
return CoordinatorClient(
|
||||
console_base_url="http://x",
|
||||
storage=storage,
|
||||
token_factory=lambda: "t",
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=bus,
|
||||
)
|
||||
|
||||
|
||||
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
|
||||
"""Subscribe-after-terminal race: the wait registers its waiter
|
||||
BEFORE the first snapshot, then re-snapshots — an already-terminal
|
||||
child must return at once without spinning the heartbeat cap.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-a"]["state"] == "idle"
|
||||
assert result["elapsed"] < 1.0
|
||||
# Waiter must be unregistered on exit so a long-lived bus doesn't
|
||||
# accumulate dead keys across many waits.
|
||||
assert "child-a" not in bus._waiters
|
||||
|
||||
|
||||
def test_wait_with_bus_wakes_on_notify(populated_storage):
|
||||
"""The core property of the refactor: a state-change ``notify``
|
||||
must wake the wait promptly — well under the legacy 0.5 s poll
|
||||
cadence AND the 2 s heartbeat cap. Test fires a state update
|
||||
+ notify after a short delay and asserts the wait returns quickly.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b starts running; flip to idle + notify after the wait
|
||||
# blocks. 100 ms is enough that the wait is parked in event.wait()
|
||||
# but short enough that the test runs fast.
|
||||
timer = _t.Timer(
|
||||
0.1,
|
||||
lambda: (
|
||||
populated_storage.update_workstream_state("child-b", "idle"),
|
||||
bus.notify("child-b"),
|
||||
),
|
||||
)
|
||||
timer.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-b"]["state"] == "idle"
|
||||
# Bus-driven wake should fire well under 1 s; legacy poll would
|
||||
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
|
||||
# plus a few ms. Generous 0.6 s budget for CI noise.
|
||||
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
|
||||
|
||||
|
||||
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
|
||||
"""A notify on a ws_id the wait isn't watching must NOT wake it —
|
||||
otherwise every state change anywhere on the system would shake
|
||||
every concurrent wait into a redundant storage snapshot.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b is running indefinitely; mode='all' will time out unless
|
||||
# a relevant notify fires. Fire only unrelated notifies — wait
|
||||
# should still hit the full timeout.
|
||||
import threading as _t
|
||||
|
||||
def _fire_unrelated() -> None:
|
||||
for _ in range(5):
|
||||
bus.notify("ws-unrelated-1")
|
||||
bus.notify("ws-unrelated-2")
|
||||
time.sleep(0.05)
|
||||
|
||||
t = _t.Thread(target=_fire_unrelated, daemon=True)
|
||||
t.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
|
||||
# Wait should burn its full timeout (give or take heartbeat
|
||||
# granularity). The bus path doesn't have a 0.5 s poll, so the
|
||||
# bound is "approximately timeout".
|
||||
assert elapsed >= 0.5
|
||||
t.join(timeout=1.0)
|
||||
|
||||
|
||||
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
|
||||
"""Without any notify, the wait must still progress through ticks
|
||||
via the heartbeat cap so ``progress_callback`` keeps firing for
|
||||
the sidebar UI. Verified by counting callback firings over an
|
||||
interval longer than the heartbeat.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Shrink the heartbeat for test speed via the ClassVar seam —
|
||||
# instance attribute shadows the class-level default. Production
|
||||
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
|
||||
# notify property in well under 1 s.
|
||||
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
|
||||
snapshots: list[dict[str, dict[str, object]]] = []
|
||||
|
||||
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
|
||||
snapshots.append(snap)
|
||||
|
||||
# child-b is running indefinitely; wait will time out at 0.4 s.
|
||||
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
|
||||
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
|
||||
# CI flakiness.
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False
|
||||
assert elapsed >= 0.4
|
||||
# At least 2 callback firings: the initial snapshot plus at least
|
||||
# one heartbeat-driven re-tick. Tight upper bound would be
|
||||
# ~ceil(0.4/0.1) + 1 = 5 firings.
|
||||
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
|
||||
|
||||
|
||||
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
|
||||
"""Both the success path and the timeout path must unregister the
|
||||
waiter — otherwise a long-lived bus accumulates dead
|
||||
``threading.Event`` instances forever.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Success path (already-terminal child).
|
||||
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert bus._waiters == {}, "success path leaked waiter"
|
||||
# Timeout path (running child, mode='all' that times out).
|
||||
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
|
||||
assert bus._waiters == {}, "timeout path leaked waiter"
|
||||
|
||||
|
||||
def test_wait_with_bus_multi_waiter_independence(populated_storage):
|
||||
"""Two concurrent waits on the same ws_id must be independent —
|
||||
one wait completing must not affect the other's wake-up state.
|
||||
Smoke-tests the multi-Event-per-bucket bus behaviour against the
|
||||
real wait-loop.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
|
||||
results: dict[str, dict[str, object]] = {}
|
||||
|
||||
def _do_wait(label: str) -> None:
|
||||
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
|
||||
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
for label in ("t0", "t1", "t2"):
|
||||
assert results[label]["complete"] is True
|
||||
assert results[label]["results"]["child-a"]["state"] == "idle"
|
||||
# All waiters must be unregistered after exit.
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — last-message bundling
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -2378,3 +2659,362 @@ def test_cleanup_dead_task_child_refs_storage_batch_failure_swallows(populated_s
|
||||
|
||||
populated_storage.get_workstreams_batch = _boom # type: ignore[method-assign]
|
||||
assert client.cleanup_dead_task_child_refs("coord-1") == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_workstream — three-tier output compression
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# A coord doing a fan-out wave against tool-heavy children would
|
||||
# otherwise blow the context budget on raw output alone. Mirrors the
|
||||
# search tool's Tier-1/Tier-2/Tier-3 ladder.
|
||||
|
||||
|
||||
def _make_inspect_result(
|
||||
*, ws_id: str = "ws-test", state: str = "running", n_messages: int = 5
|
||||
) -> dict[str, Any]:
|
||||
"""Build an inspect-result dict shaped like ``coordinator_client.inspect()``.
|
||||
|
||||
Production output keys (``ws_id``, ``skill_id``) mirror the storage
|
||||
row that ``inspect()`` spreads from ``get_workstream``. Tests that
|
||||
synthesize an inspect result must match these keys — otherwise a
|
||||
formatter that looks at the production keys silently emits null
|
||||
values against a fixture that uses different ones (real bug-1
|
||||
regression source: skeleton tier read ``skill`` from a fixture
|
||||
that wrote ``skill`` while production wrote ``skill_id``).
|
||||
"""
|
||||
return {
|
||||
"ws_id": ws_id,
|
||||
"state": state,
|
||||
"title": "test workstream",
|
||||
"skill_id": "researcher",
|
||||
"messages": [
|
||||
{"role": "user" if i % 2 == 0 else "assistant", "content": f"msg {i} content"}
|
||||
for i in range(n_messages)
|
||||
],
|
||||
"verdicts": [],
|
||||
}
|
||||
|
||||
|
||||
def test_format_inspect_tiered_full_fits_returns_full_tier():
|
||||
"""Small payloads pass through with `_tier='full'` — no compression."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
result = _make_inspect_result(n_messages=3)
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "full"
|
||||
# Every message verbatim.
|
||||
assert len(parsed["messages"]) == 3
|
||||
assert parsed["messages"][0]["content"] == "msg 0 content"
|
||||
|
||||
|
||||
def test_format_inspect_tiered_compact_when_full_exceeds_budget():
|
||||
"""Large messages trigger the compact tier — head/tail-snipped
|
||||
content with the rest of the row intact."""
|
||||
from turnstone.console.coordinator_client import (
|
||||
_INSPECT_MSG_CONTENT_HEAD,
|
||||
_INSPECT_MSG_CONTENT_TAIL,
|
||||
_INSPECT_OUTPUT_BUDGET,
|
||||
_format_inspect_tiered,
|
||||
)
|
||||
|
||||
# Each message ~5KB; with 20 messages, full tier blows the 32KB budget.
|
||||
fat = "X" * 5000
|
||||
result = {
|
||||
"id": "ws-fat",
|
||||
"state": "running",
|
||||
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "compact"
|
||||
# Every message preserved (compact keeps the count, just snips content).
|
||||
assert len(parsed["messages"]) == 20
|
||||
# Head/tail snip kicked in.
|
||||
msg_content = parsed["messages"][0]["content"]
|
||||
assert msg_content.startswith("X" * _INSPECT_MSG_CONTENT_HEAD)
|
||||
assert msg_content.endswith("X" * _INSPECT_MSG_CONTENT_TAIL)
|
||||
assert "chars elided" in msg_content
|
||||
# Budget invariant — the load-bearing contract of the formatter.
|
||||
# Without this assertion, a future change to ``_tier_note`` or
|
||||
# ``_compact_message`` could push the output over budget and the
|
||||
# ``_truncate_output`` head+tail safety net would silently mask
|
||||
# the regression, re-introducing the middle-message-drop pathology.
|
||||
assert len(out) <= _INSPECT_OUTPUT_BUDGET
|
||||
|
||||
|
||||
def test_format_inspect_tiered_compact_when_content_below_snip_threshold():
|
||||
"""When per-message content is below the snip threshold but the
|
||||
message COUNT alone overflows the budget, compact tier must still
|
||||
stay within budget — by trimming the message list (head + tail of
|
||||
messages) rather than degrading straight to skeleton. Bug-3
|
||||
regression cover: with 400 × 100-char messages, the original
|
||||
formatter fell through to skeleton because adding ``_tier_note``
|
||||
to an un-snipped tier-2 produced output strictly larger than
|
||||
tier-1 (both over budget). The fix preserves messages from both
|
||||
ends of the list and inserts an ``_omitted`` sentinel."""
|
||||
from turnstone.console.coordinator_client import (
|
||||
_INSPECT_OUTPUT_BUDGET,
|
||||
_format_inspect_tiered,
|
||||
)
|
||||
|
||||
# 400 × ~100 chars → Tier-1 ~53 KB (over budget), per-message
|
||||
# content under the 964-char snip threshold so content-snipping
|
||||
# saves nothing. Without the list-trim rung the formatter would
|
||||
# fall to skeleton and drop all 400 messages.
|
||||
smallish = "S" * 100
|
||||
result = {
|
||||
"ws_id": "ws-many-small",
|
||||
"state": "running",
|
||||
"messages": [
|
||||
{"role": "assistant" if i % 2 == 0 else "user", "content": smallish} for i in range(400)
|
||||
],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
# Should NOT fall through to skeleton — message-list trim preserves
|
||||
# head + tail of the conversation.
|
||||
assert parsed["_tier"] == "compact"
|
||||
assert "messages" in parsed
|
||||
# Some messages must survive; the trim shape is head + tail with an
|
||||
# ``_omitted`` sentinel between them.
|
||||
assert len(parsed["messages"]) > 0
|
||||
assert len(parsed["messages"]) < 400
|
||||
# Budget invariant.
|
||||
assert len(out) <= _INSPECT_OUTPUT_BUDGET
|
||||
|
||||
|
||||
def test_format_inspect_tiered_skeleton_when_compact_also_exceeds_budget():
|
||||
"""Tier 3 fallback: counts + last assistant preview only. Trigger by
|
||||
flooding with messages whose content is a multi-block list — the
|
||||
snipper correctly leaves non-string content unchanged (mirrors
|
||||
Anthropic/OpenAI multi-block content shape), so even after the
|
||||
(5, 10) message-list trim the surviving 15 messages don't fit in
|
||||
the 32 KB budget."""
|
||||
from turnstone.console.coordinator_client import (
|
||||
_INSPECT_OUTPUT_BUDGET,
|
||||
_format_inspect_tiered,
|
||||
)
|
||||
|
||||
# 50 messages × multi-block content (~30 KB each — list-shape
|
||||
# content bypasses the head/tail string snipper because lists
|
||||
# aren't strings). Even (5, 10) trim leaves 15 × 30 KB which
|
||||
# blows the 32 KB budget — forces skeleton.
|
||||
fat_block = {"type": "text", "text": "Y" * 3000}
|
||||
result = {
|
||||
"ws_id": "ws-flood",
|
||||
"state": "running",
|
||||
"title": "flood",
|
||||
"skill_id": "researcher",
|
||||
"messages": [
|
||||
{
|
||||
"role": "assistant" if i % 2 == 0 else "user",
|
||||
"content": [fat_block] * 10,
|
||||
}
|
||||
for i in range(50)
|
||||
],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "skeleton"
|
||||
assert parsed["message_count"] == 50
|
||||
# Role distribution surfaces — the "what shape of activity" signal.
|
||||
assert parsed["roles"]["assistant"] == 25
|
||||
assert parsed["roles"]["user"] == 25
|
||||
# No `messages` field at skeleton tier — only the aggregate signal.
|
||||
assert "messages" not in parsed
|
||||
# Budget invariant.
|
||||
assert len(out) <= _INSPECT_OUTPUT_BUDGET
|
||||
|
||||
|
||||
def test_format_inspect_tiered_skeleton_keeps_terminal_state_fields():
|
||||
"""``close_reason`` / ``last_error`` survive the skeleton fall — they're
|
||||
small, load-bearing, and the operator needs them to understand WHY
|
||||
a terminal child landed in its state."""
|
||||
from turnstone.console.coordinator_client import (
|
||||
_INSPECT_OUTPUT_BUDGET,
|
||||
_format_inspect_tiered,
|
||||
)
|
||||
|
||||
# Same flood pattern as the bare-skeleton test (multi-block content
|
||||
# bypasses the string snipper) — paired with terminal-state fields
|
||||
# that must survive the skeleton fall.
|
||||
fat_block = {"type": "text", "text": "Z" * 3000}
|
||||
result = {
|
||||
"ws_id": "ws-closed",
|
||||
"state": "closed",
|
||||
"title": "done",
|
||||
"skill_id": "researcher",
|
||||
"messages": [{"role": "user", "content": [fat_block] * 10} for _ in range(50)],
|
||||
"verdicts": [],
|
||||
"close_reason": "task complete: report attached",
|
||||
"live": None, # filtered by truthy check
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "skeleton"
|
||||
assert parsed["close_reason"] == "task complete: report attached"
|
||||
# Falsy ``live`` doesn't bleed through.
|
||||
assert "live" not in parsed
|
||||
assert len(out) <= _INSPECT_OUTPUT_BUDGET
|
||||
|
||||
|
||||
def test_format_inspect_tiered_error_shapes_bypass_tiering():
|
||||
"""Cross-tenant / not-found responses keep their original shape — they
|
||||
carry no messages, are already tiny, and changing them would break
|
||||
callers that key on the ``error`` field."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
result = {"error": "workstream not found", "ws_id": "ws-foreign"}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed == {"error": "workstream not found", "ws_id": "ws-foreign"}
|
||||
# No `_tier` annotation — error shapes are self-describing.
|
||||
assert "_tier" not in parsed
|
||||
|
||||
|
||||
def test_format_inspect_tiered_compact_preserves_tool_call_linkage():
|
||||
"""Compact tier keeps ``tool_name`` / ``tool_call_id`` / ``name`` so a
|
||||
model reading the snipped trace can still pair a tool call to its
|
||||
response — the linkage is load-bearing for "what happened" signal."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
fat = "Q" * 5000
|
||||
result = {
|
||||
"ws_id": "ws-tools",
|
||||
"state": "running",
|
||||
"messages": [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": fat,
|
||||
"tool_name": "bash",
|
||||
"tool_call_id": "call-1",
|
||||
}
|
||||
for _ in range(20)
|
||||
],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "compact"
|
||||
first = parsed["messages"][0]
|
||||
assert first["tool_name"] == "bash"
|
||||
assert first["tool_call_id"] == "call-1"
|
||||
|
||||
|
||||
def test_format_inspect_tiered_compact_preserves_assistant_tool_calls():
|
||||
"""Compact tier must preserve the assistant-side ``tool_calls`` list
|
||||
(OpenAI shape: ``[{id, type, function: {name, arguments}}]``) so a
|
||||
model reading the snipped trace can see WHICH tool was called and
|
||||
pair it with the corresponding result row via ``id`` ↔ ``tool_call_id``.
|
||||
Bug-2 regression cover: the pre-fix compactor stripped ``tool_calls``,
|
||||
leaving the audit reader with a tool-result orphan against an
|
||||
invisible call.
|
||||
|
||||
``function.arguments`` strings are snipped head/tail (analogous to
|
||||
content) because they can be multi-KB JSON; ``id`` and
|
||||
``function.name`` are preserved verbatim — they're the linkage."""
|
||||
from turnstone.console.coordinator_client import (
|
||||
_INSPECT_TOOL_ARG_HEAD,
|
||||
_INSPECT_TOOL_ARG_TAIL,
|
||||
_format_inspect_tiered,
|
||||
)
|
||||
|
||||
fat_content = "C" * 5000 # forces compact tier
|
||||
fat_args = "A" * 5000 # forces argument snipping
|
||||
tool_calls = [
|
||||
{
|
||||
"id": "call-abc-123",
|
||||
"type": "function",
|
||||
"function": {"name": "bash", "arguments": fat_args},
|
||||
},
|
||||
{
|
||||
"id": "call-def-456",
|
||||
"type": "function",
|
||||
"function": {"name": "read_file", "arguments": fat_args},
|
||||
},
|
||||
]
|
||||
result = {
|
||||
"ws_id": "ws-tool-calls",
|
||||
"state": "running",
|
||||
"messages": [
|
||||
{"role": "assistant", "content": fat_content, "tool_calls": tool_calls}
|
||||
for _ in range(20)
|
||||
],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "compact"
|
||||
first = parsed["messages"][0]
|
||||
# tool_calls survives compaction.
|
||||
assert "tool_calls" in first
|
||||
assert len(first["tool_calls"]) == 2
|
||||
# Linkage fields verbatim.
|
||||
assert first["tool_calls"][0]["id"] == "call-abc-123"
|
||||
assert first["tool_calls"][0]["function"]["name"] == "bash"
|
||||
assert first["tool_calls"][1]["id"] == "call-def-456"
|
||||
assert first["tool_calls"][1]["function"]["name"] == "read_file"
|
||||
# arguments snipped head/tail — both prefix and suffix preserved.
|
||||
snipped_args = first["tool_calls"][0]["function"]["arguments"]
|
||||
assert snipped_args.startswith("A" * _INSPECT_TOOL_ARG_HEAD)
|
||||
assert snipped_args.endswith("A" * _INSPECT_TOOL_ARG_TAIL)
|
||||
assert "chars elided" in snipped_args
|
||||
|
||||
|
||||
def test_format_inspect_tiered_compact_passes_small_messages_through_unsnipped():
|
||||
"""Messages under the snip threshold pass through verbatim at compact
|
||||
tier — snipping a 100-byte message costs more bytes (the elision
|
||||
marker) than it saves."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
# Mix: a few large messages force compact tier; small messages must
|
||||
# not be snipped.
|
||||
big = "B" * 5000
|
||||
small = "S" * 50
|
||||
result = {
|
||||
"id": "ws-mixed",
|
||||
"state": "running",
|
||||
"messages": [{"role": "assistant", "content": big} for _ in range(15)]
|
||||
+ [{"role": "user", "content": small}],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "compact"
|
||||
# The trailing small message is exact, not snipped.
|
||||
assert parsed["messages"][-1]["content"] == small
|
||||
|
||||
|
||||
def test_format_inspect_tiered_emits_tier_note_when_compressed():
|
||||
"""The ``_tier_note`` advisory tells the LLM how to ask for a tighter
|
||||
or fuller view next time — actionable feedback rather than a bare
|
||||
"we compressed your output" signal."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
fat = "F" * 5000
|
||||
result = {
|
||||
"id": "ws-noted",
|
||||
"state": "running",
|
||||
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
|
||||
"verdicts": [],
|
||||
}
|
||||
out = _format_inspect_tiered(result)
|
||||
parsed = json.loads(out)
|
||||
assert "_tier_note" in parsed
|
||||
assert "message_limit" in parsed["_tier_note"]
|
||||
|
||||
|
||||
def test_format_inspect_tiered_full_tier_omits_tier_note():
|
||||
"""When the full tier fits, no note is emitted — the absence of a
|
||||
note is the signal that nothing was compressed."""
|
||||
from turnstone.console.coordinator_client import _format_inspect_tiered
|
||||
|
||||
out = _format_inspect_tiered(_make_inspect_result(n_messages=2))
|
||||
parsed = json.loads(out)
|
||||
assert parsed["_tier"] == "full"
|
||||
assert "_tier_note" not in parsed
|
||||
|
||||
@@ -40,6 +40,7 @@ from turnstone.console.server import (
|
||||
_require_coord_mgr,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.session_manager import SessionManager
|
||||
from turnstone.core.session_routes import (
|
||||
SessionEndpointConfig,
|
||||
@@ -286,6 +287,7 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
|
||||
coord_ws_id="coord-42",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
# spawn ---------------------------------------------------------------
|
||||
@@ -387,6 +389,7 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-root",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -215,7 +215,12 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
|
||||
summary tempted callers to write ``if result["status"] == "idle"``
|
||||
which silently never matched. The summary now omits the field
|
||||
entirely; lifecycle state lives on the workstream row and is read
|
||||
via inspect_workstream."""
|
||||
via inspect_workstream.
|
||||
|
||||
Also asserts the return key is ``child_ws_id`` (not ``ws_id``) so
|
||||
the coordinator LLM doesn't recency-bias toward feeding the spawn
|
||||
output back into another ``spawn_workstream(ws_id=...)`` call.
|
||||
"""
|
||||
sess, coord, _ui = coord_session
|
||||
coord.spawn.return_value = {
|
||||
"ws_id": "child-7",
|
||||
@@ -227,8 +232,9 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
|
||||
_call_id, output = sess._exec_spawn_workstream(item)
|
||||
body = json.loads(output)
|
||||
assert "status" not in body
|
||||
assert "ws_id" not in body
|
||||
# The substantive fields are still here.
|
||||
assert body["ws_id"] == "child-7"
|
||||
assert body["child_ws_id"] == "child-7"
|
||||
assert body["node_id"] == "node-1"
|
||||
|
||||
|
||||
@@ -248,6 +254,10 @@ def test_spawn_batch_exec_does_not_surface_misleading_status_field(coord_session
|
||||
body = json.loads(output)
|
||||
assert "0" in body["results"]
|
||||
assert "status" not in body["results"]["0"]
|
||||
# Per-result entries surface ``child_ws_id``, not ``ws_id`` — same
|
||||
# recency-bias rationale as the spawn_workstream test above.
|
||||
assert body["results"]["0"]["child_ws_id"] == "c-x"
|
||||
assert "ws_id" not in body["results"]["0"]
|
||||
|
||||
|
||||
def test_spawn_exec_surfaces_client_error(coord_session):
|
||||
@@ -260,6 +270,21 @@ def test_spawn_exec_surfaces_client_error(coord_session):
|
||||
assert ui.tool_results[-1][3] is True # is_error
|
||||
|
||||
|
||||
def test_spawn_exec_treats_missing_ws_id_on_success_path_as_error(coord_session):
|
||||
"""A malformed upstream response (200-success-shape with no
|
||||
``ws_id``) used to emit ``{"child_ws_id": null}`` to the LLM,
|
||||
which then chased a null id through follow-up tools. Now matches
|
||||
the matching guard in ``_exec_spawn_batch``: surface as a tool
|
||||
error so the model retries instead of acting on garbage."""
|
||||
sess, coord, ui = coord_session
|
||||
# No ``error`` field, but ``ws_id`` is missing — the silent-null path.
|
||||
coord.spawn.return_value = {"name": "c", "node_id": "node-1", "status": 200}
|
||||
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
|
||||
_call_id, output = sess._exec_spawn_workstream(item)
|
||||
assert "no ws_id" in output
|
||||
assert ui.tool_results[-1][3] is True # is_error
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1410,9 +1435,13 @@ def test_spawn_batch_exec_serialises_spawns_and_returns_results(coord_session):
|
||||
assert body["denied"] == []
|
||||
# Keyed by input index (stringified).
|
||||
assert set(body["results"].keys()) == {"0", "1", "2"}
|
||||
assert body["results"]["0"]["ws_id"] == "child-0"
|
||||
assert body["results"]["0"]["child_ws_id"] == "child-0"
|
||||
assert body["results"]["1"]["node_id"] == "n-1"
|
||||
assert body["results"]["2"]["ws_id"] == "child-2"
|
||||
assert body["results"]["2"]["child_ws_id"] == "child-2"
|
||||
# Confirm we don't leak the old ``ws_id`` key alongside the new
|
||||
# ``child_ws_id`` — see test_spawn_exec_does_not_surface_misleading_status_field
|
||||
# for the rationale on the rename.
|
||||
assert "ws_id" not in body["results"]["0"]
|
||||
|
||||
|
||||
def test_spawn_batch_exec_surfaces_per_item_errors_in_denied(coord_session):
|
||||
|
||||
@@ -453,3 +453,225 @@ class TestDecorateAdvisoryExtraction:
|
||||
assert tool_msg["advisories"] == [
|
||||
{"type": "user_interjection", "text": "check the logs", "priority": "notice"}
|
||||
]
|
||||
|
||||
|
||||
class TestExtractReasoningForHistory:
|
||||
"""``extract_reasoning_for_history`` — Phase 1 surfaces stored
|
||||
Anthropic thinking blocks on assistant messages and strips
|
||||
``_provider_content`` from the wire payload.
|
||||
|
||||
Drives through the real ``AnthropicProvider.extract_reasoning_text``
|
||||
(no mock-of-extractor) — the helper test and the provider unit
|
||||
test (``tests/test_provider_anthropic_reasoning.py``) together
|
||||
catch a regression at either layer distinctly.
|
||||
"""
|
||||
|
||||
def _anthropic_thinking_msg(self, text: str = "let me think") -> dict[str, object]:
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": text, "signature": "sig"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
}
|
||||
|
||||
def test_extract_thinking_surfaces_reasoning_field(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [self._anthropic_thinking_msg("let me think")]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "let me think"
|
||||
|
||||
def test_strips_provider_content_after_extraction(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [self._anthropic_thinking_msg("anything")]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_strips_provider_content_when_flag_false(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [self._anthropic_thinking_msg("anything")]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=False)
|
||||
# Strip is unconditional; reasoning is the conditional bit.
|
||||
assert "_provider_content" not in messages[0]
|
||||
assert "reasoning" not in messages[0]
|
||||
|
||||
def test_first_block_thinking_dispatches_to_anthropic(self) -> None:
|
||||
# Even when text and tool_use blocks follow, the first-block-type
|
||||
# discriminator routes thinking-prefixed payloads correctly.
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "first", "signature": "s"},
|
||||
{"type": "text", "text": "spoken"},
|
||||
{"type": "tool_use", "id": "t1", "name": "f", "input": {}},
|
||||
],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "first"
|
||||
|
||||
def test_first_block_reasoning_dispatches_to_openai_responses(self) -> None:
|
||||
# Phase 3: dispatcher routes type=="reasoning" to the
|
||||
# OpenAI Responses extractor, which now returns the
|
||||
# summary[*].text concatenation. Pre-Phase-3 this asserted
|
||||
# "" (the stub); the assertion was tightened once the wire
|
||||
# path landed.
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [
|
||||
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "s"}]}
|
||||
],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "s"
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_unknown_first_block_type_no_op(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [{"type": "text", "text": "no reasoning here"}],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "reasoning" not in messages[0]
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_skips_messages_without_provider_content(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [{"role": "assistant", "content": "plain"}]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "reasoning" not in messages[0]
|
||||
assert messages[0]["content"] == "plain"
|
||||
|
||||
def test_user_and_tool_messages_untouched(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages: list[dict[str, object]] = [
|
||||
{"role": "user", "content": "hi"},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "out"},
|
||||
self._anthropic_thinking_msg("only this one"),
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "reasoning" not in messages[0]
|
||||
assert "reasoning" not in messages[1]
|
||||
assert messages[2]["reasoning"] == "only this one"
|
||||
|
||||
def test_empty_provider_content_no_extraction(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [{"role": "assistant", "content": "x", "_provider_content": []}]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "reasoning" not in messages[0]
|
||||
# Empty-list provider_content is still stripped from the wire.
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_first_block_not_a_dict_skipped(self) -> None:
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages: list[dict[str, object]] = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": ["bogus"],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert "reasoning" not in messages[0]
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_first_block_reasoning_text_dispatches_to_openai_chat(self) -> None:
|
||||
# Phase 3 path 3: synthetic ``reasoning_text`` blocks (stamped
|
||||
# by ChatSession._maybe_synth_reasoning_block for vLLM /
|
||||
# llama.cpp / Gemini-compat conversations) dispatch to
|
||||
# OpenAIChatCompletionsProvider.extract_reasoning_text.
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "answer",
|
||||
"_provider_content": [
|
||||
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
|
||||
],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "synth thought"
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_dispatcher_scans_past_unrecognized_first_blocks(self) -> None:
|
||||
# Regression for Copilot finding: dispatcher used to inspect
|
||||
# only provider_content[0]['type']. OpenAI Responses captures
|
||||
# EVERY output_item.done event into provider_blocks (not just
|
||||
# reasoning), so a hypothetical [message, reasoning, ...]
|
||||
# ordering would have silently dropped the reasoning. Now
|
||||
# walks the list for the first recognised reasoning-bearing
|
||||
# type and dispatches the whole list to that provider.
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "answer",
|
||||
"_provider_content": [
|
||||
# First block is a non-reasoning OpenAI Responses item.
|
||||
{"type": "message", "role": "assistant", "content": "answer"},
|
||||
# Reasoning sits later in the list.
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "deferred"}],
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "deferred"
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
def test_first_block_redacted_thinking_dispatches_to_anthropic(self) -> None:
|
||||
# Anthropic's extended-thinking API documents that
|
||||
# ``redacted_thinking`` blocks (sealed by the safety system)
|
||||
# can appear before, after, or interleaved with regular
|
||||
# ``thinking`` blocks. When the redacted block lands first,
|
||||
# the dispatcher must still route to AnthropicProvider so the
|
||||
# surrounding real thinking text surfaces — without this the
|
||||
# reasoning bubble silently disappears on history rehydration.
|
||||
# Pinned by registering "redacted_thinking" as a second key
|
||||
# in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the Anthropic
|
||||
# factory; Anthropic's extractor's type=="thinking" filter
|
||||
# already correctly skips the redacted block.
|
||||
from turnstone.core.history_decoration import extract_reasoning_for_history
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "answer",
|
||||
"_provider_content": [
|
||||
{"type": "redacted_thinking", "data": "sealed-blob"},
|
||||
{"type": "thinking", "thinking": "real thought", "signature": "s"},
|
||||
{"type": "text", "text": "answer"},
|
||||
],
|
||||
}
|
||||
]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == "real thought"
|
||||
assert "_provider_content" not in messages[0]
|
||||
|
||||
@@ -55,6 +55,12 @@ class _FakeUI:
|
||||
pass
|
||||
|
||||
# ChatSession callbacks (no-op for this test)
|
||||
def on_turn_start(self) -> None:
|
||||
pass
|
||||
|
||||
def on_turn_committed(self) -> None:
|
||||
pass
|
||||
|
||||
def on_thinking_start(self) -> None:
|
||||
pass
|
||||
|
||||
|
||||
@@ -777,6 +777,58 @@ class TestModelAliasResolution:
|
||||
assert judge._client_factory_args["api_key"] == "alias-key"
|
||||
assert judge._client_factory_args["provider_name"] == "openai"
|
||||
|
||||
def test_unknown_alias_inherits_session_model(self):
|
||||
"""``judge.model`` is alias-only. A value that doesn't resolve
|
||||
through the registry inherits the session model (same path as
|
||||
an empty config.model) rather than getting pinned onto the
|
||||
session provider as a raw model id — that legacy behavior
|
||||
silently broke whenever the session provider didn't speak the
|
||||
configured model id (Anthropic session, ``judge.model =
|
||||
"gpt-5-mini"`` → every verdict came back as ``llm_fallback``)."""
|
||||
session_provider = _make_mock_provider()
|
||||
session_provider.provider_name = "anthropic"
|
||||
session_client = MagicMock()
|
||||
session_client.base_url = "https://session.example/v1"
|
||||
session_client.api_key = "session-key"
|
||||
|
||||
registry = MagicMock()
|
||||
registry.has_alias.return_value = False # judge.model isn't an alias
|
||||
|
||||
config = JudgeConfig(enabled=True, model="gpt-5-mini")
|
||||
judge = IntentJudge(
|
||||
config=config,
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
model_registry=registry,
|
||||
)
|
||||
|
||||
assert judge._provider is session_provider
|
||||
assert judge._model == "session-default-model"
|
||||
# Context window mirrors the session, not the (uncalled) caps lookup.
|
||||
assert judge._judge_context_window == 100_000
|
||||
|
||||
def test_empty_model_inherits_session_model(self):
|
||||
"""Empty ``config.model`` is the documented self-consistency path."""
|
||||
session_provider = _make_mock_provider()
|
||||
session_provider.provider_name = "openai"
|
||||
session_client = MagicMock()
|
||||
session_client.base_url = "https://session.example/v1"
|
||||
session_client.api_key = "session-key"
|
||||
|
||||
config = JudgeConfig(enabled=True, model="")
|
||||
judge = IntentJudge(
|
||||
config=config,
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
)
|
||||
|
||||
assert judge._provider is session_provider
|
||||
assert judge._model == "session-default-model"
|
||||
|
||||
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
|
||||
"""Happy-path regression for coordinator tool calls: with a properly
|
||||
resolved provider, the verdict tier must be ``llm`` — the
|
||||
|
||||
+122
-1
@@ -51,7 +51,11 @@ class TestIntentVerdictCRUD:
|
||||
assert v["tier"] == "heuristic"
|
||||
assert v["judge_model"] == ""
|
||||
assert v["latency_ms"] == 2
|
||||
assert v["user_decision"] == ""
|
||||
# ``user_decision`` defaults to ``"pending"`` (not the empty
|
||||
# string) so an audit reader can distinguish in-flight rows
|
||||
# from pre-convention legacy rows that carry the column's
|
||||
# server_default of ``""``.
|
||||
assert v["user_decision"] == "pending"
|
||||
assert "created" in v
|
||||
|
||||
def test_get_nonexistent(self, db):
|
||||
@@ -114,6 +118,123 @@ class TestIntentVerdictCRUD:
|
||||
assert ok is False
|
||||
|
||||
|
||||
class TestIntentVerdictUpsert:
|
||||
"""``upsert_intent_verdict`` — the LLM-tier-aware persistence path.
|
||||
|
||||
Backs the heuristic → llm_fallback "upgrade in place" pattern.
|
||||
The async judge's fallback verdicts deliberately reuse the
|
||||
heuristic ``verdict_id``; a plain INSERT would collide on the
|
||||
PK and the upgrade would be lost to a silently-swallowed
|
||||
exception (Postgres logged ``intent_verdicts_pkey`` violations
|
||||
for every fallback delivery on stable/1.5 smoke tests).
|
||||
"""
|
||||
|
||||
def test_upsert_on_fresh_id_inserts(self, db):
|
||||
"""No conflict — behaves like a regular INSERT."""
|
||||
db.upsert_intent_verdict(**_make_verdict_kwargs())
|
||||
v = db.get_intent_verdict("v_001")
|
||||
assert v is not None
|
||||
assert v["tier"] == "heuristic"
|
||||
assert v["user_decision"] == "pending"
|
||||
|
||||
def test_upsert_on_conflict_upgrades_tier_reasoning_judge_model(self, db):
|
||||
"""On PK conflict: tier, reasoning, judge_model update — every
|
||||
other field is preserved. Mirrors what the judge emits when
|
||||
promoting heuristic → llm_fallback."""
|
||||
db.upsert_intent_verdict(
|
||||
**_make_verdict_kwargs(
|
||||
tier="heuristic",
|
||||
reasoning="initial heuristic reasoning",
|
||||
judge_model="",
|
||||
)
|
||||
)
|
||||
db.upsert_intent_verdict(
|
||||
**_make_verdict_kwargs(
|
||||
tier="llm_fallback",
|
||||
reasoning="initial heuristic reasoning (LLM judge did not return a verdict)",
|
||||
judge_model="gpt-5-judge",
|
||||
)
|
||||
)
|
||||
v = db.get_intent_verdict("v_001")
|
||||
assert v is not None
|
||||
# The three fields that should change.
|
||||
assert v["tier"] == "llm_fallback"
|
||||
assert "LLM judge did not return" in v["reasoning"]
|
||||
assert v["judge_model"] == "gpt-5-judge"
|
||||
|
||||
def test_upsert_on_conflict_preserves_user_decision(self, db):
|
||||
"""LOAD-BEARING: a manually-resolved approval (user_decision=
|
||||
``"approved"``) or auto-approve-stamped row (user_decision=
|
||||
``"policy"``/``"blanket"``/etc.) must NOT be clobbered back to
|
||||
``"pending"`` when the late LLM-fallback verdict lands.
|
||||
``IntentVerdict.to_dict()`` doesn't project user_decision, so
|
||||
the upsert's defaulted ``"pending"`` would silently overwrite
|
||||
the real value if user_decision were in the on-conflict
|
||||
SET clause."""
|
||||
db.upsert_intent_verdict(**_make_verdict_kwargs())
|
||||
ok = db.update_intent_verdict("v_001", user_decision="approved")
|
||||
assert ok is True
|
||||
# Simulate the late LLM-fallback delivery — same verdict_id,
|
||||
# default user_decision (the IntentVerdict.to_dict() shape).
|
||||
db.upsert_intent_verdict(
|
||||
**_make_verdict_kwargs(
|
||||
tier="llm_fallback",
|
||||
reasoning="extended (LLM judge did not return a verdict)",
|
||||
judge_model="gpt-5-judge",
|
||||
)
|
||||
)
|
||||
v = db.get_intent_verdict("v_001")
|
||||
assert v is not None
|
||||
assert v["user_decision"] == "approved" # NOT clobbered to "pending"
|
||||
assert v["tier"] == "llm_fallback" # but the upgrade did land
|
||||
|
||||
def test_upsert_on_conflict_preserves_identity_and_carried_fields(self, db):
|
||||
"""Identity columns (ws_id, call_id, func_name, func_args) and
|
||||
carried-verbatim columns (intent_summary, risk_level,
|
||||
confidence, recommendation, evidence, latency_ms) are
|
||||
excluded from the on-conflict SET — verify they aren't
|
||||
changed even when the second upsert passes different values
|
||||
(defensive against a future judge bug that ships divergent
|
||||
carried fields)."""
|
||||
db.upsert_intent_verdict(**_make_verdict_kwargs())
|
||||
db.upsert_intent_verdict(
|
||||
**_make_verdict_kwargs(
|
||||
# Same verdict_id (conflict trigger), divergent everything else.
|
||||
ws_id="ws-different",
|
||||
call_id="tc_different",
|
||||
func_name="bash_v2",
|
||||
func_args='{"command":"rm -rf /"}',
|
||||
intent_summary="totally different summary",
|
||||
risk_level="critical",
|
||||
confidence=0.0,
|
||||
recommendation="deny",
|
||||
evidence='["dangerous"]',
|
||||
latency_ms=99999,
|
||||
# The three fields that DO update.
|
||||
tier="llm_fallback",
|
||||
reasoning="upgraded reasoning",
|
||||
judge_model="judge-v2",
|
||||
)
|
||||
)
|
||||
v = db.get_intent_verdict("v_001")
|
||||
assert v is not None
|
||||
# All preserved from the first upsert (identity + carried).
|
||||
assert v["ws_id"] == "ws-abc"
|
||||
assert v["call_id"] == "tc_001"
|
||||
assert v["func_name"] == "bash"
|
||||
assert v["func_args"] == '{"command":"echo hello"}'
|
||||
assert v["intent_summary"] == "Echo a greeting to stdout"
|
||||
assert v["risk_level"] == "low"
|
||||
assert v["confidence"] == 0.85
|
||||
assert v["recommendation"] == "approve"
|
||||
assert v["evidence"] == '["The command only prints text."]'
|
||||
assert v["latency_ms"] == 2
|
||||
# Only the three updated.
|
||||
assert v["tier"] == "llm_fallback"
|
||||
assert v["reasoning"] == "upgraded reasoning"
|
||||
assert v["judge_model"] == "judge-v2"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bulk insert
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Integration tests for the Phase 9 admin bulk-revoke endpoint.
|
||||
|
||||
POST /v1/api/admin/mcp-servers/{name}/bulk-revoke clears every user's
|
||||
OAuth token for a server (admin-side counterpart to the per-user
|
||||
DELETE /v1/api/mcp/oauth/connections/{server_name} that shipped in
|
||||
Phase 8).
|
||||
|
||||
Coverage:
|
||||
- requires ``admin.mcp`` permission (401/403 without).
|
||||
- 404 when the named server is missing.
|
||||
- 400 when the server's ``auth_type`` is not ``oauth_user``.
|
||||
- 200 + ``rows_deleted`` + ``consented_users_before`` on success.
|
||||
- Audit row written with
|
||||
``upstream_revoke_outcome="bulk_admin_no_upstream"``.
|
||||
- Token rows are gone from ``mcp_user_tokens`` post-call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Mount, Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import admin_mcp_bulk_revoke
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
class _InjectAdminMcp(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id="admin-user",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="config",
|
||||
permissions=frozenset({"read", "write", "approve", "admin.mcp"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class _InjectNoAdminMcp(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id="regular-user",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="jwt",
|
||||
permissions=frozenset({"read", "write", "approve"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
def _build_app(storage: SQLiteBackend, *, with_admin_mcp: bool = True) -> Starlette:
|
||||
mw = _InjectAdminMcp if with_admin_mcp else _InjectNoAdminMcp
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Mount(
|
||||
"/v1",
|
||||
routes=[
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{name}/bulk-revoke",
|
||||
admin_mcp_bulk_revoke,
|
||||
methods=["POST"],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(mw)],
|
||||
)
|
||||
app.state.auth_storage = storage
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path: Any) -> SQLiteBackend:
|
||||
return SQLiteBackend(str(tmp_path / "test.db"))
|
||||
|
||||
|
||||
def _seed_oauth_server(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
name: str = "srv-oauth",
|
||||
server_id: str = "srv-oauth-id",
|
||||
) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id=server_id,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="oauth_user",
|
||||
)
|
||||
|
||||
|
||||
def _seed_static_server(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
name: str = "srv-static",
|
||||
server_id: str = "srv-static-id",
|
||||
) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id=server_id,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="static",
|
||||
)
|
||||
|
||||
|
||||
def _seed_user_tokens(backend: SQLiteBackend, server_name: str, users: int) -> None:
|
||||
for i in range(users):
|
||||
backend.create_mcp_user_token(
|
||||
f"user-{i}",
|
||||
server_name,
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=None,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
|
||||
|
||||
def test_requires_admin_mcp_permission(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage, with_admin_mcp=False))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_404_on_missing_server(storage: SQLiteBackend) -> None:
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/never-existed/bulk-revoke")
|
||||
assert resp.status_code == 404
|
||||
assert resp.json() == {"error": "No such server"}
|
||||
|
||||
|
||||
def test_400_on_static_server(storage: SQLiteBackend) -> None:
|
||||
_seed_static_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-static/bulk-revoke")
|
||||
assert resp.status_code == 400
|
||||
body = resp.json()
|
||||
assert "oauth_user" in body["error"]
|
||||
|
||||
|
||||
def test_400_on_invalid_server_name(storage: SQLiteBackend) -> None:
|
||||
# double-underscore is reserved for the prefixed-tool-name encoding.
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/bad__name/bulk-revoke")
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_200_on_success_with_no_consented_users(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["status"] == "ok"
|
||||
assert body["rows_deleted"] == 0
|
||||
assert body["consented_users_before"] == 0
|
||||
|
||||
|
||||
def test_200_clears_all_user_tokens(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_user_tokens(storage, "srv-oauth", users=3)
|
||||
# Token for another server must survive the bulk-revoke.
|
||||
_seed_oauth_server(storage, name="srv-other", server_id="srv-other-id")
|
||||
_seed_user_tokens(storage, "srv-other", users=2)
|
||||
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["status"] == "ok"
|
||||
assert body["rows_deleted"] == 3
|
||||
assert body["consented_users_before"] == 3
|
||||
|
||||
# Target server's tokens are gone; bystander's tokens survive.
|
||||
assert storage.count_mcp_consented_users_by_server("srv-oauth") == 0
|
||||
assert storage.count_mcp_consented_users_by_server("srv-other") == 2
|
||||
|
||||
|
||||
def test_audits_with_bulk_admin_no_upstream(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_user_tokens(storage, "srv-oauth", users=2)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
|
||||
# Pull the most-recent audit row for the bulk_revoked action and
|
||||
# verify it carries the deferral marker.
|
||||
events = storage.list_audit_events(limit=10)
|
||||
bulk_rows = [e for e in events if e.get("action") == "mcp_server.oauth.bulk_revoked"]
|
||||
assert len(bulk_rows) == 1
|
||||
detail = bulk_rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("upstream_revoke_outcome") == "bulk_admin_no_upstream"
|
||||
assert detail.get("rows_deleted") == 2
|
||||
assert detail.get("consented_users_before") == 2
|
||||
assert detail.get("name") == "srv-oauth"
|
||||
+146
-1
@@ -788,7 +788,11 @@ class TestSessionIntegration:
|
||||
assert call_id == "call_789"
|
||||
assert output == "result text"
|
||||
mock_mcp.call_tool_sync.assert_called_once_with(
|
||||
"mcp__test__search", {"query": "hello"}, user_id=None, timeout=30
|
||||
"mcp__test__search",
|
||||
{"query": "hello"},
|
||||
user_id=None,
|
||||
timeout=30,
|
||||
is_interactive_for_consent=True,
|
||||
)
|
||||
|
||||
def test_exec_mcp_tool_error(self, tmp_db):
|
||||
@@ -1056,6 +1060,147 @@ class TestRefreshServer:
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
class TestLastRefreshTracking:
|
||||
"""Phase 9 admin status pill — ``_last_refresh`` is written on every
|
||||
refresh path so the admin UI reflects manual-refresh AND auto-
|
||||
reconnect outcomes uniformly. This test class pins the contract.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _seed_minimal(mgr: MCPClientManager, name: str = "srv") -> MagicMock:
|
||||
mock_session = MagicMock()
|
||||
mock_session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
|
||||
mock_session.list_resources = AsyncMock(return_value=MagicMock(resources=[]))
|
||||
mock_session.list_resource_templates = AsyncMock(
|
||||
return_value=MagicMock(resourceTemplates=[])
|
||||
)
|
||||
mock_session.list_prompts = AsyncMock(return_value=MagicMock(prompts=[]))
|
||||
_seed_static_state(
|
||||
mgr,
|
||||
name,
|
||||
session=mock_session,
|
||||
tools=[],
|
||||
supports_resources=True,
|
||||
supports_prompts=True,
|
||||
)
|
||||
return mock_session
|
||||
|
||||
def test_last_refresh_written_on_success(self) -> None:
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
self._seed_minimal(mgr)
|
||||
assert "srv" not in mgr._last_refresh
|
||||
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
ts, outcome = entry
|
||||
assert outcome == "ok"
|
||||
assert isinstance(ts, float) and ts > 0
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_last_refresh_written_on_tool_refresh_failure(self) -> None:
|
||||
"""When ``_refresh_server_tools`` raises, the outcome reflects
|
||||
the exception class and the exception still propagates."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
mock_session = self._seed_minimal(mgr)
|
||||
mock_session.list_tools = AsyncMock(side_effect=RuntimeError("upstream down"))
|
||||
|
||||
with pytest.raises(RuntimeError, match="upstream down"):
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
_, outcome = entry
|
||||
assert outcome == "error:RuntimeError"
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_last_refresh_records_first_exception_when_multiple_fail(
|
||||
self,
|
||||
) -> None:
|
||||
"""``return_exceptions=True`` lets sibling tasks complete; the
|
||||
outcome reflects the FIRST exception encountered."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
mock_session = self._seed_minimal(mgr)
|
||||
# Tools succeeds; resources raises first (gather preserves
|
||||
# argument order in its results list, so resources is the
|
||||
# first failure regardless of which awaitable finished first
|
||||
# in wall-clock terms).
|
||||
mock_session.list_resources = AsyncMock(side_effect=ValueError("res boom"))
|
||||
mock_session.list_prompts = AsyncMock(side_effect=KeyError("prompts boom"))
|
||||
|
||||
with pytest.raises((ValueError, KeyError)):
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
_, outcome = entry
|
||||
# Either of the two failing tasks could be "first" in
|
||||
# gather's results list ordering — the order is positional
|
||||
# so resources (arg #2) comes before prompts (arg #3).
|
||||
assert outcome == "error:ValueError"
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_refresh_all_overwrites_stale_ok_on_reconnect_failure(
|
||||
self,
|
||||
) -> None:
|
||||
"""The chokepoint bug-1 fix: a prior successful refresh's ``'ok'``
|
||||
entry MUST be overwritten when a subsequent reconnect fails —
|
||||
otherwise the admin pill shows misleading "ok" while the server
|
||||
is in fact broken."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
# Server is configured but has no live session — _refresh_all
|
||||
# routes to the reconnect branch.
|
||||
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
|
||||
# Pre-seed a stale "ok" from an earlier successful refresh.
|
||||
mgr._last_refresh["srv"] = (1000.0, "ok")
|
||||
|
||||
async def _raise(*_a: object, **_kw: object) -> None:
|
||||
raise ConnectionError("reconnect failed")
|
||||
|
||||
mgr._connect_one = _raise # type: ignore[assignment]
|
||||
|
||||
await mgr._refresh_all("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
ts, outcome = entry
|
||||
# Outcome reflects the new failure, not the stale ok.
|
||||
assert outcome == "error:ConnectionError"
|
||||
assert ts > 1000.0
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_get_server_status_surfaces_last_refresh_fields(self) -> None:
|
||||
"""``get_server_status`` surfaces ``last_refresh_at`` and
|
||||
``last_refresh_outcome`` for the admin pill — null when no
|
||||
refresh has occurred yet, populated after one."""
|
||||
mgr = MCPClientManager({})
|
||||
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
|
||||
|
||||
# No refresh yet — fields must be present and null so the JS
|
||||
# renderer can branch on absence cleanly.
|
||||
status = mgr.get_server_status("srv")
|
||||
assert status["last_refresh_at"] is None
|
||||
assert status["last_refresh_outcome"] is None
|
||||
|
||||
# Populate the tuple directly and re-read.
|
||||
mgr._last_refresh["srv"] = (12345.5, "ok")
|
||||
status = mgr.get_server_status("srv")
|
||||
assert status["last_refresh_at"] == 12345.5
|
||||
assert status["last_refresh_outcome"] == "ok"
|
||||
|
||||
|
||||
class TestListeners:
|
||||
def test_add_and_notify(self):
|
||||
mgr = MCPClientManager({})
|
||||
|
||||
@@ -612,6 +612,97 @@ class TestCallback:
|
||||
assert plain is not None
|
||||
assert plain["refresh_token"] is None
|
||||
|
||||
def test_callback_clears_pending_consent_on_success(
|
||||
self, storage: SQLiteBackend, http_client_mock: MagicMock
|
||||
) -> None:
|
||||
"""Successful callback must drop any ``mcp_pending_consent`` rows
|
||||
for the just-consented ``(user, server)`` (Phase 9 lifecycle
|
||||
contract). Regression guard for the dashboard-stays-stale-after-
|
||||
consent invariant.
|
||||
"""
|
||||
_seed_oauth_user_server(storage)
|
||||
self._seed_pending(storage)
|
||||
# Seed a deferred-consent record that a prior non-interactive run
|
||||
# would have left behind. Plus a cross-tenant record that must
|
||||
# NOT be touched.
|
||||
storage.upsert_mcp_pending_consent(
|
||||
user_id="user-1",
|
||||
server_name="srv-oauth",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id="ws-1",
|
||||
last_tool_call_id="tool-1",
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
storage.upsert_mcp_pending_consent(
|
||||
user_id="other-user",
|
||||
server_name="srv-oauth",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
token_store = _make_token_store(storage)
|
||||
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
|
||||
http_client_mock.post.return_value = _mk_response(
|
||||
200,
|
||||
{"access_token": "opaque-aaa", "expires_in": 3600},
|
||||
)
|
||||
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
with _public_addr_patch():
|
||||
resp = client.get(
|
||||
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
|
||||
follow_redirects=False,
|
||||
)
|
||||
assert resp.status_code == 302
|
||||
# Callback completed → user-1's deferred-consent row was cleared.
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
# Cross-tenant row survives — clear is per-(user, server).
|
||||
other = storage.list_mcp_pending_consent_by_user("other-user")
|
||||
assert len(other) == 1
|
||||
assert other[0]["server_name"] == "srv-oauth"
|
||||
|
||||
def test_callback_storage_failure_does_not_block_redirect(
|
||||
self, storage: SQLiteBackend, http_client_mock: MagicMock
|
||||
) -> None:
|
||||
"""If the post-persist ``delete_mcp_pending_consent`` raises, the
|
||||
callback's redirect still completes (best-effort contract). The
|
||||
stale badge is preferred over a broken consent flow.
|
||||
"""
|
||||
_seed_oauth_user_server(storage)
|
||||
self._seed_pending(storage)
|
||||
token_store = _make_token_store(storage)
|
||||
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
|
||||
http_client_mock.post.return_value = _mk_response(
|
||||
200,
|
||||
{"access_token": "opaque-aaa", "expires_in": 3600},
|
||||
)
|
||||
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
original_delete = storage.delete_mcp_pending_consent
|
||||
|
||||
def _raise(*_a: Any, **_kw: Any) -> bool:
|
||||
raise RuntimeError("storage offline")
|
||||
|
||||
storage.delete_mcp_pending_consent = _raise # type: ignore[method-assign]
|
||||
try:
|
||||
with _public_addr_patch():
|
||||
resp = client.get(
|
||||
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
|
||||
follow_redirects=False,
|
||||
)
|
||||
finally:
|
||||
storage.delete_mcp_pending_consent = original_delete # type: ignore[method-assign]
|
||||
|
||||
assert resp.status_code == 302
|
||||
# Token persistence still succeeded — the user-visible contract.
|
||||
plain = token_store.get_user_token("user-1", "srv-oauth")
|
||||
assert plain is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 503 paths when mcp_token_store is None
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
"""Boundary tests for the Phase 9 pending-consent write path.
|
||||
|
||||
Drives ``MCPClientManager._dispatch_pool_sync`` (and the helper it
|
||||
calls, ``_record_pending_consent_best_effort``) and asserts that
|
||||
deferred-consent records reach storage only on non-interactive callers.
|
||||
|
||||
Per ``feedback_tests_through_boundaries.md``, at least one test must
|
||||
drive the real sync dispatcher → real ``_is_structured_error`` →
|
||||
real ``_record_pending_consent_best_effort`` plumb-through; the
|
||||
``_helpers`` unit tests below cover the classifier in isolation, but
|
||||
the end-to-end test is the structural gate that catches
|
||||
plumb-through regressions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import threading
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import make_mcp_token_cipher
|
||||
from turnstone.core.mcp_client import (
|
||||
_PENDING_CONSENT_PERSIST_CODES,
|
||||
MCPClientManager,
|
||||
_parse_pending_consent_envelope,
|
||||
)
|
||||
from turnstone.core.mcp_crypto import MCPTokenStore
|
||||
from turnstone.core.mcp_oauth import TokenLookupResult
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper-level unit tests (cheap, no event loop)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseEnvelope:
|
||||
def test_consent_required_no_scopes(self) -> None:
|
||||
env = json.dumps({"error": {"code": "mcp_consent_required", "server": "x", "detail": "d"}})
|
||||
assert _parse_pending_consent_envelope(env) == ("mcp_consent_required", None)
|
||||
|
||||
def test_insufficient_scope_with_scopes(self) -> None:
|
||||
env = json.dumps(
|
||||
{
|
||||
"error": {
|
||||
"code": "mcp_insufficient_scope",
|
||||
"server": "x",
|
||||
"detail": "d",
|
||||
"scopes_required": ["read", "write"],
|
||||
}
|
||||
}
|
||||
)
|
||||
assert _parse_pending_consent_envelope(env) == (
|
||||
"mcp_insufficient_scope",
|
||||
["read", "write"],
|
||||
)
|
||||
|
||||
def test_operator_codes_filtered(self) -> None:
|
||||
# Key-unknown / url-insecure / *_forbidden are operator-actionable,
|
||||
# NOT user-consent-shaped. They must not produce pending-consent
|
||||
# rows, regardless of whether the caller is interactive.
|
||||
for code in (
|
||||
"mcp_token_undecryptable_key_unknown",
|
||||
"mcp_oauth_url_insecure",
|
||||
"mcp_tool_call_forbidden",
|
||||
"mcp_resource_read_forbidden",
|
||||
"mcp_prompt_get_forbidden",
|
||||
):
|
||||
env = json.dumps({"error": {"code": code, "server": "x", "detail": "d"}})
|
||||
assert _parse_pending_consent_envelope(env) is None, code
|
||||
|
||||
def test_malformed_json_returns_none(self) -> None:
|
||||
assert _parse_pending_consent_envelope("not json") is None
|
||||
assert _parse_pending_consent_envelope("") is None
|
||||
|
||||
def test_persist_codes_set_is_expected(self) -> None:
|
||||
# Pin the contract — adding a new persistable code here is a
|
||||
# deliberate design decision and should require a test update.
|
||||
assert {
|
||||
"mcp_consent_required",
|
||||
"mcp_insufficient_scope",
|
||||
} == _PENDING_CONSENT_PERSIST_CODES
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end plumb-through (drives _dispatch_pool_sync)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_oauth_server(backend: Any, *, name: str = "pool-srv") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com/mcp",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-" + name, auth_type="oauth_user")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def running_loop_mgr():
|
||||
cfg: dict[str, Any] = {}
|
||||
mgr = MCPClientManager(cfg)
|
||||
loop = asyncio.new_event_loop()
|
||||
thread = threading.Thread(target=loop.run_forever, daemon=True, name="phase9-test-loop")
|
||||
thread.start()
|
||||
mgr._loop = loop
|
||||
try:
|
||||
yield mgr, loop, thread
|
||||
finally:
|
||||
|
||||
async def _drain(m: MCPClientManager) -> None:
|
||||
task = m._user_pool_eviction_task
|
||||
if task is not None:
|
||||
task.cancel()
|
||||
with contextlib.suppress(BaseException):
|
||||
await task
|
||||
m._user_pool_eviction_task = None
|
||||
|
||||
with contextlib.suppress(Exception):
|
||||
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
|
||||
loop.call_soon_threadsafe(loop.stop)
|
||||
thread.join(timeout=2)
|
||||
|
||||
|
||||
def _wire_mgr(mgr: MCPClientManager, backend: Any) -> None:
|
||||
cipher = make_mcp_token_cipher()
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
app_state = SimpleNamespace(
|
||||
auth_storage=backend,
|
||||
mcp_token_store=MCPTokenStore(backend, cipher, node_id="test"),
|
||||
mcp_oauth_http_client=MagicMock(),
|
||||
mcp_oauth_refresh_locks={},
|
||||
mcp_oauth_metadata_cache={},
|
||||
)
|
||||
mgr.set_storage(backend)
|
||||
mgr.set_app_state(app_state)
|
||||
|
||||
|
||||
def test_dispatch_persists_pending_for_non_interactive_caller(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Non-interactive caller hits ``mcp_consent_required`` → a
|
||||
``mcp_pending_consent`` row appears for ``(user_id, server_name)``."""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
|
||||
# Structured error envelope surfaces as RuntimeError to the caller.
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_consent_required"
|
||||
|
||||
# Persistent row written for the dashboard badge.
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
assert r["user_id"] == "user-a"
|
||||
assert r["server_name"] == "pool-srv"
|
||||
assert r["error_code"] == "mcp_consent_required"
|
||||
assert r["occurrence_count"] == 1
|
||||
|
||||
|
||||
def test_dispatch_does_not_persist_for_interactive_caller(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Interactive caller hits the same error path → NO row written.
|
||||
|
||||
Interactive (WEB / CLI) sessions surface the consent prompt in-flight
|
||||
via the Phase 8 SSE renderer; persisting would just produce
|
||||
immediately-stale dashboard badges.
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError),
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=True,
|
||||
)
|
||||
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
|
||||
|
||||
def test_dispatch_returns_envelope_unchanged_on_storage_failure(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""When ``upsert_mcp_pending_consent`` raises, the agent-observable
|
||||
contract is unchanged: the structured-error ``RuntimeError`` still
|
||||
surfaces with the original ``mcp_consent_required`` code. The doc-
|
||||
string promises best-effort persistence; this test pins that
|
||||
promise so a regression that propagates the storage exception would
|
||||
fail visibly.
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
original_upsert = backend.upsert_mcp_pending_consent
|
||||
|
||||
def _raise(*_a: Any, **_kw: Any) -> None:
|
||||
raise RuntimeError("storage offline")
|
||||
|
||||
backend.upsert_mcp_pending_consent = _raise # type: ignore[method-assign]
|
||||
try:
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
finally:
|
||||
backend.upsert_mcp_pending_consent = original_upsert # type: ignore[method-assign]
|
||||
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_consent_required"
|
||||
|
||||
|
||||
def test_dispatch_does_not_persist_for_operator_actionable_code(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Decrypt-failure → operator-actionable; even non-interactive callers
|
||||
must NOT produce a user-facing pending-consent record (the user can't
|
||||
resolve this by re-consenting).
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _decrypt_failure(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="decrypt_failure")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_decrypt_failure,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_token_undecryptable_key_unknown"
|
||||
# The operator-actionable code does NOT produce a pending-consent row.
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
@@ -0,0 +1,259 @@
|
||||
"""HTTP tests for the Phase 9 pending-consent endpoints.
|
||||
|
||||
Covers:
|
||||
- ``GET /v1/api/mcp/oauth/pending`` (install gate + read path)
|
||||
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` (single clear)
|
||||
- ``DELETE /v1/api/mcp/oauth/pending`` (bulk clear)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Mount, Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.mcp_oauth import (
|
||||
handle_mcp_oauth_clear_all_pending,
|
||||
handle_mcp_oauth_clear_pending,
|
||||
handle_mcp_oauth_list_pending,
|
||||
)
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
class _InjectAuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Stamp a fixed authenticated user on every request."""
|
||||
|
||||
def __init__(self, app: Any, user_id: str = "user-1") -> None:
|
||||
super().__init__(app)
|
||||
self._user_id = user_id
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=self._user_id,
|
||||
scopes=frozenset({"write"}),
|
||||
token_source="config",
|
||||
permissions=frozenset({"read", "write"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
def _build_app(storage: SQLiteBackend, *, user_id: str = "user-1") -> Starlette:
|
||||
class _Mw(_InjectAuthMiddleware):
|
||||
def __init__(self, app: Any) -> None:
|
||||
super().__init__(app, user_id=user_id)
|
||||
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Mount(
|
||||
"/v1",
|
||||
routes=[
|
||||
Route("/api/mcp/oauth/pending", handle_mcp_oauth_list_pending),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending",
|
||||
handle_mcp_oauth_clear_all_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending/{server_name}",
|
||||
handle_mcp_oauth_clear_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_Mw)],
|
||||
)
|
||||
app.state.auth_storage = storage
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path: Any) -> SQLiteBackend:
|
||||
backend = SQLiteBackend(str(tmp_path / "test.db"))
|
||||
backend.create_user("user-1", "user1", "User One", "hash")
|
||||
backend.create_user("user-2", "user2", "User Two", "hash")
|
||||
return backend
|
||||
|
||||
|
||||
def _seed_oauth_server(backend: SQLiteBackend, *, name: str = "srv-x") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-id-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="oauth_user",
|
||||
)
|
||||
|
||||
|
||||
def _seed_pending(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
user_id: str = "user-1",
|
||||
server_name: str = "srv-x",
|
||||
error_code: str = "mcp_consent_required",
|
||||
now_iso: str = "2026-05-11T12:00:00",
|
||||
) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id=user_id,
|
||||
server_name=server_name,
|
||||
error_code=error_code,
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=now_iso,
|
||||
)
|
||||
|
||||
|
||||
class TestListPending:
|
||||
def test_install_gate_short_circuits_on_no_oauth_servers(self, storage: SQLiteBackend) -> None:
|
||||
# Seed a pending row but NO oauth_user MCP server — the gate
|
||||
# must short-circuit to {pending: 0} regardless.
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"pending": 0, "servers": []}
|
||||
|
||||
def test_lists_pending_records_for_authenticated_user(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["pending"] == 1
|
||||
assert len(body["servers"]) == 1
|
||||
assert body["servers"][0]["server_name"] == "srv-x"
|
||||
assert body["servers"][0]["error_code"] == "mcp_consent_required"
|
||||
|
||||
def test_does_not_leak_cross_user_records(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage, user_id="user-2")
|
||||
client = TestClient(_build_app(storage, user_id="user-1"))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"pending": 0, "servers": []}
|
||||
|
||||
|
||||
class TestClearPending:
|
||||
def test_delete_single(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
|
||||
def test_delete_missing_still_returns_204(self, storage: SQLiteBackend) -> None:
|
||||
# Idempotent — must not leak cross-user existence info via 404.
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
|
||||
assert resp.status_code == 204
|
||||
|
||||
def test_delete_does_not_touch_cross_user_rows(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage, user_id="user-1")
|
||||
_seed_pending(storage, user_id="user-2")
|
||||
client = TestClient(_build_app(storage, user_id="user-1"))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
# User-2's row survives.
|
||||
assert len(storage.list_mcp_pending_consent_by_user("user-2")) == 1
|
||||
|
||||
|
||||
class TestAuditTrail:
|
||||
def test_single_dismiss_audits(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "single"
|
||||
assert detail.get("cleared") == 1
|
||||
|
||||
def test_single_dismiss_audits_even_when_no_row_existed(self, storage: SQLiteBackend) -> None:
|
||||
# Cross-tenant non-observability requires a 204 in the never-existed
|
||||
# case — the audit row distinguishes a real dismiss from a stuffed
|
||||
# attempt by recording ``cleared=0``.
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
|
||||
assert resp.status_code == 204
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "single"
|
||||
assert detail.get("cleared") == 0
|
||||
|
||||
def test_bulk_dismiss_audits(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_oauth_server(storage, name="srv-y")
|
||||
_seed_pending(storage, server_name="srv-x")
|
||||
_seed_pending(storage, server_name="srv-y")
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 2}
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "bulk"
|
||||
assert detail.get("cleared") == 2
|
||||
|
||||
|
||||
class TestClearAllPending:
|
||||
def test_bulk_clear(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_oauth_server(storage, name="srv-y")
|
||||
_seed_pending(storage, server_name="srv-x")
|
||||
_seed_pending(storage, server_name="srv-y")
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 2}
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
|
||||
def test_bulk_clear_zero_when_empty(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 0}
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Storage CRUD tests for the Phase 9 ``mcp_pending_consent`` table.
|
||||
|
||||
Validates protocol additions backing the dashboard pending-consent badge:
|
||||
|
||||
- ``upsert_mcp_pending_consent`` — insert + on-conflict refresh
|
||||
- ``list_mcp_pending_consent_by_user`` — read path
|
||||
- ``delete_mcp_pending_consent`` — single-row clear
|
||||
- ``delete_all_mcp_pending_consent_by_user`` — bulk clear
|
||||
- ``count_mcp_consented_users_by_server`` — admin status pill
|
||||
- ``any_oauth_user_mcp_servers`` — install-level gate
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def _iso(ts: str = "2026-05-11T12:00:00") -> str:
|
||||
return ts
|
||||
|
||||
|
||||
class TestUpsertAndList:
|
||||
def test_insert_round_trip(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required="read write",
|
||||
last_ws_id="ws-1",
|
||||
last_tool_call_id="tool-1",
|
||||
now_iso=_iso(),
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
assert r["user_id"] == "user-a"
|
||||
assert r["server_name"] == "srv-x"
|
||||
assert r["error_code"] == "mcp_consent_required"
|
||||
assert r["scopes_required"] == "read write"
|
||||
assert r["last_ws_id"] == "ws-1"
|
||||
assert r["last_tool_call_id"] == "tool-1"
|
||||
assert r["occurrence_count"] == 1
|
||||
assert r["first_seen_at"] == r["last_seen_at"]
|
||||
|
||||
def test_upsert_bumps_count_and_refreshes_recency(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_insufficient_scope",
|
||||
scopes_required="read",
|
||||
last_ws_id="ws-2",
|
||||
last_tool_call_id="tool-2",
|
||||
now_iso="2026-05-11T13:00:00",
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
# Recency fields refreshed to the second call's values; count bumped.
|
||||
assert r["occurrence_count"] == 2
|
||||
assert r["error_code"] == "mcp_insufficient_scope"
|
||||
assert r["scopes_required"] == "read"
|
||||
assert r["last_ws_id"] == "ws-2"
|
||||
assert r["last_tool_call_id"] == "tool-2"
|
||||
assert r["last_seen_at"] == "2026-05-11T13:00:00"
|
||||
# first_seen_at preserved — that's the load-bearing audit value.
|
||||
assert r["first_seen_at"] == "2026-05-11T12:00:00"
|
||||
|
||||
def test_list_orders_by_last_seen_desc(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-old",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T10:00:00",
|
||||
)
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-new",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T11:00:00",
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert [r["server_name"] for r in rows] == ["srv-new", "srv-old"]
|
||||
|
||||
def test_per_user_isolation(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.list_mcp_pending_consent_by_user("user-b") == []
|
||||
|
||||
|
||||
class TestDelete:
|
||||
def test_delete_single(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
# Second delete returns False (no row).
|
||||
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is False
|
||||
|
||||
def test_delete_missing_returns_false(self, backend) -> None:
|
||||
assert backend.delete_mcp_pending_consent("never", "missing") is False
|
||||
|
||||
def test_delete_all_by_user(self, backend) -> None:
|
||||
for name in ("srv-a", "srv-b", "srv-c"):
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name=name,
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
# Cross-user row that must NOT be touched.
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-b",
|
||||
server_name="srv-z",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
assert len(backend.list_mcp_pending_consent_by_user("user-b")) == 1
|
||||
|
||||
|
||||
class TestCountConsentedUsersByServer:
|
||||
def _seed_server(self, backend, name: str = "srv-x") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-id-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com/mcp",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-id-" + name, auth_type="oauth_user")
|
||||
|
||||
def test_counts_distinct_non_expired_users(self, backend) -> None:
|
||||
self._seed_server(backend)
|
||||
future = "2099-01-01T00:00:00"
|
||||
backend.create_mcp_user_token(
|
||||
"alice",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=future,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
backend.create_mcp_user_token(
|
||||
"bob",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=None, # null treated as non-expired
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
# Different server — must not count.
|
||||
self._seed_server(backend, name="srv-y")
|
||||
backend.create_mcp_user_token(
|
||||
"carol",
|
||||
"srv-y",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=future,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
assert backend.count_mcp_consented_users_by_server("srv-x") == 2
|
||||
assert backend.count_mcp_consented_users_by_server("srv-y") == 1
|
||||
|
||||
def test_excludes_expired(self, backend) -> None:
|
||||
self._seed_server(backend)
|
||||
backend.create_mcp_user_token(
|
||||
"alice",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at="2020-01-01T00:00:00", # well in the past
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
assert backend.count_mcp_consented_users_by_server("srv-x") == 0
|
||||
|
||||
def test_zero_when_no_rows(self, backend) -> None:
|
||||
assert backend.count_mcp_consented_users_by_server("missing") == 0
|
||||
|
||||
|
||||
class TestInstallGate:
|
||||
def test_any_oauth_user_returns_false_on_empty(self, backend) -> None:
|
||||
assert backend.any_oauth_user_mcp_servers() is False
|
||||
|
||||
def test_any_oauth_user_ignores_static_rows(self, backend) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-1",
|
||||
name="static-only",
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com",
|
||||
headers='{"Authorization": "Bearer x"}',
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
assert backend.any_oauth_user_mcp_servers() is False
|
||||
|
||||
def test_any_oauth_user_returns_true_when_one_exists(self, backend) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-2",
|
||||
name="oauth-srv",
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-2", auth_type="oauth_user")
|
||||
assert backend.any_oauth_user_mcp_servers() is True
|
||||
@@ -212,3 +212,73 @@ class TestModelDefinitionStorage:
|
||||
m = db.get_model_definition(did)
|
||||
assert m is not None
|
||||
assert m["temperature"] is None
|
||||
|
||||
def test_reasoning_flags_default(self, db: SQLiteBackend) -> None:
|
||||
"""surface_persisted_reasoning defaults True; replay_reasoning_to_model defaults False."""
|
||||
did = _make_id()
|
||||
db.create_model_definition(definition_id=did, alias="reason-default", model="gpt-5")
|
||||
m = db.get_model_definition(did)
|
||||
assert m is not None
|
||||
assert m["surface_persisted_reasoning"] is True
|
||||
assert m["replay_reasoning_to_model"] is False
|
||||
|
||||
def test_create_with_explicit_reasoning_flags(self, db: SQLiteBackend) -> None:
|
||||
did = _make_id()
|
||||
db.create_model_definition(
|
||||
definition_id=did,
|
||||
alias="reason-explicit",
|
||||
model="claude-opus-4-7",
|
||||
surface_persisted_reasoning=False,
|
||||
replay_reasoning_to_model=True,
|
||||
)
|
||||
m = db.get_model_definition(did)
|
||||
assert m is not None
|
||||
assert m["surface_persisted_reasoning"] is False
|
||||
assert m["replay_reasoning_to_model"] is True
|
||||
# Same values must round-trip via the alias lookup too.
|
||||
m_alias = db.get_model_definition_by_alias("reason-explicit")
|
||||
assert m_alias is not None
|
||||
assert m_alias["surface_persisted_reasoning"] is False
|
||||
assert m_alias["replay_reasoning_to_model"] is True
|
||||
|
||||
def test_update_surface_persisted_reasoning(self, db: SQLiteBackend) -> None:
|
||||
did = _make_id()
|
||||
db.create_model_definition(definition_id=did, alias="upd-persist", model="gpt-5")
|
||||
ok = db.update_model_definition(did, surface_persisted_reasoning=False)
|
||||
assert ok is True
|
||||
m = db.get_model_definition(did)
|
||||
assert m is not None
|
||||
assert m["surface_persisted_reasoning"] is False
|
||||
assert m["replay_reasoning_to_model"] is False # untouched
|
||||
|
||||
def test_update_replay_reasoning_to_model(self, db: SQLiteBackend) -> None:
|
||||
did = _make_id()
|
||||
db.create_model_definition(definition_id=did, alias="upd-replay", model="gpt-5")
|
||||
ok = db.update_model_definition(did, replay_reasoning_to_model=True)
|
||||
assert ok is True
|
||||
m = db.get_model_definition(did)
|
||||
assert m is not None
|
||||
assert m["surface_persisted_reasoning"] is True # untouched
|
||||
assert m["replay_reasoning_to_model"] is True
|
||||
|
||||
def test_list_returns_reasoning_flags(self, db: SQLiteBackend) -> None:
|
||||
db.create_model_definition(
|
||||
definition_id=_make_id(),
|
||||
alias="list-a",
|
||||
model="gpt-5",
|
||||
surface_persisted_reasoning=True,
|
||||
replay_reasoning_to_model=False,
|
||||
)
|
||||
db.create_model_definition(
|
||||
definition_id=_make_id(),
|
||||
alias="list-b",
|
||||
model="claude-opus-4-7",
|
||||
surface_persisted_reasoning=False,
|
||||
replay_reasoning_to_model=True,
|
||||
)
|
||||
models = db.list_model_definitions()
|
||||
by_alias = {m["alias"]: m for m in models}
|
||||
assert by_alias["list-a"]["surface_persisted_reasoning"] is True
|
||||
assert by_alias["list-a"]["replay_reasoning_to_model"] is False
|
||||
assert by_alias["list-b"]["surface_persisted_reasoning"] is False
|
||||
assert by_alias["list-b"]["replay_reasoning_to_model"] is True
|
||||
|
||||
@@ -76,6 +76,23 @@ class TestModelConfig:
|
||||
assert cfg.temperature == 0.0
|
||||
assert cfg.temperature is not None
|
||||
|
||||
def test_reasoning_flags_default(self) -> None:
|
||||
cfg = ModelConfig(alias="x", base_url="x", api_key="x", model="x")
|
||||
assert cfg.surface_persisted_reasoning is True
|
||||
assert cfg.replay_reasoning_to_model is False
|
||||
|
||||
def test_reasoning_flags_set(self) -> None:
|
||||
cfg = ModelConfig(
|
||||
alias="x",
|
||||
base_url="x",
|
||||
api_key="x",
|
||||
model="x",
|
||||
surface_persisted_reasoning=False,
|
||||
replay_reasoning_to_model=True,
|
||||
)
|
||||
assert cfg.surface_persisted_reasoning is False
|
||||
assert cfg.replay_reasoning_to_model is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ModelRegistry
|
||||
@@ -314,7 +331,11 @@ class TestLoadModelRegistry:
|
||||
api_key="dummy",
|
||||
model="local-model",
|
||||
)
|
||||
assert reg.count == 2 # "openai" + "default"
|
||||
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
|
||||
# populates configs — only the explicit alias survives.
|
||||
assert reg.count == 1
|
||||
assert reg.has_alias("openai")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.default == "openai"
|
||||
_, model, _ = reg.resolve()
|
||||
assert model == "gpt-4o"
|
||||
@@ -545,7 +566,12 @@ class TestLoadModelRegistryWithDB:
|
||||
assert cfg.source == "config"
|
||||
|
||||
def test_db_only_models_coexist(self) -> None:
|
||||
"""DB models coexist alongside config.toml models."""
|
||||
"""DB models coexist alongside config.toml models.
|
||||
|
||||
The CLI ``"default"`` shim is suppressed when DB / config models
|
||||
already populate the registry — see
|
||||
``test_cli_default_shim_skipped_when_db_models_present``.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
@@ -569,7 +595,7 @@ class TestLoadModelRegistryWithDB:
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.has_alias("db-only")
|
||||
assert reg.has_alias("config-only")
|
||||
assert reg.has_alias("default")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.get_config("db-only").source == "db"
|
||||
assert reg.get_config("config-only").source == "config"
|
||||
|
||||
@@ -589,10 +615,12 @@ class TestLoadModelRegistryWithDB:
|
||||
}
|
||||
]
|
||||
)
|
||||
# The CLI default shim is suppressed when the DB row populates
|
||||
# configs, so only the DB-sourced alias exists here.
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.get_config("from-db").source == "db"
|
||||
assert reg.get_config("default").source == ""
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_disabled_db_models_excluded(self) -> None:
|
||||
"""Disabled DB models are not loaded."""
|
||||
@@ -686,6 +714,53 @@ class TestLoadModelRegistryWithDB:
|
||||
assert cfg.max_tokens is None
|
||||
assert cfg.reasoning_effort is None
|
||||
|
||||
def test_db_reasoning_flags_loaded(self) -> None:
|
||||
"""Per-model reasoning flags from DB are carried in ModelConfig."""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
"alias": "anth-thinking",
|
||||
"model": "claude-opus-4-7",
|
||||
"provider": "anthropic",
|
||||
"base_url": "",
|
||||
"api_key": "sk-anth",
|
||||
"context_window": 200000,
|
||||
"capabilities": "{}",
|
||||
"enabled": True,
|
||||
"surface_persisted_reasoning": False,
|
||||
"replay_reasoning_to_model": True,
|
||||
}
|
||||
]
|
||||
)
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
cfg = reg.get_config("anth-thinking")
|
||||
assert cfg.surface_persisted_reasoning is False
|
||||
assert cfg.replay_reasoning_to_model is True
|
||||
|
||||
def test_db_reasoning_flags_default_when_absent(self) -> None:
|
||||
"""Pre-052 rows without the columns degrade to dataclass defaults."""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
"alias": "legacy-row",
|
||||
"model": "gpt-5",
|
||||
"provider": "openai",
|
||||
"base_url": "",
|
||||
"api_key": "",
|
||||
"context_window": 32768,
|
||||
"capabilities": "{}",
|
||||
"enabled": True,
|
||||
# surface_persisted_reasoning + replay_reasoning_to_model intentionally absent
|
||||
}
|
||||
]
|
||||
)
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
cfg = reg.get_config("legacy-row")
|
||||
assert cfg.surface_persisted_reasoning is True
|
||||
assert cfg.replay_reasoning_to_model is False
|
||||
|
||||
def test_db_default_alias_not_clobbered(self) -> None:
|
||||
"""DB model with alias='default' is not overwritten by CLI args."""
|
||||
storage = _MockStorage(
|
||||
@@ -869,6 +944,8 @@ class _FakeUI:
|
||||
self.infos: list[str] = []
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_turn_start(self) -> None: ...
|
||||
def on_turn_committed(self) -> None: ...
|
||||
def on_thinking_start(self) -> None: ...
|
||||
def on_thinking_stop(self) -> None: ...
|
||||
def on_reasoning_token(self, text: str) -> None: ...
|
||||
@@ -1609,6 +1686,58 @@ class TestLoadModelRegistryDBOnly:
|
||||
reg = load_model_registry(model="", storage=storage)
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
|
||||
"""An auto-detected ``--model`` does NOT synthesise a ``default``
|
||||
alias when the DB already contributes models.
|
||||
|
||||
Regression for the silent bypass of ``model.task_alias`` /
|
||||
``model.plan_alias``: a synthesised ``default`` aliased to whatever
|
||||
``--base-url`` was at boot leaks into the LLM-visible alias list,
|
||||
and the LLM picks it for ``task_agent(model="default")`` — which
|
||||
then routes around the operator-configured per-role default.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
"alias": "gh200",
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"provider": "openai",
|
||||
"base_url": "http://gh200:8000/v1",
|
||||
"api_key": "sk-gh200",
|
||||
"context_window": 1048576,
|
||||
"capabilities": "{}",
|
||||
"enabled": True,
|
||||
}
|
||||
]
|
||||
)
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry(
|
||||
base_url="http://flatspark:8000/v1",
|
||||
api_key="sk-flatspark",
|
||||
model="qwen3.6-35B-A3B", # populated by ``detect_model``
|
||||
storage=storage,
|
||||
)
|
||||
assert reg.has_alias("gh200")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
|
||||
"""Same shim suppression when only ``[models.*]`` populates configs."""
|
||||
fake_cfg: dict[str, Any] = {
|
||||
"models": {"local": {"model": "qwen3-32b"}},
|
||||
}
|
||||
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
|
||||
reg = load_model_registry("http://x/v1", "x", "fallback-model")
|
||||
assert reg.has_alias("local")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
|
||||
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
|
||||
the back-compat ``default`` alias."""
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "lone-model")
|
||||
assert reg.has_alias("default")
|
||||
assert reg.get_config("default").model == "lone-model"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# server._effective_routing / _apply_routing_overrides
|
||||
|
||||
@@ -197,6 +197,7 @@ _EXPECTED_AFFECTING_KEYS = frozenset(
|
||||
"coordinator.model_alias",
|
||||
"coordinator.reasoning_effort",
|
||||
"judge.model",
|
||||
"channels.default_model_alias",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,400 @@
|
||||
"""Tests for the console-side ``NotifyDispatcher``.
|
||||
|
||||
Exercises the dispatcher against the SQLite synthetic-sweep path so the
|
||||
suite runs without a Postgres dependency. The PG path is shaped the
|
||||
same way (same handler invocation semantics) — the only difference is
|
||||
the underlying stream's wake-up source, which is covered separately in
|
||||
``test_storage_notify.py::TestPostgresNotify``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def dispatcher_factory(storage):
|
||||
"""Yield a factory that constructs + tracks dispatchers for teardown."""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
created: list[NotifyDispatcher] = []
|
||||
|
||||
def _make(*, channels: list[str]) -> NotifyDispatcher:
|
||||
d = NotifyDispatcher(storage, channels=channels)
|
||||
created.append(d)
|
||||
return d
|
||||
|
||||
yield _make
|
||||
|
||||
for d in created:
|
||||
d.stop(timeout=2.0)
|
||||
|
||||
|
||||
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
|
||||
"""Poll ``predicate`` until True or timeout. Returns bool."""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return False
|
||||
|
||||
|
||||
def _start_ready(d, *, timeout: float = 5.0) -> None:
|
||||
"""``d.start()`` + assert the listener is actually listening.
|
||||
|
||||
Closes the start-vs-notify race for backends where ``storage.listen``
|
||||
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
|
||||
connection): without the sync, a same-thread ``storage.notify`` can
|
||||
fire before the LISTEN registers and the notification is lost.
|
||||
"""
|
||||
d.start()
|
||||
if not d.wait_until_ready(timeout=timeout):
|
||||
msg = f"dispatcher listener did not open within {timeout}s"
|
||||
raise AssertionError(msg)
|
||||
|
||||
|
||||
class TestSubscribe:
|
||||
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
# Fire a notify via the storage layer — dispatcher delivers to handler.
|
||||
storage.notify("alpha", "hello")
|
||||
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
|
||||
|
||||
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
with pytest.raises(ValueError, match="not declared"):
|
||||
d.subscribe("beta", lambda n: None)
|
||||
|
||||
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
unsub = d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "first")
|
||||
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
|
||||
unsub()
|
||||
# After unsubscribe, the handler no longer fires. Drain old hits
|
||||
# so the next notify-vs-handler-count check is unambiguous.
|
||||
seen.clear()
|
||||
storage.notify("alpha", "second")
|
||||
# Give the dispatcher a beat to deliver if it were going to.
|
||||
time.sleep(0.2)
|
||||
assert not any(n.payload == "second" for n in seen)
|
||||
|
||||
def test_construction_requires_at_least_one_channel(self, storage):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
with pytest.raises(ValueError, match="at least one"):
|
||||
NotifyDispatcher(storage, channels=[])
|
||||
|
||||
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
|
||||
assert d.channels == ["alpha", "beta"]
|
||||
|
||||
|
||||
class TestDispatch:
|
||||
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("alpha", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "shared")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert seen_a[0].payload == "shared"
|
||||
assert seen_b[0].payload == "shared"
|
||||
|
||||
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
survived: list = []
|
||||
|
||||
def _broken(_n):
|
||||
msg = "boom"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
d.subscribe("alpha", _broken)
|
||||
d.subscribe("alpha", lambda n: survived.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "after_broken")
|
||||
# The second handler runs even though the first raised.
|
||||
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
|
||||
|
||||
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha", "beta"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("beta", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "for_a")
|
||||
storage.notify("beta", "for_b")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert all(n.payload == "for_a" for n in seen_a)
|
||||
assert all(n.payload == "for_b" for n in seen_b)
|
||||
|
||||
|
||||
class TestReconnect:
|
||||
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
|
||||
|
||||
Uses a stub storage that owns its own listen stream so the test can
|
||||
drive a controlled stream-error sequence — the SQLite path can't
|
||||
raise :class:`NotifyConnectionError`, and the PG path requires a
|
||||
real database outage to exercise this code, neither of which fits a
|
||||
unit test. The dispatcher's threading and reconcile-pending logic
|
||||
are storage-agnostic — the dispatcher sees the same
|
||||
:class:`NotifyStream` Protocol regardless of backend.
|
||||
"""
|
||||
|
||||
def test_reconcile_fires_after_reopen_not_before(self):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
# State machine: open -> first poll raises NotifyConnectionError
|
||||
# -> dispatcher waits backoff then reopens -> second open's first
|
||||
# poll blocks forever (test stops the dispatcher before then).
|
||||
# The fix: synthetic reconcile fires AFTER the second open
|
||||
# succeeds, not after the first open fails.
|
||||
sequence: list[str] = []
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self, fail_first_poll: bool):
|
||||
self._fail = fail_first_poll
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
if self._fail:
|
||||
self._fail = False
|
||||
sequence.append("poll_raises")
|
||||
msg = "fake-disconnect"
|
||||
raise NotifyConnectionError(msg)
|
||||
sequence.append("poll_returns")
|
||||
# Block until close to simulate a quiet steady-state.
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
self._open_count += 1
|
||||
sequence.append(f"open_{self._open_count}")
|
||||
if self._open_count == 2:
|
||||
reopen_event.set()
|
||||
stream = _StubStream(fail_first_poll=(self._open_count == 1))
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
# Speed up backoff so the reopen happens promptly in the test.
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list[Notify] = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
# Wait for the second open (post-reconnect).
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
|
||||
# Reconcile should be delivered shortly after the reopen.
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
f"no reconcile delivered; sequence={sequence}, got={got}"
|
||||
)
|
||||
# The reconcile must NOT fire before the second open —
|
||||
# if it did, the index of 'open_2' in sequence would
|
||||
# come after any reconcile-emitting work. Check ordering:
|
||||
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
|
||||
# inside the with-block of the SECOND open).
|
||||
ix_open_1 = sequence.index("open_1")
|
||||
ix_raises = sequence.index("poll_raises")
|
||||
ix_open_2 = sequence.index("open_2")
|
||||
assert ix_open_1 < ix_raises < ix_open_2
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
def test_generic_exception_path_also_synthesizes_reconcile(self):
|
||||
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
|
||||
|
||||
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
|
||||
shape, which doesn't go through the stream's exception
|
||||
translator and would hit the generic ``except Exception``
|
||||
branch. Pre-fix, that branch emitted no reconcile.
|
||||
"""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self):
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
self._open_count += 1
|
||||
if self._open_count == 1:
|
||||
# First open raises a generic exception (e.g.
|
||||
# ``psycopg.OperationalError`` from a failed connect)
|
||||
# — landing in the dispatcher's generic except branch.
|
||||
msg = "fake-connect-failure"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
reopen_event.set()
|
||||
stream = _StubStream()
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
"no reconcile delivered after generic-exception recovery"
|
||||
)
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
|
||||
class TestCoalescing:
|
||||
"""Same-channel burst collapses to one handler invocation per batch."""
|
||||
|
||||
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
invocations: list = []
|
||||
# Slow handler to ensure all bursts queue up before the first
|
||||
# call returns — gives the dispatch loop time to coalesce.
|
||||
coalesce_gate = threading.Event()
|
||||
|
||||
def _slow_handler(n):
|
||||
invocations.append(n)
|
||||
coalesce_gate.wait(0.05)
|
||||
|
||||
d.subscribe("alpha", _slow_handler)
|
||||
_start_ready(d)
|
||||
# Burst of 10 notifies on the same channel — should coalesce
|
||||
# down to many fewer handler invocations.
|
||||
for i in range(10):
|
||||
storage.notify("alpha", str(i))
|
||||
# Wait until the dispatch settles (handler is called at least once
|
||||
# and the queue empties).
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if invocations and d._dispatch_queue.empty():
|
||||
time.sleep(0.1) # allow any final coalesced call to land
|
||||
break
|
||||
time.sleep(0.02)
|
||||
coalesce_gate.set()
|
||||
# At least one handler call; well fewer than 10 (coalescing
|
||||
# collapsed the burst). Exact count depends on timing — typical
|
||||
# is 1-2 invocations per burst on a fast machine.
|
||||
assert invocations, "handler never fired"
|
||||
assert len(invocations) < 10, (
|
||||
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
|
||||
)
|
||||
|
||||
|
||||
class TestLifecycle:
|
||||
def test_start_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.start() # No-op, no thread doubling
|
||||
# Single listener + single dispatch thread are spawned regardless.
|
||||
# Inspect by name so we don't depend on the exact thread count of
|
||||
# the test runner.
|
||||
listener_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
|
||||
]
|
||||
dispatch_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
|
||||
]
|
||||
assert len(listener_threads) == 1
|
||||
assert len(dispatch_threads) == 1
|
||||
|
||||
def test_stop_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.stop(timeout=2.0)
|
||||
d.stop(timeout=2.0) # No-op, no error
|
||||
|
||||
def test_stop_without_start_is_noop(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.stop(timeout=1.0) # No-op, no thread to join
|
||||
|
||||
def test_stop_joins_threads(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
# Capture thread references then stop and assert they exited.
|
||||
threads_before = [
|
||||
t
|
||||
for t in threading.enumerate()
|
||||
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
|
||||
]
|
||||
assert threads_before
|
||||
d.stop(timeout=3.0)
|
||||
time.sleep(0.05)
|
||||
for t in threads_before:
|
||||
assert not t.is_alive(), f"{t.name} still alive after stop"
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
@@ -401,8 +402,10 @@ class TestValidation:
|
||||
|
||||
|
||||
class TestValidUntil:
|
||||
"""``valid_until`` predicate: drain re-checks freshness; falsy /
|
||||
raising predicates drop the entry without delivery.
|
||||
"""``valid_until`` predicate: drain re-checks freshness. Falsy
|
||||
predicates drop the entry without delivery and log at ``info``
|
||||
(normal lifecycle outcome); raising predicates drop the entry and
|
||||
log at ``warning`` with ``exc_info`` (misbehaving predicate).
|
||||
"""
|
||||
|
||||
def test_valid_until_true_delivers(self):
|
||||
@@ -411,26 +414,52 @@ class TestValidUntil:
|
||||
out = q.drain({"any"})
|
||||
assert out == [("a", "1", None)]
|
||||
|
||||
def test_valid_until_false_drops_silently(self):
|
||||
def test_valid_until_false_drops_with_info_log(self, caplog: pytest.LogCaptureFixture):
|
||||
q = NudgeQueue()
|
||||
q.enqueue("a", "1", "any", valid_until=lambda: False)
|
||||
out = q.drain({"any"})
|
||||
with caplog.at_level(logging.INFO, logger="turnstone.core.nudge_queue"):
|
||||
out = q.drain({"any"})
|
||||
assert out == []
|
||||
# Already removed from queue (drain partition removes BEFORE
|
||||
# predicate check — falsy doesn't return to queue).
|
||||
assert len(q) == 0
|
||||
# The drop emits a structured info record so a wiring
|
||||
# regression (a predicate that always returns False) is still
|
||||
# observable, without spamming ``warning`` for the routine
|
||||
# lifecycle case where ``valid_until`` is doing its job.
|
||||
# structlog renders the event name + extras into ``msg`` as a
|
||||
# single rendered string, so substring-match like the
|
||||
# ``watch_dispatch.queue_full`` assertion in
|
||||
# tests/test_watch_dispatch.py.
|
||||
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
|
||||
assert len(drops) == 1
|
||||
assert drops[0].levelno == logging.INFO
|
||||
assert "predicate_false" in drops[0].getMessage()
|
||||
assert "'nudge_type': 'a'" in drops[0].getMessage()
|
||||
assert "'channel': 'any'" in drops[0].getMessage()
|
||||
assert "'text_len': 1" in drops[0].getMessage()
|
||||
|
||||
def test_valid_until_exception_drops_silently(self):
|
||||
def test_valid_until_exception_drops_with_warning(self, caplog: pytest.LogCaptureFixture):
|
||||
q = NudgeQueue()
|
||||
|
||||
def boom() -> bool:
|
||||
raise RuntimeError("predicate crash")
|
||||
|
||||
q.enqueue("a", "1", "any", valid_until=boom)
|
||||
out = q.drain({"any"})
|
||||
with caplog.at_level(logging.WARNING, logger="turnstone.core.nudge_queue"):
|
||||
out = q.drain({"any"})
|
||||
assert out == []
|
||||
# Crash-on-predicate is treated as "no longer valid" — drop, not propagate.
|
||||
assert len(q) == 0
|
||||
# Stays at ``warning`` (with ``exc_info``) because a raising
|
||||
# predicate is a bug, not a normal lifecycle outcome.
|
||||
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
|
||||
assert len(drops) == 1
|
||||
assert drops[0].levelno == logging.WARNING
|
||||
rendered = drops[0].getMessage()
|
||||
assert "predicate_raised" in rendered
|
||||
assert "RuntimeError" in rendered
|
||||
assert "predicate crash" in rendered
|
||||
|
||||
def test_valid_until_evaluated_outside_lock(self):
|
||||
"""The predicate may do non-trivial work (e.g. storage I/O)
|
||||
|
||||
@@ -11,6 +11,12 @@ from turnstone.core.session import ChatSession, _render_template
|
||||
class NullUI:
|
||||
"""UI adapter that discards all output."""
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Tests for ``AnthropicProvider.extract_reasoning_text``.
|
||||
|
||||
Phase 1 of the optional-reasoning-persistence feature: provider-side
|
||||
extractor that walks stored ``provider_blocks`` and returns the
|
||||
concatenated thinking text, capped at the operator-friendly UI display
|
||||
size.
|
||||
|
||||
These tests drive through the real ``AnthropicProvider`` instance — no
|
||||
mocks of the extractor itself — using fixture-shaped blocks that match
|
||||
what ``_iter_anthropic_stream`` actually accumulates at
|
||||
``_anthropic.py:713-724`` (``thinking_delta`` + ``signature_delta``
|
||||
combined into ``{"type": "thinking", "thinking": <text>, "signature":
|
||||
<sig>}``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
from turnstone.core.providers._protocol import (
|
||||
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def anthropic() -> AnthropicProvider:
|
||||
return AnthropicProvider()
|
||||
|
||||
|
||||
class TestExtractReasoningText:
|
||||
def test_none_input_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
|
||||
assert anthropic.extract_reasoning_text(None) == ""
|
||||
|
||||
def test_empty_list_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
|
||||
assert anthropic.extract_reasoning_text([]) == ""
|
||||
|
||||
def test_no_thinking_blocks_returns_empty(self, anthropic: AnthropicProvider) -> None:
|
||||
blocks: list[dict[str, object]] = [
|
||||
{"type": "text", "text": "hello"},
|
||||
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
|
||||
]
|
||||
assert anthropic.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_single_thinking_block_returns_text(self, anthropic: AnthropicProvider) -> None:
|
||||
blocks = [{"type": "thinking", "thinking": "Let me think about this.", "signature": "abc"}]
|
||||
assert anthropic.extract_reasoning_text(blocks) == "Let me think about this."
|
||||
|
||||
def test_multiple_thinking_blocks_joined_with_newline(
|
||||
self, anthropic: AnthropicProvider
|
||||
) -> None:
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "first thought", "signature": "s1"},
|
||||
{"type": "thinking", "thinking": "second thought", "signature": "s2"},
|
||||
]
|
||||
assert anthropic.extract_reasoning_text(blocks) == "first thought\nsecond thought"
|
||||
|
||||
def test_mixed_blocks_extracts_only_thinking(self, anthropic: AnthropicProvider) -> None:
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "reason A", "signature": "s"},
|
||||
{"type": "text", "text": "visible answer"},
|
||||
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
|
||||
{"type": "thinking", "thinking": "reason B", "signature": "s"},
|
||||
]
|
||||
assert anthropic.extract_reasoning_text(blocks) == "reason A\nreason B"
|
||||
|
||||
def test_thinking_block_without_thinking_field_skipped(
|
||||
self, anthropic: AnthropicProvider
|
||||
) -> None:
|
||||
blocks = [{"type": "thinking", "signature": "s"}]
|
||||
assert anthropic.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_thinking_block_with_empty_text_skipped(self, anthropic: AnthropicProvider) -> None:
|
||||
blocks = [{"type": "thinking", "thinking": "", "signature": "s"}]
|
||||
assert anthropic.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_truncation_at_64kib_cap(self, anthropic: AnthropicProvider) -> None:
|
||||
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
|
||||
blocks = [{"type": "thinking", "thinking": long_text, "signature": "s"}]
|
||||
result = anthropic.extract_reasoning_text(blocks)
|
||||
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
|
||||
|
||||
def test_just_under_cap_not_truncated(self, anthropic: AnthropicProvider) -> None:
|
||||
text = "y" * (_MAX_REASONING_DISPLAY_CHARS - 1)
|
||||
blocks = [{"type": "thinking", "thinking": text, "signature": "s"}]
|
||||
assert anthropic.extract_reasoning_text(blocks) == text
|
||||
|
||||
def test_malformed_block_entry_skipped(self, anthropic: AnthropicProvider) -> None:
|
||||
# A defensive sanity check — we should not crash if some
|
||||
# entry isn't a dict (e.g. a corrupted JSON payload).
|
||||
blocks = [
|
||||
"not a dict", # type: ignore[list-item]
|
||||
{"type": "thinking", "thinking": "good one", "signature": "s"},
|
||||
]
|
||||
assert anthropic.extract_reasoning_text(blocks) == "good one" # type: ignore[arg-type]
|
||||
|
||||
def test_non_list_input_returns_empty(self, anthropic: AnthropicProvider) -> None:
|
||||
# Defensive against a corrupted provider_data payload.
|
||||
assert anthropic.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
|
||||
assert anthropic.extract_reasoning_text({"type": "thinking"}) == "" # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestOtherProvidersDefault:
|
||||
"""Non-Anthropic providers return "" for the same fixture shapes."""
|
||||
|
||||
def test_openai_chat_returns_empty(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
|
||||
assert provider.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_openai_responses_returns_empty(self) -> None:
|
||||
provider = OpenAIResponsesProvider()
|
||||
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
|
||||
assert provider.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_openai_responses_extracts_reasoning_summary(self) -> None:
|
||||
# Phase 3: extractor now walks reasoning items captured via
|
||||
# include=["reasoning.encrypted_content"] and returns the
|
||||
# summary[*].text concatenation. Pre-Phase-3 this returned
|
||||
# "" — the stub was replaced once the wire path landed.
|
||||
provider = OpenAIResponsesProvider()
|
||||
blocks = [{"type": "reasoning", "summary": [{"type": "summary_text", "text": "x"}]}]
|
||||
assert provider.extract_reasoning_text(blocks) == "x"
|
||||
@@ -0,0 +1,532 @@
|
||||
"""Tests for Phase 2 wire-build replay flag + shape filter on AnthropicProvider.
|
||||
|
||||
Phase 2 of optional reasoning persistence wraps the verbatim
|
||||
``_provider_content`` replay path at ``_anthropic.py:_convert_messages``
|
||||
with two gates:
|
||||
|
||||
1. ``ANTHROPIC_VALID_BLOCK_TYPES`` per-block shape filter — foreign-
|
||||
shaped blocks (OpenAI Responses ``type="reasoning"``, Gemini thought
|
||||
parts, the synthetic ``reasoning_text`` from path-3 capture) are
|
||||
dropped individually; valid Anthropic blocks in the same message
|
||||
still ride the verbatim path. When NO valid blocks survive, the
|
||||
converter falls through to the text+tool_calls rebuild path.
|
||||
2. ``replay_reasoning_to_model`` operator flag — when False (the
|
||||
``model_definitions`` server_default), thinking blocks are
|
||||
stripped before the wire payload is built. Tool_use /
|
||||
server_tool_use / web_search_tool_result blocks (which carry
|
||||
web-search ``encrypted_content``) intentionally survive — the
|
||||
strip predicate is narrow by design.
|
||||
|
||||
Drives through the real ``AnthropicProvider._convert_messages`` with
|
||||
fixture-shaped messages, no mocks of the converter. Edge cases come
|
||||
from the briefing's "Edges & validation memo" sections.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.providers._anthropic import (
|
||||
ANTHROPIC_REASONING_BLOCK_TYPES,
|
||||
ANTHROPIC_VALID_BLOCK_TYPES,
|
||||
AnthropicProvider,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> AnthropicProvider:
|
||||
return AnthropicProvider()
|
||||
|
||||
|
||||
def _assistant_with_thinking(content: str = "Final answer.") -> dict[str, object]:
|
||||
"""Build an assistant message with a thinking + text + tool_use shape
|
||||
matching what the streaming layer captures at ``_anthropic.py:713-724``."""
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": content,
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "let me think", "signature": "sig"},
|
||||
{"type": "text", "text": content},
|
||||
{
|
||||
"type": "tool_use",
|
||||
"id": "call_abc",
|
||||
"name": "search",
|
||||
"input": {"q": "x"},
|
||||
},
|
||||
],
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_abc",
|
||||
"type": "function",
|
||||
"function": {"name": "search", "arguments": '{"q": "x"}'},
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class TestReplayFlagStripsThinking:
|
||||
"""``replay_reasoning_to_model=False`` strips thinking; ``True`` preserves."""
|
||||
|
||||
def test_replay_true_preserves_thinking_block(self, provider: AnthropicProvider) -> None:
|
||||
msg = _assistant_with_thinking()
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" in types_present
|
||||
assert "text" in types_present
|
||||
assert "tool_use" in types_present
|
||||
|
||||
def test_replay_false_strips_thinking_block(self, provider: AnthropicProvider) -> None:
|
||||
msg = _assistant_with_thinking()
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" not in types_present
|
||||
assert "text" in types_present # final answer survives
|
||||
assert "tool_use" in types_present # tool dispatch survives
|
||||
|
||||
def test_replay_false_strips_redacted_thinking_too(self, provider: AnthropicProvider) -> None:
|
||||
# Anthropic emits redacted_thinking blocks when the safety system
|
||||
# rewrites a thinking block. Phase 2 strip predicate must include
|
||||
# both shapes.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Answer.",
|
||||
"_provider_content": [
|
||||
{"type": "redacted_thinking", "data": "redacted-blob"},
|
||||
{"type": "text", "text": "Answer."},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "redacted_thinking" not in types_present
|
||||
assert "text" in types_present
|
||||
|
||||
def test_default_kwarg_preserves_existing_behaviour(self, provider: AnthropicProvider) -> None:
|
||||
"""Pre-Phase-2 callers that don't pass the kwarg get the verbatim
|
||||
replay (default True), matching the behaviour all production
|
||||
Anthropic-with-thinking turns shipped with for months."""
|
||||
msg = _assistant_with_thinking()
|
||||
_, converted = provider._convert_messages([msg]) # no kwarg
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" in types_present
|
||||
|
||||
|
||||
class TestWebSearchBlocksSurviveStrip:
|
||||
"""Edge 14: Anthropic web-search ``encrypted_content`` rides on
|
||||
``server_tool_use`` / ``web_search_tool_result`` blocks (NOT
|
||||
thinking blocks). Strip predicate is intentionally narrow."""
|
||||
|
||||
def test_server_tool_use_survives(self, provider: AnthropicProvider) -> None:
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "From search: ...",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "I should search", "signature": "s"},
|
||||
{
|
||||
"type": "server_tool_use",
|
||||
"id": "stu_1",
|
||||
"name": "web_search",
|
||||
"input": {"query": "turnstone bird"},
|
||||
},
|
||||
{
|
||||
"type": "web_search_tool_result",
|
||||
"tool_use_id": "stu_1",
|
||||
"content": [{"type": "web_search_result", "url": "https://e.com"}],
|
||||
"encrypted_content": "abc123encrypted",
|
||||
"encrypted_index": "idx456encrypted",
|
||||
},
|
||||
{"type": "text", "text": "From search: ..."},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" not in types_present # stripped
|
||||
assert "server_tool_use" in types_present # survives
|
||||
assert "web_search_tool_result" in types_present # survives
|
||||
assert "text" in types_present # survives
|
||||
# encrypted_content rides through intact — required for round-trip continuity
|
||||
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
|
||||
assert wsr["encrypted_content"] == "abc123encrypted"
|
||||
|
||||
def test_tool_use_block_survives(self, provider: AnthropicProvider) -> None:
|
||||
# Plain tool_use (not server-side) — used by client-side function
|
||||
# tools. Strip predicate must not touch these.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Calling tool",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "I should call tool", "signature": "s"},
|
||||
{"type": "tool_use", "id": "tu_1", "name": "f", "input": {"a": 1}},
|
||||
],
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "tu_1",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": '{"a": 1}'},
|
||||
}
|
||||
],
|
||||
}
|
||||
# Provide the tool result so orphan-tool detection doesn't synthesize
|
||||
msgs = [
|
||||
msg,
|
||||
{"role": "tool", "tool_call_id": "tu_1", "content": "ok"},
|
||||
]
|
||||
_, converted = provider._convert_messages(msgs, replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" not in types_present
|
||||
assert "tool_use" in types_present
|
||||
|
||||
def test_orphan_tool_use_synthesized_after_strip(self, provider: AnthropicProvider) -> None:
|
||||
"""Pin the post-strip orphan-tool branch at _anthropic.py:397-433.
|
||||
|
||||
The implementation comment specifically calls out reading
|
||||
``provider_content`` (not ``wire_blocks``) for the orphan-tool
|
||||
ID walk after the strip — keeping the read on the source-of-
|
||||
truth list so a future refactor that swapped them would still
|
||||
get the same set of tool_use IDs. This test exercises that
|
||||
branch end-to-end: replay=False strips the thinking block,
|
||||
AND the message has a tool_use whose result is missing. The
|
||||
converter must synthesize a 'cancelled' tool_result for the
|
||||
orphaned tool_use ID (matching the existing pre-Phase-2
|
||||
behaviour for the verbatim path).
|
||||
"""
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Calling tool",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "let me think", "signature": "s"},
|
||||
{"type": "tool_use", "id": "orphan_tu", "name": "f", "input": {"a": 1}},
|
||||
],
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "orphan_tu",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": '{"a": 1}'},
|
||||
}
|
||||
],
|
||||
}
|
||||
# NO tool result follows — orphan branch must synthesize one.
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
# Synthetic tool_result lands as a user-role message immediately
|
||||
# after the assistant turn (per existing behaviour at
|
||||
# _anthropic.py:421-430).
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
# Stripped: thinking gone, tool_use survives.
|
||||
a_types = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" not in a_types
|
||||
assert "tool_use" in a_types
|
||||
# Synthesized: cancelled tool_result for orphan_tu attached to a
|
||||
# following user-role message.
|
||||
user_msgs_after = [m for m in converted if m["role"] == "user"]
|
||||
assert user_msgs_after, (
|
||||
"Expected a synthetic user message carrying the cancelled "
|
||||
"tool_result for the orphaned tool_use"
|
||||
)
|
||||
flat_results = [
|
||||
block
|
||||
for um in user_msgs_after
|
||||
if isinstance(um["content"], list)
|
||||
for block in um["content"]
|
||||
if isinstance(block, dict) and block.get("type") == "tool_result"
|
||||
]
|
||||
synth = next(
|
||||
(b for b in flat_results if b.get("tool_use_id") == "orphan_tu"),
|
||||
None,
|
||||
)
|
||||
assert synth is not None, f"Expected synthetic tool_result for orphan_tu in {flat_results}"
|
||||
assert synth.get("is_error") is True
|
||||
assert "cancelled" in synth.get("content", "").lower()
|
||||
|
||||
|
||||
class TestShapeFilterFallthrough:
|
||||
"""Foreign / empty / mixed-shape ``_provider_content`` falls through
|
||||
to the text+tool_calls rebuild path rather than reaching the wire
|
||||
as a malformed block."""
|
||||
|
||||
def test_foreign_shape_openai_reasoning_falls_through(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# OpenAI Responses style block (Phase 3 will land this shape into
|
||||
# _provider_content via include=["reasoning.encrypted_content"]).
|
||||
# Mid-workstream model switch from OpenAI -> Anthropic must NOT
|
||||
# reach the API with an OpenAI-shaped block (which would 400).
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Final answer from openai turn.",
|
||||
"_provider_content": [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"summary": [{"type": "summary_text", "text": "I reasoned..."}],
|
||||
"encrypted_content": "openai-encrypted",
|
||||
}
|
||||
],
|
||||
"tool_calls": [],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
# Rebuilt from text — no foreign block reached the wire.
|
||||
for b in assistant["content"]:
|
||||
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES, (
|
||||
f"Foreign block type leaked through: {b}"
|
||||
)
|
||||
# And the foreign block specifically is NOT present.
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "reasoning" not in types_present
|
||||
|
||||
def test_mixed_shape_drops_foreign_keeps_valid(self, provider: AnthropicProvider) -> None:
|
||||
# Per-block filter: a single foreign block in a mostly-Anthropic
|
||||
# payload no longer forces fall-through. Valid Anthropic blocks
|
||||
# ride the verbatim path; the foreign block is dropped.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Mixed.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "anth shape", "signature": "s"},
|
||||
{"type": "text", "text": "Mixed."},
|
||||
{"type": "reasoning", "summary": []}, # foreign
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "reasoning" not in types_present # foreign dropped
|
||||
assert "thinking" in types_present # valid + replay=True kept
|
||||
assert "text" in types_present
|
||||
for b in assistant["content"]:
|
||||
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
|
||||
def test_mixed_shape_preserves_web_search_encrypted_content(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# The motivating case for per-block (vs all-or-nothing) filter:
|
||||
# cross-model resumption stamps a foreign ``reasoning`` block
|
||||
# alongside Anthropic web-search blocks carrying encrypted
|
||||
# citations. An all-or-nothing filter would discard the whole
|
||||
# message and rebuild from text+tool_calls — silently losing
|
||||
# the encrypted_content the API needs for round-trip continuity.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "From search: ...",
|
||||
"_provider_content": [
|
||||
{"type": "reasoning", "summary": []}, # foreign (e.g. OpenAI)
|
||||
{
|
||||
"type": "server_tool_use",
|
||||
"id": "stu_1",
|
||||
"name": "web_search",
|
||||
"input": {"query": "x"},
|
||||
},
|
||||
{
|
||||
"type": "web_search_tool_result",
|
||||
"tool_use_id": "stu_1",
|
||||
"content": [{"type": "web_search_result", "url": "https://e.com"}],
|
||||
"encrypted_content": "encrypted-blob-must-survive",
|
||||
"encrypted_index": "encrypted-idx-must-survive",
|
||||
},
|
||||
{"type": "text", "text": "From search: ..."},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "reasoning" not in types_present # foreign dropped
|
||||
assert "server_tool_use" in types_present
|
||||
assert "web_search_tool_result" in types_present
|
||||
assert "text" in types_present
|
||||
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
|
||||
assert wsr["encrypted_content"] == "encrypted-blob-must-survive"
|
||||
assert wsr["encrypted_index"] == "encrypted-idx-must-survive"
|
||||
|
||||
def test_all_foreign_blocks_fall_through_to_rebuild(self, provider: AnthropicProvider) -> None:
|
||||
# When every block is foreign-shaped (no Anthropic-valid block
|
||||
# survives the per-block filter), the converter still falls
|
||||
# through to text+tool_calls rebuild rather than emitting an
|
||||
# empty assistant turn.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "reasoning", "summary": []},
|
||||
{"type": "reasoning_text", "text": "synthetic"}, # path-3 shape
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
# Rebuild path: msg.content lifted into a single text block.
|
||||
assert assistant["content"] == [{"type": "text", "text": "Final answer."}]
|
||||
|
||||
def test_empty_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Plain text answer.",
|
||||
"_provider_content": [],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg])
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
# Falls through to text rebuild
|
||||
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
|
||||
|
||||
def test_none_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Plain text answer.",
|
||||
"_provider_content": None,
|
||||
}
|
||||
_, converted = provider._convert_messages([msg])
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
|
||||
|
||||
def test_provider_content_not_a_list_falls_through(self, provider: AnthropicProvider) -> None:
|
||||
# Defensive against a corrupted provider_data deserialization.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Plain.",
|
||||
"_provider_content": "not a list",
|
||||
}
|
||||
_, converted = provider._convert_messages([msg])
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
assert assistant["content"] == [{"type": "text", "text": "Plain."}]
|
||||
|
||||
def test_non_dict_and_missing_type_blocks_are_dropped(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# Defensive branches in the per-block walk: a stray non-dict
|
||||
# element (corrupted JSON) or a dict with no/None ``type`` key
|
||||
# (provider drift) must be silently dropped without raising.
|
||||
# Valid blocks in the same list still ride the verbatim path.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "ok",
|
||||
"_provider_content": [
|
||||
{"type": "text", "text": "ok"},
|
||||
"stray-string", # non-dict
|
||||
{"type": None, "text": "huh"}, # None type
|
||||
{"no_type_key": 1}, # missing type
|
||||
{"type": "thinking", "thinking": "t", "signature": "s"},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b.get("type") for b in assistant["content"]]
|
||||
assert types_present == ["text", "thinking"]
|
||||
|
||||
|
||||
class TestLegacyAnthropicRowsNoRegression:
|
||||
"""Critical property: rows persisted before Phase 2 carry valid
|
||||
Anthropic-shape _provider_content (only Anthropic captured this lane
|
||||
historically). They must stay in the verbatim path and keep their
|
||||
thinking context across the migration boundary when replay=True
|
||||
(the legacy default).
|
||||
"""
|
||||
|
||||
def test_legacy_thinking_row_preserved_with_default_kwarg(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
"""No kwarg passed (matches the pre-Phase-2 production call site)."""
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Old answer from months ago.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "old reasoning", "signature": "s"},
|
||||
{"type": "text", "text": "Old answer from months ago."},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg])
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
types_present = [b["type"] for b in assistant["content"]]
|
||||
assert "thinking" in types_present # preserved -> no regression
|
||||
# The thinking block IS the same dict as the source (verbatim path).
|
||||
assert assistant["content"][0]["thinking"] == "old reasoning"
|
||||
|
||||
def test_replay_false_only_strips_when_explicitly_requested(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# Operator flips persist+replay flags off. Strip fires.
|
||||
# Pinning that the strip is gated on the explicit flag value,
|
||||
# not silently triggered by some other condition.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "stripped", "signature": "s"},
|
||||
{"type": "text", "text": "Answer."},
|
||||
],
|
||||
}
|
||||
_, converted_default = provider._convert_messages([msg])
|
||||
_, converted_strip = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
default_types = [b["type"] for b in converted_default[0]["content"]]
|
||||
strip_types = [b["type"] for b in converted_strip[0]["content"]]
|
||||
assert "thinking" in default_types
|
||||
assert "thinking" not in strip_types
|
||||
|
||||
|
||||
class TestStripAllBlocksFallthrough:
|
||||
"""When the message is 100% thinking (no text, no tool_use) and
|
||||
replay=False strips everything, the message falls through to the
|
||||
text+tool_calls rebuild path. If both are also empty, the assistant
|
||||
turn is silently skipped — correct: stripped reasoning has nothing
|
||||
to replay."""
|
||||
|
||||
def test_only_thinking_strip_falls_to_rebuild_with_text(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# Provider_content = only thinking; msg.content has the spoken text.
|
||||
# Strip drops thinking; rebuild path picks up the content as a
|
||||
# text block. No information lost.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "Spoken answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "internal", "signature": "s"},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
assert assistant["content"] == [{"type": "text", "text": "Spoken answer."}]
|
||||
|
||||
def test_only_thinking_strip_with_no_content_skips_message(
|
||||
self, provider: AnthropicProvider
|
||||
) -> None:
|
||||
# Edge: provider_content was 100% thinking AND msg.content is
|
||||
# empty AND no tool_calls. The rebuild path sees nothing to
|
||||
# emit — assistant turn silently skipped. Anthropic's API
|
||||
# would reject an empty assistant content array anyway.
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "only", "signature": "s"},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
|
||||
# Assistant turn skipped — no entry for it in `converted`.
|
||||
assert all(m["role"] != "assistant" for m in converted)
|
||||
|
||||
|
||||
class TestConstants:
|
||||
"""Pin the constant contents so a future edit doesn't accidentally
|
||||
widen the strip set or narrow the valid set."""
|
||||
|
||||
def test_reasoning_block_types_is_narrow(self) -> None:
|
||||
# Strip predicate MUST cover only reasoning shapes. Adding
|
||||
# tool_use here would break web-search round-trip.
|
||||
assert frozenset({"thinking", "redacted_thinking"}) == ANTHROPIC_REASONING_BLOCK_TYPES
|
||||
|
||||
def test_valid_block_types_includes_web_search(self) -> None:
|
||||
# Without server_tool_use / web_search_tool_result, Anthropic
|
||||
# web-search results would fall through to the rebuild path
|
||||
# and lose their encrypted_content.
|
||||
assert "server_tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
assert "web_search_tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
assert "tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
assert "tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
|
||||
def test_reasoning_subset_of_valid(self) -> None:
|
||||
# The strip set must be a subset of the valid set — otherwise
|
||||
# the strip predicate would never match anything (we only
|
||||
# strip after shape validity passes).
|
||||
assert ANTHROPIC_REASONING_BLOCK_TYPES.issubset(ANTHROPIC_VALID_BLOCK_TYPES)
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Tests for OpenAI Responses reasoning capture + replay (Phase 3 path 2).
|
||||
|
||||
Phase 3 wires:
|
||||
1. ``include=["reasoning.encrypted_content"]`` on the request when
|
||||
the operator flag AND the model capability both allow.
|
||||
2. ``_convert_messages`` round-tripping stored reasoning items as
|
||||
``ResponseReasoningItemParam`` input items on subsequent turns.
|
||||
3. ``OpenAIResponsesProvider.extract_reasoning_text`` walking
|
||||
reasoning items and returning concatenated summary + content text.
|
||||
|
||||
All tests drive through the real ``OpenAIResponsesProvider`` — no
|
||||
mocks of the converter/build_kwargs themselves; only the SDK boundary
|
||||
is mocked where relevant.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.providers._openai_responses import (
|
||||
OpenAIResponsesProvider,
|
||||
_reasoning_item_for_input,
|
||||
)
|
||||
from turnstone.core.providers._protocol import (
|
||||
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
|
||||
)
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def provider() -> OpenAIResponsesProvider:
|
||||
return OpenAIResponsesProvider()
|
||||
|
||||
|
||||
def _capable_caps() -> ModelCapabilities:
|
||||
"""Capability fixture for a reasoning-replay-capable model."""
|
||||
return ModelCapabilities(
|
||||
context_window=400000,
|
||||
max_output_tokens=128000,
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_reasoning_replay=True,
|
||||
)
|
||||
|
||||
|
||||
class TestExtractReasoningText:
|
||||
def test_none_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
|
||||
assert provider.extract_reasoning_text(None) == ""
|
||||
|
||||
def test_empty_list_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
|
||||
assert provider.extract_reasoning_text([]) == ""
|
||||
|
||||
def test_no_reasoning_items_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
|
||||
blocks = [
|
||||
{"type": "message", "role": "assistant", "content": "hi"},
|
||||
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == ""
|
||||
|
||||
def test_summary_text_extracted(self, provider: OpenAIResponsesProvider) -> None:
|
||||
# Per ResponseReasoningItem (response_reasoning_item.py:31-62):
|
||||
# summary is always present; content is optional.
|
||||
blocks = [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [
|
||||
{"type": "summary_text", "text": "I considered X"},
|
||||
{"type": "summary_text", "text": "then Y"},
|
||||
],
|
||||
}
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "I considered X\nthen Y"
|
||||
|
||||
def test_content_text_extracted_alongside_summary(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
blocks = [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "summary line"}],
|
||||
"content": [{"type": "reasoning_text", "text": "raw reasoning"}],
|
||||
}
|
||||
]
|
||||
# Order: summary first, then content (matches the order the SDK
|
||||
# surfaces them via streaming events).
|
||||
result = provider.extract_reasoning_text(blocks)
|
||||
assert "summary line" in result
|
||||
assert "raw reasoning" in result
|
||||
|
||||
def test_truncation_at_64kib_cap(self, provider: OpenAIResponsesProvider) -> None:
|
||||
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
|
||||
blocks = [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": long_text}],
|
||||
}
|
||||
]
|
||||
result = provider.extract_reasoning_text(blocks)
|
||||
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
|
||||
|
||||
def test_malformed_summary_entry_skipped(self, provider: OpenAIResponsesProvider) -> None:
|
||||
blocks = [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [
|
||||
"not a dict",
|
||||
{"type": "summary_text"}, # missing text
|
||||
{"type": "summary_text", "text": ""}, # empty text
|
||||
{"type": "summary_text", "text": "good"},
|
||||
],
|
||||
}
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "good"
|
||||
|
||||
def test_non_list_input_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
|
||||
assert provider.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
|
||||
|
||||
def test_other_block_types_skipped_in_walk(self, provider: OpenAIResponsesProvider) -> None:
|
||||
# Mixed payload: only the reasoning block contributes.
|
||||
blocks = [
|
||||
{"type": "message", "role": "assistant", "content": "hi"},
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "thought"}],
|
||||
},
|
||||
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "thought"
|
||||
|
||||
|
||||
class TestReasoningItemForInput:
|
||||
"""``_reasoning_item_for_input`` projects a stored ``ResponseReasoningItem``
|
||||
dict into ``ResponseReasoningItemParam`` shape (drops server-only
|
||||
``status``)."""
|
||||
|
||||
def test_minimal_item_round_trip(self) -> None:
|
||||
stored = {
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "x"}],
|
||||
"status": "completed",
|
||||
}
|
||||
result = _reasoning_item_for_input(stored)
|
||||
assert result["type"] == "reasoning"
|
||||
assert result["id"] == "r_1"
|
||||
assert result["summary"] == [{"type": "summary_text", "text": "x"}]
|
||||
# status NOT round-tripped (server-only field per
|
||||
# ResponseReasoningItemParam at response_reasoning_item_param.py).
|
||||
assert "status" not in result
|
||||
|
||||
def test_encrypted_content_round_trips_when_present(self) -> None:
|
||||
stored = {
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "x"}],
|
||||
"encrypted_content": "opaque-blob",
|
||||
}
|
||||
result = _reasoning_item_for_input(stored)
|
||||
assert result["encrypted_content"] == "opaque-blob"
|
||||
|
||||
def test_encrypted_content_omitted_when_absent(self) -> None:
|
||||
stored = {
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "x"}],
|
||||
}
|
||||
result = _reasoning_item_for_input(stored)
|
||||
assert "encrypted_content" not in result
|
||||
|
||||
def test_content_round_trips_when_present(self) -> None:
|
||||
stored = {
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "s"}],
|
||||
"content": [{"type": "reasoning_text", "text": "raw"}],
|
||||
}
|
||||
result = _reasoning_item_for_input(stored)
|
||||
assert result["content"] == [{"type": "reasoning_text", "text": "raw"}]
|
||||
|
||||
|
||||
class TestBuildKwargsInclude:
|
||||
"""``_build_kwargs`` adds ``include=["reasoning.encrypted_content"]``
|
||||
when the resolved operator flag is True. The capability AND-gate
|
||||
lives upstream in ``ChatSession._resolve_replay_reasoning_to_model``
|
||||
(single source of truth across providers); the provider trusts the
|
||||
bool it receives. See
|
||||
``test_session_replay_reasoning.py::TestSessionToOpenAIResponsesBoundaryIntegration``
|
||||
for the end-to-end gate test."""
|
||||
|
||||
def test_include_added_when_flag_true(self, provider: OpenAIResponsesProvider) -> None:
|
||||
kwargs = provider._build_kwargs(
|
||||
model="gpt-5",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
tools=None,
|
||||
max_tokens=1024,
|
||||
temperature=0.5,
|
||||
reasoning_effort="medium",
|
||||
deferred_names=None,
|
||||
capabilities=_capable_caps(),
|
||||
replay_reasoning_to_model=True,
|
||||
)
|
||||
assert kwargs.get("include") == ["reasoning.encrypted_content"]
|
||||
|
||||
def test_include_omitted_when_flag_false(self, provider: OpenAIResponsesProvider) -> None:
|
||||
kwargs = provider._build_kwargs(
|
||||
model="gpt-5",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
tools=None,
|
||||
max_tokens=1024,
|
||||
temperature=0.5,
|
||||
reasoning_effort="medium",
|
||||
deferred_names=None,
|
||||
capabilities=_capable_caps(),
|
||||
replay_reasoning_to_model=False,
|
||||
)
|
||||
assert "include" not in kwargs
|
||||
|
||||
|
||||
class TestConvertMessagesReasoningReplay:
|
||||
"""``_convert_messages`` round-trips stored reasoning items as input."""
|
||||
|
||||
def test_reasoning_item_emitted_before_assistant_when_replay_true(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
messages = [
|
||||
{"role": "user", "content": "explain"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "I thought"}],
|
||||
"encrypted_content": "abc",
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "follow up"},
|
||||
]
|
||||
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
|
||||
# Find the reasoning input item.
|
||||
types = [it.get("type") for it in items]
|
||||
# Expected: user, reasoning, message (assistant), user.
|
||||
assert types == ["message", "reasoning", "message", "message"]
|
||||
reasoning_idx = types.index("reasoning")
|
||||
r_item = items[reasoning_idx]
|
||||
assert r_item["id"] == "r_1"
|
||||
assert r_item["encrypted_content"] == "abc"
|
||||
# And the reasoning item appears immediately BEFORE the
|
||||
# assistant message it belongs to.
|
||||
assert items[reasoning_idx + 1]["role"] == "assistant"
|
||||
|
||||
def test_reasoning_item_dropped_when_replay_false(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Answer.",
|
||||
"_provider_content": [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "thought"}],
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
_, items = provider._convert_messages(messages, replay_reasoning_to_model=False)
|
||||
types = [it.get("type") for it in items]
|
||||
assert "reasoning" not in types
|
||||
|
||||
def test_no_reasoning_items_when_provider_content_lacks_reasoning(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
# Anthropic-shaped _provider_content reaching OpenAI Responses
|
||||
# (cross-provider — operator switch from Anthropic to GPT-5):
|
||||
# no type=="reasoning" items, so nothing emitted.
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "anth", "signature": "s"},
|
||||
],
|
||||
},
|
||||
]
|
||||
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
|
||||
types = [it.get("type") for it in items]
|
||||
assert "reasoning" not in types
|
||||
|
||||
def test_default_replay_reasoning_false_omits_reasoning(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
# Pre-Phase-3 callers (no kwarg) get the back-compat behaviour:
|
||||
# reasoning items are silently dropped (sanitize_messages was
|
||||
# already stripping _provider_content anyway).
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": "x"}],
|
||||
}
|
||||
],
|
||||
},
|
||||
]
|
||||
_, items = provider._convert_messages(messages) # no kwarg
|
||||
types = [it.get("type") for it in items]
|
||||
assert "reasoning" not in types
|
||||
@@ -0,0 +1,333 @@
|
||||
"""Audit-log discipline test for reasoning text.
|
||||
|
||||
Phase 1 of optional reasoning persistence surfaces stored thinking
|
||||
blocks on the ``/history`` payload (UI rehydration). The bytes ride
|
||||
through the helper (``extract_reasoning_for_history``), through the
|
||||
provider extractor (``AnthropicProvider.extract_reasoning_text``), and
|
||||
through the server build path (``_build_history``).
|
||||
|
||||
This test pins the security-sensitive contract:
|
||||
|
||||
Reasoning text MAY land on ``msg["reasoning"]`` (UI-bound),
|
||||
but MUST NOT appear in any ``Logger.info`` / ``warning`` /
|
||||
``error`` payload at any layer in the pipeline.
|
||||
|
||||
The test mocks the standard-library ``logging.Logger`` info/warning/
|
||||
error methods, runs a thinking-bearing turn through the relevant
|
||||
extractors and history build, then asserts no captured log call's
|
||||
positional args or kwargs contain the unique marker string. Replaces
|
||||
the v4 grep-the-output approach (fragile when log strings are
|
||||
formatted) with a structural mock-and-assert (tests the actual
|
||||
contract rather than the rendered text).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
from tests._session_helpers import make_session
|
||||
from turnstone.core.history_decoration import (
|
||||
extract_reasoning_for_history,
|
||||
extract_reasoning_text_from_provider_content,
|
||||
)
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
from turnstone.core.providers._protocol import StreamChunk, UsageInfo
|
||||
from turnstone.server import _build_history
|
||||
|
||||
_MARKER = "SECRET_REASONING_MARKER_xyz123_unlikely_collision"
|
||||
|
||||
|
||||
def _payload_contains_marker(args: tuple[Any, ...], kwargs: dict[str, Any]) -> bool:
|
||||
"""Walk a captured log call's args + kwargs for the marker string.
|
||||
|
||||
Logger.info-style calls accept a format string + positional substitution
|
||||
args; the marker could appear in either the format string itself or
|
||||
the substitution values. Format-time strings (``%`` substitution) are
|
||||
NOT inspected because they're a stdlib formatting concern, not a
|
||||
callable our pipeline reaches into. The structural check is "no
|
||||
user-controlled marker appears in any arg slot we passed".
|
||||
"""
|
||||
for a in args:
|
||||
if isinstance(a, str) and _MARKER in a:
|
||||
return True
|
||||
# Defensive — a list/dict/exception arg might carry the marker too.
|
||||
try:
|
||||
if _MARKER in repr(a):
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
for v in kwargs.values():
|
||||
if isinstance(v, str) and _MARKER in v:
|
||||
return True
|
||||
try:
|
||||
if _MARKER in repr(v):
|
||||
return True
|
||||
except Exception:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def _capture_log_calls():
|
||||
"""Capture every Logger.info / warning / error call into a single list."""
|
||||
captured: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = []
|
||||
|
||||
def make_recorder(level: str):
|
||||
def _rec(*args: Any, **kwargs: Any) -> None:
|
||||
captured.append((level, args, kwargs))
|
||||
|
||||
return _rec
|
||||
|
||||
return captured, [
|
||||
patch.object(logging.Logger, "info", side_effect=make_recorder("info"), autospec=True),
|
||||
patch.object(
|
||||
logging.Logger, "warning", side_effect=make_recorder("warning"), autospec=True
|
||||
),
|
||||
patch.object(logging.Logger, "error", side_effect=make_recorder("error"), autospec=True),
|
||||
]
|
||||
|
||||
|
||||
class TestReasoningAuditLogDiscipline:
|
||||
"""Reasoning text never lands at INFO+ severity on any logger."""
|
||||
|
||||
def _thinking_msg(self, text: str = _MARKER) -> dict[str, Any]:
|
||||
return {
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": text, "signature": "sig"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
}
|
||||
|
||||
def test_anthropic_extractor_does_not_log_reasoning(self) -> None:
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
provider = AnthropicProvider()
|
||||
text = provider.extract_reasoning_text(
|
||||
[{"type": "thinking", "thinking": _MARKER, "signature": "s"}]
|
||||
)
|
||||
assert text == _MARKER # extractor IS allowed to return it
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"AnthropicProvider.extract_reasoning_text leaked reasoning text "
|
||||
f"into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_dispatch_helper_does_not_log_reasoning(self) -> None:
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
text = extract_reasoning_text_from_provider_content(
|
||||
[{"type": "thinking", "thinking": _MARKER, "signature": "s"}]
|
||||
)
|
||||
assert text == _MARKER
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"extract_reasoning_text_from_provider_content leaked reasoning "
|
||||
f"text into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_list_helper_does_not_log_reasoning(self) -> None:
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
messages = [self._thinking_msg(_MARKER)]
|
||||
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
|
||||
assert messages[0]["reasoning"] == _MARKER # UI-bound is allowed
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"extract_reasoning_for_history leaked reasoning text into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_build_history_does_not_log_reasoning(self) -> None:
|
||||
registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(surface_persisted_reasoning=True)
|
||||
)
|
||||
session = SimpleNamespace(
|
||||
messages=[self._thinking_msg(_MARKER)],
|
||||
_ws_id="ws-audit",
|
||||
_registry=registry,
|
||||
_model_alias="claude-opus-4-7",
|
||||
)
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
with patch(
|
||||
"turnstone.server._load_verdict_indexes",
|
||||
return_value=({}, {}),
|
||||
):
|
||||
history = _build_history(session)
|
||||
assert history[0]["reasoning"] == _MARKER # UI-bound is allowed
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], f"_build_history leaked reasoning text into INFO+ logs: {offending}"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Phase 2 + Phase 3 surfaces — added in response to a code-review
|
||||
# finding that the original 4-test coverage missed every code path
|
||||
# introduced after Phase 1. Each new test mirrors the structure
|
||||
# above: capture every Logger.info / warning / error call across
|
||||
# the operation, assert the marker doesn't appear in any captured
|
||||
# payload (UI-bound returns IS allowed; logging at INFO+ is NOT).
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_openai_responses_extractor_does_not_log_reasoning(self) -> None:
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
provider = OpenAIResponsesProvider()
|
||||
blocks = [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_1",
|
||||
"summary": [{"type": "summary_text", "text": _MARKER}],
|
||||
}
|
||||
]
|
||||
text = provider.extract_reasoning_text(blocks)
|
||||
assert _MARKER in text # UI-bound return is allowed
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"OpenAIResponsesProvider.extract_reasoning_text leaked reasoning "
|
||||
f"text into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_openai_chat_extractor_does_not_log_reasoning(self) -> None:
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [{"type": "reasoning_text", "text": _MARKER, "source": "vllm"}]
|
||||
text = provider.extract_reasoning_text(blocks)
|
||||
assert text == _MARKER
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"OpenAIChatCompletionsProvider.extract_reasoning_text leaked "
|
||||
f"reasoning text into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_synth_reasoning_block_via_stream_response_does_not_log_reasoning(
|
||||
self,
|
||||
) -> None:
|
||||
"""Drives ChatSession._stream_response (which calls
|
||||
_maybe_synth_reasoning_block at end-of-stream) with a fake
|
||||
``reasoning_delta=_MARKER`` chunk; asserts no log call carried
|
||||
the marker text."""
|
||||
session = make_session()
|
||||
chunks = [
|
||||
StreamChunk(reasoning_delta=_MARKER, is_first=True),
|
||||
StreamChunk(content_delta="answer"),
|
||||
StreamChunk(
|
||||
finish_reason="stop",
|
||||
usage=UsageInfo(prompt_tokens=10, completion_tokens=20, total_tokens=30),
|
||||
),
|
||||
]
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
msg = session._stream_response(iter(chunks))
|
||||
# Synth block stamped onto _provider_content with the marker.
|
||||
assert msg["_provider_content"][0]["text"] == _MARKER
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"_stream_response + _maybe_synth_reasoning_block leaked reasoning "
|
||||
f"text into INFO+ logs: {offending}"
|
||||
)
|
||||
|
||||
def test_anthropic_convert_messages_strip_does_not_log_reasoning(self) -> None:
|
||||
"""Drives the Phase 2 strip predicate
|
||||
(``replay_reasoning_to_model=False``) which walks thinking
|
||||
blocks to filter them out before the wire payload is built;
|
||||
asserts no log call carried the marker text."""
|
||||
captured, patchers = _capture_log_calls()
|
||||
for p in patchers:
|
||||
p.start()
|
||||
try:
|
||||
provider = AnthropicProvider()
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": _MARKER, "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
},
|
||||
]
|
||||
_, converted = provider._convert_messages(messages, replay_reasoning_to_model=False)
|
||||
# Strip fired — thinking block dropped from wire.
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
block_types = [b.get("type") for b in assistant["content"]]
|
||||
assert "thinking" not in block_types
|
||||
finally:
|
||||
for p in patchers:
|
||||
p.stop()
|
||||
offending = [
|
||||
(lvl, args, kwargs)
|
||||
for lvl, args, kwargs in captured
|
||||
if _payload_contains_marker(args, kwargs)
|
||||
]
|
||||
assert offending == [], (
|
||||
f"AnthropicProvider._convert_messages strip predicate leaked "
|
||||
f"reasoning text into INFO+ logs: {offending}"
|
||||
)
|
||||
+668
-19
@@ -255,12 +255,17 @@ function makeEl(tag) {
|
||||
setAttribute(k, v) { this._attrs[k] = v; },
|
||||
getAttribute(k) { return this._attrs[k] !== undefined ? this._attrs[k] : null; },
|
||||
get classList() {
|
||||
// Real DOMTokenList is array-like (length + indexed access) AND
|
||||
// exposes add/remove/contains. The hljs language-extraction
|
||||
// loop reads .length + [j], so we return a fresh Array snapshot
|
||||
// each get + bolt the mutator methods on. add/remove operate on
|
||||
// the live _classes set so subsequent reads see updates.
|
||||
const self = this;
|
||||
return {
|
||||
add(...c) { c.forEach(x => self._classes.add(x)); },
|
||||
remove(...c) { c.forEach(x => self._classes.delete(x)); },
|
||||
contains(c) { return self._classes.has(c); },
|
||||
};
|
||||
const arr = Array.from(self._classes);
|
||||
arr.add = (...c) => c.forEach((x) => self._classes.add(x));
|
||||
arr.remove = (...c) => c.forEach((x) => self._classes.delete(x));
|
||||
arr.contains = (c) => self._classes.has(c);
|
||||
return arr;
|
||||
},
|
||||
get className() { return Array.from(this._classes).join(' '); },
|
||||
set className(v) {
|
||||
@@ -269,9 +274,33 @@ function makeEl(tag) {
|
||||
get textContent() {
|
||||
return this._textContent || this.children.map(c => c.textContent || '').join('');
|
||||
},
|
||||
set textContent(v) { this._textContent = v; this.children = []; },
|
||||
set textContent(v) {
|
||||
// Real DOM: assigning textContent ALSO replaces innerHTML with
|
||||
// an entity-escaped representation of the same text. escapeHtml
|
||||
// (utils.js) round-trips via this side effect — without it,
|
||||
// every escapeHtml() call returns '' and renderMarkdown emits
|
||||
// empty <p> tags.
|
||||
this._textContent = v;
|
||||
this.children = [];
|
||||
this._innerHTML = String(v)
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
},
|
||||
get innerHTML() { return this._innerHTML; },
|
||||
set innerHTML(v) { this._innerHTML = v; this.children = []; },
|
||||
set innerHTML(v) {
|
||||
// Real DOM invalidates the previous textContent when innerHTML
|
||||
// is replaced — leaving _textContent intact would return stale
|
||||
// data from subsequent textContent reads and mask bugs that
|
||||
// depend on innerHTML/textContent consistency. We don't HTML-
|
||||
// parse here, so the cheap correct behavior is to clear
|
||||
// _textContent and let the children-derived fallback in the
|
||||
// textContent getter (which is empty after this children = [])
|
||||
// take over.
|
||||
this._innerHTML = v;
|
||||
this.children = [];
|
||||
this._textContent = '';
|
||||
},
|
||||
get isConnected() {
|
||||
// In real DOM this checks attachment to the document; for the
|
||||
// test harness we approximate via the parent chain. After
|
||||
@@ -304,17 +333,28 @@ function makeEl(tag) {
|
||||
this.parent = null;
|
||||
},
|
||||
querySelectorAll(selector) {
|
||||
// Only supports the literal "pre code.language-mermaid"
|
||||
// selector that postRenderMermaid uses.
|
||||
// Supports the two selectors the post-render passes use:
|
||||
// "pre code.language-mermaid" (postRenderMermaid)
|
||||
// "pre code[class*='language-']" (postRenderHljs)
|
||||
const out = [];
|
||||
const wantsMermaid = selector === "pre code.language-mermaid";
|
||||
function matchesLangAttr(el) {
|
||||
for (const cls of el._classes) {
|
||||
if (cls.startsWith('language-')) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function walk(node) {
|
||||
for (const c of (node.children || [])) {
|
||||
if (
|
||||
const isCodeInPre =
|
||||
c.tagName === 'CODE' &&
|
||||
c.parent && c.parent.tagName === 'PRE' &&
|
||||
c._classes.has('language-mermaid')
|
||||
) {
|
||||
out.push(c);
|
||||
c.parent && c.parent.tagName === 'PRE';
|
||||
if (isCodeInPre) {
|
||||
if (wantsMermaid) {
|
||||
if (c._classes.has('language-mermaid')) out.push(c);
|
||||
} else if (matchesLangAttr(c)) {
|
||||
out.push(c);
|
||||
}
|
||||
}
|
||||
walk(c);
|
||||
}
|
||||
@@ -352,6 +392,20 @@ global.mermaid = {
|
||||
},
|
||||
};
|
||||
|
||||
// hljs stub. highlightElement mutates the element in place: replaces
|
||||
// innerHTML with a deterministic synthetic span keyed by the source,
|
||||
// and adds the hljs class — same surface postRenderHljs depends on.
|
||||
// hljsHighlightCallCount lets tests assert "ran N times" semantics.
|
||||
let hljsHighlightCallCount = 0;
|
||||
global.hljs = {
|
||||
configure: () => {},
|
||||
highlightElement: (el) => {
|
||||
hljsHighlightCallCount++;
|
||||
el._classes.add('hljs');
|
||||
el._innerHTML = '<span class="hljs-tok">' + el._textContent + '</span>';
|
||||
},
|
||||
};
|
||||
|
||||
vm.runInThisContext(fs.readFileSync(%(utils)s, 'utf8'));
|
||||
vm.runInThisContext(fs.readFileSync(%(renderer)s, 'utf8'));
|
||||
|
||||
@@ -514,7 +568,7 @@ def test_mermaid_cache_evicts_oldest_at_cap() -> None:
|
||||
scenario = """
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
@@ -536,10 +590,10 @@ def test_mermaid_overwrite_does_not_evict() -> None:
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
// Fill exactly to cap.
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
// Overwrite an existing entry — must not evict src-0.
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null}, cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
hasOldest: _mermaidSvgCache.has('src-0'),
|
||||
@@ -558,8 +612,8 @@ def test_mermaid_cache_cleared_on_init() -> None:
|
||||
— the rendered output depends on themeVariables which change
|
||||
on init."""
|
||||
scenario = """
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null});
|
||||
_cacheMermaidEntry(_mermaidErrorCache, 'src-bad', 'old error');
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null}, _MERMAID_CACHE_MAX);
|
||||
_cacheFifoEntry(_mermaidErrorCache, 'src-bad', 'old error', _MERMAID_CACHE_MAX);
|
||||
_initMermaid();
|
||||
process.stdout.write(JSON.stringify({
|
||||
svgSize: _mermaidSvgCache.size,
|
||||
@@ -624,3 +678,598 @@ def test_streaming_render_invokes_mermaid_post_render() -> None:
|
||||
"_streamingRenderApply must call postRenderMermaid for "
|
||||
"progressive diagram rendering during streaming"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _normalizeMermaidSource — autoquote labels with bare shape-delimiter
|
||||
# chars. Mermaid rejects unquoted ( ) [ ] { } inside other labels with
|
||||
# a "got 'PS'" parse error (paren-start in shape context). The two
|
||||
# diagrams in the screenshot regression case are encoded here verbatim.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_normalize(source: str) -> str:
|
||||
"""Drive _normalizeMermaidSource against the JS harness and return
|
||||
its output. The function is pure, so no container / mermaid stub
|
||||
setup is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = _normalizeMermaidSource(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
# Diagram 1 from the screenshot regression — unquoted edge labels with
|
||||
# parens and <br/> markers. Mermaid rejects both edge labels with
|
||||
# "got 'PS'"; quoting them resolves it.
|
||||
_SCREENSHOT_DIAGRAM_1_IN = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
" -->|22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)|"
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
" B -->|13 jasl patches<br/>(Python only:<br/>tunings, kernels,"
|
||||
"<br/>warmup, etc.)|"
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_1_OUT = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
' -->|"22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)"|'
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
' B -->|"13 jasl patches<br/>(Python only:<br/>tunings, kernels,'
|
||||
'<br/>warmup, etc.)"|'
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
|
||||
# Diagram 2 from the screenshot regression — unquoted RECTANGLE node
|
||||
# label `D[untouched<br/>(.so, _version.py,<br/>install-vendored)]`.
|
||||
# Same parser failure mode; quoting the bracket label fixes it.
|
||||
_SCREENSHOT_DIAGRAM_2_IN = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
" B -->|not in archive| D[untouched<br/>(.so, _version.py,"
|
||||
"<br/>install-vendored)]\n"
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_2_OUT = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
' B -->|not in archive| D["untouched<br/>(.so, _version.py,'
|
||||
'<br/>install-vendored)"]\n'
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("source", "expected"),
|
||||
[
|
||||
(_SCREENSHOT_DIAGRAM_1_IN, _SCREENSHOT_DIAGRAM_1_OUT),
|
||||
(_SCREENSHOT_DIAGRAM_2_IN, _SCREENSHOT_DIAGRAM_2_OUT),
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_fixes_screenshot_diagrams(source: str, expected: str) -> None:
|
||||
"""The two exact diagrams from the screenshot regression. If
|
||||
these stop being rewritten with quoted labels, mermaid will
|
||||
again reject them with `Expecting ... got 'PS'` during live
|
||||
streaming."""
|
||||
assert _run_normalize(source) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"source",
|
||||
[
|
||||
# Clean diagram — no shape delimiters in any label.
|
||||
"graph TD\n A[foo] --> B[bar]",
|
||||
# Edge label with no special chars.
|
||||
"A --> B\nA -->|plain text| B",
|
||||
# Already-correctly-quoted node label.
|
||||
'A["already (quoted)"] --> B',
|
||||
# Already-correctly-quoted edge label.
|
||||
'A -->|"already (quoted)"| B',
|
||||
# Cylinder shape — inner () is part of the shape syntax.
|
||||
"A[(database)] --> B",
|
||||
# Subroutine shape — inner [] is part of the shape syntax.
|
||||
"A[[subroutine]] --> B",
|
||||
# Trapezoid shape — inner / is part of the shape syntax.
|
||||
"A[/trapezoid/] --> B",
|
||||
# Reverse trapezoid.
|
||||
"A[\\trap\\] --> B",
|
||||
# Mermaid directive — braces here are config, not a label.
|
||||
'%%{init: {"theme": "dark"}}%%\ngraph TD\n A --> B',
|
||||
# <br/> tags on their own don't trip quoting.
|
||||
"A[line1<br/>line2] --> B",
|
||||
# Sequence diagram — different grammar; we only target labels
|
||||
# in shape/edge syntax that match the regex anchors.
|
||||
"sequenceDiagram\n A->>B: hello",
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_leaves_valid_source_alone(source: str) -> None:
|
||||
"""The autoquoter must not rewrite syntactically valid Mermaid —
|
||||
a false positive here would break a working diagram. Each case
|
||||
covers a syntax form whose delimiters are intentional and must
|
||||
not be wrapped."""
|
||||
assert _run_normalize(source) == source
|
||||
|
||||
|
||||
def test_mermaid_autoquote_edge_label_with_parens() -> None:
|
||||
"""Bare-parens edge label gets wrapped. The bare `(` would
|
||||
otherwise re-enter Mermaid's shape parser."""
|
||||
src = "A -->|note (with parens)| B"
|
||||
assert _run_normalize(src) == 'A -->|"note (with parens)"| B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_parens() -> None:
|
||||
"""Bare-parens node label gets wrapped."""
|
||||
src = "D[label (foo, bar)]"
|
||||
assert _run_normalize(src) == 'D["label (foo, bar)"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_braces() -> None:
|
||||
"""Bare-braces in a rectangle label get wrapped. (Diamond {}
|
||||
shapes are left alone — only single-bracket [] labels are
|
||||
rewritten.)"""
|
||||
src = "A[config {key: value}]"
|
||||
assert _run_normalize(src) == 'A["config {key: value}"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_preserves_br_tag_with_parens() -> None:
|
||||
"""`<br/>` inside a label that also has parens stays — only the
|
||||
quoting needs to be added around the whole label."""
|
||||
src = "A[line1<br/>(line2)] --> B"
|
||||
assert _run_normalize(src) == 'A["line1<br/>(line2)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_skips_label_with_internal_quote() -> None:
|
||||
"""If a label contains a literal `"`, wrapping would produce
|
||||
nested unescaped quotes. The autoquoter must punt — leaving the
|
||||
parse error to surface, rather than silently producing a worse
|
||||
one."""
|
||||
src = 'A[he said "hi" (lol)]'
|
||||
assert _run_normalize(src) == src
|
||||
|
||||
|
||||
def test_mermaid_autoquote_multiple_edges_on_one_line() -> None:
|
||||
"""Both edge labels on a single line get rewritten independently."""
|
||||
src = "A -->|first (paren)| B -->|second (paren)| C"
|
||||
expected = 'A -->|"first (paren)"| B -->|"second (paren)"| C'
|
||||
assert _run_normalize(src) == expected
|
||||
|
||||
|
||||
def test_mermaid_autoquote_normalized_source_hits_cache() -> None:
|
||||
"""The SVG cache keys on the normalized source — same malformed
|
||||
input that the LLM streamed earlier still hits the cache on
|
||||
re-render rather than re-invoking mermaid.render every tick."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
renderCalls: renderCallCount,
|
||||
normalized: container.children[0]._attrs['data-mermaid-source'],
|
||||
}));
|
||||
}, 0);
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["renderCalls"] == 1, "second render bypassed the cache"
|
||||
assert out["normalized"] == 'A["label (with parens)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_populates_on_first_call() -> None:
|
||||
"""First postRenderMermaid call populates _mermaidNormalizeCache
|
||||
with a raw→normalized entry. A second call on identical raw
|
||||
textContent then hits the memo (size stays at 1, no second
|
||||
normalize call), which is the perf-1 fix — avoids re-running
|
||||
split + per-line regex per rAF tick when the diagram hasn't
|
||||
changed."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
const sizeAfterFirst = _mermaidNormalizeCache.size;
|
||||
const cachedNorm = _mermaidNormalizeCache.get(sources[0]);
|
||||
// Re-render on a fresh container with the same source.
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
sizeAfterFirst: sizeAfterFirst,
|
||||
cachedNorm: cachedNorm,
|
||||
sizeAfterSecond: _mermaidNormalizeCache.size,
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sizeAfterFirst"] == 1, "first call didn't populate normalize memo"
|
||||
assert out["cachedNorm"] == 'A["label (with parens)"] --> B'
|
||||
assert out["sizeAfterSecond"] == 1, (
|
||||
"second call added a new entry — memo missed on identical source"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_is_consulted_before_normalize() -> None:
|
||||
"""Pre-seed _mermaidNormalizeCache with a sentinel value for a
|
||||
raw source. postRenderMermaid must use the sentinel rather than
|
||||
re-running _normalizeMermaidSource. Catches a regression where
|
||||
the memo gets populated but the lookup path is skipped."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
sentinel = "SENTINEL_FROM_MEMO --> X"
|
||||
raw_js = json.dumps(bad)
|
||||
sentinel_js = json.dumps(sentinel)
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ f"""
|
||||
_mermaidNormalizeCache.set({raw_js}, {sentinel_js});
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {{
|
||||
process.stdout.write(JSON.stringify({{
|
||||
sourceAttr: container.children[0]._attrs['data-mermaid-source'],
|
||||
}}));
|
||||
}}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sourceAttr"] == sentinel, (
|
||||
"postRenderMermaid bypassed the normalize memo and re-ran normalize"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_distinct_sources_cache_separately() -> None:
|
||||
"""Two distinct raw sources produce two memo entries. Confirms
|
||||
the memo keys on raw textContent, not on something coarser like
|
||||
container identity."""
|
||||
bad1 = "A[label (with parens)] --> B"
|
||||
bad2 = "C[other (label)] --> D"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad1, bad2])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidNormalizeCache.size,
|
||||
hasBad1: _mermaidNormalizeCache.has(sources[0]),
|
||||
hasBad2: _mermaidNormalizeCache.has(sources[1]),
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 2
|
||||
assert out["hasBad1"] is True
|
||||
assert out["hasBad2"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Code-fence pairing — close requires \n / EOS, content can't cross
|
||||
# another close-pattern. Repros the streaming bug where ```mermaid +
|
||||
# later ```python were paired by the regex, handing mermaid a
|
||||
# truncated source.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _render_md(source: str) -> str:
|
||||
"""Drive renderMarkdown against the JS harness and return the
|
||||
rendered HTML. The function is a pure string transform; no DOM
|
||||
container scaffolding is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = renderMarkdown(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
_FENCE = "```"
|
||||
|
||||
|
||||
def test_fence_partial_open_emits_no_code_block() -> None:
|
||||
"""While a fence is still open and there's no other ``` later in
|
||||
the buffer, no <code> block is emitted — the open fence stays as
|
||||
plain markdown text until the real close arrives."""
|
||||
src = "Intro\n" + _FENCE + 'mermaid\nA["x"] -->|note (with parens)| B["y"]\nstill streaming'
|
||||
html = _render_md(src)
|
||||
assert "<code" not in html, f"open fence should not emit <code> mid-stream: {html!r}"
|
||||
|
||||
|
||||
def test_fence_partial_with_later_open_does_not_pair_wrongly() -> None:
|
||||
"""Before the fence-pair fix: an unclosed ```mermaid followed by
|
||||
a ```python (also unclosed) would have paired up as
|
||||
<code class=mermaid>...</code>python..., handing mermaid a
|
||||
truncated source. With the new regex, neither fence emits a
|
||||
block until its OWN closing line arrives."""
|
||||
src = "Intro\n" + _FENCE + "mermaid\nA --> B\n" + _FENCE + 'python\nprint("hi")'
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, (
|
||||
f"mermaid fence should not emit while open: {html!r}"
|
||||
)
|
||||
assert 'class="language-python"' not in html, (
|
||||
f"python fence should not emit while open: {html!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_fence_close_paired_with_next_open_is_rejected() -> None:
|
||||
"""Repro of the live-streaming failure: mermaid fence open, then
|
||||
```python opens and ``` closes the python block. Without the
|
||||
fix, the regex paired mermaid's open with python's *open* (or
|
||||
backtracked all the way to python's close), producing
|
||||
<code class=mermaid>truncated</code>. With the fix mermaid stays
|
||||
open (content can't cross another \\1 run; close must be at line
|
||||
boundary) and only python's pair matches."""
|
||||
src = (
|
||||
"Intro\n"
|
||||
+ _FENCE
|
||||
+ 'mermaid\nA["x"] -->|note (with parens)| B["y"]\n'
|
||||
+ _FENCE
|
||||
+ 'python\nprint("hi")\n'
|
||||
+ _FENCE
|
||||
)
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, f"mermaid fence misparing reintroduced: {html!r}"
|
||||
assert 'class="language-python"' in html, f"python fence on its own should match: {html!r}"
|
||||
|
||||
|
||||
def test_fence_closed_emits_code_block() -> None:
|
||||
"""Baseline: a properly closed fence with its close on its own
|
||||
line emits the <code> block as expected — the anchor doesn't
|
||||
break the normal case."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + "\nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_at_end_of_buffer_emits() -> None:
|
||||
"""A fence that closes at the very end of the buffer (no trailing
|
||||
newline) still emits — the anchor accepts end-of-string as a
|
||||
valid line boundary, so the rehydration / static-render path
|
||||
where the buffer ends cleanly at ``` still works."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_with_trailing_whitespace_emits() -> None:
|
||||
"""A close followed only by spaces / tabs before \\n still counts
|
||||
— CommonMark allows trailing whitespace on the close line."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + " \nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# postRenderHljs — progressive syntax highlighting + source-keyed cache
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_hljs_container_js(blocks: list[tuple[str, str]]) -> str:
|
||||
"""Build a container with <pre><code class="language-LANG"> blocks.
|
||||
|
||||
``blocks`` is a list of ``(language, source)`` tuples — the language
|
||||
becomes the ``language-X`` class, the source becomes textContent."""
|
||||
arr = "[" + ", ".join(f"[{json.dumps(lang)}, {json.dumps(src)}]" for lang, src in blocks) + "]"
|
||||
return f"""
|
||||
function buildHljsContainer(blocks) {{
|
||||
const container = document.createElement('div');
|
||||
for (const [lang, src] of blocks) {{
|
||||
const pre = document.createElement('pre');
|
||||
const code = document.createElement('code');
|
||||
code.classList.add('language-' + lang);
|
||||
code.textContent = src;
|
||||
pre.appendChild(code);
|
||||
container.appendChild(pre);
|
||||
}}
|
||||
return container;
|
||||
}}
|
||||
const blocks = {arr};
|
||||
const container = buildHljsContainer(blocks);
|
||||
"""
|
||||
|
||||
|
||||
def test_hljs_cache_hit_skips_highlight_call() -> None:
|
||||
"""Two postRenderHljs calls on identical source must invoke
|
||||
hljs.highlightElement exactly once — the second call hits the
|
||||
cache and applies the stored markup synchronously. Mirrors the
|
||||
mermaid SVG-cache invariant that lets streamingRender fire on
|
||||
every rAF tick without re-tokenizing every code block."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
const container2 = buildHljsContainer(blocks);
|
||||
postRenderHljs(container2);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
firstHtml: container.children[0].children[0]._innerHTML,
|
||||
secondHtml: container2.children[0].children[0]._innerHTML,
|
||||
secondHasHljsClass: container2.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1, (
|
||||
"second postRenderHljs call invoked highlightElement — cache miss"
|
||||
)
|
||||
assert out["cacheSize"] == 1
|
||||
assert out["firstHtml"] == out["secondHtml"]
|
||||
assert out["secondHasHljsClass"] is True
|
||||
|
||||
|
||||
def test_hljs_distinct_sources_highlight_independently() -> None:
|
||||
"""Distinct sources each trigger one highlight and cache one entry.
|
||||
Cache key includes the source string, not e.g. just the language."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os"), ("python", "print('hi')")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_cache_separates_by_language() -> None:
|
||||
"""Same source text under different language fences must NOT
|
||||
collide in the cache — language is part of the key. Otherwise a
|
||||
`python` block of `foo` and a `ruby` block of `foo` would share
|
||||
a single (wrongly-highlighted) cache entry."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "foo"), ("ruby", "foo")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_skips_no_highlight_langs() -> None:
|
||||
"""language-mermaid / language-text / language-plaintext etc. must
|
||||
get the `nohighlight` class without invoking hljs.highlightElement.
|
||||
Highlighting plaintext or mermaid source would be both wasteful
|
||||
and ugly."""
|
||||
scenario = (
|
||||
_build_hljs_container_js(
|
||||
[("mermaid", "graph TD\\nA-->B"), ("text", "plain"), ("plaintext", "p")]
|
||||
)
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
mermaidNoHighlight: container.children[0].children[0]._classes.has('nohighlight'),
|
||||
textNoHighlight: container.children[1].children[0]._classes.has('nohighlight'),
|
||||
plaintextNoHighlight: container.children[2].children[0]._classes.has('nohighlight'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 0
|
||||
assert out["cacheSize"] == 0
|
||||
assert out["mermaidNoHighlight"] is True
|
||||
assert out["textNoHighlight"] is True
|
||||
assert out["plaintextNoHighlight"] is True
|
||||
|
||||
|
||||
def test_hljs_terminal_lang_marks_pre_for_terminal_styling() -> None:
|
||||
"""Shell-family languages (bash / sh / zsh / console / terminal)
|
||||
must add the `code-terminal` class to the parent <pre>, so the
|
||||
stylesheet can give them the terminal look-and-feel."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("bash", "echo hi")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
preHasTerminalClass: container.children[0]._classes.has('code-terminal'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["preHasTerminalClass"] is True
|
||||
|
||||
|
||||
def test_hljs_cache_evicts_oldest_at_cap() -> None:
|
||||
"""FIFO eviction at _HLJS_CACHE_MAX. Mirrors the mermaid cache —
|
||||
prevents unbounded growth on long sessions with many distinct
|
||||
code blocks."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
hasNewest: _hljsCache.has('key-' + (cap + 4)),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is False
|
||||
assert out["hasNewest"] is True
|
||||
|
||||
|
||||
def test_hljs_overwrite_does_not_evict() -> None:
|
||||
"""Overwriting an existing key is an in-place update, not a new
|
||||
insertion — must not evict the oldest unrelated entry. Same
|
||||
invariant as the mermaid cache."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
_cacheFifoEntry(_hljsCache, 'key-5', 'val-updated', cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
updated: _hljsCache.get('key-5'),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is True, "overwrite evicted oldest unnecessarily"
|
||||
assert out["updated"] == "val-updated"
|
||||
|
||||
|
||||
def test_post_render_markdown_invokes_hljs() -> None:
|
||||
"""postRenderMarkdown is the public end-of-stream entry point and
|
||||
must still run syntax highlighting after the postRenderHljs
|
||||
refactor — regression guard for the public API surface that
|
||||
app.js / coordinator code already call."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderMarkdown(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
hasHljsClass: container.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["hasHljsClass"] is True
|
||||
|
||||
|
||||
def test_streaming_render_invokes_hljs() -> None:
|
||||
"""_streamingRenderApply must call postRenderHljs so closed code
|
||||
fences appear progressively (syntax-highlighted) during streaming,
|
||||
not only at stream_end via streamingRenderFinalize. The cache
|
||||
keeps the per-tick cost down to a synchronous lookup."""
|
||||
body = _RENDERER_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function _streamingRenderApply")
|
||||
hljs_call = body.find("postRenderHljs(el)", start, start + 4000)
|
||||
assert hljs_call != -1, (
|
||||
"_streamingRenderApply must call postRenderHljs for progressive "
|
||||
"syntax highlighting during streaming"
|
||||
)
|
||||
|
||||
@@ -14,6 +14,12 @@ from turnstone.core.session import ChatSession
|
||||
class NullUI:
|
||||
"""UI adapter that discards all output."""
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
|
||||
@@ -297,6 +297,49 @@ def test_extra_fields_ignored():
|
||||
assert e.text == "hi"
|
||||
|
||||
|
||||
def test_in_progress_snapshot_event_round_trip():
|
||||
from turnstone.sdk.events import InProgressSnapshotEvent
|
||||
|
||||
payload = {
|
||||
"type": "in_progress_snapshot",
|
||||
"ws_id": "ws1",
|
||||
"content": "Partial content...",
|
||||
"reasoning": "Partial reasoning...",
|
||||
}
|
||||
e = ServerEvent.from_dict(payload)
|
||||
assert isinstance(e, InProgressSnapshotEvent)
|
||||
assert e.ws_id == "ws1"
|
||||
assert e.content == "Partial content..."
|
||||
assert e.reasoning == "Partial reasoning..."
|
||||
|
||||
|
||||
def test_in_progress_snapshot_event_strips_internal_seq():
|
||||
"""``_seq`` is server-internal plumbing — even if a stray copy
|
||||
leaks through, ``from_dict`` must drop it (not a declared field)."""
|
||||
from turnstone.sdk.events import InProgressSnapshotEvent
|
||||
|
||||
e = ServerEvent.from_dict(
|
||||
{
|
||||
"type": "in_progress_snapshot",
|
||||
"ws_id": "ws1",
|
||||
"content": "x",
|
||||
"reasoning": "",
|
||||
"_seq": 42,
|
||||
}
|
||||
)
|
||||
assert isinstance(e, InProgressSnapshotEvent)
|
||||
assert not hasattr(e, "_seq")
|
||||
|
||||
|
||||
def test_state_change_event_round_trip():
|
||||
from turnstone.sdk.events import StateChangeEvent
|
||||
|
||||
e = ServerEvent.from_dict({"type": "state_change", "ws_id": "ws1", "state": "thinking"})
|
||||
assert isinstance(e, StateChangeEvent)
|
||||
assert e.state == "thinking"
|
||||
assert e.ws_id == "ws1"
|
||||
|
||||
|
||||
def test_missing_type_defaults_to_base():
|
||||
e = ServerEvent.from_dict({"ws_id": "ws1"})
|
||||
assert type(e) is ServerEvent
|
||||
|
||||
@@ -70,6 +70,12 @@ class RecordingUI:
|
||||
self.errors: list[str] = []
|
||||
self.infos: list[str] = []
|
||||
|
||||
def on_turn_start(self):
|
||||
self.events.append(("turn_start",))
|
||||
|
||||
def on_turn_committed(self):
|
||||
self.events.append(("turn_committed",))
|
||||
|
||||
def on_thinking_start(self):
|
||||
self.events.append(("thinking_start",))
|
||||
|
||||
|
||||
+542
-3
@@ -14,6 +14,12 @@ from turnstone.core.session import _IMAGE_EXTENSIONS, _IMAGE_SIZE_CAP, ChatSessi
|
||||
class NullUI:
|
||||
"""UI adapter that discards all output. Used for testing."""
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
@@ -452,6 +458,213 @@ class TestPlanExec:
|
||||
assert messages[0]["content"] == ChatSession._PLAN_IDENTITY
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests — _exec_task (optional skill substitutes the hardcoded identity)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTaskExec:
|
||||
"""Tests for _exec_task: optional skill= replaces the default persona,
|
||||
but operating guidance (one-shot, tool-use over narration, no follow-ups)
|
||||
is always preserved."""
|
||||
|
||||
@staticmethod
|
||||
def _capture_exec_messages(session, item):
|
||||
"""Run _exec_task with _run_agent patched; return system message text."""
|
||||
captured: dict = {}
|
||||
|
||||
def fake_run_agent(messages, **kwargs):
|
||||
captured["messages"] = list(messages)
|
||||
return "done"
|
||||
|
||||
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
|
||||
session._exec_task(item)
|
||||
return captured["messages"][0]["content"]
|
||||
|
||||
def test_known_skill_renders_into_system_message(self, tmp_db) -> None:
|
||||
"""Validated skill content (with template vars resolved) replaces
|
||||
the default '# Task Agent' persona, but the operating guidance
|
||||
(the numbered list) is preserved — those are sub-agent semantics
|
||||
that a persona should layer on top of, not replace.
|
||||
|
||||
Covers the full prepare→exec round-trip so a future regression
|
||||
in either half (skill not stored on the item, or exec ignoring it)
|
||||
is caught."""
|
||||
session = _make_session()
|
||||
skill = {
|
||||
"name": "research",
|
||||
"content": "# Research Agent\nws={{ws_id}} model={{model}} node={{node_id}}",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
|
||||
# Item carries the minimized projection — name/content/risk_level
|
||||
# only — not the raw prompt_templates row.
|
||||
assert item["skill"] == {
|
||||
"name": "research",
|
||||
"content": skill["content"],
|
||||
"risk_level": "",
|
||||
}
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: research" in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
# Skill persona rendered with template vars resolved
|
||||
assert "# Research Agent" in sys_msg
|
||||
assert f"ws={session._ws_id}" in sys_msg
|
||||
assert f"model={session.model}" in sys_msg
|
||||
# Default persona is gone — skill substitutes for it.
|
||||
assert "# Task Agent" not in sys_msg
|
||||
assert "autonomous task agent with full tool access" not in sys_msg
|
||||
# Operating guidance survives regardless of skill.
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
|
||||
def test_omitted_skill_uses_hardcoded_identity(self, tmp_db) -> None:
|
||||
"""Regression guard: without skill=, the default '# Task Agent'
|
||||
persona AND the operating guidance both appear verbatim.
|
||||
|
||||
Pins the no-skill path so the substitution branch can't
|
||||
accidentally swallow the default case."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
# Default-persona literals also present (sanity check on the constant).
|
||||
assert "# Task Agent" in sys_msg
|
||||
assert "autonomous task agent with full tool access" in sys_msg
|
||||
|
||||
@pytest.mark.parametrize("skill_value", ["", " ", "\t\n"])
|
||||
def test_prepare_task_empty_or_whitespace_skill_treated_as_omitted(
|
||||
self, tmp_db, skill_value
|
||||
) -> None:
|
||||
"""Documented contract: ``skill=""`` (and whitespace-only) behaves
|
||||
identically to omitting the skill arg. LLMs sometimes echo empty
|
||||
strings rather than omit the field; this pins the documented
|
||||
behavior so a future refactor of the ``(args.get("skill") or "").strip()``
|
||||
chokepoint can't quietly diverge."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": skill_value})
|
||||
assert item.get("needs_approval") is True
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
def test_prepare_task_unknown_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Unknown skill name → clean error item, no approval needed.
|
||||
|
||||
Skill validation lives in _prepare_task so an LLM passing a
|
||||
bogus name fails fast at approval time rather than at exec."""
|
||||
session = _make_session()
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=None):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "ghost"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown skill 'ghost'" in item["error"]
|
||||
assert "skill(action='search')" in item["error"]
|
||||
|
||||
def test_prepare_task_disabled_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Disabled skill → distinct error, mirrors the enabled gate that
|
||||
``_exec_skill(action='load')`` (session.py:8404) and skill-search
|
||||
already apply. Distinct from the unknown-skill phrasing so the
|
||||
LLM's recovery path can tell 'not found' from 'quarantined'."""
|
||||
session = _make_session()
|
||||
disabled_skill = {
|
||||
"name": "retired",
|
||||
"content": "# Retired",
|
||||
"enabled": False,
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=disabled_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "retired"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "is disabled" in item["error"]
|
||||
# Distinct wording from the unknown-skill error, so the LLM can
|
||||
# tell them apart at recovery time.
|
||||
assert "unknown skill" not in item["error"]
|
||||
|
||||
def test_prepare_task_high_risk_skill_surfaces_in_header(self, tmp_db, caplog) -> None:
|
||||
"""High/critical risk skills surface the tier in the approval header
|
||||
and emit a structured warning, mirroring the signal ``_load_skills``
|
||||
emits for session-level skills (session.py:1336)."""
|
||||
import logging
|
||||
|
||||
session = _make_session()
|
||||
risky_skill = {
|
||||
"name": "danger",
|
||||
"content": "# Danger",
|
||||
"enabled": True,
|
||||
"risk_level": "critical",
|
||||
}
|
||||
with (
|
||||
caplog.at_level(logging.WARNING, logger="turnstone.core.session"),
|
||||
patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill),
|
||||
):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "danger"})
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: danger" in item["header"]
|
||||
assert "risk: critical" in item["header"]
|
||||
warning_seen = any("high_risk_skill" in r.getMessage() for r in caplog.records)
|
||||
assert warning_seen, "expected task_agent.high_risk_skill warning"
|
||||
|
||||
def test_prepare_task_normal_risk_skill_omits_tier_from_header(self, tmp_db) -> None:
|
||||
"""Header only surfaces high/critical — low/medium/safe skills don't
|
||||
pollute the approval line."""
|
||||
session = _make_session()
|
||||
ok_skill = {
|
||||
"name": "research",
|
||||
"content": "# Research",
|
||||
"enabled": True,
|
||||
"risk_level": "low",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=ok_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "research"})
|
||||
assert "skill: research" in item["header"]
|
||||
assert "risk:" not in item["header"]
|
||||
|
||||
def test_evaluate_intent_projects_skill_for_task_agent(self, tmp_db, monkeypatch) -> None:
|
||||
"""Judge projection includes the skill name so heuristic arg_patterns
|
||||
can match on it and the audit row records which persona was chosen.
|
||||
|
||||
Mirrors the long-standing ``spawn_workstream`` projection at
|
||||
session.py:4603 — without it, policy rules targeting risky
|
||||
skills via ``task_agent`` silently no-op."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
skill = {"name": "research", "content": "# Research", "enabled": True}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == "research"
|
||||
assert fa["prompt"] == "investigate X"
|
||||
|
||||
def test_evaluate_intent_projects_empty_skill_when_omitted(self, tmp_db, monkeypatch) -> None:
|
||||
"""Symmetric regression guard: no-skill case projects skill="" so
|
||||
the func_args shape is stable across both branches (the judge can
|
||||
always read ``func_args["skill"]`` without a KeyError)."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == ""
|
||||
assert fa["prompt"] == "do x"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-call model override on plan_agent / task_agent
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -498,9 +711,51 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
# The error guidance must list the available aliases so the LLM can retry.
|
||||
for alias in ("default", "smart", "fast"):
|
||||
# Error guidance lists the aliases the LLM may retry, intentionally
|
||||
# excluding ``default`` — that alias is operator-only (see
|
||||
# ``test_prepare_plan_default_model_rejected``). Surfacing it here
|
||||
# would re-enable the per-role-override bypass even though the
|
||||
# tool description hides it.
|
||||
for alias in ("smart", "fast"):
|
||||
assert alias in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected(self, tmp_db) -> None:
|
||||
"""``model="default"`` is rejected even when the alias exists in
|
||||
the registry — bypasses the operator-configured ``plan_alias``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
assert "Omit `model=`" in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected_with_whitespace(self, tmp_db) -> None:
|
||||
"""The ``default`` rejection runs after ``strip()`` so leading/
|
||||
trailing whitespace can't sneak the alias past the carve-out."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": " default "})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
def test_prepare_plan_unknown_model_with_only_default_in_registry(self, tmp_db) -> None:
|
||||
"""When the registry holds only the reserved ``default`` alias
|
||||
(single-CLI-model back-compat), the unknown-alias error must say
|
||||
'(no alternative aliases configured — omit `model=`)' — not the
|
||||
misleading '(no registry configured)' that suggests routing isn't
|
||||
wired up at all."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "bogus"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "no alternative aliases configured" in item["error"]
|
||||
assert "no registry configured" not in item["error"]
|
||||
|
||||
# ---- _prepare_task ----
|
||||
|
||||
@@ -520,6 +775,15 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_task_default_model_rejected(self, tmp_db) -> None:
|
||||
"""Symmetric carve-out for task_agent — see
|
||||
``test_prepare_plan_default_model_rejected``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
# ---- tool description rendering ----
|
||||
|
||||
@@ -538,8 +802,11 @@ class TestAgentModelOverride:
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None, f"{name} missing from session tools"
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
for alias in ("default", "smart", "fast"):
|
||||
for alias in ("smart", "fast"):
|
||||
assert f"`{alias}`" in desc, f"alias {alias} missing from {desc!r}"
|
||||
# ``default`` is intentionally hidden — see
|
||||
# ``test_render_omits_default_alias_from_description``.
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_no_op_without_registry(self, tmp_db) -> None:
|
||||
"""No registry → leave the placeholder description untouched."""
|
||||
@@ -570,6 +837,82 @@ class TestAgentModelOverride:
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`bigboi`" in desc
|
||||
|
||||
def test_render_omits_default_alias_from_description(self, tmp_db) -> None:
|
||||
"""The ``default`` alias is filtered from the LLM-facing alias list.
|
||||
|
||||
Reading "default" as English ("use the default") and passing it
|
||||
explicitly bypasses the operator-configured per-role plan_alias /
|
||||
task_alias. The LLM should reach the per-role default by omitting
|
||||
``model=`` instead.
|
||||
"""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"gh200": ModelConfig("gh200", "x", "x", "m"),
|
||||
"opus-4.7": ModelConfig("opus-4.7", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
for name in ("plan_agent", "task_agent"):
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`gh200`" in desc
|
||||
assert "`opus-4.7`" in desc
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_falls_back_to_base_when_only_default_alias(self, tmp_db) -> None:
|
||||
"""Single-CLI-model registries (only ``default`` in registry) leave
|
||||
the base description untouched — the LLM sees ``"No alternative
|
||||
aliases configured"`` rather than an empty alias list."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_refresh_into_only_default_resets_to_base(self, tmp_db) -> None:
|
||||
"""A reload that drops the registry to only ``default`` must clear
|
||||
stale alias names from the previously-rendered tool descriptions —
|
||||
not return early and leave them in place."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"smart": ModelConfig("smart", "x", "x", "m"),
|
||||
"fast": ModelConfig("fast", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
# Sanity: initial render carries the non-default aliases.
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" in desc and "`fast`" in desc
|
||||
|
||||
# Reload the registry down to only ``default`` (admin removed
|
||||
# every other model definition).
|
||||
reg.reload({"default": ModelConfig("default", "x", "x", "m")}, "default")
|
||||
session.refresh_agent_tool_schemas()
|
||||
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "`fast`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_module_level_constants_not_mutated(self, tmp_db) -> None:
|
||||
"""Rendering must not pollute the module-level TOOLS list shared
|
||||
across all sessions."""
|
||||
@@ -1968,6 +2311,202 @@ class TestCoordinatorMemoryScope:
|
||||
assert scopes == ["workstream", "user", "global"]
|
||||
|
||||
|
||||
class TestMemoryToolAudit:
|
||||
"""Mutating memory tool actions emit audit rows.
|
||||
|
||||
Closes the gap that masked the May 2026 vllm_fork_overlay_pattern
|
||||
investigation: only the admin-console DELETE route emitted
|
||||
``memory.delete``, so a long-running session whose memory was
|
||||
deleted via the admin UI couldn't tell from logs alone whether the
|
||||
row had been deleted out-of-band, never persisted, or was never
|
||||
visible. Read actions (get/search/list) intentionally stay
|
||||
un-audited — auditing reads would multiply audit volume without
|
||||
forensic value.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _audit_rows(action: str) -> list[dict]:
|
||||
from turnstone.core.storage._registry import get_storage
|
||||
|
||||
return get_storage().list_audit_events(action=action)
|
||||
|
||||
def test_save_new_emits_memory_save(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha content",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row["user_id"] == "user-1"
|
||||
assert row["resource_type"] == "memory"
|
||||
assert row["resource_id"] # memory_id was populated
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["scope_id"] == "user-1"
|
||||
assert detail["type"] == "reference"
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
# The "create" path must NOT also stamp an update row.
|
||||
assert self._audit_rows("memory.update") == []
|
||||
|
||||
def test_save_global_scope_emits_empty_scope_id(self, tmp_db):
|
||||
"""Global memories have no scope_id — the audit row's detail
|
||||
must still carry the key (with value ``""``) so a forensic
|
||||
consumer can distinguish ``scope='global'`` from a row that
|
||||
forgot to populate ``scope_id`` for a scoped write."""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_global",
|
||||
"content": "shared content",
|
||||
"scope": "global",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["scope"] == "global"
|
||||
assert detail["scope_id"] == ""
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
|
||||
def test_save_upsert_emits_memory_update(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
for content in ("first", "second"):
|
||||
item = session._prepare_memory(
|
||||
"call_x",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": content,
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(item)
|
||||
|
||||
saves = self._audit_rows("memory.save")
|
||||
updates = self._audit_rows("memory.update")
|
||||
assert len(saves) == 1
|
||||
assert len(updates) == 1
|
||||
# Same memory_id on both rows — the update audits the row save created.
|
||||
assert saves[0]["resource_id"] == updates[0]["resource_id"]
|
||||
|
||||
def test_delete_emits_memory_delete(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
save_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(save_item)
|
||||
saved_memory_id = self._audit_rows("memory.save")[0]["resource_id"]
|
||||
|
||||
delete_item = session._prepare_memory(
|
||||
"call_2",
|
||||
{"action": "delete", "name": "fact_one", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "Deleted memory" in msg
|
||||
|
||||
rows = self._audit_rows("memory.delete")
|
||||
assert len(rows) == 1
|
||||
# resource_id must point at the same row save audited — proves
|
||||
# delete-by-name resolved to the right row before recording.
|
||||
assert rows[0]["resource_id"] == saved_memory_id
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["type"] == "reference"
|
||||
|
||||
def test_delete_not_found_emits_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
delete_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{"action": "delete", "name": "no_such_mem", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "not found" in msg
|
||||
assert self._audit_rows("memory.delete") == []
|
||||
|
||||
def test_reads_emit_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
session._exec_memory(
|
||||
session._prepare_memory(
|
||||
"call_save",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
for spec in (
|
||||
{"action": "get", "name": "fact_one", "scope": "user"},
|
||||
{"action": "search", "query": "fact"},
|
||||
{"action": "list"},
|
||||
):
|
||||
item = session._prepare_memory("call_read", spec)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
# Only the save above should have audited.
|
||||
save_count = len(self._audit_rows("memory.save"))
|
||||
update_count = len(self._audit_rows("memory.update"))
|
||||
delete_count = len(self._audit_rows("memory.delete"))
|
||||
assert (save_count, update_count, delete_count) == (1, 0, 0)
|
||||
|
||||
def test_audit_failure_does_not_break_tool_call(self, tmp_db):
|
||||
"""A blow-up inside record_audit must not propagate to the LLM.
|
||||
|
||||
Auditing is best-effort instrumentation; a storage hiccup that
|
||||
prevents the audit row from landing must not also lose the
|
||||
save/delete the user actually asked for.
|
||||
"""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
with patch(
|
||||
"turnstone.core.audit.record_audit",
|
||||
side_effect=RuntimeError("audit storage exploded"),
|
||||
):
|
||||
_, msg = session._exec_memory(item)
|
||||
assert "Saved memory 'fact_one'" in msg
|
||||
# The save itself still landed.
|
||||
from turnstone.core.memory import get_structured_memory_by_name
|
||||
|
||||
assert get_structured_memory_by_name("fact_one", "user", "user-1") is not None
|
||||
|
||||
|
||||
class TestPerKindToolVariants:
|
||||
"""Verify the ``kind_variants`` metadata applies per-kind tool overrides.
|
||||
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
"""Tests for :meth:`ChatSession._format_backend_error`.
|
||||
|
||||
The helper turns bare backend-boundary exceptions (httpx ``ReadTimeout``,
|
||||
OpenAI SDK ``APITimeoutError`` / ``APIConnectionError`` /
|
||||
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``) into
|
||||
operator-actionable messages that include the provider, base URL, and
|
||||
model. We bind the method to lightweight stubs rather than constructing
|
||||
a full :class:`ChatSession`: the helper only reads ``self.client``,
|
||||
``self._provider``, ``self.model``, and ``self._model_alias``, so a
|
||||
SimpleNamespace stub exercises the same surface without dragging in the
|
||||
storage / prompt composition fixtures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
|
||||
def _stub(
|
||||
*,
|
||||
base_url: str = "http://192.168.0.5:8000/v1",
|
||||
provider_name: str = "openai-compatible",
|
||||
model: str = "flatspark",
|
||||
model_alias: str | None = "flatspark",
|
||||
client_attr: str = "base_url",
|
||||
) -> Any:
|
||||
"""Build a minimal session-like stub for ``_format_backend_error``.
|
||||
|
||||
``client_attr`` selects which attribute on the client carries the
|
||||
URL — both ``base_url`` (OpenAI / Anthropic SDK public surface) and
|
||||
``_base_url`` (httpx fallback) are exercised by the helper.
|
||||
"""
|
||||
client_kwargs: dict[str, Any] = {client_attr: base_url}
|
||||
return SimpleNamespace(
|
||||
client=SimpleNamespace(**client_kwargs),
|
||||
_provider=SimpleNamespace(provider_name=provider_name),
|
||||
model=model,
|
||||
_model_alias=model_alias,
|
||||
)
|
||||
|
||||
|
||||
def _format(stub: Any, exc: BaseException) -> str | None:
|
||||
"""Invoke the method as if on a real session — ``__func__`` skips
|
||||
the descriptor protocol so we can pass any object as ``self``."""
|
||||
return ChatSession._format_backend_error(stub, exc) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synthetic exception classes — class name is what the helper matches on,
|
||||
# so we don't need real httpx / openai imports here.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# N818 (Error suffix on Exception names) is intentionally suppressed
|
||||
# for the four classes below — they exist to impersonate httpx /
|
||||
# Anthropic SDK exception class names verbatim, since the formatter
|
||||
# matches by class name. Renaming them defeats the test.
|
||||
|
||||
|
||||
class ReadTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class WriteTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APITimeoutError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectError(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class ConnectTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APIConnectionError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class NotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class AuthenticationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PermissionDeniedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class RateLimitError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SomeUnrelatedError(Exception):
|
||||
"""Outside the recognised set — should fall through to ``None``."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known categories — each branch produces an operator-actionable message
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ReadTimeout, WriteTimeout, APITimeoutError])
|
||||
def test_timeout_message_names_backend_and_model(exc_cls):
|
||||
msg = _format(_stub(), exc_cls())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "openai-compatible" in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
assert "model=flatspark" in msg
|
||||
assert "wedged" in msg
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ConnectError, ConnectTimeout, APIConnectionError])
|
||||
def test_connect_message_says_unreachable(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("dial tcp: i/o timeout"))
|
||||
assert msg is not None
|
||||
assert "Backend unreachable" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
# Raw exception text is preserved as a tail for grep-correlation.
|
||||
assert "dial tcp: i/o timeout" in msg
|
||||
|
||||
|
||||
def test_not_found_points_at_model_name_mismatch():
|
||||
msg = _format(_stub(model="flatspark"), NotFoundError("model flatspark not found"))
|
||||
assert msg is not None
|
||||
assert "Backend reports model not loaded" in msg
|
||||
assert "no model named 'flatspark'" in msg
|
||||
assert "/v1/models" in msg # operator hint
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [AuthenticationError, PermissionDeniedError])
|
||||
def test_auth_message_mentions_api_key(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("invalid api key"))
|
||||
assert msg is not None
|
||||
assert "Backend rejected credentials" in msg
|
||||
assert "API key" in msg
|
||||
|
||||
|
||||
def test_rate_limit_message():
|
||||
msg = _format(_stub(), RateLimitError("limit exceeded"))
|
||||
assert msg is not None
|
||||
assert "Backend rate-limited" in msg
|
||||
assert "limit exceeded" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fall-through + degradation behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unknown_exception_returns_none():
|
||||
assert _format(_stub(), SomeUnrelatedError("anything")) is None
|
||||
|
||||
|
||||
def test_unknown_exception_value_error_returns_none():
|
||||
assert _format(_stub(), ValueError("not a backend error")) is None
|
||||
|
||||
|
||||
def test_trailing_slash_and_query_string_stripped():
|
||||
msg = _format(
|
||||
_stub(base_url="http://node-a:8000/v1/?api_key=secret&foo=1"),
|
||||
ReadTimeout(),
|
||||
)
|
||||
assert msg is not None
|
||||
assert "http://node-a:8000/v1" in msg
|
||||
# Query string (which may carry credentials) is stripped before the
|
||||
# message is built — sanitize_error_text is a second line of defence
|
||||
# but the helper itself must not embed query params verbatim.
|
||||
assert "api_key" not in msg
|
||||
assert "secret" not in msg
|
||||
|
||||
|
||||
def test_missing_provider_degrades_to_placeholder():
|
||||
stub = _stub()
|
||||
stub._provider = None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
# No exception, no NoneType formatting leaking through.
|
||||
assert "Backend timeout" in msg
|
||||
assert "from ?" in msg or "openai-compatible" not in msg
|
||||
|
||||
|
||||
def test_client_base_url_raises_degrades_gracefully():
|
||||
class _BadClient:
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
stub = SimpleNamespace(
|
||||
client=_BadClient(),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
)
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
# base_url accessor blew up — message still renders with placeholder.
|
||||
assert "at ?" in msg
|
||||
|
||||
|
||||
def test_httpx_underscore_base_url_fallback():
|
||||
# httpx client carries ``_base_url`` on some versions instead of
|
||||
# ``base_url`` — the helper checks both.
|
||||
stub = _stub(base_url="http://alt-host:9000", client_attr="_base_url")
|
||||
# SimpleNamespace exposes the attr; remove the public one so the
|
||||
# fallback path is exercised.
|
||||
delattr(stub.client, "base_url") if hasattr(stub.client, "base_url") else None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "http://alt-host:9000" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration with _record_fatal_error — original bare-class string is
|
||||
# replaced by the enriched message when the exception type is recognised.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _record_fatal_stub(ui: Any, captured: dict[str, str]) -> Any:
|
||||
"""Build a stub for the ``_record_fatal_error`` integration tests.
|
||||
|
||||
``_record_fatal_error`` calls ``self._format_backend_error(...)``
|
||||
internally, so the stub binds the unbound method to itself rather
|
||||
than relying on Python's descriptor protocol (which only kicks in
|
||||
when ``self`` is a real instance of the class)."""
|
||||
stub = SimpleNamespace(
|
||||
client=SimpleNamespace(base_url="http://192.168.0.5:8000/v1"),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
_ws_id="ws-test",
|
||||
_has_persisted_error=False,
|
||||
ui=ui,
|
||||
_emit_state=lambda state: captured.setdefault("state", state),
|
||||
)
|
||||
stub._format_backend_error = lambda exc: ChatSession._format_backend_error(stub, exc)
|
||||
return stub
|
||||
|
||||
|
||||
def test_record_fatal_uses_enriched_message_for_known(monkeypatch):
|
||||
"""End-to-end: a recognised exception flows through
|
||||
``_record_fatal_error`` and the enriched text reaches both the UI
|
||||
and the persist hook."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
# Skip the credential-redaction module (and its module-level
|
||||
# regex compile) by returning the input verbatim — the helper
|
||||
# under test produces no credentials.
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ReadTimeout()) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors, "UI never received error"
|
||||
assert "Backend timeout" in ui.errors[0]
|
||||
assert "ReadTimeout" in ui.errors[0]
|
||||
assert captured["persist"] == ui.errors[0]
|
||||
assert captured["state"] == "error"
|
||||
assert stub._has_persisted_error is True
|
||||
|
||||
|
||||
def test_record_fatal_falls_back_for_unknown(monkeypatch):
|
||||
"""An unrecognised exception keeps the legacy
|
||||
``f"{type(exc).__name__}: {exc}"`` shape so we don't regress
|
||||
existing call sites that grep on it."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ValueError("plain old error")) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors == ["ValueError: plain old error"]
|
||||
assert captured["persist"] == "ValueError: plain old error"
|
||||
@@ -0,0 +1,648 @@
|
||||
"""Tests for session-level ``replay_reasoning_to_model`` plumbing.
|
||||
|
||||
Phase 2 of optional reasoning persistence reads the per-model
|
||||
``ModelConfig.replay_reasoning_to_model`` flag at the wire-build call
|
||||
site and threads it through ``provider.create_streaming`` /
|
||||
``provider.create_completion``. These tests pin:
|
||||
|
||||
1. The resolver helper (``ChatSession._resolve_replay_reasoning_to_model``)
|
||||
walks the registry correctly and falls back to ``False`` (the
|
||||
conservative default matching the migration server_default) when
|
||||
the lookup fails.
|
||||
2. The streaming wire-build call site at ``session.py:_try_stream``
|
||||
actually passes the resolved flag down — without this, the Phase
|
||||
2 work is dead code (the strip-when-False predicate never fires).
|
||||
3. The non-streaming wire-build call site at
|
||||
``session.py:_utility_completion`` does the same.
|
||||
|
||||
Drives through the real ``ChatSession._resolve_replay_reasoning_to_model``
|
||||
with a stub registry, then captures the kwarg passed to a mock provider
|
||||
to verify the flow end-to-end.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._session_helpers import make_session as _make_session
|
||||
|
||||
|
||||
def _registry_with_flag(persist: bool = True, replay: bool = False) -> Any:
|
||||
"""Stub registry returning a ModelConfig-shaped object with the
|
||||
flags under test."""
|
||||
return SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
surface_persisted_reasoning=persist,
|
||||
replay_reasoning_to_model=replay,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class TestResolveReplayReasoningToModel:
|
||||
"""Direct unit tests for the resolver."""
|
||||
|
||||
def test_returns_false_when_no_registry(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = None
|
||||
session._model_alias = "anything"
|
||||
assert session._resolve_replay_reasoning_to_model() is False
|
||||
|
||||
def test_returns_false_when_no_alias(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = ""
|
||||
assert session._resolve_replay_reasoning_to_model() is False
|
||||
|
||||
def test_returns_false_default(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=False)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
assert session._resolve_replay_reasoning_to_model() is False
|
||||
|
||||
def test_returns_true_when_flag_set(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
assert session._resolve_replay_reasoning_to_model() is True
|
||||
|
||||
def test_explicit_alias_arg_overrides_default(self) -> None:
|
||||
session = _make_session()
|
||||
|
||||
def per_alias(alias: str) -> Any:
|
||||
return SimpleNamespace(
|
||||
replay_reasoning_to_model=(alias == "needs-replay"),
|
||||
)
|
||||
|
||||
session._registry = SimpleNamespace(get_config=per_alias)
|
||||
session._model_alias = "primary"
|
||||
# Default reads session._model_alias → False.
|
||||
assert session._resolve_replay_reasoning_to_model() is False
|
||||
# Explicit alias arg → True for "needs-replay".
|
||||
assert session._resolve_replay_reasoning_to_model("needs-replay") is True
|
||||
|
||||
def test_returns_false_on_registry_exception(self) -> None:
|
||||
session = _make_session()
|
||||
|
||||
def boom(alias: str) -> Any:
|
||||
raise KeyError(alias)
|
||||
|
||||
session._registry = SimpleNamespace(get_config=boom)
|
||||
session._model_alias = "missing"
|
||||
# Conservative fallback — losing the strip is a UX nuisance,
|
||||
# but accepting wire-side reasoning replay against an unknown
|
||||
# operator preference is a worse default.
|
||||
assert session._resolve_replay_reasoning_to_model() is False
|
||||
|
||||
def test_caps_none_preserves_back_compat(self) -> None:
|
||||
# When ``caps`` is omitted, the resolver returns the operator
|
||||
# flag unchanged — matching pre-PR behaviour for any caller
|
||||
# that hasn't been updated to thread caps yet.
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
assert session._resolve_replay_reasoning_to_model() is True
|
||||
assert session._resolve_replay_reasoning_to_model(caps=None) is True
|
||||
|
||||
def test_caps_supports_replay_true_passes_through(self) -> None:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
caps = ModelCapabilities(supports_reasoning_replay=True)
|
||||
assert session._resolve_replay_reasoning_to_model(caps=caps) is True
|
||||
|
||||
def test_caps_supports_replay_false_blocks_replay(self) -> None:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
# Operator flipped replay=True but the model's capability
|
||||
# advertises supports_reasoning_replay=False — AND-gate blocks
|
||||
# replay so the strip predicate runs at the wire build.
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "hypothetical-no-replay-claude"
|
||||
caps = ModelCapabilities(supports_reasoning_replay=False)
|
||||
assert session._resolve_replay_reasoning_to_model(caps=caps) is False
|
||||
|
||||
def test_caps_supports_replay_true_does_not_force_replay(self) -> None:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
# Capability True but operator flag False — result must be
|
||||
# False (the AND has to be False on either side).
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=False)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
caps = ModelCapabilities(supports_reasoning_replay=True)
|
||||
assert session._resolve_replay_reasoning_to_model(caps=caps) is False
|
||||
|
||||
|
||||
class TestStreamingCallSitePassesFlag:
|
||||
"""Pin that ``_try_stream`` actually passes the resolved flag to
|
||||
``provider.create_streaming`` — without this the Phase 2 work is
|
||||
dead code at the call site."""
|
||||
|
||||
def test_replay_true_propagates_to_provider(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
# Stub provider: capture the kwargs passed to create_streaming.
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def capture_streaming(**kwargs: Any) -> Any:
|
||||
captured.update(kwargs)
|
||||
return iter([])
|
||||
|
||||
mock_provider = MagicMock()
|
||||
mock_provider.create_streaming = capture_streaming
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
session._try_stream(
|
||||
client=MagicMock(),
|
||||
model="claude-opus-4-7",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=mock_provider,
|
||||
model_alias="claude-opus-4-7",
|
||||
)
|
||||
assert captured["replay_reasoning_to_model"] is True
|
||||
|
||||
def test_replay_false_propagates_to_provider(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=False)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def capture_streaming(**kwargs: Any) -> Any:
|
||||
captured.update(kwargs)
|
||||
return iter([])
|
||||
|
||||
mock_provider = MagicMock()
|
||||
mock_provider.create_streaming = capture_streaming
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
session._try_stream(
|
||||
client=MagicMock(),
|
||||
model="claude-opus-4-7",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=mock_provider,
|
||||
model_alias="claude-opus-4-7",
|
||||
)
|
||||
assert captured["replay_reasoning_to_model"] is False
|
||||
|
||||
def test_fallback_alias_uses_its_own_flag(self) -> None:
|
||||
# When the primary fails and we fall back to an alias with a
|
||||
# different flag, the flag MUST track the resolved alias —
|
||||
# not the session's primary alias.
|
||||
session = _make_session()
|
||||
|
||||
def per_alias(alias: str) -> Any:
|
||||
return SimpleNamespace(
|
||||
replay_reasoning_to_model=(alias == "fallback-with-replay"),
|
||||
)
|
||||
|
||||
session._registry = SimpleNamespace(get_config=per_alias)
|
||||
session._model_alias = "primary" # primary has replay=False
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def capture_streaming(**kwargs: Any) -> Any:
|
||||
captured.update(kwargs)
|
||||
return iter([])
|
||||
|
||||
mock_provider = MagicMock()
|
||||
mock_provider.create_streaming = capture_streaming
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
session._try_stream(
|
||||
client=MagicMock(),
|
||||
model="fallback-model",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=mock_provider,
|
||||
model_alias="fallback-with-replay",
|
||||
)
|
||||
# Resolved against the FALLBACK alias, not the session's primary.
|
||||
assert captured["replay_reasoning_to_model"] is True
|
||||
|
||||
|
||||
class TestSessionToWireBoundaryIntegration:
|
||||
"""End-to-end integration: session._try_stream -> real
|
||||
AnthropicProvider.create_streaming -> captured Anthropic SDK
|
||||
boundary call. Verifies the strip-when-False predicate actually
|
||||
fires at the wire payload, not just at the captured kwarg.
|
||||
|
||||
The bare-function-stub tests above (TestStreamingCallSitePassesFlag)
|
||||
pin that ``_try_stream`` PASSES the flag; this test pins that the
|
||||
real provider USES it. Together they catch:
|
||||
- kwarg renamed at provider boundary -> stub-tests still pass,
|
||||
this one fails on its real-provider assertion.
|
||||
- _convert_messages stops reading the kwarg -> stub-tests still
|
||||
pass, this one fails because the wire payload still carries
|
||||
the thinking block.
|
||||
- _try_stream stops calling create_streaming -> stub-tests fail
|
||||
on the captured kwarg, this one fails because the SDK boundary
|
||||
was never reached.
|
||||
|
||||
Drives through the real ``AnthropicProvider`` with a mock client
|
||||
whose ``client.messages.stream`` is captured — the smallest possible
|
||||
surface that crosses the session->provider->wire boundary chain.
|
||||
|
||||
Negative-tested: temporarily reverting
|
||||
``_anthropic.py:create_streaming``'s
|
||||
``self._convert_messages(messages, replay_reasoning_to_model=...)``
|
||||
call to drop the kwarg makes the wire payload carry the thinking
|
||||
block again; ``test_replay_false_strips_thinking_at_wire`` then
|
||||
fails with ``Strip predicate did not fire at wire boundary``.
|
||||
Restoring the kwarg makes it pass — confirming the test gates the
|
||||
actual wire-build invariant rather than the captured kwarg.
|
||||
"""
|
||||
|
||||
def _stub_anthropic_client(self) -> tuple[MagicMock, dict[str, object]]:
|
||||
"""Build a mock Anthropic client + captured-kwargs dict.
|
||||
|
||||
``client.messages.stream(**kwargs)`` returns a context manager
|
||||
whose ``__enter__`` yields an iterable of zero events — enough
|
||||
to satisfy the ``_iter_with_cleanup`` shape without exercising
|
||||
actual streaming protocol.
|
||||
"""
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def stream(**kwargs: object) -> object:
|
||||
captured.update(kwargs)
|
||||
cm = MagicMock()
|
||||
cm.__enter__ = MagicMock(return_value=iter([]))
|
||||
cm.__exit__ = MagicMock(return_value=False)
|
||||
return cm
|
||||
|
||||
client = MagicMock()
|
||||
client.messages.stream = stream
|
||||
return client, captured
|
||||
|
||||
def _drive_session_through_anthropic(
|
||||
self,
|
||||
replay_flag: bool,
|
||||
msgs: list[dict[str, object]],
|
||||
) -> dict[str, object]:
|
||||
"""Run session._try_stream against a real AnthropicProvider with
|
||||
the resolver pre-set to *replay_flag*. Returns the kwargs
|
||||
dict that reached the (mocked) Anthropic SDK boundary.
|
||||
"""
|
||||
pytest.importorskip("anthropic")
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=replay_flag)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
client, captured = self._stub_anthropic_client()
|
||||
real_provider = AnthropicProvider()
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="claude-opus-4-7",
|
||||
msgs=msgs,
|
||||
provider=real_provider,
|
||||
model_alias="claude-opus-4-7",
|
||||
)
|
||||
# Iterate the stream to drain the (empty) generator and ensure
|
||||
# _ensure_anthropic / convert / build_kwargs all ran.
|
||||
list(stream)
|
||||
return captured
|
||||
|
||||
def test_replay_false_strips_thinking_at_wire(self) -> None:
|
||||
msgs: list[dict[str, object]] = [
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "secret reasoning", "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "ack"},
|
||||
]
|
||||
captured = self._drive_session_through_anthropic(False, msgs)
|
||||
# Anthropic SDK was called.
|
||||
wire_msgs = captured.get("messages")
|
||||
assert isinstance(wire_msgs, list), (
|
||||
f"Expected messages= list at SDK boundary, got {captured}"
|
||||
)
|
||||
# Walk the wire payload — the thinking block must NOT be present
|
||||
# in the assistant turn's content blocks.
|
||||
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
|
||||
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
|
||||
assert "thinking" not in block_types, (
|
||||
f"Strip predicate did not fire at wire boundary: blocks={block_types}"
|
||||
)
|
||||
# Defense-in-depth: the secret reasoning text must not appear
|
||||
# anywhere in the wire payload.
|
||||
flat = repr(captured)
|
||||
assert "secret reasoning" not in flat, "Reasoning text leaked into the SDK boundary payload"
|
||||
|
||||
def test_replay_true_preserves_thinking_at_wire(self) -> None:
|
||||
msgs: list[dict[str, object]] = [
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "kept reasoning", "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "ack"},
|
||||
]
|
||||
captured = self._drive_session_through_anthropic(True, msgs)
|
||||
wire_msgs = captured.get("messages")
|
||||
assert isinstance(wire_msgs, list)
|
||||
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
|
||||
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
|
||||
assert "thinking" in block_types, (
|
||||
f"Replay-true did not preserve thinking at wire: blocks={block_types}"
|
||||
)
|
||||
|
||||
def test_capability_false_strips_thinking_even_when_operator_flag_true(self) -> None:
|
||||
# Mirror of the OpenAI Responses ``test_capability_false_omits_
|
||||
# include_even_when_flag_true`` test below: operator flips
|
||||
# replay=True but the model's capability advertises
|
||||
# supports_reasoning_replay=False. AND-gate at the resolver
|
||||
# blocks replay, so the strip predicate fires at the wire and
|
||||
# the thinking block does NOT reach the SDK boundary.
|
||||
pytest.importorskip("anthropic")
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
msgs: list[dict[str, object]] = [
|
||||
{"role": "user", "content": "hello"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{"type": "thinking", "thinking": "secret reasoning", "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "ack"},
|
||||
]
|
||||
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True) # operator opted in
|
||||
session._model_alias = "hypothetical-no-replay-claude"
|
||||
caps = ModelCapabilities(supports_reasoning_replay=False)
|
||||
client, captured = self._stub_anthropic_client()
|
||||
real_provider = AnthropicProvider()
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="hypothetical-no-replay-claude",
|
||||
msgs=msgs,
|
||||
provider=real_provider,
|
||||
capabilities=caps,
|
||||
model_alias="hypothetical-no-replay-claude",
|
||||
)
|
||||
list(stream)
|
||||
|
||||
wire_msgs = captured.get("messages")
|
||||
assert isinstance(wire_msgs, list), (
|
||||
f"Expected messages= list at SDK boundary, got {captured}"
|
||||
)
|
||||
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
|
||||
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
|
||||
assert "thinking" not in block_types, (
|
||||
"Capability gate did not block replay: thinking block reached the wire "
|
||||
f"despite supports_reasoning_replay=False (blocks={block_types})"
|
||||
)
|
||||
flat = repr(captured)
|
||||
assert "secret reasoning" not in flat, (
|
||||
"Reasoning text leaked into the SDK boundary payload despite capability gate"
|
||||
)
|
||||
|
||||
|
||||
class TestSessionToOpenAIResponsesBoundaryIntegration:
|
||||
"""End-to-end integration: session._try_stream -> real
|
||||
OpenAIResponsesProvider.create_streaming -> captured Responses
|
||||
SDK boundary call. Mirrors the AnthropicProvider test above
|
||||
but for the path-2 (Responses API) replay flow.
|
||||
|
||||
Pins the include= request kwarg + reasoning input-item emission
|
||||
actually fire at the wire boundary when the operator flag and
|
||||
model capability both allow.
|
||||
"""
|
||||
|
||||
def _stub_responses_client(self) -> tuple[MagicMock, dict[str, object]]:
|
||||
"""Mock OpenAI Responses client. ``client.responses.create``
|
||||
captures kwargs and returns an empty stream iterator."""
|
||||
captured: dict[str, object] = {}
|
||||
|
||||
def create(**kwargs: object) -> object:
|
||||
captured.update(kwargs)
|
||||
return iter([])
|
||||
|
||||
client = MagicMock()
|
||||
client.responses.create = create
|
||||
return client, captured
|
||||
|
||||
def _registry_with_reasoning_capability(
|
||||
self, replay: bool = True, supports_replay: bool = True
|
||||
) -> Any:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
return SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
replay_reasoning_to_model=replay,
|
||||
capabilities={}, # no overrides
|
||||
),
|
||||
_caps=ModelCapabilities(
|
||||
context_window=400000,
|
||||
supports_temperature=False,
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
default_reasoning_effort="medium",
|
||||
supports_reasoning_replay=supports_replay,
|
||||
),
|
||||
)
|
||||
|
||||
def test_replay_true_adds_include_to_responses_request(self) -> None:
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
|
||||
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=True)
|
||||
session = _make_session()
|
||||
session._registry = registry
|
||||
session._model_alias = "gpt-5"
|
||||
client, captured = self._stub_responses_client()
|
||||
real_provider = OpenAIResponsesProvider()
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="gpt-5",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=real_provider,
|
||||
capabilities=registry._caps,
|
||||
model_alias="gpt-5",
|
||||
)
|
||||
list(stream)
|
||||
assert captured.get("include") == ["reasoning.encrypted_content"]
|
||||
|
||||
def test_replay_false_omits_include(self) -> None:
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
|
||||
registry = self._registry_with_reasoning_capability(replay=False, supports_replay=True)
|
||||
session = _make_session()
|
||||
session._registry = registry
|
||||
session._model_alias = "gpt-5"
|
||||
client, captured = self._stub_responses_client()
|
||||
real_provider = OpenAIResponsesProvider()
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="gpt-5",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=real_provider,
|
||||
capabilities=registry._caps,
|
||||
model_alias="gpt-5",
|
||||
)
|
||||
list(stream)
|
||||
assert "include" not in captured
|
||||
|
||||
def test_capability_false_omits_include_even_when_flag_true(self) -> None:
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
|
||||
# Operator flips replay=True but the model has
|
||||
# supports_reasoning_replay=False (e.g. gpt-4o via Responses).
|
||||
# Capability gate prevents the include= from being sent.
|
||||
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=False)
|
||||
session = _make_session()
|
||||
session._registry = registry
|
||||
session._model_alias = "gpt-4o"
|
||||
client, captured = self._stub_responses_client()
|
||||
real_provider = OpenAIResponsesProvider()
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="gpt-4o",
|
||||
msgs=[{"role": "user", "content": "hi"}],
|
||||
provider=real_provider,
|
||||
capabilities=registry._caps,
|
||||
model_alias="gpt-4o",
|
||||
)
|
||||
list(stream)
|
||||
assert "include" not in captured
|
||||
|
||||
def test_replay_true_emits_reasoning_input_item(self) -> None:
|
||||
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
|
||||
|
||||
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=True)
|
||||
session = _make_session()
|
||||
session._registry = registry
|
||||
session._model_alias = "gpt-5"
|
||||
client, captured = self._stub_responses_client()
|
||||
real_provider = OpenAIResponsesProvider()
|
||||
# Multi-turn conversation with stored reasoning on assistant turn.
|
||||
msgs: list[dict[str, object]] = [
|
||||
{"role": "user", "content": "explain"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Final answer.",
|
||||
"_provider_content": [
|
||||
{
|
||||
"type": "reasoning",
|
||||
"id": "r_xyz",
|
||||
"summary": [{"type": "summary_text", "text": "I thought"}],
|
||||
"encrypted_content": "blob",
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "user", "content": "follow-up"},
|
||||
]
|
||||
with (
|
||||
patch.object(session, "_get_active_tools", return_value=None),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
patch.object(session, "_get_deferred_names", return_value=frozenset()),
|
||||
patch.object(session, "_check_cancelled"),
|
||||
):
|
||||
stream = session._try_stream(
|
||||
client=client,
|
||||
model="gpt-5",
|
||||
msgs=msgs,
|
||||
provider=real_provider,
|
||||
capabilities=registry._caps,
|
||||
model_alias="gpt-5",
|
||||
)
|
||||
list(stream)
|
||||
# Walk the wire input items — one of them must be the reasoning
|
||||
# round-trip (id matches what we stored).
|
||||
wire_input = captured.get("input")
|
||||
assert isinstance(wire_input, list)
|
||||
reasoning_items = [it for it in wire_input if it.get("type") == "reasoning"]
|
||||
assert len(reasoning_items) == 1
|
||||
assert reasoning_items[0]["id"] == "r_xyz"
|
||||
assert reasoning_items[0]["encrypted_content"] == "blob"
|
||||
|
||||
|
||||
class TestUtilityCompletionPassesFlag:
|
||||
"""Non-streaming utility path (title gen, compaction, extraction) —
|
||||
same plumbing requirement as streaming."""
|
||||
|
||||
def test_utility_completion_passes_resolved_flag(self) -> None:
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
session = _make_session()
|
||||
session._registry = _registry_with_flag(replay=True)
|
||||
session._model_alias = "claude-opus-4-7"
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def capture_completion(**kwargs: Any) -> Any:
|
||||
captured.update(kwargs)
|
||||
return SimpleNamespace(content="title", finish_reason="stop", usage=None)
|
||||
|
||||
mock_provider = MagicMock()
|
||||
mock_provider.create_completion = capture_completion
|
||||
session._provider = mock_provider
|
||||
caps = ModelCapabilities(max_output_tokens=0, supports_reasoning_replay=True)
|
||||
with (
|
||||
patch.object(session, "_get_capabilities", return_value=caps),
|
||||
patch.object(session, "_provider_extra_params", return_value=None),
|
||||
):
|
||||
session._utility_completion(
|
||||
messages=[{"role": "user", "content": "summarize"}],
|
||||
max_tokens=512,
|
||||
temperature=0.3,
|
||||
)
|
||||
assert captured["replay_reasoning_to_model"] is True
|
||||
@@ -0,0 +1,355 @@
|
||||
"""Tests for ChatSession synthetic ``reasoning_text`` block stamping (Phase 3 path 3).
|
||||
|
||||
Path 3 covers OpenAI Chat Completions endpoints — vLLM with
|
||||
``--reasoning-parser``, llama.cpp with ``reasoning_format``, Gemini's
|
||||
``/v1beta/openai/`` endpoint, and any other server that surfaces
|
||||
``delta.reasoning_content`` Pydantic extras. These have no native
|
||||
provider_blocks shape on the wire, so ``ChatSession._stream_response``
|
||||
captures the streamed reasoning text into ``reasoning_parts`` and
|
||||
``_maybe_synth_reasoning_block`` stamps it onto ``_provider_content``
|
||||
as a synthetic ``{type: "reasoning_text"}`` block at the end of the
|
||||
turn.
|
||||
|
||||
These tests pin:
|
||||
1. The synthesizer fires only when no native blocks were emitted AND
|
||||
reasoning was captured (Anthropic + OpenAI Responses bypass it).
|
||||
2. ``source`` field is tagged with the active model's server_type
|
||||
(informational; pulled from ``server_compat.server_type``).
|
||||
3. ``OpenAIChatCompletionsProvider.extract_reasoning_text`` round-trips
|
||||
the synthetic block on history rehydration.
|
||||
4. The synthetic shape is NOT in ``ANTHROPIC_VALID_BLOCK_TYPES`` so
|
||||
cross-model resumption (local-model → Anthropic) falls through
|
||||
cleanly to the text+tool_calls rebuild path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
from tests._session_helpers import make_session as _make_session
|
||||
from turnstone.core.providers._anthropic import (
|
||||
ANTHROPIC_VALID_BLOCK_TYPES,
|
||||
AnthropicProvider,
|
||||
)
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
|
||||
|
||||
class TestMaybeSynthReasoningBlock:
|
||||
"""Direct unit tests for ``ChatSession._maybe_synth_reasoning_block``."""
|
||||
|
||||
def test_no_synth_when_provider_blocks_present(self) -> None:
|
||||
# Anthropic / OpenAI Responses path — native blocks already
|
||||
# carry the reasoning, no synth needed.
|
||||
session = _make_session()
|
||||
existing = [{"type": "thinking", "thinking": "x"}]
|
||||
out = session._maybe_synth_reasoning_block(existing, ["should not be added"])
|
||||
assert out is existing
|
||||
|
||||
def test_no_synth_when_reasoning_parts_empty(self) -> None:
|
||||
session = _make_session()
|
||||
out = session._maybe_synth_reasoning_block([], [])
|
||||
assert out == []
|
||||
|
||||
def test_no_synth_when_reasoning_parts_only_whitespace(self) -> None:
|
||||
session = _make_session()
|
||||
out = session._maybe_synth_reasoning_block([], [" ", "\n\t"])
|
||||
assert out == []
|
||||
|
||||
def test_synth_creates_reasoning_text_block(self) -> None:
|
||||
session = _make_session()
|
||||
out = session._maybe_synth_reasoning_block([], ["thought ", "process"])
|
||||
assert len(out) == 1
|
||||
assert out[0]["type"] == "reasoning_text"
|
||||
assert out[0]["text"] == "thought process"
|
||||
|
||||
def test_synth_omits_source_when_no_server_type(self) -> None:
|
||||
session = _make_session()
|
||||
# No registry / no server_compat → source field omitted.
|
||||
out = session._maybe_synth_reasoning_block([], ["text"])
|
||||
assert "source" not in out[0]
|
||||
|
||||
def test_synth_includes_source_when_server_type_resolvable(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
capabilities={"server_compat": {"server_type": "vllm"}},
|
||||
)
|
||||
)
|
||||
session._model_alias = "qwen3-32b"
|
||||
out = session._maybe_synth_reasoning_block([], ["text"])
|
||||
assert out[0]["source"] == "vllm"
|
||||
|
||||
def test_synth_handles_registry_exception(self) -> None:
|
||||
# _resolve_server_type silently returns "" on any lookup error
|
||||
# — synth still fires but omits the source field.
|
||||
class BrokenRegistry:
|
||||
def get_config(self, alias: str) -> Any:
|
||||
raise KeyError(alias)
|
||||
|
||||
session = _make_session()
|
||||
session._registry = BrokenRegistry()
|
||||
session._model_alias = "missing"
|
||||
out = session._maybe_synth_reasoning_block([], ["text"])
|
||||
assert out[0]["text"] == "text"
|
||||
assert "source" not in out[0]
|
||||
|
||||
def test_synth_appends_when_provider_blocks_are_non_reasoning(self) -> None:
|
||||
# GoogleProvider attaches raw tool_call dicts as provider_blocks
|
||||
# on the finish chunk (for thought_signature round-trip). When
|
||||
# the same turn streamed reasoning_delta (Gemini's reasoning_
|
||||
# content extra), the synthesizer must APPEND the synthetic
|
||||
# reasoning block rather than skip synthesis — otherwise the
|
||||
# reasoning text is shown live but lost on page reload.
|
||||
session = _make_session()
|
||||
existing = [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "search", "arguments": "{}"},
|
||||
"thought_signature": "sig123",
|
||||
}
|
||||
]
|
||||
out = session._maybe_synth_reasoning_block(existing, ["I should search"])
|
||||
assert len(out) == 2
|
||||
assert out[0] is existing[0] # tool_call fidelity block survives intact
|
||||
assert out[1]["type"] == "reasoning_text"
|
||||
assert out[1]["text"] == "I should search"
|
||||
|
||||
def test_no_synth_when_openai_responses_reasoning_already_present(self) -> None:
|
||||
# OpenAI Responses native reasoning item — synth must NOT fire
|
||||
# even though provider_blocks contains ALSO non-reasoning items
|
||||
# (e.g. message blocks). The reasoning-bearing block satisfies
|
||||
# the persistence contract on its own.
|
||||
session = _make_session()
|
||||
existing = [
|
||||
{"type": "reasoning", "summary": [{"text": "openai reasoning"}]},
|
||||
{"type": "message", "role": "assistant", "content": "answer"},
|
||||
]
|
||||
out = session._maybe_synth_reasoning_block(existing, ["live reasoning text"])
|
||||
assert out is existing
|
||||
|
||||
def test_no_synth_when_non_reasoning_blocks_but_reasoning_parts_empty(self) -> None:
|
||||
# Google tool_calls with no reasoning streamed — return as-is.
|
||||
session = _make_session()
|
||||
existing = [
|
||||
{"id": "call_1", "type": "function", "function": {"name": "f", "arguments": "{}"}}
|
||||
]
|
||||
out = session._maybe_synth_reasoning_block(existing, [])
|
||||
assert out is existing
|
||||
|
||||
|
||||
class TestSyntheticBlockShapeContract:
|
||||
"""The synthetic block shape MUST stay outside Anthropic's valid
|
||||
block types so cross-model resumption falls through cleanly."""
|
||||
|
||||
def test_reasoning_text_not_in_anthropic_valid_types(self) -> None:
|
||||
# If this assertion ever fails, the cross-model resumption
|
||||
# safety story breaks: a synthetic block from a local-model
|
||||
# session would reach Anthropic's wire as a malformed block.
|
||||
assert "reasoning_text" not in ANTHROPIC_VALID_BLOCK_TYPES
|
||||
|
||||
def test_synthetic_block_falls_through_anthropic_shape_filter(self) -> None:
|
||||
# Cross-model resumption regression: turn 1 was on a local
|
||||
# model (synthetic block stamped), then the operator switched
|
||||
# to Anthropic. The shape filter must reject the synthetic
|
||||
# block and fall through to text+tool_calls rebuild.
|
||||
provider = AnthropicProvider()
|
||||
msg = {
|
||||
"role": "assistant",
|
||||
"content": "spoken answer",
|
||||
"_provider_content": [
|
||||
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
|
||||
],
|
||||
}
|
||||
_, converted = provider._convert_messages([msg])
|
||||
assistant = next(m for m in converted if m["role"] == "assistant")
|
||||
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
|
||||
# Foreign block did NOT reach Anthropic's wire. Rebuilt from
|
||||
# text only.
|
||||
assert "reasoning_text" not in block_types
|
||||
assert assistant["content"] == [{"type": "text", "text": "spoken answer"}]
|
||||
|
||||
|
||||
class TestOpenAIChatExtractReasoningText:
|
||||
"""``OpenAIChatCompletionsProvider.extract_reasoning_text`` reads
|
||||
the synthetic block back out for UI rehydration."""
|
||||
|
||||
def test_reads_synthetic_reasoning_text_block(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [{"type": "reasoning_text", "text": "captured thought"}]
|
||||
assert provider.extract_reasoning_text(blocks) == "captured thought"
|
||||
|
||||
def test_concatenates_multiple_blocks(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [
|
||||
{"type": "reasoning_text", "text": "first"},
|
||||
{"type": "reasoning_text", "text": "second"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "first\nsecond"
|
||||
|
||||
def test_skips_other_block_types(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "anth"},
|
||||
{"type": "reasoning", "summary": [{"text": "openai"}]},
|
||||
{"type": "reasoning_text", "text": "chat"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "chat"
|
||||
|
||||
def test_handles_empty_text_field(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [
|
||||
{"type": "reasoning_text", "text": ""},
|
||||
{"type": "reasoning_text", "text": "kept"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "kept"
|
||||
|
||||
def test_handles_missing_text_field(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [
|
||||
{"type": "reasoning_text"}, # no text
|
||||
{"type": "reasoning_text", "text": "kept"},
|
||||
]
|
||||
assert provider.extract_reasoning_text(blocks) == "kept"
|
||||
|
||||
def test_returns_empty_for_no_synth_blocks(self) -> None:
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
blocks = [{"type": "thinking", "thinking": "x"}]
|
||||
assert provider.extract_reasoning_text(blocks) == ""
|
||||
|
||||
|
||||
class TestStreamResponseSynthBlockIntegration:
|
||||
"""Integration test: drives a fake reasoning-emitting stream
|
||||
through ``ChatSession._stream_response`` and asserts the
|
||||
synthesizer wires up correctly. Pins the call site at
|
||||
``session.py`` (where ``_maybe_synth_reasoning_block`` is invoked
|
||||
on the assembled provider_blocks before stamping ``_provider_content``)
|
||||
— without this, a future refactor that drops the synthesizer call
|
||||
would silently break path-3 capture (vLLM/llama.cpp/Gemini-compat
|
||||
reasoning would be visible live but invisible on history reload).
|
||||
"""
|
||||
|
||||
def _make_stream(self, content: str, reasoning: str) -> Any:
|
||||
"""Build an iterator of StreamChunks that mimic a path-3
|
||||
capture (reasoning_delta chunks, content chunks, no
|
||||
provider_blocks emitted).
|
||||
"""
|
||||
from turnstone.core.providers._protocol import StreamChunk, UsageInfo
|
||||
|
||||
chunks = []
|
||||
# Reasoning first (matches live SSE order).
|
||||
if reasoning:
|
||||
chunks.append(StreamChunk(reasoning_delta=reasoning, is_first=True))
|
||||
# Content next.
|
||||
if content:
|
||||
chunks.append(
|
||||
StreamChunk(
|
||||
content_delta=content,
|
||||
is_first=not reasoning,
|
||||
)
|
||||
)
|
||||
# Final chunk with finish_reason + usage.
|
||||
chunks.append(
|
||||
StreamChunk(
|
||||
finish_reason="stop",
|
||||
usage=UsageInfo(prompt_tokens=10, completion_tokens=20, total_tokens=30),
|
||||
)
|
||||
)
|
||||
return iter(chunks)
|
||||
|
||||
def test_stream_response_stamps_synth_block_when_path3_reasoning_captured(
|
||||
self,
|
||||
) -> None:
|
||||
"""Drive a fake stream emitting reasoning_delta chunks (no
|
||||
native provider_blocks) through ``_stream_response``; assert
|
||||
the resulting assistant_msg carries a synthetic reasoning_text
|
||||
block stamped onto ``_provider_content``."""
|
||||
session = _make_session()
|
||||
# No registry → source field omitted from synth block.
|
||||
stream = self._make_stream(content="Final answer.", reasoning="path-3 reasoning")
|
||||
msg = session._stream_response(stream)
|
||||
assert msg["role"] == "assistant"
|
||||
assert msg["content"] == "Final answer."
|
||||
# Synthetic block should be stamped onto _provider_content.
|
||||
provider_content = msg.get("_provider_content")
|
||||
assert isinstance(provider_content, list)
|
||||
assert len(provider_content) == 1
|
||||
assert provider_content[0]["type"] == "reasoning_text"
|
||||
assert provider_content[0]["text"] == "path-3 reasoning"
|
||||
|
||||
def test_stream_response_no_synth_when_no_reasoning_captured(self) -> None:
|
||||
"""Stream emits only content (no reasoning_delta). No synth
|
||||
block stamped — _provider_content key absent on assistant_msg."""
|
||||
session = _make_session()
|
||||
stream = self._make_stream(content="just content", reasoning="")
|
||||
msg = session._stream_response(stream)
|
||||
assert msg["content"] == "just content"
|
||||
# No synth block (and no native blocks either) → key absent.
|
||||
assert "_provider_content" not in msg
|
||||
|
||||
def test_stream_response_synth_block_carries_source_when_server_type_resolvable(
|
||||
self,
|
||||
) -> None:
|
||||
"""When the active model has server_compat.server_type set,
|
||||
the synth block carries it as the ``source`` field."""
|
||||
session = _make_session()
|
||||
session._registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
capabilities={"server_compat": {"server_type": "vllm"}},
|
||||
)
|
||||
)
|
||||
session._model_alias = "qwen3-32b"
|
||||
stream = self._make_stream(content="answer", reasoning="reasoning text")
|
||||
msg = session._stream_response(stream)
|
||||
provider_content = msg.get("_provider_content")
|
||||
assert isinstance(provider_content, list)
|
||||
assert provider_content[0]["source"] == "vllm"
|
||||
|
||||
|
||||
class TestResolveServerType:
|
||||
"""Direct unit tests for the helper that pulls server_type from
|
||||
the active model's capabilities dict."""
|
||||
|
||||
def test_returns_empty_when_no_registry(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = None
|
||||
assert session._resolve_server_type() == ""
|
||||
|
||||
def test_returns_empty_when_no_alias(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(capabilities={})
|
||||
)
|
||||
session._model_alias = ""
|
||||
assert session._resolve_server_type() == ""
|
||||
|
||||
def test_returns_server_type_when_present(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
capabilities={"server_compat": {"server_type": "llama.cpp"}}
|
||||
)
|
||||
)
|
||||
session._model_alias = "local-model"
|
||||
assert session._resolve_server_type() == "llama.cpp"
|
||||
|
||||
def test_returns_empty_when_server_compat_missing(self) -> None:
|
||||
session = _make_session()
|
||||
session._registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
capabilities={"context_window": 32768},
|
||||
)
|
||||
)
|
||||
session._model_alias = "local-model"
|
||||
assert session._resolve_server_type() == ""
|
||||
|
||||
def test_returns_empty_on_exception(self) -> None:
|
||||
class BrokenRegistry:
|
||||
def get_config(self, alias: str) -> Any:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
session = _make_session()
|
||||
session._registry = BrokenRegistry()
|
||||
session._model_alias = "x"
|
||||
assert session._resolve_server_type() == ""
|
||||
@@ -167,8 +167,8 @@ def test_on_intent_verdict_persists_verdict_row() -> None:
|
||||
}
|
||||
with _patch_get_storage(storage):
|
||||
ui.on_intent_verdict(verdict)
|
||||
storage.create_intent_verdict.assert_called_once()
|
||||
kwargs = storage.create_intent_verdict.call_args.kwargs
|
||||
storage.upsert_intent_verdict.assert_called_once()
|
||||
kwargs = storage.upsert_intent_verdict.call_args.kwargs
|
||||
assert kwargs["verdict_id"] == "v1"
|
||||
assert kwargs["ws_id"] == "ws-1"
|
||||
assert kwargs["call_id"] == "c1"
|
||||
@@ -352,6 +352,192 @@ def test_resolve_approval_stamps_all_pending_verdicts() -> None:
|
||||
assert ui._last_verdict_decision == "denied"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# user_decision value space — pending / approved / denied / timeout
|
||||
# / auto-approve reasons (policy / blanket / skill / always / auto_approve_tools).
|
||||
# Guards the "user_decision is never empty for new rows" invariant.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resolve_approval_timeout_kwarg_writes_timeout_value() -> None:
|
||||
"""``resolve_approval(False, ..., timeout=True)`` writes
|
||||
``user_decision="timeout"`` so the audit trail can distinguish a
|
||||
passive timeout expiry from an active user denial — the feedback
|
||||
string used to carry this distinction but the column alone could not."""
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
with _patch_get_storage(storage):
|
||||
ui.on_intent_verdict({"verdict_id": "v1", "call_id": "c1"})
|
||||
with _patch_get_storage(storage):
|
||||
ui.resolve_approval(False, "expired", timeout=True)
|
||||
storage.update_intent_verdict.assert_any_call("v1", user_decision="timeout")
|
||||
assert ui._last_verdict_decision == "timeout"
|
||||
|
||||
|
||||
def test_resolve_approval_timeout_with_approved_raises() -> None:
|
||||
"""``timeout=True`` is mutually exclusive with ``approved=True`` —
|
||||
the combination would land a row whose audit column says
|
||||
``"timeout"`` while the SSE event reports ``approved=True``. Fail
|
||||
loud so the inconsistency can't ship silently."""
|
||||
import pytest
|
||||
|
||||
ui = _make_ui()
|
||||
with pytest.raises(ValueError, match="timeout"):
|
||||
ui.resolve_approval(True, timeout=True)
|
||||
|
||||
|
||||
def test_record_auto_approves_populates_reason_lookup() -> None:
|
||||
"""``_record_auto_approves`` must seed
|
||||
``_auto_approve_reasons[call_id]`` with the per-item reason so a
|
||||
late-arriving LLM judge verdict can recover the auto-approve
|
||||
reason via ``on_intent_verdict``."""
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
items = [
|
||||
{
|
||||
"call_id": "c-policy",
|
||||
"func_name": "bash",
|
||||
"auto_approved": True,
|
||||
"auto_approve_reason": "policy",
|
||||
},
|
||||
{
|
||||
"call_id": "c-blanket",
|
||||
"func_name": "list_workstreams",
|
||||
"auto_approved": True,
|
||||
"auto_approve_reason": "blanket",
|
||||
},
|
||||
]
|
||||
with _patch_get_storage(storage):
|
||||
ui._record_auto_approves(items)
|
||||
assert "c-policy" in ui._auto_approve_reasons
|
||||
assert "c-blanket" in ui._auto_approve_reasons
|
||||
assert ui._auto_approve_reasons["c-policy"][0] == "policy"
|
||||
assert ui._auto_approve_reasons["c-blanket"][0] == "blanket"
|
||||
|
||||
|
||||
def test_on_intent_verdict_consumes_auto_approve_reason() -> None:
|
||||
"""A late LLM verdict for a previously auto-approved call_id picks
|
||||
up the reason from ``_auto_approve_reasons``, stamps it on the
|
||||
verdict before persist, and pops the entry so re-use isn't
|
||||
possible. Closes the misdiagnosis bug where auto-approved tools
|
||||
landed verdict rows with ``user_decision=""``."""
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
ui._auto_approve_reasons["c-x"] = ("auto_approve_tools", 0.0)
|
||||
with _patch_get_storage(storage):
|
||||
ui.on_intent_verdict({"verdict_id": "v-x", "call_id": "c-x"})
|
||||
storage.upsert_intent_verdict.assert_called_once()
|
||||
kwargs = storage.upsert_intent_verdict.call_args.kwargs
|
||||
assert kwargs["user_decision"] == "auto_approve_tools"
|
||||
# Consumed on read so the same call_id can't double-stamp later.
|
||||
assert "c-x" not in ui._auto_approve_reasons
|
||||
# Auto-stamped verdicts must NOT join _pending_verdicts — the
|
||||
# row's final decision is already set; appending would let a
|
||||
# later resolve_approval overwrite the auto-reason with the
|
||||
# manual decision (real audit-trail clobber bug).
|
||||
assert ui._pending_verdicts == []
|
||||
|
||||
|
||||
def test_on_intent_verdict_auto_reason_survives_resolve_cycle() -> None:
|
||||
"""Mixed-batch case: one tool was auto-approved (policy), another
|
||||
needs manual approval. The LLM judge fires for the auto-approved
|
||||
sibling DURING the manual-approval wait. The verdict must land
|
||||
with ``user_decision="policy"`` and stay that way even after
|
||||
``resolve_approval`` fires for the pending sibling — the prior
|
||||
bug was that the auto-stamped row got overwritten with
|
||||
``"approved"``/``"denied"`` by the resolve path."""
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
ui._auto_approve_reasons["c-auto"] = ("policy", 0.0)
|
||||
with _patch_get_storage(storage):
|
||||
# LLM verdict fires for the auto-approved sibling.
|
||||
ui.on_intent_verdict({"verdict_id": "v-auto", "call_id": "c-auto"})
|
||||
# Now the pending sibling gets a verdict + manual resolve.
|
||||
ui.on_intent_verdict({"verdict_id": "v-pending", "call_id": "c-pending"})
|
||||
ui.resolve_approval(True, "looks good")
|
||||
# Only the pending verdict should be UPDATEd to "approved" — the
|
||||
# auto-stamped one stays "policy" via its INSERT.
|
||||
update_calls = {
|
||||
c.args[0]: c.kwargs.get("user_decision")
|
||||
for c in storage.update_intent_verdict.call_args_list
|
||||
}
|
||||
assert update_calls == {"v-pending": "approved"}
|
||||
# The auto verdict's INSERT carried the policy reason.
|
||||
insert_calls = {
|
||||
c.kwargs["verdict_id"]: c.kwargs["user_decision"]
|
||||
for c in storage.upsert_intent_verdict.call_args_list
|
||||
}
|
||||
assert insert_calls["v-auto"] == "policy"
|
||||
assert insert_calls["v-pending"] == "pending"
|
||||
|
||||
|
||||
def test_persist_auto_approved_heuristic_verdicts_stamps_reason() -> None:
|
||||
"""The auto-approve early-return branches in ``approve_tools`` used
|
||||
to drop heuristic verdicts on the floor — auditors couldn't tell
|
||||
whether the judge ran or the call was simply silently auto-approved.
|
||||
``_persist_auto_approved_heuristic_verdicts`` closes that gap and
|
||||
stamps each verdict with the item's reason."""
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
items = [
|
||||
{
|
||||
"call_id": "c-1",
|
||||
"auto_approved": True,
|
||||
"auto_approve_reason": "blanket",
|
||||
"_heuristic_verdict": {
|
||||
"verdict_id": "v-1",
|
||||
"call_id": "c-1",
|
||||
"risk_level": "low",
|
||||
"recommendation": "review",
|
||||
},
|
||||
},
|
||||
# No _heuristic_verdict — skipped (judge didn't run for this item).
|
||||
{"call_id": "c-2", "auto_approved": True, "auto_approve_reason": "blanket"},
|
||||
# Not auto_approved — skipped (this helper only handles auto-approved).
|
||||
{
|
||||
"call_id": "c-3",
|
||||
"_heuristic_verdict": {"verdict_id": "v-3", "call_id": "c-3"},
|
||||
},
|
||||
]
|
||||
with _patch_get_storage(storage):
|
||||
ui._persist_auto_approved_heuristic_verdicts(items)
|
||||
storage.create_intent_verdicts_bulk.assert_called_once()
|
||||
rows = storage.create_intent_verdicts_bulk.call_args.args[0]
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["verdict_id"] == "v-1"
|
||||
assert rows[0]["user_decision"] == "blanket"
|
||||
|
||||
|
||||
def test_auto_approve_reasons_ttl_prune_drops_stale_entries() -> None:
|
||||
"""Lazy TTL eviction at write time: entries older than
|
||||
``_AUTO_APPROVE_REASON_TTL`` are pruned on the next
|
||||
``_record_auto_approves`` call. Without this, a session with the
|
||||
LLM judge disabled would accumulate entries that never get
|
||||
consumed."""
|
||||
import time as time_module
|
||||
|
||||
storage = MagicMock()
|
||||
ui = _make_ui()
|
||||
# Seed two stale entries (well past the TTL).
|
||||
stale_ts = time_module.time() - ui._AUTO_APPROVE_REASON_TTL - 30.0
|
||||
ui._auto_approve_reasons["c-stale-1"] = ("policy", stale_ts)
|
||||
ui._auto_approve_reasons["c-stale-2"] = ("blanket", stale_ts)
|
||||
items = [
|
||||
{
|
||||
"call_id": "c-fresh",
|
||||
"auto_approved": True,
|
||||
"auto_approve_reason": "skill",
|
||||
"func_name": "bash",
|
||||
}
|
||||
]
|
||||
with _patch_get_storage(storage):
|
||||
ui._record_auto_approves(items)
|
||||
# Stale entries pruned; only the fresh one remains.
|
||||
assert "c-stale-1" not in ui._auto_approve_reasons
|
||||
assert "c-stale-2" not in ui._auto_approve_reasons
|
||||
assert "c-fresh" in ui._auto_approve_reasons
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Output guard persistence
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -872,3 +1058,413 @@ def test_concurrent_enqueue_and_listener_registration() -> None:
|
||||
# intent survives optimization-mode assertion stripping.
|
||||
assert not producer.is_alive()
|
||||
assert all(not s.is_alive() for s in subscribers)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-turn inflight buffers — SSE refresh-resume snapshot path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_on_content_token_writes_to_both_buffers() -> None:
|
||||
"""``on_content_token`` writes to the multi-turn buffer (IDLE
|
||||
piggyback) AND the per-turn inflight buffer (SSE snapshot)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("hello")
|
||||
assert ui._ws_turn_content == ["hello"]
|
||||
assert ui._ws_inflight_content == ["hello"]
|
||||
assert ui._ws_inflight_seq == 1
|
||||
|
||||
|
||||
def test_on_reasoning_token_writes_to_inflight_buffer_only() -> None:
|
||||
"""Reasoning has no multi-turn IDLE piggyback — only the inflight
|
||||
buffer + the seq counter."""
|
||||
ui = _make_ui()
|
||||
ui.on_reasoning_token("thinking...")
|
||||
assert ui._ws_inflight_reasoning == ["thinking..."]
|
||||
assert ui._ws_inflight_seq == 1
|
||||
# Multi-turn buffer is content-only and untouched by reasoning.
|
||||
assert ui._ws_turn_content == []
|
||||
|
||||
|
||||
def test_inflight_seq_advances_on_every_emit_even_at_cap() -> None:
|
||||
"""Cap-hit content tokens MUST advance ``_ws_inflight_seq``,
|
||||
even though the buffer rejected the append. If seq stalled at
|
||||
high-water-pre-cap, a subscriber registering AFTER the cap is
|
||||
hit would capture ``snap_seq == stalled_seq`` and every
|
||||
subsequent live token (also tagged with the stalled seq) would
|
||||
be filter-dropped by the events handler — silently losing the
|
||||
rest of the stream. The cap is a buffer-size limit, not a
|
||||
"stop streaming" signal."""
|
||||
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
|
||||
|
||||
ui = _make_ui()
|
||||
chunk = "x" * 1024
|
||||
while ui._ws_inflight_content_size < _MAX_TURN_CONTENT_CHARS:
|
||||
ui.on_content_token(chunk)
|
||||
seq_at_cap = ui._ws_inflight_seq
|
||||
|
||||
# Cap-hit token: seq MUST advance (no buffer append, but the
|
||||
# event still gets a fresh seq for the dedup filter).
|
||||
ui.on_content_token(chunk)
|
||||
assert ui._ws_inflight_seq == seq_at_cap + 1
|
||||
# Buffer remains bounded — the cap-hit token is NOT in inflight.
|
||||
assert ui._ws_inflight_content_size <= _MAX_TURN_CONTENT_CHARS + len(chunk)
|
||||
|
||||
|
||||
def test_subscriber_after_cap_hit_receives_subsequent_tokens() -> None:
|
||||
"""Regression for Copilot's cap+seq finding: a subscriber that
|
||||
connects AFTER the inflight buffer is at cap must still receive
|
||||
live tokens past the cap. Past-cap tokens are absent from
|
||||
``snap.content`` (the snapshot text was truncated at cap) but
|
||||
the live stream past them must NOT be filter-dropped."""
|
||||
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
|
||||
|
||||
ui = _make_ui()
|
||||
chunk = "x" * 1024
|
||||
while ui._ws_inflight_content_size < _MAX_TURN_CONTENT_CHARS:
|
||||
ui.on_content_token(chunk)
|
||||
# Stream a few tokens PAST the cap before subscribing.
|
||||
for _ in range(3):
|
||||
ui.on_content_token(chunk)
|
||||
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
snap_seq = snap["seq"]
|
||||
|
||||
# Live token past cap.
|
||||
ui.on_content_token(chunk)
|
||||
ev = lq.get_nowait()
|
||||
assert ev["type"] == "content"
|
||||
# The critical invariant: seq advances per-emit, so the new
|
||||
# event's _seq is strictly greater than the snap_seq the
|
||||
# subscriber captured. Without this, the events handler's
|
||||
# ``seq <= snap_seq`` filter would drop every token past the
|
||||
# cap (silent token loss for refresh-past-cap).
|
||||
assert ev["_seq"] > snap_seq, (
|
||||
f"Token past cap has _seq={ev['_seq']} which is <= "
|
||||
f"snap_seq={snap_seq} — would be silently dropped after a "
|
||||
f"refresh past the cap."
|
||||
)
|
||||
|
||||
|
||||
def test_subscriber_after_reasoning_cap_hit_receives_subsequent_tokens() -> None:
|
||||
"""Same invariant as content cap: reasoning subscribers past
|
||||
cap must keep receiving live reasoning tokens."""
|
||||
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
|
||||
|
||||
ui = _make_ui()
|
||||
chunk = "x" * 1024
|
||||
while ui._ws_inflight_reasoning_size < _MAX_TURN_CONTENT_CHARS:
|
||||
ui.on_reasoning_token(chunk)
|
||||
for _ in range(3):
|
||||
ui.on_reasoning_token(chunk)
|
||||
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
snap_seq = snap["seq"]
|
||||
|
||||
ui.on_reasoning_token(chunk)
|
||||
ev = lq.get_nowait()
|
||||
assert ev["type"] == "reasoning"
|
||||
assert ev["_seq"] > snap_seq
|
||||
|
||||
|
||||
def test_on_turn_committed_resets_inflight_after_commit() -> None:
|
||||
"""``on_turn_committed`` fires immediately after each
|
||||
``messages.append(assistant_msg)`` in the send loop. Without it,
|
||||
the inflight buffer keeps the just-committed turn's content
|
||||
during the post-commit tool-execution window — and a refresh in
|
||||
that window would show the assistant turn TWICE (history list
|
||||
+ in_progress_snapshot)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("Just-finished turn ")
|
||||
ui.on_reasoning_token("Reasoning for the turn ")
|
||||
# Sanity: buffer is populated pre-commit.
|
||||
assert ui._ws_inflight_content == ["Just-finished turn "]
|
||||
assert ui._ws_inflight_reasoning == ["Reasoning for the turn "]
|
||||
|
||||
ui.on_turn_committed()
|
||||
|
||||
# Inflight content + reasoning reset; seq stays monotonic.
|
||||
assert ui._ws_inflight_content == []
|
||||
assert ui._ws_inflight_reasoning == []
|
||||
# Multi-turn buffer is NOT reset by commit (it drains at idle).
|
||||
assert ui._ws_turn_content == ["Just-finished turn "]
|
||||
|
||||
|
||||
def test_inflight_snapshot_empty_during_post_commit_tool_window() -> None:
|
||||
"""Models the user-reported bug: refresh during a tool-execution
|
||||
window between commit and the next stream. Pre-fix: snapshot has
|
||||
the just-committed turn's text → double-renders against history.
|
||||
Post-fix: snapshot is empty → no double-render. Seq stays
|
||||
monotonic (carries the high-water mark across turn boundaries)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("Calling tool with these args: ")
|
||||
seq_pre_commit = ui._ws_inflight_seq
|
||||
ui.on_turn_committed() # session.py fires this after messages.append
|
||||
# We're now in the tool-execution window. A reconnecting client
|
||||
# would call register_listener_with_in_progress_snapshot.
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == ""
|
||||
assert snap["reasoning"] == ""
|
||||
# Seq did NOT reset — must remain monotonic across turns.
|
||||
assert snap["seq"] == seq_pre_commit
|
||||
|
||||
|
||||
def test_on_turn_start_resets_inflight_content_and_reasoning() -> None:
|
||||
"""``on_turn_start`` clears the per-turn content + reasoning
|
||||
buffers but does NOT touch the multi-turn ``_ws_turn_content``
|
||||
(which the dashboard's IDLE-piggyback payload depends on) and
|
||||
does NOT reset the seq counter (must remain monotonic across
|
||||
turn boundaries — see ``test_inflight_seq_monotonic_across_turn_boundaries``)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("turn-1 ")
|
||||
ui.on_reasoning_token("reasoning-1 ")
|
||||
multi_pre = list(ui._ws_turn_content)
|
||||
multi_pre_size = ui._ws_turn_content_size
|
||||
|
||||
ui.on_turn_start()
|
||||
|
||||
assert ui._ws_inflight_content == []
|
||||
assert ui._ws_inflight_content_size == 0
|
||||
assert ui._ws_inflight_reasoning == []
|
||||
assert ui._ws_inflight_reasoning_size == 0
|
||||
# Multi-turn untouched.
|
||||
assert ui._ws_turn_content == multi_pre
|
||||
assert ui._ws_turn_content_size == multi_pre_size
|
||||
|
||||
|
||||
def test_register_listener_with_in_progress_snapshot_empty() -> None:
|
||||
ui = _make_ui()
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert isinstance(lq, queue.Queue)
|
||||
assert lq in ui._listeners
|
||||
assert snap == {"content": "", "reasoning": "", "seq": 0}
|
||||
|
||||
|
||||
def test_register_listener_with_in_progress_snapshot_populated() -> None:
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("Hello, ")
|
||||
ui.on_content_token("world!")
|
||||
ui.on_reasoning_token("planning a greeting")
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == "Hello, world!"
|
||||
assert snap["reasoning"] == "planning a greeting"
|
||||
# seq counts every successful append across BOTH buffers.
|
||||
assert snap["seq"] == 3
|
||||
# Listener is registered — later live tokens land in lq.
|
||||
ui.on_content_token(" Goodbye.")
|
||||
ev = lq.get_nowait()
|
||||
assert ev["type"] == "content"
|
||||
assert ev["text"] == " Goodbye."
|
||||
assert ev["_seq"] == 4
|
||||
|
||||
|
||||
def test_register_listener_with_in_progress_snapshot_only_inflight_not_multi_turn() -> None:
|
||||
"""The snapshot reflects the in-progress turn only — anything
|
||||
cleared by ``on_turn_start`` (a prior committed turn within the
|
||||
same send) must NOT appear in the snapshot, even though the
|
||||
multi-turn buffer still has it."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("PRIOR_TURN ")
|
||||
ui.on_turn_start() # commit boundary — inflight reset
|
||||
ui.on_content_token("CURRENT")
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == "CURRENT"
|
||||
# Multi-turn buffer still has both turns (drives the IDLE piggyback).
|
||||
assert "".join(ui._ws_turn_content) == "PRIOR_TURN CURRENT"
|
||||
|
||||
|
||||
def test_seq_filter_dedup_round_trip() -> None:
|
||||
"""End-to-end dedup invariant: every token appears exactly once
|
||||
when reconstructing from snapshot + listener queue under live
|
||||
writes that race the registration. Models the events handler."""
|
||||
ui = _make_ui()
|
||||
for ch in "abcde":
|
||||
ui.on_content_token(ch)
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
for ch in "fgh":
|
||||
ui.on_content_token(ch)
|
||||
|
||||
reconstructed = snap["content"]
|
||||
while True:
|
||||
try:
|
||||
ev = lq.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if ev.get("_seq", 0) <= snap["seq"]:
|
||||
continue
|
||||
reconstructed += ev["text"]
|
||||
assert reconstructed == "abcdefgh"
|
||||
|
||||
|
||||
def test_seq_filter_drops_overlap_when_register_lands_after_writer() -> None:
|
||||
"""Race: writer appends + emits while a second register snapshots
|
||||
after the writer. The live event has _seq <= snap.seq → must be
|
||||
dropped to avoid double-render."""
|
||||
ui = _make_ui()
|
||||
# Register a first listener so the writer's enqueue lands somewhere.
|
||||
lq1, _ = ui.register_listener_with_in_progress_snapshot()
|
||||
ui.on_content_token("X")
|
||||
# Second register snapshots AFTER the write — snap has "X" AND
|
||||
# the writer's enqueue is in lq1.
|
||||
_, snap2 = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap2["content"] == "X"
|
||||
# Drain lq1 with the filter against snap2.seq — duplicate dropped.
|
||||
duped: list[str] = []
|
||||
while True:
|
||||
try:
|
||||
ev = lq1.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if ev.get("_seq", 0) <= snap2["seq"]:
|
||||
continue
|
||||
duped.append(ev["text"])
|
||||
assert duped == []
|
||||
|
||||
|
||||
def test_inflight_seq_monotonic_across_turn_boundaries() -> None:
|
||||
"""Regression: a subscriber registered mid-turn-N must still
|
||||
receive turn N+1's tokens. The seq counter is monotonic across
|
||||
turn boundaries — resetting it at on_turn_committed/on_turn_start
|
||||
would silently drop turn N+1's first M tokens (M = the snap_seq
|
||||
captured mid-turn-N) via the events handler's `seq <= snap_seq`
|
||||
filter."""
|
||||
ui = _make_ui()
|
||||
# Turn N: stream tokens, register a listener mid-turn.
|
||||
ui.on_content_token("turn-N tok1 ")
|
||||
ui.on_content_token("turn-N tok2 ")
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
snap_seq = snap["seq"]
|
||||
assert snap_seq == 2
|
||||
# Turn N completes, turn N+1 begins.
|
||||
ui.on_turn_committed()
|
||||
ui.on_turn_start()
|
||||
# Turn N+1's first content token. With the q-1 fix, seq is
|
||||
# monotonic (3), not reset to 1. The events handler's
|
||||
# `seq <= snap_seq` filter must NOT swallow it.
|
||||
ui.on_content_token("turn-N+1 tok1 ")
|
||||
ev = lq.get_nowait()
|
||||
assert ev["type"] == "content"
|
||||
assert ev["text"] == "turn-N+1 tok1 "
|
||||
assert ev["_seq"] > snap_seq, (
|
||||
f"Token from turn N+1 has _seq={ev['_seq']} which is <= "
|
||||
f"snap_seq={snap_seq} — the events handler's dedup filter "
|
||||
f"would silently drop it on a long-lived SSE subscription."
|
||||
)
|
||||
|
||||
|
||||
def test_snapshot_and_consume_drains_inflight_at_idle() -> None:
|
||||
"""Regression for the cancel/error path: ``on_turn_committed`` is
|
||||
NOT called from cancel handlers, but every exit path eventually
|
||||
fires ``_emit_state("idle")`` (cancel) or ``_emit_state("error")``
|
||||
(exception). The IDLE/ERROR branches of
|
||||
``snapshot_and_consume_state_payload`` must drain the inflight
|
||||
buffers so a refresh post-cancel doesn't double-render the
|
||||
cancelled fragment against history's marker'd version."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("partial cancelled text ")
|
||||
ui.on_reasoning_token("partial reasoning ")
|
||||
assert ui._ws_inflight_content_size > 0
|
||||
assert ui._ws_inflight_reasoning_size > 0
|
||||
|
||||
ui.snapshot_and_consume_state_payload("idle")
|
||||
|
||||
assert ui._ws_inflight_content == []
|
||||
assert ui._ws_inflight_content_size == 0
|
||||
assert ui._ws_inflight_reasoning == []
|
||||
assert ui._ws_inflight_reasoning_size == 0
|
||||
|
||||
|
||||
def test_snapshot_and_consume_drains_inflight_at_error() -> None:
|
||||
"""Regression for the exception path: ERROR-branch must drain
|
||||
inflight too (parallel to the IDLE branch)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("partial errored text ")
|
||||
ui.on_reasoning_token("partial errored reasoning ")
|
||||
|
||||
ui.snapshot_and_consume_state_payload("error")
|
||||
|
||||
assert ui._ws_inflight_content == []
|
||||
assert ui._ws_inflight_reasoning == []
|
||||
|
||||
|
||||
def test_snapshot_and_consume_does_not_reset_seq_at_idle_or_error() -> None:
|
||||
"""The IDLE/ERROR drain clears content + reasoning but must NOT
|
||||
reset the seq counter — long-lived subscribers' snap_seq must
|
||||
stay valid across turn boundaries (see the q-1 invariant test)."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("a")
|
||||
ui.on_content_token("b")
|
||||
assert ui._ws_inflight_seq == 2
|
||||
|
||||
ui.snapshot_and_consume_state_payload("idle")
|
||||
assert ui._ws_inflight_seq == 2
|
||||
|
||||
ui.snapshot_and_consume_state_payload("error")
|
||||
assert ui._ws_inflight_seq == 2
|
||||
|
||||
|
||||
def test_listeners_share_dict_reference_warning() -> None:
|
||||
"""Pinning the shape that necessitated the events-handler shallow
|
||||
copy: ``_enqueue`` puts ONE dict reference into every listener
|
||||
queue. If multiple SSE coroutines mutate (e.g. ``del event[\"_seq\"]``)
|
||||
without copying first, they corrupt each other's view. The fix
|
||||
in make_events_handler is ``event = dict(event)`` immediately
|
||||
after ``client_queue.get`` — verify the underlying invariant
|
||||
here so a future refactor of ``_enqueue`` can't silently break
|
||||
the assumption the events handler relies on."""
|
||||
ui = _make_ui()
|
||||
lq1, _ = ui.register_listener_with_in_progress_snapshot()
|
||||
lq2, _ = ui.register_listener_with_in_progress_snapshot()
|
||||
ui.on_content_token("X")
|
||||
ev1 = lq1.get_nowait()
|
||||
ev2 = lq2.get_nowait()
|
||||
# Same reference today — consumers MUST shallow-copy before any
|
||||
# mutation. If a future _enqueue change makes this no longer
|
||||
# true, the events handler's defensive copy becomes redundant
|
||||
# but harmless; if this assertion suddenly fails the underlying
|
||||
# invariant has shifted and the handler comment should be updated.
|
||||
assert ev1 is ev2
|
||||
|
||||
|
||||
def test_concurrent_writer_and_register_with_snapshot_no_loss_no_dup() -> None:
|
||||
"""Stress: many tokens streaming + a register_with_snapshot landing
|
||||
at a random point. End state: snapshot ∪ filtered_live == every
|
||||
token written, exactly once."""
|
||||
ui = _make_ui()
|
||||
n_tokens = 500
|
||||
snap_box: dict[str, Any] = {}
|
||||
lq_box: dict[str, queue.Queue[Any]] = {}
|
||||
|
||||
def _writer() -> None:
|
||||
for i in range(n_tokens):
|
||||
ui.on_content_token(f"{i},")
|
||||
|
||||
def _registrar() -> None:
|
||||
# Tiny sleep so the writer is mid-flight.
|
||||
threading.Event().wait(0.001)
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
snap_box["snap"] = snap
|
||||
lq_box["lq"] = lq
|
||||
|
||||
w = threading.Thread(target=_writer)
|
||||
r = threading.Thread(target=_registrar)
|
||||
w.start()
|
||||
r.start()
|
||||
w.join()
|
||||
r.join()
|
||||
|
||||
snap = snap_box["snap"]
|
||||
lq = lq_box["lq"]
|
||||
reconstructed = snap["content"]
|
||||
while True:
|
||||
try:
|
||||
ev = lq.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if ev.get("_seq", 0) <= snap["seq"]:
|
||||
continue
|
||||
reconstructed += ev["text"]
|
||||
expected = "".join(f"{i}," for i in range(n_tokens))
|
||||
assert reconstructed == expected, (
|
||||
f"reconstruction mismatch: len(rec)={len(reconstructed)}, len(exp)={len(expected)}"
|
||||
)
|
||||
|
||||
@@ -24,6 +24,12 @@ from turnstone.core.storage._registry import get_storage
|
||||
class NullUI:
|
||||
"""UI adapter that discards all output."""
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
|
||||
@@ -34,6 +34,12 @@ from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
class NullUI:
|
||||
"""UI adapter that discards all output."""
|
||||
|
||||
def on_turn_start(self):
|
||||
pass
|
||||
|
||||
def on_turn_committed(self):
|
||||
pass
|
||||
|
||||
def on_thinking_start(self):
|
||||
pass
|
||||
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
"""Tests for the storage layer's cross-process ``notify`` / ``listen`` API.
|
||||
|
||||
Covers SQLite (synthetic-sweep + in-process fan-out) and PostgreSQL
|
||||
(real ``LISTEN``/``NOTIFY``). The PG-only cases are gated on the
|
||||
``--storage-backend=postgresql`` flag so they no-op on default CI runs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _drain_until(stream, predicate, deadline_sec: float = 5.0):
|
||||
"""Poll ``stream`` until ``predicate`` matches one of the drained notifies.
|
||||
|
||||
Returns the matching notify or raises ``TimeoutError``. Tests use
|
||||
this so timing flakes against the bounded-blocking ``poll`` shape
|
||||
don't masquerade as logic bugs.
|
||||
"""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
remaining = max(0.05, deadline - time.monotonic())
|
||||
for n in stream.poll(min(0.5, remaining)):
|
||||
if predicate(n):
|
||||
return n
|
||||
msg = "no matching notify drained before deadline"
|
||||
raise TimeoutError(msg)
|
||||
|
||||
|
||||
class TestSqliteNotify:
|
||||
"""SQLite path: in-process fan-out + synthetic sweep."""
|
||||
|
||||
def test_notify_no_listeners_is_noop(self, storage):
|
||||
# No exception, no side effect — safe to always call from dispatch.
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
|
||||
def test_notify_delivers_to_in_process_listener(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
got = _drain_until(stream, lambda n: n.payload == '{"op": "INSERT"}')
|
||||
assert got.channel == "services"
|
||||
# ``pid`` is 0 on the SQLite synthetic path and the sending
|
||||
# backend's PID on Postgres — both are valid notify shapes,
|
||||
# so don't assert on the value here.
|
||||
|
||||
def test_notify_filters_by_channel(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("other_channel", "ignored")
|
||||
storage.notify("services", "wanted")
|
||||
got = _drain_until(stream, lambda n: True)
|
||||
assert got.payload == "wanted"
|
||||
|
||||
def test_multiple_listeners_each_get_event(self, storage):
|
||||
# Two streams open on the same channel; each gets its own copy.
|
||||
with storage.listen(["services"]) as s1, storage.listen(["services"]) as s2:
|
||||
storage.notify("services", "broadcast")
|
||||
got1 = _drain_until(s1, lambda n: True)
|
||||
got2 = _drain_until(s2, lambda n: True)
|
||||
assert got1.payload == "broadcast"
|
||||
assert got2.payload == "broadcast"
|
||||
|
||||
def test_close_stops_stream(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
pass
|
||||
# After context exit, the stream is closed; poll returns [] without
|
||||
# blocking. A second close() is idempotent.
|
||||
assert stream.poll(0.05) == []
|
||||
stream.close()
|
||||
|
||||
def test_synthetic_sweep_emits_after_interval(self, storage, _is_sqlite):
|
||||
# Synthetic sweep is fundamentally SQLite-specific — the PG path
|
||||
# uses real ``LISTEN``/``NOTIFY`` and has no sweep tick. Gate
|
||||
# so the test doesn't false-fail by waiting for a "sweep" notify
|
||||
# that the PG stream will never produce.
|
||||
with storage.listen(["services"], sweep_interval=0.1) as stream:
|
||||
# First poll: not yet at the interval, so likely empty.
|
||||
stream.poll(0.05)
|
||||
# Wait past the interval, then poll again — should emit a
|
||||
# synthetic-sweep notify per declared channel.
|
||||
time.sleep(0.15)
|
||||
got = _drain_until(stream, lambda n: n.payload == "sweep")
|
||||
assert got.channel == "services"
|
||||
assert got.payload == "sweep"
|
||||
|
||||
def test_empty_channel_list_yields_empty_stream(self, storage):
|
||||
with storage.listen([]) as stream:
|
||||
# No channels — poll returns [] regardless of how long we wait.
|
||||
assert stream.poll(0.05) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PostgreSQL path — gated on --storage-backend=postgresql.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_postgres(storage):
|
||||
"""Skip the wrapped test when the active backend isn't Postgres."""
|
||||
if storage.__class__.__name__ != "PostgreSQLBackend":
|
||||
pytest.skip("PostgreSQL-specific test")
|
||||
return True
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_sqlite(storage):
|
||||
"""Skip the wrapped test when the active backend isn't SQLite."""
|
||||
if storage.__class__.__name__ != "SQLiteBackend":
|
||||
pytest.skip("SQLite-specific test")
|
||||
return True
|
||||
|
||||
|
||||
class TestPostgresNotify:
|
||||
def test_round_trip(self, storage, _is_postgres):
|
||||
# Open a listener, fire a notify on a regular pooled connection,
|
||||
# drain the listener within a reasonable bound (PG NOTIFY is
|
||||
# typically sub-100ms on a local socket).
|
||||
with storage.listen(["pytest_round_trip"]) as stream:
|
||||
# Tiny sleep so the LISTEN settles before the NOTIFY fires —
|
||||
# otherwise the notify can arrive on the connection before
|
||||
# the LISTEN is registered (race only visible in tests).
|
||||
time.sleep(0.05)
|
||||
storage.notify("pytest_round_trip", '{"hello": "world"}')
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "pytest_round_trip"
|
||||
assert got.payload == '{"hello": "world"}'
|
||||
assert got.pid > 0
|
||||
|
||||
def test_concurrent_notifies_all_arrive(self, storage, _is_postgres):
|
||||
with storage.listen(["pytest_concurrent"]) as stream:
|
||||
time.sleep(0.05)
|
||||
for i in range(5):
|
||||
storage.notify("pytest_concurrent", str(i))
|
||||
seen: set[str] = set()
|
||||
deadline = time.monotonic() + 3.0
|
||||
while len(seen) < 5 and time.monotonic() < deadline:
|
||||
for n in stream.poll(0.2):
|
||||
seen.add(n.payload)
|
||||
assert seen == {"0", "1", "2", "3", "4"}
|
||||
|
||||
def test_close_aborts_blocked_poll(self, storage, _is_postgres):
|
||||
# poll() should return promptly once close() runs on another thread.
|
||||
with storage.listen(["pytest_close"]) as stream:
|
||||
done = threading.Event()
|
||||
result: list[list] = []
|
||||
|
||||
def _poll_long():
|
||||
result.append(stream.poll(5.0))
|
||||
done.set()
|
||||
|
||||
t = threading.Thread(target=_poll_long, daemon=True)
|
||||
t.start()
|
||||
time.sleep(0.1)
|
||||
stream.close()
|
||||
assert done.wait(2.0), "close() did not unblock poll()"
|
||||
# No notify arrived, so the polled batch is empty — but the
|
||||
# poll loop must have exited well under the 5 s timeout.
|
||||
assert result == [[]]
|
||||
|
||||
|
||||
class TestServicesTriggerFilter:
|
||||
"""Migration 053's trigger: fires on real changes, quiet on heartbeats.
|
||||
|
||||
PG-only — the SQLite path has no trigger and is covered by
|
||||
:class:`TestSqliteNotify`. Verifies the in-trigger ``IS NOT DISTINCT
|
||||
FROM`` filter — a heartbeat-only UPDATE (same url + same metadata,
|
||||
only ``last_heartbeat`` changed) must NOT emit a NOTIFY, since
|
||||
``register_service`` runs the same UPSERT on every 30 s tick × N
|
||||
nodes and the channel would otherwise flood.
|
||||
"""
|
||||
|
||||
def test_insert_fires_notify(self, storage, _is_postgres):
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node", "http://127.0.0.1:1")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "services"
|
||||
assert '"op": "INSERT"' in got.payload or "INSERT" in got.payload
|
||||
# Cleanup so concurrent suites don't pick up the row.
|
||||
storage.deregister_service("server", "pytest-trigger-node")
|
||||
|
||||
def test_delete_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-del", "http://127.0.0.1:2")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.deregister_service("server", "pytest-trigger-node-del")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "DELETE" in got.payload
|
||||
|
||||
def test_url_change_update_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:3")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
# UPSERT with different url — UPDATE path with url diff,
|
||||
# trigger must fire.
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:9")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "UPDATE" in got.payload
|
||||
storage.deregister_service("server", "pytest-trigger-node-url")
|
||||
|
||||
def test_heartbeat_only_update_is_quiet(self, storage, _is_postgres):
|
||||
# Open the LISTEN session FIRST so PG delivers the INSERT NOTIFY
|
||||
# to this connection — pg_notify routes only to sessions that
|
||||
# have LISTENed at COMMIT time, so an INSERT committed before the
|
||||
# listen opens would be lost and the drain would time out instead
|
||||
# of exercising the heartbeat-quiet check below.
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node-hb", "http://127.0.0.1:4")
|
||||
# Drain the INSERT notify so subsequent polls see only what
|
||||
# heartbeats emit (if anything).
|
||||
_drain_until(stream, lambda n: True, deadline_sec=2.0)
|
||||
# Now fire a heartbeat tick — same url + same metadata,
|
||||
# only last_heartbeat updates. Trigger must NOT emit.
|
||||
storage.heartbeat_service("server", "pytest-trigger-node-hb")
|
||||
# Poll long enough that any spurious notify would have
|
||||
# arrived; the channel must stay silent.
|
||||
spurious = stream.poll(0.5)
|
||||
assert spurious == [], f"heartbeat-only update emitted unexpected notify: {spurious}"
|
||||
storage.deregister_service("server", "pytest-trigger-node-hb")
|
||||
@@ -232,59 +232,52 @@ class TestSoftCap:
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# valid_until predicate
|
||||
# Predicate independence
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestValidUntil:
|
||||
"""The ``valid_until`` predicate captured at dispatch time re-checks
|
||||
the watch's ``active`` flag at drain time, so a cancelled watch's
|
||||
last splat doesn't ride out a future wake.
|
||||
class TestPredicateIndependence:
|
||||
"""The watch closure does NOT wire a ``valid_until`` predicate.
|
||||
|
||||
Earlier the closure wired ``_still_active`` (re-reading
|
||||
``is_watch_active`` at drain time). That predicate raced
|
||||
``WatchRunner._poll_watch``'s commit of ``active=False`` and silently
|
||||
dropped every terminal fire. The closure now enqueues without a
|
||||
predicate; entries survive drain regardless of the row's ``active``
|
||||
column state.
|
||||
"""
|
||||
|
||||
def test_valid_until_drops_when_watch_inactive(self, tmp_db, monkeypatch):
|
||||
def test_drain_delivers_even_when_storage_reports_inactive(self, tmp_db, monkeypatch):
|
||||
session = _make_session_for_dispatch()
|
||||
_runner, dispatch = _register_runner(session)
|
||||
|
||||
# Storage stub returns False at drain time.
|
||||
is_active_calls = patch_session_storage(monkeypatch, active=False)
|
||||
|
||||
dispatch(_reminder("body"), "watch-1")
|
||||
# Drain fires the predicate; entry should NOT be delivered.
|
||||
out = session._nudge_queue.drain({"any"})
|
||||
assert out == []
|
||||
# Predicate ran once with the dispatched watch_id.
|
||||
assert is_active_calls == ["watch-1"]
|
||||
|
||||
def test_valid_until_drops_when_storage_raises(self, tmp_db, monkeypatch):
|
||||
"""The closure's broad-except in the predicate translates a
|
||||
storage-layer exception to ``False`` so the drain doesn't
|
||||
propagate; the predicate captured ``watch_id`` correctly
|
||||
(otherwise storage wouldn't even be touched).
|
||||
"""
|
||||
session = _make_session_for_dispatch()
|
||||
_runner, dispatch = _register_runner(session)
|
||||
|
||||
patch_session_storage(monkeypatch, raise_on_is_active=True)
|
||||
|
||||
dispatch(_reminder("body"), "watch-bound-id")
|
||||
out = session._nudge_queue.drain({"any"})
|
||||
assert out == []
|
||||
|
||||
def test_valid_until_delivers_when_watch_active(self, tmp_db, monkeypatch):
|
||||
"""Happy-path counter-test for the predicate above: the entry
|
||||
DOES drain when the watch is still active.
|
||||
"""
|
||||
session = _make_session_for_dispatch()
|
||||
_runner, dispatch = _register_runner(session)
|
||||
|
||||
patch_session_storage(monkeypatch, active=True)
|
||||
# Even if storage reports active=False, the entry should still
|
||||
# drain — no predicate to drop it.
|
||||
patch_session_storage(monkeypatch, active=False)
|
||||
|
||||
dispatch(_reminder("body"), "watch-1")
|
||||
out = session._nudge_queue.drain({"any"})
|
||||
assert len(out) == 1
|
||||
assert out[0][0] == "watch_triggered"
|
||||
|
||||
def test_dispatch_never_calls_is_watch_active(self, tmp_db, monkeypatch):
|
||||
"""Pin the invariant directly: the closure must NOT consult
|
||||
``storage.is_watch_active`` anywhere along the enqueue + drain
|
||||
path. Without this assertion, a future change that re-wires
|
||||
an ``is_watch_active`` predicate would silently bring back the
|
||||
bug that motivates this whole module.
|
||||
"""
|
||||
session = _make_session_for_dispatch()
|
||||
_runner, dispatch = _register_runner(session)
|
||||
|
||||
is_active_calls = patch_session_storage(monkeypatch, active=True)
|
||||
|
||||
dispatch(_reminder("body"), "watch-bound-id")
|
||||
session._nudge_queue.drain({"any"})
|
||||
assert is_active_calls == [], (
|
||||
f"watch closure must not call is_watch_active; got {is_active_calls!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrency
|
||||
|
||||
@@ -24,11 +24,15 @@ the structural integration gate for the watch switchover.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._helpers import patch_session_storage
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.storage import get_storage
|
||||
from turnstone.core.watch import WatchRunner
|
||||
|
||||
|
||||
@@ -272,3 +276,283 @@ def test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session(tmp_db, m
|
||||
# Original session's queue stays empty — the dispatch did NOT
|
||||
# accidentally route back to it.
|
||||
assert len(original._nudge_queue) == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("stop_on", "max_polls", "label"),
|
||||
[
|
||||
('"HIT" in output', 100, "stop_on_fired"),
|
||||
(None, 1, "max_polls_reached"),
|
||||
],
|
||||
)
|
||||
def test_poll_watch_terminal_fire_survives_drain(
|
||||
tmp_db: str,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
stop_on: str | None,
|
||||
max_polls: int,
|
||||
label: str,
|
||||
) -> None:
|
||||
"""Regression for the dispatch-ordering bug.
|
||||
|
||||
With the broken ordering (``update_watch(active=False)`` before
|
||||
``_dispatch_result``) plus the ``_still_active`` ``valid_until``
|
||||
predicate that re-reads ``is_watch_active`` at drain time, every
|
||||
terminal watch fire was silently dropped — the closure enqueued
|
||||
the entry but the predicate immediately invalidated it because
|
||||
the row's ``active`` flag had already been flipped to ``0`` in
|
||||
the same poll. The model never saw the fire.
|
||||
|
||||
This test drives a REAL ``WatchRunner._poll_watch`` against a real
|
||||
``tmp_db`` watch row (no ``patch_session_storage(active=True)``
|
||||
stub — that stub is exactly what masked the bug in earlier tests).
|
||||
Covers both terminal paths: ``stop_on`` condition matched and
|
||||
``poll_count >= max_polls`` reached.
|
||||
"""
|
||||
session = _make_session()
|
||||
storage = get_storage()
|
||||
|
||||
runner = WatchRunner(storage=storage, node_id="test-node")
|
||||
session.set_watch_runner(runner)
|
||||
|
||||
storage.create_watch(
|
||||
watch_id=f"w-regression-{label}",
|
||||
ws_id=session._ws_id,
|
||||
node_id="test-node",
|
||||
name=f"regression-{label}",
|
||||
command="echo HIT",
|
||||
interval_secs=10.0,
|
||||
stop_on=stop_on,
|
||||
max_polls=max_polls,
|
||||
created_by="model",
|
||||
next_poll="1970-01-01T00:00:00",
|
||||
)
|
||||
|
||||
# Spy ``enqueue`` so the assertion can distinguish "dispatch never
|
||||
# called" (a different bug class) from "dispatch enqueued but the
|
||||
# predicate dropped it at drain" (this bug).
|
||||
enqueue_calls: list[tuple[str, str, str]] = []
|
||||
real_enqueue = session._nudge_queue.enqueue
|
||||
|
||||
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
|
||||
enqueue_calls.append((args[0], args[1][:40], args[2]))
|
||||
return real_enqueue(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
|
||||
|
||||
# For the max_polls=1 case the first poll has prev_output=None and
|
||||
# would not normally fire on output change; the max_polls branch
|
||||
# at watch.py:412-414 still marks is_final=True so dispatch runs.
|
||||
due = storage.list_due_watches("2099-01-01T00:00:00")
|
||||
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
|
||||
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
|
||||
runner._poll_watch(matching[0])
|
||||
|
||||
assert len(enqueue_calls) == 1, (
|
||||
f"_poll_watch did not enqueue exactly one fire (got {enqueue_calls!r}); "
|
||||
"this is a different bug from the predicate-drop regression"
|
||||
)
|
||||
assert enqueue_calls[0][0] == "watch_triggered"
|
||||
|
||||
assert storage.is_watch_active(f"w-regression-{label}") is False, (
|
||||
"terminal fire should have committed active=False on the row"
|
||||
)
|
||||
|
||||
# The key assertion: drain delivers the entry. Pre-fix this
|
||||
# returned ``[]`` because the ``_still_active`` predicate re-read
|
||||
# ``active=0``. Post-fix the watch closure no longer wires a
|
||||
# predicate and the entry survives.
|
||||
out = session._nudge_queue.drain({"any"})
|
||||
assert len(out) == 1, (
|
||||
"Watch fire was enqueued but never reached drain — dispatch-ordering "
|
||||
"regression. Check that WatchRunner._poll_watch dispatches BEFORE "
|
||||
"committing active=False, and that the watch closure in "
|
||||
"ChatSession.set_watch_runner does not wire an is_watch_active "
|
||||
"predicate."
|
||||
)
|
||||
nt, text, _meta = out[0]
|
||||
assert nt == "watch_triggered"
|
||||
assert "HIT" in text
|
||||
|
||||
|
||||
def test_cancel_reports_already_completed_for_auto_cancelled_watch(tmp_db: str) -> None:
|
||||
"""After a watch fires and auto-cancels, the cancel-by-name path
|
||||
should report 'already completed' rather than 'not found'.
|
||||
|
||||
Pre-fix, ``_exec_watch`` cancel looked the watch up via
|
||||
``list_watches_for_ws`` which filters ``active==1``, so a recently-
|
||||
auto-cancelled row was invisible and the model got the same
|
||||
'not found' message it would for a typo'd name. Post-fix the
|
||||
cancel path uses ``find_watch_by_name`` (no active filter) and
|
||||
branches on ``row["active"]``.
|
||||
"""
|
||||
session = _make_session()
|
||||
storage = get_storage()
|
||||
|
||||
storage.create_watch(
|
||||
watch_id="w-completed-1",
|
||||
ws_id=session._ws_id,
|
||||
node_id="test-node",
|
||||
name="completed-watch",
|
||||
command="echo x",
|
||||
interval_secs=10.0,
|
||||
stop_on=None,
|
||||
max_polls=100,
|
||||
created_by="model",
|
||||
next_poll="",
|
||||
)
|
||||
# Simulate the post-fire state.
|
||||
storage.update_watch("w-completed-1", active=False, next_poll="")
|
||||
|
||||
_call_id, msg = session._exec_watch(
|
||||
{"call_id": "c1", "action": "cancel", "watch_name": "completed-watch"}
|
||||
)
|
||||
|
||||
assert "not found" not in msg.lower()
|
||||
assert "completed" in msg.lower()
|
||||
|
||||
|
||||
def test_cancel_reports_not_found_for_unknown_watch(tmp_db: str) -> None:
|
||||
"""The 'not found' message still applies when the watch genuinely
|
||||
does not exist — make sure the new ``find_watch_by_name`` path
|
||||
didn't accidentally turn every cancel into 'already completed'.
|
||||
"""
|
||||
session = _make_session()
|
||||
|
||||
_call_id, msg = session._exec_watch(
|
||||
{"call_id": "c1", "action": "cancel", "watch_name": "ghost-watch"}
|
||||
)
|
||||
|
||||
assert "not found" in msg.lower()
|
||||
|
||||
|
||||
def test_poll_watch_retry_deactivate_after_update_watch_failure(
|
||||
tmp_db: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""``_terminal_dispatched`` lifecycle: if ``update_watch`` raises
|
||||
AFTER ``_dispatch_result`` shipped the reminder for a terminal
|
||||
fire, the next ``_poll_watch`` tick MUST retry the row write
|
||||
(so the row stops appearing in ``list_due_watches``) and MUST NOT
|
||||
re-dispatch the reminder the model already saw.
|
||||
|
||||
This is the keystone path that prevents duplicate-fire under
|
||||
transient storage failure. Pre-this-test, the entire branch was
|
||||
unexercised.
|
||||
"""
|
||||
session = _make_session()
|
||||
storage = get_storage()
|
||||
runner = WatchRunner(storage=storage, node_id="test-node")
|
||||
session.set_watch_runner(runner)
|
||||
|
||||
watch_id = "w-retry-1"
|
||||
storage.create_watch(
|
||||
watch_id=watch_id,
|
||||
ws_id=session._ws_id,
|
||||
node_id="test-node",
|
||||
name="retry-watch",
|
||||
command="echo HIT",
|
||||
interval_secs=10.0,
|
||||
stop_on='"HIT" in output',
|
||||
max_polls=100,
|
||||
created_by="model",
|
||||
next_poll="1970-01-01T00:00:00",
|
||||
)
|
||||
|
||||
enqueue_calls: list[tuple[str, str]] = []
|
||||
real_enqueue = session._nudge_queue.enqueue
|
||||
|
||||
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
|
||||
enqueue_calls.append((args[0], args[1][:32]))
|
||||
return real_enqueue(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
|
||||
|
||||
# Stage 1 — first poll. ``update_watch`` raises AFTER dispatch.
|
||||
real_update = storage.update_watch
|
||||
update_raise = {"armed": True}
|
||||
|
||||
def _failing_update(wid: str, **fields: Any) -> bool:
|
||||
if update_raise["armed"]:
|
||||
raise RuntimeError("simulated transient storage failure")
|
||||
return real_update(wid, **fields)
|
||||
|
||||
monkeypatch.setattr(storage, "update_watch", _failing_update)
|
||||
|
||||
due = storage.list_due_watches("2099-01-01T00:00:00")
|
||||
matching = [r for r in due if r["watch_id"] == watch_id]
|
||||
assert len(matching) == 1
|
||||
# ``_poll_watch`` doesn't catch the storage error; the outer
|
||||
# ``_tick`` would log it. Suppress here so the test owns the
|
||||
# boundary and continues to its assertions.
|
||||
with contextlib.suppress(RuntimeError):
|
||||
runner._poll_watch(matching[0])
|
||||
|
||||
# Dispatch ran exactly once and the watch_id sits in the
|
||||
# terminal-dispatched set awaiting retry.
|
||||
assert len(enqueue_calls) == 1
|
||||
assert enqueue_calls[0][0] == "watch_triggered"
|
||||
assert watch_id in runner._terminal_dispatched
|
||||
|
||||
# The row is still active=1 because update_watch raised. It
|
||||
# would re-appear in list_due_watches on the next tick.
|
||||
assert storage.is_watch_active(watch_id) is True
|
||||
|
||||
# Stage 2 — second poll. Storage now succeeds; retry-deactivate
|
||||
# branch must commit active=False WITHOUT re-dispatching.
|
||||
update_raise["armed"] = False
|
||||
|
||||
due = storage.list_due_watches("2099-01-01T00:00:00")
|
||||
matching = [r for r in due if r["watch_id"] == watch_id]
|
||||
assert len(matching) == 1
|
||||
runner._poll_watch(matching[0])
|
||||
|
||||
# Exactly one dispatch in total — the retry path took the
|
||||
# short-circuit return at the top of _poll_watch.
|
||||
assert len(enqueue_calls) == 1, f"retry-deactivate must not re-dispatch; got {enqueue_calls!r}"
|
||||
# Row is now inactive (the retry path's update_watch landed).
|
||||
assert storage.is_watch_active(watch_id) is False
|
||||
# Set is cleared so future watches with the same id (unlikely) /
|
||||
# process memory doesn't accumulate.
|
||||
assert watch_id not in runner._terminal_dispatched
|
||||
|
||||
|
||||
def test_cancel_clears_pending_terminal_dispatched_entry(
|
||||
tmp_db: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""If ``update_watch`` raised after dispatch, leaving a pending
|
||||
entry in ``_terminal_dispatched``, and the user then cancels the
|
||||
watch out-of-band, the retry-deactivate branch never gets to run
|
||||
(the cancel sets ``next_poll=""`` which removes the row from
|
||||
``list_due_watches``). The cancel path itself must discard the
|
||||
pending entry; otherwise the runner leaks ``watch_id``s for the
|
||||
process lifetime.
|
||||
"""
|
||||
session = _make_session()
|
||||
storage = get_storage()
|
||||
runner = WatchRunner(storage=storage, node_id="test-node")
|
||||
session.set_watch_runner(runner)
|
||||
|
||||
watch_id = "w-leak-1"
|
||||
storage.create_watch(
|
||||
watch_id=watch_id,
|
||||
ws_id=session._ws_id,
|
||||
node_id="test-node",
|
||||
name="leak-watch",
|
||||
command="echo x",
|
||||
interval_secs=10.0,
|
||||
stop_on=None,
|
||||
max_polls=100,
|
||||
created_by="model",
|
||||
next_poll="",
|
||||
)
|
||||
# Simulate: dispatch shipped, update_watch raised, watch_id sits
|
||||
# in the runner's pending set.
|
||||
with runner._terminal_dispatched_lock:
|
||||
runner._terminal_dispatched.add(watch_id)
|
||||
|
||||
# User cancels. Because the cancel writes active=False, next_poll="",
|
||||
# the row leaves list_due_watches and the runner's retry-deactivate
|
||||
# branch never executes for it. The cancel must discard the entry.
|
||||
storage.update_watch(watch_id, active=False, next_poll="")
|
||||
session._exec_watch({"call_id": "c1", "action": "cancel", "watch_name": "leak-watch"})
|
||||
|
||||
assert watch_id not in runner._terminal_dispatched
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from turnstone.core.storage._schema import watches as watches_table
|
||||
|
||||
|
||||
def _make_watch_kwargs(**overrides):
|
||||
"""Build default kwargs for create_watch."""
|
||||
@@ -101,6 +105,91 @@ class TestWatchListQueries:
|
||||
db.update_watch("w1", active=False)
|
||||
assert db.list_watches_for_ws("ws-1") == []
|
||||
|
||||
def test_find_by_name_returns_inactive(self, db):
|
||||
"""``find_watch_by_name`` ignores the active filter — that is
|
||||
what lets the cancel-by-name UX distinguish 'already completed'
|
||||
from 'no such watch.'
|
||||
"""
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="completed"))
|
||||
db.update_watch("w1", active=False)
|
||||
|
||||
row = db.find_watch_by_name("ws-1", "completed")
|
||||
assert row is not None
|
||||
assert row["watch_id"] == "w1"
|
||||
assert not row["active"]
|
||||
|
||||
def test_find_by_name_matches_watch_id_prefix(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="abcdef123", ws_id="ws-1", name="x"))
|
||||
row = db.find_watch_by_name("ws-1", "abc")
|
||||
assert row is not None
|
||||
assert row["watch_id"] == "abcdef123"
|
||||
|
||||
def test_find_by_name_scoped_to_ws(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="shared"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", ws_id="ws-2", name="shared"))
|
||||
|
||||
row = db.find_watch_by_name("ws-1", "shared")
|
||||
assert row is not None
|
||||
assert row["watch_id"] == "w1"
|
||||
|
||||
def test_find_by_name_returns_none_when_missing(self, db):
|
||||
assert db.find_watch_by_name("ws-1", "ghost") is None
|
||||
|
||||
def test_find_by_name_empty_input_returns_none(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="x"))
|
||||
assert db.find_watch_by_name("ws-1", "") is None
|
||||
|
||||
def test_find_by_name_treats_percent_as_literal(self, db):
|
||||
"""A model-supplied '%' must NOT match arbitrary watch_ids.
|
||||
|
||||
Pre-escape, ``watch_id.like(f"{name_or_prefix}%")`` would
|
||||
interpret '%' as 'match anything' and pick up the first row in
|
||||
the workstream regardless of name.
|
||||
"""
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="real-watch"))
|
||||
assert db.find_watch_by_name("ws-1", "%") is None
|
||||
|
||||
def test_find_by_name_treats_underscore_as_literal(self, db):
|
||||
"""Same as the '%' case for the single-char LIKE wildcard."""
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="abcd", ws_id="ws-1", name="real-watch"))
|
||||
# '_' would otherwise match any single char, picking up
|
||||
# watch_ids beginning with 'a', 'b', etc.
|
||||
assert db.find_watch_by_name("ws-1", "_") is None
|
||||
|
||||
def test_find_by_name_prefers_active_over_newer_inactive(self, db):
|
||||
"""If a same-name pair exists where the inactive row is NEWER
|
||||
than the active row, find_watch_by_name must still return the
|
||||
active row. Pre-fix the query was ``ORDER BY created DESC
|
||||
LIMIT 1`` — which would return the newer inactive row and
|
||||
cause the cancel UX to report 'already completed' for a name
|
||||
whose live row is still polling.
|
||||
|
||||
Reachable in practice because storage allows out-of-band
|
||||
writes (e.g. ``delete_watches_for_ws`` cleanup followed by
|
||||
re-create, an admin manually flipping ``active``, or test
|
||||
scaffolding) that bypass the create-time duplicate-name
|
||||
guard.
|
||||
"""
|
||||
# Older active watch.
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w-active", ws_id="ws-1", name="recurring"))
|
||||
# Newer inactive watch with the same name. ``create_watch``
|
||||
# stamps ``created`` to ``now`` at second resolution, so we
|
||||
# bypass the API to give the inactive row a deterministically
|
||||
# later timestamp.
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w-inactive", ws_id="ws-1", name="recurring"))
|
||||
with db._conn() as conn:
|
||||
conn.execute(
|
||||
sa.update(watches_table)
|
||||
.where(watches_table.c.watch_id == "w-inactive")
|
||||
.values(active=0, next_poll="", created="2099-01-01T00:00:00")
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
row = db.find_watch_by_name("ws-1", "recurring")
|
||||
assert row is not None
|
||||
assert row["watch_id"] == "w-active"
|
||||
assert row["active"]
|
||||
|
||||
def test_list_for_node(self, db):
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w1", node_id="n1"))
|
||||
db.create_watch(**_make_watch_kwargs(watch_id="w2", node_id="n1"))
|
||||
|
||||
@@ -29,7 +29,7 @@ class TestVersionHtml:
|
||||
def test_vendored_katex_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">'
|
||||
html = '<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
@@ -43,14 +43,14 @@ class TestVersionHtml:
|
||||
def test_vendored_mermaid_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<script src="/shared/mermaid-11.14.0/mermaid.min.js"></script>'
|
||||
html = '<script src="/shared/mermaid-11.15.0/mermaid.min.js"></script>'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
def test_vendored_hls_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<script src="/shared/hls-1.6.15/hls.min.js"></script>'
|
||||
html = '<script src="/shared/hls-1.6.16/hls.min.js"></script>'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
@@ -76,7 +76,7 @@ class TestVersionHtml:
|
||||
|
||||
html = (
|
||||
'<link rel="stylesheet" href="/shared/base.css">\n'
|
||||
'<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">\n'
|
||||
'<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">\n'
|
||||
'<link rel="stylesheet" href="/static/style.css">\n'
|
||||
'<script src="/shared/utils.js"></script>\n'
|
||||
'<script src="/shared/hljs-11.11.1/highlight.min.js"></script>\n'
|
||||
@@ -88,7 +88,7 @@ class TestVersionHtml:
|
||||
assert f'/shared/utils.js?v={__version__}"' in result
|
||||
assert f'/static/app.js?v={__version__}"' in result
|
||||
# Vendored libs unchanged
|
||||
assert '/shared/katex-0.16.44/katex.min.css"' in result
|
||||
assert '/shared/katex-0.16.47/katex.min.css"' in result
|
||||
assert '/shared/hljs-11.11.1/highlight.min.js"' in result
|
||||
|
||||
def test_version_matches_package(self):
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import queue
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -1762,3 +1764,155 @@ class TestTenantCheckOnReadEndpoints:
|
||||
assert cold_check in offloaded, (
|
||||
f"tenant_check must be invoked through asyncio.to_thread; got {offloaded}"
|
||||
)
|
||||
|
||||
|
||||
class TestHistoryReasoningRehydration:
|
||||
"""The lifted ``GET /v1/api/workstreams/{ws_id}/history`` surfaces
|
||||
stored Anthropic thinking blocks on assistant messages so a page
|
||||
refresh re-renders the reasoning bubble. Drives through the real
|
||||
``AnthropicProvider.extract_reasoning_text`` and the storage
|
||||
``reconstruct_messages`` boundary that JSON-decodes
|
||||
``provider_data`` into ``_provider_content``.
|
||||
"""
|
||||
|
||||
def test_history_handler_surfaces_reasoning_for_anthropic_thinking(self, _inject_storage):
|
||||
ws_id = "ws-reason-1"
|
||||
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
|
||||
provider_data = json.dumps(
|
||||
[
|
||||
{"type": "thinking", "thinking": "let me reason", "signature": "s"},
|
||||
{"type": "text", "text": "Final answer."},
|
||||
]
|
||||
)
|
||||
_inject_storage.save_message(
|
||||
ws_id, "assistant", "Final answer.", provider_data=provider_data
|
||||
)
|
||||
# No live session — exercises the storage-only path which
|
||||
# falls back to default surface_persisted_reasoning=True.
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = None
|
||||
client = _build_history_app(mock_mgr, _inject_storage)
|
||||
|
||||
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
|
||||
assert r.status_code == 200
|
||||
msgs = r.json()["messages"]
|
||||
assistant = next(m for m in msgs if m.get("role") == "assistant")
|
||||
assert assistant["reasoning"] == "let me reason"
|
||||
|
||||
def test_history_handler_strips_provider_content(self, _inject_storage):
|
||||
ws_id = "ws-reason-2"
|
||||
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
|
||||
provider_data = json.dumps([{"type": "thinking", "thinking": "x", "signature": "s"}])
|
||||
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = None
|
||||
client = _build_history_app(mock_mgr, _inject_storage)
|
||||
|
||||
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
|
||||
assert r.status_code == 200
|
||||
for m in r.json()["messages"]:
|
||||
assert "_provider_content" not in m
|
||||
|
||||
def test_history_handler_with_persist_flag_false_via_live_session(self, _inject_storage):
|
||||
"""Operator-flipped ``surface_persisted_reasoning=False`` on the active
|
||||
model suppresses the reasoning field even when the data is
|
||||
stored. ``_provider_content`` is still stripped from the wire.
|
||||
"""
|
||||
ws_id = "ws-reason-3"
|
||||
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
|
||||
provider_data = json.dumps([{"type": "thinking", "thinking": "hidden", "signature": "s"}])
|
||||
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
|
||||
live_session = SimpleNamespace(
|
||||
id=ws_id,
|
||||
_registry=SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(surface_persisted_reasoning=False)
|
||||
),
|
||||
_model_alias="claude-opus-4-7",
|
||||
)
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = live_session
|
||||
client = _build_history_app(mock_mgr, _inject_storage)
|
||||
|
||||
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
|
||||
assert r.status_code == 200
|
||||
for m in r.json()["messages"]:
|
||||
if m.get("role") == "assistant":
|
||||
assert "reasoning" not in m
|
||||
assert "_provider_content" not in m
|
||||
|
||||
def test_history_handler_cold_workstream_resolves_via_workstream_config(self, _inject_storage):
|
||||
"""Cold workstream (no live session) — the handler walks
|
||||
``workstream_config.model_alias`` (persisted at first send by
|
||||
the SessionManager rehydrate path) and looks up the active
|
||||
model's ``surface_persisted_reasoning`` flag through the global registry
|
||||
on ``app.state``. Operator flag-flip is honored uniformly
|
||||
across live and cold workstreams.
|
||||
"""
|
||||
ws_id = "ws-reason-cold"
|
||||
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
|
||||
# Simulate the model alias persisted by the rehydrate path
|
||||
# (session_manager.py:628-629 reads it back via the same key).
|
||||
_inject_storage.save_workstream_config(ws_id, {"model_alias": "claude-opus-4-7"})
|
||||
provider_data = json.dumps(
|
||||
[{"type": "thinking", "thinking": "should not surface", "signature": "s"}]
|
||||
)
|
||||
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
|
||||
# No live session — handler falls back to workstream_config + registry.
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = None
|
||||
|
||||
# Build the app with a global registry that reports persist=False
|
||||
# for the saved alias.
|
||||
cfg = _interactive_endpoint_cfg(mock_mgr)
|
||||
handler = make_history_handler(cfg)
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Mount(
|
||||
"/v1",
|
||||
routes=[
|
||||
Route(
|
||||
"/api/workstreams/{ws_id}/history",
|
||||
handler,
|
||||
methods=["GET"],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_InjectAuthMiddleware)],
|
||||
)
|
||||
app.state.workstreams = mock_mgr
|
||||
app.state.auth_storage = _inject_storage
|
||||
app.state.registry = SimpleNamespace(
|
||||
get_config=lambda alias: SimpleNamespace(
|
||||
surface_persisted_reasoning=(alias != "claude-opus-4-7"),
|
||||
)
|
||||
)
|
||||
client = TestClient(app)
|
||||
|
||||
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
|
||||
assert r.status_code == 200
|
||||
# Flag-flip on the saved alias is honored: reasoning suppressed.
|
||||
for m in r.json()["messages"]:
|
||||
if m.get("role") == "assistant":
|
||||
assert "reasoning" not in m
|
||||
assert "_provider_content" not in m
|
||||
|
||||
def test_history_handler_cold_workstream_no_alias_defaults_true(self, _inject_storage):
|
||||
"""A workstream that pre-dates the rehydrate-time alias persist
|
||||
(or one that simply has no workstream_config row) falls through
|
||||
to the conservative default ``True``. Reasoning surfaces.
|
||||
"""
|
||||
ws_id = "ws-reason-cold-no-alias"
|
||||
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
|
||||
provider_data = json.dumps(
|
||||
[{"type": "thinking", "thinking": "default-true wins", "signature": "s"}]
|
||||
)
|
||||
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
|
||||
mock_mgr = MagicMock()
|
||||
mock_mgr.get.return_value = None
|
||||
client = _build_history_app(mock_mgr, _inject_storage)
|
||||
|
||||
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
|
||||
assert r.status_code == 200
|
||||
assistant = next(m for m in r.json()["messages"] if m.get("role") == "assistant")
|
||||
assert assistant["reasoning"] == "default-true wins"
|
||||
|
||||
@@ -62,6 +62,13 @@
|
||||
[database]
|
||||
# url = "" # postgres://user:pass@host/db or /path/to.db
|
||||
# env: TURNSTONE_DB_URL
|
||||
# listen_url = "" # direct-to-postgres URL for the console's
|
||||
# dedicated LISTEN connection. Set this when
|
||||
# `url` points at pgbouncer in transaction
|
||||
# pooling mode (LISTEN holds session state and
|
||||
# is incompatible with transaction pooling —
|
||||
# see docs/pgbouncer.md). Defaults to `url`
|
||||
# when unset. env: TURNSTONE_DB_LISTEN_URL
|
||||
# SSL params (passed through to SQLAlchemy connection):
|
||||
# sslmode = "prefer" # disable, allow, prefer, require, verify-ca, verify-full
|
||||
# sslrootcert = "" # path to CA cert for verify-ca/verify-full
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "1.5.10"
|
||||
__version__ = "1.5.18"
|
||||
|
||||
+40
-14
@@ -12,14 +12,36 @@ import uuid
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _get_storage() -> Any:
|
||||
"""Initialize and return the storage backend."""
|
||||
def _get_storage(args: argparse.Namespace) -> Any:
|
||||
"""Initialize and return the storage backend.
|
||||
|
||||
Precedence (matches turnstone-server): CLI / config.toml ``[database]``
|
||||
> ``TURNSTONE_DB_*`` env vars > hardcoded defaults.
|
||||
"""
|
||||
from turnstone.core.storage import init_storage
|
||||
|
||||
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
db_url = os.environ.get("TURNSTONE_DB_URL", "")
|
||||
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
|
||||
return init_storage(db_backend, path=db_path, url=db_url)
|
||||
def _pick(arg_name: str, env_name: str, default: str = "") -> Any:
|
||||
# `is not None` (not truthy) so a legitimate falsy TOML value
|
||||
# like `pool_size = 0` or `url = ""` still beats the env fallback.
|
||||
val = getattr(args, arg_name, None)
|
||||
if val is not None:
|
||||
return val
|
||||
return os.environ.get(env_name, default)
|
||||
|
||||
db_backend = str(_pick("db_backend", "TURNSTONE_DB_BACKEND", "sqlite"))
|
||||
db_url = str(_pick("db_url", "TURNSTONE_DB_URL"))
|
||||
db_path = str(_pick("db_path", "TURNSTONE_DB_PATH"))
|
||||
db_pool_size = int(_pick("db_pool_size", "TURNSTONE_DB_POOL_SIZE", "2"))
|
||||
return init_storage(
|
||||
db_backend,
|
||||
path=db_path,
|
||||
url=db_url,
|
||||
pool_size=db_pool_size,
|
||||
sslmode=str(_pick("db_sslmode", "TURNSTONE_DB_SSLMODE")),
|
||||
sslrootcert=str(_pick("db_sslrootcert", "TURNSTONE_DB_SSLROOTCERT")),
|
||||
sslcert=str(_pick("db_sslcert", "TURNSTONE_DB_SSLCERT")),
|
||||
sslkey=str(_pick("db_sslkey", "TURNSTONE_DB_SSLKEY")),
|
||||
)
|
||||
|
||||
|
||||
def _cmd_create_user(args: argparse.Namespace) -> None:
|
||||
@@ -37,7 +59,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
|
||||
print("Error: invalid username (1-64 chars: letters, digits, . _ -)", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
user_id = uuid.uuid4().hex
|
||||
|
||||
# Prompt for password
|
||||
@@ -76,7 +98,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
|
||||
def _cmd_create_token(args: argparse.Namespace) -> None:
|
||||
from turnstone.core.auth import generate_token, hash_token, token_prefix
|
||||
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
|
||||
if storage.get_user(args.user) is None:
|
||||
print(f"Error: user {args.user} not found", file=sys.stderr)
|
||||
@@ -110,7 +132,7 @@ def _cmd_create_token(args: argparse.Namespace) -> None:
|
||||
|
||||
|
||||
def _cmd_list_users(args: argparse.Namespace) -> None:
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
users = storage.list_users()
|
||||
if not users:
|
||||
print("No users found.")
|
||||
@@ -120,7 +142,7 @@ def _cmd_list_users(args: argparse.Namespace) -> None:
|
||||
|
||||
|
||||
def _cmd_list_tokens(args: argparse.Namespace) -> None:
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
tokens = storage.list_api_tokens(args.user)
|
||||
if not tokens:
|
||||
print(f"No tokens found for user {args.user}.")
|
||||
@@ -134,7 +156,7 @@ def _cmd_list_tokens(args: argparse.Namespace) -> None:
|
||||
|
||||
|
||||
def _cmd_revoke_token(args: argparse.Namespace) -> None:
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
if storage.delete_api_token(args.token_id):
|
||||
print(f"Revoked token {args.token_id}")
|
||||
else:
|
||||
@@ -297,7 +319,7 @@ def _cmd_list_node_metadata(args: argparse.Namespace) -> None:
|
||||
"""List metadata for a node."""
|
||||
import json
|
||||
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
rows = storage.get_node_metadata(args.node_id)
|
||||
if not rows:
|
||||
print(f"No metadata for node: {args.node_id}")
|
||||
@@ -324,7 +346,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
|
||||
"""Set a metadata key on a node."""
|
||||
import json
|
||||
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
|
||||
# Check for auto-source conflict
|
||||
existing = storage.get_node_metadata(args.node_id)
|
||||
@@ -345,7 +367,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
|
||||
|
||||
def _cmd_delete_node_metadata(args: argparse.Namespace) -> None:
|
||||
"""Delete a metadata key from a node."""
|
||||
storage = _get_storage()
|
||||
storage = _get_storage(args)
|
||||
|
||||
existing = storage.get_node_metadata(args.node_id)
|
||||
for r in existing:
|
||||
@@ -395,6 +417,10 @@ def main() -> None:
|
||||
prog="turnstone-admin",
|
||||
description="Turnstone user and token administration",
|
||||
)
|
||||
from turnstone.core.config import add_config_arg, apply_config
|
||||
|
||||
add_config_arg(parser)
|
||||
apply_config(parser, ["database"])
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
p_cu = sub.add_parser("create-user", help="Create a new user")
|
||||
|
||||
@@ -908,6 +908,8 @@ class ModelDefinitionInfo(BaseModel):
|
||||
temperature: float | None = None
|
||||
max_tokens: int | None = None
|
||||
reasoning_effort: str | None = None
|
||||
surface_persisted_reasoning: bool = True
|
||||
replay_reasoning_to_model: bool = False
|
||||
source: str = ""
|
||||
created_by: str = ""
|
||||
created: str = ""
|
||||
@@ -926,6 +928,8 @@ class CreateModelDefinitionRequest(BaseModel):
|
||||
temperature: float | None = None
|
||||
max_tokens: int | None = None
|
||||
reasoning_effort: str | None = None
|
||||
surface_persisted_reasoning: bool = True
|
||||
replay_reasoning_to_model: bool = False
|
||||
|
||||
|
||||
class UpdateModelDefinitionRequest(BaseModel):
|
||||
@@ -940,6 +944,8 @@ class UpdateModelDefinitionRequest(BaseModel):
|
||||
temperature: float | None = None
|
||||
max_tokens: int | None = None
|
||||
reasoning_effort: str | None = None
|
||||
surface_persisted_reasoning: bool | None = None
|
||||
replay_reasoning_to_model: bool | None = None
|
||||
|
||||
|
||||
class ListModelDefinitionsResponse(BaseModel):
|
||||
@@ -990,6 +996,8 @@ class ListAvailableModelsResponse(BaseModel):
|
||||
models: list[AvailableModelInfo] = Field(default_factory=list)
|
||||
default_alias: str = ""
|
||||
channel_default_alias: str = ""
|
||||
coordinator_default_alias: str = ""
|
||||
judge_default_alias: str = ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -110,6 +110,14 @@ class TerminalUI(SessionUI):
|
||||
self.auto_approve = False
|
||||
self.auto_approve_tools: set[str] = set()
|
||||
|
||||
def on_turn_start(self) -> None:
|
||||
# Terminal UI has no inflight buffer to reset.
|
||||
pass
|
||||
|
||||
def on_turn_committed(self) -> None:
|
||||
# Terminal UI has no inflight buffer to reset.
|
||||
pass
|
||||
|
||||
def on_thinking_start(self) -> None:
|
||||
self.spinner = Spinner("Thinking")
|
||||
self.spinner.start()
|
||||
|
||||
@@ -25,9 +25,13 @@ import httpx_sse
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
from turnstone.core.auth import ServiceTokenManager
|
||||
from turnstone.core.storage._notify import Notify
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = logging.getLogger("turnstone.console.collector")
|
||||
@@ -73,6 +77,7 @@ class ClusterCollector:
|
||||
tls_cert: tuple[str, str] | None = None,
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: NotifyDispatcher | None = None,
|
||||
):
|
||||
self._storage = storage
|
||||
self._discovery_interval = discovery_interval
|
||||
@@ -82,6 +87,8 @@ class ClusterCollector:
|
||||
self._console_metrics = console_metrics
|
||||
self._tls_verify = tls_verify
|
||||
self._tls_cert = tls_cert
|
||||
self._notify_dispatcher = notify_dispatcher
|
||||
self._notify_unsubscribe: Callable[[], None] | None = None
|
||||
|
||||
self._lock = threading.Lock()
|
||||
self._nodes: dict[str, NodeSnapshot] = {}
|
||||
@@ -128,6 +135,15 @@ class ClusterCollector:
|
||||
def start(self) -> None:
|
||||
"""Start background threads."""
|
||||
self._running = True
|
||||
# Subscribe to the ``services`` channel for reactive node discovery.
|
||||
# NOTIFY-driven wake-ups bring new-node visibility from up-to-60 s
|
||||
# (next discovery tick) down to ~500 ms on Postgres; the 60 s
|
||||
# discovery loop still runs as the backstop for crash-shaped node
|
||||
# loss (NOTIFY only fires on actual writes, not on crash exits).
|
||||
if self._notify_dispatcher is not None:
|
||||
self._notify_unsubscribe = self._notify_dispatcher.subscribe(
|
||||
"services", self._on_services_notify
|
||||
)
|
||||
for target, name in [
|
||||
(self._discovery_loop, "console-discovery"),
|
||||
(self._sse_manager_thread, "console-sse"),
|
||||
@@ -145,6 +161,10 @@ class ClusterCollector:
|
||||
its ``finally`` cleanup (cancel tasks, close AsyncClient).
|
||||
"""
|
||||
self._running = False
|
||||
if self._notify_unsubscribe is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
self._notify_unsubscribe()
|
||||
self._notify_unsubscribe = None
|
||||
# Request cancellation of all SSE tasks so they don't block the
|
||||
# manager's cleanup. The manager coroutine exits when _running is
|
||||
# False and handles remaining task cancellation in its finally block.
|
||||
@@ -156,6 +176,26 @@ class ClusterCollector:
|
||||
t.join(timeout=5)
|
||||
log.info("ClusterCollector stopped")
|
||||
|
||||
def _on_services_notify(self, notify: Notify) -> None:
|
||||
"""Run a discovery tick when the ``services`` channel fires.
|
||||
|
||||
The dispatcher delivers both real Postgres notifications and
|
||||
synthetic ``reconcile`` wake-ups after a reconnect — both shape
|
||||
the same way: re-read ``services`` and diff against in-memory
|
||||
state. Re-uses :meth:`_discover_nodes` so the timer-driven
|
||||
backstop and the NOTIFY-driven fast-path share one code path.
|
||||
"""
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
self._discover_nodes()
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("Node discovery error (notify-driven)")
|
||||
|
||||
def _fanout(self, event: dict[str, Any]) -> None:
|
||||
"""Copy an event to all registered SSE listener queues."""
|
||||
with self._listeners_lock:
|
||||
|
||||
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.adapters._ui_cleanup import cleanup_session_ui
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.child_source import ClusterChildSource
|
||||
from turnstone.core.children_registry import ChildrenRegistry
|
||||
from turnstone.core.log import get_logger
|
||||
@@ -67,6 +68,14 @@ class CoordinatorAdapter:
|
||||
# method names which still exist as thin shims for the
|
||||
# cluster-routing + cleanup callers.
|
||||
self._registry = ChildrenRegistry()
|
||||
# In-process wakeup primitive for ``wait_for_workstream``. The
|
||||
# dispatch sink (:meth:`_dispatch_child_event`) calls
|
||||
# ``notify(child_ws_id)`` after each translated child event;
|
||||
# waiters block on per-call ``threading.Event``s instead of
|
||||
# polling storage. Owned by the adapter so the manager-level
|
||||
# exposure can simply delegate; ``CoordinatorClient`` picks it
|
||||
# up via the coord client factory closure.
|
||||
self._child_event_bus = ChildEventBus()
|
||||
# Cross-node child events arrive via ``ClusterChildSource``
|
||||
# (Stage 3 Step 2): a strategy that subscribes to the
|
||||
# collector's listener channel and runs a daemon thread that
|
||||
@@ -77,6 +86,17 @@ class CoordinatorAdapter:
|
||||
# the collector reference is available.
|
||||
self._child_source: ClusterChildSource | None = None
|
||||
|
||||
@property
|
||||
def child_event_bus(self) -> ChildEventBus:
|
||||
"""In-process wakeup bus consumed by ``wait_for_workstream``.
|
||||
|
||||
Exposed so the coord client factory in the console bootstrap
|
||||
can pass it to :class:`CoordinatorClient` without reaching
|
||||
into a private attr, and so :class:`SessionManager` can
|
||||
delegate its own ``child_event_bus`` property here.
|
||||
"""
|
||||
return self._child_event_bus
|
||||
|
||||
def attach(self, manager: SessionManager) -> None:
|
||||
"""Late-bind the owning :class:`SessionManager`.
|
||||
|
||||
@@ -639,6 +659,13 @@ class CoordinatorAdapter:
|
||||
"detail": event.get("detail") or {},
|
||||
}
|
||||
_enqueue_on_ui(owning_ws.ui, coord_id, child_event)
|
||||
# Wake any in-process ``wait_for_workstream`` subscriber on
|
||||
# this child. Notify runs AFTER the UI enqueue so the SSE
|
||||
# fan-out keeps priority (a wait that wakes early sees the
|
||||
# state already enqueued for its owning dashboard). The bus
|
||||
# is a no-op when no waiter is registered — the steady
|
||||
# state for the hot dispatch path.
|
||||
self._child_event_bus.notify(ws_id)
|
||||
|
||||
|
||||
def _enqueue_on_ui(ui: Any, coord_ws_id: str, payload: dict[str, Any]) -> None:
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Coordinator alias resolution shared by the placeholder API and the
|
||||
session factory.
|
||||
|
||||
Both ``/v1/api/models`` (advertises the resolved default to the home
|
||||
composer) and ``console/session_factory.py:factory`` (resolves the
|
||||
alias new coordinator sessions launch on) walk the same three-tier
|
||||
chain. Centralising it here means the tier names and the tier-2
|
||||
validation policy live once — the prior arrangement was two
|
||||
implementations coupled by a "keep these in sync" comment, which is
|
||||
exactly the drift trap that produced the historical bug where the
|
||||
home composer advertised one alias while sessions ran on another.
|
||||
|
||||
Tiers, in priority order:
|
||||
|
||||
1. **Explicit pin** — per-call ``model_alias`` arg (factory only) or
|
||||
the ``coordinator.model_alias`` ConfigStore setting.
|
||||
2. **System default** — ``model.default_alias`` ConfigStore setting,
|
||||
admin-managed in the Models tab. Validated against
|
||||
``registry.has_alias()`` — a stale or typo'd value falls through
|
||||
with a logged warning rather than 503ing.
|
||||
3. **Registry default** — ``registry.default`` (config.toml
|
||||
``[model].default``), guaranteed by the registry to resolve.
|
||||
|
||||
Tier 1 is intentionally passed through unvalidated by default: an
|
||||
explicit operator pin should surface as 503 at ``registry.resolve``
|
||||
when stale, not silently fall through to a different alias. Callers
|
||||
that need stricter filtering (the placeholder API restricts to
|
||||
enabled DB rows so the home composer doesn't advertise a model the
|
||||
workstream picker can't offer) supply an ``alias_filter`` predicate
|
||||
applied to every tier.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.config_store import ConfigStore
|
||||
from turnstone.core.model_registry import ModelRegistry
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def resolve_coordinator_alias(
|
||||
*,
|
||||
explicit: str | None,
|
||||
config_store: ConfigStore,
|
||||
registry: ModelRegistry,
|
||||
alias_filter: Callable[[str], bool] | None = None,
|
||||
) -> str:
|
||||
"""Resolve the effective coordinator alias through the three tiers.
|
||||
|
||||
See module docstring for the full chain. Returns the concrete
|
||||
alias name, or ``""`` when every tier failed (rare — only when
|
||||
``registry.default`` itself fails the filter).
|
||||
"""
|
||||
|
||||
def _accept(alias: str) -> bool:
|
||||
if not alias:
|
||||
return False
|
||||
return alias_filter(alias) if alias_filter is not None else True
|
||||
|
||||
explicit_alias = (explicit or "").strip()
|
||||
if not explicit_alias:
|
||||
explicit_alias = (config_store.get("coordinator.model_alias") or "").strip()
|
||||
if _accept(explicit_alias):
|
||||
return explicit_alias
|
||||
|
||||
fallback_alias = (config_store.get("model.default_alias") or "").strip()
|
||||
if fallback_alias and not registry.has_alias(fallback_alias):
|
||||
log.warning(
|
||||
"coord_alias.model_default_alias_unknown alias=%r "
|
||||
"— falling through to registry.default",
|
||||
fallback_alias,
|
||||
)
|
||||
fallback_alias = ""
|
||||
if _accept(fallback_alias):
|
||||
return fallback_alias
|
||||
|
||||
registry_default = registry.default or ""
|
||||
if _accept(registry_default):
|
||||
return registry_default
|
||||
|
||||
return ""
|
||||
@@ -73,12 +73,27 @@ WAIT_MAX_WS_IDS: int = 32
|
||||
# wait_for_workstream again with the same ws_ids — each call re-arms freshly.
|
||||
WAIT_MAX_TIMEOUT: float = 600.0
|
||||
|
||||
# Storage-poll cadence. 500ms is short enough that the wait terminates
|
||||
# promptly after a child finishes (well under the human-perceptible-latency
|
||||
# floor), and long enough that a 60s wait incurs at most 120 cheap row
|
||||
# reads — still cheaper than the 20+ inspect_workstream model turns the
|
||||
# tool replaces.
|
||||
WAIT_POLL_INTERVAL: float = 0.5
|
||||
# Maximum ``event.wait`` interval in the bus-driven wait loop.
|
||||
# A long-running stuck child would otherwise look dead in the sidebar UI
|
||||
# because the ``wait_progress`` SSE emission piggybacks on the wait loop
|
||||
# — capping at 2 s keeps the heartbeat visible without flooding storage.
|
||||
# Today's polling effectively snapshots every 500 ms; 2 s preserves a
|
||||
# similar liveness feel while cutting per-listener SSE traffic ~4x in the
|
||||
# steady-state-quiescent case. Tunable post-merge if profiling shows
|
||||
# storage-read pressure on state-change wakes.
|
||||
#
|
||||
# **Worst-case completion latency**: 2 s. ``SessionManager.set_state``
|
||||
# buffers non-ERROR storage writes through ``StateWriter`` (async-flushed
|
||||
# at ~1 s cadence) while ``emit_state`` fans the event out immediately —
|
||||
# a bus-driven wake can therefore beat the flusher and read pre-transition
|
||||
# state on a terminal transition, then re-block on ``event.wait`` until
|
||||
# the heartbeat cap fires. Pre-bus the 0.5 s poll bounded this at 0.5 s.
|
||||
# Going to 2 s is intentional: the 4x SSE-traffic reduction in the
|
||||
# steady-state-quiescent case outweighs the worst-case latency
|
||||
# regression on the most common terminal transition, and a model issuing
|
||||
# a follow-up ``inspect_workstream`` (the pre-bus pattern this tool
|
||||
# replaces) was already paying multi-second model-turn latency per probe.
|
||||
WAIT_HEARTBEAT_INTERVAL: float = 2.0
|
||||
|
||||
# Per-ws cap on the inline ``message`` field bundled into wait_for_workstream
|
||||
# results. Sized so a fan-out of 32 children at the cap is ~320 KiB of
|
||||
@@ -184,6 +199,7 @@ def load_task_envelope(storage: Any, ws_id: str) -> tuple[dict[str, Any], bool]:
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -313,6 +329,7 @@ class CoordinatorClient:
|
||||
user_id: str,
|
||||
timeout: float = 30.0,
|
||||
http_client: httpx.Client | None = None,
|
||||
child_event_bus: ChildEventBus,
|
||||
) -> None:
|
||||
self._base_url = console_base_url.rstrip("/")
|
||||
self._storage = storage
|
||||
@@ -325,6 +342,12 @@ class CoordinatorClient:
|
||||
# with the coordinator session.
|
||||
self._http = http_client or httpx.Client(timeout=timeout)
|
||||
self._owns_http = http_client is None
|
||||
# In-process wakeup bus for ``wait_for_workstream``. The wait
|
||||
# loop blocks on a ``threading.Event`` keyed by ws_id and only
|
||||
# re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. Owned by ``CoordinatorAdapter`` in production; tests
|
||||
# pass their own instance.
|
||||
self._child_event_bus = child_event_bus
|
||||
# tasks per-ws lock cache — populated lazily by _task_lock().
|
||||
# Single-session so a plain dict behind a coarse lock is fine;
|
||||
# WeakValueDictionary isn't needed (entries live as long as the
|
||||
@@ -447,6 +470,27 @@ class CoordinatorClient:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
def _row_in_own_subtree(self, ws_id: str, row: dict[str, Any] | None) -> bool:
|
||||
"""Row-level subtree predicate sharing one home for read paths.
|
||||
|
||||
Both :meth:`wait_for_workstream`'s pre-loop ownership filter and
|
||||
its inner ``_snapshot_all`` already have the workstream row in
|
||||
hand (from ``get_workstreams_batch``). Funneling them through
|
||||
the same 4-line check keeps the predicate in lockstep with
|
||||
:meth:`_is_own_subtree` (used by mutating ops) — both require
|
||||
``parent_ws_id`` AND ``user_id`` parity so a corrupted or
|
||||
forged ``parent_ws_id`` alone can't satisfy the gate on either
|
||||
path. Returns False on a missing / None row so callers can
|
||||
safely pass ``rows.get(wid)``.
|
||||
"""
|
||||
if ws_id == self._coord_ws_id:
|
||||
return True
|
||||
if row is None:
|
||||
return False
|
||||
if row.get("parent_ws_id") != self._coord_ws_id:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
# -- model-invoked mutating ops (HTTP) ---------------------------------
|
||||
|
||||
def spawn(
|
||||
@@ -562,7 +606,7 @@ class CoordinatorClient:
|
||||
_WAIT_TERMINAL_STATES: ClassVar[frozenset[str]] = WAIT_TERMINAL_STATES
|
||||
_WAIT_MAX_WS_IDS: ClassVar[int] = WAIT_MAX_WS_IDS
|
||||
_WAIT_MAX_TIMEOUT: ClassVar[float] = WAIT_MAX_TIMEOUT
|
||||
_WAIT_POLL_INTERVAL: ClassVar[float] = WAIT_POLL_INTERVAL
|
||||
_WAIT_HEARTBEAT_INTERVAL: ClassVar[float] = WAIT_HEARTBEAT_INTERVAL
|
||||
|
||||
def wait_for_workstream(
|
||||
self,
|
||||
@@ -721,12 +765,7 @@ class CoordinatorClient:
|
||||
snaps: dict[str, dict[str, Any]] = {}
|
||||
for wid in cleaned:
|
||||
row = rows.get(wid)
|
||||
if row is None:
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
is_self = wid == self._coord_ws_id
|
||||
is_own_child = row.get("parent_ws_id") == self._coord_ws_id
|
||||
if not (is_self or is_own_child):
|
||||
if row is None or not self._row_in_own_subtree(wid, row):
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
snaps[wid] = {
|
||||
@@ -760,54 +799,119 @@ class CoordinatorClient:
|
||||
|
||||
last_results: dict[str, dict[str, Any]] = {}
|
||||
complete = False
|
||||
while True:
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
# Subscribe to in-process state-change events for the watched
|
||||
# ws_ids when the bus is wired. ``register_waiter`` returns a
|
||||
# single ``threading.Event`` registered against every id so a
|
||||
# wait on [A, B, C] wakes on any of A/B/C changing. Bus is
|
||||
# optional so test fixtures that don't wire it fall back to the
|
||||
# legacy ``time.sleep`` cadence with no behaviour change.
|
||||
#
|
||||
# **Defense-in-depth ownership filter**: ``_dispatch_child_event``
|
||||
# fires ``bus.notify(ws_id)`` for every ws_id in *any* coord's
|
||||
# registry on this console process, so a foreign ws_id passed by
|
||||
# an untrusted coord LLM (prompt injection) would otherwise leak
|
||||
# wake-up timing as a side channel — _snapshot_all returns
|
||||
# ``denied`` for the content, but the *time* at which the wait
|
||||
# un-blocked would correlate with the foreign ws_id's next
|
||||
# state-class event. Filter ``cleaned`` to own-subtree ids
|
||||
# before registering; foreign / missing ws_ids stay in the
|
||||
# snapshot list so they still surface as ``denied`` in
|
||||
# ``_snapshot_all`` and exit via the pure-denied short-circuit
|
||||
# below. Predicate shared with ``_snapshot_all`` via
|
||||
# :meth:`_row_in_own_subtree`.
|
||||
try:
|
||||
pre_rows = self._storage.get_workstreams_batch(cleaned)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.ownership_filter_failed", exc_info=True)
|
||||
pre_rows = {wid: None for wid in cleaned}
|
||||
own_subtree = [wid for wid in cleaned if self._row_in_own_subtree(wid, pre_rows.get(wid))]
|
||||
bus = self._child_event_bus
|
||||
wake_event = bus.register_waiter(own_subtree) if own_subtree else None
|
||||
try:
|
||||
while True:
|
||||
# Clear BEFORE the storage snapshot to close the
|
||||
# subscribe/check race: any ``notify`` between clear
|
||||
# and the next ``wake_event.wait`` leaves the Event
|
||||
# set, so the wait returns immediately and the loop
|
||||
# re-snapshots without losing the wake-up.
|
||||
if wake_event is not None:
|
||||
wake_event.clear()
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
time.sleep(min(self._WAIT_POLL_INTERVAL, remaining))
|
||||
if wake_event is not None:
|
||||
# Block until a child state-change notify fires OR
|
||||
# the heartbeat cap expires (so a stuck child still
|
||||
# emits a periodic ``wait_progress`` for the
|
||||
# sidebar UX). Heartbeat cap is the only timer —
|
||||
# the bus is the wake source. See
|
||||
# ``WAIT_HEARTBEAT_INTERVAL`` (module top) for the
|
||||
# worst-case completion-latency rationale: 2 s is
|
||||
# a deliberate 4x trade vs the pre-bus 0.5 s poll.
|
||||
wake_event.wait(min(remaining, self._WAIT_HEARTBEAT_INTERVAL))
|
||||
else:
|
||||
# Pure-foreign / pure-denied list: every cleaned
|
||||
# ws_id was filtered out of ``own_subtree`` so the
|
||||
# bus has nothing to wake on. The pure-denied
|
||||
# short-circuit above exits ``mode='any'`` on the
|
||||
# first tick; ``mode='all'`` falls through to here
|
||||
# and must burn the timeout. Use the heartbeat
|
||||
# cadence for the deadline carve-up so
|
||||
# ``progress_callback`` keeps firing.
|
||||
time.sleep(min(self._WAIT_HEARTBEAT_INTERVAL, remaining))
|
||||
finally:
|
||||
# Always unregister so a crash mid-wait can't leak the
|
||||
# registration past one wait's lifetime. Bus discards
|
||||
# empty buckets so long-lived buses don't accumulate dead
|
||||
# keys after many waits. Unregister against the same
|
||||
# ``own_subtree`` list the register call used — passing
|
||||
# ``cleaned`` here would silently no-op for foreign ids
|
||||
# but pass an unknown bucket to ``unregister_waiter``.
|
||||
if wake_event is not None:
|
||||
bus.unregister_waiter(own_subtree, wake_event)
|
||||
# Bundle each terminal child's last assistant message inline so the
|
||||
# coordinator LLM doesn't have to follow up with one
|
||||
# ``inspect_workstream`` per ws. Only ``idle`` / ``error`` ws_ids
|
||||
@@ -816,7 +920,7 @@ class CoordinatorClient:
|
||||
# subset across a small thread pool — at the WAIT_MAX_WS_IDS=32
|
||||
# cap, 8 workers cuts a worst-case all-idle fan-out from 32
|
||||
# sequential storage round-trips down to 4 batches, which lands
|
||||
# inside the WAIT_POLL_INTERVAL the model already tolerates
|
||||
# inside the WAIT_HEARTBEAT_INTERVAL the model already tolerates
|
||||
# between ticks. Storage backends use SQLAlchemy with
|
||||
# ``check_same_thread=False`` (SQLite) / a connection pool
|
||||
# (Postgres), so concurrent reads from the worker pool are safe.
|
||||
@@ -1174,21 +1278,29 @@ class CoordinatorClient:
|
||||
allowed_tools: list[str] = [str(t) for t in allowed_full[:_SKILL_TOOLS_PROJECTION_CAP]]
|
||||
if len(allowed_full) > _SKILL_TOOLS_PROJECTION_CAP:
|
||||
allowed_tools.append(f"+{len(allowed_full) - _SKILL_TOOLS_PROJECTION_CAP} more")
|
||||
skills.append(
|
||||
{
|
||||
"name": r.get("name") or "",
|
||||
"category": r.get("category") or "",
|
||||
"tags": tags,
|
||||
"version": r.get("version") or "",
|
||||
"description": r.get("description") or "",
|
||||
"model": r.get("model") or "",
|
||||
"enabled": bool(r.get("enabled")),
|
||||
"risk_level": r.get("risk_level") or "",
|
||||
"activation": r.get("activation") or "",
|
||||
"kind": r["kind"],
|
||||
"allowed_tools": allowed_tools,
|
||||
}
|
||||
)
|
||||
skill_row: dict[str, Any] = {
|
||||
"name": r.get("name") or "",
|
||||
"category": r.get("category") or "",
|
||||
"tags": tags,
|
||||
"version": r.get("version") or "",
|
||||
"description": r.get("description") or "",
|
||||
"model": r.get("model") or "",
|
||||
"enabled": bool(r.get("enabled")),
|
||||
"risk_level": r.get("risk_level") or "",
|
||||
"activation": r.get("activation") or "",
|
||||
"kind": r["kind"],
|
||||
}
|
||||
# Omit ``allowed_tools`` when empty: an empty list reads as
|
||||
# "no tools are usable by this skill" to a model that doesn't
|
||||
# know the semantics, but the actual meaning is "no tools are
|
||||
# pre-approved (auto-approve exemption list)". Real
|
||||
# misdiagnosis happened in testing when a code-review skill
|
||||
# with no auto-approve allowlist looked like it had been
|
||||
# spawned with zero tool access. Dropping the key altogether
|
||||
# when empty removes the ambiguity at the source.
|
||||
if allowed_tools:
|
||||
skill_row["allowed_tools"] = allowed_tools
|
||||
skills.append(skill_row)
|
||||
return {"skills": skills, "truncated": truncated}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -1687,6 +1799,293 @@ def _serialize_verdicts(rows: list[Any]) -> list[dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# inspect_workstream — tiered output compression
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# A coord doing a fan-out wave of inspect_workstream calls against
|
||||
# tool-heavy children can blow the context budget on raw output alone
|
||||
# (one child with a 100 KB bash result × N children). The previous
|
||||
# safety net was ``_truncate_output``'s head+tail strategy, which
|
||||
# silently drops *middle* messages — exactly the wrong shape for a
|
||||
# coordinator trying to understand a child's trajectory (the LAST
|
||||
# message tells the model what the child concluded; the FIRST sets
|
||||
# the brief; the middle is the connective tissue).
|
||||
#
|
||||
# The three-tier degradation pattern matches the ``search`` tool's
|
||||
# Tier-1/Tier-2/Tier-3 ladder at ``session.py:_format_search_results``.
|
||||
# First tier whose serialized size fits the budget wins; the LLM
|
||||
# learns which tier it got via the ``_tier`` field in the response
|
||||
# (no API change to the coordinator tool).
|
||||
#
|
||||
# Budget chosen well under ``tool_truncation`` (typically 256 KB+) so
|
||||
# the head+tail safety net never fires for inspect_workstream — that
|
||||
# strategy silently drops middle messages, which is exactly the
|
||||
# pathology this formatter exists to avoid.
|
||||
|
||||
_INSPECT_OUTPUT_BUDGET: int = 32_768
|
||||
# Per-message head/tail snip when Tier 2 needs to compress content.
|
||||
# Head dominates because the first ~600 chars of an assistant message
|
||||
# usually contains the conclusion / direction; the tail is the
|
||||
# follow-through. Tool results compress similarly: head shows what
|
||||
# the tool was asked / what it found at the top; tail shows the final
|
||||
# state / error suffix.
|
||||
_INSPECT_MSG_CONTENT_HEAD: int = 600
|
||||
_INSPECT_MSG_CONTENT_TAIL: int = 300
|
||||
# Skeleton-tier preview length on the last assistant message. Single
|
||||
# value because the skeleton wants ONE meaningful signal ("what did
|
||||
# the child last say"), not a head/tail snip.
|
||||
_INSPECT_SKELETON_LAST_PREVIEW: int = 400
|
||||
|
||||
# Snip lengths for tool-call ``function.arguments`` strings on
|
||||
# assistant turns. Tighter than content snipping because tool calls
|
||||
# often appear in clusters (10+ per turn for a fan-out) and the
|
||||
# arguments JSON is dense — keep just enough to see what was invoked
|
||||
# and the head of the args structure.
|
||||
_INSPECT_TOOL_ARG_HEAD: int = 300
|
||||
_INSPECT_TOOL_ARG_TAIL: int = 100
|
||||
|
||||
# Bytes ``_snip_head_tail`` reserves for the elision marker itself
|
||||
# (``\n...[N chars elided]...\n``). A text shorter than
|
||||
# ``head + tail + this margin`` passes through unsnipped — snipping
|
||||
# would cost more bytes (the marker) than it saves.
|
||||
_INSPECT_ELISION_MARGIN: int = 64
|
||||
|
||||
# Message-list trim ladder for the compact tier when per-message
|
||||
# content snipping alone doesn't free enough budget. Each rung is
|
||||
# ``(head_count, tail_count)`` — keep the first N + last M messages,
|
||||
# elide the middle as ``{"_omitted": K}``. Tail-weighted because the
|
||||
# last assistant turn carries the load-bearing "what did the child
|
||||
# conclude" signal (same rationale as ``_inspect_skeleton``'s
|
||||
# last-assistant preview). Tried in order; first rung whose
|
||||
# serialized emission fits the budget wins. Mirrors the per-file
|
||||
# sample ladder in ``_format_search_results`` at session.py:254.
|
||||
_INSPECT_LIST_TRIM_LADDER: tuple[tuple[int, int], ...] = ((20, 30), (10, 20), (5, 10))
|
||||
|
||||
|
||||
def _snip_head_tail(text: str, head: int, tail: int) -> str:
|
||||
"""Head/tail snip with elision marker; passthrough when shorter than threshold."""
|
||||
if not isinstance(text, str) or len(text) <= head + tail + _INSPECT_ELISION_MARGIN:
|
||||
return text
|
||||
elided = len(text) - head - tail
|
||||
return text[:head] + f"\n...[{elided} chars elided]...\n" + text[-tail:]
|
||||
|
||||
|
||||
def _compact_tool_calls(tool_calls: Any) -> Any:
|
||||
"""Snip ``function.arguments`` on each tool-call entry; keep ``id`` and
|
||||
``function.name`` verbatim.
|
||||
|
||||
OpenAI shape: ``[{"id": ..., "type": "function", "function":
|
||||
{"name": ..., "arguments": "<json-string>"}}, ...]``. The
|
||||
arguments string is the dominant size term on a fan-out turn that
|
||||
issued many tool calls with multi-KB JSON arguments each;
|
||||
preserving them verbatim re-opens the same size pressure the
|
||||
compact tier is trying to relieve. Non-list / non-dict entries
|
||||
pass through so a future shape change doesn't crash the formatter.
|
||||
"""
|
||||
if not isinstance(tool_calls, list):
|
||||
return tool_calls
|
||||
out: list[Any] = []
|
||||
for call in tool_calls:
|
||||
if not isinstance(call, dict):
|
||||
out.append(call)
|
||||
continue
|
||||
compact_call: dict[str, Any] = {}
|
||||
for k in ("id", "type"):
|
||||
v = call.get(k)
|
||||
if v:
|
||||
compact_call[k] = v
|
||||
func = call.get("function")
|
||||
if isinstance(func, dict):
|
||||
compact_func: dict[str, Any] = {}
|
||||
name = func.get("name")
|
||||
if name:
|
||||
compact_func["name"] = name
|
||||
args = func.get("arguments", "")
|
||||
if args:
|
||||
compact_func["arguments"] = _snip_head_tail(
|
||||
args, _INSPECT_TOOL_ARG_HEAD, _INSPECT_TOOL_ARG_TAIL
|
||||
)
|
||||
compact_call["function"] = compact_func
|
||||
out.append(compact_call)
|
||||
return out
|
||||
|
||||
|
||||
def _compact_message(msg: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Tier-2 per-message projection: keep role + identifier keys, snip content + tool_calls.
|
||||
|
||||
Tool-call linkage is the load-bearing "what happened" signal:
|
||||
``tool_call_id`` on the result side matches an ``id`` in
|
||||
``tool_calls`` on the issuing assistant turn. Stripping
|
||||
``tool_calls`` (the pre-fix shape) left tool results dangling
|
||||
against an invisible call — the audit reader could see "bash
|
||||
returned X" but not "the assistant asked for ``ls /tmp``". The
|
||||
``arguments`` string is the size offender, so we snip it head/tail
|
||||
rather than dropping the call entirely.
|
||||
"""
|
||||
content = msg.get("content", "")
|
||||
snipped = _snip_head_tail(content, _INSPECT_MSG_CONTENT_HEAD, _INSPECT_MSG_CONTENT_TAIL)
|
||||
compact: dict[str, Any] = {"role": msg.get("role"), "content": snipped}
|
||||
# Tool-result linkage (result-side keys).
|
||||
for k in ("tool_name", "tool_call_id", "name"):
|
||||
v = msg.get(k)
|
||||
if v:
|
||||
compact[k] = v
|
||||
# Tool-call request linkage (issuing-side list), snipped per-call.
|
||||
tool_calls = msg.get("tool_calls")
|
||||
if tool_calls:
|
||||
compact["tool_calls"] = _compact_tool_calls(tool_calls)
|
||||
return compact
|
||||
|
||||
|
||||
def _inspect_skeleton(result: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Tier-3 fallback: state + counts + last assistant preview + terminal info.
|
||||
|
||||
Drops every message, keeping only aggregate signal: state, message
|
||||
count, role distribution, verdict count + risk distribution, and a
|
||||
short preview of the most recent assistant turn (the "what did this
|
||||
child last say" signal). Terminal-state fields (``close_reason``,
|
||||
``last_error``) and the ``live`` block pass through unchanged
|
||||
because they're already small and load-bearing.
|
||||
"""
|
||||
messages = result.get("messages") or []
|
||||
verdicts = result.get("verdicts") or []
|
||||
role_counts: dict[str, int] = {}
|
||||
for m in messages:
|
||||
role = m.get("role") if isinstance(m, dict) else None
|
||||
if role:
|
||||
role_counts[role] = role_counts.get(role, 0) + 1
|
||||
verdicts_by_risk: dict[str, int] = {}
|
||||
for v in verdicts:
|
||||
if isinstance(v, dict):
|
||||
risk = v.get("risk_level") or "unknown"
|
||||
verdicts_by_risk[risk] = verdicts_by_risk.get(risk, 0) + 1
|
||||
last_preview = ""
|
||||
for m in reversed(messages):
|
||||
if not isinstance(m, dict) or m.get("role") != "assistant":
|
||||
continue
|
||||
c = m.get("content", "")
|
||||
if isinstance(c, str) and c:
|
||||
last_preview = c[:_INSPECT_SKELETON_LAST_PREVIEW]
|
||||
if len(c) > _INSPECT_SKELETON_LAST_PREVIEW:
|
||||
last_preview += "..."
|
||||
break
|
||||
skeleton: dict[str, Any] = {
|
||||
# Storage row keys verbatim from ``get_workstreams_batch``
|
||||
# (the projection backing ``get_workstream`` → ``inspect()``):
|
||||
# ``ws_id``, ``skill_id``. No fallback to ``id`` / ``skill``
|
||||
# — fail loud on storage column drift rather than silently
|
||||
# emitting null.
|
||||
"ws_id": result["ws_id"],
|
||||
"state": result.get("state"),
|
||||
"title": result.get("title"),
|
||||
"skill": result["skill_id"],
|
||||
"message_count": len(messages),
|
||||
"roles": role_counts,
|
||||
"verdict_count": len(verdicts),
|
||||
"verdicts_by_risk": verdicts_by_risk,
|
||||
"last_assistant_preview": last_preview,
|
||||
"_tier": "skeleton",
|
||||
"_tier_note": (
|
||||
"Output exceeded the inspect_workstream budget at both full and compact "
|
||||
"tiers; skeleton-only. Re-call with a smaller ``message_limit`` to fit "
|
||||
"the compact tier, or read individual messages via the storage admin path."
|
||||
),
|
||||
}
|
||||
for k in ("close_reason", "last_error", "live"):
|
||||
v = result.get(k)
|
||||
if v:
|
||||
skeleton[k] = v
|
||||
return skeleton
|
||||
|
||||
|
||||
def _format_inspect_tiered(result: dict[str, Any], *, budget: int = _INSPECT_OUTPUT_BUDGET) -> str:
|
||||
"""Serialize an ``inspect_workstream`` result with tiered degradation.
|
||||
|
||||
Tier 1 (full): every message verbatim — used when the size fits.
|
||||
Tier 2 (compact): per-message ``{role, head/tail-snipped content,
|
||||
tool linkage, snipped tool_calls.arguments}`` for
|
||||
every message, then a head+tail message-list trim
|
||||
ladder when content snipping alone doesn't free
|
||||
enough budget.
|
||||
Tier 3 (skeleton): no messages — counts + last assistant preview only.
|
||||
|
||||
First emission whose JSON serialization fits ``budget`` wins.
|
||||
``_tier`` appears on every non-error emission so the coordinator
|
||||
LLM (and any audit reader) can see which compression rung the
|
||||
output landed on without inferring from length. Error-shape
|
||||
results (missing or cross-tenant ws_id) bypass tiering entirely —
|
||||
they're already small and the ``error`` key signals the shape.
|
||||
|
||||
The intermediate Tier-2 list-trim rungs exist because content
|
||||
snipping alone fails on workloads where many small messages
|
||||
overflow the budget by sheer count (``message_limit=200`` × a few
|
||||
hundred chars each). In that regime, dropping content-snipping
|
||||
saves zero bytes per message, so without the list-trim ladder
|
||||
Tier-2 produces output strictly larger than Tier-1 (added
|
||||
``_tier_note``) and the formatter fell through to skeleton —
|
||||
losing every message when a head+tail message-list trim would
|
||||
have preserved dozens. Mirrors the per-file sample ladder in
|
||||
``_format_search_results`` (session.py:_SEARCH_TIER2_SAMPLE_LADDER).
|
||||
"""
|
||||
if "error" in result:
|
||||
# Cross-tenant guard / not-found responses — pass through.
|
||||
return json.dumps(result, default=str, separators=(",", ":"))
|
||||
tier1 = {**result, "_tier": "full"}
|
||||
out1 = json.dumps(tier1, default=str, separators=(",", ":"))
|
||||
if len(out1) <= budget:
|
||||
return out1
|
||||
messages = result.get("messages") or []
|
||||
compact_msgs = [_compact_message(m) if isinstance(m, dict) else m for m in messages]
|
||||
tier2_note_full = (
|
||||
"Output exceeded the inspect_workstream budget at the full tier; messages "
|
||||
"are head/tail-snipped at "
|
||||
f"{_INSPECT_MSG_CONTENT_HEAD}/{_INSPECT_MSG_CONTENT_TAIL} chars. Re-call "
|
||||
"with a smaller ``message_limit`` for a tighter tail, or include_provider_"
|
||||
"content=False if it was on."
|
||||
)
|
||||
tier2 = {
|
||||
**result,
|
||||
"messages": compact_msgs,
|
||||
"_tier": "compact",
|
||||
"_tier_note": tier2_note_full,
|
||||
}
|
||||
out2 = json.dumps(tier2, default=str, separators=(",", ":"))
|
||||
if len(out2) <= budget:
|
||||
return out2
|
||||
# Tier-2 list-trim ladder: keep head N + tail M, elide the middle.
|
||||
# Tail-weighted because the recent turns carry the load-bearing
|
||||
# signal ("what did the child conclude") — same reason
|
||||
# ``_inspect_skeleton`` keeps a last-assistant preview rather than
|
||||
# a first-user preview.
|
||||
total = len(compact_msgs)
|
||||
for head_n, tail_n in _INSPECT_LIST_TRIM_LADDER:
|
||||
if head_n + tail_n >= total:
|
||||
# Rung doesn't actually trim — would re-emit Tier-2 verbatim.
|
||||
continue
|
||||
omitted = total - head_n - tail_n
|
||||
trimmed: list[Any] = (
|
||||
compact_msgs[:head_n] + [{"_omitted": omitted}] + compact_msgs[-tail_n:]
|
||||
)
|
||||
tier2_trim_note = (
|
||||
f"Output exceeded the inspect_workstream budget at the compact tier; "
|
||||
f"keeping first {head_n} + last {tail_n} of {total} messages, eliding "
|
||||
f"{omitted} middle messages. Re-call with a smaller ``message_limit`` "
|
||||
"to fit the full compact tier."
|
||||
)
|
||||
tier2_trim = {
|
||||
**result,
|
||||
"messages": trimmed,
|
||||
"_tier": "compact",
|
||||
"_tier_note": tier2_trim_note,
|
||||
}
|
||||
out2_trim = json.dumps(tier2_trim, default=str, separators=(",", ":"))
|
||||
if len(out2_trim) <= budget:
|
||||
return out2_trim
|
||||
skeleton = _inspect_skeleton(result)
|
||||
return json.dumps(skeleton, default=str, separators=(",", ":"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — last-message extraction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Console-side multiplexer for PostgreSQL ``LISTEN``/``NOTIFY`` events.
|
||||
|
||||
Holds a single dedicated listen connection (via :meth:`StorageBackend.listen`),
|
||||
drains it on a listener thread, and fans notifications out to per-channel
|
||||
handlers on a dedicated dispatch thread so a slow handler doesn't back up
|
||||
the connection.
|
||||
|
||||
Consumers register at construction time by passing their channel in
|
||||
:attr:`channels`, then call :meth:`subscribe` to attach a handler.
|
||||
Registering an undeclared channel raises — the construction list is the
|
||||
single source of truth so wire-in is explicit (each future consumer
|
||||
touches the dispatcher construction call site at
|
||||
``turnstone/console/server.py::main`` to add its channel).
|
||||
|
||||
On connection loss the listener wakes its handlers with a synthetic
|
||||
``Notify(channel, payload="reconcile", pid=0)`` so every consumer
|
||||
re-reads the underlying rows; their normal "reconcile on any wake-up"
|
||||
code path covers both real notifications and reconnect recovery
|
||||
identically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
# Backoff (seconds) between reconnect attempts after :class:`NotifyConnectionError`.
|
||||
# Doubles each failure, capped at the max — long enough that a Postgres outage
|
||||
# doesn't burn CPU on reconnect spins, short enough that recovery is fast.
|
||||
_RECONNECT_BACKOFF_INITIAL: float = 1.0
|
||||
_RECONNECT_BACKOFF_MAX: float = 30.0
|
||||
|
||||
# Poll cadence on the listener thread. Short enough that ``stop`` lands
|
||||
# promptly without joining a long-blocked notifies() call; long enough
|
||||
# that we don't burn CPU on empty polls.
|
||||
_LISTENER_POLL_TIMEOUT: float = 1.0
|
||||
|
||||
# Cap on the inter-thread dispatch queue. Drops oldest if a slow handler
|
||||
# falls behind (logs once per drop bucket). Sized larger than the expected
|
||||
# steady-state notification rate (services trigger fires only on
|
||||
# register/restart/deregister — order of hundreds per hour at the 100-node
|
||||
# design ceiling).
|
||||
_DISPATCH_QUEUE_MAX: int = 1024
|
||||
|
||||
|
||||
class NotifyDispatcher:
|
||||
"""Holds the dedicated listen connection and fans events to handlers.
|
||||
|
||||
Lifecycle: construct with the declared channel list, attach
|
||||
handlers via :meth:`subscribe`, then call :meth:`start`. :meth:`stop`
|
||||
closes the connection and joins the worker threads. Idempotent in
|
||||
both directions so console teardown can call stop unconditionally.
|
||||
"""
|
||||
|
||||
def __init__(self, storage: StorageBackend, channels: Iterable[str]) -> None:
|
||||
ch_list = [str(c) for c in channels if c]
|
||||
if not ch_list:
|
||||
msg = "NotifyDispatcher requires at least one declared channel"
|
||||
raise ValueError(msg)
|
||||
self._storage = storage
|
||||
self._channels: list[str] = list(dict.fromkeys(ch_list)) # de-dupe, preserve order
|
||||
self._handlers: dict[str, list[Callable[[Notify], None]]] = {
|
||||
ch: [] for ch in self._channels
|
||||
}
|
||||
self._handlers_lock = threading.Lock()
|
||||
self._lifecycle_lock = threading.Lock()
|
||||
self._started = False
|
||||
self._stopping = threading.Event()
|
||||
self._listener_thread: threading.Thread | None = None
|
||||
self._dispatch_thread: threading.Thread | None = None
|
||||
self._dispatch_queue: queue.Queue[Notify | None] = queue.Queue(maxsize=_DISPATCH_QUEUE_MAX)
|
||||
self._drop_count = 0
|
||||
# Set inside :meth:`_listener_loop` after each successful
|
||||
# ``storage.listen`` open; cleared on disconnect. Callers use
|
||||
# :meth:`wait_until_ready` after :meth:`start` to block until the
|
||||
# listener is actually listening (matters when the next caller
|
||||
# action is a ``notify`` whose delivery requires the LISTEN to
|
||||
# already be in place — e.g. tests, or any startup-path traffic
|
||||
# that should be reactive from the first event).
|
||||
self._listener_ready = threading.Event()
|
||||
|
||||
@property
|
||||
def channels(self) -> list[str]:
|
||||
"""Snapshot copy of declared channels."""
|
||||
return list(self._channels)
|
||||
|
||||
def subscribe(self, channel: str, handler: Callable[[Notify], None]) -> Callable[[], None]:
|
||||
"""Attach ``handler`` to ``channel``; return an unsubscribe callable.
|
||||
|
||||
Safe to call before or after :meth:`start`. Raises if the
|
||||
channel was not declared at construction time — the channel
|
||||
list is fixed so the dispatcher knows up-front which LISTENs
|
||||
to issue (consumers added in follow-up PRs touch the
|
||||
construction call site).
|
||||
"""
|
||||
if channel not in self._handlers:
|
||||
msg = (
|
||||
f"channel {channel!r} not declared at construction; "
|
||||
f"declared channels: {sorted(self._handlers)}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
with self._handlers_lock:
|
||||
self._handlers[channel].append(handler)
|
||||
|
||||
def _unsubscribe() -> None:
|
||||
with self._handlers_lock, contextlib.suppress(ValueError):
|
||||
self._handlers[channel].remove(handler)
|
||||
|
||||
return _unsubscribe
|
||||
|
||||
def start(self) -> None:
|
||||
"""Open the listen stream and start the listener + dispatch threads.
|
||||
|
||||
Idempotent — repeat calls log a debug line and return without
|
||||
spawning a second listener.
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if self._started:
|
||||
log.debug("notify_dispatcher.start_noop_already_started")
|
||||
return
|
||||
self._started = True
|
||||
self._stopping.clear()
|
||||
# Clear ready so a stop/start cycle's wait_until_ready only
|
||||
# returns True after the new listener has actually opened.
|
||||
self._listener_ready.clear()
|
||||
self._listener_thread = threading.Thread(
|
||||
target=self._listener_loop,
|
||||
name="notify-dispatcher-listener",
|
||||
daemon=True,
|
||||
)
|
||||
self._dispatch_thread = threading.Thread(
|
||||
target=self._dispatch_loop,
|
||||
name="notify-dispatcher-dispatch",
|
||||
daemon=True,
|
||||
)
|
||||
self._listener_thread.start()
|
||||
self._dispatch_thread.start()
|
||||
log.info(
|
||||
"notify_dispatcher.started",
|
||||
channels=self._channels,
|
||||
)
|
||||
|
||||
def wait_until_ready(self, timeout: float = 5.0) -> bool:
|
||||
"""Block until the listener has opened its stream, or ``timeout`` elapses.
|
||||
|
||||
Returns ``True`` when the listener is ready (``LISTEN`` issued
|
||||
for every declared channel on PG; subscriber queues registered
|
||||
on SQLite), ``False`` on timeout. Cleared automatically on
|
||||
disconnect — call again after a reconnect to wait for the next
|
||||
successful reopen.
|
||||
|
||||
Doesn't replace :meth:`start` — call ``start()`` first, then
|
||||
``wait_until_ready()`` for the explicit sync point. Production
|
||||
startup typically doesn't need this (the first real event tends
|
||||
to arrive well after the listener is up); tests use it to close
|
||||
the start-vs-notify race window.
|
||||
"""
|
||||
return self._listener_ready.wait(timeout=timeout)
|
||||
|
||||
def stop(self, timeout: float = 5.0) -> None:
|
||||
"""Signal shutdown and join the worker threads.
|
||||
|
||||
Idempotent — safe to call multiple times. Workers exit on the
|
||||
next iteration of their poll loops; :meth:`stop` blocks up to
|
||||
``timeout`` seconds per thread before giving up (the threads are
|
||||
daemons so the process can exit regardless).
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if not self._started:
|
||||
return
|
||||
self._stopping.set()
|
||||
listener = self._listener_thread
|
||||
dispatcher = self._dispatch_thread
|
||||
# Sentinel wakes the dispatch loop out of queue.get().
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(None)
|
||||
if listener is not None:
|
||||
listener.join(timeout=timeout)
|
||||
if dispatcher is not None:
|
||||
dispatcher.join(timeout=timeout)
|
||||
with self._lifecycle_lock:
|
||||
self._listener_thread = None
|
||||
self._dispatch_thread = None
|
||||
self._started = False
|
||||
log.info("notify_dispatcher.stopped")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal threading
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _listener_loop(self) -> None:
|
||||
"""Drain the storage stream onto the dispatch queue, reconnecting on loss.
|
||||
|
||||
After any disconnect — whether surfaced through the stream's
|
||||
:class:`NotifyConnectionError` (post-open ``poll`` failure) or
|
||||
through the generic exception path (``psycopg.connect`` /
|
||||
initial ``LISTEN`` execute failures during reopen, which are
|
||||
NOT wrapped by the stream) — the loop sets a ``reconcile_pending``
|
||||
flag, waits the backoff, then enqueues one synthetic ``reconcile``
|
||||
notify per channel ONLY after the next stream successfully
|
||||
reopens. Handlers see the synthetic notify and re-read the
|
||||
relevant rows on the same code path they use for any real event,
|
||||
closing the missed-notification window regardless of which
|
||||
exception type caused the disconnect.
|
||||
"""
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
reconcile_pending = False
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
with self._storage.listen(self._channels) as stream:
|
||||
log.debug(
|
||||
"notify_dispatcher.stream_open",
|
||||
channels=self._channels,
|
||||
)
|
||||
# Stream is open — reset backoff for the next outage
|
||||
# and flush any pending reconcile so consumers see a
|
||||
# wake-up against a now-live DB.
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
if reconcile_pending:
|
||||
self._synthesize_reconcile()
|
||||
reconcile_pending = False
|
||||
# Signal ``wait_until_ready`` callers that LISTEN is
|
||||
# in place (PG) / subscriber queues are bound
|
||||
# (SQLite). Must come AFTER the synthesize so any
|
||||
# post-reconnect reconcile reaches handlers before
|
||||
# the caller assumes "fresh notifies will deliver".
|
||||
self._listener_ready.set()
|
||||
while not self._stopping.is_set():
|
||||
batch = stream.poll(_LISTENER_POLL_TIMEOUT)
|
||||
for n in batch:
|
||||
self._enqueue(n)
|
||||
except NotifyConnectionError as exc:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.warning(
|
||||
"notify_dispatcher.connection_lost",
|
||||
error=str(exc),
|
||||
backoff_seconds=backoff,
|
||||
)
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
except Exception:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.exception("notify_dispatcher.listener_unexpected_error")
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
log.debug("notify_dispatcher.listener_exiting")
|
||||
|
||||
def _synthesize_reconcile(self) -> None:
|
||||
"""Push one synthetic ``reconcile`` notify per channel on reconnect.
|
||||
|
||||
Reconcile-on-wake is the same logic handlers run for any real
|
||||
notification, so a single synthetic event per channel covers
|
||||
any notifications missed during the connection-loss window.
|
||||
"""
|
||||
for ch in self._channels:
|
||||
self._enqueue(Notify(channel=ch, payload="reconcile", pid=0))
|
||||
|
||||
def _enqueue(self, notify: Notify) -> None:
|
||||
"""Put a notify on the dispatch queue, dropping oldest on overflow."""
|
||||
try:
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
except queue.Full:
|
||||
# Drop oldest to make room — a slow handler shouldn't be able
|
||||
# to silently block the listener thread. Log once per power
|
||||
# of two so a sustained backpressure problem shows up
|
||||
# in logs without flooding.
|
||||
self._drop_count += 1
|
||||
if self._drop_count & (self._drop_count - 1) == 0:
|
||||
log.warning(
|
||||
"notify_dispatcher.dispatch_queue_full_dropping_oldest",
|
||||
drops_total=self._drop_count,
|
||||
channel=notify.channel,
|
||||
)
|
||||
with contextlib.suppress(queue.Empty):
|
||||
self._dispatch_queue.get_nowait()
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
|
||||
def _dispatch_loop(self) -> None:
|
||||
"""Pull notifies off the queue and invoke handlers per channel.
|
||||
|
||||
Notifies queued on the same channel coalesce per dispatch batch:
|
||||
after blocking ``get()`` returns one notify, the loop drains
|
||||
whatever else is already queued and collapses to one
|
||||
``per-channel`` notify before invoking handlers. The payload is
|
||||
signal-only by design (handlers reconcile by re-reading the
|
||||
underlying rows), so N same-channel notifies have the same
|
||||
observable effect as one — coalescing turns an N-node deploy
|
||||
burst into a single ``_discover_nodes`` per channel instead of N.
|
||||
|
||||
Each handler runs under exception suppression so one buggy
|
||||
consumer can't take down the dispatch thread.
|
||||
"""
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
first = self._dispatch_queue.get(timeout=_LISTENER_POLL_TIMEOUT)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if first is None:
|
||||
# Sentinel from :meth:`stop`.
|
||||
return
|
||||
# Coalesce by channel: keep the most recent payload per
|
||||
# channel from this drain batch. Drops a stop sentinel
|
||||
# silently — the next loop iteration will see _stopping set
|
||||
# and exit anyway, so we don't need to re-queue the sentinel.
|
||||
per_channel: dict[str, Notify] = {first.channel: first}
|
||||
stop_seen = False
|
||||
while True:
|
||||
try:
|
||||
nxt = self._dispatch_queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if nxt is None:
|
||||
stop_seen = True
|
||||
continue
|
||||
per_channel[nxt.channel] = nxt
|
||||
for notify in per_channel.values():
|
||||
with self._handlers_lock:
|
||||
handlers = list(self._handlers.get(notify.channel, ()))
|
||||
for handler in handlers:
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
handler(notify)
|
||||
except Exception:
|
||||
log.exception(
|
||||
"notify_dispatcher.handler_failed",
|
||||
channel=notify.channel,
|
||||
)
|
||||
else:
|
||||
elapsed_ms = (time.monotonic() - t0) * 1000.0
|
||||
if elapsed_ms > 100.0:
|
||||
log.debug(
|
||||
"notify_dispatcher.handler_slow",
|
||||
channel=notify.channel,
|
||||
elapsed_ms=round(elapsed_ms, 1),
|
||||
)
|
||||
if stop_seen:
|
||||
return
|
||||
log.debug("notify_dispatcher.dispatch_exiting")
|
||||
@@ -319,6 +319,12 @@ class TaskScheduler:
|
||||
user_id=task.get("created_by", ""),
|
||||
skill=task.get("skill", ""),
|
||||
notify_targets=task.get("notify_targets", "[]"),
|
||||
# Mark the resulting ChatSession as non-interactive-for-
|
||||
# consent so OAuth-MCP errors get persisted to
|
||||
# ``mcp_pending_consent`` for later dashboard surfacing,
|
||||
# rather than relying on an in-flight SSE redirect the
|
||||
# absent user can't complete.
|
||||
client_type="scheduled",
|
||||
)
|
||||
ws_id = resp.ws_id
|
||||
except Exception:
|
||||
|
||||
+384
-18
@@ -41,6 +41,7 @@ from starlette.staticfiles import StaticFiles
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
from turnstone.api.docs import make_docs_handler, make_openapi_handler
|
||||
from turnstone.console.collector import ClusterCollector
|
||||
from turnstone.console.coordinator_alias import resolve_coordinator_alias
|
||||
from turnstone.console.coordinator_client import load_task_envelope
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
@@ -1651,6 +1652,27 @@ async def mcp_oauth_revoke_connection(request: Request) -> Response:
|
||||
return await handle_mcp_oauth_revoke_connection(request)
|
||||
|
||||
|
||||
async def mcp_oauth_list_pending(request: Request) -> Response:
|
||||
"""GET /v1/api/mcp/oauth/pending — list deferred-consent records (Phase 9)."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_list_pending
|
||||
|
||||
return await handle_mcp_oauth_list_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending/{server_name} — dismiss a deferred-consent record."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_all_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending — bulk-dismiss deferred-consent records."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_all_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_all_pending(request)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Route handlers — available models (lightweight, no admin permission)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1671,20 +1693,55 @@ async def list_available_models(request: Request) -> JSONResponse:
|
||||
# Include effective defaults for clients (web UI, channel gateway).
|
||||
default_alias = ""
|
||||
channel_default_alias = ""
|
||||
coordinator_default_alias = ""
|
||||
judge_default_alias = ""
|
||||
cs = getattr(request.app.state, "config_store", None)
|
||||
if cs is not None:
|
||||
default_alias = cs.get("model.default_alias") or ""
|
||||
channel_default_alias = cs.get("channels.default_model_alias") or ""
|
||||
judge_default_alias = (cs.get("judge.model") or "").strip()
|
||||
enabled_aliases = {r["alias"] for r in rows}
|
||||
if default_alias and default_alias not in enabled_aliases:
|
||||
default_alias = ""
|
||||
if channel_default_alias and channel_default_alias not in enabled_aliases:
|
||||
channel_default_alias = ""
|
||||
# Coordinator default walks the standard three-tier chain (see
|
||||
# :func:`turnstone.console.coordinator_alias.resolve_coordinator_alias`).
|
||||
# The placeholder restricts every tier to enabled DB rows so the home
|
||||
# composer doesn't advertise a model the workstream-creation picker
|
||||
# can't actually offer — the session factory uses the same chain
|
||||
# without that filter so explicit operator pins surface as 503 at
|
||||
# ``registry.resolve`` instead of being silently swapped out.
|
||||
coord_registry = getattr(request.app.state, "coord_registry", None)
|
||||
if cs is not None and coord_registry is not None:
|
||||
coordinator_default_alias = resolve_coordinator_alias(
|
||||
explicit=cs.get("coordinator.model_alias"),
|
||||
config_store=cs,
|
||||
registry=coord_registry,
|
||||
alias_filter=lambda a: a in enabled_aliases,
|
||||
)
|
||||
elif coord_registry is not None:
|
||||
# ConfigStore failed lifespan but coord_registry is still bound —
|
||||
# fall through to ``registry.default`` (filtered) so the home
|
||||
# composer isn't blank. Mirrors the helper's tier 3 with the
|
||||
# placeholder's enabled-rows filter applied.
|
||||
registry_default = getattr(coord_registry, "default", "") or ""
|
||||
if registry_default in enabled_aliases:
|
||||
coordinator_default_alias = registry_default
|
||||
# Judge falls back to the resolved coordinator alias when
|
||||
# ``judge.model`` is empty *or* not a registered alias — judge.model
|
||||
# is alias-only (matches IntentJudge.__init__), so an unknown value is
|
||||
# operator misconfiguration that the judge itself silently inherits
|
||||
# the session model on.
|
||||
if not judge_default_alias or judge_default_alias not in enabled_aliases:
|
||||
judge_default_alias = coordinator_default_alias
|
||||
return JSONResponse(
|
||||
{
|
||||
"models": models,
|
||||
"default_alias": default_alias,
|
||||
"channel_default_alias": channel_default_alias,
|
||||
"coordinator_default_alias": coordinator_default_alias,
|
||||
"judge_default_alias": judge_default_alias,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -2600,10 +2657,67 @@ async def proxy_shared_static(request: Request) -> Response:
|
||||
return JSONResponse({"error": "Node unreachable"}, status_code=502)
|
||||
|
||||
|
||||
# Auth endpoints the console handles locally instead of forwarding to
|
||||
# the upstream node. Single source of truth for the dispatch table and
|
||||
# the path set used by both the 405 short-circuit and the
|
||||
# test parametrize, so a new entry can't drift between code and tests.
|
||||
#
|
||||
# Values are handler NAMES (strings) rather than function references.
|
||||
# ``proxy_api`` resolves them via ``globals()`` at call time so test
|
||||
# ``patch("turnstone.console.server.auth_login")`` is observed; a dict
|
||||
# of refs would capture the original function at module load.
|
||||
_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str] = {
|
||||
("POST", "auth/login"): "auth_login",
|
||||
("POST", "auth/logout"): "auth_logout",
|
||||
("POST", "auth/setup"): "auth_setup",
|
||||
("POST", "auth/refresh"): "auth_refresh",
|
||||
("GET", "auth/status"): "auth_status",
|
||||
("GET", "auth/whoami"): "auth_whoami",
|
||||
("GET", "auth/oidc/authorize"): "oidc_authorize",
|
||||
("GET", "auth/oidc/callback"): "oidc_callback",
|
||||
}
|
||||
_PROXY_AUTH_LOCAL_PATHS: frozenset[str] = frozenset(path for _, path in _PROXY_AUTH_LOCAL_HANDLERS)
|
||||
|
||||
|
||||
async def proxy_api(request: Request) -> Response:
|
||||
"""Proxy API requests to target node. Detects SSE vs regular."""
|
||||
"""Proxy API requests to target node, with two exceptions handled in-process:
|
||||
|
||||
1. ``auth/*`` endpoints in ``_PROXY_AUTH_LOCAL_HANDLERS`` are
|
||||
dispatched to the console's own auth handlers so JWTs carry
|
||||
``JWT_AUD_CONSOLE`` and Set-Cookie lands on the console origin.
|
||||
Forwarding upstream would mint ``JWT_AUD_SERVER`` tokens that the
|
||||
console's ``AuthMiddleware`` rejects on the next proxied call,
|
||||
locking the user out of the proxied UI — and ``_proxy_post``
|
||||
drops Set-Cookie when forwarding anyway. ``refresh`` and
|
||||
``whoami`` are intentionally NOT in ``PUBLIC_PATHS`` (caller must
|
||||
still hold a valid cookie); local dispatch is about cookie-origin
|
||||
and audience, not public access.
|
||||
2. SSE endpoints (per-ws + global events) stream via ``_proxy_sse``.
|
||||
|
||||
Everything else is forwarded to ``server_url`` via ``_proxy_post`` /
|
||||
``_proxy_get``.
|
||||
"""
|
||||
node_id = request.path_params["node_id"]
|
||||
path = request.path_params["path"]
|
||||
|
||||
handler_name = _PROXY_AUTH_LOCAL_HANDLERS.get((request.method, path))
|
||||
if handler_name is not None:
|
||||
# Resolve via ``globals()`` so ``patch("...auth_login")`` in
|
||||
# tests is observed. A direct function-ref dict would have
|
||||
# captured the original at module load.
|
||||
handler = globals()[handler_name]
|
||||
return await handler(request) # type: ignore[no-any-return]
|
||||
# Path matches a local-dispatch auth endpoint but the method does not:
|
||||
# short-circuit with 405 so the request can't fall through to
|
||||
# ``_proxy_post`` / ``_proxy_get`` and reach the upstream
|
||||
# authenticated as the console's service token
|
||||
# (``_proxy_auth_headers`` falls back to the service identity when
|
||||
# there's no user context). Harmless today because every upstream
|
||||
# auth route 405s on the wrong method too, but kept tight so a
|
||||
# future upstream patch can't widen the surface by accident.
|
||||
if path in _PROXY_AUTH_LOCAL_PATHS:
|
||||
return JSONResponse({"error": "Method not allowed"}, status_code=405)
|
||||
|
||||
server_url = _get_server_url(request, node_id)
|
||||
if not server_url:
|
||||
return JSONResponse({"error": "Node not found"}, status_code=404)
|
||||
@@ -4104,6 +4218,7 @@ def _coord_idle_cleanup_thread(
|
||||
mgr: SessionManager,
|
||||
timeout_sec: float,
|
||||
stop_event: threading.Event | None = None,
|
||||
min_sweep_interval: float = 5.0,
|
||||
) -> None:
|
||||
"""Periodically reap idle + DB-orphan coordinator workstreams.
|
||||
|
||||
@@ -4114,7 +4229,7 @@ def _coord_idle_cleanup_thread(
|
||||
and which aren't currently loaded. The latter pass catches coords left
|
||||
behind by prior console process incarnations.
|
||||
|
||||
Runs an initial sweep BEFORE the first sleep so cold-start orphans are
|
||||
Runs an initial sweep BEFORE the first wait so cold-start orphans are
|
||||
reaped immediately rather than waiting one ``check_every`` interval (~30
|
||||
min on default 2h timeout). This intentionally diverges from the regular
|
||||
server pattern, which has no initial sweep — the regular server runs
|
||||
@@ -4122,26 +4237,90 @@ def _coord_idle_cleanup_thread(
|
||||
is a small fixed-size cache where orphans dominate the row count after
|
||||
a cold boot.
|
||||
|
||||
Wait shape: subscribes a callback to ``mgr._state_subscribers`` that
|
||||
sets a ``tick_now`` event; the loop blocks on ``tick_now.wait(check_every)``
|
||||
so any workstream state-change wakes the sweeper without waiting a
|
||||
full check interval, AND the timeout still fires the periodic sweep
|
||||
even when no activity happens (catching the DB-orphan-only case).
|
||||
Net: blocked most of the time instead of repeating storage scans.
|
||||
|
||||
``min_sweep_interval`` is the hard floor between successive
|
||||
``close_idle`` calls (default 5 s) — without it, sustained
|
||||
state-change activity (each turn typically fires
|
||||
thinking/running/attention/idle on the coord SessionManager) would
|
||||
cause every ``tick_now.set`` mid-sweep to leave the next ``wait``
|
||||
returning immediately, and the loop would tight-spin ``close_idle``
|
||||
at the rate of its own DB latency (~20-50 calls/sec). The floor
|
||||
bounds DB-call traffic at ``1 / min_sweep_interval`` per second
|
||||
under any external activity while still letting a quiet system
|
||||
fire on every state-change wake-up. Tests inject 0.0 to keep the
|
||||
suite fast.
|
||||
|
||||
Default 5 s is a 6x improvement on the pre-refactor fixed 30 s
|
||||
cadence while bounding DB-call traffic at ~0.2 calls/sec under
|
||||
sustained activity — an order of magnitude below ``close_idle``'s
|
||||
DB-latency budget, but tight enough that idle-row reaping still
|
||||
feels prompt to a human watching the sidebar. Tunable post-merge
|
||||
if profiling shows close_idle latency dominates the cadence.
|
||||
|
||||
``stop_event`` is for tests — when set, the thread exits cleanly after
|
||||
the next loop check. Production callers pass ``None`` (the daemon is
|
||||
process-lifetime).
|
||||
"""
|
||||
check_every = min(300.0, timeout_sec / 4)
|
||||
# Initial sweep — runs once before entering the sleep loop.
|
||||
tick_now = threading.Event()
|
||||
|
||||
def _on_state_change(_ws_id: str, _state: Any) -> None:
|
||||
# Any workstream state-change resets the idle clock for that
|
||||
# ws AND may make a different ws newly-eligible (close-idle
|
||||
# pass 2 evaluates DB rows by timestamp). Cheap signal, full
|
||||
# re-evaluation deferred to the next loop iteration.
|
||||
tick_now.set()
|
||||
|
||||
mgr.subscribe_to_state(_on_state_change)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
time.sleep(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Initial sweep — runs once before entering the wait loop.
|
||||
# ``tick_now`` is intentionally not cleared here: any
|
||||
# state-change event that arrives between subscribe and the
|
||||
# first ``wait`` should fire close_idle immediately, not be
|
||||
# discarded.
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
tick_now.wait(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Clear BEFORE the cadence floor so any state-change event
|
||||
# arriving during the cooldown (or during the close_idle
|
||||
# below) leaves ``tick_now`` set — the next loop iteration
|
||||
# then re-enters ``wait`` already-set and re-evaluates
|
||||
# promptly. close_idle is idempotent so a spurious extra
|
||||
# tick is just one redundant scan.
|
||||
tick_now.clear()
|
||||
# Cadence floor — see docstring for the tight-spin
|
||||
# hazard rationale. Cooldown uses ``stop_event.wait``
|
||||
# (not ``time.sleep``) so the test stop hook still
|
||||
# terminates promptly during the cooldown window.
|
||||
since_last = time.monotonic() - last_sweep_at
|
||||
if since_last < min_sweep_interval:
|
||||
gap = min_sweep_interval - since_last
|
||||
if stop_event is not None:
|
||||
if stop_event.wait(gap):
|
||||
return
|
||||
else:
|
||||
time.sleep(gap)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
finally:
|
||||
mgr.unsubscribe_from_state(_on_state_change)
|
||||
|
||||
|
||||
# Guards concurrent attempts to bootstrap the coord subsystem from the
|
||||
@@ -4203,12 +4382,22 @@ def _bootstrap_coord_subsystem(
|
||||
def _token_factory() -> str:
|
||||
return tm.token
|
||||
|
||||
# ``coord_adapter`` is bound later in this same
|
||||
# ``_bootstrap_coord_subsystem`` call, after the adapter and
|
||||
# manager are constructed but before any session is created
|
||||
# — so this factory is *defined* before the adapter exists but
|
||||
# only ever *called* after it does. The free-variable lookup
|
||||
# at call time resolves to the adapter built in this same
|
||||
# bootstrap pass, giving the client a handle to the in-process
|
||||
# wakeup bus the dispatch sink notifies on every child
|
||||
# state-change event.
|
||||
return CoordinatorClient(
|
||||
console_base_url=console_bind_url,
|
||||
storage=storage,
|
||||
token_factory=_token_factory,
|
||||
coord_ws_id=ws_id,
|
||||
user_id=user_id,
|
||||
child_event_bus=coord_adapter.child_event_bus,
|
||||
)
|
||||
|
||||
# Pre-compute config-derived integers BEFORE any thread starts so a
|
||||
@@ -4734,6 +4923,12 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
|
||||
|
||||
await close_oidc_state(app.state)
|
||||
app.state.collector.stop()
|
||||
# Stop the dispatcher after the collector — collector.stop() drops its
|
||||
# subscription, so the dispatcher's dispatch thread won't fire into a
|
||||
# half-torn-down collector during shutdown.
|
||||
notify_dispatcher = getattr(app.state, "notify_dispatcher", None)
|
||||
if notify_dispatcher is not None:
|
||||
notify_dispatcher.stop()
|
||||
audit_exec_shutdown = getattr(app.state, "audit_executor", None)
|
||||
if audit_exec_shutdown is not None:
|
||||
_set_audit_executor(None)
|
||||
@@ -7884,6 +8079,7 @@ _MODEL_AFFECTING_SETTING_KEYS: frozenset[str] = frozenset(
|
||||
"coordinator.model_alias",
|
||||
"coordinator.reasoning_effort",
|
||||
"judge.model",
|
||||
"channels.default_model_alias",
|
||||
}
|
||||
)
|
||||
|
||||
@@ -8643,10 +8839,19 @@ def _mask_mcp_secrets(server: dict[str, Any], reveal: bool = False) -> dict[str,
|
||||
def _mcp_server_to_detail(
|
||||
server: dict[str, Any],
|
||||
node_statuses: dict[str, dict[str, Any]] | None = None,
|
||||
consented_users_count: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Convert a storage dict to a McpServerDetail-shaped dict."""
|
||||
"""Convert a storage dict to a McpServerDetail-shaped dict.
|
||||
|
||||
*consented_users_count* is the Phase 9 admin pill data — distinct
|
||||
non-expired tokens issued for this ``(server_name)``. Omitted
|
||||
(``None``) when the row's ``auth_type`` is not ``oauth_user``, so
|
||||
static / none rows don't carry an irrelevant ``0``.
|
||||
"""
|
||||
d = dict(server)
|
||||
d["status"] = node_statuses or {}
|
||||
if consented_users_count is not None:
|
||||
d["consented_users_count"] = consented_users_count
|
||||
return d
|
||||
|
||||
|
||||
@@ -8697,8 +8902,31 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
|
||||
reveal = str(request.query_params.get("reveal", "")).lower() in ("true", "1")
|
||||
servers = storage.list_mcp_servers()
|
||||
|
||||
# Collect live status from all nodes
|
||||
node_statuses = await _collect_mcp_status(request)
|
||||
# Phase 9: bulk-aggregate consented-users-count across all oauth_user
|
||||
# rows in a single GROUP BY query (rather than N per-row sync DB
|
||||
# round-trips inside this async handler). Run in parallel with the
|
||||
# cross-node HTTP status fan-out below — neither has a data
|
||||
# dependency on the other, so awaiting them sequentially would
|
||||
# stack the DB latency on top of the fan-out latency. Skipped
|
||||
# entirely when no row is oauth_user so static-only installs
|
||||
# exercise zero new storage queries.
|
||||
has_oauth_user = any(s.get("auth_type") == "oauth_user" for s in servers)
|
||||
status_task: asyncio.Task[dict[str, dict[str, dict[str, Any]]]] = asyncio.create_task(
|
||||
_collect_mcp_status(request)
|
||||
)
|
||||
count_task: asyncio.Task[dict[str, int]] | None = (
|
||||
asyncio.create_task(asyncio.to_thread(storage.count_mcp_consented_users_grouped_by_server))
|
||||
if has_oauth_user
|
||||
else None
|
||||
)
|
||||
|
||||
node_statuses = await status_task
|
||||
consent_counts: dict[str, int] = {}
|
||||
if count_task is not None:
|
||||
try:
|
||||
consent_counts = await count_task
|
||||
except Exception:
|
||||
log.debug("admin.mcp_consented_users_bulk_count_failed", exc_info=True)
|
||||
|
||||
db_names: set[str] = set()
|
||||
result = []
|
||||
@@ -8710,8 +8938,14 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
|
||||
status = node_servers.get(s["name"])
|
||||
if status:
|
||||
per_node[node_id] = status
|
||||
# Phase 9: surface the consented-users-count pill for
|
||||
# oauth_user rows. Aggregate was pre-computed above with a
|
||||
# single bulk GROUP BY query; we just look up here.
|
||||
consent_count: int | None = None
|
||||
if s.get("auth_type") == "oauth_user":
|
||||
consent_count = consent_counts.get(s["name"], 0)
|
||||
s = _mask_mcp_secrets(s, reveal)
|
||||
result.append(_mcp_server_to_detail(s, per_node))
|
||||
result.append(_mcp_server_to_detail(s, per_node, consent_count))
|
||||
|
||||
# Merge config-sourced servers visible on nodes but not in DB
|
||||
config_names: set[str] = set()
|
||||
@@ -9352,6 +9586,92 @@ async def admin_mcp_reconnect_one(request: Request) -> JSONResponse:
|
||||
return await _admin_mcp_action(request, "reconnect")
|
||||
|
||||
|
||||
async def admin_mcp_bulk_revoke(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/admin/mcp-servers/{name}/bulk-revoke — clear every user's token (Phase 9).
|
||||
|
||||
Admin-side counterpart to the per-user
|
||||
``DELETE /v1/api/mcp/oauth/connections/{server_name}`` revoke that
|
||||
shipped in Phase 8. Used to drop orphaned tokens after an
|
||||
``auth_type`` transition (oauth_user → static) or after rotating
|
||||
the configured OAuth client.
|
||||
|
||||
Authoritative local delete via
|
||||
:meth:`StorageBackend.delete_mcp_oauth_rows_by_server_name` —
|
||||
purges both ``mcp_user_tokens`` and ``mcp_oauth_pending`` rows for
|
||||
the named server. Upstream RFC 7009 revoke is intentionally NOT
|
||||
attempted in bulk (would require per-row decrypt + N upstream HTTP
|
||||
calls); operators who need upstream cleanup should use the per-
|
||||
user revoke endpoint or let tokens expire naturally. The audit
|
||||
detail records ``upstream_revoke_outcome="bulk_admin_no_upstream"``
|
||||
so the deferral is visible.
|
||||
|
||||
Pool eviction is NOT performed by this handler. Per-user revoke
|
||||
has a per-(user, server) eviction primitive
|
||||
(``MCPClientManager.evict_user_session``); bulk-revoke would need a
|
||||
per-server iteration over consented users that no current primitive
|
||||
supports. Stale in-flight sessions surface as a per-user 401 on
|
||||
the next dispatch, which refreshes through the now-empty token row
|
||||
and emits ``mcp_consent_required`` — the documented v1 fallback.
|
||||
See :func:`turnstone.core.mcp_client._dispatch_pool` retry path.
|
||||
"""
|
||||
from turnstone.core.audit import record_audit
|
||||
from turnstone.core.auth import require_permission
|
||||
from turnstone.core.web_helpers import require_storage_or_503
|
||||
|
||||
storage, err = require_storage_or_503(request)
|
||||
if err:
|
||||
return err
|
||||
err = require_permission(request, "admin.mcp")
|
||||
if err:
|
||||
return err
|
||||
|
||||
name = request.path_params.get("name", "").strip()
|
||||
if not name or "__" in name:
|
||||
return JSONResponse({"error": "invalid server name"}, status_code=400)
|
||||
|
||||
existing = storage.get_mcp_server_by_name(name)
|
||||
if existing is None:
|
||||
return JSONResponse({"error": "No such server"}, status_code=404)
|
||||
if existing.get("auth_type") != "oauth_user":
|
||||
return JSONResponse(
|
||||
{"error": "bulk-revoke is only valid for auth_type=oauth_user servers"},
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
target_id = existing.get("server_id", name)
|
||||
consented_before = 0
|
||||
try:
|
||||
consented_before = storage.count_mcp_consented_users_by_server(name)
|
||||
except Exception:
|
||||
log.debug("admin.mcp_bulk_revoke_pre_count_failed server=%s", name, exc_info=True)
|
||||
|
||||
deleted = storage.delete_mcp_oauth_rows_by_server_name(name)
|
||||
|
||||
audit_uid, ip = _audit_context(request)
|
||||
record_audit(
|
||||
storage,
|
||||
audit_uid,
|
||||
"mcp_server.oauth.bulk_revoked",
|
||||
"mcp_server",
|
||||
target_id,
|
||||
{
|
||||
"name": name,
|
||||
"rows_deleted": deleted,
|
||||
"consented_users_before": consented_before,
|
||||
"upstream_revoke_outcome": "bulk_admin_no_upstream",
|
||||
},
|
||||
ip,
|
||||
)
|
||||
|
||||
return JSONResponse(
|
||||
{
|
||||
"status": "ok",
|
||||
"rows_deleted": deleted,
|
||||
"consented_users_before": consented_before,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
async def admin_import_mcp_config(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/admin/mcp-servers/import — import from pasted JSON config."""
|
||||
import uuid
|
||||
@@ -9904,6 +10224,9 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
|
||||
if not reasoning_effort:
|
||||
reasoning_effort = None
|
||||
|
||||
surface_persisted_reasoning = bool(body.get("surface_persisted_reasoning", True))
|
||||
replay_reasoning_to_model = bool(body.get("replay_reasoning_to_model", False))
|
||||
|
||||
storage.create_model_definition(
|
||||
definition_id=definition_id,
|
||||
alias=alias,
|
||||
@@ -9918,6 +10241,8 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
reasoning_effort=reasoning_effort,
|
||||
surface_persisted_reasoning=surface_persisted_reasoning,
|
||||
replay_reasoning_to_model=replay_reasoning_to_model,
|
||||
)
|
||||
|
||||
record_audit(
|
||||
@@ -10079,6 +10404,10 @@ async def admin_update_model_definition(request: Request) -> JSONResponse:
|
||||
)
|
||||
else:
|
||||
updates["reasoning_effort"] = re_val
|
||||
if "surface_persisted_reasoning" in body:
|
||||
updates["surface_persisted_reasoning"] = bool(body["surface_persisted_reasoning"])
|
||||
if "replay_reasoning_to_model" in body:
|
||||
updates["replay_reasoning_to_model"] = bool(body["replay_reasoning_to_model"])
|
||||
|
||||
if updates:
|
||||
storage.update_model_definition(definition_id, **updates)
|
||||
@@ -11836,6 +12165,7 @@ def create_app(
|
||||
console_url: str = "",
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: Any = None,
|
||||
) -> Starlette:
|
||||
"""Build the Starlette ASGI application for the console dashboard."""
|
||||
_spec = build_console_spec()
|
||||
@@ -12062,6 +12392,17 @@ def create_app(
|
||||
mcp_oauth_revoke_connection,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/mcp/oauth/pending", mcp_oauth_list_pending),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending",
|
||||
mcp_oauth_clear_all_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending/{server_name}",
|
||||
mcp_oauth_clear_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/admin/users", admin_list_users),
|
||||
Route("/api/admin/users", admin_create_user, methods=["POST"]),
|
||||
Route("/api/admin/users/{user_id}", admin_delete_user, methods=["DELETE"]),
|
||||
@@ -12247,6 +12588,11 @@ def create_app(
|
||||
admin_mcp_reconnect_one,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{name}/bulk-revoke",
|
||||
admin_mcp_bulk_revoke,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{server_id}",
|
||||
admin_get_mcp_server,
|
||||
@@ -12462,6 +12808,7 @@ def create_app(
|
||||
lifespan=_lifespan,
|
||||
)
|
||||
app.state.collector = collector
|
||||
app.state.notify_dispatcher = notify_dispatcher
|
||||
app.state.jwt_secret = jwt_secret
|
||||
app.state.auth_storage = auth_storage
|
||||
app.state.proxy_token_mgr = proxy_token_mgr
|
||||
@@ -12557,7 +12904,7 @@ def main() -> None:
|
||||
from turnstone.core.config import add_config_arg, apply_config
|
||||
|
||||
add_config_arg(parser)
|
||||
apply_config(parser, ["console", "auth"])
|
||||
apply_config(parser, ["console", "auth", "database"])
|
||||
args = parser.parse_args()
|
||||
|
||||
from turnstone.core.log import configure_logging_from_args
|
||||
@@ -12576,6 +12923,13 @@ def main() -> None:
|
||||
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
db_url = os.environ.get("TURNSTONE_DB_URL", "")
|
||||
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
|
||||
# Optional dedicated LISTEN URL — config.toml ``[database] listen_url``
|
||||
# (lifted onto args by ``apply_config``) wins over env, and an empty
|
||||
# value falls through to the main DB URL inside the storage layer.
|
||||
# Only used by the ``NotifyDispatcher``; ignored on SQLite.
|
||||
db_listen_url = getattr(args, "db_listen_url", None) or os.environ.get(
|
||||
"TURNSTONE_DB_LISTEN_URL", ""
|
||||
)
|
||||
auth_storage = init_storage(
|
||||
db_backend,
|
||||
path=db_path,
|
||||
@@ -12584,6 +12938,7 @@ def main() -> None:
|
||||
sslrootcert=os.environ.get("TURNSTONE_DB_SSLROOTCERT", ""),
|
||||
sslcert=os.environ.get("TURNSTONE_DB_SSLCERT", ""),
|
||||
sslkey=os.environ.get("TURNSTONE_DB_SSLKEY", ""),
|
||||
listen_url=db_listen_url,
|
||||
)
|
||||
except Exception:
|
||||
log.info("Console storage not available — admin API disabled, JWT-only auth")
|
||||
@@ -12613,11 +12968,21 @@ def main() -> None:
|
||||
router = ConsoleRouter(storage=auth_storage)
|
||||
console_metrics = ConsoleMetrics()
|
||||
|
||||
# NotifyDispatcher multiplexes the dedicated LISTEN connection for all
|
||||
# console-side consumers. Currently one channel: ``services`` for
|
||||
# reactive node discovery. Followup PRs (ConfigStore live reload,
|
||||
# scheduler immediate dispatch) add additional channels here.
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
notify_dispatcher = NotifyDispatcher(auth_storage, channels=["services"])
|
||||
notify_dispatcher.start()
|
||||
|
||||
collector = ClusterCollector(
|
||||
storage=auth_storage,
|
||||
token_manager=collector_token_mgr,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
collector.start()
|
||||
|
||||
@@ -12709,6 +13074,7 @@ def main() -> None:
|
||||
console_url=console_url,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
|
||||
log.info("Console starting on %s", console_url)
|
||||
|
||||
@@ -21,6 +21,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.console.coordinator_alias import resolve_coordinator_alias
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
@@ -96,18 +97,19 @@ def build_console_session_factory(
|
||||
f"console session factory only supports kind=COORDINATOR, got {kind!r}"
|
||||
)
|
||||
|
||||
# Resolve coordinator.model_alias from settings if caller didn't
|
||||
# override. Unset ``coordinator.model_alias`` falls back to the
|
||||
# model registry's default alias — operators get a working
|
||||
# coordinator on a freshly-provisioned console without an extra
|
||||
# manual setting. Resolve to the CONCRETE alias name
|
||||
# (``registry.default``) rather than passing None downstream:
|
||||
# ``ChatSession.__init__`` reads ``registry.get_provider(alias)``
|
||||
# to pick the right provider class, and passing None makes it
|
||||
# fall through to a generic OpenAI-compat provider — which
|
||||
# mismatches when the default is Anthropic/Google-backed.
|
||||
explicit_alias = model_alias or (config_store.get("coordinator.model_alias") or "").strip()
|
||||
effective_alias = explicit_alias or registry.default
|
||||
# Resolve to the CONCRETE alias name rather than passing None
|
||||
# downstream: ``ChatSession.__init__`` reads
|
||||
# ``registry.get_provider(alias)`` to pick the right provider
|
||||
# class, and passing None makes it fall through to a generic
|
||||
# OpenAI-compat provider — which mismatches when the default is
|
||||
# Anthropic/Google-backed. See
|
||||
# :func:`turnstone.console.coordinator_alias.resolve_coordinator_alias`
|
||||
# for the three-tier chain shared with the placeholder API.
|
||||
effective_alias = resolve_coordinator_alias(
|
||||
explicit=model_alias,
|
||||
config_store=config_store,
|
||||
registry=registry,
|
||||
)
|
||||
|
||||
r_client, r_model, r_cfg = registry.resolve(effective_alias)
|
||||
|
||||
|
||||
@@ -1142,12 +1142,54 @@ function _populateScheduleSelect(selectId, url, labelKey, valueKey, opts) {
|
||||
sel.appendChild(opt);
|
||||
});
|
||||
if (opts && opts.selected) sel.value = opts.selected;
|
||||
// Caller hook for placeholder annotation / other post-load tweaks.
|
||||
// Used by the schedule modals to rewrite the bare "Default model"
|
||||
// placeholder with the resolved alias so the label matches the
|
||||
// home composer (see app.js _populateHomeModelDropdowns).
|
||||
if (opts && typeof opts.afterPopulate === "function") {
|
||||
try {
|
||||
opts.afterPopulate(sel, data, items);
|
||||
} catch (_e) {
|
||||
/* hook errors must not break the dropdown */
|
||||
}
|
||||
}
|
||||
})
|
||||
.catch(function () {
|
||||
/* dropdown stays with placeholder or temporary option */
|
||||
});
|
||||
}
|
||||
|
||||
// Update the schedule-model placeholder option (first <option>) to
|
||||
// "Default — alias (model)" using /v1/api/models's resolved
|
||||
// default_alias, mirroring the home composer. Schedules don't carry
|
||||
// a coordinator/judge split, so they consume the workstream-creation
|
||||
// default rather than coordinator_default_alias / judge_default_alias.
|
||||
// Em-dash separator (rather than nested parens) keeps the alias's
|
||||
// "(model)" suffix legible.
|
||||
function _decorateScheduleModelPlaceholder(sel, data) {
|
||||
if (!sel || sel.options.length === 0) return;
|
||||
var alias = (data && data.default_alias) || "";
|
||||
if (!alias) return;
|
||||
var match = null;
|
||||
var models = (data && data.models) || [];
|
||||
for (var i = 0; i < models.length; i++) {
|
||||
if (models[i].alias === alias) {
|
||||
match = models[i];
|
||||
break;
|
||||
}
|
||||
}
|
||||
var label;
|
||||
if (match) {
|
||||
label =
|
||||
match.alias === match.model
|
||||
? match.alias
|
||||
: match.alias + " (" + match.model + ")";
|
||||
} else {
|
||||
label = alias;
|
||||
}
|
||||
sel.options[0].textContent = "Default — " + label;
|
||||
}
|
||||
|
||||
// Channel platforms shown in admin notify-target rows. Mirror server-side
|
||||
// channel adapters; expand here when a new adapter ships (Discord / Slack
|
||||
// today, MS Teams / etc. later).
|
||||
@@ -1333,6 +1375,7 @@ function showCreateScheduleModal() {
|
||||
display: function (m) {
|
||||
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
|
||||
},
|
||||
afterPopulate: _decorateScheduleModelPlaceholder,
|
||||
});
|
||||
// Populate skill dropdown
|
||||
_populateScheduleSelect(
|
||||
@@ -1487,6 +1530,7 @@ function showEditScheduleModal(taskId) {
|
||||
display: function (m) {
|
||||
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
|
||||
},
|
||||
afterPopulate: _decorateScheduleModelPlaceholder,
|
||||
});
|
||||
// Populate skill dropdown with current value pre-selected
|
||||
_populateScheduleSelect(
|
||||
@@ -2637,7 +2681,7 @@ function loadSettings() {
|
||||
|
||||
// Merge values + schema. Skip role-assignment settings owned by
|
||||
// the Models → Roles sub-tab (judge.* settings still live on the
|
||||
// Judge tab; the four model-tab roles render only there).
|
||||
// Judge tab; the model-tab roles render only there).
|
||||
var merged = {};
|
||||
var roleKeys = {
|
||||
"coordinator.model_alias": 1,
|
||||
@@ -2646,6 +2690,7 @@ function loadSettings() {
|
||||
"model.plan_effort": 1,
|
||||
"model.task_alias": 1,
|
||||
"model.task_effort": 1,
|
||||
"channels.default_model_alias": 1,
|
||||
};
|
||||
for (var j = 0; j < valuesArr.length; j++) {
|
||||
var v = valuesArr[j];
|
||||
@@ -3269,6 +3314,11 @@ function _renderMcpServers(items) {
|
||||
var totalTools = 0,
|
||||
totalRes = 0,
|
||||
totalPrompts = 0;
|
||||
// Phase 9: aggregate the most-recent refresh entry across nodes
|
||||
// so the admin pill reflects "the freshest known state" rather
|
||||
// than picking an arbitrary node.
|
||||
var newestRefreshAt = null;
|
||||
var newestRefreshOutcome = null;
|
||||
for (var j = 0; j < nodeIds.length; j++) {
|
||||
var ns = statusEntries[nodeIds[j]];
|
||||
if (ns.connected) {
|
||||
@@ -3281,6 +3331,13 @@ function _renderMcpServers(items) {
|
||||
anyError = true;
|
||||
if (!firstError) firstError = ns.error;
|
||||
}
|
||||
if (
|
||||
typeof ns.last_refresh_at === "number" &&
|
||||
(newestRefreshAt === null || ns.last_refresh_at > newestRefreshAt)
|
||||
) {
|
||||
newestRefreshAt = ns.last_refresh_at;
|
||||
newestRefreshOutcome = ns.last_refresh_outcome || null;
|
||||
}
|
||||
}
|
||||
|
||||
var dotClass = "mcp-status-dot disabled";
|
||||
@@ -3306,6 +3363,42 @@ function _renderMcpServers(items) {
|
||||
statusText = "idle";
|
||||
}
|
||||
|
||||
// Phase 9: refresh pill shows the short-relative age (e.g. "12m") with
|
||||
// outcome-tinted color (ok vs err) and the full ISO timestamp + outcome
|
||||
// in the tooltip. Pill is omitted (and the cell stays unchanged from
|
||||
// its pre-Phase-9 shape) when no node has yet recorded a refresh
|
||||
// outcome for this server.
|
||||
var refreshPill = "";
|
||||
if (newestRefreshAt !== null) {
|
||||
var ageSeconds = Math.max(
|
||||
0,
|
||||
Math.floor(Date.now() / 1000 - newestRefreshAt),
|
||||
);
|
||||
var ageShort;
|
||||
if (ageSeconds < 60) ageShort = ageSeconds + "s";
|
||||
else if (ageSeconds < 3600) ageShort = Math.floor(ageSeconds / 60) + "m";
|
||||
else if (ageSeconds < 86400)
|
||||
ageShort = Math.floor(ageSeconds / 3600) + "h";
|
||||
else ageShort = Math.floor(ageSeconds / 86400) + "d";
|
||||
var outcomeText = newestRefreshOutcome || "unknown";
|
||||
var pillCls =
|
||||
outcomeText === "ok" ? "mcp-refresh-pill-ok" : "mcp-refresh-pill-err";
|
||||
var pillTitle =
|
||||
"Last refresh " +
|
||||
new Date(newestRefreshAt * 1000).toISOString() +
|
||||
" (" +
|
||||
outcomeText +
|
||||
")";
|
||||
refreshPill =
|
||||
' <span class="mcp-refresh-pill ' +
|
||||
pillCls +
|
||||
'" title="' +
|
||||
escapeHtml(pillTitle) +
|
||||
'">' +
|
||||
escapeHtml(ageShort) +
|
||||
"</span>";
|
||||
}
|
||||
|
||||
var transportCls =
|
||||
s.transport === "stdio" ? "mcp-transport-stdio" : "mcp-transport-http";
|
||||
var toolsVal = anyConnected
|
||||
@@ -3340,6 +3433,24 @@ function _renderMcpServers(items) {
|
||||
'<button class="admin-btn-action" data-mcp-oauth-connect="' +
|
||||
escapeHtml(s.name) +
|
||||
'">connect</button>';
|
||||
// Phase 9: surface the consented-users count + bulk-revoke
|
||||
// affordance only when at least one user has consented.
|
||||
var consentCount =
|
||||
typeof s.consented_users_count === "number"
|
||||
? s.consented_users_count
|
||||
: 0;
|
||||
if (consentCount > 0) {
|
||||
actionBtns +=
|
||||
'<button class="admin-btn-danger" data-mcp-bulk-revoke="' +
|
||||
escapeHtml(s.name) +
|
||||
'" data-mcp-consent-count="' +
|
||||
consentCount +
|
||||
'" title="Drop all ' +
|
||||
consentCount +
|
||||
' user consents for this server">bulk-revoke (' +
|
||||
consentCount +
|
||||
")</button>";
|
||||
}
|
||||
}
|
||||
var actions = isConfig
|
||||
? actionBtns
|
||||
@@ -3384,6 +3495,7 @@ function _renderMcpServers(items) {
|
||||
dotClass +
|
||||
'" aria-hidden="true"></span>' +
|
||||
escapeHtml(statusText) +
|
||||
refreshPill +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-mactions">' +
|
||||
actions +
|
||||
@@ -3462,6 +3574,42 @@ function _renderMcpServers(items) {
|
||||
window.open(url, "_blank", "noopener");
|
||||
});
|
||||
});
|
||||
el.querySelectorAll("[data-mcp-bulk-revoke]").forEach(function (btn) {
|
||||
btn.addEventListener("click", function () {
|
||||
var name = this.getAttribute("data-mcp-bulk-revoke");
|
||||
var count = this.getAttribute("data-mcp-consent-count") || "?";
|
||||
showConfirmModal(
|
||||
"Bulk-revoke MCP consents",
|
||||
"Drop all " +
|
||||
count +
|
||||
' user consents for server "' +
|
||||
name +
|
||||
'"? Users will need to re-consent on next use. Upstream revoke is not attempted in bulk; tokens at the authorization server will expire naturally.',
|
||||
"Bulk-revoke",
|
||||
function () {
|
||||
authFetch(
|
||||
"/v1/api/admin/mcp-servers/" +
|
||||
encodeURIComponent(name) +
|
||||
"/bulk-revoke",
|
||||
{ method: "POST" },
|
||||
)
|
||||
.then(function (r) {
|
||||
if (!r.ok) throw new Error();
|
||||
return r.json();
|
||||
})
|
||||
.then(function (j) {
|
||||
showToast(
|
||||
"Bulk-revoked " + (j.rows_deleted || 0) + " row(s) for " + name,
|
||||
);
|
||||
loadAdminMcp();
|
||||
})
|
||||
.catch(function () {
|
||||
showToast("Failed to bulk-revoke " + name);
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
el.querySelectorAll("[data-mcp-delete]").forEach(function (btn) {
|
||||
btn.addEventListener("click", function () {
|
||||
var sid = this.getAttribute("data-mcp-delete");
|
||||
@@ -4639,6 +4787,14 @@ var _modelCreateTrigger = null;
|
||||
// setting render a second selector inline. Adding a new role (e.g.
|
||||
// ``perception.audio.model``) is purely additive: drop a row here once
|
||||
// the SettingDef lands in turnstone/core/settings_registry.py.
|
||||
// ``fallbackKind`` controls how the empty/blank option in the alias
|
||||
// dropdown is labelled. Coordinator and Judge fall back to a single
|
||||
// well-defined alias (model.default_alias / coordinator alias) so we
|
||||
// surface that concrete model in the placeholder. Plan/Task agents
|
||||
// cascade through ``[model].plan_model → [model].agent_model →
|
||||
// session model`` per turnstone/core/settings_registry.py — there's
|
||||
// no single "default" to advertise, so the blank reads "(inherit)"
|
||||
// to match the vocabulary of the reasoning-effort dropdowns.
|
||||
var MODEL_ROLES = [
|
||||
{
|
||||
label: "Coordinator",
|
||||
@@ -4646,12 +4802,14 @@ var MODEL_ROLES = [
|
||||
"Console-hosted coordinator sessions that drive child workstreams.",
|
||||
aliasKey: "coordinator.model_alias",
|
||||
effortKey: "coordinator.reasoning_effort",
|
||||
fallbackKind: "default",
|
||||
},
|
||||
{
|
||||
label: "Judge",
|
||||
description:
|
||||
"Intent-validation judge that scores tool calls before approval.",
|
||||
aliasKey: "judge.model",
|
||||
fallbackKind: "default",
|
||||
},
|
||||
{
|
||||
label: "Plan agent",
|
||||
@@ -4659,6 +4817,7 @@ var MODEL_ROLES = [
|
||||
"plan_agent sub-agent — produces high-level plans before task dispatch.",
|
||||
aliasKey: "model.plan_alias",
|
||||
effortKey: "model.plan_effort",
|
||||
fallbackKind: "inherit",
|
||||
},
|
||||
{
|
||||
label: "Task agent",
|
||||
@@ -4666,6 +4825,14 @@ var MODEL_ROLES = [
|
||||
"task_agent sub-agent — runs autonomous subtasks dispatched by the parent.",
|
||||
aliasKey: "model.task_alias",
|
||||
effortKey: "model.task_effort",
|
||||
fallbackKind: "inherit",
|
||||
},
|
||||
{
|
||||
label: "Channel adapter",
|
||||
description:
|
||||
"Workstreams created by channel adapters (Discord, Slack) when no model is specified at creation time.",
|
||||
aliasKey: "channels.default_model_alias",
|
||||
fallbackKind: "default",
|
||||
},
|
||||
];
|
||||
|
||||
@@ -4848,11 +5015,40 @@ function _renderModelRoles(container, values, schema) {
|
||||
"aria-label",
|
||||
role.label + " model (empty = default)",
|
||||
);
|
||||
// Format the blank/inherit option in the same "alias (model)" shape
|
||||
// as the other rows so the dropdown reads consistently — without
|
||||
// this the empty row was bare "(default — flatspark)" while every
|
||||
// other row carried a "(/models/...)" suffix. Plan/Task agent
|
||||
// fall back through a multi-step chain (config.toml → agent_model
|
||||
// → session) that has no single concrete "default", so they get
|
||||
// a plain "(inherit)" instead of the misleading
|
||||
// "(default — <coordinator-alias>)".
|
||||
var blank = document.createElement("option");
|
||||
blank.value = "";
|
||||
blank.textContent = _modelDefaultAlias
|
||||
? "(default — " + _modelDefaultAlias + ")"
|
||||
: "(default)";
|
||||
if (role.fallbackKind === "inherit") {
|
||||
blank.textContent = "(inherit)";
|
||||
} else {
|
||||
var defaultDef = null;
|
||||
if (_modelDefaultAlias) {
|
||||
for (var dm = 0; dm < enabledAliases.length; dm++) {
|
||||
if (enabledAliases[dm].alias === _modelDefaultAlias) {
|
||||
defaultDef = enabledAliases[dm];
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (defaultDef) {
|
||||
var defLabel =
|
||||
defaultDef.alias === defaultDef.model
|
||||
? defaultDef.alias
|
||||
: defaultDef.alias + " (" + defaultDef.model + ")";
|
||||
blank.textContent = "(default — " + defLabel + ")";
|
||||
} else if (_modelDefaultAlias) {
|
||||
blank.textContent = "(default — " + _modelDefaultAlias + ")";
|
||||
} else {
|
||||
blank.textContent = "(default)";
|
||||
}
|
||||
}
|
||||
aliasSel.appendChild(blank);
|
||||
var currentAlias = aliasInfo.value || "";
|
||||
var matched = false;
|
||||
@@ -5008,6 +5204,11 @@ function _renderModels(items) {
|
||||
if (m.max_tokens != null) overrides.push("max_tok=" + m.max_tokens);
|
||||
if (m.reasoning_effort != null)
|
||||
overrides.push("effort=" + m.reasoning_effort);
|
||||
// Reasoning persistence flags surface only when non-default
|
||||
// (persist=False is the operator opt-out; replay=True is the
|
||||
// operator opt-in). Default values are silent.
|
||||
if (m.surface_persisted_reasoning === false) overrides.push("surface=off");
|
||||
if (m.replay_reasoning_to_model === true) overrides.push("replay=on");
|
||||
if (overrides.length) {
|
||||
var ovrSpan = document.createElement("span");
|
||||
ovrSpan.className = "model-overrides-hint";
|
||||
@@ -5196,6 +5397,8 @@ function showCreateModelModal() {
|
||||
el.style.borderColor = "";
|
||||
});
|
||||
document.getElementById("model-enabled").checked = true;
|
||||
document.getElementById("model-surface-persisted-reasoning").checked = true;
|
||||
document.getElementById("model-replay-reasoning").checked = false;
|
||||
document.getElementById("model-detect-result").style.display = "none";
|
||||
document.getElementById("model-detect-btn").disabled = false;
|
||||
document.getElementById("model-detect-btn").textContent = "Detect";
|
||||
@@ -5278,6 +5481,13 @@ function showEditModelModal(definitionId) {
|
||||
document.getElementById("model-capabilities").value =
|
||||
capsText === "{}" ? "" : capsText;
|
||||
document.getElementById("model-enabled").checked = m.enabled !== false;
|
||||
// Reasoning persistence flags — defaults match the dataclass
|
||||
// defaults (persist=true, replay=false) when the API returns
|
||||
// them as undefined (legacy / pre-052 row).
|
||||
document.getElementById("model-surface-persisted-reasoning").checked =
|
||||
m.surface_persisted_reasoning !== false;
|
||||
document.getElementById("model-replay-reasoning").checked =
|
||||
m.replay_reasoning_to_model === true;
|
||||
_applyProviderDefaults();
|
||||
})
|
||||
.catch(function () {
|
||||
@@ -5419,6 +5629,16 @@ function submitCreateModel() {
|
||||
form.reasoning_effort = null;
|
||||
}
|
||||
|
||||
// Reasoning persistence flags — always serialize so a flip from
|
||||
// default takes effect on PUT (the server's update path keys off
|
||||
// "field present in body").
|
||||
form.surface_persisted_reasoning = document.getElementById(
|
||||
"model-surface-persisted-reasoning",
|
||||
).checked;
|
||||
form.replay_reasoning_to_model = document.getElementById(
|
||||
"model-replay-reasoning",
|
||||
).checked;
|
||||
|
||||
var apiKey = document.getElementById("model-api-key").value;
|
||||
if (apiKey) form.api_key = apiKey;
|
||||
|
||||
|
||||
@@ -1761,11 +1761,11 @@ function _mountHomeCoordComposer() {
|
||||
id: "judge_model",
|
||||
label: "Judge Model",
|
||||
type: "select",
|
||||
// Neutral label — the actual default is ConfigStore
|
||||
// ``judge.model`` when set, IntentJudge's agent-model
|
||||
// fallback when not. "Default judge model" doesn't
|
||||
// mislead either way.
|
||||
choices: [{ value: "", text: "Default judge model" }],
|
||||
// Initial placeholder; _populateHomeModelDropdowns rewrites this
|
||||
// to "Default model (<alias>)" once /v1/api/models reports the
|
||||
// resolved judge alias (judge.model when set, otherwise the
|
||||
// session model — see IntentJudge.__init__).
|
||||
choices: [{ value: "", text: "Default model" }],
|
||||
},
|
||||
],
|
||||
},
|
||||
@@ -1801,6 +1801,21 @@ function _populateHomeSkillDropdown() {
|
||||
});
|
||||
}
|
||||
|
||||
// Format a resolved alias with its model suffix the same way as the
|
||||
// dropdown rows ("alias (model)", or just "alias" when they coincide).
|
||||
// Returns "" when alias is empty or unknown so callers can fall back
|
||||
// to a neutral placeholder.
|
||||
function _resolveModelLabel(alias, models) {
|
||||
if (!alias) return "";
|
||||
for (var i = 0; i < (models || []).length; i++) {
|
||||
var m = models[i];
|
||||
if (m.alias === alias) {
|
||||
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
// Populate Model + Judge Model dropdowns from /v1/api/models — same
|
||||
// list the interactive new-ws modal uses. Empty/default option stays
|
||||
// at the top so submitting without a choice falls back to the
|
||||
@@ -1819,6 +1834,30 @@ function _populateHomeModelDropdowns() {
|
||||
});
|
||||
_homeCoordComposer.setOptionChoices("model", choices);
|
||||
_homeCoordComposer.setOptionChoices("judge_model", choices);
|
||||
// Both placeholders use the same "Default — alias (model)"
|
||||
// template — the field-row labels (MODEL / JUDGE MODEL) already
|
||||
// carry the role context, so an asymmetric "Default judge model"
|
||||
// reads awkwardly alongside the plain "Default model" line above
|
||||
// it. Em-dash separator (rather than nested parens) keeps the
|
||||
// alias's "(model)" suffix legible and matches the
|
||||
// ``(default — alias (model))`` pattern used in the admin Roles
|
||||
// tab.
|
||||
var coordDefault = _resolveModelLabel(
|
||||
data.coordinator_default_alias || "",
|
||||
data.models || [],
|
||||
);
|
||||
var judgeDefault = _resolveModelLabel(
|
||||
data.judge_default_alias || "",
|
||||
data.models || [],
|
||||
);
|
||||
_homeCoordComposer.setOptionPlaceholder(
|
||||
"model",
|
||||
coordDefault ? "Default — " + coordDefault : "Default model",
|
||||
);
|
||||
_homeCoordComposer.setOptionPlaceholder(
|
||||
"judge_model",
|
||||
judgeDefault ? "Default — " + judgeDefault : "Default model",
|
||||
);
|
||||
})
|
||||
.catch(function () {
|
||||
/* defaults still work even without the dropdown populated */
|
||||
|
||||
@@ -579,6 +579,13 @@
|
||||
if (callId && toolRows.has(callId)) {
|
||||
const entry = toolRows.get(callId);
|
||||
_appendResultToRow(entry.row, output, isError, opts);
|
||||
// The batch may have been --running (live tool_info auto path,
|
||||
// approval_resolved approved path, or replay-time orphan).
|
||||
// Drop --running once every row in the batch has a result so
|
||||
// the kicker text + visual style flip back to the post-execution
|
||||
// state. Per-row check (not a counter) keeps the logic
|
||||
// resilient to out-of-order replay + late SSE deliveries.
|
||||
_unsetBatchRunningIfAllResults(entry.batch);
|
||||
// Result blocks grow scrollHeight; without this the user pinned
|
||||
// at the bottom loses their pin when the row inflates. appendMsg
|
||||
// already routes through _scheduleScroll on the legacy path; this
|
||||
@@ -1166,6 +1173,37 @@
|
||||
return status;
|
||||
}
|
||||
|
||||
function _setBatchRunning(batch) {
|
||||
if (!batch) return;
|
||||
batch.classList.add("coord-tool-batch--running");
|
||||
const kicker = batch.querySelector(".coord-tool-batch-kicker");
|
||||
if (kicker) {
|
||||
const rowCount = batch.querySelectorAll(".coord-tool-row").length;
|
||||
kicker.textContent =
|
||||
rowCount >= 2 ? "Running · Parallel " + rowCount : "Running";
|
||||
}
|
||||
}
|
||||
|
||||
function _unsetBatchRunningIfAllResults(batch) {
|
||||
// Remove ``--running`` once every row in the batch has rendered a
|
||||
// result block. Caller invokes after each tool_result; the test
|
||||
// is "did THIS result complete the batch?" — cheap DOM walk over
|
||||
// the same handful of rows we already track.
|
||||
if (!batch) return;
|
||||
if (!batch.classList.contains("coord-tool-batch--running")) return;
|
||||
const rows = batch.querySelectorAll(".coord-tool-row");
|
||||
for (const row of rows) {
|
||||
if (!row.querySelector(".coord-tool-row-result")) return;
|
||||
}
|
||||
batch.classList.remove("coord-tool-batch--running");
|
||||
const kicker = batch.querySelector(".coord-tool-batch-kicker");
|
||||
if (kicker && !batch.classList.contains("coord-tool-batch--pending")) {
|
||||
const rowCount = rows.length;
|
||||
kicker.textContent =
|
||||
rowCount >= 2 ? "Parallel · " + rowCount + " tools" : "Tool";
|
||||
}
|
||||
}
|
||||
|
||||
function _morphBatchResolved(batch, opts) {
|
||||
if (!batch) return;
|
||||
batch.classList.remove("coord-tool-batch--pending");
|
||||
@@ -1283,9 +1321,15 @@
|
||||
_announceAssertive(_approvalAriaLabel(items));
|
||||
} else if (
|
||||
opts.auto &&
|
||||
existing.classList.contains("coord-tool-batch--running")
|
||||
existing.classList.contains("coord-tool-batch--running") &&
|
||||
!existing.classList.contains("coord-tool-batch--auto")
|
||||
) {
|
||||
existing.classList.remove("coord-tool-batch--running");
|
||||
// SSE tool_info clarifies an existing --running batch as
|
||||
// auto-approved. Keep --running (the tool is still in
|
||||
// flight; tool_result will remove it) and add --auto so the
|
||||
// batch reflects BOTH "auto-approved" + "running" — historical
|
||||
// behaviour swapped --running out, which lost the running
|
||||
// indicator the moment tool_info clarified the approval state.
|
||||
existing.classList.add("coord-tool-batch--auto");
|
||||
} else if (opts.pending) {
|
||||
// Already pending — keep the action row, just refresh
|
||||
@@ -1328,7 +1372,6 @@
|
||||
);
|
||||
if (opts.pending) batch.classList.add("coord-tool-batch--pending");
|
||||
else if (opts.auto) batch.classList.add("coord-tool-batch--auto");
|
||||
else if (opts.running) batch.classList.add("coord-tool-batch--running");
|
||||
else if (opts.resolved) {
|
||||
batch.classList.add(
|
||||
opts.resolved.approved
|
||||
@@ -1336,6 +1379,12 @@
|
||||
: "coord-tool-batch--denied",
|
||||
);
|
||||
}
|
||||
// ``running`` is additive — coexists with ``auto`` (auto-approved
|
||||
// and currently in flight) or stands alone (replay-time orphan
|
||||
// before SSE clarifies the approval state). Removed by
|
||||
// :func:`_unsetBatchRunningIfAllResults` when every row in the
|
||||
// batch has a tool_result.
|
||||
if (opts.running) batch.classList.add("coord-tool-batch--running");
|
||||
|
||||
const head = document.createElement("div");
|
||||
head.className = "coord-tool-batch-head";
|
||||
@@ -1960,6 +2009,47 @@
|
||||
case "reasoning":
|
||||
appendReasoningToken(ev.text || "");
|
||||
break;
|
||||
case "in_progress_snapshot":
|
||||
// One-shot replay of the in-progress turn's reasoning + content
|
||||
// when this client connects mid-stream (page refresh while the
|
||||
// model is generating). Idempotent on EventSource auto-reconnect:
|
||||
// skip overwrite when the current buffer is already at-or-past
|
||||
// the snapshot length, so a stale replay can't reset the live-
|
||||
// streamed view back to a shorter prefix.
|
||||
if (ev.reasoning && ev.reasoning.length > currentReasoningBuf.length) {
|
||||
if (!currentReasoningEl) {
|
||||
currentReasoningEl = appendMsg("reasoning", "", {
|
||||
label: "reasoning",
|
||||
});
|
||||
messagesEl.setAttribute("aria-live", "off");
|
||||
}
|
||||
currentReasoningBuf = ev.reasoning;
|
||||
var rbody = currentReasoningEl.querySelector(".msg-body");
|
||||
if (rbody) rbody.textContent = currentReasoningBuf;
|
||||
_scheduleScroll();
|
||||
}
|
||||
if (ev.content && ev.content.length > currentAssistantBuf.length) {
|
||||
if (!currentAssistantEl) {
|
||||
currentAssistantEl = appendMsg("assistant", "", {
|
||||
label: "assistant",
|
||||
});
|
||||
messagesEl.setAttribute("aria-live", "off");
|
||||
}
|
||||
currentAssistantBuf = ev.content;
|
||||
var abody = currentAssistantEl.querySelector(".msg-body");
|
||||
if (abody && typeof streamingRender === "function") {
|
||||
try {
|
||||
streamingRender(abody, currentAssistantBuf);
|
||||
} catch (e) {
|
||||
console.warn("coordinator streamingRender failed", e);
|
||||
abody.textContent = currentAssistantBuf;
|
||||
}
|
||||
} else if (abody) {
|
||||
abody.textContent = currentAssistantBuf;
|
||||
}
|
||||
_scheduleScroll();
|
||||
}
|
||||
break;
|
||||
case "stream_end":
|
||||
finishAssistantStream();
|
||||
break;
|
||||
@@ -2007,11 +2097,19 @@
|
||||
const wasAlways =
|
||||
ev.always === true ||
|
||||
(ev.always === undefined && target.dataset.requestedAlways === "1");
|
||||
const approved = ev.approved !== false;
|
||||
_morphBatchResolved(target, {
|
||||
approved: ev.approved !== false,
|
||||
approved,
|
||||
always: wasAlways,
|
||||
feedback: ev.feedback || null,
|
||||
});
|
||||
// Approved batches start running the moment the user clicks
|
||||
// approve — mirror the auto path so the live RUNNING
|
||||
// indicator shows during execution (not just on refresh).
|
||||
// Denied batches don't run at all, so no --running.
|
||||
if (approved) {
|
||||
_setBatchRunning(target);
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
@@ -2202,8 +2300,12 @@
|
||||
// for both kinds, matching the interactive payload name. All
|
||||
// items in a single ``tool_info`` envelope share a dispatch
|
||||
// turn, so render them as one batch construct (parallel when
|
||||
// ≥2, solo otherwise) rather than N separate bubbles.
|
||||
appendToolBatch(ev.items || [], { auto: true });
|
||||
// ≥2, solo otherwise) rather than N separate bubbles. ``auto``
|
||||
// marks the approval-state class; ``running`` marks "in flight"
|
||||
// — the tool starts executing the moment auto-approval lands,
|
||||
// and the batch should show the same RUNNING indicator the
|
||||
// replay path renders for an unresolved committed turn.
|
||||
appendToolBatch(ev.items || [], { auto: true, running: true });
|
||||
break;
|
||||
// Child-workstream fan-out routed through the coordinator's own
|
||||
// SSE stream. CoordinatorManager filters the cluster event bus
|
||||
@@ -4093,6 +4195,20 @@
|
||||
appendUserMessageWithAttachments(text, [], { label: "user" });
|
||||
});
|
||||
} else if (role === "assistant") {
|
||||
// Reasoning bubble (Phase 1 reasoning persistence) — render
|
||||
// BEFORE the content card so the visual order matches the
|
||||
// live SSE flow (reasoning_delta arrives before content_delta
|
||||
// for thinking-enabled models). Mirrors the live ":1524" /
|
||||
// snapshot ":2021" call sites — same appendMsg("reasoning")
|
||||
// helper, just driven from history-render rather than the
|
||||
// SSE handler. Only present when the active model's
|
||||
// surface_persisted_reasoning flag is true and the message round-tripped
|
||||
// a thinking lane.
|
||||
if (typeof m.reasoning === "string" && m.reasoning.length) {
|
||||
const rEl = appendMsg("reasoning", "", { label: "reasoning" });
|
||||
const rBody = rEl && rEl.querySelector(".msg-body");
|
||||
if (rBody) rBody.textContent = m.reasoning;
|
||||
}
|
||||
// Render content BEFORE the tool batch so DOM order matches
|
||||
// chronological order (the model emits text first, then
|
||||
// dispatches tools). Whitespace-only content (e.g. "\n\n"
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
<link rel="stylesheet" href="/shared/base.css">
|
||||
<link rel="stylesheet" href="/shared/ui-base.css">
|
||||
<link rel="stylesheet" href="/shared/chat.css">
|
||||
<link rel="stylesheet" href="/shared/katex-0.16.45/katex.min.css">
|
||||
<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">
|
||||
<link rel="stylesheet" href="/static/style.css">
|
||||
<link rel="stylesheet" href="/static/coordinator/coordinator.css">
|
||||
<style>
|
||||
@@ -634,7 +634,7 @@
|
||||
<script src="/shared/composer_attachments.js"></script>
|
||||
<script src="/shared/composer_queue.js"></script>
|
||||
<script src="/shared/status_bar.js"></script>
|
||||
<script src="/shared/katex-0.16.45/katex.min.js"></script>
|
||||
<script src="/shared/katex-0.16.47/katex.min.js"></script>
|
||||
<script src="/shared/hljs-11.11.1/highlight.min.js"></script>
|
||||
<script src="/shared/renderer.js"></script>
|
||||
<script src="/static/coordinator/coordinator.js"></script>
|
||||
|
||||
@@ -143,46 +143,91 @@ function _renderGovRoles(items) {
|
||||
}
|
||||
}
|
||||
|
||||
// All permission names for the checkbox UI
|
||||
var _ALL_PERMISSIONS = [
|
||||
"read",
|
||||
"write",
|
||||
"approve",
|
||||
"admin.users",
|
||||
"admin.roles",
|
||||
"admin.orgs",
|
||||
"admin.policies",
|
||||
"admin.skills",
|
||||
"admin.audit",
|
||||
"admin.usage",
|
||||
"admin.schedules",
|
||||
"admin.watches",
|
||||
"admin.judge",
|
||||
"admin.memories",
|
||||
"admin.settings",
|
||||
"admin.mcp",
|
||||
"tools.approve",
|
||||
"workstreams.create",
|
||||
"workstreams.close",
|
||||
// Permission inventory grouped by namespace so the role modal can
|
||||
// render each section under its own heading. Sectioning prevents the
|
||||
// row-flow grid from slicing a namespace mid-column (e.g. half of
|
||||
// ``admin.*`` ending up in column 1, the rest in column 2) and lets
|
||||
// readers who don't yet know the permission taxonomy scan by
|
||||
// concept. Each section's permissions render as a 2-column grid;
|
||||
// the ``Scopes`` and ``Workstreams & Tools`` sections are short
|
||||
// enough to fit one row, ``Admin`` carries the bulk.
|
||||
var _PERMISSION_SECTIONS = [
|
||||
{
|
||||
label: "Scopes",
|
||||
permissions: ["read", "write", "approve"],
|
||||
},
|
||||
{
|
||||
label: "Admin",
|
||||
permissions: [
|
||||
"admin.users",
|
||||
"admin.roles",
|
||||
"admin.orgs",
|
||||
"admin.policies",
|
||||
"admin.skills",
|
||||
"admin.audit",
|
||||
"admin.usage",
|
||||
"admin.schedules",
|
||||
"admin.watches",
|
||||
"admin.judge",
|
||||
"admin.memories",
|
||||
"admin.settings",
|
||||
"admin.mcp",
|
||||
],
|
||||
},
|
||||
{
|
||||
label: "Workstreams & Tools",
|
||||
permissions: ["workstreams.create", "workstreams.close", "tools.approve"],
|
||||
},
|
||||
];
|
||||
|
||||
function _buildPermCheckboxes(prefix, selected) {
|
||||
var html = '<div class="perm-grid">';
|
||||
for (var i = 0; i < _ALL_PERMISSIONS.length; i++) {
|
||||
var p = _ALL_PERMISSIONS[i];
|
||||
var checked = selected && selected.indexOf(p) >= 0 ? " checked" : "";
|
||||
html +=
|
||||
'<label class="perm-checkbox"><input type="checkbox" value="' +
|
||||
p +
|
||||
'" name="' +
|
||||
prefix +
|
||||
'-perm"' +
|
||||
checked +
|
||||
"> " +
|
||||
escapeHtml(p) +
|
||||
"</label>";
|
||||
// Flat list — kept for any caller that wants the full permission
|
||||
// inventory without caring about sectioning.
|
||||
var _ALL_PERMISSIONS = (function () {
|
||||
var flat = [];
|
||||
for (var i = 0; i < _PERMISSION_SECTIONS.length; i++) {
|
||||
flat = flat.concat(_PERMISSION_SECTIONS[i].permissions);
|
||||
}
|
||||
return flat;
|
||||
})();
|
||||
|
||||
function _buildPermCheckboxes(prefix, selected) {
|
||||
// Emits the toggle-switch component used elsewhere in the admin
|
||||
// modals so each permission reads as a deliberate on/off rather
|
||||
// than a generic checkbox. Sections are wrapped in a
|
||||
// ``.perm-section`` block with a caps-styled heading so the
|
||||
// typographic system inside the role modal stays consistent (the
|
||||
// surrounding label cadence is also caps + 0.08em letter-spacing).
|
||||
// The underlying ``<input type="checkbox" name="{prefix}-perm">``
|
||||
// shape is preserved so ``_collectPermCheckboxes`` still picks
|
||||
// them up regardless of section.
|
||||
var html = "";
|
||||
for (var s = 0; s < _PERMISSION_SECTIONS.length; s++) {
|
||||
var section = _PERMISSION_SECTIONS[s];
|
||||
html +=
|
||||
'<div class="perm-section">' +
|
||||
'<div class="perm-section-label">' +
|
||||
escapeHtml(section.label) +
|
||||
"</div>" +
|
||||
'<div class="perm-grid">';
|
||||
for (var i = 0; i < section.permissions.length; i++) {
|
||||
var p = section.permissions[i];
|
||||
var checked = selected && selected.indexOf(p) >= 0 ? " checked" : "";
|
||||
html +=
|
||||
'<label class="toggle-switch perm-toggle">' +
|
||||
'<input type="checkbox" value="' +
|
||||
p +
|
||||
'" name="' +
|
||||
prefix +
|
||||
'-perm"' +
|
||||
checked +
|
||||
">" +
|
||||
'<span class="toggle-track" aria-hidden="true"></span>' +
|
||||
'<span class="toggle-label">' +
|
||||
escapeHtml(p) +
|
||||
"</span></label>";
|
||||
}
|
||||
html += "</div></div>";
|
||||
}
|
||||
html += "</div>";
|
||||
return html;
|
||||
}
|
||||
|
||||
@@ -346,20 +391,27 @@ function showUserRolesModal(userId) {
|
||||
var assigned = {};
|
||||
for (var i = 0; i < userRoles.length; i++)
|
||||
assigned[userRoles[i].role_id] = true;
|
||||
var html = "";
|
||||
// Role-assignment rows reuse the toggle-switch component for
|
||||
// consistency with the rest of the admin UX. Role display names
|
||||
// are human-readable text, so no monospace override is needed.
|
||||
var html = '<div class="user-roles-list">';
|
||||
for (var j = 0; j < allRoles.length; j++) {
|
||||
var r = allRoles[j];
|
||||
var checked = assigned[r.role_id] ? " checked" : "";
|
||||
html +=
|
||||
'<label class="perm-checkbox"><input type="checkbox" value="' +
|
||||
'<label class="toggle-switch user-role-toggle">' +
|
||||
'<input type="checkbox" value="' +
|
||||
escapeHtml(r.role_id) +
|
||||
'" name="ur-role"' +
|
||||
checked +
|
||||
"> " +
|
||||
">" +
|
||||
'<span class="toggle-track" aria-hidden="true"></span>' +
|
||||
'<span class="toggle-label">' +
|
||||
escapeHtml(r.display_name) +
|
||||
"</label>";
|
||||
"</span></label>";
|
||||
}
|
||||
container.innerHTML = html;
|
||||
html += "</div>";
|
||||
container.innerHTML = html; // values escaped via escapeHtml above
|
||||
})
|
||||
.catch(function () {
|
||||
container.innerHTML =
|
||||
@@ -3199,18 +3251,27 @@ function renderJudgeSettings() {
|
||||
var isDefault = s.source === "default";
|
||||
|
||||
if (s.type === "bool") {
|
||||
// Toggle switch — same component used by the admin modals.
|
||||
// ``onchange`` reads the box's new ``.checked`` and writes via
|
||||
// saveJudgeSetting. The ``.toggle-label`` is a static "Enabled"
|
||||
// because the slider position is the truth — flipping the
|
||||
// caption text on save round-tripped through reload, so the
|
||||
// slider moved instantly while the caption lagged 50-300ms and
|
||||
// looked broken. ``aria-label`` carries the setting name so
|
||||
// screen readers get the row context inline.
|
||||
inputHtml =
|
||||
'<label class="toggle-label" style="display:flex;align-items:center;gap:8px;cursor:pointer">' +
|
||||
'<label class="toggle-switch toggle--flush">' +
|
||||
'<input type="checkbox" data-key="' +
|
||||
s.key +
|
||||
'" aria-label="' +
|
||||
escapeHtml(shortKey) +
|
||||
'" ' +
|
||||
(currentVal ? "checked" : "") +
|
||||
" onchange=\"saveJudgeSetting('" +
|
||||
s.key +
|
||||
'\',this.checked)" style="width:16px;height:16px">' +
|
||||
'<span style="font-size:12px">' +
|
||||
(currentVal ? "Enabled" : "Disabled") +
|
||||
"</span></label>";
|
||||
"',this.checked)\">" +
|
||||
'<span class="toggle-track" aria-hidden="true"></span>' +
|
||||
'<span class="toggle-label">Enabled</span></label>';
|
||||
} else if (s.type === "float") {
|
||||
inputHtml =
|
||||
'<div style="display:flex;gap:8px;align-items:center">' +
|
||||
|
||||
+136
-100
@@ -2507,10 +2507,11 @@
|
||||
rows="3"
|
||||
placeholder="What should the workstream do?"
|
||||
></textarea>
|
||||
<label class="admin-checkbox"
|
||||
><input id="cs-autoapprove" type="checkbox" /> Auto-approve tool
|
||||
calls</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="cs-autoapprove" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Auto-approve tool calls</span>
|
||||
</label>
|
||||
<label
|
||||
>Notify on completion
|
||||
<span class="label-hint">optional</span></label
|
||||
@@ -2609,13 +2610,16 @@
|
||||
</select>
|
||||
<label for="es-message">Initial message</label>
|
||||
<textarea id="es-message" rows="3"></textarea>
|
||||
<label class="admin-checkbox"
|
||||
><input id="es-autoapprove" type="checkbox" /> Auto-approve tool
|
||||
calls</label
|
||||
>
|
||||
<label class="admin-checkbox"
|
||||
><input id="es-enabled" type="checkbox" /> Enabled</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="es-autoapprove" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Auto-approve tool calls</span>
|
||||
</label>
|
||||
<label class="toggle-switch">
|
||||
<input id="es-enabled" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
<label
|
||||
>Notify on completion
|
||||
<span class="label-hint">optional</span></label
|
||||
@@ -2859,9 +2863,11 @@
|
||||
</select>
|
||||
<label for="ep-priority">Priority</label>
|
||||
<input id="ep-priority" type="number" value="0" min="0" max="9999" />
|
||||
<label class="admin-checkbox"
|
||||
><input id="ep-enabled" type="checkbox" checked /> Enabled</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="ep-enabled" type="checkbox" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
<div class="modal-buttons">
|
||||
<button class="modal-cancel" onclick="hideEditPolicyModal()">
|
||||
Cancel
|
||||
@@ -2963,9 +2969,11 @@
|
||||
<textarea id="epp-content" rows="10" spellcheck="false"></textarea>
|
||||
<label for="epp-priority">Priority</label>
|
||||
<input id="epp-priority" type="number" value="0" min="0" max="9999" />
|
||||
<label class="admin-checkbox"
|
||||
><input id="epp-enabled" type="checkbox" checked /> Enabled</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="epp-enabled" type="checkbox" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
<div class="modal-buttons">
|
||||
<button class="modal-cancel" onclick="hideEditPromptPolicyModal()">
|
||||
Cancel
|
||||
@@ -3083,10 +3091,11 @@
|
||||
</option>
|
||||
<option value="search">Search — BM25 discoverable</option>
|
||||
</select>
|
||||
<label class="admin-checkbox"
|
||||
><input id="ctm-default" type="checkbox" /> Apply to new
|
||||
workstreams by default</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="ctm-default" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Apply to new workstreams by default</span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
<div class="skill-spec-col skill-spec-col-content">
|
||||
@@ -3189,10 +3198,11 @@
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<label class="admin-checkbox"
|
||||
><input id="csk-auto-approve" type="checkbox" /> Auto-approve all
|
||||
tools</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="csk-auto-approve" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Auto-approve all tools</span>
|
||||
</label>
|
||||
<label for="csk-allowed-tools"
|
||||
>Allowed Tools
|
||||
<span class="label-hint"
|
||||
@@ -3222,9 +3232,11 @@
|
||||
style="display: block; margin-top: 3px"
|
||||
>JSON array. Each: channel_type + channel_id or user_id</span
|
||||
>
|
||||
<label class="admin-checkbox"
|
||||
><input id="csk-enabled" type="checkbox" checked /> Enabled</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="csk-enabled" type="checkbox" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
</details>
|
||||
<details class="admin-details">
|
||||
<summary>
|
||||
@@ -3395,10 +3407,11 @@
|
||||
</option>
|
||||
<option value="search">Search — BM25 discoverable</option>
|
||||
</select>
|
||||
<label class="admin-checkbox"
|
||||
><input id="etm-default" type="checkbox" /> Apply to new
|
||||
workstreams by default</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="etm-default" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Apply to new workstreams by default</span>
|
||||
</label>
|
||||
</div>
|
||||
</div>
|
||||
<div class="skill-spec-col skill-spec-col-content">
|
||||
@@ -3480,10 +3493,11 @@
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<label class="admin-checkbox"
|
||||
><input id="esk-auto-approve" type="checkbox" /> Auto-approve all
|
||||
tools</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="esk-auto-approve" type="checkbox" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Auto-approve all tools</span>
|
||||
</label>
|
||||
<label for="esk-allowed-tools"
|
||||
>Allowed Tools
|
||||
<span class="label-hint"
|
||||
@@ -3513,9 +3527,11 @@
|
||||
style="display: block; margin-top: 3px"
|
||||
>JSON array. Each: channel_type + channel_id or user_id</span
|
||||
>
|
||||
<label class="admin-checkbox"
|
||||
><input id="esk-enabled" type="checkbox" checked /> Enabled</label
|
||||
>
|
||||
<label class="toggle-switch">
|
||||
<input id="esk-enabled" type="checkbox" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
</details>
|
||||
<div id="etm-scan-section" style="display: none" class="admin-field">
|
||||
<span class="admin-field-heading" id="etm-scan-heading"
|
||||
@@ -3699,43 +3715,48 @@
|
||||
<legend style="font-size: 12px; padding: 0 6px">
|
||||
Multitenant Authorization
|
||||
</legend>
|
||||
<label
|
||||
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
|
||||
>
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-none"
|
||||
value="none"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
style="margin-right: 6px"
|
||||
/>No authorization
|
||||
</label>
|
||||
<label
|
||||
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
|
||||
>
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-static"
|
||||
value="static"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
checked
|
||||
style="margin-right: 6px"
|
||||
/>Static headers (single shared identity)
|
||||
</label>
|
||||
<label
|
||||
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
|
||||
>
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-oauth"
|
||||
value="oauth_user"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
style="margin-right: 6px"
|
||||
/>Per-user OAuth 2.1 (recommended)
|
||||
</label>
|
||||
<div class="segmented-control" role="radiogroup" aria-label="Multitenant Authorization">
|
||||
<label class="segmented-option">
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-none"
|
||||
value="none"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
/>
|
||||
<span class="segmented-indicator" aria-hidden="true"></span>
|
||||
<span class="segmented-text">No authorization</span>
|
||||
</label>
|
||||
<label class="segmented-option">
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-static"
|
||||
value="static"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
checked
|
||||
/>
|
||||
<span class="segmented-indicator" aria-hidden="true"></span>
|
||||
<span class="segmented-text"
|
||||
>Static headers
|
||||
<span class="segmented-hint">single shared identity</span></span
|
||||
>
|
||||
</label>
|
||||
<label class="segmented-option">
|
||||
<input
|
||||
type="radio"
|
||||
name="mcp-auth-type"
|
||||
id="mcp-auth-oauth"
|
||||
value="oauth_user"
|
||||
onchange="toggleMcpAuthFields()"
|
||||
/>
|
||||
<span class="segmented-indicator" aria-hidden="true"></span>
|
||||
<span class="segmented-text"
|
||||
>Per-user OAuth 2.1
|
||||
<span class="segmented-hint">recommended</span></span
|
||||
>
|
||||
</label>
|
||||
</div>
|
||||
<div id="mcp-oauth-fields" style="display: none; margin-top: 8px">
|
||||
<label for="mcp-oauth-as-url"
|
||||
>Authorization Server URL
|
||||
@@ -3796,22 +3817,17 @@
|
||||
/>
|
||||
</div>
|
||||
</fieldset>
|
||||
<div style="display: flex; gap: 20px; margin-top: 14px">
|
||||
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
|
||||
><input
|
||||
type="checkbox"
|
||||
id="mcp-auto-approve"
|
||||
style="margin-right: 5px"
|
||||
/>Auto-approve tools</label
|
||||
>
|
||||
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
|
||||
><input
|
||||
type="checkbox"
|
||||
id="mcp-enabled"
|
||||
checked
|
||||
style="margin-right: 5px"
|
||||
/>Enabled</label
|
||||
>
|
||||
<div class="toggle-stack">
|
||||
<label class="toggle-switch">
|
||||
<input type="checkbox" id="mcp-auto-approve" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Auto-approve tools</span>
|
||||
</label>
|
||||
<label class="toggle-switch">
|
||||
<input type="checkbox" id="mcp-enabled" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Enabled</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="modal-buttons">
|
||||
<button class="modal-cancel" onclick="hideCreateMcpModal()">
|
||||
@@ -3936,6 +3952,14 @@
|
||||
<h2 id="model-create-title">Add Model</h2>
|
||||
<div id="model-create-error" role="alert" aria-live="assertive" aria-atomic="true"></div>
|
||||
<input type="hidden" id="model-edit-id" value="" />
|
||||
<label
|
||||
class="toggle-switch toggle--flush"
|
||||
title="Disable to hide this alias from every model dropdown (workstreams, schedules, channel adapters, role assignments) without removing the definition."
|
||||
>
|
||||
<input type="checkbox" id="model-enabled" checked />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Active</span>
|
||||
</label>
|
||||
<label for="model-alias">Alias</label>
|
||||
<input
|
||||
type="text"
|
||||
@@ -4116,15 +4140,27 @@
|
||||
placeholder='{"supports_vision": true}'
|
||||
style="font-family: var(--font-mono); font-size: 11px"
|
||||
></textarea>
|
||||
<div style="display: flex; gap: 20px; margin-top: 14px">
|
||||
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
|
||||
><input
|
||||
type="checkbox"
|
||||
id="model-enabled"
|
||||
checked
|
||||
style="margin-right: 5px"
|
||||
/>Enabled</label
|
||||
<div class="toggle-stack">
|
||||
<label
|
||||
class="toggle-switch"
|
||||
title="Surface stored reasoning text on /history responses (UI bubble on page reload). Storage of reasoning bytes is unaffected by this flag — they ride in provider_data regardless."
|
||||
>
|
||||
<input
|
||||
type="checkbox"
|
||||
id="model-surface-persisted-reasoning"
|
||||
checked
|
||||
/>
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Surface persisted reasoning</span>
|
||||
</label>
|
||||
<label
|
||||
class="toggle-switch"
|
||||
title="Replay stored reasoning blocks back to the model on subsequent provider calls. Capability-dependent; off by default for cost/spec compliance."
|
||||
>
|
||||
<input type="checkbox" id="model-replay-reasoning" />
|
||||
<span class="toggle-track" aria-hidden="true"></span>
|
||||
<span class="toggle-label">Replay reasoning to model</span>
|
||||
</label>
|
||||
</div>
|
||||
<div id="model-detect-area" style="margin-top: 14px">
|
||||
<button
|
||||
|
||||
@@ -1572,23 +1572,284 @@
|
||||
padding-left: 12px;
|
||||
}
|
||||
|
||||
/* Checkbox labels inside admin modals */
|
||||
.admin-modal label.admin-checkbox {
|
||||
/* Toggle switch — modern replacement for boolean checkboxes inside
|
||||
* admin modals. The native <input type="checkbox"> stays in the
|
||||
* markup (visually hidden but keyboard-focusable) so existing JS
|
||||
* that reads `.checked` keeps working; the .toggle-track + ::before
|
||||
* pseudo render the slider, .toggle-label carries the caption.
|
||||
*
|
||||
* Usage:
|
||||
* <label class="toggle-switch">
|
||||
* <input type="checkbox" id="..." />
|
||||
* <span class="toggle-track" aria-hidden="true"></span>
|
||||
* <span class="toggle-label">Enabled</span>
|
||||
* </label>
|
||||
*/
|
||||
/* Selector is doubled with .admin-modal so we win the specificity
|
||||
* battle against ``.admin-modal label`` (0,1,1) — without that, the
|
||||
* parent rule's display:block + text-transform:uppercase + margins
|
||||
* cascade and we lose the inline-flex layout.
|
||||
*
|
||||
* Default margin-top: 14px matches the .admin-modal label cadence
|
||||
* for toggles that sit directly between regular labelled rows
|
||||
* (schedule/policy/skill modals). When a toggle is inside an
|
||||
* explicit ``.toggle-stack`` flex container, the stack resets the
|
||||
* margin so the parent's gap controls spacing on its own. */
|
||||
.admin-modal label.toggle-switch,
|
||||
label.toggle-switch {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
margin: 14px 0 0 0;
|
||||
padding: 0;
|
||||
text-transform: none;
|
||||
letter-spacing: 0;
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
color: var(--fg);
|
||||
}
|
||||
.admin-modal .toggle-stack > label.toggle-switch,
|
||||
.toggle-stack > label.toggle-switch {
|
||||
margin: 0;
|
||||
}
|
||||
/* Modifier for toggles that should sit flush against the surrounding
|
||||
* rhythm rather than carry the default 14px top margin — used when a
|
||||
* toggle is the first row of a modal (under the h2) or when it lives
|
||||
* in a dynamically-rendered row that already supplies its own
|
||||
* spacing (e.g. judge bool settings). */
|
||||
.admin-modal label.toggle-switch.toggle--flush,
|
||||
label.toggle-switch.toggle--flush {
|
||||
margin-top: 0;
|
||||
}
|
||||
.admin-modal .toggle-stack,
|
||||
.toggle-stack {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
margin-top: 16px;
|
||||
}
|
||||
/* Hidden-input + track rules also need the .admin-modal prefix to
|
||||
* outrank ``.admin-modal input:not([type="hidden"])`` (same
|
||||
* specificity 0,2,1; that one comes later in source so without the
|
||||
* bump it wins and forces width:100% on the hidden input, popping it
|
||||
* back into the layout). */
|
||||
.admin-modal .toggle-switch input[type="checkbox"],
|
||||
.toggle-switch input[type="checkbox"] {
|
||||
/* Visually hidden but still focusable + click-targetable via the
|
||||
* <label> wrap. Avoids display:none, which would strip the input
|
||||
* from the keyboard tab order. */
|
||||
position: absolute;
|
||||
width: 1px;
|
||||
height: 1px;
|
||||
margin: -1px;
|
||||
padding: 0;
|
||||
border: 0;
|
||||
overflow: hidden;
|
||||
clip: rect(0 0 0 0);
|
||||
white-space: nowrap;
|
||||
}
|
||||
.admin-modal .toggle-switch .toggle-track,
|
||||
.toggle-switch .toggle-track {
|
||||
position: relative;
|
||||
flex: 0 0 auto;
|
||||
/* 40×22 meets WCAG 2.5.5 (Level AAA) target size when combined with
|
||||
* the label's hit area, and reads as a deliberate switch on touch
|
||||
* targets without dominating the form rhythm. */
|
||||
width: 40px;
|
||||
height: 22px;
|
||||
/* Off state: depressed inset on the modal background so the track
|
||||
* silhouette stays visible at 1.5+ contrast ratio. The earlier
|
||||
* ``var(--border-strong)`` solid fill was ~1.18:1 against the modal
|
||||
* surface — visible to most readers, invisible to anyone on a
|
||||
* glare-y screen or at ``prefers-contrast: more``. */
|
||||
background: var(--bg);
|
||||
box-shadow: inset 0 0 0 1px var(--border-strong);
|
||||
border-radius: 11px;
|
||||
transition:
|
||||
background 0.18s ease,
|
||||
box-shadow 0.18s ease;
|
||||
}
|
||||
.admin-modal .toggle-switch .toggle-track::before,
|
||||
.toggle-switch .toggle-track::before {
|
||||
content: "";
|
||||
position: absolute;
|
||||
top: 3px;
|
||||
left: 3px;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
background: var(--fg);
|
||||
border-radius: 50%;
|
||||
transition: transform 0.18s ease;
|
||||
}
|
||||
.admin-modal .toggle-switch input:checked + .toggle-track,
|
||||
.toggle-switch input:checked + .toggle-track {
|
||||
background: var(--accent);
|
||||
box-shadow: none;
|
||||
}
|
||||
.admin-modal .toggle-switch input:checked + .toggle-track::before,
|
||||
.toggle-switch input:checked + .toggle-track::before {
|
||||
transform: translateX(18px);
|
||||
background: var(--bg);
|
||||
}
|
||||
.admin-modal .toggle-switch input:focus-visible + .toggle-track,
|
||||
.toggle-switch input:focus-visible + .toggle-track {
|
||||
box-shadow: 0 0 0 3px var(--accent-dim);
|
||||
}
|
||||
.admin-modal .toggle-switch input:disabled + .toggle-track,
|
||||
.toggle-switch input:disabled + .toggle-track {
|
||||
opacity: 0.4;
|
||||
}
|
||||
.toggle-switch:has(input:disabled) {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.7;
|
||||
}
|
||||
.toggle-switch .toggle-label {
|
||||
display: inline-block;
|
||||
/* The label text rides at the modal's normal label cadence — caps
|
||||
* + letter-spacing so it sits next to the surrounding form rows
|
||||
* without shifting the visual rhythm. */
|
||||
font-family: var(--font-ui);
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
color: var(--fg-dim);
|
||||
}
|
||||
|
||||
/* Hair divider for separating conceptually-grouped toggles inside a
|
||||
* stack — used between the lone "Enabled" toggle and the paired
|
||||
* Reasoning toggles in the Add Model modal so the grouping reads
|
||||
* without needing a subheading or indent. Margin: 0 because the
|
||||
* .toggle-stack flex container already supplies a 10px gap on each
|
||||
* side; adding a margin on top would visually separate the divider
|
||||
* twice. */
|
||||
.admin-modal hr.toggle-group-divider,
|
||||
hr.toggle-group-divider {
|
||||
border: 0;
|
||||
border-top: 1px solid var(--border);
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
/* Segmented option list — vertical card group for radio choices that
|
||||
* benefit from a strong selected-state highlight. The native
|
||||
* ``<input type="radio">`` is visually hidden but stays focusable;
|
||||
* the ``.segmented-indicator`` pseudo-circle and the row's
|
||||
* background carry the selected state.
|
||||
*
|
||||
* Usage:
|
||||
* <div class="segmented-control" role="radiogroup">
|
||||
* <label class="segmented-option">
|
||||
* <input type="radio" name="..." value="..." />
|
||||
* <span class="segmented-indicator" aria-hidden="true"></span>
|
||||
* <span class="segmented-text">Label
|
||||
* <span class="segmented-hint">(hint)</span>
|
||||
* </span>
|
||||
* </label>
|
||||
* ...
|
||||
* </div>
|
||||
*/
|
||||
.admin-modal .segmented-control,
|
||||
.segmented-control {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
border: 1px solid var(--border-strong);
|
||||
border-radius: var(--radius-sm);
|
||||
overflow: hidden;
|
||||
background: var(--bg);
|
||||
margin-top: 8px;
|
||||
}
|
||||
.admin-modal .segmented-option,
|
||||
.segmented-option {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
font-size: 12px;
|
||||
gap: 10px;
|
||||
padding: 11px 14px;
|
||||
cursor: pointer;
|
||||
margin: 0;
|
||||
font-family: var(--font-ui);
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
text-transform: none;
|
||||
letter-spacing: 0;
|
||||
color: var(--fg);
|
||||
cursor: pointer;
|
||||
margin-top: 14px;
|
||||
color: var(--fg-dim);
|
||||
border-top: 1px solid var(--border);
|
||||
transition:
|
||||
background 0.15s ease,
|
||||
color 0.15s ease;
|
||||
}
|
||||
.admin-modal label.admin-checkbox input[type="checkbox"],
|
||||
.admin-modal label.admin-checkbox input[type="radio"] {
|
||||
width: auto;
|
||||
margin: 0;
|
||||
.segmented-option:first-of-type {
|
||||
border-top: none;
|
||||
}
|
||||
.segmented-option input[type="radio"] {
|
||||
/* Visually hidden but focusable + click-targetable via the label
|
||||
* wrap. Keyboard arrows still cycle within the radiogroup. */
|
||||
position: absolute;
|
||||
width: 1px;
|
||||
height: 1px;
|
||||
margin: -1px;
|
||||
padding: 0;
|
||||
border: 0;
|
||||
overflow: hidden;
|
||||
clip: rect(0 0 0 0);
|
||||
}
|
||||
.segmented-option .segmented-indicator {
|
||||
position: relative;
|
||||
flex: 0 0 auto;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
border-radius: 50%;
|
||||
background: var(--bg);
|
||||
box-shadow: inset 0 0 0 1px var(--border-strong);
|
||||
transition:
|
||||
background 0.15s ease,
|
||||
box-shadow 0.15s ease;
|
||||
}
|
||||
.segmented-option .segmented-indicator::after {
|
||||
content: "";
|
||||
position: absolute;
|
||||
top: 4px;
|
||||
left: 4px;
|
||||
width: 6px;
|
||||
height: 6px;
|
||||
border-radius: 50%;
|
||||
background: transparent;
|
||||
transition: background 0.15s ease;
|
||||
}
|
||||
.segmented-option:hover {
|
||||
background: var(--bg-highlight);
|
||||
color: var(--fg);
|
||||
}
|
||||
.segmented-option:has(input:checked) {
|
||||
background: var(--accent-dim);
|
||||
color: var(--fg);
|
||||
}
|
||||
.segmented-option:has(input:checked) .segmented-indicator {
|
||||
box-shadow: inset 0 0 0 1px var(--accent);
|
||||
}
|
||||
.segmented-option:has(input:checked) .segmented-indicator::after {
|
||||
background: var(--accent);
|
||||
}
|
||||
.segmented-option:has(input:focus-visible) {
|
||||
/* Bright accent (not --accent-dim) so the ring stays visible even
|
||||
* on the currently-selected row, which already paints --accent-dim
|
||||
* as its background. */
|
||||
box-shadow: inset 0 0 0 2px var(--accent);
|
||||
}
|
||||
.segmented-option .segmented-text {
|
||||
flex: 1 1 auto;
|
||||
min-width: 0;
|
||||
}
|
||||
.segmented-option .segmented-hint {
|
||||
font-size: 11px;
|
||||
font-weight: 400;
|
||||
color: var(--fg-dim);
|
||||
margin-left: 4px;
|
||||
}
|
||||
.segmented-option:has(input:checked) .segmented-hint {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
/* Admin modals */
|
||||
@@ -1692,9 +1953,6 @@
|
||||
border-color: var(--border);
|
||||
color: var(--fg-dim);
|
||||
}
|
||||
.admin-modal label.admin-checkbox input:disabled {
|
||||
opacity: 0.4;
|
||||
}
|
||||
.admin-modal input::placeholder,
|
||||
.admin-modal textarea::placeholder {
|
||||
color: var(--fg-dim);
|
||||
@@ -2755,28 +3013,64 @@ textarea.skill-content-area {
|
||||
padding: 0;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
/* Permission section grouping — keeps each namespace contiguous so
|
||||
* the row-flow grid below doesn't slice ``admin.*`` mid-column. The
|
||||
* caps-styled ``.perm-section-label`` re-anchors the toggles to the
|
||||
* modal's typographic system (matches ``.admin-modal label``: 10px
|
||||
* caps, 0.08em letter-spacing, fg-dim). */
|
||||
.perm-section {
|
||||
margin-top: 12px;
|
||||
}
|
||||
.perm-section:first-of-type {
|
||||
margin-top: 0;
|
||||
}
|
||||
.perm-section-label {
|
||||
font-family: var(--font-ui);
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
color: var(--fg-dim);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
.perm-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 4px 16px;
|
||||
padding: 8px 0;
|
||||
gap: 8px 16px;
|
||||
padding: 4px 0 0;
|
||||
}
|
||||
.perm-checkbox {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 11px;
|
||||
/* Permission rows inherit the toggle-switch component but render the
|
||||
* permission name in monospace + lower case (it's an identifier, not
|
||||
* a heading) — overrides the .toggle-label's caps/letter-spacing
|
||||
* cadence used elsewhere in admin modals. */
|
||||
.admin-modal .perm-grid label.toggle-switch.perm-toggle,
|
||||
.perm-grid label.toggle-switch.perm-toggle {
|
||||
margin: 0;
|
||||
}
|
||||
.perm-grid .toggle-switch.perm-toggle .toggle-label {
|
||||
font-family: var(--font-mono);
|
||||
color: var(--fg);
|
||||
padding: 3px 0;
|
||||
cursor: pointer;
|
||||
font-size: 11px;
|
||||
font-weight: 500;
|
||||
text-transform: none;
|
||||
letter-spacing: normal;
|
||||
color: var(--fg);
|
||||
}
|
||||
.perm-checkbox input[type="checkbox"] {
|
||||
width: auto;
|
||||
|
||||
/* User-role assignment list (Users tab → Manage roles modal). Same
|
||||
* toggle-switch component as elsewhere, but the labels are
|
||||
* human-readable role display names so we keep ui-font + caps-on so
|
||||
* the stack reads like a settings list. Only override the layout
|
||||
* margin (the modal already provides outer padding). */
|
||||
.admin-modal .user-roles-list,
|
||||
.user-roles-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 6px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
.admin-modal .user-roles-list label.toggle-switch.user-role-toggle,
|
||||
.user-roles-list label.toggle-switch.user-role-toggle {
|
||||
margin: 0;
|
||||
accent-color: var(--accent);
|
||||
}
|
||||
|
||||
/* ==========================================================================
|
||||
@@ -3308,6 +3602,28 @@ textarea.skill-content-area {
|
||||
opacity: 0.4;
|
||||
}
|
||||
|
||||
/* Phase 9 — last-refresh pill in the MCP status cell. Compact age +
|
||||
outcome indicator inline with the existing status text. Uses the
|
||||
semantic theme tokens (--bg-highlight, --fg-dim, --warn) defined in
|
||||
shared_static/base.css so the pill follows dark/light theme swaps. */
|
||||
.mcp-refresh-pill {
|
||||
display: inline-block;
|
||||
margin-left: 6px;
|
||||
padding: 0 4px;
|
||||
border-radius: var(--radius-sm);
|
||||
font-size: 0.8em;
|
||||
font-variant-numeric: tabular-nums;
|
||||
opacity: 0.85;
|
||||
}
|
||||
.mcp-refresh-pill-ok {
|
||||
background: var(--bg-highlight);
|
||||
color: var(--fg-dim);
|
||||
}
|
||||
.mcp-refresh-pill-err {
|
||||
background: color-mix(in srgb, var(--warn) 15%, transparent);
|
||||
color: var(--warn);
|
||||
}
|
||||
|
||||
.mcp-detail-modal::before {
|
||||
background: linear-gradient(
|
||||
90deg,
|
||||
|
||||
+15
-1
@@ -513,7 +513,21 @@ def is_public_path(path: str) -> bool:
|
||||
normalized = _strip_version_prefix(path)
|
||||
if normalized in PUBLIC_PATHS:
|
||||
return True
|
||||
return any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES)
|
||||
if any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
# Console proxy: a public proxied path is still public. Without this
|
||||
# the login/status/setup endpoints are unreachable from inside a
|
||||
# ``/node/{id}/...`` proxied page once the cookie expires — the
|
||||
# AuthMiddleware 401s the login POST before any handler runs and the
|
||||
# user is locked out of the proxied UI.
|
||||
if normalized.startswith("/node/"):
|
||||
proxied = _extract_proxied_path(normalized)
|
||||
if proxied is not None:
|
||||
if proxied in PUBLIC_PATHS:
|
||||
return True
|
||||
if any(proxied.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def required_scope(method: str, path: str) -> str:
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Per-workstream wakeup primitive for in-process child state-change subscribers.
|
||||
|
||||
Retires the polling pattern in ``CoordinatorClient.wait_for_workstream``,
|
||||
where the coord LLM's wait tool issued a storage snapshot every 0.5 s
|
||||
regardless of whether anything had changed. The dispatch path
|
||||
(:meth:`turnstone.console.coordinator_adapter.CoordinatorAdapter._dispatch_child_event`)
|
||||
now calls :meth:`ChildEventBus.notify` after each translated child event;
|
||||
waiters block on a per-call :class:`threading.Event` returned by
|
||||
:meth:`register_waiter` and re-read storage only when an event fires or
|
||||
the heartbeat cap expires.
|
||||
|
||||
Bus is in-process only. Cross-process / cross-node child events are
|
||||
already merged into ``_dispatch_child_event`` via the cluster collector's
|
||||
SSE multiplex before the bus sees them — there is no locality branching
|
||||
in the bus itself.
|
||||
|
||||
Design constraints:
|
||||
|
||||
- Waiter primitive is :class:`threading.Event` because the wait tool runs
|
||||
on the coordinator's sync worker thread, not an asyncio loop.
|
||||
- Concurrent ``register`` / ``unregister`` / ``notify`` is safe — a
|
||||
single ``threading.Lock`` guards the dict. ``Event.set`` itself is
|
||||
thread-safe and is called outside the lock so a slow waker can't block
|
||||
registration.
|
||||
- ``notify`` with no subscribers is a no-op (the steady state — most
|
||||
state-change events fire while no wait tool is active).
|
||||
- A waiter watching multiple ws_ids fires once on any of them; the
|
||||
caller's ``_snapshot_all`` re-read resolves which one changed.
|
||||
- Empty per-ws_id buckets are popped on unregister so long-lived buses
|
||||
don't accumulate dead keys after wait churn.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterable
|
||||
|
||||
|
||||
class ChildEventBus:
|
||||
"""Fan ``notify(child_ws_id)`` to every :class:`threading.Event`
|
||||
registered against that ws_id.
|
||||
|
||||
Use :meth:`register_waiter` once per wait call to obtain an Event,
|
||||
then call :meth:`unregister_waiter` in a ``finally`` so a crash mid-
|
||||
wait doesn't leak the registration. The dispatch side calls
|
||||
:meth:`notify` on every translated child state-change event; an
|
||||
empty bucket is a cheap dict lookup + immediate return.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._waiters: dict[str, set[threading.Event]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def register_waiter(self, child_ws_ids: Iterable[str]) -> threading.Event:
|
||||
"""Return a fresh Event registered against every listed ws_id.
|
||||
|
||||
A wait on ``[A, B, C]`` returns a single Event that fires when
|
||||
*any* of A/B/C changes. The caller's snapshot re-read resolves
|
||||
which one. Empty / falsy ids are silently skipped — callers that
|
||||
clean their input upstream (e.g. ``wait_for_workstream``'s
|
||||
dedup + cap) don't need to filter again here.
|
||||
"""
|
||||
event = threading.Event()
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
self._waiters.setdefault(wid, set()).add(event)
|
||||
return event
|
||||
|
||||
def unregister_waiter(
|
||||
self,
|
||||
child_ws_ids: Iterable[str],
|
||||
event: threading.Event,
|
||||
) -> None:
|
||||
"""Remove ``event`` from each listed ws_id's waiter set.
|
||||
|
||||
Idempotent — already-removed Events silently no-op. Pops empty
|
||||
sets so a long-lived bus doesn't accumulate dead keys after
|
||||
many waits have come and gone. Must be called from the same
|
||||
``finally`` that paired with :meth:`register_waiter` so a
|
||||
crash mid-wait doesn't leak the registration past one wait's
|
||||
lifetime.
|
||||
"""
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
bucket = self._waiters.get(wid)
|
||||
if bucket is None:
|
||||
continue
|
||||
bucket.discard(event)
|
||||
if not bucket:
|
||||
self._waiters.pop(wid, None)
|
||||
|
||||
def notify(self, child_ws_id: str) -> None:
|
||||
"""Wake every Event registered for ``child_ws_id``.
|
||||
|
||||
Called from the coord dispatch sink after each translated child
|
||||
event. Snapshot the bucket under the lock, then call
|
||||
``Event.set`` outside the lock so a slow waker doesn't block
|
||||
``register`` / ``unregister`` / further ``notify``. ``Event.set``
|
||||
is thread-safe and idempotent — re-firing a still-set Event is
|
||||
a no-op.
|
||||
|
||||
Empty / falsy ws_ids are silently dropped; the same hot path
|
||||
runs for every dispatched event regardless of whether anyone's
|
||||
waiting, so the empty-bucket case must stay cheap.
|
||||
"""
|
||||
if not child_ws_id:
|
||||
return
|
||||
with self._lock:
|
||||
bucket = self._waiters.get(child_ws_id)
|
||||
if not bucket:
|
||||
return
|
||||
events = list(bucket)
|
||||
for event in events:
|
||||
event.set()
|
||||
@@ -54,6 +54,27 @@ def set_config_path(path: str) -> None:
|
||||
_cache = None # invalidate cache so next load_config() re-reads
|
||||
|
||||
|
||||
def _warn_if_world_readable(cfg_path: Path) -> None:
|
||||
"""Warn once if config.toml is group- or world-readable.
|
||||
|
||||
DB passwords, OIDC client secrets, and TLS key paths live in this
|
||||
file — operators usually want it at 0600. POSIX-only; no-ops where
|
||||
``stat()`` modes are meaningless (Windows).
|
||||
"""
|
||||
try:
|
||||
mode = cfg_path.stat().st_mode & 0o777
|
||||
except OSError:
|
||||
return
|
||||
if mode & 0o077:
|
||||
log.warning(
|
||||
"%s is mode %04o (group/world-readable); secrets live here — "
|
||||
"run `chmod 0600 %s` to restrict access",
|
||||
cfg_path,
|
||||
mode,
|
||||
cfg_path,
|
||||
)
|
||||
|
||||
|
||||
def load_config(section: str | None = None) -> dict[str, Any]:
|
||||
"""Load config.toml and return the full dict or a specific section.
|
||||
|
||||
@@ -66,6 +87,7 @@ def load_config(section: str | None = None) -> dict[str, Any]:
|
||||
cfg_path = _resolve_config_path()
|
||||
if cfg_path.is_file():
|
||||
try:
|
||||
_warn_if_world_readable(cfg_path)
|
||||
_cache = tomllib.loads(cfg_path.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
log.warning("Failed to parse %s: %s", cfg_path, exc)
|
||||
@@ -143,6 +165,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
|
||||
"sslrootcert": "db_sslrootcert",
|
||||
"sslcert": "db_sslcert",
|
||||
"sslkey": "db_sslkey",
|
||||
"listen_url": "db_listen_url",
|
||||
},
|
||||
"judge": {
|
||||
"enabled": "judge_enabled",
|
||||
|
||||
@@ -19,7 +19,7 @@ either an async caller (via ``asyncio.to_thread``) or a sync hook.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.tool_advisory import (
|
||||
@@ -268,6 +268,161 @@ def extract_advisories_from_tool_envelope(
|
||||
return _entity_decode_wrapper_tags(inner), advisories
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.providers._protocol import LLMProvider
|
||||
|
||||
|
||||
def _make_provider_factory(module_path: str, class_name: str) -> Callable[[], LLMProvider]:
|
||||
"""Build a thread-unsafe lazy-init factory for a provider singleton.
|
||||
|
||||
Each block-type entry in ``_BLOCK_TYPE_PROVIDER_FACTORY`` closes
|
||||
over its own (module_path, class_name) pair. Adding a fourth
|
||||
provider is a single tuple in the dict, not a new 9-line getter.
|
||||
|
||||
Uses ``nonlocal`` instead of ``functools.lru_cache`` so the cache
|
||||
state stays inside this closure (lru_cache would attach state to
|
||||
the inner function object, which is correct but adds a per-call
|
||||
hash lookup on a bound key for what's effectively a single-slot
|
||||
cache).
|
||||
"""
|
||||
instance: LLMProvider | None = None
|
||||
|
||||
def factory() -> LLMProvider:
|
||||
nonlocal instance
|
||||
if instance is None:
|
||||
import importlib
|
||||
|
||||
module = importlib.import_module(module_path)
|
||||
instance = getattr(module, class_name)()
|
||||
return instance
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
# Block-type → provider factory. Routing is structural — block shape
|
||||
# is non-overlapping across providers by API design. Recognised
|
||||
# block types today:
|
||||
#
|
||||
# * ``"thinking"`` — Anthropic native (Phase 1). Walks the
|
||||
# ``thinking`` field on each block.
|
||||
# * ``"redacted_thinking"`` — Anthropic native (Phase 1). Anthropic's
|
||||
# safety system rewrites a thinking block into a sealed
|
||||
# ``redacted_thinking`` block; the Anthropic docs note these can
|
||||
# appear before, after, or interleaved with regular ``thinking``
|
||||
# blocks. Same factory: AnthropicProvider's extractor walks the
|
||||
# full block list and filters to ``type == "thinking"``, so the
|
||||
# redacted blocks are correctly skipped while the surrounding
|
||||
# real thinking text still surfaces.
|
||||
# * ``"reasoning"`` — OpenAI Responses native (Phase 3). Walks
|
||||
# ``summary[*].text`` (always present) and ``content[*].text``
|
||||
# (present when ``include=["reasoning.encrypted_content"]`` is
|
||||
# requested AND the response carries raw reasoning text).
|
||||
# * ``"reasoning_text"`` — synthetic, stamped by
|
||||
# ``ChatSession._maybe_synth_reasoning_block`` for Chat Completions
|
||||
# paths (vLLM, llama.cpp, Gemini-compat) where reasoning surfaces
|
||||
# only as ``reasoning_delta`` chunks with no native block shape.
|
||||
_anthropic_factory = _make_provider_factory(
|
||||
"turnstone.core.providers._anthropic", "AnthropicProvider"
|
||||
)
|
||||
_BLOCK_TYPE_PROVIDER_FACTORY: dict[str, Callable[[], LLMProvider]] = {
|
||||
"thinking": _anthropic_factory,
|
||||
"redacted_thinking": _anthropic_factory,
|
||||
"reasoning": _make_provider_factory(
|
||||
"turnstone.core.providers._openai_responses", "OpenAIResponsesProvider"
|
||||
),
|
||||
"reasoning_text": _make_provider_factory(
|
||||
"turnstone.core.providers._openai_chat", "OpenAIChatCompletionsProvider"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def extract_reasoning_text_from_provider_content(provider_content: Any) -> str:
|
||||
"""Dispatch reasoning extraction by scanning for a recognised block type.
|
||||
|
||||
Walks ``provider_content`` looking for the first block whose
|
||||
``type`` is in :data:`_BLOCK_TYPE_PROVIDER_FACTORY`, then dispatches
|
||||
the WHOLE list to that provider's ``extract_reasoning_text``. Each
|
||||
provider's extractor already filters internally by its own block
|
||||
type (Anthropic walks ``thinking``, OpenAI Responses walks
|
||||
``reasoning``, OpenAI Chat walks ``reasoning_text``), so passing
|
||||
the full list is correct — interleaved foreign blocks are ignored.
|
||||
|
||||
Returns ``""`` for empty / missing / non-list input or when no
|
||||
recognised reasoning-bearing block type appears anywhere in the
|
||||
list.
|
||||
|
||||
Why scan instead of just inspecting ``provider_content[0]``: the
|
||||
OpenAI Responses streaming layer captures EVERY ``output_item.done``
|
||||
event into ``provider_blocks`` (``_openai_responses.py:415-420``),
|
||||
not just reasoning items. In practice the order is usually
|
||||
``[reasoning, message, ...]`` but the API doesn't guarantee that —
|
||||
a hypothetical ``[message, reasoning]`` ordering would silently
|
||||
drop the reasoning under an index-only check. Same robustness
|
||||
point for Anthropic's hypothetical mixed-order outputs.
|
||||
|
||||
Pure transform — safe from any thread. Both history surfaces
|
||||
(interactive ``_build_history`` and lifted ``make_history_handler``)
|
||||
call this directly. See ``_BLOCK_TYPE_PROVIDER_FACTORY`` above
|
||||
for the recognised block types and the providers that own them.
|
||||
"""
|
||||
if not isinstance(provider_content, list) or not provider_content:
|
||||
return ""
|
||||
for block in provider_content:
|
||||
if not isinstance(block, dict):
|
||||
continue
|
||||
block_type = block.get("type")
|
||||
if not isinstance(block_type, str):
|
||||
continue
|
||||
factory = _BLOCK_TYPE_PROVIDER_FACTORY.get(block_type)
|
||||
if factory is not None:
|
||||
return factory().extract_reasoning_text(provider_content)
|
||||
return ""
|
||||
|
||||
|
||||
def extract_reasoning_for_history(
|
||||
messages: list[dict[str, Any]],
|
||||
surface_persisted_reasoning_flag: bool,
|
||||
) -> None:
|
||||
"""Surface stored reasoning text on each assistant message; strip the
|
||||
raw provider content from the wire payload.
|
||||
|
||||
For the ``make_history_handler`` REST path where the response
|
||||
payload IS the messages list returned from ``storage.load_messages``
|
||||
— both extraction source and stamp destination are the same dict.
|
||||
Walks *messages* in place: for every assistant message, dispatches
|
||||
via :func:`extract_reasoning_text_from_provider_content` and stamps
|
||||
``msg["reasoning"]`` when *surface_persisted_reasoning_flag* is True and the
|
||||
dispatcher returned non-empty text. Strips ``_provider_content``
|
||||
unconditionally — the field is internal and never read by either UI.
|
||||
|
||||
The interactive ``_build_history`` surface DOES NOT call this
|
||||
helper; it builds new entry dicts from scratch and calls
|
||||
:func:`extract_reasoning_text_from_provider_content` directly per
|
||||
assistant message, stamping ``entry["reasoning"]`` inline. The two
|
||||
surfaces converge on the same dispatcher; only the mutation shape
|
||||
differs.
|
||||
|
||||
Pure transform. Safe to call from ``asyncio.to_thread``.
|
||||
"""
|
||||
for msg in messages:
|
||||
if msg.get("role") != "assistant":
|
||||
continue
|
||||
provider_content = msg.get("_provider_content")
|
||||
# Always strip the internal lane before the wire payload leaves
|
||||
# the helper, even when surface_persisted_reasoning_flag is False or the
|
||||
# field is empty/missing. The strip is the contract; reasoning
|
||||
# surfacing is conditional on top of it.
|
||||
if "_provider_content" in msg:
|
||||
del msg["_provider_content"]
|
||||
if not surface_persisted_reasoning_flag:
|
||||
continue
|
||||
text = extract_reasoning_text_from_provider_content(provider_content)
|
||||
if text:
|
||||
msg["reasoning"] = text
|
||||
|
||||
|
||||
def decorate_history_messages(
|
||||
messages: list[dict[str, Any]],
|
||||
verdicts_by_call_id: dict[str, dict[str, Any]],
|
||||
|
||||
+20
-13
@@ -910,7 +910,17 @@ class IntentJudge:
|
||||
self._context_window = context_window
|
||||
self._rule_registry = rule_registry
|
||||
|
||||
# Resolve judge model via ModelRegistry alias, falling back to session
|
||||
# Resolve judge model via ModelRegistry alias, otherwise self-
|
||||
# consistency on the session model. ``judge.model`` is alias-only
|
||||
# — same contract as ``coordinator.model_alias`` /
|
||||
# ``model.plan_alias`` / ``model.task_alias``. A non-alias value
|
||||
# used to be accepted as a raw model id pinned onto the session
|
||||
# provider, but that path silently broke whenever the session
|
||||
# provider didn't speak that model id (e.g. coordinator on
|
||||
# Anthropic, ``judge.model = "gpt-5-mini"`` → every judge call
|
||||
# returned ``llm_fallback``). Operators register an alias
|
||||
# instead; an unknown value here logs a warning and inherits the
|
||||
# session model.
|
||||
resolved = False
|
||||
if config.model and model_registry is not None:
|
||||
try:
|
||||
@@ -928,18 +938,15 @@ class IntentJudge:
|
||||
except Exception:
|
||||
log.debug("Model alias resolution failed for %r, falling back", config.model)
|
||||
|
||||
if not resolved and config.model:
|
||||
# Model name override with session provider
|
||||
self._provider = session_provider
|
||||
self._client_factory_args = self._extract_client_config(
|
||||
session_client,
|
||||
session_provider.provider_name,
|
||||
)
|
||||
self._model = config.model
|
||||
caps = self._provider.get_capabilities(self._model)
|
||||
self._judge_context_window = caps.context_window
|
||||
elif not resolved:
|
||||
# Self-consistency: same model as session
|
||||
if not resolved:
|
||||
if config.model:
|
||||
log.warning(
|
||||
"judge.model=%r is not a registered alias — falling back to "
|
||||
"session model %r. Register the model in the Models tab and "
|
||||
"set judge.model to its alias.",
|
||||
config.model,
|
||||
session_model,
|
||||
)
|
||||
self._provider = session_provider
|
||||
self._client_factory_args = self._extract_client_config(
|
||||
session_client,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user