mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-13 07:22:24 -06:00
Compare commits
33 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c1281b9721 | |||
| 3124dbe52f | |||
| a8a1e738ca | |||
| 27768abecc | |||
| 1e5017293a | |||
| b2c688f3b1 | |||
| 09b1f07e18 | |||
| 1de8f6b4c7 | |||
| 4f89c1c3f3 | |||
| 6898860f18 | |||
| 3d07cad272 | |||
| 3737ddf89f | |||
| 464450b9e2 | |||
| 1fba80a9b5 | |||
| 28a5914be0 | |||
| bb2515eddf | |||
| 19978bfd89 | |||
| 8eec44d809 | |||
| f1cf516eb6 | |||
| 1df1e739ef | |||
| fbbb21012a | |||
| 617de2488f | |||
| 1d5189bb8d | |||
| 89a282f86b | |||
| 59c943b83a | |||
| 14b7516b3f | |||
| 5d0ec99449 | |||
| 6dfd1b5c18 | |||
| 658c65aee8 | |||
| 641ce8e7f6 | |||
| d76c57e687 | |||
| 58bf811a0e | |||
| df942e375e |
+523
-3
@@ -8,13 +8,533 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
|
||||
|
||||
Three release tracks are maintained:
|
||||
|
||||
- **`stable/1.0`** — patch-only (`v1.0.x`)
|
||||
- **`stable/1.3`** — patch-only (`v1.3.x`)
|
||||
- **`stable/1.4`** — patch-only (`v1.4.x`)
|
||||
- **`main`** — experimental (`v1.5.0aN`)
|
||||
- **`stable/1.5`** — patch-only (`v1.5.x`)
|
||||
- **`main`** — experimental (next major)
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.5.15]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Admin console blank-page on MCP server rows with consented users** — a
|
||||
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
|
||||
delimiters on the bulk-revoke button HTML literal, but the literal embeds
|
||||
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
|
||||
into bare tokens, and the resulting `SyntaxError` wiped out every global
|
||||
in `admin.js` — `showAdmin` and all other admin entry points became
|
||||
undefined, so the console UI was non-functional whenever the rendered MCP
|
||||
server list contained at least one row with `consented_users_count > 0`.
|
||||
Switch the literal to single-quote delimiters to match the surrounding
|
||||
block.
|
||||
|
||||
## [1.5.14]
|
||||
|
||||
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
|
||||
|
||||
### Added
|
||||
|
||||
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
|
||||
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
|
||||
dispatchers now upsert into a new `mcp_pending_consent` table on
|
||||
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
|
||||
(scheduled / channel) that hits an unconsented server surfaces the deferred
|
||||
prompt to the user on their next dashboard load via the gear-icon badge —
|
||||
rows are cleared automatically by the OAuth callback handler on consent
|
||||
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
|
||||
admin row gains a `consented_users_count` pill and a two-step-confirm
|
||||
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
|
||||
revoke is intentionally not attempted in bulk to avoid N synchronous
|
||||
round-trips against the provider). Operator-facing docs land at
|
||||
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
|
||||
|
||||
Introduces forward-only migrations `054_mcp_pending_consent` and
|
||||
`055_mcp_user_tokens_server_index`.
|
||||
|
||||
## [1.5.13]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
|
||||
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
|
||||
the dispatcher uses in-process fan-out).
|
||||
|
||||
### Added
|
||||
|
||||
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
|
||||
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
|
||||
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
|
||||
per-channel handlers on a separate dispatch thread. The cluster collector
|
||||
subscribes to a new `services` channel and reacts to node register /
|
||||
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
|
||||
loop; the 60 s loop is retained as the backstop for crash-shaped loss
|
||||
(NOTIFY only fires on real writes). The storage layer also gains a uniform
|
||||
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
|
||||
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
|
||||
`[database] listen_url` in `config.toml`) points the dispatcher at a
|
||||
direct-to-Postgres URL; defaults to the main DB URL when unset.
|
||||
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
|
||||
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
|
||||
whenever a child state change is dispatched to the UI, and the wait loop
|
||||
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
|
||||
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
|
||||
storage ~2400 times now wakes only on real state transitions, with ~4× lower
|
||||
SSE traffic in the quiescent case.
|
||||
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
|
||||
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
|
||||
route previously emitted only `memory.delete`, so tool-initiated mutations
|
||||
had no audit footprint). All emissions are best-effort and never break the
|
||||
tool call itself.
|
||||
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
|
||||
an optional `skill=<name>` argument that loads the named skill's content as
|
||||
the sub-agent's persona in place of the hardcoded identity statement. The
|
||||
fixed operating-guidance block (one-shot, tool-use over narration,
|
||||
no follow-up questions) is still layered on top of every persona. High- and
|
||||
critical-risk skills surface their risk tier in the approval header and
|
||||
emit a `task_agent.high_risk_skill` warning, matching the existing
|
||||
session-load gate.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
|
||||
back-compat `default` alias auto-synthesised by `load_model_registry`
|
||||
remained visible to the model even when an operator had configured
|
||||
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
|
||||
routed to whichever backend the synthesised alias was attached to at boot
|
||||
instead of the configured per-role default. The synthesised alias is now
|
||||
only added when neither the DB nor `[models.*]` populates the registry,
|
||||
filtered out of the LLM-visible alias list, and explicitly rejected at the
|
||||
validator chokepoint as defense-in-depth.
|
||||
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
|
||||
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
|
||||
node labels were re-entering the shape parser and producing
|
||||
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
|
||||
affected label forms (`|content|` and `ID[content]`) before the SVG cache
|
||||
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
|
||||
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
|
||||
highlights code blocks progressively as they stream rather than only after
|
||||
completion.
|
||||
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
|
||||
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
|
||||
POST went to `/v1/api/auth/login` and was rewritten to
|
||||
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
|
||||
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
|
||||
public path, and `proxy_api` would have forwarded the login request to the
|
||||
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
|
||||
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
|
||||
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
|
||||
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
|
||||
handlers. The dispatch table is a single `(method, path) → handler` mapping
|
||||
so the test parametrize list can't drift from the implementation.
|
||||
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
|
||||
overlay was covering the entire appbar, hiding the proxy-injected node
|
||||
picker. The overlay now starts at `top: 48px` and the dashboard's role
|
||||
downgrades from `dialog+aria-modal` to `region` so the appbar above it
|
||||
remains reachable. The gear icon converts from a direct settings-panel
|
||||
click into a dropdown with "MCP connections" and "Logout" (the latter with
|
||||
`.destructive` styling). The settings-menu keydown handler is now attached
|
||||
synchronously so `Escape` can't fall through the brief window between the
|
||||
menu opening and its listeners being installed.
|
||||
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
|
||||
`services_notify` trigger lives only in the alembic chain, but the test
|
||||
fixture creates tables via `metadata.create_all`. The trigger function +
|
||||
trigger are now declared in `_schema.py` and attached via
|
||||
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
|
||||
PostgreSQL dialect, with the same SQL constants imported by migration 053
|
||||
so there's a single source of truth.
|
||||
|
||||
## [1.5.12]
|
||||
|
||||
### Added
|
||||
|
||||
- **Enriched backend error messages** — provider name and attempted URL are now
|
||||
included in session error responses, so operators can triage connectivity
|
||||
failures without enabling debug logging.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
|
||||
had no messages remaining after a rewind the history event was skipped,
|
||||
leaving connected UIs with stale content and blocking edit-and-resend flows.
|
||||
|
||||
## [1.5.11]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`052_model_reasoning_persistence` — `surface_persisted_reasoning` and
|
||||
`replay_reasoning_to_model` flag columns on `model_definitions`.
|
||||
|
||||
### Added
|
||||
|
||||
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
|
||||
restore) now receive an `in_progress_snapshot` event carrying the buffered
|
||||
partial response, so the UI can resume rendering the in-flight turn without
|
||||
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
|
||||
counter so a reconnecting client can skip events it already saw.
|
||||
- **Reasoning persistence** (Phases 1–4) — model reasoning text can now be
|
||||
persisted to conversation history and optionally replayed to the model on
|
||||
subsequent turns. Phase 1 persists reasoning text on the history payload.
|
||||
Phase 2 wires a build-time shape filter and a per-model
|
||||
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
|
||||
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
|
||||
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
|
||||
new per-model capability flags (`surface_persisted_reasoning`,
|
||||
`replay_reasoning_to_model`) both default `False` on unknown and
|
||||
local-server models.
|
||||
- **Console home composer: placeholders + toggle** — the console landing-page
|
||||
composer now shows context-aware placeholder text and a toggle component for
|
||||
advanced options; an admin polish pass tightened spacing and focus behaviour
|
||||
across the form.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`judge.model` now requires a named alias** — raw provider model IDs on
|
||||
`judge.model` in config are no longer accepted; the judge must reference an
|
||||
alias registered in the model registry. The session-provider raw-model
|
||||
fallback is removed. Existing configs using an unregistered model ID need a
|
||||
corresponding alias entry.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
|
||||
flag for a model that does not declare reasoning-replay support now silently
|
||||
no-ops instead of forwarding reasoning blocks and triggering a provider error.
|
||||
- **Coordinator alias resolution unified across placeholder + factory** — a
|
||||
placeholder coordinator and the real coordinator factory could previously
|
||||
resolve to different model aliases, producing a visible mismatch in the model
|
||||
display. Both paths now share the same resolution logic.
|
||||
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
|
||||
under-initialised coordinator state no longer 500s when the models endpoint
|
||||
is hit before the coordinator subsystem is fully bootstrapped.
|
||||
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
|
||||
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
|
||||
counter that broke `state_change` / `in_progress_snapshot` ordering on
|
||||
reconnect.
|
||||
- **Reasoning persistence shape + replay fixes** — per-block
|
||||
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
|
||||
synthesised alongside non-reasoning `provider_blocks` so both appear
|
||||
together in the history payload.
|
||||
|
||||
## [1.5.10]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`051_skill_notify_on_complete_array_default` — backfills
|
||||
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Skills unlock action** — operators can unlock an installed skill to allow
|
||||
local customisation. Once unlocked, the skill's resource content, system
|
||||
prompt additions, and notify configuration are editable through the admin UI.
|
||||
Skills shipped as part of a bundle remain locked (read-only) until explicitly
|
||||
unlocked; the unlock is logged to the audit trail. A lock icon in the
|
||||
top-right of the Skills detail pane doubles as the unlock trigger.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`skills.sh` install endpoint** — the install script was targeting an
|
||||
endpoint removed in an earlier refactor; switched to `/api/download`.
|
||||
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
|
||||
(object) instead of `[]` (array), causing notify configurations to be
|
||||
rejected at schema validation.
|
||||
- **Skills admin UI modal errors** — `.is-visible` class used consistently
|
||||
instead of inline `style.display`; stale error text is cleared on submit;
|
||||
designer-review lock-icon UX applied.
|
||||
|
||||
## [1.5.9]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`repair=False` on all display-read `load_messages` call sites** —
|
||||
passing `repair=True` on display paths was silently mutating the stored
|
||||
message list, causing divergence between what the UI showed and what the
|
||||
model received on the next turn.
|
||||
|
||||
## [1.5.8]
|
||||
|
||||
This release introduces two forward-only schema migrations:
|
||||
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
|
||||
`050_conversations_source_and_reminders` — `_source` and `_reminders` columns
|
||||
on `conversations`.
|
||||
|
||||
### Added
|
||||
|
||||
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
|
||||
configured with a client ID and secret through the admin UI. The full token
|
||||
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
|
||||
stored encrypted at rest using a key derived from the JWT secret. The consent
|
||||
flow runs in-browser via a provider redirect. Rolled out in phases:
|
||||
|
||||
- Minimum admin form and OAuth schema (`21663d15`).
|
||||
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
|
||||
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
|
||||
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
|
||||
- SDK 401/403 introspection via httpx response hook (`bde09134`).
|
||||
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
|
||||
their OAuth token is permitted to call (`cfc8a6c8`).
|
||||
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
|
||||
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
|
||||
use of an OAuth-gated server and can revoke consent from their profile;
|
||||
admins see per-server consent counts in the MCP Servers tab (`61051339`).
|
||||
|
||||
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
|
||||
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
|
||||
that delivers at most one nudge per turn, preventing multi-channel pile-ups
|
||||
that inflate context. Observable changes:
|
||||
|
||||
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
|
||||
through to the system message so the model can reason about recency.
|
||||
- Coordinator idle-children observer: a coordinator with no in-flight
|
||||
children for longer than the configured idle threshold receives a nudge.
|
||||
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
|
||||
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
|
||||
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
|
||||
rather than the previous `_watch_pending` list, giving them the same
|
||||
delivery guarantees and priority handling as other advisories.
|
||||
|
||||
- **Structured watch-result card** — the UI renders watch results as a styled
|
||||
card with a system-nudge marker, distinct from the assistant message body.
|
||||
On history replay, system-nudge turns are visually distinguished from normal
|
||||
assistant turns.
|
||||
- **Side-channel persistence** — `_source` and `_reminders` side-channel
|
||||
fields are persisted to the `conversations` storage table and restored on
|
||||
session resume, so metacognitive context survives process restarts. A
|
||||
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Replay consistency** — queued user messages captured mid-loop are now
|
||||
persisted and replayed in the correct order on a subsequent `events`
|
||||
subscription. Coordinator history replay fixed: blank assistant cards and
|
||||
out-of-order tool results on the coordinator tree no longer occur when the
|
||||
coordinator has mixed queued + delivered messages.
|
||||
- **Session reminder preservation on fork + resume** — `_source` and
|
||||
`_reminders` are carried through workstream fork and restored from storage
|
||||
on resume.
|
||||
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
|
||||
columns; `_source` and `_reminders` now strip NUL bytes on write.
|
||||
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
|
||||
now committed atomically on first model add; startup teardown is offloaded
|
||||
to avoid blocking the event loop.
|
||||
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
|
||||
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
|
||||
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
|
||||
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
|
||||
now replaces the pool entry with a fresh session; the carrier token is
|
||||
owned by the pool entry to prevent a race between the 401 handler and a
|
||||
concurrent request.
|
||||
- **OIDC hardening** — multiple security and correctness fixes:
|
||||
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
|
||||
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
|
||||
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
|
||||
callback robustness — typed exceptions, shape checks, log sanitization, JS
|
||||
race (bug-4–6, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
|
||||
stranded-user self-heal on role-mapping failure (cumulative bug-1).
|
||||
|
||||
## [1.5.7]
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline node picker** — a compact node-switcher dropdown in the console
|
||||
header replaces the "← Back to console" banner, so operators can switch
|
||||
between nodes without a full navigation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Queued user messages injected mid-loop** — messages queued while a
|
||||
generation was in progress were not being delivered at the correct seam and
|
||||
could be dropped or reordered when the worker consumed the queue.
|
||||
- **Search tool output bounded** — pathological inputs (very long lines with
|
||||
no whitespace) could produce search results exceeding the context budget.
|
||||
Output is now clamped before reaching the message.
|
||||
|
||||
## [1.5.6]
|
||||
|
||||
### Added
|
||||
|
||||
- **`api_surface` toggle** — model definitions gain an `api_surface` field
|
||||
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
|
||||
the provider client uses. Enables Mistral Medium reasoning via the Responses
|
||||
surface; Chat Completions remains the default for all other models.
|
||||
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
|
||||
includes a `healthy_aliases` list per node, so the coordinator and operators
|
||||
can see which model aliases are currently reachable without a separate
|
||||
per-model health probe.
|
||||
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
|
||||
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
|
||||
configure per-kind reasoning effort and alias overrides from the UI rather
|
||||
than editing `config.toml`. Live-refresh dropdowns update in place when
|
||||
model definitions change.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
|
||||
query-aware candidate-set selection, dramatically improving recall for
|
||||
queries whose terms span multiple stored entries.
|
||||
- **Workstream model + config preserved on rehydrate** — reopening a closed
|
||||
workstream no longer overwrites the model alias and per-workstream config
|
||||
with session defaults.
|
||||
- **Console home composer: attachments + user-message pills** — multipart
|
||||
attachments in the home composer were not forwarded correctly; user-message
|
||||
pills in the coordinator chat pane were missing.
|
||||
|
||||
## [1.5.5]
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Saved-workstream tool result rendering** — tool results in closed
|
||||
workstreams were not rendering on history replay. Audit-trail decoration for
|
||||
tool calls is now applied on the replay path.
|
||||
|
||||
## [1.5.4]
|
||||
|
||||
### Added
|
||||
|
||||
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
|
||||
first-class citizens in the cluster event bus. `child_ws_state` events are
|
||||
pushed through the cluster SSE stream so the console tree view updates in
|
||||
real time without polling. `list_children` and `get_child` primitives on
|
||||
`SessionManager` provide a consistent cross-node view of the coordinator's
|
||||
spawn tree.
|
||||
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
|
||||
in the console admin panel now supports checkbox multi-select with a
|
||||
bulk-delete action.
|
||||
|
||||
## [1.5.3]
|
||||
|
||||
This release introduces one forward-only schema migration:
|
||||
`048_workstream_reaper_index` — partial composite index on `workstreams` for
|
||||
the orphan-reaper query.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
|
||||
`close_idle` pass now scopes the DB-orphan reaper by
|
||||
`services.last_heartbeat` so workstreams belonging to a live node are not
|
||||
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
|
||||
storage primitives added; a partial composite index keeps the reaper scan
|
||||
cheap.
|
||||
- **Coordinator pool idle cleanup** — a periodic task on the console now
|
||||
closes coordinator pool entries whose session has gone idle past the
|
||||
configurable threshold, preventing pool exhaustion on long-running consoles.
|
||||
|
||||
## [1.5.2]
|
||||
|
||||
### Added
|
||||
|
||||
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
|
||||
nudges are rendered as a distinct styled bubble rather than being injected
|
||||
inline into the assistant message, making it easier to distinguish model
|
||||
output from metacognitive annotations. The CLI REPL gains matching
|
||||
`on_user_reminder` / `on_tool_reminder` callbacks.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
|
||||
now fires correctly on the third repetition; a write-success-clear that
|
||||
reset the counter after a successful tool call (preventing streaks across
|
||||
mixed-outcome sequences) was removed.
|
||||
- **Metacog reminders isolated to side-channel** — reminder text no longer
|
||||
appears in the user content turn; it flows through a dedicated side-channel
|
||||
the session injects into the system context, preventing the model from
|
||||
attributing it to the user.
|
||||
|
||||
## [1.5.1]
|
||||
|
||||
### Added
|
||||
|
||||
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
|
||||
now receive the child's pending approval detail in `child_ws_state` events,
|
||||
enabling the coordinator to surface approval prompts without a separate poll.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Coordinator registry auto-refresh** — the console coordinator registry now
|
||||
refreshes when model definitions change, so a newly added alias is visible
|
||||
to coordinators without restarting.
|
||||
- **Coordinator fan-out default** — coordinators now fan out to independent
|
||||
child workstreams by default instead of serialising them, matching the
|
||||
documented contract for parallel-work patterns.
|
||||
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
|
||||
summaries and tool results from child workstreams were silently truncated at
|
||||
the previous 4 KiB cap.
|
||||
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
|
||||
polling now runs on a dedicated 200-thread executor, matching interactive's
|
||||
`sse_executor`, so coordinator long-poll blocking no longer contends with
|
||||
storage and routing workers on the default pool.
|
||||
|
||||
## [1.5.0]
|
||||
|
||||
User-visible additions: a unified workstream HTTP surface (interactive and
|
||||
coordinator under one URL family), inline child approvals, coordinator
|
||||
composer parity, progressive rendering, OIDC authentication, MCP OAuth
|
||||
foundations, and a redesigned UI built on the Design System v1 token layer.
|
||||
|
||||
This release removes the pre-1.5 body-keyed and query-keyed URL family.
|
||||
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
|
||||
|
||||
This release introduces the following forward-only schema migrations that the
|
||||
server applies automatically on first startup. All are additive; no data loss.
|
||||
|
||||
- `039_workstream_kind` — `kind` + `parent_ws_id` columns on `workstreams`.
|
||||
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
|
||||
`admin.cluster.inspect` to the builtin-admin role.
|
||||
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
|
||||
introduced by 039.
|
||||
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
|
||||
builtin-admin.
|
||||
- `043_skill_description_required` — backfills empty `description` rows in
|
||||
`prompt_templates`.
|
||||
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
|
||||
(`interactive` / `coordinator` / `any`).
|
||||
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status` →
|
||||
`risk_level`.
|
||||
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
|
||||
by rendezvous routing in 1.4.
|
||||
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
|
||||
rows removed from the coordinator in 1.5.0a4.
|
||||
|
||||
### Added
|
||||
|
||||
- **Inline child approvals** — pending tool approvals on coordinator child
|
||||
workstreams surface directly in the coordinator tree view. A risk pill shows
|
||||
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
|
||||
buttons appear inline so operators do not need to navigate to the child's
|
||||
workstream. `pending_approval_detail` is exposed on
|
||||
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
|
||||
so all connected clients render approval prompts simultaneously. LLM judge
|
||||
verdicts are cached client-side and replayed on SSE reconnect.
|
||||
- **Coordinator composer parity** — the coordinator composer now supports
|
||||
Stop, Send-to-queue, and Attach (file upload), matching the interactive
|
||||
workstream composer feature set.
|
||||
- **Per-call model and judge override on coordinator composer** — operators
|
||||
can override the model alias and judge model for a single coordinator send
|
||||
from the composer, without changing the node-wide or role-wide defaults. Bad
|
||||
aliases return a corrective error listing available choices.
|
||||
- **Coordinator status bar + richer history replay** — each coordinator
|
||||
workstream gains a per-coordinator status bar showing active children, token
|
||||
spend, and generation state. History replay in the coordinator panel is
|
||||
extended to include tool results and thinking blocks.
|
||||
- **Coordinator child error surfacing + memory tool** — child workstream
|
||||
errors are surfaced as distinct error rows in the coordinator tree view
|
||||
rather than disappearing silently. The coordinator gains access to a
|
||||
`memory` tool (same interface as interactive) for retrieving stored facts.
|
||||
- **Coordinator inline tool-batch construct** — the coordinator tool approval
|
||||
UI replaces the separate approval dock with an inline batch construct that
|
||||
groups all pending tool calls for a given turn into a single review card.
|
||||
- **Node capability auto-detection** — nodes report kernel-level capabilities
|
||||
(available memory, CPU count, accelerator presence) via
|
||||
`/v1/api/node/capabilities` at startup, enabling the console to filter model
|
||||
aliases offered to coordinators routing to that node.
|
||||
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
|
||||
a `SKILL.md` file's content into the modal auto-populates the name,
|
||||
description, and configuration fields.
|
||||
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
|
||||
soon as a complete diagram block is detected in the stream rather than
|
||||
waiting for the full response; the diagram re-renders in place as the model
|
||||
extends it.
|
||||
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
|
||||
math delimiters are now recognised alongside the existing `$$` fences.
|
||||
|
||||
### Removed (BREAKING — 1.5.0)
|
||||
|
||||
- **Legacy body-keyed and query-keyed URL family for the workstream
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ FROM python:3.14-slim
|
||||
LABEL org.opencontainers.image.title="turnstone" \
|
||||
org.opencontainers.image.description="Multi-node AI orchestration platform"
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.11.14 /uv /usr/local/bin/uv
|
||||
|
||||
# Remove the slim image's man page exclusion so man-db has actual content
|
||||
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
|
||||
|
||||
@@ -91,7 +91,7 @@ turnstone/
|
||||
discord/ Discord adapter (bot, cog, views, streaming, config)
|
||||
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
|
||||
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
|
||||
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
katex-0.16.46/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
ui/
|
||||
colors.py ANSI color constants with NO_COLOR support
|
||||
markdown.py Streaming terminal markdown renderer (line-buffered)
|
||||
|
||||
@@ -84,6 +84,7 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|
||||
|----------|---------|-------------|
|
||||
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
|
||||
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
|
||||
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
|
||||
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
|
||||
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
|
||||
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# MCP OAuth — per-user authorization for MCP servers
|
||||
|
||||
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
|
||||
|
||||
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
|
||||
|
||||
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
|
||||
|
||||
---
|
||||
|
||||
## When to use which `auth_type`
|
||||
|
||||
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
|
||||
|
||||
| `auth_type` | What it means | When to use |
|
||||
|---|---|---|
|
||||
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
|
||||
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
|
||||
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
|
||||
|
||||
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites for `auth_type=oauth_user`
|
||||
|
||||
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
|
||||
|
||||
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
|
||||
|
||||
3. **OAuth client registration**. Two paths:
|
||||
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
|
||||
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
|
||||
|
||||
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Per-server fields (admin UI)
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
|
||||
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
|
||||
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
|
||||
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
|
||||
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
|
||||
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
|
||||
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
|
||||
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
|
||||
|
||||
### Encryption key
|
||||
|
||||
```toml
|
||||
[security]
|
||||
mcp_token_encryption_key = "base64-fernet-key"
|
||||
# For rotation, list the keys in priority order — first is used for new
|
||||
# writes, all are tried for reads.
|
||||
# mcp_token_encryption_keys = ["new-key", "old-key"]
|
||||
```
|
||||
|
||||
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
|
||||
|
||||
---
|
||||
|
||||
## Lifecycle
|
||||
|
||||
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
|
||||
|
||||
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
|
||||
|
||||
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
|
||||
|
||||
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
|
||||
|
||||
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
|
||||
|
||||
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
|
||||
|
||||
---
|
||||
|
||||
## Admin status indicators
|
||||
|
||||
The MCP Servers admin tab shows per-server status pills (Phase 9):
|
||||
|
||||
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
|
||||
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
|
||||
|
||||
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
|
||||
|
||||
---
|
||||
|
||||
## Auth-type transitions
|
||||
|
||||
| From | To | What happens |
|
||||
|---|---|---|
|
||||
| `none` / `static` → `oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
|
||||
| `oauth_user` → `none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
|
||||
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
|
||||
|
||||
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---|---|---|
|
||||
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
|
||||
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
|
||||
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
|
||||
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
|
||||
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
|
||||
|
||||
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
|
||||
@@ -0,0 +1,29 @@
|
||||
# MCP OAuth in headless / scheduled / channel-driven runs
|
||||
|
||||
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
|
||||
|
||||
**Affected surfaces**:
|
||||
|
||||
- Scheduled workstreams (`turnstone-console` task scheduler).
|
||||
- Discord adapter runs.
|
||||
- Slack adapter runs.
|
||||
- Any future channel adapter without an interactive browser session.
|
||||
|
||||
**What happens when consent is missing**:
|
||||
|
||||
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
|
||||
|
||||
**Recovery**:
|
||||
|
||||
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
|
||||
|
||||
**Pre-consent recipe**:
|
||||
|
||||
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
|
||||
|
||||
1. Open the dashboard.
|
||||
2. Open the settings modal (gear icon).
|
||||
3. Click Connect on each MCP server the schedule will use.
|
||||
4. Confirm consent in the popup.
|
||||
|
||||
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
|
||||
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
|
||||
not use server-side prepared statements by default, so this is not an
|
||||
issue.
|
||||
|
||||
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
|
||||
transaction pooling assigns a real server connection only for the
|
||||
duration of each transaction, then returns it to the pool. PostgreSQL
|
||||
`LISTEN` is session state — a transaction-pooled client can't hold the
|
||||
multi-statement session a long-lived `LISTEN` needs. The console's
|
||||
`NotifyDispatcher` (reactive node discovery via the `services` channel)
|
||||
therefore opens a **dedicated, direct-to-Postgres** connection that
|
||||
bypasses PgBouncer.
|
||||
|
||||
Configure via `config.toml` `[database] listen_url` (preferred —
|
||||
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
|
||||
(config.toml wins when both are set). Defaults to the main DB URL when
|
||||
unset.
|
||||
|
||||
| Setting | Behaviour |
|
||||
|---|---|
|
||||
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
|
||||
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
|
||||
|
||||
Set this whenever PgBouncer is in transaction mode (the recommended
|
||||
setting per this doc). The override only adds one long-lived PG
|
||||
connection per console process — sized into the cluster's
|
||||
`max_connections` budget alongside the pool.
|
||||
|
||||
See also: [Docker deployment](docker.md) · [Security](security.md)
|
||||
|
||||
+3
-3
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "1.5.11"
|
||||
version = "1.5.16"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
@@ -82,9 +82,9 @@ include = [
|
||||
"turnstone/console/static/coordinator/*.js",
|
||||
"turnstone/shared_static/*.css",
|
||||
"turnstone/shared_static/*.js",
|
||||
"turnstone/shared_static/katex-0.16.45/**/*",
|
||||
"turnstone/shared_static/katex-0.16.46/**/*",
|
||||
"turnstone/shared_static/hljs-11.11.1/**/*",
|
||||
"turnstone/shared_static/mermaid-11.14.0/**/*",
|
||||
"turnstone/shared_static/mermaid-11.15.0/**/*",
|
||||
"turnstone/shared_static/hls-1.6.16/**/*",
|
||||
"turnstone/sdk/py.typed",
|
||||
"turnstone/deploy/*.yaml",
|
||||
|
||||
@@ -50,11 +50,14 @@ update_refs() {
|
||||
local old_pattern="$1" # e.g. katex-0.16.38
|
||||
local new_pattern="$2" # e.g. katex-0.16.39
|
||||
|
||||
# Find all files with version references (excludes vendored JS and worktrees)
|
||||
# Find all files with version references. Excludes the old versioned vendor
|
||||
# directory itself (about to be rm -rf'd anyway) so we don't bother rewriting
|
||||
# self-references inside it — but does NOT exclude all of shared_static/,
|
||||
# because shared_static/renderer.js loads the vendored libs and needs the bump.
|
||||
local files
|
||||
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' \
|
||||
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' --include='*.py' \
|
||||
-F "$old_pattern" . \
|
||||
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir='shared_static' \
|
||||
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir="$old_pattern" \
|
||||
2>/dev/null || true)
|
||||
for f in $files; do
|
||||
sed -i "s|${old_pattern}|${new_pattern}|g" "$f"
|
||||
|
||||
Generated
+127
-127
@@ -74,9 +74,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-project/types": {
|
||||
"version": "0.127.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
|
||||
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
|
||||
"version": "0.129.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.129.0.tgz",
|
||||
"integrity": "sha512-3oz8m3FGdr2nDXVqmFUw7jolKliC4MoyXYIG2c7gpjBnzUWQpUGIYcXYKxTdTi+N2jusvt610ckTMkxdwHkYEg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -84,9 +84,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-android-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-TWMZnRLMe63C2Lhyicviu7ZHaU4kxa6PS3rofvc9GmcvptzNN11BcfQ4Sl7MwTOsisQoa2keB/EBdNCAnUo8vA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -101,9 +101,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-6XcD+8k0gPVItNagEw78/qqcBDwKcwDYS8V2hRmVsfUSIrd8cWe/CBvRDI5toqFyPfj+FJr6t8U6Xj2P2prEew==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -118,9 +118,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0.tgz",
|
||||
"integrity": "sha512-iN/tWVXRQDWvmZlKdceP1Dwug9GDpEymhb9p4xnEe6zvCg5lFmzVljl+1qR1NVx3yfGpr2Na+CuLmv5IU8uzfQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -135,9 +135,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-freebsd-x64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0.tgz",
|
||||
"integrity": "sha512-jjQMDvvwSOuhOwMszD/klSOjyWMM3zI64hWTj9KT5x4MxRbZAf+7vLQ6qouRhtsLVFHr3f0ILaJAfgENPiQdAQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -152,9 +152,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0.tgz",
|
||||
"integrity": "sha512-d//Dtg2x6/m3mbV64yUGNnDGNZaDGRpDLLNGerHQUVObuNaIQaaDp25yUiqGXtHEXX+NP2d0wAlmKgpYgIAJ2A==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
@@ -169,9 +169,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-n7Ofp0mx+aB2cC+Sdy5YtMnXtY9lchnHbY+3Yt0uq9JsWQExf4f5Whu0tK0R8Jdc9S6RchTHjIFY7uc92puOVQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -189,9 +189,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0.tgz",
|
||||
"integrity": "sha512-EIVjy2cgd7uuMMo94FVkBp7F6DhcZAUwNURkSG3RwUmvAXR6s0ISxM81U+IydcZByPG0pZIHsf1b6kTxoFDgJA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -209,9 +209,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-JEwwOPcwTLAcpDQlqSmjEmfs63xJnSiUNIGvLcDLUHCWK4XowpS/7c7tUsUH6uT/ct6bMUTdXKfI8967FYj6mg==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
@@ -229,9 +229,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-s390x-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-0wjCFhLrihtAubnT9iA0N++0pSV0z5Hg7tNGdNJ4RFaINceHadoF+kiFGyY1qSSNVIAZtLotG8Ju1bgDPkjnFA==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
@@ -249,9 +249,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-gnu": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0.tgz",
|
||||
"integrity": "sha512-Dfn7iak9BcMMePxcoJfpSbWqnEyrp/dRF63/8qW/eHBdOZov6x5aShLLEYGYdIeSJ6vMLK/XCVB+lGIxm41bQA==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -269,9 +269,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-musl": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0.tgz",
|
||||
"integrity": "sha512-5/utzzDmD/pD/bmuaUcbTf/sZYy0aztwIVlfpoW1fTjCZ0BaPOMVWGZL1zvgxyi7ZIVYWlxKONHmSbHuiOh8Jw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -289,9 +289,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-openharmony-arm64": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0.tgz",
|
||||
"integrity": "sha512-ouJs8VcUomfLfpbUECqFMRqdV4x6aeAK3MA4m6vTrJJjKyWTV5KnxZx7Jd9G+GlDaQQxubcba00x16OyJ1meig==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -306,9 +306,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-wasm32-wasi": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0.tgz",
|
||||
"integrity": "sha512-E+oHKGiDA+lsKMmFtffDDw91EryDT7uJocrIuCHqhm6bCTM6xFK+3gaCkYOHfPwQr0cCNarSM2xaELoQDz9jJg==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
@@ -325,9 +325,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-arm64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0.tgz",
|
||||
"integrity": "sha512-yYK02n8Rngo+gbm1y6G0+7jk1sJ/2Wt7K0me0Y7k/ErBpyf+LJ2gFpqWVTcRV1rUepBlQRmpgWkTQCiiwrK0Ow==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -342,9 +342,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-x64-msvc": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0.tgz",
|
||||
"integrity": "sha512-14bpChMahXRRXiTwahSl+zzHPW6qQTXtkMuJBFlbo+pqSAews2d4BdCSHfrJ/MBsCZtpmTafsY+1QhBzitcmdg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -359,9 +359,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/pluginutils": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0.tgz",
|
||||
"integrity": "sha512-aKs/3GSWyV0mrhNmt/96/Z3yczC3yvrzYATCiCXQebBsGyYzjNdUphRVLeJQ67ySKVXRfMxt2lm12pmXvbPFQQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -402,23 +402,23 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@types/estree": {
|
||||
"version": "1.0.8",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
|
||||
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
|
||||
"version": "1.0.9",
|
||||
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
|
||||
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/@vitest/expect": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz",
|
||||
"integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.6.tgz",
|
||||
"integrity": "sha512-7EHDquPthALSV0jhhjgEW8FXaviMx7rSqu8W6oqCoAuOhKov814P99QDV1pxMA3QPv21YudvJngIhjrNI4opLg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@standard-schema/spec": "^1.1.0",
|
||||
"@types/chai": "^5.2.2",
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"chai": "^6.2.2",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
@@ -427,13 +427,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/mocker": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz",
|
||||
"integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.6.tgz",
|
||||
"integrity": "sha512-MCFc63czMjEInOlcY2cpQCvCN+KgbAn+60xu9cMgP4sKaLC5JNAKw7JH8QdAnoAC88hW1IiSNZ+GgVXlN1UcMQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"estree-walker": "^3.0.3",
|
||||
"magic-string": "^0.30.21"
|
||||
},
|
||||
@@ -454,9 +454,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/pretty-format": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz",
|
||||
"integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.6.tgz",
|
||||
"integrity": "sha512-h5SxD/IzNhZYnrSZRsUZQIC+vD0GY8cUvq0iwsmkFKixRCKLLWqCXa/FIQ4S1R+sI+PGoojkHsdNrbZiM9Qpgw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -467,13 +467,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/runner": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz",
|
||||
"integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.6.tgz",
|
||||
"integrity": "sha512-nOPCmn2+yD0ZNmKdsXGv/UxMMWbMuKeD6GyYncNwdkYDxpQvrPSKYj2rWuDjC2Y4b6w6hjip5dBKFzEUuZe3vA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"pathe": "^2.0.3"
|
||||
},
|
||||
"funding": {
|
||||
@@ -481,14 +481,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/snapshot": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz",
|
||||
"integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.6.tgz",
|
||||
"integrity": "sha512-YhsdE6xAVfTDmzjxL2ZDUvjj+ZsgyOKe+TdQzqkD72wIOmHka8NuGQ6NpTNZv9D2Z63fbwWKJPeVpEw4EQgYxw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"magic-string": "^0.30.21",
|
||||
"pathe": "^2.0.3"
|
||||
},
|
||||
@@ -497,9 +497,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/spy": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz",
|
||||
"integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.6.tgz",
|
||||
"integrity": "sha512-JFKxMx6udhwKh/Ldo270e17QX710vgunMkuPAvXjHSvC6oqLWAHhVhjg/I71q0u0CBSErIODV1Kjv0FQNSWjdg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -507,13 +507,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vitest/utils": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz",
|
||||
"integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.6.tgz",
|
||||
"integrity": "sha512-FxIY+U81R3LGKCxaHHFRQ5+g6/iRgGLmeHWdp2Amj4ljQRrEIWHmZyDfDYBRZlpyqA7qKxtS9DD1dhk8RnRIVQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"convert-source-map": "^2.0.0",
|
||||
"tinyrainbow": "^3.1.0"
|
||||
},
|
||||
@@ -959,9 +959,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.5.13",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.13.tgz",
|
||||
"integrity": "sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==",
|
||||
"version": "8.5.14",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
|
||||
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
|
||||
"dev": true,
|
||||
"funding": [
|
||||
{
|
||||
@@ -988,14 +988,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/rolldown": {
|
||||
"version": "1.0.0-rc.17",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
|
||||
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0.tgz",
|
||||
"integrity": "sha512-yD986aXDESFGS95spT1LAv0jssywP4npMEjmMHyN2/5+eE8qQJUype2AaKkRiLgBgyD0LFlubwAht7VmY8rGoA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@oxc-project/types": "=0.127.0",
|
||||
"@rolldown/pluginutils": "1.0.0-rc.17"
|
||||
"@oxc-project/types": "=0.129.0",
|
||||
"@rolldown/pluginutils": "1.0.0"
|
||||
},
|
||||
"bin": {
|
||||
"rolldown": "bin/cli.mjs"
|
||||
@@ -1004,21 +1004,21 @@
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
|
||||
"@rolldown/binding-android-arm64": "1.0.0",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/siginfo": {
|
||||
@@ -1119,16 +1119,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "8.0.10",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
|
||||
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
|
||||
"version": "8.0.12",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.12.tgz",
|
||||
"integrity": "sha512-w2dDofOWv2QB09ZITZBsvKTVAlYvPR4IAmrY/v0ir9KvLs0xybR7i48wxhM1/oyBWO34wPns+bPGw5ZrZqDpZg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.4",
|
||||
"postcss": "^8.5.10",
|
||||
"rolldown": "1.0.0-rc.17",
|
||||
"postcss": "^8.5.14",
|
||||
"rolldown": "1.0.0",
|
||||
"tinyglobby": "^0.2.16"
|
||||
},
|
||||
"bin": {
|
||||
@@ -1145,7 +1145,7 @@
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@types/node": "^20.19.0 || >=22.12.0",
|
||||
"@vitejs/devtools": "^0.1.0",
|
||||
"@vitejs/devtools": "^0.1.18",
|
||||
"esbuild": "^0.27.0 || ^0.28.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "^4.0.0",
|
||||
@@ -1197,19 +1197,19 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vitest": {
|
||||
"version": "4.1.5",
|
||||
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz",
|
||||
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
|
||||
"version": "4.1.6",
|
||||
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.6.tgz",
|
||||
"integrity": "sha512-6lvjbS3p9b4CrdCmguzbh2/4uoXhGE2q71R4OX5sqF9R1bo9Xd6fGrMAfvp5wnCzlBnFVdCOp6onuTQVbo8iUQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@vitest/expect": "4.1.5",
|
||||
"@vitest/mocker": "4.1.5",
|
||||
"@vitest/pretty-format": "4.1.5",
|
||||
"@vitest/runner": "4.1.5",
|
||||
"@vitest/snapshot": "4.1.5",
|
||||
"@vitest/spy": "4.1.5",
|
||||
"@vitest/utils": "4.1.5",
|
||||
"@vitest/expect": "4.1.6",
|
||||
"@vitest/mocker": "4.1.6",
|
||||
"@vitest/pretty-format": "4.1.6",
|
||||
"@vitest/runner": "4.1.6",
|
||||
"@vitest/snapshot": "4.1.6",
|
||||
"@vitest/spy": "4.1.6",
|
||||
"@vitest/utils": "4.1.6",
|
||||
"es-module-lexer": "^2.0.0",
|
||||
"expect-type": "^1.3.0",
|
||||
"magic-string": "^0.30.21",
|
||||
@@ -1237,12 +1237,12 @@
|
||||
"@edge-runtime/vm": "*",
|
||||
"@opentelemetry/api": "^1.9.0",
|
||||
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
|
||||
"@vitest/browser-playwright": "4.1.5",
|
||||
"@vitest/browser-preview": "4.1.5",
|
||||
"@vitest/browser-webdriverio": "4.1.5",
|
||||
"@vitest/coverage-istanbul": "4.1.5",
|
||||
"@vitest/coverage-v8": "4.1.5",
|
||||
"@vitest/ui": "4.1.5",
|
||||
"@vitest/browser-playwright": "4.1.6",
|
||||
"@vitest/browser-preview": "4.1.6",
|
||||
"@vitest/browser-webdriverio": "4.1.6",
|
||||
"@vitest/coverage-istanbul": "4.1.6",
|
||||
"@vitest/coverage-v8": "4.1.6",
|
||||
"@vitest/ui": "4.1.6",
|
||||
"happy-dom": "*",
|
||||
"jsdom": "*",
|
||||
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
|
||||
|
||||
+75
-4
@@ -340,7 +340,7 @@ def test_phase8_no_unsafe_dom_write_in_settings_panel() -> None:
|
||||
|
||||
|
||||
def test_phase8_settings_button_in_index_html() -> None:
|
||||
"""The gear-icon entry-point for the settings panel must remain
|
||||
"""The gear-icon entry-point for the settings menu must remain
|
||||
in the appbar's actions span. The console proxy IIFE prepends a
|
||||
node pill to ``header.firstChild`` (turnstone/console/server.py:
|
||||
202); our button is appended inside ``<span class='appbar-actions'>``
|
||||
@@ -351,9 +351,10 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
"index.html must keep the #settings-btn — onclick handlers "
|
||||
"and the consent badge target it by id."
|
||||
)
|
||||
assert 'onclick="openSettingsPanel()"' in body, (
|
||||
"settings-btn must wire onclick=openSettingsPanel() — losing "
|
||||
"the binding leaves the panel unreachable."
|
||||
assert 'onclick="toggleSettingsMenu(this)"' in body, (
|
||||
"settings-btn must wire onclick=toggleSettingsMenu(this) — "
|
||||
"the gear opens a dropdown with MCP connections + Logout; "
|
||||
"losing the binding leaves the menu unreachable."
|
||||
)
|
||||
# The button must live inside <span class="appbar-actions"> so the
|
||||
# console proxy's header.insertBefore(pill, header.firstChild)
|
||||
@@ -366,6 +367,76 @@ def test_phase8_settings_button_in_index_html() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_settings_menu_handlers_defined() -> None:
|
||||
"""The gear-icon dropdown exposes a toggle/open/close trio that the
|
||||
inline ``onclick="toggleSettingsMenu(this)"`` in index.html depends
|
||||
on, plus the menu items themselves must wire to existing entry
|
||||
points (``openSettingsPanel`` for MCP connections, ``logout`` for
|
||||
sign-out). Pin all four so a rename or deletion fails loudly here
|
||||
instead of silently leaving the gear's menu broken or wired to a
|
||||
stale function."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
for name in [
|
||||
"function toggleSettingsMenu",
|
||||
"function openSettingsMenu",
|
||||
"function closeSettingsMenu",
|
||||
]:
|
||||
assert name in body, f"Missing required handler: {name}"
|
||||
# Bound to the settings-menu region so we don't accidentally match
|
||||
# an unrelated openSettingsPanel/logout call elsewhere in the file.
|
||||
start = body.index("function openSettingsMenu(")
|
||||
end = body.index("function closeSettingsMenu(", start)
|
||||
section = body[start:end]
|
||||
assert "openSettingsPanel()" in section, (
|
||||
"Settings menu's MCP-connections item must call openSettingsPanel() "
|
||||
"— otherwise the existing settings overlay is unreachable from the "
|
||||
"new dropdown."
|
||||
)
|
||||
assert "logout()" in section, (
|
||||
"Settings menu's Logout item must call logout() — that's the "
|
||||
"shared auth.js entry point that clears the cookie + session state."
|
||||
)
|
||||
|
||||
|
||||
def test_dashboard_overlay_is_region_not_dialog() -> None:
|
||||
"""The dashboard overlay must be role='region' (not role='dialog' +
|
||||
aria-modal='true'). The role downgrade is what allows ui-header to
|
||||
stay interactive while the dashboard is open — see the comment at
|
||||
showDashboard() in app.js. A revert to role='dialog' + aria-modal
|
||||
would re-trap focus and break the gear/theme buttons + the console
|
||||
proxy's node-picker pill while the dashboard is open."""
|
||||
body = _INDEX_HTML.read_text(encoding="utf-8")
|
||||
idx = body.index('id="dashboard"')
|
||||
# Bound to ~600 chars after the tag so we only check this element's
|
||||
# attributes — same shape as test_phase8_settings_modal_in_index_html.
|
||||
chunk = body[idx : idx + 600]
|
||||
assert 'role="region"' in chunk, (
|
||||
"dashboard must be role='region' — see showDashboard() comment."
|
||||
)
|
||||
assert "aria-modal" not in chunk, (
|
||||
"dashboard must NOT be aria-modal — re-trapping focus breaks "
|
||||
"the appbar's interactive controls (theme toggle, settings menu, "
|
||||
"proxy node-picker pill) while the dashboard is open."
|
||||
)
|
||||
|
||||
|
||||
def test_close_settings_menu_resets_aria() -> None:
|
||||
"""closeSettingsMenu must reset aria-expanded='false' AND remove
|
||||
aria-controls from the gear trigger. Without the reset the gear
|
||||
keeps reporting 'expanded' to assistive tech after the menu closes;
|
||||
without the removal aria-controls points at a dead DOM id."""
|
||||
body = _APP_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function closeSettingsMenu(")
|
||||
# Bound to ~600 chars so we don't catch unrelated handlers.
|
||||
section = body[start : start + 600]
|
||||
assert 'setAttribute("aria-expanded", "false")' in section, (
|
||||
"closeSettingsMenu must set aria-expanded='false' on the gear."
|
||||
)
|
||||
assert 'removeAttribute("aria-controls")' in section, (
|
||||
"closeSettingsMenu must remove aria-controls from the gear."
|
||||
)
|
||||
|
||||
|
||||
def test_phase8_settings_modal_in_index_html() -> None:
|
||||
"""Both the settings overlay and the revoke-confirmation overlay
|
||||
must remain in the modal area. The Escape-key deferral list in
|
||||
|
||||
@@ -83,6 +83,39 @@ class TestIsPublicPath:
|
||||
def test_shared_static_public(self):
|
||||
assert is_public_path("/shared/base.css") is True
|
||||
|
||||
# Console proxy: a public proxied path must still be public, otherwise
|
||||
# the login modal can never re-authenticate from inside a ``/node/{id}/``
|
||||
# proxied page once the cookie expires.
|
||||
def test_proxy_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/login") is True
|
||||
|
||||
def test_proxy_no_v1_login_public(self):
|
||||
assert is_public_path("/node/node-a/api/auth/login") is True
|
||||
|
||||
def test_proxy_v1_status_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/status") is True
|
||||
|
||||
def test_proxy_v1_setup_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
|
||||
|
||||
def test_proxy_v1_logout_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
|
||||
|
||||
def test_proxy_v1_oidc_authorize_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
|
||||
|
||||
def test_proxy_v1_oidc_callback_public(self):
|
||||
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
|
||||
|
||||
def test_proxy_v1_workstreams_still_not_public(self):
|
||||
"""Proxy prefix must not turn protected paths into public ones."""
|
||||
assert is_public_path("/node/node-a/v1/api/workstreams") is False
|
||||
|
||||
def test_proxy_v1_refresh_still_requires_auth(self):
|
||||
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
|
||||
a valid cookie. Proxy-prefix shouldn't change that."""
|
||||
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestRequiredRole
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""ChatSession interactivity flag tests (Phase 9).
|
||||
|
||||
Validates that ``ChatSession._is_interactive_for_consent`` is computed
|
||||
correctly from ``client_type`` on construction. This is the front of
|
||||
the Phase 9 plumb-through: the flag flows from here to
|
||||
``_dispatch_pool_sync`` to the structured-error → pending-consent
|
||||
write path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from tests._session_helpers import make_session
|
||||
from turnstone.prompts import INTERACTIVE_CONSENT_CLIENT_TYPES, ClientType
|
||||
|
||||
|
||||
def test_web_is_interactive() -> None:
|
||||
s = make_session(client_type=ClientType.WEB)
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_cli_is_interactive() -> None:
|
||||
s = make_session(client_type=ClientType.CLI)
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_chat_is_not_interactive() -> None:
|
||||
# Discord / Slack adapters cannot drive a browser redirect from
|
||||
# inside the channel — consent prompts must be deferred to the
|
||||
# dashboard badge.
|
||||
s = make_session(client_type=ClientType.CHAT)
|
||||
assert s._is_interactive_for_consent is False
|
||||
|
||||
|
||||
def test_scheduled_is_not_interactive() -> None:
|
||||
# The scheduler runs autonomously; the user isn't online to
|
||||
# complete the OAuth redirect.
|
||||
s = make_session(client_type=ClientType.SCHEDULED)
|
||||
assert s._is_interactive_for_consent is False
|
||||
|
||||
|
||||
def test_interactive_set_matches_module_constant() -> None:
|
||||
# Pin the module-level frozenset against the flag computation —
|
||||
# a future reorganisation that drifts the set vs the per-session
|
||||
# logic would silently break the gating.
|
||||
for ct in ClientType:
|
||||
s = make_session(client_type=ct)
|
||||
assert s._is_interactive_for_consent == (ct in INTERACTIVE_CONSENT_CLIENT_TYPES), ct
|
||||
|
||||
|
||||
def test_default_client_type_is_cli_interactive() -> None:
|
||||
# Defaults preserved — make_session uses ChatSession's default
|
||||
# which is CLI. Sanity check that the default user experience
|
||||
# stays interactive-for-consent.
|
||||
s = make_session()
|
||||
assert s._client_type == ClientType.CLI
|
||||
assert s._is_interactive_for_consent is True
|
||||
|
||||
|
||||
def test_scheduled_env_file_exists() -> None:
|
||||
"""The SCHEDULED env module must exist; otherwise
|
||||
``compose_system_message`` for a scheduled session would 500."""
|
||||
from turnstone.prompts import _load
|
||||
|
||||
text = _load("env/scheduled.md")
|
||||
assert "Output Environment" in text
|
||||
assert "consent" in text.lower()
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
|
||||
|
||||
The bus is the in-process wakeup primitive for ``wait_for_workstream``
|
||||
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
|
||||
of ws_id → set[threading.Event] under a lock — focused tests for
|
||||
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
|
||||
multi-child waiter, and concurrent register/notify (smoke). End-to-end
|
||||
integration with the dispatch sink lives in
|
||||
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
|
||||
def test_register_returns_event_that_starts_unset() -> None:
|
||||
"""A waiter must not see leftover state from before it registered —
|
||||
a fresh wait should always block until the first notify."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert isinstance(event, threading.Event)
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
|
||||
"""Different ws_ids must keep independent waiter sets — a notify on
|
||||
a stranger ws can't wake the wait or the bus stops being keyed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify("ws-other")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_notify_with_no_subscribers_is_noop() -> None:
|
||||
"""The dispatch sink calls notify on every translated event; the
|
||||
steady state has no wait tool active. Must not raise."""
|
||||
bus = ChildEventBus()
|
||||
bus.notify("ws-nobody-cares") # no exception
|
||||
|
||||
|
||||
def test_multi_waiter_each_gets_independent_event() -> None:
|
||||
"""Two waits on the same ws_id must wake independently — clearing
|
||||
one Event must not silence the other."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
assert e1 is not e2
|
||||
bus.notify("ws-1")
|
||||
assert e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
|
||||
"""A wait on [A, B, C] returns a single Event registered against
|
||||
all three. Notify on ANY of A/B/C must wake the wait — the
|
||||
caller's snapshot re-read disambiguates which one changed."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
|
||||
bus.notify("ws-b")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
|
||||
"""After unregister, notify on any of the previously-watched ws_ids
|
||||
must NOT wake the Event — leaks would mean every future notify on
|
||||
that ws_id wakes a long-dead wait."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-a", "ws-b"])
|
||||
bus.unregister_waiter(["ws-a", "ws-b"], event)
|
||||
bus.notify("ws-a")
|
||||
bus.notify("ws-b")
|
||||
assert not event.is_set()
|
||||
|
||||
|
||||
def test_unregister_is_idempotent() -> None:
|
||||
"""A double-unregister must silently no-op — finally blocks may
|
||||
run twice in odd shutdown paths, the bus must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
bus.unregister_waiter(["ws-1"], event) # no exception
|
||||
|
||||
|
||||
def test_unregister_pops_empty_buckets() -> None:
|
||||
"""Empty per-ws_id buckets must be popped so a long-lived bus
|
||||
doesn't accumulate dead keys after many waits have churned through.
|
||||
Reaches into the private state — the property is structural, not
|
||||
behavioral, so the assertion is also."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
assert "ws-1" in bus._waiters
|
||||
bus.unregister_waiter(["ws-1"], event)
|
||||
assert "ws-1" not in bus._waiters
|
||||
|
||||
|
||||
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
|
||||
"""Removing one waiter from a multi-waiter bucket must not drop
|
||||
the others — popping the bucket would silently disable notifies
|
||||
for every concurrent wait on the same ws_id."""
|
||||
bus = ChildEventBus()
|
||||
e1 = bus.register_waiter(["ws-1"])
|
||||
e2 = bus.register_waiter(["ws-1"])
|
||||
bus.unregister_waiter(["ws-1"], e1)
|
||||
bus.notify("ws-1")
|
||||
assert not e1.is_set()
|
||||
assert e2.is_set()
|
||||
|
||||
|
||||
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
|
||||
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
|
||||
is reachable from other callers in future use; falsy ids should be
|
||||
silently dropped, not registered against an empty-string key."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["", "ws-1", ""])
|
||||
# Only the real ws_id should bucket the waiter.
|
||||
assert list(bus._waiters.keys()) == ["ws-1"]
|
||||
bus.notify("") # no crash, no spurious wake
|
||||
assert not event.is_set()
|
||||
bus.notify("ws-1")
|
||||
assert event.is_set()
|
||||
|
||||
|
||||
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
|
||||
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
|
||||
must return promptly after a notify on a watched ws_id. This is
|
||||
the property that retires the 0.5s polling cadence."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
woken_at = [0.0]
|
||||
|
||||
def _waiter() -> None:
|
||||
event.wait(timeout=2.0)
|
||||
woken_at[0] = time.monotonic()
|
||||
|
||||
t = threading.Thread(target=_waiter, daemon=True)
|
||||
t.start()
|
||||
# Give the waiter a beat to enter Event.wait, then notify.
|
||||
time.sleep(0.05)
|
||||
notified_at = time.monotonic()
|
||||
bus.notify("ws-1")
|
||||
t.join(timeout=1.0)
|
||||
assert not t.is_alive(), "waiter did not wake within 1s of notify"
|
||||
# Latency budget is generous; the contract is "well under the legacy
|
||||
# 0.5s poll cadence", not microsecond timing.
|
||||
assert woken_at[0] - notified_at < 0.2
|
||||
|
||||
|
||||
def test_clear_before_check_race_does_not_lose_wake() -> None:
|
||||
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
|
||||
A notify between clear and wait must leave the Event set, so the
|
||||
next wait returns immediately and the loop re-snapshots. Same
|
||||
standard subscribe/check race the wait loop guards against."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
# Simulate wait-loop ordering: clear, then notify "between" clear
|
||||
# and the next wait.
|
||||
event.clear()
|
||||
bus.notify("ws-1")
|
||||
# The next wait must return True immediately (set is sticky until
|
||||
# the next clear).
|
||||
assert event.wait(timeout=0.1) is True
|
||||
|
||||
|
||||
def test_concurrent_register_and_notify_is_safe() -> None:
|
||||
"""Smoke test: many threads registering / notifying / unregistering
|
||||
in parallel must not raise or deadlock. Doesn't assert specific
|
||||
interleavings — only structural safety of the lock discipline."""
|
||||
bus = ChildEventBus()
|
||||
stop = threading.Event()
|
||||
errors: list[BaseException] = []
|
||||
|
||||
def _worker(ws_id: str) -> None:
|
||||
try:
|
||||
for _ in range(200):
|
||||
if stop.is_set():
|
||||
return
|
||||
ev = bus.register_waiter([ws_id])
|
||||
bus.notify(ws_id)
|
||||
bus.unregister_waiter([ws_id], ev)
|
||||
except BaseException as e: # noqa: BLE001
|
||||
errors.append(e)
|
||||
|
||||
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
stop.set()
|
||||
assert not errors, f"worker threads raised: {errors!r}"
|
||||
# All buckets should have been popped (every register paired with
|
||||
# unregister).
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ws_id", ["", None])
|
||||
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
|
||||
"""Defensive: the dispatch sink already guards against empty
|
||||
ws_ids, but a falsy slip-through must not raise."""
|
||||
bus = ChildEventBus()
|
||||
event = bus.register_waiter(["ws-1"])
|
||||
bus.notify(ws_id) # type: ignore[arg-type]
|
||||
assert not event.is_set()
|
||||
@@ -3,11 +3,13 @@
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.console.collector import ClusterCollector, NodeSnapshot
|
||||
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
|
||||
|
||||
# Shared test auth — JWT-based
|
||||
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
|
||||
@@ -149,6 +151,78 @@ class TestCollectorDiscovery:
|
||||
assert c._nodes["node-a"].started == 1234567890.0
|
||||
|
||||
|
||||
class TestCollectorNotifyWireIn:
|
||||
"""NotifyDispatcher-driven discovery — reactive node visibility."""
|
||||
|
||||
def test_start_subscribes_to_services_channel(self):
|
||||
# Stub dispatcher records subscriptions without spawning threads.
|
||||
class _StubDispatcher:
|
||||
def __init__(self):
|
||||
self.subscriptions: list[tuple[str, Any]] = []
|
||||
|
||||
def subscribe(self, channel, handler):
|
||||
self.subscriptions.append((channel, handler))
|
||||
return lambda: None
|
||||
|
||||
stub = _StubDispatcher()
|
||||
storage = MockStorage()
|
||||
c = ClusterCollector(
|
||||
storage=storage,
|
||||
discovery_interval=999,
|
||||
notify_dispatcher=stub,
|
||||
)
|
||||
try:
|
||||
c.start()
|
||||
assert len(stub.subscriptions) == 1
|
||||
channel, handler = stub.subscriptions[0]
|
||||
assert channel == "services"
|
||||
assert handler == c._on_services_notify
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_no_dispatcher_means_no_subscribe(self):
|
||||
# Collector without a dispatcher (single-node / SQLite dev) just
|
||||
# falls back to the 60 s discovery-loop polling — no error.
|
||||
c = _make_collector(MockStorage())
|
||||
try:
|
||||
c.start()
|
||||
assert c._notify_unsubscribe is None
|
||||
finally:
|
||||
c.stop()
|
||||
|
||||
def test_on_notify_runs_discovery(self):
|
||||
# Construct a synthetic Notify and invoke the handler directly —
|
||||
# asserts the wire-in delegates back to ``_discover_nodes``.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
c = _make_collector(storage)
|
||||
c._running = True # bypass start() so we don't spawn threads
|
||||
q: queue.Queue[dict[str, Any]] = queue.Queue()
|
||||
c.register_listener(q)
|
||||
|
||||
storage.services = [
|
||||
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
|
||||
]
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
|
||||
event = q.get_nowait()
|
||||
assert event["type"] == "node_joined"
|
||||
assert event["node_id"] == "node-z"
|
||||
|
||||
def test_on_notify_when_not_running_is_noop(self):
|
||||
# If a stray notify arrives after stop, the handler doesn't run
|
||||
# discovery on a half-torn-down collector.
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
storage = MockStorage()
|
||||
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
|
||||
c = _make_collector(storage)
|
||||
# _running stays False (never called start()).
|
||||
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
|
||||
assert c.get_overview()["nodes"] == 0
|
||||
|
||||
|
||||
class TestCollectorSnapshot:
|
||||
"""Applying node_snapshot SSE events."""
|
||||
|
||||
@@ -1489,6 +1563,142 @@ class TestConsoleProxy:
|
||||
assert sse_mock.await_count == 1
|
||||
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Proxied auth endpoints — handled locally by the console, not
|
||||
# forwarded to the upstream node. Cases derive directly from
|
||||
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
|
||||
# added without a matching test (or vice versa). See proxy_api's
|
||||
# docstring for the JWT-audience reasoning.
|
||||
# -------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "handler_name"),
|
||||
[
|
||||
(method, path, handler_name)
|
||||
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
|
||||
],
|
||||
)
|
||||
def test_proxy_auth_endpoint_dispatches_to_local_handler(
|
||||
self, client, method, path, handler_name
|
||||
):
|
||||
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
|
||||
local console handler and never reach the upstream proxy. The
|
||||
lockout class of bug this dispatch was added to fix is exactly
|
||||
what a regression here would reintroduce silently — covering all
|
||||
eight branches keeps each path tied to its handler."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
f"turnstone.console.server.{handler_name}",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
resp = client.request(method, f"/node/node-a/v1/api/{path}")
|
||||
assert resp.status_code == 200
|
||||
assert local_mock.await_count == 1
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
|
||||
"""Without this fix the AuthMiddleware 401s before any handler
|
||||
runs — the user is locked out of the proxied UI once the cookie
|
||||
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import _load_static, create_app
|
||||
|
||||
_load_static()
|
||||
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
|
||||
unauth_client = TestClient(app, raise_server_exceptions=False)
|
||||
try:
|
||||
with patch(
|
||||
"turnstone.console.server.auth_login",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "ok"}),
|
||||
) as local_mock:
|
||||
resp = unauth_client.post(
|
||||
"/node/node-a/v1/api/auth/login",
|
||||
json={"username": "x", "password": "y"},
|
||||
)
|
||||
# AuthMiddleware must classify the proxied login path as
|
||||
# public (is_public_path change) AND proxy_api must
|
||||
# dispatch to the local handler (proxy_api change).
|
||||
assert resp.status_code == 200, (
|
||||
f"login locked out: got {resp.status_code}, body={resp.text}"
|
||||
)
|
||||
assert local_mock.await_count == 1
|
||||
finally:
|
||||
unauth_client.close()
|
||||
|
||||
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
|
||||
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
|
||||
must short-circuit with 405 instead of falling through to the
|
||||
upstream proxy — falling through would forward the request
|
||||
authenticated as the console's service token (``_proxy_auth_headers``
|
||||
fallback)."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.console.server._proxy_post",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as post_mock,
|
||||
patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"status": "should-not-be-called"}),
|
||||
) as get_mock,
|
||||
):
|
||||
# PUT on a POST-only auth path → 405
|
||||
put_resp = client.put("/node/node-a/v1/api/auth/login")
|
||||
assert put_resp.status_code == 405
|
||||
# POST on a GET-only auth path → 405
|
||||
post_resp = client.post("/node/node-a/v1/api/auth/status")
|
||||
assert post_resp.status_code == 405
|
||||
assert post_mock.await_count == 0
|
||||
assert get_mock.await_count == 0
|
||||
|
||||
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
|
||||
"""Sanity: only auth/* paths intercept. Other API paths still
|
||||
forward to the upstream node."""
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
mock_collector.get_node_detail.return_value = {
|
||||
"node_id": "node-a",
|
||||
"server_url": "http://a:8080",
|
||||
"reachable": True,
|
||||
}
|
||||
with patch(
|
||||
"turnstone.console.server._proxy_get",
|
||||
new_callable=AsyncMock,
|
||||
return_value=JSONResponse({"ok": True}),
|
||||
) as proxy_mock:
|
||||
resp = client.get("/node/node-a/v1/api/workstreams")
|
||||
assert resp.status_code == 200
|
||||
assert proxy_mock.await_count == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Proxy URL rewriting unit tests (no HTTP needed)
|
||||
|
||||
@@ -5,11 +5,15 @@ lifting is in ``SessionManager.close_idle`` (covered in
|
||||
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
|
||||
in ``test_storage_sqlite.py``). These tests verify the glue:
|
||||
|
||||
- the helper runs an initial sweep BEFORE its first sleep (cold-start
|
||||
- the helper runs an initial sweep BEFORE its first wait (cold-start
|
||||
cleanup without blocking the lifespan),
|
||||
- the helper swallows exceptions so a transient DB blip can't kill the
|
||||
daemon thread,
|
||||
- the helper exits cleanly when ``stop_event`` is set.
|
||||
- the helper exits cleanly when ``stop_event`` is set,
|
||||
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
|
||||
event wakes the next sweep early (event-driven, not polling),
|
||||
- the helper unsubscribes when the thread exits so the subscriber
|
||||
doesn't leak past one cleanup-thread lifetime.
|
||||
|
||||
The ``stop_event`` parameter is exclusively for tests — production
|
||||
callers pass ``None`` and the daemon runs for process lifetime.
|
||||
@@ -17,28 +21,36 @@ callers pass ``None`` and the daemon runs for process lifetime.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import threading
|
||||
from unittest.mock import patch
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.console.server import _coord_idle_cleanup_thread
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
|
||||
class _StubMgr:
|
||||
"""Minimal SessionManager substitute exposing only what the cleanup
|
||||
thread touches: ``close_idle``, ``subscribe_to_state``,
|
||||
``unsubscribe_from_state``. Records call ordering for assertions
|
||||
and lets the test fire state-change events manually via
|
||||
:meth:`fire_state_change`.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
|
||||
) -> None:
|
||||
self.calls: list[float] = []
|
||||
self.sleep_calls_at_each_close: list[int] = []
|
||||
self._stop_event = stop_event
|
||||
self._expected = expected_calls
|
||||
self._raise_after = raise_after
|
||||
self._sleep_count = 0
|
||||
self._subscribers: list[Callable[[str, object], None]] = []
|
||||
self._sub_lock = threading.Lock()
|
||||
|
||||
def close_idle(self, timeout_sec: float) -> list[str]:
|
||||
# Snapshot how many sleeps preceded this close — lets the
|
||||
# "initial sweep" test verify the first close_idle ran with
|
||||
# zero preceding sleeps.
|
||||
self.sleep_calls_at_each_close.append(self._sleep_count)
|
||||
self.calls.append(timeout_sec)
|
||||
try:
|
||||
if 0 <= self._raise_after < len(self.calls):
|
||||
@@ -50,39 +62,78 @@ class _StubMgr:
|
||||
self._stop_event.set()
|
||||
return []
|
||||
|
||||
def record_sleep(self, _seconds: float) -> None:
|
||||
self._sleep_count += 1
|
||||
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock:
|
||||
self._subscribers.append(callback)
|
||||
|
||||
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
|
||||
with self._sub_lock, contextlib.suppress(ValueError):
|
||||
self._subscribers.remove(callback)
|
||||
|
||||
@property
|
||||
def subscribers_count(self) -> int:
|
||||
with self._sub_lock:
|
||||
return len(self._subscribers)
|
||||
|
||||
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
|
||||
with self._sub_lock:
|
||||
snapshot = list(self._subscribers)
|
||||
for cb in snapshot:
|
||||
cb(ws_id, state)
|
||||
|
||||
|
||||
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
|
||||
with patch("turnstone.console.server.time.sleep", mgr.record_sleep):
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
# ``min_sweep_interval=0.0`` disables the production cadence floor
|
||||
# (default 5 s) so tests can fire many close_idle calls back-to-back
|
||||
# without waiting real time between them. The floor is exercised
|
||||
# in its own dedicated test below.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, timeout_sec, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "helper failed to exit on stop_event"
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_sleep() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first time.sleep —
|
||||
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
|
||||
"""The first close_idle call must happen BEFORE the first wait —
|
||||
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
|
||||
on default 2h timeout) for the first reap. Crucial because the
|
||||
lifespan no longer does a synchronous initial sweep."""
|
||||
lifespan no longer does a synchronous initial sweep.
|
||||
|
||||
Verified structurally: a single ``expected_calls=1`` run completes
|
||||
in well under one ``check_every`` (here 0.04 s timeout → 0.01 s
|
||||
check_every), so the initial sweep must have happened before any
|
||||
real wait could have blocked it.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
assert mgr.sleep_calls_at_each_close == [0], "first close_idle should run before any sleep"
|
||||
started = time.monotonic()
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
elapsed = time.monotonic() - started
|
||||
assert len(mgr.calls) == 1
|
||||
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
|
||||
# gated behind one full wait would have taken ~0.01+ s anyway, so
|
||||
# the upper bound here is "much less than one check_every plus
|
||||
# process noise" — the explicit 1.0 s gives generous CI headroom
|
||||
# while still asserting the test is testing the right thing.
|
||||
assert elapsed < 1.0
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
|
||||
"""Heartbeat path: with no state-change events, close_idle fires
|
||||
each ``check_every`` interval. Test uses a tiny timeout so the
|
||||
test runs fast — the contract under test is "the loop iterates",
|
||||
not the production cadence.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert len(mgr.calls) == 3
|
||||
assert all(t == 120.0 for t in mgr.calls)
|
||||
assert all(t == 0.04 for t in mgr.calls)
|
||||
|
||||
|
||||
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
@@ -91,7 +142,7 @@ def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
|
||||
blip would silently leak orphans forever."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
# All four calls must have fired despite calls 2-4 raising.
|
||||
assert len(mgr.calls) == 4
|
||||
|
||||
@@ -102,5 +153,154 @@ def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
|
||||
daemon-process termination."""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=120.0)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert stop_event.is_set()
|
||||
|
||||
|
||||
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
|
||||
"""The event-driven path is the whole point of the refactor: a
|
||||
workstream state-change must wake the cleanup sweep without
|
||||
waiting one ``check_every`` interval. Tested with a long
|
||||
timeout_sec so the heartbeat would NOT have fired in the test
|
||||
window — the close_idle call past the initial sweep must come
|
||||
from a state-change wake.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
|
||||
# window. Any close_idle call past the initial sweep must come
|
||||
# from a fire_state_change-driven wake-up.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
# Wait for the initial sweep to complete AND the thread to enter
|
||||
# its first ``tick_now.wait`` (signalled here by the subscriber
|
||||
# being registered + calls advancing to 1).
|
||||
deadline = time.monotonic() + 1.0
|
||||
while time.monotonic() < deadline:
|
||||
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
|
||||
break
|
||||
time.sleep(0.01)
|
||||
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
|
||||
assert len(mgr.calls) == 1, "initial sweep didn't fire"
|
||||
# One state-change fire wakes the first ``wait`` → close_idle runs
|
||||
# again → stop_event is set (expected_calls=2) → thread exits.
|
||||
mgr.fire_state_change()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
|
||||
# 2 = initial + state-change-driven. If the state change weren't
|
||||
# being honoured, close_idle would have stalled on the 30 s wait
|
||||
# and the thread.join would have timed out.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_subscriber_unregisters_when_thread_exits() -> None:
|
||||
"""The cleanup thread's state-change subscriber must be removed
|
||||
when the thread exits — otherwise long-running processes that
|
||||
restart their cleanup threads (admin model-CRUD path, tests) leak
|
||||
subscribers and every state change fires N stale callbacks.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
|
||||
_run_until_done(mgr, stop_event, timeout_sec=0.04)
|
||||
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
|
||||
|
||||
|
||||
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
|
||||
"""A state-change fired during the initial sweep (e.g. close_idle's
|
||||
own ``close()`` calls firing subscribers) must wake the next
|
||||
``tick_now.wait`` rather than being lost to the clear-before-sweep
|
||||
ordering. The clear runs INSIDE the loop just before close_idle,
|
||||
so a fire during the initial sweep — which precedes the loop —
|
||||
arrives at an already-set event that the first wait sees set and
|
||||
returns on immediately.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
|
||||
# One-shot fire during the initial sweep, mirroring what
|
||||
# close_idle's own close() calls do in production (set_state →
|
||||
# state-change subscribers).
|
||||
fired = [False]
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
result = real_close_idle(timeout_sec)
|
||||
if not fired[0]:
|
||||
fired[0] = True
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.0},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=2.0)
|
||||
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
|
||||
# 2 = initial sweep + state-change-driven follow-up. Without the
|
||||
# event surviving the clear-before-sweep ordering, the thread
|
||||
# would have blocked on the 30 s ``wait`` and the test would have
|
||||
# timed out at thread.join.
|
||||
assert len(mgr.calls) == 2
|
||||
|
||||
|
||||
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
|
||||
"""Cadence floor: even when state-change events keep firing
|
||||
``tick_now.set()``, ``close_idle`` must not run more often than
|
||||
``min_sweep_interval`` — otherwise the loop tight-spins close_idle
|
||||
at the rate of its own DB latency, doing 600-1500x more DB work
|
||||
than the pre-refactor fixed-30 s cadence.
|
||||
|
||||
Wires a state-change subscriber that fires another state change
|
||||
from inside close_idle, so the bus would tick forever if not
|
||||
floored. Asserts the elapsed-between-sweeps is at least
|
||||
``min_sweep_interval`` modulo small wall-clock noise.
|
||||
"""
|
||||
stop_event = threading.Event()
|
||||
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
|
||||
|
||||
real_close_idle = mgr.close_idle
|
||||
sweep_times: list[float] = []
|
||||
|
||||
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
|
||||
sweep_times.append(time.monotonic())
|
||||
result = real_close_idle(timeout_sec)
|
||||
# Always fire another state-change to simulate sustained
|
||||
# activity (each turn fires thinking/running/attention/idle).
|
||||
# If the floor were absent, the next wake would race the next
|
||||
# close_idle immediately and ``sweep_times`` deltas would be
|
||||
# bounded by close_idle latency (microseconds), not the floor.
|
||||
mgr.fire_state_change()
|
||||
return result
|
||||
|
||||
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
|
||||
|
||||
# 0.15 s floor keeps the test fast (~0.3 s total) while still
|
||||
# representing a meaningful gap relative to close_idle's
|
||||
# near-zero stub latency.
|
||||
thread = threading.Thread(
|
||||
target=_coord_idle_cleanup_thread,
|
||||
args=(mgr, 120.0, stop_event),
|
||||
kwargs={"min_sweep_interval": 0.15},
|
||||
daemon=True,
|
||||
)
|
||||
thread.start()
|
||||
thread.join(timeout=3.0)
|
||||
assert not thread.is_alive(), "thread didn't exit"
|
||||
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
|
||||
# Gap between sweep 1 (post-initial) and sweep 2 must respect
|
||||
# the floor. Initial sweep at sweep_times[0] is unfloored
|
||||
# (no prior sweep to compare against), so the meaningful
|
||||
# assertion is on sweep_times[1] - sweep_times[0].
|
||||
gap = sweep_times[1] - sweep_times[0]
|
||||
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
|
||||
|
||||
@@ -716,3 +716,82 @@ class TestCoordinatorAdapterDispatchChildEvent:
|
||||
},
|
||||
)
|
||||
assert recorder.enqueued == []
|
||||
|
||||
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
|
||||
"""Every translated state-class event must call
|
||||
``ChildEventBus.notify(ws_id)`` so a registered
|
||||
``wait_for_workstream`` waiter wakes promptly. Notify fires
|
||||
AFTER the UI enqueue so the SSE fan-out keeps priority — the
|
||||
order assertion here is structural (one notify call, matching
|
||||
ws_id) since the bus side-effect lookup is what guards against
|
||||
regressions, not the relative event ordering.
|
||||
"""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
|
||||
|
||||
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
|
||||
"""The dispatch sink translates six event types into the
|
||||
``child_ws_*`` SSE shape; all six must also fire the bus so
|
||||
a wait on any of them wakes. ``ws_created`` is intentionally
|
||||
NOT in this set — waiters register against ws_ids they already
|
||||
know exist (the wait tool takes a pre-known list)."""
|
||||
for etype, extra in [
|
||||
("cluster_state", {"state": "running"}),
|
||||
("ws_closed", {"reason": "evicted"}),
|
||||
("ws_rename", {"name": "renamed"}),
|
||||
("intent_verdict", {"verdict": {"call_id": "c1"}}),
|
||||
("approval_resolved", {"approved": True}),
|
||||
("approve_request", {"detail": {}}),
|
||||
]:
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-a1"])
|
||||
adapter._dispatch_child_event(
|
||||
{"type": etype, "ws_id": "child-a1", **extra},
|
||||
)
|
||||
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
|
||||
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
|
||||
waiter registered against ws Y, or every state change anywhere
|
||||
in the system would shake every concurrent wait."""
|
||||
adapter, _, _ = self._setup()
|
||||
adapter._registry.merge_children("coord-a", ["child-a1"])
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["child-other"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "child-a1",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
|
||||
|
||||
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
|
||||
"""Events whose ws_id isn't in any coord's registry are dropped
|
||||
BEFORE the bus notify (early return at ``coord_id is None``).
|
||||
Notify only fires for events the dispatch sink fully translated,
|
||||
keeping the bus side-effect aligned with the UI enqueue."""
|
||||
adapter, _, _ = self._setup()
|
||||
bus = adapter.child_event_bus
|
||||
event = bus.register_waiter(["ws-orphan"])
|
||||
adapter._dispatch_child_event(
|
||||
{
|
||||
"type": "cluster_state",
|
||||
"ws_id": "ws-orphan",
|
||||
"state": "idle",
|
||||
}
|
||||
)
|
||||
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
|
||||
|
||||
@@ -9,6 +9,7 @@ storage-call path.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import httpx
|
||||
@@ -20,6 +21,7 @@ from turnstone.console.coordinator_client import (
|
||||
CoordinatorTokenManager,
|
||||
)
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -145,6 +147,7 @@ def _mock_client(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
return client, captured
|
||||
|
||||
@@ -470,6 +473,7 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -663,6 +667,7 @@ def _make_client_with_cluster_response(
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
@@ -1443,6 +1448,23 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
|
||||
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
|
||||
matches the coordinator but whose ``user_id`` belongs to a
|
||||
different tenant must collapse to ``denied`` — otherwise a
|
||||
forged / migration-era / pre-tenant-gate row would let a
|
||||
coordinator's LLM observe foreign-tenant state through
|
||||
``wait_for_workstream``. The ``populated_storage`` fixture's
|
||||
``cross-tenant-child`` row has exactly this shape
|
||||
(parent_ws_id="coord-1", user_id="user-2").
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
|
||||
assert result["results"]["cross-tenant-child"]["state"] == "denied"
|
||||
assert result["complete"] is False
|
||||
assert result["elapsed"] < 1.0
|
||||
|
||||
|
||||
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
|
||||
"""A ws_id that doesn't exist collapses into the same 'denied'
|
||||
shape as a foreign ws_id so wait can't be used as an existence
|
||||
@@ -1531,10 +1553,22 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
|
||||
assert list(result["results"].keys()) == ["child-a"]
|
||||
|
||||
|
||||
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
|
||||
"""Per-tick polling must issue batched storage calls — at the
|
||||
documented cap (32 ws_ids over a 600s wait) the naive per-id
|
||||
shape produced ~38k row reads. Guard against regression."""
|
||||
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
|
||||
populated_storage, monkeypatch
|
||||
):
|
||||
"""All storage reads issued by ``wait_for_workstream`` must go
|
||||
through the batched paths. At the documented cap (32 ws_ids over
|
||||
a 600 s wait) the naive per-id shape produced ~38k row reads, so
|
||||
a regression to per-id is the meaningful failure mode this test
|
||||
guards against.
|
||||
|
||||
The primary safety net is the ``pytest.fail`` mock on the per-id
|
||||
``get_workstream`` / ``sum_workstream_tokens`` paths — any call
|
||||
there blows up loudly with the regression message. The
|
||||
additional ``batch_calls`` / ``sum_calls`` assertions cover the
|
||||
subtler regression where the call IS batched but only covers a
|
||||
subset of ws_ids (e.g. one ws_id per call in a loop).
|
||||
"""
|
||||
client = _make_read_client(populated_storage)
|
||||
batch_calls: list[list[str]] = []
|
||||
sum_calls: list[list[str]] = []
|
||||
@@ -1565,11 +1599,16 @@ def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monke
|
||||
|
||||
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
# One tick is enough since child-a is already idle (terminal).
|
||||
assert len(batch_calls) == 1
|
||||
assert len(sum_calls) == 1
|
||||
assert set(batch_calls[0]) == {"child-a", "child-b"}
|
||||
assert set(sum_calls[0]) == {"child-a", "child-b"}
|
||||
# Every batched call carried the full ws_id set. The exact count
|
||||
# (currently 2: one pre-loop ownership filter + one snapshot tick)
|
||||
# is incidental; if either gains another batched read it stays
|
||||
# batched, which is the property under test.
|
||||
assert batch_calls, "no batched get_workstreams_batch call observed"
|
||||
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
|
||||
first_batch = set(batch_calls[0])
|
||||
first_sum = set(sum_calls[0])
|
||||
assert first_batch == {"child-a", "child-b"}
|
||||
assert first_sum == {"child-a", "child-b"}
|
||||
|
||||
|
||||
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
@@ -1581,6 +1620,205 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
|
||||
assert "invalid mode" in result["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — event-driven (ChildEventBus wired in)
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# When the coord adapter wires its ``child_event_bus`` into the client,
|
||||
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
|
||||
# and only re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. The legacy ``time.sleep`` poll path remains intact for tests
|
||||
# that don't wire the bus (above), so this section adds focused
|
||||
# coverage of the bus-driven behaviour without re-running the full
|
||||
# matrix of mode / since / cross-tenant cases.
|
||||
|
||||
|
||||
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
|
||||
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
|
||||
|
||||
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
|
||||
simulate the dispatch-sink wake-up.
|
||||
"""
|
||||
transport = httpx.MockTransport(lambda r: httpx.Response(200))
|
||||
http = httpx.Client(transport=transport)
|
||||
return CoordinatorClient(
|
||||
console_base_url="http://x",
|
||||
storage=storage,
|
||||
token_factory=lambda: "t",
|
||||
coord_ws_id="coord-1",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=bus,
|
||||
)
|
||||
|
||||
|
||||
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
|
||||
"""Subscribe-after-terminal race: the wait registers its waiter
|
||||
BEFORE the first snapshot, then re-snapshots — an already-terminal
|
||||
child must return at once without spinning the heartbeat cap.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-a"]["state"] == "idle"
|
||||
assert result["elapsed"] < 1.0
|
||||
# Waiter must be unregistered on exit so a long-lived bus doesn't
|
||||
# accumulate dead keys across many waits.
|
||||
assert "child-a" not in bus._waiters
|
||||
|
||||
|
||||
def test_wait_with_bus_wakes_on_notify(populated_storage):
|
||||
"""The core property of the refactor: a state-change ``notify``
|
||||
must wake the wait promptly — well under the legacy 0.5 s poll
|
||||
cadence AND the 2 s heartbeat cap. Test fires a state update
|
||||
+ notify after a short delay and asserts the wait returns quickly.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b starts running; flip to idle + notify after the wait
|
||||
# blocks. 100 ms is enough that the wait is parked in event.wait()
|
||||
# but short enough that the test runs fast.
|
||||
timer = _t.Timer(
|
||||
0.1,
|
||||
lambda: (
|
||||
populated_storage.update_workstream_state("child-b", "idle"),
|
||||
bus.notify("child-b"),
|
||||
),
|
||||
)
|
||||
timer.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is True
|
||||
assert result["results"]["child-b"]["state"] == "idle"
|
||||
# Bus-driven wake should fire well under 1 s; legacy poll would
|
||||
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
|
||||
# plus a few ms. Generous 0.6 s budget for CI noise.
|
||||
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
|
||||
|
||||
|
||||
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
|
||||
"""A notify on a ws_id the wait isn't watching must NOT wake it —
|
||||
otherwise every state change anywhere on the system would shake
|
||||
every concurrent wait into a redundant storage snapshot.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# child-b is running indefinitely; mode='all' will time out unless
|
||||
# a relevant notify fires. Fire only unrelated notifies — wait
|
||||
# should still hit the full timeout.
|
||||
import threading as _t
|
||||
|
||||
def _fire_unrelated() -> None:
|
||||
for _ in range(5):
|
||||
bus.notify("ws-unrelated-1")
|
||||
bus.notify("ws-unrelated-2")
|
||||
time.sleep(0.05)
|
||||
|
||||
t = _t.Thread(target=_fire_unrelated, daemon=True)
|
||||
t.start()
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
|
||||
# Wait should burn its full timeout (give or take heartbeat
|
||||
# granularity). The bus path doesn't have a 0.5 s poll, so the
|
||||
# bound is "approximately timeout".
|
||||
assert elapsed >= 0.5
|
||||
t.join(timeout=1.0)
|
||||
|
||||
|
||||
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
|
||||
"""Without any notify, the wait must still progress through ticks
|
||||
via the heartbeat cap so ``progress_callback`` keeps firing for
|
||||
the sidebar UI. Verified by counting callback firings over an
|
||||
interval longer than the heartbeat.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Shrink the heartbeat for test speed via the ClassVar seam —
|
||||
# instance attribute shadows the class-level default. Production
|
||||
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
|
||||
# notify property in well under 1 s.
|
||||
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
|
||||
snapshots: list[dict[str, dict[str, object]]] = []
|
||||
|
||||
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
|
||||
snapshots.append(snap)
|
||||
|
||||
# child-b is running indefinitely; wait will time out at 0.4 s.
|
||||
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
|
||||
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
|
||||
# CI flakiness.
|
||||
start = time.monotonic()
|
||||
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
|
||||
elapsed = time.monotonic() - start
|
||||
assert result["complete"] is False
|
||||
assert elapsed >= 0.4
|
||||
# At least 2 callback firings: the initial snapshot plus at least
|
||||
# one heartbeat-driven re-tick. Tight upper bound would be
|
||||
# ~ceil(0.4/0.1) + 1 = 5 firings.
|
||||
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
|
||||
|
||||
|
||||
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
|
||||
"""Both the success path and the timeout path must unregister the
|
||||
waiter — otherwise a long-lived bus accumulates dead
|
||||
``threading.Event`` instances forever.
|
||||
"""
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
# Success path (already-terminal child).
|
||||
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
assert bus._waiters == {}, "success path leaked waiter"
|
||||
# Timeout path (running child, mode='all' that times out).
|
||||
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
|
||||
assert bus._waiters == {}, "timeout path leaked waiter"
|
||||
|
||||
|
||||
def test_wait_with_bus_multi_waiter_independence(populated_storage):
|
||||
"""Two concurrent waits on the same ws_id must be independent —
|
||||
one wait completing must not affect the other's wake-up state.
|
||||
Smoke-tests the multi-Event-per-bucket bus behaviour against the
|
||||
real wait-loop.
|
||||
"""
|
||||
import threading as _t
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
|
||||
bus = ChildEventBus()
|
||||
client = _make_read_client_with_bus(populated_storage, bus)
|
||||
|
||||
results: dict[str, dict[str, object]] = {}
|
||||
|
||||
def _do_wait(label: str) -> None:
|
||||
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
|
||||
|
||||
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=5.0)
|
||||
for label in ("t0", "t1", "t2"):
|
||||
assert results[label]["complete"] is True
|
||||
assert results[label]["results"]["child-a"]["state"] == "idle"
|
||||
# All waiters must be unregistered after exit.
|
||||
assert bus._waiters == {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# wait_for_workstream — last-message bundling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -40,6 +40,7 @@ from turnstone.console.server import (
|
||||
_require_coord_mgr,
|
||||
)
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.session_manager import SessionManager
|
||||
from turnstone.core.session_routes import (
|
||||
SessionEndpointConfig,
|
||||
@@ -286,6 +287,7 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
|
||||
coord_ws_id="coord-42",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
# spawn ---------------------------------------------------------------
|
||||
@@ -387,6 +389,7 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
|
||||
coord_ws_id="coord-root",
|
||||
user_id="user-1",
|
||||
http_client=http,
|
||||
child_event_bus=ChildEventBus(),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Integration tests for the Phase 9 admin bulk-revoke endpoint.
|
||||
|
||||
POST /v1/api/admin/mcp-servers/{name}/bulk-revoke clears every user's
|
||||
OAuth token for a server (admin-side counterpart to the per-user
|
||||
DELETE /v1/api/mcp/oauth/connections/{server_name} that shipped in
|
||||
Phase 8).
|
||||
|
||||
Coverage:
|
||||
- requires ``admin.mcp`` permission (401/403 without).
|
||||
- 404 when the named server is missing.
|
||||
- 400 when the server's ``auth_type`` is not ``oauth_user``.
|
||||
- 200 + ``rows_deleted`` + ``consented_users_before`` on success.
|
||||
- Audit row written with
|
||||
``upstream_revoke_outcome="bulk_admin_no_upstream"``.
|
||||
- Token rows are gone from ``mcp_user_tokens`` post-call.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Mount, Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.console.server import admin_mcp_bulk_revoke
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
class _InjectAdminMcp(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id="admin-user",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="config",
|
||||
permissions=frozenset({"read", "write", "approve", "admin.mcp"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
class _InjectNoAdminMcp(BaseHTTPMiddleware):
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id="regular-user",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="jwt",
|
||||
permissions=frozenset({"read", "write", "approve"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
def _build_app(storage: SQLiteBackend, *, with_admin_mcp: bool = True) -> Starlette:
|
||||
mw = _InjectAdminMcp if with_admin_mcp else _InjectNoAdminMcp
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Mount(
|
||||
"/v1",
|
||||
routes=[
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{name}/bulk-revoke",
|
||||
admin_mcp_bulk_revoke,
|
||||
methods=["POST"],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(mw)],
|
||||
)
|
||||
app.state.auth_storage = storage
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path: Any) -> SQLiteBackend:
|
||||
return SQLiteBackend(str(tmp_path / "test.db"))
|
||||
|
||||
|
||||
def _seed_oauth_server(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
name: str = "srv-oauth",
|
||||
server_id: str = "srv-oauth-id",
|
||||
) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id=server_id,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="oauth_user",
|
||||
)
|
||||
|
||||
|
||||
def _seed_static_server(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
name: str = "srv-static",
|
||||
server_id: str = "srv-static-id",
|
||||
) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id=server_id,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="static",
|
||||
)
|
||||
|
||||
|
||||
def _seed_user_tokens(backend: SQLiteBackend, server_name: str, users: int) -> None:
|
||||
for i in range(users):
|
||||
backend.create_mcp_user_token(
|
||||
f"user-{i}",
|
||||
server_name,
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=None,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
|
||||
|
||||
def test_requires_admin_mcp_permission(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage, with_admin_mcp=False))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 403
|
||||
|
||||
|
||||
def test_404_on_missing_server(storage: SQLiteBackend) -> None:
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/never-existed/bulk-revoke")
|
||||
assert resp.status_code == 404
|
||||
assert resp.json() == {"error": "No such server"}
|
||||
|
||||
|
||||
def test_400_on_static_server(storage: SQLiteBackend) -> None:
|
||||
_seed_static_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-static/bulk-revoke")
|
||||
assert resp.status_code == 400
|
||||
body = resp.json()
|
||||
assert "oauth_user" in body["error"]
|
||||
|
||||
|
||||
def test_400_on_invalid_server_name(storage: SQLiteBackend) -> None:
|
||||
# double-underscore is reserved for the prefixed-tool-name encoding.
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/bad__name/bulk-revoke")
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_200_on_success_with_no_consented_users(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["status"] == "ok"
|
||||
assert body["rows_deleted"] == 0
|
||||
assert body["consented_users_before"] == 0
|
||||
|
||||
|
||||
def test_200_clears_all_user_tokens(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_user_tokens(storage, "srv-oauth", users=3)
|
||||
# Token for another server must survive the bulk-revoke.
|
||||
_seed_oauth_server(storage, name="srv-other", server_id="srv-other-id")
|
||||
_seed_user_tokens(storage, "srv-other", users=2)
|
||||
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["status"] == "ok"
|
||||
assert body["rows_deleted"] == 3
|
||||
assert body["consented_users_before"] == 3
|
||||
|
||||
# Target server's tokens are gone; bystander's tokens survive.
|
||||
assert storage.count_mcp_consented_users_by_server("srv-oauth") == 0
|
||||
assert storage.count_mcp_consented_users_by_server("srv-other") == 2
|
||||
|
||||
|
||||
def test_audits_with_bulk_admin_no_upstream(storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_user_tokens(storage, "srv-oauth", users=2)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
|
||||
assert resp.status_code == 200
|
||||
|
||||
# Pull the most-recent audit row for the bulk_revoked action and
|
||||
# verify it carries the deferral marker.
|
||||
events = storage.list_audit_events(limit=10)
|
||||
bulk_rows = [e for e in events if e.get("action") == "mcp_server.oauth.bulk_revoked"]
|
||||
assert len(bulk_rows) == 1
|
||||
detail = bulk_rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("upstream_revoke_outcome") == "bulk_admin_no_upstream"
|
||||
assert detail.get("rows_deleted") == 2
|
||||
assert detail.get("consented_users_before") == 2
|
||||
assert detail.get("name") == "srv-oauth"
|
||||
+146
-1
@@ -788,7 +788,11 @@ class TestSessionIntegration:
|
||||
assert call_id == "call_789"
|
||||
assert output == "result text"
|
||||
mock_mcp.call_tool_sync.assert_called_once_with(
|
||||
"mcp__test__search", {"query": "hello"}, user_id=None, timeout=30
|
||||
"mcp__test__search",
|
||||
{"query": "hello"},
|
||||
user_id=None,
|
||||
timeout=30,
|
||||
is_interactive_for_consent=True,
|
||||
)
|
||||
|
||||
def test_exec_mcp_tool_error(self, tmp_db):
|
||||
@@ -1056,6 +1060,147 @@ class TestRefreshServer:
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
class TestLastRefreshTracking:
|
||||
"""Phase 9 admin status pill — ``_last_refresh`` is written on every
|
||||
refresh path so the admin UI reflects manual-refresh AND auto-
|
||||
reconnect outcomes uniformly. This test class pins the contract.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _seed_minimal(mgr: MCPClientManager, name: str = "srv") -> MagicMock:
|
||||
mock_session = MagicMock()
|
||||
mock_session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
|
||||
mock_session.list_resources = AsyncMock(return_value=MagicMock(resources=[]))
|
||||
mock_session.list_resource_templates = AsyncMock(
|
||||
return_value=MagicMock(resourceTemplates=[])
|
||||
)
|
||||
mock_session.list_prompts = AsyncMock(return_value=MagicMock(prompts=[]))
|
||||
_seed_static_state(
|
||||
mgr,
|
||||
name,
|
||||
session=mock_session,
|
||||
tools=[],
|
||||
supports_resources=True,
|
||||
supports_prompts=True,
|
||||
)
|
||||
return mock_session
|
||||
|
||||
def test_last_refresh_written_on_success(self) -> None:
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
self._seed_minimal(mgr)
|
||||
assert "srv" not in mgr._last_refresh
|
||||
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
ts, outcome = entry
|
||||
assert outcome == "ok"
|
||||
assert isinstance(ts, float) and ts > 0
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_last_refresh_written_on_tool_refresh_failure(self) -> None:
|
||||
"""When ``_refresh_server_tools`` raises, the outcome reflects
|
||||
the exception class and the exception still propagates."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
mock_session = self._seed_minimal(mgr)
|
||||
mock_session.list_tools = AsyncMock(side_effect=RuntimeError("upstream down"))
|
||||
|
||||
with pytest.raises(RuntimeError, match="upstream down"):
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
_, outcome = entry
|
||||
assert outcome == "error:RuntimeError"
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_last_refresh_records_first_exception_when_multiple_fail(
|
||||
self,
|
||||
) -> None:
|
||||
"""``return_exceptions=True`` lets sibling tasks complete; the
|
||||
outcome reflects the FIRST exception encountered."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
mock_session = self._seed_minimal(mgr)
|
||||
# Tools succeeds; resources raises first (gather preserves
|
||||
# argument order in its results list, so resources is the
|
||||
# first failure regardless of which awaitable finished first
|
||||
# in wall-clock terms).
|
||||
mock_session.list_resources = AsyncMock(side_effect=ValueError("res boom"))
|
||||
mock_session.list_prompts = AsyncMock(side_effect=KeyError("prompts boom"))
|
||||
|
||||
with pytest.raises((ValueError, KeyError)):
|
||||
await mgr._refresh_server("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
_, outcome = entry
|
||||
# Either of the two failing tasks could be "first" in
|
||||
# gather's results list ordering — the order is positional
|
||||
# so resources (arg #2) comes before prompts (arg #3).
|
||||
assert outcome == "error:ValueError"
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_refresh_all_overwrites_stale_ok_on_reconnect_failure(
|
||||
self,
|
||||
) -> None:
|
||||
"""The chokepoint bug-1 fix: a prior successful refresh's ``'ok'``
|
||||
entry MUST be overwritten when a subsequent reconnect fails —
|
||||
otherwise the admin pill shows misleading "ok" while the server
|
||||
is in fact broken."""
|
||||
|
||||
async def _run() -> None:
|
||||
mgr = MCPClientManager({})
|
||||
# Server is configured but has no live session — _refresh_all
|
||||
# routes to the reconnect branch.
|
||||
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
|
||||
# Pre-seed a stale "ok" from an earlier successful refresh.
|
||||
mgr._last_refresh["srv"] = (1000.0, "ok")
|
||||
|
||||
async def _raise(*_a: object, **_kw: object) -> None:
|
||||
raise ConnectionError("reconnect failed")
|
||||
|
||||
mgr._connect_one = _raise # type: ignore[assignment]
|
||||
|
||||
await mgr._refresh_all("srv")
|
||||
|
||||
entry = mgr._last_refresh.get("srv")
|
||||
assert entry is not None
|
||||
ts, outcome = entry
|
||||
# Outcome reflects the new failure, not the stale ok.
|
||||
assert outcome == "error:ConnectionError"
|
||||
assert ts > 1000.0
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_get_server_status_surfaces_last_refresh_fields(self) -> None:
|
||||
"""``get_server_status`` surfaces ``last_refresh_at`` and
|
||||
``last_refresh_outcome`` for the admin pill — null when no
|
||||
refresh has occurred yet, populated after one."""
|
||||
mgr = MCPClientManager({})
|
||||
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
|
||||
|
||||
# No refresh yet — fields must be present and null so the JS
|
||||
# renderer can branch on absence cleanly.
|
||||
status = mgr.get_server_status("srv")
|
||||
assert status["last_refresh_at"] is None
|
||||
assert status["last_refresh_outcome"] is None
|
||||
|
||||
# Populate the tuple directly and re-read.
|
||||
mgr._last_refresh["srv"] = (12345.5, "ok")
|
||||
status = mgr.get_server_status("srv")
|
||||
assert status["last_refresh_at"] == 12345.5
|
||||
assert status["last_refresh_outcome"] == "ok"
|
||||
|
||||
|
||||
class TestListeners:
|
||||
def test_add_and_notify(self):
|
||||
mgr = MCPClientManager({})
|
||||
|
||||
@@ -612,6 +612,97 @@ class TestCallback:
|
||||
assert plain is not None
|
||||
assert plain["refresh_token"] is None
|
||||
|
||||
def test_callback_clears_pending_consent_on_success(
|
||||
self, storage: SQLiteBackend, http_client_mock: MagicMock
|
||||
) -> None:
|
||||
"""Successful callback must drop any ``mcp_pending_consent`` rows
|
||||
for the just-consented ``(user, server)`` (Phase 9 lifecycle
|
||||
contract). Regression guard for the dashboard-stays-stale-after-
|
||||
consent invariant.
|
||||
"""
|
||||
_seed_oauth_user_server(storage)
|
||||
self._seed_pending(storage)
|
||||
# Seed a deferred-consent record that a prior non-interactive run
|
||||
# would have left behind. Plus a cross-tenant record that must
|
||||
# NOT be touched.
|
||||
storage.upsert_mcp_pending_consent(
|
||||
user_id="user-1",
|
||||
server_name="srv-oauth",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id="ws-1",
|
||||
last_tool_call_id="tool-1",
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
storage.upsert_mcp_pending_consent(
|
||||
user_id="other-user",
|
||||
server_name="srv-oauth",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
token_store = _make_token_store(storage)
|
||||
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
|
||||
http_client_mock.post.return_value = _mk_response(
|
||||
200,
|
||||
{"access_token": "opaque-aaa", "expires_in": 3600},
|
||||
)
|
||||
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
with _public_addr_patch():
|
||||
resp = client.get(
|
||||
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
|
||||
follow_redirects=False,
|
||||
)
|
||||
assert resp.status_code == 302
|
||||
# Callback completed → user-1's deferred-consent row was cleared.
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
# Cross-tenant row survives — clear is per-(user, server).
|
||||
other = storage.list_mcp_pending_consent_by_user("other-user")
|
||||
assert len(other) == 1
|
||||
assert other[0]["server_name"] == "srv-oauth"
|
||||
|
||||
def test_callback_storage_failure_does_not_block_redirect(
|
||||
self, storage: SQLiteBackend, http_client_mock: MagicMock
|
||||
) -> None:
|
||||
"""If the post-persist ``delete_mcp_pending_consent`` raises, the
|
||||
callback's redirect still completes (best-effort contract). The
|
||||
stale badge is preferred over a broken consent flow.
|
||||
"""
|
||||
_seed_oauth_user_server(storage)
|
||||
self._seed_pending(storage)
|
||||
token_store = _make_token_store(storage)
|
||||
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
|
||||
http_client_mock.post.return_value = _mk_response(
|
||||
200,
|
||||
{"access_token": "opaque-aaa", "expires_in": 3600},
|
||||
)
|
||||
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
|
||||
original_delete = storage.delete_mcp_pending_consent
|
||||
|
||||
def _raise(*_a: Any, **_kw: Any) -> bool:
|
||||
raise RuntimeError("storage offline")
|
||||
|
||||
storage.delete_mcp_pending_consent = _raise # type: ignore[method-assign]
|
||||
try:
|
||||
with _public_addr_patch():
|
||||
resp = client.get(
|
||||
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
|
||||
follow_redirects=False,
|
||||
)
|
||||
finally:
|
||||
storage.delete_mcp_pending_consent = original_delete # type: ignore[method-assign]
|
||||
|
||||
assert resp.status_code == 302
|
||||
# Token persistence still succeeded — the user-visible contract.
|
||||
plain = token_store.get_user_token("user-1", "srv-oauth")
|
||||
assert plain is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 503 paths when mcp_token_store is None
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
"""Boundary tests for the Phase 9 pending-consent write path.
|
||||
|
||||
Drives ``MCPClientManager._dispatch_pool_sync`` (and the helper it
|
||||
calls, ``_record_pending_consent_best_effort``) and asserts that
|
||||
deferred-consent records reach storage only on non-interactive callers.
|
||||
|
||||
Per ``feedback_tests_through_boundaries.md``, at least one test must
|
||||
drive the real sync dispatcher → real ``_is_structured_error`` →
|
||||
real ``_record_pending_consent_best_effort`` plumb-through; the
|
||||
``_helpers`` unit tests below cover the classifier in isolation, but
|
||||
the end-to-end test is the structural gate that catches
|
||||
plumb-through regressions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import threading
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.conftest import make_mcp_token_cipher
|
||||
from turnstone.core.mcp_client import (
|
||||
_PENDING_CONSENT_PERSIST_CODES,
|
||||
MCPClientManager,
|
||||
_parse_pending_consent_envelope,
|
||||
)
|
||||
from turnstone.core.mcp_crypto import MCPTokenStore
|
||||
from turnstone.core.mcp_oauth import TokenLookupResult
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper-level unit tests (cheap, no event loop)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseEnvelope:
|
||||
def test_consent_required_no_scopes(self) -> None:
|
||||
env = json.dumps({"error": {"code": "mcp_consent_required", "server": "x", "detail": "d"}})
|
||||
assert _parse_pending_consent_envelope(env) == ("mcp_consent_required", None)
|
||||
|
||||
def test_insufficient_scope_with_scopes(self) -> None:
|
||||
env = json.dumps(
|
||||
{
|
||||
"error": {
|
||||
"code": "mcp_insufficient_scope",
|
||||
"server": "x",
|
||||
"detail": "d",
|
||||
"scopes_required": ["read", "write"],
|
||||
}
|
||||
}
|
||||
)
|
||||
assert _parse_pending_consent_envelope(env) == (
|
||||
"mcp_insufficient_scope",
|
||||
["read", "write"],
|
||||
)
|
||||
|
||||
def test_operator_codes_filtered(self) -> None:
|
||||
# Key-unknown / url-insecure / *_forbidden are operator-actionable,
|
||||
# NOT user-consent-shaped. They must not produce pending-consent
|
||||
# rows, regardless of whether the caller is interactive.
|
||||
for code in (
|
||||
"mcp_token_undecryptable_key_unknown",
|
||||
"mcp_oauth_url_insecure",
|
||||
"mcp_tool_call_forbidden",
|
||||
"mcp_resource_read_forbidden",
|
||||
"mcp_prompt_get_forbidden",
|
||||
):
|
||||
env = json.dumps({"error": {"code": code, "server": "x", "detail": "d"}})
|
||||
assert _parse_pending_consent_envelope(env) is None, code
|
||||
|
||||
def test_malformed_json_returns_none(self) -> None:
|
||||
assert _parse_pending_consent_envelope("not json") is None
|
||||
assert _parse_pending_consent_envelope("") is None
|
||||
|
||||
def test_persist_codes_set_is_expected(self) -> None:
|
||||
# Pin the contract — adding a new persistable code here is a
|
||||
# deliberate design decision and should require a test update.
|
||||
assert {
|
||||
"mcp_consent_required",
|
||||
"mcp_insufficient_scope",
|
||||
} == _PENDING_CONSENT_PERSIST_CODES
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end plumb-through (drives _dispatch_pool_sync)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_oauth_server(backend: Any, *, name: str = "pool-srv") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com/mcp",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-" + name, auth_type="oauth_user")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def running_loop_mgr():
|
||||
cfg: dict[str, Any] = {}
|
||||
mgr = MCPClientManager(cfg)
|
||||
loop = asyncio.new_event_loop()
|
||||
thread = threading.Thread(target=loop.run_forever, daemon=True, name="phase9-test-loop")
|
||||
thread.start()
|
||||
mgr._loop = loop
|
||||
try:
|
||||
yield mgr, loop, thread
|
||||
finally:
|
||||
|
||||
async def _drain(m: MCPClientManager) -> None:
|
||||
task = m._user_pool_eviction_task
|
||||
if task is not None:
|
||||
task.cancel()
|
||||
with contextlib.suppress(BaseException):
|
||||
await task
|
||||
m._user_pool_eviction_task = None
|
||||
|
||||
with contextlib.suppress(Exception):
|
||||
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
|
||||
loop.call_soon_threadsafe(loop.stop)
|
||||
thread.join(timeout=2)
|
||||
|
||||
|
||||
def _wire_mgr(mgr: MCPClientManager, backend: Any) -> None:
|
||||
cipher = make_mcp_token_cipher()
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
app_state = SimpleNamespace(
|
||||
auth_storage=backend,
|
||||
mcp_token_store=MCPTokenStore(backend, cipher, node_id="test"),
|
||||
mcp_oauth_http_client=MagicMock(),
|
||||
mcp_oauth_refresh_locks={},
|
||||
mcp_oauth_metadata_cache={},
|
||||
)
|
||||
mgr.set_storage(backend)
|
||||
mgr.set_app_state(app_state)
|
||||
|
||||
|
||||
def test_dispatch_persists_pending_for_non_interactive_caller(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Non-interactive caller hits ``mcp_consent_required`` → a
|
||||
``mcp_pending_consent`` row appears for ``(user_id, server_name)``."""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
|
||||
# Structured error envelope surfaces as RuntimeError to the caller.
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_consent_required"
|
||||
|
||||
# Persistent row written for the dashboard badge.
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
assert r["user_id"] == "user-a"
|
||||
assert r["server_name"] == "pool-srv"
|
||||
assert r["error_code"] == "mcp_consent_required"
|
||||
assert r["occurrence_count"] == 1
|
||||
|
||||
|
||||
def test_dispatch_does_not_persist_for_interactive_caller(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Interactive caller hits the same error path → NO row written.
|
||||
|
||||
Interactive (WEB / CLI) sessions surface the consent prompt in-flight
|
||||
via the Phase 8 SSE renderer; persisting would just produce
|
||||
immediately-stale dashboard badges.
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError),
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=True,
|
||||
)
|
||||
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
|
||||
|
||||
def test_dispatch_returns_envelope_unchanged_on_storage_failure(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""When ``upsert_mcp_pending_consent`` raises, the agent-observable
|
||||
contract is unchanged: the structured-error ``RuntimeError`` still
|
||||
surfaces with the original ``mcp_consent_required`` code. The doc-
|
||||
string promises best-effort persistence; this test pins that
|
||||
promise so a regression that propagates the storage exception would
|
||||
fail visibly.
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="missing")
|
||||
|
||||
original_upsert = backend.upsert_mcp_pending_consent
|
||||
|
||||
def _raise(*_a: Any, **_kw: Any) -> None:
|
||||
raise RuntimeError("storage offline")
|
||||
|
||||
backend.upsert_mcp_pending_consent = _raise # type: ignore[method-assign]
|
||||
try:
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_missing_token,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
finally:
|
||||
backend.upsert_mcp_pending_consent = original_upsert # type: ignore[method-assign]
|
||||
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_consent_required"
|
||||
|
||||
|
||||
def test_dispatch_does_not_persist_for_operator_actionable_code(
|
||||
running_loop_mgr: Any, backend: Any
|
||||
) -> None:
|
||||
"""Decrypt-failure → operator-actionable; even non-interactive callers
|
||||
must NOT produce a user-facing pending-consent record (the user can't
|
||||
resolve this by re-consenting).
|
||||
"""
|
||||
mgr, _loop, _ = running_loop_mgr
|
||||
_seed_oauth_server(backend)
|
||||
_wire_mgr(mgr, backend)
|
||||
|
||||
async def _decrypt_failure(**kwargs: Any) -> TokenLookupResult:
|
||||
return TokenLookupResult(kind="decrypt_failure")
|
||||
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.mcp_client.get_user_access_token_classified",
|
||||
side_effect=_decrypt_failure,
|
||||
),
|
||||
pytest.raises(RuntimeError) as exc_info,
|
||||
):
|
||||
mgr.call_tool_sync(
|
||||
"mcp__pool-srv__echo",
|
||||
{"payload": "hi"},
|
||||
user_id="user-a",
|
||||
timeout=10,
|
||||
is_interactive_for_consent=False,
|
||||
)
|
||||
|
||||
payload = json.loads(str(exc_info.value)).get("error", {})
|
||||
assert payload.get("code") == "mcp_token_undecryptable_key_unknown"
|
||||
# The operator-actionable code does NOT produce a pending-consent row.
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
@@ -0,0 +1,259 @@
|
||||
"""HTTP tests for the Phase 9 pending-consent endpoints.
|
||||
|
||||
Covers:
|
||||
- ``GET /v1/api/mcp/oauth/pending`` (install gate + read path)
|
||||
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` (single clear)
|
||||
- ``DELETE /v1/api/mcp/oauth/pending`` (bulk clear)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
from starlette.applications import Starlette
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.routing import Mount, Route
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
from turnstone.core.auth import AuthResult
|
||||
from turnstone.core.mcp_oauth import (
|
||||
handle_mcp_oauth_clear_all_pending,
|
||||
handle_mcp_oauth_clear_pending,
|
||||
handle_mcp_oauth_list_pending,
|
||||
)
|
||||
from turnstone.core.storage._sqlite import SQLiteBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
class _InjectAuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Stamp a fixed authenticated user on every request."""
|
||||
|
||||
def __init__(self, app: Any, user_id: str = "user-1") -> None:
|
||||
super().__init__(app)
|
||||
self._user_id = user_id
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Any) -> Response:
|
||||
request.state.auth_result = AuthResult(
|
||||
user_id=self._user_id,
|
||||
scopes=frozenset({"write"}),
|
||||
token_source="config",
|
||||
permissions=frozenset({"read", "write"}),
|
||||
)
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
def _build_app(storage: SQLiteBackend, *, user_id: str = "user-1") -> Starlette:
|
||||
class _Mw(_InjectAuthMiddleware):
|
||||
def __init__(self, app: Any) -> None:
|
||||
super().__init__(app, user_id=user_id)
|
||||
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Mount(
|
||||
"/v1",
|
||||
routes=[
|
||||
Route("/api/mcp/oauth/pending", handle_mcp_oauth_list_pending),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending",
|
||||
handle_mcp_oauth_clear_all_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending/{server_name}",
|
||||
handle_mcp_oauth_clear_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
middleware=[Middleware(_Mw)],
|
||||
)
|
||||
app.state.auth_storage = storage
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def storage(tmp_path: Any) -> SQLiteBackend:
|
||||
backend = SQLiteBackend(str(tmp_path / "test.db"))
|
||||
backend.create_user("user-1", "user1", "User One", "hash")
|
||||
backend.create_user("user-2", "user2", "User Two", "hash")
|
||||
return backend
|
||||
|
||||
|
||||
def _seed_oauth_server(backend: SQLiteBackend, *, name: str = "srv-x") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-id-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
auth_type="oauth_user",
|
||||
)
|
||||
|
||||
|
||||
def _seed_pending(
|
||||
backend: SQLiteBackend,
|
||||
*,
|
||||
user_id: str = "user-1",
|
||||
server_name: str = "srv-x",
|
||||
error_code: str = "mcp_consent_required",
|
||||
now_iso: str = "2026-05-11T12:00:00",
|
||||
) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id=user_id,
|
||||
server_name=server_name,
|
||||
error_code=error_code,
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=now_iso,
|
||||
)
|
||||
|
||||
|
||||
class TestListPending:
|
||||
def test_install_gate_short_circuits_on_no_oauth_servers(self, storage: SQLiteBackend) -> None:
|
||||
# Seed a pending row but NO oauth_user MCP server — the gate
|
||||
# must short-circuit to {pending: 0} regardless.
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"pending": 0, "servers": []}
|
||||
|
||||
def test_lists_pending_records_for_authenticated_user(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["pending"] == 1
|
||||
assert len(body["servers"]) == 1
|
||||
assert body["servers"][0]["server_name"] == "srv-x"
|
||||
assert body["servers"][0]["error_code"] == "mcp_consent_required"
|
||||
|
||||
def test_does_not_leak_cross_user_records(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage, user_id="user-2")
|
||||
client = TestClient(_build_app(storage, user_id="user-1"))
|
||||
resp = client.get("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"pending": 0, "servers": []}
|
||||
|
||||
|
||||
class TestClearPending:
|
||||
def test_delete_single(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
|
||||
def test_delete_missing_still_returns_204(self, storage: SQLiteBackend) -> None:
|
||||
# Idempotent — must not leak cross-user existence info via 404.
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
|
||||
assert resp.status_code == 204
|
||||
|
||||
def test_delete_does_not_touch_cross_user_rows(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage, user_id="user-1")
|
||||
_seed_pending(storage, user_id="user-2")
|
||||
client = TestClient(_build_app(storage, user_id="user-1"))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
# User-2's row survives.
|
||||
assert len(storage.list_mcp_pending_consent_by_user("user-2")) == 1
|
||||
|
||||
|
||||
class TestAuditTrail:
|
||||
def test_single_dismiss_audits(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_pending(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
|
||||
assert resp.status_code == 204
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "single"
|
||||
assert detail.get("cleared") == 1
|
||||
|
||||
def test_single_dismiss_audits_even_when_no_row_existed(self, storage: SQLiteBackend) -> None:
|
||||
# Cross-tenant non-observability requires a 204 in the never-existed
|
||||
# case — the audit row distinguishes a real dismiss from a stuffed
|
||||
# attempt by recording ``cleared=0``.
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
|
||||
assert resp.status_code == 204
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "single"
|
||||
assert detail.get("cleared") == 0
|
||||
|
||||
def test_bulk_dismiss_audits(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_oauth_server(storage, name="srv-y")
|
||||
_seed_pending(storage, server_name="srv-x")
|
||||
_seed_pending(storage, server_name="srv-y")
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 2}
|
||||
|
||||
events = storage.list_audit_events(limit=10)
|
||||
rows = [
|
||||
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
|
||||
]
|
||||
assert len(rows) == 1
|
||||
detail = rows[0].get("detail")
|
||||
if isinstance(detail, str):
|
||||
import json as _json
|
||||
|
||||
detail = _json.loads(detail)
|
||||
assert detail.get("mode") == "bulk"
|
||||
assert detail.get("cleared") == 2
|
||||
|
||||
|
||||
class TestClearAllPending:
|
||||
def test_bulk_clear(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
_seed_oauth_server(storage, name="srv-y")
|
||||
_seed_pending(storage, server_name="srv-x")
|
||||
_seed_pending(storage, server_name="srv-y")
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 2}
|
||||
assert storage.list_mcp_pending_consent_by_user("user-1") == []
|
||||
|
||||
def test_bulk_clear_zero_when_empty(self, storage: SQLiteBackend) -> None:
|
||||
_seed_oauth_server(storage)
|
||||
client = TestClient(_build_app(storage))
|
||||
resp = client.delete("/v1/api/mcp/oauth/pending")
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"cleared": 0}
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Storage CRUD tests for the Phase 9 ``mcp_pending_consent`` table.
|
||||
|
||||
Validates protocol additions backing the dashboard pending-consent badge:
|
||||
|
||||
- ``upsert_mcp_pending_consent`` — insert + on-conflict refresh
|
||||
- ``list_mcp_pending_consent_by_user`` — read path
|
||||
- ``delete_mcp_pending_consent`` — single-row clear
|
||||
- ``delete_all_mcp_pending_consent_by_user`` — bulk clear
|
||||
- ``count_mcp_consented_users_by_server`` — admin status pill
|
||||
- ``any_oauth_user_mcp_servers`` — install-level gate
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def _iso(ts: str = "2026-05-11T12:00:00") -> str:
|
||||
return ts
|
||||
|
||||
|
||||
class TestUpsertAndList:
|
||||
def test_insert_round_trip(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required="read write",
|
||||
last_ws_id="ws-1",
|
||||
last_tool_call_id="tool-1",
|
||||
now_iso=_iso(),
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
assert r["user_id"] == "user-a"
|
||||
assert r["server_name"] == "srv-x"
|
||||
assert r["error_code"] == "mcp_consent_required"
|
||||
assert r["scopes_required"] == "read write"
|
||||
assert r["last_ws_id"] == "ws-1"
|
||||
assert r["last_tool_call_id"] == "tool-1"
|
||||
assert r["occurrence_count"] == 1
|
||||
assert r["first_seen_at"] == r["last_seen_at"]
|
||||
|
||||
def test_upsert_bumps_count_and_refreshes_recency(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T12:00:00",
|
||||
)
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_insufficient_scope",
|
||||
scopes_required="read",
|
||||
last_ws_id="ws-2",
|
||||
last_tool_call_id="tool-2",
|
||||
now_iso="2026-05-11T13:00:00",
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert len(rows) == 1
|
||||
r = rows[0]
|
||||
# Recency fields refreshed to the second call's values; count bumped.
|
||||
assert r["occurrence_count"] == 2
|
||||
assert r["error_code"] == "mcp_insufficient_scope"
|
||||
assert r["scopes_required"] == "read"
|
||||
assert r["last_ws_id"] == "ws-2"
|
||||
assert r["last_tool_call_id"] == "tool-2"
|
||||
assert r["last_seen_at"] == "2026-05-11T13:00:00"
|
||||
# first_seen_at preserved — that's the load-bearing audit value.
|
||||
assert r["first_seen_at"] == "2026-05-11T12:00:00"
|
||||
|
||||
def test_list_orders_by_last_seen_desc(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-old",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T10:00:00",
|
||||
)
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-new",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso="2026-05-11T11:00:00",
|
||||
)
|
||||
rows = backend.list_mcp_pending_consent_by_user("user-a")
|
||||
assert [r["server_name"] for r in rows] == ["srv-new", "srv-old"]
|
||||
|
||||
def test_per_user_isolation(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.list_mcp_pending_consent_by_user("user-b") == []
|
||||
|
||||
|
||||
class TestDelete:
|
||||
def test_delete_single(self, backend) -> None:
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name="srv-x",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
# Second delete returns False (no row).
|
||||
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is False
|
||||
|
||||
def test_delete_missing_returns_false(self, backend) -> None:
|
||||
assert backend.delete_mcp_pending_consent("never", "missing") is False
|
||||
|
||||
def test_delete_all_by_user(self, backend) -> None:
|
||||
for name in ("srv-a", "srv-b", "srv-c"):
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-a",
|
||||
server_name=name,
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
# Cross-user row that must NOT be touched.
|
||||
backend.upsert_mcp_pending_consent(
|
||||
user_id="user-b",
|
||||
server_name="srv-z",
|
||||
error_code="mcp_consent_required",
|
||||
scopes_required=None,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=_iso(),
|
||||
)
|
||||
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
|
||||
assert backend.list_mcp_pending_consent_by_user("user-a") == []
|
||||
assert len(backend.list_mcp_pending_consent_by_user("user-b")) == 1
|
||||
|
||||
|
||||
class TestCountConsentedUsersByServer:
|
||||
def _seed_server(self, backend, name: str = "srv-x") -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-id-" + name,
|
||||
name=name,
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com/mcp",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-id-" + name, auth_type="oauth_user")
|
||||
|
||||
def test_counts_distinct_non_expired_users(self, backend) -> None:
|
||||
self._seed_server(backend)
|
||||
future = "2099-01-01T00:00:00"
|
||||
backend.create_mcp_user_token(
|
||||
"alice",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=future,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
backend.create_mcp_user_token(
|
||||
"bob",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=None, # null treated as non-expired
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
# Different server — must not count.
|
||||
self._seed_server(backend, name="srv-y")
|
||||
backend.create_mcp_user_token(
|
||||
"carol",
|
||||
"srv-y",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at=future,
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
assert backend.count_mcp_consented_users_by_server("srv-x") == 2
|
||||
assert backend.count_mcp_consented_users_by_server("srv-y") == 1
|
||||
|
||||
def test_excludes_expired(self, backend) -> None:
|
||||
self._seed_server(backend)
|
||||
backend.create_mcp_user_token(
|
||||
"alice",
|
||||
"srv-x",
|
||||
access_token_ct=b"ct",
|
||||
refresh_token_ct=None,
|
||||
expires_at="2020-01-01T00:00:00", # well in the past
|
||||
scopes=None,
|
||||
as_issuer="https://as.example.com",
|
||||
audience="https://example.com/mcp",
|
||||
)
|
||||
assert backend.count_mcp_consented_users_by_server("srv-x") == 0
|
||||
|
||||
def test_zero_when_no_rows(self, backend) -> None:
|
||||
assert backend.count_mcp_consented_users_by_server("missing") == 0
|
||||
|
||||
|
||||
class TestInstallGate:
|
||||
def test_any_oauth_user_returns_false_on_empty(self, backend) -> None:
|
||||
assert backend.any_oauth_user_mcp_servers() is False
|
||||
|
||||
def test_any_oauth_user_ignores_static_rows(self, backend) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-1",
|
||||
name="static-only",
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com",
|
||||
headers='{"Authorization": "Bearer x"}',
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
assert backend.any_oauth_user_mcp_servers() is False
|
||||
|
||||
def test_any_oauth_user_returns_true_when_one_exists(self, backend) -> None:
|
||||
backend.create_mcp_server(
|
||||
server_id="srv-2",
|
||||
name="oauth-srv",
|
||||
transport="streamable-http",
|
||||
command="",
|
||||
args="[]",
|
||||
url="https://example.com",
|
||||
headers="{}",
|
||||
env="{}",
|
||||
auto_approve=False,
|
||||
enabled=True,
|
||||
created_by="admin",
|
||||
)
|
||||
backend.update_mcp_server("srv-2", auth_type="oauth_user")
|
||||
assert backend.any_oauth_user_mcp_servers() is True
|
||||
@@ -331,7 +331,11 @@ class TestLoadModelRegistry:
|
||||
api_key="dummy",
|
||||
model="local-model",
|
||||
)
|
||||
assert reg.count == 2 # "openai" + "default"
|
||||
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
|
||||
# populates configs — only the explicit alias survives.
|
||||
assert reg.count == 1
|
||||
assert reg.has_alias("openai")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.default == "openai"
|
||||
_, model, _ = reg.resolve()
|
||||
assert model == "gpt-4o"
|
||||
@@ -562,7 +566,12 @@ class TestLoadModelRegistryWithDB:
|
||||
assert cfg.source == "config"
|
||||
|
||||
def test_db_only_models_coexist(self) -> None:
|
||||
"""DB models coexist alongside config.toml models."""
|
||||
"""DB models coexist alongside config.toml models.
|
||||
|
||||
The CLI ``"default"`` shim is suppressed when DB / config models
|
||||
already populate the registry — see
|
||||
``test_cli_default_shim_skipped_when_db_models_present``.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
@@ -586,7 +595,7 @@ class TestLoadModelRegistryWithDB:
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.has_alias("db-only")
|
||||
assert reg.has_alias("config-only")
|
||||
assert reg.has_alias("default")
|
||||
assert not reg.has_alias("default")
|
||||
assert reg.get_config("db-only").source == "db"
|
||||
assert reg.get_config("config-only").source == "config"
|
||||
|
||||
@@ -606,10 +615,12 @@ class TestLoadModelRegistryWithDB:
|
||||
}
|
||||
]
|
||||
)
|
||||
# The CLI default shim is suppressed when the DB row populates
|
||||
# configs, so only the DB-sourced alias exists here.
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
|
||||
assert reg.get_config("from-db").source == "db"
|
||||
assert reg.get_config("default").source == ""
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_disabled_db_models_excluded(self) -> None:
|
||||
"""Disabled DB models are not loaded."""
|
||||
@@ -1675,6 +1686,58 @@ class TestLoadModelRegistryDBOnly:
|
||||
reg = load_model_registry(model="", storage=storage)
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
|
||||
"""An auto-detected ``--model`` does NOT synthesise a ``default``
|
||||
alias when the DB already contributes models.
|
||||
|
||||
Regression for the silent bypass of ``model.task_alias`` /
|
||||
``model.plan_alias``: a synthesised ``default`` aliased to whatever
|
||||
``--base-url`` was at boot leaks into the LLM-visible alias list,
|
||||
and the LLM picks it for ``task_agent(model="default")`` — which
|
||||
then routes around the operator-configured per-role default.
|
||||
"""
|
||||
storage = _MockStorage(
|
||||
[
|
||||
{
|
||||
"alias": "gh200",
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"provider": "openai",
|
||||
"base_url": "http://gh200:8000/v1",
|
||||
"api_key": "sk-gh200",
|
||||
"context_window": 1048576,
|
||||
"capabilities": "{}",
|
||||
"enabled": True,
|
||||
}
|
||||
]
|
||||
)
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry(
|
||||
base_url="http://flatspark:8000/v1",
|
||||
api_key="sk-flatspark",
|
||||
model="qwen3.6-35B-A3B", # populated by ``detect_model``
|
||||
storage=storage,
|
||||
)
|
||||
assert reg.has_alias("gh200")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
|
||||
"""Same shim suppression when only ``[models.*]`` populates configs."""
|
||||
fake_cfg: dict[str, Any] = {
|
||||
"models": {"local": {"model": "qwen3-32b"}},
|
||||
}
|
||||
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
|
||||
reg = load_model_registry("http://x/v1", "x", "fallback-model")
|
||||
assert reg.has_alias("local")
|
||||
assert not reg.has_alias("default")
|
||||
|
||||
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
|
||||
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
|
||||
the back-compat ``default`` alias."""
|
||||
with patch("turnstone.core.model_registry.load_config", return_value={}):
|
||||
reg = load_model_registry("http://x/v1", "x", "lone-model")
|
||||
assert reg.has_alias("default")
|
||||
assert reg.get_config("default").model == "lone-model"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# server._effective_routing / _apply_routing_overrides
|
||||
|
||||
@@ -0,0 +1,400 @@
|
||||
"""Tests for the console-side ``NotifyDispatcher``.
|
||||
|
||||
Exercises the dispatcher against the SQLite synthetic-sweep path so the
|
||||
suite runs without a Postgres dependency. The PG path is shaped the
|
||||
same way (same handler invocation semantics) — the only difference is
|
||||
the underlying stream's wake-up source, which is covered separately in
|
||||
``test_storage_notify.py::TestPostgresNotify``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def dispatcher_factory(storage):
|
||||
"""Yield a factory that constructs + tracks dispatchers for teardown."""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
created: list[NotifyDispatcher] = []
|
||||
|
||||
def _make(*, channels: list[str]) -> NotifyDispatcher:
|
||||
d = NotifyDispatcher(storage, channels=channels)
|
||||
created.append(d)
|
||||
return d
|
||||
|
||||
yield _make
|
||||
|
||||
for d in created:
|
||||
d.stop(timeout=2.0)
|
||||
|
||||
|
||||
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
|
||||
"""Poll ``predicate`` until True or timeout. Returns bool."""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return False
|
||||
|
||||
|
||||
def _start_ready(d, *, timeout: float = 5.0) -> None:
|
||||
"""``d.start()`` + assert the listener is actually listening.
|
||||
|
||||
Closes the start-vs-notify race for backends where ``storage.listen``
|
||||
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
|
||||
connection): without the sync, a same-thread ``storage.notify`` can
|
||||
fire before the LISTEN registers and the notification is lost.
|
||||
"""
|
||||
d.start()
|
||||
if not d.wait_until_ready(timeout=timeout):
|
||||
msg = f"dispatcher listener did not open within {timeout}s"
|
||||
raise AssertionError(msg)
|
||||
|
||||
|
||||
class TestSubscribe:
|
||||
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
# Fire a notify via the storage layer — dispatcher delivers to handler.
|
||||
storage.notify("alpha", "hello")
|
||||
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
|
||||
|
||||
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
with pytest.raises(ValueError, match="not declared"):
|
||||
d.subscribe("beta", lambda n: None)
|
||||
|
||||
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen: list = []
|
||||
unsub = d.subscribe("alpha", lambda n: seen.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "first")
|
||||
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
|
||||
unsub()
|
||||
# After unsubscribe, the handler no longer fires. Drain old hits
|
||||
# so the next notify-vs-handler-count check is unambiguous.
|
||||
seen.clear()
|
||||
storage.notify("alpha", "second")
|
||||
# Give the dispatcher a beat to deliver if it were going to.
|
||||
time.sleep(0.2)
|
||||
assert not any(n.payload == "second" for n in seen)
|
||||
|
||||
def test_construction_requires_at_least_one_channel(self, storage):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
with pytest.raises(ValueError, match="at least one"):
|
||||
NotifyDispatcher(storage, channels=[])
|
||||
|
||||
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
|
||||
assert d.channels == ["alpha", "beta"]
|
||||
|
||||
|
||||
class TestDispatch:
|
||||
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("alpha", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "shared")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert seen_a[0].payload == "shared"
|
||||
assert seen_b[0].payload == "shared"
|
||||
|
||||
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
survived: list = []
|
||||
|
||||
def _broken(_n):
|
||||
msg = "boom"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
d.subscribe("alpha", _broken)
|
||||
d.subscribe("alpha", lambda n: survived.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "after_broken")
|
||||
# The second handler runs even though the first raised.
|
||||
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
|
||||
|
||||
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha", "beta"])
|
||||
seen_a: list = []
|
||||
seen_b: list = []
|
||||
d.subscribe("alpha", lambda n: seen_a.append(n))
|
||||
d.subscribe("beta", lambda n: seen_b.append(n))
|
||||
_start_ready(d)
|
||||
storage.notify("alpha", "for_a")
|
||||
storage.notify("beta", "for_b")
|
||||
assert _wait_for(lambda: seen_a and seen_b)
|
||||
assert all(n.payload == "for_a" for n in seen_a)
|
||||
assert all(n.payload == "for_b" for n in seen_b)
|
||||
|
||||
|
||||
class TestReconnect:
|
||||
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
|
||||
|
||||
Uses a stub storage that owns its own listen stream so the test can
|
||||
drive a controlled stream-error sequence — the SQLite path can't
|
||||
raise :class:`NotifyConnectionError`, and the PG path requires a
|
||||
real database outage to exercise this code, neither of which fits a
|
||||
unit test. The dispatcher's threading and reconcile-pending logic
|
||||
are storage-agnostic — the dispatcher sees the same
|
||||
:class:`NotifyStream` Protocol regardless of backend.
|
||||
"""
|
||||
|
||||
def test_reconcile_fires_after_reopen_not_before(self):
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
# State machine: open -> first poll raises NotifyConnectionError
|
||||
# -> dispatcher waits backoff then reopens -> second open's first
|
||||
# poll blocks forever (test stops the dispatcher before then).
|
||||
# The fix: synthetic reconcile fires AFTER the second open
|
||||
# succeeds, not after the first open fails.
|
||||
sequence: list[str] = []
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self, fail_first_poll: bool):
|
||||
self._fail = fail_first_poll
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
if self._fail:
|
||||
self._fail = False
|
||||
sequence.append("poll_raises")
|
||||
msg = "fake-disconnect"
|
||||
raise NotifyConnectionError(msg)
|
||||
sequence.append("poll_returns")
|
||||
# Block until close to simulate a quiet steady-state.
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
self._open_count += 1
|
||||
sequence.append(f"open_{self._open_count}")
|
||||
if self._open_count == 2:
|
||||
reopen_event.set()
|
||||
stream = _StubStream(fail_first_poll=(self._open_count == 1))
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
# Speed up backoff so the reopen happens promptly in the test.
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list[Notify] = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
# Wait for the second open (post-reconnect).
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
|
||||
# Reconcile should be delivered shortly after the reopen.
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
f"no reconcile delivered; sequence={sequence}, got={got}"
|
||||
)
|
||||
# The reconcile must NOT fire before the second open —
|
||||
# if it did, the index of 'open_2' in sequence would
|
||||
# come after any reconcile-emitting work. Check ordering:
|
||||
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
|
||||
# inside the with-block of the SECOND open).
|
||||
ix_open_1 = sequence.index("open_1")
|
||||
ix_raises = sequence.index("poll_raises")
|
||||
ix_open_2 = sequence.index("open_2")
|
||||
assert ix_open_1 < ix_raises < ix_open_2
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
def test_generic_exception_path_also_synthesizes_reconcile(self):
|
||||
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
|
||||
|
||||
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
|
||||
shape, which doesn't go through the stream's exception
|
||||
translator and would hit the generic ``except Exception``
|
||||
branch. Pre-fix, that branch emitted no reconcile.
|
||||
"""
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
reopen_event = threading.Event()
|
||||
|
||||
class _StubStream:
|
||||
def __init__(self):
|
||||
self._closed = False
|
||||
|
||||
def poll(self, _timeout):
|
||||
if self._closed:
|
||||
return []
|
||||
time.sleep(0.5)
|
||||
return []
|
||||
|
||||
def close(self):
|
||||
self._closed = True
|
||||
|
||||
class _StubStorage:
|
||||
def __init__(self):
|
||||
self._open_count = 0
|
||||
|
||||
def listen(self, _channels):
|
||||
import contextlib as _contextlib
|
||||
|
||||
self._open_count += 1
|
||||
if self._open_count == 1:
|
||||
# First open raises a generic exception (e.g.
|
||||
# ``psycopg.OperationalError`` from a failed connect)
|
||||
# — landing in the dispatcher's generic except branch.
|
||||
msg = "fake-connect-failure"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
@_contextlib.contextmanager
|
||||
def _cm():
|
||||
reopen_event.set()
|
||||
stream = _StubStream()
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
return _cm()
|
||||
|
||||
import turnstone.console.notify_dispatcher as nd_mod
|
||||
|
||||
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
|
||||
try:
|
||||
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
|
||||
got: list = []
|
||||
d.subscribe("alpha", lambda n: got.append(n))
|
||||
d.start()
|
||||
try:
|
||||
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if any(n.payload == "reconcile" for n in got):
|
||||
break
|
||||
time.sleep(0.02)
|
||||
assert any(n.payload == "reconcile" for n in got), (
|
||||
"no reconcile delivered after generic-exception recovery"
|
||||
)
|
||||
finally:
|
||||
d.stop(timeout=2.0)
|
||||
finally:
|
||||
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
|
||||
|
||||
|
||||
class TestCoalescing:
|
||||
"""Same-channel burst collapses to one handler invocation per batch."""
|
||||
|
||||
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
invocations: list = []
|
||||
# Slow handler to ensure all bursts queue up before the first
|
||||
# call returns — gives the dispatch loop time to coalesce.
|
||||
coalesce_gate = threading.Event()
|
||||
|
||||
def _slow_handler(n):
|
||||
invocations.append(n)
|
||||
coalesce_gate.wait(0.05)
|
||||
|
||||
d.subscribe("alpha", _slow_handler)
|
||||
_start_ready(d)
|
||||
# Burst of 10 notifies on the same channel — should coalesce
|
||||
# down to many fewer handler invocations.
|
||||
for i in range(10):
|
||||
storage.notify("alpha", str(i))
|
||||
# Wait until the dispatch settles (handler is called at least once
|
||||
# and the queue empties).
|
||||
deadline = time.monotonic() + 2.0
|
||||
while time.monotonic() < deadline:
|
||||
if invocations and d._dispatch_queue.empty():
|
||||
time.sleep(0.1) # allow any final coalesced call to land
|
||||
break
|
||||
time.sleep(0.02)
|
||||
coalesce_gate.set()
|
||||
# At least one handler call; well fewer than 10 (coalescing
|
||||
# collapsed the burst). Exact count depends on timing — typical
|
||||
# is 1-2 invocations per burst on a fast machine.
|
||||
assert invocations, "handler never fired"
|
||||
assert len(invocations) < 10, (
|
||||
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
|
||||
)
|
||||
|
||||
|
||||
class TestLifecycle:
|
||||
def test_start_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.start() # No-op, no thread doubling
|
||||
# Single listener + single dispatch thread are spawned regardless.
|
||||
# Inspect by name so we don't depend on the exact thread count of
|
||||
# the test runner.
|
||||
listener_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
|
||||
]
|
||||
dispatch_threads = [
|
||||
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
|
||||
]
|
||||
assert len(listener_threads) == 1
|
||||
assert len(dispatch_threads) == 1
|
||||
|
||||
def test_stop_is_idempotent(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
d.stop(timeout=2.0)
|
||||
d.stop(timeout=2.0) # No-op, no error
|
||||
|
||||
def test_stop_without_start_is_noop(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.stop(timeout=1.0) # No-op, no thread to join
|
||||
|
||||
def test_stop_joins_threads(self, dispatcher_factory):
|
||||
d = dispatcher_factory(channels=["alpha"])
|
||||
d.start()
|
||||
# Capture thread references then stop and assert they exited.
|
||||
threads_before = [
|
||||
t
|
||||
for t in threading.enumerate()
|
||||
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
|
||||
]
|
||||
assert threads_before
|
||||
d.stop(timeout=3.0)
|
||||
time.sleep(0.05)
|
||||
for t in threads_before:
|
||||
assert not t.is_alive(), f"{t.name} still alive after stop"
|
||||
+668
-19
@@ -255,12 +255,17 @@ function makeEl(tag) {
|
||||
setAttribute(k, v) { this._attrs[k] = v; },
|
||||
getAttribute(k) { return this._attrs[k] !== undefined ? this._attrs[k] : null; },
|
||||
get classList() {
|
||||
// Real DOMTokenList is array-like (length + indexed access) AND
|
||||
// exposes add/remove/contains. The hljs language-extraction
|
||||
// loop reads .length + [j], so we return a fresh Array snapshot
|
||||
// each get + bolt the mutator methods on. add/remove operate on
|
||||
// the live _classes set so subsequent reads see updates.
|
||||
const self = this;
|
||||
return {
|
||||
add(...c) { c.forEach(x => self._classes.add(x)); },
|
||||
remove(...c) { c.forEach(x => self._classes.delete(x)); },
|
||||
contains(c) { return self._classes.has(c); },
|
||||
};
|
||||
const arr = Array.from(self._classes);
|
||||
arr.add = (...c) => c.forEach((x) => self._classes.add(x));
|
||||
arr.remove = (...c) => c.forEach((x) => self._classes.delete(x));
|
||||
arr.contains = (c) => self._classes.has(c);
|
||||
return arr;
|
||||
},
|
||||
get className() { return Array.from(this._classes).join(' '); },
|
||||
set className(v) {
|
||||
@@ -269,9 +274,33 @@ function makeEl(tag) {
|
||||
get textContent() {
|
||||
return this._textContent || this.children.map(c => c.textContent || '').join('');
|
||||
},
|
||||
set textContent(v) { this._textContent = v; this.children = []; },
|
||||
set textContent(v) {
|
||||
// Real DOM: assigning textContent ALSO replaces innerHTML with
|
||||
// an entity-escaped representation of the same text. escapeHtml
|
||||
// (utils.js) round-trips via this side effect — without it,
|
||||
// every escapeHtml() call returns '' and renderMarkdown emits
|
||||
// empty <p> tags.
|
||||
this._textContent = v;
|
||||
this.children = [];
|
||||
this._innerHTML = String(v)
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>');
|
||||
},
|
||||
get innerHTML() { return this._innerHTML; },
|
||||
set innerHTML(v) { this._innerHTML = v; this.children = []; },
|
||||
set innerHTML(v) {
|
||||
// Real DOM invalidates the previous textContent when innerHTML
|
||||
// is replaced — leaving _textContent intact would return stale
|
||||
// data from subsequent textContent reads and mask bugs that
|
||||
// depend on innerHTML/textContent consistency. We don't HTML-
|
||||
// parse here, so the cheap correct behavior is to clear
|
||||
// _textContent and let the children-derived fallback in the
|
||||
// textContent getter (which is empty after this children = [])
|
||||
// take over.
|
||||
this._innerHTML = v;
|
||||
this.children = [];
|
||||
this._textContent = '';
|
||||
},
|
||||
get isConnected() {
|
||||
// In real DOM this checks attachment to the document; for the
|
||||
// test harness we approximate via the parent chain. After
|
||||
@@ -304,17 +333,28 @@ function makeEl(tag) {
|
||||
this.parent = null;
|
||||
},
|
||||
querySelectorAll(selector) {
|
||||
// Only supports the literal "pre code.language-mermaid"
|
||||
// selector that postRenderMermaid uses.
|
||||
// Supports the two selectors the post-render passes use:
|
||||
// "pre code.language-mermaid" (postRenderMermaid)
|
||||
// "pre code[class*='language-']" (postRenderHljs)
|
||||
const out = [];
|
||||
const wantsMermaid = selector === "pre code.language-mermaid";
|
||||
function matchesLangAttr(el) {
|
||||
for (const cls of el._classes) {
|
||||
if (cls.startsWith('language-')) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
function walk(node) {
|
||||
for (const c of (node.children || [])) {
|
||||
if (
|
||||
const isCodeInPre =
|
||||
c.tagName === 'CODE' &&
|
||||
c.parent && c.parent.tagName === 'PRE' &&
|
||||
c._classes.has('language-mermaid')
|
||||
) {
|
||||
out.push(c);
|
||||
c.parent && c.parent.tagName === 'PRE';
|
||||
if (isCodeInPre) {
|
||||
if (wantsMermaid) {
|
||||
if (c._classes.has('language-mermaid')) out.push(c);
|
||||
} else if (matchesLangAttr(c)) {
|
||||
out.push(c);
|
||||
}
|
||||
}
|
||||
walk(c);
|
||||
}
|
||||
@@ -352,6 +392,20 @@ global.mermaid = {
|
||||
},
|
||||
};
|
||||
|
||||
// hljs stub. highlightElement mutates the element in place: replaces
|
||||
// innerHTML with a deterministic synthetic span keyed by the source,
|
||||
// and adds the hljs class — same surface postRenderHljs depends on.
|
||||
// hljsHighlightCallCount lets tests assert "ran N times" semantics.
|
||||
let hljsHighlightCallCount = 0;
|
||||
global.hljs = {
|
||||
configure: () => {},
|
||||
highlightElement: (el) => {
|
||||
hljsHighlightCallCount++;
|
||||
el._classes.add('hljs');
|
||||
el._innerHTML = '<span class="hljs-tok">' + el._textContent + '</span>';
|
||||
},
|
||||
};
|
||||
|
||||
vm.runInThisContext(fs.readFileSync(%(utils)s, 'utf8'));
|
||||
vm.runInThisContext(fs.readFileSync(%(renderer)s, 'utf8'));
|
||||
|
||||
@@ -514,7 +568,7 @@ def test_mermaid_cache_evicts_oldest_at_cap() -> None:
|
||||
scenario = """
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
@@ -536,10 +590,10 @@ def test_mermaid_overwrite_does_not_evict() -> None:
|
||||
const cap = _MERMAID_CACHE_MAX;
|
||||
// Fill exactly to cap.
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
|
||||
}
|
||||
// Overwrite an existing entry — must not evict src-0.
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null});
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null}, cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidSvgCache.size,
|
||||
hasOldest: _mermaidSvgCache.has('src-0'),
|
||||
@@ -558,8 +612,8 @@ def test_mermaid_cache_cleared_on_init() -> None:
|
||||
— the rendered output depends on themeVariables which change
|
||||
on init."""
|
||||
scenario = """
|
||||
_cacheMermaidEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null});
|
||||
_cacheMermaidEntry(_mermaidErrorCache, 'src-bad', 'old error');
|
||||
_cacheFifoEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null}, _MERMAID_CACHE_MAX);
|
||||
_cacheFifoEntry(_mermaidErrorCache, 'src-bad', 'old error', _MERMAID_CACHE_MAX);
|
||||
_initMermaid();
|
||||
process.stdout.write(JSON.stringify({
|
||||
svgSize: _mermaidSvgCache.size,
|
||||
@@ -624,3 +678,598 @@ def test_streaming_render_invokes_mermaid_post_render() -> None:
|
||||
"_streamingRenderApply must call postRenderMermaid for "
|
||||
"progressive diagram rendering during streaming"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _normalizeMermaidSource — autoquote labels with bare shape-delimiter
|
||||
# chars. Mermaid rejects unquoted ( ) [ ] { } inside other labels with
|
||||
# a "got 'PS'" parse error (paren-start in shape context). The two
|
||||
# diagrams in the screenshot regression case are encoded here verbatim.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_normalize(source: str) -> str:
|
||||
"""Drive _normalizeMermaidSource against the JS harness and return
|
||||
its output. The function is pure, so no container / mermaid stub
|
||||
setup is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = _normalizeMermaidSource(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
# Diagram 1 from the screenshot regression — unquoted edge labels with
|
||||
# parens and <br/> markers. Mermaid rejects both edge labels with
|
||||
# "got 'PS'"; quoting them resolves it.
|
||||
_SCREENSHOT_DIAGRAM_1_IN = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
" -->|22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)|"
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
" B -->|13 jasl patches<br/>(Python only:<br/>tunings, kernels,"
|
||||
"<br/>warmup, etc.)|"
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_1_OUT = (
|
||||
"flowchart LR\n"
|
||||
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
|
||||
' -->|"22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)"|'
|
||||
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
|
||||
' B -->|"13 jasl patches<br/>(Python only:<br/>tunings, kernels,'
|
||||
'<br/>warmup, etc.)"|'
|
||||
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
|
||||
)
|
||||
|
||||
# Diagram 2 from the screenshot regression — unquoted RECTANGLE node
|
||||
# label `D[untouched<br/>(.so, _version.py,<br/>install-vendored)]`.
|
||||
# Same parser failure mode; quoting the bracket label fixes it.
|
||||
_SCREENSHOT_DIAGRAM_2_IN = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
" B -->|not in archive| D[untouched<br/>(.so, _version.py,"
|
||||
"<br/>install-vendored)]\n"
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
_SCREENSHOT_DIAGRAM_2_OUT = (
|
||||
"flowchart LR\n"
|
||||
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
|
||||
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
|
||||
' B -->|not in archive| D["untouched<br/>(.so, _version.py,'
|
||||
'<br/>install-vendored)"]\n'
|
||||
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("source", "expected"),
|
||||
[
|
||||
(_SCREENSHOT_DIAGRAM_1_IN, _SCREENSHOT_DIAGRAM_1_OUT),
|
||||
(_SCREENSHOT_DIAGRAM_2_IN, _SCREENSHOT_DIAGRAM_2_OUT),
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_fixes_screenshot_diagrams(source: str, expected: str) -> None:
|
||||
"""The two exact diagrams from the screenshot regression. If
|
||||
these stop being rewritten with quoted labels, mermaid will
|
||||
again reject them with `Expecting ... got 'PS'` during live
|
||||
streaming."""
|
||||
assert _run_normalize(source) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"source",
|
||||
[
|
||||
# Clean diagram — no shape delimiters in any label.
|
||||
"graph TD\n A[foo] --> B[bar]",
|
||||
# Edge label with no special chars.
|
||||
"A --> B\nA -->|plain text| B",
|
||||
# Already-correctly-quoted node label.
|
||||
'A["already (quoted)"] --> B',
|
||||
# Already-correctly-quoted edge label.
|
||||
'A -->|"already (quoted)"| B',
|
||||
# Cylinder shape — inner () is part of the shape syntax.
|
||||
"A[(database)] --> B",
|
||||
# Subroutine shape — inner [] is part of the shape syntax.
|
||||
"A[[subroutine]] --> B",
|
||||
# Trapezoid shape — inner / is part of the shape syntax.
|
||||
"A[/trapezoid/] --> B",
|
||||
# Reverse trapezoid.
|
||||
"A[\\trap\\] --> B",
|
||||
# Mermaid directive — braces here are config, not a label.
|
||||
'%%{init: {"theme": "dark"}}%%\ngraph TD\n A --> B',
|
||||
# <br/> tags on their own don't trip quoting.
|
||||
"A[line1<br/>line2] --> B",
|
||||
# Sequence diagram — different grammar; we only target labels
|
||||
# in shape/edge syntax that match the regex anchors.
|
||||
"sequenceDiagram\n A->>B: hello",
|
||||
],
|
||||
)
|
||||
def test_mermaid_autoquote_leaves_valid_source_alone(source: str) -> None:
|
||||
"""The autoquoter must not rewrite syntactically valid Mermaid —
|
||||
a false positive here would break a working diagram. Each case
|
||||
covers a syntax form whose delimiters are intentional and must
|
||||
not be wrapped."""
|
||||
assert _run_normalize(source) == source
|
||||
|
||||
|
||||
def test_mermaid_autoquote_edge_label_with_parens() -> None:
|
||||
"""Bare-parens edge label gets wrapped. The bare `(` would
|
||||
otherwise re-enter Mermaid's shape parser."""
|
||||
src = "A -->|note (with parens)| B"
|
||||
assert _run_normalize(src) == 'A -->|"note (with parens)"| B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_parens() -> None:
|
||||
"""Bare-parens node label gets wrapped."""
|
||||
src = "D[label (foo, bar)]"
|
||||
assert _run_normalize(src) == 'D["label (foo, bar)"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_node_label_with_braces() -> None:
|
||||
"""Bare-braces in a rectangle label get wrapped. (Diamond {}
|
||||
shapes are left alone — only single-bracket [] labels are
|
||||
rewritten.)"""
|
||||
src = "A[config {key: value}]"
|
||||
assert _run_normalize(src) == 'A["config {key: value}"]'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_preserves_br_tag_with_parens() -> None:
|
||||
"""`<br/>` inside a label that also has parens stays — only the
|
||||
quoting needs to be added around the whole label."""
|
||||
src = "A[line1<br/>(line2)] --> B"
|
||||
assert _run_normalize(src) == 'A["line1<br/>(line2)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_autoquote_skips_label_with_internal_quote() -> None:
|
||||
"""If a label contains a literal `"`, wrapping would produce
|
||||
nested unescaped quotes. The autoquoter must punt — leaving the
|
||||
parse error to surface, rather than silently producing a worse
|
||||
one."""
|
||||
src = 'A[he said "hi" (lol)]'
|
||||
assert _run_normalize(src) == src
|
||||
|
||||
|
||||
def test_mermaid_autoquote_multiple_edges_on_one_line() -> None:
|
||||
"""Both edge labels on a single line get rewritten independently."""
|
||||
src = "A -->|first (paren)| B -->|second (paren)| C"
|
||||
expected = 'A -->|"first (paren)"| B -->|"second (paren)"| C'
|
||||
assert _run_normalize(src) == expected
|
||||
|
||||
|
||||
def test_mermaid_autoquote_normalized_source_hits_cache() -> None:
|
||||
"""The SVG cache keys on the normalized source — same malformed
|
||||
input that the LLM streamed earlier still hits the cache on
|
||||
re-render rather than re-invoking mermaid.render every tick."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
renderCalls: renderCallCount,
|
||||
normalized: container.children[0]._attrs['data-mermaid-source'],
|
||||
}));
|
||||
}, 0);
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["renderCalls"] == 1, "second render bypassed the cache"
|
||||
assert out["normalized"] == 'A["label (with parens)"] --> B'
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_populates_on_first_call() -> None:
|
||||
"""First postRenderMermaid call populates _mermaidNormalizeCache
|
||||
with a raw→normalized entry. A second call on identical raw
|
||||
textContent then hits the memo (size stays at 1, no second
|
||||
normalize call), which is the perf-1 fix — avoids re-running
|
||||
split + per-line regex per rAF tick when the diagram hasn't
|
||||
changed."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
const sizeAfterFirst = _mermaidNormalizeCache.size;
|
||||
const cachedNorm = _mermaidNormalizeCache.get(sources[0]);
|
||||
// Re-render on a fresh container with the same source.
|
||||
const container2 = buildContainer(sources);
|
||||
postRenderMermaid(container2);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
sizeAfterFirst: sizeAfterFirst,
|
||||
cachedNorm: cachedNorm,
|
||||
sizeAfterSecond: _mermaidNormalizeCache.size,
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sizeAfterFirst"] == 1, "first call didn't populate normalize memo"
|
||||
assert out["cachedNorm"] == 'A["label (with parens)"] --> B'
|
||||
assert out["sizeAfterSecond"] == 1, (
|
||||
"second call added a new entry — memo missed on identical source"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_is_consulted_before_normalize() -> None:
|
||||
"""Pre-seed _mermaidNormalizeCache with a sentinel value for a
|
||||
raw source. postRenderMermaid must use the sentinel rather than
|
||||
re-running _normalizeMermaidSource. Catches a regression where
|
||||
the memo gets populated but the lookup path is skipped."""
|
||||
bad = "A[label (with parens)] --> B"
|
||||
sentinel = "SENTINEL_FROM_MEMO --> X"
|
||||
raw_js = json.dumps(bad)
|
||||
sentinel_js = json.dumps(sentinel)
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ f"""
|
||||
_mermaidNormalizeCache.set({raw_js}, {sentinel_js});
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {{
|
||||
process.stdout.write(JSON.stringify({{
|
||||
sourceAttr: container.children[0]._attrs['data-mermaid-source'],
|
||||
}}));
|
||||
}}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["sourceAttr"] == sentinel, (
|
||||
"postRenderMermaid bypassed the normalize memo and re-ran normalize"
|
||||
)
|
||||
|
||||
|
||||
def test_mermaid_normalize_memo_distinct_sources_cache_separately() -> None:
|
||||
"""Two distinct raw sources produce two memo entries. Confirms
|
||||
the memo keys on raw textContent, not on something coarser like
|
||||
container identity."""
|
||||
bad1 = "A[label (with parens)] --> B"
|
||||
bad2 = "C[other (label)] --> D"
|
||||
scenario = (
|
||||
_build_mermaid_container_js([bad1, bad2])
|
||||
+ _MERMAID_DRAIN_JS
|
||||
+ """
|
||||
postRenderMermaid(container);
|
||||
setTimeout(() => setTimeout(() => {
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _mermaidNormalizeCache.size,
|
||||
hasBad1: _mermaidNormalizeCache.has(sources[0]),
|
||||
hasBad2: _mermaidNormalizeCache.has(sources[1]),
|
||||
}));
|
||||
}, 0), 0);
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 2
|
||||
assert out["hasBad1"] is True
|
||||
assert out["hasBad2"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Code-fence pairing — close requires \n / EOS, content can't cross
|
||||
# another close-pattern. Repros the streaming bug where ```mermaid +
|
||||
# later ```python were paired by the regex, handing mermaid a
|
||||
# truncated source.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _render_md(source: str) -> str:
|
||||
"""Drive renderMarkdown against the JS harness and return the
|
||||
rendered HTML. The function is a pure string transform; no DOM
|
||||
container scaffolding is required."""
|
||||
scenario = f"""
|
||||
const input = {json.dumps(source)};
|
||||
const output = renderMarkdown(input);
|
||||
process.stdout.write(JSON.stringify({{ output: output }}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
return str(out["output"])
|
||||
|
||||
|
||||
_FENCE = "```"
|
||||
|
||||
|
||||
def test_fence_partial_open_emits_no_code_block() -> None:
|
||||
"""While a fence is still open and there's no other ``` later in
|
||||
the buffer, no <code> block is emitted — the open fence stays as
|
||||
plain markdown text until the real close arrives."""
|
||||
src = "Intro\n" + _FENCE + 'mermaid\nA["x"] -->|note (with parens)| B["y"]\nstill streaming'
|
||||
html = _render_md(src)
|
||||
assert "<code" not in html, f"open fence should not emit <code> mid-stream: {html!r}"
|
||||
|
||||
|
||||
def test_fence_partial_with_later_open_does_not_pair_wrongly() -> None:
|
||||
"""Before the fence-pair fix: an unclosed ```mermaid followed by
|
||||
a ```python (also unclosed) would have paired up as
|
||||
<code class=mermaid>...</code>python..., handing mermaid a
|
||||
truncated source. With the new regex, neither fence emits a
|
||||
block until its OWN closing line arrives."""
|
||||
src = "Intro\n" + _FENCE + "mermaid\nA --> B\n" + _FENCE + 'python\nprint("hi")'
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, (
|
||||
f"mermaid fence should not emit while open: {html!r}"
|
||||
)
|
||||
assert 'class="language-python"' not in html, (
|
||||
f"python fence should not emit while open: {html!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_fence_close_paired_with_next_open_is_rejected() -> None:
|
||||
"""Repro of the live-streaming failure: mermaid fence open, then
|
||||
```python opens and ``` closes the python block. Without the
|
||||
fix, the regex paired mermaid's open with python's *open* (or
|
||||
backtracked all the way to python's close), producing
|
||||
<code class=mermaid>truncated</code>. With the fix mermaid stays
|
||||
open (content can't cross another \\1 run; close must be at line
|
||||
boundary) and only python's pair matches."""
|
||||
src = (
|
||||
"Intro\n"
|
||||
+ _FENCE
|
||||
+ 'mermaid\nA["x"] -->|note (with parens)| B["y"]\n'
|
||||
+ _FENCE
|
||||
+ 'python\nprint("hi")\n'
|
||||
+ _FENCE
|
||||
)
|
||||
html = _render_md(src)
|
||||
assert 'class="language-mermaid"' not in html, f"mermaid fence misparing reintroduced: {html!r}"
|
||||
assert 'class="language-python"' in html, f"python fence on its own should match: {html!r}"
|
||||
|
||||
|
||||
def test_fence_closed_emits_code_block() -> None:
|
||||
"""Baseline: a properly closed fence with its close on its own
|
||||
line emits the <code> block as expected — the anchor doesn't
|
||||
break the normal case."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + "\nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_at_end_of_buffer_emits() -> None:
|
||||
"""A fence that closes at the very end of the buffer (no trailing
|
||||
newline) still emits — the anchor accepts end-of-string as a
|
||||
valid line boundary, so the rehydration / static-render path
|
||||
where the buffer ends cleanly at ``` still works."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
assert "import os" in html
|
||||
|
||||
|
||||
def test_fence_close_with_trailing_whitespace_emits() -> None:
|
||||
"""A close followed only by spaces / tabs before \\n still counts
|
||||
— CommonMark allows trailing whitespace on the close line."""
|
||||
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + " \nAfter"
|
||||
html = _render_md(src)
|
||||
assert 'class="language-python"' in html
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# postRenderHljs — progressive syntax highlighting + source-keyed cache
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_hljs_container_js(blocks: list[tuple[str, str]]) -> str:
|
||||
"""Build a container with <pre><code class="language-LANG"> blocks.
|
||||
|
||||
``blocks`` is a list of ``(language, source)`` tuples — the language
|
||||
becomes the ``language-X`` class, the source becomes textContent."""
|
||||
arr = "[" + ", ".join(f"[{json.dumps(lang)}, {json.dumps(src)}]" for lang, src in blocks) + "]"
|
||||
return f"""
|
||||
function buildHljsContainer(blocks) {{
|
||||
const container = document.createElement('div');
|
||||
for (const [lang, src] of blocks) {{
|
||||
const pre = document.createElement('pre');
|
||||
const code = document.createElement('code');
|
||||
code.classList.add('language-' + lang);
|
||||
code.textContent = src;
|
||||
pre.appendChild(code);
|
||||
container.appendChild(pre);
|
||||
}}
|
||||
return container;
|
||||
}}
|
||||
const blocks = {arr};
|
||||
const container = buildHljsContainer(blocks);
|
||||
"""
|
||||
|
||||
|
||||
def test_hljs_cache_hit_skips_highlight_call() -> None:
|
||||
"""Two postRenderHljs calls on identical source must invoke
|
||||
hljs.highlightElement exactly once — the second call hits the
|
||||
cache and applies the stored markup synchronously. Mirrors the
|
||||
mermaid SVG-cache invariant that lets streamingRender fire on
|
||||
every rAF tick without re-tokenizing every code block."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
const container2 = buildHljsContainer(blocks);
|
||||
postRenderHljs(container2);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
firstHtml: container.children[0].children[0]._innerHTML,
|
||||
secondHtml: container2.children[0].children[0]._innerHTML,
|
||||
secondHasHljsClass: container2.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1, (
|
||||
"second postRenderHljs call invoked highlightElement — cache miss"
|
||||
)
|
||||
assert out["cacheSize"] == 1
|
||||
assert out["firstHtml"] == out["secondHtml"]
|
||||
assert out["secondHasHljsClass"] is True
|
||||
|
||||
|
||||
def test_hljs_distinct_sources_highlight_independently() -> None:
|
||||
"""Distinct sources each trigger one highlight and cache one entry.
|
||||
Cache key includes the source string, not e.g. just the language."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os"), ("python", "print('hi')")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_cache_separates_by_language() -> None:
|
||||
"""Same source text under different language fences must NOT
|
||||
collide in the cache — language is part of the key. Otherwise a
|
||||
`python` block of `foo` and a `ruby` block of `foo` would share
|
||||
a single (wrongly-highlighted) cache entry."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "foo"), ("ruby", "foo")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 2
|
||||
assert out["cacheSize"] == 2
|
||||
|
||||
|
||||
def test_hljs_skips_no_highlight_langs() -> None:
|
||||
"""language-mermaid / language-text / language-plaintext etc. must
|
||||
get the `nohighlight` class without invoking hljs.highlightElement.
|
||||
Highlighting plaintext or mermaid source would be both wasteful
|
||||
and ugly."""
|
||||
scenario = (
|
||||
_build_hljs_container_js(
|
||||
[("mermaid", "graph TD\\nA-->B"), ("text", "plain"), ("plaintext", "p")]
|
||||
)
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
cacheSize: _hljsCache.size,
|
||||
mermaidNoHighlight: container.children[0].children[0]._classes.has('nohighlight'),
|
||||
textNoHighlight: container.children[1].children[0]._classes.has('nohighlight'),
|
||||
plaintextNoHighlight: container.children[2].children[0]._classes.has('nohighlight'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 0
|
||||
assert out["cacheSize"] == 0
|
||||
assert out["mermaidNoHighlight"] is True
|
||||
assert out["textNoHighlight"] is True
|
||||
assert out["plaintextNoHighlight"] is True
|
||||
|
||||
|
||||
def test_hljs_terminal_lang_marks_pre_for_terminal_styling() -> None:
|
||||
"""Shell-family languages (bash / sh / zsh / console / terminal)
|
||||
must add the `code-terminal` class to the parent <pre>, so the
|
||||
stylesheet can give them the terminal look-and-feel."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("bash", "echo hi")])
|
||||
+ """
|
||||
postRenderHljs(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
preHasTerminalClass: container.children[0]._classes.has('code-terminal'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["preHasTerminalClass"] is True
|
||||
|
||||
|
||||
def test_hljs_cache_evicts_oldest_at_cap() -> None:
|
||||
"""FIFO eviction at _HLJS_CACHE_MAX. Mirrors the mermaid cache —
|
||||
prevents unbounded growth on long sessions with many distinct
|
||||
code blocks."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap + 5; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
hasNewest: _hljsCache.has('key-' + (cap + 4)),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is False
|
||||
assert out["hasNewest"] is True
|
||||
|
||||
|
||||
def test_hljs_overwrite_does_not_evict() -> None:
|
||||
"""Overwriting an existing key is an in-place update, not a new
|
||||
insertion — must not evict the oldest unrelated entry. Same
|
||||
invariant as the mermaid cache."""
|
||||
scenario = """
|
||||
const cap = _HLJS_CACHE_MAX;
|
||||
for (let i = 0; i < cap; i++) {
|
||||
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
|
||||
}
|
||||
_cacheFifoEntry(_hljsCache, 'key-5', 'val-updated', cap);
|
||||
process.stdout.write(JSON.stringify({
|
||||
size: _hljsCache.size,
|
||||
hasOldest: _hljsCache.has('key-0'),
|
||||
updated: _hljsCache.get('key-5'),
|
||||
}));
|
||||
"""
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["size"] == 64
|
||||
assert out["hasOldest"] is True, "overwrite evicted oldest unnecessarily"
|
||||
assert out["updated"] == "val-updated"
|
||||
|
||||
|
||||
def test_post_render_markdown_invokes_hljs() -> None:
|
||||
"""postRenderMarkdown is the public end-of-stream entry point and
|
||||
must still run syntax highlighting after the postRenderHljs
|
||||
refactor — regression guard for the public API surface that
|
||||
app.js / coordinator code already call."""
|
||||
scenario = (
|
||||
_build_hljs_container_js([("python", "import os")])
|
||||
+ """
|
||||
postRenderMarkdown(container);
|
||||
process.stdout.write(JSON.stringify({
|
||||
highlightCalls: hljsHighlightCallCount,
|
||||
hasHljsClass: container.children[0].children[0]._classes.has('hljs'),
|
||||
}));
|
||||
"""
|
||||
)
|
||||
out = _run_mermaid_scenario(scenario)
|
||||
assert out["highlightCalls"] == 1
|
||||
assert out["hasHljsClass"] is True
|
||||
|
||||
|
||||
def test_streaming_render_invokes_hljs() -> None:
|
||||
"""_streamingRenderApply must call postRenderHljs so closed code
|
||||
fences appear progressively (syntax-highlighted) during streaming,
|
||||
not only at stream_end via streamingRenderFinalize. The cache
|
||||
keeps the per-tick cost down to a synchronous lookup."""
|
||||
body = _RENDERER_JS.read_text(encoding="utf-8")
|
||||
start = body.index("function _streamingRenderApply")
|
||||
hljs_call = body.find("postRenderHljs(el)", start, start + 4000)
|
||||
assert hljs_call != -1, (
|
||||
"_streamingRenderApply must call postRenderHljs for progressive "
|
||||
"syntax highlighting during streaming"
|
||||
)
|
||||
|
||||
+536
-3
@@ -458,6 +458,213 @@ class TestPlanExec:
|
||||
assert messages[0]["content"] == ChatSession._PLAN_IDENTITY
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests — _exec_task (optional skill substitutes the hardcoded identity)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTaskExec:
|
||||
"""Tests for _exec_task: optional skill= replaces the default persona,
|
||||
but operating guidance (one-shot, tool-use over narration, no follow-ups)
|
||||
is always preserved."""
|
||||
|
||||
@staticmethod
|
||||
def _capture_exec_messages(session, item):
|
||||
"""Run _exec_task with _run_agent patched; return system message text."""
|
||||
captured: dict = {}
|
||||
|
||||
def fake_run_agent(messages, **kwargs):
|
||||
captured["messages"] = list(messages)
|
||||
return "done"
|
||||
|
||||
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
|
||||
session._exec_task(item)
|
||||
return captured["messages"][0]["content"]
|
||||
|
||||
def test_known_skill_renders_into_system_message(self, tmp_db) -> None:
|
||||
"""Validated skill content (with template vars resolved) replaces
|
||||
the default '# Task Agent' persona, but the operating guidance
|
||||
(the numbered list) is preserved — those are sub-agent semantics
|
||||
that a persona should layer on top of, not replace.
|
||||
|
||||
Covers the full prepare→exec round-trip so a future regression
|
||||
in either half (skill not stored on the item, or exec ignoring it)
|
||||
is caught."""
|
||||
session = _make_session()
|
||||
skill = {
|
||||
"name": "research",
|
||||
"content": "# Research Agent\nws={{ws_id}} model={{model}} node={{node_id}}",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
|
||||
# Item carries the minimized projection — name/content/risk_level
|
||||
# only — not the raw prompt_templates row.
|
||||
assert item["skill"] == {
|
||||
"name": "research",
|
||||
"content": skill["content"],
|
||||
"risk_level": "",
|
||||
}
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: research" in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
# Skill persona rendered with template vars resolved
|
||||
assert "# Research Agent" in sys_msg
|
||||
assert f"ws={session._ws_id}" in sys_msg
|
||||
assert f"model={session.model}" in sys_msg
|
||||
# Default persona is gone — skill substitutes for it.
|
||||
assert "# Task Agent" not in sys_msg
|
||||
assert "autonomous task agent with full tool access" not in sys_msg
|
||||
# Operating guidance survives regardless of skill.
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
|
||||
def test_omitted_skill_uses_hardcoded_identity(self, tmp_db) -> None:
|
||||
"""Regression guard: without skill=, the default '# Task Agent'
|
||||
persona AND the operating guidance both appear verbatim.
|
||||
|
||||
Pins the no-skill path so the substitution branch can't
|
||||
accidentally swallow the default case."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
sys_msg = self._capture_exec_messages(session, item)
|
||||
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
|
||||
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
|
||||
# Default-persona literals also present (sanity check on the constant).
|
||||
assert "# Task Agent" in sys_msg
|
||||
assert "autonomous task agent with full tool access" in sys_msg
|
||||
|
||||
@pytest.mark.parametrize("skill_value", ["", " ", "\t\n"])
|
||||
def test_prepare_task_empty_or_whitespace_skill_treated_as_omitted(
|
||||
self, tmp_db, skill_value
|
||||
) -> None:
|
||||
"""Documented contract: ``skill=""`` (and whitespace-only) behaves
|
||||
identically to omitting the skill arg. LLMs sometimes echo empty
|
||||
strings rather than omit the field; this pins the documented
|
||||
behavior so a future refactor of the ``(args.get("skill") or "").strip()``
|
||||
chokepoint can't quietly diverge."""
|
||||
session = _make_session()
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": skill_value})
|
||||
assert item.get("needs_approval") is True
|
||||
assert item["skill"] is None
|
||||
assert "skill:" not in item["header"]
|
||||
|
||||
def test_prepare_task_unknown_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Unknown skill name → clean error item, no approval needed.
|
||||
|
||||
Skill validation lives in _prepare_task so an LLM passing a
|
||||
bogus name fails fast at approval time rather than at exec."""
|
||||
session = _make_session()
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=None):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "ghost"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown skill 'ghost'" in item["error"]
|
||||
assert "skill(action='search')" in item["error"]
|
||||
|
||||
def test_prepare_task_disabled_skill_returns_error(self, tmp_db) -> None:
|
||||
"""Disabled skill → distinct error, mirrors the enabled gate that
|
||||
``_exec_skill(action='load')`` (session.py:8404) and skill-search
|
||||
already apply. Distinct from the unknown-skill phrasing so the
|
||||
LLM's recovery path can tell 'not found' from 'quarantined'."""
|
||||
session = _make_session()
|
||||
disabled_skill = {
|
||||
"name": "retired",
|
||||
"content": "# Retired",
|
||||
"enabled": False,
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=disabled_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "retired"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "is disabled" in item["error"]
|
||||
# Distinct wording from the unknown-skill error, so the LLM can
|
||||
# tell them apart at recovery time.
|
||||
assert "unknown skill" not in item["error"]
|
||||
|
||||
def test_prepare_task_high_risk_skill_surfaces_in_header(self, tmp_db, caplog) -> None:
|
||||
"""High/critical risk skills surface the tier in the approval header
|
||||
and emit a structured warning, mirroring the signal ``_load_skills``
|
||||
emits for session-level skills (session.py:1336)."""
|
||||
import logging
|
||||
|
||||
session = _make_session()
|
||||
risky_skill = {
|
||||
"name": "danger",
|
||||
"content": "# Danger",
|
||||
"enabled": True,
|
||||
"risk_level": "critical",
|
||||
}
|
||||
with (
|
||||
caplog.at_level(logging.WARNING, logger="turnstone.core.session"),
|
||||
patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill),
|
||||
):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "danger"})
|
||||
assert item.get("needs_approval") is True
|
||||
assert "skill: danger" in item["header"]
|
||||
assert "risk: critical" in item["header"]
|
||||
warning_seen = any("high_risk_skill" in r.getMessage() for r in caplog.records)
|
||||
assert warning_seen, "expected task_agent.high_risk_skill warning"
|
||||
|
||||
def test_prepare_task_normal_risk_skill_omits_tier_from_header(self, tmp_db) -> None:
|
||||
"""Header only surfaces high/critical — low/medium/safe skills don't
|
||||
pollute the approval line."""
|
||||
session = _make_session()
|
||||
ok_skill = {
|
||||
"name": "research",
|
||||
"content": "# Research",
|
||||
"enabled": True,
|
||||
"risk_level": "low",
|
||||
}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=ok_skill):
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "skill": "research"})
|
||||
assert "skill: research" in item["header"]
|
||||
assert "risk:" not in item["header"]
|
||||
|
||||
def test_evaluate_intent_projects_skill_for_task_agent(self, tmp_db, monkeypatch) -> None:
|
||||
"""Judge projection includes the skill name so heuristic arg_patterns
|
||||
can match on it and the audit row records which persona was chosen.
|
||||
|
||||
Mirrors the long-standing ``spawn_workstream`` projection at
|
||||
session.py:4603 — without it, policy rules targeting risky
|
||||
skills via ``task_agent`` silently no-op."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
skill = {"name": "research", "content": "# Research", "enabled": True}
|
||||
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
|
||||
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == "research"
|
||||
assert fa["prompt"] == "investigate X"
|
||||
|
||||
def test_evaluate_intent_projects_empty_skill_when_omitted(self, tmp_db, monkeypatch) -> None:
|
||||
"""Symmetric regression guard: no-skill case projects skill="" so
|
||||
the func_args shape is stable across both branches (the judge can
|
||||
always read ``func_args["skill"]`` without a KeyError)."""
|
||||
session = _make_session()
|
||||
fake_verdict = MagicMock()
|
||||
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
|
||||
fake_judge = MagicMock()
|
||||
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
|
||||
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
|
||||
|
||||
item = session._prepare_task("c1", {"prompt": "do x"})
|
||||
session._evaluate_intent([item])
|
||||
|
||||
fa = item["func_args"]
|
||||
assert fa["skill"] == ""
|
||||
assert fa["prompt"] == "do x"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-call model override on plan_agent / task_agent
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -504,9 +711,51 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
# The error guidance must list the available aliases so the LLM can retry.
|
||||
for alias in ("default", "smart", "fast"):
|
||||
# Error guidance lists the aliases the LLM may retry, intentionally
|
||||
# excluding ``default`` — that alias is operator-only (see
|
||||
# ``test_prepare_plan_default_model_rejected``). Surfacing it here
|
||||
# would re-enable the per-role-override bypass even though the
|
||||
# tool description hides it.
|
||||
for alias in ("smart", "fast"):
|
||||
assert alias in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected(self, tmp_db) -> None:
|
||||
"""``model="default"`` is rejected even when the alias exists in
|
||||
the registry — bypasses the operator-configured ``plan_alias``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
assert "Omit `model=`" in item["error"]
|
||||
|
||||
def test_prepare_plan_default_model_rejected_with_whitespace(self, tmp_db) -> None:
|
||||
"""The ``default`` rejection runs after ``strip()`` so leading/
|
||||
trailing whitespace can't sneak the alias past the carve-out."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": " default "})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
def test_prepare_plan_unknown_model_with_only_default_in_registry(self, tmp_db) -> None:
|
||||
"""When the registry holds only the reserved ``default`` alias
|
||||
(single-CLI-model back-compat), the unknown-alias error must say
|
||||
'(no alternative aliases configured — omit `model=`)' — not the
|
||||
misleading '(no registry configured)' that suggests routing isn't
|
||||
wired up at all."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
item = session._prepare_plan("c1", {"goal": "do x", "model": "bogus"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "no alternative aliases configured" in item["error"]
|
||||
assert "no registry configured" not in item["error"]
|
||||
|
||||
# ---- _prepare_task ----
|
||||
|
||||
@@ -526,6 +775,15 @@ class TestAgentModelOverride:
|
||||
assert item.get("needs_approval") is False
|
||||
assert "error" in item
|
||||
assert "unknown model alias 'bogus'" in item["error"]
|
||||
assert "default" not in item["error"]
|
||||
|
||||
def test_prepare_task_default_model_rejected(self, tmp_db) -> None:
|
||||
"""Symmetric carve-out for task_agent — see
|
||||
``test_prepare_plan_default_model_rejected``."""
|
||||
session = _make_session(registry=self._registry(), model_alias="default")
|
||||
item = session._prepare_task("c1", {"prompt": "do x", "model": "default"})
|
||||
assert item.get("needs_approval") is False
|
||||
assert "'default' is not a selectable model alias" in item["error"]
|
||||
|
||||
# ---- tool description rendering ----
|
||||
|
||||
@@ -544,8 +802,11 @@ class TestAgentModelOverride:
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None, f"{name} missing from session tools"
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
for alias in ("default", "smart", "fast"):
|
||||
for alias in ("smart", "fast"):
|
||||
assert f"`{alias}`" in desc, f"alias {alias} missing from {desc!r}"
|
||||
# ``default`` is intentionally hidden — see
|
||||
# ``test_render_omits_default_alias_from_description``.
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_no_op_without_registry(self, tmp_db) -> None:
|
||||
"""No registry → leave the placeholder description untouched."""
|
||||
@@ -576,6 +837,82 @@ class TestAgentModelOverride:
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`bigboi`" in desc
|
||||
|
||||
def test_render_omits_default_alias_from_description(self, tmp_db) -> None:
|
||||
"""The ``default`` alias is filtered from the LLM-facing alias list.
|
||||
|
||||
Reading "default" as English ("use the default") and passing it
|
||||
explicitly bypasses the operator-configured per-role plan_alias /
|
||||
task_alias. The LLM should reach the per-role default by omitting
|
||||
``model=`` instead.
|
||||
"""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"gh200": ModelConfig("gh200", "x", "x", "m"),
|
||||
"opus-4.7": ModelConfig("opus-4.7", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
for name in ("plan_agent", "task_agent"):
|
||||
tool = self._agent_tool(session, name)
|
||||
assert tool is not None
|
||||
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`gh200`" in desc
|
||||
assert "`opus-4.7`" in desc
|
||||
assert "`default`" not in desc
|
||||
|
||||
def test_render_falls_back_to_base_when_only_default_alias(self, tmp_db) -> None:
|
||||
"""Single-CLI-model registries (only ``default`` in registry) leave
|
||||
the base description untouched — the LLM sees ``"No alternative
|
||||
aliases configured"`` rather than an empty alias list."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={"default": ModelConfig("default", "x", "x", "m")},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_refresh_into_only_default_resets_to_base(self, tmp_db) -> None:
|
||||
"""A reload that drops the registry to only ``default`` must clear
|
||||
stale alias names from the previously-rendered tool descriptions —
|
||||
not return early and leave them in place."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
reg = ModelRegistry(
|
||||
models={
|
||||
"default": ModelConfig("default", "x", "x", "m"),
|
||||
"smart": ModelConfig("smart", "x", "x", "m"),
|
||||
"fast": ModelConfig("fast", "x", "x", "m"),
|
||||
},
|
||||
default="default",
|
||||
)
|
||||
session = _make_session(registry=reg, model_alias="default")
|
||||
# Sanity: initial render carries the non-default aliases.
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" in desc and "`fast`" in desc
|
||||
|
||||
# Reload the registry down to only ``default`` (admin removed
|
||||
# every other model definition).
|
||||
reg.reload({"default": ModelConfig("default", "x", "x", "m")}, "default")
|
||||
session.refresh_agent_tool_schemas()
|
||||
|
||||
plan_tool = self._agent_tool(session, "plan_agent")
|
||||
assert plan_tool is not None
|
||||
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
|
||||
assert "`smart`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "`fast`" not in desc, f"stale alias survived reload: {desc!r}"
|
||||
assert "No alternative aliases configured" in desc
|
||||
|
||||
def test_module_level_constants_not_mutated(self, tmp_db) -> None:
|
||||
"""Rendering must not pollute the module-level TOOLS list shared
|
||||
across all sessions."""
|
||||
@@ -1974,6 +2311,202 @@ class TestCoordinatorMemoryScope:
|
||||
assert scopes == ["workstream", "user", "global"]
|
||||
|
||||
|
||||
class TestMemoryToolAudit:
|
||||
"""Mutating memory tool actions emit audit rows.
|
||||
|
||||
Closes the gap that masked the May 2026 vllm_fork_overlay_pattern
|
||||
investigation: only the admin-console DELETE route emitted
|
||||
``memory.delete``, so a long-running session whose memory was
|
||||
deleted via the admin UI couldn't tell from logs alone whether the
|
||||
row had been deleted out-of-band, never persisted, or was never
|
||||
visible. Read actions (get/search/list) intentionally stay
|
||||
un-audited — auditing reads would multiply audit volume without
|
||||
forensic value.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _audit_rows(action: str) -> list[dict]:
|
||||
from turnstone.core.storage._registry import get_storage
|
||||
|
||||
return get_storage().list_audit_events(action=action)
|
||||
|
||||
def test_save_new_emits_memory_save(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha content",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row["user_id"] == "user-1"
|
||||
assert row["resource_type"] == "memory"
|
||||
assert row["resource_id"] # memory_id was populated
|
||||
detail = json.loads(row["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["scope_id"] == "user-1"
|
||||
assert detail["type"] == "reference"
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
# The "create" path must NOT also stamp an update row.
|
||||
assert self._audit_rows("memory.update") == []
|
||||
|
||||
def test_save_global_scope_emits_empty_scope_id(self, tmp_db):
|
||||
"""Global memories have no scope_id — the audit row's detail
|
||||
must still carry the key (with value ``""``) so a forensic
|
||||
consumer can distinguish ``scope='global'`` from a row that
|
||||
forgot to populate ``scope_id`` for a scoped write."""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_global",
|
||||
"content": "shared content",
|
||||
"scope": "global",
|
||||
},
|
||||
)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
rows = self._audit_rows("memory.save")
|
||||
assert len(rows) == 1
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["scope"] == "global"
|
||||
assert detail["scope_id"] == ""
|
||||
assert detail["ws_id"] == "ws-1"
|
||||
|
||||
def test_save_upsert_emits_memory_update(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
for content in ("first", "second"):
|
||||
item = session._prepare_memory(
|
||||
"call_x",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": content,
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(item)
|
||||
|
||||
saves = self._audit_rows("memory.save")
|
||||
updates = self._audit_rows("memory.update")
|
||||
assert len(saves) == 1
|
||||
assert len(updates) == 1
|
||||
# Same memory_id on both rows — the update audits the row save created.
|
||||
assert saves[0]["resource_id"] == updates[0]["resource_id"]
|
||||
|
||||
def test_delete_emits_memory_delete(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
save_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
"type": "reference",
|
||||
},
|
||||
)
|
||||
session._exec_memory(save_item)
|
||||
saved_memory_id = self._audit_rows("memory.save")[0]["resource_id"]
|
||||
|
||||
delete_item = session._prepare_memory(
|
||||
"call_2",
|
||||
{"action": "delete", "name": "fact_one", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "Deleted memory" in msg
|
||||
|
||||
rows = self._audit_rows("memory.delete")
|
||||
assert len(rows) == 1
|
||||
# resource_id must point at the same row save audited — proves
|
||||
# delete-by-name resolved to the right row before recording.
|
||||
assert rows[0]["resource_id"] == saved_memory_id
|
||||
detail = json.loads(rows[0]["detail"])
|
||||
assert detail["name"] == "fact_one"
|
||||
assert detail["scope"] == "user"
|
||||
assert detail["type"] == "reference"
|
||||
|
||||
def test_delete_not_found_emits_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
delete_item = session._prepare_memory(
|
||||
"call_1",
|
||||
{"action": "delete", "name": "no_such_mem", "scope": "user"},
|
||||
)
|
||||
_, msg = session._exec_memory(delete_item)
|
||||
assert "not found" in msg
|
||||
assert self._audit_rows("memory.delete") == []
|
||||
|
||||
def test_reads_emit_no_audit(self, tmp_db):
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
session._exec_memory(
|
||||
session._prepare_memory(
|
||||
"call_save",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
for spec in (
|
||||
{"action": "get", "name": "fact_one", "scope": "user"},
|
||||
{"action": "search", "query": "fact"},
|
||||
{"action": "list"},
|
||||
):
|
||||
item = session._prepare_memory("call_read", spec)
|
||||
assert "error" not in item
|
||||
session._exec_memory(item)
|
||||
|
||||
# Only the save above should have audited.
|
||||
save_count = len(self._audit_rows("memory.save"))
|
||||
update_count = len(self._audit_rows("memory.update"))
|
||||
delete_count = len(self._audit_rows("memory.delete"))
|
||||
assert (save_count, update_count, delete_count) == (1, 0, 0)
|
||||
|
||||
def test_audit_failure_does_not_break_tool_call(self, tmp_db):
|
||||
"""A blow-up inside record_audit must not propagate to the LLM.
|
||||
|
||||
Auditing is best-effort instrumentation; a storage hiccup that
|
||||
prevents the audit row from landing must not also lose the
|
||||
save/delete the user actually asked for.
|
||||
"""
|
||||
session = _make_session(ws_id="ws-1", user_id="user-1")
|
||||
item = session._prepare_memory(
|
||||
"call_1",
|
||||
{
|
||||
"action": "save",
|
||||
"name": "fact_one",
|
||||
"content": "alpha",
|
||||
"scope": "user",
|
||||
},
|
||||
)
|
||||
with patch(
|
||||
"turnstone.core.audit.record_audit",
|
||||
side_effect=RuntimeError("audit storage exploded"),
|
||||
):
|
||||
_, msg = session._exec_memory(item)
|
||||
assert "Saved memory 'fact_one'" in msg
|
||||
# The save itself still landed.
|
||||
from turnstone.core.memory import get_structured_memory_by_name
|
||||
|
||||
assert get_structured_memory_by_name("fact_one", "user", "user-1") is not None
|
||||
|
||||
|
||||
class TestPerKindToolVariants:
|
||||
"""Verify the ``kind_variants`` metadata applies per-kind tool overrides.
|
||||
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
"""Tests for :meth:`ChatSession._format_backend_error`.
|
||||
|
||||
The helper turns bare backend-boundary exceptions (httpx ``ReadTimeout``,
|
||||
OpenAI SDK ``APITimeoutError`` / ``APIConnectionError`` /
|
||||
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``) into
|
||||
operator-actionable messages that include the provider, base URL, and
|
||||
model. We bind the method to lightweight stubs rather than constructing
|
||||
a full :class:`ChatSession`: the helper only reads ``self.client``,
|
||||
``self._provider``, ``self.model``, and ``self._model_alias``, so a
|
||||
SimpleNamespace stub exercises the same surface without dragging in the
|
||||
storage / prompt composition fixtures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
|
||||
|
||||
def _stub(
|
||||
*,
|
||||
base_url: str = "http://192.168.0.5:8000/v1",
|
||||
provider_name: str = "openai-compatible",
|
||||
model: str = "flatspark",
|
||||
model_alias: str | None = "flatspark",
|
||||
client_attr: str = "base_url",
|
||||
) -> Any:
|
||||
"""Build a minimal session-like stub for ``_format_backend_error``.
|
||||
|
||||
``client_attr`` selects which attribute on the client carries the
|
||||
URL — both ``base_url`` (OpenAI / Anthropic SDK public surface) and
|
||||
``_base_url`` (httpx fallback) are exercised by the helper.
|
||||
"""
|
||||
client_kwargs: dict[str, Any] = {client_attr: base_url}
|
||||
return SimpleNamespace(
|
||||
client=SimpleNamespace(**client_kwargs),
|
||||
_provider=SimpleNamespace(provider_name=provider_name),
|
||||
model=model,
|
||||
_model_alias=model_alias,
|
||||
)
|
||||
|
||||
|
||||
def _format(stub: Any, exc: BaseException) -> str | None:
|
||||
"""Invoke the method as if on a real session — ``__func__`` skips
|
||||
the descriptor protocol so we can pass any object as ``self``."""
|
||||
return ChatSession._format_backend_error(stub, exc) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synthetic exception classes — class name is what the helper matches on,
|
||||
# so we don't need real httpx / openai imports here.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# N818 (Error suffix on Exception names) is intentionally suppressed
|
||||
# for the four classes below — they exist to impersonate httpx /
|
||||
# Anthropic SDK exception class names verbatim, since the formatter
|
||||
# matches by class name. Renaming them defeats the test.
|
||||
|
||||
|
||||
class ReadTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class WriteTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APITimeoutError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ConnectError(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class ConnectTimeout(Exception): # noqa: N818
|
||||
pass
|
||||
|
||||
|
||||
class APIConnectionError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class NotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class AuthenticationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class PermissionDeniedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class RateLimitError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class SomeUnrelatedError(Exception):
|
||||
"""Outside the recognised set — should fall through to ``None``."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Known categories — each branch produces an operator-actionable message
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ReadTimeout, WriteTimeout, APITimeoutError])
|
||||
def test_timeout_message_names_backend_and_model(exc_cls):
|
||||
msg = _format(_stub(), exc_cls())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "openai-compatible" in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
assert "model=flatspark" in msg
|
||||
assert "wedged" in msg
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [ConnectError, ConnectTimeout, APIConnectionError])
|
||||
def test_connect_message_says_unreachable(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("dial tcp: i/o timeout"))
|
||||
assert msg is not None
|
||||
assert "Backend unreachable" in msg
|
||||
assert exc_cls.__name__ in msg
|
||||
assert "http://192.168.0.5:8000/v1" in msg
|
||||
# Raw exception text is preserved as a tail for grep-correlation.
|
||||
assert "dial tcp: i/o timeout" in msg
|
||||
|
||||
|
||||
def test_not_found_points_at_model_name_mismatch():
|
||||
msg = _format(_stub(model="flatspark"), NotFoundError("model flatspark not found"))
|
||||
assert msg is not None
|
||||
assert "Backend reports model not loaded" in msg
|
||||
assert "no model named 'flatspark'" in msg
|
||||
assert "/v1/models" in msg # operator hint
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exc_cls", [AuthenticationError, PermissionDeniedError])
|
||||
def test_auth_message_mentions_api_key(exc_cls):
|
||||
msg = _format(_stub(), exc_cls("invalid api key"))
|
||||
assert msg is not None
|
||||
assert "Backend rejected credentials" in msg
|
||||
assert "API key" in msg
|
||||
|
||||
|
||||
def test_rate_limit_message():
|
||||
msg = _format(_stub(), RateLimitError("limit exceeded"))
|
||||
assert msg is not None
|
||||
assert "Backend rate-limited" in msg
|
||||
assert "limit exceeded" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fall-through + degradation behaviour
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unknown_exception_returns_none():
|
||||
assert _format(_stub(), SomeUnrelatedError("anything")) is None
|
||||
|
||||
|
||||
def test_unknown_exception_value_error_returns_none():
|
||||
assert _format(_stub(), ValueError("not a backend error")) is None
|
||||
|
||||
|
||||
def test_trailing_slash_and_query_string_stripped():
|
||||
msg = _format(
|
||||
_stub(base_url="http://node-a:8000/v1/?api_key=secret&foo=1"),
|
||||
ReadTimeout(),
|
||||
)
|
||||
assert msg is not None
|
||||
assert "http://node-a:8000/v1" in msg
|
||||
# Query string (which may carry credentials) is stripped before the
|
||||
# message is built — sanitize_error_text is a second line of defence
|
||||
# but the helper itself must not embed query params verbatim.
|
||||
assert "api_key" not in msg
|
||||
assert "secret" not in msg
|
||||
|
||||
|
||||
def test_missing_provider_degrades_to_placeholder():
|
||||
stub = _stub()
|
||||
stub._provider = None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
# No exception, no NoneType formatting leaking through.
|
||||
assert "Backend timeout" in msg
|
||||
assert "from ?" in msg or "openai-compatible" not in msg
|
||||
|
||||
|
||||
def test_client_base_url_raises_degrades_gracefully():
|
||||
class _BadClient:
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
stub = SimpleNamespace(
|
||||
client=_BadClient(),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
)
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "Backend timeout" in msg
|
||||
# base_url accessor blew up — message still renders with placeholder.
|
||||
assert "at ?" in msg
|
||||
|
||||
|
||||
def test_httpx_underscore_base_url_fallback():
|
||||
# httpx client carries ``_base_url`` on some versions instead of
|
||||
# ``base_url`` — the helper checks both.
|
||||
stub = _stub(base_url="http://alt-host:9000", client_attr="_base_url")
|
||||
# SimpleNamespace exposes the attr; remove the public one so the
|
||||
# fallback path is exercised.
|
||||
delattr(stub.client, "base_url") if hasattr(stub.client, "base_url") else None
|
||||
msg = _format(stub, ReadTimeout())
|
||||
assert msg is not None
|
||||
assert "http://alt-host:9000" in msg
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration with _record_fatal_error — original bare-class string is
|
||||
# replaced by the enriched message when the exception type is recognised.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _record_fatal_stub(ui: Any, captured: dict[str, str]) -> Any:
|
||||
"""Build a stub for the ``_record_fatal_error`` integration tests.
|
||||
|
||||
``_record_fatal_error`` calls ``self._format_backend_error(...)``
|
||||
internally, so the stub binds the unbound method to itself rather
|
||||
than relying on Python's descriptor protocol (which only kicks in
|
||||
when ``self`` is a real instance of the class)."""
|
||||
stub = SimpleNamespace(
|
||||
client=SimpleNamespace(base_url="http://192.168.0.5:8000/v1"),
|
||||
_provider=SimpleNamespace(provider_name="openai-compatible"),
|
||||
model="flatspark",
|
||||
_model_alias="flatspark",
|
||||
_ws_id="ws-test",
|
||||
_has_persisted_error=False,
|
||||
ui=ui,
|
||||
_emit_state=lambda state: captured.setdefault("state", state),
|
||||
)
|
||||
stub._format_backend_error = lambda exc: ChatSession._format_backend_error(stub, exc)
|
||||
return stub
|
||||
|
||||
|
||||
def test_record_fatal_uses_enriched_message_for_known(monkeypatch):
|
||||
"""End-to-end: a recognised exception flows through
|
||||
``_record_fatal_error`` and the enriched text reaches both the UI
|
||||
and the persist hook."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
# Skip the credential-redaction module (and its module-level
|
||||
# regex compile) by returning the input verbatim — the helper
|
||||
# under test produces no credentials.
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ReadTimeout()) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors, "UI never received error"
|
||||
assert "Backend timeout" in ui.errors[0]
|
||||
assert "ReadTimeout" in ui.errors[0]
|
||||
assert captured["persist"] == ui.errors[0]
|
||||
assert captured["state"] == "error"
|
||||
assert stub._has_persisted_error is True
|
||||
|
||||
|
||||
def test_record_fatal_falls_back_for_unknown(monkeypatch):
|
||||
"""An unrecognised exception keeps the legacy
|
||||
``f"{type(exc).__name__}: {exc}"`` shape so we don't regress
|
||||
existing call sites that grep on it."""
|
||||
|
||||
captured: dict[str, str] = {}
|
||||
|
||||
def fake_persist(ws_id: str, msg: str) -> None:
|
||||
captured["persist"] = msg
|
||||
|
||||
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
|
||||
return text
|
||||
|
||||
import turnstone.core.memory as memory_mod
|
||||
|
||||
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
|
||||
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
|
||||
|
||||
class _UI:
|
||||
def __init__(self) -> None:
|
||||
self.errors: list[str] = []
|
||||
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
ui = _UI()
|
||||
stub = _record_fatal_stub(ui, captured)
|
||||
|
||||
ChatSession._record_fatal_error(stub, ValueError("plain old error")) # type: ignore[arg-type]
|
||||
|
||||
assert ui.errors == ["ValueError: plain old error"]
|
||||
assert captured["persist"] == "ValueError: plain old error"
|
||||
@@ -0,0 +1,222 @@
|
||||
"""Tests for the storage layer's cross-process ``notify`` / ``listen`` API.
|
||||
|
||||
Covers SQLite (synthetic-sweep + in-process fan-out) and PostgreSQL
|
||||
(real ``LISTEN``/``NOTIFY``). The PG-only cases are gated on the
|
||||
``--storage-backend=postgresql`` flag so they no-op on default CI runs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def _drain_until(stream, predicate, deadline_sec: float = 5.0):
|
||||
"""Poll ``stream`` until ``predicate`` matches one of the drained notifies.
|
||||
|
||||
Returns the matching notify or raises ``TimeoutError``. Tests use
|
||||
this so timing flakes against the bounded-blocking ``poll`` shape
|
||||
don't masquerade as logic bugs.
|
||||
"""
|
||||
deadline = time.monotonic() + deadline_sec
|
||||
while time.monotonic() < deadline:
|
||||
remaining = max(0.05, deadline - time.monotonic())
|
||||
for n in stream.poll(min(0.5, remaining)):
|
||||
if predicate(n):
|
||||
return n
|
||||
msg = "no matching notify drained before deadline"
|
||||
raise TimeoutError(msg)
|
||||
|
||||
|
||||
class TestSqliteNotify:
|
||||
"""SQLite path: in-process fan-out + synthetic sweep."""
|
||||
|
||||
def test_notify_no_listeners_is_noop(self, storage):
|
||||
# No exception, no side effect — safe to always call from dispatch.
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
|
||||
def test_notify_delivers_to_in_process_listener(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("services", '{"op": "INSERT"}')
|
||||
got = _drain_until(stream, lambda n: n.payload == '{"op": "INSERT"}')
|
||||
assert got.channel == "services"
|
||||
# ``pid`` is 0 on the SQLite synthetic path and the sending
|
||||
# backend's PID on Postgres — both are valid notify shapes,
|
||||
# so don't assert on the value here.
|
||||
|
||||
def test_notify_filters_by_channel(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
storage.notify("other_channel", "ignored")
|
||||
storage.notify("services", "wanted")
|
||||
got = _drain_until(stream, lambda n: True)
|
||||
assert got.payload == "wanted"
|
||||
|
||||
def test_multiple_listeners_each_get_event(self, storage):
|
||||
# Two streams open on the same channel; each gets its own copy.
|
||||
with storage.listen(["services"]) as s1, storage.listen(["services"]) as s2:
|
||||
storage.notify("services", "broadcast")
|
||||
got1 = _drain_until(s1, lambda n: True)
|
||||
got2 = _drain_until(s2, lambda n: True)
|
||||
assert got1.payload == "broadcast"
|
||||
assert got2.payload == "broadcast"
|
||||
|
||||
def test_close_stops_stream(self, storage):
|
||||
with storage.listen(["services"]) as stream:
|
||||
pass
|
||||
# After context exit, the stream is closed; poll returns [] without
|
||||
# blocking. A second close() is idempotent.
|
||||
assert stream.poll(0.05) == []
|
||||
stream.close()
|
||||
|
||||
def test_synthetic_sweep_emits_after_interval(self, storage, _is_sqlite):
|
||||
# Synthetic sweep is fundamentally SQLite-specific — the PG path
|
||||
# uses real ``LISTEN``/``NOTIFY`` and has no sweep tick. Gate
|
||||
# so the test doesn't false-fail by waiting for a "sweep" notify
|
||||
# that the PG stream will never produce.
|
||||
with storage.listen(["services"], sweep_interval=0.1) as stream:
|
||||
# First poll: not yet at the interval, so likely empty.
|
||||
stream.poll(0.05)
|
||||
# Wait past the interval, then poll again — should emit a
|
||||
# synthetic-sweep notify per declared channel.
|
||||
time.sleep(0.15)
|
||||
got = _drain_until(stream, lambda n: n.payload == "sweep")
|
||||
assert got.channel == "services"
|
||||
assert got.payload == "sweep"
|
||||
|
||||
def test_empty_channel_list_yields_empty_stream(self, storage):
|
||||
with storage.listen([]) as stream:
|
||||
# No channels — poll returns [] regardless of how long we wait.
|
||||
assert stream.poll(0.05) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PostgreSQL path — gated on --storage-backend=postgresql.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_postgres(storage):
|
||||
"""Skip the wrapped test when the active backend isn't Postgres."""
|
||||
if storage.__class__.__name__ != "PostgreSQLBackend":
|
||||
pytest.skip("PostgreSQL-specific test")
|
||||
return True
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _is_sqlite(storage):
|
||||
"""Skip the wrapped test when the active backend isn't SQLite."""
|
||||
if storage.__class__.__name__ != "SQLiteBackend":
|
||||
pytest.skip("SQLite-specific test")
|
||||
return True
|
||||
|
||||
|
||||
class TestPostgresNotify:
|
||||
def test_round_trip(self, storage, _is_postgres):
|
||||
# Open a listener, fire a notify on a regular pooled connection,
|
||||
# drain the listener within a reasonable bound (PG NOTIFY is
|
||||
# typically sub-100ms on a local socket).
|
||||
with storage.listen(["pytest_round_trip"]) as stream:
|
||||
# Tiny sleep so the LISTEN settles before the NOTIFY fires —
|
||||
# otherwise the notify can arrive on the connection before
|
||||
# the LISTEN is registered (race only visible in tests).
|
||||
time.sleep(0.05)
|
||||
storage.notify("pytest_round_trip", '{"hello": "world"}')
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "pytest_round_trip"
|
||||
assert got.payload == '{"hello": "world"}'
|
||||
assert got.pid > 0
|
||||
|
||||
def test_concurrent_notifies_all_arrive(self, storage, _is_postgres):
|
||||
with storage.listen(["pytest_concurrent"]) as stream:
|
||||
time.sleep(0.05)
|
||||
for i in range(5):
|
||||
storage.notify("pytest_concurrent", str(i))
|
||||
seen: set[str] = set()
|
||||
deadline = time.monotonic() + 3.0
|
||||
while len(seen) < 5 and time.monotonic() < deadline:
|
||||
for n in stream.poll(0.2):
|
||||
seen.add(n.payload)
|
||||
assert seen == {"0", "1", "2", "3", "4"}
|
||||
|
||||
def test_close_aborts_blocked_poll(self, storage, _is_postgres):
|
||||
# poll() should return promptly once close() runs on another thread.
|
||||
with storage.listen(["pytest_close"]) as stream:
|
||||
done = threading.Event()
|
||||
result: list[list] = []
|
||||
|
||||
def _poll_long():
|
||||
result.append(stream.poll(5.0))
|
||||
done.set()
|
||||
|
||||
t = threading.Thread(target=_poll_long, daemon=True)
|
||||
t.start()
|
||||
time.sleep(0.1)
|
||||
stream.close()
|
||||
assert done.wait(2.0), "close() did not unblock poll()"
|
||||
# No notify arrived, so the polled batch is empty — but the
|
||||
# poll loop must have exited well under the 5 s timeout.
|
||||
assert result == [[]]
|
||||
|
||||
|
||||
class TestServicesTriggerFilter:
|
||||
"""Migration 053's trigger: fires on real changes, quiet on heartbeats.
|
||||
|
||||
PG-only — the SQLite path has no trigger and is covered by
|
||||
:class:`TestSqliteNotify`. Verifies the in-trigger ``IS NOT DISTINCT
|
||||
FROM`` filter — a heartbeat-only UPDATE (same url + same metadata,
|
||||
only ``last_heartbeat`` changed) must NOT emit a NOTIFY, since
|
||||
``register_service`` runs the same UPSERT on every 30 s tick × N
|
||||
nodes and the channel would otherwise flood.
|
||||
"""
|
||||
|
||||
def test_insert_fires_notify(self, storage, _is_postgres):
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node", "http://127.0.0.1:1")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert got.channel == "services"
|
||||
assert '"op": "INSERT"' in got.payload or "INSERT" in got.payload
|
||||
# Cleanup so concurrent suites don't pick up the row.
|
||||
storage.deregister_service("server", "pytest-trigger-node")
|
||||
|
||||
def test_delete_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-del", "http://127.0.0.1:2")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.deregister_service("server", "pytest-trigger-node-del")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "DELETE" in got.payload
|
||||
|
||||
def test_url_change_update_fires_notify(self, storage, _is_postgres):
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:3")
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
# UPSERT with different url — UPDATE path with url diff,
|
||||
# trigger must fire.
|
||||
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:9")
|
||||
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
|
||||
assert "UPDATE" in got.payload
|
||||
storage.deregister_service("server", "pytest-trigger-node-url")
|
||||
|
||||
def test_heartbeat_only_update_is_quiet(self, storage, _is_postgres):
|
||||
# Open the LISTEN session FIRST so PG delivers the INSERT NOTIFY
|
||||
# to this connection — pg_notify routes only to sessions that
|
||||
# have LISTENed at COMMIT time, so an INSERT committed before the
|
||||
# listen opens would be lost and the drain would time out instead
|
||||
# of exercising the heartbeat-quiet check below.
|
||||
with storage.listen(["services"]) as stream:
|
||||
time.sleep(0.05)
|
||||
storage.register_service("server", "pytest-trigger-node-hb", "http://127.0.0.1:4")
|
||||
# Drain the INSERT notify so subsequent polls see only what
|
||||
# heartbeats emit (if anything).
|
||||
_drain_until(stream, lambda n: True, deadline_sec=2.0)
|
||||
# Now fire a heartbeat tick — same url + same metadata,
|
||||
# only last_heartbeat updates. Trigger must NOT emit.
|
||||
storage.heartbeat_service("server", "pytest-trigger-node-hb")
|
||||
# Poll long enough that any spurious notify would have
|
||||
# arrived; the channel must stay silent.
|
||||
spurious = stream.poll(0.5)
|
||||
assert spurious == [], f"heartbeat-only update emitted unexpected notify: {spurious}"
|
||||
storage.deregister_service("server", "pytest-trigger-node-hb")
|
||||
@@ -29,7 +29,7 @@ class TestVersionHtml:
|
||||
def test_vendored_katex_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">'
|
||||
html = '<link rel="stylesheet" href="/shared/katex-0.16.46/katex.min.css">'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
@@ -43,14 +43,14 @@ class TestVersionHtml:
|
||||
def test_vendored_mermaid_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<script src="/shared/mermaid-11.14.0/mermaid.min.js"></script>'
|
||||
html = '<script src="/shared/mermaid-11.15.0/mermaid.min.js"></script>'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
def test_vendored_hls_skipped(self):
|
||||
from turnstone.core.web_helpers import version_html
|
||||
|
||||
html = '<script src="/shared/hls-1.6.15/hls.min.js"></script>'
|
||||
html = '<script src="/shared/hls-1.6.16/hls.min.js"></script>'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged
|
||||
|
||||
@@ -76,7 +76,7 @@ class TestVersionHtml:
|
||||
|
||||
html = (
|
||||
'<link rel="stylesheet" href="/shared/base.css">\n'
|
||||
'<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">\n'
|
||||
'<link rel="stylesheet" href="/shared/katex-0.16.46/katex.min.css">\n'
|
||||
'<link rel="stylesheet" href="/static/style.css">\n'
|
||||
'<script src="/shared/utils.js"></script>\n'
|
||||
'<script src="/shared/hljs-11.11.1/highlight.min.js"></script>\n'
|
||||
@@ -88,7 +88,7 @@ class TestVersionHtml:
|
||||
assert f'/shared/utils.js?v={__version__}"' in result
|
||||
assert f'/static/app.js?v={__version__}"' in result
|
||||
# Vendored libs unchanged
|
||||
assert '/shared/katex-0.16.44/katex.min.css"' in result
|
||||
assert '/shared/katex-0.16.46/katex.min.css"' in result
|
||||
assert '/shared/hljs-11.11.1/highlight.min.js"' in result
|
||||
|
||||
def test_version_matches_package(self):
|
||||
|
||||
@@ -62,6 +62,13 @@
|
||||
[database]
|
||||
# url = "" # postgres://user:pass@host/db or /path/to.db
|
||||
# env: TURNSTONE_DB_URL
|
||||
# listen_url = "" # direct-to-postgres URL for the console's
|
||||
# dedicated LISTEN connection. Set this when
|
||||
# `url` points at pgbouncer in transaction
|
||||
# pooling mode (LISTEN holds session state and
|
||||
# is incompatible with transaction pooling —
|
||||
# see docs/pgbouncer.md). Defaults to `url`
|
||||
# when unset. env: TURNSTONE_DB_LISTEN_URL
|
||||
# SSL params (passed through to SQLAlchemy connection):
|
||||
# sslmode = "prefer" # disable, allow, prefer, require, verify-ca, verify-full
|
||||
# sslrootcert = "" # path to CA cert for verify-ca/verify-full
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "1.5.11"
|
||||
__version__ = "1.5.16"
|
||||
|
||||
@@ -25,9 +25,13 @@ import httpx_sse
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.console.metrics import ConsoleMetrics
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
from turnstone.console.router import ConsoleRouter
|
||||
from turnstone.core.auth import ServiceTokenManager
|
||||
from turnstone.core.storage._notify import Notify
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = logging.getLogger("turnstone.console.collector")
|
||||
@@ -73,6 +77,7 @@ class ClusterCollector:
|
||||
tls_cert: tuple[str, str] | None = None,
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: NotifyDispatcher | None = None,
|
||||
):
|
||||
self._storage = storage
|
||||
self._discovery_interval = discovery_interval
|
||||
@@ -82,6 +87,8 @@ class ClusterCollector:
|
||||
self._console_metrics = console_metrics
|
||||
self._tls_verify = tls_verify
|
||||
self._tls_cert = tls_cert
|
||||
self._notify_dispatcher = notify_dispatcher
|
||||
self._notify_unsubscribe: Callable[[], None] | None = None
|
||||
|
||||
self._lock = threading.Lock()
|
||||
self._nodes: dict[str, NodeSnapshot] = {}
|
||||
@@ -128,6 +135,15 @@ class ClusterCollector:
|
||||
def start(self) -> None:
|
||||
"""Start background threads."""
|
||||
self._running = True
|
||||
# Subscribe to the ``services`` channel for reactive node discovery.
|
||||
# NOTIFY-driven wake-ups bring new-node visibility from up-to-60 s
|
||||
# (next discovery tick) down to ~500 ms on Postgres; the 60 s
|
||||
# discovery loop still runs as the backstop for crash-shaped node
|
||||
# loss (NOTIFY only fires on actual writes, not on crash exits).
|
||||
if self._notify_dispatcher is not None:
|
||||
self._notify_unsubscribe = self._notify_dispatcher.subscribe(
|
||||
"services", self._on_services_notify
|
||||
)
|
||||
for target, name in [
|
||||
(self._discovery_loop, "console-discovery"),
|
||||
(self._sse_manager_thread, "console-sse"),
|
||||
@@ -145,6 +161,10 @@ class ClusterCollector:
|
||||
its ``finally`` cleanup (cancel tasks, close AsyncClient).
|
||||
"""
|
||||
self._running = False
|
||||
if self._notify_unsubscribe is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
self._notify_unsubscribe()
|
||||
self._notify_unsubscribe = None
|
||||
# Request cancellation of all SSE tasks so they don't block the
|
||||
# manager's cleanup. The manager coroutine exits when _running is
|
||||
# False and handles remaining task cancellation in its finally block.
|
||||
@@ -156,6 +176,26 @@ class ClusterCollector:
|
||||
t.join(timeout=5)
|
||||
log.info("ClusterCollector stopped")
|
||||
|
||||
def _on_services_notify(self, notify: Notify) -> None:
|
||||
"""Run a discovery tick when the ``services`` channel fires.
|
||||
|
||||
The dispatcher delivers both real Postgres notifications and
|
||||
synthetic ``reconcile`` wake-ups after a reconnect — both shape
|
||||
the same way: re-read ``services`` and diff against in-memory
|
||||
state. Re-uses :meth:`_discover_nodes` so the timer-driven
|
||||
backstop and the NOTIFY-driven fast-path share one code path.
|
||||
"""
|
||||
from turnstone.core.storage._registry import StorageUnavailableError
|
||||
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
self._discover_nodes()
|
||||
except StorageUnavailableError:
|
||||
pass # already logged by storage layer
|
||||
except Exception:
|
||||
log.exception("Node discovery error (notify-driven)")
|
||||
|
||||
def _fanout(self, event: dict[str, Any]) -> None:
|
||||
"""Copy an event to all registered SSE listener queues."""
|
||||
with self._listeners_lock:
|
||||
|
||||
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.adapters._ui_cleanup import cleanup_session_ui
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.child_source import ClusterChildSource
|
||||
from turnstone.core.children_registry import ChildrenRegistry
|
||||
from turnstone.core.log import get_logger
|
||||
@@ -67,6 +68,14 @@ class CoordinatorAdapter:
|
||||
# method names which still exist as thin shims for the
|
||||
# cluster-routing + cleanup callers.
|
||||
self._registry = ChildrenRegistry()
|
||||
# In-process wakeup primitive for ``wait_for_workstream``. The
|
||||
# dispatch sink (:meth:`_dispatch_child_event`) calls
|
||||
# ``notify(child_ws_id)`` after each translated child event;
|
||||
# waiters block on per-call ``threading.Event``s instead of
|
||||
# polling storage. Owned by the adapter so the manager-level
|
||||
# exposure can simply delegate; ``CoordinatorClient`` picks it
|
||||
# up via the coord client factory closure.
|
||||
self._child_event_bus = ChildEventBus()
|
||||
# Cross-node child events arrive via ``ClusterChildSource``
|
||||
# (Stage 3 Step 2): a strategy that subscribes to the
|
||||
# collector's listener channel and runs a daemon thread that
|
||||
@@ -77,6 +86,17 @@ class CoordinatorAdapter:
|
||||
# the collector reference is available.
|
||||
self._child_source: ClusterChildSource | None = None
|
||||
|
||||
@property
|
||||
def child_event_bus(self) -> ChildEventBus:
|
||||
"""In-process wakeup bus consumed by ``wait_for_workstream``.
|
||||
|
||||
Exposed so the coord client factory in the console bootstrap
|
||||
can pass it to :class:`CoordinatorClient` without reaching
|
||||
into a private attr, and so :class:`SessionManager` can
|
||||
delegate its own ``child_event_bus`` property here.
|
||||
"""
|
||||
return self._child_event_bus
|
||||
|
||||
def attach(self, manager: SessionManager) -> None:
|
||||
"""Late-bind the owning :class:`SessionManager`.
|
||||
|
||||
@@ -639,6 +659,13 @@ class CoordinatorAdapter:
|
||||
"detail": event.get("detail") or {},
|
||||
}
|
||||
_enqueue_on_ui(owning_ws.ui, coord_id, child_event)
|
||||
# Wake any in-process ``wait_for_workstream`` subscriber on
|
||||
# this child. Notify runs AFTER the UI enqueue so the SSE
|
||||
# fan-out keeps priority (a wait that wakes early sees the
|
||||
# state already enqueued for its owning dashboard). The bus
|
||||
# is a no-op when no waiter is registered — the steady
|
||||
# state for the hot dispatch path.
|
||||
self._child_event_bus.notify(ws_id)
|
||||
|
||||
|
||||
def _enqueue_on_ui(ui: Any, coord_ws_id: str, payload: dict[str, Any]) -> None:
|
||||
|
||||
@@ -73,12 +73,27 @@ WAIT_MAX_WS_IDS: int = 32
|
||||
# wait_for_workstream again with the same ws_ids — each call re-arms freshly.
|
||||
WAIT_MAX_TIMEOUT: float = 600.0
|
||||
|
||||
# Storage-poll cadence. 500ms is short enough that the wait terminates
|
||||
# promptly after a child finishes (well under the human-perceptible-latency
|
||||
# floor), and long enough that a 60s wait incurs at most 120 cheap row
|
||||
# reads — still cheaper than the 20+ inspect_workstream model turns the
|
||||
# tool replaces.
|
||||
WAIT_POLL_INTERVAL: float = 0.5
|
||||
# Maximum ``event.wait`` interval in the bus-driven wait loop.
|
||||
# A long-running stuck child would otherwise look dead in the sidebar UI
|
||||
# because the ``wait_progress`` SSE emission piggybacks on the wait loop
|
||||
# — capping at 2 s keeps the heartbeat visible without flooding storage.
|
||||
# Today's polling effectively snapshots every 500 ms; 2 s preserves a
|
||||
# similar liveness feel while cutting per-listener SSE traffic ~4x in the
|
||||
# steady-state-quiescent case. Tunable post-merge if profiling shows
|
||||
# storage-read pressure on state-change wakes.
|
||||
#
|
||||
# **Worst-case completion latency**: 2 s. ``SessionManager.set_state``
|
||||
# buffers non-ERROR storage writes through ``StateWriter`` (async-flushed
|
||||
# at ~1 s cadence) while ``emit_state`` fans the event out immediately —
|
||||
# a bus-driven wake can therefore beat the flusher and read pre-transition
|
||||
# state on a terminal transition, then re-block on ``event.wait`` until
|
||||
# the heartbeat cap fires. Pre-bus the 0.5 s poll bounded this at 0.5 s.
|
||||
# Going to 2 s is intentional: the 4x SSE-traffic reduction in the
|
||||
# steady-state-quiescent case outweighs the worst-case latency
|
||||
# regression on the most common terminal transition, and a model issuing
|
||||
# a follow-up ``inspect_workstream`` (the pre-bus pattern this tool
|
||||
# replaces) was already paying multi-second model-turn latency per probe.
|
||||
WAIT_HEARTBEAT_INTERVAL: float = 2.0
|
||||
|
||||
# Per-ws cap on the inline ``message`` field bundled into wait_for_workstream
|
||||
# results. Sized so a fan-out of 32 children at the cap is ~320 KiB of
|
||||
@@ -184,6 +199,7 @@ def load_task_envelope(storage: Any, ws_id: str) -> tuple[dict[str, Any], bool]:
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -313,6 +329,7 @@ class CoordinatorClient:
|
||||
user_id: str,
|
||||
timeout: float = 30.0,
|
||||
http_client: httpx.Client | None = None,
|
||||
child_event_bus: ChildEventBus,
|
||||
) -> None:
|
||||
self._base_url = console_base_url.rstrip("/")
|
||||
self._storage = storage
|
||||
@@ -325,6 +342,12 @@ class CoordinatorClient:
|
||||
# with the coordinator session.
|
||||
self._http = http_client or httpx.Client(timeout=timeout)
|
||||
self._owns_http = http_client is None
|
||||
# In-process wakeup bus for ``wait_for_workstream``. The wait
|
||||
# loop blocks on a ``threading.Event`` keyed by ws_id and only
|
||||
# re-snapshots storage on state-change wakes or the heartbeat
|
||||
# cap. Owned by ``CoordinatorAdapter`` in production; tests
|
||||
# pass their own instance.
|
||||
self._child_event_bus = child_event_bus
|
||||
# tasks per-ws lock cache — populated lazily by _task_lock().
|
||||
# Single-session so a plain dict behind a coarse lock is fine;
|
||||
# WeakValueDictionary isn't needed (entries live as long as the
|
||||
@@ -447,6 +470,27 @@ class CoordinatorClient:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
def _row_in_own_subtree(self, ws_id: str, row: dict[str, Any] | None) -> bool:
|
||||
"""Row-level subtree predicate sharing one home for read paths.
|
||||
|
||||
Both :meth:`wait_for_workstream`'s pre-loop ownership filter and
|
||||
its inner ``_snapshot_all`` already have the workstream row in
|
||||
hand (from ``get_workstreams_batch``). Funneling them through
|
||||
the same 4-line check keeps the predicate in lockstep with
|
||||
:meth:`_is_own_subtree` (used by mutating ops) — both require
|
||||
``parent_ws_id`` AND ``user_id`` parity so a corrupted or
|
||||
forged ``parent_ws_id`` alone can't satisfy the gate on either
|
||||
path. Returns False on a missing / None row so callers can
|
||||
safely pass ``rows.get(wid)``.
|
||||
"""
|
||||
if ws_id == self._coord_ws_id:
|
||||
return True
|
||||
if row is None:
|
||||
return False
|
||||
if row.get("parent_ws_id") != self._coord_ws_id:
|
||||
return False
|
||||
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
|
||||
|
||||
# -- model-invoked mutating ops (HTTP) ---------------------------------
|
||||
|
||||
def spawn(
|
||||
@@ -562,7 +606,7 @@ class CoordinatorClient:
|
||||
_WAIT_TERMINAL_STATES: ClassVar[frozenset[str]] = WAIT_TERMINAL_STATES
|
||||
_WAIT_MAX_WS_IDS: ClassVar[int] = WAIT_MAX_WS_IDS
|
||||
_WAIT_MAX_TIMEOUT: ClassVar[float] = WAIT_MAX_TIMEOUT
|
||||
_WAIT_POLL_INTERVAL: ClassVar[float] = WAIT_POLL_INTERVAL
|
||||
_WAIT_HEARTBEAT_INTERVAL: ClassVar[float] = WAIT_HEARTBEAT_INTERVAL
|
||||
|
||||
def wait_for_workstream(
|
||||
self,
|
||||
@@ -721,12 +765,7 @@ class CoordinatorClient:
|
||||
snaps: dict[str, dict[str, Any]] = {}
|
||||
for wid in cleaned:
|
||||
row = rows.get(wid)
|
||||
if row is None:
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
is_self = wid == self._coord_ws_id
|
||||
is_own_child = row.get("parent_ws_id") == self._coord_ws_id
|
||||
if not (is_self or is_own_child):
|
||||
if row is None or not self._row_in_own_subtree(wid, row):
|
||||
snaps[wid] = {"state": "denied", "tokens": 0}
|
||||
continue
|
||||
snaps[wid] = {
|
||||
@@ -760,54 +799,119 @@ class CoordinatorClient:
|
||||
|
||||
last_results: dict[str, dict[str, Any]] = {}
|
||||
complete = False
|
||||
while True:
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
# Subscribe to in-process state-change events for the watched
|
||||
# ws_ids when the bus is wired. ``register_waiter`` returns a
|
||||
# single ``threading.Event`` registered against every id so a
|
||||
# wait on [A, B, C] wakes on any of A/B/C changing. Bus is
|
||||
# optional so test fixtures that don't wire it fall back to the
|
||||
# legacy ``time.sleep`` cadence with no behaviour change.
|
||||
#
|
||||
# **Defense-in-depth ownership filter**: ``_dispatch_child_event``
|
||||
# fires ``bus.notify(ws_id)`` for every ws_id in *any* coord's
|
||||
# registry on this console process, so a foreign ws_id passed by
|
||||
# an untrusted coord LLM (prompt injection) would otherwise leak
|
||||
# wake-up timing as a side channel — _snapshot_all returns
|
||||
# ``denied`` for the content, but the *time* at which the wait
|
||||
# un-blocked would correlate with the foreign ws_id's next
|
||||
# state-class event. Filter ``cleaned`` to own-subtree ids
|
||||
# before registering; foreign / missing ws_ids stay in the
|
||||
# snapshot list so they still surface as ``denied`` in
|
||||
# ``_snapshot_all`` and exit via the pure-denied short-circuit
|
||||
# below. Predicate shared with ``_snapshot_all`` via
|
||||
# :meth:`_row_in_own_subtree`.
|
||||
try:
|
||||
pre_rows = self._storage.get_workstreams_batch(cleaned)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.ownership_filter_failed", exc_info=True)
|
||||
pre_rows = {wid: None for wid in cleaned}
|
||||
own_subtree = [wid for wid in cleaned if self._row_in_own_subtree(wid, pre_rows.get(wid))]
|
||||
bus = self._child_event_bus
|
||||
wake_event = bus.register_waiter(own_subtree) if own_subtree else None
|
||||
try:
|
||||
while True:
|
||||
# Clear BEFORE the storage snapshot to close the
|
||||
# subscribe/check race: any ``notify`` between clear
|
||||
# and the next ``wake_event.wait`` leaves the Event
|
||||
# set, so the wait returns immediately and the loop
|
||||
# re-snapshots without losing the wake-up.
|
||||
if wake_event is not None:
|
||||
wake_event.clear()
|
||||
results = _snapshot_all()
|
||||
last_results = results
|
||||
if progress_callback is not None:
|
||||
try:
|
||||
progress_callback(results, time.monotonic() - start)
|
||||
except Exception:
|
||||
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
|
||||
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
|
||||
settled = [_is_settled(snap) for snap in results.values()]
|
||||
# ``since`` — orthogonal to mode. If the caller supplied a
|
||||
# prior snapshot, any diff on a ws_id that IS in ``since_map``
|
||||
# exits the wait so a follow-up call doesn't re-count
|
||||
# already-terminal children. ws_ids absent from ``since_map``
|
||||
# are ignored for the diff-exit check — they fall through to
|
||||
# the normal mode='any' / mode='all' conditions below. This
|
||||
# prevents a disjoint since-dict from exiting on tick one
|
||||
# with complete=True (previous shape did, silently).
|
||||
if since_map and any(
|
||||
_diff_since(snap, since_map[wid])
|
||||
for wid, snap in results.items()
|
||||
if wid in since_map
|
||||
):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
if mode == "any":
|
||||
if any(real_terminal):
|
||||
complete = True
|
||||
break
|
||||
# Pure-denied list: every snap is settled but none is a
|
||||
# real terminal — no work to wait for. Short-circuit so
|
||||
# the model sees the denied results immediately rather
|
||||
# than spinning the timeout (``complete=False`` because
|
||||
# the wait condition never had a real chance to fire).
|
||||
if all(settled):
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
else: # mode == "all"
|
||||
if all(settled):
|
||||
# Every ws_id is settled (real-terminal or denied).
|
||||
# The wait condition is met — the model gets the
|
||||
# full results dict and decides what each terminal
|
||||
# state means.
|
||||
complete = True
|
||||
break
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
time.sleep(min(self._WAIT_POLL_INTERVAL, remaining))
|
||||
if wake_event is not None:
|
||||
# Block until a child state-change notify fires OR
|
||||
# the heartbeat cap expires (so a stuck child still
|
||||
# emits a periodic ``wait_progress`` for the
|
||||
# sidebar UX). Heartbeat cap is the only timer —
|
||||
# the bus is the wake source. See
|
||||
# ``WAIT_HEARTBEAT_INTERVAL`` (module top) for the
|
||||
# worst-case completion-latency rationale: 2 s is
|
||||
# a deliberate 4x trade vs the pre-bus 0.5 s poll.
|
||||
wake_event.wait(min(remaining, self._WAIT_HEARTBEAT_INTERVAL))
|
||||
else:
|
||||
# Pure-foreign / pure-denied list: every cleaned
|
||||
# ws_id was filtered out of ``own_subtree`` so the
|
||||
# bus has nothing to wake on. The pure-denied
|
||||
# short-circuit above exits ``mode='any'`` on the
|
||||
# first tick; ``mode='all'`` falls through to here
|
||||
# and must burn the timeout. Use the heartbeat
|
||||
# cadence for the deadline carve-up so
|
||||
# ``progress_callback`` keeps firing.
|
||||
time.sleep(min(self._WAIT_HEARTBEAT_INTERVAL, remaining))
|
||||
finally:
|
||||
# Always unregister so a crash mid-wait can't leak the
|
||||
# registration past one wait's lifetime. Bus discards
|
||||
# empty buckets so long-lived buses don't accumulate dead
|
||||
# keys after many waits. Unregister against the same
|
||||
# ``own_subtree`` list the register call used — passing
|
||||
# ``cleaned`` here would silently no-op for foreign ids
|
||||
# but pass an unknown bucket to ``unregister_waiter``.
|
||||
if wake_event is not None:
|
||||
bus.unregister_waiter(own_subtree, wake_event)
|
||||
# Bundle each terminal child's last assistant message inline so the
|
||||
# coordinator LLM doesn't have to follow up with one
|
||||
# ``inspect_workstream`` per ws. Only ``idle`` / ``error`` ws_ids
|
||||
@@ -816,7 +920,7 @@ class CoordinatorClient:
|
||||
# subset across a small thread pool — at the WAIT_MAX_WS_IDS=32
|
||||
# cap, 8 workers cuts a worst-case all-idle fan-out from 32
|
||||
# sequential storage round-trips down to 4 batches, which lands
|
||||
# inside the WAIT_POLL_INTERVAL the model already tolerates
|
||||
# inside the WAIT_HEARTBEAT_INTERVAL the model already tolerates
|
||||
# between ticks. Storage backends use SQLAlchemy with
|
||||
# ``check_same_thread=False`` (SQLite) / a connection pool
|
||||
# (Postgres), so concurrent reads from the worker pool are safe.
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Console-side multiplexer for PostgreSQL ``LISTEN``/``NOTIFY`` events.
|
||||
|
||||
Holds a single dedicated listen connection (via :meth:`StorageBackend.listen`),
|
||||
drains it on a listener thread, and fans notifications out to per-channel
|
||||
handlers on a dedicated dispatch thread so a slow handler doesn't back up
|
||||
the connection.
|
||||
|
||||
Consumers register at construction time by passing their channel in
|
||||
:attr:`channels`, then call :meth:`subscribe` to attach a handler.
|
||||
Registering an undeclared channel raises — the construction list is the
|
||||
single source of truth so wire-in is explicit (each future consumer
|
||||
touches the dispatcher construction call site at
|
||||
``turnstone/console/server.py::main`` to add its channel).
|
||||
|
||||
On connection loss the listener wakes its handlers with a synthetic
|
||||
``Notify(channel, payload="reconcile", pid=0)`` so every consumer
|
||||
re-reads the underlying rows; their normal "reconcile on any wake-up"
|
||||
code path covers both real notifications and reconnect recovery
|
||||
identically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Iterable
|
||||
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
# Backoff (seconds) between reconnect attempts after :class:`NotifyConnectionError`.
|
||||
# Doubles each failure, capped at the max — long enough that a Postgres outage
|
||||
# doesn't burn CPU on reconnect spins, short enough that recovery is fast.
|
||||
_RECONNECT_BACKOFF_INITIAL: float = 1.0
|
||||
_RECONNECT_BACKOFF_MAX: float = 30.0
|
||||
|
||||
# Poll cadence on the listener thread. Short enough that ``stop`` lands
|
||||
# promptly without joining a long-blocked notifies() call; long enough
|
||||
# that we don't burn CPU on empty polls.
|
||||
_LISTENER_POLL_TIMEOUT: float = 1.0
|
||||
|
||||
# Cap on the inter-thread dispatch queue. Drops oldest if a slow handler
|
||||
# falls behind (logs once per drop bucket). Sized larger than the expected
|
||||
# steady-state notification rate (services trigger fires only on
|
||||
# register/restart/deregister — order of hundreds per hour at the 100-node
|
||||
# design ceiling).
|
||||
_DISPATCH_QUEUE_MAX: int = 1024
|
||||
|
||||
|
||||
class NotifyDispatcher:
|
||||
"""Holds the dedicated listen connection and fans events to handlers.
|
||||
|
||||
Lifecycle: construct with the declared channel list, attach
|
||||
handlers via :meth:`subscribe`, then call :meth:`start`. :meth:`stop`
|
||||
closes the connection and joins the worker threads. Idempotent in
|
||||
both directions so console teardown can call stop unconditionally.
|
||||
"""
|
||||
|
||||
def __init__(self, storage: StorageBackend, channels: Iterable[str]) -> None:
|
||||
ch_list = [str(c) for c in channels if c]
|
||||
if not ch_list:
|
||||
msg = "NotifyDispatcher requires at least one declared channel"
|
||||
raise ValueError(msg)
|
||||
self._storage = storage
|
||||
self._channels: list[str] = list(dict.fromkeys(ch_list)) # de-dupe, preserve order
|
||||
self._handlers: dict[str, list[Callable[[Notify], None]]] = {
|
||||
ch: [] for ch in self._channels
|
||||
}
|
||||
self._handlers_lock = threading.Lock()
|
||||
self._lifecycle_lock = threading.Lock()
|
||||
self._started = False
|
||||
self._stopping = threading.Event()
|
||||
self._listener_thread: threading.Thread | None = None
|
||||
self._dispatch_thread: threading.Thread | None = None
|
||||
self._dispatch_queue: queue.Queue[Notify | None] = queue.Queue(maxsize=_DISPATCH_QUEUE_MAX)
|
||||
self._drop_count = 0
|
||||
# Set inside :meth:`_listener_loop` after each successful
|
||||
# ``storage.listen`` open; cleared on disconnect. Callers use
|
||||
# :meth:`wait_until_ready` after :meth:`start` to block until the
|
||||
# listener is actually listening (matters when the next caller
|
||||
# action is a ``notify`` whose delivery requires the LISTEN to
|
||||
# already be in place — e.g. tests, or any startup-path traffic
|
||||
# that should be reactive from the first event).
|
||||
self._listener_ready = threading.Event()
|
||||
|
||||
@property
|
||||
def channels(self) -> list[str]:
|
||||
"""Snapshot copy of declared channels."""
|
||||
return list(self._channels)
|
||||
|
||||
def subscribe(self, channel: str, handler: Callable[[Notify], None]) -> Callable[[], None]:
|
||||
"""Attach ``handler`` to ``channel``; return an unsubscribe callable.
|
||||
|
||||
Safe to call before or after :meth:`start`. Raises if the
|
||||
channel was not declared at construction time — the channel
|
||||
list is fixed so the dispatcher knows up-front which LISTENs
|
||||
to issue (consumers added in follow-up PRs touch the
|
||||
construction call site).
|
||||
"""
|
||||
if channel not in self._handlers:
|
||||
msg = (
|
||||
f"channel {channel!r} not declared at construction; "
|
||||
f"declared channels: {sorted(self._handlers)}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
with self._handlers_lock:
|
||||
self._handlers[channel].append(handler)
|
||||
|
||||
def _unsubscribe() -> None:
|
||||
with self._handlers_lock, contextlib.suppress(ValueError):
|
||||
self._handlers[channel].remove(handler)
|
||||
|
||||
return _unsubscribe
|
||||
|
||||
def start(self) -> None:
|
||||
"""Open the listen stream and start the listener + dispatch threads.
|
||||
|
||||
Idempotent — repeat calls log a debug line and return without
|
||||
spawning a second listener.
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if self._started:
|
||||
log.debug("notify_dispatcher.start_noop_already_started")
|
||||
return
|
||||
self._started = True
|
||||
self._stopping.clear()
|
||||
# Clear ready so a stop/start cycle's wait_until_ready only
|
||||
# returns True after the new listener has actually opened.
|
||||
self._listener_ready.clear()
|
||||
self._listener_thread = threading.Thread(
|
||||
target=self._listener_loop,
|
||||
name="notify-dispatcher-listener",
|
||||
daemon=True,
|
||||
)
|
||||
self._dispatch_thread = threading.Thread(
|
||||
target=self._dispatch_loop,
|
||||
name="notify-dispatcher-dispatch",
|
||||
daemon=True,
|
||||
)
|
||||
self._listener_thread.start()
|
||||
self._dispatch_thread.start()
|
||||
log.info(
|
||||
"notify_dispatcher.started",
|
||||
channels=self._channels,
|
||||
)
|
||||
|
||||
def wait_until_ready(self, timeout: float = 5.0) -> bool:
|
||||
"""Block until the listener has opened its stream, or ``timeout`` elapses.
|
||||
|
||||
Returns ``True`` when the listener is ready (``LISTEN`` issued
|
||||
for every declared channel on PG; subscriber queues registered
|
||||
on SQLite), ``False`` on timeout. Cleared automatically on
|
||||
disconnect — call again after a reconnect to wait for the next
|
||||
successful reopen.
|
||||
|
||||
Doesn't replace :meth:`start` — call ``start()`` first, then
|
||||
``wait_until_ready()`` for the explicit sync point. Production
|
||||
startup typically doesn't need this (the first real event tends
|
||||
to arrive well after the listener is up); tests use it to close
|
||||
the start-vs-notify race window.
|
||||
"""
|
||||
return self._listener_ready.wait(timeout=timeout)
|
||||
|
||||
def stop(self, timeout: float = 5.0) -> None:
|
||||
"""Signal shutdown and join the worker threads.
|
||||
|
||||
Idempotent — safe to call multiple times. Workers exit on the
|
||||
next iteration of their poll loops; :meth:`stop` blocks up to
|
||||
``timeout`` seconds per thread before giving up (the threads are
|
||||
daemons so the process can exit regardless).
|
||||
"""
|
||||
with self._lifecycle_lock:
|
||||
if not self._started:
|
||||
return
|
||||
self._stopping.set()
|
||||
listener = self._listener_thread
|
||||
dispatcher = self._dispatch_thread
|
||||
# Sentinel wakes the dispatch loop out of queue.get().
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(None)
|
||||
if listener is not None:
|
||||
listener.join(timeout=timeout)
|
||||
if dispatcher is not None:
|
||||
dispatcher.join(timeout=timeout)
|
||||
with self._lifecycle_lock:
|
||||
self._listener_thread = None
|
||||
self._dispatch_thread = None
|
||||
self._started = False
|
||||
log.info("notify_dispatcher.stopped")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Internal threading
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _listener_loop(self) -> None:
|
||||
"""Drain the storage stream onto the dispatch queue, reconnecting on loss.
|
||||
|
||||
After any disconnect — whether surfaced through the stream's
|
||||
:class:`NotifyConnectionError` (post-open ``poll`` failure) or
|
||||
through the generic exception path (``psycopg.connect`` /
|
||||
initial ``LISTEN`` execute failures during reopen, which are
|
||||
NOT wrapped by the stream) — the loop sets a ``reconcile_pending``
|
||||
flag, waits the backoff, then enqueues one synthetic ``reconcile``
|
||||
notify per channel ONLY after the next stream successfully
|
||||
reopens. Handlers see the synthetic notify and re-read the
|
||||
relevant rows on the same code path they use for any real event,
|
||||
closing the missed-notification window regardless of which
|
||||
exception type caused the disconnect.
|
||||
"""
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
reconcile_pending = False
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
with self._storage.listen(self._channels) as stream:
|
||||
log.debug(
|
||||
"notify_dispatcher.stream_open",
|
||||
channels=self._channels,
|
||||
)
|
||||
# Stream is open — reset backoff for the next outage
|
||||
# and flush any pending reconcile so consumers see a
|
||||
# wake-up against a now-live DB.
|
||||
backoff = _RECONNECT_BACKOFF_INITIAL
|
||||
if reconcile_pending:
|
||||
self._synthesize_reconcile()
|
||||
reconcile_pending = False
|
||||
# Signal ``wait_until_ready`` callers that LISTEN is
|
||||
# in place (PG) / subscriber queues are bound
|
||||
# (SQLite). Must come AFTER the synthesize so any
|
||||
# post-reconnect reconcile reaches handlers before
|
||||
# the caller assumes "fresh notifies will deliver".
|
||||
self._listener_ready.set()
|
||||
while not self._stopping.is_set():
|
||||
batch = stream.poll(_LISTENER_POLL_TIMEOUT)
|
||||
for n in batch:
|
||||
self._enqueue(n)
|
||||
except NotifyConnectionError as exc:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.warning(
|
||||
"notify_dispatcher.connection_lost",
|
||||
error=str(exc),
|
||||
backoff_seconds=backoff,
|
||||
)
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
except Exception:
|
||||
if self._stopping.is_set():
|
||||
return
|
||||
self._listener_ready.clear()
|
||||
log.exception("notify_dispatcher.listener_unexpected_error")
|
||||
reconcile_pending = True
|
||||
if self._stopping.wait(backoff):
|
||||
return
|
||||
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
|
||||
log.debug("notify_dispatcher.listener_exiting")
|
||||
|
||||
def _synthesize_reconcile(self) -> None:
|
||||
"""Push one synthetic ``reconcile`` notify per channel on reconnect.
|
||||
|
||||
Reconcile-on-wake is the same logic handlers run for any real
|
||||
notification, so a single synthetic event per channel covers
|
||||
any notifications missed during the connection-loss window.
|
||||
"""
|
||||
for ch in self._channels:
|
||||
self._enqueue(Notify(channel=ch, payload="reconcile", pid=0))
|
||||
|
||||
def _enqueue(self, notify: Notify) -> None:
|
||||
"""Put a notify on the dispatch queue, dropping oldest on overflow."""
|
||||
try:
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
except queue.Full:
|
||||
# Drop oldest to make room — a slow handler shouldn't be able
|
||||
# to silently block the listener thread. Log once per power
|
||||
# of two so a sustained backpressure problem shows up
|
||||
# in logs without flooding.
|
||||
self._drop_count += 1
|
||||
if self._drop_count & (self._drop_count - 1) == 0:
|
||||
log.warning(
|
||||
"notify_dispatcher.dispatch_queue_full_dropping_oldest",
|
||||
drops_total=self._drop_count,
|
||||
channel=notify.channel,
|
||||
)
|
||||
with contextlib.suppress(queue.Empty):
|
||||
self._dispatch_queue.get_nowait()
|
||||
with contextlib.suppress(queue.Full):
|
||||
self._dispatch_queue.put_nowait(notify)
|
||||
|
||||
def _dispatch_loop(self) -> None:
|
||||
"""Pull notifies off the queue and invoke handlers per channel.
|
||||
|
||||
Notifies queued on the same channel coalesce per dispatch batch:
|
||||
after blocking ``get()`` returns one notify, the loop drains
|
||||
whatever else is already queued and collapses to one
|
||||
``per-channel`` notify before invoking handlers. The payload is
|
||||
signal-only by design (handlers reconcile by re-reading the
|
||||
underlying rows), so N same-channel notifies have the same
|
||||
observable effect as one — coalescing turns an N-node deploy
|
||||
burst into a single ``_discover_nodes`` per channel instead of N.
|
||||
|
||||
Each handler runs under exception suppression so one buggy
|
||||
consumer can't take down the dispatch thread.
|
||||
"""
|
||||
while not self._stopping.is_set():
|
||||
try:
|
||||
first = self._dispatch_queue.get(timeout=_LISTENER_POLL_TIMEOUT)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if first is None:
|
||||
# Sentinel from :meth:`stop`.
|
||||
return
|
||||
# Coalesce by channel: keep the most recent payload per
|
||||
# channel from this drain batch. Drops a stop sentinel
|
||||
# silently — the next loop iteration will see _stopping set
|
||||
# and exit anyway, so we don't need to re-queue the sentinel.
|
||||
per_channel: dict[str, Notify] = {first.channel: first}
|
||||
stop_seen = False
|
||||
while True:
|
||||
try:
|
||||
nxt = self._dispatch_queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
if nxt is None:
|
||||
stop_seen = True
|
||||
continue
|
||||
per_channel[nxt.channel] = nxt
|
||||
for notify in per_channel.values():
|
||||
with self._handlers_lock:
|
||||
handlers = list(self._handlers.get(notify.channel, ()))
|
||||
for handler in handlers:
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
handler(notify)
|
||||
except Exception:
|
||||
log.exception(
|
||||
"notify_dispatcher.handler_failed",
|
||||
channel=notify.channel,
|
||||
)
|
||||
else:
|
||||
elapsed_ms = (time.monotonic() - t0) * 1000.0
|
||||
if elapsed_ms > 100.0:
|
||||
log.debug(
|
||||
"notify_dispatcher.handler_slow",
|
||||
channel=notify.channel,
|
||||
elapsed_ms=round(elapsed_ms, 1),
|
||||
)
|
||||
if stop_seen:
|
||||
return
|
||||
log.debug("notify_dispatcher.dispatch_exiting")
|
||||
@@ -319,6 +319,12 @@ class TaskScheduler:
|
||||
user_id=task.get("created_by", ""),
|
||||
skill=task.get("skill", ""),
|
||||
notify_targets=task.get("notify_targets", "[]"),
|
||||
# Mark the resulting ChatSession as non-interactive-for-
|
||||
# consent so OAuth-MCP errors get persisted to
|
||||
# ``mcp_pending_consent`` for later dashboard surfacing,
|
||||
# rather than relying on an in-flight SSE redirect the
|
||||
# absent user can't complete.
|
||||
client_type="scheduled",
|
||||
)
|
||||
ws_id = resp.ws_id
|
||||
except Exception:
|
||||
|
||||
+338
-18
@@ -1652,6 +1652,27 @@ async def mcp_oauth_revoke_connection(request: Request) -> Response:
|
||||
return await handle_mcp_oauth_revoke_connection(request)
|
||||
|
||||
|
||||
async def mcp_oauth_list_pending(request: Request) -> Response:
|
||||
"""GET /v1/api/mcp/oauth/pending — list deferred-consent records (Phase 9)."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_list_pending
|
||||
|
||||
return await handle_mcp_oauth_list_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending/{server_name} — dismiss a deferred-consent record."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_all_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending — bulk-dismiss deferred-consent records."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_all_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_all_pending(request)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Route handlers — available models (lightweight, no admin permission)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -2636,10 +2657,67 @@ async def proxy_shared_static(request: Request) -> Response:
|
||||
return JSONResponse({"error": "Node unreachable"}, status_code=502)
|
||||
|
||||
|
||||
# Auth endpoints the console handles locally instead of forwarding to
|
||||
# the upstream node. Single source of truth for the dispatch table and
|
||||
# the path set used by both the 405 short-circuit and the
|
||||
# test parametrize, so a new entry can't drift between code and tests.
|
||||
#
|
||||
# Values are handler NAMES (strings) rather than function references.
|
||||
# ``proxy_api`` resolves them via ``globals()`` at call time so test
|
||||
# ``patch("turnstone.console.server.auth_login")`` is observed; a dict
|
||||
# of refs would capture the original function at module load.
|
||||
_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str] = {
|
||||
("POST", "auth/login"): "auth_login",
|
||||
("POST", "auth/logout"): "auth_logout",
|
||||
("POST", "auth/setup"): "auth_setup",
|
||||
("POST", "auth/refresh"): "auth_refresh",
|
||||
("GET", "auth/status"): "auth_status",
|
||||
("GET", "auth/whoami"): "auth_whoami",
|
||||
("GET", "auth/oidc/authorize"): "oidc_authorize",
|
||||
("GET", "auth/oidc/callback"): "oidc_callback",
|
||||
}
|
||||
_PROXY_AUTH_LOCAL_PATHS: frozenset[str] = frozenset(path for _, path in _PROXY_AUTH_LOCAL_HANDLERS)
|
||||
|
||||
|
||||
async def proxy_api(request: Request) -> Response:
|
||||
"""Proxy API requests to target node. Detects SSE vs regular."""
|
||||
"""Proxy API requests to target node, with two exceptions handled in-process:
|
||||
|
||||
1. ``auth/*`` endpoints in ``_PROXY_AUTH_LOCAL_HANDLERS`` are
|
||||
dispatched to the console's own auth handlers so JWTs carry
|
||||
``JWT_AUD_CONSOLE`` and Set-Cookie lands on the console origin.
|
||||
Forwarding upstream would mint ``JWT_AUD_SERVER`` tokens that the
|
||||
console's ``AuthMiddleware`` rejects on the next proxied call,
|
||||
locking the user out of the proxied UI — and ``_proxy_post``
|
||||
drops Set-Cookie when forwarding anyway. ``refresh`` and
|
||||
``whoami`` are intentionally NOT in ``PUBLIC_PATHS`` (caller must
|
||||
still hold a valid cookie); local dispatch is about cookie-origin
|
||||
and audience, not public access.
|
||||
2. SSE endpoints (per-ws + global events) stream via ``_proxy_sse``.
|
||||
|
||||
Everything else is forwarded to ``server_url`` via ``_proxy_post`` /
|
||||
``_proxy_get``.
|
||||
"""
|
||||
node_id = request.path_params["node_id"]
|
||||
path = request.path_params["path"]
|
||||
|
||||
handler_name = _PROXY_AUTH_LOCAL_HANDLERS.get((request.method, path))
|
||||
if handler_name is not None:
|
||||
# Resolve via ``globals()`` so ``patch("...auth_login")`` in
|
||||
# tests is observed. A direct function-ref dict would have
|
||||
# captured the original at module load.
|
||||
handler = globals()[handler_name]
|
||||
return await handler(request) # type: ignore[no-any-return]
|
||||
# Path matches a local-dispatch auth endpoint but the method does not:
|
||||
# short-circuit with 405 so the request can't fall through to
|
||||
# ``_proxy_post`` / ``_proxy_get`` and reach the upstream
|
||||
# authenticated as the console's service token
|
||||
# (``_proxy_auth_headers`` falls back to the service identity when
|
||||
# there's no user context). Harmless today because every upstream
|
||||
# auth route 405s on the wrong method too, but kept tight so a
|
||||
# future upstream patch can't widen the surface by accident.
|
||||
if path in _PROXY_AUTH_LOCAL_PATHS:
|
||||
return JSONResponse({"error": "Method not allowed"}, status_code=405)
|
||||
|
||||
server_url = _get_server_url(request, node_id)
|
||||
if not server_url:
|
||||
return JSONResponse({"error": "Node not found"}, status_code=404)
|
||||
@@ -4140,6 +4218,7 @@ def _coord_idle_cleanup_thread(
|
||||
mgr: SessionManager,
|
||||
timeout_sec: float,
|
||||
stop_event: threading.Event | None = None,
|
||||
min_sweep_interval: float = 5.0,
|
||||
) -> None:
|
||||
"""Periodically reap idle + DB-orphan coordinator workstreams.
|
||||
|
||||
@@ -4150,7 +4229,7 @@ def _coord_idle_cleanup_thread(
|
||||
and which aren't currently loaded. The latter pass catches coords left
|
||||
behind by prior console process incarnations.
|
||||
|
||||
Runs an initial sweep BEFORE the first sleep so cold-start orphans are
|
||||
Runs an initial sweep BEFORE the first wait so cold-start orphans are
|
||||
reaped immediately rather than waiting one ``check_every`` interval (~30
|
||||
min on default 2h timeout). This intentionally diverges from the regular
|
||||
server pattern, which has no initial sweep — the regular server runs
|
||||
@@ -4158,26 +4237,90 @@ def _coord_idle_cleanup_thread(
|
||||
is a small fixed-size cache where orphans dominate the row count after
|
||||
a cold boot.
|
||||
|
||||
Wait shape: subscribes a callback to ``mgr._state_subscribers`` that
|
||||
sets a ``tick_now`` event; the loop blocks on ``tick_now.wait(check_every)``
|
||||
so any workstream state-change wakes the sweeper without waiting a
|
||||
full check interval, AND the timeout still fires the periodic sweep
|
||||
even when no activity happens (catching the DB-orphan-only case).
|
||||
Net: blocked most of the time instead of repeating storage scans.
|
||||
|
||||
``min_sweep_interval`` is the hard floor between successive
|
||||
``close_idle`` calls (default 5 s) — without it, sustained
|
||||
state-change activity (each turn typically fires
|
||||
thinking/running/attention/idle on the coord SessionManager) would
|
||||
cause every ``tick_now.set`` mid-sweep to leave the next ``wait``
|
||||
returning immediately, and the loop would tight-spin ``close_idle``
|
||||
at the rate of its own DB latency (~20-50 calls/sec). The floor
|
||||
bounds DB-call traffic at ``1 / min_sweep_interval`` per second
|
||||
under any external activity while still letting a quiet system
|
||||
fire on every state-change wake-up. Tests inject 0.0 to keep the
|
||||
suite fast.
|
||||
|
||||
Default 5 s is a 6x improvement on the pre-refactor fixed 30 s
|
||||
cadence while bounding DB-call traffic at ~0.2 calls/sec under
|
||||
sustained activity — an order of magnitude below ``close_idle``'s
|
||||
DB-latency budget, but tight enough that idle-row reaping still
|
||||
feels prompt to a human watching the sidebar. Tunable post-merge
|
||||
if profiling shows close_idle latency dominates the cadence.
|
||||
|
||||
``stop_event`` is for tests — when set, the thread exits cleanly after
|
||||
the next loop check. Production callers pass ``None`` (the daemon is
|
||||
process-lifetime).
|
||||
"""
|
||||
check_every = min(300.0, timeout_sec / 4)
|
||||
# Initial sweep — runs once before entering the sleep loop.
|
||||
tick_now = threading.Event()
|
||||
|
||||
def _on_state_change(_ws_id: str, _state: Any) -> None:
|
||||
# Any workstream state-change resets the idle clock for that
|
||||
# ws AND may make a different ws newly-eligible (close-idle
|
||||
# pass 2 evaluates DB rows by timestamp). Cheap signal, full
|
||||
# re-evaluation deferred to the next loop iteration.
|
||||
tick_now.set()
|
||||
|
||||
mgr.subscribe_to_state(_on_state_change)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
time.sleep(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Initial sweep — runs once before entering the wait loop.
|
||||
# ``tick_now`` is intentionally not cleared here: any
|
||||
# state-change event that arrives between subscribe and the
|
||||
# first ``wait`` should fire close_idle immediately, not be
|
||||
# discarded.
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
while True:
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
tick_now.wait(check_every)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
return
|
||||
# Clear BEFORE the cadence floor so any state-change event
|
||||
# arriving during the cooldown (or during the close_idle
|
||||
# below) leaves ``tick_now`` set — the next loop iteration
|
||||
# then re-enters ``wait`` already-set and re-evaluates
|
||||
# promptly. close_idle is idempotent so a spurious extra
|
||||
# tick is just one redundant scan.
|
||||
tick_now.clear()
|
||||
# Cadence floor — see docstring for the tight-spin
|
||||
# hazard rationale. Cooldown uses ``stop_event.wait``
|
||||
# (not ``time.sleep``) so the test stop hook still
|
||||
# terminates promptly during the cooldown window.
|
||||
since_last = time.monotonic() - last_sweep_at
|
||||
if since_last < min_sweep_interval:
|
||||
gap = min_sweep_interval - since_last
|
||||
if stop_event is not None:
|
||||
if stop_event.wait(gap):
|
||||
return
|
||||
else:
|
||||
time.sleep(gap)
|
||||
try:
|
||||
mgr.close_idle(timeout_sec)
|
||||
except Exception:
|
||||
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
|
||||
last_sweep_at = time.monotonic()
|
||||
finally:
|
||||
mgr.unsubscribe_from_state(_on_state_change)
|
||||
|
||||
|
||||
# Guards concurrent attempts to bootstrap the coord subsystem from the
|
||||
@@ -4239,12 +4382,22 @@ def _bootstrap_coord_subsystem(
|
||||
def _token_factory() -> str:
|
||||
return tm.token
|
||||
|
||||
# ``coord_adapter`` is bound later in this same
|
||||
# ``_bootstrap_coord_subsystem`` call, after the adapter and
|
||||
# manager are constructed but before any session is created
|
||||
# — so this factory is *defined* before the adapter exists but
|
||||
# only ever *called* after it does. The free-variable lookup
|
||||
# at call time resolves to the adapter built in this same
|
||||
# bootstrap pass, giving the client a handle to the in-process
|
||||
# wakeup bus the dispatch sink notifies on every child
|
||||
# state-change event.
|
||||
return CoordinatorClient(
|
||||
console_base_url=console_bind_url,
|
||||
storage=storage,
|
||||
token_factory=_token_factory,
|
||||
coord_ws_id=ws_id,
|
||||
user_id=user_id,
|
||||
child_event_bus=coord_adapter.child_event_bus,
|
||||
)
|
||||
|
||||
# Pre-compute config-derived integers BEFORE any thread starts so a
|
||||
@@ -4770,6 +4923,12 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
|
||||
|
||||
await close_oidc_state(app.state)
|
||||
app.state.collector.stop()
|
||||
# Stop the dispatcher after the collector — collector.stop() drops its
|
||||
# subscription, so the dispatcher's dispatch thread won't fire into a
|
||||
# half-torn-down collector during shutdown.
|
||||
notify_dispatcher = getattr(app.state, "notify_dispatcher", None)
|
||||
if notify_dispatcher is not None:
|
||||
notify_dispatcher.stop()
|
||||
audit_exec_shutdown = getattr(app.state, "audit_executor", None)
|
||||
if audit_exec_shutdown is not None:
|
||||
_set_audit_executor(None)
|
||||
@@ -8680,10 +8839,19 @@ def _mask_mcp_secrets(server: dict[str, Any], reveal: bool = False) -> dict[str,
|
||||
def _mcp_server_to_detail(
|
||||
server: dict[str, Any],
|
||||
node_statuses: dict[str, dict[str, Any]] | None = None,
|
||||
consented_users_count: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Convert a storage dict to a McpServerDetail-shaped dict."""
|
||||
"""Convert a storage dict to a McpServerDetail-shaped dict.
|
||||
|
||||
*consented_users_count* is the Phase 9 admin pill data — distinct
|
||||
non-expired tokens issued for this ``(server_name)``. Omitted
|
||||
(``None``) when the row's ``auth_type`` is not ``oauth_user``, so
|
||||
static / none rows don't carry an irrelevant ``0``.
|
||||
"""
|
||||
d = dict(server)
|
||||
d["status"] = node_statuses or {}
|
||||
if consented_users_count is not None:
|
||||
d["consented_users_count"] = consented_users_count
|
||||
return d
|
||||
|
||||
|
||||
@@ -8734,8 +8902,31 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
|
||||
reveal = str(request.query_params.get("reveal", "")).lower() in ("true", "1")
|
||||
servers = storage.list_mcp_servers()
|
||||
|
||||
# Collect live status from all nodes
|
||||
node_statuses = await _collect_mcp_status(request)
|
||||
# Phase 9: bulk-aggregate consented-users-count across all oauth_user
|
||||
# rows in a single GROUP BY query (rather than N per-row sync DB
|
||||
# round-trips inside this async handler). Run in parallel with the
|
||||
# cross-node HTTP status fan-out below — neither has a data
|
||||
# dependency on the other, so awaiting them sequentially would
|
||||
# stack the DB latency on top of the fan-out latency. Skipped
|
||||
# entirely when no row is oauth_user so static-only installs
|
||||
# exercise zero new storage queries.
|
||||
has_oauth_user = any(s.get("auth_type") == "oauth_user" for s in servers)
|
||||
status_task: asyncio.Task[dict[str, dict[str, dict[str, Any]]]] = asyncio.create_task(
|
||||
_collect_mcp_status(request)
|
||||
)
|
||||
count_task: asyncio.Task[dict[str, int]] | None = (
|
||||
asyncio.create_task(asyncio.to_thread(storage.count_mcp_consented_users_grouped_by_server))
|
||||
if has_oauth_user
|
||||
else None
|
||||
)
|
||||
|
||||
node_statuses = await status_task
|
||||
consent_counts: dict[str, int] = {}
|
||||
if count_task is not None:
|
||||
try:
|
||||
consent_counts = await count_task
|
||||
except Exception:
|
||||
log.debug("admin.mcp_consented_users_bulk_count_failed", exc_info=True)
|
||||
|
||||
db_names: set[str] = set()
|
||||
result = []
|
||||
@@ -8747,8 +8938,14 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
|
||||
status = node_servers.get(s["name"])
|
||||
if status:
|
||||
per_node[node_id] = status
|
||||
# Phase 9: surface the consented-users-count pill for
|
||||
# oauth_user rows. Aggregate was pre-computed above with a
|
||||
# single bulk GROUP BY query; we just look up here.
|
||||
consent_count: int | None = None
|
||||
if s.get("auth_type") == "oauth_user":
|
||||
consent_count = consent_counts.get(s["name"], 0)
|
||||
s = _mask_mcp_secrets(s, reveal)
|
||||
result.append(_mcp_server_to_detail(s, per_node))
|
||||
result.append(_mcp_server_to_detail(s, per_node, consent_count))
|
||||
|
||||
# Merge config-sourced servers visible on nodes but not in DB
|
||||
config_names: set[str] = set()
|
||||
@@ -9389,6 +9586,92 @@ async def admin_mcp_reconnect_one(request: Request) -> JSONResponse:
|
||||
return await _admin_mcp_action(request, "reconnect")
|
||||
|
||||
|
||||
async def admin_mcp_bulk_revoke(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/admin/mcp-servers/{name}/bulk-revoke — clear every user's token (Phase 9).
|
||||
|
||||
Admin-side counterpart to the per-user
|
||||
``DELETE /v1/api/mcp/oauth/connections/{server_name}`` revoke that
|
||||
shipped in Phase 8. Used to drop orphaned tokens after an
|
||||
``auth_type`` transition (oauth_user → static) or after rotating
|
||||
the configured OAuth client.
|
||||
|
||||
Authoritative local delete via
|
||||
:meth:`StorageBackend.delete_mcp_oauth_rows_by_server_name` —
|
||||
purges both ``mcp_user_tokens`` and ``mcp_oauth_pending`` rows for
|
||||
the named server. Upstream RFC 7009 revoke is intentionally NOT
|
||||
attempted in bulk (would require per-row decrypt + N upstream HTTP
|
||||
calls); operators who need upstream cleanup should use the per-
|
||||
user revoke endpoint or let tokens expire naturally. The audit
|
||||
detail records ``upstream_revoke_outcome="bulk_admin_no_upstream"``
|
||||
so the deferral is visible.
|
||||
|
||||
Pool eviction is NOT performed by this handler. Per-user revoke
|
||||
has a per-(user, server) eviction primitive
|
||||
(``MCPClientManager.evict_user_session``); bulk-revoke would need a
|
||||
per-server iteration over consented users that no current primitive
|
||||
supports. Stale in-flight sessions surface as a per-user 401 on
|
||||
the next dispatch, which refreshes through the now-empty token row
|
||||
and emits ``mcp_consent_required`` — the documented v1 fallback.
|
||||
See :func:`turnstone.core.mcp_client._dispatch_pool` retry path.
|
||||
"""
|
||||
from turnstone.core.audit import record_audit
|
||||
from turnstone.core.auth import require_permission
|
||||
from turnstone.core.web_helpers import require_storage_or_503
|
||||
|
||||
storage, err = require_storage_or_503(request)
|
||||
if err:
|
||||
return err
|
||||
err = require_permission(request, "admin.mcp")
|
||||
if err:
|
||||
return err
|
||||
|
||||
name = request.path_params.get("name", "").strip()
|
||||
if not name or "__" in name:
|
||||
return JSONResponse({"error": "invalid server name"}, status_code=400)
|
||||
|
||||
existing = storage.get_mcp_server_by_name(name)
|
||||
if existing is None:
|
||||
return JSONResponse({"error": "No such server"}, status_code=404)
|
||||
if existing.get("auth_type") != "oauth_user":
|
||||
return JSONResponse(
|
||||
{"error": "bulk-revoke is only valid for auth_type=oauth_user servers"},
|
||||
status_code=400,
|
||||
)
|
||||
|
||||
target_id = existing.get("server_id", name)
|
||||
consented_before = 0
|
||||
try:
|
||||
consented_before = storage.count_mcp_consented_users_by_server(name)
|
||||
except Exception:
|
||||
log.debug("admin.mcp_bulk_revoke_pre_count_failed server=%s", name, exc_info=True)
|
||||
|
||||
deleted = storage.delete_mcp_oauth_rows_by_server_name(name)
|
||||
|
||||
audit_uid, ip = _audit_context(request)
|
||||
record_audit(
|
||||
storage,
|
||||
audit_uid,
|
||||
"mcp_server.oauth.bulk_revoked",
|
||||
"mcp_server",
|
||||
target_id,
|
||||
{
|
||||
"name": name,
|
||||
"rows_deleted": deleted,
|
||||
"consented_users_before": consented_before,
|
||||
"upstream_revoke_outcome": "bulk_admin_no_upstream",
|
||||
},
|
||||
ip,
|
||||
)
|
||||
|
||||
return JSONResponse(
|
||||
{
|
||||
"status": "ok",
|
||||
"rows_deleted": deleted,
|
||||
"consented_users_before": consented_before,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
async def admin_import_mcp_config(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/admin/mcp-servers/import — import from pasted JSON config."""
|
||||
import uuid
|
||||
@@ -11882,6 +12165,7 @@ def create_app(
|
||||
console_url: str = "",
|
||||
router: ConsoleRouter | None = None,
|
||||
console_metrics: ConsoleMetrics | None = None,
|
||||
notify_dispatcher: Any = None,
|
||||
) -> Starlette:
|
||||
"""Build the Starlette ASGI application for the console dashboard."""
|
||||
_spec = build_console_spec()
|
||||
@@ -12108,6 +12392,17 @@ def create_app(
|
||||
mcp_oauth_revoke_connection,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/mcp/oauth/pending", mcp_oauth_list_pending),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending",
|
||||
mcp_oauth_clear_all_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending/{server_name}",
|
||||
mcp_oauth_clear_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/admin/users", admin_list_users),
|
||||
Route("/api/admin/users", admin_create_user, methods=["POST"]),
|
||||
Route("/api/admin/users/{user_id}", admin_delete_user, methods=["DELETE"]),
|
||||
@@ -12293,6 +12588,11 @@ def create_app(
|
||||
admin_mcp_reconnect_one,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{name}/bulk-revoke",
|
||||
admin_mcp_bulk_revoke,
|
||||
methods=["POST"],
|
||||
),
|
||||
Route(
|
||||
"/api/admin/mcp-servers/{server_id}",
|
||||
admin_get_mcp_server,
|
||||
@@ -12508,6 +12808,7 @@ def create_app(
|
||||
lifespan=_lifespan,
|
||||
)
|
||||
app.state.collector = collector
|
||||
app.state.notify_dispatcher = notify_dispatcher
|
||||
app.state.jwt_secret = jwt_secret
|
||||
app.state.auth_storage = auth_storage
|
||||
app.state.proxy_token_mgr = proxy_token_mgr
|
||||
@@ -12603,7 +12904,7 @@ def main() -> None:
|
||||
from turnstone.core.config import add_config_arg, apply_config
|
||||
|
||||
add_config_arg(parser)
|
||||
apply_config(parser, ["console", "auth"])
|
||||
apply_config(parser, ["console", "auth", "database"])
|
||||
args = parser.parse_args()
|
||||
|
||||
from turnstone.core.log import configure_logging_from_args
|
||||
@@ -12622,6 +12923,13 @@ def main() -> None:
|
||||
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
|
||||
db_url = os.environ.get("TURNSTONE_DB_URL", "")
|
||||
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
|
||||
# Optional dedicated LISTEN URL — config.toml ``[database] listen_url``
|
||||
# (lifted onto args by ``apply_config``) wins over env, and an empty
|
||||
# value falls through to the main DB URL inside the storage layer.
|
||||
# Only used by the ``NotifyDispatcher``; ignored on SQLite.
|
||||
db_listen_url = getattr(args, "db_listen_url", None) or os.environ.get(
|
||||
"TURNSTONE_DB_LISTEN_URL", ""
|
||||
)
|
||||
auth_storage = init_storage(
|
||||
db_backend,
|
||||
path=db_path,
|
||||
@@ -12630,6 +12938,7 @@ def main() -> None:
|
||||
sslrootcert=os.environ.get("TURNSTONE_DB_SSLROOTCERT", ""),
|
||||
sslcert=os.environ.get("TURNSTONE_DB_SSLCERT", ""),
|
||||
sslkey=os.environ.get("TURNSTONE_DB_SSLKEY", ""),
|
||||
listen_url=db_listen_url,
|
||||
)
|
||||
except Exception:
|
||||
log.info("Console storage not available — admin API disabled, JWT-only auth")
|
||||
@@ -12659,11 +12968,21 @@ def main() -> None:
|
||||
router = ConsoleRouter(storage=auth_storage)
|
||||
console_metrics = ConsoleMetrics()
|
||||
|
||||
# NotifyDispatcher multiplexes the dedicated LISTEN connection for all
|
||||
# console-side consumers. Currently one channel: ``services`` for
|
||||
# reactive node discovery. Followup PRs (ConfigStore live reload,
|
||||
# scheduler immediate dispatch) add additional channels here.
|
||||
from turnstone.console.notify_dispatcher import NotifyDispatcher
|
||||
|
||||
notify_dispatcher = NotifyDispatcher(auth_storage, channels=["services"])
|
||||
notify_dispatcher.start()
|
||||
|
||||
collector = ClusterCollector(
|
||||
storage=auth_storage,
|
||||
token_manager=collector_token_mgr,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
collector.start()
|
||||
|
||||
@@ -12755,6 +13074,7 @@ def main() -> None:
|
||||
console_url=console_url,
|
||||
router=router,
|
||||
console_metrics=console_metrics,
|
||||
notify_dispatcher=notify_dispatcher,
|
||||
)
|
||||
|
||||
log.info("Console starting on %s", console_url)
|
||||
|
||||
@@ -3314,6 +3314,11 @@ function _renderMcpServers(items) {
|
||||
var totalTools = 0,
|
||||
totalRes = 0,
|
||||
totalPrompts = 0;
|
||||
// Phase 9: aggregate the most-recent refresh entry across nodes
|
||||
// so the admin pill reflects "the freshest known state" rather
|
||||
// than picking an arbitrary node.
|
||||
var newestRefreshAt = null;
|
||||
var newestRefreshOutcome = null;
|
||||
for (var j = 0; j < nodeIds.length; j++) {
|
||||
var ns = statusEntries[nodeIds[j]];
|
||||
if (ns.connected) {
|
||||
@@ -3326,6 +3331,13 @@ function _renderMcpServers(items) {
|
||||
anyError = true;
|
||||
if (!firstError) firstError = ns.error;
|
||||
}
|
||||
if (
|
||||
typeof ns.last_refresh_at === "number" &&
|
||||
(newestRefreshAt === null || ns.last_refresh_at > newestRefreshAt)
|
||||
) {
|
||||
newestRefreshAt = ns.last_refresh_at;
|
||||
newestRefreshOutcome = ns.last_refresh_outcome || null;
|
||||
}
|
||||
}
|
||||
|
||||
var dotClass = "mcp-status-dot disabled";
|
||||
@@ -3351,6 +3363,42 @@ function _renderMcpServers(items) {
|
||||
statusText = "idle";
|
||||
}
|
||||
|
||||
// Phase 9: refresh pill shows the short-relative age (e.g. "12m") with
|
||||
// outcome-tinted color (ok vs err) and the full ISO timestamp + outcome
|
||||
// in the tooltip. Pill is omitted (and the cell stays unchanged from
|
||||
// its pre-Phase-9 shape) when no node has yet recorded a refresh
|
||||
// outcome for this server.
|
||||
var refreshPill = "";
|
||||
if (newestRefreshAt !== null) {
|
||||
var ageSeconds = Math.max(
|
||||
0,
|
||||
Math.floor(Date.now() / 1000 - newestRefreshAt),
|
||||
);
|
||||
var ageShort;
|
||||
if (ageSeconds < 60) ageShort = ageSeconds + "s";
|
||||
else if (ageSeconds < 3600) ageShort = Math.floor(ageSeconds / 60) + "m";
|
||||
else if (ageSeconds < 86400)
|
||||
ageShort = Math.floor(ageSeconds / 3600) + "h";
|
||||
else ageShort = Math.floor(ageSeconds / 86400) + "d";
|
||||
var outcomeText = newestRefreshOutcome || "unknown";
|
||||
var pillCls =
|
||||
outcomeText === "ok" ? "mcp-refresh-pill-ok" : "mcp-refresh-pill-err";
|
||||
var pillTitle =
|
||||
"Last refresh " +
|
||||
new Date(newestRefreshAt * 1000).toISOString() +
|
||||
" (" +
|
||||
outcomeText +
|
||||
")";
|
||||
refreshPill =
|
||||
' <span class="mcp-refresh-pill ' +
|
||||
pillCls +
|
||||
'" title="' +
|
||||
escapeHtml(pillTitle) +
|
||||
'">' +
|
||||
escapeHtml(ageShort) +
|
||||
"</span>";
|
||||
}
|
||||
|
||||
var transportCls =
|
||||
s.transport === "stdio" ? "mcp-transport-stdio" : "mcp-transport-http";
|
||||
var toolsVal = anyConnected
|
||||
@@ -3385,6 +3433,24 @@ function _renderMcpServers(items) {
|
||||
'<button class="admin-btn-action" data-mcp-oauth-connect="' +
|
||||
escapeHtml(s.name) +
|
||||
'">connect</button>';
|
||||
// Phase 9: surface the consented-users count + bulk-revoke
|
||||
// affordance only when at least one user has consented.
|
||||
var consentCount =
|
||||
typeof s.consented_users_count === "number"
|
||||
? s.consented_users_count
|
||||
: 0;
|
||||
if (consentCount > 0) {
|
||||
actionBtns +=
|
||||
'<button class="admin-btn-danger" data-mcp-bulk-revoke="' +
|
||||
escapeHtml(s.name) +
|
||||
'" data-mcp-consent-count="' +
|
||||
consentCount +
|
||||
'" title="Drop all ' +
|
||||
consentCount +
|
||||
' user consents for this server">bulk-revoke (' +
|
||||
consentCount +
|
||||
")</button>";
|
||||
}
|
||||
}
|
||||
var actions = isConfig
|
||||
? actionBtns
|
||||
@@ -3429,6 +3495,7 @@ function _renderMcpServers(items) {
|
||||
dotClass +
|
||||
'" aria-hidden="true"></span>' +
|
||||
escapeHtml(statusText) +
|
||||
refreshPill +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-mactions">' +
|
||||
actions +
|
||||
@@ -3507,6 +3574,42 @@ function _renderMcpServers(items) {
|
||||
window.open(url, "_blank", "noopener");
|
||||
});
|
||||
});
|
||||
el.querySelectorAll("[data-mcp-bulk-revoke]").forEach(function (btn) {
|
||||
btn.addEventListener("click", function () {
|
||||
var name = this.getAttribute("data-mcp-bulk-revoke");
|
||||
var count = this.getAttribute("data-mcp-consent-count") || "?";
|
||||
showConfirmModal(
|
||||
"Bulk-revoke MCP consents",
|
||||
"Drop all " +
|
||||
count +
|
||||
' user consents for server "' +
|
||||
name +
|
||||
'"? Users will need to re-consent on next use. Upstream revoke is not attempted in bulk; tokens at the authorization server will expire naturally.',
|
||||
"Bulk-revoke",
|
||||
function () {
|
||||
authFetch(
|
||||
"/v1/api/admin/mcp-servers/" +
|
||||
encodeURIComponent(name) +
|
||||
"/bulk-revoke",
|
||||
{ method: "POST" },
|
||||
)
|
||||
.then(function (r) {
|
||||
if (!r.ok) throw new Error();
|
||||
return r.json();
|
||||
})
|
||||
.then(function (j) {
|
||||
showToast(
|
||||
"Bulk-revoked " + (j.rows_deleted || 0) + " row(s) for " + name,
|
||||
);
|
||||
loadAdminMcp();
|
||||
})
|
||||
.catch(function () {
|
||||
showToast("Failed to bulk-revoke " + name);
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
el.querySelectorAll("[data-mcp-delete]").forEach(function (btn) {
|
||||
btn.addEventListener("click", function () {
|
||||
var sid = this.getAttribute("data-mcp-delete");
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
<link rel="stylesheet" href="/shared/base.css">
|
||||
<link rel="stylesheet" href="/shared/ui-base.css">
|
||||
<link rel="stylesheet" href="/shared/chat.css">
|
||||
<link rel="stylesheet" href="/shared/katex-0.16.45/katex.min.css">
|
||||
<link rel="stylesheet" href="/shared/katex-0.16.46/katex.min.css">
|
||||
<link rel="stylesheet" href="/static/style.css">
|
||||
<link rel="stylesheet" href="/static/coordinator/coordinator.css">
|
||||
<style>
|
||||
@@ -634,7 +634,7 @@
|
||||
<script src="/shared/composer_attachments.js"></script>
|
||||
<script src="/shared/composer_queue.js"></script>
|
||||
<script src="/shared/status_bar.js"></script>
|
||||
<script src="/shared/katex-0.16.45/katex.min.js"></script>
|
||||
<script src="/shared/katex-0.16.46/katex.min.js"></script>
|
||||
<script src="/shared/hljs-11.11.1/highlight.min.js"></script>
|
||||
<script src="/shared/renderer.js"></script>
|
||||
<script src="/static/coordinator/coordinator.js"></script>
|
||||
|
||||
@@ -3602,6 +3602,28 @@ textarea.skill-content-area {
|
||||
opacity: 0.4;
|
||||
}
|
||||
|
||||
/* Phase 9 — last-refresh pill in the MCP status cell. Compact age +
|
||||
outcome indicator inline with the existing status text. Uses the
|
||||
semantic theme tokens (--bg-highlight, --fg-dim, --warn) defined in
|
||||
shared_static/base.css so the pill follows dark/light theme swaps. */
|
||||
.mcp-refresh-pill {
|
||||
display: inline-block;
|
||||
margin-left: 6px;
|
||||
padding: 0 4px;
|
||||
border-radius: var(--radius-sm);
|
||||
font-size: 0.8em;
|
||||
font-variant-numeric: tabular-nums;
|
||||
opacity: 0.85;
|
||||
}
|
||||
.mcp-refresh-pill-ok {
|
||||
background: var(--bg-highlight);
|
||||
color: var(--fg-dim);
|
||||
}
|
||||
.mcp-refresh-pill-err {
|
||||
background: color-mix(in srgb, var(--warn) 15%, transparent);
|
||||
color: var(--warn);
|
||||
}
|
||||
|
||||
.mcp-detail-modal::before {
|
||||
background: linear-gradient(
|
||||
90deg,
|
||||
|
||||
+15
-1
@@ -513,7 +513,21 @@ def is_public_path(path: str) -> bool:
|
||||
normalized = _strip_version_prefix(path)
|
||||
if normalized in PUBLIC_PATHS:
|
||||
return True
|
||||
return any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES)
|
||||
if any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
# Console proxy: a public proxied path is still public. Without this
|
||||
# the login/status/setup endpoints are unreachable from inside a
|
||||
# ``/node/{id}/...`` proxied page once the cookie expires — the
|
||||
# AuthMiddleware 401s the login POST before any handler runs and the
|
||||
# user is locked out of the proxied UI.
|
||||
if normalized.startswith("/node/"):
|
||||
proxied = _extract_proxied_path(normalized)
|
||||
if proxied is not None:
|
||||
if proxied in PUBLIC_PATHS:
|
||||
return True
|
||||
if any(proxied.startswith(prefix) for prefix in PUBLIC_PREFIXES):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def required_scope(method: str, path: str) -> str:
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Per-workstream wakeup primitive for in-process child state-change subscribers.
|
||||
|
||||
Retires the polling pattern in ``CoordinatorClient.wait_for_workstream``,
|
||||
where the coord LLM's wait tool issued a storage snapshot every 0.5 s
|
||||
regardless of whether anything had changed. The dispatch path
|
||||
(:meth:`turnstone.console.coordinator_adapter.CoordinatorAdapter._dispatch_child_event`)
|
||||
now calls :meth:`ChildEventBus.notify` after each translated child event;
|
||||
waiters block on a per-call :class:`threading.Event` returned by
|
||||
:meth:`register_waiter` and re-read storage only when an event fires or
|
||||
the heartbeat cap expires.
|
||||
|
||||
Bus is in-process only. Cross-process / cross-node child events are
|
||||
already merged into ``_dispatch_child_event`` via the cluster collector's
|
||||
SSE multiplex before the bus sees them — there is no locality branching
|
||||
in the bus itself.
|
||||
|
||||
Design constraints:
|
||||
|
||||
- Waiter primitive is :class:`threading.Event` because the wait tool runs
|
||||
on the coordinator's sync worker thread, not an asyncio loop.
|
||||
- Concurrent ``register`` / ``unregister`` / ``notify`` is safe — a
|
||||
single ``threading.Lock`` guards the dict. ``Event.set`` itself is
|
||||
thread-safe and is called outside the lock so a slow waker can't block
|
||||
registration.
|
||||
- ``notify`` with no subscribers is a no-op (the steady state — most
|
||||
state-change events fire while no wait tool is active).
|
||||
- A waiter watching multiple ws_ids fires once on any of them; the
|
||||
caller's ``_snapshot_all`` re-read resolves which one changed.
|
||||
- Empty per-ws_id buckets are popped on unregister so long-lived buses
|
||||
don't accumulate dead keys after wait churn.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterable
|
||||
|
||||
|
||||
class ChildEventBus:
|
||||
"""Fan ``notify(child_ws_id)`` to every :class:`threading.Event`
|
||||
registered against that ws_id.
|
||||
|
||||
Use :meth:`register_waiter` once per wait call to obtain an Event,
|
||||
then call :meth:`unregister_waiter` in a ``finally`` so a crash mid-
|
||||
wait doesn't leak the registration. The dispatch side calls
|
||||
:meth:`notify` on every translated child state-change event; an
|
||||
empty bucket is a cheap dict lookup + immediate return.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._waiters: dict[str, set[threading.Event]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def register_waiter(self, child_ws_ids: Iterable[str]) -> threading.Event:
|
||||
"""Return a fresh Event registered against every listed ws_id.
|
||||
|
||||
A wait on ``[A, B, C]`` returns a single Event that fires when
|
||||
*any* of A/B/C changes. The caller's snapshot re-read resolves
|
||||
which one. Empty / falsy ids are silently skipped — callers that
|
||||
clean their input upstream (e.g. ``wait_for_workstream``'s
|
||||
dedup + cap) don't need to filter again here.
|
||||
"""
|
||||
event = threading.Event()
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
self._waiters.setdefault(wid, set()).add(event)
|
||||
return event
|
||||
|
||||
def unregister_waiter(
|
||||
self,
|
||||
child_ws_ids: Iterable[str],
|
||||
event: threading.Event,
|
||||
) -> None:
|
||||
"""Remove ``event`` from each listed ws_id's waiter set.
|
||||
|
||||
Idempotent — already-removed Events silently no-op. Pops empty
|
||||
sets so a long-lived bus doesn't accumulate dead keys after
|
||||
many waits have come and gone. Must be called from the same
|
||||
``finally`` that paired with :meth:`register_waiter` so a
|
||||
crash mid-wait doesn't leak the registration past one wait's
|
||||
lifetime.
|
||||
"""
|
||||
with self._lock:
|
||||
for wid in child_ws_ids:
|
||||
if not wid:
|
||||
continue
|
||||
bucket = self._waiters.get(wid)
|
||||
if bucket is None:
|
||||
continue
|
||||
bucket.discard(event)
|
||||
if not bucket:
|
||||
self._waiters.pop(wid, None)
|
||||
|
||||
def notify(self, child_ws_id: str) -> None:
|
||||
"""Wake every Event registered for ``child_ws_id``.
|
||||
|
||||
Called from the coord dispatch sink after each translated child
|
||||
event. Snapshot the bucket under the lock, then call
|
||||
``Event.set`` outside the lock so a slow waker doesn't block
|
||||
``register`` / ``unregister`` / further ``notify``. ``Event.set``
|
||||
is thread-safe and idempotent — re-firing a still-set Event is
|
||||
a no-op.
|
||||
|
||||
Empty / falsy ws_ids are silently dropped; the same hot path
|
||||
runs for every dispatched event regardless of whether anyone's
|
||||
waiting, so the empty-bucket case must stay cheap.
|
||||
"""
|
||||
if not child_ws_id:
|
||||
return
|
||||
with self._lock:
|
||||
bucket = self._waiters.get(child_ws_id)
|
||||
if not bucket:
|
||||
return
|
||||
events = list(bucket)
|
||||
for event in events:
|
||||
event.set()
|
||||
@@ -143,6 +143,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
|
||||
"sslrootcert": "db_sslrootcert",
|
||||
"sslcert": "db_sslcert",
|
||||
"sslkey": "db_sslkey",
|
||||
"listen_url": "db_listen_url",
|
||||
},
|
||||
"judge": {
|
||||
"enabled": "judge_enabled",
|
||||
|
||||
@@ -27,6 +27,7 @@ import urllib.parse
|
||||
import uuid
|
||||
from contextlib import AsyncExitStack
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
@@ -45,6 +46,7 @@ from turnstone.core.config import load_config
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.mcp_http_parsers import (
|
||||
MAX_INSUFFICIENT_SCOPE_REPORTED,
|
||||
is_valid_scope_token,
|
||||
parse_www_authenticate_error,
|
||||
parse_www_authenticate_scope,
|
||||
)
|
||||
@@ -510,6 +512,20 @@ class MCPClientManager:
|
||||
# Notification debounce (per-server)
|
||||
self._last_notification_refresh: dict[str, float] = {}
|
||||
|
||||
# Last refresh outcome (Phase 9 — admin status indicator). Per-
|
||||
# server tuple of ``(unix_ts, outcome)`` where outcome is one of
|
||||
# ``ok`` or ``error:<ExceptionClassName>``. Populated by
|
||||
# ``_refresh_server`` on every call (success and failure paths),
|
||||
# which means manual operator-driven refresh (``refresh_sync``)
|
||||
# AND the ``_cb_auto_reconnect`` follow-up that schedules
|
||||
# ``_refresh_server`` directly both populate the field — adding
|
||||
# a future schedule site only needs to call ``_refresh_server``
|
||||
# to participate. Read by the admin status endpoint to render
|
||||
# the per-server "last refresh" pill. No initial entry is
|
||||
# created at server-register time — absence surfaces as ``null``
|
||||
# in the admin JSON, which the UI renders as "never".
|
||||
self._last_refresh: dict[str, tuple[float, str]] = {}
|
||||
|
||||
# Per-(user, server) state for auth_type=oauth_user. Loop-bound:
|
||||
# mutated only on the mcp-loop. Sync threads interact via
|
||||
# ``asyncio.run_coroutine_threadsafe``.
|
||||
@@ -2131,14 +2147,51 @@ class MCPClientManager:
|
||||
|
||||
Returns ``(added_tools, removed_tools)`` names (tool diff only,
|
||||
for backward compatibility with ``/mcp refresh`` output).
|
||||
|
||||
Writes the ``_last_refresh`` entry on every call so the Phase 9
|
||||
admin status pill reflects every refresh path — manual
|
||||
operator-driven ``refresh_sync`` AND the ``_cb_auto_reconnect``
|
||||
follow-up that schedules ``_refresh_server`` directly.
|
||||
Centralising the write here means future schedule sites
|
||||
automatically populate the field.
|
||||
|
||||
Uses ``asyncio.gather(return_exceptions=True)`` so that a failure
|
||||
in one of the three concurrent sub-refreshes does NOT orphan the
|
||||
others mid-mutation: every sibling reaches completion (success
|
||||
or per-task failure) before the outcome is computed. The
|
||||
``_last_refresh`` write is ``"ok"`` iff all three succeeded; on
|
||||
any failure the outcome is ``f"error:{type(first_exc).__name__}"``
|
||||
and the first exception is re-raised so the outer caller's error
|
||||
path (``_refresh_all``'s except, or the manual-refresh sync
|
||||
wrapper) sees the same shape it did before this rework.
|
||||
Partial-success mutations of ``state.tools`` / ``state.resources``
|
||||
/ ``state.prompts`` are bounded to whichever sub-refresh
|
||||
succeeded — the documented trade-off vs leaving orphan tasks
|
||||
running after the error is observed.
|
||||
"""
|
||||
tool_diff, _, _ = await asyncio.gather(
|
||||
results = await asyncio.gather(
|
||||
self._refresh_server_tools(name),
|
||||
self._refresh_server_resources(name),
|
||||
self._refresh_server_prompts(name),
|
||||
return_exceptions=True,
|
||||
)
|
||||
first_exc: BaseException | None = next(
|
||||
(r for r in results if isinstance(r, BaseException)), None
|
||||
)
|
||||
if first_exc is not None:
|
||||
self._last_refresh[name] = (
|
||||
time.time(),
|
||||
f"error:{type(first_exc).__name__}",
|
||||
)
|
||||
raise first_exc
|
||||
tool_diff = results[0]
|
||||
# ``return_exceptions=True`` widens the static type; on the all-
|
||||
# success path each entry is the awaited result. We narrow the
|
||||
# tool-diff entry to the documented ``(added, removed)`` shape.
|
||||
assert isinstance(tool_diff, tuple)
|
||||
added, removed = tool_diff
|
||||
self._last_error.pop(name, None)
|
||||
self._last_refresh[name] = (time.time(), "ok")
|
||||
return added, removed
|
||||
|
||||
async def _refresh_all(
|
||||
@@ -2167,6 +2220,7 @@ class MCPClientManager:
|
||||
[t["function"]["name"] for t in post.tools] if post is not None else []
|
||||
)
|
||||
results[name] = (new_names, [])
|
||||
self._last_refresh[name] = (time.time(), "ok")
|
||||
continue
|
||||
added, removed = await self._refresh_server(name)
|
||||
self._cb_record_success(name)
|
||||
@@ -2175,6 +2229,20 @@ class MCPClientManager:
|
||||
log.warning("Refresh failed for MCP server '%s'", name, exc_info=True)
|
||||
self._set_error(name, f"Refresh failed: {exc}")
|
||||
results[name] = ([], [])
|
||||
# Overwrite unconditionally with the freshest observed
|
||||
# outcome. Two cases produce the write:
|
||||
# (1) Reconnect branch: ``_connect_one`` raised before
|
||||
# ``_refresh_server`` could write — no prior entry
|
||||
# from this iteration exists yet, so the write is
|
||||
# the only fresh signal.
|
||||
# (2) ``_refresh_server`` branch: it already wrote a
|
||||
# fresh ``error:<ClassName>`` before re-raising, so
|
||||
# the outer overwrite is a no-op for the value.
|
||||
# Using ``setdefault`` here would preserve a stale prior
|
||||
# ``"ok"`` from the previous successful refresh when the
|
||||
# current attempt fails — the admin pill would show
|
||||
# "ok" for a broken server.
|
||||
self._last_refresh[name] = (time.time(), f"error:{type(exc).__name__}")
|
||||
|
||||
# Final sync to clean up templates from servers that are no longer connected
|
||||
try:
|
||||
@@ -2952,6 +3020,7 @@ class MCPClientManager:
|
||||
transport = cfg.get("type", "stdio")
|
||||
cb_deadline = self._circuit_open_until.get(name)
|
||||
cb_open = cb_deadline is not None and time.monotonic() < cb_deadline
|
||||
last_refresh = self._last_refresh.get(name)
|
||||
# Inline predicate (instead of reusing ``connected``) so mypy narrows
|
||||
# ``state`` for the attribute reads — a separate boolean wouldn't.
|
||||
return {
|
||||
@@ -2967,6 +3036,10 @@ class MCPClientManager:
|
||||
"url": cfg.get("url", "") if transport != "stdio" else "",
|
||||
"circuit_open": cb_open,
|
||||
"consecutive_failures": self._consecutive_failures.get(name, 0),
|
||||
# Phase 9 admin status: last manual / auto-reconnect refresh.
|
||||
# ``null`` when no refresh has occurred since process start.
|
||||
"last_refresh_at": last_refresh[0] if last_refresh is not None else None,
|
||||
"last_refresh_outcome": last_refresh[1] if last_refresh is not None else None,
|
||||
}
|
||||
|
||||
def get_all_server_status(self) -> dict[str, dict[str, Any]]:
|
||||
@@ -3303,6 +3376,7 @@ class MCPClientManager:
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
timeout: int = 120,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> str:
|
||||
"""Execute an MCP tool call synchronously (blocks the calling thread).
|
||||
|
||||
@@ -3348,6 +3422,7 @@ class MCPClientManager:
|
||||
arguments=arguments,
|
||||
server_row=pool_target[2],
|
||||
timeout=timeout,
|
||||
is_interactive_for_consent=is_interactive_for_consent,
|
||||
)
|
||||
|
||||
if mapping is None or server_name is None or original_name is None:
|
||||
@@ -3500,6 +3575,54 @@ class MCPClientManager:
|
||||
return None
|
||||
return server_name, original, row
|
||||
|
||||
def _record_pending_consent_best_effort(
|
||||
self,
|
||||
*,
|
||||
user_id: str,
|
||||
server_name: str,
|
||||
result: str,
|
||||
) -> None:
|
||||
"""Persist a deferred-consent row for non-interactive callers.
|
||||
|
||||
Called from the three sync dispatchers when the dispatch returns
|
||||
a structured-error envelope AND the caller is not interactive
|
||||
(CHAT / SCHEDULED). Filters out structured-error codes that
|
||||
aren't user-consent-shaped (key-unknown, url-insecure,
|
||||
*_forbidden) — those are operator-actionable and outside the
|
||||
scope of the dashboard pending-consent badge.
|
||||
|
||||
Best-effort. A storage exception is logged but never raised:
|
||||
the structured-error envelope must reach the agent unchanged
|
||||
regardless of whether the pending-consent row was persisted, so
|
||||
a transient DB failure doesn't change the agent-observable
|
||||
contract.
|
||||
"""
|
||||
if self._storage is None:
|
||||
return
|
||||
parsed = _parse_pending_consent_envelope(result)
|
||||
if parsed is None:
|
||||
return
|
||||
code, scopes = parsed
|
||||
scopes_str = " ".join(scopes) if scopes else None
|
||||
try:
|
||||
self._storage.upsert_mcp_pending_consent(
|
||||
user_id=user_id,
|
||||
server_name=server_name,
|
||||
error_code=code,
|
||||
scopes_required=scopes_str,
|
||||
last_ws_id=None,
|
||||
last_tool_call_id=None,
|
||||
now_iso=datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
)
|
||||
except Exception:
|
||||
log.warning(
|
||||
"mcp_pool.pending_consent_persist_failed user=%s server=%s code=%s",
|
||||
user_id,
|
||||
server_name,
|
||||
code,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
def _dispatch_pool_sync(
|
||||
self,
|
||||
*,
|
||||
@@ -3509,6 +3632,7 @@ class MCPClientManager:
|
||||
arguments: dict[str, Any],
|
||||
server_row: dict[str, Any],
|
||||
timeout: int,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> str:
|
||||
"""Synchronous wrapper for pool dispatch.
|
||||
|
||||
@@ -3581,6 +3705,10 @@ class MCPClientManager:
|
||||
server_row=server_row,
|
||||
)
|
||||
if _is_structured_error(result):
|
||||
if not is_interactive_for_consent:
|
||||
self._record_pending_consent_best_effort(
|
||||
user_id=user_id, server_name=server_name, result=result
|
||||
)
|
||||
raise RuntimeError(result)
|
||||
return result
|
||||
|
||||
@@ -3636,6 +3764,7 @@ class MCPClientManager:
|
||||
uri: str,
|
||||
server_row: dict[str, Any],
|
||||
timeout: int,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> str:
|
||||
"""Synchronous wrapper for pool resource read.
|
||||
|
||||
@@ -3679,6 +3808,10 @@ class MCPClientManager:
|
||||
server_row=server_row,
|
||||
)
|
||||
if _is_structured_error(result):
|
||||
if not is_interactive_for_consent:
|
||||
self._record_pending_consent_best_effort(
|
||||
user_id=user_id, server_name=server_name, result=result
|
||||
)
|
||||
raise RuntimeError(result)
|
||||
return result
|
||||
|
||||
@@ -3722,6 +3855,7 @@ class MCPClientManager:
|
||||
arguments: dict[str, str] | None,
|
||||
server_row: dict[str, Any],
|
||||
timeout: int,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Synchronous wrapper for pool prompt invocation.
|
||||
|
||||
@@ -3765,6 +3899,10 @@ class MCPClientManager:
|
||||
# Structured-error path — surface as RuntimeError so the
|
||||
# agent-loop renders the JSON via its except-Exception
|
||||
# handler.
|
||||
if not is_interactive_for_consent:
|
||||
self._record_pending_consent_best_effort(
|
||||
user_id=user_id, server_name=server_name, result=result
|
||||
)
|
||||
raise RuntimeError(result)
|
||||
return result
|
||||
|
||||
@@ -4604,7 +4742,12 @@ class MCPClientManager:
|
||||
return best
|
||||
|
||||
def read_resource_sync(
|
||||
self, uri: str, *, user_id: str | None = None, timeout: int = 120
|
||||
self,
|
||||
uri: str,
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
timeout: int = 120,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> str:
|
||||
"""Read a resource by URI synchronously (blocks the calling thread).
|
||||
|
||||
@@ -4627,6 +4770,7 @@ class MCPClientManager:
|
||||
uri=pool_target[1],
|
||||
server_row=pool_target[2],
|
||||
timeout=timeout,
|
||||
is_interactive_for_consent=is_interactive_for_consent,
|
||||
)
|
||||
|
||||
mapping = self._resource_map.get(uri)
|
||||
@@ -4689,6 +4833,7 @@ class MCPClientManager:
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
timeout: int = 30,
|
||||
is_interactive_for_consent: bool = True,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Invoke an MCP prompt synchronously and return expanded messages.
|
||||
|
||||
@@ -4727,6 +4872,7 @@ class MCPClientManager:
|
||||
arguments=arguments,
|
||||
server_row=pool_target[2],
|
||||
timeout=timeout,
|
||||
is_interactive_for_consent=is_interactive_for_consent,
|
||||
)
|
||||
|
||||
if static_mapping is None:
|
||||
@@ -4912,6 +5058,64 @@ def _structured_error(
|
||||
return json.dumps({"error": err})
|
||||
|
||||
|
||||
# Structured-error codes that represent a deferred-consent need. When
|
||||
# encountered on a non-interactive call (chat / scheduled), the sync
|
||||
# dispatcher persists a row to ``mcp_pending_consent`` so the dashboard
|
||||
# badge can surface the deferred work later. Operator-actionable codes
|
||||
# (key-unknown, url-insecure, *_forbidden) are intentionally excluded —
|
||||
# the user cannot resolve them by completing a consent flow.
|
||||
_PENDING_CONSENT_PERSIST_CODES: frozenset[str] = frozenset(
|
||||
{"mcp_consent_required", "mcp_insufficient_scope"}
|
||||
)
|
||||
|
||||
|
||||
def _parse_pending_consent_envelope(
|
||||
result: str,
|
||||
) -> tuple[str, list[str] | None] | None:
|
||||
"""Extract ``(error_code, scopes_required)`` from a structured-error JSON.
|
||||
|
||||
Returns ``None`` when the envelope's ``code`` is not in
|
||||
:data:`_PENDING_CONSENT_PERSIST_CODES`. Callers should already have
|
||||
gated on :func:`_is_structured_error`; this helper deliberately
|
||||
re-parses (cheap on the failure path) rather than threading the
|
||||
decoded dict through the sync-dispatcher hot path.
|
||||
|
||||
Defends against non-dict JSON values (``null``, strings, numbers)
|
||||
via the same ``isinstance(decoded, dict)`` guard
|
||||
:func:`_is_structured_error` uses, so a misuse from a future caller
|
||||
that bypasses the structured-error contract surfaces as a clean
|
||||
``None`` rather than an ``AttributeError`` propagating out of the
|
||||
sync dispatcher's hot path.
|
||||
"""
|
||||
try:
|
||||
decoded = json.loads(result)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return None
|
||||
if not isinstance(decoded, dict):
|
||||
return None
|
||||
err = decoded.get("error")
|
||||
if not isinstance(err, dict):
|
||||
return None
|
||||
code = err.get("code", "")
|
||||
if code not in _PENDING_CONSENT_PERSIST_CODES:
|
||||
return None
|
||||
scopes = err.get("scopes_required")
|
||||
if isinstance(scopes, list):
|
||||
# Defense-in-depth scope filter — production paths construct
|
||||
# this list via ``parse_www_authenticate_scope`` which already
|
||||
# validates and caps, but the helper is reusable; re-applying
|
||||
# the predicate here forecloses any future caller that bypasses
|
||||
# the upstream filter from landing attacker-controlled bytes in
|
||||
# ``mcp_pending_consent.scopes_required``. Type-filter BEFORE
|
||||
# ``is_valid_scope_token`` so non-string entries (``None``,
|
||||
# ints) don't slip through as their ``str()`` repr (e.g.
|
||||
# ``None`` → ``"None"`` passes the ASCII grammar). Cap mirrors
|
||||
# ``MAX_INSUFFICIENT_SCOPE_REPORTED`` semantics.
|
||||
cleaned = [s for s in scopes if isinstance(s, str) and is_valid_scope_token(s)]
|
||||
return code, cleaned[:MAX_INSUFFICIENT_SCOPE_REPORTED]
|
||||
return code, None
|
||||
|
||||
|
||||
def _is_structured_error(result: str) -> bool:
|
||||
"""Return True if *result* parses as a :func:`_structured_error` envelope.
|
||||
|
||||
|
||||
@@ -2350,6 +2350,22 @@ async def _handle_mcp_oauth_callback_inner(request: Request) -> Response:
|
||||
},
|
||||
)
|
||||
|
||||
# Phase 9 — clear any deferred-consent records for this (user,
|
||||
# server) now that consent has completed. Best-effort: a storage
|
||||
# failure here doesn't change the user-observable callback success;
|
||||
# the worst case is a stale badge that the user can dismiss
|
||||
# manually. ``delete_mcp_pending_consent`` returns False on
|
||||
# no-such-row (the common case for interactive consent flows that
|
||||
# never deferred), which is fine.
|
||||
try:
|
||||
await asyncio.to_thread(storage.delete_mcp_pending_consent, user_id, server_name)
|
||||
except Exception:
|
||||
log.debug(
|
||||
"mcp_server.oauth.pending_consent_clear_failed",
|
||||
server_name=server_name,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
return RedirectResponse(pending["return_url"] or "/", status_code=302)
|
||||
|
||||
|
||||
@@ -2618,6 +2634,153 @@ async def _handle_mcp_oauth_revoke_connection_inner(request: Request) -> Respons
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pending-consent endpoints (Phase 9)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def handle_mcp_oauth_list_pending(request: Request) -> Response:
|
||||
"""``GET /v1/api/mcp/oauth/pending``.
|
||||
|
||||
Returns the authenticated user's deferred-consent records — populated
|
||||
by the pool dispatchers when a non-interactive run (scheduled /
|
||||
channel) hits ``mcp_consent_required`` or ``mcp_insufficient_scope``.
|
||||
Used by the dashboard badge to surface deferred consent needs on
|
||||
next login.
|
||||
|
||||
Install-level gate: when no ``mcp_servers`` row has
|
||||
``auth_type='oauth_user'``, the entire feature is dark — we
|
||||
short-circuit to ``{pending: 0, servers: []}`` without querying the
|
||||
pending table at all. This keeps local-auth installs on a
|
||||
zero-new-storage-query path.
|
||||
"""
|
||||
return _apply_security_headers(await _handle_mcp_oauth_list_pending_inner(request))
|
||||
|
||||
|
||||
_INSTALL_GATE_CACHE_TTL_S = 60.0
|
||||
|
||||
|
||||
async def _install_gate_passes(app_state: Any, storage: Any) -> bool:
|
||||
"""Cached install-level gate for OAuth-MCP features.
|
||||
|
||||
Returns True iff at least one ``mcp_servers`` row has
|
||||
``auth_type='oauth_user'``. Result is cached on ``app_state`` for
|
||||
:data:`_INSTALL_GATE_CACHE_TTL_S` seconds — admin-rare transitions
|
||||
don't justify a per-request DB round-trip on every dashboard load.
|
||||
|
||||
Reset semantics: cache is invalidated by time only. Operators who
|
||||
just enabled an ``oauth_user`` row see the gate flip within the TTL
|
||||
window. False positives (cache says True but the row was just
|
||||
deleted) are bounded by the same window — the downstream list
|
||||
query already filters by user, so the cost is at most one cheap
|
||||
user-scoped read.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
cached = getattr(app_state, "_mcp_install_gate_cache", None)
|
||||
if cached is not None:
|
||||
cached_value, cached_at = cached
|
||||
if (now - cached_at) < _INSTALL_GATE_CACHE_TTL_S:
|
||||
return bool(cached_value)
|
||||
value = bool(await asyncio.to_thread(storage.any_oauth_user_mcp_servers))
|
||||
app_state._mcp_install_gate_cache = (value, now)
|
||||
return value
|
||||
|
||||
|
||||
async def _handle_mcp_oauth_list_pending_inner(request: Request) -> Response:
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
user_id = _require_user_id(request)
|
||||
if user_id is None:
|
||||
return JSONResponse({"error": "Authentication required"}, status_code=401)
|
||||
|
||||
storage = _get_storage(request.app.state)
|
||||
if storage is None:
|
||||
return JSONResponse({"pending": 0, "servers": []})
|
||||
|
||||
if not await _install_gate_passes(request.app.state, storage):
|
||||
return JSONResponse({"pending": 0, "servers": []})
|
||||
|
||||
rows = await asyncio.to_thread(storage.list_mcp_pending_consent_by_user, user_id)
|
||||
return JSONResponse({"pending": len(rows), "servers": list(rows)})
|
||||
|
||||
|
||||
async def handle_mcp_oauth_clear_pending(request: Request) -> Response:
|
||||
"""``DELETE /v1/api/mcp/oauth/pending/{server_name}``.
|
||||
|
||||
Manual user-initiated dismissal of a single deferred-consent record.
|
||||
Called from the dashboard settings modal when the user opts to clear
|
||||
the entry without completing consent (e.g., the underlying
|
||||
auth_type was changed and the deferred record is now stale).
|
||||
|
||||
Returns 204 in both the existed-and-deleted and never-existed cases
|
||||
to keep cross-tenant existence non-observable.
|
||||
"""
|
||||
return _apply_security_headers(await _handle_mcp_oauth_clear_pending_inner(request))
|
||||
|
||||
|
||||
async def _handle_mcp_oauth_clear_pending_inner(request: Request) -> Response:
|
||||
from starlette.responses import JSONResponse, Response
|
||||
|
||||
user_id = _require_user_id(request)
|
||||
if user_id is None:
|
||||
return JSONResponse({"error": "Authentication required"}, status_code=401)
|
||||
|
||||
server_name = request.path_params.get("server_name", "").strip()
|
||||
if not server_name:
|
||||
return JSONResponse({"error": "Missing server_name"}, status_code=400)
|
||||
|
||||
storage = _get_storage(request.app.state)
|
||||
if storage is None:
|
||||
return JSONResponse({"error": "Storage unavailable"}, status_code=503)
|
||||
|
||||
cleared = bool(
|
||||
await asyncio.to_thread(storage.delete_mcp_pending_consent, user_id, server_name)
|
||||
)
|
||||
# Audit even on no-op deletes (returns 204 either way for cross-tenant
|
||||
# non-observability) so an attacker who tries to scrub deferred-consent
|
||||
# breadcrumbs leaves an audit trail of the attempts.
|
||||
await _audit_event(
|
||||
request.app.state,
|
||||
user_id=user_id,
|
||||
action="mcp_server.oauth.pending_consent_dismissed",
|
||||
server_name=server_name,
|
||||
detail={"mode": "single", "cleared": 1 if cleared else 0},
|
||||
)
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
async def handle_mcp_oauth_clear_all_pending(request: Request) -> Response:
|
||||
"""``DELETE /v1/api/mcp/oauth/pending``.
|
||||
|
||||
Bulk dismiss of every deferred-consent record for the authenticated
|
||||
user. Returns the count cleared so the dashboard can update its
|
||||
badge in one round-trip.
|
||||
"""
|
||||
return _apply_security_headers(await _handle_mcp_oauth_clear_all_pending_inner(request))
|
||||
|
||||
|
||||
async def _handle_mcp_oauth_clear_all_pending_inner(request: Request) -> Response:
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
user_id = _require_user_id(request)
|
||||
if user_id is None:
|
||||
return JSONResponse({"error": "Authentication required"}, status_code=401)
|
||||
|
||||
storage = _get_storage(request.app.state)
|
||||
if storage is None:
|
||||
return JSONResponse({"error": "Storage unavailable"}, status_code=503)
|
||||
|
||||
cleared = await asyncio.to_thread(storage.delete_all_mcp_pending_consent_by_user, user_id)
|
||||
await _audit_event(
|
||||
request.app.state,
|
||||
user_id=user_id,
|
||||
action="mcp_server.oauth.pending_consent_dismissed",
|
||||
server_name="(bulk)",
|
||||
detail={"mode": "bulk", "cleared": cleared},
|
||||
)
|
||||
return JSONResponse({"cleared": cleared})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lifespan integration
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -2674,7 +2837,10 @@ __all__ = [
|
||||
"get_user_access_token_classified",
|
||||
"handle_mcp_oauth_authorize",
|
||||
"handle_mcp_oauth_callback",
|
||||
"handle_mcp_oauth_clear_all_pending",
|
||||
"handle_mcp_oauth_clear_pending",
|
||||
"handle_mcp_oauth_list_connections",
|
||||
"handle_mcp_oauth_list_pending",
|
||||
"handle_mcp_oauth_revoke_connection",
|
||||
"initialize_mcp_oauth_state",
|
||||
"pop_pending_state",
|
||||
|
||||
@@ -500,10 +500,14 @@ def load_model_registry(
|
||||
server_compat=entry_server_compat,
|
||||
)
|
||||
|
||||
# 3. Ensure a "default" entry from CLI args (only if not already defined
|
||||
# by config.toml or DB — those take precedence, and only when a CLI
|
||||
# model was actually provided)
|
||||
if "default" not in configs and model:
|
||||
# 3. Back-compat shim: synthesize a "default" alias from CLI/auto-detected
|
||||
# ``--base-url`` + ``--model`` only when no DB or config.toml models exist.
|
||||
# Auto-creating "default" alongside DB models leaks a non-routing alias
|
||||
# into the public list — the LLM picks it in plan_agent / task_agent
|
||||
# ``model=`` and silently bypasses the operator's per-role
|
||||
# plan_alias / task_alias overrides (the "default" alias points at
|
||||
# whatever LLM_BASE_URL was at boot, not at the configured default).
|
||||
if not configs and model:
|
||||
configs["default"] = ModelConfig(
|
||||
alias="default",
|
||||
base_url=base_url,
|
||||
|
||||
+420
-40
@@ -50,7 +50,7 @@ from turnstone.core.log import get_logger
|
||||
from turnstone.core.memory import (
|
||||
count_structured_memories,
|
||||
delete_messages_after,
|
||||
delete_structured_memory,
|
||||
delete_structured_memory_by_id,
|
||||
delete_workstream,
|
||||
get_skill_by_name,
|
||||
get_structured_memory_by_name,
|
||||
@@ -112,7 +112,12 @@ from turnstone.core.tools import (
|
||||
from turnstone.core.watch import WATCH_REMINDER_OPTIONAL_KEYS
|
||||
from turnstone.core.web import check_ssrf, strip_html
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
from turnstone.prompts import ClientType, SessionContext, compose_system_message
|
||||
from turnstone.prompts import (
|
||||
INTERACTIVE_CONSENT_CLIENT_TYPES,
|
||||
ClientType,
|
||||
SessionContext,
|
||||
compose_system_message,
|
||||
)
|
||||
from turnstone.ui.colors import DIM, GRAY, GREEN, RED, RESET, YELLOW, bold, cyan, dim
|
||||
|
||||
log = get_logger(__name__)
|
||||
@@ -599,6 +604,45 @@ _REASONING_BEARING_BLOCK_TYPES: frozenset[str] = frozenset(
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backend boundary exception classification
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# ``_record_fatal_error`` routes a fatal exception through
|
||||
# ``_format_backend_error`` (defined on :class:`ChatSession`) which
|
||||
# matches the exception's class name against the sets below. Matching
|
||||
# by name keeps the helper free of httpx / openai / anthropic imports —
|
||||
# the three SDKs each define their own subclasses, but the names
|
||||
# (``ReadTimeout``, ``APITimeoutError``, …) are stable across them and
|
||||
# the OpenAI and Anthropic SDKs use the same names.
|
||||
#
|
||||
# Lifted to module scope (rather than ``ClassVar`` constants on
|
||||
# ``ChatSession``) so the test suite can bind ``_format_backend_error``
|
||||
# to lightweight stubs that don't subclass the session — keeping the
|
||||
# helper testable without the full ChatSession construction surface
|
||||
# (storage init, prompt composition, registry plumbing).
|
||||
|
||||
_BACKEND_TIMEOUT_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"ReadTimeout", "WriteTimeout", "PoolTimeout", "APITimeoutError"}
|
||||
)
|
||||
_BACKEND_CONNECT_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"ConnectTimeout", "ConnectError", "APIConnectionError"}
|
||||
)
|
||||
_BACKEND_NOT_FOUND_EXC_NAMES: frozenset[str] = frozenset({"NotFoundError"})
|
||||
_BACKEND_AUTH_EXC_NAMES: frozenset[str] = frozenset(
|
||||
{"AuthenticationError", "PermissionDeniedError"}
|
||||
)
|
||||
_BACKEND_RATE_LIMIT_EXC_NAMES: frozenset[str] = frozenset({"RateLimitError"})
|
||||
|
||||
_BACKEND_KNOWN_EXC_NAMES: frozenset[str] = (
|
||||
_BACKEND_TIMEOUT_EXC_NAMES
|
||||
| _BACKEND_CONNECT_EXC_NAMES
|
||||
| _BACKEND_NOT_FOUND_EXC_NAMES
|
||||
| _BACKEND_AUTH_EXC_NAMES
|
||||
| _BACKEND_RATE_LIMIT_EXC_NAMES
|
||||
)
|
||||
|
||||
|
||||
class SessionUI(Protocol):
|
||||
def on_turn_start(self) -> None: ...
|
||||
def on_turn_committed(self) -> None: ...
|
||||
@@ -812,6 +856,15 @@ class ChatSession:
|
||||
self._mcp_user_id: str | None = user_id or None
|
||||
self._username = username
|
||||
self._client_type = client_type
|
||||
# Whether the user is online to complete an in-flight OAuth
|
||||
# consent redirect. WEB and CLI users are; CHAT (Discord /
|
||||
# Slack) and SCHEDULED (autonomous runs) are not — their
|
||||
# consent-required errors must be persisted to
|
||||
# ``mcp_pending_consent`` by the pool dispatchers for later
|
||||
# surfacing on the dashboard badge, rather than relying on the
|
||||
# in-flight SSE rendering path that Phase 8 ships for
|
||||
# interactive surfaces.
|
||||
self._is_interactive_for_consent: bool = client_type in INTERACTIVE_CONSENT_CLIENT_TYPES
|
||||
self._config_store = config_store
|
||||
# Initialize rule registry for configurable judge rules
|
||||
self._rule_registry = None
|
||||
@@ -1477,9 +1530,13 @@ class ChatSession:
|
||||
"""
|
||||
if self._registry is None:
|
||||
return
|
||||
aliases = sorted(self._registry.list_aliases())
|
||||
if not aliases:
|
||||
return
|
||||
# Hide ``default`` from the alias list — the LLM reads the English
|
||||
# word and picks it explicitly, which routes to whichever model
|
||||
# carries that alias rather than the operator-configured per-role
|
||||
# default (plan_alias / task_alias). Omitting ``model=`` already
|
||||
# selects the per-role default; offering the literal name as an
|
||||
# alternative invites the bypass.
|
||||
aliases = sorted(a for a in self._registry.list_aliases() if a != "default")
|
||||
aliases_str = ", ".join(f"`{a}`" for a in aliases)
|
||||
|
||||
new_tools: list[dict[str, Any]] = []
|
||||
@@ -1493,11 +1550,22 @@ class ChatSession:
|
||||
new_tool = copy.deepcopy(tool)
|
||||
props = new_tool.get("function", {}).get("parameters", {}).get("properties", {})
|
||||
if "model" in props:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
f"Omit to use the operator-configured {kind}. "
|
||||
f"Available aliases: {aliases_str}."
|
||||
)
|
||||
# Always rewrite — a reload that filters down to no
|
||||
# alternatives (only ``default`` remains in the registry)
|
||||
# must clear any stale alias names left over from a prior
|
||||
# render, not return early and leave them in place.
|
||||
if aliases:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
f"Omit to use the operator-configured {kind}. "
|
||||
f"Available aliases: {aliases_str}."
|
||||
)
|
||||
else:
|
||||
props["model"]["description"] = (
|
||||
f"Optional model alias to run this {name} on. "
|
||||
"Omit to use the current session model. "
|
||||
"(No alternative aliases configured in this session.)"
|
||||
)
|
||||
new_tools.append(new_tool)
|
||||
self._tools = new_tools
|
||||
|
||||
@@ -2536,10 +2604,20 @@ class ChatSession:
|
||||
whose ``str()`` carries the credentials verbatim, and they'd
|
||||
otherwise land in (a) the dashboard via ``on_error`` and (b)
|
||||
the coord LLM's prompt via inspect/wait.
|
||||
|
||||
Known backend boundary exceptions (httpx read/connect timeouts,
|
||||
OpenAI/Anthropic SDK ``APITimeoutError`` / ``APIConnectionError``
|
||||
/ ``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``)
|
||||
get rewritten into operator-actionable text that includes the
|
||||
provider name, base URL, and model — the bare ``ReadTimeout:
|
||||
timed out`` shape produced by ``f"{type(exc).__name__}: {exc}"``
|
||||
leaves the user with no way to tell whether a model server hung,
|
||||
the URL is wrong, or the model isn't loaded. Unknown exceptions
|
||||
fall through to the default formatting unchanged.
|
||||
"""
|
||||
from turnstone.core.memory import persist_last_error, sanitize_error_text
|
||||
|
||||
raw = f"{type(exc).__name__}: {exc}"
|
||||
raw = self._format_backend_error(exc) or f"{type(exc).__name__}: {exc}"
|
||||
safe = sanitize_error_text(raw)
|
||||
try:
|
||||
self.ui.on_error(safe)
|
||||
@@ -2549,6 +2627,95 @@ class ChatSession:
|
||||
self._has_persisted_error = True
|
||||
self._emit_state("error")
|
||||
|
||||
def _format_backend_error(self, exc: BaseException) -> str | None:
|
||||
"""Return an enriched message for known backend boundary errors.
|
||||
|
||||
Returns ``None`` for exceptions outside the recognised set so the
|
||||
caller falls back to the bare ``f"{type(exc).__name__}: {exc}"``
|
||||
shape. Matching is by class name (see the
|
||||
``_BACKEND_*_EXC_NAMES`` sets above) so the same helper covers
|
||||
httpx ``ReadTimeout`` / ``ConnectError``, OpenAI SDK
|
||||
``APITimeoutError`` / ``APIConnectionError`` /
|
||||
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``,
|
||||
and the Anthropic SDK equivalents (which share names).
|
||||
|
||||
Bad input (a ``base_url`` accessor that raises, a missing
|
||||
``_provider``) silently degrades to a ``"?"`` placeholder rather
|
||||
than failing — this helper runs from the fatal-error path and
|
||||
must never itself raise. The returned text still goes through
|
||||
:func:`sanitize_error_text` in the caller, so credentials in the
|
||||
base URL are redacted before display / persist.
|
||||
"""
|
||||
name = type(exc).__name__
|
||||
if name not in _BACKEND_KNOWN_EXC_NAMES:
|
||||
return None
|
||||
|
||||
# Pull backend identity — every branch swallows so a bad
|
||||
# accessor on a partially-initialised session can't hide the
|
||||
# original exception behind a NoneType error.
|
||||
base_url = "?"
|
||||
try:
|
||||
raw_url = str(
|
||||
getattr(self.client, "base_url", None)
|
||||
or getattr(self.client, "_base_url", None)
|
||||
or "?"
|
||||
)
|
||||
base_url = raw_url.split("?")[0].rstrip("/")
|
||||
except Exception:
|
||||
log.debug("session.fatal.base_url_lookup_failed", exc_info=True)
|
||||
provider_label = "?"
|
||||
try:
|
||||
prov = self._provider
|
||||
provider_label = (
|
||||
getattr(prov, "provider_name", None) or type(prov).__name__ if prov else "?"
|
||||
)
|
||||
except Exception:
|
||||
log.debug("session.fatal.provider_lookup_failed", exc_info=True)
|
||||
model_label = self.model or self._model_alias or "?"
|
||||
# Original exception text (often empty for httpx.ReadTimeout —
|
||||
# the message is on the class name alone) — included as a
|
||||
# "raw=" tail so operators reading logs can still grep the
|
||||
# underlying SDK error.
|
||||
raw_msg = str(exc).strip()
|
||||
raw_tail = f" raw={raw_msg!r}" if raw_msg else ""
|
||||
|
||||
if name in _BACKEND_TIMEOUT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend timeout ({name}): no response from {provider_label} "
|
||||
f"at {base_url} for model={model_label}. "
|
||||
f"The model server may be wedged — check it's accepting completion requests."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_CONNECT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend unreachable ({name}): cannot reach {provider_label} "
|
||||
f"at {base_url} for model={model_label}. "
|
||||
f"Check the URL, that the server is running, and that this host can reach it."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_NOT_FOUND_EXC_NAMES:
|
||||
return (
|
||||
f"Backend reports model not loaded ({name}): {provider_label} "
|
||||
f"at {base_url} has no model named '{model_label}'. "
|
||||
f"Confirm the served model name matches the alias configuration "
|
||||
f"(GET /v1/models on the backend lists what it actually has)."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_AUTH_EXC_NAMES:
|
||||
return (
|
||||
f"Backend rejected credentials ({name}): {provider_label} "
|
||||
f"at {base_url} (model={model_label}). "
|
||||
f"Check the API key configured for this model alias."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
if name in _BACKEND_RATE_LIMIT_EXC_NAMES:
|
||||
return (
|
||||
f"Backend rate-limited ({name}): {provider_label} "
|
||||
f"at {base_url} (model={model_label})."
|
||||
f"{raw_tail}"
|
||||
)
|
||||
return None # unreachable — `name` is in _BACKEND_KNOWN_EXC_NAMES by construction
|
||||
|
||||
def _provider_extra_params(
|
||||
self,
|
||||
provider: LLMProvider | None = None,
|
||||
@@ -4439,7 +4606,18 @@ class ChatSession:
|
||||
elif name == "notify":
|
||||
it["func_args"] = {"message": (it.get("message") or "")[:200]}
|
||||
elif name == "task_agent":
|
||||
it["func_args"] = {"prompt": (it.get("prompt") or "")[:200]}
|
||||
# Pending items reach this point already shaped by
|
||||
# ``_prepare_task``, so ``it["skill"]`` is the resolved
|
||||
# skill_data dict (or ``None``), not the raw string the
|
||||
# LLM passed. Mirror the ``spawn_workstream`` projection
|
||||
# so heuristic ``arg_pattern`` rules can match on skill
|
||||
# name and the judge / audit row sees which persona was
|
||||
# selected.
|
||||
skill_dict = it.get("skill") or {}
|
||||
it["func_args"] = {
|
||||
"prompt": (it.get("prompt") or "")[:200],
|
||||
"skill": skill_dict.get("name", "") if isinstance(skill_dict, dict) else "",
|
||||
}
|
||||
elif name == "plan_agent":
|
||||
it["func_args"] = {"goal": (it.get("prompt") or "")[:200]}
|
||||
# Coordinator tool args — only the ``needs_approval=True`` set
|
||||
@@ -6013,9 +6191,39 @@ class ChatSession:
|
||||
alias = str(raw).strip()
|
||||
if not alias:
|
||||
return None, None
|
||||
# ``default`` is operator-only — the alias either back-compat-shims
|
||||
# a single-CLI-model registry or aliases a hand-named DB row, and
|
||||
# in both cases an LLM that explicitly routes here bypasses the
|
||||
# operator-configured ``plan_alias`` / ``task_alias`` per-role
|
||||
# default. Symmetric with the description filter at
|
||||
# ``_render_agent_tool_descriptions`` — closes the loophole where
|
||||
# an LLM that learned the alias name out-of-band (training data,
|
||||
# prior turn, prompt injection) can re-issue it directly.
|
||||
if alias == "default":
|
||||
return None, {
|
||||
"call_id": call_id,
|
||||
"func_name": func_name,
|
||||
"header": f"\u2717 {func_name}: 'default' is not selectable",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
"Error: 'default' is not a selectable model alias for "
|
||||
f"{func_name}. Omit `model=` to use the operator-configured "
|
||||
"per-role default."
|
||||
),
|
||||
}
|
||||
if self._registry is None or not self._registry.has_alias(alias):
|
||||
available = sorted(self._registry.list_aliases()) if self._registry is not None else []
|
||||
available_str = ", ".join(available) if available else "(no registry configured)"
|
||||
# ``default`` excluded from the retry list so an LLM probing
|
||||
# with a bogus alias can't enumerate it back from the error.
|
||||
if self._registry is None:
|
||||
available_str = "(no registry configured)"
|
||||
else:
|
||||
available = sorted(a for a in self._registry.list_aliases() if a != "default")
|
||||
available_str = (
|
||||
", ".join(available)
|
||||
if available
|
||||
else "(no alternative aliases configured — omit `model=`)"
|
||||
)
|
||||
return None, {
|
||||
"call_id": call_id,
|
||||
"func_name": func_name,
|
||||
@@ -6041,17 +6249,77 @@ class ChatSession:
|
||||
model_override, err = self._validate_agent_model_override(call_id, "task_agent", args)
|
||||
if err is not None:
|
||||
return err
|
||||
skill_arg = (args.get("skill") or "").strip()
|
||||
skill_data: dict[str, Any] | None = None
|
||||
if skill_arg:
|
||||
skill_data = get_skill_by_name(skill_arg)
|
||||
if skill_data is None:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": f"\u2717 task_agent: unknown skill '{skill_arg}'",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
f"Error: unknown skill '{skill_arg}'. "
|
||||
"Use skill(action='search') to find available names."
|
||||
),
|
||||
}
|
||||
# ``enabled=False`` is an admin's quarantine flag \u2014 mirror the
|
||||
# gate that ``_exec_skill(action='load')`` and skill-search
|
||||
# apply so task_agent can't sidestep it. Distinct from the
|
||||
# not-found case so the LLM's recovery path can tell them apart.
|
||||
if not skill_data.get("enabled", True):
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": f"\u2717 task_agent: skill '{skill_arg}' is disabled",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
f"Error: skill '{skill_arg}' is disabled and cannot be used. "
|
||||
"Use skill(action='search') to find available names."
|
||||
),
|
||||
}
|
||||
# ``get_skill_by_name`` returns the full prompt_templates row
|
||||
# (~30 columns including ``scan_report``, ``installed_by``,
|
||||
# ``source_url``, etc.). Project to the minimal field set
|
||||
# that ``_exec_task`` and ``_evaluate_intent`` actually read,
|
||||
# so the approval item doesn't drag governance metadata
|
||||
# through any future audit serializer.
|
||||
skill_data = {
|
||||
"name": skill_data["name"],
|
||||
"content": skill_data["content"],
|
||||
"risk_level": skill_data.get("risk_level", ""),
|
||||
}
|
||||
preview_text = prompt[:300] + ("..." if len(prompt) > 300 else "")
|
||||
header = "\u2699 task_agent (autonomous agent"
|
||||
if skill_data:
|
||||
header += f", skill: {skill_data['name']}"
|
||||
# Surface high/critical risk at approval time so the operator
|
||||
# sees the same signal ``_load_skills`` emits for session-level
|
||||
# skills (session.py:1336). Log mirrors that path's structured
|
||||
# event for forensic continuity.
|
||||
risk_tier = skill_data.get("risk_level", "")
|
||||
if risk_tier in ("high", "critical"):
|
||||
header += f", risk: {risk_tier}"
|
||||
log.warning(
|
||||
"task_agent.high_risk_skill",
|
||||
skill=skill_data["name"],
|
||||
risk_level=risk_tier,
|
||||
)
|
||||
header += ")"
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task_agent",
|
||||
"header": "\u2699 task_agent (autonomous agent)",
|
||||
"header": header,
|
||||
"preview": f" {preview_text}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "task_agent",
|
||||
"execute": self._exec_task,
|
||||
"prompt": prompt,
|
||||
"model_override": model_override,
|
||||
"skill": skill_data,
|
||||
}
|
||||
|
||||
def _prepare_plan(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
|
||||
@@ -8341,6 +8609,7 @@ class ChatSession:
|
||||
args,
|
||||
user_id=self._mcp_user_id,
|
||||
timeout=self.tool_timeout,
|
||||
is_interactive_for_consent=self._is_interactive_for_consent,
|
||||
)
|
||||
except TimeoutError:
|
||||
output = f"MCP tool timed out after {self.tool_timeout}s"
|
||||
@@ -8423,7 +8692,10 @@ class ChatSession:
|
||||
# 401 / 403 / consent-required handling. Otherwise the
|
||||
# static path runs byte-identical (invariant 1).
|
||||
output = self._mcp_client.read_resource_sync(
|
||||
uri, user_id=self._mcp_user_id, timeout=self.tool_timeout
|
||||
uri,
|
||||
user_id=self._mcp_user_id,
|
||||
timeout=self.tool_timeout,
|
||||
is_interactive_for_consent=self._is_interactive_for_consent,
|
||||
)
|
||||
except TimeoutError:
|
||||
output = f"MCP resource read timed out after {self.tool_timeout}s"
|
||||
@@ -8520,6 +8792,7 @@ class ChatSession:
|
||||
arguments or None,
|
||||
user_id=self._mcp_user_id,
|
||||
timeout=self.tool_timeout,
|
||||
is_interactive_for_consent=self._is_interactive_for_consent,
|
||||
)
|
||||
output = "\n\n".join(f"[{m['role']}]: {m['content']}" for m in messages)
|
||||
except TimeoutError:
|
||||
@@ -9237,35 +9510,68 @@ class ChatSession:
|
||||
self.ui.on_info(f"[{label} done] {len(content)} chars")
|
||||
return content
|
||||
|
||||
_TASK_DEFAULT_IDENTITY = (
|
||||
"# Task Agent\n\n"
|
||||
"You are an autonomous task agent with full tool access. "
|
||||
"You can use bash, read_file, write_file, edit_file, search, "
|
||||
"math, web_fetch, and web_search."
|
||||
)
|
||||
# Operating guidance always applies — these are sub-agent semantics
|
||||
# (one-shot, tool-use over narration, no follow-up questions) that a
|
||||
# persona skill should layer on top of, not replace.
|
||||
_TASK_OPERATING_GUIDANCE = (
|
||||
"1. **Follow through on actions:** Do not describe changes — "
|
||||
"use the tools to make them. After read_file, call edit_file "
|
||||
"or write_file.\n\n"
|
||||
"2. **Tool selection:**\n"
|
||||
" - Use read_file before edit_file on existing files.\n"
|
||||
" - Use write_file for new files (not bash).\n"
|
||||
" - Use bash for shell commands (git, python, tests).\n"
|
||||
" - Use search to find code across files.\n\n"
|
||||
"3. **Complete the task fully.** Do not ask follow-up "
|
||||
"questions — execute the work as described in the prompt."
|
||||
)
|
||||
|
||||
def _exec_task(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Delegate to a general-purpose autonomous sub-agent."""
|
||||
call_id, prompt = item["call_id"], item["prompt"]
|
||||
task_instruction = {
|
||||
"role": "system",
|
||||
"content": (
|
||||
"# Task Agent\n\n"
|
||||
"You are an autonomous task agent with full tool access. "
|
||||
"You can use bash, read_file, write_file, edit_file, search, "
|
||||
"math, web_fetch, and web_search.\n\n"
|
||||
"1. **Follow through on actions:** Do not describe changes — "
|
||||
"use the tools to make them. After read_file, call edit_file "
|
||||
"or write_file.\n\n"
|
||||
"2. **Tool selection:**\n"
|
||||
" - Use read_file before edit_file on existing files.\n"
|
||||
" - Use write_file for new files (not bash).\n"
|
||||
" - Use bash for shell commands (git, python, tests).\n"
|
||||
" - Use search to find code across files.\n\n"
|
||||
"3. **Complete the task fully.** Do not ask follow-up "
|
||||
"questions — execute the work as described in the prompt."
|
||||
),
|
||||
}
|
||||
skill_data = item.get("skill")
|
||||
if skill_data:
|
||||
# Structured forensic record naming the skill the LLM ran
|
||||
# under. The approval row captures the choice at consent
|
||||
# time; this log captures it at exec time so post-incident
|
||||
# search ("which sessions ran skill X?") doesn't have to
|
||||
# cross-walk approval and exec tables.
|
||||
log.info(
|
||||
"task_agent.skill_invoked",
|
||||
skill=skill_data["name"],
|
||||
risk_level=skill_data.get("risk_level", ""),
|
||||
ws_id=self._ws_id,
|
||||
)
|
||||
context = {
|
||||
"model": self.model,
|
||||
"ws_id": self._ws_id,
|
||||
"node_id": self._node_id or "",
|
||||
}
|
||||
persona = _render_template(skill_data["content"], context)
|
||||
if len(persona) > _MAX_SKILL_CONTENT:
|
||||
log.warning(
|
||||
"skill_content.truncated",
|
||||
length=len(persona),
|
||||
agent="task",
|
||||
skill=skill_data.get("name", ""),
|
||||
)
|
||||
persona = persona[:_MAX_SKILL_CONTENT]
|
||||
else:
|
||||
persona = self._TASK_DEFAULT_IDENTITY
|
||||
identity = persona + "\n\n" + self._TASK_OPERATING_GUIDANCE
|
||||
# Task agent gets the base system prompt (tool patterns) merged
|
||||
# with its own identity in a single system message. No conversation
|
||||
# history — it's an autonomous sub-agent. Merged to avoid
|
||||
# multi-system-message errors on models like Qwen.
|
||||
base = self._agent_system_messages[0]["content"] if self._agent_system_messages else ""
|
||||
agent_messages = [
|
||||
{"role": "system", "content": base + "\n\n" + task_instruction["content"]},
|
||||
{"role": "system", "content": base + "\n\n" + identity},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
try:
|
||||
@@ -9500,6 +9806,56 @@ class ChatSession:
|
||||
|
||||
return content
|
||||
|
||||
def _audit_memory_event(
|
||||
self,
|
||||
action: str,
|
||||
memory_id: str,
|
||||
*,
|
||||
name: str,
|
||||
scope: str,
|
||||
scope_id: str,
|
||||
mem_type: str,
|
||||
) -> None:
|
||||
"""Emit an audit row for a mutating memory tool action.
|
||||
|
||||
Closes the audit gap that previously masked out-of-band deletes
|
||||
when investigating "save reports success but get returns
|
||||
not-found": only the admin-console DELETE route emitted
|
||||
``memory.delete`` rows, so a long-running session whose row was
|
||||
deleted by the console UI couldn't tell from logs alone whether
|
||||
the row had been deleted, never persisted, or was never visible.
|
||||
|
||||
``scope_id`` is the empty string for ``scope='global'`` and the
|
||||
actor's user_id / ws_id for the other scopes — written as-is so
|
||||
forensic queries can filter on it. ``ws_id`` always rides in
|
||||
the detail (``self._ws_id`` is unconditional on ChatSession).
|
||||
|
||||
Best-effort: failures log at debug and swallow so an audit hiccup
|
||||
never breaks the tool call itself. Reads (get/search/list) are
|
||||
intentionally not audited — they'd multiply audit volume
|
||||
without forensic value.
|
||||
"""
|
||||
try:
|
||||
from turnstone.core.audit import record_audit
|
||||
|
||||
detail: dict[str, Any] = {
|
||||
"name": name,
|
||||
"scope": scope,
|
||||
"scope_id": scope_id,
|
||||
"type": mem_type,
|
||||
"ws_id": self._ws_id,
|
||||
}
|
||||
record_audit(
|
||||
get_storage(),
|
||||
self._user_id,
|
||||
action,
|
||||
"memory",
|
||||
memory_id,
|
||||
detail,
|
||||
)
|
||||
except Exception:
|
||||
log.debug("memory.audit_failed action=%s name=%s", action, name, exc_info=True)
|
||||
|
||||
def _exec_memory(self, item: dict[str, Any]) -> tuple[str, str]:
|
||||
"""Execute a memory tool action."""
|
||||
call_id = item["call_id"]
|
||||
@@ -9521,6 +9877,14 @@ class ChatSession:
|
||||
return call_id, msg
|
||||
self._invalidate_memory_cache()
|
||||
self._init_system_messages()
|
||||
self._audit_memory_event(
|
||||
"memory.update" if old is not None else "memory.save",
|
||||
memory_id,
|
||||
name=item["name"],
|
||||
scope=item["scope"],
|
||||
scope_id=item["scope_id"],
|
||||
mem_type=item["mem_type"],
|
||||
)
|
||||
if old is not None:
|
||||
msg = f"Updated memory '{item['name']}' (type={item['mem_type']}, scope={item['scope']})"
|
||||
else:
|
||||
@@ -9553,20 +9917,36 @@ class ChatSession:
|
||||
|
||||
if action == "delete":
|
||||
scopes = item["scopes_to_try"]
|
||||
deleted = False
|
||||
deleted: dict[str, str] | None = None
|
||||
deleted_scope = ""
|
||||
deleted_scope_id = ""
|
||||
# Look up first so the audit row can record the deleted
|
||||
# memory_id + type (delete-by-name returns only a bool).
|
||||
# Falling back through the scope walk keeps the current
|
||||
# narrowest-first IC semantics; coord sessions only see
|
||||
# ``coordinator`` here.
|
||||
for scope, scope_id in scopes:
|
||||
if delete_structured_memory(item["name"], scope, scope_id):
|
||||
deleted = True
|
||||
existing = get_structured_memory_by_name(item["name"], scope, scope_id)
|
||||
if existing and delete_structured_memory_by_id(existing["memory_id"]):
|
||||
deleted = existing
|
||||
deleted_scope = scope
|
||||
deleted_scope_id = scope_id
|
||||
break
|
||||
if not deleted:
|
||||
if deleted is None:
|
||||
tried = ", ".join(s for s, _ in scopes)
|
||||
msg = f"Error: memory '{item['name']}' not found (searched scopes: {tried})"
|
||||
self._report_tool_result(call_id, "memory", msg, is_error=True)
|
||||
else:
|
||||
self._invalidate_memory_cache()
|
||||
self._init_system_messages()
|
||||
self._audit_memory_event(
|
||||
"memory.delete",
|
||||
deleted["memory_id"],
|
||||
name=item["name"],
|
||||
scope=deleted_scope,
|
||||
scope_id=deleted_scope_id,
|
||||
mem_type=deleted.get("type", ""),
|
||||
)
|
||||
msg = f"Deleted memory '{item['name']}' (scope={deleted_scope})"
|
||||
self._report_tool_result(call_id, "memory", msg)
|
||||
return call_id, msg
|
||||
|
||||
@@ -22,6 +22,7 @@ from turnstone.core.workstream import Workstream, WorkstreamKind, WorkstreamStat
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.child_event_bus import ChildEventBus
|
||||
from turnstone.core.session import ChatSession, SessionUI
|
||||
from turnstone.core.state_writer import StateWriter
|
||||
from turnstone.core.storage._protocol import StorageBackend
|
||||
@@ -252,6 +253,19 @@ class SessionManager:
|
||||
def kind(self) -> WorkstreamKind:
|
||||
return self._adapter.kind
|
||||
|
||||
@property
|
||||
def child_event_bus(self) -> ChildEventBus | None:
|
||||
"""Delegate to the adapter's per-workstream wakeup bus.
|
||||
|
||||
Returns ``None`` for adapters that don't host one (today only the
|
||||
coord adapter does; interactive's child surface is degenerate
|
||||
and has nothing to wait on yet). Manager-level property gives
|
||||
adapter-agnostic callers (tests, future cross-kind tools) a
|
||||
stable lookup that doesn't depend on knowing which adapter is
|
||||
attached.
|
||||
"""
|
||||
return getattr(self._adapter, "child_event_bus", None)
|
||||
|
||||
@property
|
||||
def _service_type(self) -> str | None:
|
||||
"""``services.service_type`` this manager's hosting process registers
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Cross-process notification primitive shared by all storage backends.
|
||||
|
||||
Provides a uniform ``notify`` / ``listen`` shape over PostgreSQL's
|
||||
``LISTEN`` / ``NOTIFY`` and a SQLite synthetic-sweep fallback.
|
||||
|
||||
Consumers subscribe to one or more channels, drain a
|
||||
:class:`NotifyStream` via :meth:`NotifyStream.poll`, and reconcile by
|
||||
re-reading the relevant rows on every wake-up. Payloads are signal-only
|
||||
(<= 8 KiB on Postgres) — full event content is delivered by SSE or
|
||||
in-process callbacks elsewhere; this primitive is the "go re-read these
|
||||
rows" wake-up channel, nothing more.
|
||||
|
||||
The PostgreSQL implementation requires a session-mode connection
|
||||
(``pgbouncer`` in transaction mode is incompatible with LISTEN). See
|
||||
the ``listen`` docs on each backend for the deployment-config detail.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Protocol
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Notify:
|
||||
"""One notification draining out of a :class:`NotifyStream`."""
|
||||
|
||||
channel: str
|
||||
payload: str
|
||||
pid: int
|
||||
|
||||
|
||||
class NotifyConnectionError(Exception):
|
||||
"""Raised when a :class:`NotifyStream`'s underlying connection drops.
|
||||
|
||||
Consumers handle this by closing the stream, reconciling against the
|
||||
relevant table (re-reading whatever rows the channel describes), and
|
||||
reopening with a fresh :meth:`StorageBackend.listen` call.
|
||||
"""
|
||||
|
||||
|
||||
class NotifyStream(Protocol):
|
||||
"""Bounded-blocking pull interface for cross-process notifications.
|
||||
|
||||
Returned by :meth:`StorageBackend.listen` as a context manager; the
|
||||
consumer drains via :meth:`poll` in a loop, typically with a short
|
||||
timeout so the loop can also observe a shutdown flag.
|
||||
"""
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
"""Wait up to ``timeout`` seconds for notifications.
|
||||
|
||||
Returns the list of notifications received during the wait
|
||||
(possibly empty on timeout). Raises :class:`NotifyConnectionError`
|
||||
if the underlying connection was dropped — the caller reconciles
|
||||
and re-listens.
|
||||
"""
|
||||
...
|
||||
|
||||
def close(self) -> None:
|
||||
"""Stop the stream; subsequent :meth:`poll` calls return ``[]``."""
|
||||
...
|
||||
@@ -3,19 +3,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterable, Iterator
|
||||
|
||||
from turnstone.core.storage._notify import Notify, NotifyStream
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._protocol import (
|
||||
MCPOAuthPendingState,
|
||||
MCPPendingConsentRow,
|
||||
MCPUserToken,
|
||||
MCPUserTokenMetadataRow,
|
||||
OIDCIdentity,
|
||||
@@ -30,6 +34,7 @@ from turnstone.core.storage._schema import (
|
||||
heuristic_rules,
|
||||
intent_verdicts,
|
||||
mcp_oauth_pending,
|
||||
mcp_pending_consent,
|
||||
mcp_servers,
|
||||
mcp_user_tokens,
|
||||
metadata,
|
||||
@@ -120,11 +125,105 @@ def _escape_ilike(s: str) -> str:
|
||||
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
|
||||
|
||||
def _resolve_pg_listen_url(override: str, sqlalchemy_url: str) -> str:
|
||||
"""Resolve the URL used by the dedicated LISTEN connection.
|
||||
|
||||
Precedence:
|
||||
|
||||
1. ``override`` — explicitly passed via :class:`PostgreSQLBackend`
|
||||
constructor, typically wired from ``[database] listen_url`` in
|
||||
``config.toml`` or ``--db-listen-url`` on the CLI.
|
||||
2. ``TURNSTONE_DB_LISTEN_URL`` environment variable.
|
||||
3. The engine's main DB URL.
|
||||
|
||||
The override is for deployments where the regular ``TURNSTONE_DB_URL``
|
||||
points at a ``pgbouncer`` running in transaction pooling mode (the
|
||||
project default per ``docs/pgbouncer.md``). LISTEN holds session
|
||||
state and is incompatible with transaction pooling; the dispatcher
|
||||
needs to bypass pgbouncer for that single connection. When neither
|
||||
override is set and ``TURNSTONE_DB_URL`` already points at Postgres
|
||||
directly (no pooler in between), the fallback uses the engine URL
|
||||
as-is.
|
||||
|
||||
The SQLAlchemy ``+psycopg`` driver suffix is stripped so the URL is
|
||||
consumable by ``psycopg.connect`` directly.
|
||||
"""
|
||||
raw = override.strip() or os.environ.get("TURNSTONE_DB_LISTEN_URL", "").strip()
|
||||
raw = raw or sqlalchemy_url
|
||||
return raw.replace("postgresql+psycopg://", "postgresql://", 1)
|
||||
|
||||
|
||||
class _PostgreSQLNotifyStream:
|
||||
"""PostgreSQL ``listen`` stream — drains ``conn.notifies`` per poll.
|
||||
|
||||
Owns a dedicated psycopg autocommit connection. Each :meth:`poll`
|
||||
waits up to ``timeout`` seconds for notifications and returns them
|
||||
as a list — empty on timeout, raises :class:`NotifyConnectionError`
|
||||
on connection loss (caller reconciles + re-listens).
|
||||
|
||||
Closing the stream from another thread is the supported abort path:
|
||||
``close`` calls ``conn.close()``, which causes the in-flight
|
||||
:meth:`poll` to wake (the next call returns ``[]`` because
|
||||
``_closed`` is set).
|
||||
"""
|
||||
|
||||
def __init__(self, conn: Any, channels: list[str]) -> None:
|
||||
self._conn = conn
|
||||
self._channels = list(channels)
|
||||
self._closed = False
|
||||
self._close_lock = threading.Lock()
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
from turnstone.core.storage._notify import Notify, NotifyConnectionError
|
||||
|
||||
if self._closed:
|
||||
return []
|
||||
out: list[Notify] = []
|
||||
try:
|
||||
# psycopg3 generator yields whatever's available within the
|
||||
# window, then stops — bounded blocking semantics. Per-call
|
||||
# generator (not a long-lived one) so close() can abort by
|
||||
# closing the connection without leaving a half-consumed
|
||||
# generator behind.
|
||||
for n in self._conn.notifies(timeout=max(0.0, timeout)):
|
||||
out.append(Notify(channel=n.channel, payload=n.payload, pid=n.pid))
|
||||
except Exception as exc:
|
||||
if self._closed:
|
||||
# Graceful close-from-another-thread surfaced as an
|
||||
# operational error inside notifies() — swallow it,
|
||||
# let the caller observe close via the next poll
|
||||
# returning ``[]``.
|
||||
return out
|
||||
raise NotifyConnectionError(str(exc)) from exc
|
||||
return out
|
||||
|
||||
def close(self) -> None:
|
||||
with self._close_lock:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
conn = self._conn
|
||||
# Best-effort UNLISTEN + close. An already-broken connection
|
||||
# raises here; the consumer's reconciliation logic will catch
|
||||
# the underlying ``NotifyConnectionError`` on the next poll if
|
||||
# any waiter is still blocked.
|
||||
with contextlib.suppress(Exception):
|
||||
conn.execute("UNLISTEN *")
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
|
||||
|
||||
class PostgreSQLBackend:
|
||||
"""PostgreSQL implementation of the StorageBackend protocol."""
|
||||
|
||||
def __init__(
|
||||
self, url: str, pool_size: int = 2, max_overflow: int = 3, *, create_tables: bool = True
|
||||
self,
|
||||
url: str,
|
||||
pool_size: int = 2,
|
||||
max_overflow: int = 3,
|
||||
*,
|
||||
create_tables: bool = True,
|
||||
listen_url: str = "",
|
||||
) -> None:
|
||||
self._engine = sa.create_engine(
|
||||
url,
|
||||
@@ -134,6 +233,12 @@ class PostgreSQLBackend:
|
||||
)
|
||||
self._db_unavailable = False
|
||||
self._db_unavailable_lock = threading.Lock()
|
||||
# Operator override for the dedicated LISTEN connection's URL.
|
||||
# Empty string means "fall back through env var, then the main
|
||||
# engine URL" — see :func:`_resolve_pg_listen_url` for the full
|
||||
# precedence rules. Threaded through ``init_storage`` from
|
||||
# ``config.toml [database] listen_url`` / ``--db-listen-url``.
|
||||
self._listen_url_override = listen_url
|
||||
if create_tables:
|
||||
metadata.create_all(self._engine)
|
||||
|
||||
@@ -1863,6 +1968,61 @@ class PostgreSQLBackend:
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""Broadcast a wake-up via ``pg_notify`` on a pooled connection.
|
||||
|
||||
``channel`` and ``payload`` are bound as parameters so this is
|
||||
safe to call with operator-supplied strings without quoting
|
||||
gymnastics. Postgres caps the payload at 8 KiB — keep payloads
|
||||
signal-only (a JSON id list, an op name) and let consumers
|
||||
re-read the underlying rows on wake-up.
|
||||
"""
|
||||
with self._conn() as conn:
|
||||
conn.execute(
|
||||
sa.text("SELECT pg_notify(:channel, :payload)"),
|
||||
{"channel": channel, "payload": payload},
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
@contextlib.contextmanager
|
||||
def listen(self, channels: Iterable[str]) -> Iterator[NotifyStream]:
|
||||
"""Subscribe to channels on a dedicated session-mode connection.
|
||||
|
||||
Opens a fresh ``psycopg`` connection in autocommit mode (the
|
||||
SQLAlchemy pool is incompatible with LISTEN — it recycles
|
||||
connections back into a pool that may be transaction-pooled by
|
||||
pgbouncer). Channel names are interpolated via
|
||||
``psycopg.sql.Identifier`` so caller-supplied channel strings
|
||||
can't inject SQL.
|
||||
|
||||
``TURNSTONE_DB_LISTEN_URL`` overrides the engine URL — see
|
||||
:func:`_resolve_pg_listen_url` for the bypass-URL rationale.
|
||||
|
||||
Yields a :class:`_PostgreSQLNotifyStream`; the connection is
|
||||
closed on context exit.
|
||||
"""
|
||||
import psycopg
|
||||
from psycopg import sql
|
||||
|
||||
ch_list = [str(c) for c in channels if c]
|
||||
sqlalchemy_url = self._engine.url.render_as_string(hide_password=False)
|
||||
listen_url = _resolve_pg_listen_url(self._listen_url_override, sqlalchemy_url)
|
||||
conn = psycopg.connect(listen_url, autocommit=True)
|
||||
stream: _PostgreSQLNotifyStream | None = None
|
||||
try:
|
||||
for ch in ch_list:
|
||||
conn.execute(sql.SQL("LISTEN {}").format(sql.Identifier(ch)))
|
||||
stream = _PostgreSQLNotifyStream(conn, ch_list)
|
||||
yield stream
|
||||
finally:
|
||||
if stream is not None:
|
||||
stream.close()
|
||||
else:
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
@@ -4150,6 +4310,136 @@ class PostgreSQLBackend:
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- MCP pending-consent (Phase 9) ----------------------------------------
|
||||
|
||||
def upsert_mcp_pending_consent(
|
||||
self,
|
||||
user_id: str,
|
||||
server_name: str,
|
||||
error_code: str,
|
||||
scopes_required: str | None,
|
||||
last_ws_id: str | None,
|
||||
last_tool_call_id: str | None,
|
||||
now_iso: str,
|
||||
) -> None:
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
stmt = postgresql.insert(mcp_pending_consent).values(
|
||||
user_id=user_id,
|
||||
server_name=server_name,
|
||||
error_code=error_code,
|
||||
scopes_required=scopes_required,
|
||||
last_ws_id=last_ws_id,
|
||||
last_tool_call_id=last_tool_call_id,
|
||||
first_seen_at=now_iso,
|
||||
last_seen_at=now_iso,
|
||||
occurrence_count=1,
|
||||
)
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=["user_id", "server_name"],
|
||||
set_={
|
||||
"error_code": stmt.excluded.error_code,
|
||||
"scopes_required": stmt.excluded.scopes_required,
|
||||
"last_ws_id": stmt.excluded.last_ws_id,
|
||||
"last_tool_call_id": stmt.excluded.last_tool_call_id,
|
||||
"last_seen_at": stmt.excluded.last_seen_at,
|
||||
"occurrence_count": mcp_pending_consent.c.occurrence_count + 1,
|
||||
},
|
||||
)
|
||||
with self._conn() as conn:
|
||||
conn.execute(stmt)
|
||||
conn.commit()
|
||||
|
||||
def list_mcp_pending_consent_by_user(self, user_id: str) -> list[MCPPendingConsentRow]:
|
||||
with self._conn() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(mcp_pending_consent)
|
||||
.where(mcp_pending_consent.c.user_id == user_id)
|
||||
.order_by(mcp_pending_consent.c.last_seen_at.desc())
|
||||
).fetchall()
|
||||
out: list[MCPPendingConsentRow] = []
|
||||
for r in rows:
|
||||
m = r._mapping
|
||||
out.append(
|
||||
MCPPendingConsentRow(
|
||||
user_id=m["user_id"],
|
||||
server_name=m["server_name"],
|
||||
error_code=m["error_code"],
|
||||
scopes_required=m["scopes_required"],
|
||||
last_ws_id=m["last_ws_id"],
|
||||
last_tool_call_id=m["last_tool_call_id"],
|
||||
first_seen_at=m["first_seen_at"],
|
||||
last_seen_at=m["last_seen_at"],
|
||||
occurrence_count=m["occurrence_count"],
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
def delete_mcp_pending_consent(self, user_id: str, server_name: str) -> bool:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.delete(mcp_pending_consent).where(
|
||||
(mcp_pending_consent.c.user_id == user_id)
|
||||
& (mcp_pending_consent.c.server_name == server_name)
|
||||
)
|
||||
)
|
||||
conn.commit()
|
||||
return bool(result.rowcount)
|
||||
|
||||
def delete_all_mcp_pending_consent_by_user(self, user_id: str) -> int:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.delete(mcp_pending_consent).where(mcp_pending_consent.c.user_id == user_id)
|
||||
)
|
||||
conn.commit()
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
def count_mcp_consented_users_by_server(self, server_name: str) -> int:
|
||||
# ``expires_at IS NULL`` => non-expired (refresh-only tokens with no
|
||||
# advertised expiry). Compare lexically against ISO-8601 strings,
|
||||
# mirroring the convention in ``mcp_user_tokens.expires_at``.
|
||||
now_iso = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.select(sa.func.count(sa.distinct(mcp_user_tokens.c.user_id)))
|
||||
.where(mcp_user_tokens.c.server_name == server_name)
|
||||
.where(
|
||||
sa.or_(
|
||||
mcp_user_tokens.c.expires_at.is_(None),
|
||||
mcp_user_tokens.c.expires_at > now_iso,
|
||||
)
|
||||
)
|
||||
).scalar()
|
||||
return int(result or 0)
|
||||
|
||||
def count_mcp_consented_users_grouped_by_server(self) -> dict[str, int]:
|
||||
now_iso = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._conn() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(
|
||||
mcp_user_tokens.c.server_name,
|
||||
sa.func.count(sa.distinct(mcp_user_tokens.c.user_id)),
|
||||
)
|
||||
.where(
|
||||
sa.or_(
|
||||
mcp_user_tokens.c.expires_at.is_(None),
|
||||
mcp_user_tokens.c.expires_at > now_iso,
|
||||
)
|
||||
)
|
||||
.group_by(mcp_user_tokens.c.server_name)
|
||||
).fetchall()
|
||||
return {row[0]: int(row[1] or 0) for row in rows}
|
||||
|
||||
def any_oauth_user_mcp_servers(self) -> bool:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.select(sa.literal(1))
|
||||
.select_from(mcp_servers)
|
||||
.where(mcp_servers.c.auth_type == "oauth_user")
|
||||
.limit(1)
|
||||
).scalar()
|
||||
return result is not None
|
||||
|
||||
# -- Model definitions -----------------------------------------------------
|
||||
|
||||
def create_model_definition(
|
||||
|
||||
@@ -5,8 +5,10 @@ from __future__ import annotations
|
||||
from typing import TYPE_CHECKING, Any, Protocol, TypedDict, runtime_checkable
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterable
|
||||
from contextlib import AbstractContextManager
|
||||
|
||||
from turnstone.core.storage._notify import NotifyStream
|
||||
from turnstone.core.workstream import WorkstreamKind
|
||||
|
||||
|
||||
@@ -91,6 +93,27 @@ class MCPOAuthPendingState(TypedDict):
|
||||
created_at: str
|
||||
|
||||
|
||||
class MCPPendingConsentRow(TypedDict):
|
||||
"""Row shape for deferred-consent records.
|
||||
|
||||
Emitted by the pool dispatchers (Phase 5+) when a non-interactive
|
||||
run (scheduled / channel) hits ``mcp_consent_required`` or
|
||||
``mcp_insufficient_scope`` and the user can't be prompted in the
|
||||
moment. Composite PK ``(user_id, server_name)`` collapses repeat
|
||||
occurrences for the same server into one row.
|
||||
"""
|
||||
|
||||
user_id: str
|
||||
server_name: str
|
||||
error_code: str
|
||||
scopes_required: str | None
|
||||
last_ws_id: str | None
|
||||
last_tool_call_id: str | None
|
||||
first_seen_at: str
|
||||
last_seen_at: str
|
||||
occurrence_count: int
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class StorageBackend(Protocol):
|
||||
"""Protocol that every storage backend adapter must implement.
|
||||
@@ -1020,6 +1043,33 @@ class StorageBackend(Protocol):
|
||||
"""Remove a service registration. Returns True if existed."""
|
||||
...
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""Broadcast a wake-up on ``channel`` to any listening process.
|
||||
|
||||
Payloads are signal-only — a JSON-encoded string identifying
|
||||
which rows to re-read, capped well below Postgres's 8 KiB
|
||||
``NOTIFY`` payload limit. Full event content is NOT delivered
|
||||
this way; consumers reconcile by reading the relevant table on
|
||||
wake-up. Safe to call from any thread.
|
||||
"""
|
||||
...
|
||||
|
||||
def listen(self, channels: Iterable[str]) -> AbstractContextManager[NotifyStream]:
|
||||
"""Subscribe to one or more channels for cross-process wake-ups.
|
||||
|
||||
Returns a context manager wrapping a :class:`NotifyStream` the
|
||||
caller drains via :meth:`NotifyStream.poll`. PostgreSQL holds a
|
||||
dedicated session-mode connection for the lifetime of the
|
||||
context (incompatible with ``pgbouncer`` transaction pooling —
|
||||
see :class:`PostgreSQLBackend.listen` for the bypass-URL config).
|
||||
SQLite emits a synthetic-sweep wake on its own cadence (see
|
||||
``_SQLITE_NOTIFY_SWEEP_INTERVAL``) per subscribed channel so
|
||||
consumer code is identical across backends.
|
||||
"""
|
||||
...
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
@@ -1836,6 +1886,89 @@ class StorageBackend(Protocol):
|
||||
"""Bulk-delete expired pending MCP OAuth rows. Returns count deleted."""
|
||||
...
|
||||
|
||||
# -- MCP pending-consent (Phase 9; deferred-consent persistence) ----------
|
||||
|
||||
def upsert_mcp_pending_consent(
|
||||
self,
|
||||
user_id: str,
|
||||
server_name: str,
|
||||
error_code: str,
|
||||
scopes_required: str | None,
|
||||
last_ws_id: str | None,
|
||||
last_tool_call_id: str | None,
|
||||
now_iso: str,
|
||||
) -> None:
|
||||
"""Insert or refresh a deferred-consent record for ``(user, server)``.
|
||||
|
||||
On insert: ``first_seen_at = last_seen_at = now_iso``,
|
||||
``occurrence_count = 1``. On conflict (existing row for the
|
||||
same composite PK): rewrites ``error_code``, ``scopes_required``,
|
||||
``last_ws_id``, ``last_tool_call_id``, ``last_seen_at`` to the
|
||||
current values; bumps ``occurrence_count`` by 1. Preserves
|
||||
``first_seen_at`` so the dashboard can show how long the
|
||||
deferred-consent need has been pending.
|
||||
"""
|
||||
...
|
||||
|
||||
def list_mcp_pending_consent_by_user(self, user_id: str) -> list[MCPPendingConsentRow]:
|
||||
"""Return all deferred-consent records for ``user_id``.
|
||||
|
||||
Ordered by ``last_seen_at`` DESC. Empty list when the user has
|
||||
none. Used by the dashboard badge endpoint to render the
|
||||
servers-need-consent list.
|
||||
"""
|
||||
...
|
||||
|
||||
def delete_mcp_pending_consent(self, user_id: str, server_name: str) -> bool:
|
||||
"""Delete the pending-consent row for ``(user, server)``. Returns True if existed.
|
||||
|
||||
Called automatically by the OAuth callback handler when consent
|
||||
completes, and manually via the user-facing DELETE endpoint.
|
||||
"""
|
||||
...
|
||||
|
||||
def delete_all_mcp_pending_consent_by_user(self, user_id: str) -> int:
|
||||
"""Bulk-delete every pending-consent row for ``user_id``. Returns count.
|
||||
|
||||
Used by the manual "dismiss all" endpoint from the settings
|
||||
modal.
|
||||
"""
|
||||
...
|
||||
|
||||
def count_mcp_consented_users_by_server(self, server_name: str) -> int:
|
||||
"""Distinct-user count of non-expired tokens for ``server_name``.
|
||||
|
||||
``expires_at IS NULL`` is treated as non-expired (refresh-only
|
||||
tokens with no advertised expiry). Used by the admin status
|
||||
indicator to show "N users consented" per MCP server row.
|
||||
"""
|
||||
...
|
||||
|
||||
def count_mcp_consented_users_grouped_by_server(self) -> dict[str, int]:
|
||||
"""Bulk distinct-user count of non-expired tokens, grouped by server.
|
||||
|
||||
Single round-trip variant of
|
||||
:meth:`count_mcp_consented_users_by_server` for the admin list
|
||||
handler — replaces the N-call loop that issued one query per
|
||||
server with one ``GROUP BY`` query returning ``{server_name:
|
||||
count}`` for every server that has at least one non-expired
|
||||
token. Servers with zero consented users are absent from the
|
||||
result; callers should ``dict.get(name, 0)`` rather than
|
||||
indexing.
|
||||
"""
|
||||
...
|
||||
|
||||
def any_oauth_user_mcp_servers(self) -> bool:
|
||||
"""Install-level gate for OAuth-MCP features.
|
||||
|
||||
Returns True iff at least one ``mcp_servers`` row has
|
||||
``auth_type='oauth_user'``. Used to short-circuit the pending-
|
||||
consent badge endpoint to ``{pending: 0}`` on local-auth installs
|
||||
with no OAuth MCP servers, so those code paths exercise zero new
|
||||
storage queries.
|
||||
"""
|
||||
...
|
||||
|
||||
# -- Model definitions -----------------------------------------------------
|
||||
|
||||
def create_model_definition(
|
||||
|
||||
@@ -34,6 +34,7 @@ def init_storage(
|
||||
sslrootcert: str = "",
|
||||
sslcert: str = "",
|
||||
sslkey: str = "",
|
||||
listen_url: str = "",
|
||||
) -> StorageBackend:
|
||||
"""Initialize the storage backend singleton.
|
||||
|
||||
@@ -43,6 +44,13 @@ def init_storage(
|
||||
url: PostgreSQL connection URL (e.g. postgresql+psycopg://user:pass@host/db)
|
||||
pool_size: Connection pool size (PostgreSQL only)
|
||||
run_migrations: Whether to run Alembic migrations on init
|
||||
listen_url: Optional dedicated PostgreSQL URL for the dispatcher's
|
||||
``LISTEN`` connection. Required only when ``url`` points at a
|
||||
``pgbouncer`` running in transaction pooling mode (LISTEN
|
||||
holds session state and is incompatible with transaction
|
||||
pooling — see ``docs/pgbouncer.md``). Empty string means
|
||||
"fall back through ``TURNSTONE_DB_LISTEN_URL`` env var, then
|
||||
the main ``url``." Ignored on SQLite.
|
||||
"""
|
||||
global _storage
|
||||
|
||||
@@ -84,7 +92,12 @@ def init_storage(
|
||||
|
||||
sep = "&" if "?" in url else "?"
|
||||
url += sep + urlencode(ssl_params)
|
||||
_storage = PostgreSQLBackend(url, pool_size=pool_size, create_tables=create_tables)
|
||||
_storage = PostgreSQLBackend(
|
||||
url,
|
||||
pool_size=pool_size,
|
||||
create_tables=create_tables,
|
||||
listen_url=listen_url,
|
||||
)
|
||||
log.info("Storage initialized: PostgreSQL")
|
||||
|
||||
else:
|
||||
|
||||
@@ -251,6 +251,76 @@ services = sa.Table(
|
||||
|
||||
sa.Index("idx_services_type_heartbeat", services.c.service_type, services.c.last_heartbeat)
|
||||
|
||||
|
||||
# -- Postgres NOTIFY trigger on services -----------------------------------
|
||||
#
|
||||
# Producer side of the ``services`` channel that the console
|
||||
# ``NotifyDispatcher`` listens on for reactive node discovery. Fires on
|
||||
# real registry changes (INSERT, DELETE, UPDATE that changes ``url`` or
|
||||
# ``metadata``) and stays quiet on heartbeat-only UPDATEs so the 30s × N
|
||||
# nodes heartbeat tick doesn't flood the channel.
|
||||
#
|
||||
# Declared in the schema (not just in migration 053) so the ``after_create``
|
||||
# DDL event installs the trigger any time ``metadata.create_all`` builds
|
||||
# the ``services`` table — covering fresh dev databases and the test
|
||||
# fixture path (``run_migrations=False``). Migration 053 covers the
|
||||
# upgrade-on-existing-DB path; the two are mutually exclusive given the
|
||||
# ``create_tables = not run_migrations`` switch in ``init_storage``, so
|
||||
# neither double-installs. SQLite has no equivalent — the in-process
|
||||
# notify fan-out and synthetic-sweep covers the dev path consumer-side.
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_FN_NAME = "turnstone_notify_services"
|
||||
SERVICES_NOTIFY_TRIGGER_NAME = "services_notify"
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_FN_SQL = f"""
|
||||
CREATE OR REPLACE FUNCTION {SERVICES_NOTIFY_TRIGGER_FN_NAME}() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
-- Skip heartbeat-only UPDATEs: same url and metadata, only
|
||||
-- ``last_heartbeat`` changed. ``register_service`` is an UPSERT
|
||||
-- (on_conflict_do_update), so node restarts that change url or
|
||||
-- metadata MUST still fire — only no-op heartbeat ticks stay
|
||||
-- quiet. IS NOT DISTINCT FROM treats NULLs as equal so a row
|
||||
-- with NULL metadata before/after doesn't trip the diff.
|
||||
IF TG_OP = 'UPDATE'
|
||||
AND OLD.url IS NOT DISTINCT FROM NEW.url
|
||||
AND OLD.metadata IS NOT DISTINCT FROM NEW.metadata THEN
|
||||
RETURN NULL;
|
||||
END IF;
|
||||
|
||||
PERFORM pg_notify(
|
||||
'services',
|
||||
json_build_object(
|
||||
'service_type', COALESCE(NEW.service_type, OLD.service_type),
|
||||
'service_id', COALESCE(NEW.service_id, OLD.service_id),
|
||||
'op', TG_OP
|
||||
)::text
|
||||
);
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
|
||||
SERVICES_NOTIFY_TRIGGER_SQL = f"""
|
||||
CREATE TRIGGER {SERVICES_NOTIFY_TRIGGER_NAME}
|
||||
AFTER INSERT OR UPDATE OR DELETE ON services
|
||||
FOR EACH ROW EXECUTE FUNCTION {SERVICES_NOTIFY_TRIGGER_FN_NAME}();
|
||||
"""
|
||||
|
||||
sa.event.listen(
|
||||
services,
|
||||
"after_create",
|
||||
sa.DDL(SERVICES_NOTIFY_TRIGGER_FN_SQL).execute_if( # type: ignore[no-untyped-call]
|
||||
dialect="postgresql"
|
||||
),
|
||||
)
|
||||
sa.event.listen(
|
||||
services,
|
||||
"after_create",
|
||||
sa.DDL(SERVICES_NOTIFY_TRIGGER_SQL).execute_if( # type: ignore[no-untyped-call]
|
||||
dialect="postgresql"
|
||||
),
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node metadata (per-node key/value with source tracking)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -737,6 +807,15 @@ mcp_user_tokens = sa.Table(
|
||||
sa.Column("last_refreshed", sa.Text, nullable=True),
|
||||
sa.PrimaryKeyConstraint("user_id", "server_name"),
|
||||
)
|
||||
# Phase 9: covers the ``WHERE server_name = ? AND (expires_at IS NULL
|
||||
# OR expires_at > now)`` shape used by ``count_mcp_consented_users_*``
|
||||
# for the admin status pill. The composite PK can't satisfy filters
|
||||
# that don't lead with ``user_id``.
|
||||
sa.Index(
|
||||
"idx_mcp_user_tokens_server",
|
||||
mcp_user_tokens.c.server_name,
|
||||
mcp_user_tokens.c.expires_at,
|
||||
)
|
||||
|
||||
mcp_oauth_pending = sa.Table(
|
||||
"mcp_oauth_pending",
|
||||
@@ -751,6 +830,32 @@ mcp_oauth_pending = sa.Table(
|
||||
|
||||
sa.Index("idx_mcp_pending_created", mcp_oauth_pending.c.created_at)
|
||||
|
||||
# Per-(user, server) pending-consent state for non-interactive contexts.
|
||||
# Populated by the pool dispatchers when a scheduled / channel-driven run
|
||||
# hits ``mcp_consent_required`` or ``mcp_insufficient_scope`` and the user
|
||||
# can't be prompted in the moment. Read on dashboard load to render the
|
||||
# "N MCP servers need consent" badge. Cleared by the OAuth callback when
|
||||
# the matching ``(user, server)`` completes consent.
|
||||
#
|
||||
# Composite PK ``(user_id, server_name)`` collapses repeat occurrences for
|
||||
# the same server into one row; ``occurrence_count`` + ``last_*`` fields
|
||||
# carry recency metadata for the dashboard without inflating row count.
|
||||
mcp_pending_consent = sa.Table(
|
||||
"mcp_pending_consent",
|
||||
metadata,
|
||||
sa.Column("user_id", sa.Text, nullable=False),
|
||||
sa.Column("server_name", sa.Text, nullable=False),
|
||||
sa.Column("error_code", sa.Text, nullable=False),
|
||||
sa.Column("scopes_required", sa.Text, nullable=True),
|
||||
sa.Column("last_ws_id", sa.Text, nullable=True),
|
||||
sa.Column("last_tool_call_id", sa.Text, nullable=True),
|
||||
sa.Column("first_seen_at", sa.Text, nullable=False),
|
||||
sa.Column("last_seen_at", sa.Text, nullable=False),
|
||||
sa.Column("occurrence_count", sa.Integer, nullable=False, server_default="1"),
|
||||
sa.PrimaryKeyConstraint("user_id", "server_name"),
|
||||
)
|
||||
sa.Index("idx_mcp_pending_consent_user", mcp_pending_consent.c.user_id)
|
||||
|
||||
# ── TLS / ACME (lacme integration) ──────────────────────────────────────────
|
||||
|
||||
tls_account_keys = sa.Table(
|
||||
|
||||
@@ -3,18 +3,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from collections.abc import Iterable, Iterator
|
||||
|
||||
from turnstone.core.storage._notify import Notify, NotifyStream
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.storage._protocol import (
|
||||
MCPOAuthPendingState,
|
||||
MCPPendingConsentRow,
|
||||
MCPUserToken,
|
||||
MCPUserTokenMetadataRow,
|
||||
OIDCIdentity,
|
||||
@@ -29,6 +34,7 @@ from turnstone.core.storage._schema import (
|
||||
heuristic_rules,
|
||||
intent_verdicts,
|
||||
mcp_oauth_pending,
|
||||
mcp_pending_consent,
|
||||
mcp_servers,
|
||||
mcp_user_tokens,
|
||||
metadata,
|
||||
@@ -129,6 +135,90 @@ def _fts5_query(query: str) -> str:
|
||||
return " ".join(safe)
|
||||
|
||||
|
||||
# Synthetic-sweep cadence for the SQLite ``listen`` fallback. SQLite is
|
||||
# the dev-only path where reactive latency isn't load-bearing — a single
|
||||
# console process, no cross-process notify semantics to recover from.
|
||||
# 300 s sits comfortably above the existing per-consumer timers
|
||||
# (cluster collector's 60 s ``discovery_interval``, any future
|
||||
# ConfigStore/scheduler reload cadences) so the sweep is a true backstop
|
||||
# rather than a duplicate tick. Future consumers that need tighter
|
||||
# SQLite-mode reactive latency should pass a custom interval through
|
||||
# :meth:`SQLiteBackend.listen` rather than lowering this default.
|
||||
_SQLITE_NOTIFY_SWEEP_INTERVAL: float = 300.0
|
||||
|
||||
|
||||
class _SQLiteNotifyStream:
|
||||
"""SQLite ``listen`` stream — synthetic sweep + in-process fan-out.
|
||||
|
||||
Each poll either drains queued in-process notifies (delivered by a
|
||||
same-process :meth:`SQLiteBackend.notify` call) or emits one
|
||||
synthetic ``Notify(channel, payload="sweep", pid=0)`` per subscribed
|
||||
channel once :attr:`_sweep_interval` has elapsed since the previous
|
||||
sweep, whichever happens first. Consumers handle both shapes the
|
||||
same way: re-read the relevant rows on every wake-up.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
backend: SQLiteBackend,
|
||||
channels: list[str],
|
||||
sweep_interval: float,
|
||||
) -> None:
|
||||
self._backend = backend
|
||||
self._channels = list(channels)
|
||||
self._sweep_interval = sweep_interval
|
||||
self._queue: queue.Queue[Any] = queue.Queue()
|
||||
self._closed = False
|
||||
self._last_sweep = time.monotonic()
|
||||
if self._channels:
|
||||
backend._notify_register(self._channels, self._queue)
|
||||
|
||||
def poll(self, timeout: float) -> list[Notify]:
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
if self._closed:
|
||||
return []
|
||||
deadline = time.monotonic() + max(0.0, timeout)
|
||||
# Emit a synthetic-sweep tick on the first poll where the sweep
|
||||
# interval has elapsed. Single tick per channel per interval —
|
||||
# PG-equivalent "one wake-up per change" semantics, not a burst.
|
||||
now = time.monotonic()
|
||||
if self._channels and now - self._last_sweep >= self._sweep_interval:
|
||||
self._last_sweep = now
|
||||
for ch in self._channels:
|
||||
with contextlib.suppress(Exception):
|
||||
self._queue.put_nowait(Notify(channel=ch, payload="sweep", pid=0))
|
||||
out: list[Notify] = []
|
||||
try:
|
||||
while True:
|
||||
if self._closed:
|
||||
break
|
||||
if out:
|
||||
# Drain everything already queued without further
|
||||
# blocking — produces "one poll returns the burst"
|
||||
# semantics so the consumer reconciles once per wake.
|
||||
item = self._queue.get_nowait()
|
||||
else:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
item = self._queue.get(timeout=remaining)
|
||||
out.append(item)
|
||||
except queue.Empty:
|
||||
# End-of-drain: the blocking get hit its deadline OR a
|
||||
# get_nowait found the queue empty. Either way we return
|
||||
# whatever was already collected.
|
||||
pass
|
||||
return out
|
||||
|
||||
def close(self) -> None:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
if self._channels:
|
||||
self._backend._notify_unregister(self._channels, self._queue)
|
||||
|
||||
|
||||
class SQLiteBackend:
|
||||
"""SQLite implementation of the StorageBackend protocol."""
|
||||
|
||||
@@ -155,6 +245,15 @@ class SQLiteBackend:
|
||||
self._fts5_available = False
|
||||
self._db_unavailable = False
|
||||
self._db_unavailable_lock = threading.Lock()
|
||||
# In-process notify fan-out: channel name -> list of stream queues.
|
||||
# SQLite has no cross-process LISTEN/NOTIFY, so notifications are
|
||||
# delivered synchronously to any open ``listen`` stream in the same
|
||||
# process. Streams register on open + unregister on close; the
|
||||
# synthetic-sweep timer below covers consumers that need a periodic
|
||||
# wake regardless of producer activity (matching the PG-side
|
||||
# discovery-loop cadence).
|
||||
self._notify_lock = threading.Lock()
|
||||
self._notify_subs: dict[str, list[queue.Queue[Any]]] = {}
|
||||
if create_tables:
|
||||
self._init_schema()
|
||||
|
||||
@@ -2009,6 +2108,75 @@ class SQLiteBackend:
|
||||
conn.commit()
|
||||
return result.rowcount > 0
|
||||
|
||||
# -- Cross-process notifications -------------------------------------------
|
||||
|
||||
def notify(self, channel: str, payload: str = "") -> None:
|
||||
"""In-process broadcast — SQLite has no cross-process channel.
|
||||
|
||||
SQLite deployments are single-process by design (no shared backend
|
||||
across nodes); the storage layer delivers to any ``listen`` stream
|
||||
open in the same process. Cross-process consumers wouldn't be
|
||||
served regardless — the synthetic-sweep wake-up in :meth:`listen`
|
||||
is the parity fallback so consumer code stays backend-agnostic.
|
||||
"""
|
||||
from turnstone.core.storage._notify import Notify
|
||||
|
||||
with self._notify_lock:
|
||||
subs = list(self._notify_subs.get(channel, ()))
|
||||
for q in subs:
|
||||
with contextlib.suppress(Exception):
|
||||
q.put(Notify(channel=channel, payload=payload, pid=0))
|
||||
|
||||
@contextlib.contextmanager
|
||||
def listen(
|
||||
self,
|
||||
channels: Iterable[str],
|
||||
*,
|
||||
sweep_interval: float = _SQLITE_NOTIFY_SWEEP_INTERVAL,
|
||||
) -> Iterator[NotifyStream]:
|
||||
"""Subscribe to channels — synthetic-sweep + in-process fan-out.
|
||||
|
||||
The returned stream wakes every ``sweep_interval`` seconds with
|
||||
one ``Notify(channel, payload="sweep", pid=0)`` per subscribed
|
||||
channel; the default (:data:`_SQLITE_NOTIFY_SWEEP_INTERVAL`)
|
||||
suits a dev backstop with a 60 s consumer-side timer. Callers
|
||||
that need a tighter cadence (e.g. a future consumer without its
|
||||
own polling timer) pass a smaller value here. In-process
|
||||
:meth:`notify` calls deliver immediately on top of the sweep.
|
||||
Either path produces a wake-up; consumers reconcile by re-reading
|
||||
the relevant rows.
|
||||
|
||||
Channel names are de-duplicated so callers passing the same name
|
||||
twice don't double-deliver each notify to a single stream.
|
||||
"""
|
||||
# de-dupe + preserve insertion order — passing the same channel
|
||||
# twice would otherwise register the stream's queue against that
|
||||
# channel twice and deliver each notify multiple times.
|
||||
ch_list = list(dict.fromkeys(str(c) for c in channels if c))
|
||||
stream = _SQLiteNotifyStream(self, ch_list, sweep_interval=sweep_interval)
|
||||
try:
|
||||
yield stream
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
def _notify_register(self, channels: list[str], q: queue.Queue[Any]) -> None:
|
||||
"""Subscribe a stream's queue to in-process notifies on ``channels``."""
|
||||
with self._notify_lock:
|
||||
for ch in channels:
|
||||
self._notify_subs.setdefault(ch, []).append(q)
|
||||
|
||||
def _notify_unregister(self, channels: list[str], q: queue.Queue[Any]) -> None:
|
||||
"""Detach a stream's queue from in-process notifies on ``channels``."""
|
||||
with self._notify_lock:
|
||||
for ch in channels:
|
||||
subs = self._notify_subs.get(ch)
|
||||
if subs is None:
|
||||
continue
|
||||
with contextlib.suppress(ValueError):
|
||||
subs.remove(q)
|
||||
if not subs:
|
||||
self._notify_subs.pop(ch, None)
|
||||
|
||||
# -- Node metadata ---------------------------------------------------------
|
||||
|
||||
def get_node_metadata(self, node_id: str) -> list[dict[str, Any]]:
|
||||
@@ -4296,6 +4464,133 @@ class SQLiteBackend:
|
||||
conn.commit()
|
||||
return result.rowcount
|
||||
|
||||
# -- MCP pending-consent (Phase 9) ----------------------------------------
|
||||
|
||||
def upsert_mcp_pending_consent(
|
||||
self,
|
||||
user_id: str,
|
||||
server_name: str,
|
||||
error_code: str,
|
||||
scopes_required: str | None,
|
||||
last_ws_id: str | None,
|
||||
last_tool_call_id: str | None,
|
||||
now_iso: str,
|
||||
) -> None:
|
||||
from sqlalchemy.dialects import sqlite as sa_sqlite
|
||||
|
||||
stmt = sa_sqlite.insert(mcp_pending_consent).values(
|
||||
user_id=user_id,
|
||||
server_name=server_name,
|
||||
error_code=error_code,
|
||||
scopes_required=scopes_required,
|
||||
last_ws_id=last_ws_id,
|
||||
last_tool_call_id=last_tool_call_id,
|
||||
first_seen_at=now_iso,
|
||||
last_seen_at=now_iso,
|
||||
occurrence_count=1,
|
||||
)
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=["user_id", "server_name"],
|
||||
set_={
|
||||
"error_code": stmt.excluded.error_code,
|
||||
"scopes_required": stmt.excluded.scopes_required,
|
||||
"last_ws_id": stmt.excluded.last_ws_id,
|
||||
"last_tool_call_id": stmt.excluded.last_tool_call_id,
|
||||
"last_seen_at": stmt.excluded.last_seen_at,
|
||||
"occurrence_count": mcp_pending_consent.c.occurrence_count + 1,
|
||||
},
|
||||
)
|
||||
with self._conn() as conn:
|
||||
conn.execute(stmt)
|
||||
conn.commit()
|
||||
|
||||
def list_mcp_pending_consent_by_user(self, user_id: str) -> list[MCPPendingConsentRow]:
|
||||
with self._conn() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(mcp_pending_consent)
|
||||
.where(mcp_pending_consent.c.user_id == user_id)
|
||||
.order_by(mcp_pending_consent.c.last_seen_at.desc())
|
||||
).fetchall()
|
||||
out: list[MCPPendingConsentRow] = []
|
||||
for r in rows:
|
||||
m = r._mapping
|
||||
out.append(
|
||||
MCPPendingConsentRow(
|
||||
user_id=m["user_id"],
|
||||
server_name=m["server_name"],
|
||||
error_code=m["error_code"],
|
||||
scopes_required=m["scopes_required"],
|
||||
last_ws_id=m["last_ws_id"],
|
||||
last_tool_call_id=m["last_tool_call_id"],
|
||||
first_seen_at=m["first_seen_at"],
|
||||
last_seen_at=m["last_seen_at"],
|
||||
occurrence_count=m["occurrence_count"],
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
def delete_mcp_pending_consent(self, user_id: str, server_name: str) -> bool:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.delete(mcp_pending_consent).where(
|
||||
(mcp_pending_consent.c.user_id == user_id)
|
||||
& (mcp_pending_consent.c.server_name == server_name)
|
||||
)
|
||||
)
|
||||
conn.commit()
|
||||
return bool(result.rowcount)
|
||||
|
||||
def delete_all_mcp_pending_consent_by_user(self, user_id: str) -> int:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.delete(mcp_pending_consent).where(mcp_pending_consent.c.user_id == user_id)
|
||||
)
|
||||
conn.commit()
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
def count_mcp_consented_users_by_server(self, server_name: str) -> int:
|
||||
now_iso = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.select(sa.func.count(sa.distinct(mcp_user_tokens.c.user_id)))
|
||||
.where(mcp_user_tokens.c.server_name == server_name)
|
||||
.where(
|
||||
sa.or_(
|
||||
mcp_user_tokens.c.expires_at.is_(None),
|
||||
mcp_user_tokens.c.expires_at > now_iso,
|
||||
)
|
||||
)
|
||||
).scalar()
|
||||
return int(result or 0)
|
||||
|
||||
def count_mcp_consented_users_grouped_by_server(self) -> dict[str, int]:
|
||||
now_iso = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
|
||||
with self._conn() as conn:
|
||||
rows = conn.execute(
|
||||
sa.select(
|
||||
mcp_user_tokens.c.server_name,
|
||||
sa.func.count(sa.distinct(mcp_user_tokens.c.user_id)),
|
||||
)
|
||||
.where(
|
||||
sa.or_(
|
||||
mcp_user_tokens.c.expires_at.is_(None),
|
||||
mcp_user_tokens.c.expires_at > now_iso,
|
||||
)
|
||||
)
|
||||
.group_by(mcp_user_tokens.c.server_name)
|
||||
).fetchall()
|
||||
return {row[0]: int(row[1] or 0) for row in rows}
|
||||
|
||||
def any_oauth_user_mcp_servers(self) -> bool:
|
||||
with self._conn() as conn:
|
||||
result = conn.execute(
|
||||
sa.select(sa.literal(1))
|
||||
.select_from(mcp_servers)
|
||||
.where(mcp_servers.c.auth_type == "oauth_user")
|
||||
.limit(1)
|
||||
).scalar()
|
||||
return result is not None
|
||||
|
||||
# -- Model definitions -----------------------------------------------------
|
||||
|
||||
def create_model_definition(
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Trigger ``pg_notify('services', ...)`` on service registry changes.
|
||||
|
||||
The console-side :class:`NotifyDispatcher` (`turnstone/console/notify_dispatcher.py`)
|
||||
holds a dedicated LISTEN connection and fans channel events out to handlers.
|
||||
This migration installs the producer side for the ``services`` channel —
|
||||
the cluster collector subscribes so new-node discovery is reactive instead
|
||||
of polling every 60 s.
|
||||
|
||||
The trigger filters heartbeat-only UPDATEs in-trigger (same url + same
|
||||
metadata, only ``last_heartbeat`` changed): ``register_service`` is an
|
||||
UPSERT, so a node restart that changes url/metadata still fires; a plain
|
||||
heartbeat tick stays quiet to avoid flooding the channel on every
|
||||
30 s × N-nodes cluster tick. Channel payload is a small JSON object —
|
||||
service_type, service_id, op — well below PG's 8 KiB NOTIFY limit; the
|
||||
handler reconciles by re-reading ``services`` rather than relying on
|
||||
the payload content.
|
||||
|
||||
SQLite is a no-op for this migration — the SQLite backend's in-process
|
||||
:meth:`notify` doesn't go through a trigger, and the synthetic-sweep
|
||||
fallback in :meth:`listen` covers consumer parity.
|
||||
|
||||
What this trigger does NOT cover: crashed-node detection. A node that
|
||||
dies without running its deregister handshake leaves a stale row that
|
||||
ages out via the existing 120 s heartbeat-expiry filter. The 60 s
|
||||
discovery loop in the collector keeps running as the backstop for
|
||||
crash-shaped node loss.
|
||||
|
||||
Revision ID: 053
|
||||
Revises: 052
|
||||
Create Date: 2026-05-10
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
from turnstone.core.storage._schema import (
|
||||
SERVICES_NOTIFY_TRIGGER_FN_NAME,
|
||||
SERVICES_NOTIFY_TRIGGER_FN_SQL,
|
||||
SERVICES_NOTIFY_TRIGGER_NAME,
|
||||
SERVICES_NOTIFY_TRIGGER_SQL,
|
||||
)
|
||||
|
||||
revision = "053"
|
||||
down_revision = "052"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name != "postgresql":
|
||||
return
|
||||
op.execute(sa.text(SERVICES_NOTIFY_TRIGGER_FN_SQL))
|
||||
op.execute(sa.text(SERVICES_NOTIFY_TRIGGER_SQL))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if bind.dialect.name != "postgresql":
|
||||
return
|
||||
op.execute(sa.text(f"DROP TRIGGER IF EXISTS {SERVICES_NOTIFY_TRIGGER_NAME} ON services"))
|
||||
op.execute(sa.text(f"DROP FUNCTION IF EXISTS {SERVICES_NOTIFY_TRIGGER_FN_NAME}()"))
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Add mcp_pending_consent table.
|
||||
|
||||
Stores per-(user, server) deferred-consent records emitted by the pool
|
||||
dispatchers when a non-interactive run (scheduled / channel) hits
|
||||
``mcp_consent_required`` or ``mcp_insufficient_scope``. Read on
|
||||
dashboard load to render the "N MCP servers need consent" badge; cleared
|
||||
by the OAuth callback handler when consent completes.
|
||||
|
||||
Composite PK ``(user_id, server_name)`` collapses repeat occurrences for
|
||||
the same server into one row. No FKs (matches the rest of the
|
||||
oauth_user schema in migration 049).
|
||||
|
||||
Revision ID: 054
|
||||
Revises: 053
|
||||
Create Date: 2026-05-11
|
||||
"""
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "054"
|
||||
down_revision = "053"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"mcp_pending_consent",
|
||||
sa.Column("user_id", sa.Text, nullable=False),
|
||||
sa.Column("server_name", sa.Text, nullable=False),
|
||||
sa.Column("error_code", sa.Text, nullable=False),
|
||||
sa.Column("scopes_required", sa.Text, nullable=True),
|
||||
sa.Column("last_ws_id", sa.Text, nullable=True),
|
||||
sa.Column("last_tool_call_id", sa.Text, nullable=True),
|
||||
sa.Column("first_seen_at", sa.Text, nullable=False),
|
||||
sa.Column("last_seen_at", sa.Text, nullable=False),
|
||||
sa.Column("occurrence_count", sa.Integer, nullable=False, server_default="1"),
|
||||
sa.PrimaryKeyConstraint("user_id", "server_name"),
|
||||
)
|
||||
op.create_index(
|
||||
"idx_mcp_pending_consent_user",
|
||||
"mcp_pending_consent",
|
||||
["user_id"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("idx_mcp_pending_consent_user", table_name="mcp_pending_consent")
|
||||
op.drop_table("mcp_pending_consent")
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Index mcp_user_tokens by (server_name, expires_at).
|
||||
|
||||
Phase 9 admin pill (``count_mcp_consented_users_*``) filters
|
||||
``mcp_user_tokens`` by ``server_name`` and ``expires_at``. The table's
|
||||
only existing index is the composite PK ``(user_id, server_name)`` —
|
||||
``user_id`` is the leading column, so a filter on ``server_name`` alone
|
||||
must full-scan the table. The bulk ``GROUP BY server_name`` variant in
|
||||
the admin list handler benefits from the same index.
|
||||
|
||||
PostgreSQL uses ``CREATE INDEX CONCURRENTLY`` inside an
|
||||
``autocommit_block`` so the build is non-blocking on a live system —
|
||||
``mcp_user_tokens`` is on the token-refresh hot path and an
|
||||
ACCESS EXCLUSIVE lock during build would stall refresh writers on
|
||||
installs with non-trivial row counts. SQLite has no concurrent build
|
||||
concept and the table-level write lock already serializes, so a plain
|
||||
``op.create_index`` is fine. Pattern mirrors migration 048
|
||||
(``idx_workstreams_reaper``).
|
||||
|
||||
Revision ID: 055
|
||||
Revises: 054
|
||||
Create Date: 2026-05-11
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision = "055"
|
||||
down_revision = "054"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
dialect = bind.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute(
|
||||
"CREATE INDEX CONCURRENTLY IF NOT EXISTS "
|
||||
"idx_mcp_user_tokens_server ON mcp_user_tokens "
|
||||
"(server_name, expires_at)"
|
||||
)
|
||||
else:
|
||||
op.create_index(
|
||||
"idx_mcp_user_tokens_server",
|
||||
"mcp_user_tokens",
|
||||
["server_name", "expires_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
dialect = bind.dialect.name
|
||||
|
||||
if dialect == "postgresql":
|
||||
with op.get_context().autocommit_block():
|
||||
op.execute("DROP INDEX CONCURRENTLY IF EXISTS idx_mcp_user_tokens_server")
|
||||
else:
|
||||
op.drop_index("idx_mcp_user_tokens_server", table_name="mcp_user_tokens")
|
||||
@@ -37,6 +37,19 @@ class ClientType(enum.StrEnum):
|
||||
WEB = "web"
|
||||
CLI = "cli"
|
||||
CHAT = "chat"
|
||||
SCHEDULED = "scheduled"
|
||||
|
||||
|
||||
# Subset of ``ClientType`` values where the user is present to complete
|
||||
# an in-flight OAuth consent flow (browser redirect + return). CHAT and
|
||||
# SCHEDULED users cannot drive a browser redirect from inside their
|
||||
# delivery surface, so consent-required errors must be persisted to
|
||||
# ``mcp_pending_consent`` for later surfacing rather than relying on the
|
||||
# in-flight SSE rendering path. Used by ``ChatSession`` to set
|
||||
# ``_is_interactive_for_consent`` at construction time.
|
||||
INTERACTIVE_CONSENT_CLIENT_TYPES: frozenset[ClientType] = frozenset(
|
||||
{ClientType.WEB, ClientType.CLI}
|
||||
)
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
@@ -56,6 +69,7 @@ _ENV_MAP: dict[ClientType, str] = {
|
||||
ClientType.WEB: "env/web.md",
|
||||
ClientType.CLI: "env/cli.md",
|
||||
ClientType.CHAT: "env/chat.md",
|
||||
ClientType.SCHEDULED: "env/scheduled.md",
|
||||
}
|
||||
|
||||
|
||||
|
||||
Vendored
+20
@@ -0,0 +1,20 @@
|
||||
## Output Environment
|
||||
|
||||
Your response is generated by a scheduled or autonomous run — no human is watching the output as it streams. The result is delivered to the user later (Discord notification, dashboard badge, or persisted workstream history) where they will see it as a static block of markdown.
|
||||
|
||||
**Implications:**
|
||||
|
||||
- The user is not online to answer mid-task clarifying questions. Make the reasonable judgment and continue; mention the assumption you made in the final output so the user can correct course on the next run if needed.
|
||||
- Tool calls that require interactive user consent (e.g., MCP servers gated on OAuth user authorization that the user has not yet completed) will return a deferred-consent error rather than block the run. Surface the deferred work in your final summary so the user knows what was skipped.
|
||||
- Optimize for a clear, scannable final summary over conversational back-and-forth — the user reads the whole transcript at once, not turn-by-turn.
|
||||
|
||||
**Available rendering:**
|
||||
|
||||
- Standard GitHub-flavored markdown is supported in the dashboard surface. Discord delivery uses the same constraints as `chat.md` (no tables, no headings beyond bold text, no Mermaid/KaTeX).
|
||||
- Default to chat-portable formatting (bold/italic/inline-code/bullets/code-blocks) unless you know the destination is the web dashboard.
|
||||
|
||||
**Formatting principles:**
|
||||
|
||||
- Lead with the outcome in one line: what was accomplished, what was skipped, and why.
|
||||
- For multi-step work, end with a concise checklist of what ran and what remains.
|
||||
- Cite specific identifiers (workstream IDs, tool names, MCP server names) so the user can resume the work without re-reading the trace.
|
||||
+39
-3
@@ -1652,11 +1652,15 @@ async def command(request: Request) -> JSONResponse:
|
||||
if history:
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
elif cmd_word in ("/rewind", "/retry"):
|
||||
# Refresh frontend with truncated history
|
||||
# Refresh frontend with truncated history. Always emit the
|
||||
# history event even when empty: editing the first message
|
||||
# rewinds to zero messages, and the frontend dispatches the
|
||||
# queued edit-and-resend from the history handler — skipping
|
||||
# the event on an empty list orphans `_pendingEditSend` and
|
||||
# leaves the composer stuck in busy.
|
||||
ui._enqueue({"type": "clear_ui"})
|
||||
history = await asyncio.to_thread(_build_history, ws.session)
|
||||
if history:
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
ui._enqueue({"type": "history", "messages": history})
|
||||
# Audit trail
|
||||
storage = getattr(request.app.state, "auth_storage", None)
|
||||
if storage:
|
||||
@@ -2818,6 +2822,27 @@ async def mcp_oauth_revoke_connection(request: Request) -> Response:
|
||||
return await handle_mcp_oauth_revoke_connection(request)
|
||||
|
||||
|
||||
async def mcp_oauth_list_pending(request: Request) -> Response:
|
||||
"""GET /v1/api/mcp/oauth/pending — list deferred-consent records (Phase 9)."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_list_pending
|
||||
|
||||
return await handle_mcp_oauth_list_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending/{server_name} — dismiss a deferred-consent record."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_pending(request)
|
||||
|
||||
|
||||
async def mcp_oauth_clear_all_pending(request: Request) -> Response:
|
||||
"""DELETE /v1/api/mcp/oauth/pending — bulk-dismiss deferred-consent records."""
|
||||
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_all_pending
|
||||
|
||||
return await handle_mcp_oauth_clear_all_pending(request)
|
||||
|
||||
|
||||
def list_interface_settings(request: Request) -> JSONResponse:
|
||||
"""GET /v1/api/admin/settings — return interface settings from ConfigStore.
|
||||
|
||||
@@ -4060,6 +4085,17 @@ def create_app(
|
||||
mcp_oauth_revoke_connection,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/mcp/oauth/pending", mcp_oauth_list_pending),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending",
|
||||
mcp_oauth_clear_all_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route(
|
||||
"/api/mcp/oauth/pending/{server_name}",
|
||||
mcp_oauth_clear_pending,
|
||||
methods=["DELETE"],
|
||||
),
|
||||
Route("/api/admin/settings", list_interface_settings),
|
||||
Route(
|
||||
"/api/admin/settings/{key:path}",
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user